# Probabilistic Programming and Bayesian Methods for Hackers Chapter 1

    Run in Google Colab

    View source on GitHub

Original content ([this Jupyter notebook](https://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter1_Introduction/Ch1_Introduction_PyMC2.ipynb)) created by Cam Davidson-Pilon ([`@Cmrn_DP`](https://twitter.com/Cmrn_DP))

Ported to [Tensorflow Probability](https://www.tensorflow.org/probability/) by Matthew McAteer ([`@MatthewMcAteer0`](https://twitter.com/MatthewMcAteer0)) and Bryan Seybold, with help from the TFP team at  Google ([`tfprobability@tensorflow.org`](mailto:tfprobability@tensorflow.org)).

Welcome to Bayesian Methods for Hackers. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

---
### Table of Contents
- Dependencies & Prerequisites
- The Philosophy of Bayesian Inference
- The Bayesian state of mind
- Bayesian Inference in Practice
- Are frequentist methods incorrect then?
- Our Bayesian framework
- Example: Mandatory coin-flip example
- Example: Bug, or just sweet, unintended feature?
- Probability Distributions
  - Discrete Case
- Continuous Case
- But what is $\lambda \;$?
  - Example: Inferring behaviour from text-message data
- Introducing our first hammer: Tensorflow Probability
- specify the joint log-density
- Specify the posterior sampler
- Execute the TF graph to sample from the posterior
- Plot the Results
- Interpretation
- Exercises
- References

### Dependencies & Prerequisites

    Tensorflow Probability is part of the colab default runtime, so you don't need to install Tensorflow or Tensorflow Probability if you're running this in the colab.

    If you're running this notebook in Jupyter on your own machine (and you have already installed Tensorflow), you can use the following

 For the most recent nightly installation: pip3 install -q tfp-nightly

 For the most recent stable TFP release: pip3 install -q --upgrade tensorflow-probability

 For the most recent stable GPU-connected version of TFP: pip3 install -q --upgrade tensorflow-probability-gpu

 For the most recent nightly GPU-connected version of TFP: pip3 install -q tfp-nightly-gpu

Again, if you are running this in a Colab, Tensorflow and TFP are already installed

```
#@title Imports and Global Variables (make sure to run this cell)  { display-mode: "form" }
from __future__ import absolute_import, division, print_function

#@markdown This sets the warning status (default is `ignore`, since this notebook runs correctly)
warning_status = "ignore" #@param ["ignore", "always", "module", "once", "default", "error"]
import warnings
warnings.filterwarnings(warning_status)
with warnings.catch_warnings():
    warnings.filterwarnings(warning_status, category=DeprecationWarning)
    warnings.filterwarnings(warning_status, category=UserWarning)

import numpy as np
import os
#@markdown This sets the styles of the plotting (default is styled like plots from [FiveThirtyeight.com](https://fivethirtyeight.com/))
matplotlib_style = 'fivethirtyeight' #@param ['fivethirtyeight', 'bmh', 'ggplot', 'seaborn', 'default', 'Solarize_Light2', 'classic', 'dark_background', 'seaborn-colorblind', 'seaborn-notebook']
import matplotlib.pyplot as plt; plt.style.use(matplotlib_style)
import matplotlib.axes as axes;
from matplotlib.patches import Ellipse
%matplotlib inline
import seaborn as sns; sns.set_context('notebook')
from IPython.core.pylabtools import figsize
#@markdown This sets the resolution of the plot outputs (`retina` is the highest resolution)
notebook_screen_res = 'retina' #@param ['retina', 'png', 'jpeg', 'svg', 'pdf']
%config InlineBackend.figure_format = notebook_screen_res

import tensorflow as tf
tfe = tf.contrib.eager

# Eager Execution
#@markdown Check the box below if you want to use [Eager Execution](https://www.tensorflow.org/guide/eager)
#@markdown Eager execution provides An intuitive interface, Easier debugging, and a control flow comparable to Numpy. You can read more about it on the [Google AI Blog](https://ai.googleblog.com/2017/10/eager-execution-imperative-define-by.html)
use_tf_eager = False #@param {type:"boolean"}

# Use try/except so we can easily re-execute the whole notebook.
if use_tf_eager:
    try:
        tf.enable_eager_execution()
    except:
        pass

import tensorflow_probability as tfp
tfd = tfp.distributions
tfb = tfp.bijectors

def evaluate(tensors):
    """Evaluates Tensor or EagerTensor to Numpy `ndarray`s.
    Args:
    tensors: Object of `Tensor` or EagerTensor`s; can be `list`, `tuple`,
      `namedtuple` or combinations thereof.

    Returns:
      ndarrays: Object with same structure as `tensors` except with `Tensor` or
        `EagerTensor`s replaced by Numpy `ndarray`s.
    """
    if tf.executing_eagerly():
        return tf.contrib.framework.nest.pack_sequence_as(
            tensors,
            [t.numpy() if tf.contrib.framework.is_tensor(t) else t
             for t in tf.contrib.framework.nest.flatten(tensors)])
    return sess.run(tensors)

class _TFColor(object):
    """Enum of colors used in TF docs."""
    red = '#F15854'
    blue = '#5DA5DA'
    orange = '#FAA43A'
    green = '#60BD68'
    pink = '#F17CB0'
    brown = '#B2912F'
    purple = '#B276B2'
    yellow = '#DECF3F'
    gray = '#4D4D4D'
    def __getitem__(self, i):
        return [
            self.red,
            self.orange,
            self.green,
            self.blue,
            self.pink,
            self.brown,
            self.purple,
            self.yellow,
            self.gray,
        ][i % 9]
TFColor = _TFColor()

def session_options(enable_gpu_ram_resizing=True, enable_xla=False):
    """
    Allowing the notebook to make use of GPUs if they're available.

    XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear
    algebra that optimizes TensorFlow computations.
    """
    config = tf.ConfigProto()
    config.log_device_placement = True
    if enable_gpu_ram_resizing:
        # `allow_growth=True` makes it possible to connect multiple colabs to your
        # GPU. Otherwise the colab malloc's all GPU ram.
        config.gpu_options.allow_growth = True
    if enable_xla:
        # Enable on XLA. https://www.tensorflow.org/performance/xla/.
        config.graph_options.optimizer_options.global_jit_level = (
            tf.OptimizerOptions.ON_1)
    return config

def reset_sess(config=None):
    """
    Convenience function to create the TF graph & session or reset them.
    """
    if config is None:
        config = session_options(enable_gpu_ram_resizing=True, enable_xla=False)
    global sess
    tf.reset_default_graph()
    try:
        sess.close()
    except:
        pass
    sess = tf.InteractiveSession(config=config)

reset_sess()
```

## The Philosophy of Bayesian Inference

>You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, even more difficult, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more confident about our code, but still not certain. Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

## The Bayesian state of mind
Bayesian inference differs from more traditional statistical inference by preserving uncertainty. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving certainty from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of believability in an event, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: Frequentist, known as the more classical version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title). For example, the probability of plane accidents under a frequentist philosophy is interpreted as the long-term frequency of plane accidents. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of belief, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate A will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an individual, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different information about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

* I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is your belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

* Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

* A medical patient is exhibiting symptoms *x*, *y* and *z*. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be trained to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the prior probability.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A|X)$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the posterior probability so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1. $P(A)$: the coin has a 50 percent chance of being Heads. $P(A|X)$: You look at the coin, observe a Heads has landed, denote this information  $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2. $P(A)$: This big, complex code likely has a bug in it. $P(A|X)$: The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3. $P(A)$: The patient could have any number of diseases. $P(A|X)$: Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we re-weighted the prior to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes less wrong. This is the alternative side of the prediction coin, where typically we try to be more right.

## Bayesian Inference in Practice
If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return probabilities.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a YES. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of YES and NO. The function might return:

>YES, with probability 0.8; NO, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument: "Often my code has bugs". This parameter is the prior. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

### Incorporating evidence
As we acquire more and more instances of evidence, our prior belief is washed out by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an infinite amount of evidence, say as $N→∞,$ our Bayesian results (often) align with frequentist results. Hence for large N, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more unstable: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we preserve the uncertainty that reflects the instability of statistical inference of a small N dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[[1]](#scrollTo=nDdph0r1ABCn), before making such a decision:

Sample sizes are never large. If $N$, is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$, is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$, is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

## Are frequentist methods incorrect then?
No.

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [[2]](#scrollTo=nDdph0r1ABCn)[[3]](#scrollTo=nDdph0r1ABCn). Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?")

The much more difficult analytic problems involve medium data and, especially troublesome, really small data. Using a similar argument as Gelman's above, if big data problems are big enough to be readily solved, then we should be more interested in the not-quite-big enough datasets.

## Our Bayesian framework
We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a prior belief in event A, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes X tests, we want to update our belief to incorporate this. We call this new belief the posterior probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

$$ P(A|X) = \frac{P(X | A) P(A) }{P(X) } $$

$$ P(A|X) \propto{P(X | A) P(A) } $$

NOTE: ($\propto$ is "proportional to")

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A|X)$.

## Example: Mandatory coin-flip example
Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it p, but have no prior opinion on what p might be.

We begin to flip a coin, and record the observations: either H or T. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips), while also demonstrating some of the best practices when it comes to evaluating tensors and plotting the data. First, the easy part: We define the values in our Tensorflow graph

```
# Build Graph
rv_coin_flip_prior = tfp.distributions.Bernoulli(probs=0.5, dtype=tf.int32)

num_trials = tf.constant([0,1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000])

coin_flip_data = rv_coin_flip_prior.sample(num_trials[-1])

# prepend a 0 onto tally of heads and tails, for zeroth flip
coin_flip_data = tf.pad(coin_flip_data,tf.constant([[1, 0,]]),"CONSTANT")

# compute cumulative headcounts from 0 to 2000 flips, and then grab them at each of num_trials intervals
cumulative_headcounts = tf.gather(tf.cumsum(coin_flip_data), num_trials)

rv_observed_heads = tfp.distributions.Beta(
    concentration1=tf.cast(1 + cumulative_headcounts, tf.float32),
    concentration0=tf.cast(1 + num_trials - cumulative_headcounts, tf.float32))

probs_of_heads = tf.linspace(start=0., stop=1., num=100, name="linspace")
observed_probs_heads = tf.transpose(rv_observed_heads.prob(probs_of_heads[:, tf.newaxis]))
```

Next we move onto executing the graph. When it comes to calculations that need to be made frequently and repeatedly, this method of first-defining and then executing graphs provides a handy speed boost. We can actually use a custom `evaluate()` function that allows us to evaluate tensors whether we are operating in TF Graph mode, or whether we have Eager mode active. The function looks like the following:

```python

def evaluate(tensors):
    """Evaluates Tensor or EagerTensor to Numpy `ndarray`s.
    Args:
    tensors: Object of `Tensor` or EagerTensor`s; can be `list`, `tuple`,
      `namedtuple` or combinations thereof.

    Returns:
      ndarrays: Object with same structure as `tensors` except with `Tensor` or
        `EagerTensor`s replaced by Numpy `ndarray`s.
    """
    if tf.executing_eagerly():
        return tf.contrib.framework.nest.pack_sequence_as(
            tensors,
            [t.numpy() if tf.contrib.framework.is_tensor(t) else t
             for t in tf.contrib.framework.nest.flatten(tensors)])
    return sess.run(tensors)

```

To plot the tensors, we need to convert them into numpy variables. One handy way of associating tensors with their corrresponding numpy variables is to append an underscore to the numpy-like arrays. For example, if the input to `evaluate()` is `variable`, then we assign that value to `variable_`. Below we see an example of how we use both `evaluate()`  and this new styling.

```
# Execute graph
[num_trials_,
probs_of_heads_,
observed_probs_heads_,
cumulative_headcounts_,
] = evaluate([
  num_trials,
  probs_of_heads,
  observed_probs_heads,
  cumulative_headcounts
])
```

Finally, we move onto plotting our evaluated tensors in matplotlib.

```
# For the already prepared, I'm using Binomial's conj. prior.
plt.figure(figsize(16, 9))
for i in range(len(num_trials_)):
    sx = plt.subplot(len(num_trials_)/2, 2, i+1)
    plt.xlabel("$p$, probability of heads") \
    if i in [0, len(num_trials_)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.plot(probs_of_heads_, observed_probs_heads_[i],
             label="observe %d tosses,\n %d heads" % (num_trials_[i], cumulative_headcounts_[i]))
    plt.fill_between(probs_of_heads_, 0, observed_probs_heads_[i],
                     color=TFColor[3], alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities", y=1.02,
             fontsize=14)
plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always peaked at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what p is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased away from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

## Example: Bug, or just sweet, unintended feature?
Let $A$ denote the event that our code has no bugs in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A)=p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X|A)$, i.e., the probability that the code passes $X$ tests given there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event X occurring even though our code indeed has bugs (denoted $∼A$, spoken not $A$), or event $X$ without bugs $(A)$. $ P(X)$ can be represented as:

$$ \begin{align*}
P(A|X) &= \frac{P(X | A) P(A) }{P(X) } \\
 P(X) &= P(X \text{ and } A) + P(X \text{ and } \sim A) \\
  &= P(X|A)P(A) + P(X | \sim A)P(\sim A) \\
  &= P(X|A)p + P(X | \sim A)(1-p) \end{align*} $$

We have already computed $P(X|A)$ above. On the other hand, $P(X|\sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A)=0.5$. Then:

$$ \begin{align*}
P(A | X) &= \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\
&= \frac{ 2 p}{1+p} \end{align*} $$

This is the posterior probability. What does it look like as a function of our prior, $p\in[0,1]$?

```
# Defining our range of probabilities
p = tf.linspace(start=0., stop=1., num=50)

# Convert from TF to numpy.
[p_] = evaluate([p])

# Visualization.
plt.figure(figsize=(12.5, 6))
plt.plot(p_, 2*p_/(1+p_), color=TFColor[3], lw=3)
#plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c=TFColor[3])
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel(r"Prior, $P(A) = p$")
plt.ylabel(r"Posterior, $P(A|X)$, with $P(A) = p$")
plt.title(r"Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there are no bugs, so $1 \text{-} p$ is the prior probability that there are bugs.

Similarly, our posterior is also a probability, with $P(A|X)$ the probability there is no bug given we saw all tests pass, hence $1 \text{-} P(A|X)$ is the probability there is a bug given all tests passed. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```
# Defining our priors and posteriors
prior = tf.constant([0.20, 0.80])
posterior = tf.constant([1./3, 2./3])

# Convert from TF to numpy.
[
    prior_,
    posterior_,
] = evaluate([
    prior,
    posterior,
])

# Our Simple Visualization
plt.figure(figsize=(12.5, 4))
colours = [TFColor[0], TFColor[3]]
plt.bar([0, .7], prior_, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])
plt.bar([0+0.25, .7+0.25], posterior_, alpha=0.7,
        width=0.25, color=colours[1],
        label=r"posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title(r"Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with probability distributions. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

## Probability Distributions
Let's quickly recall what a probability distribution is: Let $Z$ be some random variable. Then associated with $Z$ is a probability distribution function that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

*  $Z$ is discrete: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

* $Z$ is continuous: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

* $Z$ is mixed: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case

If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```
# Build graph.
x = tf.range (start=0., limit=16.)
lambdas = tf.constant([1.5, 4.25])

poi_pmf = tfd.Poisson(
  rate=lambdas[:, tf.newaxis]).prob(x)

# Execute graph
[
  x_,
  lambdas_,
  poi_pmf_,
] = evaluate([
  x,
  lambdas,
  poi_pmf,
])

plt.figure(figsize=(12.5, 8))

# Display results in two different histograms, for easier comparison
colours = [TFColor[0], TFColor[3]]
for i in [0,1]:
  ax = plt.subplot(2,1,i+1)
  ax.set_autoscaley_on(False)
  plt.title("Probability mass function of a Poisson random variable");

  plt.bar(x_,
          poi_pmf_[i],
          color=colours[i],
          label=r"$\lambda = %.1f$" % lambdas_[i], alpha=0.60,
          edgecolor=colours[i], lw="3")
  plt.xticks(x_ + 0.4, x_)
  plt.ylim([0, .5])
  plt.legend()
  plt.ylabel(r"probability of $k$")
  plt.xlabel(r"$k$")
```

### Continuous Case

Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```
# Defining our Data and assumptions (use tf.linspace for continuous)
a = tf.range(start=0., limit=4., delta=0.04)
a = a[..., tf.newaxis]
lambdas = tf.constant([0.5, 1.])

# Now we use TFP to compute probabilities in a vectorized manner.
expo_pdf = tfd.Exponential(rate=lambdas).prob(a)

# Convert from TF to numpy
[
    a_,
    lambdas_,
    expo_pdf_,
] = evaluate([
    a,
    lambdas,
    expo_pdf,
])

# Visualizing our results
plt.figure(figsize=(12.5, 4))
for i in range(lambdas_.size):
    plt.plot(a_.T[0], expo_pdf_.T[[i]][0],
             lw=3, color=TFColor[i], label=r"$\lambda = %.1f$" % lambdas_[i])
    plt.fill_between(a_.T[0], expo_pdf_.T[[i]][0],
                         color=TFColor[i], alpha=.33)
plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title(r"Probability density function of an Exponential random variable; differing $\lambda$");

```

## But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

#### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```
# Defining our Data and assumptions
count_data = tf.constant([
    13,  24,   8,  24,   7,  35,  14,  11,  15,  11,  22,  22,  11,  57,
    11,  19,  29,   6,  19,  12,  22,  12,  18,  72,  32,   9,   7,  13,
    19,  23,  27,  20,   6,  17,  13,  10,  14,   6,  16,  15,   7,   2,
    15,  15,  19,  70,  49,   7,  53,  22,  21,  31,  19,  11,  18,  20,
    12,  35,  17,  23,  17,   4,   2,  31,  30,  13,  27,   0,  39,  37,
    5,  14,  13,  22,
], dtype=tf.float32)
n_count_data = tf.shape(count_data)
days = tf.range(n_count_data[0])

# Convert from TF to numpy.

[
    count_data_,
    n_count_data_,
    days_,
] = evaluate([
    count_data,
    n_count_data,
    days,
])

# Visualizing the Results

plt.figure(figsize=(12.5, 4))
plt.bar(days_, count_data_, color="#5DA5DA")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data_[0]);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$\lambda =
\begin{cases} \lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.
$$
\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}
$$
$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying
$$
\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}
$$
So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to [TensorFlow Probability](https://tensorflow.org/probability), a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

## Introducing our first hammer: TensorFlow Probability

TensorFlow Probability (TFP) is a Python library for programming Bayesian analysis. It is intended for data scientists, statisticians, machine learning practitioners, and scientists. Since it is built on the TensorFlow (TF) stack, it brings the runtime benefits of TF to Bayesian analysis. These include write-once run-many (ability to run your development model in production) and speedups via state-of-the-art hardware (GPUs and TPUs).

Since TFP is relatively new, the TFP community is actively developing documentation,
especially docs and examples that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why TFP is so cool.

We will model the problem above using TFP. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components.

B. Cronin [[4]](#scrollTo=nDdph0r1ABCn) has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

TFP code is easy to read. The only novel thing should be the syntax. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables.

## Specify the joint log-density

We'll assume the data is a consequence of the following generative model:

$$\begin{align*}
\lambda_{1}^{(0)} &\sim \text{Exponential}(\text{rate}=\alpha) \\
\lambda_{2}^{(0)} &\sim \text{Exponential}(\text{rate}=\alpha) \\
\tau &\sim \text{Uniform}[\text{low}=0,\text{high}=1) \\
\text{for }  i &= 1\ldots N: \\
\lambda_i &= \begin{cases} \lambda_{1}^{(0)}, & \tau > i/N \\ \lambda_{2}^{(0)}, & \text{otherwise}\end{cases}\\
 X_i &\sim \text{Poisson}(\text{rate}=\lambda_i)
\end{align*}$$

Happily, this model can be easily implemented using TF and TFP's distributions:

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. The [gather](https://https://www.tensorflow.org/api_docs/python/tf/gather) function assigns `lambda_1` or `lambda_2` as the value of `lambda_`, depending on what side of `tau` we are on. The values of `lambda_` up until `tau` are `lambda_1` and the values afterwards are `lambda_2`.

Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

TFP performs probabilistic inference by evaluating the model parameters using a joint_log_prob function, which we'll describe more in Chapter 2.

```
def joint_log_prob(count_data, lambda_1, lambda_2, tau):
    tfd = tfp.distributions

    alpha = np.array(1. / count_data.mean(), np.float32)
    rv_lambda_1 = tfd.Exponential(rate=alpha)
    rv_lambda_2 = tfd.Exponential(rate=alpha)

    rv_tau = tfd.Uniform()

    lambda_ = tf.gather(
         [lambda_1, lambda_2],
         indices=tf.to_int32(tau * count_data.size <= np.arange(count_data.size)))
    rv_observation = tfd.Poisson(rate=lambda_)

    return (
         rv_lambda_1.log_prob(lambda_1)
         + rv_lambda_2.log_prob(lambda_2)
         + rv_tau.log_prob(tau)
         + tf.reduce_sum(rv_observation.log_prob(count_data))
    )
```

Notice that the implementation is arguably very close to being a 1:1 translation of the mathematical model. The main difference is merely that once we've specified the probabilistic model, we return the sum of the log_probs.

## Specify the posterior sampler

The code below will be explained in Chapter 3, but we show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which we also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```
# Set the chain's start state.
initial_chain_state = [
    tf.cast(tf.reduce_mean(count_data), tf.float32) * tf.ones([], dtype=tf.float32, name="init_lambda1"),
    tf.cast(tf.reduce_mean(count_data), tf.float32) * tf.ones([], dtype=tf.float32, name="init_lambda2", tf.float32),
    0.5 * tf.ones([], dtype=tf.float32, name="init_tau"),
]

# Since HMC operates over unconstrained space, we need to transform the
# samples so they live in real-space.
unconstraining_bijectors = [
    tfp.bijectors.Exp(),       # Maps a positive real to R.
    tfp.bijectors.Exp(),       # Maps a positive real to R.
    tfp.bijectors.Sigmoid(),   # Maps [0,1] to R.
]

def joint_log_prob(count_data, lambda_1, lambda_2, tau):
    tfd = tfp.distributions

    alpha = (1. / tf.reduce_mean(count_data))
    rv_lambda_1 = tfd.Exponential(rate=alpha)
    rv_lambda_2 = tfd.Exponential(rate=alpha)

    rv_tau = tfd.Uniform()

    lambda_ = tf.gather(
         [lambda_1, lambda_2],
         indices=tf.to_int32(tau * tf.cast(tf.size(count_data), tf.float32) <= tf.cast(tf.range(tf.size(count_data)), tf.float32)))
    rv_observation = tfd.Poisson(rate=lambda_)

    return (
         rv_lambda_1.log_prob(lambda_1)
         + rv_lambda_2.log_prob(lambda_2)
         + rv_tau.log_prob(tau)
         + tf.reduce_sum(rv_observation.log_prob(count_data))
    )

# Define a closure over our joint_log_prob.
def unnormalized_log_posterior(lambda1, lambda2, tau):
    return joint_log_prob(count_data, lambda1, lambda2, tau)

# Initialize the step_size. (It will be automatically adapted.)
with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE):
    step_size = tf.get_variable(
        name='step_size',
        initializer=tf.constant(0.05, dtype=tf.float32),
        trainable=False,
        use_resource=True
    )

# Sample from the chain.
[
    lambda_1_samples,
    lambda_2_samples,
    posterior_tau,
], kernel_results = tfp.mcmc.sample_chain(
    num_results=100000,
    num_burnin_steps=10000,
    current_state=initial_chain_state,
    kernel=tfp.mcmc.TransformedTransitionKernel(
        inner_kernel=tfp.mcmc.HamiltonianMonteCarlo(
            target_log_prob_fn=unnormalized_log_posterior,
            num_leapfrog_steps=2,
            step_size=step_size,
            step_size_update_fn=tfp.mcmc.make_simple_step_size_update_policy(),
            state_gradients_are_stopped=True),
        bijector=unconstraining_bijectors))

tau_samples = tf.floor(posterior_tau * tf.cast(tf.size(count_data)), tf.float32)

# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tf.shape(tau_samples)[0]
expected_texts_per_day = tf.zeros(n_count_data)

# Initialize any created variables.
init_g = tf.global_variables_initializer()
init_l = tf.local_variables_initializer()

```

## Executing the TF graph to sample from the posterior

```
evaluate(init_g)
evaluate(init_l)
[
    lambda_1_samples_,
    lambda_2_samples_,
    tau_samples_,
    kernel_results_,
    N_,
    expected_texts_per_day_,
] = evaluate([
    lambda_1_samples,
    lambda_2_samples,
    tau_samples,
    kernel_results,
    N,
    expected_texts_per_day,
])

print("acceptance rate: {}".format(
    kernel_results_.inner_results.is_accepted.mean()))
print("final step size: {}".format(
    kernel_results_.inner_results.extra.step_size_assign[-100:].mean()))

```

    acceptance rate: 0.58777
    final step size: 0.034716054797172546

## Plot the Results

```
plt.figure(figsize=(12.5, 15))
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples_, histtype='stepfilled', bins=30, alpha=0.85,
         label=r"posterior of $\lambda_1$", color=TFColor[0], density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel(r"$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples_, histtype='stepfilled', bins=30, alpha=0.85,
         label=r"posterior of $\lambda_2$", color=TFColor[6], density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel(r"$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples_.shape[0] * np.ones_like(tau_samples_)
plt.hist(tau_samples_, bins=n_count_data_[0], alpha=1,
         label=r"posterior of $\tau$",
         color=TFColor[2], weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data_[0]))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data_)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel(r"probability");
```

## Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```
plt.figure(figsize=(12.5, 9))

for day in range(0, n_count_data_[0]):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples_
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day_[day] = (lambda_1_samples_[ix].sum()
                                   + lambda_2_samples_[~ix].sum()) / N_

plt.plot(range(n_count_data_[0]), expected_texts_per_day_, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data_[0])
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data_)), count_data_, color="#5DA5DA", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

## Exercises

1.   Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```
#type your code here.
```

2.   What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`

```
#type your code here.
```

3. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45? That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the TFP part. Just consider all instances where `tau_samples < 45`.)

```
#type your code here.
```

## References

[1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg)

[2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).

[3] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.

[4] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()

```


Table of Contents

1  Demonstration Dimensionless Learning for Keyhole Dynamics

2  Import libraries

3  Parametric space analysis

3.1  Load dataset

3.2  Calculate dimension matrix

3.3  Calculate basis vectors

4  Helper functions

5  Best representation learning discovery

6  Reference

# Demonstration Dimensionless Learning for Keyhole Dynamics

- **Authors**: Xiaoyu Xie, Zhengtao Gan
- **Contact**: xiaoyuxie2020@u.northwestern.edu
- **Date**: Oct. 2021

# Import libraries

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import matrix_rank
from numpy.linalg import inv
import pandas as pd
import pysindy as ps
import random
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from scipy.optimize import minimize
```

# Parametric space analysis

Paraemter list:

$f(\eta P,V_s,r_0, C_p, \alpha, \rho, T_l-T_0)=e$

## Load dataset

```python
data = np.loadtxt(
    open("../dataset/dataset_keyhole.csv","rb"),
    delimiter=',',
    skiprows=1,
    usecols = (2,3,5,7,8,9,12,14)
)
X = data[:,0:7]
Y = data[:,7]
```

## Calculate dimension matrix

Dimension matrix (input):

\begin{equation}
D_{in}=
\left[
  \begin{array}{ccc}
    2 & 1 & 1 & 2 &2&-3&0\\
    -3 & -1 & 0 & -2 &-1&0&0\\
    1& 0 & 0 & 0 &0&1&0\\
    0& 0 & 0 & -1 &0&0&1\\
  \end{array}
\right]
\end{equation}

Dimension matrix (output):

\begin{equation}
D_{out}=
\left[
  \begin{array}{ccc}
    1 \\
    0 \\
    0 \\
    0 \\
  \end{array}
\right]
\end{equation}

```python
D_in = np.mat('2,1,1,2,2,-3,0; -3,-1,0,-2,-1,0,0; 1,0,0,0,0,1,0;0,0,0,-1,0,0,1')
D_out = np.mat('1;0;0;0')
D_in_rank = matrix_rank(D_in)
print(D_in_rank)
```

    4

## Calculate basis vectors

Calculate three basis vectors for equation:
$
D_{in}x=0
$

```python
Din1 = D_in[:, 0:4]
Din2 = D_in[:, 4:8]

x2 = np.mat('-1; 0; 0')
x1 = -inv(Din1) * Din2 * x2
basis1_in = np.vstack((x1, x2))
print(f'basis1_in: \n{basis1_in}')

x2 = np.mat('0;-1;0')
x1 = -inv(Din1) * Din2 * x2
basis2_in = np.vstack((x1, x2))
print(f'basis2_in: \n{basis2_in}')

x2 = np.mat('0; 0; -1')
x1 = -inv(Din1) * Din2 * x2
basis3_in = np.vstack((x1, x2))
print(f'basis3_in: \n{basis3_in}')
```

    basis1_in:
    [[ 0.]
     [ 1.]
     [ 1.]
     [ 0.]
     [-1.]
     [ 0.]
     [ 0.]]
    basis2_in:
    [[ 1.]
     [-3.]
     [-2.]
     [ 0.]
     [ 0.]
     [-1.]
     [ 0.]]
    basis3_in:
    [[ 0.00000000e+00]
     [ 2.00000000e+00]
     [-3.70074342e-17]
     [-1.00000000e+00]
     [ 0.00000000e+00]
     [ 0.00000000e+00]
     [-1.00000000e+00]]

# Helper functions

```python
def calc_pi(a):
    '''
    Calculate pi
    Note that the best coef for keyhole is [0.5, 1, 1]
    '''
    coef_pi = 0.5 * basis1_in + a[0] * basis2_in + a[1] * basis3_in
    pi_mat = np.exp(np.log(X).dot(coef_pi))
    pi = np.squeeze(np.asarray(pi_mat))
    return pi

def calc_y(a, w):
    '''
    Calculate the prediction y using a polynomial function
    '''
    pi = calc_pi(a)
    y = w[0] + w[1] * pi + w[2] * pi**2 + w[3] * pi**3 + w[4] * pi**4 + w[5] * pi**5
    return y

def objective(a, w):
    '''
    Calculate objective(loss)
    '''
    return np.square(pi2 - calc_y(a, w)).mean()

def ploter(pi1, pi2, iteration):
    fig = plt.figure()
    plt.scatter(pi1, pi2)
    plt.xlabel('pi1', fontsize=16)
    plt.ylabel('pi2', fontsize=16)
    plt.title(f'iteration: {iteration}', fontsize=24)
    plt.show()
```

# Best representation learning discovery

```python
random.seed(3)

niter = 3000
ninital = 1

a = np.zeros(2)
w = np.zeros(6)

global pi2
pi2 = Y / X[:,2]

feature_library = ps.PolynomialLibrary(degree=5)
optimizer =LinearRegression(fit_intercept=False)

for j in range(ninital):
    a[0] = 2 * random.random()
    a[1] = 2 * random.random()
    print(f'Initial a={a}')

    info = {}
    info['initial'] = a
    a_history = np.zeros((niter, 2))

    model = ps.SINDy(feature_library=feature_library, optimizer=optimizer)

    for i in range(niter):
        # update coefficient w for polynomials
        pi1 = calc_pi(a)

        model.fit(pi1, x_dot=pi2);
        coeffi = model.coefficients()
        w = coeffi[0]
        y_recover = calc_y(a, w)
        r2 = r2_score(y_recover, pi2)

        # update coefficient a for pi
        solution = minimize(objective, a, method='BFGS', tol=1e-3, args=w, options={'maxiter':1})
        a = solution.x
        y_recover = calc_y(a,w)
        r2 = r2_score(y_recover, pi2)

        a_history[i,:] = a
        if i % 1000 == 0:
            # Note that the best coef in keyhole case is [0.5, 1, 1]
            # here we only optimize the last two coefficients
            print(f'Iteration: {i}, coef: {a}, Objective: {objective(a,w)}')
            ploter(pi1, pi2, i)
```

# Reference

- [1] **Xie, X.**, Liu, W. K., & **Gan, Z.** (2021). Data-driven discovery of dimensionless numbers and scaling laws from experimental measurements. ArXiv:2111.03583 [Physics]. http://arxiv.org/abs/2111.03583
- [2] **Gan, Z.**, Kafka, O. L., Parab, N., Zhao, C., Fang, L., Heinonen, O., Sun, T., & Liu, W. K. (2021). Universal scaling laws of keyhole stability and porosity in 3D printing of metals. Nature Communications, 12(1), 2379. https://doi.org/10.1038/s41467-021-22704-0
- [3] Saha, S., **Gan, Z.**, Cheng, L., Gao, J., Kafka, O. L., **Xie, X.**, Li, H., Tajdari, M., Kim, H. A., & Liu, W. K. (2021). Hierarchical Deep Learning Neural Network (HiDeNN): An artificial intelligence (AI) framework for computational science and engineering. Computer Methods in Applied Mechanics and Engineering, 373, 113452. https://doi.org/10.1016/j.cma.2020.113452

```python

```


```python
"""
importing the necessary libraries, do not modify
"""
%matplotlib inline
# plots graphs within the notebook
%config InlineBackend.figure_format='svg' # not sure what this does, may be default images to svg format

from IPython.display import display,Image, Latex, Math
from __future__ import division
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')

import time
from IPython.display import clear_output

import SchemDraw as schem
import SchemDraw.elements as e

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

from IPython.core.display import HTML
def header(text):
    raw_html = '<h4>' + str(text) + '</h4>'
    return raw_html

def box(text):
    raw_html = '<div style="border:1px dotted black;padding:2em;">'+str(text)+'</div>'
    return HTML(raw_html)

def nobox(text):
    raw_html = '<p>'+str(text)+'</p>'
    return HTML(raw_html)

def addContent(raw_html):
    global htmlContent
    htmlContent += raw_html

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

font = {'family' : 'serif',
        'color'  : 'black',
        'weight' : 'normal',
        'size'   : 18,
        }

from scipy.constants import convert_temperature
def C2K(T):
    return convert_temperature(T,'Celsius','Kelvin')
def C2F(T):
    return convert_temperature(T,'Celsius','Fahrenheit')
def F2K(T):
    return convert_temperature(T,'Fahrenheit','Kelvin')
def F2C(T):
    return convert_temperature(T,'Fahrenheit','Celsius')
def K2F(T):
    return convert_temperature(T,'Kelvin','Fahrenheit')
def K2C(T):
    return convert_temperature(T,'Kelvin','Celsius')

```

PB 8.31

To cool a summer home without using a vapor-compression refrigeration cycle, air is routed through a plastic pipe ($k=0.15\text{ W/m.K}$, $D_i=0.15\text{ m}$, $D_o=0.17\text{ m}$) that is submerged in an adjoining body of water. The water temperature is nominally at $T_\infty= 17^\circ\text{C}$, and a convection coefﬁcient of $h_o\approx 1500\text{ W/m$^2$. K}$ is maintained at the outer surface of the pipe.

If air from the home enters the pipe at a temperature of $T_{m,i}= 29^\circ\text{C}$ and a volumetric ﬂow rate of $\dot{\forall}_i= 0.025\text{ m$^3$/s}$, what pipe length $L$ is needed to provide a discharge temperature of $T_{m,o}=21^\circ\text{C}$? What is the fan power required
to move the air through this length of pipe if its inner surface is smooth?

```python
from NewLibraries import thermodynamics as thermo
from NewLibraries import HT_thermal_resistance as res
from NewLibraries import HT_external_convection as extconv
from NewLibraries import HT_internal_convection as intconv
D_i = 0.15 #m
D_o = 0.17 #m
k_pipe = 0.15 #W/(m.K)
T_mi = 29. #C
T_mo = 21. #C
T_infty = 17 #C
Qdot = 0.025 #kg/s
U_m = Qdot/(np.pi*D_i**2/4.)
T_mean_air = (T_mi+T_mo)/2.
h_o = 1500 #W/m^2K

T_m = (T_mi+T_mo)/2.
print("Average Temperature over the pipe length T_m= %2.1f C" %T_m)

airprop_m = thermo.Fluid("air",C2K(T_m))

?intconv.PipeFlow

airflow = intconv.PipeFlow(D_i, Um = U_m, nu = airprop_m.nu )

```

    Average Temperature over the pipe length T_m= 25.0 C

    Init signature: intconv.PipeFlow(D, Re=0.0, Um=0.0, mdot=0.0, nu=0.0, rho=0.0, L=1.0)
    Docstring:
    Determination of Nu, pressure drop, mean temperature for internal convection
    import HT_internal_convection.py as intconv

    pipe =intconv.PipeFlow(D, Re=0.0, Um = 0.0, mdot = 0.0, nu = 0.0, rho = 0.0)
    where
    D is the only required input and one of the following combination (Re, nu) or (Um, nu) or (mdot, rho, nu)
    Hence the minimum calls for PipeFlow are
    pipe =intconv.PipeFlow(D, Re= Re_m, nu = nu_m) outputs pipe.Um
    pipe =intconv.PipeFlow(D, Re= Re_m, nu = nu_m, rho = rho_m) outputs pipe.Um (bulk velocity)
            and pipe.mdot (mass flow)
    pipe =intconv.PipeFlow(D, Um = 0.0, nu = 0.0) outputs pipe.Re
    pipe =intconv.PipeFlow(D, Um = Um, nu = nu_m, rho = rho_m) outputs pipe.Re, pipe.mdot
    pipe =intconv.PipeFlow(D, mdot = 0.0, nu = 0.0, rho = 0.0) outputs pipe.Re, pipe.Um

    pipe.f_laminar(Re) outputs the friction factor for laminar flow pipe.f
    pipe.f_turbulent(Re,eps = 0.0, nu = 0.0) outputs the friction factor for turbulent flow pipe.f

    The following correlations output pipe.Nu
    pipe.laminar_isothermal for isothermal wall boundary condition
    pipe.laminar_isoflux for isoflux wall boundary condition
    pipe.Dittus_Boelter(mode, Pr, Re = 0.) for turbulent flow where mode is either "heating" or "cooling"
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    pipe.Sieder_Tate(Pr,mu,mu_s, Re = 0.0) mu and mu_s are the mean and wall dynamics viscosities
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    pipe.Gnielinski( Pr, f,Re = 0.0): where f is the friction factor
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    File:           ~/gitrepo/UVM-ME144-Heat-Transfer/NewLibraries/HT_internal_convection.py
    Type:           type

For a section of pipe of length $\Delta x$, where the fluid is cooled by the fluid outside of the pipe, the heat transfer $dq_{conv}$ between the flow and the wall is:

$$
-dq_{conv}=\dot{m}C_p\Delta T_m
$$

This radial heat transfer can also be expressed as a function of the local heat flux $q''(x)$ assumed to vary little over the small distance $\Delta x$,

$$
dq_{conv}=q'(x)\Delta x
$$

and

$$
q'(x) = \frac{1}{R'_{tot}}\left(T_{m,x}-T_\infty\right)
$$

leading to

$$
\dot{m}C_p\Delta T_m = -\frac{\Delta x}{R'_{tot}}\left(T_{m,x}-T_\infty\right)
$$

Note that in cylindrical system, the heat flux (heat rate per unit area, $W/m^2$) is not conserved due to the increase of surface area though which radial heat transfer travels as a function of increasing radial position. Consequently, one can only calculate heat rate per unit length of the pipe ($W/m$) or heat rate ($W$) in cylindrical coordinates. In spherical coordinates, one can only calculate heat rate.

In the present case, the thermal circuit looks as follows:

```python
R = []
#?res.Resistance
R.append(res.Resistance(r"$\frac{1}{h_iA_i}$","W/m"))
R.append(res.Resistance(r"$\frac{\ln(D_o/D_i)}{2\pi k}$","W/m"))
R.append(res.Resistance(r"$\frac{1}{h_oA_o}$","W/m"))

d= schem.Drawing()
d.add( e.DOT, label = r"$T_{m,x}$")
d.add( e.RES, d = 'right', botlabel = R[0].name)
d.add( e.RES, d = 'right', botlabel = R[1].name)
d.add( e.RES, d = 'right', botlabel = R[2].name)
d.add( e.DOT, label = r"$T_{\infty}$")
L1 = d.add(e.LINE, lftlabel = "$q'$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(L1, arrowofst = 0)
d.draw()

```

```python
#determination of h_i:
print("Re= %5.0f" %airflow.Re)
```

    Re= 13625

The flow is turbulent. We choose to use the Dittus Boetler correlation, in cooling mode.

All correlations for fully developed flows are valid when
$$
\frac{L}{D}\gtrsim 10
$$

```python
?intconv.PipeFlow
airflow.Dittus_Boelter("cooling",airprop_m.Pr)
h_i = airflow.Nu*airprop_m.k/D_i

print("h_i= %1.2f W/m^2.K" %h_i)
A_i = np.pi*D_i
R[0].convection(h_i,A_i)
R[1].conduction("cylindrical",k_pipe, r_a = D_i/2, r_b = D_o/2, L_pipe = 1.0)
A_o = np.pi*D_o
R[2].convection(h_o,A_o)
R_tot = res.serial_sum(R,0,2)
```

    h_i= 7.32 W/m^2.K

    Init signature: intconv.PipeFlow(D, Re=0.0, Um=0.0, mdot=0.0, nu=0.0, rho=0.0, L=1.0)
    Docstring:
    Determination of Nu, pressure drop, mean temperature for internal convection
    import HT_internal_convection.py as intconv

    pipe =intconv.PipeFlow(D, Re=0.0, Um = 0.0, mdot = 0.0, nu = 0.0, rho = 0.0)
    where
    D is the only required input and one of the following combination (Re, nu) or (Um, nu) or (mdot, rho, nu)
    Hence the minimum calls for PipeFlow are
    pipe =intconv.PipeFlow(D, Re= Re_m, nu = nu_m) outputs pipe.Um
    pipe =intconv.PipeFlow(D, Re= Re_m, nu = nu_m, rho = rho_m) outputs pipe.Um (bulk velocity)
            and pipe.mdot (mass flow)
    pipe =intconv.PipeFlow(D, Um = 0.0, nu = 0.0) outputs pipe.Re
    pipe =intconv.PipeFlow(D, Um = Um, nu = nu_m, rho = rho_m) outputs pipe.Re, pipe.mdot
    pipe =intconv.PipeFlow(D, mdot = 0.0, nu = 0.0, rho = 0.0) outputs pipe.Re, pipe.Um

    pipe.f_laminar(Re) outputs the friction factor for laminar flow pipe.f
    pipe.f_turbulent(Re,eps = 0.0, nu = 0.0) outputs the friction factor for turbulent flow pipe.f

    The following correlations output pipe.Nu
    pipe.laminar_isothermal for isothermal wall boundary condition
    pipe.laminar_isoflux for isoflux wall boundary condition
    pipe.Dittus_Boelter(mode, Pr, Re = 0.) for turbulent flow where mode is either "heating" or "cooling"
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    pipe.Sieder_Tate(Pr,mu,mu_s, Re = 0.0) mu and mu_s are the mean and wall dynamics viscosities
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    pipe.Gnielinski( Pr, f,Re = 0.0): where f is the friction factor
        The Re is optional if omitted, the Reynolds number calculated in the object PipeFlow will be used
    File:           ~/gitrepo/UVM-ME144-Heat-Transfer/NewLibraries/HT_internal_convection.py
    Type:           type

In the limit $\Delta T_m\rightarrow 0$ and $\Delta x\rightarrow 0$, this equation defines the longitudinal temperature gradient of the mean temperature

$$
\frac{d T_{m,x}}{dx} = -\frac{1}{\dot{m}C_pR'_{tot}}\left(T_{m,x}-T_\infty\right)
$$

which is equivalent to

$$
\frac{d \left(T_{m,x}-T_\infty\right)}{dx} = -\frac{1}{\dot{m}C_pR'_{tot}}\left(T_{m,x}-T_\infty\right)
$$

Integrating from the inlet to the outlet,

$$
\int_{T_{m,i}}^{T_{m,o}}\frac{d \left(T_{m,x}-T_\infty\right)}{\left(T_{m,x}-T_\infty\right)} = -\int_0^L\frac{1}{\dot{m}C_pR'_{tot}}dx
$$

yields the equation for the exponential relation between the mean outlet and inlet temperature:

$$
\ln\left(\frac{T_{m,o}-T_\infty}{T_{m,i}-T_\infty}\right) = -\frac{L}{\dot{m}C_pR'_{tot}}
$$

or

$$
\frac{T_{m,o}-T_\infty}{T_{m,i}-T_\infty} = \exp\left(-\frac{L}{\dot{m}C_pR'_{tot}}\right)
$$

Note that this integration supposed that $R'_{tot}$ is constant, which is only true if the convection coefficients used to calculate the total thermal resistance are averaged coefficients:
$$
\bar{h}=\frac{1}{L}\int_0^Lh(x)dx
$$
Similarly, the thermodynamic properties are estimated at the average temperature of the inlet and outlet mean temperatures:
$$
T_m=\frac{T_{m,i}+T_{m,o}}{2}
$$

L is then easily obtained from
$$
L = -\dot{m}C_pR'_{tot}\ln\left(\frac{T_{m,o}-T_\infty}{T_{m,i}-T_\infty}\right)
$$

```python
mdot = airprop_m.rho*Qdot
P_i = np.pi*D_i
L = -mdot*airprop_m.Cp*R_tot*np.log((T_mo-T_infty)/(T_mi-T_infty))
print("To achieve T_m,o = %2.1f F, the required length is %2.2f m" %(C2F(T_mo),L))

```

    To achieve T_m,o = 69.8 F, the required length is 13.87 m

The pressure drop
$$
\Delta P = f\frac{L}{D_i}\frac{\rho U_m^2}{2}
$$
requires the calculation of the friction factor $f$ using

```python
airflow.f_turbulent(airflow.Re)
print("friction factor: %1.3f" %airflow.f)
Delta_P = airflow.f*L/D_i*airprop_m.rho*airflow.Um**2/2.

```

    Pipe wall is assumed to be hydrodynamically smooth
    friction factor: 0.028

 The fan power is
$$
{\cal P}=\dot{Q}\Delta P
$$

```python
Power = Qdot*Delta_P
print("Fan power: %1.3f W" %Power)
```

    Fan power: 0.078 W

 PB 8.36

Hot water at mean temperature $T_m=50\text{$^\circ$C}$ is routed from one building in which it is generated to an adjoining building in which it is used for space heating. Transfer between the buildings occurs in a steel pipe ($k=60\text{ W/m.K}$) of $100 \text{ mm}$ outside diameter and 8-mm wall thickness. During the winter, representative environmental conditions involve air at $T_\infty= -5\text{$^\circ$C}$ and $V_\infty=3\text{ m/s}$ in cross ﬂow over the pipe.
Using the Churchill Bernstein and Dittus Boehler correlations, calculate the total heat transfer rate per unit length $q'$, the daily energy cost $Q'=q'\times 24\text{ h/d}$ per meter and the cost per day and per meter assuming an electricity cost of $\text{\$}0.05\text{/kW.h}$.

```python
D_o = 100.e-3 #mm
D_i = D_o - 2* 8e-3 #mm
k_p = 60. # W/m.K
T_infty = -5 #C
V_infty = 3 #m/s
T_m = 50. #C
k_pipe = 60 #W/m.K
U_m = 0.5 #m/s

T_m_air = (T_infty + T_m)/2.

air = thermo.Fluid("air",C2K(T_m_air))

water = thermo.Fluid("water",C2K(T_m))

waterflow = intconv.PipeFlow(D_i,Um = U_m, nu = water.nu)
print("Re_i= %5.0f" %waterflow.Re)
waterflow.Dittus_Boelter("cooling",water.Pr)
h_i = waterflow.Nu*water.k/D_i
print("h_i= %4.1f W/m" %h_i)

# external convection
Re_o = V_infty*D_o/air.nu
airflow = extconv.CircularCylinder('Churchill-Bernstein',Re_o,air.Pr)
h_o = airflow.Nu*air.k/D_o
print("h_o= %4.1f W/m" %h_o)
```

    Re_i= 75882
    h_i= 2067.0 W/m
    h_o= 20.3 W/m

```python
R = []
#?res.Resistance
R.append(res.Resistance(r"$\frac{1}{h_iA_i}$","W/m"))
R.append(res.Resistance(r"$\frac{\ln(D_o/D_i)}{2\pi k}$","W/m"))
R.append(res.Resistance(r"$\frac{1}{h_oA_o}$","W/m"))

d= schem.Drawing()
d.add( e.DOT, label = r"$T_{m,x}$")
d.add( e.RES, d = 'right', botlabel = R[0].name)
d.add( e.RES, d = 'right', botlabel = R[1].name)
d.add( e.RES, d = 'right', botlabel = R[2].name)
d.add( e.DOT, label = r"$T_{\infty}$")
L1 = d.add(e.LINE, lftlabel = "$q'_{conv}$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(L1, arrowofst = 0)
d.draw()

R[0].convection(h_i,np.pi*D_i)
R[1].conduction("cylindrical",k_pipe,r_a = D_i/2., r_b = D_o/2., L_pipe = 1.0)
R[2].convection(h_o,np.pi*D_o)

```

```python
q_prime = (T_m - T_infty)/res.serial_sum(R,0,2)
print("total heat rate per unit length of the pipe: %3.0f W/m" %q_prime)
Cost = q_prime *24 * 0.05e-3
print("Cost per day per pipe unit length: $%1.2f /m.d" %Cost)
```

    total heat rate per unit length of the pipe: 345 W/m
    Cost per day per pipe unit length: $0.41 /m.d

PB 9.64

Consider the hot water pipe of Problem 8.36, but under conditions for which the ambient air is not in cross flow over the pipe and is, instead, quiescent. Accounting for the effect of radiation with a pipe emissivity of 􏰇$\varepsilon_p=0.6$, what is the corresponding daily cost of heat loss per unit length of the uninsulated pipe?

```python
?res.Resistance
R = []
#?res.Resistance
R.append(res.Resistance(r"$\frac{1}{h_i\pi D_i}$","W/m"))
R.append(res.Resistance(r"$\frac{\ln(D_o/D_i)}{2\pi k}$","W/m"))
R.append(res.Resistance(r"$\frac{1}{h_o\pi D_o}$","W/m"))
R.append(res.Resistance(r"$\frac{1}{h_r\pi D_o}$","W/m"))

d = schem.Drawing()
d.add( e.DOT, label= r"$T_m$")
d.add( e.RES, d = 'right', botlabel = R[0].name)
d.add( e.DOT)
d.add( e.RES, d = 'right', botlabel = R[1].name)
d.add( e.DOT, rgtlabel = r"$T_{s,o}$")
L0 = d.add ( e.LINE, d = 'up', l = 1.5)
d.add( e.RES, d = 'right', botlabel = R[2].name)
d.add( e.LINE, d = 'down', l = 1.5)
d.add( e.LINE, d = 'down', xy = L0.end)
d.add( e.RES, d = 'right', botlabel = R[3].name)
d.add( e.LINE, d = 'up', l = 1.5)
d.add( e.LINE, d = 'right', l = 1.5)
d.add( e.DOT, label = r"$T_\infty$")
L1 = d.add(e.LINE, lftlabel = "$q'$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(L1, arrowofst = 0)
d.draw()
```

    Init signature: res.Resistance(name, units)
    Docstring:
    Defines thermal resistances for conduction, convection and radiation heat transfer.
    First define the object attached with class with the name used in the thermal circuit
    and the units, which can only be 'W', 'W/m' or 'W/m^2'
    Second use self.conduction, self.convection or self.radiation to calculate your
    resistance. Each mode requires different arguments:

    from Libraries import HT_thermal_resistance as res
    R = []
    R.append(res.Resistance("$label$", "units")) where units = 'W', 'W/m' or 'W/m^2'

    then

    R.conduction(geo, k, thickness = 0.0, A = 1.0, L_pipe = 1.0, r_a = 0., r_b = 0.,k_name = "k",                   thickness_name = "L", L_pipe_name = "L", r_a_name = "r_a",r_b_name = "r_b",A_name = "A",                   T_a_name = "T_a",T_b_name = "T_b"),
    where geo can only be 'plane','cylindrical' or 'spherical'
    The minimum number of arguments are:
    R.conduction("plane", k, thickness = a) for heat flux (where a>0)
    R.conduction("plane", k, thickness = a, A = lengthorarea) for heat rate by unit length (m) or for heat rate (m^2)
    R.conduction("cylindrical",k, r_a = a, r_b = b) for heat rate per unit length of the pipe
    R.conduction("cylindrical",k, r_a = a, r_b = b, L_pipe = L) for heat rate
    R.conduction("spherical",k, r_a = a, r_b = b) for heat rate

    thickness is the thickness of the material for plane conduction.
    r_a is the inner radius of the cylinder/sphere, r_b is the outer radius of the cylinder/sphere.
    A is the surface area of the system for plane conduction
    L_pipe is the pipe length for cylindrical conduction.
    All arguments ending with _name are used to write heat flux/rate equations(they are strings preferably
    LaTeX formatted, without $$)

    R.convection(h,A,h_name = "h",A_name = "A",T_a_name = "T_a",Tb_name = "T_b"), where h is the convection coefficient (W/m^2K) and A is
    the surface area. All arguments ending with _name are used to write the flux equations(they are strings
    preferably LaTeX formatted)
    The minimum number of arguments are:
    R.convection(h,A)

    R.radiation(eps,T_s,T_sur,A,h_name = "h_r",A_name = "A",Ts_name = "T_s",Tsur_name = "T_{sur}"), where eps is the permissivity of the material, T_s
    the surface temperature, T_sur the far away surface temperature, A the surface area.
    The minimum number of arguments are:
    R.radiation(eps,T_s,T_sur,A)

    R.contact(R,A,R_name= "R_{t}",A_name = "A",T_a_name = "T_a",Tb_name = "T_b"), where R is the contact resistance, typically obtained from a table
    A is the surface area
    The minimum number of arguments are:
    R.contact(R,A)

    R.display_equation(index) displays the heat flux/rate equations for a given resistance. index is the number of
    your resistance (you specify)

    Outputs include R[i].R the resistance of element i, R[i].h the convection or radiation coefficient.

    Functions include
    R_tot = res.serial_sum(R,first_resistance,last_resistance) sums serial resistance
    R_tot = res.parallel_sum(R,first_resistance,last_resistance) sums parallel resistance
    File:           ~/gitrepo/UVM-ME144-Heat-Transfer/NewLibraries/HT_thermal_resistance.py
    Type:           type

 To solve this problem, we need $T_{s,o}$ to determine the radiation heat transfer resistance and the film temperature for thermodynamic properties. Unfortunately solving for $T_{s,o}$ requires calculating $q'$. Once again, we need to make a guess.

An educated guess takes into consideration the fact that steel conducts heat really well. Therefore we assume that $T_{s,o}\approx T_m$. This assumption must be tested at the end.

```python

R[0].convection(h_i,np.pi*D_i)
R[1].conduction("cylindrical",k_pipe,r_a = D_i/2., r_b = D_o/2., L_pipe = 1.0)

T_so = T_m
R[3].radiation(0.6,T_so,T_infty,np.pi*D_o)
T_f = 0.5*(T_m+T_infty)
air = thermo.Fluid('air',C2K(T_f))

#
from NewLibraries import HT_natural_convection as natconv
?natconv.HorizontalCylinder
Ra_o = natconv.Ra(9.81,air.beta,T_so-T_infty,D_o,air.nu,air.alpha)
print("Ra= %1.2e" %Ra_o)
airflow_o = natconv.HorizontalCylinder("Churchill-Chu", Ra_o, Pr = air.Pr)
print("Nu = %2.2f" %airflow_o.Nu)
h_o = airflow_o.Nu*air.k/D_o
print("Convection coefficient in air h_o %3.2f W/m^2.K" %h_o)
R[2].convection(h_o,np.pi*D_o)
R_tot = res.serial_sum(R,0,1)+res.parallel_sum(R,2,3)

q_prime = (T_m - T_infty)/R_tot
print("Heat rate per unit length of pipe: %3.2f W/m" %q_prime)
New_T_so = T_m - q_prime*res.serial_sum(R,0,1)
print("New surface temperature: %2.2f C" %New_T_so)

Cost = q_prime *24 * 0.05e-3
print("Cost per day per pipe unit length: $%1.2f /m.d" %Cost)
```

    Ra= 5.52e+06
    Nu = 23.73
    Convection coefficient in air h_o 6.14 W/m^2.K
    Heat rate per unit length of pipe: 105.70 W/m
    New surface temperature: 49.76 C
    Cost per day per pipe unit length: $0.13 /m.d

    Init signature: natconv.HorizontalCylinder(correlation='Morgan', Ra=0.0, Pr=0.0)
    Docstring:
    Natural convection about a horizontal cylinder
    from NewLibraries import HT_natural_convection as natconv
    cyl = natconv.HorizontalCylinder(correlation, Ra, Pr = 0.0)
    where correlation is "Morgan" or "Churchill-Chu"
    cyl = natconv.HorizontalCylinder("Morgan", Ra)
    cyl = natconv.HorizontalCylinder("Churchill-Chu", Ra, Pr = xx)
    File:           ~/gitrepo/UVM-ME144-Heat-Transfer/NewLibraries/HT_natural_convection.py
    Type:           type

The computed surface temperature is very close to our initial guess. There is no need for further iteration.

Note that the heat transfer with natural convection is less than with force convection, as expected


##### Copyright 2020 Masahiko Isshiki - https://blog.masahiko.info/.

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# 「TensorFlowやPyTorchで自動微分して勾配を取得する方法」

## ■準備

### ●Pythonバージョン：3.x

```
import sys
print('Python', sys.version)
# Python 3.6.9 (default, Nov  7 2019, 10:44:02)  …… などと表示される
```

    Python 3.6.9 (default, Nov  7 2019, 10:44:02)
    [GCC 8.3.0]

### ●TensorFlowバージョン

```
# Google Colabで最新の2.xを使う場合、2.xに切り替える（Colab専用）
%tensorflow_version 2.x
```

```
import tensorflow as tf
print('TensorFlow', tf.__version__)
# TensorFlow 2.1.0 ……などと表示される
```

    TensorFlow 2.1.0

### ●PyTorchバージョン

```
import torch
print('PyTorch', torch.__version__)
# PyTorch 1.4.0 ……などと表示される
```

    PyTorch 1.4.0

## ■実装方法

### ●例として使う計算式

  $$
  y=2x^3+4x^2+5x+6
  $$

#### ○これをPythonの式にすると...

```
def calc(x):
  return (2 * x ** 3) + (4 * x ** 2) + (5 * x) + 6
```

### ●微分する導関数の式は...

$$
\frac{dy}{dx}=6x^2+8x+5
$$

#### ○これをPythonの式にすると...

```
def der_calc(x):
  return (6 * x ** 2) + (8 * x) + 5
```

### ●導関数に$x=0.5$を入力すると...

$$
\begin{align}
\frac{dy}{dx}&=6\times(0.5)^2+8\times(0.5)+5 \\
&=(6\times0.25)+(8\times0.5)+5 \\
&=1.5+4+5 \\
&=10.5
\end{align}
$$

#### ○Pythonのコードで計算すると...

```
result = der_calc(0.5)
result # 10.5
```

    10.5

$10.5$が出力される。これと同じことをPyTorchとTensorFlowの自動微分機能を使って行う方法を紹介する。

## ■PyTorch編

- データをPyTorchテンソル化して、計算式を作る
- `backward`メソッドで逆伝播する（自動微分）
- データ（テンソル）の`grad`変数で微分係数（＝勾配）を取得できる

という流れで書けばよい。具体的には以下の通り。

```
import torch
import torch.nn as nn  # 「ニューラルネットワーク」モジュールの別名定義

x = torch.tensor(0.5, requires_grad=True)  # 変数に対して「勾配計算が必要」とマーク
y = calc(x)    # 既存の計算式から計算グラフを動的に構築
print(y)       # tensor(9.7500, grad_fn=<AddBackward0>) ……などと表示される

y.backward()   # 逆伝播の処理として、上記式から微分係数（＝勾配）を計算（自動微分：Autograd）
g = x.grad     # 与えられた入力（x）によって計算された勾配の値（grad）を取得
print(g)       # tensor(10.5000)  ……などと表示される
```

    tensor(9.7500, grad_fn=<AddBackward0>)
    tensor(10.5000)

### ●グラフを描画

先ほどは単一のスカラーだったので、今度は複数のスカラー値で計算してみる。

```
# PyTorchの微分グラフ
import numpy as np
import matplotlib.pyplot as plt

x = torch.arange(-4.0, 4.0, 0.001, requires_grad=True)
y = calc(x)
y.backward(gradient=torch.ones_like(y)) # スカラーテンソルの場合は勾配を格納する場所が必要なのでテンソルを仮生成【重要】
g = x.grad
x_numpy = x.detach().numpy()  # デタッチして（＝追跡しないようにして）からndarray化する
g_numpy = g.detach().numpy()
plt.plot(x_numpy, g_numpy, label = "Derivative")
plt.xlim(-4, 4)
plt.ylim(-2.0, 30.0)
plt.grid()
plt.legend()
plt.show()
```

## ■TensorFlow編

- データをTensorテンソル化して、勾配テープ（GradientTape）内で計算式を作る
- 勾配テープ（tape）の`gradient`メソッドで逆伝播する（自動微分）
- 逆伝播により計算された微分係数（＝勾配）が取得される

という流れで書けばよい。具体的には以下の通り。

```
import tensorflow as tf
import tensorflow.keras.layers as layers  # 「レイヤー」モジュールの別名定義

x = tf.Variable(0.5)
with tf.GradientTape() as tape:  # 計算式に対して「勾配計算が必要」とマーク
  y = calc(x)  # 既存の計算式から計算グラフを動的に構築
print(y)       # tf.Tensor(9.75, shape=(), dtype=float32) ……などと表示される

g = tape.gradient(y, x)   # 逆伝播の処理として、上記式から微分係数（＝勾配）を計算（自動微分：Autograd）
print(g)       # tf.Tensor(10.5, shape=(), dtype=float32)  ……などと表示される
```

    tf.Tensor(9.75, shape=(), dtype=float32)
    tf.Tensor(10.5, shape=(), dtype=float32)

### ●グラフを描画

先ほどは単一のスカラーだったので、今度は複数のスカラー値で計算してみる。

```
# TensorFlowの微分グラフ
import numpy as np
import matplotlib.pyplot as plt

x = tf.Variable(tf.range(-4.0, 4.0, 0.001), trainable=True)
with tf.GradientTape() as tape:
  #tape.watch(x) # tf.constantなど「trainable=False」な状態であれば x を記録する（手動追跡）
  # ↑tf.Variableであれば既に訓練可能な状態（自動追跡）なので不要
  y = calc(x)
g = tape.gradient(y, x)
x_numpy = x.numpy()
g_numpy = g.numpy()
plt.plot(x_numpy, g_numpy, label = "Derivative")
plt.xlim(-4, 4)
plt.ylim(-2.0, 30.0)
plt.grid()
plt.legend()
plt.show()
```

# 簡単ですね。以上で終わりです。


Naive Bayes Notebook

@Author: Zhu Xu

@Date: July/2019

**Useful links**:

- [ ] [朴素贝叶斯](https://blog.csdn.net/u012162613/article/details/48323777)
- [ ] [朴素贝叶斯方法原理及实现](https://blog.csdn.net/tanhongguang1/article/details/45016421)
- [ ] [朴素贝叶斯算法](https://blog.csdn.net/fuqiuai/article/details/79458943)

简单的分类算法，例如文本分类（垃圾邮件过滤）

理论基础

贝叶斯定理

$P(A|B)$ 表示事件B已经发生的前提下，事件A发生的概率，叫做事件B发生下事件A的条件概率:
$$P(A|B)=\frac{P(AB)}{P(B)}\quad\Rightarrow\quad P(AB)=P(A|B)\cdot P(B)$$

贝叶斯定理便是基于条件概率，通过$P(A|B)$来求$P(B|A)$：

$$P(B|A)=\frac{P(BA)}{P(A)}=\frac{P(A|B)P(B)}{P(A)}$$

p.s. 全概率公式：$P(A)=\sum_{i=1}^{n}P(B_i)P(A|B_i)$

特征条件独立假设

给定训练数据集$(X,Y)$:
* 每个样本x都包括n维特征，即$x=(x_1,x_2,x_3,...,x_n)$
* 类标记集合含有k种类别，即$y=(y_1,y_2,...,y_k)$。

假设现在有一个新样本x，**从概率角度来看，就等价于给定x，它属于那个类别的概率最大**，将问题转化为求解$P(y_1|x),P(y_2|x),\cdots,P(y_k|x)$中最大的那个，即求后验概率最大的输出:$argmax_{y_k}P(y_k|x)$
$$P(y_k|x)=\frac{P(x|y_k)P(y_k)}{P(x)}$$

$$\Rightarrow P(y_k|x)=\frac{P(x|y_k)P(y_k)}{\sum_{k}P(x|y_k)P(y_k)}$$

---
* 分子中的$P(y_k)$是先验概率，根据训练集就可以计算出来
* 条件概率$P(x|y_k)=P(x_1,x_2,\cdots x_n|y_k)$，它的参数规模是指数数量级别的，假设第i维特征$x_i$可取值的个数有$S_i$个，类别取值个数为k个，那么参数的个数为$k\prod_{i=1}^{n}S_i$
    * 因此，**针对这个问题，朴素贝叶斯算法对条件概率分布做出了独立性的假设**，就是假设各个维度的特征$x_1,x_2,\cdots x_n$相互独立。
* 在特征条件独立假设的前提下，条件概率$$P(x|y_k)=P(x_1,x_2,\cdots x_n|y_k)=\prod_{i=1}^{n}P(x_i|y_k)$$这样，参数的规模就下降到$k\sum_{i=1}^{n}S_i$

至此就解决了先验概率$P(y_k)$和条件概率$P(x|y_k)$的求解问题。

---

$
\begin{align}
P(y_k|x)&=\frac{P(x|y_k)P(y_k)}{\sum_{k}P(x|y_k)P(y_k)}\\
        &=\frac{p(y_k)\prod_{i=1}^{n}P(x_i|y_k)}{\sum_{k}P(y_k)\prod_{i=1}^{n}P(x_i|y_k)}
\end{align}
$

因此朴素贝叶斯分类器可以表示为
$$f(x)=argmax_{y_k}P(y_k|x)=argmax_{y_k}\frac{p(y_k)\prod_{i=1}^{n}P(x_i|y_k)}{\sum_{k}P(y_k)\prod_{i=1}^{n}P(x_i|y_k)}$$

注意全加符号$\sum$，对于所有的$y_k$，上式中的分母值都是一样的，所以可以忽略分母部分:
$$f(x)=argmax_{y_k} p(y_k)\prod_{i=1}^{n}P(x_i|y_k)$$

最终归于$P(y_k), P(x_i|y_k)$的求解问题。

三种模型及编程实现

* 多项式模型 (Multinomial Model)

**NOTE:**
* **Training Set**: $\{ (x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N) \}$ 包含N条训练数据。
* $x_i=(x_i^{(1)}, x_i^{(2)}, \cdots, x_i^{(m)})^T$ m维向量
* $y_i\in \{ c_1, c_2, \cdots, c_k \}$

多项式模型 (Multinomial Model)

当特征是**离散**的时候，使用多项式模型。多项式模型在计算先验概率$P(y_k)$和条件概率$P(x_i|y_k)$时，会做一些**平滑处理**:

**先验概率 $P(y_k)\rightarrow P(y_i=c_k)$**:

$$P(y_k)=\frac{N_{y_k}+\alpha}{N+k\alpha}$$

$$P(y=c_k)=\frac{\sum_{i=1}^{N}I(y_i=c_k)+\alpha}{N+k\alpha}$$

* N: 样本总数
* k: 类别总数
* $N_{y_k}$: 类别为$y_i=c_k\rightarrow y_k$的样本数
* $\alpha$: 平滑值

**条件概率 $P(x^{(j)}_i|y_k)\rightarrow P(x_i^{(j)}=a_{ij}|y_i=c_k)$**:

$$P(x^{(j)}_i|y_k)=\frac{N_{x_i,y_k}+\alpha}{N_{y_k}+L_{j}\alpha}$$

$$P(x_i^{(j)}=a_{ij}|y_i=c_k)=\frac{\sum_{i=1}^{N}I(x^j_i=a_{ij}, y_i=c_k)+\alpha}{\sum_{i=1}^{N}I(y_i=c_k)+L_j\alpha}$$

* $N_{y_k}$: 类别为$y_k$的样本数
* $N_{y_k,x_i}$: 类别为$y_k$的样本中，第i维特征的值是$x_i$的样本数
* $L_j$: 第j维特征的最大取值数
* $\alpha$: 平滑值

**当α=1时，称作Laplace平滑; 当0<α<1时，称作Lidstone平滑; α=0时不做平滑**。

**Note**:

如果不做平滑，当某一维特征的值$x_i$没在训练样本中出现过时，会导致$P(x_i|y)k)=0$，从而导致后验概率为0。加上平滑就可以克服这个问题。

---

Example:

有如下训练数据，15个样本，2维特征$x^{(1)},x^{(2)}$，每个维度的最大取值种类数都为3种，2种类别-1，1。

给定测试样本$x=(2,S)^T$，判断其类别。

|$\space$|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|$x^{(1)}$|1|1|1|1|1|2|2|2|2|2|3|3|3|3|3|
|$x^{(2)}$|S|M|M|S|S|S|M|M|L|L|L|M|M|L|L|
|$Y\quad$|-1|-1|1|1|-1|-1|-1|1|1|1|1|1|1|1|-1|

理论分析

**Multinomial Model**: *Let $\alpha=1$*

1. 计算先验概率 $P(y_k)=\frac{\sum_{i=1}^{N}I(y_i=c_k)+\alpha}{N+k\alpha}$:

$P(Y=1)=\frac{9+1}{15+2\cdot1}=\frac{10}{17};\quad P(Y=-1)=\frac{6+1}{15+2\cdot1}=\frac{7}{17}$

2. 计算条件概率 $P(x^j_i|y_k)=\frac{\sum_{i=1}^{N} I(x^{j}_i=a_{ij},\quad y_i=c_k)+\alpha}{\sum_{i=1}^{N}I(y_i=c_k)+L_j\alpha}$:

$P(x_1=1|Y=1)=\frac{2+1}{9+3\cdot1}=\frac{3}{12}$

$P(x_1=2|Y=1)=\frac{4}{12}, P(x_1=3|Y=1)=\frac{5}{12}$

$P(x_1=1|Y=-1)=\frac{4}{9}, P(x_1=2|Y=-1)=\frac{3}{9}, P(x_1=3|Y=-1)=\frac{2}{9}$

$P(x_2=S|Y=1)=\frac{2}{12}, P(x_2=M|Y=1)=\frac{5}{12}, P(x_2=L|Y=1)=\frac{5}{12}$

$P(x_2=S|Y=-1)=\frac{4}{9}, P(x_2=M|Y=-1)=\frac{3}{9}, P(x_2=L|Y=-1)=\frac{2}{9}$

3. 对于$x=(2,S)^T$:

$P(y_k|x)=\frac{P(y_k,x)}{P(x)}=\frac{P(x|y_k)P(y_k)}{\sum_{k}P(x|y_k)P(y_k)}\rightarrow \frac{P(y_k)\prod P(x_i|y_k)}{\sum P(y_k)\prod P(x_i|y_k)}\rightarrow P(y_k)\prod_{i=1}^{2} P(x_i|y_k)$

$P(y_1|x)=P(Y=1)\cdot P(x_1=2|Y=1)\cdot P(x_2=S|Y=1)=\frac{10}{17}\cdot\frac{4}{12}\cdot\frac{2}{12}=0.0327$

$P(y_2|x)=P(Y=-1)\cdot P(x_1=2|Y=-1)\cdot P(x_2=S|Y=-1)=\frac{7}{17}\cdot\frac{3}{9}\cdot\frac{4}{9}=0.0610$

Thus $\Rightarrow\quad$ for $ x=(2,S)^T$, $y=-1$

编程实现 (in Python)

当给定训练集时，先计算出所有的先验概率$P(y_k)$和条件概率$P(x_i|y_k)$，然后把它们存起来（当成一个查找表）。当来一个测试样本时，计算它所有可能的后验概率$P(y_k|x)$，最大的那个对应的就是测试样本的类别，而后验概率的计算就是在查找表里查找需要的值。

以下代码就是根据这个思想来写的:

定义一个`MultinomialModel类`，它有两个主要的方法：`fit(X,y)`和`predict(X)`。
* fit方法其实就是训练，调用fit方法时，做的工作就是构建查找表。
* predict方法就是预测，调用predict方法时，做的工作就是求解所有后验概率并找出最大的那个。

此外，类的`构造函数__init__()`中，允许设定$\alpha$的值，以及设定先验概率的值。

---

**This Is TSET LINE**

---

```python
'''
Multinomial Model
'''

import numpy as np

# 类名通常是大写开头的单词，紧接着是(object)，表示该类是从哪个类继承下来的。通常，如果没有合适的继承类，就使用object类，这是所有类最终都会继承的类
class MultinomialModel(object):
    '''
    Naive bayes classifer for multinomial models
    The Multinomial Naive Bayes classifier is suitable for classification with discrete features

    Parameters
    ----------
    * alpha: float, optional(default=1.0)
        Setting alpha=0 for no smoothing
        Setting 0<alpha<1 is called Lidstone smoothing
        Setting alpha=1 is called Laplace smoothing

    * fit_prior: boolean
        Whether to learn class prior probabilities or not
        If False, a uniform prior will be used

    * class_prior: array-like, size(n_classes)
        Prior probabilities of the classes.
        If specified, the priors are not adjusted according to the data

    Attributes
    ----------
    * fix(X,y):
        X and y are array-like, represent features and labels
        call fit() method to train Naive Bayes classifier

    * predict(X):

    '''

    # init方法的第一个参数永远是self，表示创建的实例本身
    # 因此，在init方法内部，就可以把各种属性绑定到self，因为self就指向创建的实例本身
    # 和普通的函数相比，在类中定义的函数只有一点不同，就是第一个参数永远是实例变量self，并且，调用时，不用传递该参数
    def __init__(self, alpha=1.0, fit_prior=True, class_prior=None):
        self.alpha = alpha
        self.fit_prior = fit_prior
        self.class_prior = class_prior
        self.classes = None
        self.conditional_prob = None

    def _calculate_feature_prob(self, feature):  # feature -> (x^{j}_{i=1...N} & y_i=c_k)
        unique_features = np.unique(feature) # feature内不重复的数量
        total_num = float(len(feature)) # y_i = c_k 的总数
        value_prob = {}
        for k in unique_features:
            value_prob[k] = ((np.sum(np.equal(feature, k)) + self.alpha) / (total_num + len(unique_features)*self.alpha))
            #
        return value_prob
        # like value_prob {c_1: 0.3, c_2: 0.2, ...}

    def fit(self, X, y):
        self.classes = np.unique(y) # 不同的类别k列表

        # calculate class prior probabilities: P(y=c_k)
        if self.class_prior == None:
            class_num = len(self.classes) # 类别k的总数 即 N 样本总数
            if not self.fit_prior: # if fit_prior = False ---> uniform prior
                self.class_prior = [1.0/class_num for _ in range (class_num)] # uniform prior
            else:
                self.class_prior = []
                sample_num = float(len(y)) # 样本总数
                for c in self.classes:
                    c_num = np.sum(np.equal(y,c)) # {y_i=c}的数量
                    self.class_prior.append((c_num+self.alpha)/(sample_num+class_num*self.alpha)) # 先验概率 P(y_i=c_k)

        # calculate conditional probability: P(x_i|y=c_k)
        self.conditional_prob={} # like {c_0: {x_0: {value0: 0.1, value1: 0.3, ...}, x_1:{...}, ...}, c_1: {...}, ...}
        for c in self.classes: # c_i
            self.conditional_prob[c] = {} # {..., c_i:{...}, ...}
            for j in range(len(X[0])): # for each feature x^{j}
                feature = X[np.equal(y, c)][:, j] # (x^{j}_{i=1...N} & y_i=c_k) -> feature
                self.conditional_prob[c][j] = self._calculate_feature_prob(feature)
        return self

    # given values_prob {value0: 0.1, value1: 0.3, ...} and target_value
    # return the probability of target_value
    def _get_xi_prob(self, values_prob, target_value):
        return values_prob[target_value]

    # predict a single sample based on (class_prior, conditional_prob)
    def _predict_single_sample(self, x):
        label = -1
        max_posterior_prob = 0

        # for each category, calculate its posterior probability: class_prior * conditional_prob
        for c_index in range(len(self.classes)):
            current_class_prior = self.class_prior[c_index]
            current_conditional_prob = 1.0
            feature_prob = self.conditional_prob[self.classes[c_index]]
            i = 0
            for feature_i in feature_prob.keys():
                current_conditional_prob *= self._get_xi_prob(feature_prob[feature_i], x[i])
                i+=1

            # compare posterior probability and update max_posterior_prob, label
            if current_class_prior * current_conditional_prob > max_posterior_prob:
                max_posterior_prob = current_class_prior * current_conditional_prob
                label = self.classes[c_index]
        return label

    # predict samples (also single sample)
    def predict(self, X):
        if X.ndim == 1:
            return self._predict_single_sample(X)
        else:
            # classify each sample
            labels = []
            for i in range(X.shape[0]):
                label = self._predict_single_sample(X[i])
                labels.append(label)
            return labels

```

```python
import numpy as np

# 用 4, 5, 6 代替 S, M, L
X = np.array([
    [1,1,1,1,1,2,2,2,2,2,3,3,3,3,3],
    [4,5,5,4,4,4,5,5,6,6,6,5,5,6,6]
])

X = X.T
y = np.array([-1,-1,1,1,-1,-1,-1,1,1,1,1,1,1,1,-1])

nb = MultinomialModel(alpha=1.0, fit_prior=True)
nb.fit(X,y)

print(nb.predict(np.array([2,4])))
```