Probabilistic Programming and Bayesian Methods for Hackers
========

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

#### Looking for a printed version of Bayesian Methods for Hackers?

_Bayesian Methods for Hackers_ is now a published book by Addison-Wesley, available on [Amazon](http://www.amazon.com/Bayesian-Methods-Hackers-Probabilistic-Addison-Wesley/dp/0133902838)!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty*. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist*, known as the more *classical* version of statistics, assumes that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

#### Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results (often) align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty* that reflects the instability of statistical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?" )

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to } )
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
"""
The book uses a custom matplotlibrc file, which provides the unique styles for
matplotlib plots. If executing this book, and you wish to use the book's
styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the
       book's styles/ dir. See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to
       update the styles in only this notebook. Try running the following code:

        import json, matplotlib
        s = json.load( open("../styles/bmh_matplotlibrc.json") )
        matplotlib.rcParams.update(s)

"""

# The code below can be passed over, as it is currently not important, plus it
# uses advanced topics we have not covered yet. LOOK AT PICTURE, MICHAEL!
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials) - 1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = np.linspace(0, 1, 50)
plt.plot(p, 2 * p / (1 + p), color="#348ABD", lw=3)
# plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2 * (0.2) / 1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Is my code bug-free?")
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1. / 3, 2. / 3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0 + 0.25, .7 + 0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.ylim(0,1)
plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

#### Expected Value
Expected value (EV) is one of the most important concepts in probability. The EV for a given probability distribution can be described as "the mean value in the long run for many repeated samples from that distribution." To borrow a metaphor from physics, a distribution's EV as like its "center of mass." Imagine repeating the same experiment many times over, and taking the average over each outcome. The more you repeat the experiment, the closer this average will become to the distributions EV. (side note: as the number of repeated experiments goes to infinity, the difference between the average outcome and the EV becomes arbitrarily small.)

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

import scipy.stats as stats
a = np.arange(16)
poi = stats.poisson
lambda_ = [1.5, 4.25]
colours = ["#348ABD", "#A60628"]

plt.bar(a, poi.pmf(a, lambda_[0]), color=colours[0],
        label="$\lambda = %.1f$" % lambda_[0], alpha=0.60,
        edgecolor=colours[0], lw="3")

plt.bar(a, poi.pmf(a, lambda_[1]), color=colours[1],
        label="$\lambda = %.1f$" % lambda_[1], alpha=0.60,
        edgecolor=colours[1], lw="3")

plt.xticks(a + 0.4, a)
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values")
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
a = np.linspace(0, 4, 100)
expo = stats.expon
lambda_ = [0.5, 1]

for l, c in zip(lambda_, colours):
    plt.plot(a, expo.pdf(a, scale=1. / l), lw=3,
             color=c, label="$\lambda = %.1f$" % l)
    plt.fill_between(a, expo.pdf(a, scale=1. / l), color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0, 1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to PyMC, a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

Introducing our first hammer: PyMC
-----

PyMC is a Python library for programming Bayesian analysis [3]. It is a fast, well-maintained library. The only unfortunate part is that its documentation is lacking in certain areas, especially those that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why PyMC is so cool.

We will model the problem above using PyMC. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components. Model components are first-class primitives within the PyMC framework.

B. Cronin [5] has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

PyMC code is easy to read. The only novel thing should be the syntax, and I will interrupt the code to explain individual sections. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables:

```python
import pymc as pm

alpha = 1.0 / count_data.mean()  # Recall count_data is the
                               # variable that holds our txt counts
lambda_1 = pm.Exponential("lambda_1", alpha)
lambda_2 = pm.Exponential("lambda_2", alpha)

tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data)
```

In the code above, we create the PyMC variables corresponding to $\lambda_1$ and $\lambda_2$. We assign them to PyMC's *stochastic variables*, so-called because they are treated by the back end as random number generators. We can demonstrate this fact by calling their built-in `random()` methods.

```python
print("Random output:", tau.random(), tau.random(), tau.random())
```

    Random output: 21 31 29

```python
@pm.deterministic
def lambda_(tau=tau, lambda_1=lambda_1, lambda_2=lambda_2):
    out = np.zeros(n_count_data)
    out[:tau] = lambda_1  # lambda before tau is lambda1
    out[tau:] = lambda_2  # lambda after (and including) tau is lambda2
    return out
```

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

`@pm.deterministic` is a decorator that tells PyMC this is a deterministic function. That is, if the arguments were deterministic (which they are not), the output would be deterministic as well. Deterministic functions will be covered in Chapter 2.

```python
observation = pm.Poisson("obs", lambda_, value=count_data, observed=True)

model = pm.Model([observation, lambda_1, lambda_2, tau])
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `value` keyword. We also set `observed = True` to tell PyMC that this should stay fixed in our analysis. Finally, PyMC wants us to collect all the variables of interest and create a `Model` instance out of them. This makes our life easier when we retrieve the results.

The code below will be explained in Chapter 3, but I show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which I also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```python
# Mysterious code to be explained in Chapter 3.
mcmc = pm.MCMC(model)
mcmc.sample(40000, 10000, 1)
```

     [-----------------100%-----------------] 40000 of 40000 complete in 5.8 sec

```python
lambda_1_samples = mcmc.trace('lambda_1')[:]
lambda_2_samples = mcmc.trace('lambda_2')[:]
tau_samples = mcmc.trace('tau')[:]
```

```python
figsize(12.5, 10)
# histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", normed=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data) - 20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability");
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
# type your code here.

lambda_1_samples.mean()
```

    17.759989917035039

```python
lambda_2_samples.mean()
```

    22.738906564964601

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
# type your code here.

np.mean(lambda_1_samples/lambda_2_samples)
```

    0.78225828246937623

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC part. Just consider all instances where `tau_samples < 45`.)

```python
# type your code here.
np.mean(lambda_1_samples[tau_samples[tau_samples<45]])
```

    16.821068474609294

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg/).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Patil, A., D. Huard and C.J. Fonnesbeck. 2010.
PyMC: Bayesian Stochastic Modelling in Python. Journal of Statistical
Software, 35(4), pp. 1-81.
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, exp
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

```python
u, u1, u2, t, a, b, c = symbols('u u1 u2 t a b c')
```

# Differential equations
# Exponential  eAt of a matrix

## Differential equations (ordinary only)

* A differential equation moves on from the previous lecture's difference equation which had finite steps to continuously changing systems (here in the time parameter *t*)
* A differential equation included a function and its derivative(s)
* It has and order based on the highest derivative that appears
* Here we are only concerned with differential equation with constant coefficients
* The simplest differential equation is the following
$$ \frac{dy}{dt}={a}{y}\left(t\right) $$
* It is simply solved in the following manner (which gives us some insight into the general solution for these equations)
$$ \frac { dy }{ dt } =ay\\ \frac { 1 }{ y } dy=adt\\ \int { \frac { 1 }{ y }  } dy=a\int {  } dt\\ \ln { \left| y \right|  } =a\left( t+{ c }_{ 1 } \right) \\ { e }^{ \ln { \left| y \right|  }  }={ e }^{ at+a{ c }_{ 1 } }\\ y={ e }^{ at }{ e }^{ a{ c }_{ 1 } }\\ y=c{ e }^{ at } $$
* We can solve for the constant(s) if we have values for the initial condition (usually *t*=0) for *y*(t) and all its derivatives (called *initial value problems*)
* We can write a system of differential equations in matrix form
$$ { y }_{ 1 }^{ ' }=3{ y }_{ 1 }\\ { y }_{ 2 }^{ ' }=-2{ y }_{ 2 }\\ { y }_{ 3 }^{ ' }=6{ y }_{ 3 }\\ \therefore \quad \begin{bmatrix} { y }_{ 1 }^{ ' } \\ { y }_{ 2 }^{ ' } \\ { y }_{ 3 }^{ ' } \end{bmatrix}=\begin{bmatrix} 3 & 0 & 0 \\ 0 & -2 & 0 \\ 0 & 0 & 6 \end{bmatrix}\begin{bmatrix} { y }_{ 1 } \\ { y }_{ 2 } \\ { y }_{ 3 } \end{bmatrix} $$

* Rewriting the above, we consider the following differential equations in this lecture
$$ \frac { d\underline { u }  }{ dt } =A\underline { u }  $$

* So suppose we have these two differential equations
$$ \frac{{d}\underline{u}_{1}}{{d}{t}} = -\underline{u}_{1}+2\underline{u}_{2} \\ \frac{{d}\underline{u}_{2}}{{d}{t}} = \underline{u}_{1}-2\underline{u}_{2} $$
* The intial consitions are given by the following
$$ \underline{u}\left({0}\right)=\begin{bmatrix}1\\0\end{bmatrix} $$

* We can write it as A**u**

```python
A = Matrix([[-1, 2], [1, -2]])
A
```

$$\left[\begin{matrix}-1 & 2\\1 & -2\end{matrix}\right]$$

```python
u_vect = Matrix([u1, u2]) # u is now a sympy mathematical symbol
# Have to use another variable name, i.e. u_vect
u_vect
```

$$\left[\begin{matrix}u_{1}\\u_{2}\end{matrix}\right]$$

* Multiplying this A**u** brings you back to the two linear equations

```python
A * u_vect
```

$$\left[\begin{matrix}- u_{1} + 2 u_{2}\\u_{1} - 2 u_{2}\end{matrix}\right]$$

```python
A.eigenvals()
```

$$\begin{Bmatrix}-3 : 1, & 0 : 1\end{Bmatrix}$$

```python
A.eigenvects() # The results give the two eigenvectors in the following format
# (eigenvalue, no of eigenvectors, eigenvector)
```

$$\begin{bmatrix}\begin{pmatrix}-3, & 1, & \begin{bmatrix}\left[\begin{matrix}-1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}2\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

```python
S, D = A.diagonalize() # For interest sake we get the matrix of eigenvectors and the diagonal matrix
S, D
```

$$\begin{pmatrix}\left[\begin{matrix}-1 & 2\\1 & 1\end{matrix}\right], & \left[\begin{matrix}-3 & 0\\0 & 0\end{matrix}\right]\end{pmatrix}$$

* To complete the solution now, we note that there are two eigenvalues, which will give us the following
    * Two constants
    * Two exponent to the power eigenvalue times *t*
    * Two eigenvectors
* It is written like this, with **x**i denoting an eigenvector
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ { \lambda  }_{ 1 }t }{ \underline { x }  }_{ 1 }+{ c }_{ 2 }{ e }^{ { \lambda  }_{ 2 }t }{ \underline { x }  }_{ 2 } $$
* This makes our solution as follows
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ -3t }\begin{bmatrix} -1 \\ 1 \end{bmatrix}+{ c }_{ 2 }{ e }^{ \left( 0 \right) t }\begin{bmatrix} 2 \\ 1 \end{bmatrix}\\ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ -3t }\begin{bmatrix} -1 \\ 1 \end{bmatrix}+{ c }_{ 2 }\begin{bmatrix} 2 \\ 1 \end{bmatrix} $$
* There is clearly a constant term and a term that approaches zero at *t* approaches infinity

* Writing this as two separate equations we have the following
$$ { u }_{ 1 }\left( t \right) =-{ c }_{ 1 }{ e }^{ -3t }+2{ c }_{ 2 }\\ { u }_{ 2 }\left( t \right) ={ c }_{ 1 }{ e }^{ -3t }+{ c }_{ 2 }  $$

* Using the initial conditions we can solve for *c*i
$$ { u }_{ 1 }\left( 0 \right) =-{ c }_{ 1 }{ e }^{ -3\left( 0 \right)  }+2{ c }_{ 2 }=1\\ { u }_{ 2 }\left( 0 \right) ={ c }_{ 1 }{ e }^{ -3\left( 0 \right)  }+{ c }_{ 2 }=0\\ -{ c }_{ 1 }+2{ c }_{ 2 }=1\\ { c }_{ 1 }+{ c }_{ 2 }=0 $$

* Let's use an augmented matrix and Gauss-Jordan elimination to solve for the two constants (or at least the python™ equivalent)

```python
C = Matrix([[-1, 2, 1], [1, 1, 0]])
C.rref()
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 0 & - \frac{1}{3}\\0 & 1 & \frac{1}{3}\end{matrix}\right], & \begin{bmatrix}0, & 1\end{bmatrix}\end{pmatrix}$$

* Which gives us the final solution
$$  { u }_{ 1 }\left( t \right) =\frac { 1 }{ 3 } { e }^{ -3t }+\frac { 2 }{ 3 } \\ { u }_{ 2 }\left( t \right) =\frac { -1 }{ 3 } { e }^{ -3t }+\frac { 1 }{ 3 }   $$

* Just to remind ourselves about the previous lecture where we had difference equations and had something like the following
$$ \underline{u}={c}_{1}{\lambda}_{1}^{k}{\underline{x}}_{1}+{c}_{2}{\lambda}_{2}^{k}{\underline{x}}_{2} $$
* Which is for finite steps, i.e. stepping by one
$$ \underline{u}_{k+1}={A}\underline{u}_{k} $$

## What can the eigenvalues tell us about these equations as *t* approaches ∞

* If both eigenvalues (real parts) are negative, the equation **t**(t) approaches **0** (called *stability*)
* If one eigenvalue (real part) is zero and the others (real parts) are less than one, the equations reach is specific value (called a *steady state*)
* If any eigenvalue (real parts) is larger than zero, the equations approach ±∞

## What can the matrix A2×2 tell us about the eigenvalues (and then what happens when *t* approaches ∞)

* The trace is equal to the sum of the eigenvalues
* The determinant is the product of the eigenvalues

* If the trace is negative **and** the determinant positive, we will have stability

## Using diagonalization

* Consider the following derivation
$$ \frac { d\underline { u }  }{ dt } =A\underline { u } \\ \because \quad A\underline { u } =S\underline { v } \\ S\frac { d\underline { v }  }{ dt } =S\underline { v } \\ \frac { d\underline { v }  }{ dt } =S\underline { v } { S }^{ -1 }=\Lambda \underline { v }  $$

* From this we have the following
$$ \underline { v } \left( t \right) ={ e }^{ \Lambda t }\underline { v } \left( 0 \right) \\ \underline { u } \left( t \right) =S{ e }^{ \Lambda t }{ S }^{ -1 }\underline { u } \left( 0 \right) \\ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$

## Matrix exponential *e*At

* How do we calculate a matrix as a power?
* Consider Taylor series expansion
$$ {e}^{At}={I}+{At} + \frac{{\left(At\right)}^{2}}{2!}+\frac{{\left(At\right)}^{3}}{3!}+\dots+\frac{{\left(At\right)}^{n}}{n!} $$
* This comes from the following
$$ { e }^{ x }=\sum _{ n=0 }^{ \infty  }{ \frac { { x }^{ n } }{ n! }  }  $$
* As the denominator increases the *n*th term approaches 0
* Remember also this (geometric) series (just for fun)
$$ \frac { 1 }{ 1-x } =\sum _{ 0 }^{ \infty  }{ { x }^{ n } } \\ { \left( I-At \right)  }^{ -1 }=I+At+{ \left( At \right)  }^{ 2 }+\dots { \left( At \right)  }^{ n } $$
* This will blow up unless then eigenvalues of A are less than 1

* Now, let calculate *e*At, remembering the following
$$ {A}^{k}={S}{\Lambda}^{k}{S}^{-1} $$
$$ { e }^{ At }=\sum _{ n=0 }^{ \infty  }{ \frac { \left( S{ \Lambda  }^{ n }{ S }^{ -1 } \right) ^{ n }{ t }^{ n } }{ n! }  }  $$

* We thus have the following
$$ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$
* This is with the assumption that A can be diagonalized (otherwise we will have to use the infinite series (above)
$$ {e}^{At}={I}+{At} + \frac{{\left(At\right)}^{2}}{2!}+\frac{{\left(At\right)}^{3}}{3!}+\dots+\frac{{\left(At\right)}^{n}}{n!} $$

* Remember that Λ is a diagonal matrix and therefor we would have the following
$$ { e }^{ \Lambda t }=\begin{bmatrix} { e }^{ { \lambda  }_{ 1 }t } & 0 & 0 & 0 \\ 0 & { e }^{ { \lambda  }_{ 2 }t } & 0 & 0 \\ \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & { e }^{ { \lambda  }_{ n }t } \end{bmatrix} $$

* The S and S-1 matrices are stable, it is therefor the Λ matrix that provides an approach to zero as *t* approaches ∞
* This is achieved by every λi having a real part less than zero

* The powers of A go to zero if the absolute value of the real part of all the λi-values is less than 1

+ Let's consider this example

$$ \underline { y } ''+b\underline { y } '+{k}\underline{y}=\underline { 0 }  $$

* We have to create a system of two first-order equations
$$ \underline{u}=\begin{bmatrix} y' \\ y \end{bmatrix} \\ \underline{u}'=\begin{bmatrix}{y}''\\{y}'\end{bmatrix}=\begin{bmatrix} -{b} & -{k} \\ 1 & 0 \end{bmatrix}\begin{bmatrix} {y}' \\ {y} \end{bmatrix}$$

## Example problems

### Example problem 1

* Find the general solutions, the matrix A, and the first column of *e*At of the following third-order, ordinary, homogeneous differential equation with constant coefficients
$$ \frac { { d }^{ 3 }y }{ d{ t }^{ 3 } } +2\frac { { d }^{ 2 }y }{ d{ t }^{ 2 } } -\frac { dy }{ dt } -2y=0 $$

#### Solution

* Since the differential equation is third order, we need to create three first-order differential equations
$$ \frac { { d }^{ 3 }y }{ d{ t }^{ 3 } } +2\frac { { d }^{ 2 }y }{ d{ t }^{ 2 } } -\frac { dy }{ dt } -2y=0\\ \therefore \quad y'''=-2y''+y'+2\\ \underline { u } =\begin{bmatrix} y'' \\ y' \\ y \end{bmatrix}\\ \underline { u } '=\begin{bmatrix} y''' \\ y'' \\ y' \end{bmatrix}\\ \underline { u } '=A\underline { u } =\begin{bmatrix} -2 & 1 & 2 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}\begin{bmatrix} y'' \\ y' \\ y \end{bmatrix} $$

* Notice that when we do the last matrix multiplication, we get exactly what we need
$$  \underline { u } '=\begin{bmatrix} y''' \\ y'' \\ y' \end{bmatrix}=\begin{bmatrix} -2y''+y'+2y \\ y'' \\ y' \end{bmatrix} $$

* We now have the matrix A and we can calculate *e*At if A is diagonalizable

```python
A = Matrix([[-2, 1, 2], [1, 0, 0], [0, 1, 0]])
A
```

$$\left[\begin{matrix}-2 & 1 & 2\\1 & 0 & 0\\0 & 1 & 0\end{matrix}\right]$$

```python
S, D = A.diagonalize()
S, D
```

$$\begin{pmatrix}\left[\begin{matrix}4 & 1 & 1\\-2 & -1 & 1\\1 & 1 & 1\end{matrix}\right], & \left[\begin{matrix}-2 & 0 & 0\\0 & -1 & 0\\0 & 0 & 1\end{matrix}\right]\end{pmatrix}$$

* We can calculate *e*At from the following
$$ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$

* Remember that Λ is a diagonal matrix and therefor we would have the following
$$ { e }^{ \Lambda t }=\begin{bmatrix} { e }^{ { \lambda  }_{ 1 }t } & 0 & 0 & 0 \\ 0 & { e }^{ { \lambda  }_{ 2 }t } & 0 & 0 \\ \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & { e }^{ { \lambda  }_{ n }t } \end{bmatrix} $$

```python
A.eigenvals()
```

$$\begin{Bmatrix}-2 : 1, & -1 : 1, & 1 : 1\end{Bmatrix}$$

* This gives us the three eigenvalues from which we can create the diagonal matrix *e*Λt

```python
e_Lamda_t = Matrix([[exp(-2 * t), 0, 0], [0, exp(-t), 0], [0, 0, exp(t)]])
e_Lamda_t
```

$$\left[\begin{matrix}e^{- 2 t} & 0 & 0\\0 & e^{- t} & 0\\0 & 0 & e^{t}\end{matrix}\right]$$

* We have to multiply the following three matrices (with which python is not comfortable, so I'll do it in two steps)

```python
S, e_Lamda_t, S.inv()
```

$$\begin{pmatrix}\left[\begin{matrix}4 & 1 & 1\\-2 & -1 & 1\\1 & 1 & 1\end{matrix}\right], & \left[\begin{matrix}e^{- 2 t} & 0 & 0\\0 & e^{- t} & 0\\0 & 0 & e^{t}\end{matrix}\right], & \left[\begin{matrix}\frac{1}{3} & 0 & - \frac{1}{3}\\- \frac{1}{2} & - \frac{1}{2} & 1\\\frac{1}{6} & \frac{1}{2} & \frac{1}{3}\end{matrix}\right]\end{pmatrix}$$

```python
first_part = S * e_Lamda_t
first_part
```

$$\left[\begin{matrix}4 e^{- 2 t} & e^{- t} & e^{t}\\- 2 e^{- 2 t} & - e^{- t} & e^{t}\\e^{- 2 t} & e^{- t} & e^{t}\end{matrix}\right]$$

```python
first_part * S.inv()
```

$$\left[\begin{matrix}\frac{e^{t}}{6} - \frac{e^{- t}}{2} + \frac{4}{3} e^{- 2 t} & \frac{e^{t}}{2} - \frac{e^{- t}}{2} & \frac{e^{t}}{3} + e^{- t} - \frac{4}{3} e^{- 2 t}\\\frac{e^{t}}{6} + \frac{e^{- t}}{2} - \frac{2}{3} e^{- 2 t} & \frac{e^{t}}{2} + \frac{e^{- t}}{2} & \frac{e^{t}}{3} - e^{- t} + \frac{2}{3} e^{- 2 t}\\\frac{e^{t}}{6} - \frac{e^{- t}}{2} + \frac{1}{3} e^{- 2 t} & \frac{e^{t}}{2} - \frac{e^{- t}}{2} & \frac{e^{t}}{3} + e^{- t} - \frac{1}{3} e^{- 2 t}\end{matrix}\right]$$

* We still need to write our general solution
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ { \lambda  }_{ 1 }t }{ \underline { x }  }_{ 1 }+{ c }_{ 2 }{ e }^{ { \lambda  }_{ 2 }t }{ \underline { x }  }_{ 2 }+{ c }_{ 3 }{ e }^{ { \lambda  }_{ 3 }t }{ \underline { x }  }_{ 3 } $$

```python
A.eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}-2, & 1, & \begin{bmatrix}\left[\begin{matrix}4\\-2\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}-1, & 1, & \begin{bmatrix}\left[\begin{matrix}1\\-1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}1, & 1, & \begin{bmatrix}\left[\begin{matrix}1\\1\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

* With these eigenvalues and eigenvectors we get
$$ \underline { u } \left( t \right) =\begin{bmatrix} { y }'' \\ y' \\ y \end{bmatrix}={ c }_{ 1 }{ e }^{ -2t }\begin{bmatrix} 4 \\ -2 \\ 1 \end{bmatrix}+{ c }_{ s }{ e }^{ -t }\begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix}+{ c }_{ 3 }{ e }^{ t }\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\\ \therefore \quad y\left( t \right) ={ c }_{ 1 }{ e }^{ -2t }+{ c }_{ 2 }{ e }^{ -t }+{ c }_{ 3 }{ e }^{ t } $$

### Example problem 2

* Solve the following second-order ordinary differential equation
$$ y''-y'-6y=0 $$

#### Solution

* We need to create two first-order equations
$$ \therefore \quad y''=y'+6y\\ \because \quad \underline { u } \left( t \right) =\begin{bmatrix} y' \\ y \end{bmatrix}\\ \underline { u } '\left( t \right) =\begin{bmatrix} y'' \\ y' \end{bmatrix}\\ \therefore \quad \underline { u } '\left( t \right) =\begin{bmatrix} 1 & 6 \\ 1 & 0 \end{bmatrix}\underline { u } \left( t \right)  $$

```python
A = Matrix([[1, 6], [1, 0]])
A
```

$$\left[\begin{matrix}1 & 6\\1 & 0\end{matrix}\right]$$

```python
A.eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}-2, & 1, & \begin{bmatrix}\left[\begin{matrix}-2\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}3, & 1, & \begin{bmatrix}\left[\begin{matrix}3\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

* The solution is thus as follows
$$ {y}\left({t}\right)={c}_{1}{e}^{-2t}+{c}_{2}{e}^{3t} $$

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_23_Differential_equations_Exponential_of_a_matrix.ipynb


```python
# %load ../../preconfig.py
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['axes.grid'] = False

import numpy as np
import pandas as pd
#import itertools

import networkx as nx

import string

import logging
logger = logging.getLogger()
```

10 Mining Social-Network Graphs
=================
1. how to identify "communities"?
   communities: strong connections, usually overlap.

2. explore efficient algorithms for discovering other properities of graphs.

### 10.1 Social Networks as Graphs
#### 10.1.1 What is a Social Network?
The essential characteristics of a social network are:

1. There is a collection of entities that participate in the network.

2. There is at least one relationship between entities of the network.

   + all-or-nothing: friends of Facebook.

   + discrete degree: friends, family as in Google plus.

   + real number: the times of talking.

3. There is an assumption of nonrandomness or locality.
   If A is related to both B and C, then there is a higher probability than average that B and C are related.

#### 10.1.2 Social Networks as Graphs
social graph: The entities are the nodes, and an edge connects two nodes if the nodes are realted by the relationship that characterizes the network.

```python
plt.imshow(plt.imread('./res/fig10_1.png'))
```

Is Fig 10.1 typical of a social network, in the sense that it exhibits locality of relationships?

Suppose X, Y and Z are nodes of Fig 10.1, with edges between X and Y and also between X and Z.

1. What would we *expect* the probability of an edge between Y and Z to be?
   The graph has 9 edges out of the $C_7^2 = 21$ pairs of nodes.

   Since we already have (X,Y) and (X,Z), the probability of an edge (Y,Z) is $(9-2)/(21-2) = 0.368$.  If the graph were large enough, that probability would be very close to the fraction of the pairs of nodes that have edges between them, i.e., $9/21 = 0.429$.

2. Then, we must compute the probability that $(Y,Z)$ exists in Fig 10.1, given that edges $(X,Y)$ and $(Y,Z)$ exist.
   We count the pairs of nodes that could be $Y$ and $Z$, without worrying about which node is $Y$ and which is $Z$.

   If $X$ is $A$, $(B,C)$ contributes one positive example. All details are in the table following. In all, the fraction of times the third edge exists in thus $9 / 16 = 0.563 > 0.368$.

| node | pos | neg |
|------|-----|-----|
| A | 1 | 0|
| C | 1 | 0|
| E | 1 | 0|
| G | 1 | 0|
| F | 2 | 1|
| B | 1 | 2|
| D | 2 | 4|
|sum| 9 | 7| 16 |

#### 10.1.3 Varieties of Social Networks
1. Telephone Networks

2. Email Networks

3. Collaboration Networks
   + Wikipedia: articles and editors.
   + published research papers and authors.

4. Other example
   + information networks (patents)
   + infrastruture networks (roads)
   + biological networks (genes)
   + product co-purchasing networks (Groupon)

### 10.1.4 Graphs with Several Node Types
There are other social phenomena that involve entities of different types. The natural way to represent such information is as a $K$-partite graph for some $k > 1$. In general, a $k$-partite graph consists of $k$ disjoint sets of nodes, with no edges between nodes of the same set.

Eg:
deli.cio.cu: there are three different kinds of entites: users, tags and pages.

```python
plt.imshow(plt.imread('./res/fig10_2.png'))
```

`#exercise`

### 10.2 Clustering of Social-Network Graphs
clustering of the graph as a way to identify communities.

#### 10.2.1 Distance Measures for Social-Network Graphs
distance measure:
\begin{align}
    d(x,y) =
    \begin{cases}
         0 \text{ or } 1 \text{ or } 1 & \text{if edge} (x,y) {exists} \\
         1 \text{ or } \infty \text{ or } 1.5 & \text{otherwise}
    \end{cases}
\end{align}

#### 10.2.2 Applying Standard Clustering Methods
They are generally unsuitable for the problem of clustering social-network graphs.

#### 10.2.3  Betweenness: define
*betweenness* of an edge $(a,b)$:
the number of pairs of nodes $x$ and $y$ such that $(a,b)$ lies on the shortest path between $x$ and $y$. And it's credicted with the fraction of all shortest paths between $x$ and $y$ if existed.

A high score indicates that $(a,b)$ runs between two different communities, namely, $a$ and $b$ do no belong to the same community.

#### 10.2.4 The Girvan-Newman Algorithm: calculate betweenness
The algorithm is aimed to calculate the number of shortest paths going through each eage.

1) Starting at a node $X$, perform a breadth-first search (BFS) of the graph to label levels of each node.
   + DAG edges: edges between levels.

   + If there is a DAG edge $(Y,Z)$, where $Y$ is at the level about $Z$, then we shall call $Y$ a *parent* of $Z$ and $Z$ a *child* of $Y$.

2) to label each node by the number of shortest paths that reach it from the root.
   + Start by labeling the root 1.
   + Then, from the top down, label each node $Y$ by the sum of the labes of its parents.

```python
plt.imshow(plt.imread('./res/fig10_4.png'))
```

3) to calculate for each edge $e$ the sum over all nodes $Y$ of the fraction of shortest paths from the root $X$ to $Y$ that go through $e$.
   + from the bottom up.

   1. Each leaf in the DAG gets a credit of 1.

   2. Each node that is not a leaf gets a credit = 1 + the sum of the credits of the DAG edges from that node to its child nodes.

   3. credit for $(Y_i, Z)$ is:
   $$\text{credit}(Y_i, Z) = \text{credit}(Z) \frac{p_i}{\sum_{j=1}^k p_i}$$
   where $Y_1, Y_2, \dotsc, Y_k$ are the parents of $Z$, and $p_i$ is the number of the shorest paths from the root to $Y_i$, namely, the labels in Step 2).

```python
plt.imshow(plt.imread('./res/fig10_6.png'))
```

4) After performing the credit calculation with *each node* as the root,
   1. we sum the credits for each edge.

   2. And then divive the credit for each edge by 2, as each shortest path will have been discovered twice.

   3. Then we get the the true betweenness.

#### 10.2.5 Using Betweenness to Find Communities
It's a process of edge removal:

1. Start with the graph and all its edges;

2. then remove edges with the highest betweenness, until the graph has broken into a suitable number of connected components.

```python
plt.imshow(plt.imread('./res/fig10_7.png'))
plt.figure()
plt.imshow(plt.imread('./res/fig10_8.png'))
```

Speeding up the Betweenness Calculation:
If the large is large, we can pick a subset of the nodes at random and use these as the root of breadth-first searches, we can get an approximation to the betweenness of each edge that will serve in most applications.

**cons**:
It is not possible to place an individual in two different communities, and everyone is assigned to a community.

```python
# Exercises for Section 10.2
```

### 10.3 Direct Discovery of Communities
#### 10.3.1 Finding Cliques
NP-complete: finding a large *clique* (a sef of nodes with edges between any two of them).

#### 10.3.2 Complete Bipartite Graphs
A *complete bipartite graph* $K_{s,t}$ consists of $s$ nodes on one side and $t$ nodes on the other side, with all $st$ possible edges between the nodes of one side and the other present.

**Idea**:

1. While it is not possible to guarantee that a graph with many edges necessarily has a large clique, it is possible to guarantee that a bipartite graph with many edges has a large complete bipartite subgraph.

2. We can regard a complete bipartite subgraph as the nucleus of a community, and add to it nodes with many edges to existing members of the community.

   + If its nodes is consisted of two or more types, construct bipartite graphs directly.

   + If all nodes have the same type, divide the node into two equal groups at random.

#### 10.3.3 Finding Complete Bipartite Subgraphs
It's possible to view the problem of finding instance of $K_{s,t}$ within $G$ as one of finding frequent itemsets.

1. "items" - left side.

2. "baskets" - right side.
    The members of the basket for node $v$ are the nodes of the left side to which $v$ is connected.

3. Let the support threshold be $s$, the number of nodes that the instance of $K_{s,t}$ has on the right side.

the problem of find $K_{s,t}$ $\to$ finding frequent itemsets $F$ of size $t$.

#### 10.3.4 Why Complete Bipartite Graphs Must Exist
Assume:

1. the graph $G$ has $n$ nodes on the left and another $n$ nodes on the right.

2. let $d$ be the average degree of all nodes.

3. the degree of the $i$th node on the right is $d_i$.

Proof:

1. The total contribution of the $n$ nodes on the right is $\sum_i \binom{d_i}{t} \geq n \binom{d}{t}$ .

2. The number of itemsets of size $t$ is $\binom{n}{t}$.

3. Thus, the average count of an itemset of size $t$ is $n \binom{d}{t} / \binom{n}{t}$ ???

\begin{align}
    n \binom{d}{t} / \binom{n}{t} &= n \frac{d!}{(d-t)! t!} \frac{t! (n-t)!}{n!} \\
        &= n \frac{d (d-1) \dotso (d-t+1)}{n (n-1) \dotso (n-t+1)} \\
        &\approx n \frac{d^t}{n^t} \quad \text{when } n >> d >> t
\end{align}

That is, if there is a community with $n$ nodes on each side, the average degree of the nodes is $d$, and $n(d/n)^t \geq s$, then this community is guaranteed to have a complete bipartite subgraph $K_{s,t}$.

```python
# Exercises for Section 10.3
```

### 10.4 Partitioning of Graphs
tools from matrix theory $\to$ minimizing the "cut" size.

#### 10.4.1 What Makes a Good Partition?
1. divide the nodes into two sets so that the *cut* (sets of edges between two groups) is minimized.

2. two sets are approximately equal in size.

```python
plt.imshow(plt.imread('./res/fig10_11.png'))
```

#### 10.4.2 Normalized Cuts
A proper definition of a "good" cut must belance the size of the cut itself against the difference in the sizes of the sets that the cut creates.

The *normalized cut* value for $S$ and $T$ is
\begin{equation}
    \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(S)} + \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(T)}
\end{equation}
where $\operatorname{Vol}(S)$ is the number of edges with at least one end in $S$, and $\operatorname{Cut}(S,T)$ is the number of edges connected between $S$ and $T$.

#### 10.4.3 Some Matrices That Describe Graphs
1. adjacent matrix:
   \begin{equation}
       A_{i,j} = \begin{cases}
                     1, & \text{if node $i$ and $j$ is connected} \\
                     0, & \text{otherwise}
                 \end{cases}
   \end{equation}

2. degree matrix:
   $D_{i,i}$ is the degree of the $i$th node.

3. Laplacian matrix:
   $L = D - A$
   Notice that each row and column sums to zero.

#### 10.4.4 Eigenvalues of the Laplacian Matrix
The smallest eignevalues and their eigenvectors reveal the information we desire.

1. The smallest eignevalues for every Laplacian matrix is 0, and its corresponding eigenvectors is $\mathbf{1}$ ones matrix.

2. the second-smallest eigenvalues of $L$ is the minimum of $x^T L x$, and the minimum is taken under the constraints:
   + $\displaystyle \sum_{i=1}^n x_i^2 = 1$
   + $x$ is orthogonal to the eigenvector associated with the smallest eigenvalue.
     $$x^T \mathbf{1} = \displaystyle \sum_{i=1}^n x_i = 0$$

In all:
$x^T L x = \sum_{i,j} (x_i - x_j)^2$

As a consequence, $x$ must have some positive and some negative components $\to$ two sets/groups.

#### 10.4.5 Alternative Partitioning Methods
1. We could set the threshold at some point other than zero.

2. We may also want to a partition into more than two components:
   + split repeatedly as far as desired.
   + use several of the eigenvectors to partition the graph.

Attention:
while each eigenvector tries to produce a minimum-sized cut, the fact that successive eigenvectors have to satisfy more and more constraints generally causes the cuts they describe to be progressively worse.

```python
# Exercises for Section 10.4
## Ex 10.4.1
### (a)

edges = [
    ('A', 'B'), ('A', 'C'), ('B', 'C'),
    ('B', 'H'), ('C', 'D'), ('H', 'I'),
    ('H', 'G'), ('D', 'E'), ('D', 'F'),
    ('I', 'G'), ('G', 'E'), ('E', 'F')
]

G = nx.Graph()
G.add_edges_from(edges)
```

```python
nx.draw(G)
```

```python
A = nx.adjacency_matrix(G).todense()
A
```

    matrix([[0, 0, 0, 0, 0, 0, 0, 1, 1],
            [0, 0, 1, 0, 0, 1, 0, 0, 0],
            [0, 1, 0, 1, 0, 1, 0, 0, 0],
            [0, 0, 1, 0, 1, 0, 1, 0, 0],
            [0, 0, 0, 1, 0, 0, 1, 0, 0],
            [0, 1, 1, 0, 0, 0, 0, 0, 1],
            [0, 0, 0, 1, 1, 0, 0, 1, 0],
            [1, 0, 0, 0, 0, 0, 1, 0, 1],
            [1, 0, 0, 0, 0, 1, 0, 1, 0]], dtype=int64)

```python
D = np.diag(np.ravel(A.sum(axis=1)))
D
```

    array([[2, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 2, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 3, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 3, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 2, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 3, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 3, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 3, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 3]], dtype=int64)

```python
L = D - A
L
```

    matrix([[ 2,  0,  0,  0,  0,  0,  0, -1, -1],
            [ 0,  2, -1,  0,  0, -1,  0,  0,  0],
            [ 0, -1,  3, -1,  0, -1,  0,  0,  0],
            [ 0,  0, -1,  3, -1,  0, -1,  0,  0],
            [ 0,  0,  0, -1,  2,  0, -1,  0,  0],
            [ 0, -1, -1,  0,  0,  3,  0,  0, -1],
            [ 0,  0,  0, -1, -1,  0,  3, -1,  0],
            [-1,  0,  0,  0,  0,  0, -1,  3, -1],
            [-1,  0,  0,  0,  0, -1,  0, -1,  3]], dtype=int64)

```python
### Ex 10.4.2
w, v = np.linalg.eig(L)
```

```python
eig_min = np.argmin(w)
w = np.delete(w, eig_min)
v =np.delete(v, eig_min, axis=0)

eig_2nd_min = np.argmin(w)
eigv_2nd_min = np.ravel(v[eig_2nd_min])
```

```python
setA = set([n for v, n in zip(eigv_2nd_min, G.nodes()) if v >= 0])
setB = set(G.nodes()) - setA
print(setA, setB)
```

    {'B', 'F', 'D', 'C', 'A', 'E'} {'I', 'H', 'G'}

### 10.5 Finding Overlapping Communities
Communities are in practice rarely disjoint.

Assume:
the probability that two individuals are connected by an edge increases as they become members of more communities in common.

#### 10.5.1 The Nature of Communities
We expect edges to be dense with any community, but we expect edges to be even denser in the intersection of two communities, three communities, and so on.

```python
plt.imshow(plt.imread('./res/fig10_19.png'))
```

#### 10.5.2 Maximum-Likelihood Estimation
We assume that the value of the parameters that gives the largest value of the likelihood is the correct model for the observed artifact.
$$\operatorname{argmax}_{\theta} P[f(\theta)]) $$

prior probabilities:
$$\operatorname{argmax}_{\theta} P[f(\theta)]) = \operatorname{argmax}_{\theta} P[f(\theta) | \theta] \, P[\theta] $$

#### 10.5.3 The Affiliation-Graph Model
affiliation-graph model: generate social graphs from communities.

community-affiliation graphs:

+ Given: $C$ communities, $N$ nodes(individuals).

+ Question: $n_i \in C_k$ ?

+ Model:
  - Parameter: the memberships in the communities, $C_k = {n_i}$.
  - Parameter: $P_{ck}$ is the probability that two members of community $C_k$ are connected by an edge, $P_{ck} = P[(u,v) \in E \, | \, u \in C_k, v \in C_k]$.

we compute the likelihood that a given graph with the proper number of nodes is generated by this mechanism.

##### membership: Y/N
1. Parameter: membership
  define membership: $n_i \in C_k$.
  0 / 1, Yes / No, decrete variable $\to$ brute search

2. Parameter: $P_{ck}$
\begin{align}
    &P_{u,v} = 1 - \displaystyle \prod_{C_k in M} (1 - P_{ck})  \quad \text{where } M = {C_i: u \in C_i, v \in C_i}\\
    &P[f(P_{ck}, C_k, E)] = \displaystyle \prod_{(u,v) \in E} P_{u,v} \, \prod_{(u,v) \notin E} (1 - P_{u,v})
\end{align}

3. Goal:
   find $\operatorname{argmax}_{C_k} P[f(P_{ck}, C_k, E)]$.

##### membership: "strength"
avoiding the use of discrete membership changes. This improvement allows us to use standard methods.

"strenght of membership":
the stronger the membership of two individuals in the same community, the more likely it is that this community will cause them to have an edge between them.

In the improved model,

1. Parameter: membership
   define membership: strength, $F_{xC} \in \mathbb{R}_{\ge 0}$

2. Parameter: $P_{ck}$
   $$P_C(u,v) = 1 - e^{- F_{u,C} F_{v,C}}$$

```python
# exercise
```

### 10.6 Simrank
The purpose of simrank is to measure the similarity between nodes of the same type, and it does so by seeing where random walkers on the graph wind up when starting at a particular node $\to$ limited in the size of graphs.

#### 10.6.1 Random Walkers on a Social Graph
Random walkers:

A walker at a node $N$ of an undirected graph will move with equal probability to any of the *neighbors* of $N$.

#### 10.6.2 Random Walks with Restart
Let $M$ be the *transition matrix* of the graph $G$: the entry in row $i$ and column $j$ of $M$ is $1/k$ if node $j$ of $G$ has degree $k$, and one of the adjacent nodes is $i$.

$\beta$ is the probability that the walker continues at random, so $1 - \beta$ is the probability the walker will teleport to the initial node $N$.

$v'$ is the probability the walker is at each of the nodes at the next round:
$$v' = \beta M v + (1 - \beta) e_N$$

```python
plt.imshow(plt.imread('./res/fig10_22.png'))
```

```python
edges = [
    ('Picture 1', 'Sky'),
    ('Picture 1', 'Tree'),
    ('Picture 2', 'Sky'),
    ('Picture 3', 'Sky'),
    ('Picture 3', 'Tree')
]

G=nx.Graph()
G.add_edges_from(edges)
nx.draw(G)
```

```python
nodelist = ['Picture 1', 'Picture 2', 'Picture 3', 'Sky', 'Tree']
adj = nx.adjacency_matrix(G, nodelist=nodelist).todense()
```

```python
adj = pd.DataFrame(adj, index=nodelist, columns=nodelist)
adj
```

      Picture 1
      Picture 2
      Picture 3
      Sky
      Tree

      Picture 1
      0
      0
      0
      1
      1

      Picture 2
      0
      0
      0
      1
      0

      Picture 3
      0
      0
      0
      1
      1

      Sky
      1
      1
      1
      0
      0

      Tree
      1
      0
      1
      0
      0

```python
M = adj.apply(lambda x: x / sum(x), axis=0)
M
```

      Picture 1
      Picture 2
      Picture 3
      Sky
      Tree

      Picture 1
      0.0
      0
      0.0
      0.333333
      0.5

      Picture 2
      0.0
      0
      0.0
      0.333333
      0.0

      Picture 3
      0.0
      0
      0.0
      0.333333
      0.5

      Sky
      0.5
      1
      0.5
      0.000000
      0.0

      Tree
      0.5
      0
      0.5
      0.000000
      0.0

```python
beta = 0.8
e_0 = np.identity(M.shape[0])[0]
e_0 = pd.DataFrame({'e_0': e_0}, index=nodelist)
e_0
```

      e_0

      Picture 1
      1

      Picture 2
      0

      Picture 3
      0

      Sky
      0

      Tree
      0

```python
def random_walk(v: pd.DataFrame, beta: float, M: pd.DataFrame, e_n: pd.DataFrame) -> pd.DataFrame:
    return beta * (M.dot(v)) + (1 - beta) * e_n
```

```python
v = [e_0]
iter_time = 50

v_ = v[0]
for k in range(iter_time):
    v_ = random_walk(v_, beta, M, e_0)
    v.append(v_)

index = list(range(iter_time+1))
pd.concat(v, axis=1).T.reset_index(drop=True).T
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50

      Picture 1
      1
      0.2
      0.466667
      0.253333
      0.418311
      0.286329
      0.391308
      0.307325
      0.374446
      0.320749
      ...
      0.344591
      0.344625
      0.344598
      0.344620
      0.344603
      0.344616
      0.344605
      0.344614
      0.344607
      0.344613

      Picture 2
      0
      0.0
      0.106667
      0.021333
      0.100978
      0.037262
      0.089448
      0.047699
      0.081228
      0.054405
      ...
      0.066326
      0.066343
      0.066329
      0.066340
      0.066331
      0.066338
      0.066333
      0.066337
      0.066333
      0.066336

      Picture 3
      0
      0.0
      0.266667
      0.053333
      0.218311
      0.086329
      0.191308
      0.107325
      0.174446
      0.120749
      ...
      0.144591
      0.144625
      0.144598
      0.144620
      0.144603
      0.144616
      0.144605
      0.144614
      0.144607
      0.144613

      Sky
      0
      0.4
      0.080000
      0.378667
      0.139733
      0.335431
      0.178873
      0.304605
      0.204019
      0.284540
      ...
      0.248785
      0.248734
      0.248774
      0.248742
      0.248768
      0.248747
      0.248764
      0.248750
      0.248761
      0.248752

      Tree
      0
      0.4
      0.080000
      0.293333
      0.122667
      0.254649
      0.149063
      0.233046
      0.165860
      0.219557
      ...
      0.195707
      0.195673
      0.195700
      0.195679
      0.195696
      0.195682
      0.195693
      0.195684
      0.195691
      0.195686

5 rows × 51 columns

1. If we wanted to know what node were most similar to another node, we would have to start the analsis over for that node.

2. notice that convergence takes time, since there is an initial oscillation.

```python
# Exercise
```

### 10.7 Counting Triangles
#### 10.7.1 Why Count Triangles?
1. to measure the extent to which a graph looks like a social network.

   If we start with $n$ nodes and add $m$ edges to a graph at random:
   + sets of three nodes: $\binom{n}{3}$.
   + The probability of an edge between any two given nodes being added is $m / \binom{n}{2} \approx 2 m / n^2$.
   + The probability that any set of three nodes has edges between each pair (independently chosen) is $(2m / n^2)^3 = 8 m^3 / n^6$.
   + expected number of triangles of random graph is $(8 m^3 / n^6) (n^3 / 6) = \frac{4}{3} (m / n)^3$.

   We expect the number of triangles to be much greater than the value for a random graph.

2. It has been demonstrated that the age of a community is related to the density of triangles.

#### 10.7.2 An Algorithm for Finding Triangles
Suppose we have a graph of $n$ nodes and $m \geq n$ edges.

nouns:
+ heavy hitter: the node whose degree is at least $\sqrt{m}$.
  note, the number of heavy-hitter nodes is nore more than $2\sqrt{m}$, since otherwise the sum of degree of nodes would be more than $2m$.

+ heavy-hitter triangle: the triangle all three of whose nodes are heavy hitter.

Assuming the graph is represented by its edges, we preprocess the graph as follows:

1. Compute the degree of each node. $O(m)$

2. Create an index on edges, with the pair of nodes at its ends as the key. constructed in $O(m)$. Query the existence of an edge $O(1)$.

3. Create another index of edges, this one with key equal to a single node. to retrieve the nodes adjacent to given node. $O(\sqrt{m})$

order the nodes: `nodes.sort_values(by=['degree', 'id'])`

##### Finding triangles:
1. Heavy-Hitter Triangles:
   Find in all heavy-hitter nodes which is only $O(\sqrt{m})$.
   time: $O(\sqrt{m}^3) = O(m^{3/2})$

2. Other Triangles:
   Consider each edge $(v_1, v_2)$:
   + if both $v_1$ and $v_2$ are heavy hitters, ignore the dege. (use $v_3$ since the computation is less).

   + if $v_1 < v_2$, query whether $(v_1.\text{adjacent nodes}), v_2)$ exits. $O(\sqrt{m} * m) = O(m^{3/2})$.

The total time of the algorithm is $O(m^{3/2})$.

#### 10.7.3 Optimality of the Triangle-Finding Algorithm
It turns out the algorithm described above is, to within an order of magnitude the best possible.

For a complete graph on $n$ nodes, it has $m = \binom{n}{2}$ edges and the number of triangles is $\binom{n}{3}$. Since we cannot enumerate triangles in less time than the number of those triangles $O(n^3) = O((\sqrt{m})^3) = O(m^{3/2})$.

For sparse graphs, we can add to the complete graph a chain of nodes with any length up to $n^2$ to convert.

#### 10.7.4 Finding Triangles Using MapReduce
multiway join technique:
$E(X, Y) \bowtie E(X, Z) \bowtie E(Y, Z)$

if we hash nodes to $b$ buckets, then there will be $b^3$ reducers, since $(h(u), h(v), z)$, $(h(u), y, h(v))$ and $(x, h(u), h(v)$ are mapped.
$\to$ The total communication required is thus $3b$ key-value pairs for each of the $m$ tuples of the edge relation $E$, namely $O(mb)$ if we use $b^3$ Reduce tasks.
$\to$ each Reduce task receives $O(mb) / b^3 = O(m / b^2)$ edges.
$\to$ If we use the algorithm of Section 10.7.2, the computation cost of each Reduce is $O((m / b^2)^{3/2})$. Thus, the total computation cost of $b^3$ Reduce is $O((m / b^2)^{3/2} * b^3) = O(m^{3/2})$.

#### 10.7.5 Using Fewer Reduce Tasks
By a judicious ordering of the nodes, we can lower the number of reduce tasks by approximately a factor of 6.

Order by "name", $(h(i), i)$. The Reduce task corresponding to list of bucket $(i, j, k)$ will be needed only if $i \leq j \leq k$.

#### 10.7.6 Exercises for Section 10.7

### 10.8 Neighborhood Properties of Graphs
#### 10.8.1 Directed Graphs and Neighborhoods
all undirected graphs can be represented by directed graphs.

**path**: a sequence of nodes in a directed graph. Its *length* is the number of arcs, instead of nodes, along the path.

The *neighborhood of radius* $d$ for $v$ is: $\{u : \operatorname{len}(u, v) \leq d\}$, denote this neighborhood by $N(v, d)$.

The *neighborhood profile* of a node $v$ is the sequence of sizes of its neighborhoods $|N(v, 1)|, |N(v, 2)|, \dotso$.

#### 10.8.2 The Diameter of a Graph
The *diameter* $d$ of a directed graph: $\max (\operatorname{len}(u, v)): u \in G, v \in G$.

for each node $v$, we can find the smallest $d$ such that $|N(v, d)| = |N(v, d+1)|$, and call it $d(v)$. $\to$ the $d$ of $G$ is: $\max_{v} d(v)$.

A graph is *strongly connected* if there is a paht from any node to any other node. If $G$ is strongly connected, the $d = \max_{v} d(v)$.

"six degrees of separation": the diameter of social graph is six. Unfortunately, not all important graphs exhibit such tight connections.

#### 10.8.3 Transitive Closure and Reachability
The *transitive closure* of a graph is: $\{(u, v) : \operatorname{len of Path}(u, v) \geq 0 \}$. denoted $\operatorname{Path}(u, v)$.

*reachability*: we say node $u$ reaches node $v$ if $\operatorname{Path}(u, v)$.

$\operatorname{Path}(u, v)$ is true if and only if $v$ is in $N(u, \infty) = \cup_{i \geq 0} N(u, i)$.

The two problems - transtive closure and reachability - are related, but there are many examples of graphs where reachability is feasible and transitive closure is not.

#### 10.8.4 Transitive Closure Via MapReduce
transitive closure is actually more readily parallelizable than is reachability.

##### calculate reachability
$\operatorname{Arc}(X, Y) = \{(x, y) \text{ where } x \to y\}$.

```
SELECT DISTINCT Arc.Y
FROM Reach, Arc
WHERE Arc.X = Reach.X
```

how many rounds this process requires depends on how far from $v$ is the furthest node that $v$ can reach.

##### calculate transitive closure
recursive-doubling method

```
SELECT DISTINCT p1.X, p2.Y
FROM Path p1, Path p2
WHERE p1.Y = p2.X
```

#### 10.8.5 Smart Transitive Closure
The above recursive-doubling method does a lot of redundant work, since there may exist many paths between two nodes.

_smart_ transitive closure:
Every path of length greater than 1 can be broken into a _head_ whose length is a power of 2, followed by a _tail_ whose length is no greater than the length of the head.

$Q(X, Y)$ holds all pairs of nodes $(x, y)$ such that the shortest path from $x$ to $y$ is of length exactly $2^i$ after the $i$th round.

Intitally, set both $Q$ and _Path_ to be copies of the relation _Arc_.

On the $(i + 1)$st round, we do the following:

1. Compute a new value for $Q$ by joining it with itself:
   ```
   SELECT DISTINCT q1.X, q2.Y
   FROM Q q1, Q q2
   WHERE q1.Y = q2.X
   ```

2. Subtract _Path_ from the relation _Q_ computed in step 1.

3. Join _Path_ with the nw value of _Q_ computed in 2:
   ```
   SELECT DISTINCT Q.X, Path.Y
   FROM Q, Path
   WHERE Q.Y = Path.X
   ```

4. Set the new value of _Path_ to be the union of the relation computed in step 3, the new value of _Q_ computed in step 1, and the old value of _Path_.

#### 10.8.6 Transitive Closure by Graph Reduction
collapse an SCC (Stronly connected components) to a single node when computing the transitive closure.

to find most of the SCC's in a graph by some random node selections followed by two breadth-first searches.

Let $G$ be the graph to be reduced, and let $G'$ be $G$ with all the arcs reversed.

1. Pick a node $v$ from $G$ at random.

2. Find $N_G(v, \infty)$, the set of nodes reachable from $v$ in $G$.

3. Find $N_{G'} (v, \infty)$, the set of nodes that $v$ reaches in the graph $G'$ that has the arcs of $G$ reversed.

4. Construct the SCC S containing $v$, which is $N_G(v, \infty) \cap N_{G'}(v, \infty)$.

5. Replace SCC S by a single node $s$ in $G$.

#### 10.8.7 Approximating the Sizes of Neighborhoods
approximation:

1. apply hash function $h$ to nodes $\{v\}$, find the longest "tail length" $R$.

2. estimate the size of the set is $2^R$.

```python
# Exercise 10.8.8
```


# Jupyter tricks, magics and extensions

## Cells

Any Jupyter notebook is organized into **cells**. A cell can be one of several types:
- *code* cell, which contains normal Python (or whatever) code,
- *Markdown* cell,
- *raw content* cell.

To execute a cell, press `Shift+Enter` (execute and go to the next cell) or `Ctrl+Enter` (execute and stay in the same cell).

Each code cell can have output, which contains whatever is displayed as a result of executing the code inside a cell:

```python
print("This cell prints some text")
a = 5
a # prints 5
```

    This cell prints some text

    5

```python
res = _
```

```python
res
```

    5

Note, that last line is printed to the output. To supress this, use `;`:

```python
a; # prints nothing
```

## Modes

Jupyter notebook is a **modal editor**: it's behavior differs depending on the mode notebook is in. There are two modes:
- **edit mode**, in which you can edit cells,
- **command mode**, in which you can execute cell-wide and notebook-wide commands with keyboard shortcuts.

To enter edit mode, press `Enter`, while focused on some cell or left click on it. To go to command mode press `Esc`.

Note, that one or more cells are always in focus or "selected", and cell-wide commands are applied to selected cells.

## Markdown recap

Jupyter allows to mix Markdown text, code, images and many other elements in one interface. Markdown cells help to tell a story about your code and to document it.

---

### Headers

You can create headers with #:

`# H1 header` gives

# H1 header

`# H2 header` gives

## H2 header

and so on. Feel free to select the cell in edit mode to see the actual markup.

---

### Lists

It's also easy to create ordered list. Just number the items:
1. first element,
2. second element.

Unordered lists are also very straightforward:
- element,
- another element.

---

### Emphasis

You may often need to emphasize some portion of text. For example, *a priori* (`*a priori*` or `_a priori_`) is usually written in italics. Or, you may decide to **stress something out** (`**stress something out**` or `__stress something out__`). Strikethrough is as simple as ~~this~~ (`~~this~~`).

### Code blocks

Code can be either `inline` (`` `inline` ``) or in block:

```
val x: Int = 4
```

Syntax highlighting? Sure:

```scala
val x: Int = 4
```

The former is produced with

    ```
    val x: Int = 4

    ```

while the latter is produced with

    ```scala
    val x: Int = 4
    ```

You can also indent your code with 4 spaces.

### Equations

If Jupyter is running with MathJax, you can easily create inline ($E=mc^2$) and displayed equations:

\begin{equation}
y = \mathbf{W}\mathbf{x} + b
\end{equation}

## Commands and keyboard shortcuts

## Keyboard shortcuts for command mode

### Generic
- `Esc`/`Ctrl+M` - go to command mode.
- `Ctrl+S`/`Ctrl+Shift+S` - save/save as.

### Change cell type
- `Y`: change to code,
- `M`: change to Markdown,
- `R`: change to raw.

### Navigate
- `J`/`K` - Down/Up,
- `Shift+J`/`Shift+K` - navigate and select.

### Inserting, splitting and joining cells
- `A`/`B` - insert above/below,
- `Shift+M` - merge selected cells.

### Copy, paste and delete cells
- `C` - copy cells,
- `V` - paste cells below,
- `D,D` - delete cells.

### Interrupt and restart notebook kernel
- `I,I` - interrupt,
- `0,0` - restart.

## Magics

Some commands in Jupyter come not from Python, but represent additional (and useful) functionality. These commands start with % (for one-liners) or %% (for cell commands) and are called **magic commands**.

```python
# List all available magic commands
%lsmagic
```

    Available line magics:
    %alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %shell  %store  %sx  %system  %tb  %tensorflow_version  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

    Available cell magics:
    %%!  %%HTML  %%SVG  %%bash  %%bigquery  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%shell  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

    Automagic is ON, % prefix IS NOT needed for line magics.

### Shell magics

You can use notebook as a terminal:

```python
!ls -l
```

    total 4
    drwxr-xr-x 1 root root 4096 Oct  8 13:45 sample_data

```python
#This also works:
files = !ls
```

```python
files
```

    ['sample_data']

### Time your code

```python
# Measure code running time, one-liner
%timeit [i*i for i in range(10000)]
```

    The slowest run took 11.14 times longer than the fastest. This could mean that an intermediate result is being cached.
    1000 loops, best of 5: 776 µs per loop

```python
# Measure code running time, one-liner, change number of runs and loops
%timeit -n 10 -r 3 [i*i for i in range(10000)]
```

    10 loops, best of 3: 757 µs per loop

The same for entire cell:

```python
%%timeit -n 10 -r 3
[i*i for i in range(10000)]
```

    10 loops, best of 3: 782 µs per loop

### Write and load code

```python
%%writefile signaling.py
import numpy as np

some_global_var = "Hello!"

def hello_world():
    print("Welcome to Python for Data Processing!")
    print("My name is hello_world and I live in Python module.")

if __name__=="__main__":
    hello_world()
```

    Writing signaling.py

```python
%load signaling.py
```

```python
%run signaling.py
```

    Welcome to Python for Data Processing!
    My name is hello_world and I live in Python module.

```python
hello_world()
```

    Welcome to Python for Data Processing!
    My name is hello_world and I live in Python module.

```python
%run "[Py4DP] [Lecture-1] Signaling.ipynb"
```

    ERROR:root:File `'[Py4DP] [Lecture-1] Signaling.ipynb.py'` not found.

```python
signaling_var
```

```python
?%run
```

```python
!rm signaling.py
```

### Getting help

```python
?hello_world
```

```python
?%timeit
```

```python

```