Chapter 2
======
______

This chapter introduces more PyMC syntax and design patterns, and ways to think about how to model a system from a Bayesian perspective. It also contains tips and data visualization techniques for assesing goodness-of-fit for your Bayesian model.

## A little more on PyMC

### Parent and Child relationships

To assist with describing Bayesian relationships, and to be consistent with PyMC's documentation, we introduce *parent and child* variables.

*  *parent variables* are variables that influence another variable.

*  *child variable* are variables that are affected by other variables, i.e. are the subject of parent variables.

A variable can be both a parent and child. For example, consider the PyMC code below.

```
import pymc as mc

parameter = mc.Exponential( "poisson_param", 1 )

data_generator = mc.Poisson("data_generator", parameter )

data_plus_one = data_generator + 1
```

`parameter` controls the parameter of `data_generator`, hence influences its values. The former is a parent of the latter. By symmetry, `data_generator` is a child of `parameter`.

Likewise, `data_generator` is a parent to the variable `data_plus_one` (hence making `data_generator` both a parent and child variable). Although it does not look like one, `data_plus_one` should be treated as a PyMC variable as it is a *function* of another PyMC variable, hence is a child variable to `data_generator`.

This nomenclature is introduced to help us describe relationships in PyMC modeling. You can access a variables children and parent variables using the `children` and `parents` attributes attached to variables.

```
print "Children of `parameter`: "
print parameter.children
print "\nParents of `data_generator`: "
print data_generator.parents
print "\nChildren of `data_generator`: "
print  data_generator.children
```

    Children of `parameter`:
    set([<pymc.distributions.Poisson 'data_generator' at 0x0000000008DF5198>])

    Parents of `data_generator`:
    {'mu': <pymc.distributions.Exponential 'poisson_param' at 0x0000000008DF57B8>}

    Children of `data_generator`:
    set([<pymc.PyMCObjects.Deterministic '(data_generator_add_1)' at 0x0000000008DE3CC0>])

Of course a child can  have more than one parent, and a parent can have many children.

### PyMC Variables

All PyMC variables also expose a `value` attribute. This method produces the *current* (possibly random) internal value of the variable. If the variable is a child variable, its value changes given the variable's parents' values. Using the same variables from before:

```
print "parameter.value =",parameter.value
print "data_generator.value =",data_generator.value
print "data_plus_one.value =", data_plus_one.value
```

    parameter.value = 2.13659208293
    data_generator.value = 3
    data_plus_one.value = 4

PyMC is concerned with two types of programming variables: `stochastic` and `deterministic`.

*  *stochastic variables* are variables that are not deterministic, i.e., even if you knew all the values of the variables' parents (if it even has any parents), it would still be random. Included in this catagory are instances of classes `Poisson`, `DiscreteUniform`, and `Exponential`.

*  *deterministic variables* are variables that are not random if the variables' parents were known. This might be confusing at first: a quick mental check is *if I knew all of variable `foo`'s parent variables, I could determine what `foo`'s value is.*

We will detail each below.

#### Initializing Stochastic variables

Initializing a stochastic variable requires a `name` argument, plus additional parameters that are class specific. For example:

`some_variable = mc.DiscreteUniform( "discrete_uni_var", 0, 4 )`

where 0,4 are the `DiscreteUniform`-specific upper and lower bound on the random variable. The [PyMC docs](http://pymc-devs.github.com/pymc/distributions.html) contain the specific parameters for stochastic variables. (Or use `??` if you are using IPython!)

The `name` attribute is used to retrieve the posterior distribution later in the analysis, so it is best to use a descriptive name. Typically, I use the Python variable's name as the name.

For multivariable problems, rather than creating a Python array of stochastic variables, addressing the `size` keyword in the call to a `Stochastic` variable creates multivariate array of (independent) stochastic variables. The array behaves like a Numpy array when used like one, and references to its `value` attribute return Numpy arrays.

The `size` argument also solves the annoying case where you may have many variables $\beta_i, \; i = 1,...,N$ you wish to model. Instead of creating arbitrary names and variables for each one, like:

    beta_1 = mc.Uniform( "beta_1", 0, 1)
    beta_2 = mc.Uniform( "beta_2", 0, 1)
    ...

we can instead wrap them into a single variable:

    betas = mc.Uniform( "betas", 0, 1, size = N )

#### Calling `random()`
We can also call on a stochastic variable's `random()` method, which (given the parent values) will generate a new, random value. Below we demonstrate this using the texting example from the previous chapter.

```
lambda_1 = mc.Exponential( "lambda_1",  1 ) #prior on first behaviour
lambda_2 = mc.Exponential( "lambda_2", 1 ) #prior on second behaviour
tau = mc.DiscreteUniform( "tau", lower = 0, upper = 10 ) #prior on behaviour change

print "lambda_1.value = %.3f"%lambda_1.value
print "lambda_2.value = %.3f"%lambda_2.value
print "tau.value = %.3f"%tau.value
print

lambda_1.random(), lambda_2.random(), tau.random()

print "After calling random() on the variables..."
print "lambda_1.value = %.3f"%lambda_1.value
print "lambda_2.value = %.3f"%lambda_2.value
print "tau.value = %.3f"%tau.value

```

    lambda_1.value = 0.450
    lambda_2.value = 0.807
    tau.value = 6.000

    After calling random() on the variables...
    lambda_1.value = 0.847
    lambda_2.value = 3.247
    tau.value = 3.000

The call to `random` stores a new value into the variable's `value` attribute. In fact, this new value is stored in the computer's cache for faster recall and efficiency.

### **Warning**: *Don't update stochastic variables' values in-place.*

Straight from the PyMC docs, we quote [4]:

> `Stochastic` objects' values should not be updated in-place. This confuses PyMC's caching scheme... The only way a stochastic variable's value should be updated is using statements of the following form:

        A.value = new_value

> The following are in-place updates and should **never** be used:

        A.value += 3
        A.value[2,1] = 5
        A.value.attribute = new_attribute_value

#### Determinstic variables

Since most variables you will be modeling are stochastic, we distinguish deterministic variables with a `pymc.deterministic` wrapper. (If you are unfamiliar with Python wrappers (also called decorators), that's no problem. Just prepend the `pymc.deterministic` decorator before the variable declaration and you're good to go. No need to know more. ) The declaration of a deterministic variable uses a Python function:

    @mc.deterministic
    def some_deterministic_var(v1=v1,):
         #jelly goes here.

For all purposes, we can treat the object `some_deterministic_var` as a variable and not a Python function.

Prepending with the wrapper is the easiest way, but not the only way, to create deterministic variables. This is not completely true: elementary operations, like addition, exponentials etc. implicity create determinsitic variables. For example, the following returns a deterministic variable:

```
type( lambda_1 + lambda_2 )
```

    pymc.PyMCObjects.Deterministic

The use of the `deterministic` wrapper was seen in the previous chapter's text-message example.  Recall the model for $\lambda$ looked like:

$$
\lambda =
\cases{
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
}
$$

And in PyMC code:

```
n_data_points = 5 # in CH1 we had ~70 data points

@mc.deterministic
def lambda_( tau = tau, lambda_1 = lambda_1, lambda_2 = lambda_2 ):
    out = np.zeros(n_data_points)
    out[:tau] = lambda_1 #lambda before tau is lambda1
    out[tau:] = lambda_2 #lambda after tau is lambda1
    return out
```

Clearly, if $\tau, \lambda_1$ and $\lambda_2$ are known, then $\lambda$ is known completely, hence it is a deterministic variable.

Inside the deterministic decorator, the `Stochastic` variables passed in behave like scalars or Numpy arrays ( if multivariable), and *not* like `Stochastic` variables. For example, running the following:

    @mc.deterministic
    def some_deterministic( stoch = some_stochastic_var ):
        return stoch.value**2

will return an `AttributeError` detailing that `stoch` does not have a `value` attribute. It simply needs to be `stoch**2`. During the learning phase, it the variables `value` that is repeatedly passed in, not the actual variable.

Notice in the creation of the deterministic function we added defaults to each variable used in the function. This is a necessary step, and all variables *must* have default values.

### Including observations in the Model

At this point, it may not look like it, but we have fully specified our priors. For example, we can ask and answer questions like "What does my prior distribution of $\lambda_1$ look like?"

```
%pylab inline
figsize(12.5, 4)

samples = [ lambda_1.random() for i in range( 20000) ]
hist( samples, bins = 70, normed=True )
plt.title( "Prior distribution for $\lambda_1$")
plt.xlim( 0, 8)
```

To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified $P(A)$. Our next goal is to include data/evidence/observations $X$ into our model.

PyMC stochastic variables have a keyword argument `observed` which accepts a boolean (`False` by default). The keyword `observed` has a very simple role: fix the variable's current value, i.e. make `value` immutable. We have to specify an intial `value` in the variable's creation, equal to the observations we wish to include, typically an array (and it should be an Numpy array for speed). For example:

```
data = np.array( [10, 5] )
fixed_variable = mc.Poisson( "fxd", 1, value = data, observed = True )
print "value: ",fixed_variable.value
print "calling .random()"
fixed_variable.random()
print "value: ",fixed_variable.value
```

    value:  [10  5]
    calling .random()
    value:  [10  5]

This is how we include data into our models: initializing a stochastic variable to have a *fixed value*.

To complete our text message example, we fix the PyMC variable `observations` to the observed dataset.

```
#we're using some fake data here
data = np.array( [ 10, 25, 15, 20, 35] )
obs = mc.Poisson( "obs", lambda_, value = data, observed = True )
print obs.value
```

    [10 25 15 20 35]

### Finally...

We wrap all the created variables into a `mc.Model` class. With this `Model` class, we can analyze the variables as a single unit.

```
model = mc.Model( [obs, lambda_, lambda_1, lambda_2, tau] )
```

## Modeling approaches

A good starting thought to Bayesian modeling is to think about *how your data might have been generated*. Position yourself in an omniscient position, and try to imagine how *you* would recreate the dataset.

In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:

1.  We started by thinking "what is the best random variable to describe this count data?" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.

2.  Next, we think, "Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameters $\lambda$.

3.  Do we know $\lambda$? No. In fact, we have a suspicion that there are *two* $\lambda$ values, one for the earlier behaviour and one for the latter behaviour. We don't know when the behaviour switches though, but call the switchpoint $\tau$.

4. What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilites to positive real numbers. Well the exponential distribution has a parameter too, call it $\alpha$.

5.  Do we know what the parameter $\alpha$ might be? No. At this point, we could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here.

    What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was shown in the last chapter.

6. We have no expert opinion of when $\tau$ might have occured. So we will suppose $\tau$ is from a discrete uniform distribution over the entire timespan.

Below we give a graphical visualization of this, where arrows denote `parent-child` relationships. (provided by the [Daft Python library](http://daft-pgm.org/) )

PyMC, and other probablistic programming languages, have been designed to tell these data-generation *stories*. More generally, B. Cronin writes [5]:

> Probabilistic programming will unlock narrative explanations of data, one of the holy grails of business analytics and the unsung hero of scientific persuasion. People think in terms of stories - thus the unreasonable power of the anecdote to drive decision-making, well-founded or not. But existing analytics largely fails to provide this kind of story; instead, numbers seemingly appear out of thin air, with little of the causal context that humans prefer when weighing their options.

### Same story; different ending.

Interestingly, we can create *new datasets* by retelling the story.
For example, if we reverse the above steps, we can simulate a possible realization of the dataset.

1\. Specify when the user's behaviour switches by sampling from $\text{DiscreteUniform}(0, 80)$:

```
tau = mc.rdiscrete_uniform(0, 80)
print tau
```

    63

2\. Draw $\lambda_1$ and $\lambda_2$ from an $\text{Exp}(\alpha)$ distribution:

```
alpha = 1./20.
lambda_1, lambda_2 = mc.rexponential( alpha, 2 )
print lambda_1, lambda_2
```

    16.7630558157 33.1568046968

3\.  For days before $\tau$, represent the user's received SMS count by sampling from $\text{Poi}(\lambda_1)$, and sample from  $\text{Poi}(\lambda_2)$ for days after $\tau$. For example:

```
data = np.r_[ mc.rpoisson( lambda_1, tau ), mc.rpoisson( lambda_2,80 - tau) ]
```

4\. Plot the artificial dataset:

```
plt.bar( np.arange( 80 ), data, color ="#348ABD"  )
plt.bar( tau-1, data[tau-1], color = "r", label = "user behaviour changed" )
plt.xlabel( "Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Artifical dataset")
plt.xlim( 0, 80 );
plt.legend();
```

It is okay that our fictional dataset does not look like our observed dataset: the probability is incredibly small it indeed would. PyMC's engine is designed to find good parameters, $\lambda_i, \tau$, that maximize this probability.

The ability to generate artifical dataset is an interesting side effect of our modeling, and we will see that this ability is a very important method of Bayesian inference. We produce a few more datasets below:

```
def plot_artificial_sms_dataset():
    tau = mc.rdiscrete_uniform(0, 80)
    alpha = 1./20.
    lambda_1, lambda_2 = mc.rexponential( alpha, 2 )
    data = np.r_[ mc.rpoisson( lambda_1, tau ), mc.rpoisson( lambda_2,80- tau) ]
    plt.bar( np.arange( 80 ), data, color ="#348ABD"  )
    plt.bar( tau-1, data[tau-1], color = "r", label = "user behaviour changed" )
    plt.xlim( 0, 80 );

figsize( 12.5,5)
plt.title("More example of artificial datasets" )
for i in range(4):
    plt.subplot(4,1,i)
    plot_artificial_sms_dataset()

```

Later we will see how we use this to make predictions and test the appropriateness of our models.

## An algorithm for human deceit

Likely the most common statistical task is estimating the frequency of events. However, there is a difference between the *observed frequency* and the *true frequency* of an event. The true frequency can be interpreted as the probability of an event occuring. For example, the true frequency of rolling a 1 on a 6-sided die is 0.166. Knowing the frequency of events like baseball home runs, frequency of social attributes, fraction of internet users with cats etc. are common requests we ask of Nature. Unfortunately, in general Nature hides the true frequency from us and we must *infer* it from observed data.

The *observed frequency* is then the frequency we observe: say rolling the die 100 times you may observe 20 rolls of 1. The observed frequency, 0.2, differs from the true frequency, 0.166. We can use Bayesian statistics to infer probable values of the true frequency using an appropriate prior and observed data.

Social data is really interesting as people are not always honest with responses, which adds a further complication into inference. For example, simply asking individuals "Have you ever cheated on a test?" will surely contain some rate of dishonesty. What you can say for certain is that the true rate is less than your observed rate (assuming individuals lie *only* about *not cheating*; I cannot imagine one who would admit "Yes" to cheating when in fact they hadn't cheated).

To present an elegant solution to circumventing this dishonesty problem, and to demonstrate Bayesian modeling, we first need to introduce the binomial distribution.

### The Binomial Distribution

The binomial distribution is one of the most popular distributions, mostly because of its simplicity and usefulness. Unlike the other distributions we have encountered thus far in the book, the binomial distribution has 2 parameters: $N$, a positive integer representing $N$ trials or number of instances of potential events, and $p$, the probability of an event occuring in a single trial. Like the Poisson distribution, it is a discrete distribution, but unlike the Poisson distribution, it only weighs integers from $0$ to $N$. The mass distribution looks like:

$$P( X = k ) =  {{N}\choose{k}}  p^k(1-p)^{N-k}$$

If $X$ is a binomial random variable with parameters $p$ and $N$, denoted $X \sim \text{Bin}(N,p)$, then $X$ is the number of events that occured in the $N$ trials (obviously $0 \le X \le N$). The larger $p$ is (while still remaining between 0 and 1), the more events are likely to occur. The expected value of a binomial is equal to $Np$. Below we plot the mass probability distribution for varying parameters.

```
figsize( 12.5, 4)

import scipy.stats as stats
binomial = stats.binom

parameters = [ (10, .4) , (10, .9) ]
colors = ["#348ABD", "#A60628", "#7A68A6"]

for i in range(2):
    N, p = parameters[i]
    _x = np.arange( N+1 )
    plt.bar( _x - 0.5,  binomial.pmf( _x, N, p ), color = colors[i],
             edgecolor=colors[i],
             alpha = 0.6,
             label = "$N$: %d, $p$: %.1f"%(N,p),
             linewidth=3)

plt.legend(loc="upper left")
plt.xlim(0, 10.5)
plt.xlabel("$k$")
plt.ylabel("$P(X = k)$")
plt.title("Probability mass distributions of binomial random variables");
```

The special case when $N = 1$ corresponds to the Bernoulli distribution. If  $ X\ \sim \text{Ber}(p)$, then $X$ is 1 with probability $p$ and 0 with probability $1-p$.
The Bernoulli distribution is useful for indicators, e.g. $Y = X\alpha + (1-X)\beta$ is $\alpha$ with probability $p$ and $\beta$ with probability $1-p$.

There is another connection between Bernoulli and Binomial random variables. If we have $X_1, X_2, ... , X_N$ Bernoulli random variables with the same $p$, then $Z = X_1 + X_2 + ... + X_N \sim \text{Binomial}(N, p )$.

The expected value of a Bernoulli random variable is $p$. This can be seen by noting the the more general Binomial random variable has expected value $Np$ and setting $N=1$.

##### Example: Cheating amoung students

We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed post-exam (answering without consequence), we will receive integer $X$ "Yes I did cheat" answers. We then find the posterior distribution of $p$, given $N$, some specified prior on $p$, and observed data $X$.

This is a completely absurd model. No student, even with a free-pass against punishment, would admit to cheating. What we need is a better *algorithm* to ask students if they had cheated. Ideally the algorithm should encourage individuals to be honest while preserving privacy. The following proposed algorithm is a solution I greatly admire for its enginuity and effectivness:

> In the interview process for each student, the student flips a coin, hidden from the interviewer. The student agrees to answer honestly if the coin comes up heads. Otherwise, if the coin comes up tails, the student (secretly) flips the coin again, and answers "Yes, I did cheat" if the coin flip lands heads, and "No, I did not cheat", if the coin flip lands tails. This way, the interviewer does not know if a "Yes" was the result of a guilty plea, or a Heads on a second coin toss. Thus privacy is preserved and the researchers receive honest answers.

I call this the Privacy Algorithm. One could of course argue that the interviewers are still receiving false data since some "Yes"'s are not confessions but instead randomness, but an alternative perspective is that the researchers are discarding approximately half of their original dataset since half of the responses will be noise. But they have gained a systematic data generation process that can be modeled. Furthermore, they do not have to incorporate (perhaps somewhat naively) the possibility of deceitful answers. We can use PyMC to dig through this noisy model, and find a posterior distribution for the true frequency of liars.

Suppose 100 students are being surveyed for cheating, and we wish to find $p$, the proportion of cheaters. There a few ways we can model this in PyMC. I'll demonstrate the most explict way, and later show a simplified version. Both versions arrive at the same inference. In our data-generation model, we sample $p$, the true proportion of cheaters, from a prior. Since we are quite ignorant about $p$, we will assign it a $\text{Uniform}(0,1)$ prior.

```
import pymc as mc

N = 100

p = mc.Uniform( "freq_cheating", 0, 1)
```

Again, thinking of our data-generation model, we assign Bernoulli random variables to the 100 students: 1 implies they cheated and 0 implies they did not.

```
true_answers = mc.Bernoulli( "truths", p, size = N)
```

If we carry out the algorithm, the next step that occurs is the first coin-flip each student makes. This can be modeled again by sampling 100 Bernoulli random variables with $p=1/2$: denote a 1 as a *Heads* and 0 a *Tails*.

```
first_coin_flips = mc.Bernoulli( "first_flips", 0.5, size = N)
print first_coin_flips.value
```

    [False False False  True False  True False  True  True False  True  True
      True False False  True  True  True  True False False  True  True False
      True  True  True False False  True  True  True  True False  True  True
     False False False False False False  True  True False  True  True False
      True False  True  True  True  True  True False False  True  True False
      True False  True False  True False False  True  True False  True False
      True  True False  True  True  True False False  True  True False  True
     False  True False  True  True False False False  True False  True False
     False False  True False]

Although *not everyone* flips a second time, we can still model the possible realization of second coin-flips:

```
second_coin_flips = mc.Bernoulli("second_flips", 0.5, size = N)
```

Using these variables, we can return a possible realization of the *observed proportion* of "Yes" responses. We do this using a PyMC `deterministic` variable:

```
@mc.deterministic
def observed_proportion( t_a = true_answers,
                         fc = first_coin_flips,
                         sc = second_coin_flips ):

    observed = fc*t_a + (1-fc)*sc
    return observed.sum()/float(N)
```

The line `fc*t_a + (1-fc)*sc` contains the heart of the Privacy algorithm. Elements in this array are 1 *if and only if* i) the first toss is heads and the student cheated or ii) the first toss is tails, and the second is heads, and are 0 else. Summing this vector and dividing by `float(N)` produces a proportion.

```
observed_proportion.value
```

    0.54000000000000004

Next we need a dataset. After performing our coin-flipped interviews the researchers received 35 "Yes" responses. To put this into a relative perspective, if there truly were no cheaters, we should expect to see on average 1/4 of all responses being a "Yes" (half chance of having first coin land Tails, and another half chance of having second coin land Heads), so about 25 responses in a cheat-free world. On the other hand, if *all students cheated*, we should expected to see on approximately 3/4 of all response be "Yes".

The researchers observe a Binomial random variable, with `N = 100` and `p = observed_proportion` with `value = 35`:

```
X = 35

observations = mc.Binomial("obs", N, observed_proportion, observed = True, value = X)
```

Below we add all the variables of interest to a `Model` container and run our black-box algorithm over the model.

```
model = mc.Model( [p, true_answers, first_coin_flips,
                   second_coin_flips, observed_proportion, observations] )

### To be explained in Chapter 3!
mcmc = mc.MCMC( model )
mcmc.sample( 120000, 80000,4 )
```

    [****************100%******************]  120000 of 120000 complete

```
figsize(12.5, 3 )
p_trace = mcmc.trace("freq_cheating")[:]
plt.hist( p_trace, histtype="stepfilled" , normed = True,
    alpha = 0.85, bins = 30, label = "posterior distribution",
    color = "#348ABD")
plt.vlines( [.05, .35], [0,0], [5,5], linestyles = "--" )
plt.xlim(0,1)
plt.legend();
```

With regards to the above plot, we are still pretty uncertain about what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.05 to 0.35 (marked by the dashed lines). This is pretty good, as *a priori* we had no idea how many students might have cheated (hence the uniform distribution for our prior). On the other hand, it is also pretty bad since there is a .3 length window the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency?

I would argue, yes, we have discovered something. It is implausible, according to our posterior, that there are *no cheaters*, i.e. the posterior assigns low probability to $p=0$. Since we started with an uniform prior, treating all values of $p$ as equally plausible, but the data ruled out $p=0$ as a possibility, we can be confident that there were cheaters.

This kind of algorithm can be used to gather private information from users and be *reasonably* confident that the data, though noisy, is truthful.

### Alternative PyMC Model

Given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes:

\begin{align}
P(\text{"Yes"}) = & P( \text{Heads on first coin} )P( \text{cheater} ) + P( \text{Tails on first coin} )P( \text{Heads on second coin} ) \\\\
& = \frac{1}{2}p + \frac{1}{2}\frac{1}{2}\\\\
& = \frac{p}{2} + \frac{1}{4}
\end{align}

Thus, knowing $p$ we know the probability a student will respond "Yes". In PyMC, we can create a deterministic function to evaluate the probability of responding "Yes", given $p$:

```
p = mc.Uniform( "freq_cheating", 0, 1)

@mc.deterministic
def p_skewed( p =  p ):
    return 0.5*p + 0.25
```

I could have typed `p_skewed  = 0.5*p + 0.25` instead for a one-liner, as the elementary operations of addition and scalar multiplication will implicity create a `deterministic` variable, but I wanted to make the deterministic boilerplate explicit for clarity's sake.

If we know the probability of respondents saying "Yes", which is `p_skewed`, and we have $N=100$ students, the number of "Yes" responses is a binomial random variable with paramters `N` and `p_skewed`.

This is were we include our observed 35 "Yes" responses. In the declaration of the `mc.Binomial`, we include `value = 35` and `observed = True`.

```
yes_responses = mc.Binomial( "number_cheaters", 100, p_skewed,
                                value = 35, observed = True )
```

Below we add all the variables of interest to a `Model` container and run our black-box algorithm over the model.

```
model = mc.Model( [yes_responses, p_skewed, p ] )

### To Be Explained in Chapter 3!
mcmc = mc.MCMC(model)
mcmc.sample( 12500, 2500 )
```

    [****************100%******************]  12500 of 12500 complete

```
figsize(12.5, 3 )
p_trace = mcmc.trace("freq_cheating")[:]
plt.hist( p_trace, histtype="stepfilled" , normed = True,
    alpha = 0.85, bins = 30, label = "posterior distribution",
    color = "#348ABD")
plt.vlines( [.05, .35], [0,0], [5,5], linestyles = "--" )
plt.xlim(0,1)
plt.legend();
```

### More PyMC Tricks

#### Protip: *Lighter* deterministic variables with `Lambda` class

Sometimes writing a deterministic function using the `@mc.deterministic` decorator can seem like a chore, especially for a small function. I have already mentioned that elementary math operations *can* produce deterministic variables implicitly, but what about operations like indexing or slicing? Built-in `Lambda` functions can handle this with the elegance and simplicity required. For example,

    beta = mc.Normal( "coefficients", 0, size=(N,1) )
    x = np.random.randn( (N,1) )
    linear_combination = mc.Lambda( lambda x=x, beta = beta: np.dot( x.T, beta ) )

#### Protip: Arrays of PyMC variables
There is no reason why we cannot store multiple heterogeneous PyMC variables in a Numpy array. Just remember to set the `dtype` of the array to `object` upon initialization. For example:

```
N = 10
x = np.empty( N , dtype=object )
for i in range(0, N):
        x[i] = mc.Exponential('x_%i' % i, (i+1)**2)
```

The remainder of this chapter examines some practical examples of PyMC and PyMC modeling:

#####Example: Challenger Space Shuttle Disaster <span id="challenger"/>

On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle program ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below (see [1]):

```
figsize( 12.5, 3.5 )
np.set_printoptions(precision=3, suppress= True)
challenger_data = np.genfromtxt("data/challenger_data.csv", skip_header = 1, \
                               usecols=[1,2], missing_values="NA", delimiter=",")
#drop the NA values
challenger_data = challenger_data[ ~np.isnan(challenger_data[:,1]) ]

#plot it, as a function of tempature (the first column)
print "Temp (F), O-Ring failure?"
print challenger_data

plt.scatter( challenger_data[:,0], challenger_data[:,1], s = 75, color="k",
 alpha = 0.5)
plt.yticks([0,1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Farhenhit)" )
plt.title("Defects of the Space Shuttle O-Rings vs temperature");

```

It looks clear that *the probability* of damage incidents occuring increases as the outside temperature decreases. We are interested in modeling the probability here because it does not look like there is a strict cutoff point between temperature and a damage incident ocurring. The best we can do is ask "At temperature $t$, what is the probability of a damage incident?". The goal of this example is to answer that question.

We need a function of temperature, call it $p(t)$, that is bounded between 0 and 1 (so as to model a probability) and changes from 1 to 0 as we increase temperature. There are actually many such functions, but the most popular choice is the *logistic function.*

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t } } $$

In this model, $\beta$ is the variable we are uncertain about. Below is the function plotted for $\beta = 1, 3, -5$.

```
figsize(12,3)

def logistic( x, beta):
    return 1.0/( 1.0 + np.exp( beta*x) )
x = np.linspace( -4, 4, 100 )
plt.plot(x, logistic( x, 1), label = r"$\beta = 1$")
plt.plot(x, logistic( x, 3), label = r"$\beta = 3$")
plt.plot(x, logistic( x, -5), label = r"$\beta = -5$")
plt.legend();
```

But something is missing. In the plot of the logistic function, the probability changes only near zero, but in our data above the probability changes around 65 to 70. We need to add a *bias* term to our logistic function:

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$

Some plots are below, with differing $\alpha$.

```
def logistic( x, beta, alpha=0):
    return 1.0/( 1.0 + np.exp( np.dot(beta, x) + alpha) )

x = np.linspace( -4, 4, 100 )

plt.plot(x, logistic( x, 1), label = r"$\beta = 1$", ls= "--", lw =1 )
plt.plot(x, logistic( x, 3), label = r"$\beta = 3$", ls= "--", lw =1)
plt.plot(x, logistic( x, -5), label = r"$\beta = -5$", ls= "--", lw =1)

plt.plot(x, logistic( x, 1,1), label = r"$\beta = 1, \alpha = 1$",
    color = "#348ABD")
plt.plot(x, logistic( x, 3, -2), label = r"$\beta = 3, \alpha = -2$",
    color="#A60628")
plt.plot(x, logistic( x, -5, 7), label = r"$\beta = -5, \alpha = 7$",
    color = "#7A68A6")

plt.legend(loc = "lower left");
```

Adding a constant term $\alpha$ amounts to shifting the curve left or right (hence why it is called a *bias*. )

Let's start modeling this in PyMC. The $\beta, \alpha$ paramters have no reason to be positive, bounded or relatively large, so they are best modeled by a *Normal random variable*, introduced next.

### Normal distributions

A Normal random variable, denoted $X \sim N(\mu, 1/\tau)$, has a distribution with two parameters: the mean, $\mu$, and the *precision*, $\tau$. Those familar with the Normal distribution already have probably seen $\sigma^2$ instead of $\tau$. They are in fact reciprocals of each other. The change was motivated by simpler mathematical analysis and is an artifact of older Bayesian methods. Just remember: The smaller $\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\tau$ is always positive.

The probability density function of a $N( \mu, 1/\tau)$ random variable is:

$$ f(x | \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau}{2} (x-\mu)^2 \right) $$

We plot some different density functions below.

```
import scipy.stats as stats
nor = stats.norm
x = np.linspace( -8, 7, 150 )
mu = (-2, 0, 3)
tau = (.7, 1, 2.8 )
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip( mu, tau, colors )

for _mu, _tau, _color in parameters:
    plt.plot( x, nor.pdf( x, _mu , scale = 1./_tau ), \
        label ="$\mu = %d,\;\\tau = %.1f$"%(_mu, _tau), color = _color )
    plt.fill_between( x, nor.pdf( x, _mu, scale =1./_tau ), color = _color, \
         alpha = .33)

plt.legend(loc = "upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random variables");
```

A Normal random variable can be take on any real number, but the variable is very likely to be relatively close to $\mu$. In fact, the expected value of a Normal is equal to its $\mu$ parameter:

$$ E[ X | \mu, \tau] = \mu$$

and its variance is equal to the inverse of $\tau$:

$$Var( X | \mu, \tau ) = \frac{1}{\tau}$$

Below we continue our modeling of the the Challenger space craft:

```
import pymc as mc

temperature = challenger_data[:,0]
D = challenger_data[:,1] #defect or not?

#notice the`value` here. We explain why below.
beta = mc.Normal( "beta", 0, 0.001, value = 0 )
alpha = mc.Normal( "alpha", 0, 0.001, value = 0 )

@mc.deterministic
def p( t = temperature, alpha = alpha, beta = beta):
    return 1.0/( 1. + np.exp( beta*t + alpha) )

```

We have our probabilities, but how do we connect them to our observed data? A *Bernoulli* random variable with parameter $p$, denoted $\text{Ber}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:

$$ \text{Defect Incident, $D_i$} \sim \text{Ber}( \;p(t_i)\; ), \;\; i=1..N$$

where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the above code we had to set the values of `beta` and `alpha` to 0. The reason for this is that if `beta` and `alpha` are very large, they make `p` equal to 1 or 0. Unfortunately, `mc.Bernoulli` does not like probabilities of exactly 0 or 1, though they are mathematically well-defined probabilties. So by setting the coefficient values to `0`, we set the variable `p` to be a reasonable starting value. This has no effect on our results, nor does it mean we are including any additional information in our prior. It is simply a computational caveat in PyMC.

```
p.value
```

    array([ 0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,
            0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,
            0.5])

```
# connect the probabilities in `p` with our observations through a
# Bernoulli random variable.
observed = mc.Bernoulli( "bernoulli_obs", p, value = D, observed=True)

model = mc.Model( [observed, beta, alpha] )

#mysterious code to be explained in Chapter 3
map_ = mc.MAP(model)
map_.fit()
mcmc = mc.MCMC( model )
mcmc.sample( 120000, 100000, 2 )
```

    [****************100%******************]  120000 of 120000 complete

We have trained our model on the observed data, now we can sample values from the posterior. Let's look at the posterior distributions for $\alpha$ and $\beta$:

```
alpha_samples = mcmc.trace( 'alpha' )[:, None] #best to make them 1d
beta_samples = mcmc.trace( 'beta' )[:, None]

figsize(12.5, 6)

#histogram of the samples:
plt.subplot(211)
plt.title(r"Posterior distributions of the variables $\alpha, \beta$")
plt.hist( beta_samples, histtype='stepfilled', bins = 35, alpha = 0.85, \
        label = r"posterior of $\beta$", color = "#7A68A6",normed = True )
plt.legend()

plt.subplot(212)
plt.hist( alpha_samples, histtype='stepfilled', bins = 35, alpha = 0.85, \
        label = r"posterior of $\alpha$", color = "#A60628",normed = True )
plt.legend();

```

Regarding the spread of the data, we are very uncertain about what the true parameters might be (though considering the low sample size and the large overlap of defects-to-nondefects this behaviour is perhaps expected).

Next, let's look at the *expected probability* for a specific value of the temperature. That is, we average over all samples from the posterior to get a likely value for $p(t_i)$.

```
t = np.linspace( temperature.min() - 5, temperature.max()+5, 50 )[:,None]
p_t= logistic( t.T, beta_samples, alpha_samples )

mean_prob_t = p_t.mean(axis=0)
```

```
figsize( 12.5, 4)
plt.plot( t, mean_prob_t, lw = 3, label = "average posterior \nprobability \
of defect")
plt.plot( t, p_t[0, :], ls="--",label="realization from posterior" )
plt.plot( t, p_t[-2, :], ls="--", label="realization from posterior"  )
plt.scatter( temperature, D, color = "k", s = 50, alpha = 0.5 )
plt.title("Posterior expected value of probability of defect; plus realizations")
plt.legend(loc= "lower left")
plt.ylim( -0.1, 1.1 )
plt.xlim( t.min(), t.max() )
plt.ylabel("probability")
plt.xlabel("temperature");
```

Above we also plotted two possible realizations of what the actual underlying system might be. Both are equally likely as any other draw. The blue line is what occurs when we average all the 20000 possible dotted lines together.

An interesting question to ask is for what temperatures are we most uncertain about the defect-probability? Below we plot the expected value line **and** the associated 95% intervals for each temperature.

```
from scipy.stats.mstats import mquantiles
# vectorized bottom and top 5% quantiles for "confidence interval"
qs = mquantiles(p_t,[0.05,0.95],axis=0)
plt.fill_between(t[:,0],*qs,alpha = 0.7,
                color = "#7A68A6")

plt.plot(t[:,0], qs[0], label="95% CI", color = "#7A68A6", alpha =0.7)

plt.plot( t, mean_prob_t, lw = 1, ls= "--", color = "k",
        label = "average posterior \nprobability of defect")

plt.xlim( t.min(), t.max() )
plt.ylim( -0.02, 1.02 )
plt.legend(loc="lower left")
plt.scatter( temperature, D, color = "k", s = 50, alpha = 0.5 )
plt.xlabel("temp, $t$")

plt.ylabel("probability estimate" )
plt.title( "Posterior probability estimates given temp. $t$" );
```

The *95% credible interval*, or 95% CI, painted in purple, represents the interval, for each temperature, that contains 95% of the distribution. For example, at 65 degrees, we can be 95% sure that the probability of defect lies between 0.25 and 0.75.

More generally, we can see that as the temperature nears 60 degrees, the CI's spread out over [0,1] quickly. As we pass 70 degrees, the CI's tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how *wide* the posterior distribution is.

### What about the day of the Challenger disaster?

On the day of the Challenger disaster, the outside temperature was 31 degrees fahrenheit. What is the posterior distribution of a defect occuring,  given this temperature? The distribution is plotted below. It looks almost guaranteed that the Challenger was going to be subject to defective O-rings.

```
figsize(12.5, 2.5)
prob_31 = logistic( 31, beta_samples, alpha_samples )

plt.xlim( 0.995, 1)
plt.hist( prob_31, bins = 1000, normed = True, histtype='stepfilled' )
plt.title( "Posterior distribution of probability of defect, given $t = 31$")
plt.xlabel( "probability of defect occuring in O-ring" );

```

### Is our model appropriate?

The skeptical reader will say "You delibrately chose the logistic function for $p(t)$ and the specific priors. Perhaps other functions or priors will give different results. How do I know I have chosen a good model?" This is absolutely true. To consider an extreme situation, what if I had chosen the function $p(t) = 1,\; \forall t$, which guarantees a defect always occuring: I would have again predicted disaster on January 28th. Yet this is clearly a poorly chosen model. On the other hand, if I did choose the logistic function for $p(t)$, but specificed all my priors to be very tight around 0, likely we would have very different posterior distributions. How do we know our model is an expression of the data? This encourages us to measure the model's **goodness of fit**.

We can think: *how can we test whether our model is a bad fit?* An idea is to compare observed data (which if we recall is a *fixed* stochastic variable) with artifical dataset which we can simulate. The rational is that if the simulated dataset does not appear similar, statistically, to the observed dataset, then likely our model is not accurately represented the observed data.

Previously in this Chapter, we simulated artifical dataset for the SMS example. To do this, we sampled values from the priors. We saw how varied the resulting datasets looked like, and rarely did they mimic our observed dataset. In the current example,  we should sample from the *posterior* distributions to create *very plausible datasets*. Luckily, our Bayesian framework makes this very easy. We only need to create a new `Stochastic` variable, that is exactly the same as our variable that stored the observations, but minus the observations themselves. If you recall, our `Stochastic` variable that stored our observed data was:

    observed = mc.Bernoulli( "bernoulli_obs", p, value = D, observed=True)

Hence we create:

    simulated_data = mc.Bernoulli("simulation_data", p )

Let's simulate 10 000:

```
simulated = mc.Bernoulli( "bernoulli_sim", p)
N = 10000

mcmc = mc.MCMC( [simulated, alpha, beta, observed ] )
mcmc.sample( N )
```

    [****************100%******************]  10000 of 10000 complete

```
figsize(12.5, 5)
simulations = mcmc.trace("bernoulli_sim")[:]
print simulations.shape

plt.title( "Simulated dataset using posterior parameters")
figsize( 12.5, 6)
for i in range(4):
    ax = subplot( 4, 1, i+1)
    plt.scatter( temperature, simulations[1000*i,:], color = "k",
        s = 50, alpha = 0.6 );

```

Note that the above plots are different (if you can think of a cleaner way to present this, please send a pull request and answer [here](http://stats.stackexchange.com/questions/53078/how-to-visualize-bayesian-goodness-of-fit-for-logistic-regression)!).

We wish to assess how good our model is. "Good" is a subjective term of course, so results must be relative to other models.

We will be doing this graphically as well, which may seem like an even less objective method. The alternative is to use *Bayesian p-values*. These are still subjective, as the proper cutoff between good and bad is artibitrary. Gelman emphasises that the graphical tests are more illuminating [7] than p-value tests. We agree.

The following graphical test is a novel data-viz approach to logistic regression. The plots are called *separation plots*[8]. For a suite of models we wish to compare, each model is plotted on an individual separation plot. I leave most of the technical details about separation plots to the very accessible [original paper](http://mdwardlab.com/sites/default/files/GreenhillWardSacks.pdf), but I'll summarize their use here.

For each model, we calculate the proportion of times the posterior simulation proposed a value of 1 for a particular temperature, i.e. compute $P( \;\text{Defect} = 1 | t, \alpha, \beta )$ by averaging. This gives us the posterior probability of a defect at each data point in our dataset. For example, for the model we used above:

```
posterior_probability = simulations.mean(axis=0)
print "posterior prob of defect | realized defect "
for i in range( len(D) ):
    print "%.2f                     |   %d"%(posterior_probability[i], D[i])
```

    posterior prob of defect | realized defect
    0.44                     |   0
    0.26                     |   1
    0.29                     |   0
    0.35                     |   0
    0.39                     |   0
    0.19                     |   0
    0.16                     |   0
    0.26                     |   0
    0.82                     |   1
    0.59                     |   1
    0.26                     |   1
    0.07                     |   0
    0.39                     |   0
    0.91                     |   1
    0.39                     |   0
    0.12                     |   0
    0.26                     |   0
    0.04                     |   0
    0.10                     |   0
    0.06                     |   0
    0.12                     |   1
    0.10                     |   0
    0.79                     |   1

Next we sort each column by the posterior probabilities:

```
ix = np.argsort( posterior_probability )
print "probb | defect "
for i in range( len(D) ):
    print "%.2f  |   %d"%(posterior_probability[ix[i]], D[ix[i]])
```

    probb | defect
    0.04  |   0
    0.06  |   0
    0.07  |   0
    0.10  |   0
    0.10  |   0
    0.12  |   1
    0.12  |   0
    0.16  |   0
    0.19  |   0
    0.26  |   1
    0.26  |   0
    0.26  |   0
    0.26  |   1
    0.29  |   0
    0.35  |   0
    0.39  |   0
    0.39  |   0
    0.39  |   0
    0.44  |   0
    0.59  |   1
    0.79  |   1
    0.82  |   1
    0.91  |   1

We can present the above data better in a figure: I've wrapped this up into a `separation_plot` function.

```
figsize( 11., 1.5 )
from separation_plot import separation_plot

separation_plot(posterior_probability, D )
```

The snaking-line is the sorted probabilities, blue bars denote defects, and empty space (or grey bars for the optimistic readers) denote non-defects.  As the probability rises, we see more and more defects occur. On the right hand side, the plot suggests that as the posterior probability is large (line close to 1), then more defects are realized. This is good behaviour. Ideally, all the blue bars *should* be close to the right-hand side, and deviations from this reflect missed predictions.

The black vertical line is the expected number of defects we should observe, given this model. This allows the user to see how the total number of events predicted by the model compares to the actual number of events in the data.

It is much more informative to compare this to separation plots for other models. Below we compare our model (top) versus three others:

1. the perfect model, which predicts the posterior probability to be equal 1 if a defect did occur.
2. a completely random model, which predicts random probabilities regardless of temperature.
3. a constant model:  where $P(D = 1 \; | \; t) = c, \;\; \forall t$. The best choice for $c$ is the observed frequency of defects, in this case 7/23.

```
figsize( 11., 1.25 )

# our temperature-dependent model
separation_plot(posterior_probability, D )
plt.title("Temperature-dependent model")

# perfect model
# i.e. the probability of defect is equal to if a defect occured or not.
p = D
separation_plot( p, D)
plt.title("Perfect model")

# random predictions
p = np.random.rand( 23 )
separation_plot( p, D)
plt.title("Random model")

# constant model
constant_prob = 7./23*np.ones(23)
separation_plot(constant_prob, D )
plt.title("Constant-prediction model");
```

In the random model, we can see that as the probability increases there is no clustering of defects to the right-hand side. Similarly for the constant model.

The the perfect model, the probability line is not well shown, as it is stuck to the bottom and top of the figure. Of course the perfect model is only for demonstration, and we cannot infer any scientific inference from it.

##### Exercises

1\. Try putting in extreme values for our observations in the cheating example. What happens if we observe 25 affirmative responses? 10? 50?

2\. Try plotting $\alpha$ samples versus $\beta$ samples.  Why might the resulting plot look like this?

```
#type your code here.
figsize(12.5, 4 )
plt.scatter( alpha_samples, beta_samples, alpha = 0.1 )
plt.title( "Why does the plot look like this?" )
plt.xlabel( r"$\alpha$")
plt.ylabel( r"$\beta$");
```

### References

-  [1] Dalal, Fowlkes and Hoadley (1989),JASA, 84, 945-957.
-  [2] German Rodriguez. Datasets. In WWS509. Retrieved 30/01/2013, from http://data.princeton.edu/wws509/datasets/#smoking.
-  [3] McLeish, Don, and Cyntha Struthers. STATISTICS 450/850 Estimation and Hypothesis Testing. Winter 2012. Waterloo, Ontario: 2012. Print.
-  [4] Fonnesbeck, Christopher. "Building Models." PyMC-Devs. N.p., n.d. Web. 26 Feb 2013. <http://pymc-devs.github.com/pymc/modelbuilding.html>.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.
- [6] S.P. Brooks, E.A. Catchpole, and B.J.T. Morgan. Bayesian animal survival estimation. Statistical Science, 15: 357–376, 2000
- [7] Gelman, Andrew. "Philosophy and the practice of Bayesian statistics." British Journal of Mathematical and Statistical Psychology. (2012): n. page. Web. 2 Apr. 2013.
- [8] Greenhill, Brian, Michael D. Ward, and Audrey Sacks. "The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models." American Journal of Political Science. 55.No.4 (2011): n. page. Web. 2 Apr. 2013.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 120%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

</style>

```

```


# Introduction

Juan Camilo Henao Londono

Exercise from [Hacker rank](https://www.hackerrank.com/challenges/python-arithmetic-operators/problem)

# Arithmetic Operators

**Task**

Read two integers from STDIN and print three lines where:

1. The first line contains the sum of the two numbers.
2. The second line contains the difference of the two numbers (first - second).
3. The third line contains the product of the two numbers.

**Input Format**

The first line contains the first integer, $a$. The second line contains the second integer, $b$.

**Constraints**

* $1 \le a \le 10^{10}$
* $1 \le b \le 10^{10}$

**Output Format**

Print the three lines as explained above.

**Sample Input 0**

```
3
2
```

**Sample Output 0**

```
5
1
6
```

**Explanation 0**

$$
\begin{align}
3 + 2 &\Rightarrow 5 \\
3 - 2 &\Rightarrow 1 \\
3 \cdot 2 &\Rightarrow 6
\end{align}
$$

## My implementation

I just print the arithmetic operations indicated ...

```python
if __name__ == '__main__':

    a = int(input())
    b = int(input())

    print()

    print(a + b)
    print(a - b)
    print(a * b)
```

    5
    6

    11
    -1
    30

## Best implementation in the discussion

Implementation using the options provided by the ```print()``` function

```python
if __name__ == '__main__':

    a = int(input())
    b = int(input())

    print()

    print((a + b), (a - b), (a * b), sep='\n')
```

    5
    6

    11
    -1
    30


# Operators

## Preliminaries

Let's start by importing wick&d and defining a Slater determinant reference

```python
import wicked as w
from IPython.display import display, Math, Latex

def latex(expr):
    """Function to render any object that has a member latex() function"""
    display(Math(expr.latex()))

w.reset_space()
w.add_space("o", "fermion", "occupied", ['i','j','k','l','m'])
w.add_space("v", "fermion", "unoccupied", ['a','b','c','d','e','f'])
```

## Canonical representation of operators in wick&d

Wick&d defines a **canonical form** for operators. A canonical operator satisfies the following conditions:
- creation operators are to the left of annihilation operators
- all operators are normal ordered with respect to the reference state
- the numerical tensor that multiplies a product of second quantized operators is antisymmetric with respect to permutation of upper (annihilation) and lower (creation) indices

In this case, you only need to indicate which type of second quantized operators and which orbital space they act on to fully specify a general operator.

For example, the one-body operator
\begin{equation}
\sum_{ij}^\mathbb{O} h_{i}^{j} \{ \hat{a}^{\dagger}_{i} \hat{a}_{j} \}
\equiv \sum_{ij}^\mathbb{O} h_{i}^{j} \{ \hat{a}^{i}_{j} \}
\end{equation}
can be created using the `op` function and passing the operator label (`h`) and a list of second quantized operators that appear in it (`['o+ o']`)

```python
Hoo = w.op('h',['o+ o'])
Hoo
```

If we specify the operators in a noncanonical order, wick&d will resort the operators and **ignore any sign change**:

```python
Hoo = w.op('h',['o o+'])
Hoo
```

The following two-electron operator
\begin{equation}
\frac{1}{4} \sum_{ij}^\mathbb{O} \sum_{ab}^\mathbb{V} v_{ij}^{ab} \{  \hat{a}^{\dagger}_{i} \hat{a}^{\dagger}_{j} \hat{a}_{b} \hat{a}_{a} \}
\end{equation}
can be requested with

```python
Voovv = w.op('v',['o+ o+ v v'])
Voovv
```

Note that we did not have to specify the numerical prefactor (1/4) since wick&d can determine it automatically by counting the number of equivalent operators

If an operator involves multiple orbital spaces, like the following operator
\begin{equation}
\sum_{ij}^\mathbb{O} f_{i}^{j} \{ \hat{a}^{i}_{j} \}
+ \sum_{ab}^\mathbb{V} f_{a}^{b} \{ \hat{a}^{a}_{b} \}
\end{equation}
we can just pass a list with all the components

```python
Fd = w.op('f',['o+ o','v+ v'])
Fd
```

## Defining operators via the utlility function `gen_op`

When we want to specify an operator that contains all the possible combination of second quantized operators we can use the `gen_op` function.
A general one-body operator with all components can be generated via:

```python
w.utils.gen_op('F',1,'ov','ov')
```

While a two-body operator can be obtained via

```python
w.utils.gen_op('V',2,'ov','ov')
```

We can also exclude diagonal components via the optional argument `diagonal=False`

```python
w.utils.gen_op('F',1,'ov','ov',diagonal=False)
```

## Operator expressions and manipulation

Wick&d provides several functions to handle operator expressions (inear combinations of products of operators).
For example, operators can be multiplied using the noncommutative multiplication operation (`@`)

```python
A = w.op('A',['o+ v'])
B = w.op('B',['v+ o'])
C = w.op('C',['o+ o'])
AB = A @ B
AB
```

We can also compute the commutator of two operators using the `commutator` function
\begin{equation}
[\hat{A},\hat{B}]
\end{equation}

```python
AB_comm = w.commutator(A,B)
AB_comm
```

The function `commutator` accepts an arbitrary number of arguments, and can be used to evaluate nested commutators of the form
\begin{equation}
[\ldots[[\hat{A},\hat{B}],\hat{C}],\ldots]
\end{equation}
For example the commutator $[[\hat{A},\hat{B}],\hat{C}]$ can be evaluated as

```python
AB_comm = w.commutator(A,B,C)
AB_comm
```

We can also compute truncated expansion of the Baker–Campbell–Hausdorff formula
\begin{equation}
\exp(-\hat{B})\hat{A}\exp(\hat{B}) = \hat{A} + \frac{1}{1!} [\hat{A},\hat{B}]+
+ \frac{1}{2!} [[\hat{A},\hat{B}],\hat{B}]
+ \frac{1}{3!} [[[\hat{A},\hat{B}],\hat{B}],\hat{B}] + \cdots
\end{equation}
For example, the series truncated to two commutators is given by

```python
AB_bch = w.bch_series(A,B,2)
AB_bch
```

When operators commute, one can simplify operator expressions by bringing all terms in a canonical order

```python
C = w.op('C',['o+ o'])
D = w.op('D',['v+ v'])
CD_bch = w.bch_series(C,D,4)
CD_bch
```

In this case the canonical ordering leads to cancellation of all product terms

```python
CD_bch.canonicalize()
CD_bch
```


# Documentation

## Introduction

Differentiation is ubiquitous in almost all aspects of computer science, mathematics, and physics. It is used for numeric root-finding as in Newton's Method, and used for optimization with different forms of gradient descent.
However, calculating analytic derivatives is difficult and can lead to exponentially growing abstract syntax trees, which makes finding the derivative infeasible in many cases. Calculating the derivative numerically using the limit definition runs into numeric instabilities due to limited machine precision. Automatic differentiation addresses both of these issues - it uses the chain rule and the fact that computers calculate any function as a sequence of elementary operations to find the derivative.

The package can also evaluate the Hessian of multivariable functions and higher order derivatives of scalar functions. This is particularly useful for optimization problems, where convergence on a function minimum is often significantly faster when Hessian information is included in the root finding algorithm. This is useful for Control Theory applications, as well as many important algorithms used in robotics. Calculating the Hessian is useful for path planning, as well as for inverse kinematics. Since it is relatively difficult to get a closed form solution for the Hessian in complicated inverse kinematics problems, it is useful to be able to use Automatic Differentiation for the Hessian.

## Installation and Usage

### Installing from Source

If you want the latest nightly version of AutoDiffX, clone from our github
repository and install the latest version directly.

```bash
git clone https://github.com/CS207-Project-Team-1/cs207-FinalProject autodiffx
cd autodiffx
pip install -r requirements.txt
python3 setup.py install
```

If you are working on a python virtual environment or Mac OSX or your user's
python distribution, this should work. If editing the system python, you may
need to run the last command with root permissions by adding `sudo`.

### Installing from pip

For the stable version, you can install our package from PyPI.

```bash
pip install autodiffx
```

## Testing

All of the tests are run using pytest. To run pytest, you want to be in the
root directory of the repository. To ensure that `pytest` gets the imports
correct, you want to run it such that it adds the current path to `PYTHONPATH`.
The easiest way to do so is:

```bash
python -m pytest
```

This should run all of the tests for the package.

Currently, our only module that we actually use lives in the `ad/` folder. The tests can be found in the different test files in `ad/tests`.

## Background

**Automatic Differentiation**

Automatic differentiation relies heavily on the principles of chain rule differentiation. A graph of elementary functions is built to calculate the values of more complex functions. Using the chain rule on the graph of elementary functions, the value of the derivative at each node can also be calculated. This gives us the ability to calculate the values of functions and their derivatives, no matter how complex, to near machine precision (a significant advantage compared to alternatives such as finite differences).

The chain rule tells us that:
\begin{align}
\frac{d}{dx}(f(g(x))) &= f'(g(x))g'(x)\\
\end{align}

Since each step in our graph is just a combination of linear operations, we can find the derivative at a node by considering the value and derivative of the expressions at the previous node. By starting with an initial 'seed' vector for the derivative (often set to 1), we can find the derivative in any desired search direction.

Below is an example of constructing a graph to find the exact values of a function and its derivative. The function we used was:

$$f\left(x, y, z\right) = \dfrac{1}{xyz} + \sin\left(\dfrac{1}{x} + \dfrac{1}{y} + \dfrac{1}{z}\right)$$

We worked through this, starting with trace elements $x_1$ for $x$,  $x_2$ for $y$ and  $x_3$ for $z$. We wanted to solve this function at $(x, y, z) = (1, 2, 3)$.

| Trace | Elementary Function | Current Value | Elementary Function Derivative | $\nabla_{x}$ Value  | $\nabla_{y}$ Value  | $\nabla_{z}$ Value  |
| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: |
| $x_{1}$ | $x$ | 1 | $\dot{x}$ | 1 | 0 | 0 |
| $x_{2}$ | $y$ | 2 | $\dot{y}$ | 0 | 1 | 0 |
| $x_{3}$ | $z$ | 3 | $\dot{z}$ | 0 | 0 | 1 |
| $x_{4}$ | $1/x_{1}$ | 1 | $-\dot{x}_{1}/x_{1}^{2}$ | $-1$ | $0$ | $0$ |
| $x_{5}$ | $1/x_{2}$ | 1/2 | $-\dot{x}_{2}/x_{2}^{2}$ | $0$ | $-1/4$ | $0$ |
| $x_{6}$ | $1/x_{3}$ | 1/3 |  $-\dot{x}_{3}/x_{3}^{2}$ | $0$ | $0$ | $-1/9$ |
| $x_{7}$ | $x_{4} + x_{5} + x_{6}$ | 11/6 |$\dot{x}_{4} + \dot{x}_{5} + \dot{x}_{6}$ | -1 | -0.25 | -0.11 |
| $x_{8}$ | $sin(x_{7})$ | 0.966 |$\dot{x}_{7}cos(x_{7})$ | 0.260 | 0.065 | 0.029 |
| $x_{9}$ | $x_{4}x_{5}x_{6}$| 1/6 |$\dot{x}_{4}x_{5}x_{6} + \dot{x}_{5}x_{4}x_{6} + \dot{x}_{6}x_{4}x_{5} $ |-0.167 | -0.083  | -0.056 |
| $x_{10}$ | $x_{8} + x_{9}$ | 1.132 |$\dot{x}_{8} + \dot{x}_{9}$ | 0.093| -0.018  | -0.027 |

This isn't a very complicated function, but it shows how we can use the most basic of functions to create a graph allowing us to find exact values and gradients.

**Higher-Order Derivatives**

An effective approach to high-order automatic differention can be obtained by considering the calculus of a Taylor series. If we define $f_k$ as follows:

$$f_k = f_{k}(x_0) = \dfrac{f^{(k)}(x_0)}{k!}$$

We can show that basic arithmetic operations are as follow, where $f$ and $g$ are separate functions with the same input variable. Full derivations of these basic operations can be found [here](https://www.sintef.no/globalassets/project/evitameeting/2010/ad2010.pdf):

$$(f + g)_k = f_k + g_k$$

$$(f - g)_k = f_k - g_k$$

$$(f \times g)_k = \sum_{i = 0}^{k} f_{i}g_{k-i} $$

$$(f \div g)_k = \dfrac{1}{g_0} \left(f_k - \sum_{i = 0}^{k - 1} (f \div g)_{i} g_{k-i}\right) $$

$$ (e^g)_k = \dfrac{1}{k} \sum_{i = 1}^{k} ig_{i}(e^g)_{k-i}$$

**Hessian**

Our package implements the Hessian for functions of the form
$$f: \mathbb{R}^m \to \mathbb{R}$$
In this case, the Hessian should be a square matrix. Since the Hessian will be a square matrix with entries of the form
$$\frac{\partial^2 f}{\partial x_i \partial x_j}$$
for some $i, j$, we return a dictionary of dictionaries for the Hessian. This makes it easy to index the Hessian as ```hessian[x][y]``` to get the second order derivative with respect to x and y.

## Simple Usage

Our package supports scalar functions with scalar or multivariable inputs.

**Scalar Input**

Suppose that we wanted to find the derivative of the following function, $f(x)$:

$$f(x) = x \exp(\cos(x) + x^2)$$

First, we have to allocate a Variable for our input variable $x$. We can do that, and give it a string identifier `"x"` for convenience later.

```python
import ad
x = ad.Variable('x')
```
Now, we have to define our function. Our module `ad` has a lot of built in functions, and all of the regular operators `+`, `-`, `*`, `/`, `**` should also work. We can make our function $f(x)$ by just writing it out in code.

```python
f = x * ad.Exp(ad.Cos(x) + x ** 2)
```

Now, we've defined our function in terms of our input variable $x$. Now, in order to evaluate our function or evaluate the derivative at a specific point we actually have to provide a value for the input. Since we later plan on handling multiple variable inputs, we pass the input in as a dictionary. To evaluate the function, we use the function ```eval```. For the derivative at a point, we use ```d```. Suppose that we wanted to evaluate the function $f$ and its derivative at $x = 0$ and $x = 1$. We can just run:

```python
>>> f.eval({x: 0})
2.718281828459045

>>> f.d({x: 0})
1.0

>>> f.eval({x: 1})
5.666000617166735

>>> f.d({x: 1})
6.405697099891925
```

It is also possible to evaluate higher order derivatives of functions with scalar inputs. This can be done using ```f.d_n(n, val)``` where n and val are arguments expressing the desired order of differentiation and the value at which the derivative should be evaluated. We can run:

```python
>>> f.d_n(n = 3, val = 2)
1902.7256925837773
```

**Multi-Variable Input**

Suppose that we wanted to find the derivative of the following function, $f(x, y)$:

$$f(x) = x \exp(\cos(y) + x^2)$$

We have to allocate a variables for our inputs $x$ and $y$. We can do that, and give them string identifiers `"x"` and `"y"` for convenience later.

```python
import ad
x = ad.Variable('x')
y = ad.Variable('y')
```
We now define our function.

```python
f = x * ad.Exp(ad.Cos(y) + x ** 2)
```

We can evaluate our function and it's derivative exactly as we evaluated our scalar input example earlier. Suppose that we wanted to evaluate the function $f$ and its derivative at $x = 0$ and $y = 1$. We can just run:

```python
>>> f.eval({x: 0, y : 1})
1.7165256995489035

>>> f.d({x: 0, y : 1})
{y: -1.4444065708474794, x: 1.0}
```

Using the Hessian is very similar to calling the derivative of a multivariable function. If the function depends on $n$ input variables, the package will return an $n \times n$ matrix in the form of a dictionary of dictionaries, where each sub-dictionary refers to one of the rows in the matrix.

Hence, for our multivariable example above, $f(x, y)$:

$$f(x) = x \exp(\cos(y) + x^2)$$

We can evaluate the Hessian by calling:

```python
>>> f.hessian({x: 0, y : 1})
{y: {y: 0.28798342608583105, x: 0.0}, x: {y: 0.0, x: 3.433051399097807}}
```
If we want the specific second order derivatives, we can evaluate specific elements from the Hessian by calling the keys. For example, if we wanted to find the second order derivative of our function with respect to $x$, we would call:

```python
>>> f.hessian({x: 0, y : 1})[x][x]
 3.433051399097807
```

More complicated demonstrations (scalar input Newton's Method, multivariable input Newton's Method with Hessian) can be found in Jupyter notebooks at the top level directory.

## Software Organization

## Directory structure

```bash
cs207project
├── LICENSE
├── README
├── docs
│   ├── Documentation
│   ├── Milestone1.ipynb
│   ├── Milestone2.ipynb
│   └── SETUP.md
├── ad
│   ├── __init__.py
│   ├── ad.py
│   ├── simple_ops.py
│   ├── activation_ops.py.../
│   └── tests
│       ├── test_complex_ops.py
│       ├── test_d_expr.py
│       ├── test_expression.py
│       ├── test_high_order.py
│       ├── test_multivar_hessian.py
│       ├── test_simple_hessian.py
│       ├── test_simple_ops.py
│       └── test_vector.py
├── demos
│   ├── Newton_Method_Demonstration.ipynb
│   ├── Hessian_Demonstration.ipynb
│   ├── Higher_Order_Demonstration.ipynb
├── requirements.txt
└── setup.py
```

## Modules

`ad`: Implementation of core structures used in our graph-based automatic differentiation library. +, -, *, /, and exponentiation is also implemented here to support simple operator overloading.

`simple_ops`: Unary operations including Sin(x), Cos(x), Tan(x), Sinh(x), Cosh(x), Tanh(x), Exp(x), Log(x), Arcsin(x), Arccos(x), Arctan(x), Logistic(x).

## Test

See Testing.

## Installment

See Installation and Usage

## Implementation Details

We implemented our automatic differentiation using a graph based structure. First, the user will build up their function in terms of elementary operations. Then, the user will be able to feed a value for their input variables, and the package will calculate the derivatives at each step.

### Automatic Differentiation Basics

Our core data structure is the computational graph. Every node in the computational graph would be an "Expression", which is our core class.

Based on "Expression", we have "Variable(Expression)", "Constant(Expression)", "Unop(Expression)" and "Biop(Expression)". "Unop" is for unary operations such as log and power. "Biop" is for binary operations such as addition. For "Expression" and its subclasses, there are two important attributes: "grad", a boolean variable indicating whether we want to calculate the derivative or not; children, a list of nodes pointing to current one, the number of children is one for "Unop" and the number of chilren for "Biop" is two.

The elementary functions we support are:

Unary operations:

* Sin(x)
* Cos(x)
* Tan(x)
* Sinh(x)
* Cosh(x)
* Tanh(x)
* Exp(x)
* Log(x)
* Arcsin(x)
* Arccos(x)
* Arctan(x)
* Logistic(x)

Binary operations:

* Addition (+)
* Substraction (-)
* Multiplication (*)
* Division (/)
* Power(x, n)

We will mainly be using `numpy` as an external dependency for mathematical and vector operations.

### Multivariate Inputs

We want to also support getting the Jacobian when we have more than one input for a scalar function. We are able to support functions of the form
$$f: \mathbb{R}^m \to \mathbb{R}$$
Our implementation for multivariate inputs is by using a `dict` to hold the different partial derivatives. We can go through an example. Suppose that a user wanted the Jacobian of a function:

$$f(x, y, z) = x  + y \cos(yz)$$

Then, we are able to find the Jacobian in code using the following syntax.

```python
>>> x = ad.Variable('x')
>>> y = ad.Variable('y')
>>> z = ad.Variable('z')

>>> f = x + y * ad.Cos(y * z)

>>> f.eval({x:6, y:1, z:0})
7
>>> f.d({x:6, y:1, z:0})
{x: 1, y: 1, z: 0}
```

### Extension Feature - Higher Order Derivatives and Hessian

**Higher Order Derivatives**

See background.

**Hessian**

See background.

An example function and calculated Hessian will be look as follows.

$$f: \mathbb{R}^3 \to \mathbb{R}$$
$$f(x, y, z) = xy + z$$

The code uses the function `hessian()` to calculate the Hessian at a certain point.

```python
>>> x = ad.Variable('x')
>>> y = ad.Variable('y')
>>> z = ad.Variable('z')

>>> f = x * y + z

>>> f.eval({x:1, y:1, z:1})
2

>>> f.hessian({x:1, y:1, z:1})
{x: {x: 0, y: 1, z:} y: {x: 1, y: 0, z: 0} z: {x: 0, y: 0, z: 0}}
```

### Future Extensions

**Multivariate Outputs**

Currently, our package only fully supports scalar functions with scalar or multivariate input.

However, the next step after supporting multivariate inputs is to also support multivariate outputs. We will create a wrapper class `ad.Vector` that essentially wraps multiple different `ad.Expression` instances together. This will allow us to combine multiple scalar functions into a multivariate function. For example, suppose that a user wanted to find the Jacobian of a function in the form:
$$f: \mathbb{R}^3 \to \mathbb{R}^3$$
and suppose that the function was in the form:
$$f(x, y, z) =
\begin{pmatrix}
x + y + z \\
\cos(yz) \\
\exp(x - y - z)
\end{pmatrix}
$$

This could be represented in code as follows:

```python
>>> x = ad.Variable('x')
>>> y = ad.Variable('y')
>>> z = ad.Variable('z')

>>> f1 = x + y + z
>>> f2 = ad.Cos(y * z)
>>> f3 = ad.Exp(x - y - z)

>>> f = ad.Vector(f1, f2, f3)
>>> f.eval({x:0, y:0, z:0})
[0, 1, 1]

>>> f.d({x:0, y:0, z:0})
{x: [1, 0, 1], y: [1, 0, -1], z: [1, 0, -1]}
```

## Future

An automatic differentiation package with Hessian and higher-order scalar derivative function will have many potential scientific applications. Root finding algorithms with relatively rapid convergence properties rely on higher order derivative information being available. These algorithms are ubiquitous across many fields, from economics to physics, and our package makes it significantly easier for users to work with these algorithms, since the big work of calculating the Hessian (often the most complex part of implementing a method such as the Newton Method) is done automatically.

**Robotics**

In particular, we are interested in its potential use in Robotics, where being able to calculate the Hessian to machine precision quickly is critically important for path planning and inverse kinematics. Recent research work has shown that using the exact Hessian in these systems gives a significant performance boost when compared with using approximate methods (such as BFGS). The research focussed on a variety of fields beyond Robotics, stretching to motion capture, character animation and computer graphics. The [paper](http://image.diku.dk/kenny/download/erleben.andrews.17.pdf) (Erleben and Andrews) showed that 'using exact Hessians can give performance advantages and higher accuracy compared to standard numerical methods used for solving these problems.'

**Control Theory**

For many control theory problems, particularly problems requiring online predictions, having the ability to quickly compute the Hessian of the control function is very important. Researchers at Warwick University [found](https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/imr_cdt/students/david_goodwin/publications/imrcdt_southampton_mar16.pdf) that using Hessian information during the control of magnetic resonance imaging led to increased accuracy in the resulting images. Similar uses of rapid Hessian calculation are possible across many different control applications.

**Computational Cost**

Calculating the Hessian becomes very expensive as the number of variables used by a function increases. For larger problems, calculating the full Hessian stops making sense, since the benefit derived from having full accuracy are outweighed by the cost of achieving such a result. For larger problems, we would like to implement approximate Hessian algorithms (such as BFGS), to make this package universally useful for researchers needing Hessian information.


# 8: sPlot

This notebook explains __sPlot__ and how to use `zfit` and `hepstats` to compute the sWeights with `compute_sweights`. Alternatively, if the probabilities are already available, `hep_ml.splot` can be used.
__sPlot__ is a way to reconstruct features of mixture components based on known properties of distributions. This method is frequently used in High Energy Physics.

If you prefer explanations without code, find them [here](http://arogozhnikov.github.io/2015/10/07/splot.html)

```python
import numpy as np
from matplotlib import pyplot as plt
import zfit
import mplhep
```

```python
size = 10000
sig_data = np.random.normal(-1, 1, size=size)
bck_data = np.random.normal(1, 1, size=size)
```

## Simple sPlot example

First we start with a simple (and not very useful in practice) example:

- Assume we have two types of particles (say, electrons and positrons).
- Distribution of some characteristic is different for them (let this be $p_x$ momentum projection).

```python
plt.hist(sig_data, color='b', alpha=0.5, bins=30, label='electron')
plt.hist(bck_data, color='r', alpha=0.5, bins=30, label='positron')
plt.xlim(-5, 5), plt.xlabel('$p_x$')
plt.legend()
```

## Observed distributions

Picture above shows how this distibution should look like,
but due to inaccuracies during classification we will observe a different picture.

Let's assume that with a probability of 80% particle is classified correctly (and we are not using $p_x$ during classification).

And when we look at distribution of px for particles which were classified as electrons or positrons, we see that they were distorted.
We lost the original shapes of distributions.

```python
n_sig1, n_bck1 = 8000, 2000
n_sig2, n_bck2 = 2000, 8000
first_bin = np.concatenate([sig_data[:n_sig1], bck_data[:n_bck1]])
second_bin = np.concatenate([sig_data[n_sig1:], bck_data[n_bck1:]])
```

```python
plt.figure(figsize=[15, 6])
plt.subplot(121)
plt.bar([0, 2], [n_sig1, n_sig2], width=1, color='b', alpha=0.5)
plt.bar([0, 2], [n_bck1, n_bck2], width=1, bottom=[n_sig1, n_sig2], color='r', alpha=0.5)
plt.xlim(-0.5, 3.5)

plt.axis('off')
plt.xticks([0.5, 2.5], ['as electrons', 'as positrons'])
plt.text(0.5, -300, 'as electron', horizontalalignment='center', verticalalignment='top', fontsize=20)
plt.text(2.5, -300, 'as positron', horizontalalignment='center', verticalalignment='top', fontsize=20)
plt.title('Proportion of events being classified as')

plt.subplot(122)
plt.hist(first_bin, alpha=0.5, bins=30, label='as electrons', color=(0.22, 0., 0.66))
plt.hist(second_bin, alpha=0.5, bins=30, label='as positrons', color=(0.66, 0., 0.22))
plt.legend()
plt.title('Distributions')
plt.xlim(-5, 5), plt.xlabel('$p_x$')
pass
```

## Applying sWeights

We can think of it in the following way: there are 2 bins. In first 80% are electrons, 20% are signal. And visa versa in second bin.

To reconstruct initial distribution, we can plot histogram, where each event from first bin has weight 0.8,
and each event from second bin has weight -0.2. This numbers are called sWeights.

So, if we had 8000 $e^{-}$ + 2000 $e^{+}$ in first bin and 8000 $e^{+}$ + 2000 $e^{-}$ ($ e^-, e^+$ are electron and positron). After summing with introduced sWeights:

$$
\big[ 8000 e^{-} + 2000 e^{+} \big] \times 0.8 + \big[ 2000 e^{-} + 8000 e^{+} \big] \times (- 0.2) =
6800 e^{-}
$$

Positrons with positive and negative weights compensated each other, and we will get pure electrons.

At this moment we ignore normalization of sWeights (because it doesn't play role when we want to reconstruct shape).

```python
def plot_with_weights(datas, weights, **kargs):
    assert len(datas) == len(weights)
    data = np.concatenate(datas)
    weight = np.concatenate([np.ones(len(d)) * w for d, w in zip(datas, weights) ])
    plt.hist(data, weights=weight, alpha=0.5, bins=30, **kargs)
```

```python
plot_with_weights([first_bin, second_bin], [n_bck2, -n_bck1], density=True, label='reconstructed electron')
plot_with_weights([first_bin, second_bin], [-n_sig2, n_sig1], density=True, color='r', label='reconstructed positron')
plt.xlabel('px')
plt.legend();
```

### Compare

let's compare reconstructed distribution for electrons with an original one:

```python
plot_with_weights([first_bin, second_bin], [n_bck2, -n_bck1], density=True, label='reconstructed electons', edgecolor='none')
plot_with_weights([sig_data], [1], density=True, label='original electons', edgecolor='none')
plt.legend();
```

## More complex case

In the case when we have only two 'bins' is simple and straightforward. But when there are more than two bins, the solution is not unique. There are many appropriate combinations of sWeights, which one to choose?

```python
plt.bar([0, 2, 4], [3, 2, 1], width=1, color='b', alpha=0.5)
plt.bar([0, 2, 4], [1, 2, 3], width=1, bottom=[3, 2, 1], color='r', alpha=0.5)

plt.xlim(-1, 6)
plt.ylim(-0.5, 5)
plt.axis('off')

plt.text(0.5, -0.5, 'Bin 1', horizontalalignment='center', verticalalignment='top', fontsize=20)
plt.text(2.5, -0.5, 'Bin 2', horizontalalignment='center', verticalalignment='top', fontsize=20)
plt.text(4.5, -0.5, 'Bin 3', horizontalalignment='center', verticalalignment='top', fontsize=20)
pass
```

Things in practice are however even more complex. We have not bins, but continuos distribution (which can be treated as many bins).

Typically this is a distribution over mass. By fitting mass we are able to split mixture into two parts: signal channel and everything else.

## Splot

This is now an demonstration of the **sPlot** algorithm, described in [Pivk:2004ty](https://arxiv.org/pdf/physics/0402083.pdf).

If a data sample is populated by different sources of events, like signal and background, **sPlot** is able to unfold the contributions of the different sources for a given variable.

Let's construct a dataset with two variables, the invariant mass and lifetime, for the resonant signal defined above and the combinatorial background. To do this, we build the model in zfit.

```python
mu = zfit.Parameter('mu', 5279, 5100, 5400)
sigma = zfit.Parameter('sigma', 20, 1, 200)

lambd = zfit.Parameter('lambda', -0.002, -0.01, 0.0001)
```

```python
obs = zfit.Space('mass', (5000, 6000))

signal_pdf = zfit.pdf.Gauss(mu=mu, sigma=sigma, obs=obs)
comb_bkg_pdf = zfit.pdf.Exponential(lambd, obs=obs)

sig_yield = zfit.Parameter('sig_yield', 25000, 0, 50000,
                                step_size=1)  # step size: default is small, use appropriate
bkg_yield = zfit.Parameter('bkg_yield', 100000, 0, 3e5, step_size=1)

# Create the extended models
extended_sig = signal_pdf.create_extended(sig_yield)
extended_bkg = comb_bkg_pdf.create_extended(bkg_yield)

# The final model is the combination of the signal and backgrond PDF
model = zfit.pdf.SumPDF([extended_bkg, extended_sig])
```

```python
# Signal distributions.
nsig_sw = 20000
np_sig_m_sw = signal_pdf.sample(nsig_sw).numpy().reshape(-1,)
np_sig_t_sw = np.random.exponential(size=nsig_sw, scale=1)

# Background distributions.
nbkg_sw = 150000
np_bkg_m_sw = comb_bkg_pdf.sample(nbkg_sw).numpy().reshape(-1,)
np_bkg_t_sw = np.random.normal(size=nbkg_sw, loc=2.0, scale=2.5)

# Lifetime cut.
t_cut = np_bkg_t_sw > 0
np_bkg_t_sw = np_bkg_t_sw[t_cut]
np_bkg_m_sw = np_bkg_m_sw[t_cut]

# Mass distribution
np_m_sw = np.concatenate([np_sig_m_sw, np_bkg_m_sw])

# Lifetime distribution
np_t_sw = np.concatenate([np_sig_t_sw, np_bkg_t_sw])

# Plots the mass and lifetime distribution.
fig, axs = plt.subplots(1, 2, figsize=(16, 6))
axs[0].hist([np_bkg_m_sw, np_sig_m_sw], bins=50, stacked=True, label=("background", "signal"), alpha=.7)
axs[0].set_xlabel("m")
axs[0].legend(fontsize=15)
axs[1].hist([np_bkg_t_sw, np_sig_t_sw], bins=50, stacked=True, label=("background", "signal"), alpha=.7)
axs[1].set_xlabel("t")
axs[1].legend(fontsize=15);
```

In this particular example we want to unfold the signal lifetime distribution. To do so **sPlot** needs a discriminant variable to determine the yields of the various sources using an extended maximum likelihood fit.

```python
# Builds the loss.
data_sw = zfit.Data.from_numpy(obs=obs, array=np_m_sw)
nll_sw = zfit.loss.ExtendedUnbinnedNLL(model, data_sw)

# This parameter was useful in the simultaneous fit but not anymore so we fix it.
sigma.floating = False

# Minimizes the loss.
minimizer = zfit.minimize.Minuit(use_minuit_grad=True)
result_sw = minimizer.minimize(nll_sw)
print(result_sw.params)
```

```python
def plot_fit_projection(model, data, nbins=30, ax=None):
    # The function will be reused.
    if ax is None:
        ax = plt.gca()

    lower, upper = data.data_range.limit1d

    # Creates and histogram of the data and plots it with mplhep.
    counts, bin_edges = np.histogram(data.unstack_x(), bins=nbins)
    mplhep.histplot(counts, bins=bin_edges, histtype="errorbar", yerr=True,
                    label="Data", ax=ax, color="black")

    binwidth = np.diff(bin_edges)[0]
    x = np.linspace(lower, upper, num=1000)  # or np.linspace

    # Line plots of the total pdf and the sub-pdfs.
    y = model.ext_pdf(x) * binwidth
    ax.plot(x, y, label="total", color="royalblue")
    for m, l, c in zip(model.get_models(), ["background", "signal"], ["forestgreen", "crimson"]):
        ym = m.ext_pdf(x) * binwidth
        ax.plot(x, ym, label=l, color=c)

    ax.set_title(data.data_range.obs[0])
    ax.set_xlim(lower, upper)
    ax.legend(fontsize=15)

    return ax
```

```python
# Visualization of the result.
zfit.param.set_values(nll_sw.get_params(), result_sw)
plot_fit_projection(model, data_sw, nbins=100)
```

**sPlot** will use the fitted yield for each sources to derive the so-called **sWeights** for each data point:

\begin{equation}
W_{n}(x) = \frac{\sum_{j=1}^{N_S} V_{nj} f_j(x)}{\sum_{k=1}^{N_S} N_{k}f_k(x)}
\end{equation}

with

\begin{equation}
V_{nj}^{-1} = \sum_{e=1}^{N} \frac{f_n(x_e) f_j(x_e)}{(\sum_{k=1}^{N_S} N_{k}f_k(x))^2}
\end{equation}

where ${N_S}$ is the number of sources in the data sample, here 2. The index $n$ represents the source, for instance $0$ is the signal and $1$ is the background, then $f_0$ and $N_0$ are the pdf and yield for the signal.

In `hepstats` the **sWeights** are computed with the `compute_sweights` function which takes as arguments the fitted extended model and the discrimant data (on which the fit was performed).

```python
from hepstats.splot import compute_sweights

weights = compute_sweights(model, data_sw)

print(weights)
```

```python
print("Sum of signal sWeights: ", np.sum(weights[sig_yield]))
print("Sum of background sWeights: ", np.sum(weights[bkg_yield]))
```

Now we can apply the signal **sWeights** on the lifetime distribution and retrieve its signal components.

```python
fig, axs = plt.subplots(1, 2, figsize=(16, 6))
nbins = 40

sorter = np_m_sw.argsort()

axs[0].plot(np_m_sw[sorter], weights[sig_yield][sorter], label="$w_\\mathrm{sig}$")
axs[0].plot(np_m_sw[sorter], weights[bkg_yield][sorter], label="$w_\\mathrm{bkg}$")
axs[0].plot(np_m_sw[sorter], weights[sig_yield][sorter] + weights[bkg_yield][sorter],
            "-k", label="$w_\\mathrm{sig} + w_\\mathrm{bkg}$")
axs[0].axhline(0, color="0.5")
axs[0].legend(fontsize=15)
axs[0].set_xlim(5000, 5600)

axs[1].hist(np_t_sw, bins=nbins, range=(0, 6), weights=weights[sig_yield], label="weighted histogram", alpha=.5)
axs[1].hist(np_sig_t_sw, bins=nbins, range=(0, 6), histtype="step", label="true histogram", lw=1.5)
axs[1].set_xlabel("t")
axs[1].legend(fontsize=15);
```

Be careful the **sPlot** technique works only on variables that are uncorrelated with the discriminant variable.

```python
print(f"Correlation between m and t: {np.corrcoef(np_m_sw, np_t_sw)[0, 1]}")
```

Let's apply to signal **sWeights** on the mass distribution to see how bad the results of **sPlot** is when applied on a variable that is correlated with the discrimant variable.

```python
plt.hist(np_m_sw, bins=100, range=(5000, 6000), weights=weights[sig_yield]);
```

## Alternative: Known probabilities

If the probabilities are already known _beforehand_, fitting a curve with `zfit` is an extra step that is not required in order to obtain the sWeights. The next example uses `hep_ml` in order to compute the sWeights; the same function as `hepstats` also uses internally.

### Building sPlot over mass

Let's show how this works. First we generate two fake distributions (signal and background) with 2 variables: mass and momentum $p$.

```python
from scipy.stats import norm, expon
```

```python
plt.figure(figsize=[15, 6])
size = 10000

sig_mass_distr = norm(loc=4, scale=1)
bck_mass_distr = expon(scale=4)

sig_mass = sig_mass_distr.rvs(size=size)
bck_mass = bck_mass_distr.rvs(size=size)
sig_p = np.random.normal(5, 1, size=size)
bck_p = np.random.normal(3, 1, size=size)

plt.subplot(121)

plt.hist(sig_mass, bins=20, density=True)
plt.hist(bck_mass, bins=20, density=True, range=(0, 10), alpha=0.5)
plt.xlabel('mass')

plt.subplot(122)

plt.hist(sig_p, bins=20, density=True)
plt.hist(bck_p, bins=20, density=True, range=(0, 10), alpha=0.5)
plt.xlabel('p');
```

### Of course we don't have labels which events are signal and which are background beforehand

And we observe the mixture of two distributions:

```python
plt.figure(figsize=[15, 6])
mass = np.concatenate([sig_mass, bck_mass])
p = np.concatenate([sig_p, bck_p])

sorter = np.argsort(mass)
mass = mass[sorter]
p = p[sorter]

plt.subplot(121)
plt.hist(mass, bins=20, range=(0, 10))
plt.xlabel('mass')

plt.subplot(122)
plt.hist(p, bins=20)
plt.xlabel('p')
```

### We have no information about real labels

But we know a priori that background is distributed as exponential distribution and signal - as gaussian (more complex models can be met in practice, but idea is the same).

After fitting the mixture (let me skip this process), we will get the following result:

```python
x = np.linspace(0, 10)
plt.hist(mass, bins=30, range=[0, 10], density=True, alpha=0.4)
plt.plot(x, norm.pdf(x, loc=4, scale=1) / 2., label='signal')
plt.plot(x, expon.pdf(x, scale=4) / 2., label='bck')
plt.plot(x, 0.5 * (norm.pdf(x, loc=4, scale=1) + expon.pdf(x, scale=4)), label='sig + bck')
plt.legend(fontsize=20)
```

## Fitting doesn't give us information about real labels

But it gives an information about probabilities, thus now we can estimate number of signal and background events within each bin.

We won't use bins, but instead we will get for each event probability that it is signal or background:

```python
import pandas
probs = pandas.DataFrame(dict(sig=sig_mass_distr.pdf(mass), bck=bck_mass_distr.pdf(mass)))
probs = probs.div(probs.sum(axis=1), axis=0)
```

```python
plt.plot(mass, probs.sig, label='sig probability')
plt.plot(mass, probs.bck, label='bck probability')
plt.xlim(0, 10), plt.legend(), plt.xlabel('mass')
```

## Appying sPlot

sPlot converts probabilities to sWeights, using an implementation from `hep_ml`:

```python
from hep_ml import splot
sWeights = splot.compute_sweights(probs)
```

As you can see, there are also negative sWeights, which are needed to compensate the contributions of other class.

```python
plt.plot(mass, sWeights.sig, label='sig sWeight')
plt.plot(mass, sWeights.bck, label='bck sWeight')
plt.xlim(0, 10), plt.legend(), plt.xlabel('mass');
```

## Using sWeights to reconstruct initial distribution

Let's check that we achieved our goal and can reconstruct momentum distribution for signal and background:

```python
plt.figure(figsize=[15, 7])
plt.subplot(121)
hist_conf = dict(bins=30, alpha=0.4, range=[0, 10])
plt.hist(sig_p, label='original sig p', **hist_conf)
plt.hist(p, weights=sWeights.sig, label='reconstructed sig p', **hist_conf)
plt.xlabel('p'), plt.legend()

plt.subplot(122)
plt.hist(bck_p, label='original bck p', **hist_conf)
plt.hist(p, weights=sWeights.bck, label='reconstructed bck p', **hist_conf)
plt.xlabel('p'), plt.legend()
pass
```

## An important requirement of sPlot

Reconstructed variable (i.e. $p$) and splotted variable (i.e. mass) shall be statistically independent for each class.

Read the line above again. Reconstructed and splotted variable are correlated:

```python
np.corrcoef(abs(mass - 4), p) [0, 1]
```

But within each class there is no correlation, so the requirement is satisfied:

```python
print(np.corrcoef(abs(sig_mass - 4), sig_p)[0, 1])
print(np.corrcoef(abs(bck_mass - 4), bck_p)[0, 1])
```

as a demonstration why this is important let's use sweights to reconstruct mass (obviously mass is correlated with mass):

```python
hist_conf = dict(bins=30, alpha=0.5, range=[-1, 7])
plt.hist(sig_mass, label='original sig mass', **hist_conf)
plt.hist(mass, weights=sWeights.sig, label='reconstructed sig mass', **hist_conf)
plt.xlabel('mass'), plt.legend()
pass
```

$\def\ps{p_s(x)}$
$\def\pb{p_b(x)}$
$\def\ws{sw_s(x)}$
$\def\wb{sw_b(x)}$

## Derivation of sWeights (optional)

Now, after we seen how this works, let's derive a formula for sWeights.

The only information we have from fitting over mass is  $ \ps $, $ \pb$ which are probabilities of event $x$ to be signal and background.

Our main goal is to correctly reconstruct histogram. Let's reconstruct the number of _signal_ events in _particular_ bin. Let's introduce unknown $p_s$ and $p_b$ - probability that signal or background event will be in the named bin.

(Since mass and reconstructed variable are statistically independent for each class, $p_s$ and $p_b$ do not depend on mass.)

The mathematical expectation should be obviously equal to $p_s N_s$, where $N_s$ is total amount of signal events available from fitting.

Let's also introduce random variable $1_{x \in bin}$, which is 1 iff event $x$ lies in selected bin.

The __estimate for number of signal event in bin__ is equal to:
$$X = \sum_x \ws \; 1_{x \in bin},$$ where $\ws$ are sPlot weights and are subject to find.

This way we can guarantee that mean contribution of background are 0 (expectation is zero, but observed number will not be zero due to statistical deviation).

### Under assumption of linearity:

*assuming* that splot weight can be computed as a linear combination of conditional probabilities:

$ \ws = a_1 \pb + a_2 \ps$

We can easily reconstruct those numbers, first let's rewrite our system:

$ \sum_x (a_1 \pb + a_2 \ps) \; \ps = 0$

$ \sum_x (a_1 \pb + a_2 \ps) \; \pb = N_{sig}$

$ a_1 V_{bb} + a_2 V_{bs} = 0$

$ a_1 V_{sb} + a_2 V_{ss} = N_{sig}$

Where
$V_{ss} = \sum_x \ps \; \ps $, $V_{bs} = V_{sb} = \sum_x \ps \; \pb$, $V_{bb} = \sum_x \pb \; \pb$

Having solved this linear equation, we get needed coefficients (as those in the paper)

NB. There is little difference between $V$ matrix I use and $V$ matrix in the paper.

### Minimization of variation
$\def\Var{\mathbb{V}\,}$

Previous part allows one to get the correct result. But there is still no explanation of reason for linearity.

Apart from having correct mean, we should also minimize variation of any reconstructed variable. Let's try to optimize it

$$ \Var X = \sum_x \ws^2 \; \Var 1_{x \in bin} = \sum_x \ws^2 \; (p_s \ps + p_b \pb)(1 - p_s \ps - p_b \pb)$$

A bit complex, isn't it? Instead of optimizing such a complex expression (which is individual for each bin), let's minimize it's __uniform upper estimate__
$$ \Var X = \sum_x \ws^2 \; \Var 1_{x \in bin} \leq  \sum_x \ws^2  $$

so if we are going to minimize this upper estimate, we should solve the following optimization problem with constraints:

$\sum_x \ws^2 \to \min $

$\sum_x \ws \; \pb = 0$

$\sum_x \ws \; \ps = N_{sig}$

Let's write lagrangian of optimization problem:

 $$ \mathcal{L} =  \sum_x \ws^2
+ \lambda_1 \left[\sum_x \ws \; \pb \right]
+ \lambda_2 \left[\sum_x \ws \; \ps - N_{sig} \right]$$

### Uncorrelatedness

The main assuption we used here is that distribution inside each bin is abolutely indentical.

In other words, we stated that there is no correlation between the index of bin and the reconstructed variable. Remember that bin corresponds to some interval in mass, and finaly we get:

__reconstructed variable shall not be correlated with mass variables (or any other splotted variable)__

## Conclusion

1. sPlot allows reconstruction of some variables.
2. the only information used is probabilities taken from fit over variable. If fact, any probability estimates fit well.
3. the source of probabilities should be statistically independent from reconstructed variable (for each class!).
4. mixture may contain more than 2 classes (this is supported by `hep_ml.splot` as well)