**Table of contents**

* [Probability Theory](#probability-theory)
    * [Sample spaces and events](#sample-spaces-and-events)
    * [Probability measure](#probability-measure)
    * [Joint probability, conditional probability, and independence](#joint-conditional-independence)
    * [Random variable](#random-variable)
    * [Probability distribution](#probability-distribution)
* [Important Distributions](#important-distributions)
    * [Uniform distribution](#uniform)
    * [Bernoulli distribution](#bernoulli)
    * [Categorical distribution](#categorical)
* [Representation](#representation)
* [Maximum Likelihood Estimation](#mle)

**Table of Exercises**

* Theoretical (max 15 points)
    * [Exercise 2-1](#ex2-1)
    * [Exercise 2-2](#ex2-2)
    * [Exercise 2-3](#ex2-3)
    * [Exercise 2-4](#ex2-4)
    * [Exercise 2-5](#ex2-5)
    * [Exercise 2-6](#ex2-6)
    * [Exercise 2-7](#ex2-7)
    * [Exercise 2-8](#ex2-8)
* Practical (max 15 points)
    * [Exercise 2-9](#ex2-9)
    * [Exercise 2-10](#ex2-10)

**General notes**

* In this notebook you are expected to use $\LaTeX$ for typesetting mathematical expressions, if you are not familiar with it check our own examples, they illustrate all commands you will need. If you need additional documentation try searching for it on Google, lots of people contribute answer to latex-related questions. A good starting point is [LaTeX/Mathematics](https://en.wikibooks.org/wiki/LaTeX/Mathematics) and for [equations](https://www.codecogs.com/latex/eqneditor.php).
* I am very picky with notation, so make an effor to respect conventions and to typeset clear unambiguous mathematical expressions. For example, if I use $\mathbb P$ to denote a measure, there is a reason for it, thus make sure to use `\mathbb P` or you will not get the correct type of font.

#  Probability Theory

Here we provide a very *lightweight*, and rather *informal*, review of some concepts from probability theory and statistical estimation that are relevant to this course. For a more complete take on probability theory, and with a *proper formal treatment*, check this [excellent course](https://github.com/BasicProbability) out.

Probability refers to a *subjective degree of confidence* (or *belief*) that an event of a certain nature will occur. For example, a weather report might say that there is a low probability of rain in the afternoon. In other words, the event of raining *might occur*. Probability theory deals with the formal foundations of how uncertainty can be quantified.

##  Sample spaces and events

Probability theory is about assigning probability values to elements of a **sample space**.

We use $\Omega$ to denote the space, and $\omega \in \Omega$ to denote members of that space. Members are usually  called **outcomes** or **samples**.

Example of sample spaces:

* in a toss coin the coin can land showing `heads` (H) or `tails` (T), thus $\Omega = \{H, T\}$;
* throwing a 6-sided die can yield an integer from 1 (included) to 6 (included), thus $\Omega = \{1,2,3,4,5,6\}$
* throwing a [10-sided die](https://en.wikipedia.org/wiki/Pentagonal_trapezohedron) followed by a 6-sided die yields pairs of numbers where the first number ranges from 1 (included) to 10 (included) and the second number ranges from 1 to 6, thus $\Omega = \{1, 2, 3, 4, 5, 6, 7, 8, 10\} \times \{1, 2, 3, 4, 5, 6\}$
    * note that here we used a shortcut, the [cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) denoted by $\times$
    * it's okay to use it, just keep in mind what it means: the cartesian product above is the set $\{(1, 2), (1, 3), (1, 4), (1, 4), (1, 6), (2, 1), \ldots, (10, 1), (10, 2), (10, 3), (10, 4), (10, 5), (10, 6) \}$

**Exercise 2-1** **[1 point]** What is the sample space associated with the [standard 52-playincard deck](https://en.wikipedia.org/wiki/Standard_52-card_deck)? Use $\clubsuit, \diamondsuit, \heartsuit, \spadesuit$ for the suits.

$\{2,3,4,5,6,7,8,9,10,J,Q,K,A\}\times\{\clubsuit,\diamondsuit,\heartsuit,\spadesuit\}$

In a sample spaces we enumerate individual outcomes of some random experiment. However, we often care more about properties of these outcomes rather than the outcomes themselves. For example, we may be interested in whether the outcome of a die roll is an even number. We call that an **event**, where an event is any subset of the sample space. That is, if we use $A$ to denote the event, then $A \subseteq \Omega$.

Example of events:

* toss a coin and get heads: $\{H\}$;
* roll a die and get 1: $\{1\}$;
* roll a die and get an even number: $\{2, 4, 6\}$;
* roll a die and get an even number bigger than 2: $\{4, 6\}$;

**Exercise 2-2** **[3 points]** Represent the following events for the 52-card sample space of [Exercise 2-1](#ex2-1)?

1. pick an `A` of $\diamondsuit$: $\{A\diamondsuit\}$
2. pick an `A`: $\{A\clubsuit,A\diamondsuit,A\heartsuit,A\spadesuit\}$
3. pick a $\diamondsuit$: $\{2\diamondsuit,3\diamondsuit,4\diamondsuit,5\diamondsuit,6\diamondsuit,7\diamondsuit,8\diamondsuit,9\diamondsuit,10\diamondsuit,J\diamondsuit,Q\diamondsuit,K\diamondsuit,A\diamondsuit \}$

We can now define the notion of **event space**. There are some formal properties that event spaces must hold, we will not discuss them here. Instead we will always just assume that for a given sample space $\Omega$, a convenient event space $\mathcal A$ always exists. That 'convenient' event space typically is the *powerset* of the sample space, denoted $\mathcal P(\Omega)$. The [powerset](https://en.wikipedia.org/wiki/Power_set) of $\Omega$ is the set of all subsets of $\Omega$, which therefore includes the emptyset $\emptyset$ and $\Omega$ itself.

##  Probability measure

In mathematics, a *measure* is a function that maps elements from a set of sets (such as events in a event space) to real numbers and for which some formal properties hold. We will not survey measure theory, but we will pay particular attention to one important measure, namely, the **probability measure**.

Let $(\Omega, \mathcal A)$ denote a sample space and an event space, and let $A \in \mathcal A$ denote an event, then a probability measure $\mathbb P: \mathcal A \rightarrow \mathbb R$ takes elements in the event space and maps them to a real number (a *probability value*) and is such that:

\begin{align}
(1) \quad\quad &  \mathbb P(A) \ge 0  \text{ for all } A \in \mathcal A & \\
(2) \quad\quad & \mathbb P \left( \bigcup_{i=1}^k = \sum_{i=1}^k \mathbb P(A_i) \right) & \text{for disjoint events } A_1, A_2, \ldots, A_k \\
(3) \quad\quad & \mathbb P(\Omega) = 1 &
\end{align}

Let's digest this.
* Property (1) means that the smallest probability value atainable by any event is 0.
* Property (2) means that the total probability assigned to $k$ [disjoint events](https://en.wikipedia.org/wiki/Disjoint_sets) is the sum of the probability values assigned to each of the $k$ events independently.
* And finally, property (3) means that if we consider the event which is the set of all possible outcomes, we have a total probability value of 1.

Let's diggest this some more, this time with an example. Suppose we are throwing a dice. The sample space is $\Omega = \{1, 2, 3, 4, 5, 6\}$. There are many events we can be interested in, for example:

* `getting 1` represented by the set $\{1\}$;
* `getting 2` represented by the set $\{2\}$;
* `getting either 1 or 2` represented by the set $\{1, 2\}$
* `getting an odd number` represented by the set $\{1, 3, 5\}$
* `getting more than 2` represented by the set $\{3, 4, 5, 6\}$
* `getting whatever` represented by the set $\{1, 2, 3, 4, 5, 6\}$ (note that this is exactly equivalent to the complete sample space $\Omega$)

There are so many things about this sample space that we may be interested in, that we just assume we may potentially be interested in any event in $\mathcal P(\Omega)$.

* Property (1) implies that the smaller probability value assigned to any single event in the event space is 0.
* Property (2) implies that if we take disjoint events such as $\{1\}$ (`getting 1`) and $\{2\}$ (`getting 2`), the total probability value assigned to the union is the sum of probability values, that is, $\mathbb P(\{1, 2\}) = \mathbb P(\{1\}) + \mathbb P(\{2\})$.
* Property (3) implies that the event $\Omega$ (`whatever`) has probability 1. This is the famous informal rule that says *probabilities must add up to 1*.

It may not look obvious at first, but from properties 1 to 3 we can conclude that:

\begin{align}
(4) \quad\quad & \mathbb P(\emptyset) = 0 & \\
(5) \quad\quad & \mathbb P(A \cup B) = \mathbb P(A) + \mathbb P(B) - \mathbb P(A \cap B) & \text{for events } A, B \in \mathcal A\\
(6) \quad\quad & \mathbb P(\Omega \setminus A) = 1 - \mathbb P(A) & \text{for an event } A \in \mathcal A
\end{align}

Let's digest these ones now.

* Property (4) states that the *empty event* has 0 probability: this is to say that if we observe a random experiment something will happen (there is no chance that nothing will happen).
* Property (5) is a generalisation of property (2) which does not require *disjoint* events. Also, it generalises trivially to more than 2 events.
* Property (6) is also called the *complement rule*. For example, the probability of `not getting 2` is $\mathbb P(\{1, 3, 4, 5, 6\})$ which is the same as 1 minus the probability of `getting 2`, that is, $1 - \mathbb P(\{2\})$.

We now define the concept of a **probability space**. A probability space is a triple $(\Omega, \mathcal A, \mathbb P)$ consisting of a sample space $\Omega$, an event space $\mathcal A$, and a probability measure $\mathbb P$.

**Exercise 2-3** **[3 points]** For the probability space $(\Omega, \mathcal P(\Omega), \mathbb P)$ where $\Omega$ is the sample space of Exercise 2-1 (52-card deck), give an expression for the probability of picking a card that complies with the following cases (use the complement rule whenever possible and make sure you provide representations for every event present in the expression):

1. Not an `A`: $1 - \mathbb P(\{A\clubsuit,A\diamondsuit,A\heartsuit,A\spadesuit\})$
2. Not a $\heartsuit$: $1 - \mathbb P(\{2,3,4,5,6,7,8,9,10,J,Q,K,A\}\times\{\heartsuit\})$
3. Neither an `A` nor a $\heartsuit$:
$1-(\mathbb P(\{A\clubsuit,A\diamondsuit,A\heartsuit,A\spadesuit\}) + \mathbb P(\{2,3,4,5,6,7,8,9,10,J,Q,K,A\}\times\{\heartsuit\})- \mathbb P(A\heartsuit))$

##  Joint probability, conditional probability, and independence

An important concept is that of **joint probability**, which concerns the probability of the intersection of events. We denote joint probability between two events $A, B$ by $\mathbb P(A \cap B)$ or equivalently $\mathbb P(A, B)$ --- where the order of enumeration is arbitrary.

We can find the joint probability by using the elementary properties above involving union and complements. For that we will need [De Morgan's law](https://en.wikipedia.org/wiki/De_Morgan's_laws).

**Exercise 2-4** **[2 points]** Give an expression for the joint probability $\mathbb P(A \cap B)$ in terms of union and complements by applying De Morgan's law. You may use one or more of the properties (1) to (6).

Joint probability help us define the concept of **conditional probability**. The probability of an event $A$ conditioned on event $B$ having happened with $\mathbb P(B) > 0$ is defined as:

\begin{equation}
    (7) \quad\quad \mathbb P(A|B) = \frac{\mathbb P(A \cap B)}{\mathbb P(B)}
\end{equation}

The interpretation of conditional probabilities is that they are the probabilities of events assuming that another event has already occurred.

If two events $A$ and $B$ are disjoint, then $A \cap B = \emptyset$. Disjoint events are said to be **independent** events, denoted by $A \perp B$. For independent events it holds that:

\begin{equation}
    (8) \quad\quad \mathbb P(A | B) = \mathbb P(A)
\end{equation}

With (7) and (8) we conclude that for *independent* events:

\begin{equation}
    (9) \quad\quad \mathbb P(A \cap B) = \mathbb P(A) \times \mathbb P(B)
\end{equation}

**Exercise 2-5** **[1 point]** Prove property (9) using property (7) and (8).

\begin{equation}
    \quad\quad \mathbb P(A|B) = \frac{\mathbb P(A \cap B)}{\mathbb P(B)} \\
    \text{using (8) we have}\\
    \quad\quad \mathbb P(A) = \frac{\mathbb P(A \cap B)}{\mathbb P(B)}\\
    \quad\quad \mathbb P(A) \times \mathbb P(B) = \mathbb P(B) \times \frac{\mathbb P(A \cap B)}{\mathbb P(B)}\\
    \quad\quad \mathbb P(A) \times \mathbb P(B) = \mathbb P(A \cap B)
\end{equation}

##  Random variable

In general, we perform random experiments to investigate particular properties of outcomes.
We will consider an example from [Schulz and Schaffner (2016)](https://github.com/BasicProbability):

> Like most people living in the north of Europe, your choice of clothing probably depends strongly on the weather. Let us assume that you have a thick winter jacket that you put on when it is cold, a light soft-shell jacket that you wear when temperatures are mild and that you simply wear a T-shirt or a sweater when it is warm. Let us also assume that the forecasting site you consult every morning reports the temperatures up to one decimal. The question is: does it matter to you whether it is 5.4 or 7.3 degrees centigrade outside when you make your decision about what to wear? Most likely you are only interested in whether it is cold, mild or warm. This may obviously vary according to your own perception of warmth and cold. Here we will assume that it is cold whenever the temperature is below 10 degrees, mild between 10 and 20 degrees (inclusive) and warm whenever the temperature rises above 20 degrees. So what you are really interested in, at least as far as clothing is concerned, is whether the temperature $t$ is $t < 10$ or $10 \le t \le 20$ or $t > 20$. What we are looking for is a way of transforming outcomes from the temperature scale into judgements of perceived temperature.

A **random variable** $X$ on a probability space $(\Omega, \mathcal A, \mathbb P)$ is a function $X: \Omega \rightarrow \mathbb R$ that maps elements in the *sample space* to real numbers. Moreover we require that:

(10) for all $r \in \mathbb R$, the set $A = \{\omega ~ |~ X(\omega) \leq r\}$ is in the *event space* $\mathcal A$
* we read (10) as: $A$ is the set of all outcomes $\omega \in \Omega$ which the function $X$ maps to a real number less or equal to $r$; and $A$ belongs to the event space, thus $A$ is an event;
* the sample space is the *domain* of the random variable
* the real line is the *co-domain* of the random variable
* it's sometimes convenient to refer directly to the assignments that a random variable may take on, that is, the particular subset of real values that the random variable maps outcomes to (the *image* of the function). We call that set the random variable's **support** and we denote it by a caligraphic version of the letter used to denote the random variable itself. That is, we usually say:
> a random variable $X$ takes on values in the set $\mathcal X$ (the support of $X$)

By convention, we denote random variables with capital Roman letters such as $X$, $Y$, or $Z$.

Example: let's define a random variable $X$ for the weather situation where temperature values $t$ are elements of our sample space.

\begin{equation}
    X(t) =
    \begin{cases}
      1 & t < 10 \\
      2 & 10 \le t \le 20 \\
      3 & t > 20
    \end{cases}
\end{equation}

Note that even though the underlying sample space is not discrete, the random variable itself is because it buckets temperature values in 3 intervals. In other words its image is the finite set $\mathcal X = \{1, 2, 3\}$. Also, note that we did not have to explicitly define a sample space, neither an event space (random variables are very handy!).

**Assignment**: whenever a random variable $X$ takes on a particular value, we write $X=x$ using the corresponding lowercase letter as a generic value. This notation hides a lot of details, so let's look into it closely:
* when we write an assignment $X=x$, we implicitly refer to the event $A = \{\omega ~| ~ X(\omega) = x\}$. Of course, all that respecting the conditions in (10), that is,  $\omega \in \Omega$ and $A \in \mathcal A$

```python
# Let us visualise the random variable X of the temperature example
#  since RVs are functions, we can plot them
%matplotlib inline
import matplotlib.pyplot as plt

plt.plot([-20, 9.99], [1, 1])
plt.plot([10, 20], [2, 2])
plt.plot([20.01, 50], [3, 3])
plt.yticks([1, 2, 3], [1, 2, 3])
plt.xlim(-20,50)
plt.ylabel('X(t)')
plt.xlabel('t (C)')
_ = plt.legend(['cold', 'mild', 'warm'])
```

**Exercise 2-6** **[3 points]** Define a random variable that models the number of heads when we toss two coins. Hint: in this case start by defining a sample space.

$\Omega = \{HH, HT, TH, TT\}$

\begin{equation}
    X(h) =
    \begin{cases}
      0 & \{TT\} \\
      1 & \{HT, TH\} \\
      2 & \{HH\}
    \end{cases}
\end{equation}

It is in general the case that for a random variable of interest there are multiple equivalent ways to define it, starting from the fact that it all depends on how we define the sample space. That's why sometime it's easier to define the random variable directly and leave the sample space *implicitly defined*.

Note that so far we have discussed random variables that can take on discrete values, i.e. one out of $k$ possibilities, where $k$ is some finite number. We call these **discrete random variables**.

##  Probability distribution

The **discrete probability distribution** of a random variable $X$ is denoted by $P_X$ and is defined as the function

\begin{equation}
    (11) \quad\quad P_X(X=x) \triangleq \mathbb P(\{\omega ~ | ~ X(\omega) = x\})
\end{equation}

It is not at all unusual to drop one or both mentions to $X$, thus denoting the probability value by $P_X(x)$, $P(X=x)$, or $P(x)$. We do that whenever there is no risk of ambiguity.

We call **probability mass function** (or *pmf*) the function
\begin{equation}
    f(a) = P_X(X=a)
\end{equation}
it simply returns the probability value of associated with $X=a$.

**Exercise 2-7** **[1 point]** If a discrete random variable $X$ takes on values in the set $\mathcal X$, what's the result of the expression below and why?

* $\sum_{x \in \mathcal X} P_X(X=x) = 1$

This function sums up the total probability space. The area under the curve of a prob space is equal to 1

The **cumulative distribution function** (or *cdf*) is used when we are interested in assessing the probability of a range of values.

The cdf of a distribution $P_X$ of the random variable $X$ is the function

\begin{align}
F(a) &\triangleq P(X \le a) \\
       &=  \sum_{x \le a} P_X(x)
\end{align}

it returns the cumulative probability over the interval $(-\infty, a]$.

Note that it is trivial then to obtain the probability of $X$ falling in the interval $[a, b]$, we just compute $F(b) - F(a)$ for $b \ge a$.

#  Important Distributions

Now we will illustrate some instances of discrete probability distributions

##  Uniform distribution

The [Uniform distribution](https://en.wikipedia.org/wiki/Discrete_uniform_distribution) defined over $k$ outcomes is such that every outcome has the same probability. Because property (3) must always hold, it's obvious to see that each outcome has to have probability $\frac{1}{k}$. We declare Uniform random variables as follows:

\begin{equation}
    X \sim \text{Uniform}(k)
\end{equation}

This is the math equivalent to: $X$ follows a **Uniform distribution** with **parameter** $k$ (number of outcomes)
* a uniform random variable takes on values in the set $\{1, ..., k\}$
* we call $k$ a *parameter* of the distribution because it helps specify it
* we immediately know that $P(x) = \frac{1}{k}$ for $x \in \{1, ..., k\}$
* it's not difficult to generalise the uniform distribution to take on values in any finite discrete interval $\{a, a + 1, \ldots, a + k\}$

Here is a visualisation of a uniform distribution over $k=5$ possible outcomes.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
k = 5
_ = plt.bar(np.arange(1, k + 1), np.ones(k) / k)
```

##  Bernoulli distribution

The [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution) is defined over exactly 2 outcomes (a *binary* event). By convention we say that the outcome is either 1 (*True* or *positive class*) or 0 (*False* or *negative class*). We fully specify the Bernoulli distribution by specifying the probability of the positive class. Then by the complement rule (6) we know that $P_X(0) = 1 - P_X(1)$. We declare Bernoulli random variables as follows:

\begin{equation}
    X \sim \text{Bernoulli}(p)
\end{equation}

This is the math equivalent to: $X$ follows a **Bernoulli distribution** with **parameter** $p$
* a Bernoulli variable takes on values in the set $\{0, 1\}$
* we call the probability of the positive class a *parameter* of the distribution because it helps specify it
* the Bernoulli parameter is the probability of the positive class, and therefore, $0 \le p \le 1$ for it to be a valid distribution
* we immediately know that $P_X(1) = p$ and $P_X(0) = 1 - p$

Suppose a Bernoulli whose positive class probability is 0.7.

```python
p1 = 0.7
plt.bar([0, 1], [1 - p1, p1])
_ = plt.xticks([0, 1], ['False', 'True'])
```

##  Categorical distribution

The [Categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) defined over $k$ values can be used to represent the distribution of any discrete  random variable whose sample space is finite.  We declare  categorical random variables as follows:

\begin{equation}
    X \sim \text{Categorical}(\theta_1, \ldots, \theta_k)
\end{equation}

This is the math equivalent to: $X$ follows a **Categorical distribution** with **parameters** $\langle \theta_1, \ldots, \theta_k \rangle$
* a Categorical variable takes on values in the finite set $\{1, \ldots, k\}$;
* we call the parameters of the categorical distribution *class probabilities*, these are probabilities associated with each outcome and they help us specify the distribution;
* we sometimes abreviate the parameters $\langle \theta_1, \ldots, \theta_k \rangle$ into $\pmb \theta$ which we call a *parameter vector* (often we drop the word *vector* and call it simply a *parameter*);
* this $k$-dimensional vector $\pmb \theta$ is no ordinary vector, each one of its $k$ values must be a probability value and therefore $0 \le \theta_x \le 1$ for every assignment $x \in \{1, \ldots, k\}$; moreover, to make a valid distribution, they must also obey $\sum_{x=1}^k \theta_x = 1$; a parameter vector that complies with such conditions is also called a *probability vector*;
* we immediately know that $P_X(x) = \theta_x$.

Let's illustrate this with an example. Suppose we have a bag with 10 marbles where 3 of them are blue, 2 of them are red, and 5 of them are white. Then the probability that the first ball drawn at random is:
* blue is 3/10 or 0.3
* red is 2/10 or 0.2
* white is 5/10 or 0.5
We can design a random variable that models the color of the first ball drawn at random from the bag:

\begin{equation}
    X(\omega) =
    \begin{cases}
      1 & \omega = \text{blue} \\
      2 & \omega = \text{red}\\
      3 & \omega = \text{white}
    \end{cases}
\end{equation}

where we arbitrarily decided to map from colors to numbers in lexicographic order (note that we have to realise this mapping because the random variable assignments should be integers from 1 to $k$ with $k=3$ in this case).

```python
# This our sample space: we use a list (rather than a set) because we want it sorted
sample_space = ['blue', 'red', 'white']
# These are the values our random variable takes on
rv_values = [1, 2, 3]
# This is the mapping from sample space (colours) to numbers (assignments of X)
rv_mapping = dict(zip(sample_space, rv_values))  # realises the mapping above
# And these are the probabilities associated with each colour in the example
probabilities = [0.3, 0.2, 0.5]
```

```python
# Let's visualise what we accomplished
print('Sample space:', sample_space)
print('Mapping function:', rv_mapping)
for omega in sample_space:
    x = rv_mapping[omega]
    theta_x = probabilities[x - 1]  # we subtract 1 because random access to lists in Python is 0-based
    print(" outcome %r mapped to %d has probability %.2f" % (omega, x, theta_x))
```

    Sample space: ['blue', 'red', 'white']
    Mapping function: {'blue': 1, 'white': 3, 'red': 2}
     outcome 'blue' mapped to 1 has probability 0.30
     outcome 'red' mapped to 2 has probability 0.20
     outcome 'white' mapped to 3 has probability 0.50

```python
# We can also plot our probability distribution, it's simply a bar plot
#  each bar represents the probability of one of the classes
plt.bar(rv_values, probabilities)
# and we can label the ticks of the x-axis with the classes (colours) rather than the assignments (integers)
_ = plt.xticks(rv_values, sample_space)
```

## Rules of probability

> From now on we will be a bit less formal about the whole sample space, event space, probability measure thing. We will specify our random variables by describing the sample space somewhat informally and we will assume an appropriate mapping exists. The point is that from now on we want to focus on manipulating probability distributions.

A very important result of probability theory is called the **chain rule**

\begin{equation}
(12) \quad\quad P(x, y) = P(x)P(y|x)
\end{equation}

this rule relates the probability of the *joint assignment* $x$ and $y$ (left hand-side) to the *marginal* probability of $x$ (first term on the right-hand side), and the *conditional* probability of $y$ given $x$ (second probability on the rigth-hand side).

This rule can be generalised to $n \ge 2$ events and the particular order in which we enumerate them is arbitrary (the result does not depend on the order).

\begin{equation}
(13) \quad\quad P(\alpha_1, \ldots, \alpha_n) = P(\alpha_1)\prod_{i = 2}^n P(\alpha_i|\alpha_1, \ldots, \alpha_{i-1})
\end{equation}

For a joint distribution over $X$ and $Y$, the **marginal probability** of an assignment $X=x$ is the result of considering the total probability associated with $X=x$ regardless of the assignment to $Y$:

\begin{equation}
(14) \quad\quad P(X=x) = \sum_{y} P(X=x, Y=y)
\end{equation}

where the sum is over all values in the image of the random variable $Y$.

Finally, another important rule shows us how to invert a conditional probability distribution, this is called the **Bayes rule**:

\begin{equation}
(15) \quad\quad P(X=x|Y=y) = \frac{P(Y=y|X=x)P(X=x)}{P(Y=y)}
\end{equation}

This follows trivially from the definition of conditional probability and the chain rule.

The next important concept to discuss is that of a random variable.

**Exercise 2-8** **[1 point]** Prove the *Bayes rule*

\begin{equation}
P(X=x | Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)} \quad\quad\text{conditional probability}\\
P(X=x, Y=y) = P(X=x)P(Y=y|X=x) \quad\quad\text{chain rule}\\
therefore\\
P(X=x | Y=y) = \frac{P(X=x)P(Y=y|X=x)}{P(Y=y)} \quad\quad\text{bayes rule}
\end{equation}

#  Representation and Maximum Likelihood Estimation

We can represent random variables in a computer by representing their distribution. Then we can *simulate/sample* values from that distribution. Here you will learn how to represent a Bernoulli variable and a Categorical variable as well as how to simulate them.

Recall that a Bernoulli variables takes exactly 1 parameter to represent, whereas a Categorical over $k$ possible events, takes exactly $k$ values.

**Bernoulli**: we will represent a bernoully variable by storing its Bernoulli parameter (a number between 0 and 1).

```python
import numpy as np
from __future__ import division

class Bernoulli:

    def __init__(self, p):
        """
            X ~ Bernoulli(p)
            where p is the probability of the positive class
        """
        if not (0 <= p <= 1):
            raise ValueError('The Bernoulli parameter must be a probability')
        self.p = p

    def sample(self):
        """Returns either 1 or 0 depending on a random simulation"""
        # We rely on numpy's pseudo random number generator to obtain a number uniformly distributed
        # between 0 and 1
        u = np.random.uniform()
        # The idea is to now transform this uniform simulation is such a way that with probability p
        #  we get 1 and with probability (1-p) we get a 0
        # For example, imagine p = 0.7 then we want to return 1 on average 70% of the times this
        #  code is executed.
        # Since `u` was uniformly generated, it did not bias any particular segment of the interval
        #  between 0 and 1. Therefore if we take the first 70% of that interval to return 1, we
        #  simulate the variable correctly. 70% of the interval between 0 and 1 can for example be represented
        #  by the interval between 0 and 0.7 (the p parameter)
        return 1 if u < self.p else 0

    def sample_n(self, N):
        """Returns an array of N samples"""
        return np.array([self.sample() for _ in range(N)])

    def pmf(self, value):
        """
        Probability mass function
            f_X(x) = P(X=x)
        value: an assignment of X
        returns: probability value for X=value
        """
        if value == 1:
            return self.p
        elif value == 0:
            return 1. - self.p
        else:
            raise ValueError('Bernoully variables can only take on 1 or 0')

    def cdf(self, a):
        """
        Cumulative probability function
            F_X(a) = \sum_{x <= a} P(X=x)
        a: an assignment of X
        returns: total probability over the interval [-inf, a]
        """
        if a == 0:
            return 1. - self.p
        elif a == 1:
            return 1.
        else:
            raise ValueError('Bernoully variables can only take on 0 or 1')

    def __repr__(self):
        return 'Bernoulli(%s)' % self.p

```

Now we can declare a Bernoulli random variable

```python
X = Bernoulli(0.7)
print('X ~', X)
for x in range(2):
    print(' P(X=%d) = %.2f F_X(%d) = %.2f' % (x, X.pmf(x), x, X.cdf(x)))
```

    ('X ~', Bernoulli(0.7))
     P(X=0) = 0.30 F_X(0) = 0.30
     P(X=1) = 0.70 F_X(1) = 1.00

Note that an rv does not give you a value (as normal variables in programming languages would), rvs give you a distribution from which you can sample an assignment.

```python
for i in range(10):
    x = X.sample()
    print('assignment {} with probability {:.2f}'.format(x, X.pmf(x)) )
```

    assignment 1 with probability 0.70
    assignment 1 with probability 0.70
    assignment 1 with probability 0.70
    assignment 0 with probability 0.30
    assignment 0 with probability 0.30
    assignment 1 with probability 0.70
    assignment 1 with probability 0.70
    assignment 1 with probability 0.70
    assignment 0 with probability 0.30
    assignment 0 with probability 0.30

We can answer some interesting questions about the distribution of a random variable once we get some dataset of observations.
So let's get a dataset of `N=1000` Bernoulli variables:

```python
N = 1000
bernoulli_observations = X.sample_n(N)
```

We can easily plot our samples to get an idea of what we obtained:

```python
%matplotlib inline
import matplotlib.pyplot as plt
_ = plt.hist(bernoulli_observations)
```

It turns out the **maximum likelihood estimate** of the Bernoulli parameter is nothing but the ratio in which we observe $X=1$. We can do that rather easily simply by counting the number of 1s and dividing by the total size of the dataset.

```python
def bernoulli_mle(observations):
    # this is the number of observations we have
    N = len(observations)
    count1 = observations.sum()
    return float(count1) / N
```

Now if we pretend we do not actually know the Bernoulli parameter, but we have Bernoulli observations, then we can derive a **parameter estimate** that is an educated guess as to what the underlying data generating process actually was.

```python
bernoulli_mle(bernoulli_observations)
```

    0.717

Note that we get something that's quite close to 0.7.

That is only the case because we have enough observations. For example, consider estimating the Bernoulli parameter for few examples (as few as `N=10`), you will notice that your estimates are not going to always look good.

```python
nb_repetitions = 10
for i in range(nb_repetitions):
    print('Attempt number {}: {}'.format(i, bernoulli_mle(X.sample_n(10))))
```

    Attempt number 0: 0.695
    Attempt number 1: 0.719
    Attempt number 2: 0.72
    Attempt number 3: 0.708
    Attempt number 4: 0.712
    Attempt number 5: 0.712
    Attempt number 6: 0.718
    Attempt number 7: 0.723
    Attempt number 8: 0.701
    Attempt number 9: 0.707

**Exercise 2-9** **[5 points]** Compare maximum likelihood estimation for a Bernoulli variable using 1, 10, 100, and 1000 samples. In order to gather enough data, repeat each experiment 100 times. Summarise the results of your investigation using a [boxplot](https://en.wikipedia.org/wiki/Box_plot) for each sample size and provide a comment on which aspects seem to improve as the sample size grows.

Your plot should look like this (give or take some small variance):

***SOLUTION*** We start with the experiment

```python
reps = 100
data = [[],[],[],[]]
for i in range(reps):
    data[0].append(bernoulli_mle(X.sample_n(1)))
    data[1].append(bernoulli_mle(X.sample_n(10)))
    data[2].append(bernoulli_mle(X.sample_n(100)))
    data[3].append(bernoulli_mle(X.sample_n(1000)))

plt.boxplot(data, 0)
plt.ylim(-0.1, 1.1)
plt.xticks(range(1,5), ('1', '10', '100', '1000'))
plt.xlabel('sample size')
plt.ylabel('MLE')
plt.show()

```

**Exercise 2-10** **[10 points]** Now you should extend the Bernoulli case to deal with Categorical random variables, that is

1. complete the skeleton class below (2 points in total)
    * note this requires completing the sampling procedure (1 point)
2. implement the maximum likelihood estimation algorithm (2 points)

Also keep in mind that your class is expected to make sure the parameters are *valid* (recall the conditions associated with categorical parameter vectors).

To test your representation (1 point):

* print the pmf and cdf values for all values the rv can take on (as we did for the Bernoulli example)

To test your MLE procedure (5 points):
* compute a maximum likelihood estimate using N samples from a Categorical distribution with parameters $[0.1, 0.2, 0.7]$
* repeat this procedure 1000 times to get the average estimate as well as error estimates: you can get average and error using `np.mean` and `np.std`
* plot the average estimate and standard deviation using barplots (with `plt.bar`)
* experiment with N = 1, N = 10, N = 100, and N = 1000
* comment on which aspects seem to improve as sample size grows
* this is all very similar to what was done in the Bernoulli case, but here we replace boxplot by barplots because our Categorical distributions has $k=3$ parameters (not just 1)

Hints:

* to simulate a categorical random variable you need to generalise the idea shown to you in the case of Bernoulli variables to K segments (rather than 2 segments); in the Bernoulli case we chopped the $[0,1]$ segment into two, namely, $[0, p)$ and $[p, 1]$. If you are not used to the notation $[a,b)$, it means a continuous interval between $a$ (included) and $b$ (excluded), that is, $a \leq x < b$. Categorical variables defined over K values require K segments chopped like this: $[0, \theta_1)$, $[\theta_1, \theta_1 + \theta_2)$, $[\theta_1 + \theta_2, \theta_1 + \theta_2 + \theta_3)$, and so on till $[\sum_{i=1}^{K-1} \theta_i, 1]$. Note how similar this is to the cdf of the categorical variable.
* the maximum likelihood estimate of a categorical variable is a vector where each position represents the ratio in which we observe the corresponding value of the random variable, that is `number of time k occurs divided by total number of data points`

Important: an alternative way to simulate a categorical variable (which will not be accepted as an aswer to this particular exercise) is to use `np.random.choice` (check its documentation with `np.random.choice?`).

Your plot should look like this (give or take some small variance):

```python
# Here we provide you with a sketch that you must complete

class Categorical:

    def __init__(self, theta):
        """
            X ~ Categorical(theta_1, ..., theta_K)
            where theta is the parameter vector and theta_k is the probability of the kth class
            recall that theta_k is bound between 0 and 1 and the sum should be exactly 1
        """
        if not (sum(theta) == 1) or not all(0 <= t <= 1 for t in theta) :
            raise ValueError("Error: The thetas should sum up to 1 and be between 0 and 1")
        self.theta = theta

    def sample(self):
        """Returns a number from 1 to K representing the sampled value"""
        p = np.random.uniform()

        s = 0
        for i, t in enumerate(self.theta):
            s += t
            if p <= s:
                return i + 1

    def sample_n(self, N):
        """Returns a numpy array containing N simulated values"""
        return np.array([self.sample() for _ in range(N)])

    def pmf(self, value):
        """
        Probability mass function evaluated at a certain value
        value: an assignment of X
        """
        if 0 < value < len(self.theta) + 1 and type(value) == int:
            return self.theta[value - 1]
        raise ValueError('Value must be one of the classes')

    def cdf(self, a):
        """
        Cumulative probability function
            F_X(a) = \sum_{x <= a} P(X=x)
        a: an assignment of X
        returns: total probability over the interval [-inf, a]
        """
        return sum(self.theta[0:a])

    def __repr__(self):
        return "Categorical(%s)" % self.theta

```

```python
def categorical_mle(observations, classes):
    N = len(observations)
    counted = [list(observations).count(k) for k in range(1,classes+1)]
    return np.array(counted) / N

categorical_mle(c.sample_n(1),3)
```

    array([ 0.,  1.,  0.])

```python
X = Categorical([0.1,0.2,0.7])
print('X ~', X)
for x in range(1, len(X.theta) + 1):

    print(' P(X=%d) = %.2f F_X(%d) = %.2f' % (x, X.pmf(x), x, X.cdf(x)))
```

    ('X ~', Categorical([0.1, 0.2, 0.7]))
     P(X=1) = 0.10 F_X(1) = 0.10
     P(X=2) = 0.20 F_X(2) = 0.30
     P(X=3) = 0.70 F_X(3) = 1.00

```python
# Plot configurations
plt.subplots()
plt.ylim(-0.3, 1.3)
plt.xlim(0.5,3.5)
plt.xticks([1,2,3])
# Plot colors, k = class number, w = width
c = ['r','b','y','g']
k = 0
w = 0.2

for N in [1,10,100,1000]:
    data = []
    for i in range(1000):
        data.append(categorical_mle(X.sample_n(N),len(X.theta)))

    data = np.array(data)
    mean = np.mean(data, axis=0)
    error = np.std(data, axis=0)
# Create bar plots for each class
    plt.bar(1 + k*w - w * 2,
            mean[0], w, color=c[k],
            yerr=error[0], ecolor='black')

    plt.bar(2 + k*w - w * 2,
            mean[1], w, color=c[k],
            yerr=error[1], ecolor='black')

    plt.bar(3 + k*w - w * 2,
            mean[2], w, color=c[k],
            yerr=error[2], ecolor='black')
    k += 1

plt.show()
```

### Comments on experiment
The error seems to get better (less) as you take more samples. This is sound as there are momre samples to average out.
You see that the mean stays more or less the same as soon as you get above N=1. At such a low sample size it is expected that the error and mean aren't representative of the distribution.


# Random Variables

$\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\EE}[2][\,\!]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\space}{\text{ }}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\CB}[1]{\left\{ #1 \right\}}$During an experiment, the quantities of interest, or the real-valued functions defined on the sample space are known as the ***Random Variables***, shortened as $r.v.$. And the value of the outcome variable is determined by the outcome of the experiment, we shall assign probabilities to the possible values of the $r.v.$.

A special case for this is the ***Indicator Variable***, denoted as $I$ for an event $E$.

$$I = \begin{cases}
1 & \text{if } E \text{ occurs} \\
0 & \text{if } E \text{ doesn't occur}
\end{cases}$$

**e.g.**
Suppose that independent trials, each of which results in any of $m$ possible outcomes with respectively probabilities $p_1, \dots, p_m$, $\sum p_i = 1$, are continually performed. Let $X$ denote the number of trials needed until each outcome has occurred at least once. What's $P\CB{X = n}$?

>Instead solve that directly, we first calculate $P\CB{X = n}$. Let $A_i$ denote the event that outcome $i$ has not yet occured after the first $n$ trials, $i = 1, \dots, m$.
>
>$\begin{align}
P\left\{X > n\right\} &= P\left( \bigcup_{i=1}^{m} A_i \right) \\
&= \sum_{i=1}^{m} P(A_i) - \underset{i<j}{\sum\sum} P(A_iA_j) \\
& \;\;\; + \underset{i<j<k}{\sum\sum\sum} P(A_iA_jA_k) - \cdots + (-1)^{m+1}P(A_1 A_2 \cdots A_m)
\end{align}$
>
>Now, $P(A_i)$ is the probability that each of the first $n$ trials results in a $\text{non-}i$ outcome, and so by independence
>
>$P(A_i) = (1 - p_i)^{n}$
>
>And similarly, $P(A_iA_j)$ is the probability that the first $n$ trials all result in a $\text{non-}i$ and $\text{non-}j$ outcome, and so
>
>$P(A_iA_j) = (1-p_i - p_j)^{n}$
>
>As all of the other possibilities are similar, we see that
>
>$\begin{align}
P\left\{X>n\right\} = \sum_{i=1}^{n}(1-p_i)^n - \underset{i<j}{\sum\sum} (1-p_i - p_j)^n + \underset{i<j<k}{\sum\sum\sum} (1 - p_i - p_j - p_k)^n - \cdots
\end{align}$
>
>Since $P\left\{X=n\right\} = P\left\{X>n-1\right\} -P\left\{X>n\right\}$, and $(1-a)^{n-1} - (1-a)^ n = a(1-a)^{n-1}$ that
>
>$\begin{align}
P\left\{X=n\right\} &= \sum_{i=1}^{m} p_i (1-p_i)^{n-1} - \underset{i<j}{\sum\sum} (p_i + p_j) (1-p_i - p_j)^{n-1} \\
&\;\;\;+ \underset{i<j<k}{\sum\sum\sum} (p_i+p_j+p_k)(1 - p_i - p_j - p_k)^n - \cdots
\end{align}$

***
Other than this **discrete** $r.v.$, we still have the **continuous** $r.v.$, like the lifetime of the car.

We also define the ***culmulative distribution function***, $F(\cdot)$, of the $r.v.$ $X$, on any real number $b$, $-\infty < b < \infty$, by $F(b) = P\left\{X \leq b\right\}$. And some properties of the cdf $F$ are:

- $F(b)$ is nondecreasing function of $b$.$\\[0.7em]$
- $\lim\limits_{b \to \infty} F(b) = F(\infty) = 1\\[0.7em]$
- $\lim\limits_{b \to -\infty} F(b) = F(-\infty) = 0$

Also, we have: $P\left\{a < X \leq b\right\} = F(b) - F(a)$ for all $a < b$. And for $P\left\{X<b\right\}$, we need a new strategy:

$$\begin{align}
P\left\{X<b\right\}&= \lim_{h \to 0^+} P\left\{X \leq b-h\right\}\\
&= \lim_{h \to 0^+} F(b-h)
\end{align}$$

just keep in mind that $P\left\{X<b\right\}$ *may not* equal to $F(b)$.

# Discrete $r.v.$

A $r.v.$ that can take on at most a **countable** number of possible values is said to be ***discrete***, say $X$. We can define its ***probability mass function*** $p(a)$ as: $p(a) = P\left\{X = a\right\}$.

Easy to find that $p(a)$ is **positive** for at most a countable number of values of $a$. So if $X$ must assume one of the values $x_1, x_2, \dots$, then $p(x_i) > 0$ for $i = 1, 2, \dots$ and $p(x_i)=0$ for all other values of $x$.

Direct conclusions would be $\sum_{i=1}^{\infty}p(x_i) = 1$ and $F(a) = \sum_{x_i \leq a}x_i$

## The Bernoulli $r.v.$

For those $r.v.$ with the probability mass function defined as

$$
\begin{cases}
p(0) = P\left\{X=0\right\} = 1-p\\[0.5em]
p(1) = P\left\{X=1\right\} = p
\end{cases}
$$

where $0 < p < 1$, namely, the probability of successful trial.

## The Binomial $r.v.$

Suppose that $n$ independent trials, each of which results in a success with probability $p$ and a failure with probability $1-p$. Let $X$ denote the **number of successes** that occur in the $n$ trials, then $X$ is said to be a ***Binomial*** $r.v.$ with **parameters** $(n,p)$.

It's probability mass function is given by $p(i) = \d{\binom{n} {i}}p^i(1-p)^{n-i}$ for $i=0,1,\dots,n$, and it's easy to verify that this holds:

$$\sum_{i=0}^{\infty}p(i) = \sum_{i=0}^{n} \binom{n} {i}p^i(1-p)^{n-i} = (p+(1-p))^{n} = 1$$

## The Geometric $r.v.$

Suppose that independent trials, each having probability $p$ of being a success and denote $X$ as the number of trials required until the first success. Then this $X$ is said to be a ***geometric*** $r.v.$ with parameter $p$. It's probability mass function is given by $p(n) = P\left\{X=n\right\} = (1-p)^{n-1}p$ for $n = 1,2,\dots$

And it's easy to verify that $\sum\limits_{n=1}^{\infty} p(n) = p\sum\limits_{n=1}^{\infty} (1-p)^{n-1} = 1$

## The Poisson Random Variable

For $r.v.$ $X$, taking on one of the values $i = 0,1,\dots$ with probability mass function given by

$$p(i) = P\left\{X=i\right\} = e^{-\lambda} \frac{\lambda^i} {i!}$$

And it's easy to verify that $\sum\limits_{i=0}^{\infty} p(i) = e^{-\lambda} \sum\limits_{i=0}^{\infty} \ffrac{\lambda^i} {i!} = e^{-\lambda}e^{\lambda} = 1$

One important application is to **approximate** a *binomial* $r.v.$, with large $n$ and small $p$.

$$\begin{align}
P_{\text{binom}}\left\{X=i\right\} &= \binom{n} {i} p^i (1-p)^{n-i} \\
&= \frac{n!} {(n-i)!i!} \left( \frac{\lambda} {n} \right)^i \left( 1 - \frac{\lambda} {n} \right)^{n-i} \\[0.6em]
&= \frac{n(n-1)(n-2) \cdots (n-i+1)} {n!} \frac{\lambda^i} {i!} \frac{(1-\lambda/n)^n} {(1-\lambda/n)^i} \\
& \approx 1 \cdot \frac{\lambda^i} {i!} \cdot \frac{e^{-\lambda}} {1} = P_{\text{poisson}}\left\{X=i\right\}
\end{align}$$

$Remark$

- $0!=1$
- For poisson distributed $X$, $\EE{X} = \lambda$

# Continuous $r.v.$

Now the $r.v.$ can take on a uncountable set of values, also say $X$. We say $X$ is a ***continuous*** $r.v.$ if there exists a **nonnegative** function $f(x)$, defined for all real $x \in (-\infty, \infty)$, having the property that for any set $B$ of real numbers,

$$P\left\{X \in B\right\} = \int_{B} f(x) \;\dd{x}$$

here $f(x)$ is the ***probability density function*** of $X$. It must sastify $P\CB{X \in \left( -\infty, \infty\right)} = \d{\int_{-\infty}^{\infty} f(x)\;\dd{x} = 1}$. And one funny thing about this is for any *particular value* assumed to $X$ like $a$, $P\CB{X = a} = \d{\int_{a}^{a}f(x)\;\dd{x}}=0$.

Also, we can use this to define the cumulative distribution $F(\cdot)$: $F(a) = \d{\int_{-\infty}^{a} f(x) \;\dd{x}}$, then we can differentiate both sides and it yields: $\ffrac{\dd{}} {\dd{a}}F(a) = f(a)$.

## The Uniform Random Variable

A $r.v.$ is said to be ***uniformly distributed*** over the interval $(0,1)$ if its pdf is given by

$$f(x) = \begin{cases}
1 & \text{if } 0 < x < 1 \\[0.6em]
0 & \text{otherwise}
\end{cases}$$

And in general, we say that $X$ is a uniform random variable on the interval $(\alpha, \beta)$ if its pdf is given by

$$f(x) = \begin{cases}
\ffrac{1} {\beta - \alpha} & \text{if } \alpha < x < \beta \\[0.6em]
0 & \text{otherwise}
\end{cases}$$

## Exponential Random Variables

A continuous $r.v.$ whose pdf is given, for some $\lambda > 0$, by,

$$f(x) = \begin{cases}
\lambda e ^{-\lambda x} & \text{if }x\geq 0 \\[0.6em]
0 & \text{if } x<0
\end{cases}$$

is said to be an ***exponential*** $r.v.$ with parameter $\lambda$. And for its cdf, we have

$$F(a) = \begin{cases}
\d{\int_{0}^{a} \lambda e ^{-\lambda x} \;\dd{x}} = 1 - e^{-\lambda a} & \text{if } a\geq 0 \\[0.6em]
0 & \text{if } a<0
\end{cases}$$

And also it's easy to verify that $F(\infty) = \d{\int_{0}^{\infty} \lambda e ^{-\lambda x} \;\dd{x} = 1}$

## Gamma Random Variables

A continuous $r.v.$ whose pdf is given, for some $\lambda > 0$ and $\alpha > 0$, by

$$f(x) = \begin{cases}
\ffrac{\lambda e ^{-\lambda x} (\lambda x)^{\alpha-1}} {\Gamma(\alpha)} & \text{if } x\geq 0 \\[0.6em]
0 & \text{if } x<0
\end{cases}$$

is said to be a ***gamma*** $r.v.$ with parameter $\alpha$, $\lambda$, and ***gamma function***, $\Gamma(\alpha) = \d{\int_{0}^{\infty} e^{-x} x^{\alpha - 1} \; \dd{x}}$.

$Remark$

By induction we can show that $\Gamma(n) = (n-1)!$ for integral $n$.

>$$\begin{align}
\Gamma(n) &= \int_{0}^{\infty} e^{-x} x^{n-1} \;\dd{x} = (n-1)! \\
&= \int_{0}^{\infty} e^{-x}\; \ffrac{\dd{x^{n}}} {n}\\
&= \left.\ffrac{e^{-x}x^n} {n}\right|_{0}^{\infty} - \int_{0}^{\infty} -e^{-x} \ffrac{x^{n}} {n} \;\dd{x}
\end{align}$$

## Normal Random Variables

We say that $X$ is a ***normal*** $r.v.$ with parameters $\mu$ and $\sigma^2$ if its pdf is given by

$$f(x) = \frac{1} {\sqrt{2\pi\sigma^2}} \exp\CB{-\ffrac{(x-\mu)^2} {2\sigma^2}}$$

with $x \in \mathbb{R}$. It's density function is a bell-shaped curve that is symmetric around $\mu$.

$Remark$

If $X$ is normally distributed with parameters $\mu$ and $\sigma^2$, then for $Y = \alpha X + \beta$, it's also normally distributed with parameters $\alpha \mu + \beta$ and $\alpha^2 \sigma^2$. and from the linearity, $Y \in \mathbb{R}$.

>$$\begin{align}
F_Y(a) &= P(Y \leq a) = P(\alpha X + \beta \leq a)\\[0.6em]
&= F_X \left( \ffrac{a-\beta} {\alpha} \right) \\
&= \int_{-\infty}^{(a - \beta)/\alpha} \frac{1} {\sqrt{2\pi\sigma^2}} \exp\CB{-\ffrac{(x-\mu)^2} {2\sigma^2}} \;\dd{x} \\
&\stackrel{ y = \alpha x + \beta} {=} \int_{-\infty}^{a}\frac{1} {\sqrt{2\pi}\sigma\alpha} \exp\CB{-\ffrac{(y-(\alpha x + \beta))^2} {2\sigma^2\alpha^2}} \;\dd{y}
\end{align}$$

$Remark$

The previous result can be applied inversely so that any normally distributed $r.v.$ $X$ can be transformed into a specific one with parameters $0$ and $1$, by conducting $Z = (X - \mu)/\sigma$

# Expectation of a Random Variable
## The Discrete Case

If $X$ is a discrete $r.v.$ having a pmf $p(x)$, the the ***expected value*** of $X$ is defined by:

$\d{\EE{X} = \sum_{x:p(x)>0}xp(x)}$

**e.g.**
Expectation of a **Bernoulli** $r.v.$

> $\EE{X} = 0 \cdot (1-p) + 1 \cdot p = p $

**e.g.**
Expectation of a **Binomial** $r.v.$

> $\begin{align}
\EE{X} &= \sum_{i=0}^{n} i \cdot p(i) = \sum_{i=0}^{n} i \cdot \binom{n} {i} p^i (1-p)^{n-i} \\
&= \sum_{i=\mathbf{1}}^{n} \ffrac{n!} {(n-i)!(i-1)!} p^i (1-p)^{n-i} \\
&= np \sum_{i=\mathbf{1}}^{n} \ffrac{(n-1)!} {(n-i)!(i-1)!} p^{i-1} (1-p)^{n-i} \\
&\stackrel{k=i-1}{=} np \sum_{k=\mathbf{0}}^{n-1} \cdot \binom{n-1} {k} p^k (1-p)^{n-1-k} \\
&= np\left[p+(1-p)\right]^{n-1} = np
\end{align}$

**e.g.**
Expectation of a **Geometric** $r.v.$

> $\begin{align}
\EE{X} &= \sum_{n=1}^{\infty} n \cdot p(1-p)^{n-1} \\
&\stackrel{q=1-p}{=} p \sum_{n=1}^{\infty}nq^{n-1} \\
&= p \sum_{n=1}^{\infty} \ffrac{\dd{}} {\dd{q}}q^{n} \\
&= p \ffrac{\dd{}} {\dd{q}} \left( \ffrac{q} {1-q} \right)\\
&= \ffrac{p} {(1-q)^2} = \ffrac{1} {p}
\end{align}$
>
> 错位相减法 also works

**e.g.**
Expectation of a **Poisson** $r.v.$

> $\begin{align}
\EE{X} &= \sum_{i=0}^{\infty} i\cdot\ffrac{e^{-\lambda}\lambda^i} {i!} \\
&= \lambda e^{-\lambda} \sum_{i=\mathbf{1}}^{\infty} \ffrac{\lambda^{i-1}} {(i-1)!} \\
&= \lambda e^{-\lambda} e^{\lambda} = \lambda
\end{align}$

***

## The Continuous Case

For $r.v.$ $X$ with pdf $f(x)$, its expected value is defined by $\EE{X} = \d{\int_{-\infty}^{\infty} xf(x) \;\dd{x}}$.

**e.g.**
Expectation of a **Uniform** $r.v.$

> $\begin{align}
\EE{X} &= \int_{\alpha}^{\beta} x \cdot \frac{1} {\beta - \alpha} \; \dd{x} \\
&= \frac{\beta^2 - \alpha^2} {2(\beta - \alpha)} = \frac{\beta + \alpha} {2}
\end{align}$

**e.g.**
Expectation of a **Exponential** $r.v.$

> $\begin{align}
\EE{X} &= \int_{0}^{\infty} x \cdot \lambda e^{-\lambda x} \;\dd{x}  = \int_{0}^{\infty} -x\;\dd{e^{-\lambda x}}\\
&= \left. -xe^{-\lambda x}\right|_{0}^{\infty} + \int_{0}^{\infty} e^{-\lambda x} \; \dd{x} \\
&= 0 - \left. \frac{e^{-\lambda x}} {\lambda} \right|_{0}^{\infty} \\
&= \frac{1} {\lambda}
\end{align}$

**e.g.**
Expectation of a **Normal** $r.v.$

> $\begin{align}
\EE{X} &= \frac{1} {\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} x \cdot \exp\CB{-\frac{(x-\mu)^2} {2\sigma^2}} \;\dd{x} \\
&\stackrel{y=x-\mu}{=}\frac{1}{\sqrt{2\pi}\sigma}\left( \int_{-\infty}^{\infty} y \exp\CB{-\frac{y^2} {2\sigma^2}} \; \dd{y} + \mu \int_{-\infty}^{\infty} \exp\CB{-\frac{(x-\mu)^2} {2\sigma^2}} \; \dd{x} \right) \\
&= \frac{1}{\sqrt{2\pi}\sigma} \int_{-\infty}^{\infty} y \exp\CB{-\frac{y^2} {2\sigma^2}} \; \dd{y} + \mu \int_{-\infty}^{\infty}  f(x) \; \dd{x}
\end{align}$

>By symmetricity of the first term we can conclude that $\EE{X} = \mu \cdot 1 = \mu$.

***

## Expectation of a Function of a $r.v.$

$Proposition$

For $X$ with pmf $p(x)$, and any real-valued function $g$, $\EE{g(X)} = \d{\sum_{x:p(x)>0}} g(x)p(x)$. And for those with pdf $f(x)$, we have $\EE{g(X)} = \d{\int_{-\infty}^{\infty}} g(x)f(x) \;\dd{x}$

$Corollary$

For constant $a$ and $b$, then $\EE{aX+b} = a\EE{X} + b$.

***

We also call the quantity $\EE{X^n}$ for $n \geq 1$ the $n\text{-th}$ ***moment*** of $X$. And the variance, defined by $\Var{X} = \EE{(X - \EE{X})^2}$.

**e.g.**
Variance of the ***Normal*** $r.v.$

> $\begin{align}
\Var{X} &= \EE{(X - \mu)^2} \\
&= \frac{1} {\sqrt{2\pi}\sigma}\int_{-\infty}^{\infty} (x-\mu)^2 \exp\CB{-\frac{(x-\mu)^2} {2\sigma^2}}\;\dd{x} \\
&\stackrel{y=(x-\mu)/\sigma}{=} \frac{\sigma^2} {\sqrt{2\pi}} \int_{-\infty}^{\infty} y^2 \exp\CB{-\frac{y^2} {2}} \;\dd{y} \\
&= \frac{\sigma^2} {\sqrt{2\pi}} \int_{-\infty}^{\infty} -y \;\dd{e^{-y^2/2}}\\
&= \frac{\sigma^2} {\sqrt{2\pi}} \left( \left.-ye^{-y^2/2}\right|_{-\infty}^{\infty} + \int_{-\infty}^{\infty} e^{-y^2/2} \;\dd{y} \right) \\
&= \frac{\sigma^2} {\sqrt{2\pi}} \cdot \int_{-\infty}^{\infty} e^{-y^2/2} \;\dd{y} \\
&= \sigma^2
\end{align}$

***

$Remark$

About the provement of $\d{\int_{-\infty}^{\infty} e^{-y^2/2} \;\dd{y}} = \sqrt{2\pi}$, u can use the method of double integral. Well, I am gonna think out of another way. (But failed finnaly... sad)

$Remark$

Another formular will connect the expectation and the variance: $\Var{X} = \EE{X^2} - (\EE{X})^2$, for both continuous case and discrete case.

# Jointly Distributed Random Variables
##  Joint Distribution Functions

For any two $r.v.$s $X$ and $Y$, we can define the ***joint cumulative probability distritbution function*** of $X$ and $Y$ by

$$F(a,b) = P\CB{X \leq a, Y \leq b}$$

for $a,b \in \mathbb{R}$. And with this we can find the ***marginal cumulative probability distribution*** like:

$$\begin{align}
F_X(a) &= P\CB{X \leq a} = P \CB{X \leq a, Y < \infty} = F(a, \infty)\\
F_Y(b) &= F(\infty, b)
\end{align}$$

In the case where $X$ and $Y$ are both discrete $r.v.$, it's also convenient to define the ***joint probability mass function*** of $X$ and $Y$ by: $p(x,y) = P\CB{X = x, Y=y}$, then following the ***marginal probability mass function*** like:

$$
p_X(x) = \sum_{y:p(x,y)>0} p(x,y) \;\lvert\; p_Y(y) = \sum_{x:p(x,y)>0} p(x,y)
$$

We say that $X$ and $Y$ are ***jointly continuous*** if there exists a function $f(x,y)$, namely, the ***joint probability density funciton*** of $X$ and $Y$, defined for all real $x$ and $y$, having the property that for all sets $A$ and $B$ of real numbers this holds:

$$\d{P\CB{X \in A, Y \in B} = \int_B \int_A f(x,y) \; \dd{x} \; \dd{y}}$$

And the ***marginal*** part:

$$\begin{align}
P\CB{X\in A} &= P\CB{X \in A, Y \in (-\infty,\infty)} \\
&= \int_{-\infty}^{\infty} \int_A f(x,y) \;\dd{x} \; \dd{y} \\
&= \int_A f_X(x)\;\dd{x}
\end{align}$$

where $f_X(x) = \d{\int_{-\infty}^{\infty} f(x,y) \; \dd{y}}$, which is how we obtain the marginal pdf of $X$.

And because $F(a,b) = P\CB{X \leq a,Y \leq b}=\d{\int_{-\infty}^{a}\int_{-\infty}^{b} f(x,y) \;\dd{y} \;\dd{x}}$, differentiation yields:

$$\ffrac{\dd{}^2} {\dd{a}\;\dd{b}}F(a,b) = f(a,b)$$

The expectation can be calculated by

$$
\EE{g(X,Y)} = \begin{cases}
\d{\sum_y \sum_x g(x,y) p(x,y)} & \text{discrete case}\\
\d{\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} g(x,y) f(x,y) \;\dd{x}\;\dd{y}} &\text{continuous case}
\end{cases}
$$

A direct application of this is

$$\EE{\sum_{i=1}^{n}a_iX_i} = \sum_{i=1}^{n}a_i\EE{X_i}$$

for $n$ $r.v.$s and $n$ constants $a_1, \dots, a_n$.

$Remark$

Only applicable for linear combination!

**(V)e.g.**
Choose $10$ letters from $A$ to $Z$. Compute the expected number of different types that are contained in a set of $10$ letters.

> It's hard to calculate that directly, so we break it apart, $10$ parts. We first define $X_i$ as

>$$X_i = \begin{cases}
1, & \text{if at least one type of letter } i \text{ is in the set of } 10\\
0, & \text{otherwise}
\end{cases}$$

>Then $X$, as the number of different types in the set of $10$ letters, we have $X = \sum X_i$. And we have

>$$\begin{align}
\EE{X_i} &= P\CB{X_i = 1} \\
&= 1 - P\CB{\text{no type of letter }i\text{ are in the set of }10} \\
&= 1 - \left(\ffrac{25} {26}\right)^{10}
\end{align}$$
>
>So that $\EE{X} = \sum\EE{X_i} = 26\left[1 - \left(\ffrac{25} {26}\right)^{10}\right]$

***

## Independent $r.v.$

$X$ and $Y$ are said to be ***independent*** if for all $a$, $b$, we have $P\CB{X \leq a, Y \leq b} = P\CB{X \leq a}\cdot P\CB{Y \leq b}$. *In other words*, the events $E_a = \CB{X \leq a}$ and $F_b = \CB{Y \leq b}$ are independent.

In terms of the joint distribution function $F$, we have that $X$ and $Y$ are independent if for $\forall a,b$,

$$F(a,b) = F_X (a) \cdot F_Y(b)$$

which can also be reduced to

$$\begin{cases}
p(x,y) &\!\!\!\!= p_X(x) \cdot p_Y(y) & \text{discrete case}\\[0.6em]
f(x,y) &\!\!\!\!= f_X(x) \cdot f_Y(y) & \text{continuous case}
\end{cases}$$

$Proposition$

If $X$ and $Y$ are independent, the for any functions $h$ and $g$: $\EE{g(X)\cdot h(Y)} = \EE{g(X)} \cdot \EE{h(Y)}$.

## Covariance and Variance of Sums of $r.v.$

The covariance of *any* two random variables $X$ and $Y$, denoted by $\Cov{X,Y}$, is defined by

$$\begin{align}
\Cov{X, Y} &= \EE{(X - \EE{X}) \cdot (Y - \EE{Y})} \\
&= \EE{XY - Y\EE{X} - X\EE{Y} + \EE{X}\EE{Y}} \\
&= \EE{XY} - \EE{X} \EE{Y}
\end{align}$$

Easy to see that if $X$ and $Y$ are independent, then $\Cov{X,Y} = 0$

$Remark$

In general it can be shown that a **positive** value of $\Cov{X,Y}$ is an **indication** that $Y$ tends to increase as $X$ does, whereas a negative value indicates that $Y$ tends to decrease as $X$ increase.

**e.g.**

Given the joint density function of $X$, $Y$, $f(x) = \ffrac{1} {y} \exp\CB{-y-\ffrac{x} {y}}$, for $0 < X,Y < \infty$. Verify that and find the covariance.

> For the verification:

>$$\begin{align}
\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} f(x,y) \;\dd{y} \;\dd{x} &= \int_{0}^{\infty}\int_{0}^{\infty} \ffrac{1} {y} \exp\CB{-y-\frac{x} {y}} \;\dd{y} \;\dd{x} \\
&= \int_{0}^{\infty} e^{-y} \int_{0}^{\infty} \frac{1} {y} \exp\CB{-\frac{x} {y}} \;\dd{x} \;\dd{y}\\
&= \int_{0}^{\infty} e^{-y} \;\dd{y} \\
&= 1
\end{align}$$

>And for the covariance we first need the expectation of separate $r.v.$s. Two ways available. For $\EE{X}$,

>$$
\begin{align}
\EE{X} &= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} x\cdot f(x,y) \;\dd{y} \;\dd{x} \\
&= \int_{0}^{\infty} e^{-y} \int_{0}^{\infty} \frac{x} {y} \exp\CB{-\frac{x} {y}} \;\dd{x} \;\dd{y}
\end{align}$$

>Note that $\d{\int_{0}^{\infty} \frac{x} {y} \exp\CB{-\frac{x} {y}} \;\dd{x}}$ is the exponential $r.v.$ with parameter $\ffrac{1}{y}$ and thus is equal to $y$. Consequently, $\EE{X} = \d{\int_{0}^{\infty} y e^{-y} \;\dd{y} = 1}$.

>***
>Then for $\EE{Y}$, we need another method. We first calculate the marginal probablity $f_Y(y)$.
>
>$f_Y(y) = e^{-y} \d{\int_{0}^{\infty} \ffrac{1} {y} \exp\CB{-\ffrac{x} {y}}\;\dd{x}} = e^{-y}$, then $\EE{Y} = 1$.
>
>***
>$$
\begin{align}
\EE{XY} &= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} xy \cdot f(x,y) \;\dd{y} \;\dd{x} \\
&= \int_{0}^{\infty} y e^{-y} \int_{0}^{\infty} \frac{x} {y} \exp\CB{-\frac{x} {y}} \;\dd{x} \;\dd{y} \\
&= \int_{0}^{\infty} y^2 e^{-y} \;\dd{y} \\
&= \int_{0}^{\infty} -y^2 \;\dd{e^{-y}} \\
&= \left.-y^2 e^{-y}\right|_{0}^{\infty} + \int_{0}^{\infty} 2ye^{-y} \;\dd{y} = 2\EE{Y} = 2
\end{align}$$

>Consequently, $\Cov{X,Y} = \EE{XY} - \EE{X}\EE{Y} = 1$

$Remark$

>Covariance equaling to $0$ can't imply that the two are independent, the inverse statement is true though.

$Other \space properties$

For any $r.v.$s $X$, $Y$, $Z$ and constant $c$, we have

- $\Cov{X,X} = \Var{X}\\[0.5em]$
- $\Cov{X,Y} = \Cov{Y,X}\\[0.5em]$
- $\Cov{cX,Y} = c \cdot\Cov{X, Y}\\[0.5em]$
- $\Cov{X,Y+Z} = \Cov{X,Y} + \Cov{X,Z}$

And the generalized forth property: $\d{\Cov{\sum_{i=1}^{n}X_i,\sum_{j=1}^{m}Y_j}=\sum_{i=1}^{n}\sum_{j=1}^{m} \Cov{X_i,Y_j}}$.

And one more application for variance

$$
\begin{align}
\Var{\sum_{i=1}^{n} X_i} &= \Cov{\sum_{i=1}^{n}X_i,\sum_{j=1}^{n}X_j} \\
&= \sum_{i=1}^{n} \sum_{j=1}^{n} \Cov{X_i, X_j} \\
&= \sum_{i=1}^{n}\Cov{X_i, X_i} + \sum_{i=1}^{n} \sum_{j \neq i} \Cov{X_i, X_j} \\
&= \sum_{i=1}^{n}\Var{X_i} + 2 \sum_{i=1}^{n} \sum_{j < i} \Cov{X_i, X_j}
\end{align}$$

Even, when $X_i$ are independent, this will reduce to $\d{\Var{\sum_{i=1}^{n}X_i} = \sum_{i=1}^{n} \Var{X_i}}$

$Def$

If $X_1, X_2, \dots, X_n$ are **independent** and **identically distributed**, we define the ***sample mean*** as

$$\bar{X} = \frac{1} {n}\sum_{i=1}^{n} {X_i}$$

$Proposition$

Suppose that $X_1, \dots, X_n$ araea independent distributed with expected value $\mu$ and variance $\sigma^2$. Then:

- $\EE{\bar{X}} = \mu$
- $\Var{\bar{X}} = \ffrac{\sigma^2}{n}\\[0.5em]$
- $\Cov{\bar{X}, X_i - \bar{X}} = 0$, $i = 1,2,\dots,n\\[0.5em]$

**e.g.**
Variance of a **Binomial** $r.v.$

>We first break it up. $X = X_1 +\cdots+X_n$, with the $n$ components from $n$ independent *Bernoulli* $r.v.$. Then we have

>$$\Var{X} = \sum \Var{X_i}$$

>Since $\Var{X_i} = \EE{X_i^2} - \left(\EE{X_i}\right)^2 = p - p^2$, $\Var{X} = np(1-p)$

***

**e.g.** ***The Hypergeometric***

Consider $N$ individuals with $p$ percent of whom are in favor of a certain proposition and the rest are opposed, where $p$ is assumed to be *unknown* and required to *estimate*. We will randomly choosing and then determining the positions of $n$ members of the population.

>We use the portion of the favored in the sample as an estimator of $p$. First we let
>
>$$X_i = \begin{cases}
1, &\text{if the }i\texttt{th}\text{ person chosen is in favor} \\[0.5em]
0, &\text{otherwise}
\end{cases}$$
>
>Then the estimator of $p$ is $\ffrac{1} {n}\sum_{i=1}^{n} X_i$. We now compute its mean and variance for a little comparison
>
>$\d{\EE{\ffrac{1} {n}\sum_{i=1}^{n} X_i} = \ffrac{1} {n}\sum_{i=1}^{n} \EE{X_i} } = p$
>
>$\d{\Var{\ffrac{1} {n}\sum_{i=1}^{n} X_i} = \ffrac{1} {n^2} \left(\sum_{i=1}^{n} \Var{X_i} + 2 \underset{i<j}{\sum\sum} \Cov{X_i, X_j}\right)}$
>
>Easy to see that $X_i$ is a **Bernoulli** $r.v.$ so that $\Var{X_i} = p(1-p)$, so now we get down to handling the covariance.
>
>$\begin{align}
\Cov{X_i,X_j} &= \EE{X_i \cdot X_j} - \EE{X_i} \cdot \EE{X_j} \\[0.5em]
&= P\CB{X_i = 1, X_j = 1} - p^2 \\[0.5em]
&= \ffrac{Np} {N} \cdot \ffrac{Np-1} {N-1} - p^2\\[0.5em]
\end{align}$
>
>$\begin{align}
\Var{\ffrac{1} {n}\sum_{i=1}^{n} X_i} &= \ffrac{1} {n^2} \left[ np(1-p) + 2\binom{n}{2} \left( \ffrac{Np} {N} \cdot \ffrac{Np-1} {N-1} - p^2 \right) \right] \\
&= \ffrac{p(1-p)} {n} - \ffrac{(n-1)p(1-p)} {n(N-1)} = \ffrac{p(1-p)(N-n)} {n(N-1)}
\end{align}$

$Remark$

When $N$ increases, the variance goes larger and the limiting value as $N \to \infty$ is $p(1-p)/n$, which is not surprising since for $N$ large enough each $X_i$ can be considered as *independent* **bernoulli** $r.v.$ and thus $\sum X_i$ can be considered as **binomial** distribution with parameter $n$ and $p$.

$Remark$

The real ***Hypergeometric*** $r.v.$ is brought out like this: Totally $N$ identities with $p$ percent with a feature and the rest not. Then we select $n$ identities from all $N$ and denote $X$ as the number of identities with that feature:

$$\d{P\CB{X=k}} = \ffrac{\d{\binom{Np} {k}\binom{N-Np} {n-k}}} {\d{\binom{N} {n}}}$$

And an easy example for this is to consider an urn with $Np$ red balls and $N-Np$ blu balls in. We take $n$ balls out and this is the *distribution* of the number of blue balls.
***

Another thing is the ***convolution*** of the distributions $F_X$ and $F_Y$, the distribution of $X+Y$, from the distributions of $X$ and $Y$, given they are **independent**.

$$
\begin{align}
F_{X+Y}(a) &= P \CB{X+Y \leq a} \\
&= \iint_{x+y \leq a} f(x) g(y) \;\dd{x} \;\dd{y} \\
&= \int_{-\infty}^{\infty} \int_{-\infty}^{a-y} f(x) g(y) \;\dd{x} \;\dd{y}\\
&= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{a-y} f(x) \;\dd{x} \right) g(y) \;\dd{y} \\
&= \int_{-\infty}^{\infty} F_X (a-y) g(y) \;\dd{y}
\end{align}$$

Then we differentiating both sides of the equation above, the pdf comes:

$$
\begin{align}
\ffrac{\dd{}} {\dd{a}} F_{X+Y}(a) &= \ffrac{\dd{}} {\dd{a}} \int_{-\infty}^{\infty} F_X (a-y) g(y) \;\dd{y} \\[0.7em]
f_{X+Y}(a) &= \int_{-\infty}^{\infty} \ffrac{\dd{}} {\dd{a}} \big(F_X (a-y)\big)g(y) \;\dd{y} \\
&= \int_{-\infty}^{\infty} f(a-y) g(y) \;\dd{y}
\end{align}$$

**(V)e.g.** **Sum** of Two Independent **Uniform** $r.v.$

Given $X$ and $Y$ are independent $r.v.$ both uniformly distributed on $(0,1)$, find the pdf of $X+Y$.

>First we have $f(z) = g(z) = \begin{cases}
1, & 0 < z < 1 \\[0.5em]
0, & \text{otherwise}
\end{cases}$, and with the previous formula we have:
>
>$$f_{X+Y} (z) = \int_{-\infty}^{\infty} f(z-y)g(y) \;\dd{y} = \int_0^1f(z-y)\;\dd{y}$$
>
>Then for $0 \leq z  \leq 1$, this yields $\d{f_{X+Y}(z) = \int_{0}^{z} \;\dd{y} = z}$. For $1 < z < 2$, we get $\d{f_{X+Y} (z) = \int_{z-1}^{1}\;\dd{y} = 2-z}$. Hence we draw the conclusion as
>
>$$
f_{X+Y}(z) = \begin{cases}
z, & 0 \leq z \leq 1 \\[0.5em]
2-z, & 1 < z < 2 \\[0.5em]
0, & \text{otherwise}
\end{cases}$$
>
>Just for fun, I also calculate the "triple" one:

>$$
f_{Z+Y}(w) = \begin{cases}
\frac{1} {2}w^2, & 0 \leq z \leq 1 \\[0.5em]
-w^2 + 3w - \frac{3} {2}, & 1 < z \leq 2 \\[0.5em]
\frac{1} {2}w^2 - 3w + \frac{9} {2} = \frac{1} {2}(w-3)^2, & 2 < z < 3 \\[0.5em]
0, & \text{otherwise}
\end{cases}$$

**e.g.** **Sum** of Independent **Poisson** $r.v.$

Let $Χ$ and $Y$ be independent **Poisson** $r.v.$ with respective means $\lambda_1$ and $\lambda_2$.

>$$\begin{align}
P\CB{X + Y = n} &= \sum_{k=0}^{n} P \CB{X=k, Y = n-k} \\
&= \sum_{k=0}^{n} P\CB{X = k} \cdot P\CB{Y = n-k} \\
&= \sum_{k=0}^{n} e^{-\lambda_1} \ffrac{\lambda_1^k} {k!} \cdot e^{-\lambda_2} \ffrac{\lambda_2^{n-k}} {(n-k)!}\\
&= \ffrac{e^{-\lambda_1 - \lambda_2}} {n!} \sum_{k=0}^{n} \ffrac{n!} {k!(n-k)!} \lambda_1^k \lambda_2^{n-k} \\
&= \ffrac{e^{-\lambda_1 - \lambda_2}} {n!} (\lambda_1 + \lambda_2)^n
\end{align}$$

>In words, $X_1+X_2$ has a **Poisson** distribution with mean $\lambda_1 + \lambda_2$.

$Remark$

The general idea of independency is for all values $a_1, a_2, \dots, a_n$, we have

$$P\CB{X_1 \leq a_1, X_2 \leq a_2, \dots, X_n \leq a_n} = P\CB{X_1 \leq a_1} \cdot P\CB{X_2 \leq a_2} \cdot  \cdots \cdot P\CB{X_n \leq a_n} $$

The ***Order Statistics***

Let $X_1, \dots, X_n$ be $i.i.d.$ continuous $r.v.$ with cdf $F$ and pdf $f = F'$. Define $X_{(i)}$ as the $i\texttt{th}$ smallest of these $r.v.$, then $X_{(1)}, \dots, X_{(n)}$ are called the ***Order Statistics*** . Find their distributions.

 $$P\CB{X_{(i)} \leq x} = \sum_{k=i}^{n} \binom{n} {k} \big(F(x)\big)^k \big( 1-F(x) \big)^{n-k}$$

Differentiation yields that the density function of $X_{(i)}$ is as follows:

$$
\begin{align}
f_{X_{(i)}} (x) &= f(x) \left( \sum_{k=i}^{n}\binom{n} {k} \cdot k \big(F(x)\big)^{k-1} \big( 1-F(x) \big)^{n-k} - \sum_{k=i}^{n}\binom{n} {k} \big(F(x)\big)^{k} \cdot (n-k) \big( 1-F(x) \big)^{n-k-1} \right) \\
&= f(x) \left( \sum_{k=i}^{n} \ffrac{n!} {(n-k)!(k-1)!} \big(F(x)\big)^{k-1} \big( 1-F(x) \big)^{n-k} - \sum_{k=i}^{n}\ffrac{n!} {(n-k-1)!k!} \big(F(x)\big)^{k} \big( 1-F(x) \big)^{n-k-1} \right) \\
&= f(x) \left( \sum_{k=i}^{n} \ffrac{n!} {(n-k)!(k-1)!} \big(F(x)\big)^{k-1} \big( 1-F(x) \big)^{n-k} - \sum_{j=i+1}^{n}\ffrac{n!} {(n-j)!(j-1)!} \big(F(x)\big)^{j-1} \big( 1-F(x) \big)^{n-j} \right) \\
&= \ffrac{n!} {(n-i)!(i-1)!} f(x) \big(F(x)\big)^{i-1}\big(1-F(x)\big)^{n-i}
\end{align}$$

***

## Joint Probability Distribution of Functions of $r.v.$

Let $X_1$ and $X_2$ be jointly continuous $r.v.$ with jointly pdf $f(x_1,x_2)$. We need to obtain the joint distribution of two new $r.v.$s $Y_1$ and $Y_2$ that arise as functions of $X_1$ and $X_2$, with $Y_1 = g_1(X_1, X_2)$ and $Y_2 = g_2(X_1, X_2)$.

$Assumptions$

1. The equations $y_1 = g_1(x_1, x_2)$ and $y_2 = g_2(x_1, x_2)$ can be *uniquely* solved for $x_1$ and $x_2$ in terms of $y_1$ and $y_2$ with solutions given by $x_1 = h_1(y_1,y_2)$ and $x_2 = h_2(y_1,y_2)$.$\\[0.7em]$
2. The functions $g_1$ and $g_2$ have *continuous partial derivatives* at *all points* $(x_1, x_2)$ and are such that the following determinant$\\[0.6em]$
$$J(x_1, x_2) = \begin{vmatrix}
\ffrac{\partial g_1}{\partial x_1} & \ffrac{\partial g_1}{\partial x_2} \\
\ffrac{\partial g_2}{\partial x_1} & \ffrac{\partial g_2}{\partial x_2}
\end{vmatrix} \equiv \ffrac{\partial g_1}{\partial x_1} \cdot \ffrac{\partial g_2}{\partial x_2} \; \: - \; \: \ffrac{\partial g_1}{\partial x_2} \cdot \ffrac{\partial g_2}{\partial x_1} \neq 0\\[0.5em]$$
at all points $(x_1, x_2)$.

Under these, $Y_1$ and $Y_2$ are jointly continuous with their joint density function given by

$$f_{Y_1,Y_2}(y_1,y_2) = g(y_1,y_2) = f_{X_1,X_2}(x_1, x_2) \big| J(x_1, x_2) \big|^{-1}$$

where $x_1 = h_1(y_1,y_2)$ and $x_2 = h_2(y_1,y_2)$. This formula can be obtained by differiate the following equationon both sides with respect to $y_1$ and $y_2$.

$$P\CB{Y_1 \leq y_1,Y_2 \leq y_2} = \iint\limits_{\d{\begin{array}{c}
(x_1,x_2): \\
g_1(x_1,x_2) \leq y_1\\
g_2(x_1,x_2) \leq y_2
\end{array}}} f_{X_1,X_2}(x_1, x_2) \;\dd{x_1}\;\dd{x_2}$$

**e.g.**

If $X$ and $Y$ are independent **gamma** $r.v.$s with parameters $(\alpha, \lambda)$ and $(\beta, \lambda)$, respectively. Find the joint density of $U = X + Y$ and $V = X/(X+Y)$.

> From their independency, we have their joint density function first
>
> $$\begin{align}
f_{X,Y}(x,y) &= f_X(x) \cdot f_Y(y) \\
&= \ffrac{\lambda e^{-\lambda x} (\lambda x)^{\alpha - 1}} {\Gamma(\alpha)} \cdot \ffrac{\lambda e^{-\lambda x} (\lambda x)^{\beta - 1}} {\Gamma(\beta)} \\
&= \ffrac{\lambda^{\alpha + \beta} } {\Gamma(\alpha)\Gamma(\beta)} e^{-\lambda(x+y)} x^{\alpha-1} y^{\beta -1}
\end{align}$$
>
>Given $g_1(x,y) = x+y, g_2(x,y) = x/(x+y)$, we have $\ffrac{\partial g_1} {\partial x} = \ffrac{\partial g_1} {y} = 1$, $\ffrac{\partial g_2}{\partial x} = \ffrac{y} {(x+y)^2}$, and $\ffrac{\partial g_2} {\partial y}=- \ffrac{x} {(x+y)^2}$, also the solutions: $x = u\upsilon$ and $y=u(1-\upsilon)$ so that:
>
>$$J(x,y) = \begin{vmatrix}
1 & 1\\[0.6em]
\ffrac{y}{\left( x+y \right )^2} & \ffrac{-x}{\left( x+y \right )^2}
\end{vmatrix} = - \ffrac{1} {x+y}$$
>
>$$
\begin{align}
f_{U,V}(u,\upsilon) &= f_{X,Y}(x,y) \cdot (x+y) \\[0.6em]
&= f_{X,Y}(u\upsilon, u(1-\upsilon)) \cdot u \\[0.6em]
&= \ffrac{\lambda^{\alpha + \beta} } {\Gamma(\alpha)\Gamma(\beta)} e^{-\lambda(u\upsilon+u(1-\upsilon))} (u\upsilon)^{\alpha-1} (u(1-\upsilon))^{\beta -1} \cdot u \\[0.6em]
&= \ffrac{\lambda^{\alpha + \beta-1}\cdot \lambda} {\Gamma(\alpha)\Gamma(\beta)} \cdot e^{-\lambda u} \cdot u^{1 + \alpha-1 + \beta -1} \cdot \ffrac{\Gamma(\alpha + \beta)} {\Gamma(\alpha + \beta)} \cdot \upsilon^{\alpha-1} \cdot (1-\upsilon)^{\beta -1} \\[0.6em]
&= \ffrac{\lambda e^{-\lambda u} (\lambda u)^{\alpha + \beta -1}} {\Gamma(\alpha + \beta)} \cdot \ffrac{ \upsilon^{\alpha-1} (1-\upsilon)^{\beta -1} \Gamma(\alpha + \beta)} {\Gamma(\alpha)\Gamma(\beta)}
\end{align}$$

$Remark$

Later we will know that $X+Y$ is also a **gamma** $r.v.$ with parameter $(\alpha + \beta , \lambda)$, thus with a pdf: $\d{f_{U}(u)  = \ffrac{\lambda e^{-\lambda u} (\lambda u)^{\alpha + \beta -1}} {\Gamma(\alpha + \beta)}}$.

Also, since $X+Y$ and $X/(X+Y)$ are independent, we can also see that: $\d{f_V{(\upsilon)} = \ffrac{\upsilon^ {\alpha-1} (1-\upsilon)^{\beta -1} \Gamma(\alpha + \beta)} {\Gamma(\alpha)\Gamma(\beta)}}$, which is called the ***beta density*** with parameters $(\alpha, \beta)$, with $0<\upsilon<1$.

$\QQQ$ The last paragraph.

***

And the same method can be applied to more than $2$ $r.v.$s. When the joint density function of the $n$ variables $X_1, X_2, \dots, X_n$ is given and we wnat to compute the joint density function of $Y_1, Y_2, \dots, Y_n$, where

$$Y_1 = g_1(X_1, X_2, \dots, X_n), Y_2 = g_2(X_1, X_2, \dots, X_n), \dots, Y_n = g_n(X_1, X_2, \dots, X_n)$$

Same assumptions required, like the continuous partial derivable and that the Jacobian determinant $J(x_1,x_2,\dots, x_n) \neq 0$ for all points $(x_1,x_2,\dots, x_n)$.

$$J(x_1,x_2,\dots, x_n) = \begin{vmatrix}
\ffrac{\partial g_1} {\partial x_1} & \ffrac{\partial g_1} {\partial x_2} & \cdots & \ffrac{\partial g_1} {\partial x_n} \\
\ffrac{\partial g_2} {\partial x_1} & \ffrac{\partial g_2} {\partial x_2} & \cdots & \ffrac{\partial g_2} {\partial x_n} \\
\vdots & \vdots & \ddots & \vdots \\
\ffrac{\partial g_n} {\partial x_1} & \ffrac{\partial g_n} {\partial x_2} & \cdots & \ffrac{\partial g_n} {\partial x_n}
\end{vmatrix}$$

and the equation set has a unique solution, $x_i = h_i(y_1,y_2,\dots,y_n)$, for $y_1 = g_1 (x_1,x_2,\dots,x_n), y_2 = g_2 (x_1,x_2,\dots,x_n), \dots, y_n = g_n (x_1,x_2,\dots,x_n)$. Under these, the joint dnsity function of the $r.v.$s $Y_i$  is given by

$$f_{Y_1,Y_2,\dots, Y_n}(y_1,y_2,\dots,y_n) = f_{X_1,X_2,\dots,X_n}(x_1,x_2,\dots, x_n)\big|J(x_1,x_2,\dots, x_n)\big|^{-1}$$

where $x_i = h_i(y_1,y_2,\dots,y_n)$.

***

# Moment Generating Functions

The ***moment generating function*** $\phi(t)$ of the $r.v.$ $X$ is defined for all values $t$ by

$$\phi(t) = \EE{e^{tX}} = \begin{cases}
\d{\sum_x e^{tx} \cdot p(x)}, & \text{if } X \text{is discrete} \\[0.5em]
\d{\int_{-\infty}^{\infty} e^{tx} \cdot f(x) \;\dd{x}}, & \text{if } X \text{is continuous}
\end{cases}$$

We can use this function to obtain all the moments of $X$ by successively differentiating $\phi(t)$.

$$\begin{align}
\phi'(t) &= \ffrac{\dd{}} {\dd{t}} \EE{e^{tX}} \\
&= \EE{\ffrac{\dd{}} {\dd{t}} e^{tX}} \\
&= \EE{Xe^{tX}} \\[0.8em]
\Longrightarrow \phi'(0) &= \EE{X}
\end{align}$$

Similarly, $\phi''(t) = \EE{X^2 e^{tX}} \; \Longrightarrow \; \phi''(0) = \EE{X^2}$. So in general, the $n\texttt{th}$ derivative of $\phi(t)$ evaluated at $t=0$ equals $\EE{X^n}$, for $n \geq 1$.

**e.g.** The **Binomial** Distribution

>$$\begin{align}
\phi(t) &= \EE{e^{tX}} \\
&= \sum_{k=0}^{n} e^{tk} \cdot \left( \binom{n} {k} p^k (1-p)^{n-k} \right)\\
&= \sum_{k=0}^{n} \binom{n} {k} (pe^t)^k (1-p)^{n-k} \\[0.5em]
&= (pe^t + 1 - p)^n
\end{align}$$

>Hence, $\EE{X} = \phi'(0) = n(pe^t + 1 - p)^{n-1} \cdot pe^t \big.\big|_{t=0} = np$ and $\EE{X^2} = \cdots = n(n-1) p^2 + np$. Thus we can also obtain the variance: $\Var{X} = \EE{X^2} - (\EE{X})^2 = \cdots = np(1-p)$.

**e.g.** The **Poisson** Distribution

>$$\begin{align}
\phi(t) &= \EE{e^{tX}} \\
&= \sum_{n=0}^{\infty} e^{tn} \cdot \ffrac{e^{-\lambda} \lambda^n} {n!}\\
&= e^{-\lambda} \sum_{n=0}^{\infty} \ffrac{\left( \lambda e^t \right)^n} {n!} \\
&= \QQQ e^{-\lambda} \cdot e^{\lambda e^t} = \exp\CB{\lambda \left(e^t - 1 \right)}
\end{align}$$

>Differentiation yields: $\phi'(t) = \lambda e^t \exp\CB{\lambda \left(e^t - 1 \right)}$ and $\phi''(t) = \left( \lambda e^t \right)^2 \exp\CB{\lambda \left(e^t - 1 \right)} + \lambda e^t \exp\CB{\lambda \left(e^t - 1 \right)}$

> and so $\EE{X} = \lambda$, $\EE{X^2} = \lambda^2 + \lambda$. And $\Var{X} = \lambda$

**e.g.** The **Exponential** Distribution

>$$\begin{align}
\phi(t) &= \EE{e^{tX}} \\
&= \int_{0}^{\infty} e^{tx} \cdot \lambda e^{-\lambda x} \;\dd{x} \\
&= \lambda \int_{0}^{\infty} e^{-\left(\lambda - t\right)x} \;\dd{x} \\
&\stackrel{t < \lambda} {=} \ffrac{\lambda} {\lambda - t}
\end{align}$$

>Differentiation of $\phi(t)$ yields $\phi'(t) = \ffrac{\lambda} {\left(\lambda - t\right)^2}, \phi''(t) = \ffrac{2\lambda} {\left(\lambda - t\right)^3}$. Thus, $\EE{X} = \phi'(0) = \ffrac{1} {\lambda}$, $\EE{X^2} = \phi''(0) = \ffrac{2} {\lambda^2}$ and the variance of $X$ is given by $\Var{X} = \EE{X^2} - \left(\EE{X}\right) ^2 = \ffrac{1} {\lambda^2}$

$Remark$

Only when $t < \lambda$ can we calculate the integral.

**e.g.** The **Normal** Distribution

>$$\begin{align}
\EE{e^{tZ}} &= \int_{-\infty}^{\infty} e^{tz} \cdot \ffrac{1} {\sqrt{2\pi}} \exp\CB{-\ffrac{z^2} {2}} \;\dd{z} \\
&= \ffrac{1} {\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp\CB{-\ffrac{z^2 - 2tz} {2}} \;\dd{z} \\
&= \ffrac{e^{t^2/2}} {\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp\CB{-\ffrac{(x-t)^2} {2}} \;\dd{z} = \exp\CB{ \ffrac{t^2}{2}}
\end{align}$$

$\space$

>Here $Z$ is a ***standard normal*** $r.v.$, so for any normal $r.v.$ $X = \sigma Z + \mu$ with parameters $\mu$ and $\sigma^2$, we have
>
>$$\phi(t) = \EE{e^{tX}} = \EE{e^{t\left(\sigma Z + \mu\right)}} = e^{t\mu} \EE{e^{t\sigma Z}} = \exp\CB{ \ffrac{\sigma^2 t^2} {2} + \mu t}$$
>
>And by differentiating we obtian $\phi'(t) = \left(\mu + t \sigma^2\right) \exp\CB{\ffrac{\sigma^2 t^2} {2} + \mu t}$, so $\EE{X} = \phi'(0) = \mu$, and $\phi''(t) = \left(\mu + t \sigma^2\right)^2 \exp\CB{\ffrac{\sigma^2 t^2} {2}} + \sigma^2 \exp\CB{\ffrac{\sigma^2 t^2} {2}}$, so $\EE{X^2} = \phi''(0) = \mu^2 + \sigma^2$, implying that $\Var{X} = \sigma^2$.

***

An important property of the **moment generating function** is that: For the sum of *independent* $r.v.$s, it's mgf is just the product of the individual mgfs. Suppose $X$ and $Y$ are independent and have mgf $\phi_X(t)$ and $\phi_Y(t)$, respectively.

$$\begin{align}
\phi_{X+Y}(t) &= \EE{e^{t\left(X+Y\right)}} \\
&= \EE{e^{tX} \cdot e^{tY}} \\
&\stackrel{\texttt{independency}} {=} \EE{e^{tX}}\cdot \EE{e^{tY}} = \phi_X(t)\phi_Y(t)
\end{align}$$

Another important property is that the mgf *uniquely* determines the distribution. It's a one-to-one correspondence.

$Remark$

More about the **Poisson** Distribution, the ***Poisson paradigm***, that the number of success in $n$ trials that are either independent or at most weakly dependent is, when the trial success probabilities are all small, approximately a **Poisson** $r.v.$.

$Remark$

***Laplace transform***, for nonnegative $r.v.$ $X$, is defined as for $t \geq 0$, $g(t) = \phi(-t) = \EE{e^{-tX}}$. This would limit the value between $0$ and $1$.

We can also define the ***joint moment generating function*** of more than just two $r.v.$s. For any $n$ $r.v.$s $X_1, X_2, \dots, X_n$, and for all real values of $t_1, t_2, \dots, t_n$ we define:

$$\phi(t_1, t_2, \dots, t_n) = \EE{\exp\CB{t_1X_1 + t_2X_2 + \cdots + t_nX_n}}$$

and it can be shown that $\phi(t_1, t_2, \dots, t_n)$ uniquely determines the joint distribution of $X_1, X_2, \dots, X_n$.

**e.g.** The ***Multivariate Normal Distribution***

Let $Z_1,\dots,Z_n$ be a set of $n$ independent standart normal random variables. If, for some constants $a_{ij}$ and $\mu_i$, $1 \leq i \leq m$, $1 \leq j \leq n$,

$$
\begin{array}{rcl}
X_1\!\!\!\! &=&\!\!\!\!a_{11}Z_1 + \cdots + a_{1n}Z_n + \mu_1 \\
X_2 \!\!\!\!&=&\!\!\!\!a_{21}Z_1 + \cdots + a_{2n}Z_n + \mu_2 \\
& \vdots & \\
X_i \!\!\!\!&=&\!\!\!\!a_{i1}Z_1 + \cdots + a_{in}Z_n + \mu_i \\
& \vdots & \\
X_m \!\!\!\!&=&\!\!\!\!a_{m1}Z_1 + \cdots + a_{mn}Z_n + \mu_m
\end{array}$$

Then the $r.v.$s $X_1, X_2, \dots, X_n$. are said to have a **Multivariate Normal Distribution**.

> Easy to see that $\EE{X_i} = \mu_i$ and $\Var{X_i} = \sum\limits_{j=1}^{n}a_{ij}^2$. Then $\EE{\sum\limits_{i=1} ^{m} t_iX_i} = \sum\limits_{i=1}^{m} t_i\mu_i$ and

>$$\Var{\sum\limits_{i=1} ^{m} t_iX_i} = \Cov{\sum\limits_{i=1} ^{m} t_iX_i,\sum\limits_{j=1} ^{m} t_jX_j} = \sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m} t_it_j\Cov{X_i,X_j}$$

>$$\phi\left(t_1,\dots,t_m\right) = \exp\CB{\sum\limits_{i=1}^{m} t_i\mu_i + \ffrac{1} {2} \sum\limits_{i=1}^{m} \sum\limits_{j=1}^{m} t_it_j\Cov{X_i,X_j}}$$

***

##  The Joint Distribution of the Sample Mean and Sample Variance from a Normal Population

$X_1,\dots,X_n$ are independent and identical distributed $r.v.$s, each with mean $\mu$ and variance $\sigma^2$. We now define the ***sample mean*** $\bar{X} =\ffrac{1} {n}\sum\limits_{i=1}^{n} X_i$ and ***sample variance***:

$$S^2 = \sum_{i=1}^{n}\ffrac{\left(X_i - \bar{X}\right)^2} {n-1}$$

With the fact that

$$\begin{align}
\sum_{i=1}^{n} \left(X_i - \bar{X}\right) &= \sum_{i=1}^{n} \left( X_i - \mu + \mu - \bar{X} \right)^2 \\
&= \left[\sum_{i=1}^{n} \left(X_i - \mu \right)^2\right] + n\left(\mu - \bar{X}\right)^2 + 2\left(\mu - \bar{X} \right)\sum_{i=1}^{n} \left(X_i - \mu \right) \\
&= \left[\sum_{i=1}^{n} \left(X_i - \mu \right)^2\right] + n\left(\mu - \bar{X}\right)^2 - 2n\left(\mu - \bar{X}\right)^2 = \left[\sum_{i=1}^{n} \left(X_i - \mu \right)^2\right] - n\left(\bar{X} - \mu\right)^2
\end{align}$$

we can calculate the expectation as

$$\begin{align}
\EE{S^2} &= \ffrac{1} {n-1} \left[\left(\sum_{i=1}^{n} \EE{(X_i - \mu)^2}\right)-n\EE{\left(\bar{X} - \mu\right)^2 }\right] \\
&= \ffrac{1} {n-1}\left(n\sigma^2 - n\Var{\bar{X}}\right) = \sigma^2
\end{align}$$

$Def$ ***Chi-Squared*** $r.v.$

If $Z_1,\dots,Z_n$ are *independent* **standard normal** $r.v.$s then the $r.v.$ $\sum Z_i^2$ is said to be a **chi-squared** $r.v.$ with $n$ ***degrees of freedom***.

We first compute its mgf, note that

$$\begin{align}
\EE{\exp\CB{tZ_i^2}} &= \int_{-\infty}^{\infty}\exp\CB{tx^2}\cdot\ffrac{1} {\sqrt{2\pi}} \exp\CB{-\ffrac{x^2} {2}} \;\dd{x} \\
&= \ffrac{1} {\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp\underset{\;\;\begin{array}{c}
\uparrow \\
\sigma^2 = \left(1-2t\right)^{-1}
\end{array}}{\CB{-\ffrac{x^2} {2\sigma^2}}} \;\dd{x} \\[0.8em]
&= \sigma = \left(1-2t\right)^{-1/2}
\end{align}$$

Hence,

$$\EE{\exp\CB{t\sum_{i=1}^{n} Z_i^2}} = \prod_{i=1}^{n} \EE{\exp\CB{tZ_i^2}} = \left(1-2t\right)^{-n/2}$$

$Remark$

Consider $Y$ be a **normal** $r.v.$ with mean $\mu$ and variance $\sigma^2/n$ that is independent of $X_1, \dots, X_n$, then the $r.v.$s $Y, X_1-\bar{X}, X_2-\bar{X},\dots,X_n-\bar{X}$ have a **multivariate normal** distribution. Since they are independent, $\Cov{Y,X_i - \bar{X}} = 0$ for $i = 1,\dots, n$. Also, $\EE{Y + X_1-\bar{X} + \cdots + X_n-\bar{X}} = \EE{Y} = \EE{\bar{X}}$.

Our conclusion is that for a **multivariate normal** distribution is *completely*, *uniquely* determined by its expected values and covariances, $\bar{X}$ is independent of the sequence of deviations $X_i - \bar{X}$, $i = 1,\dots, n$.

So that it's also independent of the **sample variance** $S^2 \equiv \ffrac{1} {n-1} \sum_{i=1}^{n}\left(X_i - \bar{X} \right)^2$ and now we're gonna determine the distribution of $S^2$.

$$\ffrac{n-1} {\sigma^2} S^2 = \left[ \sum_{i=1}^{n} \ffrac{\left(X_i - \mu \right)^2} {\sigma^2}\right] - \ffrac{n\left(\bar{X} - \mu\right)^2} {\sigma^2} \Rightarrow \ffrac{(n-1)S^2} {\sigma^2} + \left( \ffrac{\bar{X} - \mu} {\sigma / \sqrt{n}} \right)^2 = \sum_{i=1}^{n} \ffrac{\left(X_i - \mu \right)^2} {\sigma^2}$$

The key is to use the mgf. We've already seen that the mgf for the right side term, the **chi-squared** $r.v.$ with $n$ degree of freedom and the second term on the left, the square of a **standard normal** $r.v.$, the **chi-squared** $r.v.$ with $1$ degree of freedom. So that

$$\EE{\exp\CB{t\cdot \ffrac{(n-1)S^2} {\sigma^2}}}(1-2t)^{-1/2} = (1-2t)^{-n/2}$$

Thus, the mgf of $\ffrac{n-1} {\sigma^2} S^2$ is the same with that of a **chi-squared** $r.v.$ with $n-1$ degrees of freedom, where we can claim the proposition

$Proposition$

If $X_1,\dots,X_n$ are $i.i.d.$ **normal** $r.v.$s with mean $\mu$ and varianc $\sigma^2$, then the **sample mean** $\bar{X}$ and **sample variance** $S^2$ are independent. $\bar{X}$ is a **normal** $r.v.$ with mean $\mu$ and variance $\sigma^2/n$; $(n-1)S^2/\sigma^2$ is a **chi-squared** $r.v.$ with $n-1$ degrees of freedom.

# The Distribution of the Number of Events that Occur

Consider arbitrary events $A_1,\dots,A_n$, and let $X$ denote the number of these events that occur. What's the pmf of $X$? We first define:

$$S_k = \sum_{\d{i_1 < \cdots < i_k}} P\left( A_{\d{i_1}},\dots,A_{\d{i_k}} \right)$$

as the sum of the probabilities of all the $\d{\binom{n} {k}}$ intersections ($\cap$) of $k$ distinct events, and note that the inclusion-exclusion identity states that

$$P\CB{X>0} = P\left(\bigcup_{i=1}^{n}A_i\right) = S_1 - S_2 + S_3 - \cdots + (-1)^{n+1} S_n$$

Now, to help understand we fix $h$ of the $n$ events, say $A_{1},\dots,A_{h}$ and let $A=\bigcap\limits_{j=1}^{h} A_{j}$ be the event that all $h$ of these events occur. Also, let $B=\bigcap\limits_{j\notin\CB{1, 2, \dots, h}} A_{j}^{c}$ be the none of the other $n-h$ events occur. Consequently, $A\cap B = AB$ is the event that $A_{1}, \dots,A_{h}$ are the *only* events to occur. Then, since $A = AB \cup AB^c$, we have $P(AB) = P(A) - P(AB^c)$.

While $B^c = \bigcup\limits_{j \notin \CB{1, 2, \dots, h}} A_j$, so that $P(AB^c) = P\left( A\bigcup\limits _{j\notin\CB{1,\dots,h}} A_j\right) = P\left( \; \bigcup\limits _{j\notin\CB{1,\dots,h}} AA_j \right)$.

Then we apply the inclusion-exclusion identity again:

$$
\begin{align}
P(AB^c) &= \sum_{\d{j\notin \CB{1, 2, \dots, h}}} P(AA_j) - \sum_{\d{j_1 <j_2 \notin \CB{1,\dots,h}}} P( AA_{\d{j_1}}A_{\d{j_2}} ) \\[1em]
&\;\;\;\;+ \sum_{\d{j_1 < j_2 < j_3 \notin \CB{1, 2, \dots, h}}} P( AA_{\d{j_1}}A_{\d{j_2}} A_{\d{j_3}} ) - \cdots
\end{align}$$

Then followed by $P(A\cap B) = P(A) - P(AB^c) = P(A_1 \cap A_2 \cap \cdots \cap A_h) - P(AB^c)$, we can approach our final generalized answer,

$\;\;\;\;
\begin{align}
P\CB{X=k} &= \sum_{\d{i_1 <\dots<i_k}} \left[ P\left(A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}}\right) - \sum _{\d{j \notin \CB{i_1,\dots, i_k}}} P\left(A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \cap A_{\d{j}}\right)\right. \\[0.8em]
&\;\;\;\;+ \sum_{\d{j_1 < j_2 \notin \CB{i_1,\dots,i_k}}} P\left( A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \cap A_{\d{j_1}} \cap A_{\d{j_2}} \right) \\
&\;\;\;\;- \left.\sum_{\d{j_1 < j_2 < j_3 \notin \CB{i_1,\dots,i_k}}} P\left( A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \cap A_{\d{j_1}} \cap A_{\d{j_2}} \cap A_{\d{j_3}} \right)+\cdots \right]
\end{align}$

Kinda complex, how to simplify this expression?

First note that $S_k = \sum\limits_{\d{i_1 <\dots<i_k}} P\left( A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \right)$. Now consider

$$\sum_{\d{i_1 < \cdots <i_k}} \; \sum_{\d{j \notin \CB{i_1,\dots,i_k}}}P\left( A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \cap A_j \right)$$

Surely, there're repetition. These $k+1$ distinct events are choosed by two steps. We let them be $A_{\d{m_1}} \cap A_{\d{m_2}} \cap \cdots \cap A_{\d{m_{k+1}}}$, thus it's easier to find out the probability of every intersection actually appear $\d{\binom{k+1}{k}}$ times in this multiple summation. Hence:

$\;\;\;\;
\begin{align}
&\sum_{\d{i_1 < \cdots <i_k}} \; \sum_{\d{j \notin \CB{i_1,\dots,i_k}}}P\left( A_{\d{i_1}} \cap A_{\d{i_2}} \cap \cdots \cap A_{\d{i_k}} \cap A_j \right) \\
=& \binom{k+1}{k} \sum_{\d{m_1 < \cdots < m_{k+1}}} P\left(A_{\d{m_1}} \cap A_{\d{m_2}} \cap \cdots \cap A_{\d{m_{k+1}}}\right)\\
=& \binom{k+1}{k} S_{k+1} \\[1em]
&\texttt{So mf easier!}
\end{align}$

Similarly, we can say

$\;\;\;\;P\CB{X=k} = S_k - \d{\binom{k+1} {k}}S_{k+1} + \cdots + (-1)^{n-k}\d{\binom{n} {k}}S_n = \d{\sum_{j=k}^{n}} (-1)^{k+j} \d{\binom{j} {k}} S_j$

Using this we will now prove that $P\CB{X \geq k} = \d{\sum_{j=k}^{n} (-1)^{k+j} \binom{j-1}{k-1}S_j}$. We will use a backwards mathematical induction that starts with $k=n$. Now, when $k=n$ the preceding identity states that

$$P\CB{X=n} = Sn$$

First step finished! So we assume that $P\CB{X \geq k+1} = \d{\sum_{j=k+1}^{n} (-1)^{k+1+j} \binom{j-1} {k}S_j}$. And then

$$\begin{align}
P\CB{X \geq k} &= P\CB{X = k} + p\CB{X \geq k+1} \\[0.8em]
&= \left[\sum_{j=k}^{n} (-1)^{k+j} \binom{j} {k} S_j\right] + \left[ \sum_{j=k+1}^{n} (-1)^{k+1+j} \binom{j-1} {k}S_j \right]\\
&= S_k + \left[ \sum_{j=k+1}^{n} (-1)^{k+j} \left[\binom{j} {k} - \binom{j-1} {k} \right] S_j \right] \\
&= S_k + \sum_{j=k+1}^{n} (-1)^{k+j} \binom{j-1} {k-1}S_j = \sum_{j=k}^{n} (-1)^{k+j} \binom{j-1} {k-1}S_j
\end{align}$$

All done!

# Limit Theorems

First we prove the **Markov's inequality**.

$Proposition$ ***Markov's inequality***

If $X$ is a $r.v.$ that takes only *nonnegative* values, then for any value $\alpha > 0$,

$$P\CB{X \geq a} \leq \ffrac{\EE{X}} {a}$$

$Proof$

This proof is for the case where $X$ is continuous with density $f$.

$$\begin{align}
\EE{X} &= \int_{0}^{\infty} x\cdot f(x) \;\dd{x}\\
&= \int_{0}^{a} x\cdot f(x) \;\dd{x} + \int_{a}^{\infty} x\cdot f(x) \;\dd{x}\\
&\geq \int_{a}^{\infty} x\cdot f(x) \;\dd{x} \geq \int_{a}^{\infty} a\cdot f(x)\;\dd{x} \\
&= a \cdot P\CB{X \geq a}
\end{align}$$

$Remark$

From the process of proving it, we can easily find that this holds for discrete $r.v.$, and a slightly different result can be made if the $r.v.$ only takes nonpositive values.

$Proposition$ ***Chebyshev's Inequality***

If $X$ is a $r.v.$ with mean $\mu$ and variance $\sigma^2$, then, for any value $k > 0$,

$$P \CB{\left|X - \mu \right| \geq k} \leq \ffrac{\sigma^2} {k^2}$$

$Proof$

Since $\left(X - \mu\right)^2$ is a nonnegative $r.v.$, we can apply the previous proposition, the **Markov's inequality** (with $a = k^2$) to obtain:

$$P\CB{\left(X - \mu\right)^2 \geq k^2} = P\CB{\left|X - \mu\right| \geq k} \leq \ffrac{\EE{\left(X-\mu\right)^2}} {k^2}$$

$Remark$

These two propositions are important for that they produce the methods to find a bound for a certain probability given limited infomations like the mean or the mean and the variane.

**e.g.**

The number of items produced in a factory during a week is a $r.v.$ with *mean* $500$.

What's the probability that this week's production will be at least $1000$?

> Let $X$ be the number of items that will be produced in a week.
>
>$P\CB{X \geq 1000} \leq\ffrac{\EE{X}} {1000} = \ffrac{500} {1000} = 0.5$

If the variance is also given with value $100$, then what's the probability that this week's production will be between $400$ and $600$?

>$P\CB{|X-500| \geq 100} \leq \ffrac{\sigma^2} {100^2} = \ffrac{1} {100}$, hence, $P\CB{400 < X < 600} = 1 - \ffrac{1} {100} = \ffrac{99} {100}$.

***

$Theorem$ ***Strong Law of Large Numbers***

Let $X_1,X_2,\dots$ be a sequence of *independent* $r.v.$ having a common distribution, and let $\EE{X_i} = \mu$. Then **with probability** $1$ (later will be shortened as $wp1$).

$$\lim_{n \to \infty}\ffrac{X_1 + X_2 + \cdots + X_n} {n} = \mu$$

$Theorem$ ***Central Limit Theorem***

Let $X_1,X_2,\dots$ be a sequence of $i.i.d.$ $r.v.$, each with mean $\mu$ and variance $\sigma^2$. Then

$$\lim_{n \to \infty} P\CB{\ffrac{X_1 + X_2 + \cdots + X_n - n \mu} {\sigma \sqrt{n}} \leq a} = \ffrac{1} {\sqrt{2 \pi}} \int_{-\infty}^{a} e^{-x^2/2} \;\dd{x}$$

$Remark$

This holds for *any* distribution of the $X_i$s! Herein lies its power! Say an **binomially** distributed $r.v.$ with parameters $n$ and $p$, $X$. Then $X$ can be seen as the sum of $n$ independent **Bernoulli** $r.v.$s, each with parameter $p$. Hence the distribution

$$\ffrac{X- \EE{X}} {\sqrt{\Var{X}}} = \ffrac{X - np} {\sqrt{np(1-p)}}$$

approaches the **standard normal** distribution as $n$ approaches $\infty$. And this normal approximation will be generally great for values of $n$ satisfying $np(1-p) \geq 10$. See the next example~

**e.g.** From **Binomial** to **Normal**

$X$ is the number of times that a fair coin, flipped $40$ times, lands *heads*. What's the probability that $X = 20$? How's the normal approximation comparing to the exact solution?

> How to approximate a discrete $r.v.$ using a continuous $r.v.$? Here's the *trick*.
>
>$$
\begin{align}
P\CB{X=20} &= P\CB{19.5 < X < 20.5} \\
&= P\CB{\ffrac{19.5-20} {\sqrt{10}} < \ffrac{X - 20} {\sqrt{10}} < \ffrac{20.5 - 20} {\sqrt{10}}} \\[0.6em]
&= P\CB{-0.16 < Z < 0.16} \\[0.7em]
& \mathbf{\approx} \Phi(0.16) - \Phi(-0.16)
\end{align}$$

$Remark$

Here, $\Phi(z)$ is the probability that the **standard normal** is less than $z$ and is given by

$$\Phi(z) = \ffrac{1} {\sqrt{2\pi}} \int_{-\infty}^{z} e^{-x^2/2}\;\dd{x} $$

>Thus by the symmetry of the **standard normal** distribution: $\Phi(-0.16) = P\CB{\N{0,1} > 0.16} = 1 - \Phi(0.16)$, where $\N{0,1}$ is a **standard normal** $r.v.$. Hence the answer is

>$$P\CB{X = 20} \approx 2\Phi(0.16) - 1 = 0.1272$$

>Then, the exact result is $P\CB{X=20} = \d{\binom{40} {20}\left(\frac{1} {2}\right)^{40}} = 0.1268$

The following will be a heuristic proof of the ***CLT***, central limit theorem. We first suppose that $X_i$ have mean $0$ and variance $1$, and their common mgf is $\EE{e^{tX}}$. Then the mgf of $\ffrac{\sum X_i} {\sqrt{n}}$ is:

$$\begin{align}
\EE{\exp\CB{t\cdot\left( \ffrac{X_1+\cdots+X_n} {\sqrt{n}} \right)}} &= \EE{e^{tX_1/\sqrt{n}} \cdots e^{tX_n/ \sqrt{n}}} \\
&= \left( \EE{e^{tX/\sqrt{n}}} \right)^n
\end{align}$$

From the **Taylor series expnsion** of $e^y$, for large $n$, we have

$$e^{tX/\sqrt{x}} = 1 + \ffrac{tX} {\sqrt{n}} + \ffrac{t^2X^2} {2n} $$

and since $\EE{X} = 0$ and $\EE{X^2} = 1$, we have

$$
\begin{align}
\EE{\exp\CB{t\cdot\left( \ffrac{X_1+\cdots+X_n} {\sqrt{n}} \right)}} &= \left( \EE{e^{tX/\sqrt{n}}} \right)^n \\
&= \left(1 + \ffrac{t^2} {2n}\right)^n \\
&\to e^{t^2/2} \;\;\;\;\text{as } n \to \infty
\end{align}$$

This is the mgf of a **standard normal** $r.v.$ with mean $0$ and variance $1$. So we can already say that the $r.v.$ $\ffrac{X_1 + \cdots + X_n} {\sqrt{n}}$ converges to the **standard normal** distribution function $\Phi$.

And then when $X_i$ have mean $\mu$ and variance $\sigma^2$, we convert them to $\ffrac{X_i - \mu} {\sigma}$ with mean $0$ and $1$. Thus the preceding shows that:

$$P\CB{\ffrac{X_1 - \mu + \cdots +X_n - \mu} {\sigma\sqrt{n}} \leq a} \to \Phi(a)$$

which proves the **CLT**.

# Stochastic Processes

A ***stochastic process*** $\CB{X(t), t \in T}$ is a *collection* of $r.v.$. The index $t$ is often interpreted as ***time*** and as a result, we refer to $X(t)$ as the ***state*** of the process at time $t$. And it must be *infinite* elements in it.

The set $T$ is the ***index set*** of the process. When $T$ is countable set, the stochastic process is said to be a ***discrete-time process***. And if $T$ is an interval of the real line, the stochastic process is said to be a ***continuous-time process***.

The ***state space*** of a stochastic process is the set of *ALL possible values* that the $r.v.$ $X(t)$ can assume.

- State Space and Time Parameter are both Discrete (Random Walk)
- State Space is continuous and Time Parameter is Discrete (Common)
- State Space is Discrete and Time Parameter is continuous (Poisson Process)
- State Space and Time Parameter are both Continuous (Brownian Motion Process)

**e.g.**

Consider a partical that moves along a set of $m+1$ nodes, labeled from $0$ to $m$, that are arranged arround a circle. At each step the particle is equally likely to move one position in either the clockwise or counterclockwise direction. That is, if $X_n$ is the position of the particle after its $n\texttt{th}$ step, then

$$P\CB{X_{n+1} = i+1 \mid X_n = i} = P \CB{X_{n+1} = i-1 \mid X_n = i} = \ffrac{1} {2}$$

where we let $i+1=0$ when $i=m$ and $i-1=m$ when $i=0$. Now the particle starts at $0$ and continues to move around according to the preceding rules until all the nodes have been visited. What is the probability that node $i$ is the last one visited?

> Consider the first time that the particle is at one of the two neighbors of node $i$ not $0$ as assumed, say, node $i − 1$. Since either node $i$ or $i+1$ has yet been visited, it follows that $i$ will be the last node visited $iff$ $i+1$ is visited before $i$. So that the particle will progress $m − 1$ steps in a specified direction before progressing one step in the other direction.

>That is, it is equal to the probability that a gambler who starts with one unit, and wins one when a fair coin turns up heads and loses one when it turns up tails, will have his fortune go up by $m − 1$ before he goes broke. Hence, because the preceding $\QQQ$ implies that the probability that node $i$ is the last node visited is the same for all $i$, and because these probabilities must sum to $1$, we obtain

>$$P \CB{ i \text{ is the last node visited}} = 1/ m$$

$Remark$

Consider that gambler again. He going down $n$ before being up $1$ is with probability $\QQQ$ $1/(n+1)$; or equivalently,

$$P\CB{\text{gambler is up }1\text{ before being down }n} = \ffrac{n} {n+1}$$

Then:

$$
\begin{align}
& P\CB{\text{gambler is up }2\text{ before being down }n} \\
=& P\CB{\text{up }2\text{ before down }n \mid \text{up }1\text{ before down }n} \cdot \ffrac{n} {n+1} \\
=& P\CB{\text{up }2\text{ before down }n+1}\cdot \ffrac{n} {n+1} \\
=& \ffrac{n+1} {n+2}\ffrac{n} {n+1} = \ffrac{n} {n+2}
\end{align}$$

Repeating this argument yields that

$$P\CB{\text{gambler is up }k\text{ before being down }n} = \ffrac{n} {n+k}$$

Probability and Statistics/Applied Random Process/Chap_02.ipynb