# Convex Nonparametric Least Squares (`CNLS`): An alternative formulation

## CNLS

Hildreth (1954) was the first to consider nonparametric regression subject to monotonicity and concavity constraints in the case of a single input variable $x$ (see also Hanson and Pledger 1976). Kuosmanen (2008) extended Hidreth’s approach to the multivariate setting with a vector-valued $\bf{x}$, and coined the term convex nonparametric least squares (`CNLS`) for this method. CNLS builds upon the assumption that the true but unknown production function f belongs to the set of continuous, monotonic increasing and globally concave functions, $F_2$, imposing exactly the same production axioms as standard DEA.

The `CNLS` estimator of function $f$ is obtained as the optimal solution to the infinite dimensional least squares problem

\begin{align}
& \underset{f} min \sum_{i=1}^n(y_i-f(\bf{x}_i))^2 \\
& \text{s.t.} \\
&  f \in F_2
\end{align}

## CNLS estimator with the observed data points ($\mathbf{x}_i$, $y_i$)

\begin{align}
& \underset{\alpha, \beta, \varepsilon} {min} \sum_{i=1}^n\varepsilon_i^2 \\
& \text{s.t.} \\
&  y_i = \alpha_i + \beta_i^{'}X_i + \varepsilon_i \quad \forall i \\
&  \alpha_i + \beta_i^{'}X_i \le \alpha_j + \beta_j^{'}X_i  \quad  \forall i, j\\
&  \beta_i \ge 0 \quad  \forall i \\
\end{align}

where $\alpha_i$ and $\beta_i$ define the intercept and slope parameters of tangent hyperplanes that characterize the estimated piece-wise linear frontier. $\varepsilon_i$ denotes the CNLS residuals.

## Alternative formulation

To estimate the above `CNLS` model on Python, we have to resort to an alternative `CNLS` formulation.

\begin{align}
& \underset{\hat{y}, \beta} {min}  \frac{1}{2} \mid\mid y_i-\hat{y}_i\mid\mid_2^2    \\
& \text{s.t.} \\
&  \hat{y}_i - \hat{y}_j \ge \beta_i^{'}(X_i -X_j)  \quad  \forall i, j\\
&  \beta_i \ge 0 \quad  \forall i \\
\end{align}

> Proof 1: The constraints in the two formulations are equivalent.

   + Replace the CNLS residuals by: $\varepsilon_i = y_i − \alpha_i - X_i \beta_i^{'} = y_i - \hat{y}_i$
   + Left-hand side in second constraint: $\hat{y}_i$
   + Right-hand side in second constraint: $\hat{y}_j - X_j \beta_j^{'} + X_i \beta_j^{'} = \hat{y}_j + \beta_j^{'} (X_i-X_j)$

    Therefore, the second constraint is redefined as:

\begin{align}
&  \alpha_i + \beta_iX_i \le \alpha_j + \beta_jX_i  \quad  \forall i, j\\
&  \hat{y}_i \le  \hat{y}_j + \beta_j^{'} (X_i-X_j) \\
&  \hat{y}_j - \hat{y}_i \ge \beta_j^{'} (X_j-X_i)  \\
&  \hat{y}_i - \hat{y}_j \ge \beta_i^{'} (X_i-X_j)  \quad \blacksquare \\
\end{align}

> Proof 2: The objective function is a standard quadratic problem (QP).

The standard `QP` formulation:
\begin{align}
\text{min} f(x) = \frac{1}{2}x^TAx + q^Tx
\end{align}

The objective function in the alternative `CNLS` formulation:
\begin{align}
& \text{min} \frac{1}{2} \mid\mid y-\hat{y}\mid\mid_2^2  \\
& = \frac{1}{2} (\hat{y} - y)^T(\hat{y} - y) \\
& = \frac{1}{2}(\hat{y}^T\hat{y} -y^T\hat{y} -\hat{y}^Ty+ y^Ty) \\
& = \frac{1}{2}\hat{y}^T\hat{y} - y^T\hat{y} + \frac{1}{2}y^Ty \\
\end{align}

since $\frac{1}{2}y^Ty$ is a contant term, it is sufficient to solve the QP-form objective function. Note that A is a identity matrix, q= -y. $\blacksquare$

Now, we can estimate the alternative `CNLS` model based on convex quadratic problem solver in Python, eg., CVXOPT.

### References

[1] Johnson, A. L. and Kuosmanen, T. (2015) An Introduction to CNLS and StoNED Methods for Efficiency Analysis: Economic Insights and Computational Aspects, in Ray, S. C., Kumbhakar, S. C., and Dua, P. (eds) Benchmarking for Performance Evaluation: A Production Frontier Approach. Springer, pp. 117–186.

[2] Kuosmanen, T., Johnson, A. and Saastamoinen, A. (2015) Stochastic Nonparametric Approach to Efficiency Analysis: A unified Framework, in Zhu, J. (ed.) Data Envelopment Analysis. Springer, pp. 191–244.


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
from scipy import stats
```

## Calculus in SymPy ##

Working with densities involves calculus which can sometimes be time-consuming. This course gives you two ways of reducing the amount of calculus involved.

- Probabilistic methods can help reduce algebra and calculus. You've seen this with algebra in the discrete case. You'll see it with calculus as we learn more about densities.
- Python has a symbolic math module called `SymPy` that does algebra, calculus, and much other symbolic math. In this section we will show you how to do calculus using `SymPy`.

We will demonstrate the methods in the context of an example. Suppose $X$ has density given by

$$
f(x) =
\begin{cases}
105x^2(1-x)^4 ~~~ \text{if } 0 \le x \le 1 \\
0 ~~~~~~~~~ \text{otherwise}
\end{cases}
$$

As you can see from its graph below, $f$ could be used to model the distribution of a random proportion that you think is likely to be somewhere between 0.2 and 0.4.

```python
# NO CODE
x_vals = np.arange(0, 1.01, 0.01)
f = 105 * x_vals**2 * (1 - x_vals)**4
plt.plot(x_vals, f, color='darkblue', lw=2)
plt.xlabel('$x$')
plt.ylabel('$f(x)$', rotation=0);
```

The density $f$ is a polynomial on the unit interval, and in principle the algebra and calculus involved in integrating it are straightforward. But they are tedious. So let's get `SymPy` to do the work.

First, we will import all the functions in `SymPy` and set up some printing methods that make the output look nicer than the retro typewritten pgf output you saw in a previous section. In future sections of this text, you can assume that this importing and initialization will have been done at the start.

```python
from sympy import *
init_printing()
```

Next, we have to create tell Python that an object is symbolic. In our example, the variable $x$ is the natural candidate to be a symbol. You can use `Symbol` for this, by using the argument `'x'`. We have assinged the symbol to the name `x`.

```python
x = Symbol('x')
```

Now we will assign the name `density` to the expression that defines $f$. The expression looks just like a numerical calculation, but the output is algebraic!

```python
density = 105 * x**2 * (1-x)**4
density
```

That's the expression for $f(x)$ defined by the equation at the start of the section. Notice that what we naturally think of as $1 - x$ is expressed as $-x + 1$. That's because `SymPy` is writing the polynomial leading with the term of highest degree.

Let's not simply accept that this function is a density. Let's check that it is a density by integrating it from 0 to 1. To display this, we use the method `Integral` that takes the name of a function and a *tuple* (a sequence in parentheses) consisting of the variable of integration and the lower and upper limits of integration. We have assigned this integral to the name `total_area`.

```python
total_area = Integral(density, (x, 0, 1))
total_area
```

The output of displays the integral, which is nice, but what we really want is its numerical value. In `SymPy`, this is achieved by abruptly instructing the method to `doit()`.

```python
total_area.doit()
```

This confirms that the function $f$ is a density.

We can use `Integral` to find the chance that $X$ is in any interval. Here is $P(0.2 < X < 0.4)$.

```python
prob_02_04 = Integral(density, (x, 0.2, 0.4)).doit()
prob_02_04
```

For $x$ in the unit interval, the cdf of $X$ is

$$
F(x) ~ = ~ P(X \le x) ~ = ~ \int_0^x f(s)ds ~ = ~ I(s)~ \Big{\rvert}_0^x ~ = ~ I(x) - I(0)
$$

where $I$ is the indefinite integral of $f$.

To get the indefinite integral, simply ask `SymPy` to integrate the density; there are no limits of integration.

```python
indefinite = Integral(density).doit()
indefinite
```

Now $F(x) = I(x) - I(0)$. You can see at a glance that $I(0) = 0$ but here is how `SymPy` would figure that out.

To evaluate $I(0)$, `SymPy` must substitute $x$ with 0 in the expression for $I$. This is achieved by the method `subs` that takes the variable as its first argument and the specified value as the second.

```python
I_0 = indefinite.subs(x, 0)
I_0
```

```python
cdf = indefinite - I_0
cdf
```

To find the value of the cdf at a specified point, say 0.4, we have to substitute $x$ with 0.4 in the formula for the cdf.

```python
cdf_at_04 = cdf.subs(x, 0.4)
cdf_at_04
```

Thus $P(X \le 0.4)$ is roughly 58%. Earlier we calulated $P(0.2 < X < 0.4) = 43.2\%$, which we can confirm by using the cdf:

```python
cdf_at_02 = cdf.subs(x, 0.2)
cdf_at_04 - cdf_at_02
```

The expectation $E(X)$ is a definite integral from 0 to 1:

```python
expectation = Integral(x*density, (x, 0, 1)).doit()
expectation
```

Notice how simple the answer is. Later in the course, you will see why.

Here is $E(X^2)$, which turns out to be another simple fraction. Clearly, the density $f$ has interesting properties. We will study them later. For now, let's just get the numerical answers.

```python
expected_square = Integral((x**2)*density, (x, 0, 1)).doit()
expected_square
```

Now you can find $SD(X)$.

```python
sd = (expected_square - expectation**2)**0.5
sd
```

### SymPy and the Exponential Density ###
One of the primary distributions in probability theory, the exponential distribution has a positive parameter $\lambda$ known as the "rate", and density given by

$$
f(t) ~ = \lambda e^{-\lambda t}, ~~~ t \ge 0
$$

The density is 0 on the negative numbers. Here is its graph when $\lambda = 3$.

```python
# NO CODE

lam = 3
tt = np.arange(0, 2.01, 0.01)
f = lam * np.exp(-1*lam*tt)
plt.plot(tt, f, color='darkblue', lw=2)
plt.xlabel('$t$')
plt.ylabel('$f(t)$', rotation=0)
plt.title('Exponential Density, Rate $\lambda = 3$');
```

To check that $f$ is a density, we have to confirm that its integral is 1. We will start by constructing two symbols, `t` and `lamda`. Notice the incorrectly spelled `lamda` instead of `lambda`. That is because `lambda` has another meaning in Python, as some of you might know.

Note the use of `positive=True` to specify that the symbol can take on only positive values.

```python
t = Symbol('t', positive=True)
lamda = Symbol('lamda', positive=True)
```

Next we construct the expression for the density. Notice the use of `exp` for the exponential function.

```python
expon_density = lamda * exp(-lamda * t)
expon_density
```

To see that the function is a density, we can check that its integral from 0 to $\infty$ is 1. The symbol that `SymPy` uses for $\infty$ is `oo`, a double lower case o. It looks very much like $\infty$.

```python
Integral(expon_density, (t, 0, oo)).doit()
```

Suppose $T$ has the exponential $(\lambda)$ density. Then for $t \ge 0$ the cdf of $T$ is

$$
F_T(t) ~ = ~ P(T \le t) ~ = ~ \int_0^t \lambda e^{-\lambda s}ds
$$

This is a straightforward integral that you can probably do in your head. However, let's get some more practice using `SymPy` to find cdf's.
We will use the same method that we used to find the cdf in the previous example.

$$
\int_0^t \lambda e^{-\lambda s}ds ~ = ~ I(t) - I(0)
$$

where $I$ is the indefinite integral of the density. To get this indefinite integral we will use `Integral` as before, except that this time we must specify `t` as the variable of integration. That is because `SymPy` sees two symbols `t` and `lamda` in the density, and doesn't know which one is the variable unless we tell it.

```python
indefinite = Integral(expon_density, t).doit()
indefinite
```

Now use $F_T(t) = I(t) - I(0)$:

```python
I_0 = indefinite.subs(t, 0)
I_0
```

```python
cdf = indefinite - I_0
cdf
```

Thus the cdf of $T$ is

$$
F_T(t) ~ = ~ 1 - e^{-\lambda t}
$$

The expectation of $T$ is

$$
E(T) ~ = ~ \int_0^\infty t \lambda e^{-\lambda t} dt ~ = ~ \frac{1}{\lambda}
$$

which you can check by integration by parts. But `SymPy` is faster:

```python
expectation = Integral(t*expon_density, (t, 0, oo)).doit()
expectation
```

Calculating $E(T^2)$ is just as easy.

```python
expected_square = Integral(t**2 * expon_density, (t, 0, oo)).doit()
expected_square
```

The variance and SD follow directly.

```python
variance = expected_square - (expectation ** 2)
variance
```

```python
sd = variance ** 0.5
sd
```

That's a pretty funny way of writing $\frac{1}{\lambda}$ but we'll take it. It's a small price to pay for not having to do all the integrals by hand.

```python

```


# Polynomial Optimization

## Technical note

The section "Sum-of-Squares approach" of notebook uses features of SumOfSquares.jl and PolyJuMP.jl that are not yet released.
Please do the following to use the "master" branch

```julia
Pkg.checkout("SumOfSquares")
Pkg.checkout("PolyJuMP")
```

You can undo these with the following two lines

```julia
Pkg.free("SumOfSquares")
Pkg.free("PolyJuMP")
```

## Introduction

Consider the polynomial optimization problem of
minimizing the polynomial $x^3 - x^2 + 2xy -y^2 + y^3$
over the polyhedron defined by the inequalities $x \ge 0, y \ge 0$ and $x + y \geq 1$.

```julia
using DynamicPolynomials
@polyvar x y
p = x^3 - x^2 + 2x*y -y^2 + y^3
using SemialgebraicSets
S = @set x >= 0 && y >= 0 && x + y >= 1
p(x=>1, y=>0), p(x=>1//2, y=>1//2), p(x=>0, y=>1)
```

    (0, 1//4, 0)

The optimal solutions are $(x, y) = (1, 0)$ and $(x, y) = (0, 1)$ with objective value $0$ but [Ipopt](https://github.com/JuliaOpt/Ipopt.jl/) only finds the local minimum $(1/2, 1/2)$ with objective value $1/4$.

```julia
using JuMP
using Ipopt
m = Model(optimizer=IpoptOptimizer(print_level=0))
@variable m a >= 0
@variable m b >= 0
@constraint m a + b >= 1
@NLobjective(m, Min, a^3 - a^2 + 2a*b - b^2 + b^3)
JuMP.optimize(m)
@show JuMP.terminationstatus(m)
@show JuMP.resultvalue(a)
@show JuMP.resultvalue(b)
@show JuMP.objectivevalue(m);
```

With the following equivalent model, [Ipopt](https://github.com/JuliaOpt/Ipopt.jl/) finds the correct optimal solution. The reason (although counter intuitive) of the difference is that with registered functions, only first order derivatives are available.

```julia
using JuMP
using Ipopt
m = Model(solver=IpoptSolver(print_level=0))
@variable m a >= 0
@variable m b >= 0
@constraint m a + b >= 1
peval(a, b) = p(x=>a, y=>a)
JuMP.register(m, :peval, 2, peval, autodiff=true)
@NLobjective(m, Min, peval(a, b))
status = solve(m)
@show status
@show getvalue(a)
@show getvalue(b)
@show getobjectivevalue(m);
```

## Sum-of-Squares approach

We will now see how to find the optimal solution using Sum of Squares Programming.
We first need to pick an SDP solver, see [here](http://www.juliaopt.org/) for a list of the available choices.

```julia
using CSDP
optimizer = CSDPOptimizer(printlevel=0);
```

```julia
using MathOptInterfaceMosek
optimizer = MosekOptimizer(LOG=0);
```

A Sum-of-Squares certificate that $p \ge \alpha$ over the domain `S`, ensures that $\alpha$ is a lower bound to the polynomial optimization problem.
The following program searches for the largest upper bound and find zero.

```julia
using JuMP
using SumOfSquares
const MOI = MathOptInterface
MOI.empty!(optimizer)
m = SOSModel(optimizer = optimizer)
@variable m α
@objective m Max α
c3 = @constraint m p >= α domain = S

JuMP.optimize(m)
@show JuMP.terminationstatus(m)
@show JuMP.objectivevalue(m);
```

    JuMP.terminationstatus(m) = Success::MathOptInterface.TerminationStatusCode = 0
    JuMP.objectivevalue(m) = -2.0092666419557759e-10

Using the solution $(1/2, 1/2)$ found by Ipopt of objective value $1/4$
and this certificate of lower bound $0$ we know that the optimal objective value is in the interval $[0, 1/4]$
but we still do not know what it is (if we consider that we did not try the solutions $(1, 0)$ and $(0, 1)$ as done in the introduction).
If the dual of the constraint `c3` was atomic, its atoms would have given optimal solutions of objective value $0$ but that is not the case.

```julia
using MultivariateMoments
μ3 = JuMP.resultdual(c3)
X3 = certificate_monomials(c3)
ν3 = matmeasure(μ3, X3)
extractatoms(ν3, 1e-3) # Returns nothing as the dual is not atomic
```

    Nullable{MultivariateMoments.AtomicMeasure{Float64,Array{DynamicPolynomials.PolyVar{true},1}}}()

Fortunately, there is a hierarchy of programs with increasingly better programs that can be solved until we get one with atom dual variables.
This comes from the way the Sum-of-Squares constraint with domain `S` is formulated.
The polynomial $p - \alpha$ is guaranteed to be nonnegative over the domain `S` if there exists Sum-of-Squares polynomials $s_0$, $s_1$, $s_2$, $s_3$ such that
$$ p - \alpha = s_0 + s_1 x + s_2 y + s_3 (x + y - 1). $$
Indeed, in the domain `S`, $x$, $y$ and $x + y - 1$ are nonnegative so the right-hand side is a sum of squares hence is nonnegative.
Once the degrees of $s_1$, $s_2$ and $s_3$ have been decided, the degree needed for $s_0$ will be determined but we have a freesom in choosing the degrees of $s_1$, $s_2$ and $s_3$.
By default, they are chosen so that the degrees of $s_1 x$, $s_2 y$ and $s_3 (x + y - 1)$ match those of $p - \alpha$ but this can be overwritten using the $maxdegree$ keyword argument.

### The maxdegree keyword argument

The maximum total degree (i.e. maximum sum of the exponents of $x$ and $y$) of the monomials of $p$ is 3 so the constraint in the program above is equivalent to `@constraint m p >= α domain = S maxdegree = 3`..
That is, since $x$, $y$ and $x + y - 1$ have total degree 1, the sum of squares polynomials $s_1$, $s_2$ and $s_3$ have been chosen with maximum total degree $2$.
Since these polynomials are sums of squares, their degree must be even so the next maximum total degree to try is 4.
For this reason, the keywords `maxdegree = 4` and `maxdegree = 5` have the same effect in this example.
In general, if the polynomials in the domain are not all odd or all even, each value of `maxdegree` has different effect in the choice of the maximum total degree of $s_i$.

```julia
using JuMP
using SumOfSquares
const MOI = MathOptInterface
MOI.empty!(optimizer)
m = SOSModel(optimizer = optimizer)
@variable m α
@objective m Max α
c5 = @constraint m p >= α domain = S maxdegree = 5

JuMP.optimize(m)
@show JuMP.terminationstatus(m)
@show JuMP.objectivevalue(m);
```

    JuMP.terminationstatus(m) = Success::MathOptInterface.TerminationStatusCode = 0
    JuMP.objectivevalue(m) = -8.707343734926098e-10

This time, the dual variable is atomic as it is the moments of the measure
$$0.5 \delta(x-1, y) + 0.5 \delta(x, y-1)$$
where $\delta(x, y)$ is the dirac measure centered at $(0, 0)$.
Therefore the program provides both a certificate that $0$ is a lower bound and a certificate that it is also an upper bound since it is attained at the global minimizers $(1, 0)$ and $(0, 1)$.

```julia
using MultivariateMoments
μ5 = JuMP.resultdual(c5)
X5 = certificate_monomials(c5)
ν5 = matmeasure(μ5, X5)
extractatoms(ν5, 1e-3)
```

    Nullable{MultivariateMoments.AtomicMeasure{Float64,Array{DynamicPolynomials.PolyVar{true},1}}}(Atomic measure on the variables x, y with 2 atoms:
     at [-0.00109073, 1.00109] with weight 0.49908199889134325
     at [0.99992, 8.0388e-5] with weight 0.5006917587661682)

## A deeper look into atom extraction

The `extractatoms` function requires a `ranktol` argument that we have set to `1e-3` in the preceding section.
This argument is used to transform the dual variable into a system of polynomials equations whose solutions give the atoms.
This transformation uses the SVD decomposition of the matrix of moments and discard the equations corresponding to a singular value lower than `ranktol`.
When this system of equation has an infinite number of solutions, `extractatoms` concludes that the measure is not atomic.
For instance, with `maxdegree = 3`, we obtain the system
$$x + y = 1$$
which contains a whole line of solution.
This explains `extractatoms` returned `nothing`.

```julia
ν3 = matmeasure(μ3, X3)
MultivariateMoments.computesupport!(ν3, 1e-3)
```

    Algebraic Set defined by 1 equality
     -x - 1.0000000000000007y + 1.0000000000569826 == 0

With `maxdegree = 5`, we obtain the system
\begin{align}
  x + y & = 1\\
  y^2 & = y\\
  xy & = 0\\
  x^2 + y & = 1
\end{align}

```julia
ν5 = matmeasure(μ5, X5)
MultivariateMoments.computesupport!(ν5, 1e-3)
```

    Algebraic Set defined by 4 equalities
     -x - 0.9999999999999996y + 1.0000000002403562 == 0
     -y^2 + 1.0011711224086954y - 8.047569308633484e-5 == 0
     -xy + 1.4187906239762362e-12y + 0.00018536057700866868 == 0
     -x^2 - 1.0011711224223037y + 1.0010906469640255 == 0

This system can be reduced to the equivalent system
\begin{align}
  x + y & = 1\\
  y^2 & = y
\end{align}
which has the solutions $(0, 1)$ and $(1, 0)$.

```julia
SemialgebraicSets.computegröbnerbasis!(ideal(get(ν5.support)))
get(ν5.support)
```

    Algebraic Set defined by 2 equalities
     x + 0.9999999999999996y - 1.0000000002403562 == 0
     y^2 - 1.0011711224086954y + 8.047569308633484e-5 == 0

The function `extractatoms` then reuse the matrix of moments to find the weights $1/2$, $1/2$ corresponding to the diracs centered respectively at $(0, 1)$ and $(1, 0)$.
This details the how the function obtained the result
$$0.5 \delta(x-1, y) + 0.5 \delta(x, y-1)$$
given in the previous section.

```julia

```


Tutorial 3: Probability

Adnane Ez-zizi, 16 Oct 2019

$\newcommand{\E}{\mathrm{E}}$
$\newcommand{\Var}{\mathrm{Var}}$
$\newcommand{\Cov}{\mathrm{Cov}}$
$\newcommand{\Corr}{\mathrm{Corr}}$

This notebook introduces the elementary concepts from the probability theory that are necessary to understand and build machine learning models. For example, we will cover notions such as a random variable, a probability distribution and independence, as well as useful properties of probability like the chain rule or Bayes theorem. We will also illustrate these notions using Python code.

## 0. Preliminary steps

```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
#%precision 3
```

## 1. Definition of probability

Probability, at its core, refers to the likelihood or the chance that a certain event will occur. For instance, when we roll a fair die, we can intuitively report that there is a probability of $\frac{1}{6}$ that one of the six possible outcomes (1, 2, 3, 4, 5 or 6) will occur.

Probability, as a field, is branch of mathematics that provides tools to deal and measure uncertainty. In machine learning, we often have to deal with uncertain or stochastic quantities, whether the system that we try to model is assumed inherently stochastic like the trajectory of a subatomic particle, or cannot be completely observed like a robot with a low quality camera trying to identify objects, or because of incomplete modelling like when have to discard some of the information that we have collected (e.g., a robot that discretises the space when making a prediction about the position of an object).

Before we move to more advanced notions, let's cover some elementary terminology that is commonly used in probability theory:

- **Experiment:** any process, usually random, that results in one of many possible results. For example in our first example, the toss of a die can be considered as an experiment.
- **Trial:** one repetition of the process of an experiment. For example, in the experiment of tossing a die 10 times and recording the sequence of results, each toss is considered as a trial.
- **Outcome:** each distinct possible result of an experiment. For example, in the experiment of tossing a die twice, one possible outcome is (3,5) where 3 is the result from the first trial and 5 is the result from the second trial.
- **Event:** consists of one or more outcomes. For example, in the experiment of tossing on die, drawing an even number can be considered as an event, which has 3 possible outcomes (2, 4 and 6).
- **Sample space** set of all possible outcome. For the die experiment, the sample space is {1,2,3,4,5,6}.

In mathematical notation, the probability of an event E is often denoted as $P(E)$ or $Pr(E)$.

## 2. Random variables

A random variable is a function that maps a possible outcome of a random process to a real number. For example, we can define a random variable that represents the outcome from a die roll, $X \in \{1,2,3,4,5,6\}$, or the sum of the outcomes from rolling two dice, $Y \in \{1,2,\cdots,12\}$.

Once we have defined a random variable, we can talk about its probability of taking a certain value or of taking a value in a given interval. Thus, the probability of observing a 3 when we roll a die is expressed as $P(X = 3)$, while the probability of observing an even outcome can be expressed as $P(X \in \{2,4,6\})$.

There are two types of random variables: discrete and continuous. A random variable is said to be discrete if it takes a finite or countably infinite number of values such as the variable $X$ representing the die roll's outcome. A continous random variable can take real values such as the value of a share in a stock exchange market.

## 3. Discrete variables

### 3.1. Probability mass function

The probability mass function, or distribution of a discrete random variable $X$ maps the possible values of $X$ ($x_1, x_2, \dots$) to their probabilities. We will denote a probability mass function by a lower-cased $p(.)$: $p(x_i) = P(X = x_i)$ for all possible values $x_i$'s of X.

For example, the probability mass function of the random variable $X$ that represents a die roll outcome is defined by:
\begin{equation}
p(x) = P(X = x) = \frac{1}{6} \;\; \text{for} \; x \in \{1, 2, 3, 4, 5, 6\}
\end{equation}

A probability mass function $p$ of a random variable $X$ must satisfy 3 conditions:

1. The domain of $p$ must be the set of all possible outcomes of $X$.
2. for all possible values $x$ of $X$, $0\leq p(x) \leq 1$, where $p(x) = 0$ if $x$ can never occur (impossible outcome) and $p(x) = 1$ if $x$ always occur.
3. $\sum_{x} {p(x)} = 1$, that is, the probabilities of all possible outcomes sum to 1.

### 3.2. Expectation and variance

#### Expectation

The expectation or expected value (or mean) of a discrete random variable $X$ is defined as:
$\E(X) = \sum_{i} {x_ip(x_i)}$. In other words, the expectation is a weighted average of the possible values $x_i$ with weights $p(x_i)$, similar to a center of mass in physics. It gives us an idea about the central value taken by the random variable.

For the die roll example above, the expectation is equal to:
\begin{equation}
\E(X) = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6} = \frac{21}{6} = 3.5
\end{equation}

The expectation has a few properties:
- The expectation of a deterministic value is equal to the value itself: $\E(x) = x$, where x is a non-random value.
- The expectation is a linear function, that is: $\E(aX + bY) = a\E(X) + b\E(Y)$.
- $\E(f(X)) = \sum_{i} {f(x_i)p(x_i)}$

#### Variance

The variance provides a measure of how much the values of a random variable $X$ vary as we sample different values of $X$ from its probability distribution. Mathematically, it is defined as: $\Var(X) = \E(\left[X - \E(X)\right]^2) = \E(X^2) - \E(X)^2$. For instance, the variance of $X$ in the die roll example is given by:
\begin{equation}
\Var(X) = \E(X^2) - \E(X)^2 = (1^2 \cdot \frac{1}{6} + 2^2 \cdot \frac{1}{6} + 3^2 \cdot \frac{1}{6} + 4^2 \cdot \frac{1}{6} + 5^2 \cdot \frac{1}{6} + 6^2 \cdot \frac{1}{6}) - \frac{21}{6} = \frac{71}{6}
\end{equation}

The variance has a few properties:
- The variance of a deterministic value is equal to 0: $\Var(x) = 0$, where $x$ is a non-random value.
- The variance preserves the sum: $\Var(X + Y) = \Var(X) + \Var(Y)$.
- The variance is a non linear function for the multiplication by a scalar: $\Var(aX) = a^2\Var(X)$.
- $\Var(f(X)) = \E(f(X)^2) - \E(f(X))^2$

The standard deviation is a related measure defined as the square root of the variance: $SD(X) = \sqrt{\Var(X)}$.

### 3.3. Some popular discrete probability distributions

#### Bernoulli distribution

Suppose that we run a random experiment whose outcome can be classified as either a success or a failure. An example of such an experiment is tossing a fair coin where a head is considered as success and tail as failure. If we let $X = 1$ when the outcome is a success (e.g. head) and $X = 0$ when it is a failure (e.g. tail), then the probability mass function of $X$ is given by:
\begin{aligned}
& p(0) = P(X = 0) = 1 − p\\
& p(1) = P(X = 1) = p
\end{aligned}
where $p$ is the probability that the experiment is a success ($0\leq p\leq 1$).

A random variable $X$ is said to be a Bernoulli random variable if its probability mass function is given by the equations
shown just above for some $p \in (0, 1)$. We also say that $X$ follows a Bernoulli distribution with parameter $p$ and write this mathematically as $X \sim Bernoulli(p)$.

```python
# Mass function of a Bernoulli(0.7) distribution
print('p(0) ={p0: 1.1f}'.format(p0 = stats.bernoulli.pmf(0, p = 0.7)))
print('p(1) ={p0: 1.1f}'.format(p0 = stats.bernoulli.pmf(1, p = 0.7)))
```

    p(0) = 0.3
    p(1) = 0.7

We can draw samples from a Bernoulli distribution, using the function `stats.bernoulli.rvs()`. For example, if we want to draw 10 values from a $Bernoulli(0.7)$ distribution, we would use:

```python
# Draw 10 values from a Bernoulli variable with p = 0.7
stats.bernoulli.rvs(p = 0.7, size = 10)
```

The expection and variance of a random variable $X \sim Bernoulli(p)$ can be computed in a straightforward manner as:
\begin{align}
& \E(X) = 0 \cdot p(0) + 1 \cdot p(1) = p\\
& \Var(X) = E(X^2) - E(X)^2 = (0^2 \cdot p(0) + 1^2 \cdot p(1)) - p^2 = p(1-p)
\end{align}

```python
# Expectation and variance of a X ~ Bernoulli(0.7)
print('E(X) ={exp: 1.2f}'.format(exp = stats.bernoulli.mean(p = 0.7)))
print('Var(X) ={var: 1.2f}'.format(var = stats.bernoulli.var(p = 0.7)))
```

#### Binomial distribution

Suppose now that we perform an experiment with $n$ independent trials, each of which results in a success with probability $p$ and in a failure with probability $1 − p$. If $X$ represents the number of successes that occur in the $n$ trials, then $X$ is said to be a binomial random variable with parameters $(n, p)$. A Bernoulli random variable is thus just a binomial random variable with parameters $(1, p)$.

The probability mass function of a binomial random variable, $X$, having parameters $(n, p)$ (we can write $X \sim B(n,p)$), is given by:
\begin{equation}
p(i) = {n \choose i} p^i (1 − p)^{n−i} \quad i = 0, 1,\dots, n
\end{equation}
where ${n \choose i}$ is the number of possible combinations that result from randomly drawing $i$ objects out of $n$ different objects. Mathematically, ${n \choose i} = \frac{n!}{i!\,(n-i)!}$, where $k! = k\times(k-1)\times(k-2)\dots\times1$.

```python
# Mass function of a B(6,0.5) distribution
for i in range(7):
    print('p({j:d}) = {p0:1.4f}'.format(j = i, p0 = stats.binom.pmf(i, n = 6, p = 0.5)))
```

    p(0) = 0.0156
    p(1) = 0.0938
    p(2) = 0.2344
    p(3) = 0.3125
    p(4) = 0.2344
    p(5) = 0.0938
    p(6) = 0.0156

```python
# Plor of the mass function
fig, ax = plt.subplots(1, 1)
n, p = 6, 0.5
x = list(range(7))
ax.plot(x, stats.binom.pmf(x, n, p), 'bo', ms=8, label='binom pmf')
ax.vlines(x, 0, stats.binom.pmf(x, n, p), colors='b', lw=5, alpha=0.5)
plt.show()
```

In code, to draw samples from a Binomial distribution, we can call `stats.binom.rvs()`. For example, drawing 3 values from a $B(6, 0.5)$ distribution can be done using the following code:

```python
# Draw 3 values from a variable following a B(6, 0.5)
print(stats.binom.rvs(n = 6, p = 0.5, size = 3))
```

    [3 5 1]

This means that we got 4 successes ($X = 1$) in each of the first two draws, and got 2 successes in the last draw.
The expection and variance of a random variable $X \sim B(n,p)$ are given by:
\begin{align}
& \E(X) = np\\
& \Var(X) = p(1-p)
\end{align}

```python
# Expectation and variance of a X ~ B(6, 0.5)
print('E(X) ={exp: 1.1f}'.format(exp = stats.binom.mean(n = 6, p = 0.5)))
print('Var(X) ={var: 1.1f}'.format(var = stats.binom.var(n = 6, p = 0.5)))
```

    E(X) = 3.0
    Var(X) = 1.5

## 4. Continuous variables

### 4.1. Probability density function

When working with a continuous random variable, we no longer want to find the probability that the variable take a specific value such as asking what is the probability that the value of a certain share in a financial market is exactly £100.684, since this probability will virtually be 0. Instead, we are more interested in the probability that the continuous random variable lies in a certain interval. For that, we normally describe the distribution of a random variable using a probability density function instead of a probability mass function. More specifically, a function $f$ is said to be the density of a random variable $X$ if, for any interval $I$, $P(X \in I) = \int_{I}{f(x)dx}$.

In general, a function $f$ must satisfy 3 conditions to be considered as a density function of a random variable $X$:

1. The domain of $f$ must be the set of all possible outcomes of $X$.
2. for all possible values $x$ of $X$, $f(x) \geq 0$.
3. $\int{f(x)dx} = 1$.

### 4.2. Expectation and variance

#### Expectation

The definition of expectation in the case of continuous random variables is very similar to that in the discrete case, where we only have to replace the summation with integration, and the probability mass function with the probability density function. More specifically, if $X\sim f$ (meaning that X follows a distribution whose density function is $f$) and the domain of $X$ is $\mathcal{X}$, then: $\E(X) = \int_{\mathcal{X}} {xf(x)dx}$.

#### Variance

For variance, the definition remains the same: $\Var(X) = \E(\left[X - \E(X)\right]^2) = \E(X^2) - \E(X)^2$.

### 4.3. Cumulative distribution function

Let $X$ be a random variable. The function $F$ defined by:

\begin{equation}
F(x) = P(X \leq x), \quad -\infty \leq x\leq \infty
\end{equation}

is called the cumulative distribution function of $X$. Thus, $F$ specifies, for all real values $x$, the probability that the random variable is less than or equal to $x$.

Similarly to the probability density function, the cumulative distribution function $F$ describes fully the distribution of a random variable, with the additional advantage that it can work with any type of random variables (discrete or continuous). Actually, F can be obtained from $f$, the probability density function, using the formulas: $F(x) = \int{f(t)dt}$

The One interesting property of $F$ is that it is a nondecreasing function; that is, if $a < b$, then $F(a) \leq F(b)$.

### 4.4. Some popular continuous probability distributions

#### Uniform distribution

A random variable $X$ is said to be uniformly distributed over the interval $(a, b)$, noted in short as $X \sim U(a,b)$, if $X$ is equally likely to fall anywhere in this interval. The probability density function of such random variable is given by:
\begin{equation}
f(x) =
\left\{
	\begin{array}{ll}
		\frac{1}{b-a}  & \mbox{if } a\leq x\leq b \\
		0 & \mbox{otherwise }
	\end{array}
\right.
\end{equation}

```python
# density function of a U(0,1) distribution
print('f(-0.2) = {fx:1.1f}'.format(fx = stats.uniform.pdf(-0.2, loc = 0, scale = 1)))
print('f(0) = {fx:1.1f}'.format(fx = stats.uniform.pdf(0, loc = 0, scale = 1)))
print('f(0.3) = {fx:1.1f}'.format(fx = stats.uniform.pdf(0.1, loc = 0, scale = 1)))
print('f(1) = {fx:1.1f}'.format(fx = stats.uniform.pdf(1, loc = 0, scale = 1)))
print('f(4.5) = {fx:1.1f}'.format(fx = stats.uniform.pdf(4.5, loc = 0, scale = 1)))
```

    f(-0.2) = 0.0
    f(0) = 1.0
    f(0.3) = 1.0
    f(1) = 1.0
    f(4.5) = 0.0

```python
# Plot of the density function
fig, ax = plt.subplots(1, 1)
a, b = 0, 1
x = np.linspace(-2, 2, 200)
ax.plot(x, stats.uniform.pdf(x, loc = a, scale = b), 'r-', lw=2, label='frozen pdf')
plt.show()
```

In code, to draw samples from a Uniform distribution, we can call `stats.uniform.rvs()`. For example, drawing 10 values from a $U(0, 1)$ distribution can be done using the following code:

```python
# Draw 10 values from a variable following a U(0, 1)
stats.uniform.rvs(loc = 0, scale = 1, size = 10)
```

Since $F(x) = \int{f(t)dt}$, it follows from the above equation that the distribution function of a uniform random variable on the interval $(a, b)$ is given by:
\begin{equation}
F(x) =
\left\{
	\begin{array}{ll}
		0  & \mbox{if } x< a \\
		\frac{x-a}{b-a}  & \mbox{if } a\leq x\leq b \\
        1 & \mbox{otherwise }
	\end{array}
\right.
\end{equation}

The expectation and variance of the $U(a,b)$ distribution are given by:
\begin{align}
& \E(X) = \frac{a+b}{2}\\
& \Var(X) = \frac{(b-a)^2}{12}
\end{align}

```python
# Expectation and variance of a X ~ U(0, 1)
print('E(X) ={exp: 1.1f}'.format(exp = stats.uniform.mean(loc = 0, scale = 1)))
print('Var(X) ={var: 1.4f}'.format(var = stats.uniform.var(loc = 0, scale = 1)))
```

    E(X) = 0.5
    Var(X) = 0.0833

#### Normal distribution

$X$ is said to be a normal random variable (or normally distributed), with parameters $\mu$ and $\sigma^2$ if the density of $X$ is given by:

\begin{equation}
f(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x-\mu)^2\right)
\end{equation}

In short, we denote a normal distribution with parameters $\mu$ and $\sigma^2$ by $\mathcal{N}(\mu,\sigma^2)$, so we will also write as before $X \sim \mathcal{N}(\mu,\sigma^2)$ to introduce a normal random variable $X$.

```python
# density function of a N(0,1) distribution
print('f(-1.96) = {fx:1.2f}'.format(fx = stats.norm.pdf(-1.96, loc = 0, scale = 1)))
print('f(0) = {fx:1.2f}'.format(fx = stats.norm.pdf(0, loc = 0, scale = 1)))
print('f(1.96) = {fx:1.2f}'.format(fx = stats.norm.pdf(1.96, loc = 0, scale = 1)))
print('f(1) = {fx:1.2f}'.format(fx = stats.norm.pdf(1, loc = 0, scale = 1)))
```

    f(-1.96) = 0.06
    f(0) = 0.40
    f(1.96) = 0.06
    f(1) = 0.24

```python
# Plot of the density function
fig, ax = plt.subplots(1, 1)
mu, sigma = 0, 1
x = np.linspace(-5, 5, 100)
ax.plot(x, stats.norm.pdf(x, loc = mu, scale = sigma), 'r-', lw=2, label='frozen pdf')
plt.show()
```

In code, to draw samples from a Uniform distribution, we can call `stats.uniform.rvs()`. For example, drawing 10 values from a $U(0, 1)$ distribution can be done using the following code:

```python
# Draw 10 values from a variable following a N(0, 1)
stats.norm.rvs(loc = 0, scale = 1, size = 10)
```

    array([-1.07286337, -0.02047795, -1.0892859 ,  0.63307728, -0.71625438,
           -0.43093896, -0.09998078, -0.32279445, -0.98224144, -1.03249923])

The expectation and variance of a normally distributed random variable, $X \sim \mathcal{N}(\mu,\sigma^2)$, are respectively equal to the paramters of the distribution, that is: $\E(X) = \mu$ and $\Var(X) = \sigma^2$. In other words a normal distribution is completely specified using its expectation and variance.

```python
# Expectation and variance of a X ~ U(0, 1)
print('E(X) ={exp: 1.1f}'.format(exp = stats.norm.mean(loc = 0, scale = 1)))
print('Var(X) ={var: 1.1f}'.format(var = stats.norm.var(loc = 0, scale = 1)))
```

    E(X) = 0.0
    Var(X) = 1.0

## 5. Joint and conditional distributions

### 5.1. Joint probability

Up to now, we have only discussed probability distributions for single random variables. But what if we are interested in the outcomes of two or more random variables? For example, suppose we toss two fair coins and wants to know the probability that the first coin turns up head and the second turns up tail, or in a formal way, $P(X_1 = H, X_2 = T)$, where $X_1$ and $X_2$ respectively represent the outcome of the first and second coin. Let's write the probabilities of all possible joint outcomes in a table:

outcome | probability
---- | -------
HT | 1/4
HH | 1/4
TH | 1/4
TT | 1/4

What we just did is to define a joint probability distribution for the random variables $X_1$ and $X2$. Technically, as before, we can talk either about a joint probability mass function in the case of discrete variables or joint probability density function (or cumulative distribution function) when the variables are continuous.

#### Joint probability mass function (discrete case):

Assume we have two discrete random variables $X$ and $Y$, which can take on values $x_1, x_2, \dots$ and $y_1, y_2, \dots$, respectively. Their joint probability mass function is defined as:

\begin{equation}
p(x_i, y_i) = P(X = x_i, Y = y_i) \; \text{for all possible values $x_i$'s and $y_i$'s of $X$ and $Y$}.
\end{equation}

A joint probability mass function $p$ of a random variable $X$ must satisfy the same 3 conditions seen before for a standard mass function:

1. The domain of $p$ must be the set of all possible joint outcomes of $X$ and $Y$.
2. for all possible values $x$ of $X$ and $y$ of $Y$, $0\leq p(x, y) \leq 1$
3. $\sum_{x,y} {p(x,y)} = 1$, that is, the probabilities of all possible outcomes sum to 1.

One of the widely-used joint distributions is the [multinomial distribution](https://en.wikipedia.org/wiki/Multinomial_distribution), which generalises the binomial distribution when we are interested in the probability of any possible combination of number of success of each of $k$ categories (instead of two) in $n$ independent trials. For example, say we roll a 6-sided die $n$ times independently. The multinomial distribution would then give the probabiliy of counts of each of the 6 sides (here, $k=6$). The parameters of a multinomial are: $n$, the number of independent trials and $p_1, \dots, p_k$ the probabilities of successes of each of the $k$ categories to compute the count of successes for. We thus often denote the multinomial distribution by $Mult(n, p_1, \dots, p_k)$

#### Marginal mass function (discrete case)

If we have a joint distribution of multiple discrete variables, we can extract the mass function of one of the variables by summing over the possible values of the other variables. The obtained distribution of that variable is called marginal distribution. Concretelely, say we have two random variables $X$ and $Y$ that represent the outcomes of coin flips. Their joint mass function is given by:

\begin{equation}
p(i,j) = \frac{1}{4} \;\text{for } i,j \in \{H,T\}
\end{equation}

The marginal mass function of $X$ can be extracted as follows:

\begin{equation}
p(i) = \sum_{j \in \{H,T\}}{p(i,j)} = \frac{1}{4} + \frac{1}{4} = \frac{1}{2}
\end{equation}

#### Joint cumulative and density functions (continuous case):

Joint cumulative distribution function can be easily generalised from the single variable case to the multiple-variable case. If $X$ and $Y$ are two continous random variables, their joint cumulative distribution function is defined as:

\begin{equation}
F(x, y) = P(X \leq x, Y \leq y), \quad -\infty \leq x,y\leq \infty
\end{equation}

We say that $f$ is the joint probabiliy density function of $X$ and $Y$ if, for any interval $I$ and $J$, $P(X \in I, Y \in J) = \int_{I,J}{f(x,y)dxdy}$. The density function $f$ also satisfies the same 3 conditions seen in the case of a single variable, that is:

1. The domain of $f$ must be the set of all possible joint outcomes of $X$ and $Y$.
2. for all possible values $x$ of $X$ and $y$ of $Y$, $f(x, y) \geq 0$.
3. $\int{f(x,y)dxdy} = 1$.

#### Marginal density function (continous case)

As in the discrete case, if we have a joint distribution of two (or multiple) continuous random variables $X$ and $Y$, we can marginalizing over $Y$ to get the marginal distrubution of $X$. This is done by integrating over the values of $Y$:

\begin{equation}
f(x) = \int{f(x,y)dy}
\end{equation}

### 5.2. Conditional probability

#### Definition

We often want to know the probability of one event, given that we know another event is true. For instance, given that a house is located in a posh area, what is the probability that its price is greater than £500,000. This is refered to as a conditional probability, and in our case, we write it as $P(\text{price } > 500000 \text{ | area is posh})$.

If we know the joint distribution of two random variables, then we can get the conditional distribution of one variable given the other variable as follows:

- In the discrete case, the conditional mass function of $X$ given $Y$ is equal to: $p(x|y) = \frac{p(x,y)}{p(y)} = \frac{p(x,y)}{\sum_{x}{p(x,y)}}$

- In the continuous case, the conditional density function of $X$ given $Y$ is equal to: $f(x|y) = \frac{f(x,y)}{f(y)} = \frac{f(x,y)}{\int{f(x,y)dx}}$

The definition can be extended easily to the case of more than two variables by simply replacing one of the variable in the formula with the multiple variables that we want to condition on or compute the conditial distribution for.

#### Chain rule

Any joint probability distribution over many random variables may be decomposed into conditional distributions over only one variable:

\begin{equation}
P(x_1, \cdots , x_n) = P(x_n|x_{n-1}, \cdots, x_1)P(x_{n-1}|x_{n-2}, \cdots, x_1) \cdots P(x_2|x_1)  P(x_1)
\end{equation}

Thus, for example, we have:

\begin{equation}
P(x,y,z) = P(x|y,z)P(y|z)P(z)
\end{equation}

#### Bayes theorem

Bayes theorem, named after the reputable Mathematician Thomas Bayes, is a simple yet a powerful formula that allows to flip a conditional probability. It basically states that:

\begin{equation}
p(y|x) = \frac{p(x|y)p(y)}{p(x)} = \frac{p(x|y)p(y)}{\sum_y{p(x|y)p(y)}}
\end{equation}

One major application of Bayes theorem is to compute the probability of a model (or the parameters of the model to be more precise) given some data. Knowing the model, we can determine the probability of the data given a specific set of model parameters. Bayes’ rule allows us to get from the probability of the data given the model $P(data|model)$, to the probability of the model given the data $P(model|data)$. In other words, it allows us to estimate the parameters of a model having collected a set of data.

### 5.3. Independence and conditional independence

#### Independence

Two random variables $X$ and $Y$ are said to be independent (often denoted as $X\perp Y$) if the values taken by one of the variables have no influence in the values taken by the other variable. Formally, this can be expressed using conditional probabilities as: $p(x|y) = p(x)$ in the discrete case, or $f(x|y) = f(x)$ in the continous case.

knowing that $p(x|y) = \frac{p(x,y)}{p(y)}$

then $\frac{p(x,y)}{p(y)} = p(x)$

which means that $p(x,y) = p(x)p(y)$

Actually, in textbooks of probability, you often find that independence is introduced using the last formula instead of using conditional probabilities.

#### Conditional independence

Two random variables $X$ and $Y$ are conditionally independent given a random variable $z$ (denoted as $X\perp Y \;|\; Z$) if the conditional joint mass (or density) function of $X$ and $Y$ is equal to the product of the conditional mass (or density) functions of the two random variables:

\begin{equation}
p(x,y|z) = p(x|z)p(y|z)
\end{equation}

### 5.4. Covariance and correlation

#### Covariance

The covariance between $X$ and $Y$, denoted by $\Cov(X,Y)$, is defined by

\begin{equation}
\Cov(X,Y) = \E\left[(X-\E(X)) (Y-\E(Y))\right]
\end{equation}

The covariance has a few properties:
- The covariance is symmetric: $\Cov(X,Y) = \Cov(Y,X)$
- The covariance is a linear function of each argument, that is: $\Cov(aX,Y) = a\,\Cov(Y,X)$.
- If $X$ and $Y$ are independent, then $\Cov(X,Y) = 0$.

#### Correlation

The correlation of two random variables $X$ and $Y$, denoted by $\Corr(X,Y)$, is defined by:

\begin{equation}
\Corr(X,Y)= \frac{\Cov(X,Y)}{\sqrt{\Var(X)\Var(Y)}}
\end{equation}

where $\Var(X)$ and $\Var(Y)$ are strictly positive. One can show that $-1\leq\Corr(X,Y)\leq 1$

The correlation coefficient is a measure of the degree of linearity between $X$ and $Y$. A value of $\Corr(X,Y)$ near $1$ or $−1$ indicates a high degree of linearity between $X$ and $Y$, whereas a value near $0$ indicates that such linearity is absent. A positive value of $\Corr(X,Y)$ indicates that $Y$ tends to increase when $X$ does, whereas a negative value indicates that $Y$ tends to decrease when $X$ increases. If $\Corr(X,Y) = 0$, then $X$ and $Y$ are said to be uncorrelated.

## 6. Law of large numbers, Central limit theorem

Here we will cover two of the most fundamental theorems in Probability

### 6.1. Law of large numbers

Suppose that we have a collection of random variables $X_1, X_2, \cdots, X_n$ that are independent and identically distributed. The strong law of large numbers states that as the number of random variables increase ($n\to \infty$), the empirical average of the random variables will get closer to the true common expected value of the variables normal distribution:

\begin{equation}
\frac{1}{n}\sum_{i=1}^n{X_i} \xrightarrow[n\to \infty]{} \E(X)
\end{equation}

where the convergence is almost sure, that is:

\begin{equation}
P\left( \lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n{X_i} = \E(X) \right) = 1
\end{equation}

### 6.2. Central limit theorem

Let's $X_1, X_2, \cdots, X_n$ be a collection of random variables that are independent and identically distributed. The central limit theorem states that if we sum them up, then the larger the collection, the closer the sum will be to a normal distribution. Mathematically, this is translated as:

\begin{equation}
\frac{1}{\sqrt{n}} \sum_{i=1}^n \frac{X_i - \mu}{\sigma} \xrightarrow[n\to \infty]{} \mathcal{N}(0, 1) \quad \left(\text{or equivalently}, \sum_{i=1}^n{X_i} \underset{n\to \infty}{\sim} \mathcal{N}\left(\mu, \frac{\sigma}{n}\right)\right)
\end{equation}

where $\mu$ and $\sigma^2$ are respectively the common expectation and variance of the random variables $X_1, X_2, \cdots, X_n$, and the convergence is in law, that is:

\begin{equation}
\lim_{n \to \infty} P\left( \frac{1}{\sqrt{n}} \sum_{i=1}^n \frac{x_i - \mu}{\sigma} \le z \right) = \int_{-\infty}^z
        (1/\sqrt{2 \pi}) \exp(-u^2/2) \, du
\end{equation}


```
%matplotlib inline
from sympy import var, Matrix, eye, init_printing, roots
init_printing()
```

```
var("a:5")
var("b:5")
var("s");
```

# Formas canónicas observador y observabilidad

Tenemos un sistema de orden $4$ con una representación de estado:

```
Ao = Matrix([[0, 0, 0, -a4], [1, 0, 0, -a3], [0, 1, 0, -a2], [0, 0, 1, -a1]])
bo = Matrix([[b4], [b3], [b2], [b1]])
co = Matrix([[0], [0], [0], [1]])
```

```
Ao
```

```
bo
```

```
co
```

La cual ya esta en la forma canónica observador, podemos obtener su polinomio caracteristico:

```
PCo = (s*eye(4) - Ao).det()
PCo
```

La matriz sistema es la siguiente:

```
MSo = (s*eye(4) - Ao).row_join(bo).col_join(-co.T.row_join(Matrix([0])))
MSo
```

y su polinomio es:

```
pMSo = MSo.det()
pMSo
```

tiene una matriz de observabilidad:

```
MOo = ((co.T.col_join(co.T*Ao)).col_join(co.T*Ao*Ao)).col_join(co.T*Ao*Ao*Ao)
MOo
```

```
MOo.det()
```

Dado este sistema, podemos transformarlo a su forma observabilidad:

```
Aob = Matrix([[-a1, -a2, -a3, -a4], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]])
beta = Matrix([[1, 0, 0, 0], [a1, 1, 0, 0], [a2, a1, 1, 0], [a3, a2, a1, 1]]).inv()*Matrix([[b1], [b2], [b3], [b4]])
bob = Matrix([[beta[3]], [beta[2]], [beta[1]], [beta[0]]])
cob = co
```

```
Aob
```

```
bob
```

```
cob
```

La cual, tiene un polinomio caracteristico:

```
PCob = (s*eye(4) - Aob).det()
PCob
```

y una matriz sistema:

```
MSob = (s*eye(4) - Aob).row_join(bob).col_join(-co.T.row_join(Matrix([0])))
MSob
```

con una matriz de observabilidad:

```
MOob = ((cob.T.col_join(cob.T*Aob)).col_join(cob.T*Aob*Aob)).col_join(cob.T*Aob*Aob*Aob)
MOob
```

```
MOob.det()
```

## Ejemplo

Tenemos el siguiente sistema:

```
from numpy import matrix
```

```
A = Matrix([[0, 1, 0], [0, 0 , 1], [0, 0, 0]])
b = Matrix([[0], [0], [1]])
c = Matrix([[-1], [0], [1]])
x0 = Matrix ([[1], [0], [0]])
```

Su polinomio caracteristico es:

```
PC = (s*eye(3) - A).det()
PC
```

Su matriz sistema es:

```
MS =(s*eye(3) - A).row_join(b).col_join(-c.T.row_join(Matrix([0])))
MS
```

y su polinomio es:

```
pMS =  MS.det()
pMS.factor()
```

su matriz de controlabilidad es:

```
MC = (b.row_join(A*b)).row_join(A*A*b)
MC
```

y su determinanate:

```
MC.det()
```

lo que quiere decir, que este sistema es controlable.

Ahora calculamos su matriz de observabilidad.

```
MO = (c.T.col_join(c.T*A)).col_join(c.T*A*A)
MO
```

y su determinante:

```
MO.det()
```

lo cual quiere decir que  es observable.

Sabemos que tenemos 3 polos en el origen, pero queremos que esten en $-1$, podemos calcular los coeficientes necesarios:

```
((s+1)**3).expand()
```

lo cual nos implica la siguiente retroalimentación de estado:

```
f = Matrix([[-1], [-3], [-3]])
```

y el siguiente sistema retroalimentado:

```
Af = A + b*f.T
Af
```

```
A.col_join(c.T)
```

pero para calcular la inyección de salida, deseamos hacerlo bajo la forma canónica observador, por lo que primero la describimos y despues calculamos el cambio de base necesario:

```
Ao = Matrix([[0, 0, 0], [1, 0, 0], [0, 1, 0]])
bo = Matrix([[-1], [0], [1]])
co = Matrix([[0], [0], [1]])
```

cuya función de transferencia será:

```
(co.T*((s*eye(3) - Ao).inv())*bo)[0].factor()
```

su matriz de observabilidad:

```
MOo = (co.T.col_join(co.T*Ao)).col_join(co.T*Ao*Ao)
MOo
```

y la matriz de cambio de base será:

```
T = MO.inv()* MOo
T
```

teniendo esto, tan solo tenemos que idear una inyección de salida.

En este caso, queremos una respuesta mas rapida, por lo que pondremos estos polos en $-10$:

```
((s+10)**3).expand()
```

por lo que nuestra inyección de salida será:

```
ko = Matrix([[-1000], [-300], [-30]])
```

y el sistema bajo inyección de salida será:

```
Ako = Ao + ko*co.T
Ako
```

y su polinomio caracteristico será:

```
(s*eye(3) - Ako).det()
```

pero esta inyección de salida esta definida en la nueva base, por lo que tenemos que convertirla a la base anterior:

```
k = T*ko
k
```

Si ahora juntamosla retroalimentación de estado y la inyección de salida, tendremos:

```
Alc = (A.row_join(b*f.T)).col_join((-k*c.T).row_join(Af + k*c.T))
Alc
```

```
blc = b.col_join(b)
blc
```

```
clc = c.col_join(Matrix([[0], [0], [0]]))
clc
```

el polinomio caracteristico de este sistema será:

```
(s*eye(6) - Alc).det().factor()
```

y el polinomio de la matriz sistema:

```
MSlc = ((s*eye(6) - Alc).row_join(blc)).col_join(-clc.T.row_join(Matrix([[0]])))
MSlc
```

```
MSlc.det().factor()
```

su matriz de controlabilidad será:

```
MClc = ((((blc.row_join(Alc*blc))
          .row_join(Alc*Alc*blc))
         .row_join(Alc*Alc*Alc*blc))
        .row_join(Alc*Alc*Alc*Alc*blc)).row_join(Alc*Alc*Alc*Alc*Alc*blc)
MClc
```

```
MClc.det()
```

```
MOlc = ((((clc.T.col_join(clc.T*Alc))
          .col_join(clc.T*Alc*Alc))
         .col_join(clc.T*Alc*Alc*Alc))
        .col_join(clc.T*Alc*Alc*Alc*Alc)).col_join(clc.T*Alc*Alc*Alc*Alc*Alc)
MOlc
```

```
MOlc.det()
```

```
Tlc, Jlc = Alc.jordan_form()
```

```
Jlc
```

```
Tlc
```

IPythonNotebooks/Teoria de Control I/Formas canonicas observador y observabilidad.ipynb


#Snippets and Programs from Chapter 7: Solving Calculus Problems

```python
#P182: Simple example of finding the limit
>>> from sympy import Limit, Symbol, S
>>> x = Symbol('x')
>>> Limit(1/x, x, S.Infinity).doit()
```

    0

```python
#P182: Specifying the limit direction: negative
>>> from sympy import Limit
>>> Limit(1/x, x, 0, dir='-').doit()
```

    -oo

```python
#P182: Specifying the limit direction: positive
>>> from sympy import Limit
>>> Limit(1/x, x, 0, dir='+').doit()
```

    oo

```python
#P183: Indeterminate limit example
>>> from sympy import Symbol, sin
>>> Limit(sin(x)/x, x, 0).doit()
```

    1

```python
#P183: Continuous Compound Interest
>>> from sympy import Symbol, Limit, S
>>> n = Symbol('n')
>>> p = Symbol('p', positive=True)
>>> r = Symbol('r', positive=True)
>>> t = Symbol('t', positive=True)
>>> Limit(p*(1+r/n)**(n*t), n, S.Infinity).doit()
```

    p*exp(r*t)

```python
#P184: Instantaneous Rate of Change
>>> from sympy import Symbol, Limit
>>> t = Symbol('t')
>>> St = 5*t**2 + 2*t + 8
>>> t1 = Symbol('t1')
>>> delta_t = Symbol('delta_t')
>>> St1 = St.subs({t: t1})
>>> St1_delta = St.subs({t: t1 + delta_t})
>>> Limit((St1_delta-St1)/delta_t, delta_t, 0).doit()
```

    10*t1 + 2

```python
#P185: Finding the derivative
>>> from sympy import Symbol, Derivative
>>> t = Symbol('t')
>>> St = 5*t**2 + 2*t + 8
>>> d = Derivative(St, t)
>>> d.doit().subs({t:t1})
```

    10*t1 + 2

```python
#P186: Derivative of a complicated arbitrary function
>>> from sympy import Derivative, Symbol
>>> x = Symbol('x')
>>> f = (x**3 + x**2 + x)*(x**2+x)
>>> Derivative(f, x).doit()
```

    (2*x + 1)*(x**3 + x**2 + x) + (x**2 + x)*(3*x**2 + 2*x + 1)

```python
#P186: Derivative calculator
'''
Derivative Calculator
'''
from sympy import Symbol, Derivative, sympify, pprint
from sympy.core.sympify import SympifyError

def derivative(f, var):
    var = Symbol(var)
    d = Derivative(f, var).doit()
    pprint(d)

if __name__=='__main__':
    f = input('Enter a function: ')
    var = input('Enter the variable to differentiate with respect to: ')
    try:
        f = sympify(f)
    except SympifyError:
        print('Invalid input')
    else:
        derivative(f, var)
```

    Enter a function: 2*x**2 + 3*x + 1
    Enter the variable to differentiate with respect to: x
    4⋅x + 3

```python
#P193: Use gradient ascent to find the angle at which a projectile has a maximum range

'''
Use gradient ascent to find the angle at which the projectile
has maximum range for a fixed velocity, 25 m/s
'''
import math
from sympy import Derivative, Symbol, sin

def grad_ascent(x0, f1x, x):
    epsilon =  1e-6
    step_size = 1e-4
    x_old = x0
    x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    while abs(x_old - x_new) > epsilon:
        x_old = x_new
        x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    return x_new

def find_max_theta(R, theta):
    # Calculate the first derivative
    R1theta = Derivative(R, theta).doit()
    theta0 = 1e-3
    theta_max = grad_ascent(theta0, R1theta, theta)
    return theta_max

if __name__ == '__main__':
    g = 9.8
    # Assume initial velocity
    u = 25
    # Expression for range
    theta = Symbol('theta')
    R = u**2*sin(2*theta)/g
    theta_max = find_max_theta(R, theta)
    print('Theta: {0}'.format(math.degrees(theta_max)))
    print('Maximum Range: {0}'.format(R.subs({theta:theta_max})))
```

    Theta: 44.997815081691805
    Maximum Range: 63.7755100185965

```python
%matplotlib inline
```

```python
#P193: This is the program corresponding to Figure 7-5
'''
Find the angle at which the projectile has maximum range for
a fixed velocty, u. This also shows the intermediate points.
'''

import math
import matplotlib.pyplot as plt
from sympy import Derivative, Symbol, sin, cos, solve

def plot_range_theta(u):
    g = 9.8
    angles = range(0, 90, 1)
    R = [u**2*math.sin(math.radians(2*angle))/g for angle in angles]
    plt.plot(angles, R)
    # Use LaTex for the X-axis label
    #plt.rc('text', usetex=True)
    plt.plot(angles, R)
    #plt.xlabel(r'$\theta$ : Angle of projection (degrees)')
    plt.ylabel('R: Distance traveled by projectile (meters)')

def grad_ascent(x0, f1x):
    theta = Symbol('theta')
    epsilon =  1e-6
    step_size = 1e-4
    x_old = x0
    x_new = x_old + step_size*f1x.subs({theta:x_old}).evalf()

    X = []
    while abs(x_old - x_new) > epsilon:
        X.append(x_new)
        x_old = x_new
        x_new = x_old + step_size*f1x.subs({theta:x_old}).evalf()

    return x_new, X

def find_max_theta(R, theta):
    # Calculate the first derivative
    R1theta = Derivative(R, theta).doit()
    theta0 = 1e-3
    theta_max, X = grad_ascent(0.001, R1theta)
    return math.degrees(theta_max.evalf()), X

if __name__ == '__main__':
    g = 9.8
    # Assume initial velocity
    u = 25
    plot_range_theta(u)
    theta = Symbol('theta')
    # Expression for range
    R = u**2*sin(2*theta)/g
    theta_max, X = find_max_theta(R, theta)

    # calculate R for all theta's traversed
    Y = [u**2*math.sin(2*angle)/g for angle in X]
    X = [math.degrees(angle) for angle in X]
    plt.plot(X, Y, 'ro')
    plt.show()
    print('Theta: {0}, Max. Range: {1}'.format(theta_max, R.subs({theta:theta_max})))
```

```python
#P195: Generic program for gradient ascent
'''
Use gradient ascent to find the maximum value of a
single variable function
'''
from sympy import Derivative, Symbol, sympify
def grad_ascent(x0, f1x, x):
    epsilon =  1e-6
    step_size = 1e-4
    x_old = x0
    x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    while abs(x_old - x_new) > epsilon:
        x_old = x_new
        x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    return x_new

if __name__ == '__main__':
    f = input('Enter a function in one variable: ')
    var = input('Enter the variable to differentiate with respect to: ')
    var0 = float(input('Enter the initial value of the variable: '))
    try:
        f = sympify(f)
    except SympifyError:
        print('Invalid function entered')
    else:
        var = Symbol(var)
        d = Derivative(f, var).doit()
        var_max = grad_ascent(var0, d, var)
        print('{0}: {1}'.format(var.name, var_max))
        print('Maximum value: {0}'.format(f.subs({var:var_max})))
```

    Enter a function in one variable: 25*25*sin(2*theta)/9.8
    Enter the variable to differentiate with respect to: theta
    Enter the initial value of the variable: 0.001
    theta: 0.785360029379083
    Maximum value: 63.7755100185965

```python
#P198:

'''
Use gradient ascent to find the maximum value of a single-variable function.
This also checks for the existence of a solution for the equation f'(x)=0.
'''
from sympy import Derivative, Symbol, sympify, solve
def grad_ascent(x0, f1x, x):
    # check if f1x=0 has a solution
    if not solve(f1x):
        print('Cannot continue, solution for {0}=0 does not exist'.format(f1x))
        return
    epsilon = 1e-6
    step_size = 1e-4
    x_old = x0
    x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    while abs(x_old - x_new) > epsilon:
        x_old = x_new
        x_new = x_old + step_size*f1x.subs({x:x_old}).evalf()
    return x_new

if __name__ == '__main__':
    f = input('Enter a function in one variable: ')
    var = input('Enter the variable to differentiate with respect to: ')
    var0 = float(input('Enter the initial value of the variable: '))
    try:
        f = sympify(f)
    except SympifyError:
        print('Invalid function entered')
    else:
        var = Symbol(var)
        d = Derivative(f, var).doit()
        var_max = grad_ascent(var0, d, var)
        if var_max:
            print('{0}: {1}'.format(var.name, var_max))
            print('Maximum value: {0}'.format(f.subs({var:var_max})))
```

    Enter a function in one variable: log(x)
    Enter the variable to differentiate with respect to: x
    Enter the initial value of the variable: 0.1
    Cannot continue, solution for 1/x=0 does not exist

```python
#P200: Basic example of finding the indefinite Integral
>>> from sympy import Integral, Symbol
>>> x = Symbol('x')
>>> k = Symbol('k')
>>> Integral(k*x, x).doit()
```

    k*x**2/2

```python
#P200: Basic example of finding the definite integral
>>> from sympy import Integral, Symbol
>>> x = Symbol('x')
>>> k = Symbol('k')
>>> Integral(k*x, (x, 0, 2)).doit()
```

    2*k

```python
#P203: Probability of the grade falling between  11 and 12
>>> from sympy import Symbol, exp, sqrt, pi, Integral
>>> x = Symbol('x')
>>> p = exp(-(x - 10)**2/2)/sqrt(2*pi)
>>> Integral(p, (x, 11, 12)).doit().evalf()
```

    0.135905121983278

```python
#P204: The probability density function is such that the integral between -infinity and infinity is 1
>>> from sympy import Symbol, exp, sqrt, pi, Integral, S
>>> x = Symbol('x')
>>> p = exp(-(x - 10)**2/2)/sqrt(2*pi)
>>> Integral(p, (x, S.NegativeInfinity, S.Infinity)).doit().evalf()
```

    1.00000000000000


# Exponential example.

In this notebook, we demonstrate the calculation of error bounds for the Swiss Army infinitesimal jackknife for a simple scalar exponential family model.  Although the model is trivial, it simplicity aids understanding the interplay of different elements of Theorem 1.

In this notebook, we pay particular attention to the influence of the choice of $\Omega_\theta$ on the resulting bounds.

```python
import autograd
import autograd.numpy as np
import scipy as sp
import paragami
import vittles

import tqdm
import time

import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# A convenience function to visually compare two variables.
def comparison_plot(x, y):
    plt.plot(x, x, 'k')
    plt.plot(x, y, 'r.')
```

First, draw some data from an exponential distribution.

```python
def generate_data(num_obs, theta_true):
    return np.random.exponential(np.exp(-1 * theta_true), num_obs)

num_obs = 10000
theta_true = np.log(0.1)

x = generate_data(num_obs, theta_true)

plt.hist(x, 100);
np.mean(x)
```

First, we specify the weighted MLE loss function.  In this case, the optimum is available in closed form.  For convenience we implement the closed form optimum and test that it matches the optimum of the loss function.

```python
def get_opt(w, x):
    return np.log(np.sum(w)) - np.log(np.sum(w * x))

def eval_log_loss(theta, w, x):
    ll = -1 * np.sum(w * (-1 * x * np.exp(theta) + theta)) / num_obs
    return ll

w = np.ones(num_obs) + 0.1 * (np.random.random(num_obs) - 0.5)

# Test the loss function and optimum with random weights.
log_loss_objective = paragami.OptimizationObjective(
    lambda theta: eval_log_loss(theta, w, x))
log_loss_objective.reset()
log_loss_objective.set_log_every(1)
opt = sp.optimize.minimize(
    fun=log_loss_objective.f,
    jac=log_loss_objective.grad,
    method='bfgs',
    x0=theta_true,
    options={'gtol': 1e-12})
print(opt.message)

assert(np.abs(opt.x[0] - get_opt(w, x)) < 1e-8)

```

    Iter 0: f = 3.30362589
    Iter 1: f = 3.30362524
    Iter 2: f = 3.30362524
    Iter 3: f = 3.30362524
    Optimization terminated successfully.

Get the optimum at the original weights.

```python
w1 = np.ones(num_obs)
theta_opt = get_opt(w1, x)
```

Define an IJ predictor object using ``vittles``.

```python
ij_predictor = \
    vittles.HyperparameterSensitivityLinearApproximation(
        lambda theta, w: eval_log_loss(theta, w, x),
        opt_par_value=np.array([theta_opt]),
        hyper_par_value=w1)
```

Get the IJ predictions and exact CV for a range of weights.

```python
w_loo = np.ones(num_obs)

theta_ij = np.full(num_obs, float('nan'))
theta_cv = np.full(num_obs, float('nan'))
for n in range(num_obs):
    w_loo[n] = 0
    theta_ij[n] = ij_predictor.predict_opt_par_from_hyper_par(w_loo)
    theta_cv[n] = get_opt(w_loo, x)
    w_loo[n] = 1

comparison_plot(theta_cv - theta_opt, theta_ij - theta_opt)
```

# Error bound calculation.

We now calculate error bounds for the IJ approximation.

For this model, we will parameterize the log likelihood of datapoint $x_n$ as follows:

$$
\begin{align}
\ell\left(x_{n},\theta\right)	=& -\exp(\theta) x_{n} + \theta \\
\end{align}
$$

By allowing $\theta$ to take values in all of $\mathbb{R}$, we simplify some calculation and avoid large derivatives at the boundary of the valid domain.

For convenience, we define

$$
\begin{align}
M\left(w\right)	=& \frac{1}{N}\sum_{n=1}^{N}w_{n}x_{n}\\
S\left(w\right)	=& \frac{1}{N}\sum_{n=1}^{N}w_{n}x_{n}^{2}\\
\bar{w}	=& \frac{1}{N}\sum_{n=1}^{N}w_{n}.\\
\end{align}
$$

Given these definitions,

$$
\begin{align}
g_{n}\left(\theta\right)&=-\exp\left(\theta\right)x_{n}+1\\
h_{n}\left(\theta\right)&=-\exp\left(\theta\right)x_{n}
\end{align}
$$

and the aggregated quantities are

$$
\begin{align}
L\left(\theta,w\right) &= -M\left(w\right)\exp\left(\theta\right)+\bar{w}\theta\\
G\left(\theta,w\right) &= -M\left(w\right)\exp\left(\theta\right)+\bar{w}\\
H\left(\theta,w\right) &= -M\left(w\right)\exp\left(\theta\right).\\
\end{align}
$$

Note that

$$
\exp\left(\hat{\theta}\right)=\frac{1}{M(w_1)}.
$$

Define $w_1 = (1, ...., 1)^T$ to be the vector of unity weights.

We'll define the domain as $\Omega_{\theta}=\left(\theta_{min}, \theta_{max}\right)$.

### Assumption 1.

In this case, the objective is continuous and infinitely differentiable.

### Assumption 2.

By direct calculation,

$$
\begin{align}
|H\left(\theta, w_1\right)|^{-1}	&= \frac{1}{\exp\left(\theta\right) M(w_1)}
    \le \frac{1}{\exp\left(\theta_{min}\right) M(w_1)}\Rightarrow \\
C_{op}&:= \frac{1}{\exp\left(\theta_{min}\right) M(w_1)}
\end{align}
$$

### Assumption 3.

We will bound Condition 1 directly without using Corollary 1, so the constants $C_g$ and $C_h$ are not needed.

### Assumption 4.

To calculate $L_h$, we need to choose a window $\Delta_\theta$ and upper bound the following quantity:

$$
\begin{align}
\sup_{\theta\in\left(\hat{\theta}-\Delta_{\theta},\hat{\theta}+\Delta_{\theta}\right)}
    \frac{\frac{1}{N} \left\Vert h\left(\theta\right)-h\left(\hat{\theta}\right) \right\Vert_{2}^{2}}
    {\left\Vert \theta-\hat{\theta} \right\Vert_{2}^{2}}.
\end{align}
$$

By convexity of $\left(\exp\left(\theta\right)-\exp\left(\hat{\theta}\right)\right)^{2}$, we have

$$
\begin{align}
\sup_{\theta\in\left(\hat{\theta}-\Delta_{\theta},\hat{\theta}+\Delta_{\theta}\right)}
    \frac{\frac{1}{N}\left\Vert h\left(\theta\right) - h\left(\hat{\theta}\right) \right\Vert_{2}^{2}}
         {\left\Vert \theta-\hat{\theta} \right\Vert_{2}^{2}}
 =& \sup_{\theta\in\left(\hat{\theta}-\Delta_{\theta},\hat{\theta}+\Delta_{\theta}\right)}
    \frac{\frac{1}{N} \sum_{n=1}^N \left( h_n(\theta) - h_n(\hat\theta)\right)^2}
         {\left\Vert \theta-\hat{\theta} \right\Vert_{2}^{2}}	\\
 =& \sup_{\theta\in\left(\hat{\theta}-\Delta_{\theta},\hat{\theta}+\Delta_{\theta}\right)}
    \frac{\frac{1}{N} \sum_{n=1}^N x_n^2 \left( \exp(\theta) - \exp(\hat\theta)\right)^2}
         {\left\Vert \theta-\hat{\theta} \right\Vert_{2}^{2}}	\\
 =& S(w_1)   \sup_{\theta\in\left(\hat{\theta}-\Delta_{\theta},\hat{\theta}+\Delta_{\theta}\right)}
    \frac{\left(\exp\left(\theta\right)-\exp\left(\hat{\theta}\right)\right)^{2}}{\left\Vert{\theta-\hat{\theta}}\right\Vert_{2}^{2}} \\
=& S(w_1) \frac{\left(\exp\left(\hat{\theta}+\Delta_{\theta}\right)-\exp\left(\hat{\theta}\right)\right)^{2}}{\Delta_{\theta}^{2}} \\
=& S(w_1) \frac{\exp\left(2\hat{\theta}\right)\left(\exp\left(\Delta_{\theta}\right)-1\right)^{2}}{\Delta_{\theta}^{2}}
	\\
=& \frac{S(w_1) }{M(w_1)^{2}}\frac{\left(\exp\left(\Delta_{\theta}\right)-1\right)^{2}}{\Delta_{\theta}^{2}}.
\end{align}
$$

So we can take

$$
L_{h}	=\frac{\sqrt{S(w_1)}}{M(w_1)}\frac{\left|\exp\left(\Delta_{\theta}\right)-1\right|}{\Delta_{\theta}}.
$$

### Assumption 5.

Because we are doing leave-one-out, $\sum_{n=1}^N (w_n - 1) = 1$ for all weight vectors, and

$$
C_{w}	=\sqrt{\frac{1}{N}\sum_{n=1}^{N}\left(w_{n}-1\right)^{2}}
	=\frac{1}{\sqrt{N}}.
$$

### The constant $C_{IJ}$.

Here, the dimension $D=1$. Combining the above results,

$$
\begin{align}
C_{IJ}	=& 1+DC_{w}L_{h}C_{op} \\
=& 1+1\cdot\frac{1}{\sqrt{N}}\frac{\sqrt{S(w_1)}}{M(w_1)}\frac{\left|\exp\left(\Delta_{\theta}\right)-1\right|}{\Delta_{\theta}}\frac{\exp\left(-\theta_{min}\right)}{M(w_1)} \\
=& 1+\frac{\sqrt{S(w_1)}}{\sqrt{N}M(w_1)^2}
    \frac{\left|\exp\left(\Delta_{\theta}\right)-1\right|}{\Delta_{\theta}}\exp\left(-\theta_{min}\right)\\
\approx& 1+\frac{\sqrt{S(w_1)}}{\sqrt{N}M(w_1)^2}\left(1+\frac{1}{2}\Delta_{\theta}\right)\exp\left(-\theta_{min}\right),
\end{align}
$$

where the final approximation holds for small $\Delta_\theta$ by a Taylor series approximation to $\exp(\Delta_\theta)$.

### Condition 1.

Define

$$
x^* = \max_{n \in [N]} x_n.
$$

For leave one out, a gradient complexity bound is

$$
\begin{align}
\sup_{w\in W}\sup_{\theta\in\Omega_{\theta}}
    \left| \frac{1}{N} \sum_{n=1}^{N}\left(w_{n}-1\right)g_{n}\left(\theta\right) \right| &\le
\frac{1}{N}\sup_{\theta\in\Omega_{\theta}}\left\Vert{g\left(\theta\right)}\right\Vert_{\infty} \\
&=\frac{1}{N}\sup_{\theta\in\Omega_{\theta}} \max_{n\in\left[N\right]}
    \left|g_{n}\left(\theta\right)\right| \\
&=\frac{1}{N}\sup_{\theta\in\Omega_{\theta}}\max_{n\in\left[N\right]}
    \left|\exp\left(\theta\right)x_{n} - 1\right| \\
&\le\frac{1}{N}\sup_{\theta\in\Omega_{\theta}}\max_{n\in\left[N\right]}
    \left(\exp\left(\theta\right) x_{n} + 1\right) \\
&=\frac{1}{N}\left(
    \left( \sup_{\theta\in\Omega_{\theta}} \exp\left(\theta\right) \right)
    \left(\max_{n\in\left[N\right]} x_{n} \right) + 1
    \right) \\
&=\frac{1}{N}\left( \exp\left(\theta_{max}\right)x^{*} + 1 \right).
\end{align}
$$

Similarly, a Hessian complexity bound is given by

$$
\begin{align}
\sup_{w\in W}\sup_{\theta\in\Omega_{\theta}}
    \left|\frac{1}{N}\sum_{n=1}^{N}\left(w_{n}-1\right)h_{n}\left(\theta\right)\right|	\le &
\frac{1}{N}\sup_{\theta\in\Omega_{\theta}}\max_{n\in\left[N\right]}
    \left|\exp\left(\theta\right)x_{n}\right| \\
\le& \frac{1}{N}\exp\left(\theta_{max}\right)x^{*}.
\end{align}
$$

The gradient bound is always looser than the Hessian bound.  Consequently we can satisfy Condition 1 with

$$
\delta	=\frac{1}{N}\left(\exp\left(\theta_{max}\right)x^{*}+1\right).
$$

### Choosing $\Delta_\theta$.

In order to apply Theorem 1, we will require

$$
\begin{align}
\delta	\le& \Delta_{\theta}C_{op}^{-1}\\
=& \Delta_{\theta}M(w_1)\exp\left(\theta_{min}\right)
\end{align}
$$

and

$$
\begin{align}
\delta	\le& \frac{1}{2}C_{IJ}^{-1}C_{op}^{-1} \\
=& \frac{1}{2}\left(1+\frac{\sqrt{S(w_1)}}{\sqrt{N}M(w_1)^2}\frac{\left|\exp\left(\Delta_{\theta}\right)-1\right|}{\Delta_{\theta}}\exp\left(-\theta_{min}\right)\right)M(w_1)\exp\left(\theta_{min}\right) \\
=& \frac{1}{2}\left(M(w_1)\exp\left(\theta_{min}\right) +
    \sqrt{\frac{S(w_1)}{N}}\frac{1}{M(w_1)}\frac{\left|\exp\left(\Delta_{\theta}\right)-1\right|}{\Delta_{\theta}}\right) \\
\approx& \frac{1}{2}\left(M(w_1)\exp\left(\theta_{min}\right)+\sqrt{\frac{S(w_1)}{N}}\frac{1}{M(w_1)}\left(1+\Delta_{\theta}\right)\right).
\end{align}
$$

The complexity bound $\delta$ is set by quantities that don't depend on $\Delta_\theta$, and both bounds are unboundedly increasing in $\Delta_\theta$, so we can always find a $\Delta_\theta$ large enough for Theorem 1 to apply.

We expect $\Delta_\theta < \frac{1}{2}$, in which case satisfying the first bound automatically satisfies the second.  Consequently, for simplicity we choose

$$
\Delta_\theta = \delta C_{op}
$$

and check that the second bound is satisfied.

We now implement a function to calculate these quantities.

```python
def get_ij_error_bounds(theta_min, theta_max, x):
    ij_error_terms = dict()

    # The objective is determined by these sufficient statistics.
    m1 = np.sum(x) / num_obs
    s1 = np.sum(x ** 2) / num_obs
    x_star = np.max(x)

    c_g = np.sqrt(np.exp(2 * theta_max) * s1 + 2 * np.exp(theta_max) * m1 + 1)
    c_h = np.sqrt(np.exp(theta_max) * s1)
    c_op = np.exp(-1 * theta_min) / m1
    c_w = 1 / np.sqrt(num_obs)
    d = 1

    delta = (np.exp(theta_max) * x_star + 1) / num_obs

    delta_theta = delta * c_op + 1e-3
    l_h = (np.sqrt(s1) / m1) * np.abs(np.exp(delta_theta) - 1) / delta_theta

    # This should be true by construction.
    assert delta <= delta_theta / c_op

    c_ij = 1 + d * c_w * l_h * c_op

    # This we have to check.
    if not delta <=  0.5 / (c_ij * c_op):
        err_msg = (
            'The IJ bound is invalid -- probably \\Omega_\\theta is too large.  ' +
            '\n\nThe bound could be made valid by numerically solving the nonlinear ' +
            'equation to find an appropriate \\Delta_\\theta, but the resulting bound ' +
            'would probably be unusably loose.')
        raise ValueError(err_msg)

    err_bound = 2 * (c_op ** 2) * c_ij * (delta ** 2)

    ij_error_terms['theta_min'] = theta_min
    ij_error_terms['theta_max'] = theta_max
    ij_error_terms['c_g'] = c_g
    ij_error_terms['c_h'] = c_h
    ij_error_terms['c_op'] = c_op
    ij_error_terms['c_w'] = c_w
    ij_error_terms['d'] = d
    ij_error_terms['delta'] = delta
    ij_error_terms['delta_theta'] = delta_theta
    ij_error_terms['l_h'] = l_h
    ij_error_terms['c_ij'] = c_ij
    ij_error_terms['err_bound'] = err_bound

    return ij_error_terms

def print_error_bounds(ij_error_terms):
    for k, v in ij_error_terms.items():
        print('{:>15} = {}'.format(k, v))
```

We take an initial guess at $\Omega_\theta$ based on the asymptotic variance.  We will check and refine this guess below.

```python
h1 = num_obs * autograd.hessian(lambda theta: eval_log_loss(theta, w1, x))(theta_opt)
theta_sd = np.sqrt(1 / h1)
print('theta standard deviation: ', theta_sd)

theta_min = theta_opt - 2 * theta_sd
theta_max = theta_opt + 2 * theta_sd

ij_error_terms = get_ij_error_bounds(theta_min, theta_max, x)
print_error_bounds(ij_error_terms)
```

    theta standard deviation:  0.009999999999999992
                  d = 1
                c_w = 0.01
        delta_theta = 0.002001847745624872
          theta_min = -2.3234720841690373
               c_ij = 1.0143914341923759
                c_h = 4.503176515483211
                c_g = 2.25995438528836
               c_op = 1.0202013400267547
          err_bound = 2.0362871443197175e-06
          theta_max = -2.2834720841690372
              delta = 0.0009820098311168644
                l_h = 1.4106464702348336

Now, we check that the bounds capture the true error and that $\hat\theta(w)$ is entirely contained in $\Omega_\theta$.  Let $\epsilon$  denote the error bound.  Note that the latter condition can be checked _without calculating $\hat\theta(w)$_ using the fact that, by the triangle inequality,

$$
\begin{align}
|\hat\theta(w) - \hat\theta)| <& |\hat\theta_{IJ}(w) - \hat\theta)| + \epsilon.
\end{align}
$$

If the quantities on the right hand side are all contained in $\Omega_\theta$, then so is $\hat\theta(\tilde{w})$ for all $\tilde{w}$ such that $\left\Vert \tilde{w} - 1 \right\Vert_2 \le \left\Vert w - w_1 \right\Vert_2$.  This is all that is needed of $\Omega_\theta$ in Theorem 1.

```python
def check_error_bound(theta_ij, theta_cv, ij_error_terms, plot=False):
    err_bound = ij_error_terms['err_bound']

    omega_sufficient = np.all(np.logical_and(
              theta_ij <= ij_error_terms['theta_max'],
              theta_ij >= ij_error_terms['theta_min']))
    print('Omega sufficient:\t', omega_sufficient)

    err = theta_cv - theta_ij
    bound_sufficient = np.all(np.abs(err) < err_bound)
    print('Error bound sufficient:\t', bound_sufficient)
    print('Max error bound ratio:\t', np.max(np.abs(err) / err_bound))

    if plot:
        plt.hist(err / err_bound);
        plt.title('Ratio of error to uniform bound');

    return bound_sufficient and omega_sufficient

assert check_error_bound(theta_ij, theta_cv, ij_error_terms, plot=True)
```

# Optimially choosing the size of $\Omega_\theta$.

As is clear from the above discussion and the definition of the function ``get_ij_error_bounds``, we have a map from the bounds $(\theta_{min}, \theta_{max}) \mapsto \epsilon$, where $|\hat\theta_{IJ}(w) - \hat\theta(w)| < \epsilon$.  Tigher bounds on $\theta$ lead to a smaller $\epsilon$.   If we find that the original choice of $\Omega_\theta$ is larger than necessary, we can reduce its size, potentially decreasing the error bound.

```python
# We can potentially decrease the size of \Omega_\theta.
def get_implied_omega_theta(theta_ij, ij_error_terms):
    err_bound = ij_error_terms['err_bound']
    pred_theta_max = np.max(theta_ij) + err_bound
    pred_theta_min = np.min(theta_ij) - err_bound
    return (pred_theta_min, pred_theta_max)
```

```python
bound_change = float('inf')
tol = 1e-4
pred_theta_min = theta_min
pred_theta_max = theta_max
print('Original Omega_theta:\t{}'.format((pred_theta_min, pred_theta_max)))
while bound_change > tol:
    print('\n')
    new_pred_theta_min, new_pred_theta_max = get_implied_omega_theta(theta_ij, ij_error_terms)
    print('Implied Omega_theta:\t{}'.format((pred_theta_min, pred_theta_max)))
    bound_change = np.abs(new_pred_theta_min - pred_theta_min) + np.abs(new_pred_theta_max - pred_theta_max)
    print('Bound change: {}'.format(bound_change))
    pred_theta_min = new_pred_theta_min
    pred_theta_max = new_pred_theta_max
    new_ij_error_terms = get_ij_error_bounds(pred_theta_min, pred_theta_max, x)

    print('Results for new bounds:')
    bounds_ok = check_error_bound(theta_ij, theta_cv, new_ij_error_terms)
    if not bounds_ok:
        print('################\nBounds not ok -- terminating.')
        bound_change = 0

```

    Original Omega_theta:	(-2.3234720841690373, -2.2834720841690372)

    Implied Omega_theta:	(-2.3234720841690373, -2.2834720841690372)
    Bound change: 0.03913138290025664
    Results for new bounds:
    Omega sufficient:	 True
    Error bound sufficient:	 True
    Max error bound ratio:	 0.19521106622236165

    Implied Omega_theta:	(-2.303574120115332, -2.3027055030155887)
    Bound change: 0.0
    Results for new bounds:
    Omega sufficient:	 True
    Error bound sufficient:	 True
    Max error bound ratio:	 0.19521106622236165


# A simple SymPy example

First we import SymPy and initialize printing:

```
from sympy import init_printing
from sympy import *
 init_printing()
```

Create a few symbols:

```
x,y,z = symbols('x y z')
```

Here is a basic expression:

```
e = x**2 + 2.0*y + sin(z); e
```

```
diff(e, x)
```

```
integrate(e, z)
```

```

```


# Functions of Random Variables

(same as: Change of Variables)

I find changes of variables to be the easiest to understand by writing down a joint probability distribution using a delta function to express the conditional probability for the new variables based on the old variables. The next step is to perform a change of variables in the delta function, and integrate. The delta function means that the relationship between the old and the new variables is deterministic. If the relationship is non-deterministic you can use whatever expression for the conditional probability is appropriate.

Let's say that you want to know the probability distribution of the function $\vec{u} = \mathbf{H}(\vec{x})$ of some random variable $\vec{x}$ for which the probability distribution is known.

I personally find it least confusing to approach this problem by thinking about the joint probability distribution, and then obtaining $p(\vec{u})$ through marginalization:

\begin{equation}
p(\vec{u}) = \int dx^n p(\vec{x},\vec{u}) = \int dx^n p_x(\vec{x}) p_u(\vec{u}|\vec{x})
\end{equation}

Since $\vec{u}$ is a deterministic function of $\vec{x}$, the conditional probabiltiy $p_u(\vec{u}|\vec{x}) = \delta(\vec{u} - \mathbf{H}(\vec{x}))$. Explicitly:

\begin{equation}
p(\vec{u}) = \int dx^n p_x(\vec{x}) \delta(\vec{u} - \mathbf{H}(\vec{x}))
\end{equation}

What this does, is to integrate over all the points $p_x(\vec{x})$ where the argument of the delta function is zero. The important thing is that care needs to be taken when the argument of the delta function is itself a function. In that case, a change of variables has to be performed, so that this is no longer the case. On wikipedia, this is done by defining the new variable $du = |\frac{d}{dx}g(x)|dx$, from which follows:

\begin{equation}
\delta(g(x)) = \sum_{x_0} \frac{\delta(x-x_0)}{|g'(x_0)|}
\end{equation}

Where it is necessary to sum over each point $x_0$ for which $g(x_0) = 0$.

Which makes sense. Except, if you rewrite in terms of the new variable, you get an expression that does not strike one as necessarily the same:

\begin{equation}
\int f(x) \delta(g(x))dx = \sum_{n}\int f(g_n^{-1}(u)) \delta(u) \left|\frac{d}{du}g^{-1}(u)\right|du
\end{equation}

Where the sum $n$ is over all functions $g_n^{-1}$ that satisfy $g(g_n^{-1}(u)) = u$. For example, if $u = g(x) = sin(x)$ then $g_n^{-1} = asin(u) + n2\pi$, for any integer $n$. The link is provided by the inverse function theorem. At any rate, I feel more comfortable with the second path.

Note that a delta function with a vector argument can be written as a product of the delta functions along each dimension.

Define a new set of variables $\vec{a} = \vec{u} - \mathbf{H}(\vec{x})$. It follows that $\vec{x} = \mathbf{H_n^{-1}}(\vec{u}-\vec{a})$ and the volume element $dx^n = \left| \frac{d}{d\vec{a}}\mathbf{H}_n^{-1}(\vec{u}-\vec{a})\right|da^n$. Here, $H^{-1}_n$ are all the functions that satisfy $\mathbf{H}(\mathbf{H}_n^{-1}(\vec{u})) = \vec{u}$. The integral becomes:

\begin{equation}
p(\vec{u}) = \int da^n p_x(\mathbf{H}^{-1}_n(\vec{u}-\vec{a}))\left| \frac{d}{d\vec{a}}\mathbf{H}_n^{-1}(\vec{u}-\vec{a})\right| \delta(\vec{a})
\end{equation}

At this point it is safe to evaluate the delta function integral. The result is:

\begin{equation}
p(\vec{u}) = p_x(\mathbf{H}^{-1}_n(\vec{u})\left| \frac{d}{d\vec{a}}\mathbf{H}_n^{-1}(\vec{u}-\vec{a})\right|_{(\vec{a}=0)}
\end{equation}

If the map $\mathbf{H}$ is not bijective, for example because $\vec{u}$ has lower dimensionality than $\vec{x}$, then a bijective map can be artifically constructed by introducing additional variables that are then also marginalized out. For example, if the goal is to calculate the probability of measuring some sum of random variables $s$, then you can define:

\begin{equation}
u_0 = s - \sum(x_i)\\
u_1 = x_1\\
u_2 = x_2\\
\vdots\\
u_{(n-1)} = x_{n-1}
\end{equation}

The inverse is:

\begin{equation}
x_1 = u_1\\
x_2 = u_2\\
\vdots\\
x_n = s - u_0 - \sum_{i<n} x_i
\end{equation}

The argument in the delta function is transformed $\delta(s - \sum(x_i)) \rightarrow \delta(u_0)$. The determinant of the Jacobian $|J| = |\frac{d}{d\vec{u}} H^{-1}(\vec{u})|$ is given by $[\frac{d}{du_1} \mathbf{H^{-1}},\frac{d}{du_2} \mathbf{H^{-1}},...,\frac{d}{du_{n-1}} \mathbf{H^{-1}}]$. That's an $nxn$ matrix where the first row is all $-1$, the lower left is an $(n-1)x(n-1)$ identity matrix, and the lower right is a $(n-1)x1$ vector of 0s. The determinant is one. Consequently the probability distribution of measuring a sum $s$ is given by:

\begin{equation}
p(s)= \int dx^{n-1} p_x(x_1,x_2,x_3,...,s-\sum_{x<n}x_i)
\end{equation}

Which turns out to be the convolution when the variables are independent.

## Experiment: Sum of Four Uniformly Distributed Independent Random Variables

I will do this in an inelegant and computationally inefficient way, because it is pedantic.

$p(x) = 1/2$ for $x \in (-1,1)$ and $p(x) = 0$ for $x\notin (-1,1)$.

And:

$p_s(s) = \int dx^{n-1} p(x_1)p(x_2)p(x_3)p(s-x_1-x_2-x_3)$

```python
import matplotlib.pyplot as plt
import numpy as np
from numpy.random import uniform

def p_x(x):
    """
    Probability density function for a single variable
    """
    if np.abs(x) < 1:
        return 1/2
    else:
        return 0

def convolution(f1,f2,s):
    """
    Convolution for dummies
    """

    D = 200
    x_values = np.linspace(-5,5,D)

    f1x_values = np.array([f1(x) for x in x_values])
    f2z_x_values = np.array([f2(s-x) for x in x_values])

    return np.sum(f1x_values*f2z_x_values)/D

def p_s(s):
    """
    Probability density function of the sum
    """
    s2 = lambda s: convolution(p_x,p_x,s)

    return convolution(s2,s2,s)
```

```python
N = 50

data = np.sum(np.vstack([uniform(-1,1,30000) for i in range(4)]),axis=0)
counts,bins, _ = plt.hist(data,density=True,bins=N)

p_s_values = np.array([p_s(s) for s in bins])
```

```python
plt.plot(bins,p_s_values/np.sum(p_s_values))
plt.plot(bins[:-1],counts/np.sum(counts))
plt.legend(['Analytical','Simulated'])
```

## Experiment: Sine and Cosine

Assume $x,y$ are sampled from a standard normal distribution with standard deviation $\pi$. Assume $u = cos(x)$ and $v = sin(x)$.

Then, the joint probability distribution is:

\begin{equation}
p(u,v) = \sum_{n,m} p( acos(u)+n2\pi)p(asin(v) + m2\pi) \frac{1}{|\sqrt{1-u^2}\sqrt{1-v^2}|}
\end{equation}

Or (following the other path with the delta functions)

\begin{equation}
p(u,v) = \sum_{n,m} p( acos(u)+n2\pi)p(asin(v) + m2\pi) \frac{1}{|sin(acos(u))cos(asin(v))|}
\end{equation}

Both are weird, but it turns out to be correct (at least, the shape is: admittedly I did not bother to check proper normalization).

Super non-inuitive -- this distribution is only intuitive when the standard veriation of $x,y << \pi$. After that, it becomes sharply peaked for $u,v \in -1,1$

```python
def p(x):
    std = 1*np.pi
    mu = 0
    return (1/np.sqrt(2*np.pi*std)) * np.exp(-0.5*(x-mu)**2/std**2)

def p_uv(u,v):

    N = 5
    M = 5

    p_ = 0*u
    for n in range(-N,N):
        for m in range(-M,M):
            p_ += p(np.arccos(u)+n*2*np.pi)*p(np.arcsin(v)+m*2*np.pi)

    a = 1/(np.sqrt(1-u**2))
    b = 1/(np.sqrt(1-v**2))
    #a = np.abs(1/np.sin(np.arccos(u)))
    #b = np.abs(1/np.cos(np.arcsin(v)))

    return a*b*p_

u_vals,v_vals = np.linspace(-0.99,0.99,100),np.linspace(-0.99,0.99,100)

U,V = np.meshgrid(u_vals,v_vals)

p_uv_vals = p_uv(U,V)
p_uv_vals = p_uv_vals #/np.sum(p_uv_vals)

plt.contourf(U,V,p_uv_vals)
plt.xlabel('u')
plt.ylabel('v')
plt.title('Analytical')
```

```python
from numpy.random import randn

std = 1*np.pi
x = 0+std*randn(10000)
y = 0+std*randn(10000)

u = np.cos(x)
v = np.sin(y)

counts, xbins, ybins, _ = plt.hist2d(u,v,bins=100)
plt.title('Simulated')
```

```python
plt.figure(figsize=(12,8))
_ = plt.hist(x,bins=20)
_ = plt.hist(np.sin(x),bins=20,alpha=1)
_ = plt.hist(np.cos(x),bins=20,alpha=0.75)

plt.legend(['X','Sin(X)','Cos(X)'],fontsize=12)
```

## Example: Lower-Dimensional Variable

This was already the case for the average of several random variables, in which case the dimensionality of the problem was reduced from many to one.

Consider the case many to fewer. Again, you just need to perform a change of variables. The new set of variables needs to have the same dimensionality as the old set of variables. The rest should follow pretty obviously.

Linear example: $\vec{y} = \mathbf{M}\vec{x}$ where $y$ has 2 dimensions and $x$ has 3.

Transform:

\begin{equation}
u_1 = y_1 - \vec{M_1}\cdot\vec{x}\\
u_2 = y_2 - \vec{M_2}\cdot\vec{x}\\
u_3 = x_3
\end{equation}

You can write this in terms of some invertible matrix $\mathbf{W} = [\mathbf{M},[0,0,1]]$ as $\vec{u} = \mathbf{W}\vec{x}$, so that $dx^n = |\mathbf{W}^{-1}|du^n|$. The integral is then:

\begin{equation}
p(y) = \int du^n p_x(\mathbf{W}^{-1}(\vec{y}-\vec{u}))\delta(u_1)\delta(u_2)|\mathbf{W}^{-1}|
\end{equation}

## Discrete Variables

In the case of discrete variables the pattern is the same. It's sums instead of integreals, and the change of variables is easier because there is no volume element involved.

Probability - Functions of Random Variables.ipynb


# Combien de numéros de mobiles français sont des nombres premiers ?

A question simple, réponse simple :

## Dépendances

```python
from sympy import isprime
```

```python
print(isprime.__doc__[:180])
```

        Test if n is a prime number (True) or not (False). For n < 2^64 the
        answer is definitive; larger n values have a small probability of actually
        being pseudoprimes.

## Réponse

```python
first_number = 6_00_00_00_00
last_number = 7_99_99_99_99

# test rapide
#last_number = first_number + 20

all_numbers = range(first_number, last_number + 1)
```

```python
def count_prime_numbers_in_range(some_range):
    count = 0
    for number in some_range:
        if isprime(number):
            count += 1
    return count
```

## Conclusion

```python
count = count_prime_numbers_in_range(all_numbers)

print(f"Pour des numéros de téléphones, nombres entre {first_number} et {last_number} (inclus), il y a {count} nombres premiers.")
```

    Pour des numéros de téléphones, nombres entre 600000000 et 799999999 (inclus), il y a 9821476 nombres premiers.

Et donc, on peut calculer la part de nombres premiers parmi les numéros de téléphones mobiles français.
De souvenir, c'était environ 5.1%, vérifions :

```python
total_number = len(all_numbers)

print(f"Pour des numéros de téléphones, nombres entre {first_number} et {last_number} (inclus), il y a {count/total_number:%} nombres premiers.")
```

    Pour des numéros de téléphones, nombres entre 600000000 et 799999999 (inclus), il y a 4.910738% nombres premiers.

Et voilà, c'était simple !