# Monty Hall with  ADG

**author: Alessio Benavoli**
 alessiobenavoli.com

This is Monty Hall problem.

Suppose you are on a game show, and you are given the choice of three doors:
1. Behind one door is a Ferrari;
2. behind the others, goats.

You pick a door, say DoorA, and the TV presenter, who knows what's behind the doors, opens another door, say No. DoorC, which has a goat. He then says to you, "Do you want to pick door No. DoorB?" Is it to your advantage to switch your choice?

To model this problem, we will use `ConditionalGambleList`

```python
%load_ext autoreload
%autoreload 2
from __future__ import absolute_import
from PyRational.models.ADG import  ADG
from PyRational.special.gambles import indicator, conditionalGambleList
import numpy as np
from sympy import symbols,  FiniteSet, Piecewise, Eq
```

*PyRational* requires numerical input variables.
We therefore encode door symbols (A,B,C) into integer data (1, 2, 3).

*PyRational* uses *Sympy* for symbolic mathematics. We need to define in *Sympy*  a `symbol` associated to the door you choose and one to the location of the Ferrari  and their relative domains {1,2, 3} (we use Sympy `FiniteSet` for the latter).

```python
door     = symbols('d');
ferrari  = symbols('f');
domain_door=FiniteSet(1,2,3)
domain_ferrari=FiniteSet(1,2,3)
```

We are now ready to build our first *PyRational* model: **ADG**.

```python
ListSymbols=[door,ferrari]
ListDomains=[domain_door,domain_ferrari]
model = ADG(ListSymbols,ListDomains)
model
```

ADG model

  List of Symbols 	[d, f]

  Domain 	Ω={1, 2, 3} x {1, 2, 3}

  List of desirable gambles 	G=[]

  Avoiding sure loss? 	  to be verified

We will use PyRational for rational reasoning.

First, we define a gambling system for helping you selecting a door. In this case, since you don't know anything, you should only accept nonnegative gambles, i.e., all gambles of the form:

g = a\*indicator('Eq(f,1)') + b\*indicator('Eq(f,2)') + c\*indicator('Eq(f,3)')

with $a,b,c\geq0$.

However, for the moment, we follow the standard modelling framework assuming that you are initially indifferent.
You accept all gambles such that

$$
a+b+c=0
$$

like these ones.

```python
g1= -1*indicator('Eq(f,1)') -1*indicator('Eq(f,2)') + 2*indicator('Eq(f,3)')
g2= -1*indicator('Eq(f,1)') +2*indicator('Eq(f,2)') - 1*indicator('Eq(f,3)')
g3=  2*indicator('Eq(f,1)') -1*indicator('Eq(f,2)') - 1*indicator('Eq(f,3)')
cg0=[g1,g2,g3]
```

What are the game's rules?

Assume Ferrari is behind DoorB, then the TV presenter can only open DoorC. Hence, in this case, if you were asked to bet on which door the presenter will open, then you should bet on all gambles that are nonnegative on d==3, e.g.,

```python
g11 = -1*indicator('Eq(d,1)') -1*indicator('Eq(d,2)') + 1*indicator('Eq(d,3)')
g21 = -1*indicator('Eq(d,1)') +1*indicator('Eq(d,2)') + 1*indicator('Eq(d,3)')
g31 = +1*indicator('Eq(d,1)') -1*indicator('Eq(d,2)') + 1*indicator('Eq(d,3)')
g41 = -1*indicator('Eq(d,1)') -1*indicator('Eq(d,2)') + 0*indicator('Eq(d,3)')
g51 = -1*indicator('Eq(d,1)') +1*indicator('Eq(d,2)') + 0*indicator('Eq(d,3)')
g61 = +1*indicator('Eq(d,1)') -1*indicator('Eq(d,2)') + 0*indicator('Eq(d,3)')

cg1=conditionalGambleList([g11,g21,g31,g41,g51,g61],'Eq(f,2)')
```

The last statement means that your are willing to accept all the above gambles conditional on $f$ being behind $2$.

Assume Ferrari is behind DoorC, then the TV presenter can only open DoorB. Hence, in this case, if you were asked to bet on which door the presenter will open, then you should bet on all gambles that are nonnegative on d==2, e.g.,

```python
g12 = -1*indicator('Eq(d,1)') -1*indicator('Eq(d,3)') + 1*indicator('Eq(d,2)')
g22 = -1*indicator('Eq(d,1)') +1*indicator('Eq(d,3)') + 1*indicator('Eq(d,2)')
g32 = +1*indicator('Eq(d,1)') -1*indicator('Eq(d,3)') + 1*indicator('Eq(d,2)')
g42 = -1*indicator('Eq(d,1)') -1*indicator('Eq(d,3)') + 0*indicator('Eq(d,2)')
g52 = -1*indicator('Eq(d,1)') +1*indicator('Eq(d,3)') + 0*indicator('Eq(d,2)')
g62 = +1*indicator('Eq(d,1)') -1*indicator('Eq(d,3)') + 0*indicator('Eq(d,2)')

cg2=conditionalGambleList([g12,g22,g32,g42,g52,g62],'Eq(f,3)')
```

Finally, in case Ferrari is behind DoorA, then the TV presenter can open any door between B and C. We assume again indifference between door 2 and door 3.

```python
g13 = -1*indicator('Eq(d,1)') + 1*indicator('Eq(d,2)') - 1*indicator('Eq(d,3)')
g23 = -1*indicator('Eq(d,1)') - 1*indicator('Eq(d,2)') + 1*indicator('Eq(d,3)')
g33 = +1*indicator('Eq(d,1)') - 1*indicator('Eq(d,2)') + 1*indicator('Eq(d,3)')
g43 = +1*indicator('Eq(d,1)') + 1*indicator('Eq(d,2)') - 1*indicator('Eq(d,3)')
g53 = -1*indicator('Eq(d,1)')

cg3=conditionalGambleList([g13,g23,g33,g43,g53],'Eq(f,1)')
```

```python
cg=cg0.copy()
cg.extend(cg1)
cg.extend(cg2)
cg.extend(cg3)
model = ADG(ListSymbols,ListDomains)
cg
```

    [-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)),
     -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)),
     2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)),
     (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)),
     -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True))]

We add all these gambles to `model` as follows:

```python
model.add_gambleList(cg)
model
```

ADG model

  List of Symbols 	[d, f]

  Domain 	Ω={1, 2, 3} x {1, 2, 3}

  List of desirable gambles 	G=[-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)), -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), 2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True))]

  Avoiding sure loss? 	  to be verified

Note that $G$ is a list that includes all Alice's desirable gambles. We now `build` the belief model.

```python
model.buildModel()
model
```

ADG model

  List of Symbols 	[d, f]

  Domain 	Ω={1, 2, 3} x {1, 2, 3}

  List of desirable gambles 	G=posi(ℒ(Ω)+      ∪    [-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)), -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), 2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True))])

  Avoiding sure loss? 	  to be verified

```python
optimoptions={ 'method_LISP': 'Cutting_plane',
               'SolverLP':'linprog',
               'LP_acc_constraints':1e-300,
               'SolverNLP':'differential_evolution',
               'NLP_alpha_cut':-0.009,
               'num_support_points': 10,
               'verbose':False}

model.check_avs(options=optimoptions)
model
```

    Belief Model avoids sure  loss

ADG model

  List of Symbols 	[d, f]

  Domain 	Ω={1, 2, 3} x {1, 2, 3}

  List of desirable gambles 	G=posi(ℒ(Ω)+      ∪    [-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)), -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), 2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)), -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True))])

  Avoiding sure loss? 	 Yes

So Alice is **rational** or, equivalently, her set of desirable gambles is coherent.

```python
f= Piecewise((1.0,Eq(ferrari,2)),(0.0,True))
f_range=(None,None)
lp=model.lower_prevision(f,f_range,options=optimoptions)
up=model.upper_prevision(f,f_range,options=optimoptions)
print(lp,up)
```

    0.3333333333333333 0.3333333333333334

## Updating: inference

Alice may want to evaluate how her inferences would change if she knew that the result of the coin toss will be Head.

This operation is called `updating` in ADG and is formally equivalent to probabilistic conditioning. It is performed by firstly defining an indicator on the event of interest:

```python
h = Piecewise((1.0,Eq(door,3)),(0.0,True))
```

and then re-computing `lower_prevision` and `upper_prevision` conditionally on this event.

```python
f= Piecewise((1.0,Eq(ferrari,2)),(0.0,True))-Piecewise((1.0,Eq(ferrari,1)),(0.0,True))
f_range=(None,None)
lp=model.lower_prevision(f,f_range,h=h,options=optimoptions)
print(lp)
```

    0.33333333333333337

Note that, since she knows that the result is Heads, then she is sure that the payoff for $f$ is one.

## Updating: model

Alice may also want to update her whole belief model conditionally on $h$. This can be done as follows:

```python
NewG = model.updating(h,options=optimoptions)
NewG
```

    [-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)) + 1.0,
     -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)) - 1.0,
     2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)) + 2.77555756156289e-17,
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666,
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666,
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666,
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300,
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300,
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300,
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)) - 2.0e-300,
     (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)) - 2.0e-300,
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)),
     (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) + 0.333333333333333,
     (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) - 0.333333333333333,
     (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) - 0.333333333333333,
     (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) + 0.333333333333333,
     -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True)) - 2.0e-300]

It returns the updated set of desirable gambles. We can use it to build a new belief model.

```python
model1 = ADG(ListSymbols,ListDomains)
model1.add_gambleList(NewG)
model1.buildModel()
model1.check_avs()
model1
```

    Warning: gamble 6 is always negative in T
    Warning: gamble 19 is always negative in T
    Belief Model avoids sure  loss

ADG model

  List of Symbols 	[d, f]

  Domain 	Ω={1, 2, 3} x {1, 2, 3}

  List of desirable gambles 	G=posi(ℒ(Ω)+      ∪    [-Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) + 2*Piecewise((1, Eq(f, 3)), (0, True)) + 1.0, -Piecewise((1, Eq(f, 1)), (0, True)) + 2*Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)) - 1.0, 2*Piecewise((1, Eq(f, 1)), (0, True)) - Piecewise((1, Eq(f, 2)), (0, True)) - Piecewise((1, Eq(f, 3)), (0, True)) + 2.77555756156289e-17, (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666, (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666, (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 0.666666666666666, (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300, (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300, (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)))*Piecewise((1, Eq(f, 2)), (0, True)) - 2.0e-300, (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)) - 2.0e-300, (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)) - 2.0e-300, (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 3)), (0, True)), (-Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) + 0.333333333333333, (-Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) - 0.333333333333333, (Piecewise((1, Eq(d, 1)), (0, True)) - Piecewise((1, Eq(d, 2)), (0, True)) + Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) - 0.333333333333333, (Piecewise((1, Eq(d, 1)), (0, True)) + Piecewise((1, Eq(d, 2)), (0, True)) - Piecewise((1, Eq(d, 3)), (0, True)))*Piecewise((1, Eq(f, 1)), (0, True)) + 0.333333333333333, -Piecewise((1, Eq(d, 1)), (0, True))*Piecewise((1, Eq(f, 1)), (0, True)) - 2.0e-300])

  Avoiding sure loss? 	 Yes

```python
model1.lower_prevision(f,f_range,options=optimoptions)
```

    Warning: gamble 6 is always negative in T
    Warning: gamble 19 is always negative in T

    0.33333333333333315

This is inference is equivalent to `model1.lower_prevision(f,f_range,h=h,options=optimoptions)`


```python
from IPython.display import Image
import sympy as sp
import math
import numpy as np
import datetime
```

```python
Image(filename='/Users/wy/Desktop/beales_function.png')
```

```python
class GoldSearch(object):

    def  __init__(self):
        self.l = 10**-5
        self.alpha = (math.sqrt(5)-1)/2.

    def g_lambda(self, a, b):
        return a+(1-self.alpha)*(b-a)

    def g_mu(self, a, b):
        return a+self.alpha*(b-a)

    def goldSearch(self, a, b,lambda_k,mu_k,function,k = 1):
        # step1
        if (b - a) < self.l:
            return (a+b)/2.

        if function(lambda_k) > function(mu_k):
            # step2
            a = lambda_k
            b = b
            lambda_k = mu_k
            mu_k = self.g_mu(a,b)
            k = k+1
            return self.goldSearch(a,b,lambda_k,mu_k,function,k)

        elif function(lambda_k) <= function(mu_k):
            # step3
            a = a
            b = mu_k
            mu_k = lambda_k
            lambda_k = self.g_lambda(a,b)
            k = k+1
            return self.goldSearch(a,b,lambda_k,mu_k,function,k)
GoldSearch = GoldSearch()
```

```python
def gradient(f):
    return [sp.lambdify((x1,x2), f.diff(x, 1), 'numpy') for x in [x1,x2]]
```

# Fletcher_Reeves
初始點 (1,1)
GoldSearch interval -5 ~ 5
e = 10**-5
number of iterations : 24
run time : 0.91s

```python
def Fletcher_Reeves(f,xj):
    lambda_j = sp.symbols('lambda_j')
    e = 10**-5
    sj = np.array(map(lambda fun : fun( xj[0],xj[1] ),gradient(f)))*(-1)
    i = 1
    while np.linalg.norm(sj) > e:
        i = i+1
        tmp = xj+lambda_j*sj
        new_f = f.subs([(x1,tmp[0]),(x2,tmp[1])])
        lambdaJ = GoldSearch.goldSearch(a,b,GoldSearch.g_lambda(a,b),GoldSearch.g_mu(a,b),sp.lambdify(lambda_j , new_f))
        xj_1 = xj+lambdaJ*sj
        sj_1 = np.array(map(lambda fun : fun( xj_1[0],xj_1[1] ),gradient(f)))*(-1)
        beta_j = np.dot(sj_1.T,sj_1)/np.dot(sj.T,sj)
        sj_1 = sj_1+beta_j*sj
        sj = sj_1
        xj = xj_1
    return xj_1,i
```

```python
a = -5
b = 5
x1,x2 = sp.symbols('x1,x2')
f = (1.5-x1*(1-x2))**2 + (2.25-x1*(1-x2**2))**2 + (2.625-x1*(1-x2**3))**2
# 初始點
xj = np.array([1,1])
start = datetime.datetime.now()
xj_1,i = Fletcher_Reeves(f,xj)
end = datetime.datetime.now()
print xj_1
print i
print end - start
```

    [ 3.00000314  0.50000078]
    24
    0:00:00.916073

# DFP
初始點 (1,1)
GoldSearch interval -5 ~ 5
e = 10**-5
number of iterations : 8
run time : 0.34s

```python
def DFP(f,xi):
    lambda_i = sp.symbols('lambda_i')
    e = 10**-3
    gradient_f = (np.array(map(lambda fun : fun( xi[0],xi[1] ),gradient(f)))).reshape(2,1)
    Bi = np.identity(2)
    i = 0
    while abs(np.linalg.norm(gradient_f)) > e:
        i = i+1
        si = (np.dot(Bi,gradient_f)*(-1)).reshape(1,2)[0]
        tmp = xi+lambda_i*si
        new_f = f.subs([(x1,tmp[0]),(x2,tmp[1])])
        lambdaI = GoldSearch.goldSearch(a,b,GoldSearch.g_lambda(a,b),GoldSearch.g_mu(a,b),sp.lambdify(lambda_i , new_f))
        xi_1 = xi+lambdaI*si
        gradient_f_1 = (np.array(map(lambda fun : fun( xi_1[0],xi_1[1] ),gradient(f)))).reshape(2,1)
        if abs(np.linalg.norm(gradient_f_1)) > e:
            gi = (gradient_f_1 - gradient_f).reshape(1,2)[0]
            Mi = (np.dot(si.reshape(2,1),si.reshape(2,1).T))*lambdaI/np.dot(si.T,gi)
            Ni = np.dot(np.dot(Bi,gi).reshape(2,1),np.dot(Bi,gi).T.reshape(1,2))*(-1)/np.dot(np.dot(gi.T,Bi),gi)
            Bi = Bi+Mi+Ni
            xi = xi_1
            gradient_f = (np.array(map(lambda fun : fun( xi[0],xi[1] ),gradient(f)))).reshape(2,1)
        else:
            return xi_1,i
```

```python
a = -5
b = 5
x1,x2 = sp.symbols('x1,x2')
f = (1.5-x1*(1-x2))**2 + (2.25-x1*(1-x2**2))**2 + (2.625-x1*(1-x2**3))**2
xi = np.array([1,1])

start = datetime.datetime.now()
xi_1,i = DFP(f,xi)
end = datetime.datetime.now()
print xi_1
print i
print end - start
```

    [ 3.00002223  0.49998904]
    8
    0:00:00.343704

# BFGS
初始點 (1,1)
GoldSearch interval -5 ~ 5
e = 10**-5
number of iterations : 8
run time : 0.38s

```python
def BFGS(f,xi):
    lambda_i = sp.symbols('lambda_i')
    e = 10**-3
    gradient_f = (np.array(map(lambda fun : fun( xi[0],xi[1] ),gradient(f)))).reshape(2,1)
    Bi = np.identity(2)
    i = 0
    while abs(np.linalg.norm(gradient_f)) > e:
        i = i+1
        si = (np.dot(Bi,gradient_f)*(-1)).reshape(1,2)[0]
        tmp = xi+lambda_i*si
        new_f = f.subs([(x1,tmp[0]),(x2,tmp[1])])
        lambdaI = GoldSearch.goldSearch(a,b,GoldSearch.g_lambda(a,b),GoldSearch.g_mu(a,b),sp.lambdify(lambda_i , new_f))
        xi_1 = xi+lambdaI*si
        gradient_f_1 = (np.array(map(lambda fun : fun( xi_1[0],xi_1[1] ),gradient(f)))).reshape(2,1)
        if abs(np.linalg.norm(gradient_f_1)) > e:
            gi = (gradient_f_1 - gradient_f).reshape(1,2)[0]
            di = xi_1-xi
            Mi = ((1 + np.dot(np.dot(gi.T,Bi),gi)/np.dot(di.T,gi))*np.dot(di.reshape(2,1),di.reshape(1,2)))/np.dot(di.T,gi)
            Ni = np.dot(np.dot(di.reshape(2,1),gi.reshape(1,2)),Bi)*(-1)/np.dot(di.T,gi)
            Qi = np.dot(np.dot(Bi,gi).reshape(2,1),di.reshape(1,2))*(-1)/np.dot(di.T,gi)
            Bi = Bi+Mi+Ni+Qi
            xi = xi_1
            gradient_f = (np.array(map(lambda fun : fun( xi[0],xi[1] ),gradient(f)))).reshape(2,1)
        else:
            return xi_1,i
```

```python
a = -5
b = 5
x1,x2 = sp.symbols('x1,x2')
f = (1.5-x1*(1-x2))**2 + (2.25-x1*(1-x2**2))**2 + (2.625-x1*(1-x2**3))**2
xi = np.array([1,1])

start = datetime.datetime.now()
xi_1,i = BFGS(f,xi)
end = datetime.datetime.now()
print xi_1
print i
print end - start
```

    [ 3.00002222  0.49998909]
    8
    0:00:00.389444

```python
from scipy.optimize import fmin

def fun(X):
    return (1.5-X[0]*(1-X[1]))**2 + (2.25-X[0]*(1-X[1]**2))**2 + (2.625-X[0]*(1-X[1]**3))**2

fmin(fun,np.array([1,1]))
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 56
             Function evaluations: 107

    array([ 3.00002489,  0.50000749])

# scipy python做科學計算的lib
出處 ： http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html#scipy.optimize.fmin
Minimize a function using the downhill simplex algorithm.
This algorithm only uses function values, not derivatives or second derivatives.


# Probability Theory

*The theory of probability consists in using human intuition to study the phenomena of our daily work. For this, we will use the basic principle of human learning, which is the idea of experiment.*

*We can classify experiments into two types: random (casual) and non-random (deterministic). The deterministic experiments are totally characterized a priori, that is, they are phenomena in which the result is known even before it occurs and in this way, we have nothing to do.*

*The experiments that we will study are the random ones, of which we do not know the result a priori, that is, they are events whose results cannot be predicted.*

*Most of this material comes from the course '[Introduction to Probability Theory](https://uspdigital.usp.br/apolo/apoObterCurso?cod_curso=550400491&cod_edicao=20001&numseqofeedi=1)' given by [Francisco A. Rodrigues](https://github.com/franciscoicmc) at ICMC - USP, and the online book '[Introduction to probability, statistics, and random processes](https://www.probabilitycourse.com/)' by H. Pishro-Nik.*

---

## Table of contents

1. [Introduction](#intro)

  1.1 [Fundamental concepts](#fundamental)

    * [Inclusion-Exclusion Principle](#inclusion)

    * [More properties](#proper)

  1.2 [Coin tossing simulation](#coin)

  1.3 [Urn problem simulation](#urn)

2. [Conditional probability and independence](#condprob)

  2.1 [Conditional probability](#cond)

  2.2 [Independence](#indep)

  2.3 [Conditional probability simulation](#condsim)

  2.4 [Law of Total Probability](#lawtotal)

  2.5 [Bayes Rule](#bayes)

  2.6 [Bayes Rule Simulation](#bayessim)

  2.7 [Base rate fallacy](#fallacy)

3. [Random variables](#rand)

  3.1 [Discrete random variables](#disc)

  3.2 [Continuous random variables](#cont)

  3.3 [Random variable simulation](#randsim)

4. [Expected value and variance](#expvar)

  4.1 [Expected value](#exp)

    * [Properties](#expproper)

    * [Expected value simulation](#expsim)

  4.2 [Variance](#variance)

    * [Variance example](#varianceex)

5. [Discrete distributions](#discretedist)

  5.1 [Uniform model](#discuniform)

  5.2 [Bernoulli model](#discbernoulli)

  5.3 [Binomial model](#discbinomial)

  5.4 [Poisson distribution](#discpoisson)

  5.5 [Geometric distribution](#discgeometric)

  5.6 [Negative Binomial distribution](#discnegative)

  5.7 [Hypergeometric distribution](#dischyper)

6. [Continuous distributions](#contdist)

  6.1 [Normal distribution](#contnormal)

  6.2 [Exponential distribution](#contexp)

  6.3 [Uniform distribution](#contuniform)

  6.4 [Gamma distribution](#contgamma)

  6.5 [Beta distribution](#contbeta)

  6.6 [Weibull distribution](#contweibull)

  6.7 [Student's t-distribution](#contstudent)

  6.8 [Chi-square distribution](#contchisquare)

---

## Introduction

### Fundamental concepts

- Outcome: A result of a random experiment;

- Sample space ($\Omega$): The set of all possible outcomes;

- Event ($A$): A subset of the sample space.

Relation between events:

- Impossible event: $\varnothing$;

- Possible event: $\Omega$;

- $A \cup B$: Event that occurs if $A$ or $B$ (or both) occur;

- $A \cap B$: Event that occurs if and only if A and B occur;

- $\bar{A}$: Event that occurs if A doesn't occur;

- Mutual exclusive event: $A \cap B = \varnothing$.

#### Inclusion-Exclusion Principle

$P(A \cup B )= P(A)+P(B)-P(A \cap B)$

We can prove by using:

$A \cup B = (A \cap \bar{B}) \cup (A \cap B) \cup (\bar{A}\cap B) \Rightarrow$

$P(A \cup B) = P(A \cap \bar{B}) + P(A\cap B) + P(\bar{A} \cap B) \Leftrightarrow$

$P(A \cup B) = P(A) - P(A\cap B) + P(A\cap B) + P(B) - P(A\cap B) = P(A) + P(B) - P(A\cap B)$

#### More properties

- $P(\varnothing) = 0$;
- $P(A) = 1 - P(\bar{A})$;
- If $A \in B$, therefore $P(A) \leq P(B)$.

### Coin tossing simulation

With these concepts, let's simulate coin tossing:

```python
import numpy as np

p = 0.7 # Probability for heads

nsim = 1000 # Number of simulations

nhead = 0 # Initializing number of heads

outcomes = [] # Store the outcomes Heads: 1, Tails: 0

for i in range(0, nsim):
    if(np.random.uniform() < p): # If less than p, we have heads
        nhead = nhead + 1 # Add counting for heads
        outcomes.append(1)
    else:
        outcomes.append(0)
```

The outcomes generated:

```python
print("Outcomes:", outcomes)
```

    Outcomes: [0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

Of course, in this case, the sample space is made of the unique values from the outcomes, Heads or Tails. $\Omega = \{1,\;0\}$.

```python
np.unique(outcomes)
```

    array([0, 1])

Using [np.random.uniform](https://numpy.org/doc/stable/reference/random/generated/numpy.random.uniform.html) we use probability density function of the uniform distribution which is:

$$p(x) = \frac{1}{b - a}$$

Therefore, any value in the interval has the same chance of being chosen (more explications about probability density will be in the notebook). So, in theory, with many simulations, we will get a frequency very similar to the given probability:

```python
print("Heads frequency:", nhead/nsim)
```

    Heads frequency: 0.69

This go hand in hand with the frequentist probability definition, which says the probability of an event is equal to its frequency occurrence in many experiments:

$$P(A) = \lim_{n \rightarrow \infty} \frac{n_{A}}{n}$$

Let's see the coin tossing simulation with this idea of frequency visually:

```python
import matplotlib.pyplot as plt

p = 0.7 # Probability for heads
vp = [] # Occurence frequency
vsim = [] # Simulation number
Nmax = 1000 # Max number of simulations

for nsim in np.arange(1, Nmax):
    nhead = 0 # Number for heads
    for i in range(1,nsim):
        if(np.random.uniform() < p):
            nhead = nhead + 1
    vp.append(nhead/nsim)
    vsim.append(nsim)

# Show simulation results graphically
plt.figure(figsize=(8, 6))
plt.plot(vsim, vp, linestyle='-', color="blue", linewidth=2,label = 'Simulated value')
plt.axhline(y=p, color='r', linestyle='--', label = 'Theoretical value')
plt.ylabel("Heads frequency", fontsize=20)
plt.xlabel("Number of simulations", fontsize=20)
plt.xlim([0.0, Nmax])
plt.ylim([0.0, 1.0])
plt.legend()
plt.show(True)
```

We can see that the higher the number of simulations, the closer to the theoretical value we get.

### Urn problem simulation

If we have an urn with 15 white balls and 3 black balls the chance of getting black ball will be:

$$P(A) = \frac{|A|}{|\Omega|} = \frac{n(A)}{n(\Omega)} = \frac{3}{18} \cong 0.1667$$

Let's simulate this experiment.

```python
import random

# 0: White ball, 1: Black ball
urn = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]

N = 100000

# Frequency for black
fBlack = 0

for i in range(0, N):
    chosenBall = random.choice(urn)
    if(chosenBall == 1):
        fBlack = fBlack + 1
fBlack = fBlack/N

print('Frequency for black:', fa)
```

    Frequency for black: 0.16799

We use the library [random](https://docs.python.org/3/library/random.html) for choosing randomly a ball from the urn. As the coin tossing example, the higher the number of simulations, the closer to the real thing we get.

## Conditional probability and independence

### Conditional probability

Let it be two events $A$ and $B$ in the same sample space $\Omega$. The conditional probability of $A$ given $B$ occurring is $P(A|B)$ and its definition:

$$P(A|B)=\frac{|A \cap B|}{|B|}=\frac{\frac{|A \cap B|}{|S|}}{\frac{|B|}{|S|}}=\frac{P(A \cap B)}{P(B)}\text{, when}\; P(B) > 0$$

The conditional probability is a way of updating the probabilities of events.

### Independence

$A$ and $B$ are independent events if and only if,

$$P(A \cap B) = P(A)P(B)$$

So, $B$ occuring doesn't affect probability for A:

$$P(A|B) = \frac{P(A\cap B)}{P(B)} = P(A)$$

### Conditional probability simulation

Let's use Pandas library to work with this concept (using this [code](https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python)). We will make a dataset for cars:

```python
import pandas as pd

df = pd.DataFrame({'company': ['ford', 'chevy', 'ford', 'ford', 'ford', 'toyota'],
                     'model': ['mustang', 'camaro', 'fiesta', 'focus', 'taurus', 'camry'],
                     'rating': ['A', 'B', 'C', 'A', 'B', 'B'],
                     'type': ['coupe', 'coupe', 'sedan', 'sedan', 'sedan', 'sedan']})

df
```

      company
      model
      rating
      type

      0
      ford
      mustang
      A
      coupe

      1
      chevy
      camaro
      B
      coupe

      2
      ford
      fiesta
      C
      sedan

      3
      ford
      focus
      A
      sedan

      4
      ford
      taurus
      B
      sedan

      5
      toyota
      camry
      B
      sedan

The probability of finding a car with a random rating:

```python
df.groupby('rating').count()['model'] / len(df)
```

    rating
    A    0.333333
    B    0.500000
    C    0.166667
    Name: model, dtype: float64

But we can group the dataframe wanting to find a car type given a specific rating, so

$$P(\text{type}|\text{rating})$$

Making this probability table:

```python
(df.groupby(['type', 'rating']).count() / df.groupby('rating').count())['model']
```

    type   rating
    coupe  A         0.500000
           B         0.333333
    sedan  A         0.500000
           B         0.666667
           C         1.000000
    Name: model, dtype: float64

For example, we have:

$$P(\text{type = sedan}|\text{rating = B}) = \frac{P(\text{type = sedan} \cap \text{rating = B})}{P(\text{rating = B})} = \frac{\frac{2}{6}}{\frac{3}{6}} = \frac{2}{3} \cong 0.6667$$

### Law of Total Probability

If $B_1$, $B_2$, $B_3$, $\cdots$ is a partition of the sample space S, then for any event A we have

$$P(A)=\sum_{i} P(A \cap B_i)=\sum_{i} P(A | B_i) P(B_i)$$

So, with this image:

We have

$$P(A) = P(A | B_1) P(B_1) + P(A | B_2) P(B_2) + P(A | B_3) P(B_3) + P(A | B_4) P(B_4) + P(A | B_5) P(B_5) + P(A | B_6) P(B_6)$$

### Bayes Rule

Bayes' Rule describes the probability of an event, based on prior knowledge of conditions that might be related to the event, in other words, Bayes theorem is what allows us to go from a sampling (or likelihood) distribution and a prior distribution to a posterior distribution.

For any two event $A$ and $B$, where $P(A) \neq 0$, we have

$$P(B|A)=\frac{P(A|B)P(B)}{P(A)}$$

if $B_1$, $B_2$, $B_3$, $\cdots$ form a partition of the sample space S, then for any event A with $P(A) \neq 0$ we have

$$\underbrace{P(B_i|A)}_{\text{Posterior}}=\frac{\overbrace{P(A|B_i)}^{\text{Likelihood}}\overbrace{P(B_i)}^{\text{Prior}}}{\underbrace{\displaystyle\sum_{j = 1}^{n} P(A | B_j) P(B_j)}_{\text{Evidence}}}$$

We can see $P(A)$ can be changed to the idea of Law of Total Probability.

### Bayes Rule Simulation

Let's solve problems with this Python function using the theorem:

```python
# calculate P(A|B)
def bayes(p_a, p_b_given_a, p_b_given_not_a):
	# calculate P(not A)
	not_a = 1 - p_a
	# calculate P(B), the evidence
	p_b = p_b_given_a * p_a + p_b_given_not_a * not_a
	# calculate P(A|B)
	p_a_given_b = (p_b_given_a * p_a) / p_b

	return p_a_given_b
```

*The problem:* A suspect has been arrested and the police are 60% sure that he is the wanted criminal. New evidence says that the wanted criminal has tattoos. If 20% of the population has tattoos, what will the police level of trust be with this new information?

```python
# prior P(A), base rate
p_a = 0.6
# likelihood P(B|A), also called sensitivity
p_b_given_a = 1
# P(B|not A)
p_b_given_not_a = 0.2
# calculate P(A|B)

print("P(A|B) = {:2f}".format(bayes(p_a, p_b_given_a, p_b_given_not_a)))
```

    P(A|B) = 0.882353

Let's see a 'trickier' problem,

*[The problem](https://www.probabilitycourse.com/chapter1/1_4_3_bayes_rule.php):* A certain disease affects about 1 out of 10,000 people. There is a test to check whether the person has the disease. The test is quite accurate. In particular, we know that

- the probability that the test result is positive (suggesting the person has the disease), given that the person does not have the disease, is only 2 percent;

- the probability that the test result is negative (suggesting the person does not have the disease), given that the person has the disease, is only 1 percent.

A random person gets tested for the disease and the result comes back positive. What is the probability that the person has the disease?

```python
# prior P(A), base rate
p_a = 1/10000
# likelihood P(B|A), also called sensitivity
p_b_given_a = 1 - 0.01
# P(B|not A)
p_b_given_not_a = 0.02
# calculate P(A|B)

print("P(A|B) = {:2f}".format(bayes(p_a, p_b_given_a, p_b_given_not_a)))
```

    P(A|B) = 0.004926

So if the result comes positive, it means that there is a very low chance that the person has the disease.

### Base rate fallacy

Also called [base rate bias](https://en.wikipedia.org/wiki/Base_rate_fallacy), the best way to demonstrate this fallacy is when we have the false positive paradox. This paradox describes situations where there are more false positive test results than true positives. For example, 50 of 1,000 people test positive for an infection, but only 10 have the infection, meaning 40 tests were false positives.

With the previous exercise we had the same situation. This is one of the motives to inspect classifier models with cautious. The famous ROC curve has the exact [same problem](http://keeganhin.es/blog/base_rate.html) to deal. Bayes' rule is a great way to understand it.

Also, other names using the idea of binary classifier terminology:

- $P(A)$: Base rate;
- $P(B|A)$: True Positive Rate (TPR), Sensitivity;
- $P(\bar{B}|\bar{A})$: True Negative Rate (TNR), Specificity;
- $P(B|\bar{A})$: False Positive Rate (FPR);
- $P(\bar{B}|A)$: False Negative Rate (FNR).

Also we have that:

- $P(B|A) + P(\bar{B}|A) = 1$
- $P(\bar{B}|\bar{A}) + P(B|\bar{A}) = 1$

A [Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) is a good way to visualize these relations.

## Random variables

A random variable is a function that maps the sample space in the real line, where each element from the sample space is mapped in a real value.

If we flip two coins, we will have the following sample space, where Heads: H, Tails: T

$$\Omega = \{HH, HT, TH, TT\}$$

A possible random variable associated to the experiment is defined by:

$$X: \text{'Number of heads'}$$

### Discrete random variables

Let $X$ be a random variable. If the number of possible values of $X$ is finite or numerable infinite, we define $X$ as a discrete random variable.

$$X\; \text{is a discrete random variable, if its range is countable.}$$

The probability distribution for a fair dice:

| $X$ | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|----|
| $P(X = x)$  | $\frac{1}{6}$  | $\frac{1}{6}$  | $\frac{1}{6}$  |  $\frac{1}{6}$ | $\frac{1}{6}$  |  $\frac{1}{6}$  |

Being $X$ a discrete random variable.

We can plot this distribution:

```python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5, 6]
prob = [1/6, 1/6, 1/6, 1/6, 1/6, 1/6]

df = pd.DataFrame({'Value': x,'Probability': prob})

df.plot.bar(x = 'Value', y = 'Probability', rot = 0, title = "Probability for dice values")

fig = plt.gcf()
fig.set_size_inches(8, 4)

plt.show(block=True)
```

### Continuous random variables

On the other side we have continuous random variables, which has a range in the form of an interval or a union of non-overlapping intervals on the real line. Also, for any $x \in \mathbb{R},\;P(X = x) = 0$.

Let $X$ be a continuous random variable with following probability density function:

$ f(x) = \left\{ \begin{array}{cl}
2x & : \ 0 < x < 1 \\
0 & : x \leq 0 \vee x \geq 1
\end{array} \right.$

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Polygon

def func(x):
    return 2 * x

a, b = 0.5, 1  # integral limits
x = np.linspace(0, 1)
y = func(x)

fig, ax = plt.subplots()
ax.plot(x, y, 'r', linewidth=2)
ax.set_ylim(bottom=0)
ax.set_xlim(0)

# Make the shaded region
ix = np.linspace(a, b)
iy = func(ix)
verts = [(a, 0), *zip(ix, iy), (b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)

ax.text(0.5 * (a + b), 0.5, r"$\int_a^b f(x)\mathrm{d}x = P(X \geq a)$", horizontalalignment='center', fontsize=14)

fig.text(0.9, 0.05, '$x$')
fig.text(0.1, 0.9, '$y$')

ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.xaxis.set_ticks_position('bottom')

ax.set_xticks((a, b))
ax.set_xticklabels(('$a$', '$b$'))
ax.set_yticks([])

plt.show()
```

So, for $P(X \geq \frac{1}{2})$ we have:

```python
import sympy as sp

x = sp.Symbol('x')
y = 2*x
sp.integrate(y,(x, 1/2, 1))
```

$\displaystyle 0.75$

### Random variable simulation

*Problem:* A player pays 5 dollars to participate in a dice game, disputing with the opponent who has the highest point. The player and the opponent each throw their dice and the following award rule is established:

(i) If the player's point is higher, he wins 2 times the difference between his point and the one obtained by his opponent.

(ii) If the player's point is less than or equal to that of the opponent, he wins nothing.

Is this game favorable to the player?

----

We can solve it simulating many games to find the frequency of the player having profit.

```python
from random import randint

bet = 5
Ns = 1000000 # Number of games
p = 0

for n in range(1,Ns):
    G = 0
    j = randint(1, 6) # Player
    b = randint(1, 6) # Oponent
    if(j > b): # If the player has a greater value
        G = 2*(j-b)
    if(G > bet): # If there's profit
        p = p + 1

p = p/Ns # Probability of profit
print('Bet:', bet, 'dollars.')
print("Probability of profit:", p)
```

    Bet: 5 dollars.
    Probability of profit: 0.166387

But we must understand that the profit is a function with the following form:

$ G = \left\{ \begin{array}{cl}
2(Y - X) & : \ Y > X \\
0 & : \ Y \leq X
\end{array} \right.$

With the following probability distribution:

| $G$ | 0 | 2 | 4 | 6 | 8 | 10 |
|---|---|---|---|---|---|----|
| $P(G = g)$  | $\frac{21}{36}$  | $\frac{5}{36}$  | $\frac{4}{36}$  |  $\frac{3}{36}$ | $\frac{2}{36}$  |  $\frac{1}{36}$  |

```python
x = [0, 2, 4, 6, 8, 10]
prob = [21/36, 5/36, 4/36, 3/36, 2/36, 1/36]

df = pd.DataFrame({'Value': x,'Probability': prob})

df.plot.bar(x = 'Value', y = 'Probability', rot = 0, title = "Probability for sum of points")

fig = plt.gcf()
fig.set_size_inches(8, 4)

plt.show(block=True)
```

For the player have some profit we need to have $P(G \geq 6)$, so:

$$P(G \geq 6) = \frac{3}{36} + \frac{2}{36} + \frac{1}{36} = \frac{1}{6} \cong 0.16667$$

## Expected value and variance

### Expected value

The expected value, also called mathematical hope or expectancy, of a random variable is the sum of the product of each probability of exit from the experience by its respective value. That is, it represents the average "expected" value of an experience if it is repeated many times.

For discrete random variables we can calculate it with:

$$EX=\sum_{x_k \in R_X} x_k P_X(x_k).$$

And for continuous random variables:

$$EX= \int_{-\infty}^{\infty} xf_X(x)dx$$

We can think intuitively that the expected value for a uniform distribution with $X \sim Uniform(a,b)$ is $\frac{a + b}{2}$, the mean and median for the distibution, using the idea that every probability has equal weight, but we can calculate using the idea of the probability density function, which is

$\begin{equation}
                           \nonumber f_X(x) = \left\{
                            \begin{array}{l l}
                              \frac{1}{b-a} & \quad  a < x < b\\
                              0 & \quad x  b
                            \end{array} \right.
                          \end{equation}$

So for the expected value:

$\begin{array}{rcl}
EX & = & \int_{-\infty}^{\infty} xf_X(x)dx \\
 & = & \int_{a}^{b} x (\frac{1}{b-a}) dx \\
 & = & \frac{1}{b-a} \bigg[ \frac{1}{2}x^2 \bigg]_{a}^{b} dx\\
 & = & \frac{a+b}{2}
\end{array}$

#### Properties

$E[aX + b] = aE[X] + b$

In the general case we have:

$E[XY] \neq E[X]E[Y]$

But if we have independent random variables $X, Y$:

$E[XY] = E[X]E[Y]$

#### Expected value simulation

We can use the example from the random variable simulation to simulate the expected value:

```python
from random import randint

bet = 5
Ns = 1000000 # Number of games
vsim = [] # Simulation number
p = 0

for n in range(1,Ns):
    G = 0
    j = randint(1, 6) # Player
    b = randint(1, 6) # Oponent
    if(j > b): # If the player has a greater value
        G = 2*(j-b)

    vsim.append(G)

E = sum(vsim)/Ns # Probability of profit
print('Bet:', bet, 'dollars.')
print("Expected value:", E)
```

    Bet: 5 dollars.
    Expected value: 1.946174

Using probability distribution found before:

| $G$ | 0 | 2 | 4 | 6 | 8 | 10 |
|---|---|---|---|---|---|----|
| $P(G = g)$  | $\frac{21}{36}$  | $\frac{5}{36}$  | $\frac{4}{36}$  |  $\frac{3}{36}$ | $\frac{2}{36}$  |  $\frac{1}{36}$  |

$E[X] = 0\cdot \frac{21}{36} + 2\cdot \frac{5}{36} + 4\cdot \frac{4}{36} + 6\cdot \frac{3}{36} + 8\cdot \frac{2}{36} + 10\cdot \frac{1}{36} = \frac{35}{18} \cong 1.944$

Beyond using probability, it becomes even more clear that it's not a good idea to make the bet.

A manufacturer produces parts such that 10 % of them are defective and 90 % are perfect. If a defective part is produced, the manufacturer loses \$1.00, while a non defective part gives him a profit of \$5.00. If $X$ is the net profit, calculate $E(X)$.

The probability distribution of the profit:
$P(X = -1) = 0.1$ and $P(X = 5) = 0.9$

Let's simulate this experiment.

```python
from random import seed
import numpy as np
seed(1) # Setting seed

N = 100000 # number of parts
X = 0 # profit by part
p = 0.9 # probability for perfect part
profit = 5 # profit for perfect part
loss = 1 # loss for defective part
for n in range(0,N):
    r = np.random.uniform()
    if(r <= p):
        X = X + profit
    else:
        X = X - loss
X = X/N # expectrd value

print('Expected profit value:', X)
```

    Expected profit value: 4.3934

The probability distribution is:

| $X$ | -1 | 5 |
|---|---|---|
| $P(X = x)$  | $0.1$  | $0.9$  |

So we have:

$$E[X] = -1\cdot 0.1 + 5\cdot 0.9 = 4.4$$

A very similar value from the simulation.

Two balls are randomly chosen without replacement from a box containing 10 white balls, 5 black balls and 5 orange balls. Suppose we get \$ 2.00 for each black ball selected and lose \$ 1.00 for each white ball selected. In the case of the orange ball, there is no gain or loss. Be $X$ the random variable that denotes our winnings. Calculate the average win.

```python
import numpy as np
from random import seed
import random

N = 10000000 # number of simulations
X = 0 # profit

for n in range(0, N):
    balls = ["W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "B", "B", "B", "B", "B", "O", "O", "O", "O", "O"]

    ball_one = random.choice(balls)
    if(ball_one == "W"):
        X = X - 1
    elif (ball_one == "B"):
        X = X + 2

    balls.remove(ball_one)

    ball_two = random.choice(balls)
    if(ball_two == "W"):
        X = X - 1
    elif (ball_two == "B"):
        X = X + 2

    balls.remove(ball_two)

X = X/N # expected value

print('Expected value for profit:', X)
```

    Expected value for profit: -0.0002915

### Variance

The variance of a random variable or stochastic process is a measure of its statistical dispersion, indicating "how far" its values are in general from the expected value.

Let $X$ be a discrete random variable. The n-th central moment is defined by:

$$E[(X - E[X])^{m}]$$

The 1st central moment is equal to zero;

The 2nd central moment is the variance:

$$Var(X) = E[(X - E[X])^{2}] = E[X^2] - E[X]^{2}$$

With $X$ being a continuous random variable we have:

$$Var(X) = E[(X - E[X])^{2}] = E[X^2] - E[X]^{2} = \int_{-\infty}^{\infty}x^{2}f(x)dx - (\int_{-\infty}^{\infty}xf(x)dx)^{2}$$

#### Variance example

With probability distribution from before:

| $G$ | 0 | 2 | 4 | 6 | 8 | 10 |
|---|---|---|---|---|---|----|
| $P(G = g)$  | $\frac{21}{36}$  | $\frac{5}{36}$  | $\frac{4}{36}$  |  $\frac{3}{36}$ | $\frac{2}{36}$  |  $\frac{1}{36}$  |

$E[X^{2}] = 0^{2}\cdot \frac{21}{36} + 2^{2}\cdot \frac{5}{36} + 4^{2}\cdot \frac{4}{36} + 6^{2}\cdot \frac{3}{36} + 8^{2}\cdot \frac{2}{36} + 10^{2}\cdot \frac{1}{36} = \frac{35}{3} \cong 11.67$

So we have:

$$Var(X) = E[X^2] - E[X]^{2} = 11.67 - (1.944)^{2} \cong 7.9$$

Using this [numpy function](https://numpy.org/doc/stable/reference/generated/numpy.var.html) we can the find the variance for the simulation array from before:

```python
import numpy as np

np.var(vsim)
```

    7.906206880333851

## Discrete distributions

### Uniform model

If we have a probability distribution that never changes and each value is equal, we have an uniform distribution. For a fair dice:

| $X$ | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|----|
| $P(X = x)$  | $\frac{1}{6}$  | $\frac{1}{6}$  | $\frac{1}{6}$  |  $\frac{1}{6}$ | $\frac{1}{6}$  |  $\frac{1}{6}$  |

We have:

$$P(X = x_j) = \frac{1}{N}, \qquad j = 1, 2, \cdots, N$$

### Bernoulli model

If we have a random variable with only two outcomes, we can represent the probability in this way:

$$P(X = x) = p^{x}(1 - p)^{1 - x} \qquad x = 0, 1$$

Being the expected value and variance:

$$E[X] = p, \qquad V(X) = p(1 - p)$$

### Binomial model

Let a random variable based on n repetitions of bernoulli process, we have:

$$P(X = k) = \frac{n!}{(n - k)!k!}p^{k}(1 - p)^{n - k}, \qquad k = 0, 1, \cdots, n$$

Being the expected value and variance:

$$E[X] = np, \qquad V(X) = np(1 - p)$$

#### Binomial simulation

We can simulate flipping a coin $n$ times using a binomial model.

```python
from random import seed
from matplotlib import pyplot as plt
import numpy as np
from scipy.stats import binom

import math

n = 100 # launching number
p = 0.5 # probability for heads

Pk = np.zeros(n)
vk = np.arange(0,n)

ns = 10000 # number of simulations

for j in range(0,ns):
    S = 0 # number of sucess
    for i in range(0, n):
        r = np.random.uniform()
        if(r <= p): # if sucess
            S = S + 1
    Pk[S] = Pk[S] + 1

Pk = Pk/sum(Pk) # standardizing

plt.figure(figsize=(10,6))
plt.xlim(0.8*np.min(vk[Pk>0]),1.2*np.max(vk[Pk>0]))
plt.bar(vk, Pk, label='Simulation')

# theoretical curve
Pkt = np.zeros(n+1)
vkt = np.arange(0,n+1)

for k in range(0, n+1):
    Pkt[k] = (math.factorial(n)/(math.factorial(n-k)*math.factorial(k)))*(p**k)*(1-p)**(n-k)

plt.plot(vkt, Pkt, 'r--', label='Theoretical probability')

plt.xlabel('k', fontsize = 15)
plt.ylabel('P(k)',fontsize = 15)
plt.legend()
plt.show(True)
```

We can with many simulations, we have something like a normal distribution. If n is large enough, then the skew of the distribution is not too great. In this case a reasonable approximation to B(n, p) is given by the normal distribution

$$\displaystyle {\mathcal {N}}(np,\,np(1-p))$$

### Poisson distribution

A random variable have a Poisson distribution with parameter $\lambda > 0$ (rate) if its probability function is given by:

$$P(X = k) = \frac{e^{-\lambda}\lambda ^{k}}{k!} \qquad k = 0, 1, \cdots$$

THe expected value and variance is given by:

$$E[X] = V(X) = \lambda$$

```python
import numpy as np
import matplotlib.pyplot as plt

lamb = 50 # rate
n = 100

P = np.random.poisson(lamb, n)
k = np.arange(0, np.max(P))
count, bins, ignored = plt.hist(P, bins=k, density=True, color='#0504aa',alpha=0.7, rwidth=0.9, label='Simulation')

# theoretical curve
Po = np.zeros(n+1)
vo = np.arange(0,n+1)
lbd = n*p

for k in range(0,n+1):
    Po[k] = (np.exp(-lamb)*lamb**k)/math.factorial(k)

plt.plot(vo, Po, 'k--', label='Theoretical probability')

plt.xlim(np.min(P) - 5 , np.max(P) + 5)
plt.xlabel('k', fontsize = 15)
plt.ylabel('P(k)',fontsize = 15)
plt.legend()
plt.show(True)
```

### Geometric distribution

A random variable $X$ that follows a geometric distribution with parameter $p$ if:

$$P(X = k) = p(1 - p)^{k - 1} \qquad k = 0, 1, 2, \cdots$$

The expected value and variance:

$$E[X] = \frac{1}{p}\;, \qquad V(X) = \frac{1 - p}{p^{2}}$$

```python
import numpy as np
import matplotlib.pyplot as plt

p = 0.2 # probability for sucess
n = 100000

X = np.random.geometric(p, size=n)
k = np.arange(0, np.max(X))
count, bins, ignored = plt.hist(X, bins=k, density=True, color='#0504aa',alpha=0.7,rwidth=0.9)
plt.xlabel('k', fontsize = 15)
plt.ylabel('P(k)',fontsize = 15)
plt.show(True)

print("Theoretical value:", 1/p)
print("Expected value simulation:", np.sum(X)/len(X))
```

### Negative Binomial Distribution

Let $X$ be the number of necessary repetitions to occurr exactly $r$ successes in a way that the $r$-th sucess occurs in the $k$-th try, so:

$$P(X = k) = \binom{k - 1}{r - 1}p^{r}(1 - p)^{k - r}, \qquad k = r, r + 1, \cdots$$

Expected value and variance:

$$E[X] = \frac{r}{p}\; , \qquad V(X) = \frac{r(1 - p)}{p^{2}}$$

### Hypergeometric Distribution

Consider a set of $N$ objects, of which $m$ are type I and $N-m$ are type II. For a random selection of r objects (r<N), without replacement, define: X: number of selected objects of type I.

$$P(X=k)=\frac{{{N_1}\choose{k}}{{N_2}\choose{r-k}}}{{{N}\choose{r}}}, \quad N = N_1 + N_2$$

```python
import numpy as np
import matplotlib.pyplot as plt

N1 = 4
N2 = 21
r = 5
n = 1000

X = np.random.hypergeometric(N2, N1, r, 1000)
k = np.arange(0, np.max(X))

print("Theoretical expected value:", N2*N1/(N1+N2))
print("Simulation expected value:", np.sum(X)/len(X))

count, bins, ignored = plt.hist(X, bins=k, density=True, color='#0504aa',alpha=0.7,
                                rwidth=0.9)
plt.xlabel('k', fontsize = 15)
plt.ylabel('P(k)',fontsize = 15)
plt.show(True)
```

## Continuous distributions

### Normal distribution

A random variable $X$ that takes all real values $-\infty < x < \infty$ follows a normal distribution if its density probability function is given by:

$$f(x) = \frac{1}{\sigma \sqrt{2\pi}}\exp \left[\frac{-1}{2}\left(\frac{x - \mu}{\sigma}\right)^{2}\right]$$

Being:

$$\mu = E[X], \qquad \sigma^2 = V(X)$$

The notation is $ X \sim \mathcal{N}(\mu, \sigma^{2})$.

In relation to the mean, using the measure of standard deviation we can see where much of data comes from. With $1\sigma$ we have 68.27% of the data:

```python
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
from matplotlib.patches import Polygon
import math

mu = 0
sigma = 2

fig, ax = plt.subplots()
x = np.linspace(mu - 5*sigma, mu + 5*sigma, 100)
ax.plot(x, stats.norm.pdf(x, mu, sigma))
ax.set_ylim(bottom=0)

ax.text(0, 0.1, r"$68.27\%$", horizontalalignment='center', fontsize=16)

a = mu - sigma
b = mu + sigma

ix = np.linspace(a, b)
iy = stats.norm.pdf(ix, mu, sigma)

verts = [(a, 0), *zip(ix, iy), (b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)

plt.show()
```

For $2\sigma$:

```python
fig, ax = plt.subplots()
x = np.linspace(mu - 5*sigma, mu + 5*sigma, 100)
ax.plot(x, stats.norm.pdf(x, mu, sigma))
ax.set_ylim(bottom=0)

ax.text(0, 0.1, r"$95.45\%$", horizontalalignment='center', fontsize=16)

a = mu - 2*sigma
b = mu + 2*sigma

ix = np.linspace(a, b)
iy = stats.norm.pdf(ix, mu, sigma)

verts = [(a, 0), *zip(ix, iy), (b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)

plt.show()
```

For $3\sigma$:

```python
fig, ax = plt.subplots()
x = np.linspace(mu - 5*sigma, mu + 5*sigma, 100)
ax.plot(x, stats.norm.pdf(x, mu, sigma))
ax.set_ylim(bottom=0)

ax.text(0, 0.1, r"$99.73\%$", horizontalalignment='center', fontsize=16)

a = mu - 3*sigma
b = mu + 3*sigma

ix = np.linspace(a, b)
iy = stats.norm.pdf(ix, mu, sigma)

verts = [(a, 0), *zip(ix, iy), (b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)

plt.show()
```

For a normal distribution we can use the standard deviation in relation to the mean to get the adequate percentile.

This is represented by the Z-Score:

$$Z = \frac{X - \mu}{\sigma}$$

For example, if we use $+0.5\sigma$ (or $Z = 0.5$) to $\mu$:

```python
stats.norm.cdf(0.5)
```

    0.6914624612740131

This is the 69th percentile, calculated with [stats.norm.cdf](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) we can see it graphically:

```python
mu = 0
sigma = 2

fig, ax = plt.subplots()
x = np.linspace(mu - 5*sigma, mu + 5*sigma, 100)
ax.plot(x, stats.norm.pdf(x, mu, sigma))
ax.set_ylim(bottom=0)

ax.text(0, 0.1, "69.15%", horizontalalignment='center', fontsize=18)

a = x[0]
b = mu + 0.5*sigma

ix = np.linspace(a, b)
iy = stats.norm.pdf(ix, mu, sigma)

verts = [(a, 0), *zip(ix, iy), (b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)

plt.show()
```

So, the disllocation in relation to the mean $\mu$ using the standard deviation $\sigma$ is the Z-Score. We can use the same library from before with the function ppf, which uses the percentile as the parameter, returning the Z-Score:

```python
stats.norm.ppf(0.6915)
```

    0.5001066271622767

### Exponential distribution

A random variable $X$ follows an exponential model if its probability density function is given by:

$\begin{equation}
                           \nonumber f(x) = \left\{
                            \begin{array}{l l}
                              \alpha e^{-\alpha x}, & \quad  x > 0\\
                              0, & \quad x < 0
                            \end{array} \right.
                          \end{equation}$

Expected value and variance:

$$E[X] = \frac{1}{\alpha}, \qquad V(X) = \frac{1}{\alpha ^{2}}$$

### Uniform distribution

A random variable $X$ follows an uniform distribution if its probability density function is given by:

$\begin{equation}
                           \nonumber f(x) = \left\{
                            \begin{array}{l l}
                              \frac{1}{b - a}, & \quad  x \in [a, b]\\
                              0, & \quad x \notin [a, b]
                            \end{array} \right.
                          \end{equation}$

Expected value and variance:

$$E[X] = \frac{a + b}{2}, \qquad V(X) = \frac{(b - a)^{2}}{12}$$

### Gamma distribution

A random variable $X$ follows a Gamma distribution if:

$$f(x; \alpha , \beta) = \frac{\beta^{\alpha}x^{\alpha - 1}e^{-\beta x}}{\Gamma (\alpha)},\; x > 0 \; \alpha , \beta > 0 \qquad \Gamma (z) = \int_{0}^{\infty}x^{z - 1}e^{-x}dx, \qquad \mathfrak{R} (z) > 0$$

The expected value and variance:

$$E[X]: \text{No closed formula}, \qquad V(X) = \frac{\alpha}{\beta^{2}}$$

The exponential distribution is a special case of the gamma distribution ($\alpha = 1$).

### Beta distribution

A random variable $X$ follows a Beta distribution if:

$$f(x; \alpha , \beta) = \frac{1}{B(\alpha, \beta)}x^{\alpha - 1}(1 - x)^{\beta - 1} \qquad B(x, y) = \int_{0}^{1}t^{x - 1}(1 - t)^{y - 1}dt$$

The expected value and variance:

$$E[X] = \frac{\alpha}{\alpha + \beta}, \qquad V(X) = \frac{\alpha \beta}{(\alpha + \beta)^{2}(\alpha + \beta + 1)}$$

### Weibull distribution

A random variable $X$ follows a Weibull distribution if:

$\begin{equation}
                           \nonumber f(x; \lambda, k) = \left\{
                            \begin{array}{l l}
                              \frac{k}{\lambda}(\frac{x}{\lambda})^{k - 1}e^{-(\frac{x}{\lambda})^{k}}, & \quad  x \geq 0\\
                              0, & \quad x < 0
                            \end{array} \right.
                          \end{equation}$

Expected value and variance:

$$E[X] = \lambda \Gamma (1 + \frac{1}{k})$$

$$V(X) = \lambda^2\left[\Gamma\left(1+\frac{2}{k}\right) - \left(\Gamma\left(1+\frac{1}{k}\right)\right)^2\right]\,$$

### Student s t-distribution

A random variable $X$ follows a Student's t-distribution if:

$$f(t) = \frac{\Gamma(\frac{\nu+1}{2})} {\sqrt{\nu\pi}\,\Gamma(\frac{\nu}{2})} \left(1+\frac{t^2}{\nu} \right)^{\!-\frac{\nu+1}{2}},\!$$

### Chi-square distribution

A random variable $X$ follows a Chi-square distribution if:

$$\displaystyle { f(x; k) = \frac {1}{2^{k/2}\Gamma (k/2)}}\;x^{k/2-1}e^{-x/2}\;$$

The expected value and variance:

$$E[X] = k, \qquad V(x) = 2k$$


<!-- dom:TITLE: Demo - Working with Functions -->
# Demo - Working with Functions
<!-- dom:AUTHOR: Mikael Mortensen Email:mikaem@math.uio.no at Department of Mathematics, University of Oslo. -->
<!-- Author: -->
**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **August 7, 2020**

**Summary.** This is a demonstration of how the Python module [shenfun](https://github.com/spectralDNS/shenfun) can be used to work with
global spectral functions in one and several dimensions.

## Construction

A global spectral function $u(x)$ is represented on a one-dimensional
domain (a line) as

$$
u(x) = \sum_{k=0}^{N-1} \hat{u}_k \psi_k(x)
$$

where $\psi_k(x)$ is the $k$'th basis function and $x$ is a
position inside the domain. $\{\hat{u}_k\}_{k=0}^{N-1}$ are the
expansion coefficient for the series, and often referred to as the
degrees of freedom. There is one degree of freedom per basis function.
We can use any number of basis functions,
and the span of the chosen basis is then a function space. Also part of the
function space is the domain, which is
specified when a function space is created. To create a function space
$T=\text{span}\{T_k\}_{k=0}^{N-1}$ for
the first N Chebyshev polynomials of the first kind on the default domain $[-1, 1]$,
do

```
from shenfun import *
N = 8
T = FunctionSpace(N, 'Chebyshev', domain=(-1, 1))
```

The function $u(x)$ can now be created with all N coefficients
equal to zero as

```
u = Function(T)
```

When using Chebyshev polynomials the computational domain is always
$[-1, 1]$. However, we can still use a different physical domain,
like

```
T = FunctionSpace(N, 'Chebyshev', domain=(0, 1))
```

and under the hood shenfun will then map this domain to the reference
domain through

$$
u(x) = \sum_{k=0}^{N-1} \hat{u}_k \psi_k(2(x-0.5))
$$

## Approximating analytical functions

The `u` function above was created with only zero
valued coefficients, which is the default. Alternatively,
a [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) may be initialized using a constant
value

```
T = FunctionSpace(N, 'Chebyshev', domain=(-1, 1))
u = Function(T, val=1)
```

but that is not very useful. A third method to initialize
a [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) is to interpolate using an analytical
Sympy function.

```
import sympy as sp
x = sp.Symbol('x', real=True)
u = Function(T, buffer=4*x**3-3*x)
print(u)
```

Here the analytical Sympy function will first be evaluated
on the entire quadrature mesh of the `T` function space,
and then forward transformed to get the coefficients. This
corresponds to a projection to `T`. The projection is

Find $u_h \in T$, such that

$$
(u_h - u, v)_w = 0 \quad \forall v \in T,
$$

where $v \in \{T_j\}_{j=0}^{N-1}$ is a test function,
$u_h=\sum_{k=0}^{N-1} \hat{u}_k T_k$ is a trial function and the
notation $(\cdot, \cdot)_w$ represents a weighted inner product.
In this projection $u_h$ is the [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function), $u$ is the sympy function and we use sympy
to exactly evaluate $u$ on all quadrature points
$\{x_j\}_{j=0}^{N-1}$. With quadrature we then have

$$
(u, v)_w = \sum_{j\in\mathcal{I}^N} u(x_j) v(x_j) w_j \forall v \in T,
$$

where $\mathcal{I}^N = (0, 1, \ldots, N-1)$ and $\{w_j\}_{j\in \mathcal{I}^N}$
are the quadrature weights. The left hand side of the projection is

$$
(u_h, v)_w = \sum_{j\in\mathcal{I}^N} u_h(x_j) v(x_j) w_j \forall v \in T.
$$

A linear system of equations arise when inserting for the
basis functions

$$
\left(u, T_i\right)_w = \tilde{u}_i \forall i \in \mathcal{I}^N,
$$

and

$$
\begin{align*}
\left(u_h, T_i \right)_w &= (\sum_{k\in \mathcal{I}^N} \hat{u}_k T_k , T_i)_w \\
  &= \sum_{k\in \mathcal{I}^N} \left( T_k, T_i\right)_w \hat{u}_k
\end{align*}
$$

with the mass matrix

$$
a_{ik} = \left( T_k, T_i\right)_w   \forall (i, k) \in \mathcal{I}^N \times \mathcal{I}^N,
$$

we can now solve to get the unknown
expansion coefficients. In matrix notation

$$
\hat{u} = A^{-1} \tilde{u},
$$

where $\hat{u}=\{\hat{u}_i\}_{i\in \mathcal{I}^N}$,
$\tilde{u}=\{\tilde{u}_i\}_{i \in \mathcal{I}^N}$ and
$A=\{a_{ki}\}_{(i,k) \in \mathcal{I}^N \times \mathcal{I}^N}$

## Adaptive function size

The number of basis functions can also be left open during creation
of the function space, through

```
T = FunctionSpace(0, 'Chebyshev', domain=(-1, 1))
```

This is useful if you want to approximate a function and
are uncertain how many basis functions that are required.
For example, you may want to approximate the function $\cos(20 x)$.
You can then find the required [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) using

```
u = Function(T, buffer=sp.cos(20*x))
print(len(u))
```

We see that $N=45$ is required to resolve this function. This agrees
well with what is reported also by [Chebfun](https://www.chebfun.org/docs/guide/guide01.html).
Note that in this process a new [FunctionSpace()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.FunctionSpace) has been
created under the hood. The function space of `u` can be
extracted using

```
Tu = u.function_space()
print(Tu.N)
```

To further show that shenfun is compatible with Chebfun we can also
approximate the Bessel function

```
T1 = FunctionSpace(0, 'Chebyshev', domain=(0, 100))
u = Function(T1, buffer=sp.besselj(0, x))
print(len(u))
```

which gives 83 basis functions, in close agreement with Chebfun (89).
The difference lies only in the cut-off criteria. We cut frequencies
with a relative tolerance of 1e-12 by default, but if we make this criteria
a little bit stronger, then we will also arrive at a slightly higher number:

```
u = Function(T1, buffer=sp.besselj(0, x), reltol=1e-14)
print(len(u))
```

Plotting the function on its quadrature points looks
a bit ragged, though:

```
%matplotlib inline

import matplotlib.pyplot as plt
Tu = u.function_space()
plt.plot(Tu.mesh(), u.backward())
```

To improve the quality of this plot we can instead evaluate the
function on more points

```
xj = np.linspace(0, 100, 1000)
plt.plot(xj, u(xj))
```

Alternatively, we can refine the function, which simply
pads zeros to $\hat{u}$

```
up = u.refine(200)
Tp = up.function_space()
plt.plot(Tp.mesh(), up.backward())
```

The padded expansion coefficients are now given as

```
print(up)
```

## More features

Since we have used a regular Chebyshev basis above, there
are many more features that could be explored simply by going through
[Numpy's Chebyshev module](https://numpy.org/doc/stable/reference/routines.polynomials.chebyshev.html).
For example, we can create a Chebyshev series like

```
import numpy.polynomial.chebyshev as cheb
c = cheb.Chebyshev(u, domain=(0, 100))
```

The Chebyshev series in Numpy has a wide range of possibilities,
see [here](https://numpy.org/doc/stable/reference/generated/numpy.polynomial.chebyshev.Chebyshev.html#numpy.polynomial.chebyshev.Chebyshev).
However, we may also work directly with the Chebyshev
coefficients already in `u`. To find the roots of the
polynomial that approximates the Bessel function on
domain $[0, 100]$, we can do

```
z = Tu.map_true_domain(cheb.chebroots(u))
```

Note that the roots are found on the reference domain $[-1, 1]$
and as such we need to move the result to the physical domain using
`map_true_domain`. The resulting roots `z` are both real and imaginary,
so to extract the real roots we need to filter a little bit

```
z2 = z[np.where((z.imag == 0)*(z.real > 0)*(z.real < 100))].real
print(z2[:5])
```

Here `np.where` returns the indices where the condition is true. The condition
is that the imaginary part is zero, whereas the real part is within the
true domain $[0, 100]$.

**Notice.**

Using directly `cheb.chebroots(c)` does not seem to work (even though the
series has been generated with the non-standard domain) because
Numpy only looks for roots in the reference domain $[-1, 1]$.

We could also use a function space with boundary conditions built
in, like

```
Td = FunctionSpace(0, 'C', bc=(sp.besselj(0, 0), sp.besselj(0, 100)), domain=(0, 100))
ud = Function(Td, buffer=sp.besselj(0, x))
print(len(ud))
```

As we can see this leads to a function space of dimension
very similar to the orthogonal space.

The major advantages of working with a space with boundary conditions
built in only comes to life when solving differential equations. As
long as we are only interested in approximating functions, we may just
as well stick to the orthogonal spaces. This goes for Legendre as
well as Chebyshev.

## Multidimensional functions

Multidimensional tensor product spaces are created
by taking the tensor products of one-dimensional function spaces.
For example

```
C0 = FunctionSpace(20, 'C')
C1 = FunctionSpace(20, 'C')
T = TensorProductSpace(comm, (C0, C1))
u = Function(T)
```

Here $\text{T} = \text{C0} \otimes \text{C1}$, the basis function is
$T_i(x) T_j(y)$ and the Function `u` is

$$
u(x, y) = \sum_{i=0}^{N-1} \sum_{j=0}^{N-1} \hat{u}_{ij} T_i(x) T_j(y).
$$

The multidimensional Functions work more or less exactly like for the
1D case. We can here interpolate 2D Sympy functions

```
y = sp.Symbol('y', real=True)
u = Function(T, buffer=sp.cos(10*x)*sp.cos(10*y))
X = T.local_mesh(True)
plt.contourf(X[0], X[1], u.backward())
```

Like for 1D the coefficients are computed through projection,
where the exact function is evaluated on all quadrature points
in the mesh.

The Cartesian mesh represents the quadrature points of the
two function spaces, and can be visualized as follows

```
X = T.mesh()
for xj in X[0]:
    for yj in X[1]:
        plt.plot((xj, xj), (X[1][0, 0], X[1][0, -1]), 'k')
        plt.plot((X[0][0], X[0][-1]), (yj, yj), 'k')
```

We may alternatively plot on a uniform mesh

```
X = T.local_mesh(broadcast=True, uniform=True)
plt.contourf(X[0], X[1], u.backward(kind='uniform'))
```

## Curvilinear coordinates

With shenfun it is possible to use curvilinear coordinates,
and not necessarily with orthogonal basis vectors. With
curvilinear coordinates the computational coordinates are
always straight lines, rectangles and cubes. But the physical
coordinates can be very complex.

Consider the unit disc with polar coordinates. Here
the position vector $\mathbf{r}$ is given by

$$
\mathbf{r} = r\cos \theta \mathbf{i} + r\sin \theta \mathbf{j}
$$

The physical domain is $\Omega = \{(x, y): x^2 + y^2 < 1\}$,
whereas the computational domain is the Cartesian product
$D = \{(r, \theta) \in [0, 1] \times [0, 2 \pi]\}$.

We create this domain in shenfun through

```
r, theta = psi = sp.symbols('x,y', real=True, positive=True)
rv = (r*sp.cos(theta), r*sp.sin(theta))
B0 = FunctionSpace(20, 'C', domain=(0, 1))
F0 = FunctionSpace(20, 'F')
T = TensorProductSpace(comm, (B0, F0), coordinates=(psi, rv))
```

Note that we are using a Fourier space for the azimuthal
direction, since the solution here needs to be periodic.
We can now create functions on the space using an
analytical function in computational coordinates

```
u = Function(T, buffer=(1-r)*r*sp.sin(sp.cos(theta)))
```

However, when this is plotted it may not be what you expect

```
X = T.local_mesh(True)
plt.contourf(X[0], X[1], u.backward(), 100)
```

We see that the function has been plotted in computational coordinates,
and not on the disc, as you probably expected. To plot on
the disc we need the physical mesh, and not the computational

```
X = T.local_cartesian_mesh()
plt.contourf(X[0], X[1], u.backward(), 100)
```

**Notice.**

The periodic plot does not wrap all around the circle. This is
not wrong, we have simply not used the same point twice, but it
does not look very good. To overcome this problem we can wrap the
grid all the way around and re-plot.

```
up = u.backward()
xp, yp, up = wrap_periodic([X[0], X[1], up], axes=[1])
plt.contourf(xp, yp, up, 100)
```

## Adaptive functions in multiple dimensions

If you want to find a good resolution for a function in multiple
dimensions, the procedure is exactly like in 1D. First create function
spaces with 0 quadrature points, and then call [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function)

```
B0 = FunctionSpace(0, 'C', domain=(0, 1))
F0 = FunctionSpace(0, 'F')
T = TensorProductSpace(comm, (B0, F0), coordinates=(psi, rv))
u = Function(T, buffer=((1-r)*r)**2*sp.sin(sp.cos(theta)))
print(u.shape)
```

The algorithm used to find the approximation in multiple dimensions
simply treat the problem one direction at the time. So in this case
we would first find a space in the first direction by using
a function ` ~ ((1-r)*r)**2`, and then along the second using
a function ` ~ sp.sin(sp.cos(theta))`.

<!-- ======= Bibliography ======= -->


# Introduction to Deep Learning with PyTorch

[구글Colab에서 실행하기](https://colab.research.google.com/github/aonekoda/DL-PyTorch/blob/master/01Basics/04NetworkBasic_ex.ipynb)

 [PyTorch](http://pytorch.org/)를 활용하여 간단한 신경망 모형을 구현해 보자. Pytorch의 tensor는 앞서 학습한 Numpy의 ndarray와 매우 유사하다. Tensor는 또한 backpropagation을 위해 자동으로 gradient를 계산하는 기능을 제공한다.

## Neural Networks

Deep Learning 은 artificial neural networks에 기반을 두고 있다. 인공 신경망 모형은 neurons으로 구성된다. 각 뉴런은 input을 받아 weight에 곱하여 이를 모두 더한다. 이를 activation function(활성화 함수)를 거쳐 최종 output을  출력한다.

수학적 기호로 표현하면 다음과 같다.:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

이것은 두 vectors의 dot/inner product 로 표현할 수 있다:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Tensors

-  vector 는 1-dimensional tensor
-  matrix is a 2-dimensional tensor
-  3-dimensional tensor

텐서는 neural networks의 기본적인 자료 구조이다.

기본을 익히기 위해 tensor를 사용해서 간단한 neural network를 구성하는 실습을 해보자.

```python
# First, import PyTorch
import torch
```

```python
def activation(x):
    """ Sigmoid activation function

        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))
```

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))
```

먼저 실습을 위해서 random한 숫자로 이루어진 tensor 데이터를 생성한다.

- `features = torch.randn((1, 5))` : shape이 `(1, 5)`인 tensor를 생성한다. 표준 정규분포(평균이 0이고 표준편차가 1인 정규분포)로부터 random 숫자로 생성된다.

- `weights = torch.randn_like(features)` : `features`와 동일한 shape을 가지는 tensor를 생성한다.

- `bias = torch.randn((1, 1))` : 단일한 숫자를 normal distribution로 부터 생성한다.

PyTorch tensors 는 Numpy arrays 처럼 더하기 빼기 등의 연산이 가능하다.

> **실습** : input features인 `features`, weights `weights`, 그리고 bias `bias`로 output 데이터 y를 생성해 보자. [`torch.sum()`](https://pytorch.org/docs/stable/torch.html#torch.sum) 또는 `.sum()` method 를 사용할 수 있다.이를 사용하여 다음 식을 구성해 보자. `activation`은 앞에서 생성한 activation function(=sigmoid)를 사용한다.
$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

```python
## TODO

y = None
```

multiplication과 sum 연산은  matrix multiplication으로 구현 가능하다. 일반적으로 matrix multiplications 을 사용하는 것이 연산에 효율적이다.

그래서 위에 했던 연산을 matrix multiplication으로 바꾸어 보자. 이를 위해 [`torch.mm()`](https://pytorch.org/docs/stable/torch.html#torch.mm) 또는 [`torch.matmul()`](https://pytorch.org/docs/stable/torch.html#torch.matmul) 을 사용한다. 연산하는 두 matrix의 shape이 맞지 않으면 다음과 error가 발생하므로 shape을 주의하자.

```python
>> torch.mm(features, weights)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> 1 torch.mm(features, weights)

RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
```

이러한 에러는 텐서의 shape(=dim)정보가 맞지 않기 때문에 발생한다.

다음과 같은 사항을 반드시 주의하자.
- 첫번째 tensor의 column의 갯수와  두번째 tensor의 row 갯수가 서로 일치해야 한다.
- 만약 `features` 가 `weights` 같은 shape `(1, 5)` 라고 하면 `weights` 의 shape를 matrix multiplication 에 맞도록 바꾸어야 한다..

**Note:** `tensor.shape`로 shape를 확인할 수 있다.

변경을 위한 옵션 : [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape), [`weights.resize_()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_), 또는 [`weights.view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view).

일반적으로 `.view()` 메소드를 많이 사용한다. 우리는 이 실습에서 `weights`를 5행 1열의 matrix로 변경하여 수행한다. `weights.view(5, 1)`.

> **실습** : 위와 동일하게 y를 matrix multiplication로 구현하시오.

```python
## TODO

y = None
```

### Stack them up!

앞선 예제에서는 단일 뉴런으로 구성했지만 인공 신경망은 이런 뉴런들을 layer로 여럿 쌓아서 작동하는 알고리즘이다. 첫번째 layer의 output이 두번째 layer의 뉴런 입력이된다. 입력 뉴런과 출력 뉴런의 개수가 많고 layer를 쌓기 위해서 matrix를 사용한다.

**input layer** : 그림에서 맨 아래쪽의 layer로 입력에 해당
**hidden layer** : 중간 layer
**output layer** : 맨 윗쪽의 layer.

network를 수학적으로  matrix multiplication 으로 표현하면 다음과 같다.
$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

output layer를 간단히 표현 하면 다음과 같다.

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 3))

# Define the size of each layer in our network
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)

# and bias terms for hidden and output layers
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))
```

> **실습** : 위에서 생성한 data를 사용하여 network의 출력값을 계산하고 화면에 출력해 보시오.

```python
### TODO

print(output)
```

`tensor([[ 0.3171]])`라는 결과가 나오는가?

## Numpy to Torch and back

PyTorch의 tensor와 numpy array는 서로 변환 가능하다.
- `torch.from_numpy()`: numpy array를 tensor로
- `.numpy()` method : tensor를 numpy array로.

```python
import numpy as np
a = np.random.rand(4,3)
a
```

**실습** : a numpy array를 tensor로 변환하여 b라는 변수에 저장하시오.

```python
## TODO
b = None
b
```

**실습** : 위의 b를 numpy array로 변환해 보시오.

```python
## TODO

```

**실습** : 위의 예제에서 Numpy array와 Torch tensor는 메모리를 공유한다.그래서 in-place 연산을 수행하면 a, b 값이 모두 변하는 것을 확인할 수 있다. 다음을 수행하여 결과를 확인하시오.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

```python
# Numpy array matches new values from Tensor
a
```