# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Inverse Transform

So far only the [(forward) Laplace transform](definition.ipynb) has been introduced. The Laplace transform features also an [inverse transform](https://en.wikipedia.org/wiki/Inverse_Laplace_transform). The inverse Laplace transform maps a complex-valued Laplace transform $X(s) \in \mathbb{C}$ with complex-valued independent variable $s \in \mathbb{C}$ into the complex-valued signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$. It can be shown that the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ is uniquely determined for most practically relevant signals. This section discusses two different techniques for the computation of the inverse Laplace transform.

### Integral Formula

Using results from complex analysis, the inverse Laplace transform is given by the following complex line integral

\begin{equation}
x(t) = \frac{1}{2 \pi j} \int_{\sigma - j \infty}^{\sigma + j \infty} X(s) \, e^{s t} \; ds
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ is assumed to be analytic in its simply connected region of convergence (ROC). The notation $\sigma \mp j \infty$ for the lower/upper integration limit denotes an arbitrary integration path which lies in the ROC and ranges from $\Im \{s\} = - \infty$ to $\Im \{s\} = + \infty$. The integration path can be chosen parallel to the imaginary axis but also all other paths in the ROC are possible. This results from [Cauchy's integral theorem](https://en.wikipedia.org/wiki/Cauchy's_integral_theorem). Two equivalent paths are shown in the following illustration

where the blue line indicates the integration path and the gray area the ROC.

### Rational Laplace Transforms

Computing the inverse Laplace transform by above integral formula can be challenging. The [Cauchy residue theorem](https://en.wikipedia.org/wiki/Residue_theorem) provides a practically tractable solution for Laplace transforms $X(s) = \mathcal{L} \{ x(t) \}$ which are given as rational functions. It states that the value of a line integral of an holomorphic function over a closed contour is given by summing up its [residues](https://en.wikipedia.org/wiki/Residue_theorem). The residue is the value of the line integral for a path enclosing a singularity. Consequently, the inverse Laplace transform of a rational Laplace transform can be computed by summing up the individual contributions from its poles. This procedure is detailed in the following.

#### Basic Procedure

A rational Laplace transform $X(s)$ can be written in terms of its numerator and denominator polynomial

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n}
\end{equation}

where $M$, $N$ denote the order of the numerator and denominator polynomial and $\beta_m$, $\alpha_n$ their coefficients, respectively. It is assumed that $\alpha_N \neq 0$ and that $M \leq N$. If $M > N$, $X(s)$ can be decomposed by [polynomial division](https://en.wikipedia.org/wiki/Polynomial_long_division) into a sum of powers of $s$ and a rational function fulfilling $M \leq N$.

Now a [partial fraction decomposition](https://en.wikipedia.org/wiki/Partial_fraction_decomposition) of $X(s)$ is performed resulting in

\begin{equation}
X(s) = A_0 + \sum_{\mu = 1}^{L} \sum_{\nu = 1}^{R_\mu} \frac{A_{\mu \nu}}{(s - s_{\infty \mu})^\nu}
\end{equation}

where $s_{\infty \mu}$ denotes the $\mu$-th unique pole of $X(s)$, $R_\mu$ its degree and $L$ the total number of different poles $\mu = 1 \dots L$. Using the known Laplace transforms (cf. [example for the modulation theorem](theorems.ipynb#Modulation-Theorem) or [table of selected transforms](table_theorems_transforms.ipynb#Selected-Transforms))

\begin{equation}
\mathcal{L} \{ t^n e^{-s_0 t} \epsilon(t) \} = \frac{n!}{(s + s_0)^{n+1}} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

and $\mathcal{L} \{ \delta(t) \} = 1$, together with the linearity of the Laplace transform yields a generic result for the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of a right-sided signal

\begin{equation}
x(t) = A_0 \cdot \delta(t) + \epsilon(t) \sum_{\mu = 1}^{L} e^{s_{\infty \mu} t} \sum_{\nu = 1}^{R_\mu} \frac{A_{\mu \nu} \, t^{\mu - 1}}{(\nu -1)!}
\end{equation}

It remains to compute the coefficients $A_0$ and $A_{\mu \nu}$ of the partial fraction decomposition. The constant coefficient $A_0$ is given as

\begin{equation}
A_0 = \lim_{s \to \infty} X(s)
\end{equation}

For a pole $s_{\infty \mu}$ with degree $R_\mu = 1$, the coefficient $A_{\mu 1}$ reads

\begin{equation}
A_{\mu 1} = \lim_{s \to s_{\infty \mu}} \left( X(s) \cdot (s - s_{\infty \mu}) \right)
\end{equation}

For a pole $s_{\infty \mu}$ of degree $R_\mu > 1$, the coefficients $A_{\mu \nu}$ are given as

\begin{equation}
A_{\mu \nu} = \frac{1}{(R_\mu - \nu)!} \lim_{s \to s_{\infty \mu}} \frac{d^{R_\mu - \nu}}{d s^{R_\mu - \nu}} \left( X(s) \cdot (s - s_{\infty \mu})^{R_\mu} \right)
\end{equation}

#### Example - Inverse transform of a rational Laplace transform

The inverse transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of

\begin{equation}
X(s) = \frac{1}{(s+1) (s+2)^2} \qquad \text{for } \Re \{s \} > -1
\end{equation}

is computed using the procedure outline above. First the function $X(s)$ is defined in `SymPy`

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

s = sym.symbols('s', complex=True)
t = sym.symbols('t', real=True)

X = 1/((s+1)*(s+2)**2)
X
```

Since $X(s)$ has two real-values poles, $x_{\infty 1} = -1$ with degree $R_1 = 1$ and $x_{\infty 2} = -2$ with degree $R_2 = 2$, the following partial fraction decomposition is chosen as ansatz in accordance with above given formula

```python
A0, A11, A21, A22 = sym.symbols('A_0 A_{11} A_{21} A_{22}', real=True)
Xp = A0 + A11/(s+1) + A21/(s+2) + A22/(s+2)**2
Xp
```

The four real-valued constants $A_0$, $A_{11}$, $A_{21}$ and $A_{22}$ of the partial fraction decomposition will be determined later. First a look is taken at the inverse Laplace transforms of the individual summands composed from the constant and the individual poles. For the constant we get

```python
x0 = sym.inverse_laplace_transform(Xp.args[0], s, t)
x0
```

For the first pole $s_{\infty 1}$ with degree zero we get

```python
x1 = sym.inverse_laplace_transform(Xp.args[1], s, t)
x1
```

For the second pole $s_{\infty 2}$ with degree two we get two contributions

```python
x2 = sym.inverse_laplace_transform(Xp.args[2], s, t)
x2
```

```python
x3 = sym.inverse_laplace_transform(Xp.args[3], s, t)
x3
```

The inverse Laplace transform of $X(s)$ is now composed from the superposition of the four individual parts

```python
x = x0 + x1 + x2 + x3
x
```

For the final solution the four coefficients are determined using the given limit formulas for the coefficients of a partial fraction decomposition

```python
coeffs = {A0 : sym.limit(X, s, sym.oo)}
coeffs.update({A11 : sym.limit(X*(s+1), s, -1)})
coeffs.update({A21 : sym.limit(sym.diff(X*(s+2)**2, s), s, -2)})
coeffs.update({A22 : sym.limit(X*(s+2)**2, s, -2)})
coeffs
```

Substitution into the inverse Laplace transform yields the final solution $x(t) = \mathcal{L}^{-1}\{ X(s) \}$

```python
x = x.subs(coeffs)
x
```

The solution is plotted for illustration

```python
sym.plot(x, (t, -1, 10), ylabel=r'$x(t)$')
```

    <sympy.plotting.plot.Plot at 0x11e8f46a0>

**Exercise**

* Derive the inverse Laplace transform of $X(s)$ by manual calculation.

#### Classification of Poles

Above procedure allows to compute the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of a rational Laplace transform $X(s)$ in a systematic way. It is well suited for an algorithmic realization. However, for manual calculus it may be more efficient to classify the poles with respect to their location in the $s$-plane and their symmetries. The classification can then be used to formulate a modified partial fraction decomposition which limits the need for later algebraic simplification of the inverse Laplace transform. Three classes of poles are typically considered

| Type | Pole-Zero Diagramm | $X(s)$ | $x(t) = \mathcal{L}^{-1} \{ X(s) \} \qquad \qquad$ |
|---|:---:|:---:|:---:|
| Single complex pole |  | $\frac{n!}{(s + s_0)^{n+1}}$ | $t^n e^{-s_0 t} \epsilon(t)$ |
| Conjugated imaginary poles |  | $\frac{A s + B}{s^2 + \omega_0^2}$ | $\begin{cases} \sin(\omega_0 t) \epsilon(t) \\ \cos(\omega_0 t) \epsilon(t) \end{cases}$ |
| Conjugated complex poles |  | $\frac{A s + B}{(s + \sigma_0)^2 + \omega_0^2}$ | $\begin{cases} e^{-\sigma_0 t} \sin(\omega_0 t) \epsilon(t) \\ e^{-\sigma_0 t} \cos(\omega_0 t) \epsilon(t) \end{cases}$ |

where $s_0 \in \mathbb{C}$ and $\omega_0, \sigma_0 \in \mathbb{R}$. The expansion coefficients $A, B \in \mathbb{R}$ can be derived by comparison of coefficients. Whether $x(t)$ contains a sine or cosine depends on the coefficient $A$. If $A \neq 0$ then $x(t)$ contains a cosine (cf. [table of selected transforms](table_theorems_transforms.ipynb#Selected-Transforms)).

#### Example - Inverse transform of a rational Laplace transform with symmetric poles

The inverse transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of

\begin{equation}
X(s) = \frac{2 s^2 + 14 s + 124}{s^3 + 8 s^2 + 46 s + 68} \qquad \text{for } \Re \{s \} > -2
\end{equation}

is computed. First the function $X(s)$ is defined in `SymPy`

```python
X = (2*s**2 + 14*s + 124)/(s**3 + 8 * s**2 + 46*s + 68)
X
```

The poles of $X(s)$ are derived by computing the roots of the denominator polynomial

```python
poles = sym.roots(sym.denom(X))
poles
```

The result is a real-valued pole and a conjugate complex pair of poles. According to above introduced classification of poles, the following ansatz is chosen for the partial fraction decomposition of the Laplace transform

\begin{equation}
X_p(s) = \frac{A}{s + 2} + \frac{B s + C}{s^2 + 6s + 34}
\end{equation}

The coefficients $A, B, C \in \mathbb{R}$ are derived by equating coefficients with $X(s)$

```python
A, B, C = sym.symbols('A B C', real=True)

Xp = A / (s+2) + (B*s + C)/(s**2 + 6*s + 34)
coeffs = sym.solve(sym.Eq(X, Xp), (A, B, C))
coeffs
```

Introducing the coefficients into $X_p(s)$ yields

```python
Xp = Xp.subs(coeffs)
Xp
```

The first fraction belongs to the complex conjugate poles. Applying [completion of the square](https://en.wikipedia.org/wiki/Completing_the_square) to the denominator, its inverse can be identified in the [table of Laplace transforms](table_theorems_transforms.ipynb#Transforms) as exponentially decaying cosine signal. Performing the inverse Laplace transform with `SymPy` yields

```python
x1 = sym.inverse_laplace_transform(Xp.args[1], s, t)
x1
```

The second fraction belongs to a real-valued pole of first degree. Its inverse Laplace transform can be looked-up directly in the [table of Laplace transforms](table_theorems_transforms.ipynb#Transforms) as exponentially decaying signal. Performing the inverse Laplace transform again with `SymPy` yields

```python
x2 = sym.inverse_laplace_transform(Xp.args[0], s, t)
x2
```

The inverse Laplace transform of $X(s)$ is given by summing up these two parts

```python
x = x1 + x2
x
```

The resulting signal is plotted for illustration

```python
sym.plot(x, (t, -0.1, 4), xlabel='$t$', ylabel='$x(t)$');
```

The same result can be derived directly from $X(s)$ by using the inverse Laplace transform of `SymPy`

```python
sym.inverse_laplace_transform(X, s, t).simplify()
```

**Exercise**

* Derive the inverse Laplace transform of $X(s)$ by manual calculation.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Project "Training a cat to catch a Mouse using Q-learning and Qiskit"

Participants:

## 1. Intoduction
### 1.1 Idea
The idea of this project is to train a novice(single agent) to catch a mouse in a grid environment 3*3.

We placed the cat in the lower right corner and the mouse in the upper left corner. Our task is to find the best way for the cat to catch a motionless (and unsuspecting danger) mouse. We want to test whether it is possible to solve such a problem using quantum computing.

    MOUSE
    EMPTY
    EMPTY

    EMPTY
    EMPTY
    EMPTY

    EMPTY
    EMPTY
    CAT

### 1.2 Q-Learning
Q-Learning is a reinforcement learning algorithm to learn the value of an action in a particular state.
When Q-learning is performed we create what’s called a **q-table** or matrix that follows the shape of *[state, action]* and we initialize our values to zero. We then update and store our q-values after an episode. This q-table becomes a reference table for our agent to select the best action based on the q-value.

An agent interacts with the environment in 1 of 2 ways. The first is to use the q-table as a reference and view all possible actions for a given state. The agent then selects the action based on the max value of those actions. This is known as **exploiting** since we use the information we have available to us to make a decision.

The second way to take action is to act randomly. This is called **exploring**. Instead of selecting actions based on the max future reward we select an action at random. Acting randomly is important because it allows the agent to explore and discover new states that otherwise may not be selected during the exploitation process. You can balance exploration/exploitation using epsilon (ε) and setting the value of how often you want to explore vs exploit. Here’s some rough code that will depend on how the state and action space are setup.

Q-table is updated in every episode of training via **Bellman Equation**:

$$
\begin{align}
\underbrace{\text{New}Q(s,a)}_{\scriptstyle\text{New Q-Value}}=Q(s,a)+\mkern-34mu\underset{\text{New Q-Value}}{\underset{\Bigl|}{\alpha}}\mkern-30mu \underbrace{R(s,a)}_{\scriptstyle\text{Reward}}+ \mkern-30mu\underset{\text{Discount rate}}{\underset{\Biggl |}{\gamma}}\mkern-75mu * \overbrace{\max Q'(s',a')}^{\scriptstyle\substack{\text{Maximum predicted reward, given} \\ \text{new state and all possible actions}}} * \mkern-45mu-Q(s,a)
\end{align}
$$

### 1.3 Variational Quantum Eigensolver (VQE)

VQE is an application of the variational method of quantum mechanics.
$$
\begin{align}
    \lambda_{min} \le \langle H \rangle_{\psi} = \langle \psi | H | \psi \rangle = \sum_{i = 1}^{N} \lambda_i | \langle \psi_i | \psi\rangle |^2
\end{align}
$$
The above equation is known as the **variational method**.

When the Hamiltonian of a system is described by the Hermitian matrix $H$ the ground state energy of that system, $E_{gs}$, is the smallest eigenvalue associated with $H$. By arbitrarily selecting a wave function $|\psi \rangle$ (called an *ansatz*) as an initial guess approximating $|\psi_{min}\rangle$, calculating its expectation value, $\langle H \rangle_{\psi}$, and iteratively updating the wave function, arbitrarily tight bounds on the ground state energy of a Hamiltonian may be obtained.

A systematic approach to varying the ansatz is required to implement the variational method on a quantum computer. VQE does so through the use of a parameterized circuit with a fixed form. Such a circuit is often called a *variational form*, and its action may be represented by the linear transformation $U(\theta)$. A variational form is applied to a starting state $|\psi\rangle$ (such as the vacuum state $|0\rangle$, or the Hartree Fock state) and generates an output state $U(\theta)|\psi\rangle\equiv |\psi(\theta)\rangle$. Iterative optimization over $|\psi(\theta)\rangle$ aims to yield an expectation value $\langle \psi(\theta)|H|\psi(\theta)\rangle \approx E_{gs} \equiv \lambda_{min}$. Ideally, $|\psi(\theta)\rangle$ will be close to $|\psi_{min}\rangle$ (where 'closeness' is characterized by either state fidelity, or Manhattan distance) although in practice, useful bounds on $E_{gs}$ can be obtained even if this is not the case.

## 2. Solution scheme via Q-learning & Quantum Calculations

The scheme of our solution a as follows: Each point of our medium of size 3*3 corresponds to a quantum circuit with configurable parameters. Each such scheme consists of several U3-gates, the type of which is shown below.

$$
\begin{align}
    U3(\theta, \phi, \lambda) = \begin{pmatrix}\cos(\frac{\theta}{2}) & -e^{i\lambda}\sin(\frac{\theta}{2}) \\ e^{i\phi}\sin(\frac{\theta}{2}) & e^{i\lambda + i\phi}\cos(\frac{\theta}{2}) \end{pmatrix}
\end{align}
$$

One of the options for implementing such a quantum circuit looks like this:

```python
from copy import deepcopy
import numpy as np
import random
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit import Aer, transpile, assemble
from qiskit.providers import backend
from qiskit.aqua.components.optimizers import COBYLA
import matplotlib.pyplot as plt
import itertools
```

### 2.3 Basic Classes

**State**

This class is needed for describing the position of the cat and operating with it.

**GridWorld**

We use this for operating with out grid where the cat and the mouse are located.

**QNet**

Here we can perfrom all the quantum computation needed for updating the q-tables in each point.

**Cat**

This class describes the cat's actions and update q-circuits accordingly.

**PetSchool**

In this class we can train our model and show the results.

```python
# TYPES:
CAT = "c"
MOUSE = "m"
EMPTY = "emp"

# ACTIONS:
UP = "000"
DOWN = "001"
LEFT = "010"
RIGHT = "011"

UPPERLEFT = "100"
UPPERRIGHT = "101"
LOWERLEFT = "110"
LOWERRIGHT = "111"

ACTIONS = [UP, DOWN, LEFT, RIGHT, UPPERLEFT, UPPERRIGHT, LOWERLEFT, LOWERRIGHT]

# random seed
seed = 10
random.seed(seed)
np.random.seed(seed)
```

```python
# state of cat
class State:
    def __init__(self, catP, mouseP):
        self.catP = catP
        self.mouseP = mouseP

    def __eq__(self, other):
        return isinstance(other, State) and self.mouseP==other.mouseP and self.catP == other.catP

    def __hash__(self):
        return hash(str(self.catP))

    def __str__(self):
        return f"State(cat_pos={self.catP})"
```

```python
# GridWorld
# e.g.
#  MOUSE | EMPTY | EMPTY
#  EMPTY | EMPTY | EMPTY
#  EMPTY | EMPTY | CAT
class GridWorld:
    def __init__(self, s, catP, mouseP):
        self.numRows = s[0]
        self.numColumns = s[1]
        self.catP = catP
        self.mouseP = mouseP
        assert(not self.compaireList(self.catP, self.mouseP))

    def getItem(self, p):
        if p[0]>=self.numRows or p[0]<0:
            return None
        if p[1]>=self.numColumns or p[1]<0:
            return None
        if self.compaireList(p, catP):
            return CAT
        elif self.compaireList(p, mouseP):
            return MOUSE
        else:
            return EMPTY

    def compaireList(self, l1,l2):
        for i, j in zip(l1, l2):
            if i!=j:
                return False
        return True

    def getNumRows(self):
        return self.numRows

    def getNumColumns(self):
        return self.numColumns

    def getMouseP(self):
        return self.mouseP

    def getCatP(self):
        return self.catP

    def setCatP(self, p):
        self.catP = p

    def setMouseP(self, p):
        self.mouseP = p

    def initCatState(self, rd = False):
        # init cat position
        if not rd:
            catP = [self.getNumRows() - 1, self.getNumColumns() - 1]
        else:
            catP = [random.randint(0, self.getNumRows()), random.randint(0, self.getNumColumns())]
            while self.getItem(catP) != EMPTY and self.getItem(catP) != CAT:
                catP = [random.randint(0, self.getNumRows()), random.randint(0, self.getNumColumns())]
        self.setCatP(catP)
        return State(catP, self.getMouseP())

    def show(self):
        output = ""
        for i in range(self.numRows):
            for j in range(self.numColumns):
                if self.compaireList([i,j], self.catP):
                    output += CAT + " "
                if self.compaireList([i,j], self.mouseP):
                    output += MOUSE + " "
                if not self.compaireList([i,j], self.catP) and not self.compaireList([i,j], self.mouseP):
                    output += EMPTY + " "
            output += "\n"
        print(output)
```

```python
# QNet
class QNet:

    def __init__(self, qTable, gridWorld:GridWorld, alpha=0.1, gamma=1.0, eps=0.2, actions=[UP, DOWN, LEFT, RIGHT], numParams=9):
        self.gw = gridWorld
        self.qt = qTable
        self.eps = eps
        self.backend = Aer.get_backend("qasm_simulator")
        self.NUM_SHOTS = 1000 # number of measurements
        self.optimizer = COBYLA(maxiter=500, tol=0.0001) # off the shelf
        self.gamma = gamma
        self.alpha = alpha
        self.ACTIONS = actions

        # self.rets = {(0,0):([0,..,0],0.0,0), ...}
        self.rets = dict() # resulting parameters after optimization for all points in the grid

        self.state = None

        for i in range(self.gw.getNumRows()):
            for j in range(self.gw.getNumColumns()):
                for m in range(self.gw.getNumRows()):
                    for n in range(self.gw.getNumColumns()):
                        self.rets[i, j, m, n] = (np.random.rand(numParams), 0.0, 0)

    def qcMaker(self, params):
        qr = QuantumRegister(3, name="q")
        cr = ClassicalRegister(3, name="c")
        qc = QuantumCircuit(qr, cr)
        start = 0
        step = 3
        assert(not len(params)%3)
        # add U3 for all qubits once
        for i in range(int(len(params)/3)):
            qc.u3(params[start], params[start + 1], params[start + 2], qr[i])
            start += step
        # qc.cx(qr[0], qr[1]) # create entanglement
        qc.measure(qr, cr)
        return qc

    def newPosition(self, state, action):
        p = deepcopy(state.catP)
        if action == UP:
            p[0] = max(0, p[0] - 1)
        elif action == DOWN:
            p[0] = min(self.gw.getNumRows() - 1, p[0]+1)
        elif action == LEFT:
            p[1] = max(0, p[1] - 1)
        elif action == RIGHT:
            p[1] = min(self.gw.getNumColumns() - 1, p[1] + 1)
        elif action == UPPERLEFT:
            p[0] = max(0, p[0] - 1)
            p[1] = max(0, p[1] - 1)
        elif action == UPPERRIGHT:
            p[0] = max(0, p[0] - 1)
            p[1] = min(self.gw.getNumColumns() - 1, p[1] + 1)
        elif action == LOWERLEFT:
            p[0] = min(self.gw.getNumRows() - 1, p[0] + 1)
            p[1] = max(0, p[1] - 1)
        elif action == LOWERRIGHT:
            p[0] = min(self.gw.getNumRows() - 1, p[0] + 1)
            p[1] = min(self.gw.getNumColumns() - 1, p[1] + 1)
        else:
            raise ValueError(f"Unkown action {action}")
        return p

    def getReward(self, p):
        grid = self.gw.getItem(p)
        if grid == EMPTY:
            reward = -1
        elif grid == MOUSE:
            reward = 1000
        elif grid == CAT:
            reward = -1
        else:
            raise ValueError(f"Unknown grid item {grid}")
        return reward

    def selectAction(self, state, training):
        if random.uniform(0, 1) < self.eps:
            return random.choice(self.ACTIONS)
        else:
            if training:
                self.state = state
                self.updateCircuit(state)
            return self.ACTIONS[np.argmax(self.qt[state.catP[0], state.catP[1], state.mouseP[0], state.mouseP[1]])]

    def lossFunction(self, params):
        action = ""
        qc = self.qcMaker(params=params)
        t_qc = transpile(qc, self.backend)
        job = assemble(t_qc, shots=self.NUM_SHOTS)
        rlt = self.backend.run(job).result()
        counts = rlt.get_counts(qc)
        # speedup training, cross the ravine
        if random.uniform(0, 1) < self.eps:
            action = random.choice(self.ACTIONS)
        else:
            action = max(counts, key = counts.get)
        nextPosition = self.newPosition(self.state, action)
        reward = self.getReward(nextPosition)
        targetQvalue = reward + self.gamma *  np.max(self.qt[nextPosition[0],nextPosition[1], self.state.mouseP[0], self.state.mouseP[1]])
        predictedQvalue = self.calculateQvalue(action, nextPosition, reward, targetQvalue, self.state)

        # update q-table
        self.updateQtable(predictedQvalue, action)
        return targetQvalue - self.qt[self.state.catP[0], self.state.catP[1], self.state.mouseP[0], self.state.mouseP[1]][int(action,2)]

    def updateQtable(self, predictedQvalue, action):
        if self.qt[self.state.catP[0], self.state.catP[1], self.state.mouseP[0], self.state.mouseP[1]][int(action,2)] < predictedQvalue:
            self.qt[self.state.catP[0], self.state.catP[1], self.state.mouseP[0], self.state.mouseP[1]][int(action,2)] = predictedQvalue

    def calculateQvalue(self, action, nextPosition, reward, targetQvalue, state:State):
        return self.qt[state.catP[0], state.catP[1], state.mouseP[0], state.mouseP[1]][int(action,2)] + self.alpha * (targetQvalue - self.qt[state.catP[0],state.catP[1],state.mouseP[0], state.mouseP[1]][int(action,2)]) # update q-table

    def updateCircuit(self, state:State):
        self.rets[state.catP[0], state.catP[1], state.mouseP[0], state.mouseP[1]] = self.optimizer.optimize(num_vars=9, objective_function=self.lossFunction, initial_point=self.rets[state.catP[0], state.catP[1], state.mouseP[0], state.mouseP[1]][0])

    def setAlpha(self, alpha):
        self.alpha = alpha
```

```python
# Agent: cat
class Cat:
    def __init__(self, qNet: QNet, training=True, eps = 0.2, actions = [UP, DOWN, LEFT, RIGHT]):
        self.eps = eps
        self.training = training
        self.qNet = qNet
        self.ACTIONS = actions
        self.state = None

    def newPosition(self, state, action):
            p = deepcopy(state.catP)
            if action == UP:
                p[0] = max(0, p[0] - 1)
            elif action == DOWN:
                p[0] = min(self.qNet.gw.getNumRows() - 1, p[0] + 1)
            elif action == LEFT:
                p[1] = max(0, p[1] - 1)
            elif action == RIGHT:
                p[1] = min(self.qNet.gw.getNumColumns() - 1, p[1] + 1)
            # in case of tough boundary then slipe
            elif action == UPPERLEFT:
                p[0] = max(0, p[0] - 1)
                p[1] = max(0, p[1] - 1)
            elif action == UPPERRIGHT:
                p[0] = max(0, p[0] - 1)
                p[1] = min(self.qNet.gw.getNumColumns() - 1, p[1] + 1)
            elif action == LOWERLEFT:
                p[0] = min(self.qNet.gw.getNumRows() - 1, p[0] + 1)
                p[1] = max(0, p[1] - 1)
            elif action == LOWERRIGHT:
                p[0] = min(self.qNet.gw.getNumRows() - 1, p[0] + 1)
                p[1] = min(self.qNet.gw.getNumColumns() - 1, p[1] + 1)
            else:
                raise ValueError(f"Unkown action {self.ACTIONS[action]}")
            return p

    def getReward(self, p):
        grid = self.qNet.gw.getItem(p)
        if grid == MOUSE:
            reward = 1000
            end = True
        elif grid == EMPTY:
            reward = -1
            end = False
        elif grid == CAT:
            reward = -1
            end = False
        else:
            raise ValueError(f"Unknown grid item {grid}")
        return reward, end

    def act(self, state, action):
        p = self.newPosition(state, action)
        reward, end = self.getReward(p)
        return p, reward, end

    def updateQtable(self, action, p, reward, state):
        pqv = self.qNet.calculateQvalue(action, p, reward, state)
        self.qNet.updateQtable(pqv, action)

    def setTraining(self, training):
        self.Training = training
```

```python
# The pet school
class PetSchool:
    def __init__(self, cat:Cat, numEpisodes, maxEpisodeSteps, training=True, minAlpha = 0.02, eps = 0.2):
        self.cat = cat
        self.training = training
        self.NUM_EPISODES = numEpisodes
        self.MAX_EPISODE_STEPS = maxEpisodeSteps
        self.alphas = np.linspace(1.0, minAlpha, self.NUM_EPISODES)
        self.eps = eps

    def train(self):
        counter = 0
        rd = True
        for e in range(self.NUM_EPISODES): #  episode: a rund for agent
            print("episode: ", e)
            if e > int(self.NUM_EPISODES/2):
                rd = False
            state = self.cat.qNet.gw.initCatState(rd=rd)
            self.cat.qNet.setAlpha(self.alphas[e])
            total_reward  = 0
            step = 0
            end = False
            for _ in range(self.MAX_EPISODE_STEPS): # step: a time step for agent
                action = self.cat.qNet.selectAction(deepcopy(state), self.training)
                p, reward, end = self.cat.act(state, action)
                self.catMoveTo(p)
                if not end: # stupid mouse may move to cat
                    end = self.mouseMove(self.cat.qNet.gw.getMouseP())
                    total_reward += 1000 # not update qtable
                total_reward += reward
                step += 1
                counter += 1
                if end:
                    print("catch the mouse!!!")
                    print("total reward: ", total_reward, "steps: ", step)
                    break
        print("counter: ", counter)

    def catMoveTo(self, p):
        self.cat.qNet.gw.setCatP(p)

    def show(self):
        self.cat.qNet.gw.show()
        print("qTable: ", self.cat.qNet.qt)
        print("\nparams: ", self.cat.qNet.rets)

    def initqTable(self, actions, size):
        d = {}
        for i in range(size[0]):
            for j in range(size[1]):
                d[i,j] = np.zeros(len(actions))
        return d

    # @Daniel-Molpe
    def mouseMove(self, oldPos, p=0.5): # goal (mouse) moves randomly with prob p every time the cat moves
        side = min(self.cat.qNet.gw.getNumColumns(), self.cat.qNet.gw.getNumRows()) # Number of cells per side of the grid
        end = False
        if np.random.random() < p:
            n = np.random.random()
            if n < 0.25:
                newPos = (max(0, oldPos[0]-1), oldPos[1])
            elif n < 0.5:
                newPos = (min(side - 1, oldPos[0]+1),oldPos[1])
            elif n < 0.75:
                newPos = (oldPos[0], max(0, oldPos[1]-1))
            else:
                newPos = (oldPos[0], min(side - 1, oldPos[1]+1))
        else:
            newPos = oldPos
        self.cat.qNet.gw.setMouseP(newPos)
        if self.cat.qNet.gw.getCatP == newPos: # mouse ends the training
            end = True
        return end
```

```python
# super parameter
gridSize = [3, 3]
catP = [gridSize[0]-1, gridSize[0]-1]
mouseP = [0, 0]
EPS = 20 # 500
MAX_EPS_STEP = 30 # 100
sizeOfParams = 9
gamma = 0.98
```

```python
def initqTable(size, actions=[UP, DOWN, LEFT, RIGHT]):
    d = {}
    for i in range(size[0]):
        for j in range(size[1]):
            for m in range(size[0]):
                for n in range(size[1]):
                    d[i, j, m, n] = np.zeros(len(actions))
    return d

# initGridWorld
gridWorld = GridWorld(gridSize, catP=catP, mouseP=mouseP)
# init q Table
qt = initqTable(gridSize, actions=ACTIONS)
# init q Circuit
qNet = QNet(qt, gridWorld, gamma=gamma, actions=ACTIONS)
# init cat
cat = Cat(qNet=qNet)
# init pet school
petSchool = PetSchool(cat, EPS, MAX_EPS_STEP)
# start training
petSchool.train()
# show what have been learned
petSchool.show()
```

    episode:  0
    catch the mouse!!!
    total reward:  1000 steps:  1
    episode:  1
    episode:  2
    episode:  3
    episode:  4
    episode:  5
    episode:  6
    episode:  7
    episode:  8
    episode:  9
    catch the mouse!!!
    total reward:  1000 steps:  1
    episode:  10
    episode:  11
    episode:  12
    episode:  13
    episode:  14
    episode:  15
    episode:  16
    episode:  17
    episode:  18
    episode:  19
    counter:  542
    emp m emp
    emp emp c
    emp emp emp

    qTable:  {(0, 0, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 0, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 1, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 1, 2): array([512.59221053,   0.        ,   0.        , 512.59221037,
             0.        , 512.59221014, 524.07368421,   0.        ]), (0, 2, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (0, 2, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 0, 0): array([1000.,    0.,  979.,    0., 1000.,    0.,    0.,    0.]), (1, 0, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 0, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 1, 2): array([  0.        ,   0.        ,   0.        ,   0.        ,
           535.78947368,   0.        ,   0.        ,   0.        ]), (1, 1, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 1, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (1, 2, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 0, 0): array([979.  , 958.42, 958.42,   0.  , 979.  ,   0.  , 958.42,   0.  ]), (2, 0, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 0, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 1, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 1, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 0, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 0, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 0, 2): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 1, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 1, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 1, 2): array([  0.        , 512.59208419,   0.        , 512.59177302,
           524.07363811,   0.        ,   0.        , 512.59123879]), (2, 2, 2, 0): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 2, 1): array([0., 0., 0., 0., 0., 0., 0., 0.]), (2, 2, 2, 2): array([0., 0., 0., 0., 0., 0., 0., 0.])}

    params:  {(0, 0, 0, 0): (array([0.77132064, 0.02075195, 0.63364823, 0.74880388, 0.49850701,
           0.22479665, 0.19806286, 0.76053071, 0.16911084]), 0.0, 0), (0, 0, 0, 1): (array([0.08833981, 0.68535982, 0.95339335, 0.00394827, 0.51219226,
           0.81262096, 0.61252607, 0.72175532, 0.29187607]), 0.0, 0), (0, 0, 0, 2): (array([0.91777412, 0.71457578, 0.54254437, 0.14217005, 0.37334076,
           0.67413362, 0.44183317, 0.43401399, 0.61776698]), 0.0, 0), (0, 0, 1, 0): (array([0.51313824, 0.65039718, 0.60103895, 0.8052232 , 0.52164715,
           0.90864888, 0.31923609, 0.09045935, 0.30070006]), 0.0, 0), (0, 0, 1, 1): (array([0.11398436, 0.82868133, 0.04689632, 0.62628715, 0.54758616,
           0.819287  , 0.19894754, 0.8568503 , 0.35165264]), 0.0, 0), (0, 0, 1, 2): (array([0.75464769, 0.29596171, 0.88393648, 0.32551164, 0.1650159 ,
           0.39252924, 0.09346037, 0.82110566, 0.15115202]), 0.0, 0), (0, 0, 2, 0): (array([0.38411445, 0.94426071, 0.98762547, 0.45630455, 0.82612284,
           0.25137413, 0.59737165, 0.90283176, 0.53455795]), 0.0, 0), (0, 0, 2, 1): (array([0.59020136, 0.03928177, 0.35718176, 0.07961309, 0.30545992,
           0.33071931, 0.7738303 , 0.03995921, 0.42949218]), 0.0, 0), (0, 0, 2, 2): (array([0.31492687, 0.63649114, 0.34634715, 0.04309736, 0.87991517,
           0.76324059, 0.87809664, 0.41750914, 0.60557756]), 0.0, 0), (0, 1, 0, 0): (array([0.51346663, 0.59783665, 0.26221566, 0.30087131, 0.02539978,
           0.30306256, 0.24207588, 0.55757819, 0.56550702]), 0.0, 0), (0, 1, 0, 1): (array([0.47513225, 0.29279798, 0.06425106, 0.97881915, 0.33970784,
           0.49504863, 0.97708073, 0.44077382, 0.31827281]), 0.0, 0), (0, 1, 0, 2): (array([0.51979699, 0.57813643, 0.85393375, 0.06809727, 0.46453081,
           0.78194912, 0.71860281, 0.58602198, 0.03709441]), 0.0, 0), (0, 1, 1, 0): (array([0.35065639, 0.56319068, 0.29972987, 0.51233415, 0.67346693,
           0.15919373, 0.05047767, 0.33781589, 0.10806377]), 0.0, 0), (0, 1, 1, 1): (array([0.17890281, 0.8858271 , 0.36536497, 0.21876935, 0.75249617,
           0.10687958, 0.74460324, 0.46978529, 0.59825567]), 0.0, 0), (0, 1, 1, 2): (array([0.14762019, 0.18403482, 0.64507213, 0.04862801, 0.24861251,
           0.54240852, 0.22677334, 0.38141153, 0.92223279]), 0.0, 0), (0, 1, 2, 0): (array([0.92535687, 0.56674992, 0.53347088, 0.01486002, 0.97789926,
           0.5730289 , 0.791757  , 0.56155736, 0.87733524]), 0.0, 0), (0, 1, 2, 1): (array([0.58419583, 0.70884983, 0.14853345, 0.42845074, 0.69389007,
           0.10461974, 0.43960524, 0.16620215, 0.50697863]), 0.0, 0), (0, 1, 2, 2): (array([0.81903586, 0.09010673, 0.80006875, 0.56512635, 0.58934771,
           0.19810066, 0.43611826, 0.29590376, 0.03755768]), 0.0, 0), (0, 2, 0, 0): (array([0.03068484, 0.453105  , 0.74486408, 0.55729541, 0.3851136 ,
           0.1680728 , 0.83826132, 0.5990518 , 0.78271482]), -1.0, 55), (0, 2, 0, 1): (array([0.84850918, 0.60316298, 0.78106062, 0.61573688, 0.02116519,
           0.75046461, 0.17604213, 0.45851421, 0.51312271]), 0.0, 0), (0, 2, 0, 2): (array([0.48402089, 0.84438579, 0.17481389, 0.01463488, 0.84876407,
           0.74267458, 0.45669754, 0.41689841, 0.11672951]), -1.0, 55), (0, 2, 1, 0): (array([0.33867913, 0.09465904, 0.71583087, 0.0770854 , 0.20595026,
           0.57377623, 0.29383156, 0.65572674, 0.80356835]), 0.0, 0), (0, 2, 1, 1): (array([0.3512135 , 0.09344038, 0.81331608, 0.78486672, 0.39341911,
           0.86447919, 0.38403077, 0.25730289, 0.82940192]), 0.0, 0), (0, 2, 1, 2): (array([4.73636293, 1.50758696, 0.64437053, 0.21318099, 0.89573799,
           0.96604745, 0.317002  , 0.86566146, 0.3101917 ]), 1.1368683772161603e-13, 74), (0, 2, 2, 0): (array([0.02526395, 0.04919516, 0.18462684, 0.06903334, 0.25747542,
           0.91358173, 0.45784951, 0.13021179, 0.80989167]), 0.0, 0), (0, 2, 2, 1): (array([0.40346984, 0.02443264, 0.85683104, 0.27429482, 0.70910596,
           0.35577234, 0.79430906, 0.84461886, 0.53814752]), 0.0, 0), (0, 2, 2, 2): (array([0.55908652, 0.12250998, 0.37764192, 0.42874733, 0.51120865,
           0.89176257, 0.30021061, 0.39644188, 0.79327323]), 0.0, 0), (1, 0, 0, 0): (array([0.41225893, 0.18487628, 0.96527097, 0.56919789, 0.0600451 ,
           0.98892001, 0.22881785, 0.61471421, 0.36807373]), 0.0, 77), (1, 0, 0, 1): (array([0.80277594, 0.62756185, 0.82121648, 0.28825314, 0.27479513,
           0.60105189, 0.9385142 , 0.70729631, 0.26340316]), 0.0, 0), (1, 0, 0, 2): (array([0.12233592, 0.45872423, 0.9748125 , 0.68057437, 0.19325373,
           0.04673923, 0.95264394, 0.85864902, 0.82790318]), 0.0, 0), (1, 0, 1, 0): (array([0.96390052, 0.59893645, 0.93091584, 0.14676881, 0.11396031,
           0.3720289 , 0.76656013, 0.32114203, 0.63803508]), 0.0, 0), (1, 0, 1, 1): (array([0.0092211 , 0.54346504, 0.33775465, 0.89802431, 0.94070704,
           0.38412185, 0.39611779, 0.89727994, 0.05882237]), 0.0, 0), (1, 0, 1, 2): (array([0.40537929, 0.13017554, 0.08696941, 0.78396429, 0.30457336,
           0.58405587, 0.87846618, 0.88079206, 0.60095421]), 0.0, 0), (1, 0, 2, 0): (array([0.64522775, 0.0566282 , 0.2974841 , 0.69177461, 0.18902945,
           0.80300754, 0.51476375, 0.75728604, 0.17778875]), 0.0, 0), (1, 0, 2, 1): (array([0.08262029, 0.48207197, 0.52885388, 0.69630827, 0.20476161,
           0.67137233, 0.79326933, 0.04173781, 0.96335751]), 0.0, 0), (1, 0, 2, 2): (array([0.97539272, 0.55066078, 0.06490698, 0.34523679, 0.02042997,
           0.80085181, 0.2079271 , 0.14325251, 0.69989398]), 0.0, 0), (1, 1, 0, 0): (array([0.05794981, 0.25660757, 0.51033178, 0.99525878, 0.14651568,
           0.44951478, 0.60144038, 0.09727249, 0.28873488]), 0.0, 0), (1, 1, 0, 1): (array([0.7207999 , 0.55080606, 0.83857702, 0.58033135, 0.18457174,
           0.61550212, 0.88695509, 0.51678928, 0.62614367]), 0.0, 0), (1, 1, 0, 2): (array([0.50505912, 0.90966258, 0.41330466, 0.53546872, 0.34264313,
           0.12957678, 0.66282175, 0.93567   , 0.61272918]), 0.0, 0), (1, 1, 1, 0): (array([0.8427385 , 0.21790844, 0.90317174, 0.00976998, 0.22581273,
           0.13205513, 0.90754295, 0.91023064, 0.58105791]), 0.0, 0), (1, 1, 1, 1): (array([0.08749305, 0.11687944, 0.77107126, 0.73279527, 0.08712836,
           0.3574632 , 0.77321162, 0.1314716 , 0.53783243]), 0.0, 0), (1, 1, 1, 2): (array([0.75472357, 0.27252618, 0.566517  , 0.47668497, 0.55662069,
           0.44073706, 0.69373736, 0.71823718, 0.75638236]), -1.0, 56), (1, 1, 2, 0): (array([0.03732201, 0.6788354 , 0.4772208 , 0.10017139, 0.61419623,
           0.83791451, 0.73389404, 0.32156558, 0.06785531]), 0.0, 0), (1, 1, 2, 1): (array([0.03733024, 0.5591253 , 0.1608527 , 0.26788819, 0.23571196,
           0.01934746, 0.15168642, 0.03390251, 0.98183486]), 0.0, 0), (1, 1, 2, 2): (array([0.36048531, 0.82546453, 0.41223741, 0.25317915, 0.23671088,
           0.7731156 , 0.67125691, 0.70633089, 0.85315541]), 0.0, 0), (1, 2, 0, 0): (array([0.52204957, 0.44276965, 0.55374844, 0.65199485, 0.78855387,
           0.89226945, 0.30910516, 0.13560977, 0.75092665]), 0.0, 0), (1, 2, 0, 1): (array([0.52683322, 0.78473858, 0.4299261 , 0.83884659, 0.53735403,
           0.25042393, 0.05889401, 0.09194928, 0.17419994]), 0.0, 0), (1, 2, 0, 2): (array([0.43925435, 0.81093423, 0.91040212, 0.57609071, 0.29115443,
           0.65178637, 0.6379207 , 0.50357474, 0.96555272]), 0.0, 0), (1, 2, 1, 0): (array([0.80669115, 0.92851936, 0.61393513, 0.98200392, 0.71498796,
           0.00865927, 0.28814558, 0.84139866, 0.44317709]), 0.0, 0), (1, 2, 1, 1): (array([0.17304972, 0.11452711, 0.7978806 , 0.64890515, 0.501746  ,
           0.28564021, 0.63669583, 0.36834191, 0.53953854]), 0.0, 0), (1, 2, 1, 2): (array([0.7620668 , 0.83988748, 0.10495741, 0.51063131, 0.64548866,
           0.15306681, 0.00175965, 0.96704575, 0.80786653]), 0.0, 0), (1, 2, 2, 0): (array([0.27406759, 0.27908961, 0.12067964, 0.48068826, 0.93230198,
           0.91492743, 0.2438751 , 0.3948583 , 0.97023953]), -1.0, 55), (1, 2, 2, 1): (array([0.27561963, 0.93784944, 0.76901216, 0.88471869, 0.87100498,
           0.14647096, 0.78918879, 0.90922225, 0.83817055]), 0.0, 0), (1, 2, 2, 2): (array([0.71203119, 0.30737942, 0.35378455, 0.2363534 , 0.08981164,
           0.19447817, 0.61580788, 0.93208368, 0.69108395]), 0.0, 0), (2, 0, 0, 0): (array([ 3.56260035,  3.37823743,  1.5271986 ,  1.77509479,  0.29520735,
            0.18319154, -0.01162405,  2.74356531,  1.038326  ]), -1.0, 72), (2, 0, 0, 1): (array([0.43367967, 0.83208476, 0.01345856, 0.50933539, 0.4795487 ,
           0.02965494, 0.01142853, 0.331844  , 0.06143108]), 0.0, 0), (2, 0, 0, 2): (array([0.59817351, 0.88592025, 0.41213378, 0.0382722 , 0.0808424 ,
           0.08191454, 0.80936733, 0.800694  , 0.01690981]), 0.0, 0), (2, 0, 1, 0): (array([0.86129433, 0.91130022, 0.40087668, 0.2868907 , 0.84304489,
           0.61704455, 0.14059873, 0.22875561, 0.84149797]), 0.0, 0), (2, 0, 1, 1): (array([0.84435847, 0.22014599, 0.81113764, 0.53595353, 0.61042845,
           0.52386988, 0.63962915, 0.00515635, 0.02473878]), 0.0, 0), (2, 0, 1, 2): (array([0.69002639, 0.17159302, 0.69393067, 0.28526562, 0.05510592,
           0.5822229 , 0.29581873, 0.32281614, 0.96698939]), 0.0, 0), (2, 0, 2, 0): (array([0.19493466, 0.34739651, 0.10401407, 0.40178185, 0.09284765,
           0.50484381, 0.74259074, 0.00529051, 0.09016117]), 0.0, 0), (2, 0, 2, 1): (array([0.05618398, 0.88432566, 0.98558586, 0.52017469, 0.19841743,
           0.93334832, 0.6545982 , 0.79179175, 0.51121568]), -1.0, 55), (2, 0, 2, 2): (array([0.78821189, 0.33479673, 0.11325313, 0.90481398, 0.00422391,
           0.07350388, 0.13229136, 0.94214992, 0.30292969]), 0.0, 0), (2, 1, 0, 0): (array([0.21331305, 0.25566255, 0.09592007, 0.40179788, 0.73102764,
           0.64820994, 0.25084558, 0.22743891, 0.49581616]), 0.0, 0), (2, 1, 0, 1): (array([0.86476497, 0.63406375, 0.75449247, 0.43712319, 0.40021209,
           0.72291631, 0.37236858, 0.55259803, 0.08046362]), 0.0, 0), (2, 1, 0, 2): (array([0.59427557, 0.28092147, 0.74244759, 0.2988272 , 0.75753668,
           0.22636014, 0.00440764, 0.13978028, 0.2957637 ]), 0.0, 0), (2, 1, 1, 0): (array([0.19559382, 0.54888204, 0.20999217, 0.77849217, 0.4846799 ,
           0.63934039, 0.16769631, 0.9246826 , 0.71733995]), 0.0, 0), (2, 1, 1, 1): (array([0.15481361, 0.72363361, 0.18640993, 0.82966457, 0.19664176,
           0.96130067, 0.54759758, 0.2228636 , 0.21272669]), 0.0, 0), (2, 1, 1, 2): (array([0.62798796, 0.63327613, 0.30327904, 0.81636775, 0.41905672,
           0.81385829, 0.26521517, 0.98342547, 0.87230532]), -1.0, 55), (2, 1, 2, 0): (array([0.44852459, 0.33538451, 0.02970605, 0.99093675, 0.68094952,
           0.18915819, 0.04504555, 0.01313955, 0.66181062]), 0.0, 0), (2, 1, 2, 1): (array([0.10751234, 0.30995764, 0.30794745, 0.11747223, 0.94693142,
           0.97739055, 0.90447925, 0.79618015, 0.06542942]), 0.0, 0), (2, 1, 2, 2): (array([0.71703863, 0.45591043, 0.15148371, 0.6467202 , 0.02923457,
           0.75352087, 0.88973295, 0.01038004, 0.32742784]), -1.0, 55), (2, 2, 0, 0): (array([0.82849109, 0.11259759, 0.02469574, 0.81435353, 0.24890527,
           0.2491966 , 0.1582368 , 0.27605595, 0.42871035]), 0.0, 0), (2, 2, 0, 1): (array([0.25842181, 0.36694704, 0.34929559, 0.33303578, 0.3033672 ,
           0.31512729, 0.27022518, 0.90029462, 0.9669742 ]), 0.0, 0), (2, 2, 0, 2): (array([0.89669855, 0.37327538, 0.64827922, 0.31144182, 0.8099721 ,
           0.8047729 , 0.79441769, 0.11143517, 0.81159814]), -1.0, 55), (2, 2, 1, 0): (array([0.77955234, 0.88508016, 0.32242105, 0.10716543, 0.16124976,
           0.67278636, 0.71414989, 0.2372342 , 0.80571737]), 0.0, 0), (2, 2, 1, 1): (array([0.9182963 , 0.98462185, 0.99704715, 0.51592396, 0.70008431,
           0.07655878, 0.9011415 , 0.81358265, 0.35906171]), -1.0, 55), (2, 2, 1, 2): (array([0.97570374, 0.3859861 , 1.01620939, 0.01867446, 0.54413954,
           0.44314632, 0.02560015, 0.36840451, 0.42012051]), -1.0, 64), (2, 2, 2, 0): (array([0.40540396, 0.99807766, 0.15667538, 0.43695891, 0.27023265,
           0.96896344, 0.58980904, 0.70418642, 0.64993667]), -1.0, 55), (2, 2, 2, 1): (array([0.75540708, 0.88531427, 0.25305273, 0.22799381, 0.60264171,
           0.23616416, 0.7309863 , 0.66399892, 0.23927492]), -1.0, 55), (2, 2, 2, 2): (array([0.18591473, 0.31277572, 0.36632897, 0.58990679, 0.75923894,
           0.32235078, 0.50202215, 0.58383107, 0.26660088]), -1.0, 55)}


```
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
```

```
scipy.stats.bernoulli?
```

    Object `scipy.stats.bernoulli` not found.

##Sprint 3 Prep Notebook Overview

Topics that may show up on Sprint Challenge 3:

Probability, Statistics, and Inference
- demonstrate understanding of fundamental concepts of set theory and probability,
  including permutations, combinations, expected value, variance, binomial distribution,
  and Bernoulli trial

- explain the importance of statistics in the context of informing a practical
  and reliable understanding of data

Sampling, Confidence Int, and Hypo Testing
- use hypothesis testing to determine an outcome is significant or just the
  result of chance

- determine the level of confidence for a possible outcome

Introduction to Bayesian Inference
- be able to derive Bayes' theorem from the Law of Conditional Probability
- calculate Bayesian confidence intervals

Real-World Experiment Design
- deploy a simple static (front-end) A/B-instrumented webpage
- evaluate and summarize findings from real-world A/B test data

###Fundamentla Concepts

**Set Theory**

A set is a collection of things (elements).
Example
The following sets are used in this book:

The set of natural numbers, ℕ={1,2,3,⋯}.

The set of integers, ℤ={⋯,−3,−2,−1,0,1,2,3,⋯}.

The set of rational numbers ℚ.

The set of real numbers ℝ.

Closed intervals on the real line. For example, [2,3] is the set of all real numbers x such that 2≤x≤3.

Open intervals on the real line. For example (−1,3) is the set of all real numbers x such that {−1<x<3}.

Similarly, [1,2) is the set of all real numbers x such that {1≤x<2.}

The set of complex numbers ℂ is the set of numbers in the form of a+bi, where a,b∈ℝ, and i=sqrt(-1).

**Probability:**

A probability is the *likelihood* of an event occurring.

We assign a probability measure P(A) to an event A. This is a value between 0 and 1 that shows how likely the event is. If P(A) is close to 0, it is very unlikely that the event A occurs. On the other hand, if P(A) is close to 1, A is very likely to occur.

P(A∩B)=P(A and B)=P(A,B),

P(A∪B)=P(A or B).

**Permutations:**

A permutation is a method to calculate the number of events occurring where order matters. To calculate a permutation, you will need to use the formula

nPr = n! / (n - r)!.

In this equation, n represents the number of items to choose from and r represents how many items are being chosen.

**Combinations:**

Combinations are a way to calculate the total outcomes of an event where order of the outcomes does not matter.

nCr = n! / r! * (n - r)!

Where n represents the total number of items, and r represents the number of items being chosen at a time.

**Expected Value:**

E(x)=∑ xi*P(xi)

In statistics and probability analysis, the expected value is calculated by multiplying each of the possible outcomes by the likelihood each outcome will occur and then summing all of those values.

Take, for example, a normal six-sided die. Once you roll the die, it has an equal one-sixth chance of landing on one, two, three, four, five, or six. Given this information, the calculation is straightforward:

(1/6 * 1) + (1/6 * 2) + (1/6 * 3) + (1/6 * 4) + (1/6 * 5) + (1/6 * 6) = 3.5

If you were to roll a six-sided die an infinite amount of times, you see the average value equals 3.5.

**Variance:**

Variance is the expectation of the squared deviation of a random variable from its mean.

Var(X) = Σx^2p − μ2

Informally, it measures how far a set of (random) numbers are spread out from their average value.

in code: {sample.var()}

**Binomial Distribution & Bernoulli Trial:**

A binomial experiment is a statistical experiment that has the following properties:

The experiment consists of n repeated trials.
Each trial can result in just two possible outcomes. We call one of these outcomes a success and the other, a failure.
The probability of success, denoted by P, is the same on every trial.
The trials are independent; that is, the outcome on one trial does not affect the outcome on other trials.
Consider the following statistical experiment. You flip a coin 2 times and count the number of times the coin lands on heads.

np.random.binomial(n, p, size=None)

or

scipy.stats.binom : probability density function, distribution or cumulative density function, etc.

binomial(n, p, size=None)

Draw samples from a binomial distribution.

Samples are drawn from a binomial distribution with specified
parameters, n trials and p probability of success where
n an integer >= 0 and p is in the interval [0,1]. (n may be
input as a float, but it is truncated to an integer in use)

A Bernoulli trial (or binomial trial) is a random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is conducted.

```
from IPython.display import display, Image

url = 'https://slideplayer.com/slide/5130463/16/images/2/Statistical+Inference.jpg'
example = Image(url=url, width=400)

display(example)
```

###Explain the importance of statistics

*How does statistics inform a pracitcal and reliable understanding of data?*

**Statistical Significance:**

Using hypothesis testing methods. If the p value is less than the  level (typically 0.05), then the results are statistically significant.

The effect observed in the sample was unlikely to have occured due to chance alone.

Strongly related to sample size. If the sample size is large enough, even tiny differences from the hypothesized value will be found statistically significant.

**Practical Significance**

Practical significance refers to the magnitude of the difference, which is known as the effect size. Results are practically significant when the difference is large enough to be meaningful in real life. What is meaningful may be subjective and may depend on the context.

In addition to the results of a hypothesis test, it is best to also report an appropriate confidence interval.

The interval illustrates the size of the effect, and can help determine if the effect has any practical importance.

##Sampling, Confidence Interval, & Hypothesis Testing (T-tests)

##Using hypothesis testing to determine if an outcome is significant or just the result of chance.

###Hypothesis Testing

The most important thing to remember is that its not up to you to pick the test, the question you want to answer is the one that picks the hypothesis test.

For example:

T-tests help you answer the question of whether the means of two samples are close to eachother. These are specifically designed to test for two means.

P-value tells you whether that level of closeness is statistically significant.

```
#we can determine this by checking for correlation (chi sq) between two sets
#two example below
#notice the P values
import numpy as np
from scipy.stats import chisquare  # One-way chi square test

# Chi square can take any crosstab/table and test the independence of rows/cols
# The null hypothesis is that the rows/cols are independent -> low chi square
# The alternative is that there is a dependence -> high chi square
# Be aware! Chi square does *not* tell you direction/causation

ind_obs = np.array([[1, 1], [2, 2]]).T
print(ind_obs)
print(chisquare(ind_obs, axis=None))

dep_obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
print(dep_obs)
print(chisquare(dep_obs, axis=None))
```

    [[1 2]
     [1 2]]
    Power_divergenceResult(statistic=0.6666666666666666, pvalue=0.8810148425137847)
    [[16 32]
     [18 24]
     [16 16]
     [14 28]
     [12 20]
     [12 24]]
    Power_divergenceResult(statistic=23.31034482758621, pvalue=0.015975692534127565)

```
#generating a random sample test...
from scipy.stats import normaltest
# Poisson models arrival times and is related to the binomial (coinflip)
#poisson(lambda, size) -> the higher to the right the further away the distribution moves
#so a lambda of 1, distribution is closer to 0, and with 5, moves away and to the right
sample = np.random.poisson(5, 1000)
sample
```

    array([ 5,  2,  0,  5,  6,  3,  7,  4,  4,  5,  5,  8,  7,  9,  7, 12,  1,
            3,  8,  6,  3,  5,  1,  4,  2,  4,  4,  8,  6,  6,  6,  6,  7,  0,
            4,  2,  6,  4,  8,  3,  3,  7,  3,  6,  2,  5,  2,  2,  6,  3,  2,
            3,  7,  9,  6,  7,  2,  3,  6,  2,  4,  2,  3,  5,  4,  8, 11,  5,
           11,  2,  8, 11,  7,  2,  5,  4,  7, 10,  6,  5,  3,  2,  1,  1,  7,
            9,  4,  3,  5,  5,  6,  7,  4,  3,  1,  7,  5,  5, 14,  1,  8,  0,
            6,  2,  7,  2,  8,  6,  2,  6,  2,  9,  5,  4,  7,  7,  7,  2,  4,
            8, 10,  7,  8,  4,  4,  5,  5,  6,  5,  2,  4,  5,  7,  4, 10,  4,
            3,  9,  2,  6,  4,  4,  4,  7,  6,  7,  4,  2,  7,  1,  4,  4,  7,
            5,  2,  8,  4,  6,  4,  6,  4,  5,  3,  5,  4,  3,  3,  5,  6,  7,
            2,  7,  8,  3,  8,  3,  7,  6,  3,  6,  4,  2,  6,  6,  5,  6,  3,
            6,  5,  9,  7,  4,  4,  5,  3,  6,  2,  4,  2,  7,  5,  6,  6,  5,
            4,  7,  8,  3,  6,  3, 10,  8,  5,  8,  5,  6,  3,  5,  4,  4,  6,
            4, 11,  5,  6,  5,  6,  4,  4,  5,  3,  5,  9,  3,  3,  7,  7,  5,
            3,  8,  8,  9,  7,  2,  3,  9,  6,  7,  5,  7,  2,  4,  2,  4,  8,
            2,  5,  4,  7,  5,  7,  3,  4, 11,  6,  0,  6,  4,  4,  3,  3,  1,
            6,  7,  4,  3,  9,  3,  5,  4,  7,  1,  7,  3,  4,  0,  5,  5,  8,
            9,  4,  3,  8,  5,  3,  8,  4,  2,  5,  2,  4,  4,  7,  4,  9,  4,
            7,  3,  6,  2,  6,  4,  6,  8,  4,  3,  1,  3,  6, 10,  8,  4,  5,
            3,  1,  5,  4,  4,  2,  1,  2,  5,  4,  7,  1,  5,  3,  6,  7,  6,
            7,  7,  4,  7,  4,  5,  6,  7,  5,  6,  4,  6,  6,  9,  4,  7,  0,
            5,  5,  2,  5,  4,  2,  3,  5,  4, 10,  3,  4,  4,  3,  5,  4,  8,
            4,  3,  4,  6,  4,  7,  7, 11,  4,  5,  3,  4,  5,  6,  4,  3,  7,
            4,  4,  5,  1, 11,  7,  9,  6,  5,  1,  3,  4,  6,  4,  3, 10,  8,
            4, 10,  5,  1,  3,  5,  7,  4, 10,  7,  3,  5,  2,  7,  1,  2,  5,
            5,  2,  5,  7,  3,  3,  2,  5,  3,  5,  5,  2,  6,  7,  6,  6,  3,
            8,  7,  5,  8,  5,  7,  5,  3,  4,  8,  4,  5,  1,  5,  4,  5,  7,
            3,  2,  7, 10,  7,  5,  7,  9,  3,  2,  7,  4,  3,  6,  4,  6,  6,
            1,  4,  4,  4,  3,  3,  3,  3,  5,  1, 11,  3,  4,  3,  4,  8,  4,
            4,  4,  7, 10,  6,  2,  6,  5, 12,  7,  2,  6,  3,  7,  5,  9,  8,
            7,  6,  6,  5,  6,  4,  6,  0,  1,  3,  4,  4,  5,  3,  4,  6,  4,
            4,  4,  3,  6,  4,  7,  3,  7,  9,  5,  8,  4,  5,  3,  6,  3,  5,
            7,  7,  3,  3,  4,  2,  8,  8,  3,  3,  7,  2,  6,  3,  2,  6,  2,
           11,  2,  3,  3,  1,  3,  6, 15,  5,  3,  5,  8,  5,  4,  9,  4,  3,
            9,  8,  3, 12,  5,  2,  6,  8,  2,  1,  6,  6,  6,  2,  5,  2,  6,
            2,  5,  3,  1,  5,  8,  8,  8,  7,  2,  6,  8,  6,  5,  5,  5,  4,
            3,  4,  8,  3,  6,  2,  1,  6, 10,  3,  6,  9,  6,  7,  6,  6,  5,
            5,  6,  5,  6,  7,  8,  5,  2,  5,  5,  3,  4,  8,  2,  8,  7,  3,
            3,  3,  4,  8,  5,  8,  6,  4,  5,  5,  3,  6,  4,  7,  3,  5,  4,
            3,  5,  4,  5,  4,  4,  7,  5,  7,  6,  6,  5,  8,  7,  5,  3,  3,
            2,  8,  6,  5,  4,  4,  6,  6,  3,  2,  5,  1,  7,  3,  3,  5,  6,
            5,  6,  3,  5,  4,  7,  9,  3,  7,  7,  6,  5,  4,  7,  4,  5,  6,
            4,  8,  3,  3,  3,  5,  5,  5,  1,  6,  5,  6,  6,  4,  6,  3,  7,
            7,  6,  3,  3,  4,  9,  4,  6,  5,  8,  5,  5,  7,  7,  6,  6,  5,
           12,  2, 12,  8,  2,  3,  8,  7,  7,  5,  8,  8,  5,  5,  5,  5,  4,
            8,  3,  2, 11,  4,  3,  6,  5,  7,  5,  7,  7, 10,  1,  5,  5,  5,
            7,  7,  2, 10,  0,  7,  2,  5,  2,  3,  6,  7,  6,  5,  8,  9,  5,
            7,  3,  7,  8,  3,  7,  4,  5,  7,  3,  6,  6,  4,  5,  7,  6,  6,
            4,  5,  3,  5,  9,  5,  2,  4,  1,  3,  7,  5,  4, 10,  7,  7,  6,
            2,  6,  6,  4,  6,  7,  9,  6,  4,  4, 15,  1,  2,  3,  7,  3,  6,
            8,  5,  5,  0,  5,  3,  3,  4,  7,  3,  5,  2,  5,  3,  5,  4,  6,
            5,  0,  6,  4,  8,  7,  7,  7,  6,  5,  4, 12,  8,  6,  4, 11,  7,
            4,  3,  3,  3,  5,  4,  6,  5,  5,  5,  6,  2,  2, 11,  5,  3,  6,
            5,  5,  3,  4,  7,  5,  5,  1,  5,  4,  3, 10,  3,  5,  3,  4,  4,
            7,  9,  4,  6,  6,  1,  7,  3,  4,  3,  4,  4,  3,  5,  3,  6,  2,
            4,  4,  3,  5,  5,  7,  4,  6, 12,  3,  4,  7,  6,  3,  4,  9,  7,
            7,  5,  9,  5,  6,  1,  3,  6,  2,  2,  2,  8,  3,  7,  3,  7,  4,
            1,  3,  6,  4,  4,  6,  2,  5,  7,  2,  8,  4,  8,  1,  5,  7,  6,
            3,  7,  3,  8,  4, 10,  6,  6,  5,  5,  4,  4,  3, 10])

```
#this data is not normally distributed
import matplotlib.pyplot as plt
plt.hist(sample);
```

```
#running a normality test on it, is the data normally distributed?
print(normaltest(sample))  # Pretty clearly not normal due to high p value
```

    NormaltestResult(statistic=53.45532530051912, pvalue=2.4678774905973204e-12)

```
#generating a normal distribution with a mean of 50 and std dev of 3
sample = np.random.normal(50, 3, 1000)
print(normaltest(sample))
```

    NormaltestResult(statistic=1.0348680605762306, pvalue=0.5960480285857734)

#T-Test Assumptions

Null hypothesis is that the means of the two distributions are the same, and running this hypothesis of P-value that is less than 0.05.

```
from scipy.stats import ttest_ind
#so lets make two data sets and check the p-value
skittles_fac_1 = np.random.normal(50, 5, size=100)
skittles_fac_1 = skittles_fac_1.round().astype(int)
skittles_fac_1[:10]
```

    array([52, 41, 51, 47, 42, 53, 49, 55, 52, 45])

```
#getting the mean
skittles_fac_1.mean()
```

    50.29

```
#first data set
plt.hist(skittles_fac_1, bins=15);
```

```
#second set of data or distribution
skittles_fac_2 = np.random.normal(49.3, 5.5, size=100)
skittles_fac_2 = skittles_fac_2.round().astype(int)
skittles_fac_2[:10]
```

    array([52, 44, 51, 45, 46, 53, 51, 54, 50, 44])

```
skittles_fac_2.mean()
```

    48.49

```
#plotting second data set
plt.hist(skittles_fac_2, bins=15);
```

```
#performing the t-test
ttest_ind(skittles_fac_1, skittles_fac_2)
```

    Ttest_indResult(statistic=2.3819263642990087, pvalue=0.018169499723603544)

#T-Test Conclusions

```
#note if you change the sample size, say make it smaller
#the p-value will increase because your deviations from the mean will be greater on average
#pulling the means away from eachother results in a lower p-value
#increasing std deviation will increase p-value

#if the p-value is less than 0.05, you can claim it as significant
#if the p-value is greater than 0.05, you can claim its the result of chance
```

#T-test doesn't say anything about the difference variances (standard deviations) - only their means

```
#illustrated here with two samples with same mean of 50
#np.random.normal(mean, std deviation, size)
skittles_fac_1 = np.random.normal(50, 2, size=5000)
skittles_fac_1 = skittles_fac_1.round().astype(int)
skittles_fac_1[:10]
skittles_fac_2 = np.random.normal(50, 4, size=5000)
skittles_fac_2 = skittles_fac_2.round().astype(int)
skittles_fac_2[:10]

#setting density=True normalizes the data so that the area underneath each is the same
plt.hist(skittles_fac_1, alpha=0.5, color='b', density=True)
plt.hist(skittles_fac_2, alpha=0.5, color='r', density=True);
```

```
#large p-value makes sense because our two means are the same
#this p-value doesn't allow us to reject the null hypothesis
#we operate under the assumption that the two means are the same
#we see that these two samples are obvisouly different though,
ttest_ind(skittles_fac_1, skittles_fac_2)
```

    Ttest_indResult(statistic=0.15054221339275364, pvalue=0.8803398856273693)

```
#if we want to detect and show that these sets are different
#we can use the KS test
#it takes this normalized historgram for two distributions
#and creates cumulative probability distribution for two samples
#and measures larger difference for two cumulative distributions
#based on this difference it computes the p-value

#null hypothesis is that the 2 distributions are the same
#the really low p-value shows that the 2 distributions are NOT the same
from scipy.stats import ks_2samp
ks_2samp(skittles_fac_1, skittles_fac_2)
```

    Ks_2sampResult(statistic=0.1634, pvalue=1.1715419467159992e-58)

```
#changing the data to make these more similar
skittles_fac_1 = np.random.normal(50, 3, size=5000)
skittles_fac_1 = skittles_fac_1.round().astype(int)
skittles_fac_1[:10]
skittles_fac_2 = np.random.normal(50, 3.3, size=5000)
skittles_fac_2 = skittles_fac_2.round().astype(int)
skittles_fac_2[:10]

#setting density=True normalizes the data so that the area underneath each is the same
plt.hist(skittles_fac_1, alpha=0.5, color='b', density=True)
plt.hist(skittles_fac_2, alpha=0.5, color='r', density=True);
```

```
#because we have very large sample sizes, we are still able to get the p-value less than 0.05
ks_2samp(skittles_fac_1, skittles_fac_2)
```

    Ks_2sampResult(statistic=0.0366, pvalue=0.002465208749609002)

```
#writing a function that creates two normal distributions
def get_two_norm_dist(mus, sigmas, sizes):
  #initializing an empty list, thats what we'll return
  dist_list = []
  #range 2 for 2 distributions
  for i in range(2):
    #distribution will generate first mu, sigma, and size
    dist = np.random.normal(mus[i], sigmas[i], size=sizes[i])
    #round it and turn it into an integer (may or may not be right thing to do depending on your data)
    dist = dist.round().astype(int)
    #append this to the list
    dist_list.append(dist)
  return dist_list

#now we can generate two distributions
dist_1, dist_2 = get_two_norm_dist(mus=(50, 50), sigmas=(3, 3.3), sizes=(1000, 500))

#and perform a ttest on them
ttest_ind(dist_1, dist_2)

#can not reject the null hypothesis because the P-value is large
```

    Ttest_indResult(statistic=0.28217542259767536, pvalue=0.7778479818587736)

##Unfair Die Example - T-test vs. KS-Test

```
unfair_die = np.array(150*[1] + 130*[2] + 50*[3] + 50*[4] + 130*[5] + 150*[6])
fair_die = np.array(110*[1] + 110*[2] + 110*[3] + 110*[4] + 110*[5] + 110*[6])
```

```
unfair_die.mean()
```

    3.5

```
fair_die.mean()
```

    3.5

```
#normal t-test comparing means
ttest_ind(unfair_die, fair_die) #This shows highest possible value for P
```

    Ttest_indResult(statistic=0.0, pvalue=1.0)

```
#ks test comparing means and distribution
ks_2samp(unfair_die, fair_die) #another example that this can detect differences in distribution
```

    Ks_2sampResult(statistic=0.09090909090909091, pvalue=0.008524685542396894)

##Central Limit Theorem
https://colab.research.google.com/drive/1HrCsKj9N3LYUqLiza2AVidg4izKY-i09#scrollTo=8V8RlyN4SaL0

#Confidence Intervals
##Determine the level of confidence for a possible outcome

Hypothesis testing is all about inferring some properties of the population, based on the sample.

Often times you don't know the size of the population, or the population is so large that you can't collect all the data about it.

One of the inferences is that the sample mean is close to the population mean.

Based on our sample data, we can say that the population mean lies within this range

```
#some madeup data... say 45% of people will vote for A, or B,... as an example
pop = pd.Series(np.random.binomial(1, p=0.45, size=1000000))
pop.sum()
```

    449476

```
from scipy import stats

def confidence_interval(data, confidence=0.95):
  """
  Calculate a confidence interval around a sample mean for given data.
  Using t-distribution and two-tailed test, default 95% confidence.

  Arguments:
    data - iterable (list or numpy array) of sample observations
    confidence - level of confidence for the interval

  Returns:
    tuple of (mean, lower bound, upper bound)
  """
  data = np.array(data)
  mean = np.mean(data)
  n = len(data)
  stderr = stats.sem(data)
  #this is a function that computes t for you
  t = stats.t.ppf((1 + confidence) / 2.0, n - 1)
  interval = stderr * t
  return (mean, mean - interval, mean + interval)

#returns mean and upper and lower bounds
```

```
#lets generate the confidence intervals manual first
#we'll do this with a sample size
sample_size = 100
sample = pop.sample(sample_size)
sample.head()
```

    563042    0
    100549    0
    270385    1
    134974    1
    905346    1
    dtype: int64

```
#lets find the mean, which tells us on average how these people will vote on average
sample_mean = sample.mean()
#remember to set the ddof to 1, it may make a difference here but mostly makes a difference in smaller sample sizes
sample_std = np.std(sample, ddof=1)
print(sample_mean, sample_std)
```

    0.45 0.49999999999999994

##Standard Error

```
#now we can find the standard error
standard_error = sample_std/np.sqrt(sample_size)
standard_error
```

    0.049999999999999996

##T-distribution table

```
#for T-distrubtions
#now we can get T, but where?
#they used to be found on tables found here
#http://uregina.ca/~gingrich/tt.pdf
#took 95% column and 100 row, found 1.984
#used 1.984 from 100 row, we don't have 100, but we have 99, so its close enough

#for Normal Distributions
#when samples get really large, use 1.96
#https://en.wikipedia.org/wiki/1.96

t = 1.984 # 95% confidence from table above
(sample_mean, sample_mean - t*standard_error, sample_mean + t*standard_error)
```

    (0.45, 0.3508, 0.5492)

```
#comparing to the function above, we get fairly close results
confidence_interval(sample, confidence=0.95)
```

    (0.45, 0.3507891524245659, 0.5492108475754341)

```
print('We can say with 95% confidence that on average, the population mean lies within 35%-54% with a mean of 45%.')
```

    We can say with 95% confidence that on average, the population mean lies within 35%-54% with a mean of 45%.

#Chi squared Test Example from Class Notes

```

```

##Probability Density Function

A function that gives you the density of probability for *continuous* distributions.

There are two types, continuous, or discrete.

https://en.wikipedia.org/wiki/Probability_density_function

##Cumulative Distribution Function
https://en.wikipedia.org/wiki/Cumulative_distribution_function

A good way to find the mean 'mu' of a normal distribution if you only have the CDF, is to look at where the area underneath the curve is 0.5.

**Summary:**

Probability Density Functions for: continuos distributions

Probability Mass Function for: discrete distributions

Cumulative Distiribution Function is independent of either.
For discrete it would look like steps up, for continuous it would be smooth.

#Derive Bayes Theorem from the Law of Conditional Probability
---

#Bayesian Inference

The general framework of bayesian inference is that it allows you to derive probabilities of certain events as new data and new information comes in. This can be very powerful.

###One event happening

By definition, the total probability of all outcomes (events) if some variable (event space) $A$ is 1. That is:

$$P(A) = \sum_n P(A_n) = 1$$

and

###Two events happening simultaneously

The law of total probability states:

$$P(A) = \sum_n P(A | B_n) P(B_n)$$

In other words - the total probability of  𝐴  is equal to the sum of the conditional probability of  𝐴  on any given event  𝐵𝑛  times the probability of that event  𝐵𝑛 , and summed over all possible events in  𝐵 .

##The Law of Conditional Probability

What's the probability of something conditioned on something else? To determine this we have to go back to set theory and think about the intersection of sets:

The formula for actual calculation:

$$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

Think of the overall rectangle as the whole probability space, $A$ as the left circle, $B$ as the right circle, and their intersection as the red area. Try to visualize the ratio being described in the above formula, and how it is different from just the $P(A)$ (not conditioned on $B$).

We can see how this relates back to the law of total probability - multiply both sides by $P(B)$ and you get $P(A|B)P(B) = P(A \cap B)$ - replaced back into the law of total probability we get $P(A) = \sum_n P(A \cap B_n)$.

This may not seem like an improvement at first, but try to relate it back to the above picture - if you think of sets as physical objects, we're saying that the total probability of $A$ given $B$ is all the little pieces of it intersected with $B$, added together. The conditional probability is then just that again, but divided by the probability of $B$ itself happening in the first place.

\begin{align}
P(A|B) &= \frac{P(A \cap B)}{P(B)}\\
\Rightarrow P(A|B)P(B) &= P(A \cap B)\\
P(B|A) &= \frac{P(B \cap A)}{P(A)}\\
\Rightarrow P(B|A)P(A) &= P(B \cap A)\\
\Rightarrow P(A|B)P(B) &= P(B|A)P(A) \\
P(A \cap B) &= P(B \cap A)\\
P(A|B) &= \frac{P(B|A) \times P(A)}{P(B)}
\end{align}

```
# 1) P(A and B) == P(B and A)
# 2) P(A given B) == P(A and B)/P(B) => P(A given B)*P(B) == P(A and B)
# 3) P(B given A) == P(B and A)/P(A) => P(B given A)*P(A) == P(B and A)

# P(A given B)*P(B) = P(B given A)*P(A)

# P(A given B) = P(B given A)*P(A)/P(B)
```

##Bayes Theorem Application - Drunk or Not?
What is the likelihood somebody really is drunk if they test positive? Some may guess it's 92% - the difference between the true positives and the false positives. But we have a prior belief of the background/true rate of drunk driving. Sounds like a job for Bayes' theorem!

$$
\begin{aligned}
P(Drunk | Positive) &= \frac{P(Positive | Drunk)P(Drunk)}{P(Positive)} \\
&= \frac{1 \times 0.001}{0.08} \\
&= 0.0125
\end{aligned}
$$

In other words, the likelihood that somebody is drunk given they tested positive with a breathalyzer in this situation is only 1.25% - probably much lower than you'd guess. This is why, in practice, it's important to have a repeated test to confirm (the probability of two false positives in a row is $0.08 * 0.08 = 0.0064$, much lower), and Bayes' theorem has been relevant in court cases where proper consideration of evidence was important.

```
#for the drunk example from class
def prob_drunk_given_positive(prob_drunk_prior, prob_positive, prob_positive_drunk):
    p_drunk = ((prob_positive_drunk * prob_drunk_prior) / (prob_positive)) * 100
    return p_drunk

print(prob_drunk_given_positive(1,.08,.001), '%')
```

    1.25 %

#99% Sensitive and 99% Specific
This example comes from [Wikipedia](https://en.wikipedia.org/wiki/Bayes%27_theorem)

Suppose that a test for using a particular drug is 99% sensitive and 99% specific. That is, the test will produce 99% true positive results for drug users and 99% true negative results for non-drug users. Suppose that 0.5% of people are users of the drug. What is the probability that a randomly selected individual with a positive test is a drug user?

Even if an individual tests positive, it is more likely that they do not use the drug than that they do. This is because the number of non-users is large compared to the number of users. The number of false positives outweighs the number of true positives. For example, if 1000 individuals are tested, there are expected to be 995 non-users and 5 users. From the 995 non-users, 0.01 × 995 ≃ 10 false positives are expected. From the 5 users, 0.99 × 5 ≈ 5 true positives are expected. Out of 15 positive results, only 5 are genuine.

The importance of specificity in this example can be seen by calculating that even if sensitivity is raised to 100% and specificity remains at 99% then the probability of the person being a drug user only rises from 33.2% to 33.4%, but if the sensitivity is held at 99% and the specificity is increased to 99.5% then the probability of the person being a drug user rises to about 49.9%.

```
#drug example from class

#4 parameters to pass in for the drug example
#Positive User
tru_pos = 0.99
#User
drug_user = 0.005
#Positive Non User
non_user = 0.01
#Non User
pos_user = .995

def prob_positive_given_random(tru_pos, drug_user, non_user, pos_user):
  p_user_plus = (tru_pos * drug_user) / ((tru_pos * drug_user) + (non_user * pos_user)) * 100
  return p_user_plus

print(round(prob_positive_given_random(tru_pos, drug_user, non_user, pos_user),3),'%')
```

    33.221 %

```
#attempting to loop this

#4 parameters to pass in for the drug example
#Positive User
tru_pos = 0.99
#User
drug_user = 0.005
#Positive Non User
non_user = 0.01
#Non User
pos_user = .995

def prob_positive_given_random(tru_pos, drug_user, non_user, pos_user):
  posterior_list = []
  p_user_plus = (tru_pos * drug_user) / ((tru_pos * drug_user) + (non_user * pos_user))
  posterior_list.append(p_user_plus)
  return p_user_plus
  return posterior_list

print(round(prob_positive_given_random(tru_pos, drug_user, non_user, pos_user),3))
```

    0.332

```
#what if we wanted another independent lab to check our results of the first test?
#we just update the prior numbers to the result of the first test, like this
#so, running a 2nd gets you closer

print(round(prob_positive_given_random(0.99, 0.33221, .01, .995),3),'%')
```

    97.063 %

```
#what about a third time?
print(round(prob_positive_given_random(0.99, 0.97063, .01, .995),3),'%')
#now we can be sure that the test is not as random as it is initially
```

    98.975 %

```
#function from class as opposed to my function written above
def bayes_theorem(prior, false_neg, false_pos, num_iter):
  #initiating an empty list
  posterior_list = []
  #range num_iter for
  for i in range(num_iter):
    posterior = ((prior * false_neg) / ((prior * false_neg) + (false_pos * num_iter))) # write your code here
    posterior_list.append(posterior)
  return posterior_list

bayes_theorem(.99, .005, .01, .995);
bayes_theorem
```

```
#alex kims
def bayes_theorem(prior, false_neg, false_pos, num_iter):
  posterior_list = []
  posterior = prior
  for i in range(num_iter):
    posterior = (1-false_neg)*posterior/((1-false_neg)*posterior + false_pos*(1-posterior))
    posterior_list.append(posterior)
  return posterior_list

bayes_theorem(0.005, 0.01, 0.01, 15)
```

    [0.33221476510067116,
     0.9801000000000001,
     0.9997949506335922,
     0.9999979283737839,
     0.9999999790744397,
     0.9999999997886307,
     0.999999999997865,
     0.9999999999999785,
     0.9999999999999998,
     1.0,
     1.0,
     1.0,
     1.0,
     1.0,
     1.0]

```
#breaking down alex's code
#posterior = ((1 - .01) * .005) / (( 1 - .01 ) * .005 ) + (.01 * 1-.005)
#posterior = (.99 * .005) / (.99 * .005) + (.01 * .995)
```

```
#paul kutrich
def bayes(bga, a, b):
  num = bga * a
  not_a = 1 - a
  denom = num + b * not_a

  return num / denom

def repeat_bayes(bga, a, b, reps):
  for _ in range(reps):
    a = bayes(bga, a, b)
    print(a)

pos_g_drunk = .99
drunk = 0.001
pos = .01

repeat_bayes(pos_g_drunk, drunk, pos, 6)
```

    0.09016393442622951
    0.9075000000000001
    0.9989714794017902
    0.9999896003148012
    0.9999998949515932
    0.9999999989389049

#Calculate Bayesian Confidence Intervals

Class Notes:
https://colab.research.google.com/drive/1cQuMCerEOQSYOOgCOO4xAdSBRBHtOUe8#scrollTo=h-OP3v-VnIRj

```
import numpy as np
from scipy import stats

np.random.seed(seed=42)
coinflips = np.random.binomial(n=1, p=0.5, size=100)
coinflips
```

    array([0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0,
           0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1,
           0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
           0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1,
           1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0])

```
#defining the confidence interval function
def confidence_interval(data, alpha=.9): #accepting two arguments, data and alpha as confidence interval
  n = len(data) #size of our sample
  mean = sum(data)/n #the mean
  data = np.array(data) #converts the list to an array, if already an array it does not effect the data
  stderr = stats.sem(data) #computing standard error
  t = stats.t.ppf((1 + alpha) / 2.0, n-1) #computing t statistics
  interval = stderr * t #getting our interval to calculate lower and upper bounds
  return (mean , mean-interval, mean+interval)

#outputting confidence interval from coinflips
confidence_interval(coinflips)
```

    (0.47, 0.3867125284491557, 0.5532874715508442)

```
#calculate this confidence interval using the bayesian method
stats.bayes_mvs(coinflips)
```

    (Mean(statistic=0.47, minmax=(0.38671252844915566, 0.5532874715508442)),
     Variance(statistic=0.25680412371134015, minmax=(0.20215017434095595, 0.323311952657888)),
     Std_dev(statistic=0.5054540733507159, minmax=(0.44961113680708126, 0.5686052696360524)))

```
#if we are only interested in the first row we can do this:
mean_conf_inter,_,_ = stats.bayes_mvs(coinflips, alpha=0.9)
mean_conf_inter
```

    Mean(statistic=0.47, minmax=(0.38671252844915566, 0.5532874715508442))

##Real World Experiment Design

Deploy a static A/B instrumented webpage

Evaluate and summarize findings from real-world A/B test data

#Type 1 Error

A type 1 error is when the null is true and you reject it.

#Type 2 Error
A type 2 error is when the null is false and you fail to reject it.

##Researchers need to confirm that certains pill contain a particular amount of active ingredient (not more, not less). What type of t-test should they use?

Two tailed test.

##Researchers need to confirm that the manufacturer produces no more than a certain percentage of defective products. What type of t-test should they use?

One tailed test.

##You want to do a study (based on people’s survey). You decided to collect data from sources that are more convenient than doing a randomized data collection. How might this choice affect your conclusion?

Biased sampling.

###A researcher wants to be thorough. She takes several samples to test her belief that the population mean is a certain value:  120.  To do so she forms an appropriate null hypothesis and alternative hypothesis: HA > 120 and selects alpha = 0.05: a) State the null hypothesis. b) She repeats the sampling process 40 times. How many times would you expect that she would make an error in her conclusion and reject H0 if indeed the population mean really is 120?

.95 * 40 = 38

40-38 = 2

##What is the CDF of a normal distribution?

The cumulative distribution function (CDF) of the standard normal distribution, usually denoted with the capital Greek letter (phi), is the integral. The related error function gives the probability of a random variable with normal distribution of mean 0 and variance 1/2 falling in the range.

This can also be described as the area under the curve.