# Electrical Power Generation 2

## Objective and Prerequisites

This example (which is an extension of the earlier ‘Electrical Power Generation 1 ‘ example) will teach you how to choose an optimal set of power stations to turn on in order to satisfy anticipated power demand over a 24-hour time horizon – but gives you the option of using hydroelectric power plants to satisfy that demand.

This model is example 16 from the fifth edition of Model Building in Mathematical Programming by H. Paul Williams on pages 271-272 and 326-327.

This example is at the intermediate level, where we assume that you know Python and the Gurobi Python API and that you have some knowledge of building mathematical optimization models.

**Download the Repository** <br />
You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

---
## Problem Description

In this problem, thermal power generation units are grouped into three distinct types, with different characteristics for each type (power output, cost per megawatt hour, startup cost, etc.).  A unit can be on or off, with a startup cost associated with transitioning from off to on, and power output that can fall anywhere between a specified minimum and maximum value when the unit is on.  There are also two hydroelectric plants available, also with different cost and power generation characteristics.  A 24-hour time horizon is divided into 5 discrete time periods, each with an expected total power demand.  The model decides which units to turn on, and when, in order to satisfy demand for each time period.  The model also captures a reserve requirement, where the selected power plants must be capable of increasing their output, while still respecting their maximum output, in order to cope with the situation where actual demand exceeds predicted demand.

A set of generators is available to satisfy power demand for the following day.  Anticipated demand is as follows:

| Time Period | Demand (megawatts) |
| --- | --- |
| 12 pm to 6 am | 15000 |
| 6 am to 9 am | 30000 |
| 9 am to 3 pm | 25000 |
| 3 pm to 6 pm | 40000 |
| 6 pm to 12 pm | 27000 |

Thermal generators are grouped into three types, with the following minimum and maximum output for each type (when they are on):

| Type | Number available | Minimum output (MW) | Maximum output (MW) |
| --- | --- | --- | --- |
| 0 | 12 |  850 | 2000 |
| 1 | 10 | 1250 | 1750 |
| 2 | 5 | 1500 | 4000 |

There are costs associated with using a thermal generator: a cost per hour when the generator is on (and generating its minimum output), a cost per megawatt hour above its minimum, and a startup cost for turning a generator on:

| Type | Cost per hour (when on) | Cost per MWh above minimum | Startup cost |
| --- | --- | --- | --- |
| 0 | $\$1000$ | $\$2.00$ | $\$2000$ |
| 1 | $\$2600$ | $\$1.30$ | $\$1000$ |
| 2 | $\$3000$ | $\$3.00$ | $\$500$ |

Two hydroelectric generators are also available, each with a fixed power output (when on):

| Hydro plant | Output (MW) |
| --- | --- |
| A | 900 |
| B | 1400 |

The costs associated with using a hydro plant are slightly different.  There's an hourly cost, but it is much smaller than the hourly cost of a thermal generator.  The real cost for a hydroelectric plant comes from depletion of the water in the reservoir, which happens at different rates for the two units.  The reservoir must be replenished before the end of the time horizon by pumping water into it, which consumes electricity.  A hydroelectric plant also has a startup cost.

| Hydro plant | Cost per hour (when on) | Startup cost | Reservoir depth reduction (m/hr) |
| --- | --- | --- | --- |
| A | $\$90$ | $\$1500$ | 0.31 |
| B | $\$150$ | $\$1200$ | 0.47 |

Pumping water into the reservoir consumes electricity at a rate of 3000 MWh of electricity per meter of height.  The height of the reservoir at the end of the time horizon must be equal to the height at the beginning.

Generators must meet predicted demand, but they must also have sufficient reserve capacity to be able to cope with the situation where actual demand exceeds predicted demand.  For this model, the set of selected thermal generators plus the set of hydro generators must be able to produce as much as 115% of predicted demand.

Which generators should be committed to meet anticipated demand in order to minimize total cost?

---
## Model Formulation

### Sets and Indices

$t \in \text{Types}=\{0,1,2\}$: Types of thermal generators.

$h \in \text{HydroUnits}=\{0,1\}$: Two hydro generators.

$p \in \text{Periods}=\{0,1,2,3,4\}$: Time periods.

### Parameters

$\text{period_hours}_{p} \in \mathbb{N}^+$: Number of hours in each time period.

$\text{demand}_p \in \mathbb{R}^+$: Total power demand for time period $p$.

$\text{generators}_t \in \mathbb{N}^+$: Number of thermal generators of type $t$.

$\text{start0} \in \mathbb{N}^+$: Number of thermal generators that are on at the beginning of the time horizon (and available in time period 0 without paying a startup cost).

$\text{min_output}_t \in \mathbb{R}^+$: Minimum output for thermal generator type $t$ (when on).

$\text{max_output}_t \in \mathbb{R}^+$: Maximum output for thermal generator type $t$.

$\text{base_cost}_t \in \mathbb{R}^+$: Minimum operating cost (per hour) for a thermal generator of type $t$.

$\text{per_mw_cost}_t \in \mathbb{R}^+$: Cost to generate one additional MW (per hour) for a thermal generator of type $t$.

$\text{startup_cost}_t \in \mathbb{R}^+$: Startup cost for thermal generator of type $t$.

$\text{hydro_load}_h \in \mathbb{R}^+$: Output for hydro generator $h$.

$\text{hydro_cost}_h \in \mathbb{R}^+$: Cost for operating hydro generator $h$.

$\text{hydro_startup_cost}_h \in \mathbb{R}^+$: Startup cost for hydro generator $h$.

$\text{hydro_height_reduction}_h \in \mathbb{R}^+$: Hourly reduction in reservoir height from operating hydro generator $h$.

### Decision Variables

$\text{ngen}_{t,p} \in \mathbb{N}^+$: Number of thermal generators of type $t$ that are on in time period $p$.

$\text{output}_{t,p} \in \mathbb{R}^+$: Total power output from thermal generators of type $t$ in time period $p$.

$\text{nstart}_{t,p} \in \mathbb{N}^+$: Number of thermal generators of type $t$ to start in time period $p$.

$\text{hydro}_{h,p} \in [0,1]$: Indicates whether hydro generators $h$ is on in time period $p$.

$\text{hydro_start}_{h,p} \in [0,1]$: Indicates whether hydro generator $h$ starts in time period $p$.

$\text{height}_{p} \in \mathbb{R}^+$: Height of reservoir in time period $p$.

$\text{pumping}_{p} \in \mathbb{R}^+$: Power used to replenish reservoir in time period $p$.

### Objective Function

- **Cost**: Minimize the cost (in USD) to satisfy the predicted electricity demand.

\begin{equation}
\text{Minimize} \quad Z_{on} + Z_{extra} + Z_{startup} + Z_{hydro} + Z_{hydro\_startup}
\end{equation}

\begin{equation}
Z_{on} = \sum_{(t,p) \in \text{Types} \times \text{Periods}}{\text{base_cost}_t*\text{ngen}_{t,p}}
\end{equation}
\begin{equation}
Z_{extra} = \sum_{(t,p) \in \text{Types} \times \text{Periods}}{\text{per_mw_cost}_t*(\text{output}_{t,p} - \text{min_load}_t})
\end{equation}
\begin{equation}
Z_{startup} = \sum_{(t,p) \in \text{Types} \times \text{Periods}}{\text{startup_cost}_t*\text{nstart}_{t,p}}
\end{equation}
\begin{equation}
Z_{hydro} = \sum_{(h,p) \in \text{HydroUnits} \times \text{Periods}}{\text{hydro_cost}_h*\text{hydro}_{h,p}}
\end{equation}
\begin{equation}
Z_{hydro\_startup} = \sum_{(h,p) \in \text{HydroUnits} \times \text{Periods}}{\text{hydro_startup_cost}_h*\text{hydro_start}_{h,p}}
\end{equation}

### Constraints

- **Available generators**: Number of generators used must be less than or equal to the number available.

\begin{equation}
\text{ngen}_{t,p} \leq \text{generators}_t  \quad \forall (t,p) \in \text{Types} \times \text{Periods}
\end{equation}

- **Demand**: Total power generated across all generator types must meet anticipated demand plus pumping for each time period $p$.

\begin{equation}
\sum_{t \in \text{Types}}{\text{output}_{t,p}} +
\sum_{h \in \text{HydroUnits}}{\text{hydro_load}_h*\text{hydro}_{h,p}} \geq
\text{demand}_p + \text{pumping}_p \quad \forall p \in \text{Periods}
\end{equation}

- **Min/max generation**: Power generation must respect thermal generator min/max values.

\begin{equation}
\text{output}_{t,p} \geq \text{min_output}_t*\text{ngen}_{t,p} \quad \forall (t,p) \in \text{Types} \times \text{Periods}
\end{equation}

\begin{equation}
\text{output}_{t,p} \leq \text{max_output}_t*\text{ngen}_{t,p} \quad \forall (t,p) \in \text{Types} \times \text{Periods}
\end{equation}

- **Reserve**: Selected thermal generators plus hydro units must be able to satisfy demand that is as much as 15% above the prediction.

\begin{equation}
\sum_{t \in \text{Types}}{\text{max_output}_t*\text{ngen}_{t,p}} +
\sum_{h \in \text{HydroUnits}}{\text{hydro_load}_h} \geq 1.15 * \text{demand}_p \quad \forall p \in \text{Periods}
\end{equation}

- **Thermal startup**: Establish relationship between number of active thermal generators and number of startups (use $start0$ for period before the time horizon starts)

\begin{equation}
\text{ngen}_{t,p} \leq \text{ngen}_{t,p-1} + \text{nstart}_{t,p} \quad \forall (t,p) \in \text{Types} \times \text{Periods}
\end{equation}

- **Hydro startup**: Establish relationship between hydro generator state and number of hydro startups (assume hydro plants are off at the beginning of the horizon)

\begin{equation}
\text{hydro}_{h,p} \leq \text{hydro}_{h,p-1} + \text{hydro_start}_{h,p} \quad \forall (h,p) \in \text{HydroUnits} \times \text{Periods}
\end{equation}

- **Reservoir height**: Track reservoir height.   Note that the height at the end of the final time period must equal the height at the beginning of the first.

- Reservoir level constraints: Height increases due to pumping activity and decreases due to hydroelectric generation.

\begin{equation}
\text{height}_{p} = \text{height}_{p-1}  + \text{period_hours}_{p}*\text{pumping}_{p}/3000 -
\sum_{h \in \text{HydroUnits}}{\text{period_hours}_{p}*\text{hydro_height_reduction}_{h}*\text{hydro}_{h,p}} \quad \forall p \in \text{Periods}
\end{equation}

- Cyclic constraint: Height at the first period must be equal to height at the last period.

\begin{equation}
\text{height}_{pfirst} = \text{height}_{plast}  + \text{period_hours}_{pfirst}*\text{pumping}_{pfirst}/3000 -
\sum_{h \in \text{HydroUnits}}{\text{period_hours}_{pfirst}*\text{hydro_height_reduction}_{h}*\text{hydro}_{h,pfirst}}
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
%pip install gurobipy
```

```python
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input Data
We define all the input data of the model.

```python
# Parameters

ntypes = 3
nperiods = 5
maxstart0 = 5
hydrounits = 2

generators = [12, 10, 5]
period_hours = [6, 3, 6, 3, 6]
demand = [15000, 30000, 25000, 40000, 27000]
min_load = [850, 1250, 1500]
max_load = [2000, 1750, 4000]
base_cost = [1000, 2600, 3000]
per_mw_cost = [2, 1.3, 3]
startup_cost = [2000, 1000, 500]

hydro_load = [900, 1400]
hydro_cost = [90, 150]
hydro_height_reduction = [0.31, 0.47]
hydro_startup_cost = [1500, 1200]
```

## Model Deployment

We create a model and the variables. For each time period, we have: an integer decision variable to capture the number of active generators of each type (ngen), an integer variable to capture the number of generators of each type we must start (nstart), a continuous decision variable to capture the total power output for each generator type (output), a binary decision variable that indicates whether a hydro unit is active (hydro), a binary decision variable that indicates whether a hydro unit must be started (hydrstart), a continuous decision variable that captures the amount of enery used to replenish the reservoir (pumping), and a continuous decision variable that captures the height of the reservoir (height).

```python
model = gp.Model('PowerGeneration2')

ngen = model.addVars(ntypes, nperiods, vtype=GRB.INTEGER, name="ngen")
nstart = model.addVars(ntypes, nperiods, vtype=GRB.INTEGER, name="nstart")
output = model.addVars(ntypes, nperiods, vtype=GRB.CONTINUOUS, name="genoutput")

hydro = model.addVars(hydrounits, nperiods, vtype=GRB.BINARY, name="hydro")
hydrostart = model.addVars(hydrounits, nperiods, vtype=GRB.BINARY, name="hydrostart")
pumping = model.addVars(nperiods, vtype=GRB.CONTINUOUS, name="pumping")
height = model.addVars(nperiods, vtype=GRB.CONTINUOUS, name="height")
```

    Using license file c:\gurobi\gurobi.lic

Next we insert the constraints:

The number of active generators can't exceed the number of generators:

```python
# Generator count

numgen = model.addConstrs(ngen[type, period] <= generators[type]
                         for type in range(ntypes) for period in range(nperiods))
```

Total power output for a thermal generator type depends on the number of generators of that type that are active.

```python
# Respect minimum and maximum output per generator type

min_output = model.addConstrs((output[type, period] >= min_load[type] * ngen[type, period])
                              for type in range(ntypes) for period in range(nperiods))

max_output = model.addConstrs((output[type, period] <= max_load[type] * ngen[type, period])
                              for type in range(ntypes) for period in range(nperiods))
```

Total generator output (thermal plus hydro) for each time period must meet predicted demand plus pumping.

```python
# Meet demand

meet_demand = model.addConstrs(gp.quicksum(output[type, period] for type in range(ntypes)) +
                               gp.quicksum(hydro_load[unit]*hydro[unit,period] for unit in range(hydrounits))
                               >= demand[period] + pumping[period]
                               for period in range(nperiods))
```

Maintain appropriate reservoir levels

```python
# Reservoir levels

reservoir = model.addConstrs(height[period] == height[period-1] + period_hours[period]*pumping[period]/3000 -
                             gp.quicksum(hydro_height_reduction[unit]*period_hours[period]*hydro[unit,period] for unit in range(hydrounits))
                             for period in range(1,nperiods))

# cyclic - height at end must equal height at beginning
reservoir0 = model.addConstr(height[0] == height[nperiods-1] + period_hours[0]*pumping[0]/3000 -
                             gp.quicksum(hydro_height_reduction[unit]*period_hours[0]*hydro[unit,0]
                             for unit in range(hydrounits)))
```

Selected generators must be able to cope with an excess of demand.

```python
# Provide sufficient reserve capacity

reserve = model.addConstrs(gp.quicksum(max_load[type]*ngen[type, period] for type in range(ntypes)) >=
                           1.15*demand[period] - sum(hydro_load)
                           for period in range(nperiods))
```

Connect the decision variables that capture active thermal generators with the decision variables that count startups.

```python
# Startup constraint

startup0 = model.addConstrs((ngen[type,0] <= maxstart0 + nstart[type,0])
                            for type in range(ntypes))

startup = model.addConstrs((ngen[type,period] <= ngen[type,period-1] + nstart[type,period])
                           for type in range(ntypes) for period in range(1,nperiods))
```

Connect hydro decision variables with hydro startup decision variables.

```python
# Hydro startup constraint

hydro_startup0 = model.addConstrs((hydro[unit,0] <= hydrostart[unit,0])
                                    for unit in range(hydrounits))

hydro_startup = model.addConstrs((hydro[unit,period] <= hydro[unit,period-1] + hydrostart[unit,period])
                           for unit in range(hydrounits) for period in range(1,nperiods))
```

Objective: minimize total cost.  Cost consists of five components: the cost for running active thermal generation units, the cost to generate power beyond the minimum for each unit, the cost to start up thermal generation units, the cost to operate hydro units, and the cost to start up hydro units.

```python
# Objective: minimize total cost

active = gp.quicksum(base_cost[type]*period_hours[period]*ngen[type,period]
                    for type in range(ntypes) for period in range(nperiods))

per_mw = gp.quicksum(per_mw_cost[type]*period_hours[period]*(output[type,period] - min_load[type]*ngen[type,period])
                       for type in range(ntypes) for period in range(nperiods))

startup_obj = gp.quicksum(startup_cost[type]*nstart[type,period]
                         for type in range(ntypes) for period in range(nperiods))

hydro_active = gp.quicksum(hydro_cost[unit]*period_hours[period]*hydro[unit,period]
                           for unit in range(hydrounits) for period in range(nperiods))

hydro_startup = gp.quicksum(hydro_startup_cost[unit]*hydrostart[unit,period]
                            for unit in range(hydrounits) for period in range(nperiods))

model.setObjective(active + per_mw + startup_obj + hydro_active + hydro_startup)
```

Next, we start the optimization and Gurobi finds the optimal solution.

```python
model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 85 rows, 75 columns and 215 nonzeros
    Model fingerprint: 0x1b27f759
    Variable types: 25 continuous, 50 integer (20 binary)
    Coefficient statistics:
      Matrix range     [1e-03, 4e+03]
      Objective range  [4e+00, 9e+03]
      Bounds range     [1e+00, 1e+00]
      RHS range        [5e+00, 4e+04]
    Presolve removed 18 rows and 3 columns
    Presolve time: 0.00s
    Presolved: 67 rows, 72 columns, 194 nonzeros
    Variable types: 25 continuous, 47 integer (18 binary)

    Root relaxation: objective 9.990143e+05, 28 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 999014.286    0    7          - 999014.286      -     -    0s
    H    0     0                    1106190.0000 999014.286  9.69%     -    0s
    H    0     0                    1030900.0000 999014.286  3.09%     -    0s
    H    0     0                    1011330.0000 999014.286  1.22%     -    0s
    H    0     0                    1008935.0000 999014.286  0.98%     -    0s
    H    0     0                    1006220.0000 999014.286  0.72%     -    0s
    H    0     0                    1005470.0000 999014.286  0.64%     -    0s
    H    0     0                    1003274.0000 999014.286  0.42%     -    0s
    H    0     0                    1000630.0000 999955.477  0.07%     -    0s
         0     0 999955.477    0   10 1000630.00 999955.477  0.07%     -    0s

    Cutting planes:
      Gomory: 1
      MIR: 4

    Explored 1 nodes (39 simplex iterations) in 0.04 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 7: 1.00063e+06 1.00327e+06 1.00547e+06 ... 1.10619e+06

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.000630000000e+06, best bound 1.000611428571e+06, gap 0.0019%

---
## Analysis

The anticipated demand for electricity over the 24-hour time window can be met for a total cost of $\$1,000,630$, as compared with the $\$1,002,540$ that was required to meet the same demand without the hydro plants.  The detailed plan for each time period is as follows.

### Unit Commitments

The following table shows the number of thermal generators of each type that are active in each time period in the optimal solution:

```python
rows = ["Thermal" + str(t) for t in range(ntypes)]
units = pd.DataFrame(columns=range(nperiods), index=rows, data=0.0)

for t in range(ntypes):
    for p in range(nperiods):
        units.loc["Thermal"+str(t), p] = ngen[t,p].x
units
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Thermal0</th>
      <td>12.0</td>
      <td>12.0</td>
      <td>12.0</td>
      <td>12.0</td>
      <td>12.0</td>
    </tr>
    <tr>
      <th>Thermal1</th>
      <td>3.0</td>
      <td>9.0</td>
      <td>9.0</td>
      <td>9.0</td>
      <td>9.0</td>
    </tr>
    <tr>
      <th>Thermal2</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>

The following shows the number of generators of each type that must be started in each time period to achieve this plan (recall that the model assumes that 5 are available at the beginning of the time horizon):

```python
rows = ["HydroA", "HydroB"]
hydrotable = pd.DataFrame(columns=range(nperiods), index=rows, data=0.0)

for p in range(nperiods):
    hydrotable.loc["HydroA", p] = int(hydro[0,p].x)
    hydrotable.loc["HydroB", p] = int(hydro[1,p].x)
hydrotable
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>HydroA</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>HydroB</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>1.0</td>
    </tr>
  </tbody>
</table>
</div>

The hydro plants are lightly used in the schedule - we only use plant B for the last two time periods.

The following shows the pumping that must be done in order to support this level of hydro activity

```python
pumptable = pd.DataFrame(columns=range(nperiods), index=["Pumping"], data=0.0)

for p in range(nperiods):
        pumptable.loc["Pumping", p] = pumping[p].x
pumptable
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Pumping</th>
      <td>815.0</td>
      <td>0.0</td>
      <td>950.0</td>
      <td>0.0</td>
      <td>350.0</td>
    </tr>
  </tbody>
</table>
</div>

It is interesting to note that the plan simultaneously operates HydroB and pumps in time period 4.
While it might appear that costs could be reduced by operating the hydro plant at a lower load, in this model the hydro plants have fixed output.  If it makes economic sense to turn a hydro plant on, then we have to replenish the water drained, even if it means using some of the generated electricity to pump.

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


# The next two cells deal with
## barebones.py

### HTML cell
The equations are given in HTML suitable for a web page. No attempt is made to derive them
### Python program cell
No explaination of how the equations are found is given. The program takes the input numbers and prints out the answers

<!DOCTYPE html>

  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width">
  <title>MathJax example</title>

    The given

R = radius of the plunger

c = the speed of a water wave

v = the speed of the simulated object (use v = 0.7c as a default)

n = number of plunger impressions

    What is to be calculated

    $$ \begin{align}
        x_{start}&=\ position\ of\ first\ impression\\
        \Delta x &=\ distance\ traveled\ between\ impressions\\
        \Delta t &=\ time\ spent\ traveling\ between\ impressions\\
        x_{tank} &=\ length\ of\ tank\\
        y_{tank} &=\ width\ of\ tank\\
    \end{align}
    $$

    The equations

    $$ \begin{align}
        x_{start}&=R\left( \frac{3}{2}-\frac{1}{4n}\left(1+\frac{v}{c}\right)\right)\\
        \Delta x &= \frac{Rv}{nc}\\
        \Delta t &= \frac{R}{nc}\\
        x_{tank} &= \frac{R}{2}\left(5-\frac{3}{n}\frac{v}{c} +2\frac{v}{c}\right) \\
        y_{tank} &= R \left(2+\frac{n-1}{n}+\frac{\sqrt{c^2-v^2}}{2nc} \right)
    \end{align}
    $$

```python
import math as m

# Input data
R = 40.0
c = 18.0
v = c*0.7
n = 4

# Calculate values
x_start = R*(3/2-(1+v/c)/4/n)
x_tank  = (R/2)*(5-3*v/n/c+2*v/c)
y_tank  = (  2*R +
             R*(n-1)/n +
             R/2/n/c*m.sqrt(c**2-v**2)
             )
dt = R/n/c
dx = R*v/n/c

# print values
print('Given:' )
print(' R = radius of plunger     = {:5.2f} cm'.format(R))
print(' c = speed of water wave   = {:5.2f} cm/s'.format(c))
print(' v = simulated velocity    = {:5.2f} cm/s'.format(v))
print(' n = number of impressions = {}'.format(n))
print('If the left wall of the tank is x = 0 then:')
print(' first impressions is at {:5.2f}'.format(x_start))
print(' the tank length is      {:5.2f}'.format(x_tank))
print(' the tank width is       {:5.2f}'.format(y_tank))
print(' the plunger will make {} impressions {:5.2f} cm appart every {:5.2f}s'.
      format(n,dx,dt))

```

    Given:
     R = radius of plunger     = 40.00 cm
     c = speed of water wave   = 18.00 cm/s
     v = simulated velocity    = 12.60 cm/s
     n = number of impressions = 4
    If the left wall of the tank is x = 0 then:
     first impressions is at 55.75
     the tank length is      117.50
     the tank width is       113.57
     the plunger will make 4 impressions  7.00 cm appart every  0.56s

# The next cell is the program
## SRbuildnumbers.py

### Hand written notes:
There is a single page of calculations: images/ballpark.png
on file to back up calculations used in program
### Python program cell
The program takes the input numbers and prints out the answers while attempting to be plain how the numbers are being generated.

```python
import math as m
import matplotlib.pyplot as plt

def not_a_num(number):
    '''Will check to see if the value passed to the functions is a float or and int'''
    if type(number)!=int and type(number)!=float:
        return True
    else:
        return False

def check_given(R,c,v,n):
    '''Will check to see if the numbers passed to functions are within
    acceptable ranges. Will print warnings and replace offencing data
    with defaults to allow program to continue.'''
    # check inputs are correct types
    if not_a_num(R):
        print('WARNING: R must be number  \nSet to default 1.0 ')
        R = 1.0
    if not_a_num(c):
        print('WARNING: c must be number  \nSet to default 18.0 ')
        c = 18.0
    if not_a_num(v):
        print('WARNING: v must be number  \nSet to default v*0.7 ')
        v= 0.7*c
    if type(n)!= int:
        print('WARNING: n must be an integer \n Set to default 2 ')
        n = 2

    # check number ranges
    if R<=0 or R>200.:
        print('WARNING: Check range of R.  \nSet to default 1.0 ')
        R = 1.0
    if c<=0:
        print('WARNING: Check range of c.  \nSet to default of 18.0 ')
        c = 18.0
    if v<=0 or v>c:
        print('WARNING: Check range of v.  \nSet to default 0.7c ')
        v = 0.7*c
    if  n<2:
        print('WARNING: Check range of n.  \nset to default 2.0 ')
        n = 2

    return R,c,v,n

def find_dx_dt():
    '''Calculates the most important values: dx and dt
    dx = how much the plunger must move
    dt = the time this move must be executed
    '''
    dt = R/n/c
    dx = R*v/n/c
    return dx,dt

def find_ellipse_data():
    '''Finds the data for the small ellipses
    a = major axis
    b = minor axis
    f = the focal length (often denoted with c in math texts but avoided here)
    (n-1) = number of small ellipses
    tf = the time required for the wave to travel 1/2 the distance from f to
    perimeter ellipse and back to second f
    '''
    b = R/2/n/c*m.sqrt(c**2-v**2)
    tf = R/2/n/c
    a = c*tf
    f = v*tf
    return a,b,f,tf

def find_timing():
    ''' Calculates additional timing details:
    1) t_str = intial time delay from first impression to actual start of
     simulation in sweetspot
    2) t_sim = length of time waves are simulating effect in sweetspot
    3) t_end = the end of the sim in the sweetspot'''
    t_str = R/c
    t_sim = 2*tf*(n-1)
    t_end = R/c+t_sim
    return t_str,t_sim,t_end

def find_sweetspot():
    '''finds the length and width of the sweetspot surrounding the
    small ellipses
    (0,0) is set at the left most focal point for the n=0 impression'''
    # Calculate the y "sweet spot" where simulation will take place
    sweet_l = -(a-f)
    sweet_r = sweet_l+2*a+2*f*(n-2)
    sweet_t =b
    sweet_b =-b
    return sweet_l,sweet_r,sweet_t,sweet_b

def find_tanksize():
    '''Calculates the dimentions the rectagular tank required to hold the simulation.
    Criteria: unwanted wave cannot reflect back into sweetspot before end of
    simulation
    '''
    tanks_t =  (R+b+c*t_sim+R)/2
    tanks_b = -tanks_t
    tanks_l = (-R -
              (a-f)-
              (R+c*t_sim))  /2
    tanks_r = 2*f*(n-1)+(R+R+(a-f))/2
    return tanks_l,tanks_r,tanks_t,tanks_b

def print_data():
    '''Prints given and all data calculated from that given'''
    print('Given:' )
    print('R = radius of plunger     = {:5.2f} cm'.format(R))
    print('c = speed of water wave   = {:5.2f} cm/s'.format(c))
    print('v = simulated velocity    = {:5.2f} cm/s'.format(v))
    print('n = number of impressions = {}'.format(n))
    print('\nImportant Results:')
    print('x_start = starting impression     = {:5.2f} cm'.
          format(R*(3/2-(1+v/c)/4/n)))
    print('dx = distance between impressions = {:5.2f} cm'.format(dx))
    print('dt = time between impressions     = {:5.2f} s'.format(dt) )

    print('\nAdditional Timing details of sim:')
    print('time after first impression to start of sim \t= {:5.2f}s'.format(t_str))
    print('time for sim                 \t\t\t= {:5.2f}s'.format( n-1,2*tf,t_sim))
    print('time after first impression to end of sim \t= {:5.2f}s'.format(t_end))
    print('\nEllipse details:')
    print('(n-1) = number of ellipses   = {:5.2f}'.format((n-1)))
    print('a   = length of major axis = {:5.2f} cm'.format(a))
    print('b   = length of minor axis = {:5.2f} cm'.format(b))
    print('f   = focal length         = {:5.2f} cm'.format(f))
    print('\nSweetspot Size:')
    print('x sweet spot\t\t= {:5.2f} cm'.format(sweet_r-sweet_l))
    print('y sweet spot\t\t= {:5.2f} cm'.format(sweet_t-sweet_b))
    print('\nTank Size')
    print('y tank \t\t\t= {:5.2f} cm'.format(tanks_t-tanks_b))
    print('x tank \t\t\t= {:5.2f} cm'.format(tanks_r-tanks_l))

def ellipse(xcenter=0.0,ycenter=0.0,a=1.0,b=1.0,num_points = 30):
    '''Lists xs and ys are cleared then data for an ellipse installed.
      1) ellipse center is on (xcenter,ycenter)
      2) a = major axis (x-axis)
      3) b = minor axis (y-axis)
      4) num_points = number of points in ellipse
    '''
    xs=[]
    ys=[]
    angle_inc = 2*m.pi/num_points
    for num in range(0,num_points+1):
        xs.append(a*m.cos(num*angle_inc+m.pi/2)+xcenter)
        ys.append(b*m.sin(num*angle_inc+m.pi/2)+ycenter)
    return xs,ys

def ellipse_plus_c(xcenter=0.0,ycenter=0.0,a=1.0,b=1.0,num_points = 30):
    '''Calls ellipse() to install data for an ellipse,
    then adds the center to coordinate list (xs,ys)'''
    xs,ys = ellipse(xcenter,ycenter,a,b,num_points)
    xs.append(xcenter)
    ys.append(ycenter)
    return xs,ys

def draw_tank(show_waves_at_end=False):
    '''draws the tank, '''
    # set defaults for image
    plt.figure(figsize=(7,7))
    color_tank = 'm'
    color_sweet = 'r'
    color_ellipses = 'r'

    # Draw in size of unwanted waves at end of simulation
    # NOTE: Used to check formulas. May be of use but
    # will likely end up as dead code with no future use.
    if show_waves_at_end:
        for i in range(0,n):
            xs,ys = ellipse_plus_c(2*i*f-tanks_l,0.0,2*R+c*((n-1)-i)*dt,
                                   2*R+c*((n-1)-i)*dt,100)
            grey = 1.0-(i+1)/n
            col=(0,0,grey)
            time = 'x= {:5.2f}cm t= {:5.2f} s'.format(2*i*f-tanks_l,i*dt)
            plt.plot(xs,ys,color=col)

    # draw in placement of impressions of plunger
    for i in range(0,n):
        xs,ys = ellipse_plus_c(2*i*f-tanks_l,0.0,R,R,100)
        grey = 1.0-(i+1)/n
        col=(0,0,grey)
        time = 'x= {:5.2f}cm t= {:5.2f} s'.format(2*i*f-tanks_l,i*dt)
        plt.plot(xs,ys,color=col,label=time)

    # draw in tank or tank
    xs=[0.0,tanks_r-tanks_l,tanks_r-tanks_l,0.0,0.0]
    ys=[tanks_t,tanks_t,tanks_b,tanks_b,tanks_t]
    plt.plot(xs,ys,'-',color = color_tank,label ='tank (min. size)')

    # small ellipses
    for i in range(0,(n-1)):
        xs,ys = ellipse(f+2*f*i-tanks_l,0.0,a,b,100)
        plt.plot(xs,ys,'-',color = color_ellipses)
    # sweetspot around small ellipses
    xs=[sweet_l-tanks_l,sweet_l-tanks_l,sweet_r-tanks_l,
        sweet_r-tanks_l,sweet_l-tanks_l]
    ys=[sweet_b,sweet_t,sweet_t,sweet_b,sweet_b]
    plt.plot(xs,ys,'-',color = color_sweet)

    plt.axis('scaled')
    plt.legend(loc='lower right')
    plt.show()

R = 40.0
c = 18.0
v = c*0.7
n = 4

R,c,v,n =check_given(R,c,v,n)
dx,dt = find_dx_dt()
a,b,f,tf = find_ellipse_data()

t_str,t_sim,t_end = find_timing()
sweet_l,sweet_r,sweet_t,sweet_b = find_sweetspot()
tanks_l,tanks_r,tanks_t,tanks_b = find_tanksize()

print_data()
draw_tank(show_waves_at_end=False)

```

```python

```


$\newcommand{\ve}[1]{\mathbf{#1}}$
$\newcommand{\ovo}{\overline{O}}$
$\def\Brack#1{\left[ #1 \right]}$
$\def\bra#1{\mathinner{\langle{#1}|}}$
$\def\ket#1{\mathinner{|{#1}\rangle}}$
$\def\braket#1{\mathinner{\langle{#1}\rangle}}$
$\def\Bra#1{\left<#1\right|}$
$\def\Ket#1{\left|#1\right>}$
$\def\KetC#1{\left|\left\{ #1 \right\} \right\rangle}$
$\def\BraC#1{\left\langle \left\{ #1 \right\} \right|}$
$\def\sen{\mathop{\mbox{\normalfont sen}}\nolimits}$
$\newcommand{\vac}{\ket{\text{vac}}}$
$\newcommand{\vacbra}{\bra{\text{vac}}}$
$\newcommand{\sinc}{\text{sinc}}$

Quantum Non Demolition Measurements (QND)

 Mediciones cuánticas no destructivas

 Outline

* Divisor de haz
* Interferómetro
* Medidas no destructivas. Ver solo un poquito.

 Divisor de haz

Un divisor de haz sin pérdidas, es como el que se muestra en la figura. Posee:
* Dos modos de campo de entrada
* Dos modos de campo de salida

* Que no haya pérdidas garantiza que la probabilidad se conserve
* Es una transformación unitaria que se puede escribir como
\begin{equation}
|\psi' \rangle = U | \psi \rangle
\end{equation}
* Se puede escribir en forma matricial
$$\mathbf{U} = e^{i\kappa}\left[\begin{array}
{rr}
te^{i\delta_r} & -re^{-i\delta_t}  \\
re^{i\delta_t} & t^{-i\delta_r} \\
\end{array}\right]
$$
La forma matricial significa que si llega un estado $|0\rangle$ o $|1\rangle$ tendrá la transformación
$$|0\rangle \rightarrow U_{00} |0\rangle + U_{10} |1\rangle $$
$$|1\rangle \rightarrow U_{01} |0\rangle + U_{11} |1\rangle$$

 Algunos hechos interesantes del divisor de haz

* $t^2 + r^2 =1$
* La diferencia de fase entre los campos reflejado y transmitido cuando el estado de entrada es $|0\rangle$ es $\delta_0=\delta_r-\delta_t$
    * Cuando el estado de entrada es |1\rangle, la diferencia de fase es $\delta_1=-\delta_r+\delta_t \pm \pi$.
* Además, todo divisor de haz cumple $\delta_0+\delta_1=\pm \pi$

 Divisores de haz especiales

$$\mathbf{U}_1 = e^{i\kappa}\left[\begin{array}
{rr}
ir & t  \\
t & ir \\
\end{array}\right], \quad
\mathbf{U}_2 = e^{i\kappa}\left[\begin{array}
{rr}
r & t  \\
t & -r \\
\end{array}\right]
$$

* $U_1$ no es simétrico en el tiempo, ¿Qué significa esto?
* Pero es espacialmente simétrica para $r=t=1/\sqrt{2}$
* ¿Es $U_2$ temporalmente simétrica?
    * ¿Y espacialmente?

## Graficando el qubit en la esfera de Bloch

Recordar que para graficar el qubit en la esfera de Bloch es necesario obtener las proyecciones en cada uno de los ejes. Dado que contamos con la matriz de densidad del sistema, y como podemos escribir el estado de un qubit de forma general en términos de las matrices de Pauli como sigue

$$
\rho = \frac{1}{2} \left( \mathbf{1}+ \vec{r} \cdot \vec{\sigma} \right),
$$

donde $\vec{\sigma} = (\sigma_1,\sigma_2, \sigma_3 )$ y $\vec{r} = (\sin \theta \cos \phi, \sin \theta \sin\phi, \cos \theta)$, por lo que obtener las componentes $r_i$ es directo de la forma

$$
r_i = \text{tr} [\rho \sigma_i]
$$

```python
import numpy as np
from scipy.linalg import expm, norm
import tensorflow as tf
import strawberryfields as sf
from strawberryfields.ops import *
from strawberryfields.backends.tfbackend.ops import partial_trace
# matrices de Pauli
s1 = np.array([[0, 1],[ 1, 0]])
s2 = np.array([[0, -1j],[1j, 0]])
s3 = np.array([[1, 0],[0, -1]])

cutoff = 2

#entrada 1 del BS
psi = np.zeros([cutoff], dtype=np.complex128)
psi[0] = 1.0
psi[1] = 1.0

psi /= np.linalg.norm(psi)

#entrada 2 del BS
phi = np.zeros([cutoff],dtype=np.complex128)
phi[0] = 1.0
phi /= np.linalg.norm(phi)
```

```python
#conversión al tipo necesario para tensor flow
psi = tf.cast(psi, tf.complex64)
phi = tf.cast(phi,tf.complex64)
in_state = tf.tensordot(psi,phi,axes=0)
eng, q = sf.Engine(2)
with eng:
    Ket(in_state) | q
    BSgate(np.pi/4,0) | q
#state_out = eng.run('tf', cutoff_dim=cutoff,eval=False,modes=[1])
state_out = eng.run('tf', cutoff_dim=cutoff)
#Matriz de densidad del sistema y las matrices reducidas
rho=state_out.dm()
rhoA = np.einsum('ijll->ij', rho)
rhoB = np.einsum('kkij->ij', rho)
#Grafica de p(n) para uno de los modos de salida
import matplotlib.pyplot as plt
plt.bar(np.arange(cutoff), height=np.real_if_close(np.diag(rhoA)))
```

```python
# Gráfica en esfera de Bloch
def M(axis, theta):
    #Función que realiza una rotación en el eje axis
    return expm(np.cross(np.eye(3), axis/norm(axis)*theta))
from qutip import Bloch
b=Bloch()
vec = [[0,0,-1],[0,1,0],[0,0,1]]
b.add_vectors(vec)
npts=10;

v5, axis, theta = [0.1,0.5,0], [0,0,1],1.2
#v=v/norm(v)
v1= np.trace(rhoA@s1)
v2= np.trace(rhoA@s2)
v3= np.trace(rhoA@s3)
v = np.real_if_close([v1,v2,v3])
b.clear()
b.vector_color = ['r']
b.view = [-40,30]
#b.add_points(np.transpose(vecv))
b.add_vectors(v)

```

```python
b.show()
```

### Ejercicio:

* Escribe las transformaciones sobre estados $|0\rangle$ y $| 1\rangle$
* Obtén las transformaciones para una combinación lineal de estados de la forma $|\psi \rangle = \alpha |0 \rangle + \beta |1\rangle$
* Haz un programa utilizando RBF que realice la transformación de estos estados.
* ¿Podemos construir alguna compuerta del set estudiado hasta ahora con lo que acabamos de ver de los divisores de haz?

NOTA PERSONAL: falta agregar la parametrización de los coeficientes

* Construya cualquier transformación unitaria mediante retardadores de fase y la compuerta $U_2$

 Inteferómetro Mach-Zender

Un interferómetro de Mach-Zender se construye como sigue

- Es equivalente a introducir un 'retardador de fase' entre dos compuertas Hadamard (Compruebe esto). Esto se puede hacer con una compuerta de fase. Esto es equivalente a $H \phi(-\alpha) H$, donde la fase $\alpha$ es una fase relativa entre los dos caminos ópticos posibles

- Considere ahora que un sólo fotón entra en el interferómetro por el brazo superior ($|0\rangle$). Luego, se detecta un fotón en el brazo superior a la salida del interferómetro. Se puede demostrar que la probabilidad de obtener un fotón está dada por
    $$p_0 = \frac{1}{2} \left( 1+ \cos \alpha\right)$$

 Mediciones nulas ("No destructivas")

Queremos detectar detectar una bomba que es tan sensible, que con sólo absorber un fotón exlota. Es decir, que si nosotros la observamos, por el hecho de saber que está allí (haberla iluminado de alguna forma para observarla) habría provocado su explosión. ¿Habrá alguna manera de resolver este paradigma?

NOTA: colocar encuesta y resultados en esta sección

####  Una medición nula: es aquella en la que el objeto a ser medido no cambia cuando cumple con ciertas características, pero se modifica cuando cumple con otras. Considere el ejemplo en la figura, donde un electrón sólo puede ser excitado si un fotón incide con la polarización adecuada.

 Implementación en StrawBerry Fields

Queremos detectar bomba que se muestra en la figura

Flujo de codificación y simulación:
* Preparar un estado inicial en $|0\rangle$.
* Hacerlo evolucionar en el BS
* Calcular la probabilidad de que el fotón se haya ido por el camino inferior, utilizando $P_1 = |\langle 1 | \psi \rangle |^2$.
* Aplicar la medición nula, es decir, volver a preparar el estado en el modo $|0\rangle$ y hacerlo evolucionar en el BS
* Calcular las probabilidades de detección finales.

```python
# se prepararn las entradas del BS
psi = np.zeros([cutoff], dtype=np.complex128)
#psi[0] = 1.0
psi[1] = 1.0

psi /= np.linalg.norm(psi)

#entrada 2 del BS
phi = np.zeros([cutoff],dtype=np.complex128)
phi[0] = 1.0
phi /= np.linalg.norm(phi)

psi = tf.cast(psi, tf.complex64)
phi = tf.cast(phi,tf.complex64)
in_state = tf.tensordot(psi,phi,axes=0)

eng, q = sf.Engine(2)
with eng:
    Ket(in_state) | q
    BSgate(np.pi/4,0) | q   #2 efecto del BS sobre la entrada
#    Measure | q[1] # esta estrategia tiene la problemática de que
state_out = eng.run('tf', cutoff_dim=cutoff)
rho1=state_out.dm()
#realizar medición
MMM= np.tensordot(np.eye(2),np.array([0,1])[np.newaxis].T@np.array([0,1])[np.newaxis],axes=0)
Prob_Bomba=np.real_if_close(np.trace(np.trace(rho1@MMM)))

# se prepara el estado nuevamente
# se hace pasar por el divisor de haz
# se hacen mediciones de n
eng, q = sf.Engine(2)
with eng:
    Ket(in_state) | q
    BSgate(np.pi/4,0) | q   #2 efecto del BS sobre la entrada
    Measure | q[0]
    Measure | q[1]

#sess=tf.Session()
#with sess.as_default():
    #psi_value=sess.run(psi)
#    MMM=MMM.eval()
 #   rhoB2=rhoB.eval()

```

```python

```

    0.4999999701976776

```python
np.trace(np.trace(state_out.dm()@np.tensordot(np.eye(2),np.eye(2),axes=0)))
```

    (0.9999999403953552+0j)

```python
np.array([0,1])[np.newaxis].T@np.array([0,1])[np.newaxis]
```

    array([[0, 0],
           [0, 1]])

```python
a = np.array([5,4])[np.newaxis]
```

```python
a
```

    array([[5, 4]])

```python
np.array([0,1])[np.newaxis].T@np.array([0,1])[np.newaxis]
```

    array([[0, 0],
           [0, 1]])

```python
np.eye(2)
```

    array([[1., 0.],
           [0., 1.]])

```python
MMM
```

    array([[[[0., 0.],
             [0., 1.]],

            [[0., 0.],
             [0., 0.]]],

           [[[0., 0.],
             [0., 0.]],

            [[0., 0.],
             [0., 1.]]]])

```python

```


# Hands on Machine Learning with Scikit-Learn and Tensorflow

## Chapter 3: Classification

### Notebook by: Arunava

```python
# Ignoring Warnings
import warnings

def warn(*args, **kwargs):
    pass
warnings.warn = warn
```

## MNIST

```python
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
mnist
```

    {'COL_NAMES': ['label', 'data'],
     'DESCR': 'mldata.org dataset: mnist-original',
     'data': array([[0, 0, 0, ..., 0, 0, 0],
            [0, 0, 0, ..., 0, 0, 0],
            [0, 0, 0, ..., 0, 0, 0],
            ...,
            [0, 0, 0, ..., 0, 0, 0],
            [0, 0, 0, ..., 0, 0, 0],
            [0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
     'target': array([0., 0., 0., ..., 9., 9., 9.])}

```python
X, y = mnist['data'], mnist['target']
print (X.shape)
print (y.shape)
```

    (70000, 784)
    (70000,)

```python
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

some_digit = X[23456].reshape(28, 28)
plt.imshow(some_digit, cmap=matplotlib.cm.binary, interpolation='nearest')
plt.axis('off')
plt.show()

print ('Label: ', y[23456])
```

```python
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]
```

```python
import numpy as np
shuffle_index = np.random.permutation(60000)
X_train, y_train = X[shuffle_index], y[shuffle_index]
```

## Training a Binary Classifier

```python
y_train_3 = (y_train == 3)
y_test_3 = (y_test == 3)
```

```python
from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_3)
sgd_clf.predict(some_digit.ravel().reshape(1, -1))
```

    array([ True])

## Performance Measures

Implementing cross-validation from scratch

```python
from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

skfolds = StratifiedKFold(n_splits = 3, random_state=42)

i = 1

for train_index, test_index in skfolds.split(X_train, y_train_3):
    clone_clf = clone(sgd_clf)

    X_train_f = X_train[train_index]
    y_train_f = y_train_3[train_index]

    X_test_f = X_train[test_index]
    y_test_f = y_train_3[test_index]

    clone_clf.fit(X_train_f, y_train_f)
    pred = clone_clf.predict(X_test_f)

    n_correct = sum(pred == y_test_f)
    acc = n_correct/len(X_test_f)
    print ('Accuracy - Iteration', i, ' :', acc)

    i += 1
```

    Accuracy - Iteration 1  : 0.9619019049047548
    Accuracy - Iteration 2  : 0.96565
    Accuracy - Iteration 3  : 0.9648482424121206

Using `cross_val_score()` along with `sgd_clf`

```python
from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_3, cv=3, scoring='accuracy')
```

    array([0.9619019 , 0.96565   , 0.96484824])

Building a dumb classifier that predicts everything in the 'not 3' class

```python
from sklearn.base import BaseEstimator

class Never3Clasifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X, y=None):
        return np.zeros((len(X), 1), dtype=bool)
```

```python
never_3_clf = Never3Clasifier()
cross_val_score(never_3_clf, X_train, y_train_3, cv=3, scoring='accuracy')
```

    array([0.8976 , 0.89935, 0.8965 ])

~90% Accuaracy!

This is simply because about 10% of the classes are `3` and rest are `not_3`.

Also, if you simply predict not `3` always then you are bound to get accuracy of `90%` all the time as only `10%` of the examples are `3`.

This is the reason that `accuracy` is not a very good measure of how good a classifier really is. (Especially in cases of skewed datasets where some classes are much more frequent than others)

## Confusion Matrix

A general idea is to count the number of times instances of class A were classified as class B and class A i.e. correct predictions and incorrect predictions.

```python
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_3, cv=3)
confusion_matrix(y_train_pred, y_train_3)
```

    array([[53260,  1543],
           [  609,  4588]])

In the confusion matrix, each **row represents an actual class** while each **columnrepresents a predicted class**.

1, 1 -> true negetives

1, 2 -> false positives

2, 1 -> false negetives

2, 2 -> true positives

A perfect classifier would have only the true positives and true negetives

```python
confusion_matrix(y_train_3, y_train_3)
```

    array([[53869,     0],
           [    0,  6131]])

A more consise metric would be look at the classifier's precision which is the accuary of the positive predictions.

\begin{align}
precision = \frac{TP}{TP+FP}
\end{align}

where, _TP_ is the number of true positives and _FP_ is the number of false positives

A trivial way to have a perfect precision is to make one single positive prediction and ensure it is correct (`precision` = `1/1` = `100%`)

This would not be very useful.

So, precision is typically used along with another metric name _recall_ (sensitivity or true positive rate (TPR)) -> this is the ratio of positive instances that are correctly detected by the classifier

\begin{align}
recall = \frac{TP}{TP+FN}
\end{align}

where _FN_ is false negetives

### Precision and Recall

```python
from sklearn.metrics import precision_score, recall_score

print ('Precision score:', precision_score(y_train_3, y_train_pred))
print ('Recall score:', recall_score(y_train_3, y_train_pred))
```

    Precision score: 0.8828170098133539
    Recall score: 0.7483281683249062

Now the 3-detector doesn't seem as good as previous.

As you can see it is correct only `82%` of the time (precision score)

and, it detects only `80%` of the `5`s

We often combine precision and recall into a single metric called the _F1 score_.

It calculates the harmonic mean which gives more weight to the low values (irrespective to regular mean where all values are treated equally)

\begin{align}
F1 = \frac{TP}{TP+\frac{FN+FP}{2}}
\end{align}

Okay, lets now compute the f1 score

```python
from sklearn.metrics import f1_score
f1_score(y_train_3, y_train_pred)
```

    0.8100282485875706

f1 score favours classifiers that have similar precision and recall. This is not always the case as in some context you mostly care about the precision and in other contexts you care about the recall.

Increasing precision reduces recall and vice versa. This is called **precision/recall tradeoff**.

## Precision/Recall Tradeoff

In the SDGClassifier you cannot set the threshold directly but you can get the decision scores which is calcualted for each instance.

Raising the threshold increases the precision and decreases the recall

Lowering the threshold decreases the precision and increases the recall.

```python
y_scores = sgd_clf.decision_function([some_digit.ravel()])
print ('y_scores: ', y_scores)
threshold = 0
y_some_digit_pred = (y_scores > threshold)
print ('y_some_digit_pred: ', y_some_digit_pred)
```

    y_scores:  [150957.38355489]
    y_some_digit_pred:  [ True]

Since, the threshold is set to `0` every instance is classified as `5` and thus whatever the score is the classifier would return `True`.

Let's raise the threshold.

```python
threshold = 230000
y_some_digit_pred = (y_scores > threshold)
print ('y_some_digit_pred: ', y_some_digit_pred)
```

    y_some_digit_pred:  [False]

So, how can we decide what threshold to use?

Let's get the scores of all instances using `cross_val_predict()`.

Note: Since we need to get the scores we need to pass `method='decision_function'` to `cross_val_predict()`

```python
y_scores = cross_val_predict(sgd_clf, X_train, y_train_3, cv=3, method='decision_function')
```

With all the scores we can now calculate `precision` and `recall` for thresholds.

```python
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_3, y_scores)
```

```python
def plot_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], 'b-', label='Precision')
    plt.plot(thresholds, recalls[:-1], 'g-', label='Recall')
    plt.xlabel('Threshold')
    plt.ylabel('Precision')
    plt.legend()
    plt.ylim([-0.2, 1.2])

plot_recall_vs_threshold(precisions, recalls, thresholds)
plt.show()
```

```python
plt.plot(precisions, recalls, 'b-')
plt.xlabel('Precision')
plt.ylabel('Recalls')
plt.show()
```

If we closely look at the precison vs threshold graph and find that to get a `~90%` precision we need to use a `threshold` of `70000` then we can just simply make the predictions as the following without the use of the `predict()`.

```python
y_train_90_pred = (y_scores > 70000)
print ('Precision score:', precision_score(y_train_3, y_train_90_pred))
print ('Recall score:', recall_score(y_train_3, y_train_90_pred))
```

    Precision score: 0.9407153356197943
    Recall score: 0.6263252324253792

Great! we now have a `91%` precision classifier. It's pretty easy to create a classifier with any precision.

But, it comes with a cost, look at the recall score!

## The ROC Curve

The receiver operating characteristic (ROC) curve is another common tool with binary classifiers.

It plots the TPR (true positive rate) against the FPR (false positive rate) i.e. `sensitivity` vs `1-specificity`.

Where

\begin{align}
TPR = sensitivity = recall
\end{align}

And,

\begin{align}
FPR = 1 - TNR
\end{align}

where

\begin{align}
FPR = false positive rate
\end{align}

\begin{align}
TNR = specificity
\end{align}

```python
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_3, y_scores)
```

```python
def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([0, 1, 0, 1])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')

plot_roc_curve(fpr, tpr)
plt.show()
```

Once again, there is a trade-off between the TPR and FPR.

One way to compare classifiers is to measure the area under the curve(AUC). A perfect classifier will have a ROC AUC equal to `1`, whereas a purely randomly classifier will have a ROC AUC equal to `0.5`.

```python
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_3, y_scores)
```

    0.9598739627145828

As a rule of thumb, we should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negetives. And the ROC curve otherwise.

Let's train a `RandomForestClassifier` and compare its ROC curve and the ROC AUC score to the `SDGClassifier`.

```python
from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(random_state=42)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_3, cv=3, method='predict_proba')
y_probas_forest
```

    array([[1. , 0. ],
           [0.9, 0.1],
           [1. , 0. ],
           ...,
           [1. , 0. ],
           [1. , 0. ],
           [0.2, 0.8]])

Since to plot ROC curve we need the scores and probabilities, a simple solution is to use the positive class's probability as the score

```python
y_scores_forest = y_probas_forest[:, 1]
fpr_f, tpr_f, thresholds_f = roc_curve(y_train_3, y_scores_forest)

plt.plot(fpr, tpr, 'b:', label='SGD')
plot_roc_curve(fpr_f, tpr_f, 'Random Forest')
plt.legend(loc='bottom right')
plt.show()
```

We can see that `RandomForestClassifier`'s ROC curve looks much better than `SGDClassifier`'s.

And if we take a look at its roc_auc_score

```python
print ('ROC AUC Score:', roc_auc_score(y_train_3, y_scores_forest))

forest_clf.fit(X_train, y_train_3)
print ('Precision Score: ', precision_score(y_train_3, forest_clf.predict(X_train)))
print ('Recall Score: ', recall_score(y_train_3, forest_clf.predict(X_train)))
```

    ROC AUC Score: 0.9872408717258868
    Precision Score:  1.0
    Recall Score:  0.9879301908334692

Great!

So, now we know how to train binary classifiers:

1) Choose appropriate metric for your task

2) Evaluate your classifiers using cross validation

3) Select precision/recall tradeoff that fits the needs

4) Compare various models using ROC curves and ROC AUC scores.

Now, let's try to detect more than `5`.

## Multiclass Classification

Some algorithms are capable of handling multiple classes directly. Others are strictly binary classifiers. However, there are various strategies that you can use to perform multiclass classification using multiple binary classifiers:

1) OvA (one-versus-all)

2) OvO (one-versus-one)

One of the advantages of OvO is that each classifier only needs to be trained on the part of the training set for the two classes that it must distinguish.

Scikit-Learn detects when we try to use a binary classifier for a multiclass classification task and it automatically runs OvA (except for SVM classifiers, which uses OvO).

Let's try this with SGDClassifier

```python
sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit.ravel()])
```

    array([3.])

As said, under the hood, scikit learn uses OvA and trains `10` different classifiers and makes a predictions based on their decision scores.

To check this is the case, we can use the `decision_function()`

```python
some_digit_scores = sgd_clf.decision_function([some_digit.ravel()])
some_digit_scores
```

    array([[-138351.24191231, -350763.52460008, -273317.67754911,
             150957.38355489, -903511.79727288,  -91347.89672036,
            -380559.78962422, -254909.5057566 , -335632.13555252,
            -400516.91000383]])

There we have `10` different classifiers each providing a decision scores. The highest score is that of the class `5`.

```python
sgd_clf.classes_[np.argmax(some_digit_scores)]
```

    3.0

If we want to force scikit-learn to use `one-versus-one` or `one-versus-all` you can use the `OneVsOneClassifier` or `OneVsRestClassifier` classes. Simply create an instance and pass a binary classifier to its contructor.

```python
from sklearn.multiclass import OneVsOneClassifier
ovo_clf = OneVsOneClassifier(SGDClassifier(random_state=42))
ovo_clf.fit(X_train, y_train)
ovo_clf.predict([some_digit.ravel()])
```

    array([3.])

```python
len(ovo_clf.estimators_)
```

    45

Training a RandomForestClassifier is just as easy.

```python
forest_clf.fit(X_train, y_train)
forest_clf.predict([some_digit.ravel()])
```

    array([3.])

This time sklearn doesn't uses OvA or OvO cause RandomForestClassifier directly classify instances into multiple classes. We can use `.predict_proba()` to get the list of probabilities that the classifier assigned to each instance for each class.

```python
forest_clf.predict_proba([some_digit.ravel()])
```

    array([[0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]])

Let's evaluate the SGDClassifier now using `cross_val_score`

```python
cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring='accuracy')
```

    array([0.84998   , 0.85824291, 0.8660299 ])

It gets over 83% on all folds. Simply scaling the inputs would improve the accuracy over 90%

```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=4, scoring='accuracy')
```

    array([0.91189603, 0.90994534, 0.90885451, 0.91137046])

WooHoo! Over `90%` accuracy!

## Error Analysis

Let's start by looking at the confusion matrix.

```python
y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx
```

    array([[5748,    3,   23,    9,   10,   35,   45,   10,   38,    2],
           [   2, 6459,   49,   24,    6,   49,    6,   13,  121,   13],
           [  58,   33, 5357,   96,   76,   22,   93,   50,  153,   20],
           [  52,   44,  139, 5344,    3,  228,   36,   53,  135,   97],
           [  18,   28,   37,    6, 5347,   10,   56,   35,   91,  214],
           [  79,   41,   34,  185,   72, 4581,  114,   28,  184,  103],
           [  36,   24,   45,    2,   45,   86, 5632,    4,   44,    0],
           [  22,   22,   74,   26,   45,   10,    6, 5789,   17,  254],
           [  56,  163,   76,  162,   13,  163,   59,   25, 4979,  155],
           [  39,   28,   30,   87,  155,   28,    3,  193,   80, 5306]])

Let's look at an image representation of the matrix.

```python
plt.matshow(conf_mx, cmap=plt.cm.gray)
plt.show()
```

Since most images are on the main diagonal which means that they were classified correctly. The 5s look a slightly darker which could mean that there are fewer images of 5s in the dataset or that the classifier does not perform as well on 5s as on others.

```python
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
```

Now let's fill the diagonal with zeros to keep only the errors.

```python
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()
```

Analyzing individual errors can also be a good way to gain insights on what your classifier is doing and why it has errors.

```python
def plot_digits(X, img_per_row):
    '''fig, ax = plt.subplots(nrows=img_per_row, ncols=len(X)//img_per_row)
    i = 0
    for row in ax:
        for col in row:
            col.imshow(X[i].reshape(28, 28), cmap=matplotlib.cm.binary)
            i += 1
    '''
    plt.figure()
    for i in range(len(X)):
        plt.subplot(5, 5, i+1)
        plt.imshow(X[i].reshape(28, 28), cmap=matplotlib.cm.binary)

cl_a, cl_b = 3, 5

X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]

plt.figure()
plt.subplot(221); plot_digits(X_aa[:25], img_per_row=5)
plt.subplot(222); plot_digits(X_ab[:25], img_per_row=5)
plt.subplot(223); plot_digits(X_ba[:25], img_per_row=5)
plt.subplot(224); plot_digits(X_bb[:25], img_per_row=5)

plt.axis('off')
plt.show()
```

For some of the images a human would also go wrong, for others it seems like a obvious mistake and the reason is that we used a SGDClassifier whcih is a linear model.

So, this classifier is quite sensitive to image shifting and rotation. So one way to reduce the 3/5 confusion would be to center the image and make sure its not too rotated.

## Multilabel Classification

Until now each instance has always been classified to just one class. In some cases we may want a classifier to output multiple labels for each instance.

An example would be to output multiple binary labels (such a system is called multilabel classification system)

```python
from sklearn.neighbors import KNeighborsClassifier

y_train_large = (y_train >= 7)
y_train_odd = (y_train % 2 == 1)
y_multilabel = np.c_[y_train_large, y_train_odd]

knn_clf = KNeighborsClassifier()
knn_clf.fit(X_train, y_multilabel)
```

    KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
               metric_params=None, n_jobs=1, n_neighbors=5, p=2,
               weights='uniform')

`KNeighborsClassifier` supports multilabel classification and we train it using multiple labels.

Now, let's make a perdiction.

```python
knn_clf.predict([some_digit.ravel()])
```

    array([[False,  True]])

And it gets it right!! The digit is `3` so its less than `7` indicated by the `False` and its odd indicated by `True`.

There are many ways to evaluate a multilabel classifier and selecting the right metric really depends on your project. One approach is to measure the F1 score for each label and compute the average.

Following computes F1 score across all labels.

```python
#y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_train, cv=3)
#f1_score(y_train, y_train_knn_pred)
```

This assumes that all labels are equally important, which may not be the case.

In case, we want to give each label a weight based on its support (the number of instances with that target label) then we can just set `average='weighted'`

## Multioutput Classification

Multioutput-multilabel classification (or simply multioutput classification) is a generalization of multilabel classification where each label can be multiclass.

To illustrate this, we will build a system that removes noise from images. So, the output will be an array of pixel intensities (multilabel) and each pixel can have multiple values (multioutput)

```python
import random as rnd

noise_tr = rnd.randint(0, 100, (len(X_train), 784))
noise_tt = rnd.randint(0, 100, (len(X_test), 784))
X_train_mod = X_train + noise_tr
X_test_mod = X_test + noise_tt
y_train_mod = X_train
y_test_mod = X_test
```

Let's take a peek at an image from both the train set and test set (note this is not recommended to do as this can introduce snooping bias)

```python
plt.figure()
plt.subplot(1, 2, 1)
plt.imshow(X_train_mod[23].reshape(28, 28), cmap=matpltolib.cm.binary)

plt.subplot(1, 2, 2)
plt.imshow(X_test_mod[23].reshape(28, 28), cmap=matpltolib.cm.binary)

plt.show()
```

On the left is the noisy image and on the right is the clean target image. Now let's train the classifier and make it a clean image :)

```python
knn_clf.fit(X_train_mod, y_train_mod)
clean_digit = knn_clf.predict([X_test_mod[23]])

plt.imshow(clean_digit.reshape(28, 28), cmap=matplotlib.cm.binary)
plt.show()
```

Looks close to the target :)

## Conclusion

And we now know how to select good metrics for classification tasks, pick the appropriate precision/recall trade off, compare classifiers and more generally build good classification systems.


# Non-linear Breit-Wheeler pair creation

The pair production rate can be written as follows:

$$
\frac{d^2N_{BW}}{d\chi_\pm dt} =  \frac{\alpha_f m_e^2 c^4}{\pi \sqrt{3} \hbar \epsilon_\gamma \chi_\gamma}
\int_x^{+\infty} \sqrt{s} K_{1/3}\left({\frac{2}{3} s^{3/2}}\right) ds - \left({ 2 - \chi_\gamma x^{3/2} }\right) K_{2/3} \left({\frac{2}{3} x^{3/2}}\right)
$$

where $x = \left( \chi_\gamma / (\chi_- \chi_+) \right)^{2/3}$.

When I ask sympy to peform the integral, it remains stuck forever...

```python
from sympy import *
s = Symbol('s')
func = besselk(1/3, 2/3 * s**(3/2))*sqrt(s)
#print(integrate(func, s))Don't do that
```

However Wolfram Alpha provides an explicit form for the solution:

$$
\int \sqrt{s} K_{1/3}\left({\frac{2}{3} s^{3/2}}\right) ds = \\
\frac{
3^{5/6} \pi ~ s ~ \Gamma\left({\frac{1}{3}}\right) \Gamma\left({\frac{5}{3}}\right)
\phantom{f}_1F_2\left({ \frac{1}{3}; \frac{2}{3}, \frac{4}{3}; \frac{s^3}{9}}\right)
-
3^{1/6} \pi ~ s^2 ~ \Gamma\left({\frac{2}{3}}\right)^2
\phantom{f}_1F_2\left({ \frac{2}{3}; \frac{4}{3}, \frac{5}{3}; \frac{s^3}{9}}\right)
}
{9 \Gamma\left({\frac{2}{3}}\right) \Gamma\left({\frac{4}{3}}\right) \Gamma\left({\frac{5}{3}}\right)}
$$

If we find a good open source implementation of generalized hypergeometric functions, we could exploit this in the code.

Anyway, this notebook contains implementations of the pair production rate functions, in order to perform quick test of the library. Let's start with some useful definitions (for code units we use same normalization as smilei):

```python
import math as m
import numpy as np

electron_mass = 9.10938356e-31
elementary_charge = 1.6021766208e-19
speed_of_light = 299792458
reduced_plank = 1.054571800e-34
vacuum_permittivity =  8.854187817e-12
fine_structure_constant =  0.0072973525664

classical_elec_radius = (1./4./np.pi/vacuum_permittivity)*( elementary_charge**2 / (electron_mass * speed_of_light**2))

lambda_laser = 0.8e-6

schwinger_field_SI = electron_mass**2 * speed_of_light**3 / (reduced_plank*elementary_charge)
print('Schiwinger field in SI units is: ' + str(schwinger_field_SI))

schwinger_field_code_units = electron_mass*speed_of_light*lambda_laser/(2.0*reduced_plank*m.pi)
print('Schiwinger field in code units is: ' + str(schwinger_field_code_units))

conversion_of_rate_from_code_to_SI = (2.0 * m.pi * speed_of_light)/lambda_laser
print('Conv. factors for prod. rate from code units to SI: ' + str(conversion_of_rate_from_code_to_SI))

conversion_of_rate_from_SI_to_code = 1.0/conversion_of_rate_from_code_to_SI
print('Conv. factors for prod. rate from SI to code units: ' + str(conversion_of_rate_from_SI_to_code))

field_reference_PIC = 2.0 * np.pi * electron_mass * speed_of_light * speed_of_light / (elementary_charge*lambda_laser)
print('Field reference PIC: {:e}'.format(field_reference_PIC))
```

    Schiwinger field in SI units is: 1.3232854660718482e+18
    Schiwinger field in code units is: 329718.7590461917
    Conv. factors for prod. rate from code units to SI: 2354564459136066.5
    Conv. factors for prod. rate from SI to code units: 4.247069967100916e-16
    Field reference PIC: 4.013376e+12

Let's define the $\chi_\gamma $ function. We will assume that all these quantities are provided in code units (same normalization as Smilei)

```python
def calc_chi_gamma(p, E, B):
    gamma_phot = np.linalg.norm(p)
    c = p/gamma_phot
    loc_field = gamma_phot * np.linalg.norm( E - np.dot(c,E)*c + np.cross(c,B))
    return loc_field/schwinger_field_code_units

#A quick tests
a = calc_chi_gamma(np.array([83.759, 139.311, -230.553]), np.array([-166.145, -78.231, -278.856]), np.array([-279.174, -158.849, -93.826]))

b = calc_chi_gamma(np.array([-201.076, -170.956, 66.094]), np.array([-201.968, -114.862, 3.580]), np.array([-195.188, 223.384, -204.352]))

c = calc_chi_gamma(np.array([-219.959, 135.362, 135.086]), np.array([110.184, 272.527, 64.938]), np.array([-127.926, -21.659, -59.226]))

d = calc_chi_gamma(np.array([-2314.45, -2356.30, 546.28]), np.array([1230.11, 1638.02, -2911.04]), np.array([-2203.66, 1243.79, -2830.99]))

e = calc_chi_gamma(np.array([9.2627, -25.4575, -10.2246]), np.array([2.9271, 10.4293, 3.6103]), np.array([1.7439, 1.9778, 17.8799]))

print(a)
print(b)
print(c)
print(d)
print(e)

```

    0.347111844317
    0.315109658214
    0.200401320634
    57.2204397969
    0.000904147405336

We now define

$$
inner(x) = \int_x^{+\infty} \sqrt{s} K_{1/3}\left({\frac{2}{3} s^{3/2}}\right) ds
$$

and

$$
X(\chi_\gamma, \chi_-) = \left( \frac{\chi_\gamma}{\chi_- (\chi_\gamma - \chi_- )}   \right)^{2/3}
$$

and

$$
T(\chi_\gamma) = \frac{1}{\pi \sqrt{3} \chi^2_\gamma} \int_0^{+\infty} inner\left({X(\chi_\gamma, \chi_-)}\right) - \left({ 2 - \chi_\gamma X(\chi_\gamma, \chi_-)^{3/2} }\right) K_{2/3} \left({\frac{2}{3} X(\chi_\gamma, \chi_-)^{3/2}}\right) d\chi_-
$$

```python
import scipy.special as spe
import scipy.integrate as integ

def inner(x):
    return integ.quad(lambda s: np.sqrt(s)*spe.kv(1./3., 2./3. * s**(3./2.)), x, np.inf)[0]

inner = np.vectorize(inner)

def X(chi_phot, chi_ele):
    if (chi_phot > chi_ele and chi_ele != 0):
        return np.power(chi_phot/(chi_ele*(chi_phot-chi_ele)), 2./3.)
    else:
        return 1.0e30

X = np.vectorize(X)

def T(chi_phot):
    coeff = 1./(np.pi * np.sqrt(3.) * chi_phot * chi_phot)
    return integ.quad(lambda chi_ele:
                      coeff*(inner(X(chi_phot, chi_ele)) -
                      (2.0 - chi_phot*np.power(X(chi_phot, chi_ele), 3./2.))*spe.kv(2./3., 2./3. *X(chi_phot, chi_ele)**(3./2.)) )
                      , 0, chi_phot)[0]
T = np.vectorize(T)

```

We can now write the total production rate as:

$$
\frac{dN_{BW}}{dt} = \frac{\alpha_f m_e^2 c^4}{\hbar \epsilon_\gamma} \chi_\gamma T(\chi_\gamma)
$$

we also note that $\epsilon_\gamma / m_e c^2$ is the  photon energy in code units.
So we can write the following function, which gives the total production rate *in SI units* as a function of quantities *in code units*.

```python
def dNBW_dt(chi_phot, e_phot):
    val = ((electron_mass*(speed_of_light)**2)*fine_structure_constant/reduced_plank)*(chi_phot/e_phot)*T(chi_phot)
    return val

px = 9000
chi = calc_chi_gamma(np.array([px,0,0]),np.array([0,500,0]),np.array([0,0,0]))
rate = dNBW_dt(calc_chi_gamma(np.array([px,0,0]),np.array([0,500,0]),np.array([0,0,0])), px)
print(str(chi) + " " + str(rate) + " " + str(rate/1e15))

p = np.array([61019.1, -24359.3, 65116.2])
E = np.array([69942.0, 38024.7, -43604.1])
B = np.array([-26990.0, 58267.8, -63485.8])
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)

p = np.array([-965.61, -3975.11, 6917.22])
E = np.array([11.17, -2117.72, -1407.19])
B = np.array([6259.79, 7557.54, 773.11])
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)
p = np.array([149.825, 933.115, -538.195])
E = np.array([931.686, -861.074, 944.652])
B = np.array([531.406, 670.933, 660.057])
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)
p = np.array([-44.4546, -0.2033, 94.5843])
E = np.array([39.8996, -29.2501, 58.7720])
B = np.array([44.3417, 15.5024, 29.4024])
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)
p = np.array([6.81696,9.68933,2.81229])
E = np.array([-4.89986,-9.65535,3.69471])
B = np.array([8.89549,-5.46574,-6.75393])
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)
p = np.array([-2552.37,3970.67,-62.9812])
E = np.array([800,800,800])
B = np.array([800,800,800])
print("chi:" + str(calc_chi_gamma(p, E, B)))
print(dNBW_dt(np.asscalar(calc_chi_gamma(p, E, B)), np.asscalar(np.linalg.norm(p)))*conversion_of_rate_from_SI_to_code)
```

    13.6479950762 9.32177159432e+14 0.932177159432
    7.63488202211
    3.51855878777
    1.50648551484
    4.69766211952e-73
    0.0
    chi:27.667906063
    1.41557597076

```python
px = np.logspace(-1, 8.0, num=100, base=10)

def ff(px):
    return dNBW_dt(calc_chi_gamma(np.array([px,0,0]),np.array([0,500,0]),np.array([0,0,0])), px)

ff = np.vectorize(ff)

def cc(px):
    return calc_chi_gamma(np.array([px,0,0]),np.array([0,500,0]),np.array([0,0,0]))

cc = np.vectorize(cc)

ff(px)

import matplotlib.pyplot as plt

plt.loglog(cc(px), ff(px)*1e-15)
axes = plt.gca()
axes.set_xlim([0.1,100000])
axes.set_ylim([0.00001,1])
plt.show()
```

    <matplotlib.figure.Figure at 0x7f78b280f080>

# Quantum synchrotron emission

```python
def calc_chi_elec(p, E, B):
    gamma_elec = sqrt(1.0 + np.dot(p,p))
    v = p/gamma_elec
    evcb = E + np.cross(v,B)
    loc_field = gamma_elec * sqrt(abs(np.dot(v,E)**2 - np.dot(evcb,evcb)))
    return loc_field/schwinger_field_code_units

calc_chi_elec(np.array([1000, 0, 0]), np.array([0, 0, 0]), np.array([0, 0, 500]))
```

    1.51644389735785

```python
chi_phot_min = 0.01

def comp_y(chi_phot, chi_part):
    if(chi_part == chi_phot):
        return 0.0
    return chi_phot/(3*chi_part*(chi_part-chi_phot))

def inner_fun(y):
    return integ.quad(lambda s: spe.kv(1./3.,s), 2*y, np.inf)[0]

def KK_int(chi_part, chi_phot):
    yy = comp_y(chi_phot, chi_part)
    inner = inner_fun(yy)
    part_2 = spe.kv(2./3.,2*yy)*(2 + 3*chi_phot*yy)
    return ( part_2 - inner)/(pi*sqrt(3))

def KK_func(chi_part):
    return integ.quad(lambda s: KK_int(chi_part,s), chi_phot_min, chi_part)[0]

tau_e = classical_elec_radius/speed_of_light

def qs_dn_dt(chi_part, gamma_part):
    return (fine_structure_constant/tau_e)*KK_func(chi_part)/(chi_part*gamma_part)

print(KK_func(0.001))
print(KK_func(0.01))
print(KK_func(0.1))
print(KK_func(1))
print(KK_func(10.))
print(KK_func(100.))
print(KK_func(10000.))
```

    /usr/lib/python3/dist-packages/scipy/integrate/quadpack.py:364: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      warnings.warn(msg, IntegrationWarning)

    nan
    nan
    0.0013032229454470994
    0.7032369286240776
    53.93181891450526
    2912.9421286618954
    6698656.806401038

# Miscellanea

```python
from decimal import Decimal

I = 1e24
I_W_m2 = I * 1e4
E = sqrt(I_W_m2 * 2 / speed_of_light / vacuum_permittivity)
print("{:6.3e}".format(E))
print("{:6.3e}".format(E/field_reference_PIC))
print("{:6.3e}".format(E/schwinger_field_SI))

B = 2500000
E = B*speed_of_light/speed_of_light
calc_chi_elec(np.array([1800, 0, 0]), np.array([0, 0, 0]), np.array([0, 0, E/field_reference_PIC]))
```

    2.745e+15
    6.839e+2
    2.074e-3

    3.40062678490544e-9

```python
a = calc_chi_elec(np.array([24.3752, -11.5710, -10.0841]), np.array([57.185, -16.6555, 22.4340]), np.array([6.6911, -23.8724, 13.9934]))
b = calc_chi_elec(np.array([4.015, 197.287, 141.705]), np.array([30.287, 115.740, 120.891]), np.array([-190.161, -129.115, -57.002]))
c = calc_chi_elec(np.array([-2534.83, 1011.54, -793.04]), np.array([741.67, -2359.97, 1463.50]), np.array([1477.19, -1448.33, 1953.68]))

print(a)
print(b)
print(c)
```

    0.00216716627219670
    0.166318112874468
    16.0114572646993


#  CS109B Data Science 2: Advanced Topics in Data Science

## Lab 02 - Review of Probability Distributions

**Harvard University**

**Spring 2022**

**Instructors:** Mark Glickman and Pavlos Protopapas

**Lab instructor and content:** Eleni Angelaki Kaxiras

---

```python
import warnings

import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import pymc3 as pm
import theano.tensor as tt

from pymc3 import summary

warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.filterwarnings('ignore')

import matplotlib.pyplot as plt
from matplotlib import gridspec
import scipy.stats as stats
import pandas as pd
import seaborn as sns
%matplotlib inline

import warnings
print('Running on PyMC3 v{}'.format(pm.__version__))
```

    Running on PyMC3 v3.11.4

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 20000;
```

```python
#pandas trick
pd.options.display.max_columns = 50  # None -> No Restrictions
pd.options.display.max_rows = 200    # None -> Be careful with this
pd.options.display.max_colwidth = 100
pd.options.display.precision = 3
```

## A review of Common Probability Distributions

### Discrete Distributions

The random variable has a **probability mass function (pmf)** which measures the probability that our random variable will take a specific value $y$, denoted $P(Y=y)$.

- **Bernoulli** (binary outcome, success has probability $\theta$, $one$ trial):
$
P(Y=k) =  \theta^k(1-\theta)^{1-k}, k \in (0, 1)
$

- **Binomial** (binary outcome, success has probability $\theta$, $k$ sucesses, $n$ trials):
\begin{equation}
P(Y=k) =  {{n}\choose{k}} \cdot \theta^k(1-\theta)^{n-k}
\end{equation}

*Note*: Binomial(1,$p$) = Bernouli($p$)

- **Poisson** (counts independent events occurring at a rate $\lambda$)
\begin{equation}
P\left( Y=y|\lambda \right) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}}
\end{equation}
y = 0,1,2,...

- **Categorical, or Multinulli** (random variables can take any of K possible categories, each having its own probability; this is a generalization of the Bernoulli distribution for a discrete variable with more than two possible outcomes, such as the roll of a die)

### Continuous Distributions

The random variable has a **probability density function (pdf)**.
- **Uniform** (variable equally likely to be near each value in interval $(a,b)$)
\begin{equation}
P(X = x) = \frac{1}{b - a}
\end{equation}
anywhere within the interval $(a, b)$, and zero elsewhere.

- **Normal** (a.k.a. Gaussian)
\begin{equation}
X \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

    A Normal distribution can be parameterized either in terms of precision $\tau$ or variance $\sigma^{2}$. The link between the two is given by
\begin{equation}
\tau = \frac{1}{\sigma^{2}}
\end{equation}
 - Expected mean $\mu$
 - Variance $\frac{1}{\tau}$ or $\sigma^{2}$
 - Parameters: `mu: float`, `sigma: float` or `tau: float`
 - Range of values (-$\infty$, $\infty$)

- **Beta** (where the variable ($\theta$) takes on values in the interval $[0,1]$, and is parametrized by two positive parameters, $\alpha$ and $\beta$ that control the shape of the distribution. Note that Beta is a good distribution to use for priors (beliefs) because its range is $[0,1]$ which is the natural range for a probability and because we can model a wide range of functions by changing the $\alpha$ and $\beta$ parameters. Its density is:

\begin{equation}
\label{eq:beta}
P(\theta|a,b) = \frac{1}{B(\alpha, \beta)} {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1} \propto {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1}
\end{equation}

where the normalisation constant, $B$, is a beta function of $\alpha$ and $\beta$,

\begin{equation}
B(\alpha, \beta) = \int_{t=0}^1 t^{\alpha - 1} (1 - t)^{\beta - 1} dt.
\end{equation}
 - 'Nice', unimodal distribution
 - Range of values $[0, 1]$

- **Exponential**

 - Range of values [$0$, $\infty$]

- **Gamma**

 #### Code Resources:
 - Statistical Distributions in numpy/scipy: [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)
 - Statistical Distributions in pyMC3: [distributions in PyMC3](https://docs.pymc.io/api/distributions.html) (we will see when we look at PyMC3).

Exercises: Discrete Probability Plots

#### Poisson
Change the value of $\lambda$ in the Poisson PMF and see how the plot changes. Remember that the y-axis in a discrete probability distribution shows the probability of the random variable having a specific value in the x-axis.

\begin{equation}
P\left( X=y \right|\lambda) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}}
\end{equation}

for $y \ge0$.

Routine is `stats.poisson.pmf(x, lambda)`. $\lambda$ is our $\theta$ in this case. $\lambda$ is also the mean in this distribution.

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 60)
for lam in [0.5, 3, 8]:
    pmf = stats.poisson.pmf(x, lam)
    plt.plot(x, pmf, alpha=0.5, label='$\lambda$ = {}'.format(lam))
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability', fontsize=12)
plt.legend(loc=1)
plt.ylim=(-0.1)
plt.show()
```

#### Binomial

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 50)
ns = [10, 17]
ps = [0.5, 0.7]
for n, p in zip(ns, ps):
    pmf = stats.binom.pmf(x, n, p)
    plt.plot(x, pmf, alpha=0.5, label='n = {}, p = {}'.format(n, p))
plt.xlabel('x', fontsize=14)
plt.ylabel('f(x)', fontsize=14)
plt.legend(loc=1)
plt.show()
```

Exercise: Continuous Distributions Plot

#### Uniform

Change the value of $\mu$ in the Uniform PDF and see how the plot changes.

Remember that the y-axis in a continuous probability distribution does not shows the actual probability of the random variable having a specific value in the x-axis because that probability is zero!. Instead, to see the probability that the variable is within a small margin we look at the integral below the curve of the PDF.

The uniform is often used as a noninformative prior.

```
Uniform - numpy.random.uniform(a=0.0, b=1.0, size)
```

$\alpha$ and $\beta$ are our parameters. `size` is how many tries to perform.
Our $\theta$ is basically the combination of the parameters a,b. We can also call it
\begin{equation}
\mu = (a+b)/2
\end{equation}

```python
from scipy.stats import uniform

r = uniform.rvs(size=1000)
plt.plot(r, uniform.pdf(r),'r-', lw=5, alpha=0.6, label='uniform pdf')
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.ylabel(r'probability density')
plt.xlabel(f'random variable')
plt.legend(loc='best', frameon=False)
plt.show()
```

#### Beta

We get an amazing set of shapes by tweaking the two parameters $a$ and $b$! Notice that for $a=b=1.$ we get a constant. From then on, as the values increase, we get a curve that looks more and more Gaussian.

```python
from scipy.stats import beta

fontsize = 15
alphas = [0.5, 0.5, 1., 3., 6.]
betas = [0.5, 1., 1., 3., 6.]
x = np.linspace(0, 1, 1000)
colors = ['red', 'green', 'blue', 'black', 'pink']

fig, ax = plt.subplots(figsize=(8, 5))

for a, b, colors in zip(alphas, betas, colors):
    dist = beta(a, b)
    plt.plot(x, dist.pdf(x), c=colors,
             label=f'a={a}, b={b}')

ax.set_ylim(0, 3)

ax.set_xlabel(r'$\theta$', fontsize=fontsize)
ax.set_ylabel(r'P ($\theta|\alpha,\beta)$', fontsize=fontsize)
ax.set_title('Beta Distribution', fontsize=fontsize*1.2)

ax.legend(loc='best')
fig.show();
```

#### Gaussian

```python
y = pm.Exponential.dist(lam=2),
#y = pm.Binomial.dist(n=10, p=0.5)
type(y)
#print(y.logp(4).eval())
#plt.plot(y.random(size=30))
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.]
sigmas = [0.4, 1., 2., 0.4]
for mu, sigma in zip(mus, sigmas):
    pdf = stats.norm.pdf(x, mu, sigma)
    plt.plot(x, pdf, label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

 At home: Prove the formula mentioned in class which gives the probability density for a Beta distribution with parameters $2$ and $5$:

$p(\theta|2,5) = 30 \cdot \theta(1 - \theta)^4$

**References**:

- [Distributions in PyMC3](https://docs.pymc.io/api/distributions.html)

Information about PyMC3 functions including descriptions of distributions, sampling methods, and other functions, is available via the `help` command.


# Construcción Curva Libor USD 3M

```python
from IPython.display import Image
```

## Valor de un Cupón Libor

Tenemos que (notebook 4):

$$
\begin{equation}
\Pi\left(t,X\right)=\mathbb{E}_t^{Q_f}\left[D_c\left(t,T\right)\Pi\left(T,X\left(T\right)\right)\right]
\end{equation}
$$

$$
\begin{equation}
D_c\left(t,T\right)=\exp\left[-\int_t^Tr_c\left(u\right)du\right]
\end{equation}
$$

$$
\begin{equation}
\mathbb{E}_t^{Q_f}\left[D_c\left(t,T\right)\right]=P_c\left(t,T\right)
\end{equation}
$$

$$
\begin{equation}
D_c\left(t,T\right)={B_c\left(t,T\right)}^{-1}
\end{equation}
$$

También vamos a necesitar:

$$
\begin{equation}
\Pi\left(t,X\right)=P_c\left(t,T\right)\mathbb{E}_{t}^{Q_f^T}\left[\Pi\left(T,X\right)\right]
\end{equation}
$$

$$
\begin{equation}
P_c\left(t,T\right)=\mathbb{E}^{Q_f}\left[D_c\left(t,T\right)\right]
\end{equation}
$$

donde $Q_f^T$ es la probabilidad asociada a $P_c\left(t,T\right)$ (la medida $T$-forward).

Sea $L\left(T\right)$ el valor de la Libor USD de $tenor=3M$ al tiempo $T$ y sea $\Delta\left(T,T_{3M}\right)$ la fracción de año asociada a esa Libor y $T_{3M}$ es la fecha $3M$ después de $T$. Para un cupón flotante a Libor (suponiendo un nocional igual a 1), tenemos que:

$$
\begin{equation}
\Pi\left(T,X\left(T\right)\right)=L\left(T\right)\cdot\Delta\left(T,T_{3M}\right)
\end{equation}
$$

Y por lo tanto,

$$
\begin{equation}
\Pi\left(t,X\left(t\right)\right)=\mathbb{E}_t^{Q_f}\left[D_c\left(t,T_{3M}\right)L\left(T\right)\cdot\Delta\left(T,T_{3M}\right)\right]
\end{equation}
$$

Finalmente, utilizando (5) y (6) y la linealidad del valor esperado se llega a:

$$
\begin{equation}
\Pi\left(t,X\left(t\right)\right)=\Delta\left(T,T_{3M}\right)\cdot P_c\left(t,T\right)\cdot\mathbb{E}_t^{Q_f^T}\left[L\left(T\right)\right]
\end{equation}
$$

## Cotizaciones Libor USD 3M vs Tasa Fija

```python
Image(url="img/libor_usd_3m_fixed_swap_quotes.gif", width=900, height=720)
```

Como se puede observar en la imagen anterior, existen cotizaciones de swaps Libor USD 3M versus tasa fija para plazos hasta 30Y. Utilizando la fórmula (9) y lo que ya vimos sobre la construcción de una curva a partir de las cotizaciones de OIS ¿Cómo se escribe la fórmula de valorización para un swap Libor?

Para fijar las ideas, consideremos el swap cuyo $tenor=1Y$. Recordemos (notebook 2, 2.2.1) que, en estos swaps, la pata fija tiene periodicidad semestral y convención de tasa 30/360, mientras que la pata flotante tiene trimestral y convención de tasa Act/360.

Tenemos que, el valor de la pata fija es:

$$
\begin{equation}
\Pi^{fija,1Y}\left(t\right)=P_c\left(t,T_1\right)\cdot r\cdot\delta\left(T_0,T_1\right)+P_c\left(t,T_2\right)\left(1+r \cdot \delta\left(T_1,T_2\right)\right)
\end{equation}
$$

Donde $P_c\left(t,T\right)$ proviene de la curva OIS (ver notebook 5) y $\delta\left(T_i,T_j\right)$ representa la fracción de año en convención 30/360 entre $T_i$ y $T_j$.

Por otro lado, el valor de la pata flotante es:

$$
\begin{equation}
\Pi^{flot,1Y}\left(T_0\right)=\sum\limits_{j=1}^{3}\left\{P_c\left(T_0,T_{3M}^j\right)\cdot\mathbb{E}_t^{Q_f^T}\left[L\left(T_j\right)\right]\cdot\Delta\left(T_j,T_{3M}^j\right)\right\}+P_c\left(T_0,T_{4}\right)\cdot\left(1+\cdot\mathbb{E}_t^{Q_f^T}\left[L\left(T_4\right)\right]\cdot\Delta\left(T_4,T_{3M}^4\right)\right)
\end{equation}
$$

Donde $\Delta\left(T_i,T_{3M}^i\right)$ representa la fracción de año en convención Act/360 correspondiente la Libor que parte en $T_i$.

Podemos observar que esta ecuación tiene 4 incógnitas, y por lo tanto, para poder aplicar un procedimiento de bootstrapping como para los OIS, se requiere disponer de data de corto plazo, que permita deducir algunos valores iniciales.

Afortunadamente, dicha data sí existe y se puede apreciar en la imagen siguiente:

```python
Image(url="img/20201016_libor_curve.gif", width=900, height=720)
```

La primera de las tasas marcadas en rojo corresponde a la Libor USD 3M del 16-10-2020. Las siguientes 6 tasas, se obtienen de las cotizaciones de futuros [eurodollar](https://www.cmegroup.com/trading/interest-rates/stir/eurodollar.html) con esos *nemotécnicos*.


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing.

## Introduction

The response of a system $y[k] = \mathcal{H} \{ x[k] \}$ to a random input signal $x[k]$ is the foundation of statistical signal processing. In the following we limit ourselves to [linear-time invariant (LTI) systems](https://en.wikipedia.org/wiki/LTI_system_theory).

In the following it is assumed that the statistical properties of the input signal $x[k]$ are known. For instance, its first- and second-order ensemble averages. It is further assumed that the impulse response $h[k]$ or the transfer function $H(e^{\,\mathrm{j}\,\Omega})$ of the LTI system is given. We are looking for the statistical properties of the output signal $y[k]$ and the joint properties between the input $x[k]$ and output $y[k]$ signal.

## Stationarity and Ergodicity

The question arises if the output signal $y[k]$ of an LTI system is (wide-sense) stationary or (wide-sense) ergodic for an input signal $x[k]$ exhibiting the same properties.

Let's assume that the input signal $x[k]$ is drawn from a stationary random process. According to the [definition of stationarity](../random_signals/stationary_ergodic.ipynb#Definition) the following relation must hold

\begin{equation}
E\{ f(x[k_1], x[k_2], \dots) \} = E\{ f(x[k_1 + \Delta], x[k_2 + \Delta], \dots) \}
\end{equation}

where $\Delta \in \mathbb{Z}$ denotes an arbitrary (temporal) shift and $f(\cdot)$ an arbitary mapping function. The condition for time-invariance of a system reads

\begin{equation}
y[k + \Delta] = \mathcal{H} \{ x[k + \Delta] \}
\end{equation}

for $y[k] = \mathcal{H} \{ x[k] \}$. Applying the system operator $\mathcal{H}\{\cdot\}$ to the left- and right-hand side of the definition of stationarity for the input signal $x[k]$ and recalling the linearity of the expectation operator $E\{\cdot\}$ yields

\begin{equation}
E\{ g(y[k_1], y[k_2], \dots) \} = E\{ g(y[k_1 + \Delta], y[k_2 + \Delta], \dots) \}
\end{equation}

where $g(\cdot)$ denotes an arbitrary mapping function that may differ from $f(\cdot)$. From the equation above, it can be concluded that the output signal of an LTI system for a stationary input signal is also stationary. The same reasoning can also be applied to an [ergodic](../random_signals/stationary_ergodic.ipynb#Ergodic-Random-Processes) input signal. Since both wide-sense stationarity (WSS) and wide-sense ergodicity are special cases of the general case, the derived results hold also here.

Summarizing, for an input signal $x[k]$ that is

* (wide-sense) stationary, the output signal $y[k]$ is (wide-sense) stationary and the in- and output is jointly (wide-sense) stationary
* (wide-sense) ergodic, the output signal $y[k]$ is (wide-sense) ergodic and the in- and output is jointly (wide-sense) ergodic

This implies for instance, that for a WSS input signal $x[k]$ measures like the auto-correlation function (ACF) can also be applied to the output signal $y[k] = \mathcal{H} \{ x[k] \}$ of an LTI system.

### Example - Response of an LTI system to a random signal

The following example computes and plots estimates of the linear mean $\mu[k]$ and auto-correlation function (ACF) $\varphi[k_1, k_2]$ for the in- and output of an LTI system. The input $x[k]$ is drawn from a normally distributed white noise process.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

L = 64  # number of random samples
N = 1000  # number of sample functions

# generate input signal (white Gaussian noise)
np.random.seed(1)
x = np.random.normal(size=(N, L))
# generate output signal
h = 2*np.fft.irfft([1,1,1,0,0,0])
y = np.asarray([np.convolve(x[n,:], h, mode='same') for n in range(N)])

# compute and plot results
def compute_plot_results(x):

    # estimate linear mean by ensemble average
    mu = 1/N * np.sum(x, 0)
    # estimate the auto-correlation function
    acf = np.zeros((L, L))
    for n in range(L):
        for m in range(L):
            acf[n, m] = 1/N * np.sum(x[:, n]*x[:, m], 0)

    plt.subplot(121)
    plt.stem(mu)
    plt.title(r'Estimate of linear mean $\hat{\mu}[k]$')
    plt.xlabel(r'$k$')
    plt.ylabel(r'$\hat{\mu}[k]$')
    plt.axis([0, L, -1.5, 1.5])

    plt.subplot(122)
    plt.pcolor(np.arange(L), np.arange(L), acf, vmin=-2, vmax=2)
    plt.title(r'Estimate of ACF $\hat{\varphi}[k_1, k_2]$')
    plt.xlabel(r'$k_1$')
    plt.ylabel(r'$k_2$')
    plt.colorbar()
    plt.autoscale(tight=True)

plt.figure(figsize = (10, 5))
plt.gcf().suptitle(r'Input signal $x[k]$', fontsize=12)
compute_plot_results(x)

plt.figure(figsize = (10, 5))
plt.gcf().suptitle(r'Output signal $y[k]$', fontsize=12)
compute_plot_results(y)
```

**Exercise**

* Are the in- and output signals WSS?
* Can the output signal $y[k]$ be assumed to be white noise?

Solution: From the shown results it can assumed for the input signal $x[k]$ that the linear mean $\mu_x[k]$ does not depend on the time-index $k$ and that the ACF $\varphi_{xx}[k_1, k_2]$ does only depend on the difference $\kappa = k_1 - k_2$ of both time indexes. The same holds also for the output signal $y[k]$. Hence both the in- and output signal are WSS. Although the input signal $x[k]$ can be assumed to be white noise, the output signal $y[k]$ is not white noise due to its ACF $\varphi_{yy}[k_1, k_2]$.