# Direct Fidelity Estimation

Using a method known as direct fidelity estimation (DFE), see [DFE1] and [DFE2], it is possible to estimate the fidelity between
* a target pure state $\rho_\psi = |\psi\rangle\langle \psi|$ and its experimental realization $\sigma$,
* a target unitary $U$ and its experimental realization $U_e$.

This can be done with a small number (relative to state and process tomography) of simple experimental settings that is independent of the system size. Such methods are useful for the experimental study of larger quantum information processing units.

In this notebook we explore some state and process DFE using the forest.benchmarking module `direct_fidelity_estimation.py`.

| [DFE1] *Practical Characterization of Quantum Devices without Tomography.*
|        Silva et al.
|        PRL 107, 210404 (2011).
|        https://doi.org/10.1103/PhysRevLett.107.210404
|        https://arxiv.org/abs/1104.3835

| [DFE2]  *Direct Fidelity Estimation from Few Pauli Measurements.*
|        Flammia et al.
|        PRL 106, 230501 (2011).
|        https://doi.org/10.1103/PhysRevLett.106.230501
|        https://arxiv.org/abs/1104.4695

## Simplistic state DFE example

Suppose we have tried to prepare the state $|0\rangle$, with state matrix $\rho_0 = |0\rangle \langle 0 |$, but in fact prepared the state

$$\sigma = \frac 1 2 (I + x X + y Y + z Z)$$

The usual way to quantify how close $\sigma$ and $\rho_0$ are is to use quantum state tomography
to estimate $\sigma$ and then calculate the fidelity between $\sigma$ and $\rho_0$.

DFE provides a way to directly estimate the fidelity without first estimating the state $\sigma$. To see this, first note that since $\rho_0$ is a pure state (our estimate $\sigma$ in general is not) we can write the [fidelity](https://en.wikipedia.org/wiki/Fidelity_of_quantum_states#Definition) as $F(\rho_0, \sigma) = \langle 0 |\sigma|0 \rangle$. This is equivalent to

$$F(\rho_0, \sigma) = {\rm Tr}[\rho_0  \sigma]$$

using the [cyclic property of the trace](https://en.wikipedia.org/wiki/Trace_(linear_algebra%29#Cyclic_property).
Next we parameterize the pure state as

$$\rho_0= |0\rangle \langle 0| =  \frac 1 2 \big (I + (0) X + (0) Y + (+1) Z \big)=  \frac 1 2 (I +  Z )$$

Finally we arrive at

$$F(\rho_0, \sigma)= {\rm Tr}[\rho_0 \sigma] = \frac 1 4 {\rm Tr}[(1+z)I]= \frac{(1+z)}{2}$$

This result shows that we only need to estimate one observable expectation $\langle Z \rangle \rightarrow \tilde z$ in order to estimate the fidelity between $\rho_0$ and $\sigma$ in this particular example.

## Generalizing state DFE, somewhat

The code that we've developed has some assumptions baked in that are important to understand. We have implemented DFE for a restricted subset of states and processes using these assumptions.

To start, we assume that our start state is always the pure state $|0 \rangle$ on all qubits. On $n$ qubits this state can be decomposed as

$$\rho_0^{\otimes n} = \left(I_0 + Z_0\right) \otimes \left(I_1 + Z_1\right) \otimes \dots \left(I_n + Z_n\right) / 2^n  = \frac{1}{2^n} \sum_{k=1}^{2^n} P^{(IZ)}_k$$

where we have expanded the tensor multiplication into one large sum over all possible of combinations $P^{(IZ)}_k$ of I and Z Pauli terms. From this start point we apply the user specified program $U$, or rather the noisy implementation $\mathcal E( \cdot )$, to prepare the state $\sigma$.

$$\sigma = \mathcal E(\rho_0^{\otimes n})$$

Meanwhile we want to calculate the fidelity between $\sigma$ and the ideally prepared state $\rho = U \rho_0^{\otimes n} U^\dagger$. By linearity we can calculate $\rho$ by conjugating each Pauli term $P^{(IZ)}$ in the sum above by the ideal program $U$.

$$\rho = \frac{1}{2^n} \sum_k U P^{(IZ)}_k U^\dagger$$

**The assumption** we make is that the ideal prep program $U$ is an element of the Clifford group. Since the Clifford group is the [normalizer](https://en.wikipedia.org/wiki/Centralizer_and_normalizer) of the Pauli group, these conjugated terms $U P^{(IZ)}_k U^\dagger$ will again be some Pauli $P'_k$ (in the code we use a `pyquil.BenchmarkConnection` to quickly calculate the resultant Pauli). Following the example above we arrive at the set of $2^n$ Paulis $\{P'_k\}_k$ whose expectations we need to estimate with respect to the physical state $\sigma$. The estimate is the average of all these Pauli expectations (see Eqn. 1 of [DFE1]).

$$F(\rho_0^{\otimes n}, \sigma)= {\rm Tr}[\rho_0^{\otimes n} \sigma] = \frac{1}{2^n} \sum_{k=1}^{2^n} {\rm Tr}[ \mathcal E(\rho_0^{\otimes n}) P'_k]$$

Note that we don't need to estimate the expectation of the trivial all $I$ term since we know its expectation is 1. We exclude this term in experiment generation but include it in the analysis 'by hand'.

## Process DFE

The [average gate fidelity](https://arxiv.org/pdf/quant-ph/0205035.pdf) between an experimental process $\mathcal E$ and the ideal (unitary) process $\mathcal U$ is given by

$$F(\mathcal U,\mathcal E) = \frac{ {\rm Tr} [\mathcal E \mathcal U^\dagger] + d} {d^2+d}$$

where the processes are represented by linear superoperators acting on vectorized
density matrices, and d is the dimension of the Hilbert space $\mathcal H$ that $\mathcal E$ and $\mathcal U$ act on. If you are unfamiliar with these terms look at [superoperator tools notebook](superoperator_tools.ipynb) and [superoperator_representations.md](../superoperator_representations.rst)

Using the $d^2$ dimensional orthonormal Pauli basis $\{P_k\}$ for superoperators on $\mathcal H$--e.g. $\{I/\sqrt{2}, X/\sqrt{2}, Y/\sqrt{2}, Z/\sqrt{2} \}$ for a single qubit--we can expand the trace and insert an identity superoperator between $\mathcal U$ and $\mathcal E$; this amounts to re-casting these superoperators in the Pauli-Liouville representation (aka Pauli Transfer Matrix). (again, see [superoperator_representations.md](../superoperator_representations.rst) if you are unfamiliar with vec notation $P_k \iff \left| P_k \rangle\rangle\langle\langle P_k\right|$):

$${\rm Tr} [\mathcal E \mathcal U^\dagger] = {\rm Tr} [\mathcal E   \left( \sum_k \left| P_k \rangle\rangle\langle\langle P_k\right| \right) \mathcal U^\dagger]$$

$$= \sum_k {\rm Tr} [  \left( \langle\langle P_k | \mathcal U^\dagger \right) \left(\mathcal E | P_k \rangle\rangle \right) ]$$

$$= \sum_k {\rm Tr} [  \left( \mathcal U |  P_k \rangle\rangle  \right)^\dagger \left(\mathcal E | P_k \rangle\rangle \right) ]$$

Now we switch representations by unveccing  $|  P_k \rangle\rangle$ and representing $\mathcal U$ by its unitary action on the matrix $P_k$.

$$= \sum_k {\rm Tr} [  \left( U P_k U^\dagger \right)^\dagger \cdot \mathcal E \left( P_k \right) ]$$

Finally we can decompose the $P_k$ acted on by $\mathcal E$ into a sum over projectors onto each eigenvector $\left|\phi_j \rangle\langle \phi_j \right|$ with the correct sign for the eigenvalue (which is $\pm 1$ for Paulis).

$$= \sum_k \left( \sum_{\phi_j^{(+)} \in \{\phi\}_k^+} \frac{1}{\sqrt{d}} {\rm Tr} [  \left( U P_k U^\dagger \right) \cdot \mathcal E \left( \left|\phi_j^{(+)}\rangle\langle\phi_j^{(+)}\right| \right) ]
- \sum_{\phi_j^{(-)} \in \{\phi\}_k^-} \frac{1}{\sqrt{d}} {\rm Tr} [  \left( U P_k U^\dagger \right) \cdot \mathcal E \left( \left|\phi_j^{(-)}\rangle\langle\phi_j^{(-)}\right| \right) ]\right)$$

At this point our **assumption** that $U$ is a Clifford element again comes into play and allows us to easily compute the conjugated Pauli $U P_k U^\dagger = \sigma_k / \sqrt{d}$. Inserting this assumption gives a simple picture where we need to estimate the expectation of each $\sigma_k$ Pauli for the state which results from applying our noisy circuit $\mathcal E$ to each eigenstate $\left| \phi_j \rangle \right.$ of $P_k$.

$$= \frac{1}{d} \sum_k \left( \sum_{\phi_j^{(+)} \in \{\phi\}_k^+} {\rm Tr} [ \sigma_k \cdot \mathcal E \left( \left|\phi_j^{(+)}\rangle\langle\phi_j^{(+)}\right| \right) ]
- \sum_{\phi_j^{(-)} \in \{\phi\}_k^-} {\rm Tr} [  \sigma_k \cdot \mathcal E \left( \left|\phi_j^{(-)}\rangle\langle\phi_j^{(-)}\right| \right) ]\right)$$

The final estimate of $F(\mathcal U,\mathcal E)$ follows from estimating these expectations, plugging them in to the sum to get ${\rm Tr} [\mathcal E \mathcal U^\dagger]$ which we insert into the equation at the beginning of the section. Note that moving a factor of $1/d$ from the Fidelity expression into the trace expression allows us to re-interpret the trace as an average of all of the $d^3$ Pauli expectations. Further, again note that we do not need to estimate the all identity Pauli expectations, which will all be 1. This reduces the number of experiment settings to $(d^2-1)d$.

## Process DFE by reduction to state DFE

We can also understand process DFE by appealing to the [Choi–Jamiołkowski isomorphism](https://en.wikipedia.org/wiki/Channel-state_duality) which says that we can represent our channel superoperators as state matrices (i.e. density matrices)  $J(\mathcal E)$ and $J(\mathcal U)$. This allows us to rewrite the average gate fidelity as

$$F(\mathcal U,\mathcal E)= \frac{d^2 {\rm Tr}[ J(\mathcal E)⋅J(\mathcal U)] + d}{d^2+d}$$

Finally, since $J(\mathcal U)$ is a pure state, we employ the same fact from above about fidelity between states to note that ${\rm Tr}[ J(\mathcal E)⋅J(\mathcal U)] = F(J(\mathcal E),J(\mathcal U))$. We have reduced process DFE to state DFE, but we need to dive into the particulars of $J(\cdot)$ to understand what we actually need to measure.

For $U$ acting on $n$ qubits the state $J(\mathcal U)$ over $2n$ qubits is given by

$$J(\mathcal U) = \left(\textrm{Id} \otimes \mathcal U\right) \left(\sum_{k=0}^{2^n-1}\sum_{k'=0}^{2^n-1} |k\rangle \langle k'| \otimes |k \rangle \langle k'|\right) = \sum_{k=0}^{2^n-1}\sum_{k'=0}^{2^n-1} |k\rangle \langle k'| \otimes U |k \rangle   \langle k'| U^\dagger$$

where $|k \rangle$ is the state over $n$ qubits corresponding to the binary representation of $k$. The state matrix for the maximally entangled state on which $\textrm{Id} \otimes \mathcal U$ acts can be decomposed as:

$$\sum_{k=0}^{d-1} \sum_{k'=0}^{d-1} | k k \rangle \langle k' k'| = \frac{1}{d}\sum_j P_j^* \otimes P_j$$

where the sum is over the complete orthonormal Pauli basis $\{P_j\}$ (i.e. $\sigma_j/\sqrt{d}$) for a $d$ dimensional Hilbert space. This gives us

$$d^2 {\rm Tr}[ J(\mathcal E)⋅J(\mathcal U)] = d^2{\rm Tr}[ \left(\frac{1}{d}\sum_k P_k^* \otimes \mathcal E(P_k)\right) \left(\frac{1}{d}\sum_j P_j^* \otimes U P_j U^\dagger\right)]$$

If we partially evaluate the trace over the first subsystem and use ${\rm Tr}[P_k^* P_j^*] = {\rm Tr}[\left(P_k P_j\right)^*] = {\rm Tr}[P_k P_j]^* = {\rm Tr}[P_k^\dagger P_j]^* = \delta_{jk}$ we get

$$= \sum_k {\rm Tr} [  \left( U P_k U^\dagger \right)^\dagger \cdot \mathcal E \left( P_k \right) ]$$

This is the same expression as from the last section, so we can proceed identically.

```python
from pyquil.paulis import ID
from pyquil.gates import I, X, MEASURE, H, CNOT, RY, CZ
from pyquil import Program, get_qc
from pyquil.api import get_benchmarker
from forest.benchmarking.direct_fidelity_estimation import ( generate_exhaustive_state_dfe_experiment,
                                                             generate_exhaustive_process_dfe_experiment,
                                                             generate_monte_carlo_state_dfe_experiment,
                                                             generate_monte_carlo_process_dfe_experiment,
                                                             acquire_dfe_data,
                                                             estimate_dfe )

import numpy as np
from matplotlib import pyplot
```

```python
# noiseless QVM
qvm = get_qc("9q-generic-qvm", as_qvm=True, noisy=False)

# noisy QVM
noisy_qvm = get_qc("9q-generic-qvm", as_qvm=True, noisy=True)

bm = get_benchmarker()
```

## Direct fidelity estimation in `forest.benchmarking`

The basic workflow is:

1. Prepare a *state* or a *process* by specifying a pyQuil program.
2. Construct a list of observables that are needed to estimate the state; we collect this into an object called an `ObservablesExperiment`.
3. Acquire the data by running the program on a QVM or QPU.
4. Apply an estimator to the data to obtain an estimate of the fidelity between the ideal and measured state or process.
5. Visualize if you wish.

## Two quick examples

### Step 1. Specify a state preparation or unitarty process with a `Program`

This is the object we will do DFE on.

The process we choose is

$$U = {\rm CNOT}(H\otimes I)=\frac{1}{\sqrt{2}}
\begin{pmatrix}
1 & 0 & 1 & 0\\
0 & 1 & 0 & 1\\
0 & 1 & 0 & -1\\
1 & 0 & -1 & 0
\end{pmatrix}$$

and the state is

$$|\Psi\rangle = {\rm CNOT}(H\otimes I)|00\rangle = \frac{1}{\sqrt{2}} \begin{pmatrix} 1\\0\\0\\ 1\end{pmatrix}$$

Perhaps we should emphasize that while the `Program` that we pass into the DFE module is the same whether we are doing process DFE on $\tilde U$ or state DFE on $\tilde \Psi$, the specific experiment construction method we choose in the next step selects one or the other and results in different `ExperimentSettings` around the same `Program`.

```python
p = Program()
prep_prog = p.inst(H(0), CNOT(0,1))
print(prep_prog)
```

    H 0
    CNOT 0 1

```python
from pyquil.gate_matrices import I as Imatrix, H as Hmatrix, CNOT as CNOTmatrix

U_ideal = CNOTmatrix @ np.kron(Hmatrix, Imatrix)
print(U_ideal)

rho_ideal = U_ideal @ np.array([[1], [0], [0], [0]]) @ np.array([[1], [0], [0], [0]]).T @ U_ideal.conj().T
print(rho_ideal)
```

    [[ 0.70710678  0.          0.70710678  0.        ]
     [ 0.          0.70710678  0.          0.70710678]
     [ 0.          0.70710678  0.         -0.70710678]
     [ 0.70710678  0.         -0.70710678  0.        ]]
    [[0.5 0.  0.  0.5]
     [0.  0.  0.  0. ]
     [0.  0.  0.  0. ]
     [0.5 0.  0.  0.5]]

### Step 2. Construct an `ObservablesExperiment` for DFE

We use the helper functions

* `generate_exhaustive_state_dfe_experiment`
* `generate_exhaustive_process_dfe_experiment`

to construct a (tomographically incomplete) set of preparations + measurements (i.e. `ExperimentSettings`) for state and process DFE respectively

```python
qubits = [0,1]

# state dfe
state_exp = generate_exhaustive_state_dfe_experiment(bm, prep_prog, qubits)

# process dfe
process_exp = generate_exhaustive_process_dfe_experiment(bm, prep_prog, qubits)
```

For state DFE we can print this experiment object out to see the $d - 1 = 2^2-1= 3$ observables whose expectation we will estimate (i.e. the measurements we will perform). Note that we could have included an additional observable `->I0I1`, but since this trivially gives an expectation of 1 we instead omit this observable in experiment generation and include its contribution by hand in the estimation methods. Be mindful of this if generating your own settings.

```python
# Let's take a look into one of these experiment objects

print('The type of the object is:', type(state_exp),'\n')
print('The program is:')
print(state_exp.program)
print('There are three different settings:')
print(state_exp.settings_string())
```

    The type of the object is: <class 'forest.benchmarking.observable_estimation.ObservablesExperiment'>

    The program is:
    H 0
    CNOT 0 1

    There are three different settings:
    0: Z+_0 * Z+_1→(1+0j)*Z0Z1
    1: Z+_0 * Z+_1→(1+0j)*X0X1
    2: Z+_0 * Z+_1→(-1+0j)*Y0Y1

Let's also look at a few of the settings for process DFE. Here our omission of trivial Identity terms means that we are summing over only the $d^2 - 1 = 15$ total unique observables measured. For each of these observables we prepare the $d = 4$ different initial preparations of eigenstates of the pre-circuit-conjugated observable. (For each Identity term of the pre-conjugated observable we still prepare two separate eigenstates, which we arbitrarily choose to be the two computational basis states on that qubit.) This gives us $60$ total settings.

Note that we have incorporated the minus sign into the observables paired with preparations of the negative eigenstates. This means that for ideal no-noise data collection we will get expectations all equal to positive 1. (we have chosen our observables so that the ideal state is always in the positive eigenvalue subspace).

```python
print('The program is, again:')
print(process_exp.program)
print('There are more settings:')
print(process_exp.settings_string())
```

    The program is, again:
    H 0
    CNOT 0 1

    There are more settings:
    0: Z+_0 * X+_1→(1+0j)*X1
    1: Z+_0 * X-_1→(-1+0j)*X1
    2: Z-_0 * X+_1→(1+0j)*X1
    3: Z-_0 * X-_1→(-1+0j)*X1
    4: Z+_0 * Y+_1→(1+0j)*Z0Y1
    5: Z+_0 * Y-_1→(-1+0j)*Z0Y1
    6: Z-_0 * Y+_1→(1+0j)*Z0Y1
    7: Z-_0 * Y-_1→(-1+0j)*Z0Y1
    8: Z+_0 * Z+_1→(1+0j)*Z0Z1
    9: Z+_0 * Z-_1→(-1+0j)*Z0Z1
    10: Z-_0 * Z+_1→(1+0j)*Z0Z1
    11: Z-_0 * Z-_1→(-1+0j)*Z0Z1
    12: X+_0 * Z+_1→(1+0j)*Z0
    13: X+_0 * Z-_1→(1+0j)*Z0
    14: X-_0 * Z+_1→(-1+0j)*Z0
    15: X-_0 * Z-_1→(-1+0j)*Z0
    16: X+_0 * X+_1→(1+0j)*Z0X1
    17: X+_0 * X-_1→(-1+0j)*Z0X1
    18: X-_0 * X+_1→(-1+0j)*Z0X1
    19: X-_0 * X-_1→(1+0j)*Z0X1
    20: X+_0 * Y+_1→(1+0j)*Y1
    21: X+_0 * Y-_1→(-1+0j)*Y1
    22: X-_0 * Y+_1→(-1+0j)*Y1
    23: X-_0 * Y-_1→(1+0j)*Y1
    24: X+_0 * Z+_1→(1+0j)*Z1
    25: X+_0 * Z-_1→(-1+0j)*Z1
    26: X-_0 * Z+_1→(-1+0j)*Z1
    27: X-_0 * Z-_1→(1+0j)*Z1
    28: Y+_0 * Z+_1→(-1+0j)*Y0X1
    29: Y+_0 * Z-_1→(-1+0j)*Y0X1
    30: Y-_0 * Z+_1→(1-0j)*Y0X1
    31: Y-_0 * Z-_1→(1-0j)*Y0X1
    32: Y+_0 * X+_1→(-1+0j)*Y0
    33: Y+_0 * X-_1→(1-0j)*Y0
    34: Y-_0 * X+_1→(1-0j)*Y0
    35: Y-_0 * X-_1→(-1+0j)*Y0
    36: Y+_0 * Y+_1→(1+0j)*X0Z1
    37: Y+_0 * Y-_1→(-1+0j)*X0Z1
    38: Y-_0 * Y+_1→(-1+0j)*X0Z1
    39: Y-_0 * Y-_1→(1+0j)*X0Z1
    40: Y+_0 * Z+_1→(-1+0j)*X0Y1
    41: Y+_0 * Z-_1→(1-0j)*X0Y1
    42: Y-_0 * Z+_1→(1-0j)*X0Y1
    43: Y-_0 * Z-_1→(-1+0j)*X0Y1
    44: Z+_0 * Z+_1→(1+0j)*X0X1
    45: Z+_0 * Z-_1→(1+0j)*X0X1
    46: Z-_0 * Z+_1→(-1+0j)*X0X1
    47: Z-_0 * Z-_1→(-1+0j)*X0X1
    48: Z+_0 * X+_1→(1+0j)*X0
    49: Z+_0 * X-_1→(-1+0j)*X0
    50: Z-_0 * X+_1→(-1+0j)*X0
    51: Z-_0 * X-_1→(1+0j)*X0
    52: Z+_0 * Y+_1→(1+0j)*Y0Z1
    53: Z+_0 * Y-_1→(-1+0j)*Y0Z1
    54: Z-_0 * Y+_1→(-1+0j)*Y0Z1
    55: Z-_0 * Y-_1→(1+0j)*Y0Z1
    56: Z+_0 * Z+_1→(-1+0j)*Y0Y1
    57: Z+_0 * Z-_1→(1-0j)*Y0Y1
    58: Z-_0 * Z+_1→(1-0j)*Y0Y1
    59: Z-_0 * Z-_1→(-1+0j)*Y0Y1

### Step 3. Acquire the data

We will use the QVM, but at this point you could use a QPU.

We use a simple wrapper, `acquire_dfe_data`, around the standard `estimate_observables` method to run each `ExperimentSetting` in our `ObservablesExperiment`. By default `acquire_dfe_data` includes an additional error mitigation step that attempts to calibrate estimation of each observable expectation in order to combat readout error.

Note that `acquire_dfe_data` returns a `list` of `ExperimentResult`s which is a dataclass defined in the module `observable_estimation.py`.

The details of the dataclass and error mitigation strategies are given in the [observable estimation ipython notebook](observable_estimation.ipynb).

```python
# get some NOISELESS data
results = acquire_dfe_data(qvm, process_exp, num_shots=1000)
```

```python
# look at the results -- we expect all expectations to be one since there is no noise.
print("Operator Expectations")
print([res.expectation for res in results])
print('\n')
print("Calibration Expectations")
print([res.calibration_expectation for res in results])
```

    Operator Expectations
    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

    Calibration Expectations
    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

```python
# get some NOISY data
n_results_proce = acquire_dfe_data(noisy_qvm, process_exp, num_shots=1000)

n_results_state = acquire_dfe_data(noisy_qvm, state_exp, num_shots=1000)
```

```python
# look at it
print("Noisy Operator Expectations")
print([np.round(res.expectation, 4) for res in n_results_proce])
print('\n')
# given some readout noise we scale up our estimates by the inverse calibration expectation
print("Noisy Calibration Expectations")
print([res.calibration_expectation for res in n_results_proce])
```

    Noisy Operator Expectations
    [1.0023, 0.9792, 1.0324, 0.9861, 0.9893, 1.0295, 1.0295, 1.0134, 0.932, 0.9924, 0.9698, 0.9975, 0.9817, 1.0252, 0.9908, 0.9908, 0.9676, 0.9925, 0.9551, 1.0, 0.9954, 0.9977, 0.9586, 0.9977, 0.9729, 0.9412, 0.9593, 0.9751, 0.9123, 0.9699, 0.9323, 0.9574, 0.9726, 0.9954, 0.9772, 1.0, 1.0026, 0.9896, 1.026, 0.9557, 0.9631, 0.9409, 0.9113, 0.9138, 0.9843, 1.0313, 1.0522, 0.9765, 0.9636, 0.9818, 0.9636, 1.0023, 0.9751, 0.9428, 0.9851, 0.9279, 1.0187, 0.9627, 0.9973, 0.984]

    Noisy Calibration Expectations
    [0.864, 0.864, 0.864, 0.864, 0.746, 0.746, 0.746, 0.746, 0.794, 0.794, 0.794, 0.794, 0.874, 0.874, 0.874, 0.874, 0.802, 0.802, 0.802, 0.802, 0.87, 0.87, 0.87, 0.87, 0.884, 0.884, 0.884, 0.884, 0.798, 0.798, 0.798, 0.798, 0.876, 0.876, 0.876, 0.876, 0.768, 0.768, 0.768, 0.768, 0.812, 0.812, 0.812, 0.812, 0.766, 0.766, 0.766, 0.766, 0.88, 0.88, 0.88, 0.88, 0.804, 0.804, 0.804, 0.804, 0.75, 0.75, 0.75, 0.75]

### Step 4. Apply some estimators to the data "do DFE"
This just performs the calculations we did at the top of the notebook.

**Process DFE**

```python
# estimate using NOISELESS data
fid_est, fid_std_err = estimate_dfe(results, 'process')

print('Fidelity point estimate is: ',fid_est)
print('The standard error of the fidelity point estimate is: ', fid_std_err)
```

    Fidelity point estimate is:  1.0
    The standard error of the fidelity point estimate is:  0.0

```python
# estimate using NOISY data
nfid_est, nfid_std_err = estimate_dfe(n_results_proce, 'process')

print('Fidelity point estimate is', np.round(nfid_est, 4))
print('The std error of the fidelity point estimate is', np.round(nfid_std_err, 4))
```

    Fidelity point estimate is 0.9853
    The std error of the fidelity point estimate is 0.0031

**State DFE**

```python
# estimate using NOISY data
nfid_est_state, nfid_std_err_state = estimate_dfe(n_results_state, 'state')

print('Fidelity point estimate is', np.round(nfid_est_state, 4))
print('The std error of the fidelity point estimate is', np.round(nfid_std_err_state, 4))
```

    Fidelity point estimate is 1.0141
    The std error of the fidelity point estimate is 0.0165

### Step 5. Visualize

**State DFE**

We will start with state DFE as it is the simplest case.

Our strategy here will be to try to reconstruct the state using a state tomography estimator. This is possible
since the settings used in DFE are a strict subset of the tomography settings. In fact, this makes the strategy of DFE clear-- we only need to estimate those components of the ideal state that are non-zero in order to compute the fidelity.

```python
from forest.benchmarking.tomography import iterative_mle_state_estimate
```

```python
rho_est = iterative_mle_state_estimate(n_results_state, qubits=[0,1])

np.round(rho_est, 3)
```

    array([[0.497+0.j, 0.   +0.j, 0.   +0.j, 0.497+0.j],
           [0.   +0.j, 0.003+0.j, 0.002+0.j, 0.   +0.j],
           [0.   +0.j, 0.002+0.j, 0.003+0.j, 0.   +0.j],
           [0.497+0.j, 0.   +0.j, 0.   +0.j, 0.497+0.j]])

```python
import matplotlib.pyplot as plt
from forest.benchmarking.utils import n_qubit_pauli_basis
from forest.benchmarking.operator_tools.superoperator_transformations import vec, computational2pauli_basis_matrix
from forest.benchmarking.plotting.state_process import plot_pauli_rep_of_state, plot_pauli_bar_rep_of_state

# convert to pauli representation
n_qubits = 2
pl_basis = n_qubit_pauli_basis(n_qubits)
c2p = computational2pauli_basis_matrix(2*n_qubits)

rho_true_pauli = np.real(c2p @ vec(rho_ideal))
rho_mle_pauli = np.real(c2p @ vec(rho_est))

fig1, (ax3, ax4) = plt.subplots(1, 2, figsize=(10,5))
title_res = f"Estimated via DFE data using MLE \n" f"DFE Fidelity = {np.round(nfid_est_state, 5)} ± {np.round(nfid_std_err_state, 6)}"
plot_pauli_bar_rep_of_state(rho_true_pauli.flatten(), ax=ax3, labels=pl_basis.labels, title='Ideal State')
plot_pauli_bar_rep_of_state(rho_mle_pauli.flatten(), ax=ax4, labels=pl_basis.labels, title=title_res)
fig1.tight_layout()
```

To be clear, each entry on the X axis corresponds to an observable. If we ran full state tomography on the noisy state then the plot on the right would likely have small values for each entry due to noise. However, we note that to compute the fidelity we only need to estimate those entries for which the ideal state is non-zero. The plot on the right shows that we estimated exactly these four entries.

**Process DFE**

The same principle applies to process DFE, but now we must remember that for processes we vary over input states and measurements both.

```python
from forest.benchmarking.tomography import pgdb_process_estimate

choi_mle_est = pgdb_process_estimate(n_results_proce, qubits)

# sneak peak at part of the estimated process
np.real_if_close(np.round(choi_mle_est, 2))[0:4]
```

    array([[ 0.5, -0. , -0. ,  0.5, -0. ,  0.5,  0.5, -0. ,  0.5, -0. ,  0. ,
            -0.5, -0. ,  0.5, -0.5,  0. ],
           [-0. , -0. ,  0. , -0. , -0. ,  0. , -0. ,  0. , -0. , -0. , -0. ,
             0. , -0. , -0. ,  0. ,  0. ],
           [-0. ,  0. , -0. , -0. ,  0. , -0. , -0. ,  0. , -0. ,  0. , -0. ,
             0. ,  0. , -0. ,  0. , -0. ],
           [ 0.5, -0. , -0. ,  0.5, -0. ,  0.5,  0.5,  0. ,  0.5, -0. ,  0. ,
            -0.5, -0. ,  0.5, -0.5,  0. ]])

```python
from forest.benchmarking.plotting.state_process import plot_pauli_transfer_matrix
from forest.benchmarking.operator_tools import choi2pauli_liouville, kraus2pauli_liouville

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,5))
title_res = f"Estimated via DFE data using MLE \n" f"DFE Fidelity = {np.round(nfid_est, 5)} ± {np.round(nfid_std_err, 6)}"
plot_pauli_transfer_matrix(np.real(kraus2pauli_liouville(U_ideal)), ax1, title='Ideal')
plot_pauli_transfer_matrix(np.real(choi2pauli_liouville(choi_mle_est)), ax2, title=title_res)
plt.tight_layout()
```

## State fidelity between $\left|1\right\rangle$ and $|\theta\rangle = R_y(\theta)\left|1\right\rangle$

In this section we check that state DFE is working correctly by comparing with an analytical calculation.
Essentially we would like to prepare $|1\rangle$ but we simulate an incorrect preparation $|\theta\rangle = R_y(\theta)|1\rangle$. The fidelity in that case is

$$\begin{align}
F\big(|1\rangle, |\theta\rangle \big) &= |\langle 1|R_y(\theta) |1\rangle |^2\\
&= \cos^2(\theta/2)\\
&= \frac{1}{2} \big (1 +\cos(\theta) \big).
\end{align}$$

So the point of this section is to try to "experimentally" plot the fidelity expression as a function of $\theta$.

```python
qubit = 0
ideal_prep_program = Program(X(qubit))

# generate state dfe experiment to estimate fidelity to |1 >
experiment = generate_exhaustive_state_dfe_experiment(bm, ideal_prep_program, [qubit])
```

To simulate the error we will modify the ideal program in the experiment to be `RY(theta, 0) X(0)` for various angles $\theta$; this is the case of unitary rotation error occurring after our ideal preparation `X(0)`.

Of course, for characterizing the actual noise on a real QPU our program would simply remain `X(0)` since this is the state preparation we hope will prepare the state $|1 \rangle$ but which in practice will be affected by noise. Here we must append our own noise `RY(theta)` to the program for the purposes of simulation.

```python
points = 10
res = []
res_std_err = []

# loop over different angles
for theta in np.linspace(0, np.pi, points):
    # reuse the same experiment object but modify its program
    # field to do the "noisy" program
    experiment.program = ideal_prep_program + RY(theta, qubit)
    ry_state_data = acquire_dfe_data(qvm, experiment, num_shots=1000)
    fid_est, fid_std_err = estimate_dfe(ry_state_data, 'state')
    res.append(fid_est)
    res_std_err.append(2*fid_std_err)
```

```python
pyplot.errorbar(np.linspace(0, np.pi, points), res, res_std_err, fmt=".", label="Simulation")
pyplot.plot(np.linspace(0, np.pi, points), (1/2+1/2*np.cos(np.linspace(0, np.pi, points))), label="Theory")
pyplot.xlabel(r"$\theta$")
pyplot.ylabel("Fidelity")
pyplot.legend()
pyplot.title(r"State fidelity between $\left|1\right\rangle$ and $R_y(\theta)\left|1\right\rangle$")
pyplot.show()
```

as expected, the fidelity decays from the ideal with increasing values of theta.

## Process fidelity between $I$ and $R_y(\theta)$

Like the state fidelity example above we will construct a DFE experiment to estimate the fidelity between the ideal identity operation and a 'noisy identity' where we insert the unitary error $R_y(\theta)$. Then we will show that process DFE is consistent with the analytical calculation.

The process fidelity is easy to calculate using Eqn 5 from [arXiv:quant-ph/0701138](https://arxiv.org/pdf/quant-ph/0701138.pdf). The fidelity between the unitary $U_0$ and the quantum operation specified by the Kraus operators $\{ G_k \}$ is

$$F(\{ G_k \}, U_0)
= \frac{1}{d(d+1)} \big ( {\rm Tr}\big [\sum_k M_k^\dagger M_k \big ] + \sum_k |{\rm Tr}[ M_k]|^2 \big )$$

where $M_k = U_0^\dagger G_k$.

In the context we care about we have a single Kraus operator $\{ G_k \} = \{R_y(\theta)\}$ and $U_0=I \implies M_k = R_y(\theta)$. So the fidelity becomes

$$F\big(R_y(\theta), I \big)
= \frac{1}{3} \big (1 +2\cos^2(\theta/2) \big )$$

where we used $R_y(\theta) = \exp[-i \theta Y /2]= \cos(\theta/2) I - i \sin(\theta/2)Y$.

```python
# here our ideal program is the Identity process
qubit = 0
ideal_process_program = Program(I(qubit))

# generate process DFE experiment to estimate fidelity to I
expt = generate_exhaustive_process_dfe_experiment(bm, ideal_process_program, [qubit])
```

Now, as with state dfe, we will simulate noisy implementation of the identity by modifying the program that is actually run in the experiment to be `RY(theta,0)` instead of just the identity.

As above we emphasize that modifying the program in this way only makes sense if you are inserting your own simulation of noise. To characterize real noise on a QPU the program should not be changed after generating the experiment, since the generation code tailors the experiment to the provided program.

**Note:** in some of the cells below there is a comment `# NBVAL_SKIP` that is used to speed up our tests by skipping that particular cell.

```python
# NBVAL_SKIP
num_points = 10
thetas = np.linspace(0, np.pi, num_points)
res = []
res_std_err = []
for theta in thetas:
    # modify the experiment object to do the noisy program instead
    expt.program = ideal_process_program + RY(theta, qubit)
    ry_proc_data = acquire_dfe_data(qvm, expt, num_shots=500)
    fid_est, fid_std_err = estimate_dfe(ry_proc_data, 'process')
    res.append(fid_est)
    res_std_err.append(2*fid_std_err)
```

```python
# NBVAL_SKIP
pyplot.errorbar(thetas, res, res_std_err, fmt=".", label="Simulation")
pyplot.plot(thetas, (1 + 2*np.cos(thetas/2)**2)/3,
            label="theory")
pyplot.xlabel(r"$\theta$")
pyplot.ylabel("Fidelity")
pyplot.legend()
pyplot.ylim(0.25, 1.05)
pyplot.title(r"Process fidelity between $I$ and $R_y(\theta)$")
pyplot.show()
```

## Advanced

### Monte Carlo Sampling of large graph states

We can do [Monte Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_method) or random sampling of `ExperimentSettings` for large states or processes.

* `generate_monte_carlo_state_dfe_experiment`
* `generate_monte_carlo_process_dfe_experiment`

generally you need to specify the number of terms you would like to sample for the desired time savings

```python
import networkx as nx
from matplotlib import pyplot as plt
from forest.benchmarking.entangled_states import create_graph_state
```

We will demonstrate state DFE on a graph state over 5 qubits. First, we will take some subgraph of the larger QC topology.

```python
nx.draw(noisy_qvm.qubit_topology(), with_labels=True)

# we will do a subgraph
graph = nx.from_edgelist([(0, 1), (0, 3), (1, 2), (1, 4), (3, 4)])
```

We use a helper to create the prep program

```python
graph_prep_prog = create_graph_state(graph)

qubits = list(graph_prep_prog.get_qubits())
print(graph_prep_prog)
```

    H 0
    H 1
    H 3
    H 2
    H 4
    CZ 0 1
    CZ 0 3
    CZ 1 2
    CZ 1 4
    CZ 3 4

Generate both exhaustive and monte_carlo experiments for comparison

```python
gstate_exp_exh = generate_exhaustive_state_dfe_experiment(bm, graph_prep_prog, qubits)
gstate_exp_mc = generate_monte_carlo_state_dfe_experiment(bm, graph_prep_prog, qubits, n_terms=8)
```

```python
num_exh_exp = len(list(gstate_exp_exh.setting_strings()))
num_mc_exp = len(list(gstate_exp_mc.setting_strings()))

print(f'In exhaustive state DFE there are {num_exh_exp} experiments.\n' )

print(f'In monte carlo state DFE we chose {num_mc_exp} experiments.' )
```

    In exhaustive state DFE there are 31 experiments.

    In monte carlo state DFE we chose 8 experiments.

Run each experiment and compare estimates and runtimes.

```python
# NBVAL_SKIP

# because of the #NBVAL_SKIP `%%time` wont work
from time import time
start_time = time()

graph_state_mc_data = acquire_dfe_data(noisy_qvm, gstate_exp_mc, num_shots=500)

fid_est, fid_std_err = estimate_dfe(graph_state_mc_data, 'state')
print(f'The five qubit graph fidelity estimate is {fid_est}.\n')
print('Monte-Carlo took ', np.round(time()-start_time, 2), 'seconds.')
```

    The five qubit graph fidelity estimate is 0.9234208202870585.

    Monte-Carlo took  41.18 seconds.

```python
# NBVAL_SKIP

start_time = time()

graph_state_exh_data = acquire_dfe_data(noisy_qvm, gstate_exp_exh, num_shots=500)

fid_est, fid_std_err = estimate_dfe(graph_state_exh_data, 'state')
print(f'The five qubit graph fidelity estimate is {fid_est}.\n')
print('Exhaustive took ', np.round(time()-start_time, 2), 'seconds.')
```

    The five qubit graph fidelity estimate is 0.9395043896868677.

    Exhaustive took  204.76 seconds.

### DFE measurements that include spectator qubits to signal cross talk.

Suppose you wanted to do process tomography on a CZ gate.

It turns out in many superconducting qubits there are many kinds of cross-talk at play, that is, noise which affects qubits not directly involved in the gate.

One way to measure such noise is to do the gate you care about and additionally specify spectator qubits not involved in the gate when creating the experiment.
If the actual process is not Identity on those qubits then DFE will pick it up.

Of course, such experiments are best done on the QPU or at the least with a noise model that includes these effects.

```python
prog = Program(CZ(0,1))
qubits = [0,1,2]
```

```python
print('The CZ gate is: CZ(0,1)')
nx.draw(nx.from_edgelist([(0, 1), (1, 2)]), with_labels=True)
```

```python
xtalk_proc_exp = generate_monte_carlo_process_dfe_experiment(bm, prog, qubits, n_terms=20)
```

```python
len(list(xtalk_proc_exp.setting_strings()))
```

    20

```python
xtalk_data = acquire_dfe_data(qvm, xtalk_proc_exp, num_shots=500)
```

```python
fid_est, fid_std_err = estimate_dfe(xtalk_data, 'process')
fid_est
```

    1.0

Again, we expect this to be 1.0 unless we are running on a real QPU or incorporate cross-talk into the QVM noise model.

If we wanted to we could "amplify" the cross talk error by applying the gate many times

```python
prog = Program([CZ(0,1)]*3)
```

### Parallel state and process DFE

The `ObservablesExperiment` [framework](observable_estimation.ipynb) allows for easy parallelization of experiments that operate on disjoint sets of qubits. Below we will demonstrate the simple example of performing process DFE on two separate bit flip processes `Program(X(0))` and `Program(X(1))`. To run each experiment in serial would require $n_1 + n_2 = 2n$ experimental runs ($n_1 = n_2 = n$ in this case), but when we run a 'parallel' experiment we need only $n$ runs.

Note that the parallel experiment is not the same as doing DFE on the program `Program(X(0), X(1))` because in the later case we need to do more data acquisition runs on the qc and we get more information back; even if each process perfectly transforms the 1q states it could still behave erroneously on some 2q states but we would only catch that if we did 2q DFE. The `ExperimentSetting`s for the 2q experiment are a superset of the parallel 1q settings. We also cannot directly compare a parallel experiment with two serial experiments, because in a parallel experiment 'cross-talk' and other multi-qubit effects can impact the overall process; that is, the physics of 'parallel' experiments cannot in general be neatly factored into two serial experiments.

See the linked [notebook](observable_estimation.ipynb) for more explanation and words of caution.

```python
from forest.benchmarking.observable_estimation import ObservablesExperiment, merge_disjoint_experiments

disjoint_sets_of_qubits = [(0,),(1,)]
programs = [Program(X(*q)) for q in disjoint_sets_of_qubits]

expts_to_parallelize = []
for qubits, program in zip(disjoint_sets_of_qubits, programs):
    expt = generate_exhaustive_process_dfe_experiment(bm, program, qubits)
    expts_to_parallelize.append(expt)

# get a merged experiment with grouped settings for parallel data acquisition
parallel_expt = merge_disjoint_experiments(expts_to_parallelize)

print(f'Original number of runs: {sum(len(expt) for expt in expts_to_parallelize)}')
print(f'Parallelized number of runs: {len(parallel_expt)}\n')
print(parallel_expt)
```

    Original number of runs: 12
    Parallelized number of runs: 6

    X 0; X 1
    0: X+_0→(1+0j)*X0, X+_1→(1+0j)*X1
    1: X-_0→(-1+0j)*X0, X-_1→(-1+0j)*X1
    2: Y+_0→(-1+0j)*Y0, Y+_1→(-1+0j)*Y1
    3: Y-_0→(1-0j)*Y0, Y-_1→(1-0j)*Y1
    4: Z+_0→(-1+0j)*Z0, Z+_1→(-1+0j)*Z1
    5: Z-_0→(1-0j)*Z0, Z-_1→(1-0j)*Z1

Collect the data. Separate the results by qubit to get back estimates for each process.

```python
from forest.benchmarking.observable_estimation import get_results_by_qubit_groups

parallel_results = acquire_dfe_data(noisy_qvm, parallel_expt, num_shots = 500)

individual_results = get_results_by_qubit_groups(parallel_results, disjoint_sets_of_qubits)

fidelity_estimates = []
process_estimates = []
for q in disjoint_sets_of_qubits:
    fidelity_estimate = estimate_dfe(individual_results[q], 'process')
    fidelity_estimates.append(fidelity_estimate)
    print(fidelity_estimate)

    proc_estimate = pgdb_process_estimate(individual_results[q], q)
    process_estimates.append(proc_estimate)

fig, axes = plt.subplots(1, len(process_estimates), figsize=(12,5))
for idx, est in enumerate(process_estimates):
    plot_pauli_transfer_matrix(choi2pauli_liouville(est), axes[idx], title=f'Estimate {idx}')

plt.tight_layout()

```

```python

```


# Introduction

This Notebook gives a quick overview of the capabilities of the visiualisation module.

- Easy drawing of plain geometric primitives (rectangles, circles)
- Easy access to interactions (useful for kinematic visualisation)
- Easy access to animation generation (time-dependent functions and configuration space)

```python
%load_ext ipydex.displaytools
import sympy as sp
from sympy import sin, cos
import numpy as np
import scipy.integrate
import symbtools as st
import symbtools.modeltools as mt
import matplotlib.pyplot as plt
import symbtools.visualisation as vt
sp.init_printing()

import importlib
```

```python
theta1, theta2 = theta = st.symb_vector('theta1:3')
dtheta1, dtheta2 = dtheta = st.time_deriv(theta, theta)
tau = sp.symbols('tau')
params = st.symb_vector('m1 l1 J1 mu1 m2 r2 J2 mu2')
st.make_global(params)
```

```python
p0 = sp.Matrix([0, 0])
p1 = sp.Matrix([l1/2*cos(theta1), l1/2*sin(theta1)])
p2 = sp.Matrix([l1*cos(theta1), l1*sin(theta1)])
p3 = p2 + sp.Matrix([r2*cos(theta1+theta2), r2*sin(theta1+theta2)])
dp1 = st.time_deriv(p1, theta)
dp2 = st.time_deriv(p2, theta)
```

```python
T_rot = J1 * dtheta1**2 / 2 + J2 * (dtheta1 + dtheta2)**2 / 2
T_trans = (m1 * dp1.T * dp1 / 2 + m2 * dp2.T * dp2 / 2)[0]
T = T_rot + T_trans
V = 0
friction1 = dtheta1 * mu1
friction2 = dtheta2 * mu2

model = mt.generate_symbolic_model(T, V, theta, [0 - friction1, tau - friction2])
model.tau = [tau]
model.calc_state_eq()

f = model.f
g = model.g
```

```python
m1_val = 0.1
m2_val = 1
l1_val = 0.5
r2_val = 0.2
J1_val = 1/12 * m1_val * l1_val**2
J2_val = 1/2 * m2_val * r2_val**2
mu1_val = 0.01
mu2_val = 0.001

param_subs = st.lzip(params, [m1_val, l1_val, J1_val, mu1_val, m2_val, r2_val, J2_val, mu2_val])

f_fun = st.expr_to_func(model.x, f.subs(param_subs))
g_fun = st.expr_to_func(model.x, g.subs(param_subs))

def tau_fun(t, x):
    return np.sin(2*np.pi*0.25*t)

def ode_fun(t, x):
    return f_fun(x[0], x[1], x[2], x[3]) + tau_fun(t, x) * g_fun(x[0], x[1], x[2], x[3])

x_init = np.array([0, 0, 0, 2])
tf = 1.0
dt = 1/30
ts = np.arange(0.0, tf, dt)
samples = len(ts)
ys = np.empty((samples, len(model.x)))
ys[0, :] = x_init

# get state trajectory
for i in range(samples - 1):
    result = scipy.integrate.solve_ivp(ode_fun, (ts[i], ts[i+1]), ys[i, :])
    ys[i+1, :] = result.y[:, -1]

# get input trajectory after the fact
us = np.empty((samples, 1))
for i in range(samples):
    us[i, 0] = tau_fun(ts[i], ys[i, :])
```

```python
importlib.reload(vt)
```

    <module 'symbtools.visualisation' from '/media/workcard/workstickdir/projekte/rst_python/symbtools-TUD-RST-Account/symbtools/visualisation.py'>

```python
vis = vt.Visualiser(theta, xlim=(-1, 1), ylim=(-1, 1))
vis.add_linkage([p0.subs(param_subs), p2.subs(param_subs)])
vis.add_disk([p2.subs(param_subs), p3.subs(param_subs)], color="red")
```

```python
vis.plot([0.0, 0.0])
```

```python
vis.interact(theta2=(-3.14, 3.14, 0.01))
```

    interactive(children=(FloatSlider(value=0.0, description='theta1', max=5.0, min=-5.0), FloatSlider(value=0.0, …

```python
simanim = vt.SimAnimation(model.x, ts, ys)
simanim.add_visualiser(vis)
simanim.display_frame()
```

```python
simanim = vt.SimAnimation(model.x, ts, ys, figsize=(8, 4))
simanim.add_visualiser(vis, 122)
simanim.add_graph(theta, 121)
simanim.display_frame()
```

```python
from matplotlib.gridspec import GridSpec

gs = GridSpec(4, 3, hspace=0.3)

simanim = vt.SimAnimation(model.x, ts, ys, start_pause=0.5, end_pause=0.5, figsize=(12, 10))
simanim.add_visualiser(vis, gs[:, 1:])
simanim.add_graph(theta, gs[0, 0], ax_kwargs=dict(title='Angles')) # most common options can probably be exposed easier
simanim.add_graph([dtheta1, dtheta2], gs[1, 0], ax_kwargs=dict(title='Velocities'))
simanim.add_graph(p3.subs(param_subs), gs[2, 0], ax_kwargs=dict(title='x-y Position'), plot_kwargs=dict(ls='--'))
simanim.add_graph(us[:, 0], gs[3, 0], ax_kwargs=dict(title='Input torque'))
simanim.display_frame()  # render just one frame to check the layout
```

```python
# simanim.display()  # show the whole animation (increases filesize of jupyter notebook)
```

```python
# this probably needs to be adapted on other systems
# (anaconda version of ffmpeg might lead to an error)

plt.rcParams['animation.ffmpeg_path'] = '/usr/bin/ffmpeg'

fname = 'demo_animation.mp4'
simanim.save(fname, dpi=50)  # save to file
```

```python
vt.display_video_file(fname)
```

```python

```


# 6 of 15 central calculations

The first year course, Descriptive Economics A, presents 15 calculations for doing basic descriptive statistics.
This project shows how to conveniently apply these calculations on selected data from Statistics Denmark to easily see the many calculation done on a short time series. We have selected six of these 15 calculations to describe the change in expenditure and income in the public sector (OFF3).

\begin{align}
    \text{Absolute change: }& x_t - x_{t-1} \\
    \text{Average absolute change: }& \frac{x_n - x_0}{n} \\
    \text{Percentage change: }& \left(\frac{x_t}{x_{t-1}}-1\right)\times 100 \\
    \text{Average percentage change: }& \left[\left(\frac{x_n}{x_0}\right)^{\frac{1}{n}}-1\right]\times 100 \\
    \text{Change in percentage points: }& \text{pct.-points}_t - \text{pct. points}_{t-1} \\
    \text{Simple index: }& \frac{x_t}{x_0}\times 100 \\
\end{align}

## Importing modules

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pydst
import pprint as pp
import ipywidgets as widgets
from ipywidgets import interact
dst = pydst.Dst(lang='en')
```

## Collecting data

```python
variables = dst.get_variables(table_id = 'OFF3') # collecting table OFF3 from Statistics Denmark
OFF3 = dst.get_data(table_id = 'OFF3', variables={'UI':['1.8','1.13','1.16','1.17','2.13','2.16','2.17','2.18',
                                                        '2.19'], # sub-categories
                                                  'Tid':['*'], # all time
                                                  'SEKTOR':['TOTAL']}) # total public sector
OFF3.sort_values(by = 'TID', inplace = True)
```

## Cleaning data

```python
del OFF3['SEKTOR'] # deleting irrelevant variable
names = OFF3['UI'][0:9] # generating names variable based on different sub-categories

# Renaming variables
columns_dict = {}
columns_dict['UI'] = 'Variable'
columns_dict['TID'] = 'Time'
columns_dict['INDHOLD'] = 'Absolute level'
OFF3.rename(columns = columns_dict,inplace=True)
```

```python
# Initialize empty dictionary
rename_dict = {}
# List of wanted names
wanted_names = ['1.2: Capital accumulation',
                '1.3: Capital expenses',
                '1.4: Current and capital expenditure (1+3)',
                '1.1: Current expenditure',
                '2.1: Current revenue',
                '2.2: Capital revenue',
                '2.3: Current plus capital revenue (1+2)',
                '2.4: Currents surplus=Gross saving (2.1-1.1)',
                '2.5: Overall surplus=Net lending/borrowing (2.3-1.4)']
# Create rename dictionary for variable names
for name, wantedname in zip(names,wanted_names):
    rename_dict[name] = wantedname
# Rename the variables
for key,value in rename_dict.items():
    OFF3.Variable.replace(key, value, inplace=True)
```

### Create subsetting booleans

```python
# Initializes list of lists for subsetting
Ilist = [[] for eachlist in range(9)]
```

```python
# List i in Ilist is the true/false boolean for name i in wanted_names
for number, name in enumerate(wanted_names):
    Ilist[number] = OFF3['Variable']==name
```

```python
names = ['capital_accumulation', 'capital_expenses', 'current_and_capital_expenditure',
         'current_expenditure', 'current_revenue', 'capital_revenue',
         'current_plus_capital_revenue', 'current_surplus', 'overall_surplus']
for i, name in enumerate(names):
    namespace = locals()
    namespace[name] = (OFF3[Ilist[i]])
```

```python
dataframes = [capital_accumulation, capital_expenses, current_and_capital_expenditure,
              current_expenditure, current_revenue, capital_revenue,
              current_plus_capital_revenue, current_surplus, overall_surplus]

# Resetting index
for i in dataframes:
    i.reset_index(drop = True, inplace = True)
```

## Calculations

We can now make each calculation, (1)-(6), on every data frame as a new variable.

```python
for i in dataframes:

    # Absolute change
    i['Absolute change'] = i['Absolute level'].diff()

    # Average absolute change
    i['Average absolute change'] = i['Absolute level'].mean()

    # Percentage change
    i['Percentage change'] = i['Absolute level'].pct_change()*100

    # Average percentage change
    i['Average percentage change'] = ((i['Absolute level'].iloc[-1]/i['Absolute level'][0])**(1/len(i['Absolute level']))-1)*100

    # Change in percentage points
    i['Change in percentage points'] = i['Percentage change'].diff()

    # Simple index
    i['Simple index'] = np.nan # Create 'Simple index'-variable with NaNs

    for j in range(0, len(i['Absolute level'])):
        i['Simple index'][j] = i['Absolute level'].iloc[j]/i['Absolute level'].iloc[0]*100
```

    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      after removing the cwd from sys.path.
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      import sys
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:10: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      # Remove the CWD from sys.path while we load stuff.
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:13: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      del sys.path[0]
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:16: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      app.launch_new_instance()
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:19: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
    /anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:22: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
    /anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3267: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      exec(code_obj, self.user_global_ns, self.user_ns)

## Plotting the data

```python
rows = ['Absolute level', 'Absolute change', 'Average absolute change', 'Percentage change',
        'Average percentage change', 'Change in percentage points', 'Simple index']

data_names = ['Capital accumulation', 'Capital expenses', 'Current and capital expenditure',
             'Current expenditure', 'Current revenue', 'Capital revenue',
             'Current plus capital revenue', 'Current surplus', 'Overall surplus']

inputoutput = {'Capital accumulation':0, 'Capital expenses':1, 'Current and capital expenditure':2,
             'Current expenditure':3, 'Current revenue':4, 'Capital revenue':5,
             'Current plus capital revenue':6, 'Current surplus':7, 'Overall surplus':8}

def view2(x, df):
    if x == 'Average absolute change':
        result = print(f'The {x.lower()} of {df.lower()} is {dataframes[inputoutput[df]][x][3]:.0f} m. DKK per year.')
    elif x == 'Average percentage change':
        result = print(f'The {x.lower()} of {df.lower()} is {dataframes[inputoutput[df]][x][3]:.2f} percent per year.')
    else:
        plt.plot(dataframes[inputoutput[df]]['Time'], dataframes[inputoutput[df]][x])
        if x == 'Percentage change':
            plt.ylabel(f'{x} (%)')
        elif x == 'Change in percentage points':
            plt.ylabel(f'{x} (%-points)')
        elif x == 'Simple index':
            plt.ylabel(f'{x}')
        else:
            plt.ylabel(f'{x} (m. DKK)')
        plt.xlabel(f'Year')
        plt.title(f'Figure: {df}')
        result = plt.show()
    return result
x1 = widgets.Dropdown(options=rows, description='Calculation')
df1 = widgets.Dropdown(options=data_names,value='Capital accumulation', description='Variable')
interact(view2, x=x1,df=df1)
```

    interactive(children=(Dropdown(description='Calculation', options=('Absolute level', 'Absolute change', 'Avera…

    <function __main__.view2(x, df)>

dataproject/dataproject/Data project - FINAL.ipynb


# Shrinkage Methods: The Lasso

## Objective and Prerequisites

In this notebook, you will learn how to:

1. Apply the Lasso using mathematical programming.
2. Perform hyper-parameter tuning using random search.

To fully understand the content of this notebook, the reader should be familiar with the following:

- Differential calculus.
- Linear algebra (matrix multiplication, transpose and inverse).
- Linear regression analysis.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Motivation

For regression problems, the standard linear model is often used to describe the relationship between a response variable and a set of features. In fact, when applied to real-world problems, it is usually competitive in relation to non-linear methods. However, it may fall short when dealing with problems that have few observations with respect to the number of features considered. In particular, provided that the true relationship is approximately linear:

- If the number of observations $n$ is much larger than the number of features $d$, i.e. $n >> d$, then the Ordinary Least Squares (OLS) estimation will tend to have low variance. Hence, the linear model will perform well on unseen data.
- If $n$ is not much larger that $d$, then there can be a lot of variability in the fitting process, resulting in overfitting and thus a poor performance on new observations.
- If $n < d$, there is no longer a unique solution to the OLS algorithm, as the variance is infinite.

Use cases that commonly have few observations per feature include:

- Genome-scale data analysis.
- Clinical trials.
- Destructive testing.
- Analysis of production systems under abnormal conditions.

To exacerbate this, oftentimes some or many features may not be associated with the response, and including them would only increase the complexity of the resulting model. This notebook will present the Least Absolute Shrinkage and Selection Operator, better known as the Lasso, a technique that has gained a lot of traction because of its ability to deal with both issues.

---
## Problem Description

Linear Regression is a supervised learning algorithm used to predict a quantitative response. It assumes that there is a linear relationship between the feature vector $x_i \in \mathbb{R}^d$ and the response $y_i \in \mathbb{R}$. Mathematically speaking, for sample $i$ we have $y_i = \beta^T x_i + \epsilon_i$, where $\beta \in \mathbb{R}^d$ is the vector of feature weights, including the intercept, and  $\epsilon_i$ is a normally-distributed random variable with zero mean and constant variance representing the error term. We can learn the weights from a training dataset with $n$ observations $\{X \in \mathbb{M}^{nxd},y \in \mathbb{R}^n\}$ by minimizing the Residual Sum of Squares (RSS): $e^Te =(y-X\beta)^T (y-X\beta)=\beta^T X^T X\beta- 2y^TX\beta+y^T y$. The Ordinary Least Squares (OLS) method achieves this by taking the derivative of this quadratic and convex function and then finding the stationary point: $\beta_{OLS}=(X^T X)^{-1} X^T y$.

As previously discussed, Linear Regression does not perform well when the observations-per-feature ratio is low. Shrinkage methods, such as the Lasso, address this problem by constraining —or shrinking— the weight estimates, thus significantly reducing the variance at the expense of a slight increase in bias.

Specifically, the Lasso fits a model containing all $d$ predictors, while incorporating a budget constraint based on the L1-norm of $\beta$, disregarding the intercept component. Mathematically speaking, this method minimizes the RSS, subject to $\sum_{l=1}^{d-1}\mathopen|\beta_l\mathclose| \leq s$, where $s$ is a hyper-parameter representing the budget that is usually tuned via cross-validation (because of this constraint, the procedure is no longer scale-invariant). This constraint has the effect of shrinking all weight estimates, also known as regularization, allowing some of them to be exactly zero when $s$ is small enough. In fact, when $s \rightarrow 0$ the null model (intercept only) is retrieved, and when $s \geq ||\beta_{OLS}||_1$ the OLS model is recovered (whenever the budget constraint is non-binding). In light of this, the Lasso implicitly assumes that some of the weights are truly equal to zero, allowing it to perform feature selection.

It is worth noting that the unconstrained version of the Lasso is more frequently used. This version solves an unconstrained optimization problem where $RSS + \lambda \sum_{l=1}^{d-1}\mathopen|\beta_l\mathclose|$ is minimized, for a given value of the —modified— lagrangian multiplier $\lambda \in \mathbb{R}^+$. However, this notebook will focus on the model presented in the previous paragraph, which may be casted as a Quadratic Program (quadratic and convex objective function with linear constraints).

---
## Solution Approach

Mathematical programming is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

- Sets and indices.
- Parameters.
- Decision variables.
- Objective function(s).
- Constraints.

Before delving into the mathematical model, note that the budget constraint involves the L1-norm of the feature weights, which is defined as the sum of their absolute values. This constraint, in its current form, is not amenable to Quadratic Programming because it is nonlinear.  However, it is possible to linearize it using a variable transformation technique, at the expense of additional decision variables.

Let $\beta_j = \beta_j^+ - \beta_j^-$ and $|\beta_j| = \beta_j^+ + \beta_j^-$, where $\beta_j^+, \beta_j^- \geq 0 \quad \forall j \in \{1,2,\dots, d\}$ (for brevity, the intercept is also transformed). Then, the budget constraint can be re-expressed as $\sum_{l=1}^{d-1}(\beta_l^+ + \beta_l^-) \leq s$. Consequently, the RSS is modified as follows:

$$
\begin{equation}
e^Te \\
=(y^T-\beta^TX)(y-X\beta) \\
=[y^T-(\beta^+-\beta^-)^TX^T][y-X(\beta^+-\beta^-)] \\
=y^Ty -y^TX\beta^+ +y^TX\beta^- -\beta^{+T}X^Ty +\beta^{+T}X^TX\beta^+ -\beta^{+T}X^TX\beta^- +\beta^{-T}X^Ty -\beta^{-T}X^TX\beta^+ +\beta^{-T}X^TX\beta^- \\
= \beta^{+T}X^TX\beta^+ +\beta^{-T}X^TX\beta^- -2\beta^{+T}X^TX\beta^- -2y^TX\beta^+ +2y^TX\beta^- +y^Ty
\end{equation}
$$

As a side note, it can be shown from the definitions that $\beta_j^+ = \frac{|\beta_j|+\beta_j}{2}$ and $\beta_j^- = \frac{|\beta_j|-\beta_j}{2}$, which in turn implies that $\beta_j^-=0$ when $\beta_j \geq 0$ and $\beta_j^+=0$ when $\beta_j \leq 0$. In other words, $\beta_j^+ \cdot \beta_j^- = 0 \quad \forall j \in \{1,2,\dots, d\}$.

We are now ready to present a QP formulation that finds the weight estimates for a linear regression problem with an L1-norm constraint on the feature weights:

### Sets and Indices
$i \in I=\{1,2,\dots,n\}$: Set of observations.

$j \in J=\{1,2,\dots,d\}$: Set of features, where the last ID corresponds to the intercept.

$l \in L = J \backslash \{d\}$: Set of features, where the intercept is excluded.

### Parameters
$s \in \mathbb{R}^+$: Budget available for the L1-norm of $\beta$.

$Q = X^T X \in \mathbb{M}^{|J|x|J|}$: Quadratic component of the objective function.

$c = y^TX \in \mathbb{R}^{|J|}$: Linear component of the objective function.

***Note:** Recall that the RSS is defined as $\beta^{+T}X^TX\beta^+ +\beta^{-T}X^TX\beta^- -2\beta^{+T}X^TX\beta^- -2y^TX\beta^+ +2y^TX\beta^- +y^Ty$. However, Since $y^T y$ is constant w.r.t. the decision variables, we can drop this term altogether.

### Decision Variables
$\beta_j = \beta_j^+ - \beta_j^- \in \mathbb{R}$: Weight of feature $j$, representing the change in the response variable per unit-change of feature $j$.

$\beta_j^+, \beta_j^- \in \mathbb{R}^+$: Auxiliary variables used to model the absolute value of $\beta_j$.

### Objective Function

- **Training error**: Minimize the Residual Sum of Squares (RSS):

\begin{equation}
\text{Min} \quad Z = \frac{1}{2}\sum_{j \in J}\sum_{j' \in J}Q_{j,j'}\beta_j^+\beta_{j'}^+ +\frac{1}{2}\sum_{j \in J}\sum_{j' \in J}Q_{j,j'}\beta_j^-\beta_{j'}^- -\sum_{j \in J}\sum_{j' \in J}Q_{j,j'}\beta_j^+\beta_{j'}^- - \sum_{j \in J}c_j\beta_j^+ + \sum_{j \in J}c_j\beta_j^-
\tag{0}
\end{equation}

Note that we use the fact that if $x^*$ is a minimizer of $f(x)$, it is also a minimizer of $a\cdot f(x)$, as long as $a > 0$.

### Constraints

- **Budget constraint**: The L1-norm of the feature weights cannot exceed the budget:

\begin{equation}
\sum_{l \in L}(\beta_l^+ + \beta_l^-) \leq s
\tag{1}
\end{equation}

---
## Python Implementation

In the following implementation, we make use of three main libraries:

- **Numpy** for scientific computing.
- **Scikit learn** for machine learning algorithms.
- **Gurobi** for mathematical optimization.

```python
# Let's import all the necessary libraries

import numpy as np
from itertools import product
from gurobipy import *

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn import linear_model
from sklearn.metrics import mean_squared_error as mse
```

```python
# Create and deploy the optimization model

def lasso(X, y, budget, verbose=False):
    """
    Deploy and optimize the QP formulation of the Lasso. It assumes that data is
    centered and scaled.
    """
    regressor = Model()
    samples, dim = X.shape
    assert samples == y.shape[0]
    assert budget >= 0

    # Append a column of ones to the feature matrix to account for the y-intercept
    X = np.concatenate([X, np.ones((samples, 1))], axis=1)

    # Decision variables. Note that a change of variable is used
    beta_plus = regressor.addVars(dim + 1, name="beta_plus") # Weight estimates
    beta_minus = regressor.addVars(dim + 1, name="beta_minus") # Weight estimates
    intercept_plus = beta_plus[dim] # Last decision variable captures the y-intercept
    intercept_minus = beta_minus[dim] # Last decision variable captures the y-intercept
    intercept_plus.varname = 'intercept_plus'
    intercept_minus.varname = 'intercept_minus'

    # Objective Function (OF): minimize 1/2 * RSS using the facts that
    # if x* is a minimizer of f(x), it is also a minimizer of k*f(x) iff k > 0
    # if x* is a minimizer of f(x), it is also a minimizer of f(x) + k
    Quad = np.dot(X.T, X)
    lin = np.dot(y.T, X)
    obj = sum(0.5 * Quad[i,j] * beta_plus[i] * beta_plus[j] for i, j in product(range(dim + 1), repeat=2))
    obj += sum(0.5 * Quad[i,j] * beta_minus[i] * beta_minus[j] for i, j in product(range(dim + 1), repeat=2))
    obj -= sum(Quad[i,j] * beta_plus[i] * beta_minus[j] for i, j in product(range(dim + 1), repeat=2))
    obj -= sum(lin[i] * beta_plus[i] for i in range(dim + 1))
    obj += sum(lin[i] * beta_minus[i] for i in range(dim + 1))
    regressor.setObjective(obj, GRB.MINIMIZE)

    # Budget constraint. Note that the intercept is not included
    regressor.addConstr(quicksum([beta_plus[j] + beta_minus[j] for j in range(dim)]) == budget)

    if not verbose:
        regressor.params.OutputFlag = 0
    regressor.params.timelimit = 60
    regressor.optimize()

    coeff_plus = np.array([beta_plus[i].X for i in range(dim)])
    coeff_minus = np.array([beta_minus[i].X for i in range(dim)])
    return intercept_plus.X - intercept_minus.X, coeff_plus - coeff_minus
```

---
## Hyper-parameter Tuning

Unlike Ordinary Least Squares (OLS), the Lasso produces a different set of weight estimates $\beta(s)$ for each value of the budget $s$. Thus, we require a method for selecting a value for that hyper-parameter. As $s \in \mathbb{R}^+$, we will:

1. Try several values using random search and compute the cross-validated Mean Squared Error (MSE) for each of them. This metric is defined as  $\text{MSE} = \sum_{i}{(y_i - \hat{y}_i)^2}$ , where $y_i$ and $\hat{y}_i$ are the observed and predicted values for the ith observation, respectively.
2. Select the budget for which the cross-validated MSE is smallest.
3. Fit a model using all of the available observations and the selected value for the budget.

***Note:** As previously stated, we can bound our search to $0 \leq s \leq ||\beta_{OLS}||_1$.

```python
def split_folds(features, response, train_mask):
    """
    Assign folds to either train or test partitions based on train_mask.
    """
    xtrain = features[train_mask,:]
    xtest = features[~train_mask,:]
    ytrain = response[train_mask]
    ytest = response[~train_mask]
    return xtrain, xtest, ytrain, ytest

def cross_validate(features, response, budget, folds, seed):
    """
    Train a Lasso model for each fold and report the cross-validated MSE.
    """
    if seed is not None:
        np.random.seed(seed)
    samples, dim = features.shape
    assert samples == response.shape[0]
    fold_size = int(np.ceil(samples / folds))
    # Randomly assign each sample to a fold
    shuffled = np.random.choice(samples, samples, replace=False)
    mse_cv = 0
    # Exclude folds from training, one at a time, to get out-of-sample estimates of the MSE
    for fold in range(folds):
        idx = shuffled[fold * fold_size : min((fold + 1) * fold_size, samples)]
        train_mask = np.ones(samples, dtype=bool)
        train_mask[idx] = False
        xtrain, xtest, ytrain, ytest = split_folds(features, response, train_mask)
        intercept, beta = lasso(xtrain, ytrain, budget)
        ypred = np.dot(xtest, beta) + intercept
        mse_cv += mse(ytest, ypred) / folds
    # Report the average out-of-sample MSE
    return mse_cv

def get_budget_UB(X, y):
    """
    Calculates the L1 norm of the OLS estimates, in log10 units.
    """
    X = np.concatenate([X, np.ones((X.shape[0], 1))], axis=1)
    arg1 = np.linalg.inv(np.dot(X.T, X))
    arg2 = np.dot(X.T, y)
    budget = np.linalg.norm(np.dot(arg1, arg2)[:-1], ord=1)
    return np.log10(budget)

def lasso_cv(features, response, trials=10, folds=5, seed=None):
    """
    Looks for a promising value for the budget on a log-linear space and
    returns the weight estimates for the corresponding Lasso model.
    """
    if seed is not None:
        np.random.seed(seed)
    upper_bound = get_budget_UB(features, response)
    best_mse = np.inf
    best_budget = None
    for i in range(trials):
        # Explore values between the null and OLS models, in log-linear space
        exponent = np.random.uniform(0, upper_bound)
        budget = np.power(10, exponent)
        val = cross_validate(features, response, budget, folds=folds, seed=seed)
        if val < best_mse:
            best_mse = val
            best_budget = budget
    intercept, beta = lasso(features, response, best_budget)
    return intercept, beta
```

---
## Benchmark

We will validate the output of the implementation presented above with the one provided by Scikit learn. The Boston dataset is used for this purpose. This dataset measures the prices of 506 houses, along with 13 features that provide insights about their neighborhoods. We will use the original feature terminology, so the interested reader can visit [this website](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) for more information.

Note that 20% of the samples are reserved for computing the out-of-sample MSE. Furthermore, as the Lasso is not scale-invariant, we first standardize all features so they are expressed in the same units. Such preprocessing entails three steps, namely:

For each feature $x_l$:
1. Compute its sample average $\mu_l$ and sample standard deviation $\sigma_l$.
2. Center by subtracting $\mu_l$ from $x_l$.
3. Scale by dividing the resulting difference by $\sigma_l$.

This procedure has the effect of coercing the sample average and variance to be equal to zero and one, respectively, across all features.

```python
# Load data and split into train (80%) and test (20%)
boston = load_boston()
X = boston['data']
y = boston['target']
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=10101)
# Center and scale data to have zero mean and variance of one
scaler = StandardScaler()
scaler.fit(Xtrain)
Xtrain = scaler.transform(Xtrain)
Xtest = scaler.transform(Xtest)
```

```python
# Hyper-parameter tuning using random search
b0, beta = lasso_cv(Xtrain, ytrain, seed=10101)
mse(ytest, np.dot(Xtest, beta) + b0)
```

    23.36210006159952

```python
# Hyper-parameter tuning performed using grid search:
# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html
lr = linear_model.LassoCV(cv=5)
lr.fit(Xtrain, ytrain)
mse(ytest, lr.predict(Xtest))
```

    23.36501227078064

```python
# intercept is not accounted for
norm_qp = np.round(np.linalg.norm(np.array(beta), 1), 2)
norm_sklearn = np.round(np.linalg.norm(lr.coef_, 1), 2)
print("The L1-norm of Qp's output is: {0}".format(norm_qp))
print("The L1-norm of Scikit-learn's output is: {0}".format(norm_sklearn))
```

    The L1-norm of Qp's output is: 22.1
    The L1-norm of Scikit-learn's output is: 22.14

We can see that both metrics, the MSE and the L1-norm of the vector of feature weights, are virtually the same in both implementations. Notice that the norms are not exactly equal because the implementation in Scikit-learn uses grid search, rather than random search.

### Final Model

The previous analysis indicates that the best model is as follows:

$$
\text{medv} = 22.56-1.00\text{crim}+1.41\text{zn}+0.49\text{chas}-1.84\text{nox}+2.57\text{rm}
$$

$$
-1.84\text{age}-3.48\text{dis}+2.52\text{rad}-2.12\text{tax}-1.85\text{ptratio}+1.01\text{b}-3.64\text{lstat}
$$

Since we standardized the data, the intercept represents the estimated median value (in thousands) of a house with mean values across features. Likewise, we can interpret $\beta_1=-1.00$ as the decrease in the house value when the per-capita crime rate increases by one standard deviation from the average value, all other things being equal (similar statements can be made for the rest of the features).

---
## Conclusion

As already shown, the Lasso is especially useful when we have few observations for the learning process, or when we want to increase the interpretability of the model.

Also, this notebook presented a mathematical model based on Quadratic Programming to find an optimal solution for the Lasso. While this approach may not be the fastest, it can easily accommodate additional linear constraints (Bertsimas, 2015), such as:

- Enforcing group sparsity among features.
- Limiting pairwise multicollinearity.
- Limiting global multicollinearity.
- Considering a fixed set of nonlinear transformations.

Thus, mathematical programming provides a systematic approach to address common aspects of the model-building process, such as imposing statistical properties, which otherwise would be hard to incorporate into specialized algorithms. Ultimately, it comes down to choosing between efficiency and versatility.

---
## References

1. Bertsimas, D., & King, A. (2015). OR forum—An algorithmic approach to linear regression. Operations Research, 64(1), 2-16.
2. Busa, J. (2012). Solving quadratic programming problem with linear constraints containing absolute values. Acta Electrotechnica et Informatica, 12(3), 11.
3. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: springer.
4. The Boston housing dataset (1996, October 10). Retrieved from https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html
5. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267-288.

Copyright © 2019 Gurobi Optimization, LLC


# Laboratorio de física con Python

## Temario
* Simulación de ODE (de primer orden) [Proximamente, de orden superior!)
* Análisis de datos
 - Transformación de datos, filtrado
 - Ajuste de modelos
 - Integración
 - Derivación
* Adquisición de datos
* Gráficos

Importamos librerías: _numpy_ para análisis numérico, _scipy_ para funciones de integración y ajuste, y _matplotlib_ para graficar. De paso, definimos características de ploteo.

```python
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
from matplotlib import animation
import scipy.optimize as opt
%matplotlib inline

#Estilos de los gráficos de matplotlib.
plt.rcParams["figure.figsize"] = (5 * (1 + np.sqrt(5)) / 2, 5)
plt.rcParams["lines.linewidth"] = 2.5
plt.rcParams["ytick.labelsize"] = 12
plt.rcParams["xtick.labelsize"] = 12
plt.rcParams["axes.labelsize"] = 20
plt.rcParams["axes.grid"] = True
```

## Explicación del circuito
En este taller vamos a analizar el siguiente circuito, que corresponde a un RC

Para encontrar la tensión en el capacitor usamos la segunda ley de Kirchoff, que nos dice que

$v_{in} = v_{R} + v_{c}$
$ v_{in}(t) = R\;i(t) + \frac{1}{C} \int_{0}^{t} i(\tau) d\tau $

que si derivamos respecto al tiempo y multiplicamos por $C$, finalmente llegamos a la ecuación diferencial

$$ RC \; \frac{d i(t)}{dt} + i(t) = C \frac{d v_{in}(t)}{dt} $$

Para una señal escalón, que se simula con una señal cuadrada, se puede demostrar (ver https://en.wikipedia.org/wiki/Heaviside_step_function) que la derivada es nula para $t>0$, que es el tiempo que nos importa, por lo que vamos a resolver finalmente la siguiente ecuación

$$ RC \; \frac{d i(t)}{dt} + i(t) = 0 $$
$$\frac{d i(t)}{dt} = - \frac{1}{RC} i(t)$$
Pasemos a la simulación propiamente dicha, para lo que definimos el parámetro
$$\tau = RC$$

## Análisis de datos
Adquiridos datos o ya adquiridos, vamos a analizarlos.

```python
data = np.loadtxt("RC.csv")  #Traigo datos ya adquiridos
```

```python
print(type(data))
#print(dir(data))  #Permite graficar las funciones del ndarray
print(data)
```

    <class 'numpy.ndarray'>
    [[  8.00000000e+00   2.70000000e+01]
     [  1.28000000e+02   9.40000000e+01]
     [  2.48000000e+02   1.58000000e+02]
     [  3.68000000e+02   2.16000000e+02]
     [  4.88000000e+02   2.71000000e+02]
     [  6.08000000e+02   3.22000000e+02]
     [  7.28000000e+02   3.69000000e+02]
     [  8.48000000e+02   4.14000000e+02]
     [  9.68000000e+02   4.55000000e+02]
     [  1.08800000e+03   4.93000000e+02]
     [  1.20800000e+03   5.29000000e+02]
     [  1.32800000e+03   5.63000000e+02]
     [  1.44800000e+03   5.94000000e+02]
     [  1.56800000e+03   6.23000000e+02]
     [  1.68800000e+03   6.50000000e+02]
     [  1.81600000e+03   6.77000000e+02]
     [  1.93600000e+03   7.00000000e+02]
     [  2.05600000e+03   7.22000000e+02]
     [  2.17600000e+03   7.43000000e+02]
     [  2.29600000e+03   7.62000000e+02]
     [  2.41600000e+03   7.79000000e+02]
     [  2.53600000e+03   7.96000000e+02]
     [  2.65600000e+03   8.11000000e+02]
     [  2.77600000e+03   8.25000000e+02]
     [  2.89600000e+03   8.39000000e+02]
     [  3.01600000e+03   8.51000000e+02]
     [  3.13600000e+03   8.63000000e+02]
     [  3.25600000e+03   8.74000000e+02]
     [  3.37600000e+03   8.84000000e+02]
     [  3.49600000e+03   8.93000000e+02]
     [  3.61600000e+03   9.02000000e+02]
     [  3.73600000e+03   9.11000000e+02]
     [  3.85600000e+03   9.18000000e+02]
     [  3.97600000e+03   9.25000000e+02]
     [  4.09600000e+03   9.32000000e+02]
     [  4.21600000e+03   9.38000000e+02]
     [  4.33600000e+03   9.44000000e+02]
     [  4.45600000e+03   9.49000000e+02]
     [  4.57600000e+03   9.54000000e+02]
     [  4.69600000e+03   9.59000000e+02]
     [  4.81600000e+03   9.63000000e+02]
     [  4.93600000e+03   9.67000000e+02]
     [  5.05600000e+03   9.71000000e+02]
     [  5.17600000e+03   9.74000000e+02]
     [  5.29600000e+03   9.78000000e+02]
     [  5.41600000e+03   9.81000000e+02]
     [  5.53600000e+03   9.84000000e+02]
     [  5.65600000e+03   9.86000000e+02]
     [  5.77600000e+03   9.89000000e+02]
     [  5.89600000e+03   9.91000000e+02]
     [  6.01600000e+03   9.93000000e+02]
     [  6.13600000e+03   9.95000000e+02]
     [  6.25600000e+03   9.97000000e+02]
     [  6.37600000e+03   9.99000000e+02]
     [  6.49600000e+03   1.00100000e+03]
     [  6.61600000e+03   1.00200000e+03]
     [  6.73600000e+03   1.00400000e+03]
     [  6.85600000e+03   1.00500000e+03]
     [  6.97600000e+03   1.00600000e+03]
     [  7.09600000e+03   1.00700000e+03]
     [  7.21600000e+03   1.00800000e+03]
     [  7.33600000e+03   1.00900000e+03]
     [  7.45600000e+03   1.01000000e+03]
     [  7.57600000e+03   1.01100000e+03]
     [  7.70400000e+03   9.94000000e+02]
     [  7.82400000e+03   9.26000000e+02]
     [  7.94400000e+03   8.63000000e+02]
     [  8.06400000e+03   8.04000000e+02]
     [  8.18400000e+03   7.49000000e+02]
     [  8.30400000e+03   6.98000000e+02]
     [  8.42400000e+03   6.51000000e+02]
     [  8.54400000e+03   6.07000000e+02]
     [  8.66400000e+03   5.66000000e+02]
     [  8.78400000e+03   5.27000000e+02]
     [  8.90400000e+03   4.92000000e+02]
     [  9.02400000e+03   4.58000000e+02]
     [  9.14400000e+03   4.27000000e+02]
     [  9.26400000e+03   3.98000000e+02]
     [  9.38400000e+03   3.71000000e+02]
     [  9.50400000e+03   3.46000000e+02]
     [  9.62400000e+03   3.22000000e+02]
     [  9.74400000e+03   3.00000000e+02]
     [  9.86400000e+03   2.80000000e+02]
     [  9.98400000e+03   2.61000000e+02]
     [  1.01040000e+04   2.43000000e+02]
     [  1.02240000e+04   2.26000000e+02]
     [  1.03440000e+04   2.11000000e+02]
     [  1.04640000e+04   1.97000000e+02]
     [  1.05840000e+04   1.83000000e+02]
     [  1.07040000e+04   1.71000000e+02]
     [  1.08240000e+04   1.54000000e+02]
     [  1.10000000e+04   1.44000000e+02]
     [  1.11400000e+04   1.33000000e+02]
     [  1.12560000e+04   1.24000000e+02]
     [  1.13760000e+04   1.16000000e+02]
     [  1.14960000e+04   1.08000000e+02]
     [  1.16160000e+04   1.00000000e+02]
     [  1.17360000e+04   9.30000000e+01]
     [  1.18560000e+04   8.70000000e+01]
     [  1.19760000e+04   8.10000000e+01]
     [  1.21120000e+04   7.50000000e+01]
     [  1.22320000e+04   7.00000000e+01]
     [  1.23520000e+04   6.50000000e+01]
     [  1.24720000e+04   6.00000000e+01]
     [  1.25920000e+04   5.60000000e+01]
     [  1.27120000e+04   5.20000000e+01]
     [  1.28320000e+04   4.90000000e+01]
     [  1.29520000e+04   4.50000000e+01]
     [  1.30720000e+04   4.20000000e+01]
     [  1.31920000e+04   3.90000000e+01]
     [  1.33120000e+04   3.60000000e+01]
     [  1.34320000e+04   3.40000000e+01]
     [  1.35520000e+04   3.10000000e+01]
     [  1.36720000e+04   2.90000000e+01]
     [  1.37920000e+04   2.70000000e+01]
     [  1.39120000e+04   2.50000000e+01]
     [  1.40320000e+04   2.30000000e+01]
     [  1.41520000e+04   2.20000000e+01]
     [  1.42720000e+04   2.00000000e+01]
     [  1.43920000e+04   1.90000000e+01]
     [  1.45120000e+04   1.70000000e+01]
     [  1.46320000e+04   1.60000000e+01]
     [  1.47520000e+04   1.50000000e+01]
     [  1.48720000e+04   1.40000000e+01]
     [  1.49920000e+04   1.30000000e+01]
     [  1.51120000e+04   1.20000000e+01]
     [  1.52320000e+04   1.10000000e+01]
     [  1.53520000e+04   1.00000000e+01]]

Ahora imprimimos los datos. Como son dos tiras de datos, debemos imprimir varios

```python
plt.plot(data[:,0], data[:,1], "ro-");
```

Vamos a hacer un poco de análisis de datos, para eso, tomemos solamente una parte de la curva verde. Index slicing al rescate! Graficamos para revistar los resultados

```python
#Hay varias opciones para sacar elementos
#fitData = data[0:64]
fitData = data[data[:, 0] < 7700]

#Grafiquemos los resultados
plt.plot(fitData[:,0], fitData[:,1], 'bo-')
```

Para ajustar el modelo, que sabemos que es
$ V = V_0 ( 1 - e^{-B t}) $
primero construimos la función

```python
f = lambda x, A, B, C: A * np.exp(- B * x) + C
```

Y luego usamos la función curve_fit de scipy

```python
T = (fitData[:,0] - fitData[:,0].min()) * 1e-6
V = fitData[:,1]
ErrV = 20  #Este error corresponde solamente al instrumental
p0 = (-1023, 1000, 1024)
p, cov = opt.curve_fit(f, T, V, p0)
#Construyo una variable auxiliar del T
t = np.linspace(T.min(), T.max(), 1000)
plt.errorbar(T, V, yerr = ErrV, fmt = 'go')
#plt.plot(T, V, 'go')
plt.plot(t, f(t, *p), 'r-') #Grafico el ajuste
#Presento de una manera "linda" el ajuste
sigma = np.sqrt(np.diag(cov)) #la diagonal de la covarianza corresponde a la varianza, el "cuadrado del error"
print("f = A exp(-B t) + C")
print("A = {:.2f} +- {:.2f}".format(p[0], sigma[0])) #Le digo que ponga solo dos digitos después de la coma
print("B = {:.2f} +- {:.2f}".format(p[1], sigma[1]))
print("C = {:.2f} +- {:.2f}".format(p[2], sigma[2]))
```

La componentes corresponde a $R = (18,0 \pm 0,2)\text{k}\Omega$, $C = (0,10 \pm 0,01)\mu\text{F}$, con lo que la constante del circuito nos queda
$\tau = \dfrac{1}{RC} = (555 \pm 62) $
que nos permite corresponder el modelo propuesto con lo experimental.

También podemos ver la "bondad del ajuste", para lo que necesitamos el $\chi$-cuadrado. La función curve_fit no lo devuelve, pero podemos calcularlo rápidamente

```python
chi2Red = (np.power((V - f(T, *p)) / np.sqrt(ErrV), 2)).sum()/(len(V) - 2)
chi2Red
```

    0.0039539953544748853

Acá tenemos el $\chi$-cuadrado reducido, que debe ser cercano a 1. Si es mucho menor que 1, los errores están sobre dimensionados; si es mucho mayor a 1, el ajuste puede ser rechazado. Para cuantificar la palabra "mucho" podemos usar el p-valor con el test de $\chi$-cuadrado (https://en.wikipedia.org/wiki/Goodness_of_fit)

## Derivación e integración

Ahora vamos a tomar datos para un integrador y un derivador. Para eso ejecuten la parte de adquisición de datos, si tienen el dispositivo, y obtengan los datos. Si no importenlos como ya hicimos

```python
data = np.loadtxt("RC_int.csv")
plt.plot(data[:,0], data[:,1], "ro-");
```

Derivemos los datos para ver que resultado tenemos. Para eso tenemos la función de _numpy.diff_. Notar que al diferenciar el resultado tiene un dato menos, por lo que se debe eliminar del "vector tiempo" data[:,0]

```python
plt.plot(data[:-1,0], np.diff(data[:,1]), 'go');
```

Se "parece" a una cuadrada, creemos datos que representen a una cuadrada e integremoslos y comparandolos. Para eso, debemos usar la función _scipy.integrate.cumtrapz_. Luego reescalamos la señal integrada para compararla con la cuadrada

```python
from scipy.integrate import cumtrapz
T = data[:,0]
N = int(data[:,1].shape[0] / 2)
V = np.concatenate((np.full(N, 1), np.zeros(N) - 1))
plt.plot(T,V,'go')
V_int = sp.integrate.cumtrapz(V, initial = 0)
V_int /= V_int.max()
plt.plot(T, V_int, 'ro')
```

Veamos ahora comparado los datos adquiridos respecto a la integración numérica, escalado para que sea comparable

```python
plt.plot(data[:,0], data[:,1] / data[:,1].max(), 'bo', T, V_int, 'go')
```

Se nota que el circuito RC se acerca a la integración numérica, pero debido a lo que se conoce como ganancia del filtro RC y la constante de tiempo $\tau$ no se llega a obtener un integrador real. Para mejorar esto, en general, se debe usar circuitos activos (ver https://en.wikipedia.org/wiki/Active_filter), pero requieren más experiencia de diseño

## Simulación del circuito

### Simulación numérica
Conociendo el comportamiento del circuito y la forma funcional de su derivada podemos usar las funciones que ya conocemos para simularlo. Recordemos que la ecuación diferencial había quedado como

$ \dfrac{d i(t)}{dt} = - \dfrac{1}{\tau} i(t)$

Esto lo podemos escribir como

```python
from scipy.integrate import odeint
tau = 10 #Varien el parámetro y vean el resultado
def f(y, t, tau):
    return (-y * tau)
t = np.linspace(0, 1, 1000)
y0 = [1]
y = odeint(f, y0, t, args = (tau,))
plt.plot(t, y, 'r-');
```

Recordemos que la ecuación diferencial que estamos resolviendo corresponde a la corriente del circuito. Si en vez de eso queremos la tensión del capacitor, debemos integrar este resultado, ya que

$$ v_{c}(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau $$

Con la librería _scipy.integrate_, y la función _cumtrapz_, podemos integrar, ya que permite aplicar acumulativamente la regla del trapesoide.

```python
from scipy.integrate import cumtrapz
y_int = cumtrapz(y.ravel(), t)
plt.plot(t[:-1], y_int.ravel(),'g-')
```

### Simulación analítica
Para completar, vamos a encontrar la expresión analítica de la corriente y la tensión del capacitor, que nos va a permitir determinar el modelo y ajustar los datos

```python
#Usamos sympy, que tiene un tutorial muy completo en
#http://docs.sympy.org/latest/tutorial/
#Nostros no hacemos más que repetir
from sympy import symbols, Function, Eq, dsolve, integrate, collect_const

#Creo los simbolos, es decir variables con significado simbolico
t, tau = symbols('t, tau')
i = symbols("i", cls=Function)
diffEq = Eq(i(t).diff(t), -i(t)/tau)
sol = dsolve(diffEq)
sol
```

    i(t) == C1*exp(-t/tau)

Esta expresión es la corriente, para la tensión del capacitor, debemos integrar este resultado

```python

C1, u, A = symbols('C1 u A')  #Creo una variable de integración, la constante de integración y una variable A
g = sol.rhs.subs(t, u)  #Tomo la parte izquierda de la igualdad, y elimino t por u para integrar
I = integrate(g, (u, 0, t))
collect_const(I.subs(tau*C1,A), A) #Esta función solo reodena el resultado, y remplazo tau*C1 -> A
```

    A*(1 - exp(-t/tau))

Por lo que el modelo utilizado previamente es coherente con la solución de la ecuación diferencial. Con esto queda un pantallazo de toda la simulación, analítica y numérica, que se puede efectuar en Python. Nada mal

## Adquisición de datos
Acá están las funciones para adquisición de datos. **No es necesario ejecutarlas**, pero si tenés un dispositivo serie que devuelva una lista de datos ASCII separada por tabulaciones (como ser algunos osciloscopios, por ejemplo), esto te va a permitir guardarlo!. Mientras, está pensado para utilizarlo con un [Arduino](http://www.arduino.cc), que fue programado por nosotros (y [acá está el código]() en Processing para ver que se hace). La librería _pandas_ tiene herramientas muy poderosas de análisis y filtrado de datos, pero es más de lo que necesitamos en general y usamos solamente _numpy_

```python
import io
import time
import serial
import pandas as pd
from serial.tools import list_ports
def inputPort():
    '''Obtiene la lista de puertos, la presenta en pantalla y da
    a elegir un puerto, devolviendo el string para conectarse
    '''
    ports = list(list_ports.comports())
    for i,p in enumerate(ports):
        print("[{}]: Puerto {}".format(i + 1, p[1]))
    port = ""
    if len(ports) > 0:
        port = ports[int(input("Ingrese el puerto serie: ")) - 1][0]
    return port

def updateData(s):
    '''Obtiene los datos que manda el Arduino/Teensy por UART USB'''
    A = []
    s.flushInput()
    s.write(b"1")
    time.sleep(2)
    while True:
        if ser.inWaiting() == 0:
            break
        A.append(s.read().decode())
    A = "".join(A)
    data = pd.read_csv(io.StringIO(A),sep="\t",names=["t","v"]).dropna(axis=0)
    return data

port = inputPort()
ser = serial.Serial(port, "9600")
data = updateData(ser).values
ser.close()
np.savetxt("data.csv", data)
```

Grafiquemos los datos para estar seguros de que obtenimos el resultado!

```python
plt.plot(data[:,0], data[:,1], "ro-");
```

Para complentar, reimplementamos la adquisición de datos, pero la hacemos un bucle y presentamos en tiempo real la adquisición, con un intervalo de refresco de 2s, aproximadamente

```python
from IPython import display #Permite borrar la salida y volverla a cargar. Sirve para toda instancia de Ipython

inputPort()
ser = serial.Serial(port, "9600")
data = updateData(ser).values
ser.close()
for i in range(50):
    plt.clf()
    data = update()
    plt.plot(data.t, data.V, "ro")
    display.clear_output(wait=True)
    display.display(plt.gcf())
    #time.sleep(0.5) #Este delay no es necesario, ya está implementado en el update
plt.close()
```


# Inaugural Project - Housing demand and taxation
### - *Mathilde Pilgaard, Klara Krogh Hammerum, Louise Albæk Jensen og Oluf Kelkjær*
A given household can spend cash $m$ on either housing or consumption $c$. Quality of housing, $h$, grants household utility and has the cost $p_{h}$ which is subject to progressive taxation and morgage cost as by equation (\ref{eq4}). The household aims to maximize utility such that optimal amounts of housing, $h^{*}$, and consumption, $c^{*}$, are met according to equation (\ref{eq1}).

\begin{align}
    c^{*},h^{*} & = \underset{c, h}{\operatorname{argmax}} c^{1-\phi}h^{\phi} \label{eq1}\tag{1} \\
    &s.t.\\
    \tilde{p}_{h} &= p_{h}\varepsilon \label{eq2}\tag{2} \\
    m &= \tau(p_{h},\tilde{p}_{h})+c \label{eq3}\tag{3} \\
    \tau(p_{h},\tilde{p}_{h})&=r p_{h}+\tau^{g}\tilde{p}_{h}+\tau^{p}max\{\tilde{p}_{h}-\bar{p},0\} \label{eq4}\tag{4}
\end{align}

The homes are taxed according to the public assesment being $\tilde{p}_{h}$. The assesment is politically decided according to equation (\ref{eq2}) by $\varepsilon$. Available cash for a given household is split between housing costs and consumption. $r$ denotes the carrying interest for purchasing a home, and $\tau^{g}$ denotes the base housing tax while the last element in equation (\ref{eq4}) homes valued above a certain cutoff, $\bar{p}$ are taxed at an additional rate $\tau^{p}$

# Q1
Construct a function that solves household's problem above. We let the market price of a home be equal to its quality:
$$ p_{h}=h $$
and assume that the household in question has cash-on-hand $m=0.5$. Notive that the monetary units is in millions DKK. In addition use the given parameter-values.
Knowing that the marketprice of a home is equal to its quality, we can use a scalar solver using the problems monotonicity.
$$ c = m - \tau(p_{h},\tilde{p}_{h})\Leftrightarrow $$
$$ c = m - rh-\tau^{g}h\varepsilon+\tau^{p}max\{h\varepsilon-\bar{p},0 \}   $$
meaning $c$ is implicit for a given level of $h$ through the budget constraint

```python
# Importing relevant packages
from scipy import optimize
import numpy as np

par1 = {'m':0.5,
        'phi':0.3,
        'epsilon': 0.5,
        'r': 0.03,
        'tau_g': 0.012,
        'tau_p': 0.004,
        'p_bar': 3
}

# Creating utility function
def u_func(c, h, phi):
    return c**(1-phi)*h**phi

# Creating objective function to minimize
def value_of_choice(h,m,phi,epsilon,r,tau_g, tau_p, p_bar):
    c = m - r*h-tau_g*h*epsilon-tau_p*max(h*epsilon-p_bar,0)
    return -u_func(c,h,phi)

```

```python
from scipy import optimize

par1 = {'m':0.5,
        'phi':0.3,
        'epsilon': 0.5,
        'r': 0.03,
        'tau_g': 0.012,
        'tau_p': 0.004,
        'p_bar': 3
}

def u_optimizer(phi, m, epsilon, r, tau_g, tau_p, p_bar, print_res = False, **kwargs):
    sol = optimize.minimize_scalar(lambda h: value_of_choice(h,phi, m, epsilon, r, tau_g, tau_p, p_bar),
                                  method = 'bounded', bounds = (0,m*10))
    h = sol.x
    h_cost = r*h+tau_g*h*epsilon+tau_p*max(h*epsilon-p_bar,0)
    c = m - h_cost
    u = u_func(c,h,phi)

    if print_res == True:
        excess_dkk = m - h_cost - c

        text = f'---'*35
        text += '\nFor parameters:\n'
        text += f'\u03C6={phi:.3f}'
        text += f' m={m:.3f}'
        text += f' \u03B5={epsilon:.3f}'
        text += f' r={r:.3f}'
        text += f' \u03C4ᵍ={tau_g:.3f}'
        text += f' \u03C4ᵖ={tau_p:.3f}'

        text += f'\nThe optimal allocation of ressources is consuming c={c:.3f} while buying a house of quality h={h:.3f} \n'
        text += f'As the household pays no downpayment and only has to service interest-rate and housing taxes, the total spending amounts to:\n'
        text += f'c={c:.3f} and buying a house of quality h={h:.3f} at a cost of {h_cost:.3f} leaving excess mDKK of {excess_dkk:.3f}\n\n'
        text += f'Above allocation of ressources results in a utility of {u:.3f}\n'
        text += f'---'*35
        print(text)

    return u, h, c

u_star, h_star, c_star = u_optimizer(**par1, print_res = True)

```

    ---------------------------------------------------------------------------------------------------------
    For parameters:
    φ=0.300 m=0.500 ε=0.500 r=0.030 τᵍ=0.012 τᵖ=0.004
    The optimal allocation of ressources is consuming c=0.350 while buying a house of quality h=4.167
    As the household pays no downpayment and only has to service interest-rate and housing taxes, the total spending amounts to:
    c=0.350 and buying a house of quality h=4.167 at a cost of 0.150 leaving excess mDKK of 0.000

    Above allocation of ressources results in a utility of 0.736
    ---------------------------------------------------------------------------------------------------------

```python
import numpy as np
N = 10000

# Initializing storage
m_array = np.linspace(0.4,4,N)
h_stars = np.empty(N)
c_stars = np.empty(N)
u_stars = np.empty(N)

# Excluding m in a new dictionary, such that it can be varied.
par1_exc = par1.copy()
del par1_exc['m']
par1_exc

for i, v in enumerate(m_array):
    result_vector = u_optimizer(**par1_exc, m=v, print_res = False)

    # loading results
    u_stars[i] = result_vector[0]
    h_stars[i] = result_vector[1]
    c_stars[i] = result_vector[2]

# print(u_stars)

# Plot housing costs
# I will probably sky-rocket or at least kink at some point.
# Problem: c can be negative, shouldn't happen
# Create bounds on h, such that cost of h can't exceed some limit.
# Perhaps make the unknown h, such that c is selected and put boundary on such that c can't exceed m.

c_stars
```

    C:\Users\olufk\Anaconda3\lib\site-packages\ipykernel_launcher.py:17: RuntimeWarning: invalid value encountered in double_scalars

    array([0.28000003, 0.28025206, 0.28050408, ..., 3.43079611, 3.43110389,
           2.76400002])

```python
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(20,5))# figsize is in inches

ax = fig.add_subplot(1,2,1)
ax.plot(c_stars,h_stars)
ax.set_xlabel(f"$c^\star$")
ax.set_ylabel(f"$h^\star$")
ax.set_title(f'Value function')

ax_right = fig.add_subplot(1,2,2)
ax_right.plot(m_array,u_stars)
ax_right.set_xlabel(f"$m$")
ax_right.set_ylabel(f"$u^\star$")
ax_right.set_title(f'Consumption function')

plt.show()
```

```python
par1_exc = par1.copy()
del par1_exc['m']
par1_exc
u_star, h_star, c_star = u_optimizer(**par1_exc, m=2)
u_star
```

    2.726625822432702


# [A rolling disc with Kane’s method ](https://docs.sympy.org/latest/modules/physics/mechanics/examples/rollingdisc_example_kane.html)

## 系統參數

### Function of time
* Angle $q_1(t),~q_2(t),~q_3(t)$
* $u_1(t),~u_2(t),~u_3(t)$

### Scalars
* radius: r
* mass: m
* gravity: g
* time: t

```python
from sympy import symbols, sin, cos, tan
from sympy.physics.mechanics import *
q1, q2, q3, u1, u2, u3  = dynamicsymbols('q1 q2 q3 u1 u2 u3')
q1d, q2d, q3d, u1d, u2d, u3d = dynamicsymbols('q1 q2 q3 u1 u2 u3', 1)
r, m, g, t = symbols('r m g t')
```

## 座標轉換

### Yaw-Row-Pitch rotation sequence(3-1-2)

```python
N = ReferenceFrame('N') # the inertia frame
Y = N.orientnew('Y', 'Axis', [q1, N.z]) # yaw angle q1
L = Y.orientnew('L', 'Axis', [q2, Y.x]) # roll angle q2
R = L.orientnew('R', 'Axis', [q3, L.y]) # pitch angle q3
```

#### ang_vel_in(otherframe):
Returns the angular velocity vector of the ReferenceFrame.

#### set_ang_vel(otherframe, value):
Define the angular velocity vector in a ReferenceFrame.

#### set_ang_acc(otherframe, value):
Define the angular acceleration Vector in a ReferenceFrame.

```python
w_R_N_qd = R.ang_vel_in(N) # angular velocity of frame R w.r.t inertia frame N
R.set_ang_vel(N, u1 * L.x + u2 * L.y + u3 * L.z) #
```

```python

```

```python
C = Point('C')  # define the disc contact point C
C.set_vel(N, 0) # velocity of disc contact point = 0
```

Form the position vector from the contact point to the disc’s center of mass.

#### v2pt_theory(otherpoint, outframe, fixedframe):
Sets the velocity of this point with the 2-point theory. The 2-point theory for point velocity looks like this:

$$ {}^{N}v^{P} = {}^{N}v^{O} + {}^{N}\omega^{B} \times r^{O/P} $$

where O and P are both points fixed in frame B, which is rotating in frame N.

```python
Dmc = C.locatenew('Dmc', r * L.z)
Dmc.v2pt_theory(C, N, R)
```

$\displaystyle r \left(\sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)\mathbf{\hat{l}_x} -  r \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\mathbf{\hat{l}_y}$

```python

```