#  Taylor problem 5.50

last revised: 21-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Here we are exploring the Fourier series for a waveform defined to be odd about the origin, so $f(-t) = -f(t)$, with period $\tau$.  That means that the integrand for the $a_m$ coefficients is odd and so all of the corresponding integrals vanish.

The particular wave of interest here is a sawtooth, such that in the interval $-\tau/2 \leq t \leq \tau/2$, the function takes the form:

$\newcommand{\fmax}{f_{\textrm{max}}}$
$\begin{align}
  f(t) = \left\{ \begin{array}{ll}
                   \fmax(t/\tau) & t < 0 \\
                   \fmax(t/\tau) & t > 0
                 \end{array}
                   \right.
\end{align}$

(we wrote it this way so it looks like the function for problem 5.49).

As already note, the $a_m$ coefficients are zero, so we only calculate the $b_m$ coefficients.  Here $\omega \equiv 2\pi/\tau$.  The result is:

$\begin{align}
  b_m = \frac{2}{\tau} \int_{-\tau/2}^{\tau/2} \sin(m\omega t) f(t)\, dt =
     % 2 \fmax \int_0^1 \sin(m\pi t) t\, dt
    %  &= - \frac{2\fmax}{(m\pi)^2)}\left[\sin(m\pi t)\right]^1_0 \\
     % =
      \left\{
      \begin{array}{ll}
        -\frac{ \fmax}{m\pi} & [m\ \mbox{even}] \\
        \frac{ \fmax}{m\pi} & [m\ \mbox{odd}]
      \end{array}
      \right.
\end{align}$

Note that the coefficients are independent of $\tau$.  Is this a general result?

## Define the functions we'll need

```python
import numpy as np
import sys
import matplotlib.pyplot as plt
from scipy.integrate import quad
```

We start by defining a function for the sawtooth wave at any $t$.  The definition here is for a scalar function.  That is, it won't work to call it with $t$ and array of time points, unlike other functions we have defined.  It is possible to make it work, but then the function will be much less clear.  When we need to evaluate it for all elements of an array, we will use the construction: `np.array([sawtooth(t) for t in t_pts])` for the array `t_pts`.

```python
def sawtooth(t, tau, f_max=1):
    """Returns the sawtooth wave of amplitude f_max and odd about the
        origin at time t.  The period is tau.  It is defined as a scalar
        function (i.e., only one value of t can be passed at a time).
    """
    if np.floor(t) % 2 == 0:
        t_adjust = t - np.floor(t + 1/2)
        return t_adjust / tau
    else:
        t_adjust = t - (np.floor(t) - 1)
        return t_adjust / tau

```

Now a function that creates an array of Fourier coefficients for the sawtooth wave up to order N_max.

```python
def sawtooth_coeffs_by_hand(N_max, tau=2., f_max=1.):
    """Fourier coefficients calculated by hand and loaded into an array.
        Note that these are independent of tau, but we pass it for
        consistency with other functions.
    """
    coeffs_array = [(0., 0.)]   # a_0 and b_0
    for n in np.arange(1, N_max, 1):
        #if (n % 2) == 0:    # for even n
        #    b_n = -f_max * (n * np.pi)
        #else:          # for odd n
        #    b_n = f_max * (n * np.pi)
        a_n = 4*f_max*(n*np.pi)**2
        b_n = 0
        coeffs_array.append((a_n, b_n))

    return np.array(coeffs_array)  # convert to a numpy array
```

We would like a general way to construct the away of Fourier coefficients given any periodic function.  Our first pass at that uses a class definition and the scipy integration function quad.

```python
class FourierSeries():
    """
    Fourier series class finds the coefficients in a Fourier series with
    period tau up to a specified order.
    Assume these imports:
        from scipy.integrate import quad
        import numpy as np
    """
    def __init__(self,
                 function,
                 tau=2,
                 N_max=10
                ):
        self.function = function
        self.tau = tau
        self.omega = 2. * np.pi / tau
        self.N_max = N_max
        try:
            N_max
        except N_max < 0:
            sys.exit(1)
        # add something to quit if N_max < 0 or not an integer (try and except)

    def a0_calc(self):
        """Calculate the constant Fourier coefficient by integration"""
        answer, error = quad(self.function, -tau/2., tau/2., args=(tau,))
        return (1./self.tau) * answer

    def an_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.cos(n * self.omega * t)

    def an_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        # note comma after n in args
        answer, error = quad(self.an_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def bn_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.sin(n * self.omega * t)

    def bn_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        answer, error = quad(self.bn_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def coeffs_upto_Nmax(self):
        """Calculate the Fourier series up to Nmax"""
        # first generate the coefficient
        coeffs_array = [(self.a0_calc(), 0)]   # a_0 and b_0

        for n in np.arange(1, N_max, 1):
            a_n = self.an_calc(n)
            b_n = self.bn_calc(n)
            coeffs_array.append((a_n, b_n))  # append a tuple of coefficients

        return np.array(coeffs_array)  # convert to a numpy array

```

Finally, we need a function that can take as input an array of t values and an array of Fourier coefficients and return the function at those t values with terms up to order N_max.

```python

def Fourier_reconstruct(t_pts, coeffs_array, tau, N_max):
    """Sum up the Fourier series up to n = N_max terms."""
    omega = 2. * np.pi / tau
    result = 0.
    # iterate over coefficients but only up to N_max
    for n, (a,b) in enumerate(coeffs_array[:N_max+1]):
        result = result + a * np.cos(n * omega * t_pts) \
                        + b * np.sin(n * omega * t_pts)
    return result
```

## Problem 5.50

Ok, now we can do problem 5.49.  Calculate the coefficients both ways.

```python
N_max = 20
tau = 2.
f_max = 1.
coeffs_by_hand = sawtooth_coeffs_by_hand(N_max, tau, f_max)

fs = FourierSeries(sawtooth, tau, N_max)
coeffs_by_quad = fs.coeffs_upto_Nmax()
```

Let's check that the exact and numerical calculation of the coefficients agree.

(Note the space in the formats, e.g., `{a1: .6f}`.  This means to leave an extra space for a positive number so that it aligns at the decimal point with negative numbers.)

```python
print(' n   a_exact     a_quad      b_exact     b_quad')
for n, ((a1,b1), (a2,b2)) in enumerate(zip(coeffs_by_hand,
                                           coeffs_by_quad)):
    print(f'{n:2d}  {a1: .6f}  {a2: .6f}    {b1: .6f}  {b2: .6f}')
```

     n   a_exact     a_quad      b_exact     b_quad
     0   0.000000   0.375000     0.000000   0.000000
     1   39.478418   0.159155     0.000000  -0.477465
     2   157.913670   0.000000     0.000000  -0.000000
     3   355.305758  -0.053052     0.000000  -0.159155
     4   631.654682   0.000000     0.000000  -0.079577
     5   986.960440   0.031831     0.000000  -0.095493
     6   1421.223034  -0.000000     0.000000   0.000000
     7   1934.442463  -0.022736     0.000000  -0.068209
     8   2526.618727  -0.000000     0.000000  -0.039789
     9   3197.751826   0.017684     0.000000  -0.053052
    10   3947.841760  -0.000000     0.000000   0.000000
    11   4776.888530  -0.014469     0.000000  -0.043406
    12   5684.892135  -0.000000     0.000000  -0.026526
    13   6671.852575   0.012243     0.000000  -0.036728
    14   7737.769850  -0.000000     0.000000   0.000000
    15   8882.643961  -0.010610     0.000000  -0.031831
    16   10106.474907  -0.000000     0.000000  -0.019894
    17   11409.262688   0.009362     0.000000  -0.028086
    18   12791.007304   0.000000     0.000000  -0.000000
    19   14251.708755  -0.008377     0.000000  -0.025130

Make the comparison plot requested: N_max = 2 vs. N_max = 6.

```python
t_pts = np.arange(-2., 6., .01)
f_pts_2 = Fourier_reconstruct(t_pts, coeffs_by_quad, tau, 2)
f_pts_6 = Fourier_reconstruct(t_pts, coeffs_by_quad, tau, 6)

# Python way to evaluate the sawtooth function at an array of points:
#   * np.array creates a numpy array;
#   * note the []s around the inner statement;
#   * sawtooth(t) for t in t_pts
#       means step through each element of t_pts, call it t, and
#       evaluate sawtooth at that t.
#   * This is called a list comprehension.  There are more compact ways,
#      but this is clear and easy to debug.
sawtooth_t_pts = np.array([sawtooth(t, tau, f_max) for t in t_pts])
```

```python

fig_1 = plt.figure(figsize=(10,5))
ax_1 = fig_1.add_subplot(1,2,1)
ax_1.plot(t_pts, f_pts_2, label='N = 2', color='blue')
ax_1.plot(t_pts, sawtooth_t_pts, label='exact', color='red')
ax_1.set_xlim(-1.1,4.1)
ax_1.set_xlabel('t')
ax_1.set_ylabel('f(t)')
ax_1.set_title('N = 2')
ax_1.legend()

ax_2 = fig_1.add_subplot(1,2,2)
ax_2.plot(t_pts, f_pts_6, label='N = 6', color='blue')
ax_2.plot(t_pts, sawtooth_t_pts, label='exact', color='red')
ax_2.set_xlim(-1.1,4.1)
ax_2.set_xlabel('t')
ax_2.set_ylabel('f(t)')
ax_2.set_title('N = 6')
ax_2.legend();

fig_1.tight_layout()
fig_1.savefig('problem_5.50.png')
```

```python

```


## Review on IR Metrics

Given the predictions $\mathbf{y}=f(\mathbf{x})$ and the corresponding ground-truth
labels $\mathbf{y}^{*}$, we use $Y^{+}=\{i|y_{i}^{*}>0\}$ and $Y^{-}=\{j|y_{j}^{*}=0\}$ to represent the sets of relevant documents and non-relevant documents, respectively. We use $\mathbf{b}^{*}=\mathbb{I}\{\mathbf{y}^{*}>0\}$ with $b_{j}^{*}=\mathbb{I}\{y_{j}^{*}>0\}$
to represent the binarized ground-truth, and the cumulative sum on
$\mathbf{b}^{*}$ is given as $B_{k}^{*}=\sum_{j=1}^{k}b_{j}^{*}$. The scoring function $f$ induces
a ranking $\bar{\mathbf{y}}$.
The corresponding ground-truth labels are  $\mathbf{y}^{**}$. Furthermore, we denote the binarized ground-truth labels as $\mathbf{b}^{**}=\mathbb{I}\{\mathbf{y}^{**}>0\}$
with $b_{j}^{**}=\mathbb{I}\{y_{j}^{**}>0\}$, and the cumulative
sum on $\mathbf{b}^{**}$ is given as $B_{k}^{**}=\sum_{j=1}^{k}b_{j}^{**}$.

To evaluate the effectiveness of a scoring function, a number of IR
metrics have been proposed to emphasize the items that are ranked
at higher positions. In general, the IR metrics are computed
based on the list of ground-truth labels $\bar{\mathbf{y}}$ induced by $f$. For example, the binary-relevance IR metrics measure the performance of
a specific ranking model based on $\mathbf{b}^{**}$, such as precision and AP. The graded-relevance IR metrics measure the performance of a specific ranking model based on $\mathbf{y}^{**}$,
such as nDCG and ERR.

## Precision

Precision@k measures the proportion
of relevant documents retrieved at a given truncation position, which is defined as:
\begin{equation}
Pre@k=\frac{1}{k}\sum_{j=1}^{k}b_{j}^{**}
\end{equation}

where $k$ denotes the truncation position.

## Average Precision

Different from Precision@k that does not take into account the position
at which a document is ranked, Average Precision (AP) is a **rank-sensitive**
metric, which builds upon Precision as follows:
\begin{equation}
AP=\frac{1}{|Y^{+}|}\sum_{j}b_{j}^{**}\times Pre@j
\end{equation}

Then Mean Average Precision (MAP) is defined
as the mean of AP scores over a set of queries.

## Normalized Discounted Cumulative Gain (nDCG)

Normalized Discounted Cumulative Gain (nDCG) is a **graded-relevance
rank-sensitive** metric. The discounted cumulative
gain (DCG) of a ranked list is given as $DCG@k=\sum_{j=1}^{k}\frac{2^{y_{j}^{**}}-1}{\log_{2}(j+1)}$,
where $G_{j}=2^{y_{j}^{**}}-1$ is usually referred to as the gain
value of the $j$-th document.
We denote the maximum DCG value attained by the ideal ranking as $DCG^{*}$,
then normalizing DCG with $DCG^{*}$ gives nDCG as follows:

\begin{equation}
nDCG@k=\frac{DCG@k}{DCG^{*}@k}
\end{equation}

## Expected Reciprocal Rank (ERR)

Expected Reciprocal Rank (ERR) is another popular graded-relevance rank-sensitive metric. Let $Pr(j)$ be the relevance probability of the document at rank
$j$. In accordance with the gain function for nDCG, the relevance
probability is commonly calculated as $Pr(j)=\frac{2^{y_{j}^{**}}-1}{2^{\max(\mathbf{y}^{**})}}$.
ERR interprets the relevance probability as the probability that the
user is satisfied with the document at a rank position. Thus the probability
that the user is dissatisfied with the documents at ranks from $1$
to $k$ is given as $Disp(1,k)=\prod_{i=1}^{k}(1-Pr(i))$. ERR is
then defined as

\begin{equation}
ERR@k=\sum_{j=1}^{k}\frac{Disp(1,j-1)\cdot Pr(j)}{j}
\end{equation}

Let $ERR^{*}$ the maximum ERR value attained by the ideal ranking, we have the normalized ERR as follows:

\begin{equation}
nERR@k=\frac{ERR@k}{ERR^{*}@k}
\end{equation}


# Linear Models

There's a few linear models out there that we felt were generally useful. This
document will highlight some of them.

## LOWESS

Lowess stands for LOcally WEighted regreSSion and has historically
been used for smoothing but you can also use it
for machine learning where you're interested in interpolating. Here's
a demo;

```python
import numpy as np
import matplotlib.pylab as plt
from sklego.linear_model import LowessRegression

n = 100
xs = np.linspace(0, np.pi, n)
ys = 1 + np.sin(xs) + np.cos(xs**2) + np.random.normal(0, 0.1, n)

mod = LowessRegression(sigma=0.1, span=0.5).fit(xs.reshape(-1, 1), ys)

xs_new = np.linspace(-1, np.pi + 1, n*2)
preds = mod.predict(xs_new.reshape(-1, 1))

plt.figure(figsize=(12, 4))
plt.scatter(xs, ys)
plt.plot(xs_new, preds, color='orange')
plt.title("Be careful with extrapolation here.");
```

The line does not look linear but that's because internally, during prediction,
many weighted linear regressions are happening. The gif below demonstrates how
the data is being weighted when we would make a prediction.

### Details on `sigma`.

We'll also show two different prediction outcomes depending on the hyperparameter
`sigma`.

You may be tempted now to think that a lower sigma always has a better fit, but you
need to be careful here. The data might have gaps and larger sigma values will be
able to properly regularize.

Note that this regression also works in higher dimensions but the main downside of
this approach is that it is really slow when making predictions.

If you really want to get advanced there's also a hyperparameter `span` but you'll
really need to know what you're doing. It was added for completeness but the authors
of this package have yet to find a proper usecase for it.

### Details on `span`.

The `span` parameter can be used to force that you'll only include a certain
  percentage of your dataset. Technically without a `span` you are still using
  all the data in your dataset, albeit with small weights if they are far away.

The effect of the `span` parameter on the weights can be seen below.

This will also effect the predictions.

You may need to squint your eyes a bit to see it, but lower spans cause more
  jiggles and less smooth curves.

## ProbWeightRegression

> Note, this is a somewhat experimental feature. We found this feature to be plausibly useful but we've not seen it cause a big "win" yet.

Let's say that you're interested in combining a few models for a regression task.
You could put them together in an ensemble. Say we've got predictions $y_1, y_2, y_3$, each of which come from respectable models, then you may want to combine these together in another linear regression. This way the new, hopefully best, prediction $y_*$ is defined via;

$$y^* = w_1 y_1 + w_2 y_2 + w_3 y_3$$

This can be a valid way of reweighing. But there's one issue:
technically the weights $w_1, w_2, w_3$ can sum to a number that isn't one. Since that's
numerically totally possible we need to be aware that we can end up in a strange situation.

The `ProbWeightRegression` adresses this. It assumes that every input it receives is
the output of a model and it will ensure that they are reweighed with a constraint in mind. For this usecase, it would optimise;

\begin{equation} \label{eq1}
\begin{split}
\text{min} & \sum_i (y_i - (w_1 y_{1,i} + w_2 y_{2,i} + w_3 y_{3,i}))^2 \\
\text{subject to} & \\
& \text{ } w_1 + w_2 + w_3 = 1 \\
& \text{ } w_1 \geq 0, w_2 \geq 0, w_3 \geq 0
\end{split}
\end{equation}

The final positivity constraint is optional in our model.

Here's an example usage.

```python
from sklearn.datasets import make_regression
import pandas as pd

X, y = make_regression(n_samples=1000, n_features=10, random_state=42)
df = pd.DataFrame(X)
```

We've turned the array into a dataframe so that we can apply the `ColumnSelector`.

```python
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.linear_model import LinearRegression
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV

from sklego.meta import EstimatorTransformer
from sklego.linear_model import ProbWeightRegression
from sklego.preprocessing import ColumnSelector

pipe = Pipeline([
    ('models', FeatureUnion([
        ('path1', Pipeline([
            ('select1', ColumnSelector([0,1,2,3,4])),
            ('pca', PCA(n_components=3)),
            ('linear', EstimatorTransformer(LinearRegression()))
        ])),
        ('path2', Pipeline([
            ('select2', ColumnSelector([5,6,7,8,9])),
            ('pca', PCA(n_components=2)),
            ('linear', EstimatorTransformer(LinearRegression()))
        ]))
    ])),
    ('prob_weight', ProbWeightRegression())
])

grid = GridSearchCV(estimator=pipe, param_grid={}, cv=3).fit(df, y)
```

```python
from sklearn import set_config
set_config(display='diagram')
grid
```

<input class="sk-toggleable__control sk-hidden--visually" id="1f1cc5db-07d1-470d-aa5a-253d7d316602" type="checkbox" >GridSearchCV

GridSearchCV(cv=3,
             estimator=Pipeline(steps=[('models',
                                        FeatureUnion(transformer_list=[('path1',
                                                                        Pipeline(steps=[('select1',
                                                                                ColumnSelector(columns=[0,
                                                                                1,
                                                                                2,
                                                                                3,
                                                                                4])),
                                                                                ('pca',
                                                                                PCA(n_components=3)),
                                                                                ('linear',
                                                                                EstimatorTransformer(estimator=LinearRegression()))])),
                                                                       ('path2',
                                                                        Pipeline(steps=[('select2',
                                                                                ColumnSelector(columns=[5,
                                                                                6,
                                                                                7,
                                                                                8,
                                                                                9])),
                                                                                ('pca',
                                                                                PCA(n_components=2)),
                                                                                ('linear',
                                                                                EstimatorTransformer(estimator=LinearRegression()))]))])),
                                       ('prob_weight',
                                        ProbWeightRegression())]),
             param_grid={})

<input class="sk-toggleable__control sk-hidden--visually" id="722adf51-b4eb-4aea-acce-ff4fd0835955" type="checkbox" >models: FeatureUnion

FeatureUnion(transformer_list=[('path1',
                                Pipeline(steps=[('select1',
                                                 ColumnSelector(columns=[0, 1,
                                                                         2, 3,
                                                                         4])),
                                                ('pca', PCA(n_components=3)),
                                                ('linear',
                                                 EstimatorTransformer(estimator=LinearRegression()))])),
                               ('path2',
                                Pipeline(steps=[('select2',
                                                 ColumnSelector(columns=[5, 6,
                                                                         7, 8,
                                                                         9])),
                                                ('pca', PCA(n_components=2)),
                                                ('linear',
                                                 EstimatorTransformer(estimator=LinearRegression()))]))])

path1

<input class="sk-toggleable__control sk-hidden--visually" id="ee9ceae5-59b7-4c88-9fbf-9505349fa887" type="checkbox" >ColumnSelector

ColumnSelector(columns=[0, 1, 2, 3, 4])

<input class="sk-toggleable__control sk-hidden--visually" id="e0129048-6eb8-47bc-a8c0-ae9d96d63cd1" type="checkbox" >PCA

PCA(n_components=3)

<input class="sk-toggleable__control sk-hidden--visually" id="fe8293b5-d914-4a59-8c39-e2b4e18851c8" type="checkbox" >linear: EstimatorTransformer

EstimatorTransformer(estimator=LinearRegression())

<input class="sk-toggleable__control sk-hidden--visually" id="41c53c2a-d2c2-4241-b549-8f7653a0029f" type="checkbox" >LinearRegression

LinearRegression()

path2

<input class="sk-toggleable__control sk-hidden--visually" id="86b9c93e-12be-4fa6-865f-bb874ff99129" type="checkbox" >ColumnSelector

ColumnSelector(columns=[5, 6, 7, 8, 9])

<input class="sk-toggleable__control sk-hidden--visually" id="ec46eb32-4807-466f-a47b-dba193ff5315" type="checkbox" >PCA

PCA(n_components=2)

<input class="sk-toggleable__control sk-hidden--visually" id="5efe2360-3f3b-4c67-bc8f-e2619f06d229" type="checkbox" >linear: EstimatorTransformer

EstimatorTransformer(estimator=LinearRegression())

<input class="sk-toggleable__control sk-hidden--visually" id="6ceb127c-c870-4944-b6c3-5f47c0b5af1b" type="checkbox" >LinearRegression

LinearRegression()

<input class="sk-toggleable__control sk-hidden--visually" id="8d1aaabb-dda3-4fe0-99dc-c4b60597b332" type="checkbox" >ProbWeightRegression

ProbWeightRegression()

You can see that the `ProbWeightRegression` indeeds sums to one.

```python
grid.best_estimator_[1].coefs_
```

    array([0.03102466, 0.96897535])

## Least Absolute Deviation Regression

Imagine that you have a dataset with some outliers.

```python
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(0)
X = np.linspace(0, 1, 20)
y = 3*X + 1 + 0.5*np.random.randn(20)
X = X.reshape(-1, 1)

y[10] = 8
y[15] = 15

plt.figure(figsize=(16, 4))
plt.scatter(X, y)
```

A simple linear regression will not do a good job since it is distracted by the outliers. That is because it optimizes the mean squared error
$$ \sum_i \left(y_i-\textrm{model}(x_i)\right)^2 $$

which penalizes a few large errors more than many tiny errors. For example, if y-model(x) = 4 for some single observation, the MSE here is 16. If there are two observations with y_1 - model(x_1) = 2 and y_2 - model(x_2) = 2, the MSE is 8 in total, which is less than for one larger error. Note that the sum of the errors is the same in both cases.

Hence, linear regression does the following:

```python
from sklearn.linear_model import LinearRegression

x = np.array([0, 1]).reshape(-1, 1)
plt.figure(figsize=(16, 4))
plt.scatter(X, y)
plt.plot(x, LinearRegression().fit(X, y).predict(x), 'r')
```

By changing the loss function to the mean absolute deviation
$$ \sum_i \left|y_i-\textrm{model}(x_i)\right|\enspace, $$

we can let the model put the same focus on each error. This yields the least absolute deviation (LAD) regression that tries to agree with the majority of the points.

```python
from sklearn.linear_model import LinearRegression
from sklego.linear_model import LADRegression

x = np.array([0, 1]).reshape(-1, 1)
plt.figure(figsize=(16, 4))
plt.scatter(X, y)
plt.plot(x, LinearRegression().fit(X, y).predict(x), 'r')
plt.plot(x, LADRegression().fit(X, y).predict(x), 'g')
```

### See also
scikit-learn tackles this problem by offering a variety of [robust regressors](https://scikit-learn.org/stable/auto_examples/linear_model/plot_robust_fit.html). Many of them use an indirect approach to reduce the effect of outliers. [RANSAC](https://en.wikipedia.org/wiki/Random_sample_consensus), for example, samples random points from the dataset until it consists of all inliers.
The closest thing to LADRegression that scikit-learn offers is the [HuberRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.HuberRegressor.html) with a loss function that is partly a squared and partly an absolute error. However, it is more complicated and requires hyperparameter tuning to unleash its full potential.

```python

```


# More on Simulating Conditional Probabilities

Simulation is a key technique used in this book to explore random phenomena. Thus, it is important to understand the simulation techniques we use for different scenarios. Consider again the Magician's Coin problem, and let $H_i$ denote the event that the coin comes up heads on the $i$th flip. In {doc}`simulating-cond-probs`, we investigated $P(H_2|H_1)$, the probability of getting Heads on a second flip of the chosen coin given that heads was observed on the first flip.  If we apply the definition of conditional probability from {doc}`definition`, then we can calculate this probability as

<!--
\begin{equation}
P(H_2|H_1) = \frac{P(H_1 \cap H_2)}{P(H_1)}.
\label{eq:p-h2-given-h1}
\end{equation}
-->
$$
P(H_2|H_1) = \frac{P(H_1 \cap H_2)}{P(H_1)}.
$$ (eq:p-h2-given-h1)

However, in {doc}`simulating-cond-probs`, we did not estimate either $P(H_1 \cap H_2)$ or $P(H_1)$.  Instead, we counted the proportion of times that we observed the outcome $H_2$ when $H_1$ was the outcome of the first flip.  Let's see why these produce the same result.

Let's see how we would estimate the probabilities on the right-hand side of {eq}`eq:p-h2-given-h1`. We will use the following notation for the counters we have in the simulation:
* $N$ is the total number of simulation trials,
* $N_1$ is the number of simulation trials in which the outcome of flip 1 was Heads (i.e., the number of times $H_1$ occurred), and
* $N_{12}$ is the number of simulation trials in which the outcomes of both flip 1 and flip 2 were Heads (i.e., the number of times $H_1 \cap H_2$ was observed.

Now, we can estimate the probability of $H_1$ using its relative frequency,

$$
P(H_1) = \frac{N_1}{N}.
$$

Similarly, we can estimate the probability of $H_1 \cap H_2$ using its relative frequency,

$$
P(H_1 \cap H_2) =\frac{N_{12}}{N}.
$$

If we substitute these into {eq}`eq:p-h2-given-h1`, we get

$$
P(H_2|H_1) = \frac{ \frac{N_{12}}{N} }{ \frac{N_1}{N} }= \frac{ N_{12} }{ N_1 }.
$$

I.e., the result is the same as the approach taken in {doc}`simulating-cond-probs`. In general, we can directly estimate the conditional probability of an event $B$ given an event $A$ using nested if statements to:
* Count the number of trials in which $A$ occurred. Call this number $N_A$
* Within those trials in which  $A$ occurred, count the number of trials in which $B$ also occurred. Call this number $N_{AB}$.
* Estimate the conditional probability of $B$ occurring given that $A$ occurred as

$$
P(B|A) \approx \frac{N_{AB}}{N_A}
$$

```{warning}

Note that when designing simulation to estimate conditional probabilities, you must be especially careful in selecting the number of trials to be simulated.

In simulations of single events, if we think we know the approximate probability of the event, we can use that probability to estimate the number of trials needed to estimate it. For example, if we are estimating a probability $P(A)$ that we believe is close to 0.1, then we can estimate that the event $A$ occurs approximately once in every $1/P_A=10$ trials. If we capture 100 instances of $A$ occurring, we will get a good estimate of $P(A)$, so we might use this information to simulate $100/P_A=1000$ trials.

This approach will not work to estimate a conditional probability!

<span style="color:red;font-weight:bold">It is not sufficient to use only a conditional probability to estimate the number of trials needed.</span>

Let's use an example to show why this is the case. Suppose $A$ and  $B$ are events for which we wish to estimate the conditional probability $P(B|A)$. Consider the scenario where $P(A)=10^{-3}$ and $P(A \cap B= 9 \times 10^{-4}$. Then

$$
P(B|A) = \frac{P(A \cap B)}{P(A)} = \frac{9 \times 10^{-4}}{10^{-3}} = 0.9.
$$

If we used only the knowledge of $P(B|A)$, then we might use $100/P(B|A) \approx 111$ trials. But remember that in the simulation, we estimate the conditional probability as

$$
P(B|A) \approx \frac{N_{AB}}{N_A}.
$$
We can estimate the values of these counters as follows. (We will formalize the meaning of these estimates later in the book when we talk about expected values/ensemble means.) $N_A$ will be approximately equal to the number of trials times $P(A)$, so it will be approximately equal to $111 P(A)= 0.11$. In other words, in 111 trials, most of the time we will not see any event $A$ occur. The estimated value of counter $N_{AB}$ will be the estimated number of trials in which we see both $A$ and $B$ occur, which will be approximately equal to $111 P(A \cap B) \approx 0.1$. **If we only use  the conditional probability information, we may severely underestimate the number of trials required to observe a sufficient number of events.**

$\Rightarrow$ When determining a sufficient number of trials for a simulation of a conditional probability $P(B|A)$, we must determine the required number of trials from the (estimated) value of $P(A \cap B)$, if known. If no information about $P(A \cap B)$ is available, the value of $N_{AB}$ should be checked at the end of the simulation to make sure that a sufficient number of events (at least 10; 100 is better) were observed.

Because of this effect, some people will code their simulation to run until $N_{AB}$ reaches a specific threshold. This will bias the estimate of the $P(B|A)$ for reasons that are beyond the scope of this book. However, the bias will be small if threshold is large (>100).

```

```python

```


# [A rolling disc with Kane’s method ](https://docs.sympy.org/latest/modules/physics/mechanics/examples/rollingdisc_example_kane.html)

## 系統參數

### Function of time
* Angle $q_1(t),~q_2(t),~q_3(t)$
* $u_1(t),~u_2(t),~u_3(t)$

### Scalars
* radius: r
* mass: m
* gravity: g
* time: t

```python
from sympy import symbols, sin, cos, tan
from sympy.physics.mechanics import *
q1, q2, q3, u1, u2, u3  = dynamicsymbols('q1 q2 q3 u1 u2 u3')
q1d, q2d, q3d, u1d, u2d, u3d = dynamicsymbols('q1 q2 q3 u1 u2 u3', 1)
r, m, g, t = symbols('r m g t')
```

## 座標轉換

### Yaw-Row-Pitch rotation sequence(3-1-2)

```python
N = ReferenceFrame('N') # the inertia frame
Y = N.orientnew('Y', 'Axis', [q1, N.z]) # yaw angle q1
L = Y.orientnew('L', 'Axis', [q2, Y.x]) # roll angle q2
R = L.orientnew('R', 'Axis', [q3, L.y]) # pitch angle q3
```

#### ang_vel_in(otherframe):
Returns the angular velocity vector of the ReferenceFrame.

#### set_ang_vel(otherframe, value):
Define the angular velocity vector in a ReferenceFrame.

#### set_ang_acc(otherframe, value):
Define the angular acceleration Vector in a ReferenceFrame.

```python
w_R_N_qd = R.ang_vel_in(N) # angular velocity of frame R w.r.t inertia frame N
R.set_ang_vel(N, u1 * L.x + u2 * L.y + u3 * L.z) #
```

```python
w_R_N_qd
```

$\displaystyle \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\mathbf{\hat{l}_y} + \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\mathbf{\hat{y}_x} + \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\mathbf{\hat{n}_z}$


```
%%capture
!pip install qiskit
!pip install pylatexenc
!apt install texlive-latex-recommended
```

```
import qiskit as qk
from qiskit.visualization import plot_bloch_multivector, plot_histogram
from qiskit.quantum_info import random_statevector, Statevector
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np
%matplotlib inline
backend = qk.Aer.get_backend('qasm_simulator')

style = {}

# Uncomment this block if you are using Colab dark theme
# style = {'backgroundcolor': "#00000000", 'linecolor': "#ffffff", 'textcolor': "#ffffff"}
# mpl.rcParams['text.color'] = "#ffffff"
# mpl.rcParams['axes.labelcolor'] = "#ffffff"
# mpl.rcParams['xtick.color'] = "#ffffff"
# mpl.rcParams['ytick.color'] = "#ffffff"
```

#**Getting an IMBQ Token**

To get started, visit [IBM Quantum Experience](https://quantum-computing.ibm.com/) and select *Create an IBMid account* to create an account. Once you've followed the steps to register for an account, you should be able to log into your dashboard. First, go to *My Account* in the top right drop-down menu.

From here you can copy your API token, or generate it if you do not already have one.

# **The Qubit**

The qubit is the fundamental unit of quantum computation, a coherent superposition of two orthonormal states:

$$
\left|\psi\right> = a \left|0\right> + b \left|1\right>  \qquad a,b \in \mathbb{C}
$$

Common examples of physical qubits are the electron spin or photon polarization.

We operate on a classical bit by changing it's state from 0 to 1 or vice versa. A qubit is operated on by rotating it's Bloch vector, altering the linear superposition.

## **The Bloch Sphere**

We represent the state of a qubit as a vector on a unit two-sphere, known as the Bloch sphere. The positive and negative $z$-axes correspond to the two orthonormal basis states of the qubit.

To see what this looks like, let's set up a simulation of a qubit. First we'll create a register with a single qubit, and add it to our quantum circuit, before initializing the register in a random state.

```
qr = qk.QuantumRegister(1) # create quantum register with single qubit
qc = qk.QuantumCircuit(qr) # add the register to our circuit
qc.initialize(random_statevector(2).data, 0) # initialize register in random state

plot_bloch_multivector(Statevector.from_instruction(qc)) # plot the qubit's Bloch sphere
```

## **Operations as Rotations on the Bloch Sphere**
The polar angle is $\theta$ and the azimuthal angle $\phi$ such that

$$
\left|\psi\right> = \cos(\theta/2) \left|0\right> + e^{i\phi}\sin(\theta/2) \left|1\right>
$$

We represent operations on the qubit as rotations of the Bloch vector. The elemental rotations are the Pauli matrices, which correspond to $\pi$ rotations about the Cartesian $x$, $y$ and $z$ axes

\begin{gather}
X =
\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}
\qquad
Y =
\begin{bmatrix}
0 & -i \\
i & 0
\end{bmatrix}
\qquad
Z =
\begin{bmatrix}
1 & 0 \\
0 & -1
\end{bmatrix} \\[1em]
\left|0\right> =
\begin{bmatrix}
1 \\
0
\end{bmatrix}
\qquad
\left|1\right> =
\begin{bmatrix}
0 \\
1
\end{bmatrix}
\end{gather}

The $X$ matrix represents a bit flip operation, switching the $\left|0\right>$ and $\left|1\right>$ states

$$
X \big( a \left|0\right> + b \left|1\right> \big) = a \left|1\right> + b \left|0\right>
$$

The Z matrix represents a phase flip operation, changing the relative sign of the superposition

$$
Z \big( a \left|0\right> \pm b \left|1\right> \big) = a \left|0\right> \mp b \left|0\right>
$$

The Y matrix represents the combination of the two
$$
Y \big( a \left|0\right> \pm b \left|1\right> \big) = ai \left|1\right> \mp bi \left|0\right>
$$

Let's try some of these rotations. First we'll initialize a register with two qubits in the state $\left|\psi\right> = \left|0\right>$, and apply the Pauli $X$ matrix to the second qubit.

```
qr = qk.QuantumRegister(2) # quantum register with two qubits
qc = qk.QuantumCircuit(qr)

qc.x(1) # apply the Pauli X operator to the second qubit

plot_bloch_multivector(Statevector.from_instruction(qc))
```

The Pauli $X$ matrix performs a rotation of $\pi$ radians, swapping the state of the qubit.

$$
X\left|0\right> = \left|1\right>
$$

The Pauli matrices are Hermitian, so applying the $X$ matrix a second time rotates the Bloch vector back to it's original position

\begin{align}
XX\left|0\right> &= X\left|1\right> = \left|0\right> \\
&= XX^\dagger\left|0\right> = \left|0\right>
\end{align}

```
qc.x(1) # apply X to the second qubit again

plot_bloch_multivector(Statevector.from_instruction(qc))
```

## **PQ Coin Toss**

The USS Enterprise is facing an imminent catastrophe, and captain Picard must find a way to save his crew. Suddenly the omnipotent Q appears, and offers to rescue the ship if Picard can beat him at a simple game. The setup is as follows:

1.   Q places a quantum coin in a small box, heads up (Picard may check this if he likes).
2.   They then take turns, starting with Q, to perform a quantum operation on the coin: either leaving the coin in the state 'heads up', $\left|0\right>$, or flipping the coin to 'tails up', $\left|1\right>$. These operations correspond to applying the identity matrix, $I$, and the Pauli $X$ matrix, respectively.
3.   After Q's second turn, they open the box, and if the coin is still heads up, then Q wins. If it is tails up, Picard wins.

It seems Picard has a 50:50 chance of winning. After Q takes his first turn Picard can choose to leave the qubit in either the state $\left|0\right>$, or the state $\left|1\right>$, thereby randomising the result of Q's final flip. No?
\
\
<centre></centre>

## The Twist

Unfortunately, Q is a tricksy individual, and he knows of a third coin flip, called the Hadamard, which will guarantee that Q wins the game *every single time*

$$
H = \frac{1}{\sqrt{2}}
\begin{bmatrix}
1 & 1 \\
1 & -1
\end{bmatrix}
$$

Using this trick, Q can ensure the qubit is always in the state $\left|0\right>$ at the end of the game, no matter whether Picard flips the coin or not. Let's see how the Hadamard works.

We initialize a quantum register with a single qubit in the $\left|0\right>$ state. We also create a classical register with a single classical bit. This will be used to store the result of the final measurement of the qubit either $\left|0\right>$ or $\left|1\right>$.

```
qr = qk.QuantumRegister(1, "q")
cr = qk.ClassicalRegister(1, "c")
qc = qk.QuantumCircuit(qr, cr)

plot_bloch_multivector(Statevector.from_instruction(qc), title="Initial State")

```

Q then applies his Hadamard flip, which acts on the quantum coin to create an equal superposition of 'heads' and 'tails'.

$$
H\left|0\right>
= \frac{1}{\sqrt{2}}
\begin{bmatrix}
1 & 1 \\
1 & -1
\end{bmatrix}
\begin{bmatrix}
1 \\
0
\end{bmatrix}
= \frac{1}{\sqrt{2}} \big( \left|0\right> + \left|1\right>\big) = \left|+\right>
$$

```
qc.h(0) # Q applies the first Hadamard gate

plot_bloch_multivector(Statevector.from_instruction(qc), title="Q's first flip")
```

Picard can either leave the qubit in the superposition state $\left|+\right>$ (applying the identity, $\mathbb{I}$), or perform the Pauli $X$ operation to flip the coin. If he applies the flip:

$$
X\left|+\right> = \frac{1}{\sqrt{2}}
\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}
\begin{bmatrix}
1 \\
1
\end{bmatrix}
=
\frac{1}{\sqrt{2}}
\begin{bmatrix}
1 \\
1
\end{bmatrix}
= \left|+\right>
$$

```
qc.x(0) # Picard applies the Pauli X gate

plot_bloch_multivector(Statevector.from_instruction(qc), title="Picard's flip")
```

In the equal superposition, the bit flip operator leaves the qubit's state unchanged. Q then simply applies the Hadamard once more, to transform the qubit back to it's original state

$$
H\left|+\right> = \left|0\right>
$$

```
qc.h(0) # Q applies a second Hadamard gate

plot_bloch_multivector(Statevector.from_instruction(qc), title="Q's second flip")
qc.measure(qr, cr)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f21ac84ab38>

Now, when we perform a measurement on the qubit, we find it in the state $\left|0\right>$ with unit probability.

```
result = qk.execute(qc, backend, shots=1).result().get_counts()
print("The result of the measurement is: |{0}>".format([*result][0]))
```

    The result of the measurement is: |0>

The quantum circuit for this series of operations, with the qubit initialized in the state $\left|0\right>$

```
qc.draw(output="mpl", style=style, scale=2)
```

## **Measurement**

At the end of the PQ coin toss game we performed a measurement of the qubit's state. Specifically, we performed a projective measurement or projection-valued measurement (PVM), in the computational basis $\{\left|0\right>, \left|1\right>\}$. This measurement is described by the Pauli-$Z$ matrix we saw earlier

\begin{align}
Z \left|0\right> &= + 1 \left|0\right> \\
Z \left|1\right> &= -1 \left|1\right>
\end{align}

and is a projective measurement along the positive $z$-axis of the Bloch sphere. The Eigenvalues $\pm 1$ correspond to the possible results of the measurement.

### **Measurement Along Arbitrary Axes**

A unit axis $\hat{n}$, described by the angles $\theta$ and $\phi$ on the Bloch sphere, has the PVM

$$
\sigma_\hat{n} =
\begin{bmatrix}
\cos\theta & e^{-i\phi}\sin\theta \\
e^{i\phi}\sin\theta & -\cos\theta
\end{bmatrix}
$$

which has an Eigenstate corresponding to the $+1$ Eigenvalue given by

$$
\left|0\right>_\hat{n} =
\begin{bmatrix}
e^{-i\frac{\phi}{2}}\cos\frac{\theta}{2} \\
e^{i\frac{\phi}{2}}\sin\frac{\theta}{2}
\end{bmatrix}
$$

### **Arbitrary Measurements in the Computational Basis**

For example, we could perform a projective measurement onto the equal superposition states, which form an orthonormal basis on the Bloch sphere

\begin{align}
\left|+\right> &= \frac{1}{\sqrt{2}} \big( \left|0\right> + \left|1\right> \big) \\
\left|-\right> &= \frac{1}{\sqrt{2}} \big( \left|0\right> - \left|1\right> \big)
\end{align}

which correspond to the Bloch vectors $\hat{n}_{+} = (\pi/2, 0)$ and $\hat{n}_{-} = (\pi/2, \pi)$, respectively. The measurement operator that projects the state of a qubit onto this basis is the Pauli $X$ matrix.

However, quantum computers performs every read-out measurement in the computational basis, $\{\left|0\right>, \left|1\right>\}$. To perform a PVM in the polar basis, we must construct the Pauli $X$ gate in terms of the measurement operator for the computational basis (the Pauli $Z$ gate)

$$
\sigma_{\left|+\right>} = X = HZH
= \frac{1}{2}
\begin{bmatrix}
1 & 1 \\
1 & -1
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\
0 & -1
\end{bmatrix}
\begin{bmatrix}
1 & 1 \\
1 & -1
\end{bmatrix}
= \begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}
$$

we do this using the Hadamard, which transforms between the computational and polar bases.

Let's see what this measurement circuit looks like...

```
def meas_polar(qubit, bit, circuit):
  '''Measure in the polar basis'''
  circuit.h(qubit)
  circuit.measure(qubit, bit)
  circuit.h(qubit)
  return circuit

qr = qk.QuantumRegister(1, "q")
cr = qk.ClassicalRegister(1, "c")
qc = qk.QuantumCircuit(qr, cr)

qc = meas_polar(qr[0], cr[0], qc)

qc.draw(output='mpl', style=style, scale=2)
```

```
qr = qk.QuantumRegister(1, "q")
cr = qk.ClassicalRegister(1, "c")
qc = qk.QuantumCircuit(qr, cr)

qc.h(0)
qc.barrier()
qc = meas_polar(qr[0], cr[0], qc)

qc.draw(output='mpl', style=style, scale=2)
```

```
job = qk.execute(qc, backend, shots=1000) # execute the circuit 1000 times
result = job.result().get_counts()
print(result)
```

    {'0': 1000}

We project the qubit onto the computational basis state, $\left|0\right>_\text{c}$, with unit probability. Thus, if Picard had suspected Q of foul play, he could have performed a measurement in the polar basis, obtaining the result $\left|0\right>_c$, and exposing Q's trickery.

What if we measure the state of the qubit in the computational basis before our measurement in the polar basis?

```
qr = qk.QuantumRegister(1, "q")
cr1 = qk.ClassicalRegister(1, "c1") # one register for each measurement
cr2 = qk.ClassicalRegister(1, "c2")
qc = qk.QuantumCircuit(qr, cr1, cr2)

qc.h(0)
qc.measure(qr[0], cr1[0])
qc.barrier()
qc = meas_polar(qr[0], cr2[0], qc)

job = qk.execute(qc, backend, shots=1000) # execute the circuit 1000 times
result = job.result().get_counts()
print(result)

qc.draw(output='mpl', style=style, scale=2)
```

When we perform a measurement of the superposition state $\left|+\right>_c$ in the computational basis we collapse the superposition into either the $\left|0\right>_c$ or $\left|1\right>_c$ state, each with probability $1/2$. In the polar basis

\begin{align}
\left|0\right>_c &= \left|+\right>_p \\
\left|1\right>_c &= \left|-\right>_p
\end{align}

So that when we measure in the computational basis, we obtain either $\left|0\right>_p$ or $\left|1\right>_p$ each with probability $1/2$.

This is the Stern-Gerlach experiment in action!

# **Combining Qubits**

## **Quantum Circuits**

Now we have a general idea of how to apply single qubit operations. We have just seen how the Hadamard gate can be used to prepare an equal superposition of states, for which a measurement obtains the results $\left|0\right>$ and $\left|1\right>$ with equal probability. But to do anything very interesting we have to network some qubits together.

To start let's initialize a quantum register with 4 qubits, and a classical register with 4 classical bits. The classical register will be used to store the results of a measurement on each of the qubits.

```
qr = qk.QuantumRegister(4, "q") # quantum register with 8 qubits
cr = qk.ClassicalRegister(4, "c") # 8 classical bits to store measurement results
qc = qk.QuantumCircuit(qr, cr)
```

Now let's try adding a Hadamard gate to each channel in the circuit in order to prepare each of the qubits in an equal superposition state

$$
\left|+\right> = \frac{1}{\sqrt{2}} \big( \left|0\right> + \left|1\right> \big)
$$

Each Hadamard gate can be followed with a measurement of the qubit's state. We perform a projective measurement onto the computational basis $\{\left|0\right> \left|1\right>\}$ which will collapse the superposition. The result of the measurement is then stored in a classical bit from the classical register. We'll also draw the circuit diagram for this quantum circuit. Can you tell what this circuit will do?

```
for i in range(4):
  qc.h(qr[i]) # add a Hadamard gate to each qubit channel
  qc.measure(qr[i], cr[i]) # measure qubit i and store result in classical bit

qc.draw(output="mpl", style=style, scale=2)
```

So far, we have been using the `qasm_simulator` backend, which simulates the behaviour of a quantum computer on a classical computer. For the circuit above to be a true random number generator (as opposed to a pseudo-random number generator) we have to perform the computation on an actual quantum computer. So now we're going to use the IBM API to send our quantum circuit to a real quantum computer.

To connect to the IBM Athens quantum computer, **replace the empty string below with your own API key**. If you don't have one then leave the string blank and the backend will revert to `qasm_simulator`, or see [getting an IBMQ token](#IBMQ_token) for instructions.

```
my_token = '' # enter your API token here, e.g my_token = '123456789'

if my_token:
  qk.IBMQ.save_account(my_token)
  acc = qk.IBMQ.load_account()
  quantum_backend = acc.get_backend('ibmq_athens') # Athens quantum computer
else:
  quantum_backend = backend
```

    configrc.store_credentials:WARNING:2020-10-28 14:55:25,682: Credentials already present. Set overwrite=True to overwrite.
    /usr/local/lib/python3.6/dist-packages/qiskit/providers/ibmq/ibmqfactory.py:192: UserWarning: Timestamps in IBMQ backend properties, jobs, and job results are all now in local time instead of UTC.
      warnings.warn('Timestamps in IBMQ backend properties, jobs, and job results '

Now we'll set up a job to send to the quantum computer. The number of `shots` determines how many times our circuit is run. Since we only want to generate a single random number, we run the circuit one time.

The results are returned as a dictionary of states for the classical register as keys, and the number of times that result is obtained as the value. In this case we only perform one run of the circuit, so we expect to obtain a single key corresponding to the state of the classical register after the measurements are performed.

```
job = qk.execute(qc, quantum_backend, shots=1) # execute the circuit 1 time
results = job.result().get_counts()
print("The results of a single shot are: {0}".format(results))
```

Finally, we convert the measurement result stored in the 4-bit classical register to an integer to obtain a random number in the range

$$0 \leq x \leq (2^4 - 1)$$

```
bit_string = [*results][0] # extract the bit string as first key in the dictionary
rand_num = int(bit_string, 2) # convert bit string to integer
print("The random number is: {0}*(2^3) + {1}*(2^2) + {2}*(2^1) + {3}*(2^0) = {4} ".format(*[i for i in bit_string], rand_num))
```

    The random number is: 1*(2^3) + 1*(2^2) + 1*(2^1) + 0*(2^0) = 14

Alter the number of shots below to see how the resultant histogram of measurement results changes as the number of repeat measurements, $N$, increases. For a random number generator, we should approach a uniform distribution as $N \rightarrow \infty$.

```
n_shots = 1e6 # number of times to run the circuit

job = qk.execute(qc, quantum_backend, shots=n_shots)
results = job.result().get_counts()

print("The counts for {0} shots are: {1}\n".format(int(n_shots), results))
plot_histogram(results, title="Histogram of {0} shots".format(int(n_shots)))
```

## **The CNOT Gate**

Controlled operations form the basis of both quantum and classical computers. They are multi-bit operations in which one or more 'control' bits inform the computer what operation to perform on one or more 'target' bits. They are logical if/else statements of the form

$$
\text{if A is true, then do B}
$$

The fundamental controlled operation is the *controlled not*, or CNOT. The CNOT gate has two inputs: the *control qubit* and the *target qubit*. It acts on the target qubit according to the state of the control qubit.

$$
\left|c\right> \left|t\right> \longrightarrow \left|t \oplus c\right>
$$

i.e if the control bit is $\left|1\right>$ then the target bit is flipped, if the control bit is $\left|0\right>$ we apply the identity. We represent the two qubits with the tensor product of their respective spaces

$$
\left|00\right> =
\begin{bmatrix}
1 \\
0 \\
0 \\
0 \\
\end{bmatrix}
\qquad
\left|10\right> =
\begin{bmatrix}
0 \\
1 \\
0 \\
0 \\
\end{bmatrix}
\qquad
\left|01\right> =
\begin{bmatrix}
0 \\
0 \\
1 \\
0 \\
\end{bmatrix}
\qquad
\left|11\right> =
\begin{bmatrix}
0 \\
0 \\
0 \\
1 \\
\end{bmatrix}
$$

The CNOT gate is written

$$
\text{CNOT} =
\begin{bmatrix}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 \\
0 & 0 & 1 & 0
\end{bmatrix} \\[1em]
$$

Lets initialise a quantum circuit with a two qubit register in the state $\left|00\right>$, we'll add a CNOT gate with the 0th qubit as the control and the 1st qubit as the target. We'll also add a classical two bit register to store the outcome of a measurement of both the qubits.

```
qr = qk.QuantumRegister(2, "q")
cr = qk.ClassicalRegister(2, "c")
qc = qk.QuantumCircuit(qr, cr)
qc.cx(qr[0], qr[1]) # CNOT gate 0th qubit is control, 1st qubit is target
qc.measure(qr, cr)

qc.draw(output="mpl", style=style, scale=2)
```

```
job = qk.execute(qc, backend, shots=1024) # we perform 1024 independent measurements of the cicuit
result = job.result()

print(result.get_counts())
```

    {'00': 1024}

The control bit enters the CNOT gate in the $\left|0\right>$ state. This tells the gate to apply no operation to the target bit. The state of the register is unchanged, and so we expect to measure the $\left|00\right>$ state after the CNOT gate with unit probability. All 1024 measurements of the quantum register after applying the CNOT gate gave the same result

$$
\text{CNOT} \left|00\right> = \left|00\right>
$$

Now let's try again, but this time we'll set the control bit to the state $\left|1\right>$ using the Pauli $X$ gate before it enters the CNOT gate.

```
# Create a quantum circuit with one quantum register and one classical register
qr = qk.QuantumRegister(2, "q")
cr = qk.ClassicalRegister(2, "c")
qc = qk.QuantumCircuit(qr, cr)

qc.x(qr[0]) # flip the control qubit |0> -> |1>
qc.cx(qr[0], qr[1]) # CNOT gate 0th qubit is control, 1st qubit is target
qc.measure(qr, cr) # measure the result of the computation

job = qk.execute(qc, backend, shots=1024)
result = job.result()
qc.draw(output="mpl", style=style, scale=2)
```

```
print("The measurement results are: ", result.get_counts())
```

    The measurement results are:  {'11': 1024}

The control bit entered the CNOT gate in the $\left|1\right>$ state, this tells the gate to flip the state of the target bit, from $\left|0\right>$ to $\left|1\right>$. When we project the state of the register onto the two-qubit basis, we measure the state $\left|11\right>$ with unit probability. Again, all 1024 measurements give

$$
\text{CNOT}\left|10\right> = \left|11\right>
$$

What happens if we prepare the control qubit in a superposition state? For example, by applying the Hadamard to the control bit initially

$$
\left|q_0\right> = H \left|0\right> = \frac{1}{\sqrt{2}} \big( \left|0\right> + \left|1\right> \big)
$$

So that the register is in the state

$$
\left|q_0 q_1 \right> = \frac{1}{\sqrt{2}} \big(\left|00\right> + \left|10\right>\big)
$$

```
qr = qk.QuantumRegister(2, "q")
cr = qk.ClassicalRegister(2, "c")
qc = qk.QuantumCircuit(qr, cr)

qc.h(qr[0]) # apply the Hadamard to the control bit
qc.cx(qr[0], qr[1])
qc.measure(qr, cr)

job = qk.execute(qc, backend, shots=1024)
result = job.result()
qc.draw(output="mpl", style=style, scale=2)
```

```
print("The measurement results are: ", result.get_counts())
```

    The measurement results are:  {'00': 513, '11': 511}

We obtain the results $\left|00\right>$ and $\left|11\right>$ each with $P \approx 0.5$.

$$
\text{CNOT} \frac{\big( \left|00\right> + \left|10\right> \big)}{\sqrt{2}} = \frac{\big( \left|00\right> + \left|11\right> \big)}{\sqrt{2}}
$$

## **Bell States and Entanglement**

We have created a very specific state here, known as a Bell state. In particular this state is one of the Bell basis states, which span the four-dimensional Hilbert space of the two-qubit register.

\begin{align}
\left|\Phi^+\right> &= \frac{1}{\sqrt{2}} \big(\left|00\right> + \left|11\right>\big) \\
\left|\Phi^-\right> &= \frac{1}{\sqrt{2}} \big(\left|00\right> - \left|11\right>\big) \\
\left|\Psi^+\right> &= \frac{1}{\sqrt{2}} \big(\left|01\right> + \left|10\right>\big) \\
\left|\Psi^-\right> &= \frac{1}{\sqrt{2}} \big(\left|01\right> - \left|10\right>\big)
\end{align}

The state we have measured above is the $\Phi^+$ Bell basis state. These basis states have the special property of being *maximally entangled* states. Entangled states have no classical analogue, and their interesting properties form the basis of quantum communication.

Let's prepare a register in the $\left|\Phi^+\right>$ state again. Except this time, after we've prepared the Bell state we'll only measure the target qubit. This is known as a local measurement, in which we project the state of the target qubit $\left|q_1\right>$ onto the single-qubit basis $\{\left|0\right>, \left|1\right>\}$.

```
qr = qk.QuantumRegister(2, "q") # quantum register has two qubits
cr = qk.ClassicalRegister(1, "c") # single bit classical register
qc = qk.QuantumCircuit(qr, cr)

qc.h(qr[0])
qc.cx(qr[0], qr[1])
qc.measure(qr[1], cr) # measure q1 and store result in c register

job = qk.execute(qc, backend, shots=1000)
result = job.result()
qc.draw(output="mpl", style=style, scale=2)
```

```
print("The measurement results are: ", result.get_counts())
```

    The measurement results are:  {'0': 473, '1': 527}

Perhaps this is not too suprising, after all, the CNOT gate appears to have prepared $q_1$ in a superposition state with equal probability of being $\left|0\right>$ or $\left|1\right>$:

$$
\text{CNOT} \frac{\big( \left|00\right> + \left|10\right> \big)}{\sqrt{2}} = \frac{\big( \left|00\right> + \left|11\right> \big)}{\sqrt{2}} = \left|\Phi^+\right>
$$

In the [Measurement](#measurement) section we saw how the single qubit superposition state $\left|+\right> = \frac{1}{\sqrt{2}}\big( \left|0\right> + \left|1\right> \big)$ can be measured in the polar basis $\left\{\left|0\right>_p, \left|1\right>_p\right\}$, with unit probability of obtaining the result $\left|0\right>_p$.

What happens if we perform the same measurement on $q_1$ in the circuit above?

```
qr = qk.QuantumRegister(2, "q") # quantum register has two qubits
cr = qk.ClassicalRegister(1, "c") # single bit classical register
qc = qk.QuantumCircuit(qr, cr)

qc.h(qr[0])
qc.cx(qr[0], qr[1])
qc = meas_polar(qr[1], cr, qc) # measure q1 and store result in c register

job = qk.execute(qc, backend, shots=1000)
result = job.result().get_counts()
print("The measurement results are: ", result)
qc.draw(output="mpl", style=style, scale=2)
```

Instead of measuring $\left|0\right>$ every time, we obtain the results $\left|0\right>_p = \left|+\right>_c$ and $\left|1\right>_p = \left|-\right>_c$ with equal probability. In the two qubit state

$$
\left|\Phi^-\right> = \frac{1}{\sqrt{2}} \big( \left|00\right> + \left|11\right> \big)
$$

A measurement of *either* qubit along *any* axis gives a completely random result. In other words, there is no single qubit basis in which it is possible to determine with certainty, the state of the qubit. Let's the check the limit as the number of shots increases

$$\lim_{N \rightarrow \infty} P = ?$$

```
probs = []
n_shots = [10**(n/4) for n in range(4, 25)]
for shots in n_shots:
  job = qk.execute(qc, backend, shots=shots)
  counts = list(job.result().get_counts().values())
  probs.append(0.5 + abs(0.5 - counts[0]/shots)) # select larger of the two probabilities
plt.plot(n_shots, probs, ls='', marker='x')
plt.xscale('log')
plt.ylabel("counts/shots")
plt.xlabel("shots")

```

Mathematically, measuring a single qubit in the entangled two-qubit register is equivalent to taking the partial trace over the control qubit of the quantum register's density matrix, $\rho_\text{AB}$. For the $\Phi^+$ Bell state this is

\begin{align}
\rho_{q_1q_2} &= \left|\Phi^-\right> \left<\Phi^-\right|
= \frac{1}{2} \big( \left|00\right>\left< 00\right| +  \left|00\right>\left< 11\right| +  \left|11\right>\left< 00\right| +  \left|11\right>\left< 11\right| \big) \\
&= \frac{1}{2}
\begin{bmatrix}
1 & 0 & 0 & 1 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
1 & 0 & 0 & 1
\end{bmatrix}
\end{align}

Such that the density matrix of the target qubit is a multiple of the identity matrix:

\begin{align}
\rho_{q_1} &= \text{Tr}_{q_0} \big(\left|\Phi^-\right> \left<\Phi^-\right| \big) = \sum_{i, j = 0, 1} \left \left<\Phi^-\right| \left|j\right>_{q_0} = \frac{1}{2}  \big( \left| 0\right>\left< 0\right| +  \left|1\right>\left< 1\right| \big) \\[.5em]
&= \frac{1}{2}
\begin{bmatrix}
1 \\
0
\end{bmatrix}
\begin{bmatrix}
1 & 0
\end{bmatrix}
+ \frac{1}{2}
\begin{bmatrix}
0 \\
1
\end{bmatrix}
\begin{bmatrix}
0 & 1
\end{bmatrix}
= \frac{1}{2}
\begin{bmatrix}
1 & 0 \\
0 & 0
\end{bmatrix}
+ \frac{1}{2}
\begin{bmatrix}
0 & 0 \\
0 & 1
\end{bmatrix} \\
&= \frac{1}{2} \mathbb{1}
\end{align}

Physically, this means that if we measure the state of $q_1$ along *any* axis, $\hat{n}(\theta, \phi)$, the result is completely random - i.e we obtain the results $\left|0\right>\left< 0\right|$ and $\left|1\right>\left< 1\right|$ each with probability 0.5

\begin{align}
P\big(q_1 = \hat{n}\big) &= \left|\left< \hat{n} | q_1 \right>\right|^2 = \left< \hat{n} | q_1 \right>\left< q_1 | \hat{n} \right> = \left< \hat{n} | \rho_{q_1} | \hat{n} \right> \\[1em]
&= \frac{1}{2}
\begin{bmatrix}
e^{-i\frac{\theta}{2}}\cos\frac{\theta}{2} &
e^{i\frac{\theta}{2}}\sin\frac{\theta}{2}
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\[1em]
0 & 1
\end{bmatrix}
\begin{bmatrix}
e^{i\frac{\theta}{2}}\cos\frac{\theta}{2} \\
e^{-i\frac{\theta}{2}}\sin\frac{\theta}{2}
\end{bmatrix} = \frac{1}{2} \left( \cos^2 \frac{\theta}{2} + \sin^2 \frac{\theta}{2} \right) \\
&= \frac{1}{2}
\end{align}

```

```


# Chapter 2 - Linear Algebra

## 2.1 Scalars, Vectors, Matrices and Tensors

### Q1 [10 Points, M]
Denote the set of all n-dimensional binary vectors with Cartesian product notation

Solution:

{0, 1}$^{n}$

### Q2 [10 Points, S]
Given the vector
$
\boldsymbol{x}=\left[\begin{array}{c}
x_{1} \\
x_{2} \\
x_{3} \\
x_{4} \\
\end{array}\right]
$,
and the set
$
S = \{2, 4\}
$,
obtain the vectors $\boldsymbol{x}_{S}$ and $\boldsymbol{x}_{-S}$

Solution:

$x_{S} = \left[\begin{array}{c}
x_{2} \\
x_{4} \\
\end{array}\right]$

$x_{-S} = \left[\begin{array}{c}
x_{1} \\
x_{3} \\
\end{array}\right]$

### Q3 [20 Points, S]
Evaluate the following expressions with broadcasting rules,

$$
\left[\begin{array}{lll}
0 & 1 & 2
\end{array}\right]+[5]=
$$

$$
\left[\begin{array}{lll}
1 & 1 & 1 \\
1 & 1 & 1 \\
1 & 1 & 1
\end{array}\right]+\left[\begin{array}{lll}
0 & 1 & 1
\end{array}\right]=
$$

$$
\left[\begin{array}{l}
0 \\
1 \\
2
\end{array}\right]+\left[\begin{array}{ll}
0 & 1 & 2
\end{array}\right]=
$$

```python
#@title
# Adapted from astroML: see http://www.astroml.org/book_figures/appendix/fig_broadcast_visual.html
import numpy as np
from matplotlib import pyplot as plt

#------------------------------------------------------------
# Draw a figure and axis with no boundary
fig = plt.figure(figsize=(6, 4.5), facecolor='w')
ax = plt.axes([0, 0, 1, 1], xticks=[], yticks=[], frameon=False)

def draw_cube(ax, xy, size, depth=0.4,
              edges=None, label=None, label_kwargs=None, **kwargs):
    """draw and label a cube.  edges is a list of numbers between
    1 and 12, specifying which of the 12 cube edges to draw"""
    if edges is None:
        edges = range(1, 13)

    x, y = xy

    if 1 in edges:
        ax.plot([x, x + size],
                [y + size, y + size], **kwargs)
    if 2 in edges:
        ax.plot([x + size, x + size],
                [y, y + size], **kwargs)
    if 3 in edges:
        ax.plot([x, x + size],
                [y, y], **kwargs)
    if 4 in edges:
        ax.plot([x, x],
                [y, y + size], **kwargs)

    if 5 in edges:
        ax.plot([x, x + depth],
                [y + size, y + depth + size], **kwargs)
    if 6 in edges:
        ax.plot([x + size, x + size + depth],
                [y + size, y + depth + size], **kwargs)
    if 7 in edges:
        ax.plot([x + size, x + size + depth],
                [y, y + depth], **kwargs)
    if 8 in edges:
        ax.plot([x, x + depth],
                [y, y + depth], **kwargs)

    if 9 in edges:
        ax.plot([x + depth, x + depth + size],
                [y + depth + size, y + depth + size], **kwargs)
    if 10 in edges:
        ax.plot([x + depth + size, x + depth + size],
                [y + depth, y + depth + size], **kwargs)
    if 11 in edges:
        ax.plot([x + depth, x + depth + size],
                [y + depth, y + depth], **kwargs)
    if 12 in edges:
        ax.plot([x + depth, x + depth],
                [y + depth, y + depth + size], **kwargs)

    if label:
        if label_kwargs is None:
            label_kwargs = {}
        ax.text(x + 0.5 * size, y + 0.5 * size, label,
                ha='center', va='center', **label_kwargs)

solid = dict(c='black', ls='-', lw=1,
             label_kwargs=dict(color='k'))
dotted = dict(c='black', ls='-', lw=0.5, alpha=0.5,
              label_kwargs=dict(color='gray'))
depth = 0.3

#------------------------------------------------------------
# Draw top operation: vector plus scalar
draw_cube(ax, (1, 10), 1, depth, [1, 2, 3, 4, 5, 6, 9], '0', **solid)
draw_cube(ax, (2, 10), 1, depth, [1, 2, 3, 6, 9], '1', **solid)
draw_cube(ax, (3, 10), 1, depth, [1, 2, 3, 6, 7, 9, 10], '2', **solid)

draw_cube(ax, (6, 10), 1, depth, [1, 2, 3, 4, 5, 6, 7, 9, 10], '5', **solid)
draw_cube(ax, (7, 10), 1, depth, [1, 2, 3, 6, 7, 9, 10, 11], '5', **dotted)
draw_cube(ax, (8, 10), 1, depth, [1, 2, 3, 6, 7, 9, 10, 11], '5', **dotted)

draw_cube(ax, (12, 10), 1, depth, [1, 2, 3, 4, 5, 6, 9], '5', **solid)
draw_cube(ax, (13, 10), 1, depth, [1, 2, 3, 6, 9], '6', **solid)
draw_cube(ax, (14, 10), 1, depth, [1, 2, 3, 6, 7, 9, 10], '7', **solid)

ax.text(5, 10.5, '+', size=12, ha='center', va='center')
ax.text(10.5, 10.5, '=', size=12, ha='center', va='center')
ax.text(1, 11.5, r'${\tt np.arange(3) + 5}$',
        size=12, ha='left', va='bottom')

#------------------------------------------------------------
# Draw middle operation: matrix plus vector

# first block
draw_cube(ax, (1, 7.5), 1, depth, [1, 2, 3, 4, 5, 6, 9], '1', **solid)
draw_cube(ax, (2, 7.5), 1, depth, [1, 2, 3, 6, 9], '1', **solid)
draw_cube(ax, (3, 7.5), 1, depth, [1, 2, 3, 6, 7, 9, 10], '1', **solid)

draw_cube(ax, (1, 6.5), 1, depth, [2, 3, 4], '1', **solid)
draw_cube(ax, (2, 6.5), 1, depth, [2, 3], '1', **solid)
draw_cube(ax, (3, 6.5), 1, depth, [2, 3, 7, 10], '1', **solid)

draw_cube(ax, (1, 5.5), 1, depth, [2, 3, 4], '1', **solid)
draw_cube(ax, (2, 5.5), 1, depth, [2, 3], '1', **solid)
draw_cube(ax, (3, 5.5), 1, depth, [2, 3, 7, 10], '1', **solid)

# second block
draw_cube(ax, (6, 7.5), 1, depth, [1, 2, 3, 4, 5, 6, 9], '0', **solid)
draw_cube(ax, (7, 7.5), 1, depth, [1, 2, 3, 6, 9], '1', **solid)
draw_cube(ax, (8, 7.5), 1, depth, [1, 2, 3, 6, 7, 9, 10], '2', **solid)

draw_cube(ax, (6, 6.5), 1, depth, range(2, 13), '0', **dotted)
draw_cube(ax, (7, 6.5), 1, depth, [2, 3, 6, 7, 9, 10, 11], '1', **dotted)
draw_cube(ax, (8, 6.5), 1, depth, [2, 3, 6, 7, 9, 10, 11], '2', **dotted)

draw_cube(ax, (6, 5.5), 1, depth, [2, 3, 4, 7, 8, 10, 11, 12], '0', **dotted)
draw_cube(ax, (7, 5.5), 1, depth, [2, 3, 7, 10, 11], '1', **dotted)
draw_cube(ax, (8, 5.5), 1, depth, [2, 3, 7, 10, 11], '2', **dotted)

# third block
draw_cube(ax, (12, 7.5), 1, depth, [1, 2, 3, 4, 5, 6, 9], '1', **solid)
draw_cube(ax, (13, 7.5), 1, depth, [1, 2, 3, 6, 9], '2', **solid)
draw_cube(ax, (14, 7.5), 1, depth, [1, 2, 3, 6, 7, 9, 10], '3', **solid)

draw_cube(ax, (12, 6.5), 1, depth, [2, 3, 4], '1', **solid)
draw_cube(ax, (13, 6.5), 1, depth, [2, 3], '2', **solid)
draw_cube(ax, (14, 6.5), 1, depth, [2, 3, 7, 10], '3', **solid)

draw_cube(ax, (12, 5.5), 1, depth, [2, 3, 4], '1', **solid)
draw_cube(ax, (13, 5.5), 1, depth, [2, 3], '2', **solid)
draw_cube(ax, (14, 5.5), 1, depth, [2, 3, 7, 10], '3', **solid)

ax.text(5, 7.0, '+', size=12, ha='center', va='center')
ax.text(10.5, 7.0, '=', size=12, ha='center', va='center')
ax.text(1, 9.0, r'${\tt np.ones((3,\, 3)) + np.arange(3)}$',
        size=12, ha='left', va='bottom')

#------------------------------------------------------------
# Draw bottom operation: vector plus vector, double broadcast

# first block
draw_cube(ax, (1, 3), 1, depth, [1, 2, 3, 4, 5, 6, 7, 9, 10], '0', **solid)
draw_cube(ax, (1, 2), 1, depth, [2, 3, 4, 7, 10], '1', **solid)
draw_cube(ax, (1, 1), 1, depth, [2, 3, 4, 7, 10], '2', **solid)

draw_cube(ax, (2, 3), 1, depth, [1, 2, 3, 6, 7, 9, 10, 11], '0', **dotted)
draw_cube(ax, (2, 2), 1, depth, [2, 3, 7, 10, 11], '1', **dotted)
draw_cube(ax, (2, 1), 1, depth, [2, 3, 7, 10, 11], '2', **dotted)

draw_cube(ax, (3, 3), 1, depth, [1, 2, 3, 6, 7, 9, 10, 11], '0', **dotted)
draw_cube(ax, (3, 2), 1, depth, [2, 3, 7, 10, 11], '1', **dotted)
draw_cube(ax, (3, 1), 1, depth, [2, 3, 7, 10, 11], '2', **dotted)

# second block
draw_cube(ax, (6, 3), 1, depth, [1, 2, 3, 4, 5, 6, 9], '0', **solid)
draw_cube(ax, (7, 3), 1, depth, [1, 2, 3, 6, 9], '1', **solid)
draw_cube(ax, (8, 3), 1, depth, [1, 2, 3, 6, 7, 9, 10], '2', **solid)

draw_cube(ax, (6, 2), 1, depth, range(2, 13), '0', **dotted)
draw_cube(ax, (7, 2), 1, depth, [2, 3, 6, 7, 9, 10, 11], '1', **dotted)
draw_cube(ax, (8, 2), 1, depth, [2, 3, 6, 7, 9, 10, 11], '2', **dotted)

draw_cube(ax, (6, 1), 1, depth, [2, 3, 4, 7, 8, 10, 11, 12], '0', **dotted)
draw_cube(ax, (7, 1), 1, depth, [2, 3, 7, 10, 11], '1', **dotted)
draw_cube(ax, (8, 1), 1, depth, [2, 3, 7, 10, 11], '2', **dotted)

# third block
draw_cube(ax, (12, 3), 1, depth, [1, 2, 3, 4, 5, 6, 9], '0', **solid)
draw_cube(ax, (13, 3), 1, depth, [1, 2, 3, 6, 9], '1', **solid)
draw_cube(ax, (14, 3), 1, depth, [1, 2, 3, 6, 7, 9, 10], '2', **solid)

draw_cube(ax, (12, 2), 1, depth, [2, 3, 4], '1', **solid)
draw_cube(ax, (13, 2), 1, depth, [2, 3], '2', **solid)
draw_cube(ax, (14, 2), 1, depth, [2, 3, 7, 10], '3', **solid)

draw_cube(ax, (12, 1), 1, depth, [2, 3, 4], '2', **solid)
draw_cube(ax, (13, 1), 1, depth, [2, 3], '3', **solid)
draw_cube(ax, (14, 1), 1, depth, [2, 3, 7, 10], '4', **solid)

ax.text(5, 2.5, '+', size=12, ha='center', va='center')
ax.text(10.5, 2.5, '=', size=12, ha='center', va='center')
ax.text(1, 4.5, r'${\tt np.arange(3).reshape((3,\, 1)) + np.arange(3)}$',
        ha='left', size=12, va='bottom')

ax.set_xlim(0, 16)
ax.set_ylim(0.5, 12.5)
print()
```

## 2.2-3 Multiplying Matrices and Vectors and Identity and Inverse Matrices

### Q4 [20 Points, H]
Let $A$ be a $2 \times 2$ matrix, if $A B=B A$ for every $B$ of the size $2 \times 2$, Prove that:
$$
A=\left[\begin{array}{ll}
a & 0 \\
0 & a
\end{array}\right], \
a \in \mathbb{R}
$$

$$
\begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix}
\begin{pmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{pmatrix}=
\begin{pmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{pmatrix}
\begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix} \Rightarrow \\
\begin{cases}\require{cancel}\cancel{a_{11}b_{11}}+a_{12}b_{21}=\cancel{b_{11}a_{11}}+b_{12}a_{21}\\
a_{11}b_{12}+a_{12}b_{22}=b_{11}a_{12}+b_{12}a_{22}\\
a_{21}b_{11}+a_{22}b_{21}=b_{21}a_{11}+b_{22}a_{21}\\
a_{21}b_{12}+\cancel{a_{22}b_{22}}=b_{21}a_{12}+\cancel{b_{22}a_{22}}\end{cases}
$$

Since $b_{12}$ and $b_{21}$ can be any number, in particular, $b_{12}=0$ and $b_{21} \neq 0,$ we get:
$a_{12}=0$
Similarly, for $b_{12} \neq 0$ and $b_{21}=0,$ we get $a_{21}=0$
From $(2),$ since $a_{12}=0$ and $b_{12}$ is an arbitrary number, we get $a_{11} b_{12}=b_{12} a_{22} \Rightarrow a_{11}=a_{22}$

## 2.4 Linear Dependence and Span

### Q5 [10 Points, H]
Prove that if a linear system of equations have two solutions, then it has infinitely many solutions.

Solution:

Assume that we have $ Ax = b$

If both $x$ and $y$ are the solutions, then $z=\alpha \boldsymbol{x}+(1-\alpha) \boldsymbol{y}$ is a solution for any real $\alpha$. Since it holds for any real $\alpha$, there are infinite number of solution possible. So, it is not possible to have more than one but less than infinetly many solutions for a particular b.

### Q6 [5 Points, M]
Given $A x=0$, where $A \in \mathbb{R}^{m \times n}$ is any matrix, and $x \in \mathbb{R}^{n}$ is a vector of unknown variables to be solved, what is the condition such that there is infinitely many solutions?

Solution:

Since it is a homogeneous system, the condition where there is infinitely many solutions is $m<n$ (i.e., where there are less equations than unknown).

## 2.5 Norms

### Q7 [15 Points, M]
Prove that **Max Norm** follows these conditions,
$$
\begin{align}
f(\boldsymbol{x})=0 \Rightarrow \boldsymbol{x}=\mathbf{0} \\
f(\boldsymbol{x}+\boldsymbol{y}) \leq f(\boldsymbol{x})+f(\boldsymbol{y}) \\
\forall \alpha \in \mathbb{R}, f(\alpha \boldsymbol{x})=|\alpha| f(\boldsymbol{x})
\end{align}
$$

Solution:

 * $f(\boldsymbol{x})=0 \Rightarrow \boldsymbol{x}=\mathbf{0}$ : By definition of max norm, the absolute value of the element is simplified with the largest magnitude in the vector (i.e., $\|x\|_{\infty}=\max _{i}\left|x_{i}\right|$). Therefore, if $f(\boldsymbol{x})=0$, $x$ should be also zero.

    We can also prove it by thinking the case where $x\neq0$ (i.e., at least one element of the vector $x$ is not equal to zero). Assume that $k$th element of x is non-zero. Then, with the max norm definition above, the max norm of any $i$th element would be greater or equal (if $k$th element is the largest magnitude) than $k$th element $\|x\|_{\infty}=\max _{i}\left|x_{i}\right| \geq |x_k|$. Because of the nature of absolute value, these all would be greater than 0.

* $f(\boldsymbol{x}+\boldsymbol{y}) \leq f(\boldsymbol{x})+f(\boldsymbol{y})$ : This is the triangle inequality. We can prove that the max norm holds triangle inequality by applying the max norm to the left hand side.

  $\begin{aligned}\|x+y\|_{\infty} &=\max _{i}\left|x_{i}+y_{i}\right| \\ & \leq \max _{i}\left(\left|x_{i}\right|+\left|y_{i}\right|\right) \\ &=\max _{i}\left|x_{i}\right|+\max _{j}\left|y_{j}\right|=\|x\|_{\infty}+\|y\|_{\infty} \end{aligned}$

* $\forall \alpha \in \mathbb{R}, f(\alpha \boldsymbol{x})=|\alpha| f(\boldsymbol{x})$

$\|\alpha x\|_{\infty}=\max _{i}\left|\alpha x_{i}\right| = \max _{i}\left|\alpha|\left|x_{i}|=| \alpha\left|\max _{i}\right| x_{i}|=| \alpha \mid\right\| x \|_{\infty}\right.$

## 2.6 Special Kinds of Matrices and Vectors

### Q8 [10 Points, M]
Solve the following system of equations,

$$
\frac{1}{2}\left[\begin{array}{cccc}
1 & 1 & 1 & 1 \\
1 & 1 & -1 & -1 \\
1 & -1 & 1 & -1 \\
1 & -1 & -1 & 1
\end{array}\right] \left[\begin{array}{c}
x_{1} \\
x_{2} \\
x_{3} \\
x_{4} \\
\end{array}\right] = \left[\begin{array}{c}
1 \\
2 \\
3 \\
4 \\
\end{array}\right]
$$

Solution:

This is a symmetric  and orthonormal matrix. As a result, $ A = A^{T} $

$$
\left[\begin{array}{c}
x_{1} \\
x_{2} \\
x_{3} \\
x_{4} \\
\end{array}\right] =
\frac{1}{2}\left[\begin{array}{cccc}
1 & 1 & 1 & 1 \\
1 & 1 & -1 & -1 \\
1 & -1 & 1 & -1 \\
1 & -1 & -1 & 1
\end{array}\right] \left[\begin{array}{c}
1 \\
2 \\
3 \\
4 \\
\end{array}\right] = \left[\begin{array}{c}
5 \\
-2 \\
-1 \\
0 \\
\end{array}\right]
$$


 INFERENTIAL STATISTICS

It help us to get the inferences about the population from picking out the samples from the population.

It's helps to estimate thr natural parameters of a probability distribution followed by the population. Estimate can be good or bad both types.

The propoerties of the good estimator is:

 It should be unbiased.

 It should have minimum variance

 Frequency Distribution  of any sample statistics plotted by doing repeated sampling is called sampling distribution of that sample statistics.

 Central Limit Theorem

Central Limit Theorem states that the frequency distribution of a sample statistics approaches pure normal distribution as the N (sample size) approaches to the infinity as well as the variance of the sampling distribution  of a sample statistics is inversely proportional to the N (sample size).

Variance of a sampling distibution of sample statistics is inversely proportional to sample size (N).

\begin{equation}
Variance\ of\ a\ sampling\ distribution\\ of sample\ statistics\ (\sigma^2) = \frac{\sigma^2}{N}
\end{equation}

\begin{equation}
Standard\ deviation\ of\ a\ sampling\ distribution\\ of sample\ statistics\ (\sigma) = \frac{\sigma}{\sqrt(N)}
\end{equation}

 Why we make sampling distibution?

To get the:

Best Estimator = Minimum Variance Unbiased (MVU) Estimator

 Python code to prove the central limit theorem:::

Let's check it using by taking an example of the coin tossing.

```python
import numpy as np
import pandas as pd
```

```python
N = 10
psuccess = 0.5
qfailure = 1-psuccess
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    6

```python
N = 100
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    49

It will stablize the value of the sample size will increase. As we know that the probability of the getting the head/tail in coin tossing is 50 % i.e., 0.5.

```python
N = 10000000
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    4999066

```python

```


#  Number representation in computers

In this chapter we are going to give elements of understanding of how numbers are represented in today's computers. We won't go too much into the details (which are perhaps not so relevant here), however it is extremely important for scientists to understand the pitfalls of the so-called "floating-point arithmetics". At the end of this lecture, you should understand what these pitfalls are and when they occur, since sooner or later one of these pitfalls will hit you.

Table of Contents

11  Number representation in computers

11.1  Binary representation of integers

11.1.1  Basics

11.1.2  Overflow

11.1.3  Extending the binary representation to non-integers: fixed-point notation

11.2  Floating point numbers

11.2.1  Introduction

11.2.2  Scientific (exponential) notation in base b

11.2.3  Floating point representation (simplified)

11.2.4  IEEE 754

11.2.5  Consequences

11.2.6  Overflow

11.3  What to do from here?

11.3.1  Alternative data types

11.3.2  Symbolic computations

11.3.3  Deal with it

11.3.3.1  Awareness

11.3.3.2  Error propagation

11.3.3.3  Safer tests

11.4  Take home points

11.5  Further reading

11.6  What's next?

## Binary representation of integers

Traditional arithmetics and mathematics rely on the **decimal** system. Every number can be decomposed in a sum of products such as: $908.2 = 9\times10^2 + 0\times10^1 + 8\times10^{0} + 2\times10^{-1}$. This system is called decimal because the factors can take the values $[0 - 9]$ while the **fixed base** (the number which is scaled by an exponent) is 10.

Nearly all modern processors however represent numbers in binary form, with each digit being represented by a two-valued physical quantity such as a "low" or "high" voltage: 0s and 1s. These binary digits are called **bits**, and are the basis of the **binary representation of numbers**.

### Basics

With only 0s and 1s available the most efficient way to represent numbers is as a sum of 2s to the power of i, with i going from 0 to N-1. It is best shown with an example, here with N=4 digits:

    0101

To convert our number to a decimal representation we compute $0\times2^3 + 1\times2^2 + 0\times2^1 + 1\times2^0 = 5$. In this convention, the leftmost digit is called the **most significant bit** and the rightmost the **least significant bit**. If all the elements from the left are zeros, the convention is to omit them when writing. This is for example what the built-in [bin](https://docs.python.org/3/library/functions.html#bin) function chooses to do:

```python
bin(5)
```

    '0b101'

which converts an integer number to a binary string prefixed with `0b`. Now it appears quite clear that the number of different integers that a computer can represent with this system depends on the size N of the binary sequence. A *positive* integer represented with a **byte** (8 bits are called a **byte**) can thus be as large as:

```python
sum([2**i for i in range(8)])  # do you understand what this line does?
```

    255

but not larger. Unless specified otherwise, the first bit is often used to give the *sign* of the integer it represents. Therefore, the actual range of numbers that a byte can represent is $[-2^7; 2^7-1]= [-128; 127]$ (the reason for this asymmetry is a matter of definition, as we will see later). If you are sure that you only want to do arithmetics with positive number you can spare this one bit and specify your binary number as being of **unsigned integer type**.

So how many bits does our computer use to represent integers? Well, it depends on the platform and programming language you are using. Many older languages (including old python versions) made the distinction between short (32 bits) and long (64 bits) integers.

 **Exercise**: what is the largest number that a 64-bit integer can represent? The smallest? And a 32-bit unsigned integer?

Now, what is the default length of binary integers in python 3? Let's try to find out:

```python
from sys import getsizeof
a = 10
getsizeof(a)
```

    28

So 28 bytes? That's a lot of bits for the number 10. This is because the [getsizeof](https://docs.python.org/3/library/sys.html#sys.getsizeof) function returns the **memory consumption** of the *object* `int(10)`. What does that mean? Well, in python numbers are not only numbers, they are also "things" (**objects**). And these things come with services, like for example the ``bit_length`` method:

```python
a.bit_length()
```

    4

 **Exercise**: what does `bit_length()` do? What is the bit length of 5? of 127?

These services have a memory cost (an "overhead"), and are required no matter how big our number is:

```python
size_int = getsizeof(int())
size_int  # size of an empty integer object
```

    24

```python
def get_int_bitsize(integer):
    """Returns the actual memory consumption of an integer (in bits) without the object overhead."""
    return (getsizeof(integer) - getsizeof(int())) * 8
```

```python
get_int_bitsize(2)
```

    32

Ha! This looks more reasonable. So python uses 32 bits to store integers? But then, the largest number it can manipulate must be very small? Let's see if we can create a number larger than `2**32-1`:

```python
12**68
```

    24228399786958997352779519807679893609907752382877715390869269071681028096

```python
get_int_bitsize(2**18), get_int_bitsize(2**68), get_int_bitsize(2**100000)
```

    (32, 96, 106688)

As shown above, it turns out that modern python versions have **no limitations** on the size of integers (other than the total memory available on your computer). The memory slot used to store your number simply depends on how large it is.

```python
get_int_bitsize(2**100000) / 8 / 1024
```

    13.0234375

So the ``2**100000`` number requires about 13 KB (**Kilobytes**) of memory.

 **Exercise**: print ``2**100000`` on screen, just for "fun".

### Overflow

This dynamic resizing of integers in python means that they cannot **overflow**. **Overflow** is a common pitfall of many numerical operations though, and to illustrate what it is we can either use floats (unlike python integers, python floats can overflow), or numpy, which uses integer types of fixed length:

```python
import numpy as np
a = np.int8(127)
a.dtype
```

    dtype('int8')

```python
a
```

    127

```python
a + np.int8(1)
```

    /home/mowglie/.pyvirtualenvs/py3/lib/python3.5/site-packages/ipykernel_launcher.py:1: RuntimeWarning: overflow encountered in byte_scalars
      """Entry point for launching an IPython kernel.

    -128

What happened here? To understand this we need to understand how binary numbers are added together first. Please read the [addition chapter](https://en.wikipedia.org/wiki/Binary_number#Addition) of the Wikipedia article on binary numbers before going on.

Basically, we added 1 (binary `00000001`) to 127 (binary `01111111`) which gives us the binary number `10000000`, i.e. -128 in [two's complement](https://en.wikipedia.org/wiki/Two%27s_complement) representation, which is the most common method of representing signed integers on computers. At least, python warned us that we are doing something wrong here. But be aware that this is not always the case:

```python
np.int32(2147483648)
```

    -2147483648

```python
a = np.array([18])
a**a
```

    array([-497033925936021504])

These are examples of **silent** overflows. Silent means that they do not warn you about the probable mistake and could happen in your code without you noticing that this happens.

### Extending the binary representation to non-integers: fixed-point notation

A more general definition of the binary representation for integers is to use negative exponents as well. With negative exponents, any rational number a can be approximated by:

$$a = \pm \sum_{i=-j}^{k} z_i b^i$$ with $b > 1$ (base) and $0 \le z_i \le b-1$, $j$ and $k$ all positive integers. The precision depends on the size of $j$ while the maximum range depends on the size of $k$.

In the notation, a fixed point separates digits of positive powers of the base, from those of negative powers. In **base 2** (binary), the number $10000.1_2$ is equal to $16.5_{10}$ in base 10.

Indeed:

$$1\times2^4 + 0\times2^3 + 0\times2^2 + 0\times2^1 + 0\times2^0 + 1\times2^{-1} = 16.5_{10}$$

Although this representation is convenient, the representable value range is heavily limited by the number of bits available to represent the number (see this week's assignments). Therefore, most computers today are relying on the **floating point notation** to represent real numbers.

## Floating point numbers

### Introduction

Because computers have a *finite* number of storage units available, they can only represent a *finite* number of distinguishable values. In fact, a memory slot with $N$ available bits cannot represent more than $2^N$ distinguishable values. The range of real (or complex) numbers is of course infinite, and therefore it becomes clear that in the computer representation of numbers **there will always be a trade-off between the range of numbers one would like to represent and their relative accuracy** (i.e. the absolute difference between two consecutive representable numbers).

Taking the **decimal representation** of the number 1/3 as an example: it can be written as ``0.33``, ``0.333``, ``0.3333``, etc. Depending on the numbers of digits available, the precision of the number will increase but never reach the exact value, at least in the decimal representation.

This fundamental limitation is the explanation for unexpected results of certain arithmetic operations. For example:

```python
0.1 + 0.1  # so far so good
```

    0.2

```python
0.1 + 0.2  # wtf?
```

    0.30000000000000004

This is a typical **rounding error**, happening because most computers do not represent numbers as decimal fractions, but as binary. Without going too much into details (which can be tricky), this chapter will give you some elements of understanding in order to prepare you for the most common pitfalls of floating point arithmetics.

### Scientific (exponential) notation in base b

In the exponential notation (used by [floating point](https://en.wikipedia.org/wiki/Floating-point_arithmetic) numbers), a number is approximated with a fixed number of significant digits (the significand) and scaled using an exponent in some fixed base; the base for the scaling is normally two, ten, or sixteen. A number that can be represented exactly is of the following form:

$\mathrm{number} = \mathrm{significand} \times \mathrm{base} ^{\mathrm{exponent}}$

For example, in base 10:

$$1.234 = 1234 \times 10^{-3}$$

The number ``1.234`` can easily be represented exactly in base 10, but the number 1/3 cannot. However, in base 3 (which is just used here as an example) 1/3 can be represented exactly by $1 \times 3^{-1}$.

To approximate a number in base 10, the rule of "the more bits you have, the more precise you are" continues to hold true: $33 \times 10^{-2}$ and $33333333333333 \times 10^{-14}$ are two ways to approximate the number, the later being more expensive in terms of storage requirements but more accurate.

The exponential notation is the most common way to represent real numbers in computers and is the basis of the floating-point number representation.

### Floating point representation (simplified)

A floating point number in any base will store three numbers:
- the sign (one bit)
- the significand ($N_s$ bits)
- the exponent ($N_e$ bits)

Computers usually work in base 2, but other bases have been experimented with as well e.g. 16 (hexadecimal). $N_s$ and $N_e$ are usually fixed beforehand. Now remember:
- the significand determines the **precision** of the representation (significant digits)
- the exponent determines the **magnitude** of the represented number

Let's make an example to illustrate this concept. We will work in base 10 for simplicity, and assume that we have 8 "memory slots" (the equivalent of bits) available, each memory slot capable of storing a number from 0 to 9 or the sign (+ or -). We attribute $N_s=6$ slots to the significand (including its sign) and $N_e=2$ slots to the exponent (including its sign).

Now, what is the smallest positive number we can represent? And the biggest? Let's try it:

- smallest: $+00001 \times 10^{-9}$
- second smallest: $+00002 \times 10^{-9}$
- biggest: $+99999 \times 10^{+9}$
- second biggest: $+99998 \times 10^{+9}$

From these examples it becomes apparent that the precision (the distance between two consecutive numbers) is better for small numbers than for large numbers. Although our example is simplified, the principle is exactly the same for "real" floating point numbers in most computers, which follow the IEEE754 convention.

### IEEE 754

From the example above, we can see that with a fixed number of memory slots, we have a trade-off between the maximum precision of a number and its size.

This precision/size trade-off raises many challenges: memory and electronics must be optimized so that computations are fast while still allowing programs to realize computations on a *wide* range of numbers in the same program. For example, atmospheric models realize computations on specific humidity values ($\mathrm{kg}\,\mathrm{kg}^{-1}$) with typical values of 10$^{-5}$ and on the geopotential with values up to several order of magnitude larger. Using different standards for each variable would be impracticable. This lead to the development of the [IEEE Standard for Floating-Point Arithmetic (IEEE 754)](https://en.wikipedia.org/wiki/IEEE_754).

The standard defines five basic formats that are named for their numeric base and the number of bits used in their encoding, as listed in [this table](https://en.wikipedia.org/wiki/IEEE_754#Basic_and_interchange_formats). For example, ``binary64`` is the famous "double precision" format, called ``float64`` in numpy and simply ``float`` in the python standard library. In this format, $N_s=53$ bits are used for the significand, and the remaining $N_e=11$ for the exponent.

It is possible to compute the approximate precision of IEEE754 floating point numbers according to their value (see also the exercises):

Source: [wikipedia](https://en.wikipedia.org/wiki/IEEE_754)

### Consequences

With the floating point format, **small numbers have a larger absolute precision than large numbers**. See this example:

```python
.99 == .98  # so far so good
```

    False

```python
999999999999999.99 == 999999999999999.98  # wtf?
```

    True

```python
999999999999999.99  # wtf?
```

    1000000000000000.0

A further direct consequence is that **when summing two numbers, precision is lost to match the size of the outcome**.

### Overflow

Like numpy integers, floating point numbers can overflow:

```python
np.float16(65510) + np.float16(20)
```

    /home/mowglie/.pyvirtualenvs/py3/lib/python3.5/site-packages/ipykernel_launcher.py:1: RuntimeWarning: overflow encountered in half_scalars
      """Entry point for launching an IPython kernel.

    inf

Fortunately, the IEEE 754 standard defines two new numbers (-inf and +inf) which are more informative (and less dangerous) than the negative reset of integer numbers. IEEE 754 also defines "Not A Number" (abbreviated NaN), which propagates through computations:

```python
np.NaN * 10
```

    nan

## What to do from here?

As we've learned, errors in floating-point numbers are unavoidable. Even if these errors are very small, simple calculations on approximated numbers can contain pitfalls that increase the error in the result way beyond just having the individual errors "add up".

### Alternative data types

In certain cases where perfect decimal accuracy is needed (for example when dealing with currencies and money), it is possible to use a decimal floating point representation instead of the default binary one:

```python
1/10*3  # not precise
```

    0.30000000000000004

```python
from decimal import Decimal
Decimal(1) / Decimal(10) * 3  # precise
```

    Decimal('0.3')

With limited-precision decimals there are no unexpected rounding errors. In practice, however, alternative datatypes are used rarely because the precision gain comes with a performance overhead: computers work best with 0s and 1s, not with numbers humans can read.

### Symbolic computations

Symbolic computations are realized *literally* (like in mathematics), not approximately. [SymPy](http://www.sympy.org) is a popular python library for symbolic mathematics:

```python
import sympy
a = sympy.sympify('1 / 3')
a + a
```

    2/3

Seams like the perfect solution, right? It probably is if you are a mathematician, but for actual numerical computations SymPy will be way too slow to use. Symbolic mathematics can only be used for problems where analytical solutions are known. Unfortunately, this is not always the case (take numerical models of the atmosphere for example).

### Deal with it

There are no simple answers to numerical rounding errors. Be aware that they occur and try to mitigate their effect.

#### Awareness

*Awareness* is mostly hindered by the string representation of floating point numbers. In practice:

```python
0.1
```

    0.1

```python
format(0.1, '.16g')  # give 16 significant digits
```

    '0.1'

```python
format(0.1, '.30g')  # give 30 significant digits
```

    '0.100000000000000005551115123126'

The default `0.1` print is therefore a "lie", but it is a useful one: in most cases you don't want to know about these insignificant digits at the end. The [numpy.finfo](https://docs.scipy.org/doc/numpy/reference/generated/numpy.finfo.html) is a useful function informing you about the machine limits for floating point types:

```python
info = np.finfo(np.float)
info.bits, info.precision, info.max
```

    (64, 15, 1.7976931348623157e+308)

#### Error propagation

Preventing rounding errors to happen is not possible, but there are a few general rules:
- Multiplication and division are "safer" operations
- Addition and subtraction are dangerous, because when numbers of different magnitudes are involved, digits of the smaller-magnitude number are lost.
- This loss of digits can be inevitable and benign (when the lost digits are also insignificant for the final result) or catastrophic (when the loss is magnified and distorts the result strongly).
- The more calculations are done (especially when they form an iterative algorithm) the more important it is to consider this kind of problem.
- A method of calculation can be stable (meaning that it tends to reduce rounding errors) or unstable (meaning that rounding errors are magnified). Very often, there are both stable and unstable solutions for a problem.

(list taken from [the floating point guide](http://floating-point-gui.de/errors/propagation/))

As illustration for the difference between addition and multiplication, see the following example:

```python
a = 10 * 0.1
b = 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1
a == b
```

    False

```python
a, b
```

    (1.0, 0.9999999999999999)

Realize **safer computations** therefore involves asking yourself at which stage the most precision is going to be lost: this is most often the case when adding numbers of very different magnitudes. When building numerical models, this should always be something you consider: is a formula leading to dangerous additions, then reformulate it and/or use other units for your variables (e.g. $\mathrm{g}\,\mathrm{kg}^{-1}$ instead of $\mathrm{kg}\,\mathrm{kg}^{-1}$ for specific humidity). Consider the following example:

```python
a = 1
b = 1e-16
c = 10
```

```python
c1 = c * (a + b)
c2 = c * a + c * b
```

```python
c1 == c2
```

    False

``c * (a + b)`` and ``c * a + c * b`` are mathematically equivalent, and the first is computationally less expensive (two operations instead of three). However, the second is less prone to rounding errors!

#### Safer tests

Fortunately, rounding errors often remain unnoticed, meaning that your computations are probably OK! In our field in particular, we often do not care if the post-processed temperature forecast is numerically precise at 0.001° since the forecast accuracy is much lower anyway. However, this can still lead to surprises when comparing arrays for equality (e.g. when testing that a temperature is equal to zero, or for matching coordinates like longitude or latitude). In these cases, prefer to use numpy's specialized functions:

```python
np.isclose(c1, c2)  # OK is you don't care about small numerical errors
```

    True

## Take home points

- computers can only represent a finite number of distinguishable values.
- the range of representable numbers depends on the size of memory allocated to store it. There is practically no limit to the size of integers in python, but there is a limit for floats. Numpy implements several types of variables named after their size in bits (e.g. ``np.float32``, ``np.float64``, ``np.float128``).
- there are many different ways to represent numbers on computers, all with different strengths and weaknesses. The vast majority of systems use the [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) standard for floating points, which is a good compromise between range and accuracy. Most systems are binary (base 2) per default, but there are other bases available: base 10 (decimal) and base 16 (hexadecimal) are frequent as well.
- rounding errors happen because of these limitations. They always happen, even for the simplest arithmetics, and you **shall not ignore them**.
- however, there are ways to mitigate the impact of these rounding errors. This includes the use of more precise datatypes (e.g. float64 instead of float32), alternative representations (decimal instead of binary), and the use of more conservative operations (e.g. multiplication before addition when dealing with numbers of different magnitude)
- floating point errors have dramatic consequences in chaotic systems. A scary example is given in [this paper](https://journals.ametsoc.org/doi/pdf/10.1175/MWR-D-12-00352.1) about the influence of floating point computations on numerical weather forecasts.

## Further reading

Because of the importance of floating point computations you will find many resources online. I **highly recommend** to go through the short [floating point guide](http://floating-point-gui.de/) website, which explains the problem to non specialists. It will give you another point of view on the topic.

Other resources:
- [Python's documentation](https://docs.python.org/3/tutorial/floatingpoint.html) on floating point arithmetic
- [What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html): kind of heavy, but a reference text

## What's next?

Back to the [table of contents](00-Introduction.ipynb#ctoc), or [jump to this week's assignment](12-Assignment-04.ipynb).