The Two Envelope Paradox - Simulated
====
####Introduction
Bayesian statistics can most naively be described as the _art of thinking conditionally_. Conditional probability leads often to unexpected outcomes and violates our basic intuition in most surprising ways. One of the most profound violations the famous [two envelope paradox](http://en.wikipedia.org/wiki/Two_envelopes_problem).

What's the paradox?
------
The paradox can be stated in a very simple way

> You are given two identical envelopes $a$ and $b$ containing an unknown amount of money $X$. You are being told that one envelope contains twice the amount of money than the other, but you don't know which one. After choosing one envelope you are allowed to open it and decide if you want to switch.
* Strategy 1: The situation seems absolutely symmetric, independent of which envelope you choose. As we don't know any details about the distribution we might argue that it doesn't matter if we switch - hence
\begin{align}
E( Y ) = E(X)
\end{align}

>However, after a while you start to doubt yourself, and you start thinking along a different line.
* Strategy 2: Say you choose envelope $a$ and observe an amount $X$. Then you know that with probablity $p=0.5$ the amount in envelope $b$ is $Y=2X$ or with $p=0.5$ it is $Y=X/2$. A quick calculation using conditional probability then reveals
\begin{align}
E( Y ) = \frac{1}{2} E(Y | Y = 2X ) + \frac{1}{2} E(Y | Y = X /2 )
\end{align}
and one might quickly come to the conclusion that
\begin{align}
 \frac{1}{2} E( 2X ) + \frac{1}{2} E( X /2 )= \frac{5}{4} E(X) > E(X)
\end{align}
and you should switch the envelope as you have a statistical 25% gain from switching to envelope $b$.

Which strategy is right? It seems that the second strategy can be argued as easily with reverse roles, which would then lead to a constant switching back and forth between the envelopes.

To answer this question let's simulate the problem. First of all we need to devise a strategy after which we decide when to switch. This can be done quite simply by chosing an arbitrary threshold $T$. In the event $X<T$ we switch envelopes, otherwise we stay with the original choice

Numerical solution of the problem
------

####Python imports of the important packages

```python
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pymc as pm
from numpy.random import choice

%matplotlib inline
matplotlib.style.use('ggplot')
matplotlib.rc_params_from_file("../styles/matplotlibrc" ).update()
```

####Helper methods

Switching the envelopes is easy

```python
def switch_envelope(chosen_envelope):
    if chosen_envelope == 'a':
        return 'b'
    else:
        return 'a'
```

We also need some helper methods to evaluate the success of our strategy

```python
'''Find out if the final envelope _actually_ contains the highest value'''
def isLargerAmount(chosen_envelope, envelope_contents):
    inverted_contents = create_inverted_envelope(envelope_contents)
    if chosen_envelope == inverted_contents.get(np.max(inverted_contents.keys())):
        return 1 # success
    else:
        return 0 # failure

'''We need an inverse lookup table to associate the highest value with an envelope'''
def create_inverted_envelope(envelope_contents):
    dct = {}
    for key in envelope_contents.keys():
        dct[envelope_contents.get(key)] = key
    return dct
```

####Implementing the actual strategy

The strategy contains of four steps:

* create two envelopes containing iid amount of money
* choose one of them at random
* check for the money and switch if it $X<T$ where $T$ is draw from a different distribution
* evaluate if the strategy was successful

```python
def singleExperiment(money_distribution, threshold_distribution):

    # create two identical envelopes with a random amount of money
    envelope_contents = {'a': money_distribution.random().item(),
                         'b': money_distribution.random().item()}

    #choose an envelope
    chosen_envelope = choice(['a','b'])

    #check for the amount and switch if necessary
    if (threshold_distribution.random().item() >= envelope_contents[chosen_envelope]):
        chosen_envelope = switch_envelope(chosen_envelope)

    #evaluate whether strategy was a success
    return isLargerAmount(chosen_envelope, envelope_contents)
```

####Performing the experiment

To keep it simple we draw the monetary values from a uniform distribution, whereas our random threshold is exponentially distributed.

```python
money = pm.DiscreteUniform('money', 100, 100000)
threshold = pm.Exponential("threshold", 0.00005)
```

Let us run the experiment $n_{trials}$ times. Since we return values $1$ and $0$ for success and failure, respectively, we can find a good approximation of the success probability by computing the mean.

```python
def run_n_experiments(n_trials, money, threshold):
    lst = []
    for idx in range(n_trials):
        lst.append(singleExperiment(money, threshold))
    return np.mean(lst)
```

```python
print 'The success probability is approximately p = %0.3f' % run_n_experiments(100, money, threshold)
```

    The success probability is approximately p = 0.680

It seems strategy 2 from above is correct! The probability is significantly higher than $0.5$ and switching seems like a good choice. How is that possible?

Resolution of the paradox
------

Looking back at strategy 2, we silently dropped the conditions when evaluating the expectation values. In fact if we let $I$ be an indicator random variable of the event $Y=2X$, then we have
\begin{align}
E( Y | Y = 2X) = E(2X | I = 1) \neq E(2X)
\end{align}

There is no reason we can drop the condition $I=1$ without ensuring that these events are independent. In fact, the above simulation shows, that they are _not_ independent. This leads us to the surprising result that observing the result of one envelope gives us information about the amount of money in the second envelope! It also shows that a naive assumption of symmetry can lead to false intuition. The symmetry argument of strategy 1 secretly assumes independence of the observations, which is clearly violated.

####Mathematical resolution of the paradox

To base the resolution is a little more formal description, we denote the observed amount of money of as $x_a$ and $x_b$ and can safely assume $x_a<x_b$. We switch the envelopes in the event $T>x_a$. Let us denote the success of the strategy as event $S$ and choosing envelope $a$ as event $A$. Assuming $T\sim Expo(1)$, we have $P(S|A) = P(T>x_a) = 1-(1-e^{-x_a}) = e^{-x_a}$ and correspondingly $P(S|A^c) = P(T<x_b) = 1- e^{-x_b}$. Adding the events correctly we obtain
\begin{align}
\frac{1}{2} P(S|A) + \frac{1}{2} P(S|A^c) = \frac{1}{2} +  \frac{1}{2} \left( e^{-x_a} - e^{-x_b}\right) > \frac{1}{2}
\end{align}

Dependency on the choice of the threshold
------

The above example was very specific on the choice of our threshold. However, it is now easy to investigate how our success rate depends on the rate parameter $\lambda$ of our distribution $T\sim Expo(\lambda)$ or on the choice of the money amounts in the envelopes.

In this section we use the global money distribution

```python
money = pm.DiscreteUniform('money', 100, 100000)
```

####Dependence on the rate parameter

We investigate the rate parameter first. To have a continous curve we need a method to repeatedly compute the success rate.

```python
def createSuccessValuesFrom(rates):
    success_values = []
    for rate in rates:
        threshold = pm.Exponential("threshold", rate)
        success_values.append(run_n_experiments(1000, money, threshold))
    return success_values
```

Now let's create a couple of rates and plot the final result

```python
array_of_rates = np.logspace(-8, 0, num=100)
plt.semilogx(array_of_rates, createSuccessValuesFrom(array_of_rates))
```

We see that this curve is pretty noisy. To smoothen the noise, we can create a bunch of curves and average them. Alternatively we can create a bunch of success rates for every threshold rate and average them.

```python
def averageSuccessRate(threshold, number_of_repetitions):
    trial = 0
    lst = []
    while trial < number_of_repetitions:
        lst.append(run_n_experiments(100, money, threshold))
        trial += 1
    return np.mean(lst), np.std(lst)

def createSmoothSuccessValuesFrom(rates, number_of_repetitions):
    success_values = []
    stddev = []
    for rate in rates:
        threshold = pm.Exponential("threshold", rate)
        success_values.append(averageSuccessRate(threshold, number_of_repetitions)[0])
        stddev.append(averageSuccessRate(threshold, number_of_repetitions)[1])
    return success_values, stddev
```

Let's plot the results!

```python
array_of_rates = np.logspace(-8, 0, num=50)
smoothened_rates = createSmoothSuccessValuesFrom(array_of_rates, 25)
plt.semilogx(array_of_rates, smoothened_rates[0])
```

We see that after the noise reduction we still have a pronounced feature. To fully appreciate that let's see what the deviation from the mean is for an eyeballed optimal rate of $\lambda = 2\, 10^{-5}$

```python
threshold = pm.Exponential('threshold', 0.00002)
print 'The success probability is approximately p = %0.3f +/- %0.3f ' % averageSuccessRate(threshold, 100)
```

    The success probability is approximately p = 0.637 +/- 0.044

We win! Even within a standard deviation the success rate is significant.

However we also see that not every value of the threshold leads to a successful stratetgy and for the wrong values of the threshold we gain no advatange and the two envelopes seem uncorrelated again.

Dependency on the distribution of the money
------

Let us see how the distribution range for the money comes into play here. Let us do the above analysis for various different upper ranges of the money distribution.

####Uniform money distribution

```python
def createSuccessValuesWithMoneyRangeFrom(rates, money):
    success_values = []
    for rate in rates:
        threshold = pm.Exponential("threshold", rate)
        success_values.append(run_n_experiments(1000, money, threshold))
    return success_values
```

```python
money_1 = pm.DiscreteUniform('money', 100, np.power(10, 3))
money_2 = pm.DiscreteUniform('money', 100, np.power(10, 6))
money_3 = pm.DiscreteUniform('money', 100000, np.power(10, 9))

array_of_rates = np.logspace(-10, 0, num=100)
plt.semilogx(array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_1),
             array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_2),
             array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_3))
```

We can clearly see that with increasing the spread of the distribution the maximum is moving towars smaller rates. In fact you can roughly say that the maximum occurs at at a rate close to the inverse of the upper boundary. This makes sense since the expectation value $E(T) = 1/\lambda$ and thus an optimal threshold choice is given by the inverse of the upper limit of the money distribution.

####Different distributions

The previous choices of the money distribution have been motivated by the idea that we have no knowledge about the value of the money in the envelope at all. This means that every value is equally likely and the natural choice is a uniform distribution.

However, what happens if we do gain some insight to the distribution? A good assumption is that the person handing out the money will do so according to a normal distribution $X\sim N(\mu, \sigma^2)$. Let's see what happens:

```python
money_1 = pm.Normal('money', np.power(10,5), 100)
money_2 = pm.Normal('money', np.power(10,5), 500)
money_3 = pm.Normal('money', np.power(10,5), 10000)

array_of_rates = np.logspace(-8, 0, num=100)
plt.semilogx(array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_1),
             array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_2),
             array_of_rates, createSuccessValuesWithMoneyRangeFrom(array_of_rates, money_3))
```

Our advantage is gone! The fact that we do know something about the money distribution mysteriously decorrelates the envelopes and renders our strategy obsolete. We might as well stick with any envelope independent of the amount of money we find in it. However, this might not be a bad trade, since we now have a rough idea of what is in either envelope before we even open it.

Summary
------

The two-envelope paradox is quite surprising. In a first hunch we would assume that the situation is quite symmetric and that there is no chance we could potentially gain knowledge from opening an envelope. However, it turns out, that we can devise a strategy that allows us to make an optimal choice whether to switch envelopes or not. The two envelopes are correlated despite what meets the eye! However, our optimal choice hinges on the range and type of distribution. We only gain an advantage if we have zero prior knowledge of the money distribution, i.e. for a uniform. If we gain knowledge in the form of the money distribution, as is the case for the normal, our strategy is rendered useless. This might not be too bad as it probably eases our minds by making the choice easier.

I hope you enjoy this notebook and have a lot of fun playing some more with it, such as exploring different distributions and maybe strategies. Contributions are more than welcome!

###Miscellenea

####References

Stylesheets and idea from
- [1] [Probabilistic Programming and Bayesian Methods for Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers)

####Stylesheet

```python
from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```

Numerical-Experimentation/The Two Envelope Paradox.ipynb


## Computing partition function of the $2-$D Ising model using tensor network
The partition function of $2-$D ferromagnetic Ising model can be written as
$$Z(\beta)=\sum_\mathbf{s}\prod_{ij}e^{\beta s_is_j}=\mathbf{Tr}\left( \mathcal{A^{(1)}} \times \mathcal{A^{(2)}}\times\cdots\times \mathcal{A^{(L\times L)}}\right).$$ Here the last term denotes tensor contraction of all the tensors, each of which is a tensor given by $$\mathcal{A}^{(i)} = I\times_1\sqrt{\mathbf{B}}\times_2\sqrt{\mathbf{B}}\times_3\sqrt{\mathbf{B}}\times_4\sqrt{\mathbf{B}},$$ where $I$ is a identity tensor with order equals to the degree of node $i$, and $B_{ij}$ defines a Boltzmann matrix with $$
\mathbf{B}=\left[\begin{array}{l}e^{\beta }& e^{-\beta } \\ e^{-\beta}& e^{\beta}\end{array}\right].$$ $\sqrt{B}$ indicates square root of matrix $B$, yileding $\sqrt{B}\times \sqrt{B} = B$.

Given eigen decomposition of the semi-definite symmetric matrix $B$, $B=V\Sigma V^T$, simple calculation gives
$$\sqrt{B}=V\sqrt{\Sigma}V^T=\frac{1}{2}\left[\begin{array}{l}\sqrt{2\cosh(\beta)}+\sqrt{2\sinh(\beta)} &\sqrt{2\cosh(\beta)} -\sqrt{2\sinh(\beta)} \\\sqrt{2\cosh(\beta)} -\sqrt{2\sinh(\beta)}& \sqrt{2\cosh(\beta)}+\sqrt{2\sinh(\beta)}\end{array}\right] $$.

Also notice that when we are considering open boundary conditions, tensors on the boundaries are contracting to $\sqrt{\mathbf{B}}$ only from boundary edges.

```python
import torch,math
import numpy as np
from scipy.linalg import sqrtm
```

## Boundary MPS method for contracting the $2-$ dimensional tensor network

* For convenidence, boundary tensors of the mps is made to a $3$-way tensor with an additional index having bond dimension $1$.

* Indexing of $3$-way and $4$-way tensors are clock-wise for tensors on the top half of the lattice.

```python
def get_lnZ(L,beta=0.44068679350977147,chi=16,mydtype=torch.float64,mydevice=torch.device('cpu')):
    assert(L%2==0)
    B=torch.tensor(sqrtm(np.array([[np.exp(beta),np.exp(-beta)],[np.exp(-beta),np.exp(beta)]])),dtype=mydtype,device=mydevice) # the Boltzmann matrix
    A2=B@B #A2=torch.einsum("ij,ab,jb->ia",[B,B,I2])
    A3=torch.einsum("i,j,k->ijk",B[:,0],B[:,0],B[:,0])+torch.einsum("i,j,k->ijk",B[:,1],B[:,1],B[:,1]) #A3=torch.einsum("ij,ab,cd,jbd->iac",[B,B,B,I3])
    A4=torch.einsum("i,j,k,l->ijkl",B[:,0],B[:,0],B[:,0],B[:,0])+torch.einsum("i,j,k,l->ijkl",B[:,1],B[:,1],B[:,1],B[:,1]) # A4=torch.einsum("ij,ab,cd,xy,jbdy->iacx",[B,B,B,B,I4])
    tensors=[]
    tensors.append( [A2[None,:,:] if i==0 else( A2[:,:,None] if i==L-1 else A3) for i in range(L)] )
    for j in range(1,L>>1):
        tensors.append( [ A3[None,:,:,:] if i==0 else( A3[:,:,None,:] if i==L-1 else A4)  for i in range(L) ] )
    lnZ=0 # log of partition function
    for head in range((L>>1)-1): # mps on the boundary is eating the next mpo, for L/2-1 times
        [res,tensors[head+1]] = compress( eat(tensors[head][:],tensors[head+1][:]) , chi)
        lnZ += res
    return 2*lnZ
```

* Appling an MPO with bound dimension $2$ to an MPS results to a new MPS with a doubled bound dimension.

```python
def eat(mps,mpo):
    return [ torch.einsum("ijk,abcj->iabkc",mps[i],mpo[i]).contiguous().view(mps[i].shape[0]*mpo[i].shape[0],2,-1) for i in range(len(mps))]

```

* Left canonicalization using sequential QR decompositions. After that, all tensors except the right most ones are isometry with a column orthogonal unforded matrix

* Then we do sequential singular value decompositions and a cut off on the spectrum to reduce bondimensions.

```python
def compress(mps,chi):
    residual=0
    for i in range(len(mps)-1): # From left to right, sweep once doing qr decompositions
        Q,R=torch.qr(mps[i].contiguous().view(mps[i].shape[0]*2,-1))
        mps[i] = Q.contiguous().view(mps[i].shape[0],2,-1)
        mps[i+1] = torch.einsum("ij,jab->iab",[R,mps[i+1]])
    for i in range(len(mps)-1,0,-1): # From right to left, sweep onece using svd on the tensor merged from two consecutive tensors.
        [U,s,V]=torch.svd( torch.einsum("ijk,kab->ijab",mps[i-1],mps[i]).view(mps[i-1].shape[0]*2,mps[i].shape[2]*2) )
        mps[i] = V[:,:chi].t().contiguous().view(-1,2,mps[i].shape[2])
        mps[i-1] = (U[:,:chi]@torch.diag(s[:chi])).contiguous().view(mps[i-1].shape[0],2,-1)
        tnorm=mps[i-1].norm()
        mps[i-1] /= tnorm
        residual += math.log(tnorm)
    return residual,mps
```

```python
import kacward
L=16
beta_c=0.44068679350977147
chi=16
print("L=",L," chi=",chi)
lnZ=get_lnZ(L=L,beta=beta_c,chi=chi);print("lnZ_TN=",lnZ/L**2)
lnZ_exact=kacward.lnZ_2d_ferro_Ising(L,beta_c);print("lnZ_Exact=",lnZ_exact/L**2)
print("|lnZ-lnZ_exact|=%.2g"%(abs(lnZ-lnZ_exact)/L**2))
```

    L= 16  chi= 16
    lnZ_TN= 0.9085922273659681
    lnZ_Exact= 0.9085922273659632
    |lnZ-lnZ_exact|=4.9e-15

```python

```

```python
%matplotlib inline
import matplotlib.pyplot as plt
L=16
lnZ_exact=kacward.lnZ_2d_ferro_Ising(L,beta_c);
chis=[2, 3, 4, 5, 6, 7, 8, 9, 10, 12 ,14];
diff=np.abs(np.array([get_lnZ(L=L,beta=beta_c,chi=chi)/L**2 for chi in chis])-lnZ_exact/L**2)
plt.rcParams['figure.figsize']=(15,3.5)
plt.rcParams['font.size']= 20
plt.plot(np.array(chis),diff,'ro-',linewidth=2,markersize=5)
plt.yscale('log')
plt.xlabel('Bond dimension',fontsize=30)
plt.ylabel('Error',fontsize=30)
plt.title('L=16',fontsize=30)
plt.show()

input("Press any key to compute L=32")
L=32
lnZ_exact=kacward.lnZ_2d_ferro_Ising(L,beta_c);
chis=[2, 3, 4, 5, 6, 7, 8, 9, 10, 12 ,14];
diff=np.abs(np.array([get_lnZ(L=L,beta=beta_c,chi=chi)/L**2 for chi in chis])-lnZ_exact/L**2)
plt.rcParams['figure.figsize']=(15,3.5)
plt.plot(np.array(chis),diff,'bx-',linewidth=2,markersize=5)
plt.yscale('log')
plt.xlabel('Bond dimension',fontsize=30)
plt.ylabel('Error',fontsize=30)
plt.title('L=32',fontsize=30);
```

## Onsager solution
What we are going to compare with is the Onsager solution, with $\ln Z$, critical temperature $\beta_c$, and the spontaneous magnetization are given as
\begin{align} \ln Z&=\ln 2 +\frac{1}{8\pi^2}\int_0^{2\pi}d\theta_1\int_0^{2\pi}d\theta_2\ln\left[  \cosh^2(2\beta)-\sinh(2\beta)\cos(\theta_1)-\sinh(2\beta)\cos(\theta_2)  \right]\\
\beta_c&=\frac{\ln (1+\sqrt{2})}{2},\\
m_{\beta \gt \beta_c}&=\left[ 1-\sinh^{-4}(2\beta) \right]^{\frac{1}{8}}.
\end{align}

```python
from scipy.integrate import nquad
def m_exact(beta):
    if(beta>0.5*math.log(1+math.sqrt(2))):
        return math.pow((1- math.pow(math.sinh(2*beta),-4)),1/8  )
    else:
        return 0
def Onsager(beta):
    if(beta>0.5*math.log(1+math.sqrt(2))):
        m = math.pow((1- math.pow(math.sinh(2*beta),-4)),1/8  )
    else:
        m=0
    def func_Osg(theta1,theta2):
        return math.log( math.cosh(2*beta)*math.cosh(2*beta) - math.sinh(2*beta)*math.cos(theta1)-math.sinh(2*beta)*math.cos(theta2) )
    lnZ=nquad(func_Osg,[[0, 2*math.pi],[0, 2*math.pi]])[0]/(8*math.pi*math.pi)+math.log(2)
    return m,lnZ
[m,lnZ_Onsager]=Onsager(beta_c)
lnZ_Onsager
```

    0.9296953983416107

```python
import time
chi=16
Ls=[4,8,16,32,64,128,170,200]
time_used=[]
plt.rcParams['figure.figsize']=(20,3.5)
plt.rcParams['font.size']= 20
g, ax = plt.subplots(1,2)
lnZ=[]
for L in Ls:
    t1=time.time()
    res=get_lnZ(L=L,beta=beta_c,chi=chi)/L**2
    lnZ.append(res)
    t=time.time()-t1
    time_used.append(t)
    print("L=%d\tLnZ=%.15f\t lnZ_Onsager=%.15f \t time %.2f sec."%(L,res,lnZ_Onsager,t))
Ls=np.array(Ls)
ax[0].plot(Ls,np.array(lnZ),'ro-',linewidth=2,markersize=5)
ax[0].plot(Ls,Ls-Ls+lnZ_Onsager,'k--',linewidth=2,markersize=5)
ax[0].set_xlabel('L',fontsize=30)
ax[0].set_xlim(min(Ls)-0.1,max(Ls)+0.1)
ax[0].set_ylabel('lnZ',fontsize=30)
ax[0].legend(['Tensor Network','Onsager'])
ax[1].plot(np.array(Ls),np.array(time_used),'bx-',linewidth=2,markersize=5)
ax[1].set_xlim(min(Ls)-0.1,max(Ls)+0.1)
ax[1].set_xlabel('L',fontsize=30)
ax[1].set_ylabel('time(sec.)',fontsize=30);

```

# Question: How to obtain $L\to\infty$ results using tensor networks efficiently?


# Controlled oscillator

The controlled oscillator is an oscillator
with an extra input that controls the frequency of the oscillation.

To implement a basic oscillator,
we would use a neural ensemble of two dimensions
that has the following dynamics:

$$
\dot{x} = \begin{bmatrix} 0 && - \omega \\ \omega && 0 \end{bmatrix} x
$$

where the frequency of oscillation is $\omega \over {2 \pi}$ Hz.

We need the neurons to represent three variables,
$x_0$, $x_1$, and $\omega$.
According the the dynamics principle of the NEF,
in order to implement some particular dynamics,
we need to convert this dynamics equation into a feedback function:

$$
\begin{align}
  \dot{x} &= f(x) \\
  &\implies f_{feedback}(x) = x + \tau f(x)
\end{align}
$$

where $\tau$ is the post-synaptic time constant of the feedback connection.

In this case, the feedback function to be computed is

$$
\begin{align}
  f_{feedback}(x) &= x + \tau
  \begin{bmatrix}
    0 && - \omega \\
    \omega && 0
  \end{bmatrix}
  x \\
  &=
  \begin{bmatrix}
    x_0 - \tau \cdot \omega \cdot x_1 \\
    x_1 + \tau \cdot \omega \cdot x_0
  \end{bmatrix}
\end{align}
$$

Since the neural ensemble represents all three variables
but the dynamics only affects the first two ($x_0$, $x_1$),
we need the feedback function to not affect that last variable.
We do this by adding a zero to the feedback function.

$$
f_{feedback}(x) = \begin{bmatrix}
  x_0 - \tau \cdot \omega \cdot x_1 \\
  x_1 + \tau \cdot \omega \cdot x_0 \\
 0 \end{bmatrix}
$$

We also generally want to keep
the ranges of variables represented within an ensemble
to be approximately the same.
In this case, if $x_0$ and $x_1$ are between -1 and 1,
$\omega$ will also be between -1 and 1,
giving a frequency range of $-1 \over {2 \pi}$ to $1 \over {2 \pi}$.
To increase this range,
we introduce a scaling factor to $\omega$ called $\omega_{max}$.

$$
f_{feedback}(x) = \begin{bmatrix}
  x_0 - \tau \cdot \omega \cdot \omega_{max} \cdot x_1 \\
  x_1 + \tau \cdot \omega \cdot \omega_{max} \cdot x_0 \\
  0 \end{bmatrix}
$$

```python
%matplotlib inline
import matplotlib.pyplot as plt

import nengo
from nengo.processes import Piecewise
```

## Step 1: Create the network

```python
tau = 0.1  # Post-synaptic time constant for feedback
w_max = 10  # Maximum frequency is w_max/(2*pi)

model = nengo.Network(label='Controlled Oscillator')
with model:
    # The ensemble for the oscillator
    oscillator = nengo.Ensemble(500, dimensions=3, radius=1.7)

    # The feedback connection
    def feedback(x):
        x0, x1, w = x  # These are the three variables stored in the ensemble
        return x0 + w * w_max * tau * x1, x1 - w * w_max * tau * x0, 0

    nengo.Connection(oscillator, oscillator, function=feedback, synapse=tau)

    # The ensemble for controlling the speed of oscillation
    frequency = nengo.Ensemble(100, dimensions=1)

    nengo.Connection(frequency, oscillator[2])
```

## Step 2: Create the input

```python
with model:
    # We need a quick input at the beginning to start the oscillator
    initial = nengo.Node(Piecewise({0: [1, 0, 0], 0.15: [0, 0, 0]}))
    nengo.Connection(initial, oscillator)

    # Vary the speed over time
    input_frequency = nengo.Node(
        Piecewise({
            0: 1,
            1: 0.5,
            2: 0,
            3: -0.5,
            4: -1
        }))

    nengo.Connection(input_frequency, frequency)
```

## Step 3: Add Probes

```python
with model:
    # Indicate which values to record
    oscillator_probe = nengo.Probe(oscillator, synapse=0.03)
```

## Step 4: Run the Model

```python
with nengo.Simulator(model) as sim:
    sim.run(5)
```

## Step 5: Plot the Results

```python
plt.figure()
plt.plot(sim.trange(), sim.data[oscillator_probe])
plt.xlabel('Time (s)')
plt.legend(['$x_0$', '$x_1$', r'$\omega$']);
```


```
from sympy import *
import numpy as np
import math
```

Strutinsky Energy Averaging Method

Define a callable class for computing the smooth part of the density of states $\tilde{g}_m(E)$ with curvature
correction of order $2M$.

```
class SmoothDOS(object):

    def __init__(self, energies):
        self.energies = energies

    def gaussian_smoothing_func(self, M, x):
        return (mpmath.laguerre(M,0.5,x**2)*exp(-x**2)/sqrt(pi)).evalf()

    def __call__(self, e, M=3, gamma=1.0):
        return sum(self.gaussian_smoothing_func(M, (e-ei)/gamma)/gamma for ei in self.energies)
```

```
def avgf(M, x):
    return (mpmath.laguerre(M,0.5,x**2)*exp(-x**2)/sqrt(pi)).evalf()
```

```
def smooth_dos(gamma, M, e, energies):
    return sum(avgf(M, (e-ei)/gamma)/gamma for ei in energies)
```

1D Simple Harmonic Oscillator DOS

```
def sho_smooth_dos(e):
    """Compute the exact smooth DOS."""
    return 1
```

```
def sho_spectrum(nmax):
    """Compute the first nmax energies of the 1D SHO."""
    return [n+0.5 for n in range(nmax)]
```

```
sho_evalues = np.linspace(0.0,20,100)
```

```
sho_dos = SmoothDOS(sho_spectrum(30))
```

```
sho_exact_dos = [sho_smooth_dos(e) for e in sho_evalues]
```

```
sho_approx_dos = [sho_dos(e,M=3,gamma=10.0) for e in sho_evalues]
```

```
plot(sho_evalues, sho_exact_dos, label="exact")
plot(sho_evalues, sho_approx_dos, label="approx")
title("Smooth part of the DOS for the 1D SHO")
xlabel("Energy"); ylabel("$g(E)$")
legend(loc=4)
```

Note how the exact smoothed DOS and the Strutinsky approximation agree once the energy is away from
0. In general, these are edge effects and are also present at the right limit if you don't use enough energy
levels.  Here we have used 30, so the right side doesn't show these artifacts.

1D Particle in a BOX (PIAB) DOS

```
def piab_smooth_dos(e):
    """Compute the exact smooth DOS."""
    return 1.0/(2.0*math.sqrt(e*math.pi**2/2))
```

```
def piab_spectrum(nmax):
    """Compute the first nmax energies of the 1D PIAB."""
    return [0.5*math.pi**2*(n+1)**2 for n in range(nmax)]
```

```
piab_evalues = linspace(1.0,1000.0,100)
```

```
piab_dos = SmoothDOS(piab_spectrum(20))
```

```
piab_exact_dos = [piab_smooth_dos(e) for e in piab_evalues]
```

```
piab_approx_dos = [piab_dos(e, M=4, gamma=150.0) for e in piab_evalues]
```

```
plot(piab_evalues, piab_exact_dos, label="exact")
plot(piab_evalues, piab_approx_dos, label="approx")
title("Smooth part of the DOS for the 1D PIAB")
xlabel("Energy"); ylabel("$g(E)$")
legend(loc=1)
```

```

```


```python
import control
import sympy
import numpy as np
import matplotlib.pyplot as plt
sympy.init_printing()
```

## Original System

```python
s = control.tf([1, 0], [0, 1])
```

```python
G = 1/((s+2)*s)
G
```

$$\frac{1}{s^2 + 2 s}$$

### Requirements

* 0 steady state error to unit ramp response
* damping ratio of sqrt(2)/2
* settling time of 2 seconds

```python
control.rlocus(G, xlim=[-3, 0], ylim=[-3, 3]);
plt.title('uncompensated root locus')
sd = -2 + 2j

plt.plot(np.real(sd), np.imag(sd), 's')
plt.plot(np.real(sd), -np.imag(sd), 's')
```

```python
Gc = G/(1+G)
t, y = control.step_response(Gc, T=np.linspace(0, 20));
plt.plot(t, y - 1, label='y')
#plt.hlines(1, 0, 20, label='u')
plt.title('uncompensated step response error')

Kv = np.real((control.minreal(s*G).horner(0))[0, 0])
Kv

plt.figure()
t, y = control.step_response(Gc/s, T=np.linspace(0, 20));
plt.plot(t, t-y, label='e')
plt.hlines(1/Kv, 0, 20)
plt.title('uncompensated unit ramp response error')
```

## PID Compensator

* Do PD first

```python
H_PD = 2*(s + 4)

```

```python
control.rlocus(G*H_PD, xlim=[-10, 1]);
plt.plot(-2, 2, 's')
plt.plot(-2, -2, 's')
plt.title('PD compensated root locus')
```

```python
Gc_PD = G*H_PD/(1+G*H_PD)
t, y = control.step_response(Gc_PD/s, T=np.linspace(0, 10, 1000));
plt.plot(t, t)
plt.plot(t, y)
plt.title('PD compensated ramp response')

plt.figure()
t, y = control.step_response(Gc_PD/s, T=np.linspace(0, 20, 1000));
plt.plot(t, t-y)
plt.hlines(0.5, 0, 20)
```

## PI Compensator

```python
z = 0.01
H_PI = (s+z)/s
```

```python
H = H_PD*H_PI
```

```python
H
```

$$\frac{2 s^2 + 8.02 s + 0.08}{s}$$

Find:

K_P = 8.02, K_I = 0.08, K_D = 2

```python
1/np.abs((G*H).horner(-2 + 2j))
```

    array([[[1.00250312]]])

```python
control.rlocus(G*H,xlim=[-10, 0], ylim=[-3, 3]);
plt.plot(-2, 2, 's')
plt.plot(-2, -2, 's')
```

```python
control.minreal(G*H_PD*s).horner(0)
```

    1 states have been removed from the model

    array([[[4.+0.j]]])

```python
control.minreal(G*H*s).horner(0)
```

    1 states have been removed from the model

    array([[[inf+nanj]]])

```python
Gc_PID = G*H/(1+G*H)
t, y = control.step_response(Gc_PID/s, T=np.linspace(0, 1000, 1000));
plt.plot(t, t)
plt.plot(t, y)
plt.title('PID compensated ramp response')

plt.figure()
plt.plot(t, t - y)
plt.hlines(t[0], t[-1], 80)
plt.title('PID compensated ramp response error')
```


## Multiple Linear Regression
We will see how multiple input variables together influence the output variable, while also learning how the calculations differ from that of Simple LR model. We will also build a regression model using Python.
At last, we will go deeper into Linear Regression and will learn things like Multicollinearity, Hypothesis Testing, Feature Selection, and much more.

```python
# Importing required libraries
import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
from mpl_toolkits.mplot3d import Axes3D
```

### Advertising Data
We are going to use Advertising data which is available on the site of USC Marshall School of Business.
The advertising data set consists of the sales of a product in 200 different markets, along with advertising budgets for three different media: TV, radio, and newspaper.

Loading and plotting the Data

```python
data = pd.read_csv("Advertising.csv")
data.drop("Unnamed: 0", axis =1,inplace=True)
data
```

      TV
      Radio
      Newspaper
      Sales

      0
      230.1
      37.8
      69.2
      22.1

      1
      44.5
      39.3
      45.1
      10.4

      2
      17.2
      45.9
      69.3
      9.3

      3
      151.5
      41.3
      58.5
      18.5

      4
      180.8
      10.8
      58.4
      12.9

      ...
      ...
      ...
      ...
      ...

      195
      38.2
      3.7
      13.8
      7.6

      196
      94.2
      4.9
      8.1
      9.7

      197
      177.0
      9.3
      6.4
      12.8

      198
      283.6
      42.0
      66.2
      25.5

      199
      232.1
      8.6
      8.7
      13.4

200 rows × 4 columns

```python
plt.scatter(data.TV, data.Sales, color='blue', label='TV', alpha=0.5)
plt.scatter(data.Radio, data.Sales, color='red', label='radio', alpha=0.5)
plt.scatter(data.Newspaper, data.Sales, color='green', label='newspaper', alpha=0.5)

plt.legend(loc="lower right")
plt.title("Sales vs. Advertising")
plt.xlabel("Advertising [1000 $]")
plt.ylabel("Sales [Thousands of units]")
plt.grid()
plt.show()
```

----

#### Mathematically, general Linear Regression model can be expressed as:
\begin{align}
Y= β_0+ β_1 X_1+ β_2 X_2+⋯+ β_p X_p
\end{align}

Here, Y is the output variable, and X terms are the corresponding input variables.The first β term (βo) is the intercept constant and is the value of Y in absence of all predictors (i.e when all X terms are 0).

Finding the values of these constants(β) is what regression model does by minimizing the error function and fitting the best line or hyperplane (depending on the number of input variables).
This is done by minimizing the Residual Sum of Squares (RSS), which is obtained by squaring the differences between actual and predicted outcomes.

\begin{align}
RSS=∑_i^n(y_i-y ̂_i )^2
\end{align}

Because this method finds the least sum of squares, it is also known as the **Ordinary Least Squares** (OLS) method.

There are two primary ways to implement the OLS algorithm: **Scikit Learn** and **Statsmodels**

#### SciKit Learn:
Just import the Linear Regression module from the Sklearn package and fit the model on the data. This method is pretty straightforward and you can see how to use it below.

```python
from sklearn.linear_model import LinearRegression
sk_model = LinearRegression()
sk_model.fit(data.drop('Sales', axis=1), data.Sales)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
print("Intercept: ", sk_model.intercept_)
print("Coefficients: ", sk_model.coef_)
```

    Intercept:  2.938889369459412
    Coefficients:  [ 0.04576465  0.18853002 -0.00103749]

#### Statsmodels
Another way is to use the Statsmodels package to implement OLS. Statsmodels is a Python package that allows performing various statistical tests on the data. We will use it here because it will be helpful for us later as well.

```python
# Importing statsmodels
import statsmodels.formula.api as sm

# Fitting the OLS on data
model = sm.ols('Sales ~ TV + Radio + Newspaper', data).fit()
print(model.params)
```

    Intercept    2.938889
    TV           0.045765
    Radio        0.188530
    Newspaper   -0.001037
    dtype: float64

    /usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

These results can be interpreted as follows:

* If we fix the budget for TV & newspaper, then increasing the radio budget by 1000 USD  will lead to an increase in sales by around **189 units**(0.189*1000).

* Similarly, by fixing the radio & newspaper, we infer an approximate rise of **46 units** of products per 1000 USD increase in the TV budget.

* However, for the newspaper budget, since the coefficient is quite negligible (close to zero), it’s evident that the newspaper is not affecting the sales. In fact, it’s on the negative side of zero(-0.001) which, if the magnitude was big enough, could have meant that this agent is rather causing the sales to fall. But we cannot make that kind of inference with such negligible value.

If sales is regressed solely on newspaper (as shown below), the slope coefficient will come out to be 0.055, which is significanly large as compared to what we saw above.

```python
# Simple Linear regression for sales vs newspaper
model_npaper = sm.ols('Sales ~ Newspaper', data).fit()
print(model_npaper.params)
```

    Intercept    12.351407
    Newspaper     0.054693
    dtype: float64

#### This is explained by **Multicollinearity**

Let's plot and observe the correlations among the variables

```python
data.corr()
```

      TV
      Radio
      Newspaper
      Sales

      TV
      1.000000
      0.054809
      0.056648
      0.782224

      Radio
      0.054809
      1.000000
      0.354104
      0.576223

      Newspaper
      0.056648
      0.354104
      1.000000
      0.228299

      Sales
      0.782224
      0.576223
      0.228299
      1.000000

#### Plotting the heatmap to visualize the Correlation

```python
# Plotting correlation heatmap
plt.ylim(-.5,3.5)
plt.imshow(data.corr(), cmap=plt.cm.GnBu, interpolation='nearest',data=True)
plt.colorbar()
tick_marks = [i for i in range(len(data.columns))]
plt.xticks(tick_marks, data.columns, rotation=45)
plt.yticks(tick_marks, data.columns, rotation=45)

# Putting annotations
for i in range(len(data.columns)):
    for j in range(len(data.columns)):
        text = '%.2f'%(data.corr().iloc[i,j])
        plt.text(i-0.2,j-0.1,text)
```

The correlation between newspaper and radio is 0.35. This indicates a fair relationship between newspaper and radio budgets.

Hence, it can be inferred that → when the radio budget is increased for a product, there’s a tendency to spend more on newspapers as well.

This is called Multicollinearity and is referred to as a situation in which two or more input variables are linearly related.
Hence, even though the Multiple Regression model shows no impact on sales by the newspaper, the Simple Regression model still does due to this multicollinearity and the absence of other input variables.

$Sales & Radio → probable causation
Newspaper & Radio → multicollinearity
Sales & Newspaper → transitive correlation$

---

### Hypothesis Testing

One of the fundamental questions that should be answered while running Multiple Linear Regression is, whether or not, at least one of the predictors is useful in predicting the output.

We saw that the three predictors TV, radio and newspaper had a different degree of linear relationship with the sales. But what if the relationship is just by chance and there is no actual impact on sales due to any of the predictors?

The model can only give us numbers to establish a close enough linear relationship between the response variable and the predictors. However, it cannot prove the credibility of these relationships.

To have some confidence, we take help from statistics and do something known as a Hypothesis Test. We start by forming a Null Hypothesis and a corresponding Alternative Hypothesis.

 NULL HYPOTHESIS
\begin{align}
H_0 : \beta_1 = \beta_2 = ... = \beta_p = 0
\end{align}
\begin{align}
H_0 : \beta_TV = \beta_radio = \beta_newspaper = 0
\end{align}

 ALTERNATIVE HYPOTHESIS
\begin{align}
H_a : At \: least \: one \: \beta_i \: is \: non-zero
\end{align}

The hypothesis test is performed by using F-Statistic. The formula for this statistic contains Residual Sum of Squares (RSS) and the Total Sum of Squares (TSS), which we can calculate using the Statsmodels.

```python
print(model.summary2())
```

                     Results: Ordinary least squares
    =================================================================
    Model:              OLS              Adj. R-squared:     0.896
    Dependent Variable: Sales            AIC:                780.3622
    Date:               2021-03-09 18:29 BIC:                793.5555
    No. Observations:   200              Log-Likelihood:     -386.18
    Df Model:           3                F-statistic:        570.3
    Df Residuals:       196              Prob (F-statistic): 1.58e-96
    R-squared:          0.897            Scale:              2.8409
    ------------------------------------------------------------------
                    Coef.   Std.Err.     t     P>|t|    [0.025  0.975]
    ------------------------------------------------------------------
    Intercept       2.9389    0.3119   9.4223  0.0000   2.3238  3.5540
    TV              0.0458    0.0014  32.8086  0.0000   0.0430  0.0485
    Radio           0.1885    0.0086  21.8935  0.0000   0.1715  0.2055
    Newspaper      -0.0010    0.0059  -0.1767  0.8599  -0.0126  0.0105
    -----------------------------------------------------------------
    Omnibus:             60.414       Durbin-Watson:          2.084
    Prob(Omnibus):       0.000        Jarque-Bera (JB):       151.241
    Skew:                -1.327       Prob(JB):               0.000
    Kurtosis:            6.332        Condition No.:          454
    =================================================================

If the value of F-statistic is equal to or very close to 1, then the results are in favor of the Null Hypothesis and we fail to reject it.

But as we can see that the F-statistic is many folds larger than 1, thus providing strong evidence against the Null Hypothesis (that all coefficients are zero). Hence, we **reject the Null Hypothesis** and are confident that at least one predictor is useful in predicting the output.

*Note that F-statistic is not suitable when the number of predictors(p) is large, or if p is greater than the number of data samples (n).*

Hence, we can say that at least one of the three advertising agents is useful in predicting sales.

But to find out, which predictor or predictors are useful and which one are not, we do **Feature Selection**

Two of the ways of doing Feature Selection are **Forward Selecton** & **Backward Selection**

Let's proceed with forward selection. We start with a model without any predictor and just the intercept term. We then perform simple linear regression for each predictor to find the best performer(lowest RSS). We then add another variable to it and check for the best 2-variable combination again by calculating the lowest RSS(Residual Sum of Squares). After that the best 3-variable combination is checked, and so on. The approach is stopped when some stopping rule is satisfied.

```python
# Defining the function to evaluate amodel
def evaluateModel (model):
    print("RSS = ", ((data.sales - model.predict())**2).sum())
    print("R2 = ", model.rsquared)
```

Single Predictor Models

```python
# For TV
model_TV = sm.ols('Sales ~ TV', data=data).fit()
print("model_TV")
print(model_TV.summary())
print("------------")

# For radio
model_radio = sm.ols('Sales ~ Radio', data=data).fit()
print("model_radio")
print(model_radio.summary())
print("------------")

# For newspaper
model_newspaper = sm.ols('Sales ~ Newspaper', data=data).fit()
print("model_newspaper")
print(model_newspaper.summary())
print("------------")
```

    model_TV
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.612
    Model:                            OLS   Adj. R-squared:                  0.610
    Method:                 Least Squares   F-statistic:                     312.1
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):           1.47e-42
    Time:                        18:29:01   Log-Likelihood:                -519.05
    No. Observations:                 200   AIC:                             1042.
    Df Residuals:                     198   BIC:                             1049.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================
    Omnibus:                        0.531   Durbin-Watson:                   1.935
    Prob(Omnibus):                  0.767   Jarque-Bera (JB):                0.669
    Skew:                          -0.089   Prob(JB):                        0.716
    Kurtosis:                       2.779   Cond. No.                         338.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------
    model_radio
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.332
    Model:                            OLS   Adj. R-squared:                  0.329
    Method:                 Least Squares   F-statistic:                     98.42
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):           4.35e-19
    Time:                        18:29:01   Log-Likelihood:                -573.34
    No. Observations:                 200   AIC:                             1151.
    Df Residuals:                     198   BIC:                             1157.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      9.3116      0.563     16.542      0.000       8.202      10.422
    Radio          0.2025      0.020      9.921      0.000       0.162       0.243
    ==============================================================================
    Omnibus:                       19.358   Durbin-Watson:                   1.946
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):               21.910
    Skew:                          -0.764   Prob(JB):                     1.75e-05
    Kurtosis:                       3.544   Cond. No.                         51.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------
    model_newspaper
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.052
    Model:                            OLS   Adj. R-squared:                  0.047
    Method:                 Least Squares   F-statistic:                     10.89
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):            0.00115
    Time:                        18:29:01   Log-Likelihood:                -608.34
    No. Observations:                 200   AIC:                             1221.
    Df Residuals:                     198   BIC:                             1227.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     12.3514      0.621     19.876      0.000      11.126      13.577
    Newspaper      0.0547      0.017      3.300      0.001       0.022       0.087
    ==============================================================================
    Omnibus:                        6.231   Durbin-Watson:                   1.983
    Prob(Omnibus):                  0.044   Jarque-Bera (JB):                5.483
    Skew:                           0.330   Prob(JB):                       0.0645
    Kurtosis:                       2.527   Cond. No.                         64.7
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------

We observe that for model_TV, the RSS is least and R² value is the most among all the models. Hence we select model_TV as our base model to move forward.

Now, we will add the radio and newspaper one by one and check the new values.

```python
# For TV & radio
model_TV_radio = sm.ols('Sales ~ TV + Radio', data=data).fit()
print("model_TV_radio")
print(model_TV_radio.summary())
print("------------")

# For TV & newspaper
model_TV_newspaper = sm.ols('Sales ~ TV + Newspaper', data=data).fit()
print("model_TV_newspaper")
print(model_TV_newspaper.summary())
print("------------")
```

    model_TV_radio
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     859.6
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):           4.83e-98
    Time:                        18:29:01   Log-Likelihood:                -386.20
    No. Observations:                 200   AIC:                             778.4
    Df Residuals:                     197   BIC:                             788.3
    Df Model:                           2
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      2.9211      0.294      9.919      0.000       2.340       3.502
    TV             0.0458      0.001     32.909      0.000       0.043       0.048
    Radio          0.1880      0.008     23.382      0.000       0.172       0.204
    ==============================================================================
    Omnibus:                       60.022   Durbin-Watson:                   2.081
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              148.679
    Skew:                          -1.323   Prob(JB):                     5.19e-33
    Kurtosis:                       6.292   Cond. No.                         425.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------
    model_TV_newspaper
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.646
    Model:                            OLS   Adj. R-squared:                  0.642
    Method:                 Least Squares   F-statistic:                     179.6
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):           3.95e-45
    Time:                        18:29:01   Log-Likelihood:                -509.89
    No. Observations:                 200   AIC:                             1026.
    Df Residuals:                     197   BIC:                             1036.
    Df Model:                           2
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      5.7749      0.525     10.993      0.000       4.739       6.811
    TV             0.0469      0.003     18.173      0.000       0.042       0.052
    Newspaper      0.0442      0.010      4.346      0.000       0.024       0.064
    ==============================================================================
    Omnibus:                        0.658   Durbin-Watson:                   1.969
    Prob(Omnibus):                  0.720   Jarque-Bera (JB):                0.415
    Skew:                          -0.093   Prob(JB):                        0.813
    Kurtosis:                       3.122   Cond. No.                         410.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------

As we can see that our values have improved tremendously in TV & radio model. RSS has increased and R² has decreased further, as compared to model_TV. It’s a good sign.

On the other hand, it's not the same for TV and newspaper. The values have improved slightly by adding newspaper too, but not as significantly as with the radio. Hence, at this step, we will proceed with the TV & radio model and will observe the difference when we add newspaper to this model.

```python
# For TV, radio & newspaper
model_all = sm.ols('Sales ~ TV + Radio + Newspaper', data=data).fit()
print("model_all")
print(model_all.summary())
print("------------")
```

    model_all
                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  Sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     570.3
    Date:                Tue, 09 Mar 2021   Prob (F-statistic):           1.58e-96
    Time:                        18:29:02   Log-Likelihood:                -386.18
    No. Observations:                 200   AIC:                             780.4
    Df Residuals:                     196   BIC:                             793.6
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    Radio          0.1885      0.009     21.893      0.000       0.172       0.206
    Newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================
    Omnibus:                       60.414   Durbin-Watson:                   2.084
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              151.241
    Skew:                          -1.327   Prob(JB):                     1.44e-33
    Kurtosis:                       6.332   Cond. No.                         454.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    ------------

The values have not improved with any significance. Hence, it’s imperative to not add newspaper and finalize the model with TV and radio as selected features.
So our final model can be expressed as below:
\begin{align}
sales = 2.92 + 0.046*TV + 0.188*radio
\end{align}

Plotting the variables TV, radio, and sales in the 3D graph, we can visualize how our model has fit a regression plane to the data.

```python
fig = plt.figure(figsize=(12,10))
ax = fig.add_subplot(211, projection='3d')

fig.suptitle('Regression: Sales ~ TV & radio Advertising')

# Defining z function (or sales in terms of TV and radio)
def z_function(x,y):
    return (2.938889 + (0.045765*y) + (0.188530*x))

X, Y = np.meshgrid(range(0,50,2),range(0,300,10))
Z = z_function(X, Y)

## Creating Wireframe
ax.plot_wireframe(X, Y, Z, color='black')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')

## Creating Surface plot
ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap='winter', edgecolor='black', alpha=0.5)

## Adding Scatter Plot
ax.scatter(data.Radio, data.TV, data.Sales, c='red', s=25)

## Adding labels
ax.set_xlabel('Radio')
ax.set_ylabel('TV')
ax.set_zlabel('Sales')
ax.text(0,150,1, '@DataScienceWithSan')

## Rotating for better view
ax.view_init(10,30)
```


$$
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert#1\right\rvert}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inner}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\bra}[1]{\left\langle#1\right|}
\newcommand{\ket}[1]{\left|#1\right\rangle}
\newcommand{\braket}[2]{\left\langle#1|#2\right\rangle}
\newcommand{\ketbra}[2]{\left|#1\right\rangle\left\langle#2\right|}
\newcommand{\angleset}[1]{\left\langle#1\right\rangle}
\newcommand{\expected}[1]{\left\langle#1\right\rangle}
\newcommand{\dv}[2]{\frac{d#1}{d#2}}
\newcommand{\real}[0]{\mathfrak{Re}}
$$

# Projective Measurement

_prepared by Israel Gelover_

### Definition 3.6 Projector

Given a subset of vectors $\set{\ket{f_i}}_{i=1}^n \subset \mathcal{H}$, we define the _Projector_ over the subspace $\mathcal{F}$ generated by them as:

\begin{equation*}
    \begin{split}
        \hat{P}:\mathcal{H} &\to \mathcal{F} \\
        \ket{\psi} &\to \sum_{i=1}^n \ket{f_i}\braket{f_i}{\psi}
    \end{split}
\end{equation*}

It is clear that what we obtain from this operator is a linear combination of $\set{\ket{f_i}}_{i=1}^n$ and therefore, the resulting vector is an element of the subspace generated by these vectors. And it is precisely this definition what we used to calculate the Wave function collapse.

### Definition 3.7 Projective Measurement

A _Projective Measurement_ is described with a self-adjoint operator

\begin{equation*}
    \hat{M} = \sum_m m\hat{P}_m
\end{equation*}

Where $\hat{P}_m$ is a projector on the subspace corresponding to the eigenvalue $m$ of $\hat{M}$.

This is known as the spectral decomposition of the $\hat{M}$ operator, and any self-adjoint operator can be expressed in terms of its spectral decomposition. We emphasize that this way of decomposing a projective measurement is very useful to us since it involves the eigenvalues and the projectors associated with these eigenvalues.

### Example

Let

\begin{equation}\label{op_h}
    \hat{H} = \ketbra{0} + i\ketbra{1}{2} - i\ketbra{2}{1}
\end{equation}

Let us recall that in the example of Time evlution we saw that this is a self-adjoint operator, therefore we can use it as a projective measurement, and the way to do it is by obtaining its spectral decomposition through the eigenvalues and eigenvectors that we already calculated. That is

\begin{equation*}
    \begin{split}
        \varepsilon_1 = 1 \qquad&\qquad \ket{\varepsilon_1} = \ket{0} \\
        \varepsilon_2 = 1 \qquad&\qquad \ket{\varepsilon_2} = \frac{1}{\sqrt{2}}(\ket{1} + i\ket{2}) \\
        \varepsilon_3 = -1 \qquad&\qquad \ket{\varepsilon_3} = \frac{1}{\sqrt{2}}(\ket{1} - i\ket{2})
    \end{split}
\end{equation*}

Note that we only have two different eigenvalues: $1$ and $-1$. The eigenvalue $1$ has multiplicity $2$ and therefore has  associated a subspace of dimension $2$, while the eigenvalue $-1$ has multiplicity $1$ and therefore has associated a subspace of dimension $1$.

Thus

\begin{equation*}
    \hat{H} = 1\cdot\hat{P_1} + (-1)\cdot\hat{P_{-1}}
\end{equation*}

Where, from Definition 3.6

\begin{equation*}
    \begin{split}
        \hat{P_1} &= \ketbra{\varepsilon_1}{\varepsilon_1} + \ketbra{\varepsilon_2}{\varepsilon_2} \\
        \hat{P_{-1}} &= \ketbra{\varepsilon_3}{\varepsilon_3}
    \end{split}
\end{equation*}

Therefore

\begin{equation*}
    \hat{H} = \ketbra{\varepsilon_1}{\varepsilon_1} + \ketbra{\varepsilon_2}{\varepsilon_2} - \ketbra{\varepsilon_3}{\varepsilon_3}
\end{equation*}

Something that may not be so clear from this result is that in $\hat{H} = \ketbra{0} + i\ketbra{1}{2} - i\ketbra{2}{1}$ we have $\hat{H}$ expressed in terms of the base $\set{\ket{0}, \ket{1}, \ket{2}}$ and what the spectral decomposition is doing is diagonalize the operator $\hat{H}$, since we are expressing it in terms of its eigenvectors and that what turns out to be is a diagonal matrix, in this case

\begin{equation*}
    \hat{H} = \begin{pmatrix}
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & -1
    \end{pmatrix}
\end{equation*}

## Measurement Related Postulates

This formalism of projective measurements allows us on the one hand to group the postulates of quantum mechanics related to measurement in a single formalism, but on the other hand, it also allows us to focus on the state that we want to measure and on the state of the system after the measurement. Let us recall that the postulates of quantum mechanics related to measurement focus on the value that we can measure, that is, on the eigenvalue of an observable that is related to a measurable physical quantity. This formalism allows us to focus on the state in which the vector (that we originally had) ends up after measurement, and so to speak, to put aside for a bit what we are measuring.

In the following proposition we are going to describe the postulates related to the measurement that we already mentioned, but in a more condensed way in two quantities.

### Proposition 3.8

Let $\hat{M} = \sum_m m\hat{P_m}$ be a projective measurement expressed in terms of its spectral decomposition. ($\hat{M}$ can be an observable)

1. If the system is in the state $\ket{\psi}$, the probability of measuring the eigenvalue $m$ is given by

\begin{equation*}
    P_\psi(m) = \bra{\psi}\hat{P_m}\ket{\psi}
\end{equation*}

2. The state of the system immediately after measuring the eigenvalue $m$ is given by

\begin{equation*}
    \ket{\psi} \to \frac{\hat{P_m}\ket{\psi}}{\sqrt{P(m)}}
\end{equation*}

**Proof:**

1. Let's verify the first statement by calculating the expected value. Recall that by Definition 3.6, the $m$ projector applied to $\ket{\psi}$ is given by

\begin{equation*}
    \hat{P_m}\ket{\psi} = \sum_{i=1}^{g_m} \ket{m_i}\braket{m_i}{\psi}
\end{equation*}

where $g_m$ is the multiplicity of the eigenvalue $m$. Thus

\begin{equation*}
    \begin{split}
        \bra{\psi}\hat{P_m}\ket{\psi} &= \bra{\psi} \sum_{i=1}^{g_m} \ket{m_i}\braket{m_i}{\psi} = \sum_{i=1}^{g_m} \braket{\psi}{m_i}\braket{m_i}{\psi} \\
        &= \sum_{i=1}^{g_m} \braket{m_i}{\psi}^*\braket{m_i}{\psi} = \sum_{i=1}^{g_m} \abs{\braket{m_i}{\psi}}^2 \\
        &= P_\psi(m)
    \end{split}
\end{equation*}

This last equality is given by Postulate V.

2. Let's remember that projecting the vector can change its norm and therefore we need to renormalize it. Let us then calculate the magnitude of the projection, calculating the internal product of the projection with itself. In the previous section we gave the expression of the $m$ projector applied to $\ket{\psi}$, let's see now that

\begin{equation*}
    \bra{\psi}\hat{P_m}^* = \sum_{i=1}^{g_m} \braket{\psi}{m_i}\bra{\psi}
\end{equation*}

Thus

\begin{equation*}
    \begin{split}
        \abs{\hat{P_m}\ket{\psi}}^2 &= \bra{\psi}\hat{P_m}^* \hat{P_m}\ket{\psi} \\
        &= \sum_{i=1}^{g_m} \braket{\psi}{m_i}\bra{\psi} \sum_{i=1}^{g_m} \ket{m_i}\braket{m_i}{\psi} \\
        &= \sum_{i=1}^{g_m} \braket{\psi}{m_i} \braket{m_i} \braket{m_i}{\psi} \\
        &= \sum_{i=1}^{g_m} \braket{\psi}{m_i}\braket{m_i}{\psi} \\
        &= \sum_{i=1}^{g_m} \braket{m_i}{\psi}^*\braket{m_i}{\psi} \\
        &= \sum_{i=1}^{g_m} \abs{\braket{m_i}{\psi}}^2 \\
        &= P_\psi(m) \\
        \implies \\
        \abs{\hat{P_m}\ket{\psi}} &= \sqrt{P_\psi(m)}
    \end{split}
\end{equation*}

### Remark

In summary, with this projector formalism, we can express the measurement-related postulates in two simpler expressions:

**1. The probability of measuring an eigenvalue is the expected value of the projector associated with the eigenvalue.**
**2. The state of the system after measurement is obtained by applying the projector to the state and renormalizing. The normalization constant is precisely the square root of the probability, calculated in the previous section.**

As we have already mentioned, this formalism is useful when we are not so interested in what we are going to measure, but rather we are interested in the state of the system after measurement. That is, instead of verifying what is the observable, calculate the eigenvalues of the observable, calculate the probability based on the eigenvectors associated with these eigenvalues, etc. All this, that is required by the quantum mechanics postulates related to measurement in order to find a probability, is already implicit in the projector formalism.

On the other hand, when we talk about measurements in quantum computing, we usually refer to measurements in the computational basis, and the computational basis is the basis of the Pauli operator $\hat{\sigma_z}$. So the observable that is almost always used in quantum computing is $\hat{\sigma_z}$, that is, when talking about measuring a qubit, we are talking about measuring the observable $\hat{\sigma_z}$ and calculating the probability to find the eigenvalue $+1$ or the eigenvalue $-1$ of $\hat{\sigma_z}$. The eigenvalue $+1$ is associated with the qubit $\ket{0}$ and the eigenvalue $-1$ is associated with the qubit $\ket{1}$.

Observables are very useful when we are interested in measuring magnitudes that have a physical interpretation such as momentum, position or energy, on the other hand, in quantum computing we are interested in knowing if when the measurement is made the system will be in a state $\ket{0}$ or in a state $\ket{1}$, beyond the measured eigenvalue or the observable with which you are working. It is for this reason that this formalism of projective measurements is particularly useful in this area.

Let's see how we can apply it in a concrete example.

### Example

Let $\ket{\psi}$ be the state

\begin{equation*}
    \ket{\psi} = \sqrt{\frac{3}{8}}\ket{00} + \frac{1}{2}\ket{01} + \frac{1}{2}\ket{10} + \frac{1}{\sqrt{8}}\ket{11} \in \mathbb{B}^{\otimes2}
\end{equation*}

We note that the state is normalized, that is, it is a valid state in $\mathbb{B}^{\otimes2}$ and thus we can answer the following questions.

**1. What is the probability of finding the first qubit in $\ket{0}$?**

To emphasize the previous comments, let's start by considering the following projective measurement, which corresponds to the expression in terms of outer products of the Pauli operator $\hat{\sigma_x}$

\begin{equation*}
    \hat{M} = (1)\hat{P_0} + (-1)\hat{P_1} \enspace \text{ where } \enspace \hat{P_0} = \ketbra{0}{0}, \enspace \hat{P_1} = \ketbra{1}{1}
\end{equation*}

We know that to answers this question we need to find the probability of measuring the eigenvalue of $\hat{M}$ associated with the qubit $\ket{0}$, but note that according to section **1.** of the previous remark, what is relevant to make this calculation is only $\hat{P_0}$. That is, we are not interested in the eigenvalue that is measured, nor the observable. To accentuate this fact, we could even have considered any other projective measurement such as

\begin{equation*}
    \hat{M} = \alpha\hat{P_0} + \beta\hat{P_1}
\end{equation*}

and this would still be a self-adjoint operator and therefore a valid projective measure, for all $\alpha,\beta \in \mathbb{R}$.

For all this to agree with the formalism of the postulates of quantum mechanics, we usually take $\hat{M} = \hat{\sigma_z}$ as we did initially, that is, formally from a physical point of view, what we will do is measure the observable $\hat{\sigma_z}$. However, from a mathematical point of view we can measure any projective measurement (self-adjoint operator) that distinguishes with a certain eigenvalue the qubit $\ket{0}$ and with a different eigenvalue the qubit $\ket{1}$.

In summary, what is really relevant for this calculation is the projector of the eigenvalue associated with the state we want to measure, in this case what we want to calculate is $\bra{\psi}\hat{P_0}\ket{\psi}$, except that we are working on $\mathbb{B}^{\otimes2}$, but that detail will be clarified below.

According to section **1.** of the previous remark, to calculate this probability, we must calculate the expected value of a projector, but we cannot simply consider $\hat{P_0}$ because of the fact that we just mentioned, that we are working on $\mathbb{B}^{\otimes2}$. Since in this case the state of the second qubit is not relevant, what we need is the following

\begin{equation*}
    \begin{split}
        &\bra{\psi}\hat{P_0}\otimes\hat{I}\ket{\psi} = \\
        &= \bra{\psi} \left[\ketbra{0}\otimes\hat{I}\left(\sqrt{\frac{3}{8}}\ket{00} + \frac{1}{2}\ket{01} + \frac{1}{2}\ket{10} + \frac{1}{\sqrt{8}}\ket{11} \right) \right] \\
        &= \bra{\psi} \left[ \sqrt{\frac{3}{8}} \ketbra{0}\otimes\hat{I}\ket{00} + \frac{1}{2} \ketbra{0}\otimes\hat{I}\ket{01} + \frac{1}{2} \ketbra{0}\otimes\hat{I}\ket{10} + \frac{1}{\sqrt{8}} \ketbra{0}\otimes\hat{I}\ket{11} \right] \\
    \end{split}
\end{equation*}

Let us recall from Definition 2.11, that $\hat{A} \otimes \hat{B}(\ket{a} \otimes \ket{b}) = (\hat{A}\ket{a})\otimes(\hat{B}\ket{b})$. This means that we must apply the projector $\ketbra{0}{0}$ to the first qubit and the operator $\hat{I}$ to the second qubit, since its state is not relevant to us. Thus

\begin{equation*}
    \begin{split}
        \bra{\psi}\hat{P_0}\otimes\hat{I}\ket{\psi} &= \bra{\psi} \left( \sqrt{\frac{3}{8}} \ket{00} + \frac{1}{2} \ket{01} \right) \\
        &= \left(\sqrt{\frac{3}{8}}\bra{00} + \frac{1}{2}\bra{01} + \frac{1}{2}\bra{10} + \frac{1}{\sqrt{8}}\bra{11}\right) \left(\sqrt{\frac{3}{8}} \ket{00} + \frac{1}{2} \ket{01}\right) \\
        &= \left(\sqrt{\frac{3}{8}}\right)\left(\sqrt{\frac{3}{8}}\right)\braket{00}{00} + \left(\frac{1}{2}\right)\left(\frac{1}{2}\right)\braket{01}{01} \\
        &= \frac{3}{8} + \frac{1}{4} = \frac{5}{8}
    \end{split}
\end{equation*}

This is congruent with the intuition given by the fact that the amplitudes associated with the states where the first qubit is $\ket{0}$ are $\sqrt{\frac{3}{8}}$ and $\frac{1}{2}$ and to calculate the probability of measuring these states, we take the modulus squared of the amplitudes, which is known as _Born's Rule_.

In summary, formally what we did was calculate the probability of measuring the eigenvalue $+1$ of the observable $\hat{\sigma_z}\otimes\hat{I}$, which is completely congruent with what the postulates tell us. But as we previously highlighted, for practical issues of carrying out this calculation, the only thing that was relevant, was the projector associated with the state we wanted to measure, we do not need to know the observable or the eigenvalue to measure. Which allows us to put aside a bit the formalism that the postulates entail.

**2. What is the status immediately after measurement?**

Section **2.** of the previous remark tells us that

\begin{equation*}
    \begin{split}
        \ket{\psi} \longrightarrow \frac{\hat{P_0}\otimes\hat{I}\ket{\psi}}{\sqrt{P(\ket{0})}} &= \frac{\hat{P_0}\otimes\hat{I}\ket{\psi}}{\sqrt{\frac{5}{8}}} \\
        &= \sqrt{\frac{8}{5}}\hat{P_0}\otimes\hat{I}\ket{\psi} \\
        &= \sqrt{\frac{8}{5}}\left(\sqrt{\frac{3}{8}}\ket{00} + \frac{1}{2}\ket{01}\right) \\
        &= \sqrt{\frac{3}{5}}\ket{00} + \sqrt{\frac{2}{5}}\ket{01}
    \end{split}
\end{equation*}

Where $P(\ket{0})$ is the probability that we just calculated in the first question. Technically it would have to be the probability of measuring the eigenvalue $+1$, but from what we explained previously, we allow ourselves to use this notation.

Note that this new state is the projection of $\ket{\psi}$ over the subspace generated by all the states that have the first qubit in $\ket{0}$, namely $\set{\ket{00}, \ket{01}}$, therefore this condition is also true. On the other hand, we note that the normalization is correct since $\abs{\sqrt{\frac{3}{5}}}^2 + \abs{\sqrt{\frac{2}{5}}}^2 = \frac{3}{5} + \frac{2}{5} = 1$.

**3. What is the probability of measuring some qubit in $\ket{1}$?**

Let us consider the following events

\begin{equation*}
    \begin{split}
        A &= \text{Measure first qubit in} \ket{1} \\
        B &= \text{Measure second qubit in} \ket{1}
    \end{split}
\end{equation*}

Recall from probability theory that

\begin{equation*}
    P(A \cup B) = P(A) + P(B) - P(A \cap B)
\end{equation*}

So what we are looking for is

\begin{equation*}
    \begin{split}
        P(A \cup B) &= \bra{\psi}\hat{P_1}\otimes\hat{I}\ket{\psi} + \bra{\psi}\hat{I}\otimes\hat{P_1}\ket{\psi} - \bra{\psi}\hat{P_1}\otimes\hat{P_1}\ket{\psi} \\
        &= \bra{\psi}\left(\frac{1}{2}\ket{10} + \frac{1}{\sqrt{8}}\ket{11}\right) + \bra{\psi}\left(\frac{1}{2}\ket{01} + \frac{1}{\sqrt{8}}\ket{11}\right) - \bra{\psi}\left(\frac{1}{\sqrt{8}}\ket{11}\right) \\
        &= \frac{1}{2}\braket{\psi}{10} + \frac{1}{\sqrt{8}}\braket{\psi}{11} + \frac{1}{2}\braket{\psi}{01} + \frac{1}{\sqrt{8}}\braket{\psi}{11} - \frac{1}{\sqrt{8}}\braket{\psi}{11} \\
        &= \left(\frac{1}{2}\right)\left(\frac{1}{2}\right) + \left(\frac{1}{\sqrt{8}}\right)\left(\frac{1}{\sqrt{8}}\right) + \left(\frac{1}{2}\right)\left(\frac{1}{2}\right) \\
        &= \frac{1}{4} + \frac{1}{8} + \frac{1}{4} = \frac{5}{8}
    \end{split}
\end{equation*}

Note that the amplitudes of the terms of $\ket{\psi}$ that have some qubit in $\ket{1}$ are precisely $\frac{1}{2}$, $\frac{1}{2}$ and $\frac{1}{\sqrt{8}}$, if we calculate the sum of its squared modules we have exactly

\begin{equation*}
    \abs{\frac{1}{2}}^2 + \abs{\frac{1}{2}}^2 + \abs{\frac{1}{\sqrt{8}}}^2 = \frac{1}{4} + \frac{1}{4} + \frac{1}{8} = \frac{5}{8} = P(AUB)
\end{equation*}

The goal of this section on projective measurement is to highlight that in quantum computing, when we talk about measuring, it is much more practical to ask about the state of the system than the value to be measured, which might not be so relevant in this context. For example, if we have a state $\ket{\psi}$ of three qubits, it is easier to think of calculating the probability of measuring $\ket{010}$ than to think of calculating the probability of measuring a certain eigenvalue of $\hat{\sigma_z}\otimes\hat{\sigma_z}\otimes\hat{\sigma_z}$, which is actually what we are doing in the background but without the formalism of the postulates of quantum mechanics. We can say that in quantum computing the state of the system (the qubits themselves) is more relevant than the eigenvalues obtained from measuring $\hat{\sigma_z}$.

It is important to note that in quantum computing, measurement can also be part of an algorithm. When this topic is addressed, it will be clear that many times a measurement is made to project to a certain state that is being sought and continue with the algorithm from that new state. Therefore, being able to know the state of a system after a certain measurement turns out to be very relevant.

### Remark 3.9

1. Non-orthogonal states cannot be reliably distinguished by a projective measurement.

Let us remember that if we have a certain state $\ket{\psi}$, we measure an observable (self-adjoint operator) and obtain an eigenvalue of that measurement, what the postulates of quantum mechanics regarding the measurement tell us, is that this state $\ket{\psi}$ will be projected on the subspace associated with the measured eigenvalue. In terms of the previous section, this means applying a projector to the $\ket{\psi}$ state.

What do we mean by reliably distinguish them? By using a projective measurement we can measure one of them with  probability $1$, and measure the other with probability $0$. For example, if we wanted to distinguish if we have the state $\ket{\varphi}$ and not state $\ket{\psi}$, we would simply want to measure the expected value of the projector $\hat{P_\varphi}$ in $\ket{\varphi}$ and get $1$ and in turn get $0$ by measuring it in $\ket{\psi}$. Let's see why we can't make this reliable distinction with two states that are not orthogonal using an example.

Let us consider the following non-orthogonal states

\begin{equation*}
    \ket{\psi} = \ket{0} \enspace \text{ y } \enspace \ket{\varphi} = \frac{1}{\sqrt{2}}(\ket{0} + \ket{1})
\end{equation*}

And the projector

\begin{equation*}
    \hat{P_\psi} = \ketbra{\psi}{\psi}
\end{equation*}

Thus we have

\begin{equation*}
    \begin{split}
        P(\ket{\psi}) &= \bra{\psi}\hat{P_\psi}\ket{\psi} = \braket{\psi}{\psi}\braket{\psi}{\psi} = 1 \\
        P(\ket{\varphi}) &= \bra{\varphi}\hat{P_\psi}\ket{\varphi} = \braket{\varphi}{\psi}\braket{\psi}{\varphi} = \frac{1}{\sqrt{2}}\frac{1}{\sqrt{2}} = \frac{1}{2}
    \end{split}
\end{equation*}

It should be clear from this particular example that we cannot have a projector that allows us to reliably distinguish two non-orthogonal states.

2. Orthogonal states can be reliably distinguished by a projective measurement.

Let's consider the states

\begin{equation*}
    \ket{\psi} = \ket{0} \enspace \text{ and } \enspace \ket{\varphi} = \ket{1}
\end{equation*}

\begin{equation*}
    \begin{split}
        \hat{P_\psi} &= \ketbra{\psi}{\psi} \\
        \implies \\
        P(\ket{\psi}) &= \bra{\psi}\hat{P_\psi}\ket{\psi} = \braket{\psi}{\psi}\braket{\psi}{\psi} = 1 \\
        P(\ket{\varphi}) &= \bra{\varphi}\hat{P_\psi}\ket{\varphi} = \braket{\varphi}{\psi}\braket{\psi}{\varphi} = 0
    \end{split}
\end{equation*}

It should be clear from this particular example that we can reliably distinguish orthogonal states.


# Data Analysis with Jupyter Notebooks.

# Tutorial 5

Benjamin J. Morgan, University of Bath.

# Contents

- [Data analysis and statistics with numpy](#data_analysis)
- [Linear regression](#linear_regression)

# Data analysis and statistics with numpy

`numpy` contains a lot of powerful functions for performing simple statistical analysis on our data. For example, consider the set of numbers 1 to 50:

>```python
import numpy as np
a = np.arange(1,51)
a
```

```python

```

To find the minimum and maximum values we can use `np.min()` and `np.max()`

>```python
np.min(a)
```

```python

```

>```python
np.max(a)
```

```python

```

To find the **sum** of all these numbers, we can use `np.sum()`

>```python
np.sum(a)
```

```python

```

The **mean** of a set of numbers is defined as

\begin{equation}
\frac{\sum_i^N x_i}{N}
\end{equation}

which we could calculate with

>```python
np.sum(a) / len(a)
# len(a) returns the length of the array `a`
```

```python

```

or with `np.mean()`

>```python
np.mean( a )
```

```python

```

The **standard deviation**, $\sigma$ quantifies how much the numbers in our set deviate from the mean.

\begin{equation}
\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^N(x_i-\mu)^2}
\end{equation}

where $\mu$ is the mean.

Again, we could write this out in code:

>```python
import math
sigma = math.sqrt( np.sum( ( a - np.mean(a))**2 ) / len(a) )
sigma
```

```python

```

Or use the `np.std()` function

>```python
np.std(a)
```

```python

```

### Linear Regression

Another commonly used data analysis technique is **linear regression**. This is used to calculate the relationship between two data sets, $X$ and $Y$, assuming that this relationship can be described by a straight line

\begin{equation}
y_i = m x_i + c.
\end{equation}

For any real data set, the data points are unlikely to all fall exactly on the same line. Linear regression is the process of calculating the line that “best fits” the given data.

Look at the following snippet, and try to work out what the result will be.

np.random.rand(10) creates an array of 10 random numbers between 0 and 1.

>```python
import matplotlib.pyplot as plt
x = np.arange(1,11)
offset = 2.0
y = x + ( np.random.rand(10) - 0.5 ) * offset
plt.plot( x, y, 'o' )
plt.show()
```

```python

```

You can see this approximately gives the straight line relationship between $y=x$. We can use linear regression to calculate the “best” straight line that describes these data.

There are a number of different ways in Python to calculate a line of best-fit. One of the simplest is to use another module, [`scipy.stats`](https://docs.scipy.org/doc/scipy-0.18.1/reference/stats.html). As you might suspect from the name, `scipy.stats` contains an large set of statistical analysis tools. We want [`linregress()`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.stats.linregress.html#scipy.stats.linregress):

>```python
from scipy.stats import linregress
linregress( x, y )
```

```python

```

You can see the output is complicated, but includes a list of values that includes the slope and the intercept. In fact you can treat the output like a [list](lists), and use indexing to select a specific result.

>```python
linregress( x, y )[0] # use indexing to get the slope
```

```python

```

Another option is to collect all five of the output values at once

>```python
slope, intercept, rvalue, pvalue, stderr = linregress( x, y )
print( "slope =", slope )
print( "intercept =", intercept )
```

```python

```

To plot the best-fit line against the original data, we generate a new data set according to $y=mx+c$, where $m$ and $c$ are set to the `slope` and `intercept`, calculated from `linregress`.

>```python
y_fit = slope * x + intercept
plt.plot( x, y, 'o' )
plt.plot( x, y_fit, '-' )
plt.xlabel( 'x' )
plt.ylabel( 'y' )
plt.title( 'y =', slope, 'x +', intercept )
plt.show()
```

```python

```


# Derivación numérica

Aunque la derivada de una función se puede obtener algorítmicamente de manera analítica, los algoritmos numéricos que usemos pueden depender de muchas derivadas o no pueden acceder a otra cosa más que la función original.

Así, comúnmente utilizaremos una aproximación numérica de la derivada en lugar de el valor real.

## Preludio matemático: series de Taylor

Sea $f:\mathbb{R} \to \mathbb{R}$ diferenciable $k$ veces en un punto $a$. Entonces :

$$
f(x) = f(a) + f'(a)(x-a)+ \frac{f''(a)}{2!} (x-a)^2 + \ldots + \frac{f^{(k)}(a)}{k!}(x-a)^k + R_{k}(x)
$$

Y $R_{k}(x)$ cumple que

\begin{equation}
\lim_{x\to a} \frac{R_{k}(x)}{(x-a)^k} = 0 \quad \left( R_k(x) \sim (x-a)^{k+1}\right)
\label{eq1}
\end{equation}

En forma aproximada, se cumple que:
$$
f(x) \approx f(a) + f'(a)(x-a)+ \frac{f''(a)}{2!} (x-a)^2 + \ldots + \frac{f^{(k)}(a)}{k!}(x-a)^k
$$

### Forma explícita de $R_k(x)$

Utilizando el teorema del punto medio, podemos expresar a $R_k(x)$ como

$$
R_k(x) = \frac{f^{(k+1)}(\xi)}{(k+1)!} (x-a)^{k+1}
$$

Con $\xi$ un número real **fijo** entre $x$ y $a$ **que desconocemos**. Con esta expresión, es claro que se la condición de $\lim_{x\to a} \frac{R_{k}(x)}{(x-a)^k} = 0 $ se cumple.

### Reescribir el teorema:

Sustitución: $x-a = h \implies x = a+h$

$$
f(a+h) = f(a) + f'(a)(h)+ \frac{f''(a)}{2!} (h)^2 + \ldots + \frac{f^{(k)}(a)}{k!}(h)^k + R_{k}(a+h)
$$

Y $R_{k}(a+h)$ cumple que

$$
\lim_{h\to 0} \frac{R_{k}(a+h)}{h^k} = 0 \quad \left( R_k(a+h) \sim h^{k+1}\right)
$$

## ¿Cómo aproximar la derivada?

Sabemos, por definición, que la derivada de $f$ en un punto $a$ es:

$$
f'(a) = \lim_{h\to 0 } \frac{f(a+h) - f(a)}{h}
$$

**No podemos hacer limites en la computadora**. ¿Qué hacemos? Fijamos un valor de $h$ muy pequeño y aproximamos la derivada. Una aproximación de  **diferencias finitas**

## Preludio computacional: funciones como objetos

Aunque en clases pasadas hemos entendido a las funciones como un objeto abstracto, podemos tratarlas como un objeto con un tipo definido arbitrario. Por ahora no podemos explicar profundamente cuál es el tipo de una función, así que solo procederemos a utilizarlas como un objeto: asignar una función a una variable, dársela a otra función como argumento, etc.

```julia
# ejemplo de asignación
function f(x)
    return sin(3*x^2)
end
```

    f (generic function with 1 method)

```julia
println(f(45))
# asigno a la variable `g` la función representada por `f`
g = f
# `g` me imprime el mismo valor que `f` al evaluarla
println(g(45))
```

    -0.7447712875753175
    -0.7447712875753175

```julia
# ejemplo de utilizar una función como argumento de otra función
function evaluarEn3(func)
    return func(3)
end
```

    evaluarEn3 (generic function with 1 method)

```julia
println(evaluarEn3(sin))
println(sin(3))
```

    0.1411200080598672
    0.1411200080598672

```julia
println(evaluarEn3(exp))
println(exp(3))
```

    20.085536923187668
    20.085536923187668

## Derivadades de Funciones unidimensionales

### Primera aproximación: diferencia hacia adelante (forward difference)

Sea $h > 0$, $h << a$, utilizando una serie de Taylor, tenemos que:

$$
f(a+h) = f(a) + f'(a)h + R_{1}(a+h)
$$

Despejo la derivada:

$$
f'(a) = \frac{f(a+h)-f(a)}{h} - \frac{R_{1}(a+h)}{h}
$$

Como desconocemos el valor exacto de $R_{1}(a+h)$, podemos despreciar ese término y así aproximar:

$$
f'(a) \approx \frac{f(a+h)-f(a)}{h}
$$

Entre menor sea el valor de $h$, mejor será la aproximación ya que, por la propiedad del residuo

$$
\lim_{h \to 0} \frac{R_1(a+h)}{h} = 0
$$

Notemos que el error absoluto de nuestra aproximación ($f'(a) - \frac{f(a+h) - f(a)}{h}$) está dado por el término $ \frac{R_1 (a+h)}{h}$, el cuál sabemos que es proporcional a $h$ (ya que $R_1(a+h) \sim h^2$)

## Notación:

En la clase de complejidad, introducimos la notación $\mathcal{O}(f(n))$ para decir que la complejidad temporal o espacial de un algoritmo está acotada superiormente por una función de la forma $C \cdot f(n)$.

En **análisis numérico**, esa misma notación se utiliza para los errores numéricos. En el caso de la diferencia hacia adelante, por la forma del error absoluto, sabemos que este es $\mathcal{O}(h)$

En general, cuando el **error de aproximación** es proporcional a $h^p$, decimos que nuestro algoritmo numérico es de clase $\mathcal{O}(h^p)$. A $h^p$ también se le conoce como **orden del error**

Pueden consultar más información sobre estas notaciones [en este link](https://courses.engr.illinois.edu/cs357/fa2019/references/ref-2-error/)

```julia
using Plots
```

```julia
function primera(x)
    return sin(2*pi*x)
end
```

    primera (generic function with 1 method)

```julia
xs = range(0,stop=1,length=100)
ys = [primera(x) for x in xs]
plot(xs,ys,title="funcion a derivar")
```

Quiero implementar la aproximación
$$
f'(a) = \frac{f(a+h) - f(a)}{h}
$$

```julia
# esta función me calcula la derivada de f en un punto a
# usando la diferencia hacia adelante con un valor de h
function difAdelante(f,a,h)
    return (f(a+h) - f(a)) / h
end
```

    difAdelante (generic function with 1 method)

Sabemos que

$$
primera'(x) = (\sin{2\pi x})' = 2 \pi \cos{2 \pi x}
$$

Podemos comparar el resultado numérico con el analítico

```julia
# derivada analítica de la funcion `primera`
function dprimera(x)
    return 2*pi*cos(2*pi*x)
end
```

    dprimera (generic function with 1 method)

```julia
print("Derivada numérica: ")
println(difAdelante(primera,0,0.01))
print("Derivada analítica: ")
println(dprimera(0))
# ver el error absoluto
print("Error absoluto: ")
println(abs(dprimera(0)-difAdelante(primera,0,0.01)))
```

    Derivada numérica: 6.279051952931337
    Derivada analítica: 6.283185307179586
    Error absoluto: 0.004133354248248899

```julia
# cambiando el valor de $h$ la aproximación cambia
h = 0.001
print("Derivada numérica: ")
println(difAdelante(primera,0,h))
print("Derivada analítica: ")
println(dprimera(0))
# ver el error absoluto
print("Error absoluto: ")
println(abs(dprimera(0)-difAdelante(primera,0,h)))
```

    Derivada numérica: 6.283143965558951
    Derivada analítica: 6.283185307179586
    Error absoluto: 4.134162063529345e-5

## La derivada aproximada como una función

```julia
# ver la derivada como función
xs = range(0,stop=1,length=101)
# grafico la derivada analítica
ys1 = [dprimera(x) for x in xs]
# tomo un $h$ fijo
h = 0.01
# hago la derivada numérica en cada punto de `xs` con ese valor de `h`
ys2 = [difAdelante(primera,x,h) for x in xs]
plot(xs,ys1,label="Valor analítico",title="Derivada")
plot!(xs,ys2,label="Diferencia hacia adelante")
```

### Ejercicio 1

Realiza una gráfica en la que compares la derivada analítica de la función $f(x) = \sin{x} \cdot \cos{x}$ con la obtenida por diferencia hacia adelante. Toma como dominio el intervalo $[-\pi,\pi]$

### Ejercicio 2

Para la función del inciso anterior, fija un valor de $x$, y gráfica, el error absoluto entre la derivada analítica y la numérica como función de $h$ ¿Qué observas? ¿Es lo esperado? Utiliza la escala log-log si es necesario.

#### ¿Qué otras aproximaciones hay?

### Segunda aproximación: diferencia hacia atrás (backwards difference)

Sea $h > 0$, $h << a$

$$
f(a-h) = f(a) + f'(a)(-h) + R_{1}(a-h)
$$

Despejo la derivada:

$$
f'(a) = \frac{f(a-h)-f(a)}{-h} - \frac{R_{1}(a-h)}{-h} = \frac{f(a)-f(a-h)}{h} + \frac{R_{1}(a-h)}{h}
$$

Aproximando de la misma forma que en la diferencia hacia adelante, despreciamos el término con $R_1$ y obtenemos:

$$
f'(a) \approx \frac{f(a)-f(a-h)}{h}
$$

$$
\lim_{h \to 0} \frac{R_1(a-h)}{h} = 0
$$

### Ejercicio 3
Define una función llamada `diferenciaAtras(f,a,h)` que aproxime la derivada de $f$ en a, $f'(a)$, utilizando  una diferencia hacia atras de orden $h$

### Tercera aproximación: diferencia centrada (central difference)

Primero expandimos $f(a+h)$ y $f(a-h)$ en serie de taylor de orden 2:

$$
f(a+h) = f(a) + f'(a) h + \frac{f''(a)}{2!} h^2 + R_2 (a+h)
$$

$$
\begin{split}
f(a-h) &= f(a) + f'(a)(- h) + \frac{f''(a)}{2!} (-h)^2 + R_2 (a-h) \\
&= f(a) - f'(a)h + \frac{f''(a)}{2!} h^2 + R_2 (a-h)
\end{split}
$$

Podemos restar ambas aproximaciones:

$$
\begin{split}
f(a+h) - f(a-h) &= (f(a) - f(a)) + (f'(a)h - (-f'(a) h)) + (\frac{f''(a)}{2!} h^2 - \frac{f''(a)}{2!} h^2) +  (R_2(a+h) - R_2(a-h)) \\
 &=  2 f'(a) h + (R_2(a+h) - R_2(a-h))
\end{split}
$$

Despejo la derivada:

$$
f'(a) = \frac{f(a+h) - f(a-h)}{2h} - \frac{R_2(a+h) - R_2(a-h)}{2h}
$$
Aproximamos igual en que en las otras aproximaciones

$$
f'(a)  \approx \frac{f(a+h) - f(a-h)}{2h}
$$
Mientras que el término despreciado cumple

$$
\lim_{h \to 0} \frac{R_2(a+h) - R_2(a-h)}{2h} = 0
$$

### Ejercicio 4

Define una función llamada `diferenciaCentrada(f,a,h)` que aproxime la derivada de $f$ en a, $f'(a)$, utilizando  una diferencia centrada de orden $h$

### Ejercicio 5

Para las siguientes funciones, realiza dos gráficas: una en la que compares las derivadas obtenidas analíticamente, con diferencia hacia adelante, diferencia hacia atrás y diferencia centrada, Y  otra gráfica en la que analices el error absoluto como función de $h$ para un punto fijo. Puedes escoger el dominio de la primera gráfica

1. $f_1(x) = 4x^3 + 10x^2 - 9$

2. $f_2(x) = e^{-x^2}\cos{(10 \pi x)}$

3. $f_3(x) = \frac{\sin{(x)}}{\sqrt[3]{x}} + \frac{x}{2}$

**Sugerencia:** tus valores de $h$ no deben ser grandes, no deberían exceder $1$

## Derivadas de orden superior

Podemos seguir el mismo esquema para obtener aproximaciones de las derivadas de orden más grande. Esto no es un proceso simple, pero para una segunda derivada podemos hacerlo de manera muy sencilla.

### Ejercicio 6

Suma las expansiones en serie de taylor a orden 2 de $f(a+h)$ y $f(a-h) $ para obtener una expresión para la segunda derivada $f''(a)$. ¿Cuál es su error de aproximación?

### Ejercicio 7

Comprueba el error de aproximación obtenido al analizar el error absoluto en la segunda derivada de la función $\sin{(6x^2+1)}$ en un punto fijo del intervalo $[1,3]$

## Funciones multidimensionales

Podemos también utilizar diferencias finitas para aproximar las derivadas parciales de una función $f:\mathbb{R^n} \to \mathbb{R}$. Hacer expresiones de las derivadas de orden superior se vuelve complicado ya que hay que hacer expansiones de Taylor multidimensionales, pero para las primeras derivadas parciales basta recordar la definición

$$
\frac{\partial f}{\partial x_k} (a_1, \ldots, a_n) = \lim_{h \to 0} \frac{f(a_1, \ldots, a_k + h, \ldots, a_n) - f(a_1, \ldots, a_k, \ldots, a_n)}{h}
$$

### Ejercicio 8

Define una función `devParcial(f,i,A,h)`, donde $f:\mathbb{R^n} \to \mathbb{R}$ es una función multivariada, que toma como argumento un solo arreglo, $A$ es un arreglo de longitud $n$ e $1\leq i \leq n$ es un natural, que te regrese la derivdad parcial $\frac{\partial f}{\partial x_i} (A)$ calculada utilizando una diferencia central de orden $h$. Pruebala con alguna función de tu elección.

### Ejercicio 9

Sea $g:\mathbb{R}^2 \to \mathbb{R}$, obtén una expresión **analítica** (i.e. no deben programar nada) de diferencias finitas para la derivada curzada $\frac{\partial^2 g}{\partial x \partial y}$. ¿Cuál es el orden de su error de aproximación?

**Sugerencia:** Hay dos maneras de obtener la expresión, y para ambas debes suponer que las parciales cruzadas son iguales $\frac{\partial^2 f}{\partial x\partial y} = \frac{\partial^2 f}{\partial y \partial x}$. La manera larga es usar una serie de Taylor Multidimensional y resolver el sistema de ecuaciones que se obtiene. La manera corta es realizar dos aproximaciones simultáneas en cada entrada.

## La visión general:

Las diferencias hacia adelante, hacia atrás y centrada, son aproximaciones malas y un tanto pobres que no se utilizan con extrema frecuencia en la vida real.

Existe una manera de derivar aproximaciones de diferencias finitas arbitrarias que sean $\mathcal{O}(h^p)$ para el $p \in \mathbb{N}$ que yo quiera. Ese método escapa al alcance del curso, pero lo pueden consultar en el libro de LeVeque de la bibliografía.

Existe también una manera de calcular las derivadas de manera exacta, llamada **diferenciación automática**. No veremos ese tema pues requiere de conocimientos más avanzados de programación.

```julia

```