```
import scipy.stats as stats
figsize( 12.5, 4)
```

#Chapter 4
______

##The greatest theorem never told

>  This relatively short chapter focuses on an idea that is always bouncing around our heads, but is rarely made explicit outside books devoted to statistics or Monte Carlo. In fact, we've been used this idea in every example so far.

###The Law of Large Numbers

Let $Z_i$ be samples from some probability distribution. According to *the Law of Large numbers*,  so long as $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=0}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear be examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=0}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=0}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for *any distribution*, minus some pathological examples that only mathematicians have fun with.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size= 100000` poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to it's parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```
figsize( 12.5, 5 )
sample_size = 100000
expected_value = lambda_ = 4.5
poi = stats.poisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi.rvs( lambda_, size = sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
    of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average), \
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend()
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait-- *compute on average*? This simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=0}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

(We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=0}^NZ_i  - 4.5 \; \right)^2 $$

i.e., we consider the average

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \approx D(N) $$ where $N_Y$ is some suitably large number.

```
figsize( 12.5, 4)
N_Y = 250
D_N_results = []
N_array = np.arange( 0, 50000, 2500 )
lambda_ = 4.5
expected_value = 4.5

def D_N( n ):
    Z = poi.rvs( lambda_, size = (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for n in N_array:
    D_N_results.append( D_N(n) )

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " )
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not choosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of converge to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainity so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=0}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probablities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i)$$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probablities using frequencies).

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would of been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distibution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give use *confidence in how unconfident we should be*. The next section deals with this issue.

## Confidence should be proportional to sample size

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: the law is treasure at the end of an infinite rainbow. While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

--------

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If included in the data is an average of some characteristic of each the geographic area, we must be concious of the Law of Large Numbers and how it can *fail* for areas with small populations.

Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 4000. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. Furthermore, we are interested in measuring the average height of individuals per county. Unbeknowst to the us, height does not vary across county, and each individual, regardless of the county he or she is currenly living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = stats.randint( 100, 4000 )
norm = stats.norm( mean_height, scale = std_height )

#generate some artificial population numbers
population = pop_generator.rvs( n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm.rvs( population[i] ).mean()

#where are the extreme populations?
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population vs. average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 4000 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 4000], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1)
```

What do we observe? *Without accounting for population* we run the risk of making an enourmous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error is that the calculated average of the small population is not a good reflection of the true expected value of the popuation (which should be $\mu =150$). The sample size/population size/$N$, whatever you want to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 4000. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 4000, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```
print "Population sizes of 10 'shortest' counties: "
print population[ np.argsort( average_across_county )[:10] ]
print
print "Population sizes of 10 'tallest' counties: "
print population[ np.argsort( -average_across_county )[:10] ]
```

    Population sizes of 10 'shortest' counties:
    [181 168 229 110 156 123 222 154 498 375]

    Population sizes of 10 'tallest' counties:
    [105 114 111 236 373 244 183 278 234 268]

Not at all uniform over 100 to 4000. This is an absolute failure of the Law of Large Numbers.

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivilants). The dataset is from a Kaggle machine learning competition some collegues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the mail-back rate versus block group population:

```
figsize( 12.5, 5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1)

```

The above is a classic phenonmenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simpley, small datasets cannot be processed using the Law of Large Numbers. Compare, again, with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or substracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

Returning to our Census return rates,  we can ask what are the population sizes of block-groups with the most extreme return rates:

```
print "Population sizes of 10 'least-responsible' block-groups: "
print data[ np.argsort( data[:,0] )[:10], 1 ].astype(int)
print
print "Population sizes of 10 'most-responsible' block-groups: "
print data[ np.argsort( data[:,0] )[-10:], 1 ].astype(int)
```

    Population sizes of 10 'least-responsible' block-groups:
    [ 241  447   11    1   13  954    2  257 1058 1187]

    Population sizes of 10 'most-responsible' block-groups:
    [ 452 3764   10    1   47    6    1   21 2380  550]

Again, we see the numbers are biased towards the left.

For further reading on the hidden dangers of the Law of Large Numbers, I would recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

#### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

By becoming Bayesian experts, we can avoid two traps of the Law of Large Numbers.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally as accurate?

```
## Enter code here
exp = stats.expon( 4 )
N = 1e5
X = exp.rvs( N )
## ...
```

2. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by there percent of non-misses. What mistake have the researchers made?

-----

### Table 3: Kicker Careers Ranked by Make Percentage

Rank 	Kicker 	Make % 	Number  of Kicks

1 	Garrett Hartley 	87.7 	57

2	 Matt Stover 	86.8 	335

3 	Robbie Gould 	86.2 	224

4 	Rob Bironas 	86.1 	223

5	 Shayne Graham 	85.4 	254

… 	… 	…

51	 Dave Rayner 	72.2 	90

52	 Nick Novak 	71.9 	64

53 	Tim Seder 	71.0 	62

54 	Jose Cortez 	70.7	 75

55 	Wade Richey 	66.1	 56

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. Web. 20 Feb. 2013.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

```

```


#Electrodiffusion of K+ through small, tortuous extracellular spaces in the presence of K+ sources and sinks
##Application of a linearized version of the Nernst-Plank equation

#Problem definition

The brain is comprised of millions of neurons that incenssantly produce cognitive computations. Unlike in the cables and chips of man-made circuits, the electric signals underlying these computations are mediated by the flux of of ions such as K+. The number of ions required to carry electric currents is minute. However, in some cases, neurons are so densely packed that the extracellular concentration of K+ ([K]o, in mol cm-3) is prone to change, making the regulation of [K]o a widespread challenge throughout the brain.

This script calculates [K]o as a function of position in a two-dimentional space in the presence of K+ sources and sinks and in the presence of an extracellular electric field. The script utilizes the numerical method of lines to calculate the rate of change in [K]o(x) (in mol cm -3 s-1), as follows.

The [continuity equation](https://en.wikipedia.org/wiki/Continuity_equation) dictates that, if the lateral flux of K+ (JKelectrodiffusion (x) , in mol cm-2 s-1) is not in balance with the sources or sinks of K+ (Qs, in mol cm-3 s-1), then [K]o (x) must be changing in time:
$
\begin{align}
\frac{\partial [K]_o(x)}{\partial t} &= -\frac{\partial J^K_{electrodiffusion}(x)}{\partial x} + Q_s(x) \tag{1}
\end{align}$

The [Nernst-plank equation](https://en.wikipedia.org/wiki/Nernst%E2%80%93Planck_equation) describes the flux of ions under the influence of both an ionic concentration gradient ($\frac{\partial [K]_o}{\partial x}$ in mol cm -2) and an electric field ($\frac{\partial Vo}{\partial x}$, in V cm -1)

$
\begin{align}
 J^K_{electrodiffusion}(x) =- D^* \bigg[\frac{\partial [K]_o}{\partial x} + r [K]_o \frac{\partial Vo}{\partial x} \bigg] \tag{2}
 \end{align}$

#Problem definition

```
#@title Libraries {display-mode: "form"}
import numpy as np
import scipy.integrate as integrate
from numpy.core._multiarray_umath import ndarray
from scipy.integrate import odeint
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from matplotlib import rc #This is to display the animation
from scipy.integrate import odeint
from scipy.integrate import solve_ivp
```

```
#@title Geometry and time {display-mode: "form"}
#@markdown Total length [um]:
length_x =   70#@param{type:"number"}
#@markdown delta x [um]:
delta_x = 0.1 #@param {type:"number"}
x_steps = round(length_x / delta_x)

#@markdown total time [ms]
tf = 200  #@param {type:"number"}
#@markdown delta t [ms]
delta_t = 1.  #@param {type:"number"}
nout = round(tf / delta_t)

```

```
#@title Physical properties {display-mode: "form"}
#@markdown #Tortuosity factor
lambda_ = 1.2 #@param {type:"number"}
#@markdown #Alpha factor
alpha_ =  1#@param {type:"number"}
#@markdown #Difussion coefficient of K in ringers [um^2 s-1]
Diffusion_coefficient = 2000  #@param {type:"number"}

#@markdown Initial [K]o [mM] at x=Ko_from [um]
Ko_left =  3#@param {type:"number"}
Ko_from =  50#@param {type:"number"}
#@markdown Initial [K]o [mM] at  x=Ko_to [um]
Ko_right = 3. #@param {type:"number"}
Ko_to =  51#@param {type:"number"}

#@markdown #Sources and sinks  <br/>
#@markdown magnitude [mM s-1]
source =   0#@param {type:"number"}
sink =  0#@param {type:"number"}

#@markdown location [um]
source_start = 0  #@param {type:"number"}
source_end = 20  #@param {type:"number"}
sink_start =20  #@param {type:"number"}
sink_end = 30  #@param {type:"number"}

#@markdown #Voltage gradient  <br/>
#@markdown Magnitude at x=0 [mV]
V_start =  0#@param {type:"number"}
#@markdown Magnitude at x=l [mV]
V_end =  1#@param {type:"number"}

Faraday = 96485.33289  # [C mol - 1]
R_ = 8.314  # [J K−1 mol−1] universal gas constant
T = 310.15  # [K] 37 C in Kelvin
r = (Faraday / (R_ * T))  # [C / J]
```

#Problem instantiation

```
#@title Initial conditions
Initial_ko_x = np.zeros(x_steps)
Vo_x = np.zeros(x_steps)
SRS_alpha = np.zeros(x_steps)
SRS_tortuosity = np.zeros(x_steps)
D_x = np.zeros(x_steps)
source_x = np.zeros(x_steps)
sink_x = np.zeros(x_steps)
x_dim = np.linspace(0, length_x, x_steps)

# Parametrize initial concentration
# n=((Ko_to-Ko_from)/delta_x)
# initial_concentration = np.linspace(Ko_left,Ko_right, int(n))  # [mM]
# Initial_ko_x[Ko_from:Ko_to] = initial_concentration
Initial_ko_x[int(Ko_from/delta_x):int(Ko_to/delta_x)] = 20

# Parametrize SRS alpha
SRS_alpha.fill(alpha_)

# Parametrize tortuosity
SRS_tortuosity.fill(lambda_)

# Calculate effective diffusion coefficient
D_x[:] = Diffusion_coefficient / SRS_tortuosity[:] ** 2

# Parametrize Vo_x
Vo_x[:] = np.linspace(V_start,V_end, x_steps) #[mV]

# Calculate dVodx
dVodx = np.gradient(Vo_x, delta_x, edge_order=2) #[mV um-1]

# Parametrize sources and sinks
source_x[round(source_start/delta_x):round(source_end/delta_x)] = source  # [mM s-1]
# print(source_x)
sink_x[round(sink_start/delta_x):round(sink_end/delta_x)] = sink  # [mM s-1]

#plot initial conditions
fig = plt.figure(figsize=(10,15))
#Initial Ko concentration
Ax0 = fig.add_subplot(3, 2, 1)
Ax0.plot(x_dim,Initial_ko_x,color="k")
Ax0.set_title("Initial [K]o profile")
Ax0.set_ylabel("Concentration (mM)")
#Alpha
Ax1 = fig.add_subplot(3, 2, 2)
color = 'tab:red'
Ax1.set_title("Alpha and lambda")
Ax1.set_ylabel("Alpha", color=color)
Ax1.plot(x_dim,SRS_alpha,color=color)
Ax1.tick_params(axis='y', labelcolor=color)
#Lambda
Ax2 = Ax1.twinx()  # instantiate a second axes that shares the same x-axis
color = 'tab:blue'
Ax2.set_ylabel('Tortuosity factor', color=color)  # we already handled the x-label with ax1
Ax2.plot(x_dim,SRS_tortuosity, color=color)
Ax2.tick_params(axis='y', labelcolor=color)
# Ax1.legend(loc='best')
#Effective diffusion coefficient
Ax3 = fig.add_subplot(3, 2, 3)
Ax3.plot(x_dim,D_x,color="k")
Ax3.set_title("Effective diffusion coefficient")
Ax3.set_ylabel("D* (um^2 s-1)")
#Effective diffusion coefficient
Ax4 = fig.add_subplot(3, 2, 4)
Ax4.plot(x_dim,Vo_x,color="k")
Ax4.set_title("Extracellular voltage")
Ax4.set_ylabel("Voltage (mV)")
#Sources and sinks
#Source
Ax5 = fig.add_subplot(3, 2, 5)
color = 'tab:red'
Ax5.set_title("K+ sources and sinks")
Ax5.set_ylabel("Source (mM s-1)", color=color)
Ax5.set_ylim(sink-1,source+1)
Ax5.plot(x_dim,source_x,color=color)
Ax5.tick_params(axis='y', labelcolor=color)
#Lambda
Ax6 = Ax5.twinx()  # instantiate a second axes that shares the same x-axis
color = 'tab:blue'
Ax6.set_ylabel('Sink (mM s-1)', color=color)  # we already handled the x-label with ax1
Ax6.set_ylim(sink-1,source+1)
Ax6.plot(x_dim,sink_x, color=color)
Ax6.tick_params(axis='y', labelcolor=color)

Ax4.set_xlabel("Distance (um)")

plt.subplots_adjust(hspace=0.5)
```

```
#@title Derivative taker
def simple_electrodif(t, u):
    Faraday = 96485.33289  # [C mol - 1]
    R_ = 8.314  # [J K−1 mol−1] universal gas constant
    T = 310.15  # [K] 37 C in Kelvin
    r = (Faraday / (R_ * T))  # [C / J]

    C_x = np.zeros(x_steps)
    Jk_diffusive = np.zeros(x_steps)
    Jk_electrophoretic = np.zeros(x_steps)
    Jk = np.zeros(x_steps)
    dydt = np.zeros(x_steps)

    C_x[:] = u[:] * 1e-15  # [mMoles um-3]
    dCdx = np.gradient(C_x, delta_x, edge_order=2) # [mMoles um-4]

    # No flux boundary conditions
    dCdx[0] = 0
    dCdx[len(dCdx) - 1] = 0

    # Calculate lateral diffusive term
    Jk_diffusive[:] = - D_x * dCdx  # [mMoles um-2 s-1]

    # Calculate electric term
    Jk_electrophoretic[:] = -D_x * r * C_x * dVodx * 1e-3  # [mMoles um-2 s-1]

    # Net electrophoretic flux
    Jk[:] = Jk_diffusive[:] + Jk_electrophoretic[:]  # [mMoles um-2 s-1]

    # Change in concentration due to lateral flux
    dCdt_electrophoretic = np.gradient(Jk, delta_x, edge_order=2) * 1e15  # [mMolar s-1]

    #Boundary conditions (Wrong?)
    # dCdt_electrophoretic[0] = 0
    # dCdt_electrophoretic[len(dCdt_electrophoretic) - 1] = 0

    dydt[:] = (source_x[:] + sink_x[:] - dCdt_electrophoretic[:]) * 1e-3 # [mMolar ms-1]

    return dydt

```

```
#Test 0th derivative
test = simple_electrodif(0, u=Initial_ko_x)
plt.plot(test)
plt.ylim((np.amin(test)-np.amin(test)*0.1),(np.amax(test)+np.amax(test)*0.1))
# print(test)
```

```
#@title Call the derivative taker
tout=np.linspace(0,nout,tf)

# out,infodict=odeint(pde4Sa,u0,tout,tfirst=True,full_output=True)
out_=solve_ivp(simple_electrodif,(0,tf),Initial_ko_x,t_eval=tout,vectorized=False,method='BDF', rtol=1e-6,atol=1e-6)

out=out_.y.T
```

```
result, Ax = plt.subplots()
Ax.plot(x_dim,out.T, color="k")
Ax.plot(x_dim,out.T[:, nout - 1], color="g")
plt.show()
```

```
# #@title Older integrator

# call the integrator
# out: ndarray = np.zeros((t_steps, x_steps))
# print("shape of out: "+str(out.shape))

t_out = np.linspace(0, tf, nout)
out = odeint(simple_electrodif, Initial_ko_x, t_out, tfirst=True)
out=np.transpose(out)
# print("shape of out: "+str(odeint(simple_electrodif, Initial_Ko_x, t_out, tfirst=True).shape))

result, Ax = plt.subplots()
Ax.plot(x_dim,out, color="k")
Ax.plot(x_dim,out[:, nout - 1], color="g")
plt.show()

```

```
#@title Make animation
from IPython.display import HTML, Image # For GIF
# rc('animation', html='jshtml')#setup the animation display
rc('animation', html='html5')

# # Set up formatting for the movie files
# Writer = animation.writers['ffmpeg']
# writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)

# create animation
k=0
fig = plt.figure(figsize=(4,3))
fig.set_dpi(100)
axis = plt.axes(xlim =(0,length_x),
                ylim =(0,20))
line, = axis.plot([], [], lw = 1)

axis.set_ylabel('concentration (mM)')
axis.set_xlabel('Distance (um)')
axis.grid(True)

def init():
    line.set_data([], [])
    return line,

# initializing empty values
# for x and y co-ordinates
xdata, ydata = [], []

# animation function
def animate(i):

    # t is a parameter which varies
    # with the frame number
    k = int(delta_t * i)

    line.set_data(x_dim,out[k])
    axis.text(3,15,'time: '+str(round(tout[k],2))+' ms',
              bbox=dict(facecolor='white',edgecolor='white'),fontsize=10)
    return line,

anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=len(tout-1), interval=20,
                               blit=True, repeat=True)

# anim = animation.FuncAnimation(fig,animate,frames=len(tout), blit=False, repeat=True)#,interval=20
#Save as GIF
# anim.save('test.gif', writer='pillow', fps=60)

# anim.save('basic_animation.mp4', fps=30, extra_args=['-vcodec', 'libx264'])
# Image(url='test.gif')# anim
# try:
#     anim.save('test.mp4',fps=10)
# except:
#     print('requires ffmpeg to save mp4 file')
#     plt.show()

# plt.show()
anim
```


# INF-510, v0.32, Claudio Torres, ctorres@inf.utfsm.cl. DI-UTFSM
## Textbook: Lloyd N. Trefethen, Spectral Methods in MATLAB, SIAM, Philadelphia, 2000

# Spectral differentiation++

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import scipy.sparse.linalg as sp
# We are adding all sympy just to reduce the length of the expressions used.
# For instance, instead of using sym.exp we just use
from sympy import *
import sympy as sym
from scipy.linalg import toeplitz
import ipywidgets as widgets
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
sym.init_printing()

def plot_matrices_with_values(ax,M):
    N=M.shape[0]
    cmap = plt.get_cmap('GnBu')
    ax.matshow(M, cmap=cmap)
    for i in np.arange(0, N):
        for j in np.arange(0, N):
            ax.text(i, j, '{:.2f}'.format(M[i,j]), va='center', ha='center', color='r')
```

Defining some symbolic variables

```python
N = 10
y = sym.Symbol('y', real=True)
f = exp(sin(pi*y))
fv = sym.lambdify([y], f, 'numpy')
fp = sym.diff(f)
fpv = sym.lambdify([y], fp, 'numpy')
fp2 = sym.diff(fp)
fp2v = sym.lambdify([y], fp2, 'numpy')
# We remove the first point since we consider periodic boundary conditions
x = np.linspace(-1,1,N+1)[1:]
xx = np.linspace(-1,1,10*N)
u = fv(x)
```

```python
# The variable M is used for changing the default size of the figures
M=8
plt.figure(figsize=(M,M))
# This plots the "continuous" function
plt.plot(xx,fv(xx),'-',label='"continuous" function')
# This plots the "discrete" function
plt.plot(x,u,'k.',markersize=20,label='"discrete" function')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.grid(True)
plt.legend(loc=2)
plt.show()
```

## My First Differentiation Matrix

```python
def my_D(N):
    h=2./(N)
    c=np.zeros(N)
    c[1]=-1
    c[-1]=1
    D=toeplitz(c/(2.*h),r=-c/(2.*h))
    return D
def show_my_D(N):
    D=my_D(N)
    h = 2./N

    # We multiply by h just to cancel out its dependency
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(2*M,M))
    # First derivative
    plot_matrices_with_values(ax1,D*h)
    # 'Second' derivative
    plot_matrices_with_values(ax2,np.dot(D,D)*(h)**2)
    plt.show()

widgets.interact(show_my_D,N=(2,10))
```

## Let's differentiate our discrete function!

```python
def show_discrete_derivate(N=10):
    D=my_D(N)
    D2=np.dot(D,D)
    x = np.linspace(-1,1,N+1)[1:]
    xx = np.linspace(-1,1,10*N)
    u = fv(x)
    plt.figure(figsize=(M,M))
    # This is the "exact" derivative
    plt.plot(xx,fpv(xx),'-',label='"exact" derivative')
    # Here we differentiate our "discrete" function
    Up=np.dot(D,u)
    plt.plot(x,Up,'k.',markersize=20,label='"discrete" derivative')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.grid(True)
    plt.legend(loc='best')
    plt.title('Is it a good approximation?')
    plt.show()
widgets.interact(show_discrete_derivate,N=(5,100,2))
```

## Let's compute the second derivate!

```python
def show_discrete_2nd_derivate(N=10):
    D=my_D(N)
    D2=np.dot(D,D)
    x = np.linspace(-1,1,N+1)[1:]
    xx = np.linspace(-1,1,10*N)
    u = fv(x)
    plt.figure(figsize=(M,M))
    plt.plot(xx,fp2v(xx),'-',label='"exact" 2nd derivative')
    plt.plot(x,np.dot(D2,u),'k.',markersize=20,label='"discrete" 2nd derivative')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.grid(True)
    plt.legend(loc='best')
    plt.title('Is it a good approximation?')
    plt.show()
widgets.interact(show_discrete_2nd_derivate,N=(5,100,1))
```

## Let's study how fast our approximation converges.

```python
Ne = 10
errors=np.zeros(Ne)
Ns = np.ceil(np.logspace(1,3,Ne))
for i in np.arange(Ne):
    N = int(Ns[i])
    D = my_D(N)
    x = np.linspace(-1,1,N+1)[1:]
    u = fv(x)
    Up = np.dot(D,u)
    up = fpv(x)
    errors[i] = np.linalg.norm(Up-up,np.inf)
plt.figure(figsize=(M,M))
plt.loglog(Ns,errors,'k.',markersize=20,label='2nd order')
plt.loglog(Ns,100*Ns**(-2),'r',markersize=20)
plt.xlabel('N',fontsize=16)
plt.ylabel('error',fontsize=16)
plt.grid(True)
plt.title('Convergence analysis of 2nd order approximation 1st derivative')
plt.legend(loc='best')
plt.show()
```

## Can we find a faster approximation?

```python
def my_D4th(N):
    h=2./N
    c=np.zeros(N)
    c[1]=-2./3.
    c[2]=1./12.
    c[-1]=2./3.
    c[-2]=-1./12.
    D=toeplitz(c/h,r=-c/h)
    return D
def show_my_D4th(N):
    D=my_D4th(N)
    h = 2./N
    # We multiply by h just to cancel out its dependency
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(2*M,M))
    # First derivative
    plot_matrices_with_values(ax1,D*h)
    # 'Second' derivative
    plot_matrices_with_values(ax2,np.dot(D,D)*(h)**2)
    plt.show()
widgets.interact(show_my_D4th,N=(3,10,1))
```

```python
Ne = 30
errors=np.zeros(Ne)
errors4th=np.zeros(Ne)
Ns = np.ceil(np.logspace(1,3.5,Ne))
for i in np.arange(Ne):
    N = int(Ns[i])
    D = my_D(N)
    D4th = my_D4th(N)
    x = np.linspace(-1,1,N+1)[1:]
    u = fv(x)
    Up = np.dot(D,u)
    Up4th = np.dot(D4th,u)
    up = fpv(x)
    errors[i] = np.linalg.norm(Up-up,np.inf)
    errors4th[i] = np.linalg.norm(Up4th-up,np.inf)

plt.figure(figsize=(M,M))
plt.loglog(Ns,errors,'k.',markersize=20,label='2nd order')
plt.loglog(Ns,errors4th,'b.',markersize=20,label='4th order')
plt.loglog(Ns,100*Ns**(-2),'r',markersize=20)
plt.loglog(Ns,2500*Ns**(-4),'g',markersize=20)
plt.xlabel('N',fontsize=16)
plt.ylabel('error',fontsize=16)
plt.grid(True)
plt.legend(loc='best')
plt.show()
```

# Can we find something even better?

```python
def my_D_spec(N):
    h=(2./N)*np.pi
    c=np.zeros(N)
    j=np.arange(1,N)
    c[1:]=0.5*((-1)**j)*(np.tan(j*h/2.)**(-1))
    r=np.zeros(N)
    r[0]=c[0]
    r[1:]=c[-1:0:-1]
    D=np.pi*toeplitz(c,r=r)
    return D
def  show_my_D_spec(N):
    D=my_D_spec(N)
    h = 2./N
    # We multiply by h just to cancel out its dependency
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(2*M,M))
    # First derivative
    plot_matrices_with_values(ax1,D*h)
    # 'Second' derivative
    plot_matrices_with_values(ax2,np.dot(D,D)*(h)**2)
    plt.show()
widgets.interact(show_my_D_spec,N=(3,10,1))
```

```python
Ne = 30
errors=np.zeros(Ne)
errors4th=np.zeros(Ne)
Ns = np.ceil(np.logspace(1,3.5,Ne))
for i in np.arange(Ne):
    N = int(Ns[i])
    D = my_D(N)
    D4th = my_D4th(N)
    x = np.linspace(-1,1,N+1)[1:]
    u = fv(x)
    Up = np.dot(D,u)
    Up4th = np.dot(D4th,u)
    up = fpv(x)
    errors[i] = np.linalg.norm(Up-up,np.inf)
    errors4th[i] = np.linalg.norm(Up4th-up,np.inf)

NsS = np.ceil(np.arange(2,51,2))
NeS= NsS.shape[0]
errorsSpectral=np.zeros(NeS)
for i in np.arange(NeS):
    N = int(NsS[i])
    DSpectral = my_D_spec(N)
    x = np.linspace(-1,1,N+1)[1:]
    u = fv(x)
    UpSpectral = np.dot(DSpectral,u)
    up = fpv(x)
    errorsSpectral[i] = np.linalg.norm(UpSpectral-up,np.inf)

plt.figure(figsize=(M,M))
plt.loglog(Ns,errors,'k.',markersize=20,label='2nd order')
plt.loglog(Ns,errors4th,'b.',markersize=20,label='4th order')
plt.loglog(NsS,errorsSpectral,'m.',markersize=20,label='spectral order')
plt.loglog(Ns,100*Ns**(-2),'r',markersize=20)
plt.loglog(Ns,500*Ns**(-4),'g',markersize=20)
plt.xlabel('N',fontsize=16)
plt.ylabel('error',fontsize=16)
plt.grid(True)
plt.legend(loc='best')
plt.show()
```

# Bonus

```python
x=np.arange(1,3001)
plt.figure(figsize=(M,M))
plt.plot(x,np.sin(x),'k.',markersize=20)
plt.xlabel('x',fontsize=16)
plt.ylabel('$\sin(x)$',fontsize=16)
plt.grid(True)
plt.show()
```

```python
x=np.arange(1,1001)
plt.figure(figsize=(M,M))
plt.plot(x,np.sin(x),'k.',markersize=20)
plt.xlabel('x',fontsize=16)
plt.ylabel('$\sin(x)$',fontsize=16)
plt.grid(True)
plt.show()
```

```python

```


# Objects and Data Structures

** Answer the following questions **

## 1. Numbers

What is the *type* of the result of the expression 3 + 1.5 + 4?

What would you use to find a number’s square root, as well as its square?

```python
# Square root:

```

```python
# Square:

```

##  Creating formulas

Write the following mathematical formula in Python:

\begin{align}
 result = 6a^3 - \frac{8b^2 }{4c} + 11
\end{align}

```python
a = 2
b = 3
c = 2
```

> write your formula as
```python
result =
```

It must verify
```python
>>> assert result == 50
```

## 2. Strings

Given the string 'hello' give an index command that returns 'e'. Enter your code in the cell below:

```python

```

Reverse the string 'hello' using slicing:

```python

```

Given the string hello, give two methods of producing the letter 'o' using indexing.

```python
# Method 1
```

```python
# Method 2:

```

## Fill missing pieces

Fill `____` pieces below to have correct values for `lower_cased`, `stripped` and `stripped_lower_case` variables.

```python
original = ' Python strings are COOL! '
lower_cased = original._____
stripped = ____.strip()
stripped_lower_cased = original._____._____
```

Let's verify that the implementation is correct by running the cell below. `assert` will raise `AssertionError` if the statement is not true.

```python
>>> assert lower_cased == ' python strings are cool! '
>>> assert stripped == 'Python strings are COOL!'
>>> assert stripped_lower_cased == 'python strings are cool!'
```

## Prettify ugly string

Use `str` methods to convert `ugly` to wanted `pretty`.

```python
ugly = ' tiTle of MY new Book\n\n'
```

> Your implementation:
```python
pretty = 'TODO'
```

Let's make sure that it does what we want. `assert` raises [`AssertionError`](https://docs.python.org/3/library/exceptions.html#AssertionError) if the statement is not `True`.

```python
>>> print('pretty: {}'.format(pretty))
>>> assert pretty == 'Title Of My New Book'
```

## Format string based on existing variables

Create `sentence` by using `verb`, `language`, and `punctuation` and any other strings you may need.

```python
verb = 'is'
language = 'Python'
punctuation = '!'
```

> Your implementation:
```python
sentence = 'TODO'
```

```python
>>> print('sentence: {}'.format(sentence))
>>> assert sentence == 'Learning Python is fun!'
```

## 3. Lists

### Build this list [0,0,0] two separate ways.

```python
# Method 1:

```

```python
# Method 2:

```

## Fill the missing pieces

Fill the `____` parts of the code below.

```python
# Let's create an empty list
my_list = ____

# Let's add some values
my_list.____('Python')
my_list.____('is ok')
my_list.____('sometimes')

# Let's remove 'sometimes'
my_list.____('sometimes')

# Let's change the second item
my_list[____] = 'is neat'
```

Let's verify that it's correct
```python
>>> assert my_list == ['Python', 'is neat']
```

```python

```

## Create a new list without modifiying the original one

```python
original = ['I', 'am', 'learning', 'hacking', 'in']
```

> Your implementation here
```python
modified =
```

```python
>>> assert original == ['I', 'am', 'learning', 'hacking', 'in']
>>> assert modified == ['I', 'am', 'learning', 'lists', 'in', 'Python']
```

```python

```

## Create a merged sorted list

```python
list1 = [6, 12, 5]
list2 = [6.2, 0, 14, 1]
list3 = [0.9]
```

> Your implementation here
```python
my_list =
```

```python
>>> assert my_list == [14, 12, 6.2, 6, 5, 1, 0.9, 0]
```

## 4. Dictionaries

#Using keys and indexing, grab the 'hello' from the following dictionaries:

```python
d = {'simple_key':'hello'}
# Grab 'hello'

```

```python
d = {'k1':{'k2':'hello'}}
# Grab 'hello'

```

```python
# Getting a little tricker
d = {'k1':[{'nest_key':['this is deep',['hello']]}]}
#Grab hello

```

## Populating a dictionary

Create a dictionary by using all the given variables.

```python
first_name = 'John'
last_name = 'Doe'
favorite_hobby = 'Python'
sports_hobby = 'gym'
age = 82
```

> Your implementation
```python
my_dict =
```

```python
>>> assert my_dict == {
        'name': 'John Doe',
        'age': 82,
        'hobbies': ['Python', 'gym']
    }
```

```python

```

## Accessing and merging dictionaries

Combine `dict1`, `dict2`, and `dict3` into `my_dict`. In addition, get the value of `special_key` from `my_dict` into a `special_value` variable. Note that original dictionaries should stay untouched and `special_key` should be removed from `my_dict`.

```python
dict1 = dict(key1='This is not that hard', key2='Python is still cool')
dict2 = {'key1': 123, 'special_key': 'secret'}
# This is also a away to initialize a dict (list of tuples)
dict3 = dict([('key2', 456), ('keyX', 'X')])
```

> Your impelementation
```python
my_dict =
special_value =
```

```python
>>> assert my_dict == {'key1': 123, 'key2': 456, 'keyX': 'X'}
>>> assert special_value == 'secret'

>>> # Let's check that the originals are untouched
>>> assert dict1 == {
        'key1': 'This is not that hard',
        'key2': 'Python is still cool'
    }
>>> assert dict2 == {'key1': 123, 'special_key': 'secret'}
>>> assert dict3 == {'key2': 456, 'keyX': 'X'}
```

```python

```

## 5. Tuples

What is the major difference between tuples and lists?

How do you create a tuple?

## 6. Sets

What is unique about a set?

Use a set to find the unique values of the list below:

```python
list5 = [1,2,2,33,4,4,11,22,3,3,2]

```

## 7. Booleans

For the following quiz questions, we will get a preview of comparison operators. In the table below, a=3 and b=4.

Operator	Description	Example

==
If the values of two operands are equal, then the condition becomes true.
 (a == b) is not true.

!=
If values of two operands are not equal, then condition becomes true.
 (a != b) is true.

>
If the value of left operand is greater than the value of right operand, then condition becomes true.
 (a > b) is not true.

<
If the value of left operand is less than the value of right operand, then condition becomes true.
 (a < b) is true.

>=
If the value of left operand is greater than or equal to the value of right operand, then condition becomes true.
 (a >= b) is not true.

<=
If the value of left operand is less than or equal to the value of right operand, then condition becomes true.
 (a <= b) is true.

What will be the resulting Boolean of the following pieces of code (answer fist then check by typing it in!)

```python
# Answer before running cell
2 > 3
```

```python
# Answer before running cell
3 <= 2
```

```python
# Answer before running cell
3 == 2.0
```

```python
# Answer before running cell
3.0 == 3
```

```python
# Answer before running cell
4**0.5 != 2
```

Final Question: What is the boolean output of the cell block below?

```python
# two nested lists
l_one = [1,2,[3,4]]
l_two = [1,2,{'k1':4}]

# True or False?
l_one[2][0] >= l_two[2]['k1']
```


```
from sympy import symbols
from sympy.physics.quantum.trace import Tr
from sympy.matrices.matrices import Matrix
from IPython.core.display import display_pretty
from sympy.printing.latex import *
from sympy.physics.quantum.cartesian import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.density import *

%load_ext sympyprinting

#TODO: Add examples of simple qubit usage
```

## Examples of Tr operations on Qubits

```
q1 = Qubit('10110')
q2 = Qubit('01010')
d = Density( [q1, 0.6], [q2, 0.4] )

# Trace one bit
t = Tr(d,[0])

display_pretty(t.doit())

# Partial trace of 3 qubits
# the 0th bit is the right-most bit
t = Tr(d,[2, 1, 3])
display_pretty(t.doit())
```

    0.4⋅(❘0101⟩, 1) + 0.6⋅(❘1011⟩, 1)

    0.4⋅(❘00⟩, 1) + 0.6⋅(❘10⟩, 1)

## Partial Tr of mixed state

```
from sympy import *
q = (1/sqrt(2)) * (Qubit('00') + Qubit('11'))

d = Density ( [q, 1.0] )
t = Tr(d, [0])
display_pretty(t.doit())
```

    0.5⋅(❘0⟩, 1) + 0.5⋅(❘1⟩, 1)

```

```


```python
import matplotlib.pyplot as plt
# install with `conda install -c conda-forge matplotlib-venn `
from matplotlib_venn import *
import numpy as np
```

$\newcommand\lf[1]{\large\color{CadetBlue}{#1}}
\newcommand\sf[1]{\small\color{grey}{#1}}
\newcommand{\mi}{\mathrm{i}}$

# Complex Numbers
https://texblog.org/2013/11/27/complex-number-symbols-in-latex/

Complex numbers are a 2-dimensional number represented on the complex plane.

1. [Number Set](#set)
2. [Complex (Argand) Plane](#plane)
3. [imaginary Numbers](#imaginary)
3. [Complex Notation](#notation)
4. [Complex Algebra](#lgebra)
5. [Complex Conjugate](#conjugate)
6. [Complex Magnitude](#magnitude)
7. [Complex Exponents](#exponent)

----
## Number Set

**Set of all complex numbers is $\mathbb{C}$**

> Set of **Imaginary** numbers ($\mathbb{I}$) have no Real ($\mathbb{R}$) component.

> Set of **Real** numbers ($\mathbb{R}$) have no Imaginary ($\mathbb{I}$) component.

> Set of **Complex** numbers $\mathbb{C}$ are a superset of both Real ($\mathbb{R}$) and Imaginary ($\mathbb{I}$) sets.

> **0** belongs to all sets

```python
# Define sets
imaginary_set = {'Imaginary', '0'}
real_set = {'Real', '0'}
complex_set = {'Imaginary', 'Real', '0','Complex'}
labels = ('', '', '')
all_sets = [imaginary_set, real_set, complex_set]

# Create diagram
v3 = venn3(all_sets, set_labels = labels)
v3.get_label_by_id('101').set_text('Imaginary')
v3.get_label_by_id('001').set_text('Complex')
v3.get_label_by_id('011').set_text('Real')
v3.get_label_by_id('111').set_text('0')

# Display
#plt.title("Real, Imaginary and Complex Number Sets as a Venn Diagram")
#plt.show()
```

----

## Complex (Argand) Plane

Complex numbers have 2 dimensions - the real component, and imaginary component.

```python
# Simple argand view fuction that plots 0 to a position on the complex plane
def argand(a):
    for x in range(len(a)):
        plt.plot([0,a[x].real],[0,a[x].imag],'ro-',label='python')
    limit=np.max(np.ceil(np.absolute(a))) # set limits for axis

    plt.xlim((-limit,limit))
    plt.ylim((-limit,limit))
    plt.ylabel('Imaginary')
    plt.xlabel('Real')
    plt.show()
```

```python
a = np.arange(5) + 1j*np.arange(6,11)

# Complex Plane
plt.subplot(2,1, 1)
argand(a)

# Polar plot
plt.subplot(2, 1, 2, polar=True)
for x in a:
    plt.polar([0,np.angle(x)],[0,abs(x)],marker='o')

plt.show()
```

----
##  Imaginary Numbers

> $\lf{ i = \sqrt{-1} }$

> $\sf{ i^2 = -1 }$

### Imaginary Exponents

> $\lf{ \large i^1 = i }$ $\sf{ \text{ and any } i^n \text{, where } n\pmod4 = 1 \text{, } n \in \mathbb{Z} }$
>
> $\lf{ \large i^2 = -1 }$ $\sf{ \text{ and any } i^n \text{, where } n\pmod4 = 2 \text{, } n \in \mathbb{Z} }$
>
> $\lf{ \large i^3 = -i }$ $\sf{ \text{ and any } i^n \text{, where } n\pmod4 = 3 \text{, } n \in \mathbb{Z} }$
>
> $\lf{ \large i^4 = 1 }$ $\sf{ \text{ and any } i^n \text{, where } n\pmod4 = 4 \text{, } n \in \mathbb{Z} }$

----
## Complex Notation

### Rectangular Notation
$\lf{ z = a + b\mi }$

#### Real Component ( $\operatorname{Re}(z)$ / $\Re(z)$ )
$\lf{ \Re(z) = a }$

#### Imaginary Component ($\operatorname{Im}(z)$ / $\Im(z)$)
$\lf{ \Im(z) = b }$

### Polar Notation:
$\lf{ z\;\angle\;\theta }$

$\sf{ \text{z is the magnitude or modulus} }$

$\sf{ \theta \text{ is the angle/argument/phase} }$

----
## Complex Algebra
https://byjus.com/complex-number-formula

### Equality of Complex Numbers Formula

> $\lf{ a+b\mi = c + d\mi \iff (a=c) \land (b = d) }$

### Addition

> $\lf{  (a+b\mi) + (c + d\mi) = (a+c) + (b+d)\mi }$

### Subtraction

> $\lf{  (a+b\mi) - (c + d\mi) = (a-c) + (b-d)\mi }$

### Multiplication

> $\lf{  (a+b\mi)d\mi = -bd + ad\mi }$

> $\lf{  (a+b\mi) \times (c + d\mi) = (ac-bd) + (ad+bc)\mi }$

### Complex Reciprocal
*Multiplying by the reciprocal, is the same as dividing*

> $\lf{ \frac{1}{z} = \frac{\bar{z}}{|z|^2} }$
>
> $\sf{ \begin{equation} \label{eq1}
\begin{split}
\frac{1}{z} & = \frac{\bar{z}}{z\bar{z}} \\
& = \frac{a - bi}{a^2 +ab\mi - ab\mi - (bi)^2} \\
& = \frac{a - bi}{a^2 + b^2} \\
& = \frac{\bar{z}}{|z|^2} \\
\end{split}
\end{equation} }$

### Division
*To remove a complex number from the denominator, multiple it by the conjugate*

> $\lf{ \frac{a+b\mi}{c+d\mi} \;=\; \frac{a+b\mi}{c+d\mi} \times \frac{c-d\mi}{c-d\mi} \;=\; \frac{ac+bd}{c^2 + d^2} + \frac{bc-ad}{c^2 + d^2}\mi }$

----
## Complex Conjugate

if $z = a + b\mi$, then it's conjugate is $\lf{ \bar{z} = a - b\mi}$

### Conjugates - Components

> $\lf{ \Re(\bar{z}) = a }$
>
> $\lf{ \Im(\bar{z}) = -b }$

### Multiply with Conjugate
>*Remove Imaginary component with conjugate*
>
>$\lf{ z\bar{z} = |z|^2 }$

>
>$\sf{ \begin{equation} \label{eq1}
\begin{split}
z\bar{z} & = (a+b\mi)(a-b\mi) \\
& = a^2 + b^2 \\
& = |z|^2
\end{split}
\end{equation} }$

### Real & Imaginary Functions
>
> $\lf{ \Re(z) \equiv \frac{z + \bar{z}}{2} }$
>
> $\lf{ \Im(z) \equiv \frac{z - \bar{z}}{2} }$

### Equality of Sum of Conjugates
> $\lf{ \overline{z + w} = \overline{z} + \overline{w} }$

----
## Complex Magnitude

$z = a + b\mi$ can be expressed as the length from the origin and the angle from Real axis $|z|\;\angle\;\theta$

### Magnitude/Modulus Component

> $\lf{ |z|^2 = a^2 + b^2}$

### Angle Component

> $\lf{ \theta = \arctan(\frac{b}{a}) }$
>
> $\lf{ z = |z|\;e^{i\theta} }$
>
> $\lf{ z = |z|\;cos(\theta) + \mi sin(\theta) }$

### Triangle Inequality

> $\lf{ |z + w| \leq |z| + |w| }$
>
> $\lf{ |zw| = |z||w| }$

### Multiplication

> $\lf{ |zw| = |z||w| }$
>
> $\lf{ \angle{zw} = \angle{z} + \angle{w} }$

----
## Complex Exponents

### Exponents

> $\lf{  z^n \implies |z|^n }$
>
>$\sf{ \begin{equation} \label{eq1}
\begin{split}
z^n & = |z||z|... \text{n times} \\
& = |z|^n \\
\end{split}
\end{equation} }$

> $\lf{ z^n \implies \angle\;{n\theta}  }$
>
>$\sf{ \begin{equation} \label{eq2}
\begin{split}
\angle{\theta^n} & = \angle{\theta} + \angle{\theta}... \text{n times} \\
& = \angle{n\theta} \\
\end{split}
\end{equation} }$

### Trig Form

> $\lf{  z^N = (a + bi)^N = |z|^N(\;cos(Nθ) + \mi sin(Nθ)\;) }$

### Exponent Form

> $\lf{ z^N = (|z|\;e^{i\theta})^N =|z|^Ne^{iN\theta} }$


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.9-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
#init_printing(order="grlex")
```

```python
# this is for rac[5,3]
#
# f(kappa)
# lambda0(alpha, beta)(gamma,delta)(epsilon)/(...+omega+rho)
#
k, l0, a, b, g, d, e, o, r = symbols('k l0 a b g d e o r', real=True, positive=True)
```

```python
# Roman's new factorization:
f1 = k**2 + 2*a**2*k + a**4+b**2
f2 = 1 + g**2*k + d**2*k**2
f3 = 1 + e**2*k
A = a**4+b**2
B = 2*a**2 + g**2*A + e**2*A

f = l0 * f1 * f2 * f3 / (A + B*k + o**2*k**2 + r**2*k**3)
f
```

```python
diff(f,l0)
```

```python
da=diff(f,a)
da.factor()
```

```python
db=diff(f,b)
db.factor()
```

```python
dg=diff(f,g)
dg.factor()
```

```python
dd=diff(f,d)
dd.factor()
```

```python
de=diff(f,e)
de.factor()
```

```python
do=diff(f,o)
do.factor()
```

```python
dr=diff(f,r)
dr.factor()
```

```python

```

```python

```

```python
solveset(f1, k)
```

```python
expand((-a**2 + 1j*b)**2)
```

```python
#
#  in f2:  d2 = 1/(a**4+b**2)  and g2=2*a**2/(a**4+b**2)
#
f3 = k**2/(a**4+b**2) + k*2*a**2/(a**4+b**2) + 1
solveset(f3, k)
```

```python

```


```python
import sympy as sp
from IPython.display import display
import inspect

class Errorpropagator:
    def __init__(self, func, name='func'):
        self.name = name
        self.F = sp.Symbol('f')
        self.f = sp.parse_expr(func)
        symbols = list(map(str, list(self.f.free_symbols)))
        self.x__ = sp.Symbol('x')
        self.symbols = []
        for symbol in symbols:
            self.symbols.append(
                [sp.Symbol(f'{symbol}'), sp.Symbol('\Delta ' + symbol)])

        df_sim = 0
        df = 0
        for i, di in self.symbols:
            df_sim += sp.simplify((sp.diff(self.f, i)*di/self.f)**2)
            df += (sp.diff(self.f, i)*di)**2

        self.df_sim = self.F*sp.sqrt(df_sim)
        for i in range(3):
            self.df_sim = sp.simplify(self.df_sim)
        self.latex_sim = sp.latex(self.df_sim)

        self.df = sp.sqrt(df)
        self.latex = sp.latex(self.df)

        ### get source code ###
        self.symbols_d = []
        for symbol in symbols:
            self.symbols_d.append(
                [sp.Symbol(f'{symbol}'), sp.Symbol('d' + symbol)])

        df = 0
        for i, di in self.symbols_d:
            df += (sp.diff(self.f, i)*di)**2
        self.df_d = sp.sqrt(df)

    def showme(self):
        class Code:
            def __init__(self, x_, f_):
                raw_code = inspect.getsource(
                    sp.lambdify(x_, f_)).split('return')[-1]
                raw_code = raw_code.replace('exp', 'np.exp')
                raw_code = raw_code.replace('sqrt', 'np.sqrt')
                raw_code = raw_code.replace('log', 'np.log')
                raw_code = raw_code.replace('ln', 'np.log')

                raw_code = raw_code.replace('sin(', 'np.sin(')
                raw_code = raw_code.replace('cos(', 'np.cos(')
                raw_code = raw_code.replace('tan(', 'np.tan(')

                raw_code = raw_code.replace('sinh(', 'np.sinh(')
                raw_code = raw_code.replace('cosh(', 'np.cosh(')
                raw_code = raw_code.replace('tanh(', 'np.tanh(')

                raw_code = raw_code.replace('atan(', 'np.arctan(')
                raw_code = raw_code.replace('asin(', 'np.arcsin(')
                raw_code = raw_code.replace('acos(', 'np.arccos(')

                raw_code = raw_code.replace('atanh(', 'np.arctanh(')
                raw_code = raw_code.replace('asinh(', 'np.arcsinh(')
                raw_code = raw_code.replace('acosh(', 'np.arccosh(')
                self.raw_code = raw_code
                t = sp.lambdify(x_, f_)

        self.free_f = sorted(list(map(str, list(self.f.free_symbols))))
        self.free_f = ['x'] + sorted([i for i in self.free_f if i != 'x'])
        self.free_df = list(map(str, list(self.df.free_symbols)))
        self.free_df = [i.replace('\\Delta ', 'd') for i in self.free_df]
        self.free_df1 = sorted(
            [i for i in self.free_df if 'd' not in i and i != 'x'])
        self.free_df2 = sorted(
            [i for i in self.free_df if 'd' in i and i != 'dx'])
        self.free_df = ['x']+self.free_df1+['dx']+self.free_df2

        self.free_df = ','.join(self.free_df)
        self.free_f = ','.join(self.free_f)

        n = 180
        print('-'+'#'*n+'-')
        print('input function:')
        print('\t', sp.latex(self.f))
        print('\njupyter only representation:')
        display(self.f)

        print('#'*n)

        print('LaTeX code of propagated error:')
        print('\t', sp.latex(self.df))
        print('\njupyter only representation:')
        display(self.df)

        print('#'*n)

        print('LaTeX code of simplified propagated error:')
        print('\t', sp.latex(self.df_sim))
        print('\njupyter only representation:')
        display(self.df_sim)

        print('#'*n)

        print('LaTeX code of simplified expanded propagated error:')
        print('\t', sp.latex(self.df_sim))
        print('\njupyter only representation:')
        display(self.df_sim.expand())

        print('#'*n)

        print('python code of input function:')
        print('def '+self.name+'('+self.free_f+'):')
        print('\treturn' + Code(self.x__, self.f).raw_code)
        print('python code of error function:')
        print('def d'+self.name+'('+self.free_df+'):')
        print('\treturn' + Code(self.x__, self.df_d).raw_code)

def main(expression, name='func'):
    '''
    this little script gives you the followings things:
        1) python code for the expression you can copy directly into your python script
        2) python code for the propagated error you can copy directly into your python script
        2) on top, you get various latex code for the expressions stated above
    params:
        :expression: enter your expression like you would in python. you can use <variable>_<index>, to index your variables. You can not use ^
        :name: optional, for easier copy and paste, give the function a name
        call Errorpropagator(<expression>).showme()
        translation table:
            np.arcsin -> asin
            np.sinh -> sinh

    '''
    E = Errorpropagator(expression, name)
    E.showme()

```


# RNA Sequencing Through Expectation Maximization

### Introduction

A classic problem from computational biology is that RNA sequencing. RNA transcripts consist of nucleotides (A, G, C, and T) that contain instructions for making proteins in organisms. In this problem, we want to estimate how many of each type of RNA transcript is present in a given cell. [High through sequencing methods](https://en.wikipedia.org/wiki/DNA_sequencing#High-throughput_methods) provide data containing _reads_, which are short, randomly sampled bits of the RNA sequence. By observing the reads, we wish to estimate the distribution of transcripts in the cell, i.e. the fraction of each read to the total number of reads.

We'll consider the following problem, simplified considerably from reality. We have a set of RNA transcripts that we know the sequense of. We take a number of reads from a cell, which are randomly selected segments of known length from any of the available RNA. We sequence the reads and match them to the full transcripts, allowing us to estimate the distribution of transcripts. If every read was uniquely associated with a single transcript, this would be a fair easy prolem. Instead, our problem is complicated by the fact that individual reads could come from multiple transcripts. In other words, there is _ambiguity_ to the reads, as illustrated in the following figure:

For instance, "Read 1" might be the sequence `AGACT`, which is present on all three transcripts illustrated here.

Now to formalize the problem. We assume that there are $m$ transcripts each of length $\ell_t$, and we take $n$ reads each of length $\ell_r$. Additionally, we assume that within the cell we are testing, the probability of selecting transcript $t$ is $\rho_t$ for $t=1,\ldots,m$. For this to be a valid probability distribution, it must be the case that

$$\sum_{t=1}^{m} \rho_t = 1$$.

We use the notation that the $i^{\text{th}}$ read comes from a randomly chosen transcript $T_i$, following the distribution $\rho_1,\ldots,\rho_m$. Each read can come from a random position within the selected transcript. The number of possible starting positions within a given transcript is

$$\ell = \ell_t - \ell_r + 1$$.

So, given a set of reads $\left\{R_1,\ldots,R_n\right\}$ (which we assume to be independent), we estimate the distribution $\rho_1,\ldots,\rho_m$ that is most likely to result in the observed reads. We do this by maximizing the following likelihood function:

$$L\left(\rho_1,\ldots,\rho_m\right) = \prod_{i=1}^n P\left(R_i=r_i;\rho_1,\ldots,\rho_m\right)$$

We assume that...

1. The sequence of nucleotides (A, G, C, T) in each transcipt is known.
2. Each read can only come from at most one location in a transcript.

### Solving a simplier problem first

As a warmup, let's consider the case that there is _not_ ambiguity in the reads. That is, every read can only come from one single transcript, as in the following figure:

Consider the following: What is the probability that the $i^{\text{th}}$ observed read $R_i$ is a particular sequence $r_i$? In other words, what is $P(R_i=r_i)$? Let $t_i$ be the transcript where sequence $r_i$ occurs, where $t_i\in\left\{1,2,\ldots,m\right\}$. By the law of total probability,

$$
P(R_i=r_i) = \sum_{j=1}^m P(R_i=r_i \cap T_i=j)
$$

However, we know that each sequence can only occur on one transcript so $P(R_i=r_i \cap T_i=j)=0$ when $j \neq t_i$. So, that eleminates all but one term in the sum. Then, we apply the definition of conditional probability as follows:

$$
\begin{align}
P(R_i=r_i) &= \sum_{j=1}^m P(R_i=r_i \cap T_i=j) \\
&= P(R_i=r_i \cap T_i=t_i) \\
&= P(R_i=r_i \mid T_i=t_i)P(T_i=t_i)
\end{align}
$$

The first term is the conditional probability that $r_i$ is observed, given that transcript $t_i$ was selected. This is straightforward. As previously mentioned, there are $\ell = \ell_t - \ell_r + 1$ possible starting positions within a transcript. Only one of them will result in the observe sequence, so

$$
P(R_i=r_i \mid T_i=t_i) = \frac{1}{\ell_t - \ell_r + 1} = \frac{1}{\ell}.
$$

Additonally, the second term is the probability of selecting that particular transcript, so

$$
P(T_i=t_i) = \rho_{t_i}.
$$

Bringing this all together, we have:

$$
P(R_i=r_i) = \frac{\rho_{t_i}}{\ell}
$$

Okay! Now let's do a sanity check. Let's suppose that there are only two possible transcript. What is the maximum likelihood estimate (MLE) of $\rho_1$ and $\rho_2$, assuming that we observe $k$ reads compatible with transcript 1 and $n-k$ reads compatible with transcript 2?

Let's start by forming the maximum likelihood equation.

$$
\begin{align}
L(\rho_1, \rho_2) &= \prod_{i=1}^{n}{  P(R_i=r_i;\rho_1, \rho_2)  } \\
&= \left(\frac{\rho_{1}}{\ell_t - \ell_r +1}\right)^k \left(\frac{\rho_{2}}{\ell_t - \ell_r +1}\right)^{n-k} \\
&= \frac{\rho_{1}^k \rho_{2}^{n-k}}{\left(\ell_t - \ell_r +1\right)^n}
\end{align}
$$

Where,

$$\ell = \ell_t - \ell_r +1$$

We can see that $L$ is maximized when its numerator is maximized:

$$\max_{\rho_1}L=\max_{\rho_1}\rho_{1}^k \left(1-\rho_{1}\right)^{n-k}$$

Take the derivative with respect to $\rho_1$ and set equal to zero:

$$\frac{d}{d \rho_1} \left(\rho_{1}^k \left(1-\rho_{1}\right)^{n-k} \right)= (k-n\rho_1) \rho_1^{k-1}\left(1-\rho_{1}\right)^{n-k-1}=0$$

The solutions for the above equation are

$$\rho_1 = 0, 1, \frac{k}{n}$$

And $L$ is maximized for $\rho_1=\frac{k}{n}$.

So,

$$\begin{align}\hat{\rho}_1 &= \frac{k}{n} \\ \hat{\rho}_2 &= 1 - \frac{k}{n} = \frac{n - k}{n} \end{align}$$

So, in the simple case that we have two transcripts and no ambiguity in reads, the MLE of the distribution is simply the fraction of observed reads compatible with the given transcript.

By following this pattern (and leaving the rigorous proof to the reader), we can expand this to the case of $m$ transcripts (still with no ambiguity). Assuming that out of $n$ reads, $k_i$ reads are compatible with transcript $i$, such that $\sum_{i=1}^{m}{k_i} = n$, the maximum likelihood estimator of $\rho_1,\ldots,\rho_m$ is

$$
\hat{\rho}_i = \frac{k_i}{n},\quad\quad\text{for }i=1,\ldots,m.
$$

Okay, that was a good warm-up. Now on to the real problem...

### Modeling the problem with ambiguity

Now, we return to the case that an observable sequence can be present in multiple transcripts. We still assume that each read can occur in at most one location on a transcript. We'll encode the information about what reads can occur in what transcripts in a _compatibility matrix_, $A$, where entry $A_{ij}$ is 1 if read $r_i$ can occur in transcript $j$ and 0 otherwise. An example is given below.

$$
A = \left[ \begin{matrix} 1 & 1 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 0 & 0 \\ 1 & 1 & 0 \end{matrix}\right]
$$

So, given an arbitrary compatibility matrix, what is the maximum likelihood expression for $\rho_1,\ldots,\rho_m$? As before, we want to consider the probability of observing a specific read, given a distribution of $\rho$'s.

$$P(R_i=r_i;\rho_1, \ldots, \rho_m) = \sum_{j=1}^m P(R_i=r_i \cap T_i=j)$$

However, this time, $r_i$ can be associated with multiple possible transcripts, so the sum does not collapse. Instead, we sum over the rows of $A$.

$$P(R_i=r_i;\rho_1, \ldots, \rho_m) = \sum_{j=1}^m A_{ij} P(R_i=r_i \cap T_i=j) $$

Lets, define the vector $\rho$:

$$\rho = \left[ \begin{matrix} \rho_1 \\ \vdots \\ \rho_m \end{matrix} \right]$$

Which allows us to put the probability of observing $r_i$ in compact form:

$$\begin{align}P(R_i=r_i;\rho_1, \ldots, \rho_m) &= \sum_{j=1}^m A_{ij} P(R_i=r_i \cap T_i=j)\\
&= \sum_{j=1}^m A_{ij} P(R_i=r_i \mid T_i=j)P(T_i=j) \\
&= \frac{1}{\ell}\sum_{j=1}^m A_{ij} P(T_i=j) \\
&= \frac{1}{\ell} e_{i}^T A \rho\end{align} $$

Where $e_{i} \in \mathbf{R}^n$ is a vector with all zeros except for a one in position $i$.

Now, we can form the maximum likelihood function.

$$\begin{align}L(\rho_1, \ldots \rho_m) &= \prod_{i=1}^{n}{  P(R_i=r_i;\rho_1, \ldots \rho_m)  } \\
&= \prod_{i=1}^{n} \frac{1}{\ell} e_{i}^T A \rho \\
&= \frac{1}{\ell^n}\prod_{i=1}^{n} e_{i}^T A \rho \end{align}$$

Let's apply this model to the example above, with $m=3$ and $n=5$. There are five terms in the product.

$$\begin{align}
e_{1}^T A \rho &= \left[\begin{matrix} 1 & 0& 0 & 0& 0\end{matrix}\right] \left[\begin{matrix} 1 & 1 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 0 & 0 \\ 1 & 1 & 0\end{matrix}\right]\left[ \begin{matrix} \rho_1 \\ \rho_2 \\ \rho_3 \end{matrix} \right] \\
&= \left[\begin{matrix} 1 & 1 & 1 \end{matrix}\right]\left[ \begin{matrix} \rho_1 \\ \rho_2 \\ \rho_3 \end{matrix} \right] \\
&= \rho_1 + \rho_2 + \rho_3 \\
e_{2}^T A \rho &= \rho_2 + \rho_3 \\
e_{3}^T A \rho &= \rho_1 + \rho_3 \\
e_{4}^T A \rho &= \rho_1 \\
e_{5}^T A \rho &= \rho_1 + \rho_2
\end{align}$$

Which gives us the likelihood equation:

$$ L(\rho_1, \ldots, \rho_3) = \frac{1}{\ell^5} (\rho_1 + \rho_2 +\rho_3)(\rho_2 + \rho_3)(\rho_1 + \rho_3)(\rho_1)(\rho_1 + \rho_2) $$

Using $\rho_1 + \rho_2 + \rho_3 = 1$, maximizing the likelihood is equivalent to maximizing the following.

$$
\begin{align}
    & (\rho_2 + \rho_3)(\rho_1 + \rho_3)\rho_1(\rho_1 + \rho_2) \\
    & = (\rho_2 + \rho_3)(1-\rho_2)(1-\rho_2-\rho_3)(1-\rho_3) \\
        & = (\rho_2 + \rho_3)(1-\rho_2-\rho_3+\rho_2\rho_3)(1-\rho_2-\rho_3)
\end{align}
$$

We can observe that whatever value is chosen for $\rho_2 + \rho_3=1-\rho_1$, the above equation is maximized when $\rho_2\rho_3$ is maximized. This implies that $\rho_2 = \rho_3$.

$$\begin{align}
L(\rho_1, \ldots, \rho_3) &= (2\rho_2)(1-2\rho_2+\rho_2^2)(1-2\rho_2) \\
&= -4\rho_2^4+1-\rho_2^3-8\rho_2^2+2\rho_2 \\
\frac{d}{d\rho_2}L(\rho_1, \ldots, \rho_3) &= -16\rho_2^3 + 30\rho_2^2 -16\rho_2 + 2 \\
&= -2\left(\rho_2-1\right)\left(8\rho_2^2-7\rho_2+1\right)
\end{align}$$

So, the likelihood function has a maxima or minima at $\rho_2=1,\frac{7}{16} \pm \frac{\sqrt{17}}{16}$. Plugging these back into the equation, we see that the function is maximized at

$$\hat{\rho}_2 = \hat{\rho}_3 = \frac{7}{16} - \frac{\sqrt{17}}{16}, \;\hat{\rho}_1 = 1-\hat{\rho}_2-\hat{\rho}_3 = \frac{2}{16} + \frac{\sqrt{17}}{8}$$.

So, in this particular case, we can solve for the probability distribution exactly. What about the general case? Enter the EM algorithm.

### The expectation-maximization (EM) algorithm

The [EM algorithm](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) is an iterative method for maximizing a likelihood function over some latent variables ($\rho_1,\ldots,\rho_m). In many cases (like this one), it is not tractable to directly maximize the likelihood function. Instead, we alternate between two steps:

1. Guess at the values that we want and calculate (E-step)
2. Maximize the expectation with respect to the latent variables.

Let's try applying the approach to this problem. Suppose for a minute that we observe a read that only appears in transcripts 1 and 4. Further, suppose we knew that $\rho_1=0.1$ and $\rho_2=0.3$. Then the probability, that the read was actually sampled from transcript 1 would be

$$
\frac{\text{probability of selecting $t_1$}}{\text{probability of selecting $t_1$ or $t_4$}} = \frac{\rho_1}{\rho_1+\rho_2} = \frac{0.1}{0.1+0.3} = 0.25
$$

In general, the probability of selecting a particular transcript given a read is

$$\begin{align}
P(T_i=j \mid R_i=r_i) & = \frac{A_{ij}\rho_j}{\sum_{k=1}^m A_{ik}\rho_k}\\
    &= \frac{A_{ij}\rho_j}{e_{i}^T A \rho}
\end{align}$$

So, if we observe that particular sequence $k$ times, we would assume that the sequence was drawn from transcript $j$

$$kP(T_i=j \mid R_i=r_i)$$

times. At this point, we ask ourselves: Given the accounting of observations, what is the most likely estimate of $\rho$? We addressed this in the section with the simplifying assumption that there is no ambiguity in reads. As we showed then, the ML estimate of $\rho_j$ is simply the empirical fraction of reads that are compatible with transcript $j$. Therefore, after observing all $n$ reads, we could update the estimate of $\rho_j$ as follows based on a fixed estimate $\bar{\rho}$:

$$\begin{align}
\hat{\rho_j} &= \frac{1}{n}\sum_{i=1}^n P(T_i=j \mid R_i=r_i) \\
&= \frac{1}{n}\sum_{i=1}^n \frac{A_{ij}\bar{\rho}_j}{e_{i}^T A \bar{\rho}}
\end{align}$$

To recap, here's our approach:

1. Proportionally assign the reads according to the current guess for $\rho$(E-step)
2. Update the estimates of the parameters based on this allocation (M-step).

Proving that this iterative algorithm converges to the ML estimate is beyond the scope of this post. Instead, we'll turn to a numerical example.

Let's start by applying EM to the previous example we worked out by hand.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set(context='talk', style='darkgrid', palette='colorblind')
```

We'll set up a function to calculate $P(T_i=j \mid R_i=r_i)$ for the E-step. In addition, we'll create a function for calculating the [log-likelihood](https://en.wikipedia.org/wiki/Likelihood_function#Log-likelihood) equation.

$$\begin{align}
\ell(\rho) &= \log\left[ \frac{1}{\ell^n}\prod_{i=1}^{n} e_{i}^T A \rho \right] \\
&= -\log\left(\ell^n\right) + \sum_{i=1}^{n} \log\left( e_{i}^T A \rho \right)
\end{align}$$

This isn't necessary for the algorithm, but it allows us to directly observe the algorithm maximizing this equation. Why have we switched to log-likelihood? Numerical stability. The likelihood function is the product of $n$ many terms, each of which is between 0 and 1. This means that the likelihood function gets vanishingly small for large $n$. By taking the natural logarithm of the likelihood, we are now summing $n$ negative numbers instead. Because the logarithm is a monotonically increasing function, we know that the log-likelihood function is maximized when the likelihood function is maximized.

```python
def Pr(Tj, Ri, rho, compat_matrix):
    rho_j = rho[Tj]
    ei = np.zeros(compat_matrix.shape[0])
    ei[Ri] = 1
    return compat_matrix[i, j] * rho_j / np.dot(ei, np.dot(compat_matrix, rho))

def log_likelihood(rho, compat_matrix, scale=1.):
    output = 0
    for i in xrange(compat_matrix.shape[0]):
        output += np.log(compat_matrix[i].dot(rho))
    return output
```

Set up the given parameters, the observed compatibility matrix, and the initial estimate of $\rho$.

```python
N_iter = 50 # number of EM iterations
A = np.array([
        [1, 1, 1],
        [0, 1, 1],
        [1, 0, 1],
        [1, 0, 0],
        [1, 1, 0]
    ])
n = A.shape[0]
m = A.shape[1]
rho_est = [1./m] * m
```

And run the algorithm...

```python
rho_old = rho_est
Ls = np.ones(N_iter, dtype=float)
for ix in range(N_iter):
    rho_new = [(1./n) * np.sum([Pr(j, i, rho_old, A) for i in xrange(n)]) for j in xrange(m)]
    rho_old = rho_new
    Ls[ix] = log_likelihood(rho_new, A)

rho_est = rho_new
```

Here's the exact result we worked out before:

```python
rho_ml = [2./16 + np.sqrt(17)/8, 7./16 - np.sqrt(17)/16, 7./16 - np.sqrt(17)/16]
```

And now we can compare the results:

```python
print 'Exact ML:    {:.5f}, {:.5f}, {:.5f}'.format(*rho_ml)
print 'EM estimate: {:.5f}, {:.5f}, {:.5f}'.format(*rho_est)
```

    Exact ML:    0.64039, 0.17981, 0.17981
    EM estimate: 0.64039, 0.17981, 0.17981

Spot on! Let's check out the learning curve.

```python
fig = plt.figure(figsize=(8,6))
plt.plot(Ls, linewidth=1)
plt.xlabel('Iteration')
plt.ylabel('Log-Likelihood function value')
plt.show()
```

So, the EM estimate matches the analytical solution, and we can see empirically that the algorithm monotonically increased the likelihood function.

Now, let's try a larger, randomly generated dataset.

Start by defining some transcripts.

```python
m = 10
l_t = 100
transcripts = [''.join(np.random.choice(list('AGCT'), l_t)) for i in xrange(m)]
transcripts[:2]
```

    ['AGAGCCGTACAGACCCCATCTCGAAAGCAGAATCTTGCGGTTACTGAGGGGTAAGATAAGTAGAGTATGGCCCGGCCAGCTTAGCTGCGCATGACATGTT',
     'CGAGCGGATCTAGCATATACAAGCAGTCAGGTTTTTGAGGCAGGTGGTGAGATTCGTATACGCTTTACAGGTGCTTCCGACACGTGAGCGGTGCAGTGCA']

Next, randomly generate the true distribution $\rho_1, \ldots, \rho_m$.

```python
rho = np.random.rand(m)
rho /= sum(rho)
plt.bar(np.arange(m) + 0.6, rho, width=0.8)
```

We'll then pick 1000 random reads of length 5 each.

```python
n = 1000 # Number of reads
l_r = 5
```

```python
def random_read(transcripts, rho, l_r):
    chosen_seq = np.random.choice(transcripts, p=rho)
    start_idx = np.random.randint(0, len(chosen_seq) - l_r)
    end_idx = start_idx + l_r
    return chosen_seq[ start_idx:end_idx ]
```

```python
reads = [random_read(transcripts, rho, l_r) for _ in xrange(n)]
print'First 10 reads...', reads[0:10]
```

    First 10 reads... ['TCAGG', 'GCCTT', 'CGAAA', 'GGTGT', 'TTATT', 'GATAA', 'TTCTC', 'CTGCC', 'GCCAG', 'ATTCT']

Based on the observed data, construct the alignment matrix.

```python
def find_all_alignments(transcripts, read):
    n = len(transcripts)
    tmp = np.zeros(n)
    for j in range(n):
        if read in transcripts[j]:
            tmp[j] = 1
    return tmp
```

```python
A = np.array([find_all_alignments(transcripts, read) for read in reads])
sns.heatmap(A[:10])
plt.title('First 10 rows of A')
plt.show()
```

And now, run the EM algorithm, same as before.

```python
N_iter = 100 # number of EM iterations
# Initialize the initial estimates
rho_old = [1./m] * m
Ls = np.ones(N_iter, dtype=float)

for ix in range(N_iter):
    rho_new = [(1./n) * np.sum([Pr(j, i, rho_old, A) for i in xrange(n)]) for j in xrange(m)]
    rho_old = rho_new
    Ls[ix] = log_likelihood(rho_new, A, scale=1e1000)

rho_est = rho_new
```

Finally, let's plot the estimated distribution next to the true distribution.

```python
plt.bar(np.arange(m) + 0.6, rho, color='blue', width=0.4, label='Real')
plt.bar(np.arange(m) + 1, rho_est, color='green', width=0.4, label='Estimated')
plt.legend()
plt.show()
```

```python
fig = plt.figure(figsize=(8,6))
plt.plot(Ls, linewidth=1)
plt.xlabel('Iteration')
plt.ylabel('Log-Likelihood function value')
plt.show()
```

We see that the algorithm has converged on approximately the correct solution!


# 13 Linear Algebra
## Motivating problem: Two masses on three strings
Two masses $M_1$ and $M_2$ are hung from a horizontal rod with length $L$ in such a way that a rope of length $L_1$ connects the left end of the rod to $M_1$, a rope of length $L_2$ connects $M_1$ and $M_2$, and a rope of length $L_3$ connects $M_2$ to the right end of the rod. The system is at rest (in equilibrium under gravity).

Find the angles that the ropes make with the rod and the tension forces in the ropes.

In class we derived the equations that govern this problem – see [13_String_Problem_lecture_notes (PDF)](13_String_Problem_lecture_notes.pdf).

We can represent the problem as system of nine coupled non-linear equations:

$$
\mathbf{f}(\mathbf{x}) = 0
$$

### Summary of equations to be solved
Treat $\sin\theta_i$ and $\cos\theta_i$ together with $T_i$, $1\leq i \leq 3$, as unknowns that have to simultaneously fulfill the nine equations
\begin{align}
-T_1 \cos\theta_1 + T_2\cos\theta_2 &= 0\\
 T_1 \sin\theta_1 - T_2\sin\theta_2 - W_1 &= 0\\
                   -T_2\cos\theta_2 + T_3\cos\theta_3 &= 0\\
                    T_2\sin\theta_2 + T_3\sin\theta_3 - W_2 &= 0\\
 L_1\cos\theta_1 + L_2\cos\theta_2 + L_3\cos\theta_3 - L &= 0\\
-L_1\sin\theta_1 - L_2\sin\theta_2 + L_3\sin\theta_3  &= 0\\
\sin^2\theta_1 + \cos^2\theta_1 - 1 &= 0\\
\sin^2\theta_2 + \cos^2\theta_2 - 1 &= 0\\
\sin^2\theta_3 + \cos^2\theta_3 - 1 &= 0
\end{align}

Consider the nine equations a vector function $\mathbf{f}$ that takes a 9-vector $\mathbf{x}$ of the unknowns as argument:
\begin{align}
\mathbf{f}(\mathbf{x}) &= 0\\
\mathbf{x} &= \left(\begin{array}{c}
x_0 \\ x_1 \\ x_2 \\
x_3 \\ x_4 \\ x_5 \\
x_6 \\ x_7 \\ x_8
\end{array}\right)
 =
\left(\begin{array}{c}
\sin\theta_1 \\ \sin\theta_2 \\ \sin\theta_3 \\
\cos\theta_1 \\ \cos\theta_2 \\ \cos\theta_3 \\
T_1 \\ T_2 \\ T_3
\end{array}\right) \\
\mathbf{L} &= \left(\begin{array}{c}
L \\ L_1 \\ L_2 \\ L_3
\end{array}\right), \quad
\mathbf{W} = \left(\begin{array}{c}
W_1 \\ W_2
\end{array}\right)
\end{align}

In more detail:

\begin{align}
f_1(\mathbf{x}) &= -x_6 x_3 + x_7 x_4 &= 0\\
f_2(\mathbf{x}) &= x_6 x_0 - x_7 x_1 - W_1 & = 0\\
\dots\\
f_8(\mathbf{x}) &= x_2^2 + x_5^2 - 1 &=0
\end{align}

We generalize the *Newton-Raphson algorithm* from the [last lecture](http://asu-compmethodsphysics-phy494.github.io/ASU-PHY494//2018/03/21/12_Root_finding/) to $n$ dimensions:

## General Newton-Raphson algorithm
Given a trial vector $\mathbf{x}$, the correction $\Delta\mathbf{x}$ can be derived from the Taylor expansion
$$
f_i(\mathbf{x} + \Delta\mathbf{x}) = f_i(\mathbf{x}) + \sum_{j=1}^{n} \left.\frac{\partial f_i}{\partial x_j}\right|_{\mathbf{x}} \, \Delta x_j + \dots
$$
or in full vector notation
\begin{align}
\mathbf{f}(\mathbf{x} + \Delta\mathbf{x}) &= \mathbf{f}(\mathbf{x}) + \left.\frac{d\mathbf{f}}{d\mathbf{x}}\right|_{\mathbf{x}} \Delta\mathbf{x} + \dots\\
  &= \mathbf{f}(\mathbf{x}) + \mathsf{J}(\mathbf{x}) \Delta\mathbf{x} + \dots
\end{align}
where $\mathsf{J}(\mathbf{x})$ is the *[Jacobian](http://mathworld.wolfram.com/Jacobian.html)* matrix of $\mathbf{f}$ at $\mathbf{x}$, the generalization of the derivative to multivariate vector functions.

Solve
$$
\mathbf{f}(\mathbf{x} + \Delta\mathbf{x}) = 0
$$
i.e.,
$$
\mathsf{J}(\mathbf{x}) \Delta\mathbf{x} = -\mathbf{f}(\mathbf{x})
$$
for the correction $\Delta x$
$$
\Delta\mathbf{x} = -\mathsf{J}(\mathbf{x})^{-1} \mathbf{f}(\mathbf{x})
$$
which has the same form as the 1D Newton-Raphson correction $\Delta x = -f'(x)^{-1} f(x)$.

These are *matrix equations* (we linearized the problem). One can either explicitly solve for the unknown vector $\Delta\mathbf{x}$ with the inverse matrix of the Jacobian or use other methods to solve the coupled system of linear equations of the general form
$$
\mathsf{A} \mathbf{x} = \mathbf{b}.
$$

## Linear algebra with `numpy.linalg`

```python
import numpy as np
```

```python
np.linalg?
```

### System of coupled linear equations
Solve the coupled system of linear equations of the general form
$$
\mathsf{A} \mathbf{x} = \mathbf{b}.
$$

```python
A = np.array([
        [1, 0, 0],
        [0, 1, 0],
        [0, 0, 2]
    ])
b = np.array([1, 0, 1])
```

What does this system of equations look like?

```python
for i in range(A.shape[0]):
    terms = []
    for j in range(A.shape[1]):
        terms.append("{1} x[{0}]".format(i, A[i, j]))
    print(" + ".join(terms), "=", b[i])
```

    1 x[0] + 0 x[0] + 0 x[0] = 1
    0 x[1] + 1 x[1] + 0 x[1] = 0
    0 x[2] + 0 x[2] + 2 x[2] = 1

Now solve it with `numpy.linalg.solve`:

```python
x = np.linalg.solve(A, b)
print(x)
```

    [ 1.   0.   0.5]

Test that it satisfies the original equation:
$$
\mathsf{A} \mathbf{x} - \mathbf{b} = 0
$$

```python
np.dot(A, x) - b
```

    array([ 0.,  0.,  0.])

#### Activity: Solving matrix equations
With
$$
\mathsf{A}_1 = \left(\begin{array}{ccc}
   +4 & -2 & +1\\
   +3 & +6 & -4\\
   +2 & +1 & +8
   \end{array}\right)
$$
and
$$
\mathbf{b}_1 = \left(\begin{array}{c}
   +12 \\ -25 \\ +32
   \end{array}\right), \quad
\mathbf{b}_2 = \left(\begin{array}{c}
   +4 \\ -1 \\ +36
   \end{array}\right), \quad
$$
solve for $\mathbf{x}_i$
$$
\mathsf{A}_1 \mathbf{x}_i = \mathbf{b}_i
$$
and *check the correctness of your answer*.

```python
A1 = np.array([
        [+4, -2, +1],
        [+3, +6, -4],
        [+2, +1, +8]
    ])
b1 = np.array([+12, -25, +32])
b2 = np.array([ 4, -1, 36])
```

```python
x1 = np.linalg.solve(A1, b1)
print(x1)
print(A1.dot(x1) - b1)
```

    [ 1. -2.  4.]
    [ 0.  0.  0.]

```python
x2 = np.linalg.solve(A1, b2)
print(x2)
print(A1.dot(x2) - b2)
```

    [ 1.  2.  4.]
    [ -8.88178420e-16   0.00000000e+00   7.10542736e-15]

### Matrix inverse
In order to solve directly we need the inverse of $\mathsf{A}$:
$$
\mathsf{A}\mathsf{A}^{-1} = \mathsf{A}^{-1}\mathsf{A} = \mathsf{1}
$$
Then
$$
\mathbf{x} = \mathsf{A}^{-1} \mathbf{b}
$$

If the inverse exists, `numpy.linalg.inv()` can calculate it:

```python
Ainv = np.linalg.inv(A)
print(Ainv)
```

    [[ 1.   0.   0. ]
     [ 0.   1.   0. ]
     [ 0.   0.   0.5]]

Check that it behaves like an inverse:

```python
Ainv.dot(A)
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

```python
A.dot(Ainv)
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

Now solve the coupled equations directly:

```python
Ainv.dot(b)
```

    array([ 1. ,  0. ,  0.5])

#### Activity: Solving coupled equations with the inverse matrix
1. Compute the inverse of $\mathsf{A}_1$ and *check the correctness*.
2. Compute $\mathbf{x}_1$ and $\mathbf{x}_2$ with $\mathsf{A}_1^{-1}$ and check the correctness of your answers.

```python
A1_inv = np.linalg.inv(A1)
print(A1_inv)
```

    [[ 0.19771863  0.06463878  0.00760456]
     [-0.121673    0.11406844  0.07224335]
     [-0.03422053 -0.03041825  0.11406844]]

```python
A1.dot(A1_inv)
```

    array([[  1.00000000e+00,   6.93889390e-18,   0.00000000e+00],
           [ -5.55111512e-17,   1.00000000e+00,   1.11022302e-16],
           [  0.00000000e+00,  -5.55111512e-17,   1.00000000e+00]])

```python
A1_inv.dot(A1)
```

    array([[  1.00000000e+00,  -3.46944695e-18,   5.55111512e-17],
           [  2.77555756e-17,   1.00000000e+00,   2.22044605e-16],
           [  2.77555756e-17,   1.38777878e-17,   1.00000000e+00]])

```python
x1 = A1_inv.dot(b1)
print(x1)
print(A1.dot(x1) - b1)
```

    [ 1. -2.  4.]
    [  0.00000000e+00   0.00000000e+00   7.10542736e-15]

```python
x2 = A1_inv.dot(b2)
print(x2)
print(A1.dot(x2) - b2)
```

    [ 1.  2.  4.]
    [  0.00000000e+00   3.55271368e-15   7.10542736e-15]

### Eigenvalue problems
The equation
\begin{gather}
\mathsf{A} \mathbf{x}_i = \lambda_i \mathbf{x}_i
\end{gather}
is the **eigenvalue problem** and a solution provides the eigenvalues $\lambda_i$ and corresponding eigenvectors $x_i$ that satisfy the equation.

#### Example 1: Principal axes of a square
The principle axes of the [moment of inertia tensor](https://en.wikipedia.org/wiki/Moment_of_inertia#The_inertia_tensor) are defined through the eigenvalue problem
$$
\mathsf{I} \mathbf{\omega}_i = \lambda_i \mathbf{\omega}_i
$$
The principal axes are the $\mathbf{\omega}_i$.

```python
Isquare = np.array([[2/3, -1/4], [-1/4, 2/3]])
```

```python
lambdas, omegas = np.linalg.eig(Isquare)
```

```python
lambdas
```

    array([ 0.91666667,  0.41666667])

```python
omegas
```

    array([[ 0.70710678,  0.70710678],
           [-0.70710678,  0.70710678]])

Note that the eigenvectors are `omegas[:, i]`! You can transpose so that axis 0 is the eigenvector index:

```python
omegas.T
```

    array([[ 0.70710678, -0.70710678],
           [ 0.70710678,  0.70710678]])

Test:
$$
(\mathsf{I} - \lambda_i \mathsf{1}) \mathbf{\omega}_i = 0
$$

```python
(Isquare - lambdas[0]*np.identity(2)).dot(omegas[:, 0])
```

    array([ 0.,  0.])

```python
(Isquare - lambdas[0]*np.identity(2)).dot(omegas.T[0])
```

    array([ 0.,  0.])

```python
(Isquare - lambdas[1]*np.identity(2)).dot(omegas.T[1])
```

    array([ 0.,  0.])

#### Example 2: Spin in a magnetic field
In quantum mechanics, a spin 1/2 particle is represented by a spinor $\chi$, a 2-component vector. The Hamiltonian operator for a stationary spin 1/2 particle in a homogenous magnetic field $B_y$ is
$$
\mathsf{H} = -\gamma \mathsf{S}_y B_y = -\gamma B_y \frac{\hbar}{2} \mathsf{\sigma_y}
   = \hbar \omega \left( \begin{array}{cc} 0 & -i \\ i & 0 \end{array}\right)
$$

Determine the *eigenvalues* and *eigenstates*
$$
\mathsf{H} \mathbf{\chi} = E \mathbf{\chi}
$$
of the spin 1/2 particle.

(To make this a purely numerical problem, divide through by $\hbar\omega$, i.e. calculate $E/\hbar\omega$.)

```python
sigma_y = np.array([[0, -1j], [1j, 0]])
E, chis = np.linalg.eig(sigma_y)
print(E)
print(chis.T)
```

    [ 1.+0.j -1.+0.j]
    [[-0.00000000-0.70710678j  0.70710678+0.j        ]
     [ 0.70710678+0.j          0.00000000-0.70710678j]]

Normalize the eigenvectors:
$$
\hat\chi = \frac{1}{\sqrt{\chi^\dagger \cdot \chi}} \chi
$$

```python
chi1 = chis.T[0]
print(chi1)
norm = np.dot(chi1.conjugate(), chi1)
chi1_hat = chi1/np.sqrt(norm)
print(chi1_hat)
```

    [-0.00000000-0.70710678j  0.70710678+0.j        ]
    [-0.00000000-0.70710678j  0.70710678+0.j        ]

```python
norm
```

    (0.99999999999999989+0j)

... they were already normalized.

#### Activity: eigenvalues
Find the eigenvalues and eigenvectors of
$$
\mathsf{A}_2 = \left(\begin{array}{ccc}
   -2 & +2 & -3\\
   +2 & +1 & -6\\
   -1 & -2 & +0
   \end{array}\right)
$$

Are the eigenvectors normalized?

Check your results.

```python
A2 = np.array([[-2, +2, -3],
   [+2, +1, -6],
   [-1, -2, +0]])
```

```python
lambdas, evecsT = np.linalg.eig(A2)
evecs = evecsT.T
print(lambdas)
print(evecs)
```

    [-3.  5. -3.]
    [[-0.95257934  0.27216553 -0.13608276]
     [ 0.40824829  0.81649658 -0.40824829]
     [ 0.05155221  0.82292764  0.5658025 ]]

```python
np.linalg.norm(evecs, axis=1)
```

    array([ 1.,  1.,  1.])

```python
Identity = np.identity(A2.shape[0])
for evalue, evec in zip(lambdas, evecs):
    print((A2 - evalue * Identity).dot(evec))
```

    [ -3.88578059e-16   1.11022302e-16  -1.11022302e-16]
    [ -6.66133815e-16   2.66453526e-15   0.00000000e+00]
    [ 0.  0.  0.]

```python

```