## preamble

```
%matplotlib inline
import numpy as np
from misc import *
import matplotlib.pyplot as plt

solar_mass_in_seconds = 4.92686088e-6

def phase(f, Mc):

        phase = -np.pi/4. + ( 3./( 128. * pow(Mc*np.pi*f, 5./3.) ) )
        return phase

def htilde(f, Mc):
        Mc *= solar_mass_in_seconds
        htilde = pow(f, -7./6.) * pow(Mc,5./6.) * np.exp(1j*phase(f,Mc))
        return htilde
```

# Brief summary

In the note I'll show how to build an ''empirical interpolant'' of a function $h(M_c;f)$, that is spanned by a basis $[e_i(f)]_{i=1}^{n}$. In the previous section, we built the ''reduced basis'' of $h(M_c;f)$: for the purpose of building the empirical interpolant it is not important how the basis was constructed, we just need to be able to write the expansion $h(M_c;f) = \sum_{i=1}^{n} \langle h(M_c;f), e_i(f) \rangle\,e_i(f)$. Given such an expansion, we want to be able to re-write it as:

$h(M_c;f) \approx \sum_{i=1}^{n} c_i(M_c)\,e_i(f)$

where $c_i(M_c)$ are some interpolating coefficients. This is a standard interpolation problem and we will use the empirical interpolation method to find these coefficients. While one could use other interpolation techniques, such as spline interpolation, or interpolation with Chebyshev polynomials, neither are optimal in any precise sense. The empirical interpolant is optimal in a precise sense, namely if the function $h(M_c;f)$ is spanned by the basis $[e_i(f)]_{i=1}^{n}$ whose projection error has an upper bound of $\sigma$, then the maximum error of the empirical interpolant $\mathcal{I}[h]$ is bounded by

$max_{M_c}|| h(M_c;f) - \mathcal{I}[h](M_c;f) || \leq \Lambda\,\sigma$

where $\Lambda$ is the Lebesgue constant: $\Lambda \approx 100$. Thus, if the largest proection error of the basis is $\sigma \approx 10^{-12}$ then the maximum error of the empirical interpolant anywhere in the parameter space $M_c$ is around $10^{-10}$. Hence, if we have an optimal basis, we can make an optimal interpolant!

Below is a brief description of the empirical interpolant (see http://arxiv.org/pdf/1308.3565v2.pdf).

The empirical interpolant, which interpolates the waveform $h(M_c;f)$ for a given parameter $M_c$, is denoted by $\mathcal{I}[h](M_c;f)$ and takes the form
\begin{equation}
\mathcal{I}[h](M_c;f) = \sum_{i=1}^n  c_i (M_c) e_i(f)\, .
\end{equation}
The coefficients $\{ c_i \}_{i=1}^n$ are defined by requiring the interpolant to equal the value of the waveform at the empirical frequency nodes $F_j$,
\begin{align}
\sum_{i=1}^n  c_i (M_c) e_i(F_j) = h(M_c;F_j) , \qquad  j=1,\dots, n \, ,
\end{align}
which is equivalent to solving an $n$-by-$n$ system
\begin{equation}
\sum_{i=1}^nV_{ji}c_i(M_c) = h (M_c;F_j)\,, \quad j=1, \ldots, n
\end{equation}
for the coefficients $\{ c_i \}_{i=1}^n$ where the interpolation matrix
\begin{equation}
  V \equiv \left(  \begin{array}{cccc}
              e_1(F_1)  &  e_2(F_1)            & \cdots & e_{n}(F_1)      \\
              e_1(F_2)  &  e_2(F_2)            & \cdots & e_{n}(F_2)       \\
              e_1(F_3)  &  e_2(F_3)          & \cdots & e_{n}(F_3)   \\
              \vdots    & \vdots             & \ddots & \vdots                       \\
              e_1(F_{n})  & e_2(F_{n})    & \cdots & e_{n}(F_{n})  \\
             \end{array}
   \right)
\end{equation}
is independent of the parameter $M_c$.

The choice of empirical nodes given by the empirical interpolation algorithm (shown later)together with the linear independence of the reduced basis ensure that $V$ in is as well-conditioned as possible and invertible so that
\begin{equation}
c_i(M_c) = \sum_{j=1}^n \left( V^{-1}\right)_{ij}h (M_c;F_j)
\end{equation}
is the unique solution for the coefficient $c_i$.
It then follows upon substitution that the empirical interpolant is

\begin{equation}
\mathcal{I}[h](M_c;f) = \sum_{j=1}^n B_j (f) h(M_c;F_j)
\end{equation}
where
\begin{equation}
B_j(f) \equiv  \sum_{i=1}^n e_i (f )  \left( V^{-1} \right)_{ij}
\end{equation}
and is independent of $M_c$.

There is a point worth mentioning: Note that in the expression for the empirical interpolant $\mathcal{I}[h](M_c;f) = \sum_{j=1}^m B_j (f) h(M_c;F_j)$, the functions $h$ appears explicity. Thus to evaluate the empirical interpolant one must be able to evaluate $h(M_c;F_j)$ at the nodes $\{ F_j \}_{j=1}^n$. Fortunately, this is the case in this example, as well as with your coupling coefficients for the mirror map work. The goal of this note is basically to get those empirical interpolation nodes $\{ F_j \}_{j=1}^n$, and everything else falls into place once you have them.

In the rest of this note, I'll show how to make the empirical interpolant for $h$. The input to the algorithm will be the reduced basis set $\{ e_i \}_{i=1}^n$ and the output will be $\{ B_i \}_{i=1}^n$ and the empirical frequency nodes $\{ F_i \}_{i=1}^n$, which is sufficient to compute the empirical interpolant. The reduced basis should have been produced from the previous note on building the reduced basis: if you haven't ran that note, please run it now so that you can proceed.

Rather than explain exactly why this algorithm works, I'll just code it up: I actually don't have any intuition as to how it works, and this seems to be a common sentiment in the community. The following is just Algorithm 2 of http://arxiv.org/pdf/1308.3565v2.pdf.

We will begin by loading in the basis, and the associated frequency series.

```
# Load the basis
e_re = np.loadtxt("basis_re.dat")
e_im = np.loadtxt("basis_im.dat")

e = e_re + 1j*e_im

# Load the frequency series and frequency step
fseries = np.loadtxt("fseries.dat")
```

Next we will create some storage for the frequency nodes, the indices of the frequency nodes (with respect to the full frequency series) and the V matrix

```
indices = []
f_nodes = []
V = np.zeros((len(e), len(e)), dtype=complex)
```

Now we'll start the empirical interpolation method (EIM) algorithm. (See Algorithm 2 of http://arxiv.org/pdf/1308.3565v2.pdf.). This gives us the empirical interpolation nodes $\{ F_i \}_{i=1}^n$.

```
from scipy.linalg import inv

# seed EIM algorithm

indices.append( int(np.argmax( np.abs(e[0]) )) ) # (2) of Algorithm 2
f_nodes.append(fseries[indices]) # (3) of Algorithm 2

for i in range(1, len(e)): #(4) of Algorithm 2

        #build empirical interpolant for e_iter
        for j in range(len(indices)): # Part of (5) of Algorithm 2: making V_{ij}
                for k in range(len(indices)): # Part of (5) of Algorithm 2: making V_{ij}
                        V[k][j] = e[j][indices[k]] # Part of (5) of Algorithm 2: making V_{ij}
        invV = inv(V[0:len(indices), 0:len(indices)]) # Part of (5) of Algorithm 2: making V_{ij}
        B = B_matrix(invV, e) # Part of (5) of Algorithm 2: making B_j(f)
        interpolant = emp_interp(B, e[i], indices) # Part of (5) of Algorithm 2: making the empirical interpolant of e
        res = interpolant - e[i] # 6 of Algorithm 2

        index = int(np.argmax(np.abs(res))) # 7 of Algorithm 2
        print "F_{%i} = %f"%(i, fseries[index])
        indices.append(index) # 8 of Algorithm 2
        f_nodes.append( fseries[index] ) # 9 of Algorithm 2

# make B matrix with all the indices
for j in range(len(indices)):
                for k in range(len(indices)):
                        V[k][j] = e[j][indices[k]]

invV = inv(V[0:len(indices), 0:len(indices)])
B = B_matrix(invV, e)

#save B matrix
np.savetxt("B_re.dat", np.matrix(B).real)
np.savetxt("B_im.dat", np.matrix(B).imag)

#save empirical interpolation node indices and nodes
np.savetxt("indices.dat", np.matrix(indices),  fmt='%i')
np.savetxt("nodes.dat", np.matrix(f_nodes))
```

Now that we have the empirical interpolation nodes, we can build the interpolant and look at its accuracy. To do this I'll make a space of waveforms which I will compare to their interpolants.

```
size = 2000 # training space of TS_size number of waveforms

########## set up waveforms for comparison ##########

Mc_min = 1.5
Mc_max = 3

Mcs_5_over_3 = np.linspace(Mc_min**(5./3.), Mc_max**(5./3.), size)
Mcs = pow(Mcs_5_over_3, 3./5.)

#### allocate memory for waveforms ####

H = np.zeros(size*len(fseries), dtype=complex).reshape(size, len(fseries)) # store training space in TS_size X len(fseries) array

for i in range(size):

        H[i] = htilde(fseries, Mcs[i])
```

As a first step, it's useful to just plot a waveform and its interpolant.

```
h_for_comparison = htilde(fseries, 2)
interpolant_for_comparison = np.inner(B.T, h_for_comparison[indices])

plt.plot(fseries, h_for_comparison.real, label='true waveform')
plt.plot(fseries, interpolant_for_comparison.real, label='interpolant')
plt.legend()
plt.xlabel('frequency (Hz)')
plt.show()
```

At least by eye the waveform and its empirical interpolant seem indistinguishable. Let's now look at the accuray of the interpolant of all the waveforms we just generated. To do this, I'll print out the the error $|| h(M_{c};f) - \mathcal{I}[h](M_c;f) ||$ for all of the $h(M_c;f)$. The claim earlier was that the largest this quantity should ever get is about $10^{-10}$:

```
list_of_errors = []
for i in range(size):
        interpolant = np.inner(B.T, H[i][indices])
        interpolant /= np.sqrt(np.vdot(interpolant, interpolant)) #normalize

        H[i] /= np.sqrt(np.vdot(H[i], H[i]) ) #normalize

        error = abs(np.vdot(H[i] - interpolant, H[i] - interpolant ))
        list_of_errors.append(error)

        print error
```

    3.78191639324e-29
    2.35035294402e-28
    1.06261980223e-27
    3.08369029505e-27
    4.18695879037e-27
    9.00194495983e-27
    2.05516111466e-26
    2.7716252319e-13
    2.30004529777e-26
    1.37643884267e-26
    3.99369182828e-13
    4.74044376971e-28
    5.41730508293e-13
    1.10186760068e-27
    3.73293814686e-13
    1.04951989614e-26
    1.43375083979e-13
    1.46962656246e-26
    8.47929095243e-27
    9.44998589626e-14
    4.13185508392e-28
    1.47863236007e-13
    2.42577409718e-27
    1.45828247586e-13
    3.86150514501e-27
    9.39244975523e-14
    1.52154926623e-27
    6.35915201531e-28
    1.85464853807e-13
    1.4256445484e-27
    8.70138931011e-13
    6.24025621081e-27
    6.90749593636e-12
    9.58325148465e-12
    1.14678828015e-27
    4.95084487221e-12
    6.32106973392e-27
    7.35747250416e-12
    1.18857988921e-26
    3.34450159693e-11
    3.943345479e-11
    5.05916040442e-28
    1.81299517887e-11
    5.68373960112e-27
    4.84997359412e-11
    4.60118920127e-11
    5.63139759554e-28
    1.32866521557e-11
    6.96797329919e-28
    1.29621738424e-11
    3.02301700521e-27
    4.31371178535e-11
    4.49882618131e-11
    4.89608230133e-28
    1.71548651848e-11
    3.18129131391e-27
    4.12785991508e-11
    3.85890258565e-11
    5.23454367126e-28
    1.23797076139e-11
    2.28552772512e-27
    2.58575254925e-11
    2.26627273411e-11
    1.44759942934e-27
    6.40944914645e-12
    7.6681874254e-27
    1.17666277525e-11
    9.61050275698e-12
    2.3659419347e-28
    2.30969381794e-12
    4.0281511012e-27
    3.39198148158e-12
    2.3727715338e-12
    1.38098912906e-27
    3.43686030824e-13
    8.76432724846e-27
    1.15107869496e-13
    5.10392121715e-27
    4.24475459555e-27
    4.91544065644e-13
    8.85883040358e-13
    5.6293710789e-28
    7.08025749235e-13
    8.87247614632e-27
    3.07248176199e-12
    3.80897051004e-12
    7.41194397452e-28
    2.20057201566e-12
    1.32083251916e-26
    9.12189146756e-12
    1.2244910639e-11
    2.6379761846e-27
    1.52252630081e-11
    1.41704811131e-11
    2.96468400164e-28
    5.60816072023e-12
    2.58636819419e-27
    1.82628908448e-11
    2.24278206405e-11
    1.77067122641e-27
    2.44196240552e-11
    2.16878012119e-11
    2.00768079333e-27
    8.08740251372e-12
    9.80823871098e-27
    2.60515713265e-11
    3.25484100538e-11
    3.46437185658e-27
    3.94590534457e-11
    3.96178230609e-11
    1.42570234625e-27
    3.34033002996e-11
    2.71702796384e-11
    8.36984652356e-28
    8.92243940953e-12
    8.64805390527e-27
    2.66002739145e-11
    3.25129565863e-11
    1.56895528114e-26
    3.90879373105e-11
    3.99542677105e-11
    2.88595610468e-27
    3.75194882061e-11
    3.44830319779e-11
    1.63371510028e-27
    2.55341238281e-11
    1.98206345414e-11
    2.75626689372e-28
    6.10113457016e-12
    8.51097664556e-27
    1.76669247393e-11
    2.15599348636e-11
    7.85600513798e-27
    2.65909288914e-11
    2.79805694773e-11
    6.27911655443e-27
    2.90373856401e-11
    2.88583771442e-11
    2.30605523523e-27
    2.75070373428e-11
    2.64420469001e-11
    3.67242469383e-27
    2.38553151974e-11
    2.24074084427e-11
    1.45571286456e-27
    1.94216642573e-11
    1.7925776232e-11
    1.85120876234e-27
    1.50600060983e-11
    1.37077118533e-11
    8.78618605464e-28
    1.12264290514e-11
    1.01008617267e-11
    1.76028003157e-28
    8.09700402111e-12
    7.21313450594e-12
    5.82448007325e-28
    5.67551812176e-12
    5.01166348598e-12
    5.00012939351e-28
    3.87682428752e-12
    3.39534362973e-12
    4.41831316683e-28
    2.58358703382e-12
    2.24403537124e-12
    1.16115510797e-27
    1.67823592289e-12
    1.4442816543e-12
    1.86166503591e-27
    1.05842116875e-12
    9.00397659128e-13
    2.5464313957e-27
    6.42037096958e-13
    5.37139465732e-13
    1.93472311972e-27
    3.67029583857e-13
    2.98548991987e-13
    6.62296308676e-27
    1.88464364128e-13
    1.44649661507e-13
    7.39618566601e-27
    7.5332785679e-14
    4.85414734588e-14
    4.35031163725e-27
    9.32239550937e-15
    1.31315773923e-27
    8.77142783507e-15
    2.12418082273e-28
    4.06221407897e-14
    5.97118797507e-14
    6.28367821544e-28
    1.04495539777e-13
    1.31384603018e-13
    9.80069823875e-28
    2.00379091908e-13
    2.47257588466e-13
    4.12955819116e-27
    3.92658533295e-13
    5.18644851505e-13
    2.59760154389e-26
    1.17970857509e-12
    2.7895250632e-12
    1.4574295132e-12
    2.94196635986e-28
    9.96707907086e-13
    9.60695535924e-13
    2.78807082162e-28
    1.06322772357e-12
    1.22312833927e-12
    8.68313945807e-27
    2.22844506758e-12
    4.81414648829e-12
    2.32273936811e-12
    2.0448442067e-27
    1.39069676914e-12
    1.26828645249e-12
    1.77835582746e-27
    1.27974927555e-12
    1.41729191195e-12
    6.74176613021e-27
    2.42785541767e-12
    5.12018954022e-12
    2.42211980703e-12
    2.95212250178e-27
    1.41175793381e-12
    1.27850537929e-12
    8.0157671642e-28
    1.28971357321e-12
    1.4389644138e-12
    8.64547264758e-27
    2.54819972865e-12
    5.52690683714e-12
    2.71672025682e-12
    3.11041490329e-28
    1.79090673344e-12
    1.79452975551e-12
    6.0780569506e-27
    2.757852763e-12
    5.68839666983e-12
    2.68214978399e-12
    1.32555574454e-27
    1.65752549825e-12
    1.61960187448e-12
    1.67376853986e-26
    2.39307375585e-12
    4.86287574464e-12
    2.26534990451e-12
    7.94701541186e-28
    1.37775528074e-12
    1.34089652832e-12
    3.20463854137e-26
    1.9823539118e-12
    4.04826785016e-12
    1.90211888329e-12
    5.68509008041e-28
    1.1927436087e-12
    1.18909242339e-12
    6.93586488313e-27
    1.89733448064e-12
    4.11713354204e-12
    2.11209264364e-12
    5.51010079973e-27
    1.94424403093e-12
    3.49269140977e-12
    1.48654194225e-12
    2.63712536581e-28
    8.04329383892e-13
    7.56819440656e-13
    5.92997216019e-27
    1.09734915889e-12
    2.28768081504e-12
    1.1320556455e-12
    2.13923404113e-27
    9.79394133787e-13
    1.71335649975e-12
    7.12027517155e-13
    1.32402242202e-27
    3.70144667549e-13
    3.42742543334e-13
    4.27516364413e-27
    4.85620585814e-13
    1.00636361477e-12
    4.9747999551e-13
    1.62592484688e-27
    4.39328447106e-13
    7.9164720496e-13
    3.47251626015e-13
    2.49512539907e-27
    2.44954278092e-13
    3.90316368233e-13
    1.47960322726e-13
    1.50411524824e-28
    6.36906975863e-14
    5.33005876504e-14
    1.29559425902e-27
    6.00052969969e-14
    1.08498477813e-13
    4.57933546054e-14
    2.90303665078e-27
    2.65193322641e-14
    3.53501411128e-14
    1.02418635759e-14
    2.34108970811e-27
    1.17643912824e-15
    1.01386787516e-27
    1.1856115506e-15
    1.43252824191e-27
    1.05390370509e-14
    3.70323962478e-14
    2.84977496006e-14
    1.64521430563e-27
    5.36918939512e-14
    1.37039317704e-13
    8.45545116817e-14
    1.21763568266e-27
    1.1935833229e-13
    2.75608219249e-13
    1.56699796772e-13
    4.75500859479e-28
    1.95143652153e-13
    4.29085236936e-13
    2.33886095662e-13
    9.74423704663e-28
    2.71990857644e-13
    5.81647313098e-13
    3.09415600748e-13
    4.33655099417e-28
    3.45680670299e-13
    7.27266304679e-13
    3.81460970925e-13
    9.24380132635e-28
    4.16746085305e-13
    8.6935109188e-13
    4.52912579093e-13
    1.05330151073e-27
    4.90553391068e-13
    1.02134771984e-12
    5.31966331453e-13
    1.33147834218e-27
    5.78932444732e-13
    1.211715017e-12
    6.35865175654e-13
    1.84226844894e-27
    7.07978406192e-13
    1.50620655335e-12
    8.06955977414e-13
    4.10861946172e-27
    9.54104004563e-13
    2.12182419945e-12
    1.20758321757e-12
    5.12343149867e-27
    1.7823294885e-12
    5.05158374614e-12
    5.15291245515e-12
    1.8915503505e-12
    2.09722211047e-28
    1.39180256915e-12
    2.55945006633e-12
    1.20834268653e-12
    5.11395397323e-28
    1.14183227121e-12
    2.2766024998e-12
    1.15298392132e-12
    5.62133734099e-28
    1.24451122557e-12
    2.66891917009e-12
    1.47268695511e-12
    3.714811441e-27
    2.07420325946e-12
    5.78443700428e-12
    5.83364440847e-12
    2.12761099963e-12
    4.30151351701e-28
    1.56961743409e-12
    2.91580337234e-12
    1.40047716212e-12
    3.82284712606e-28
    1.40689867035e-12
    2.94651609467e-12
    1.59814790008e-12
    7.28725457156e-27
    2.21333487776e-12
    6.1723188949e-12
    6.26165025926e-12
    2.31265355648e-12
    2.68088315902e-28
    1.79448418986e-12
    3.47934189859e-12
    1.7765532576e-12
    1.79258713773e-27
    2.24837956158e-12
    6.05564517208e-12
    5.9616413683e-12
    2.14500664201e-12
    4.14989398102e-28
    1.5944738895e-12
    3.03751066019e-12
    1.52709842606e-12
    5.67098023351e-27
    1.88463589064e-12
    5.02566150798e-12
    4.9064072152e-12
    1.75337203229e-12
    1.11580235876e-27
    1.29170065344e-12
    2.45508949287e-12
    1.23331985791e-12
    3.58287236098e-27
    1.52717617817e-12
    4.08971338953e-12
    4.0170086332e-12
    1.44728445423e-12
    1.27851475503e-28
    1.09172584921e-12
    2.10892403e-12
    1.08075125586e-12
    2.15303117815e-27
    1.41388244608e-12
    3.93391877916e-12
    4.05908229802e-12
    1.56185632328e-12
    2.88160702333e-27
    1.48889305102e-12
    3.69213468862e-12
    3.41858849319e-12
    1.17544245147e-12
    9.94704628865e-28
    8.28597197606e-13
    1.56231861426e-12
    7.85482595738e-13
    2.58533416144e-27
    1.00276399124e-12
    2.77450084293e-12
    2.85969651156e-12
    1.10451623297e-12
    1.1292060685e-27
    1.07996881903e-12
    2.74488752354e-12
    2.63533543266e-12
    9.55560134237e-13
    7.26745291943e-28
    8.30410075198e-13
    1.98547237737e-12
    1.78146700389e-12
    5.96127121387e-13
    1.9031143636e-28
    4.02439767355e-13
    7.46306895241e-13
    3.70188744779e-13
    2.44720081424e-27
    4.64024277493e-13
    1.27769461781e-12
    1.31459826073e-12
    5.0848212357e-13
    1.45355533142e-27
    5.03659894478e-13
    1.29565699205e-12
    1.26483619949e-12
    4.68860846551e-13
    1.20148132269e-27
    4.34652831949e-13
    1.08848981618e-12
    1.03735098639e-12
    3.76214756522e-13
    8.53776603392e-28
    3.35423151061e-13
    8.2519016419e-13
    7.73199154044e-13
    2.75878841881e-13
    7.09417738222e-28
    2.38379124201e-13
    5.77514579037e-13
    5.32889981746e-13
    1.87221765096e-13
    7.69335475017e-28
    1.56721256797e-13
    3.73438648871e-13
    3.38669794415e-13
    1.16835502602e-13
    1.05892071428e-27
    9.39478037498e-14
    2.18860184592e-13
    1.93629660084e-13
    6.49934425308e-14
    1.54700948466e-27
    4.89469526195e-14
    1.09550457937e-13
    9.25029173708e-14
    2.93810190429e-14
    3.00287931344e-27
    1.9066413906e-14
    3.84484612445e-14
    2.83172892629e-14
    7.45414787241e-15
    4.09749950265e-27
    2.33037493525e-15
    1.83538743208e-15
    2.96015782897e-27
    1.83846273969e-15
    2.33869667753e-15
    7.09486663271e-29
    7.53670520416e-15
    2.88423912684e-14
    3.95772855201e-14
    1.9907344009e-14
    5.46155305772e-28
    3.20781372705e-14
    1.04528468751e-13
    1.29569588649e-13
    6.14965424366e-14
    5.65588172733e-27
    9.93753943927e-14
    3.51728935248e-13
    5.32114779612e-13
    4.36573583483e-13
    1.53379241785e-13
    6.64852462414e-28
    1.49050710861e-13
    3.98973753946e-13
    4.14379707647e-13
    1.6652018714e-13
    4.31131120153e-28
    1.90874514002e-13
    5.45095157469e-13
    6.0351535791e-13
    2.59934382717e-13
    2.61852518069e-27
    3.59562147076e-13
    1.19675643265e-12
    1.71871068521e-12
    1.35053513865e-12
    4.58362369184e-13
    6.03803701244e-28
    4.26704287782e-13
    1.13391541595e-12
    1.18257296633e-12
    4.84075838318e-13
    1.7833939211e-27
    6.16730041196e-13
    1.98469100825e-12
    2.76663834108e-12
    2.1172940036e-12
    7.01962106676e-13
    8.90375515536e-28
    6.28597176984e-13
    1.64424238819e-12
    1.69160420689e-12
    6.84495999741e-13
    1.28895310791e-27
    8.57320646482e-13
    2.7436656925e-12
    3.81111288556e-12
    2.91224159548e-12
    9.66091845879e-13
    9.54528352963e-28
    8.71987955934e-13
    2.2984400423e-12
    2.38965923266e-12
    9.80346197047e-13
    3.9268387649e-27
    1.27711129689e-12
    4.19992732941e-12
    6.03612408863e-12
    4.81556188631e-12
    1.68865966578e-12
    1.32228458442e-27
    1.81796322662e-12
    5.59300454891e-12
    7.59438623028e-12
    5.76064994863e-12
    1.92842951458e-12
    3.21407057431e-27
    1.90243439626e-12
    5.601520477e-12
    7.2617197166e-12
    5.23424814107e-12
    1.65164304338e-12
    7.17496973857e-28
    1.37545352305e-12
    3.5103902008e-12
    3.54969955922e-12
    1.42208382865e-12
    3.27237262531e-27
    1.78601750101e-12
    5.79634056576e-12
    8.24779892729e-12
    6.53574934156e-12
    2.28377078215e-12
    3.93820677475e-27
    2.46485341656e-12
    7.63057982576e-12
    1.04636932323e-11
    8.04771686089e-12
    2.74355609214e-12
    4.40364342684e-27
    2.84988864397e-12
    8.69103035773e-12
    1.17649926591e-11
    8.94833205475e-12
    3.02139087435e-12
    2.38368246369e-27
    3.09060795394e-12
    9.36876920317e-12
    1.26194556941e-11
    9.55982410328e-12
    3.21793248096e-12
    1.37936675275e-27
    3.28009399041e-12
    9.93837772434e-12
    1.33918680091e-11
    1.0158146872e-11
    3.427091291e-12
    2.05404248208e-27
    3.52006063894e-12
    1.07243723536e-11
    1.45496993656e-11
    1.11284880557e-11
    3.79243117612e-12
    1.10129390403e-27
    4.00078247318e-12
    1.24057188876e-11
    1.71968078468e-11
    1.35087514081e-11
    4.7617253225e-12
    1.65561642357e-27
    5.57085328465e-12
    1.88107751545e-11
    2.97874337351e-11
    2.97482336482e-11
    1.87359616816e-11
    5.53402367163e-12
    9.19616841663e-29
    4.71213300971e-12
    1.33631623169e-11
    1.7027045061e-11
    1.23100093144e-11
    3.98389272921e-12
    1.02881331705e-27
    3.82421577611e-12
    1.13357986571e-11
    1.50170758671e-11
    1.1252106819e-11
    3.76791700125e-12
    1.17722174705e-27
    3.87065365488e-12
    1.18907373462e-11
    1.63707284112e-11
    1.2803939878e-11
    4.50496104631e-12
    3.11440188368e-27
    5.291703306e-12
    1.79747825906e-11
    2.87120567229e-11
    2.90098627726e-11
    1.85446869734e-11
    5.579540895e-12
    5.66690750289e-28
    4.99097852741e-12
    1.46150510228e-11
    1.93494665876e-11
    1.46495382254e-11
    5.01532980107e-12
    2.63216722891e-27
    5.64497778559e-12
    1.88605190122e-11
    2.9712316166e-11
    2.96780716107e-11
    1.87963885674e-11
    5.61444835215e-12
    1.98459696829e-27
    4.97884506513e-12
    1.45577715032e-11
    1.92819345016e-11
    1.4634250712e-11
    5.03330663595e-12
    7.34826311396e-27
    5.76022363804e-12
    1.94907066028e-11
    3.12078791803e-11
    3.1823177332e-11
    2.06938629034e-11
    6.39520024439e-12
    5.352534255e-28
    6.30315525823e-12
    2.0108413769e-11
    3.05020799358e-11
    2.95233185971e-11
    1.82116281635e-11
    5.32066188022e-12
    8.57391863032e-29
    4.5615199269e-12
    1.31724539456e-11
    1.72756419516e-11
    1.30139937542e-11
    4.45303335604e-12
    1.28092645707e-27
    5.07842880291e-12
    1.72123085186e-11
    2.76692590609e-11
    2.83946499183e-11
    1.86294405161e-11
    5.82476405141e-12
    6.38400821681e-28
    5.93261859852e-12
    1.93480018041e-11
    3.01410426308e-11
    3.01301764849e-11
    1.93313131902e-11
    5.9286887068e-12
    1.10466348928e-27
    5.85016014221e-12
    1.8829703211e-11
    2.89910850127e-11
    2.8676663777e-11
    1.82248888329e-11
    5.54175473435e-12
    1.35922656839e-27
    5.38907100868e-12
    1.7238639424e-11
    2.63949690265e-11
    2.59803072954e-11
    1.64393332685e-11
    4.97972640175e-12
    1.02229043916e-27
    4.81313457008e-12
    1.53609782326e-11
    2.34771040723e-11
    2.30767501399e-11
    1.4588828381e-11
    4.41721750641e-12
    6.59166885215e-28
    4.27177012196e-12
    1.36470257435e-11
    2.08893792918e-11
    2.05754252596e-11
    1.30416105338e-11
    3.96145030018e-12
    4.52794132095e-28
    3.86316661824e-12
    1.24066179452e-11
    1.91060546215e-11
    1.89504405561e-11
    1.21081344344e-11
    3.71187936801e-12
    1.45173806056e-27
    3.70317479678e-12
    1.20620736883e-11
    1.88844509642e-11
    1.90988096372e-11
    1.24905172238e-11
    3.93934645273e-12
    2.83757666524e-27
    4.2614226988e-12
    1.47539776313e-11
    2.51924637544e-11
    2.91505497893e-11
    2.427460628e-11
    1.37001580296e-11
    3.81444799715e-12
    7.41997464175e-29
    3.28247001353e-12
    1.00557358311e-11
    1.48740620459e-11
    1.42473815466e-11
    8.83026318076e-12
    2.63566365273e-12
    2.22830783493e-28
    2.51597206998e-12
    8.04795013562e-12
    1.24018949034e-11
    1.23712294176e-11
    7.99558105752e-12
    2.49664383023e-12
    4.69306142761e-27
    2.66146117342e-12
    9.17110590188e-12
    1.56127073252e-11
    1.80428274754e-11
    1.50327548081e-11
    8.50469446907e-12
    2.37841525441e-12
    7.19215488375e-28
    2.07906572209e-12
    6.44480543214e-12
    9.676925954e-12
    9.44503866023e-12
    5.99280248775e-12
    1.84212108336e-12
    2.17497944891e-27
    1.9157770328e-12
    6.53869149194e-12
    1.10429843776e-11
    1.26790958095e-11
    1.05097500968e-11
    5.92313170198e-12
    1.65225446521e-12
    2.99150270413e-28
    1.44250736251e-12
    4.47751445974e-12
    6.74103459333e-12
    6.60636100677e-12
    4.21504624164e-12
    1.30497349613e-12
    1.72819494967e-27
    1.38441739232e-12
    4.78818942923e-12
    8.21667757161e-12
    9.61650743608e-12
    8.15684434151e-12
    4.72682605284e-12
    1.36414611982e-12
    6.38390377209e-28
    1.3110121366e-12
    4.3689017445e-12
    7.26101758412e-12
    8.26223252675e-12
    6.8340462092e-12
    3.87104102434e-12
    1.09408391317e-12
    1.23691745509e-27
    1.01293429677e-12
    3.31882962286e-12
    5.42794320838e-12
    6.08252634682e-12
    4.95776583644e-12
    2.76878434015e-12
    7.71901861588e-13
    8.47396411254e-28
    6.96090458109e-13
    2.25182534378e-12
    3.63691715446e-12
    4.0252150815e-12
    3.24067178353e-12
    1.78771147106e-12
    4.92295076621e-13
    3.78007543992e-28
    4.33052860665e-13
    1.38333671149e-12
    2.20572808874e-12
    2.40946176985e-12
    1.91398052865e-12
    1.0413631332e-12
    2.82705406936e-13
    7.93243663276e-28
    2.41278527376e-13
    7.58380669488e-13
    1.18887725591e-12
    1.27560928737e-12
    9.94209063448e-13
    5.30085762816e-13
    1.40817106543e-13
    1.09893629639e-27
    1.14477265118e-13
    3.50054631653e-13
    5.32475505427e-13
    5.52669478863e-13
    4.15186685557e-13
    2.12452765697e-13
    5.38862932655e-14
    1.7916421014e-27
    3.91324018296e-14
    1.11577976408e-13
    1.56329283121e-13
    1.47091722399e-13
    9.80692002103e-14
    4.32518115824e-14
    9.0646793313e-15
    1.71074648186e-27
    3.5450469991e-15
    5.7038244048e-15
    2.62887039927e-15
    6.85473070104e-28
    3.42186898326e-15
    1.04968122144e-14
    1.28002546873e-14
    6.28479301297e-15
    2.45567546687e-28
    1.30522761833e-14
    5.90871721703e-14
    1.29316606869e-13
    1.89541349359e-13
    1.98800493053e-13
    1.41253001474e-13
    4.97405123606e-14
    1.74441139466e-27
    7.11117678478e-14
    2.91290956753e-13
    6.03097879267e-13
    8.76427764654e-13
    9.71600141152e-13
    8.22309362164e-13
    4.8963416018e-13
    1.47959760747e-13
    6.02235876712e-28
    1.61775979172e-13
    5.82890090806e-13
    1.05522258673e-12
    1.31703931366e-12
    1.20319681519e-12
    7.5821493285e-13
    2.40316860431e-13
    2.84483423416e-28
    2.88207984451e-13
    1.09652127205e-12
    2.12479499791e-12
    2.90927484574e-12
    3.05672147413e-12
    2.46484743273e-12
    1.40498868731e-12
    4.08187355362e-13
    3.89882291607e-28
    4.17308170908e-13
    1.46151614667e-12
    2.57999570447e-12
    3.14954158843e-12
    2.82244303489e-12
    1.74964461225e-12
    5.47041254444e-13
    4.1296499469e-27
    6.43760015688e-13
    2.43659371115e-12
    4.71129793876e-12
    6.45751943806e-12
    6.8160108486e-12
    5.54349580447e-12
    3.20170398032e-12
    9.47685255446e-13
    1.20199954234e-27
    1.0284748777e-12
    3.77270469336e-12
    7.09949192797e-12
    9.50176652828e-12
    9.81942092738e-12
    7.83653053791e-12
    4.44965822385e-12
    1.29695527143e-12
    6.74720058803e-28
    1.37051095615e-12
    4.96968044013e-12
    9.25428423969e-12
    1.22681182745e-11
    1.25691670087e-11
    9.95312924027e-12
    5.61214145258e-12
    1.62565982769e-12
    1.74564957302e-28
    1.70043038057e-12
    6.14132473573e-12
    1.13984071443e-11
    1.50716503772e-11
    1.54132152726e-11
    1.21923067971e-11
    6.87302734916e-12
    1.99213921162e-12
    4.65262533036e-28
    2.09240394669e-12
    7.58499459458e-12
    1.41478292717e-11
    1.88269964651e-11
    1.94093649874e-11
    1.55081773572e-11
    8.85165040933e-12
    2.60554210341e-12
    2.27039010691e-27
    2.85873505285e-12
    1.06930704441e-11
    2.07892320547e-11
    2.92850079981e-11
    3.27897019866e-11
    2.98476567127e-11
    2.15978336411e-11
    1.1325694653e-11
    3.08783369892e-12
    9.04647114753e-28
    2.93065436168e-12
    1.01683351905e-11
    1.82075908861e-11
    2.33091317874e-11
    2.31488871397e-11
    1.78296657187e-11
    9.80950343538e-12
    2.78086881769e-12
    1.7257152696e-27
    2.80980730373e-12
    1.00154453259e-11
    1.8397395428e-11
    2.41451772618e-11
    2.45836740499e-11
    1.9425186539e-11
    1.09790670793e-11
    3.20429608998e-12
    3.74664829867e-27
    3.46925258599e-12
    1.29147834216e-11
    2.50198672687e-11
    3.51637911529e-11
    3.93316578899e-11
    3.58124309804e-11
    2.59559723258e-11
    1.36523010885e-11
    3.73897860461e-12
    2.4721777827e-28
    3.59819860482e-12
    1.26050392754e-11
    2.28345869217e-11
    2.96401991855e-11
    2.99220232658e-11
    2.34942990292e-11
    1.32218790214e-11
    3.84960162262e-12
    1.67568545367e-27
    4.17064728582e-12
    1.557345908e-11
    3.03204567674e-11
    4.2911134371e-11
    4.84376592059e-11
    4.46155297801e-11
    3.28010179343e-11
    1.75564676189e-11
    4.91160764476e-12
    7.41077103088e-28
    5.00910516241e-12
    1.82655402792e-11
    3.48330632811e-11
    4.84045605596e-11
    5.37549832837e-11
    4.87928859616e-11
    3.53996082295e-11
    1.87201287082e-11
    5.17972282192e-12
    9.39383300264e-28
    5.18088549799e-12
    1.87302480171e-11
    3.54365360525e-11
    4.88823631157e-11
    5.39174231121e-11
    4.86332348264e-11
    3.50791368607e-11
    1.84513114729e-11
    5.08014121961e-12
    3.75084715821e-28
    5.03709011069e-12
    1.81408171897e-11
    3.42019135197e-11
    4.70306199949e-11
    5.17281964983e-11
    4.65413500518e-11
    3.34963518086e-11
    1.75853691722e-11
    4.8339811172e-12
    1.51219933965e-28
    4.78186241368e-12
    1.72089579027e-11
    3.24303466268e-11
    4.45872707625e-11
    4.90472782174e-11
    4.41485145196e-11
    3.17980396927e-11
    1.67116853221e-11
    4.6002772314e-12
    4.94650385477e-28
    4.56818049228e-12
    1.64803508707e-11
    3.11452530587e-11
    4.29585819301e-11
    4.74278730142e-11
    4.28655884604e-11
    3.10154203306e-11
    1.63836997024e-11
    4.53567763078e-12
    5.25245282062e-28
    4.56455943813e-12
    1.65964816095e-11
    3.16387250042e-11
    4.40644870833e-11
    4.91793502916e-11
    4.49934699563e-11
    3.30065353337e-11
    1.77112809338e-11
    4.99256973054e-12
    1.615052578e-27
    5.26051385636e-12
    1.97094521999e-11
    3.89909530232e-11
    5.6928339793e-11
    6.76493961493e-11
    6.76174918583e-11
    5.68525330662e-11
    3.89150327815e-11
    1.96667612481e-11
    5.25081938952e-12
    3.94990044257e-28
    4.99752898429e-12
    1.77790757045e-11
    3.3266239686e-11
    4.55951034907e-11
    5.01952376194e-11
    4.53916929532e-11
    3.29754114393e-11
    1.75534099391e-11
    4.91668923353e-12
    4.86656527093e-28
    5.13925405843e-12
    1.92226532733e-11
    3.80230698499e-11
    5.55975759836e-11
    6.62784223567e-11
    6.6579545745e-11
    5.63736500461e-11
    3.89463627348e-11
    1.99173042503e-11
    5.39754262313e-12
    4.49356207378e-28
    5.35609483567e-12
    1.96158143543e-11
    3.80815978768e-11
    5.47565057075e-11
    6.42920460102e-11
    6.3696415467e-11
    5.32523286242e-11
    3.63621444767e-11
    1.83954583261e-11
    4.93521920038e-12
    3.67590924855e-28
    4.80922765022e-12
    1.7468811598e-11
    3.36527765174e-11
    4.8038746147e-11
    5.6020923426e-11
    5.51467229073e-11
    4.58266811559e-11
    3.11141467093e-11
    1.56563908531e-11
    4.17922836445e-12
    1.38558184156e-27
    4.03516535705e-12
    1.45957549947e-11
    2.80074760837e-11
    3.98330925619e-11
    4.62922038337e-11
    4.54239963712e-11
    3.76350013723e-11
    2.54821548e-11
    1.27899715009e-11
    3.40617320479e-12
    1.38072086113e-27
    3.27563859018e-12
    1.1828610945e-11
    2.26646626445e-11
    3.21946482338e-11
    3.73773871476e-11
    3.66475420618e-11
    3.03463380806e-11
    2.05400452917e-11
    1.03081662166e-11
    2.74550419737e-12
    8.80642939771e-28
    2.64264907933e-12
    9.55059224689e-12
    1.83194849495e-11
    2.60577606682e-11
    3.0302723299e-11
    2.97697657324e-11
    2.47082971603e-11
    1.67688208246e-11
    8.44146574873e-12
    2.25620451819e-12
    8.33289720109e-29
    2.19008157652e-12
    7.95498917979e-12
    1.53456189747e-11
    2.19673320673e-11
    2.57299933904e-11
    2.54830053503e-11
    2.13451539554e-11
    1.46381623854e-11
    7.45732665819e-12
    2.02078598676e-12
    7.08317616278e-28
    2.0313199311e-12
    7.54728601143e-12
    1.49670210526e-11
    2.21783704393e-11
    2.71608442343e-11
    2.85622468582e-11
    2.60617851193e-11
    2.04221726119e-11
    1.32287031936e-11
    6.405050249e-12
    1.65591235114e-12
    6.36217326546e-28
    1.52230373387e-12
    5.40546081639e-12
    1.02173544322e-11
    1.43598545064e-11
    1.65415180556e-11
    1.61364388444e-11
    1.33312383558e-11
    9.02847233521e-12
    4.54748744523e-12
    1.21967638398e-12
    3.26239736736e-28
    1.20482725459e-12
    4.44433166347e-12
    8.7590817418e-12
    1.29124653332e-11
    1.57487863282e-11
    1.65126683352e-11
    1.50414228541e-11
    1.17826514811e-11
    7.64167614309e-12
    3.71109919483e-12
    9.64372726465e-13
    1.85347442952e-28
    9.03167731417e-13
    3.25517378434e-12
    6.27866399476e-12
    9.0704219987e-12
    1.08525469789e-11
    1.11721440064e-11
    9.99862327842e-12
    7.69953195471e-12
    4.91096745962e-12
    2.34628307492e-12
    5.99963649351e-13
    1.18722695485e-27
    5.44267445166e-13
    1.93070474531e-12
    3.66503142373e-12
    5.2101571611e-12
    6.13318738718e-12
    6.21033783989e-12
    5.46525479035e-12
    4.13682057496e-12
    2.59247102873e-12
    1.2163363262e-12
    3.05261716158e-13
    7.88532455664e-28
    2.66193648001e-13
    9.2465827123e-13
    1.71710550544e-12
    2.38530384935e-12
    2.74037640681e-12
    2.70432728437e-12
    2.31570792701e-12
    1.70251090335e-12
    1.03419889323e-12
    4.69253739741e-13
    1.13591794996e-13
    6.60723000705e-28
    9.1280511837e-14
    3.02626438879e-13
    5.33850350587e-13
    7.00569088648e-13
    7.55329521766e-13
    6.94006494905e-13
    5.48006854795e-13
    3.67147781103e-13
    2.0023921063e-13
    8.00371755993e-14
    1.66469521408e-14
    2.60448050178e-28
    8.79880625259e-15
    2.14982608557e-14
    2.48512863179e-14
    1.71219150476e-14
    5.54616879503e-15
    1.7534439341e-27
    6.25201951872e-15
    2.21071012405e-14
    3.83089510837e-14
    4.38526033388e-14
    3.3293464822e-14
    1.25448623365e-14
    2.30218467236e-27
    2.15156666159e-14
    1.00290850841e-13
    2.45040799884e-13
    4.41199820282e-13
    6.49525673731e-13
    8.14346580227e-13
    8.80383966723e-13
    8.13671552346e-13
    6.1976428806e-13
    3.52282219014e-13
    1.06897233187e-13
    1.47885815504e-27
    1.36118899479e-13
    5.72511243374e-13
    1.29158536494e-12
    2.19113395085e-12
    3.09834743689e-12
    3.80722557263e-12
    4.13179245783e-12
    3.96179026362e-12
    3.30504721432e-12
    2.30273676912e-12
    1.20989213109e-12
    3.42249660613e-13
    6.53240821973e-28
    3.86344846939e-13
    1.54218984713e-12
    3.31632116515e-12
    5.38296061883e-12
    7.30711265899e-12
    8.64511234857e-12
    9.05721887667e-12
    8.40379947626e-12
    6.79867744081e-12
    4.60261708548e-12
    2.35394604192e-12
    6.49218836897e-13
    6.23469598804e-28
    6.9971251239e-13
    2.73451690018e-12
    5.763770773e-12
    9.18019537838e-12
    1.22404498535e-11
    1.42381797838e-11
    1.46790505623e-11
    1.34141354674e-11
    1.06964658191e-11
    7.14293800613e-12
    3.60608565566e-12
    9.82417208741e-13
    6.54377664471e-28
    1.03512333879e-12
    4.00343727655e-12
    8.35577532337e-12
    1.31855313345e-11
    1.74275544751e-11
    2.01051083132e-11
    2.05671582979e-11
    1.86581395621e-11
    1.47766252898e-11
    9.80475056617e-12
    4.9205489386e-12
    1.33315917267e-12
    9.95555491758e-28
    1.39110869613e-12
    5.3576107332e-12
    1.1139819391e-11
    1.75195777206e-11
    2.30875030443e-11
    2.65668904155e-11
    2.71195389987e-11
    2.45602620523e-11
    1.94260255351e-11
    1.28790794978e-11
    6.46108570521e-12
    1.7507844533e-12
    1.53794651333e-27
    1.83030511982e-12
    7.06173661928e-12
    1.47184381321e-11
    2.32182427378e-11
    3.07114143933e-11
    3.54972710998e-11
    3.64256576034e-11
    3.31895420349e-11
    2.64369855687e-11
    1.76703538427e-11
    8.94839162867e-12
    2.45127098629e-12
    1.07334825105e-27
    2.6336108729e-12
    1.03382416689e-11
    2.19933363704e-11
    3.55566233257e-11
    4.84588973499e-11
    5.81320613266e-11
    6.25605319638e-11
    6.07358298627e-11
    5.29111532406e-11
    4.05946413148e-11
    2.62760204812e-11
    1.29390090786e-11
    3.45768939828e-12
    2.75482897598e-28
    3.56111445361e-12
    1.37254454046e-11
    2.87121558239e-11
    4.5703327232e-11
    6.13957279815e-11
    7.26680066852e-11
    7.72267524592e-11
    7.40948648591e-11
    6.38362955591e-11
    4.84661421962e-11
    3.10617158977e-11
    1.51526405829e-11
    4.01330482289e-12
    5.51676428311e-28
    4.0656033141e-12
    1.55502681968e-11
    3.22932676316e-11
    5.10486890312e-11
    6.81262366718e-11
    8.01312564854e-11
    8.46535666394e-11
    8.07633372753e-11
    6.9209485893e-11
    5.22787302964e-11
    3.33435580868e-11
    1.61912829305e-11
    4.2697778204e-12
    1.4025784105e-27
    4.29094604415e-12
    1.63523580303e-11
    3.38432155596e-11
    5.33291171792e-11
    7.09606641745e-11
    8.32397387549e-11
    8.77201919888e-11
    8.3500826758e-11
    7.14096686251e-11
    5.38418511085e-11
    3.4284296753e-11
    1.66239872593e-11
    4.37835494267e-12
    1.63945407028e-27
    4.39146203598e-12
    1.67240593107e-11
    3.45963550879e-11
    5.45026923527e-11
    7.25218431744e-11
    8.50916123434e-11
    8.97164238186e-11
    8.54662410772e-11
    7.31662350005e-11
    5.52389483545e-11
    3.52304737853e-11
    1.71154112559e-11
    4.51782467703e-12
    7.85054048855e-28
    4.5562421054e-12
    1.74089498455e-11
    3.61471058018e-11
    5.71829013418e-11
    7.64416574164e-11
    9.01543195694e-11
    9.5599428218e-11
    9.16495089986e-11
    7.90124612095e-11
    6.01190804563e-11
    3.86762989204e-11
    1.89717439799e-11
    5.06230414835e-12
    5.40519833351e-28
    5.24001369023e-12
    2.03411001703e-11
    4.30148379355e-11
    6.95172701645e-11
    9.53143231833e-11
    1.15901768078e-10
    1.2762990452e-10
    1.2838033105e-10
    1.17967360564e-10
    9.81879456883e-11
    7.25096448635e-11
    4.54546846902e-11
    2.17939268622e-11
    5.69823501135e-12
    1.00826707446e-27
    5.69513182656e-12
    2.17718148816e-11
    4.53940393072e-11
    7.24061116673e-11
    9.80684693738e-11
    1.17894333498e-10
    1.28438275165e-10
    1.27896311792e-10
    1.16410612219e-10
    9.60267942552e-11
    7.03148385122e-11
    4.37263480506e-11
    2.08062738457e-11
    5.40083481744e-12
    8.61092543653e-28
    5.32601069081e-12
    2.02343789174e-11
    4.19391225491e-11
    6.65186676335e-11
    8.96113656117e-11
    1.07177896808e-10
    1.16196953328e-10
    1.15173257356e-10
    1.04371253918e-10
    8.57377857574e-11
    6.25332198777e-11
    3.87417206834e-11
    1.83690118803e-11
    4.75213191117e-12
    6.5243073742e-28
    4.65732048618e-12
    1.76438324482e-11
    3.64728800391e-11
    5.77062493724e-11
    7.75630891483e-11
    9.25757737892e-11
    1.00179258568e-10
    9.91332640232e-11
    8.9707697144e-11
    7.36038786636e-11
    5.36313929834e-11
    3.32023437788e-11
    1.57347873231e-11
    4.06963064692e-12
    1.03736482649e-28
    3.98948699136e-12
    1.51219538961e-11
    3.12857771015e-11
    4.95563293188e-11
    6.67082985183e-11
    7.9768931747e-11
    8.65177186022e-11
    8.58490518362e-11
    7.7938937418e-11
    6.4191792334e-11
    4.69815653302e-11
    2.92360520088e-11
    1.39382491818e-11
    3.63002524061e-12
    5.54634174909e-28
    3.62080087724e-12
    1.38744773106e-11
    2.90732917207e-11
    4.67515186489e-11
    6.40770284209e-11
    7.83130590879e-11
    8.72541383764e-11
    8.95656953008e-11
    8.49746194558e-11
    7.4283879716e-11
    5.921462303e-11
    4.210700196e-11
    2.55334972706e-11
    1.18899832229e-11
    3.03027357869e-12
    5.02403729011e-28
    2.9071173439e-12
    1.09434300608e-11
    2.25474248924e-11
    3.5677329612e-11
    4.8147264354e-11
    5.7971498965e-11
    6.36622041e-11
    6.4435864091e-11
    6.0299709298e-11
    5.20101401984e-11
    4.09163739756e-11
    2.87201091815e-11
    1.719403671e-11
    7.90571232513e-12
    1.98964528122e-12
    3.16792951144e-28
    1.86164959756e-12
    6.92105746893e-12
    1.40829624384e-11
    2.20065560671e-11
    2.93271354028e-11
    3.48672165929e-11
    3.78048072567e-11
    3.77749881704e-11
    3.48933131563e-11
    2.97025776815e-11
    2.30568538011e-11
    1.59658274011e-11
    9.42710366992e-12
    4.27381091963e-12
    1.0602012157e-12
    6.94303720343e-28
    9.62751364711e-13
    3.52387550102e-12
    7.05619024161e-12
    1.08450602304e-11
    1.42071854057e-11
    1.65938181888e-11
    1.76633723825e-11
    1.73144307155e-11
    1.56773683894e-11
    1.30697920409e-11
    9.92653440311e-12
    6.71816243469e-12
    3.87249395065e-12
    1.71167588941e-12
    4.13399154032e-13
    6.71959817591e-28
    3.54091061312e-13
    1.2551760539e-12
    2.42878886307e-12
    3.59852635463e-12
    4.53186548805e-12
    5.07267597545e-12
    5.1564005715e-12
    4.80731601396e-12
    4.12063403519e-12
    3.23452729345e-12
    2.29854695244e-12
    1.44475974957e-12
    7.66634274132e-13
    3.08638087205e-13
    6.70163853841e-14
    8.87008140338e-28
    4.40098485347e-14
    1.31647224151e-13
    2.073934007e-13
    2.3789945346e-13
    2.15255491746e-13
    1.53368347913e-13
    7.9148052749e-14
    2.13115267124e-14
    6.0794487697e-28
    2.00366961655e-14
    6.96164183613e-14
    1.24757477626e-13
    1.58331074541e-13
    1.51233852384e-13
    1.02621137622e-13
    3.61516769915e-14
    4.23914785278e-28
    5.97355558792e-14
    2.84823125461e-13
    7.31124561899e-13
    1.42301124137e-12
    2.33927198172e-12
    3.40669790076e-12
    4.50405789711e-12
    5.47728835752e-12
    6.16443574377e-12
    6.4266353127e-12
    6.17978661244e-12
    5.42067464087e-12
    4.24177370308e-12
    2.83036661777e-12
    1.45019527747e-12
    4.06680277691e-13
    2.03889574686e-28
    4.72751483458e-13
    1.96070166725e-12
    4.45547547779e-12
    7.78693752889e-12
    1.16305691649e-11
    1.55413566257e-11
    1.90114207104e-11
    2.15442200683e-11
    2.27348073264e-11
    2.23433247327e-11
    2.03493944521e-11
    1.69767397496e-11
    1.26817331158e-11
    8.10477184757e-12
    3.9891623528e-12
    1.07753399435e-12
    3.31787760349e-28
    1.17038810385e-12
    4.70685087828e-12
    1.03906598754e-11
    1.7672252399e-11
    2.57271824074e-11
    3.35572749405e-11
    4.01249773763e-11
    4.45030800985e-11
    4.60182665442e-11
    4.43665108091e-11
    3.96814324846e-11
    3.2542627095e-11
    2.39194110452e-11
    1.5054876739e-11
    7.30391502123e-12
    1.94625782932e-12
    1.6482951619e-28
    2.06211267493e-12
    8.19994286589e-12
    1.79117670082e-11
    3.01656604231e-11
    4.35156449861e-11
    5.62827845398e-11
    6.67792399101e-11
    7.35456247936e-11
    7.55686305146e-11
    7.24469424081e-11
    6.44795221128e-11
    5.26602958833e-11
    3.85761220637e-11
    2.42182469242e-11
    1.1730154238e-11
    3.12354270346e-12
    4.66018678002e-28
    3.31571892867e-12
    1.32223915662e-11
    2.90084257406e-11
    4.9152349376e-11
    7.14860723708e-11
    9.34509088401e-11
    1.12416145038e-10
    1.26020242175e-10
    1.32489739989e-10
    1.30892264853e-10
    1.21287493366e-10
    1.04752735438e-10
    8.32762639904e-11
    5.95292639555e-11
    3.65438305369e-11
    1.73370202178e-11
    4.52842115022e-12
    1.6369210881e-27
    4.64190155773e-12
    1.82177885142e-11
    3.9368843449e-11
    6.57587968302e-11
    9.43449391994e-11
    1.21743969739e-10
    1.44648709429e-10
    1.60244066088e-10
    1.6656898259e-10
    1.62777016674e-10
    1.49260021111e-10
    1.2761689845e-10
    1.00469367486e-10
    7.11469173509e-11
    4.32800012716e-11
    2.03527276302e-11
    5.27099273024e-12
    2.30192925718e-27
    5.31610752715e-12
    2.07036839052e-11
    4.44098853054e-11
    7.36512768603e-11
    1.04947727831e-10
    1.34543709417e-10
    1.58865892707e-10
    1.74961417285e-10
    1.80862399081e-10
    1.75831379806e-10
    1.60456524256e-10
    1.36584358351e-10
    1.07098127363e-10
    7.55694796722e-11
    4.58267844214e-11
    2.14937935743e-11
    5.55494292958e-12
    1.94002428981e-27
    5.59000317827e-12
    2.17713045477e-11
    4.67463839257e-11
    7.76913874942e-11
    1.110931007e-10
    1.43164246478e-10
    1.70287080484e-10
    1.8943235278e-10
    1.98503764165e-10
    1.96567984155e-10
    1.83943982114e-10
    1.62143156068e-10
    1.33670123852e-10
    1.01710865476e-10
    6.97481237382e-11
    4.11517724574e-11
    1.87936410576e-11
    4.73189116453e-12
    1.02086772755e-27
    4.52218873232e-12
    1.71610986937e-11
    3.58880404826e-11
    5.80518924286e-11
    8.07119905099e-11
    1.00993726896e-10
    1.16424049655e-10
    1.25207750096e-10
    1.26413311462e-10
    1.2004930226e-10
    1.07025605572e-10
    8.90086849419e-11
    6.81921311875e-11
    4.70137983888e-11
    2.78558327876e-11
    1.27645200868e-11
    3.22275773522e-12
    1.24660689714e-27
    3.09385757556e-12
    1.17659546356e-11
    2.46631264771e-11
    4.00054210214e-11
    5.58152217878e-11
    7.01582078246e-11
    8.13668944266e-11
    8.8220660783e-11
    9.00631205638e-11
    8.68457014138e-11
    7.9096290691e-11
    6.78209160401e-11
    5.43543732172e-11
    4.01812888489e-11
    2.67514207093e-11
    1.5312354104e-11
    6.77899690636e-12
    1.65322733696e-12
    2.35297820203e-27
    1.47840609478e-12
    5.41957336585e-12
    1.09379459972e-11
    1.70593186052e-11
    2.28481841912e-11
    2.7518351369e-11
    3.05142001333e-11
    3.15546345841e-11
    3.0637362634e-11
    2.80065344354e-11
    2.40914526176e-11
    1.94273630927e-11
    1.45709905566e-11
    1.00230493068e-11
    6.16798624487e-12
    3.23774944412e-12
    1.30229783897e-12
    2.85324499031e-13
    1.87684731358e-27
    1.9699375362e-13
    6.15818204677e-13
    1.03082494181e-12
    1.28479615906e-12
    1.30568581269e-12
    1.10523052644e-12
    7.61343409433e-13
    3.88946384212e-13
    1.05727970465e-13
    1.06035056767e-27
    1.07242022463e-13
    4.00181220975e-13
    7.94640020187e-13
    1.17034838384e-12
    1.40295202211e-12
    1.40111210673e-12
    1.14124621877e-12
    6.92396889715e-13
    2.25036961869e-13
    2.71585426817e-28
    3.37045894462e-13
    1.56609954276e-12
    3.96755922275e-12
    7.71060782114e-12
    1.27997949306e-11
    1.90399451306e-11
    2.60276569901e-11
    3.3174415396e-11
    3.97619418353e-11
    4.50256207153e-11
    4.82571123498e-11
    4.89134305423e-11
    4.67174818482e-11
    4.1734687821e-11
    3.44123146324e-11
    2.55717892919e-11
    1.63501613951e-11
    8.09316450744e-12
    2.20927367936e-12
    4.60604631093e-28
    2.48614526729e-12
    1.0251877113e-11
    2.33279445838e-11
    4.11321865163e-11
    6.24795000845e-11
    8.56658433644e-11
    1.0862187388e-10
    1.29122745732e-10
    1.45032872928e-10
    1.54558653817e-10
    1.56479197136e-10
    1.50326111187e-10
    1.36487748299e-10
    1.16221020209e-10
    9.15641475089e-11
    6.51549509428e-11
    3.99710291654e-11
    1.90179179819e-11
    4.99838989644e-12
    4.6196368125e-28
    5.23699537263e-12
    2.0878137721e-11
    4.5983268966e-11
    7.8559997735e-11
    1.15738528863e-10
    1.54049410254e-10
    1.89776502806e-10
    2.19348352284e-10
    2.39725144098e-10
    2.48738478998e-10
    2.45344283382e-10
    2.29759357027e-10
    2.03463495784e-10
    1.69064862501e-10
    1.30041251431e-10
    9.03843611127e-11
    5.41850441177e-11
    2.52046684e-11
    6.47931592874e-12
    2.761504833e-28
    6.50372294648e-12
    2.53979179893e-11
    5.48260327954e-11
    9.18664774504e-11
    1.32842621792e-10
    1.73709254458e-10
    2.10473025172e-10
    2.39600837299e-10
    2.58372038929e-10
    2.65137117302e-10
    2.59456703249e-10
    2.42108444804e-10
    2.14964471667e-10
    1.80757094121e-10
    1.427617382e-10
    1.04435625087e-10
    6.90535872815e-11
    3.93815585752e-11
    1.7422235463e-11
    4.25749698501e-12
    1.96878082558e-28
    3.85315228988e-12
    1.42662296851e-11
    2.91610939724e-11
    4.62000562556e-11
    6.306136215e-11
    7.76900871049e-11
    8.84971168926e-11
    9.44870152673e-11
    9.53055131399e-11
    9.12077491225e-11
    8.29583745809e-11
    7.16818827823e-11
    5.86866609796e-11
    4.52874972414e-11
    3.26490979356e-11
    2.16686209891e-11
    1.29079917719e-11
    6.57905020729e-12
    2.57703103773e-12
    5.5152899657e-13
    8.46162237227e-28
    3.67301130915e-13
    1.1357685157e-12
    1.8936174527e-12
    2.37427695955e-12
    2.46475484247e-12
    2.18642155787e-12
    1.65575947709e-12
    1.03515192161e-12
    4.84269526031e-13
    1.21269039961e-13
    4.76226242896e-28
    1.05651073109e-13
    3.67269542344e-13
    6.82286590518e-13
    9.4611513495e-13
    1.0794102619e-12
    1.04671060311e-12
    8.62589818131e-13
    5.8461785477e-13
    2.95564531212e-13
    7.97859461157e-14
    4.42496469312e-28
    8.03937726549e-14
    3.00354120576e-13
    6.00362290187e-13
    8.98456396232e-13
    1.11287537334e-12
    1.18477252005e-12
    1.09471668699e-12
    8.68210935715e-13
    5.6835416974e-13
    2.77357423183e-13
    7.19707136336e-14
    4.22067171774e-28
    6.52614713334e-14
    2.26257076226e-13
    4.09936121533e-13
    5.37092942005e-13
    5.51633804734e-13
    4.43666903887e-13
    2.57196597486e-13
    7.74799225551e-14
    1.02461703116e-28
    9.05830024906e-14
    3.51678718318e-13
    7.10484551639e-13
    1.03873230156e-12
    1.20197725418e-12
    1.12150687985e-12
    8.20573115756e-13
    4.25940128353e-13
    1.11269361257e-13
    3.94988712652e-27
    8.09897868582e-14
    2.06714894644e-13
    2.10917742871e-13
    8.2264425814e-14
    9.73812786229e-27
    4.7300983273e-14
    1.17831609412e-26

and plot the results:

```
plt.scatter(np.linspace(0, size, size), np.log10(list_of_errors))
plt.ylabel('log10 interpolation error')
plt.show()
```

and we see that the error indeed has an upper bound of around $10^{-10}$, so we have a pretty damn accurate interpolant!!

```

```

ROQ/Making the empirical interpolant.ipynb


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

A decision tree is the easiest classifer to understand, interpret, and explain.
A decision tree is constructed by recursively splitting the data set into a
sequence of subsets based on if-then questions.  The training set consists of
pairs $(\mathbf{x},y)$ where $\mathbf{x}\in \mathbb{R}^d$ where $d$ is the
number of features available and where $y$ is the corresponding label. The
learning method splits the training set into groups based on $\mathbf{x}$ while
attempting to keep the assignments in each group as uniform as possible. In order to
do this, the learning method must pick a feature and an associated threshold for that feature upon which to
divide the data.  This is tricky to explain in words, but easy to see with
an example. First, let's set up the Scikit-learn classifer,

```python
%matplotlib inline

from matplotlib.pylab import subplots
from numpy import ma
import numpy as np
np.random.seed(12345678)
```

```python
from sklearn import tree
clf = tree.DecisionTreeClassifier()
```

 Let's also create some example data,

```python
import numpy as np
M=np.fromfunction(lambda i,j:j>=2,(4,4)).astype(int)
print(M)
```

    [[0 0 1 1]
     [0 0 1 1]
     [0 0 1 1]
     [0 0 1 1]]

**Programming Tip.**

The `fromfunction` creates Numpy arrays using the indicies as inputs
to a function whose value is the corresponding array entry.

 We want to classify the elements of the matrix based on their
respective positions in the matrix. By just looking at the matrix, the
classification is pretty simple --- classify as `0` for any positions in the
first two columns of the matrix, and classify `1` otherwise.  Let's walk
through this formally and see if this solution emerges from the decision
tree.  The values of the array are the labels for the training set and the
indicies of those values are the elements of $\mathbf{x}$. Specifically, the
training set has $\mathcal{X} = \left\{(i,j)\right\}$ and
$\mathcal{Y}=\left\{0,1\right\}$ Now, let's extract those elements and construct
the training set.

```python
i,j = np.where(M==0)
x=np.vstack([i,j]).T # build nsamp by nfeatures
y = j.reshape(-1,1)*0 # 0 elements
print(x)
print(y)
```

    [[0 0]
     [0 1]
     [1 1]
     [2 0]
     [2 1]
     [3 0]
     [3 1]]
    [[0]
     [0]
     [0]
     [0]
     [0]
     [0]
     [0]]

 Thus, the elements of `x` are the two-dimensional indicies of the
values of `y`. For example, `M[x[0,0],x[0,1]]=y[0,0]`. Likewise,
to complete the training set, we just need to stack the rest of the data to cover
all the cases,

```python
i,j = np.where(M==1)
x=np.vstack([np.vstack([i,j]).T,x ]) # build nsamp x nfeatures
y=np.vstack([j.reshape(-1,1)*0+1,y]) # 1 elements
```

 With all that established, all we have to do is train the classifer,

```python
clf.fit(x,y)
```

    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                           max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort=False,
                           random_state=None, splitter='best')

 To evaluate how the classifer performed, we can report the score,

```python
clf.score(x,y)
```

    1.0

 For this classifer, the *score* is the accuracy, which is
defined as the ratio of the sum of the true-positive ($TP$) and
true-negatives ($TN$) divided by the sum of all the terms, including
the false terms,

$$
\texttt{accuracy}=\frac{TP+TN}{TP+TN+FN+FP}
$$

 In this case, the classifier gets every point correctly, so
$FN=FP=0$. On a related note, two other common names from information retrieval
theory are  *recall* (a.k.a. sensitivity) and *precision* (a.k.a. positive
predictive value, $TP/(TP+FP)$).  We can visualize this tree in [Figure](#fig:example_tree_001). The Gini coefficients (a.k.a. categorical variance)
in the figure are a measure of the purity of each so-determined class. This
coefficient is defined as,

Example decision tree. The Gini coefficient in each branch measures the purity of the partition in each node.  The samples item in the box shows the number of items in the corresponding node in the decision tree.

$$
\texttt{Gini}_m = \sum_k p_{m,k}(1-p_{m,k})
$$

 where

$$
p_{m,k} = \frac{1}{N_m} \sum_{x_i\in R_m} I(y_i=k)
$$

 which is the proportion of observations labeled $k$ in the $m^{th}$
node and $I(\cdot)$ is the usual indicator function. Note that the maximum value of
the Gini coefficient is $\max{\texttt{Gini}_{m}}=1-1/m$. For our simple
example, half of the sixteen samples are in category `0` and the other half are
in the `1` category.  Using the notation above, the top box corresponds to the
$0^{th}$ node, so $p_{0,0} =1/2 = p_{0,1}$. Then, $\texttt{Gini}_0=0.5$. The
next layer of nodes in [Figure](#fig:example_tree_001) is determined by
whether or not the second dimension of the $\mathbf{x}$ data is greater than
`1.5`. The Gini coefficients for each of these child nodes is zero because
after the prior split, each subsequent category is pure. The `value` list in
each of the nodes shows the distribution of elements in each category at each
node.

To make this example more interesting, we can contaminate the data slightly,

```python
M[1,0]=1 # put in different class
print(M) # now contaminated
```

    [[0 0 1 1]
     [1 0 1 1]
     [0 0 1 1]
     [0 0 1 1]]

  Now we have a `1` entry in the previously
pure first column's second  row. Let's re-do
the analysis as in the following,

```python
i,j = np.where(M==0)
x=np.vstack([i,j]).T
y = j.reshape(-1,1)*0
i,j = np.where(M==1)
x=np.vstack([np.vstack([i,j]).T,x])
y = np.vstack([j.reshape(-1,1)*0+1,y])
clf.fit(x,y)
```

    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                           max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort=False,
                           random_state=None, splitter='best')

 The result is shown
in [Figure](#fig:example_tree_002). Note the tree has grown
significantly due to this one change! The $0^{th}$ node has the
following parameters, $p_{0,0} =7/16$ and $p_{0,1}=9/16$. This makes
the Gini coefficient for the $0^{th}$ node equal to
$\frac{7}{16}\left(1-\frac{7}{16}\right)+\frac{9}{16}(1-\frac{9}{16})
= 0.492$.  As before, the root node splits on $X[1] \leq 1.5$. Let's
see if we can reconstruct the succeeding layer of nodes manually, as
in the following,

```python
y[x[:,1]>1.5] # first node on the right
```

    array([[1],
           [1],
           [1],
           [1],
           [1],
           [1],
           [1],
           [1]])

 This obviously has a zero Gini coefficient. Likewise, the node on the
left contains the following,

```python
y[x[:,1]<=1.5] # first node on the left
```

    array([[1],
           [0],
           [0],
           [0],
           [0],
           [0],
           [0],
           [0]])

 The Gini coefficient in this case is computed as
`(1/8)*(1-1/8)+(7/8)*(1-7/8)=0.21875`. This node splits based on `X[1]<0.5`.
The child node to the right derives from the following equivalent logic,

```python
np.logical_and(x[:,1]<=1.5,x[:,1]>0.5)
```

    array([False, False, False, False, False, False, False, False, False,
           False,  True,  True, False,  True, False,  True])

 with corresponding classes,

```python
y[np.logical_and(x[:,1]<=1.5,x[:,1]>0.5)]
```

    array([[0],
           [0],
           [0],
           [0]])

**Programming Tip.**

The  `logical_and` in Numpy provides element-wise logical conjuction. It is not
possible to accomplish this with something like `0.5< x[:,1] <=1.5` because
of the way Python parses this syntax.

Decision tree for contaminated data. Note that just one change in the training data caused the tree to grow five times as large as before!

Notice that for this example as well as for the previous one, the decision tree
was exactly able memorize (overfit) the data with perfect accuracy. From our
discussion of machine learning theory, this is an indication potential problems
in generalization.

The key step in building the decision tree is to come up with the
initial split. There are the number of algorithms that can build
decision trees based on different criteria, but the general idea is to
control the information *entropy* as the tree is developed.  In
practical terms, this means that the algorithms attempt to build trees
that are not excessively deep. It is well-established that this is a
very hard problem to solve completely and there are many approaches to
it.  This is because the algorithms must make global decisions at each
node of the tree using the local data available up to that point.

```python
_=clf.fit(x,y)
fig,axs=subplots(2,2,sharex=True,sharey=True)
ax=axs[0,0]
ax.set_aspect(1)
_=ax.axis((-1,4,-1,4))
ax.invert_yaxis()

# same background all on axes
for ax in axs.flat:
    _=ax.plot(ma.masked_array(x[:,1],y==1),ma.masked_array(x[:,0],y==1),'ow',mec='k')
    _=ax.plot(ma.masked_array(x[:,1],y==0),ma.masked_array(x[:,0],y==0),'o',color='gray')

lines={'h':[],'v':[]}
nc=0
for i,j,ax in zip(clf.tree_.feature,clf.tree_.threshold,axs.flat):
    _=ax.set_title('node %d'%(nc))
    nc+=1
    if i==0:   _=lines['h'].append(j)
    elif i==1: _=lines['v'].append(j)
    for l in lines['v']: _=ax.vlines(l,-1,4,lw=3)
    for l in lines['h']: _=ax.hlines(l,-1,4,lw=3)

fig.savefig('fig-machine_learning/example_tree_003.png')
```

The decision tree divides the training set into regions by splitting successively along each dimension until each region is as pure as possible.

For this example, the decision tree partitions the $\mathcal{X}$ space into
different regions corresponding to different $\mathcal{Y}$ labels as shown in
[Figure](#fig:example_tree_003). The root node at the top of [Figure](#fig:example_tree_002) splits the input data based on $X[1] \leq 1.5$. This
corresponds to the top left panel in [Figure](#fig:example_tree_003) (i.e.,
`node 0`) where the vertical line divides the training data shown into two
regions, corresponding to the two subsequent child nodes. The next split
happens with $X[1] \leq 0.5$ as shown in the next panel of [Figure](#fig:example_tree_003) titled `node 1`. This continues until the last panel
on the lower right, where the contaminated element we injected has been
isolated into its own sub-region. Thus, the last panel is a representation of
[Figure](#fig:example_tree_002), where the horizontal/vertical lines
correspond to successive splits in the decision tree.

```python
i,j = np.indices((5,5))
x=np.vstack([i.flatten(),j.flatten()]).T
y=(x[:,0]>=x[:,1]).astype(int).reshape((-1,1))
_=clf.fit(x,y)
fig,ax=subplots()
_=ax.axis((-1,5,-1,5))
ax.set_aspect(1)
ax.invert_yaxis()

_=ax.plot(ma.masked_array(x[:,1],y==1),ma.masked_array(x[:,0],y==1),'ow',mec='k',ms=15)
_=ax.plot(ma.masked_array(x[:,1],y==0),ma.masked_array(x[:,0],y==0),'o',color='gray',ms=15)

for i,j in zip(clf.tree_.feature,clf.tree_.threshold):
    if i==1:
        _=ax.hlines(j,-1,6,lw=3.)
    else:
        _=ax.vlines(j,-1,6,lw=3.)

fig.savefig('fig-machine_learning/example_tree_004.png')
```

The decision tree fitted to this triangular matrix is very complex, as shown by the number of horizontal and vertical partitions. Thus, even though the pattern in the training data is visually clear, the decision tree cannot automatically uncover it.

[Figure](#fig:example_tree_004) shows another example, but now using a simple
triangular matrix.  As shown by the number of vertical and horizontal
partitioning lines, the decision tree that corresponds to this figure is tall
and complex. Notice that if we apply a simple rotational transform to the
training data, we can obtain [Figure](#fig:example_tree_005), which requires a
trivial decision tree to fit. Thus, there may be transformations of the
training data that simplify the decision tree, but these are very difficult to
derive in general. Nonetheless, this highlights a key weakness of decision
trees wherein they may be easy to understand, to train, and to deploy, but may
be completely blind to such time-saving and complexity-saving transformations.
Indeed, in higher dimensions, it may be impossible to even visualize the
potential of such latent transformations.  Thus, the advantages of decision
trees can be easily outmatched by other methods that we will study later that
*do* have the ability to uncover useful transformations, but which will
necessarily be harder to train.  Another disadvantage is that because of how
decision trees are built, even a single misplaced data point can cause the tree
to grow very differently. This is a symptom of high variance.

In all of our examples, the decision tree was able to
memorize the training data exactly, as we discussed earlier,
this is a sign of potentially high generalization errors.  There
are pruning algorithms that strategically remove some of the
deepest nodes. but these are not yet fully implemented in
Scikit-learn, as of this writing. Alternatively, restricting
the maximum depth of the decision tree can have a similar
effect. The `DecisionTreeClassifier` and
`DecisionTreeRegressor` in Scikit-learn both have keyword
arguments that specify maximum depth.

## Random Forests

It is possible to combine a set of decision trees into a larger
composite tree that has better performance than its individual
components by using ensemble learning. This is implemented in
Scikit-learn as `RandomForestClassifier`.  The composite tree helps
mitigate the primary weakness of decision trees --- high variance.
Random forest classifiers help by averaging out the predictions of
many constituent trees to minimize this variance by randomly selecting
subsets of the training set to train the  embedded trees.   On the
other hand, this randomization can increase bias because there may be
a subset of the training set that yields an excellent decision tree,
but the averaging effect over randomized training samples washes this
out in the same averaging that reduces the variance. This is a key
trade-off.  The following code implements a simple random forest
classifer from our last example.

```python
from numpy import sin, cos, pi
rotation_matrix=np.matrix([[cos(pi/4),-sin(pi/4)],
                           [sin(pi/4),cos(pi/4)]])
xr=(rotation_matrix*(x.T)).T
xr=np.array(xr)

fig,ax=subplots()
ax.set_aspect(1)
_=ax.axis(xmin=-2,xmax=7,ymin=-4,ymax=4)

_=ax.plot(ma.masked_array(xr[:,1],y==1),ma.masked_array(xr[:,0],y==1),'ow',mec='k',ms=15)
_=ax.plot(ma.masked_array(xr[:,1],y==0),ma.masked_array(xr[:,0],y==0),'o',color='gray',ms=15)

_=clf.fit(xr,y)

for i,j in zip(clf.tree_.feature,clf.tree_.threshold):
    if i==1:
        _=ax.vlines(j,-1,6,lw=3.)
    elif i==0:
        _=ax.hlines(j,-1,6,lw=3.)

fig.savefig('fig-machine_learning/example_tree_005.png')
```

Using a simple rotation on the training data in [Figure](#fig:example_tree_004), the decision tree can now easily fit the training data with a single partition.

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score
X_train,X_test,y_train,y_test=train_test_split(x,y,random_state=1)
clf = tree.DecisionTreeClassifier(max_depth=2)
_=clf.fit(X_train,y_train)
rfc = RandomForestClassifier(n_estimators=4,max_depth=2)
_=rfc.fit(X_train,y_train.flat)
```

```python
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(n_estimators=4,max_depth=2)
rfc.fit(X_train,y_train.flat)
```

    RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                           max_depth=2, max_features='auto', max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, n_estimators=4,
                           n_jobs=None, oob_score=False, random_state=None,
                           verbose=0, warm_start=False)

```python
def draw_board(x,y,clf,ax=None):
    if ax is None: fig,ax=subplots()
    xm,ymn=x.min(0).T
    ax.axis(xmin=xm-1,ymin=ymn-1)
    xx,ymx=x.max(0).T
    _=ax.axis(xmax=xx+1,ymax=ymx+1)
    _=ax.set_aspect(1)
    _=ax.invert_yaxis()
    _=ax.plot(ma.masked_array(x[:,1],y==1),ma.masked_array(x[:,0],y==1),'ow',mec='k')
    _=ax.plot(ma.masked_array(x[:,1],y==0),ma.masked_array(x[:,0],y==0),'o',color='gray')
    for i,j in zip(clf.tree_.feature,clf.tree_.threshold):
        if i==1:
            _=ax.vlines(j,-1,6,lw=3.)
        elif i==0:
            _=ax.hlines(j,-1,6,lw=3.)
    return ax
```

```python
fig,axs = subplots(2,2)
# draw constituent decision trees
for est,ax in zip(rfc.estimators_,axs.flat):
    _=draw_board(X_train,y_train,est,ax=ax)

fig.savefig('fig-machine_learning/example_tree_006.png')
```

 Note that we have constrained the maximum depth `max_depth=2` to help
with generalization. To keep things simple we have only set up a forest with
four individual classifers [^seedNote].  [Figure](#fig:example_tree_006) shows
the individual classifers in the forest that have been trained above. Even
though all the constituent decision trees share the same training data, the
random forest  algorithm randomly picks feature subsets (with replacement) upon
which to train individual trees. This helps avoid the tendency of decision
trees to become too deep and lopsided, which hurts both performance and
generalization. At the prediction step, the individual outputs of each of the
constituent decision trees are put to a majority vote for the final
classification. To estimate generalization errors without using
cross-validation, the training elements *not* used for a particular constituent
tree can be used to test that tree and form a collaborative estimate of
generalization errors.  This is called the *out-of-bag* estimate.

[^seedNote]: We have also set the random seed to a fixed value
to make the figures reproducible in the Jupyter Notebook corresponding to
this section.

The constituent decision trees of the random forest and how they partitioned the training set are shown in these four panels. The random forest classifier uses the individual outputs of each of the constituent trees to produce a collaborative  final estimate.

The main advantage of random forest classifiers is that they require very
little tuning and provide a way to trade-off bias and variance via averaging
and randomization. Furthermore, they are fast and easy to train in parallel
(see the `n_jobs` keyword argument) and fast to predict. On the downside, they
are less interpretable than simple decision trees.  There are many other
powerful tree methods in Scikit-learn like `ExtraTrees` and Gradient Boosted
Regression Trees `GradientBoostingRegressor` which are discussed in the online
documentation.

## Boosting Trees

To understand additive modeling using trees, recall the
Gram-Schmidt orthogonalization procedure for vectors.  The
purpose of this orthogonalization procedure is to create an
orthogonal set of vectors starting with a given vector
$\mathbf{u}_1$. We have already discussed the projection
operator in the section [ch:prob:sec:projection](#ch:prob:sec:projection).  The
Gram-Schmidt orthogonalization procedure starts with a
vector $\mathbf{v}_1$, which we define as the following:

$$
\mathbf{u}_1 = \mathbf{v}_1
$$

 with the corresponding projection operator $proj_{\mathbf{u}_1}$. The
next step in the procedure is to remove the residual of  $\mathbf{u}_1$
from $\mathbf{v}_2$, as in the following:

$$
\mathbf{u}_2 = \mathbf{v}_2 -  proj_{\mathbf{u}_1}(\mathbf{v}_2)
$$

 This procedure continues for $\mathbf{v}_3$ as in the following:

$$
\mathbf{u}_3 = \mathbf{v}_3 - proj_{\mathbf{u}_1}(\mathbf{v}_3) - proj_{\mathbf{u}_2}(\mathbf{v}_3)
$$

 and so on. The important aspect of this procedure is that new
incoming vectors (i.e., $\mathbf{v}_k$) are stripped of any pre-existing
components already present in the set of $\lbrace\mathbf{u}_1,\mathbf{u}_2,\ldots,\mathbf{u}_M \rbrace$.

Note that this procedure is sequential. That is, the *order*
of the incoming $\mathbf{v}_i$ matters [^rotation]. Thus,
any new vector can be expressed using the so-constructed
$\lbrace\mathbf{u}_1,\mathbf{u}_2,\ldots,\mathbf{u}_M
\rbrace$ basis set, as in the following:

[^rotation]: At least up to a rotation of the resulting
orthonormal basis.

$$
\mathbf{x} = \sum \alpha_i \mathbf{u}_i
$$

 The idea behind additive trees is to reproduce this procedure for
trees instead of vectors. There are many natural topological and algebraic
properties that we lack for the general problem, however. For example, we already
have well established methods for measuring distances between vectors for
the Gram-Schmidt procedure outlined above (namely, the $L_2$
distance), which we lack here.  Thus, we need the concept of *loss function*,
which is a way of measuring how well the process is working out at each
sequential step. This loss function is parameterized by the training data and
by the classification function under consideration: $ L_{\mathbf{y}}(f(x)) $.
For example, if we want a classifier ($f$) that selects the label $y_i$
based upon the input data $\mathbf{x}_i$ ($f :\mathbf{x}_i \rightarrow y_i)$,
then the squared error loss function would be the following:

$$
L_{\mathbf{y}}(f(x))  = \sum_i (y_i - f(x_i))^2
$$

 We represent the classifier in terms of a set of basis trees:

$$
f(x) = \sum_k \alpha_k u_{\mathbf{x}}(\theta_k)
$$

 The general algorithm for forward stagewise  additive modeling is
the following:

   * Initialize $f(x)=0$

   * For $m=1$ to $m=M$, compute the following:

$$
(\beta_m,\gamma_m) = \argmin_{\beta,\gamma} \sum_i  L(y_i,f_{m-1}(x_i)+\beta b(x_i;\gamma))
$$

* Set $f_m(x) = f_{m-1}(x) +\beta_m b(x;\gamma_m) $

The key point is that the residuals from the prior step are used to fit the
basis function for the subsequent iteration.  That is, the following
equation is being sequentially approximated.

$$
f_m(x) - f_{m-1}(x) =\beta_m b(x_i;\gamma_m)
$$

 Let's see how this works for decision trees and the exponential loss
function.

$$
L(x,f(x)) = \exp(-y f(x))
$$

 Recall that for the classification problem, $y\in \lbrace -1,1 \rbrace$. For
AdaBoost, the basis functions are the individual classifiers, $G_m(x)\mapsto \lbrace -1,1 \rbrace$
The key step in the algorithm is the minimization step for
the objective function

$$
J(\beta,G) = \sum_i \exp(y_i(f_{m-1}(x_i)+\beta G(x_i)))
$$

$$
(\beta_m,G_m) = \argmin_{\beta,G} \sum_i \exp(y_i(f_{m-1}(x_i)+\beta G(x_i)))
$$

 Now, because of the exponential, we can factor out the following:

$$
w_i^{(m)} = \exp(y_i f_{m-1}(x_i))
$$

 as a weight on each data element and re-write the objective function as the following:

$$
J(\beta,G) = \sum_i w_i^{(m)} \exp(y_i \beta G(x_i))
$$

 The important observation here is that $y_i G(x_i)\mapsto 1$ if
the tree classifies $x_i$ correctly and $y_i G(x_i)\mapsto -1$ otherwise.
Thus, the above sum has terms like the following:

$$
J(\beta,G) = \sum_{y_i\ne G(x_i)} w_i^{(m)} \exp(-\beta) + \sum_{y_i= G(x_i)} w_i^{(m)}\exp(\beta)
$$

 For $\beta>0$, this means that  the best $G(x)$ is
the one that incorrectly classifies for the largest weights. Thus,
the minimizer is the following:

$$
G_m = \argmin_G \sum_i w_i^{(m)} I(y_i \ne G(x_i))
$$

 where $I$ is the indicator function (i.e.,
$I(\texttt{True})=1,I(\texttt{False})=0$).

 For $\beta>0$, we can re-write the objective function
as the following:

$$
J= (\exp(\beta)-\exp(-\beta))\sum_i w_i^{(m)} I(y_i \ne G(x_i)) + \exp(-\beta) \sum_i w_i^{(m)}
$$

 and substitute $\theta=\exp(-\beta)$ so that

$$
\begin{equation}
\frac{J}{\sum_i w_i^{(m)}} = \left(\frac{1}{\theta} - \theta\right)\epsilon_m+\theta
\label{eq:bt001} \tag{1}
\end{equation}
$$

 where

$$
\epsilon_m = \frac{\sum_i w_i^{(m)}I(y_i\ne G(x_i))}{\sum_i w_i^{(m)}}
$$

  is the error rate of the classifier with $0\le \epsilon_m\le 1$. Now,
finding $\beta$ is a straight-forward calculus minimization exercise on the
right side of Equation ([1](#eq:bt001)), which gives the following:

$$
\beta_m  = \frac{1}{2}\log \frac{1-\epsilon_m}{\epsilon_m}
$$

 Importantly, $\beta_m$ can become negative if
$\epsilon_m<\frac{1}{2}$, which would would violate our assumptions on $\beta$.
This is captured in the requirement that the base learner be better than just
random guessing, which would correspond to $\epsilon_m>\frac{1}{2}$.
Practically speaking, this means that boosting cannot fix a base learner that
is no better than a random guess. Formally speaking, this is known as the
*empirical weak learning assumption* [[schapire2012boosting]](#schapire2012boosting).

Now we can move to the iterative weight update. Recall that

$$
w_i^{(m+1)} = \exp(y_i f_{m}(x_i)) = w_i^{(m)}\exp(y_i \beta_m G_m(x_i))
$$

 which we can rewrite as the following:

$$
w_i^{(m+1)} = w_i^{(m)}\exp(\beta_m)\exp(-2\beta_m I(G_m(x_i)=y_i))
$$

 This means that the data elements that are incorrectly classified have their
corresponding weights increased by $\exp(\beta_m)$ and those that are correctly
classified have their corresponding weights reduced by $\exp(-\beta_m)$.
The reason for the choice of the exponential loss function comes from the
following:

$$
f^{*}(x) = \argmin_{f(x)} \mathbb{E}_{Y\vert x}(\exp(-Y f(x))) = \frac{1}{2}\log \frac{\mathbb{P}(Y=1\vert x)}{\mathbb{P}(Y=\minus 1\vert x)}
$$

 This means that boosting is approximating a $f(x)$ that is actually
half the log-odds of the conditional class probabilities.  This can be
rearranged as the following

$$
\mathbb{P}(Y=1\vert x) = \frac{1}{1+\exp(\minus 2f^{*}(x))}
$$

 The important benefit of this general formulation for boosting, as a sequence
of additive approximations,  is that it opens the door to other choices of loss
function, especially loss functions that are based on robust statistics that
can account for errors in the training data (c.f. Hastie).

### Gradient Boosting

Given a differentiable loss function, the optimization process can be
formulated using numerical gradients. The fundamental idea is to treat the
$f(x_i)$ as a scalar parameter to be optimized over. Generally speaking,
we can think of the following loss function,

$$
L(f) = \sum_{i=1}^N L(y_i,f(x_i))
$$

 as a vectorized quantity

$$
\mathbf{f} = \lbrace f(x_1),f(x_2),\ldots, f(x_N) \rbrace
$$

 so that the optimization is over this vector

$$
\hat{\mathbf{f}} = \argmin_{\mathbf{f}} L(\mathbf{f})
$$

 With this general formulation we can use numerical optimization methods
to solve for the optimal $\mathbf{f}$ as a sum of component vectors
as in the following:

$$
\mathbf{f}_M = \sum_{m=0}^M \mathbf{h}_m
$$

 Note that this leaves aside the prior assumption that $f$
is parameterized as a sum of individual decision trees.

$$
g_{i,m} = \left[ \frac{\partial L(y_i,f(x_i))}{\partial f(x_i)} \right]_{f(x_i)=f_{m-1}(x_i)}
$$


```
%matplotlib inline
from matplotlib.pyplot import plot, figure, legend, style
from control import ss, step_response, forced_response, tf, acker
from numpy import linspace, matrix, sign, hstack, vstack, roots
from numpy.linalg import det, eig
from sympy import var, Matrix, eye, init_printing, roots
init_printing()
```

Fecha de Nacimiento: 18-07-86

Sea $d$ el mayor digito de su dia de nacimiento, $m$ el mayor digito de su mes de nacimiento y $r$ el mayor de estos dos.

Sea un sistema lineal invariante en el tiempo, descrito por la siguiente EDO:

$$
\left( \frac{d^3}{dt^3} + \frac{d^2}{dt^2} \right) y(t) = \left( \frac{d}{dt} - d \right) \left( \frac{d}{dt} + m \right) u(t)
$$

donde $y(t)$ es la salida y $u(t)$ es la entrada.

Para nuestro  exámen en particular:

$$
\left( \frac{d^3}{dt^3} + \frac{d^2}{dt^2} \right) y(t) = \left( \frac{d}{dt} - 8 \right) \left( \frac{d}{dt} + 7 \right) u(t)
$$

# 1 - Representación de estado

* Dar una representación de estado,
$$
\frac{dx(t)}{dt} = A x(t) + b u(t)
$$
$$
y(t) = c^T x(t)
$$
del sistema descrito, tal que su matriz de controlabilidad sea de rango pleno por filas.

* De la representación de estado propuesta calcule
    * Su función de transferencia
    * Su espectro (valores propios)
    * Los ceros del sistema
    * El determinante de su matriz controlabilidad
    * El determinante de su matriz de observabilidad

$s^3 + s^2$

$s^2 - as + ab$

```
G = tf([0, 1, -1, 56], [1, 1, 0, 0])
```

```
RE = ss(G)
```

```
A = RE.A
A
```

    matrix([[  4.44089210e-16,  -4.61404088e-16,   3.31856603e-16],
            [ -1.00000000e+00,  -4.69305777e-16,   3.15808984e-17],
            [  0.00000000e+00,   1.00000000e+00,  -1.00000000e+00]])

```
b = RE.B
b
```

    matrix([[-5.6],
            [-0.1],
            [ 0.1]])

```
c = RE.C
c
```

    matrix([[ 0.,  0., 10.]])

```
Co = hstack([b ,A*b, A*A*b])
```

```
det(Co)
```

```
d=RE.D
d
```

    matrix([[ 0.]])

```
A = matrix([[0, 1, 0], [0, 0, 1], [0, 0, -1]])
b = matrix([[0], [0], [1]])
c = matrix([[-9], [0], [1]])
```

```
Co = hstack([b ,A*b, A*A*b])
Co
```

    matrix([[ 0,  0,  1],
            [ 0,  1, -1],
            [ 1, -1,  1]])

```
det(Co)
```

```
var("s")
```

```
c.T*(s*eye(3) - A).inv()*b
```

```
eig(A)
```

    (array([ 0.,  0., -1.]),
     matrix([[  1.00000000e+000,  -1.00000000e+000,   5.77350269e-001],
             [  0.00000000e+000,   3.00625254e-292,  -5.77350269e-001],
             [  0.00000000e+000,   0.00000000e+000,   5.77350269e-001]]))

```
E = vstack([hstack([s*eye(3) - A, b]), hstack([c.T, d])])
E
```

    matrix([[s, -1, 0, 0],
            [0, s, -1, 0],
            [0, 0, s + 1, 1],
            [-9.0, 0.0, 1.0, 0.0]], dtype=object)

```
Matrix(E).det()
```

```
roots([-1, 0,  9])
```

```
Ob = vstack([c.T, c.T*A, c.T*A*A])
Ob
```

    matrix([[-9,  0,  1],
            [ 0, -9, -1],
            [ 0,  0, -8]])

```
det(Ob)
```

# 2 - Retroalimentación de estado

Diseñe una retroalimentación de estado

$$
u(t) = f^T x(t) + v(t)
$$

tal que:

* La función de transferencia en lazo cerrado sea $\frac{s-d}{(s+m)^2}$
* ¿Es factible que la función de transferencia en lazo cerrado: $\frac{1}{s+m}$? Justifique su respuesta

queremos que el denominador quede $(s+m)^3$

```
((s+7)**3).expand()
```

```
f = acker(A, b, [-3, -3, -3])
f = f.T
f
```

    matrix([[ 27.],
            [ 27.],
            [  8.]])

```
Af = A + b*f.T
Af
```

    matrix([[  9.,  18.,  -9.],
            [  1.,   0.,   0.],
            [  1.,   1.,  -2.]])

```
PCf = (s*eye(3) - Af).det()
PCf.factor()
```

```
roots([1, -7, -27, -27])
```

```
c.T * (s*eye(3) - Af).inv()*b
```

```
A
```

    matrix([[ 0,  1,  0],
            [ 0,  0,  1],
            [ 0,  0, -1]])

```
b
```

    matrix([[0],
            [0],
            [1]])

```
RE.feedback()
```

    <control.statesp.StateSpace instance at 0x10976f950>

```

```

IPythonNotebooks/Teoria de Control I/Examen I - Representacion de estado.ipynb