# Vertical Line Test

```
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
```

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.

```
x = [1,2]
y = [2,1]

plt.plot(x,y)
plt.axvline(x=1.5, color='orange')
plt.title('Vertical Line Test Pass');
```

```
# made a parabola thanks to some googl fu

x1 = np.linspace(-9, 9, 400)
y1 = np.linspace(-5, 5, 400)
x1, y1 = np.meshgrid(x1, y1)

def axes():
    plt.axhline(0, alpha=.1)
    plt.axvline(0, alpha=.1)

a = .3
axes()
plt.contour(x1, y1, (y1**2 - 4*a*x1), [0], colors='purple')
plt.axvline(x=2, color='orange')
plt.show()
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

At the core, the reason is because a traditional function will have an x value assigned to a y value. If a function fails the vertical line test and it has an x value associated with more than one y-value, this is indicative that x is dependent on more than just one function of y.

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

Functions: Relation 2.

This would pass the vertical line test(only 1 x value per y, where as the rest have repeated x values and thus would fail the vertical line test and therefore NOT be functions.

```
x = [1,3,1]
y = [2, 2, 3]

plt.plot(x,y)
plt.title('Plot of Relation 1')
plt.show()
# for relation 1 here, see how it would fail the vertical line test, at a glance you can see that
# the rest would also fail the vertical line test excluding relation 2
```

# Functions as a mapping between dimensions

## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}

m inputs: 3, m outputs 3

n inputs: 4, m outputs 2

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

Upon doing some googl fu, I'm inclined to say that YES! It is possible to for a function to map from a lower dimension to a higher dimension.

Example: Within our Data Science scope, this could be as simple as adding an additional column. Not a duplicate column but a new column of information. Whilst this addition does add complexity, it could potentially serve to make the data analysis better.

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

Part 1)

\begin{align}
T = \begin{bmatrix} 1 & 3 \\ 2 & -1 \end{bmatrix}
\end{align}

Part 2)

\begin{align}
T = \begin{bmatrix} 4 & 1 & 2 \\ 2 & -1 & 3 \\ 5 & -2 & 1 \end{bmatrix}
\end{align}

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.

```

```

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

Think of the globe intuition. A location on the equator covers a certain distance per rotation. Whereas a location closer to one of the poles covers a smaller distance per the same rotation. And the axis with which the globe spins on doesn't actually move any distance per rotation.(Although this is relative as is the case with physics) With that said, the purpose of Eigen values is through some calculations and number crunching, we are able to analyze both features without the magnitude of the raw numbers granting more importance to any particular feature.

Another nice example from lecture on how this would be useful: if you're analyzing a used car, and some features you're comparing are price and # of past owners. The price will naturally be in the thousands whereas the number of owners is most likely gonna be countable on one hand, yet in your analysis, you want to take both factors into consideration. Especially when it comes to machine learning, we would use conepts of eigen values and eigen vectors such that we could use machine learning to do our analysis WITHOUT having the sheer magnitude of the raw numbers(1000s, to a handful) skew the data away from reality.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

For starts, working in any spacial dimension greater than 4 becomes difficult to conceptualize for humans outside of a numerical standpoint.

When it comes to our actual work here as data scientists, refering back to square vs cube vs hypercube example, it is possible to create a 4 dimensional hyper cube but its starts to get extremely convoluted fast.

Next is when it actually comes to working with data, if we have a dataset with say 1000 columns, even doing something like a seaborn pairplot is gonna be undoable unless you have a NASA computer. Heck even in my own project, there was a point where I tried to simply do a scatter plot that had 70,000 entries, it takes a few moments to load that every time. Now just imagine trying to do 1000 times that.

As we go down the list of issues, there is a point of diminishing returns with high dimensionality, for example, the pixels on a screen. It gets to a point where its so fine tuned that with human eye balls, you probably couldn't even see further improvements.

ALSO, if you have a low number of observations in relation to the number of dimensions, you'll run into the issue of overfitting with whatever analytical approach you're taking. This is however, always fixable with... MORE OBSERVATIONS.

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

A general rule of thumb is to have 5 times the number of observations as parameters.

However there is debate that goes as high as observations should be parameters squared.

# Principal Component Analysis

## 7.1 Load the UCI Machine Learning Repository's [Iris Dataset](https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/d546eaee765268bf2f487608c537c05e22e4b221/iris.csv) and use PCA to isolate the dataset's first and second principal components and plot them on a graph.

```
from numpy import array
from numpy import mean
from numpy import std
from numpy import cov
from numpy.linalg import eig
```

```
df = pd.read_csv('https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/d546eaee765268bf2f487608c537c05e22e4b221/iris.csv')
df.head()
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      species

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

```
# Step One, Seperate your data into x and y variables, in this case, in an Array

X = [df['sepal_length'], df['sepal_width']]
mean(X)
```

    4.448666666666667

```
# Step Two, center each column at 0 by subtracting the mean

def means(X):
  means = mean(X, axis=1)
  return

print('Means:\n', means)
print(df['sepal_length'].mean())
print(df['sepal_width'].mean())
```

    Means:
     <function means at 0x7f1a2859bae8>
    5.843333333333335
    3.0540000000000007

```
df.mean()
```

    sepal_length    5.843333
    sepal_width     3.054000
    petal_length    3.758667
    petal_width     1.198667
    dtype: float64

```
# Step Three, Divide each column by its standard deviation

# finding standard dev
std_devs = np.std(X, axis=1)
print('Standard Deviation:\n', std_devs)

# dividing each (column - mean) by std
Z = (X - mean(X)) / std(X)   # need to get this part to subtract the mean of each column, not mean of DF
print('\nStandardized Data: Z\n', Z.T)

# lol, a lot of stuff here
```

    Standard Deviation:
     [0.82530129 0.43214658]

    Standardized Data: Z
     [[ 0.42228262 -0.61505441]
     [ 0.29261549 -0.93922224]
     [ 0.16294836 -0.80955511]
     [ 0.09811479 -0.87438867]
     [ 0.35744905 -0.55022085]
     [ 0.61678331 -0.35572016]
     [ 0.09811479 -0.67988798]
     [ 0.35744905 -0.67988798]
     [-0.03155233 -1.0040558 ]
     [ 0.29261549 -0.87438867]
     [ 0.61678331 -0.48538729]
     [ 0.22778192 -0.67988798]
     [ 0.22778192 -0.93922224]
     [-0.0963859  -0.93922224]
     [ 0.87611757 -0.29088659]
     [ 0.811284   -0.03155233]
     [ 0.61678331 -0.35572016]
     [ 0.42228262 -0.61505441]
     [ 0.811284   -0.42055372]
     [ 0.42228262 -0.42055372]
     [ 0.61678331 -0.67988798]
     [ 0.42228262 -0.48538729]
     [ 0.09811479 -0.55022085]
     [ 0.42228262 -0.74472154]
     [ 0.22778192 -0.67988798]
     [ 0.35744905 -0.93922224]
     [ 0.35744905 -0.67988798]
     [ 0.48711618 -0.61505441]
     [ 0.48711618 -0.67988798]
     [ 0.16294836 -0.80955511]
     [ 0.22778192 -0.87438867]
     [ 0.61678331 -0.67988798]
     [ 0.48711618 -0.22605303]
     [ 0.68161687 -0.16121946]
     [ 0.29261549 -0.87438867]
     [ 0.35744905 -0.80955511]
     [ 0.68161687 -0.61505441]
     [ 0.29261549 -0.87438867]
     [-0.03155233 -0.93922224]
     [ 0.42228262 -0.67988798]
     [ 0.35744905 -0.61505441]
     [ 0.03328123 -1.39305719]
     [-0.03155233 -0.80955511]
     [ 0.35744905 -0.61505441]
     [ 0.42228262 -0.42055372]
     [ 0.22778192 -0.93922224]
     [ 0.42228262 -0.42055372]
     [ 0.09811479 -0.80955511]
     [ 0.55194974 -0.48538729]
     [ 0.35744905 -0.74472154]
     [ 1.65412034 -0.80955511]
     [ 1.26511895 -0.80955511]
     [ 1.58928678 -0.87438867]
     [ 0.68161687 -1.39305719]
     [ 1.32995252 -1.06888936]
     [ 0.811284   -1.06888936]
     [ 1.20028539 -0.74472154]
     [ 0.29261549 -1.32822362]
     [ 1.39478608 -1.0040558 ]
     [ 0.48711618 -1.13372293]
     [ 0.35744905 -1.58755788]
     [ 0.94095113 -0.93922224]
     [ 1.0057847  -1.45789075]
     [ 1.07061826 -1.0040558 ]
     [ 0.74645044 -1.0040558 ]
     [ 1.45961965 -0.87438867]
     [ 0.74645044 -0.93922224]
     [ 0.87611757 -1.13372293]
     [ 1.13545182 -1.45789075]
     [ 0.74645044 -1.26339006]
     [ 0.94095113 -0.80955511]
     [ 1.07061826 -1.06888936]
     [ 1.20028539 -1.26339006]
     [ 1.07061826 -1.06888936]
     [ 1.26511895 -1.0040558 ]
     [ 1.39478608 -0.93922224]
     [ 1.52445321 -1.06888936]
     [ 1.45961965 -0.93922224]
     [ 1.0057847  -1.0040558 ]
     [ 0.811284   -1.19855649]
     [ 0.68161687 -1.32822362]
     [ 0.68161687 -1.32822362]
     [ 0.87611757 -1.13372293]
     [ 1.0057847  -1.13372293]
     [ 0.61678331 -0.93922224]
     [ 1.0057847  -0.67988798]
     [ 1.45961965 -0.87438867]
     [ 1.20028539 -1.39305719]
     [ 0.74645044 -0.93922224]
     [ 0.68161687 -1.26339006]
     [ 0.68161687 -1.19855649]
     [ 1.07061826 -0.93922224]
     [ 0.87611757 -1.19855649]
     [ 0.35744905 -1.39305719]
     [ 0.74645044 -1.13372293]
     [ 0.811284   -0.93922224]
     [ 0.811284   -1.0040558 ]
     [ 1.13545182 -1.0040558 ]
     [ 0.42228262 -1.26339006]
     [ 0.811284   -1.06888936]
     [ 1.20028539 -0.74472154]
     [ 0.87611757 -1.13372293]
     [ 1.7189539  -0.93922224]
     [ 1.20028539 -1.0040558 ]
     [ 1.32995252 -0.93922224]
     [ 2.04312173 -0.93922224]
     [ 0.29261549 -1.26339006]
     [ 1.84862103 -1.0040558 ]
     [ 1.45961965 -1.26339006]
     [ 1.78378747 -0.55022085]
     [ 1.32995252 -0.80955511]
     [ 1.26511895 -1.13372293]
     [ 1.52445321 -0.93922224]
     [ 0.811284   -1.26339006]
     [ 0.87611757 -1.06888936]
     [ 1.26511895 -0.80955511]
     [ 1.32995252 -0.93922224]
     [ 2.10795529 -0.42055372]
     [ 2.10795529 -1.19855649]
     [ 1.0057847  -1.45789075]
     [ 1.58928678 -0.80955511]
     [ 0.74645044 -1.06888936]
     [ 2.10795529 -1.06888936]
     [ 1.20028539 -1.13372293]
     [ 1.45961965 -0.74472154]
     [ 1.78378747 -0.80955511]
     [ 1.13545182 -1.06888936]
     [ 1.07061826 -0.93922224]
     [ 1.26511895 -1.06888936]
     [ 1.78378747 -0.93922224]
     [ 1.9134546  -1.06888936]
     [ 2.23762242 -0.42055372]
     [ 1.26511895 -1.06888936]
     [ 1.20028539 -1.06888936]
     [ 1.07061826 -1.19855649]
     [ 2.10795529 -0.93922224]
     [ 1.20028539 -0.67988798]
     [ 1.26511895 -0.87438867]
     [ 1.0057847  -0.93922224]
     [ 1.58928678 -0.87438867]
     [ 1.45961965 -0.87438867]
     [ 1.58928678 -0.87438867]
     [ 0.87611757 -1.13372293]
     [ 1.52445321 -0.80955511]
     [ 1.45961965 -0.74472154]
     [ 1.45961965 -0.93922224]
     [ 1.20028539 -1.26339006]
     [ 1.32995252 -0.93922224]
     [ 1.13545182 -0.67988798]
     [ 0.94095113 -0.93922224]]

```
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
print(scaler.fit(X))
print(scaler.transform(X))
```

    StandardScaler(copy=True, with_mean=True, with_std=True)
    [[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
       1.  1.  1.  1.  1.  1.]
     [-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
      -1. -1. -1. -1. -1. -1.]]

```
# Step Four Calculate Variance-Covariance MAtrix of Z

covariance_matrix = cov(Z)
print('Covariance Matrix:\n ', covariance_matrix)

```

    Covariance Matrix:
      [[ 0.2882238  -0.01650607]
     [-0.01650607  0.07902544]]

```
# Step Five, Calculate Eigen vectors and Eigenvalues of Variance-Covariance Matrix

values, vectors = eig(covariance_matrix)
print('Eigenvectors:\n', vectors)
print('\nEigenvalues:\n',values)

# step six is just sorting them, so the largest Eigenvalue corresponds to the
# directino of the greatest variance in the data.
# the eigenvalue eigenvector pair with the largest eigen value becomes... PC1!!
# (thats Principle Component 1)
```

    Eigenvectors:
     [[ 0.99693955  0.07817635]
     [-0.07817635  0.99693955]]

    Eigenvalues:
     [0.28951814 0.0777311 ]

```
# Step Seven Use a matrix transformation to project the datapoints onto our eigen vector subspaces

projected = vectors.dot(X)
print('Projected Data:\n', projected.T)
```

    Projected Data:
     [[5.35800892 3.090589  ]
     [5.11953284 2.6077545 ]
     [4.9357802  2.82277768]
     [4.82826861 2.73090136]
     [5.2661326  3.19810059]
     [5.68836133 3.46591192]
     [4.85172151 3.02998323]
     [5.25049733 2.99871268]
     [4.61324543 2.54714872]
     [5.12735047 2.70744846]
     [5.67272606 3.26652401]
     [5.05110942 3.01434796]
     [5.01983888 2.61557214]
     [4.52136911 2.65466031]
     [6.09495478 3.53433533]
     [6.02653137 3.94092878]
     [5.68836133 3.46591192]
     [5.35800892 3.090589  ]
     [5.97962556 3.34276505]
     [5.38146183 3.38967087]
     [5.64927315 2.96744214]
     [5.37364419 3.28997691]
     [4.86735678 3.22937114]
     [5.34237365 2.89120109]
     [5.05110942 3.01434796]
     [5.21922679 2.59993687]
     [5.25049733 2.99871268]
     [5.45770288 3.08277137]
     [5.44988524 2.98307741]
     [4.9357802  2.82277768]
     [5.02765652 2.71526609]
     [5.64927315 2.96744214]
     [5.50460869 3.6809351 ]
     [5.81150819 3.75717614]
     [5.12735047 2.70744846]
     [5.23486206 2.79932478]
     [5.75678474 3.05931846]
     [5.12735047 2.70744846]
     [4.62106306 2.64684268]
     [5.35019129 2.99089505]
     [5.25831497 3.09840664]
     [4.66603357 1.94116736]
     [4.63669833 2.84623059]
     [5.25831497 3.09840664]
     [5.38146183 3.38967087]
     [5.01983888 2.61557214]
     [5.38146183 3.38967087]
     [4.83608624 2.83059532]
     [5.5730321  3.27434164]
     [5.2426797  2.89901873]
     [7.22874115 2.64297207]
     [6.63057743 2.68987788]
     [7.12122956 2.55109575]
     [5.66297312 1.86299101]
     [6.69900084 2.28328443]
     [5.9014492  2.34582551]
     [6.53870111 2.79738947]
     [5.07262702 2.00959077]
     [6.80651243 2.37516074]
     [5.39516179 2.28521973]
     [5.14105044 1.60299732]
     [6.11647238 2.52957815]
     [6.15362525 1.72420888]
     [6.30804266 2.41424892]
     [5.80957288 2.4533371 ]
     [6.92184165 2.56673102]
     [5.81739052 2.55303105]
     [5.99332552 2.23831392]
     [6.35301316 1.7085736 ]
     [5.77830234 2.05456128]
     [6.13210765 2.72896606]
     [6.30022502 2.31455497]
     [6.47616002 1.99983783]
     [6.30022502 2.31455497]
     [6.60712452 2.39079602]
     [6.81433006 2.4748547 ]
     [6.9980827  2.25983152]
     [6.91402402 2.46703706]
     [6.2083487  2.42206656]
     [5.88581393 2.1464376 ]
     [5.67079075 1.96268496]
     [5.67079075 1.96268496]
     [5.99332552 2.23831392]
     [6.19271343 2.22267865]
     [5.61800261 2.56866632]
     [6.24743688 2.92053633]
     [6.92184165 2.56673102]
     [6.46052475 1.80044992]
     [5.81739052 2.55303105]
     [5.67860839 2.06237892]
     [5.68642602 2.16207287]
     [6.31586029 2.51394288]
     [5.98550789 2.13861996]
     [5.16450334 1.90207918]
     [5.79393761 2.25394919]
     [5.91708447 2.54521342]
     [5.90926684 2.44551946]
     [6.40773661 2.40643129]
     [5.27983257 2.09364946]
     [5.9014492  2.34582551]
     [6.53870111 2.79738947]
     [5.99332552 2.23831392]
     [7.31279984 2.43576652]
     [6.50743056 2.39861365]
     [6.71463611 2.48267233]
     [7.81126961 2.39667835]
     [5.08044466 2.10928473]
     [7.50437011 2.3204373 ]
     [6.87493584 1.96856729]
     [7.4593996  3.02611261]
     [6.73027138 2.68206024]
     [6.59148925 2.19140811]
     [7.01371797 2.45921943]
     [5.8779963  2.04674365]
     [6.00114316 2.33800787]
     [6.63057743 2.68987788]
     [6.71463611 2.48267233]
     [7.97350465 3.18641235]
     [7.87969302 1.99008489]
     [6.15362525 1.72420888]
     [7.1290472  2.6507897 ]
     [5.80175525 2.35364314]
     [7.89532829 2.1894728 ]
     [6.49179529 2.19922574]
     [6.93747692 2.76611893]
     [7.42812906 2.6273368 ]
     [6.39991897 2.30673733]
     [6.31586029 2.51394288]
     [6.59930688 2.29110206]
     [7.41249379 2.42794889]
     [7.59624643 2.21292571]
     [8.17289256 3.17077708]
     [6.59930688 2.29110206]
     [6.49961293 2.2989197 ]
     [6.28458975 2.11516706]
     [7.91096356 2.38886071]
     [6.54651874 2.89708342]
     [6.62275979 2.59018392]
     [6.21616634 2.52176051]
     [7.12122956 2.55109575]
     [6.92184165 2.56673102]
     [7.12122956 2.55109575]
     [5.99332552 2.23831392]
     [7.02935324 2.65860734]
     [6.93747692 2.76611893]
     [6.91402402 2.46703706]
     [6.47616002 1.99983783]
     [6.71463611 2.48267233]
     [6.44682479 2.90490106]
     [6.11647238 2.52957815]]

Now to Run a PCA via library functions to check my answers

```
from sklearn.decomposition import PCA
```

```
# assigning x and z for examples sake even though they're in the cell above
X = array([df['sepal_length'], df['sepal_width']])
Z = (X - mean(X)) / std(X)
print("Data: \n", Z)

# Create PCA instance
pca = PCA(2)

# fit on data
pca.fit(Z)

# Access eigen vectors and eigen values
print('\n Eigen Vectors: \n', pca.components_)
print('\n Eigen Values: \n', pca.explained_variance_)

# transform the data
B = pca.transform(X)
print('\n Projected Data: \n', B)

```

    Data:
     [[ 0.42228262  0.29261549  0.16294836  0.09811479  0.35744905  0.61678331
       0.09811479  0.35744905 -0.03155233  0.29261549  0.61678331  0.22778192
       0.22778192 -0.0963859   0.87611757  0.811284    0.61678331  0.42228262
       0.811284    0.42228262  0.61678331  0.42228262  0.09811479  0.42228262
       0.22778192  0.35744905  0.35744905  0.48711618  0.48711618  0.16294836
       0.22778192  0.61678331  0.48711618  0.68161687  0.29261549  0.35744905
       0.68161687  0.29261549 -0.03155233  0.42228262  0.35744905  0.03328123
      -0.03155233  0.35744905  0.42228262  0.22778192  0.42228262  0.09811479
       0.55194974  0.35744905  1.65412034  1.26511895  1.58928678  0.68161687
       1.32995252  0.811284    1.20028539  0.29261549  1.39478608  0.48711618
       0.35744905  0.94095113  1.0057847   1.07061826  0.74645044  1.45961965
       0.74645044  0.87611757  1.13545182  0.74645044  0.94095113  1.07061826
       1.20028539  1.07061826  1.26511895  1.39478608  1.52445321  1.45961965
       1.0057847   0.811284    0.68161687  0.68161687  0.87611757  1.0057847
       0.61678331  1.0057847   1.45961965  1.20028539  0.74645044  0.68161687
       0.68161687  1.07061826  0.87611757  0.35744905  0.74645044  0.811284
       0.811284    1.13545182  0.42228262  0.811284    1.20028539  0.87611757
       1.7189539   1.20028539  1.32995252  2.04312173  0.29261549  1.84862103
       1.45961965  1.78378747  1.32995252  1.26511895  1.52445321  0.811284
       0.87611757  1.26511895  1.32995252  2.10795529  2.10795529  1.0057847
       1.58928678  0.74645044  2.10795529  1.20028539  1.45961965  1.78378747
       1.13545182  1.07061826  1.26511895  1.78378747  1.9134546   2.23762242
       1.26511895  1.20028539  1.07061826  2.10795529  1.20028539  1.26511895
       1.0057847   1.58928678  1.45961965  1.58928678  0.87611757  1.52445321
       1.45961965  1.45961965  1.20028539  1.32995252  1.13545182  0.94095113]
     [-0.61505441 -0.93922224 -0.80955511 -0.87438867 -0.55022085 -0.35572016
      -0.67988798 -0.67988798 -1.0040558  -0.87438867 -0.48538729 -0.67988798
      -0.93922224 -0.93922224 -0.29088659 -0.03155233 -0.35572016 -0.61505441
      -0.42055372 -0.42055372 -0.67988798 -0.48538729 -0.55022085 -0.74472154
      -0.67988798 -0.93922224 -0.67988798 -0.61505441 -0.67988798 -0.80955511
      -0.87438867 -0.67988798 -0.22605303 -0.16121946 -0.87438867 -0.80955511
      -0.61505441 -0.87438867 -0.93922224 -0.67988798 -0.61505441 -1.39305719
      -0.80955511 -0.61505441 -0.42055372 -0.93922224 -0.42055372 -0.80955511
      -0.48538729 -0.74472154 -0.80955511 -0.80955511 -0.87438867 -1.39305719
      -1.06888936 -1.06888936 -0.74472154 -1.32822362 -1.0040558  -1.13372293
      -1.58755788 -0.93922224 -1.45789075 -1.0040558  -1.0040558  -0.87438867
      -0.93922224 -1.13372293 -1.45789075 -1.26339006 -0.80955511 -1.06888936
      -1.26339006 -1.06888936 -1.0040558  -0.93922224 -1.06888936 -0.93922224
      -1.0040558  -1.19855649 -1.32822362 -1.32822362 -1.13372293 -1.13372293
      -0.93922224 -0.67988798 -0.87438867 -1.39305719 -0.93922224 -1.26339006
      -1.19855649 -0.93922224 -1.19855649 -1.39305719 -1.13372293 -0.93922224
      -1.0040558  -1.0040558  -1.26339006 -1.06888936 -0.74472154 -1.13372293
      -0.93922224 -1.0040558  -0.93922224 -0.93922224 -1.26339006 -1.0040558
      -1.26339006 -0.55022085 -0.80955511 -1.13372293 -0.93922224 -1.26339006
      -1.06888936 -0.80955511 -0.93922224 -0.42055372 -1.19855649 -1.45789075
      -0.80955511 -1.06888936 -1.06888936 -1.13372293 -0.74472154 -0.80955511
      -1.06888936 -0.93922224 -1.06888936 -0.93922224 -1.06888936 -0.42055372
      -1.06888936 -1.06888936 -1.19855649 -0.93922224 -0.67988798 -0.87438867
      -0.93922224 -0.87438867 -0.87438867 -0.87438867 -1.13372293 -0.80955511
      -0.74472154 -0.93922224 -1.26339006 -0.93922224 -0.67988798 -0.93922224]]

     Eigen Vectors:
     [[-4.42242130e-02 -5.25162529e-02 -4.14601997e-02 -4.14601997e-02
      -3.86961864e-02 -4.14601997e-02 -3.31681597e-02 -4.42242130e-02
      -4.14601997e-02 -4.97522396e-02 -4.69882263e-02 -3.86961864e-02
      -4.97522396e-02 -3.59321730e-02 -4.97522396e-02 -3.59321730e-02
      -4.14601997e-02 -4.42242130e-02 -5.25162529e-02 -3.59321730e-02
      -5.52802662e-02 -3.86961864e-02 -2.76401331e-02 -4.97522396e-02
      -3.86961864e-02 -5.52802662e-02 -4.42242130e-02 -4.69882263e-02
      -4.97522396e-02 -4.14601997e-02 -4.69882263e-02 -5.52802662e-02
      -3.04041464e-02 -3.59321730e-02 -4.97522396e-02 -4.97522396e-02
      -5.52802662e-02 -4.97522396e-02 -3.86961864e-02 -4.69882263e-02
      -4.14601997e-02 -6.08082928e-02 -3.31681597e-02 -4.14601997e-02
      -3.59321730e-02 -4.97522396e-02 -3.59321730e-02 -3.86961864e-02
      -4.42242130e-02 -4.69882263e-02 -1.05032506e-01 -8.84484260e-02
      -1.05032506e-01 -8.84484260e-02 -1.02268493e-01 -8.01563860e-02
      -8.29203993e-02 -6.91003328e-02 -1.02268493e-01 -6.91003328e-02
      -8.29203993e-02 -8.01563860e-02 -1.05032506e-01 -8.84484260e-02
      -7.46283594e-02 -9.95044792e-02 -7.18643461e-02 -8.56844126e-02
      -1.10560532e-01 -8.56844126e-02 -7.46283594e-02 -9.12124393e-02
      -1.05032506e-01 -9.12124393e-02 -9.67404659e-02 -9.95044792e-02
      -1.10560532e-01 -1.02268493e-01 -8.56844126e-02 -8.56844126e-02
      -8.56844126e-02 -8.56844126e-02 -8.56844126e-02 -9.12124393e-02
      -6.63363195e-02 -7.18643461e-02 -9.95044792e-02 -1.10560532e-01
      -7.18643461e-02 -8.29203993e-02 -8.01563860e-02 -8.56844126e-02
      -8.84484260e-02 -7.46283594e-02 -8.01563860e-02 -7.46283594e-02
      -7.73923727e-02 -9.12124393e-02 -7.18643461e-02 -8.01563860e-02
      -8.29203993e-02 -8.56844126e-02 -1.13324546e-01 -9.39764526e-02
      -9.67404659e-02 -1.27144612e-01 -6.63363195e-02 -1.21616586e-01
      -1.16088559e-01 -9.95044792e-02 -9.12124393e-02 -1.02268493e-01
      -1.05032506e-01 -8.84484260e-02 -8.29203993e-02 -8.84484260e-02
      -9.67404659e-02 -1.07796519e-01 -1.40964679e-01 -1.05032506e-01
      -1.02268493e-01 -7.73923727e-02 -1.35436652e-01 -9.95044792e-02
      -9.39764526e-02 -1.10560532e-01 -9.39764526e-02 -8.56844126e-02
      -9.95044792e-02 -1.16088559e-01 -1.27144612e-01 -1.13324546e-01
      -9.95044792e-02 -9.67404659e-02 -9.67404659e-02 -1.29908626e-01
      -8.01563860e-02 -9.12124393e-02 -8.29203993e-02 -1.05032506e-01
      -9.95044792e-02 -1.05032506e-01 -8.56844126e-02 -9.95044792e-02
      -9.39764526e-02 -1.02268493e-01 -1.05032506e-01 -9.67404659e-02
      -7.73923727e-02 -8.01563860e-02]
     [-9.69563911e-01 -2.32852711e-02 -2.81628983e-04 -1.30197073e-02
       1.50228403e-02 -1.30197073e-02 -5.32053453e-03 -1.55860983e-02
      -1.30197073e-02  3.02334333e-02  7.32366750e-03  2.77609187e-02
       4.75727656e-03  3.03273096e-02  4.75727656e-03  3.03273096e-02
      -1.30197073e-02  9.89005844e-03  2.19088563e-03  4.85115289e-03
      -3.75505311e-04 -1.04533164e-02  1.25503257e-02  4.75727656e-03
       2.77609187e-02 -3.75505311e-04 -1.55860983e-02  7.32366750e-03
       4.75727656e-03 -2.81628983e-04  3.27998242e-02 -3.75505311e-04
      -2.75414359e-03  4.85115289e-03  3.02334333e-02  4.75727656e-03
      -3.75505311e-04  3.02334333e-02  1.50228403e-02  7.32366750e-03
      -2.81628983e-04 -5.50828719e-03  7.41754383e-03 -2.81628983e-04
       4.85115289e-03  4.75727656e-03  4.85115289e-03 -2.31913948e-02
       9.89005844e-03  7.32366750e-03  4.38177125e-03 -3.11721966e-02
       4.38177125e-03 -3.11721966e-02  6.94816219e-03  2.74792897e-02
      -5.63257966e-04  1.22686967e-02  6.94816219e-03 -1.32074600e-02
      -5.63257966e-04  2.74792897e-02  4.38177125e-03 -3.11721966e-02
       7.13591485e-03  9.51455313e-03  9.70230578e-03 -2.86058056e-02
      -7.51010621e-04 -2.86058056e-02  3.26120716e-02 -3.37385875e-02
       4.38177125e-03 -3.37385875e-02  1.20809441e-02  9.51455313e-03
      -7.51010621e-04  6.94816219e-03 -2.86058056e-02 -2.86058056e-02
      -2.86058056e-02 -2.86058056e-02 -2.86058056e-02 -3.37385875e-02
       1.48350877e-02  9.70230578e-03  9.51455313e-03 -7.51010621e-04
       9.70230578e-03 -5.63257966e-04  2.00313297e-03 -2.86058056e-02
      -3.11721966e-02  3.26120716e-02  2.00313297e-03  3.26120716e-02
       3.00456806e-02  1.72137259e-02  3.51784625e-02  2.74792897e-02
      -5.63257966e-04 -2.86058056e-02  4.76349119e-02  1.46473350e-02
       1.20809441e-02  3.48029572e-02 -1.06410691e-02  3.99357391e-02
      -5.88379250e-03  9.51455313e-03 -3.37385875e-02  6.94816219e-03
       4.38177125e-03  1.97801169e-02 -5.63257966e-04 -3.11721966e-02
       1.20809441e-02 -4.91369331e-02  2.19710025e-02  4.38177125e-03
       6.94816219e-03  3.00456806e-02  7.80560978e-02  9.51455313e-03
       1.46473350e-02 -7.51010621e-04  1.46473350e-02 -2.86058056e-02
       9.51455313e-03 -5.88379250e-03 -1.61493562e-02 -3.31740156e-03
       9.51455313e-03  1.20809441e-02  1.20809441e-02  3.22365663e-02
       2.00313297e-03  1.72137259e-02 -5.63257966e-04  4.38177125e-03
       9.51455313e-03  4.38177125e-03 -2.86058056e-02  9.51455313e-03
       1.46473350e-02  6.94816219e-03  4.38177125e-03  1.20809441e-02
       3.00456806e-02  2.74792897e-02]]

     Eigen Values:
     [2.75099335e+02 9.49560500e-30]

     Projected Data:
     [[-69.89722815  -1.87878642]
     [-33.71795232  -1.87878642]]

```

```

```

```

```

```

```

```

```

```

```

```

```
# here it is from an article online that loads things into traing and test features

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"
# load dataset into Pandas DataFrame
df = pd.read_csv(url, names=['sepal length','sepal width','petal length','petal width','target'])

from sklearn.preprocessing import StandardScaler
features = ['sepal length', 'sepal width', 'petal length', 'petal width']
# Separating out the features
x = df.loc[:, features].values
# Separating out the target
y = df.loc[:,['target']].values
# Standardizing the features
x = StandardScaler().fit_transform(x)

```

```
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
principalComponents = pca.fit_transform(x)
principalDf = pd.DataFrame(data = principalComponents
             , columns = ['principal component 1', 'principal component 2'])

finalDf = pd.concat([principalDf, df[['target']]], axis = 1)

```

```
fig = plt.figure(figsize = (8,8))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
targets = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica']
colors = ['r', 'g', 'b']
for target, color in zip(targets,colors):
    indicesToKeep = finalDf['target'] == target
    ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']
               , finalDf.loc[indicesToKeep, 'principal component 2']
               , c = color
               , s = 50)
ax.legend(targets)
ax.grid()
```

# Stretch Goal

## 1) Do NOT work on the stretch goal until you feel like you have a firm grasp of eigenvectors, eigenvalues, and PCA. Prioritize self-study over the stretch goal if you are not comfortable with those topics yet.

## 2) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

You don't necessarily have to use this resource, but this will get you started:
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)


# Yet antoher re-implementation and cross-check for absorption on disk
In this notebook I will implement the formulas for the $\gamma \gamma$ absorption on disk presented in Finke 2016 and Dermer 2009 and comapre them.

```python
import numpy as np
import astropy.units as u
from astropy.constants import m_e, c, G, M_sun
import matplotlib.pyplot as plt
import pkg_resources
from IPython.display import Image
import sys
sys.path.append("../../")
from agnpy.targets import SSDisk
from agnpy.absorption import sigma
from agnpy.utils.math import axes_reshaper
from agnpy.utils.conversion import nu_to_epsilon_prime, to_R_g_units
```

```python
# useful constants
# electron radius
r_e = 2.81794 * 1e-15 * u.cm
```

## Formula derived from [Finke 2016](https://iopscience.iop.org/article/10.3847/0004-637X/830/2/94/pdf)
Let us consider the general formula for absorption

\begin{equation}
\tau_{\gamma \gamma}(\hat{\nu}_1) =
    \int_{r}^{\infty} {\rm d}l \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{-1}^{1}  {\rm d}\mu \, (1 - \cos\psi) \,
    \int_{0}^{\infty} {\rm d}\epsilon \,
    \frac{\underline{u}(\epsilon, \Omega; l)}{\epsilon m_{\rm e} c^2} \,
    \sigma_{\gamma \gamma}(s),
\end{equation}

where, for the Skakura Sunyaev Disk

\begin{equation}
    \underline{u}(\epsilon, \Omega; r) = \frac{3 G M \dot{m}}{(4 \pi)^2 c R^3}\varphi(R) \, \delta(\epsilon - \epsilon_0(R))
\end{equation}

replacing, and simplifying

\begin{equation}
\begin{split}
\tau_{\gamma \gamma}(\hat{\nu}_1) &=
    \int_{r}^{\infty} {\rm d}l \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{\mu_{\rm min}}^{\mu_{\rm max}}  {\rm d}\mu \, (1 - \cos\psi) \,
    \int_{0}^{\infty} {\rm d}\epsilon \,
    \frac{3 G M \dot{m}}{(4 \pi)^2 c R^3}\varphi(R) \, \delta(\epsilon - \epsilon_0(R))
    \frac{1}{\epsilon m_{\rm e} c^2} \sigma_{\gamma \gamma}(s) \\
    &= \frac{3 G M \dot{m}}{(4 \pi)^2 m_{\rm e} c^3}
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{\mu_{\rm min}}^{\mu_{\rm max}}  {\rm d}\mu \, (1 - \cos\psi) \,
    \frac{1}{R^3}\frac{\varphi(R)}{\epsilon(R)} \sigma_{\gamma \gamma}(s).
\end{split}
\end{equation}

The extremes of integration in cosine $(\mu_{\rm in}, \mu_{\rm out})$ will change depending on the distance $l$, so let us change to an integration in $R$: $\mu = \frac{1}{\sqrt{1 + \frac{R^2}{r^2}}} \Rightarrow \frac{{\rm d}\mu}{{\rm d}R} = - \frac{1}{2}\frac{1}{\left(1 + \frac{R^2}{r^2}\right)^{3/2}}\frac{2R}{r^2} = - \frac{1}{\mu^3}\frac{R}{r^2}$.

\begin{equation}
\begin{split}
\tau_{\gamma \gamma}(\hat{\nu}_1) &=
    \frac{3 G M \dot{m}}{(4 \pi)^2 m_{\rm e} c^3}
    \int_{r}^{\infty} {\rm d}l \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{R_{\rm out}}^{R_{\rm in}} -\frac{1}{\mu^3}\frac{R}{l^2}{\rm d}R \,
    (1 - \cos\psi) \, \frac{1}{R^3}\frac{\varphi(R)}{\epsilon(R)} \sigma_{\gamma \gamma}(s) \\
    &= \frac{3 G M \dot{m}}{(4 \pi)^2 m_{\rm e} c^3}
    \int_{r}^{\infty} {\rm d}l \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{R_{\rm in}}^{R_{\rm out}}{\rm d}R \,
    (1 - \cos\psi) \, \frac{1}{\mu^3}\frac{1}{l^2}\frac{1}{R^2}
    \frac{\varphi(R)}{\epsilon(R)} \sigma_{\gamma \gamma}(s) \\
    &= \frac{3 G M \dot{m}}{(4 \pi)^2 m_{\rm e} c^3 R_g^2}
    \int_{r}^{\infty} \frac{{\rm d}l}{R_g} \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{R_{\rm in}}^{R_{\rm out}} \frac{{\rm d}R}{R_g} \,
    (1 - \cos\psi) \, \frac{1}{\mu^3}\frac{R_g^2}{l^2}\frac{R_g^2}{R^2}
    \frac{\varphi(R)}{\epsilon(R)} \sigma_{\gamma \gamma}(s) \\
    &= \frac{3 \dot{m}}{(4 \pi)^2 m_{\rm e} c R_g}
    \int_{\tilde{r}}^{\infty} {\rm d}\tilde{l} \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{\tilde{R}_{\rm in}}^{\tilde{R}_{\rm out}} {\rm d}\tilde{R} \,
    (1 - \cos\psi) \, \frac{1}{\mu^3}\frac{1}{\tilde{l}^2}\frac{1}{\tilde{R}^2}
    \frac{\varphi(\tilde{R})}{\epsilon(\tilde{R})} \sigma_{\gamma \gamma}(s) \\
    &= \frac{3 L_{\rm disk}}{(4 \pi)^2 \eta m_{\rm e} c^3 R_g}
    \int_{\tilde{r}}^{\infty} {\rm d}\tilde{l} \,
    \int_{0}^{2\pi} {\rm d}\phi \,
    \int_{\tilde{R}_{\rm in}}^{\tilde{R}_{\rm out}} {\rm d}\tilde{R} \,
    (1 - \cos\psi) \, \frac{1}{\mu^3}\frac{1}{\tilde{l}^2}\frac{1}{\tilde{R}^2}
    \frac{\varphi(\tilde{R})}{\epsilon(\tilde{R})} \sigma_{\gamma \gamma}(s),
\end{split}
\end{equation}

and we have used $R_g=GM/c^2$ and $L_{\rm disk} = \eta \dot{m} c^2$.

Assuming $\mu_s=1 \Rightarrow \cos\psi=\mu$ and
\begin{equation}
\tau_{\gamma \gamma}(\hat{\nu}_1) = \frac{3 L_{\rm disk}}{8 \pi^2 \eta m_{\rm e} c^3 R_g}
    \int_{\tilde{r}}^{\infty} {\rm d}\tilde{l} \,
    \int_{\tilde{R}_{\rm in}}^{\tilde{R}_{\rm out}} {\rm d}\tilde{R} \,
    \frac{(1 - \mu)}{\mu^3} \frac{1}{\tilde{l}^2 \tilde{R}^2}
    \frac{\varphi(\tilde{R})}{\epsilon(\tilde{R})} \sigma_{\gamma \gamma}(s).
\end{equation}

```python
def evaluate_tau_disk_finke_2016(
    nu,
    z,
    M_BH,
    L_disk,
    eta,
    R_in,
    R_out,
    r,
    R_tilde_size=100,
    l_tilde_size=50,
):
    """expression of the disk absorption derived from the formulas in Finke 2016"""
    # conversions
    R_g = (G * M_BH / c ** 2).to("cm")
    r_tilde = to_R_g_units(r, M_BH)
    R_in_tilde = to_R_g_units(R_in, M_BH)
    R_out_tilde = to_R_g_units(R_out, M_BH)
    # multidimensional integration
    R_tilde = np.linspace(R_in_tilde, R_out_tilde, R_tilde_size)
    l_tilde = np.logspace(0, 6, l_tilde_size) * r_tilde
    epsilon_1 = nu_to_epsilon_prime(nu, z)
    _R_tilde, _l_tilde, _epsilon_1 = axes_reshaper(R_tilde, l_tilde, epsilon_1)
    _epsilon = SSDisk.evaluate_epsilon(L_disk, M_BH, eta, _R_tilde)
    _phi_disk = 1 - (R_in_tilde / _R_tilde) ** (1 / 2)
    _mu = (1 + _R_tilde ** 2 / _l_tilde ** 2) ** (-1 / 2)
    s = _epsilon * _epsilon_1 * (1 - _mu) / 2
    integrand = (
        (1 - _mu)
        / _mu ** 3
        / _l_tilde ** 2
        / _R_tilde ** 2
        * _phi_disk
        / _epsilon
        * sigma(s)
    )
    integral_R_tilde = np.trapz(integrand, R_tilde, axis=0)
    integral = np.trapz(integral_R_tilde, l_tilde, axis=0)
    prefactor = 3 * L_disk / (8 * np.pi**2 * eta * m_e * c**3 * R_g)
    return (prefactor * integral).to_value("")
```

```python
# disk parameters as in Finke 2016
M_BH = 1.2 * 1e9 * M_sun.cgs
L_disk = 2 * 1e46 * u.Unit("erg s-1")
R_g = (G * M_BH / c**2).to("cm")
eta = 1 / 12
R_in = 6 * R_g
R_out = 200 * R_g

disk = SSDisk(M_BH, L_disk, eta, R_in, R_out)
print(disk)

# parameters of 3C454.3
z = 0.859
R_line = 1.1 * 1e17 * u.cm
# distance of the disk
r = 0.1 * R_line
```

    * Shakura Sunyaev accretion disk:
     - M_BH (central black hole mass): 2.39e+42 g
     - L_disk (disk luminosity): 2.00e+46 erg / s
     - eta (accretion efficiency): 8.33e-02
     - dot(m) (mass accretion rate): 2.67e+26 g / s
     - R_in (disk inner radius): 1.06e+15 cm
     - R_out (disk inner radius): 3.54e+16 cm

```python
# let us try to reproduce the data shared by Finke
for l in ["1e-1", "1e0", "1e1", "1e2"]:
    # read the reference SED in Figure 14 of Finke 2016
    data_file_ref_abs = f"../../agnpy/data/reference_taus/finke_2016/figure_14_left/tau_SSdisk_r_{l}_R_Ly_alpha.txt"
    data_ref = np.loadtxt(data_file_ref_abs, delimiter=",")
    # read energies and opacity values
    E_ref = data_ref[:, 0] * u.GeV
    tau_ref = data_ref[:, 1]
    nu_ref = E_ref.to("Hz", equivalencies=u.spectral())
    # compute the tau
    _r = float(l) * R_line
    tau = evaluate_tau_disk_finke_2016(nu_ref, z, M_BH, L_disk, eta, R_in, R_out, _r)
    # plot it against the reference
    plt.loglog(E_ref, tau_ref, ls="--", color="k", label="reference")
    plt.loglog(E_ref, tau, ls="-", color="crimson", label="agnpy")
    plt.ylim([1e-8, 1e6])
    plt.title(f"r = {l} R(Ly alpha)")
    plt.ylabel(r"$\tau_{\gamma\gamma}$")
    plt.xlabel("E / GeV")
    plt.legend()
    plt.show()
```

## Formula from [Dermer et al. 2009](https://iopscience.iop.org/article/10.1088/0004-637X/692/1/32/pdf), Eq. 80
Let us directly take Eq. (80) in Dermer et al. 2009 reporting the same quantity (again we are assuming $\mu_s = 1$).
\begin{equation}
\tau_{\gamma \gamma}(\hat{\nu}_1) = 3 \times 10^6 \frac{l_{\rm Edd}^{3/4} M_8^{1/4}}{\eta^{3/4}}
    \int_{\tilde{r}}^{\infty} \frac{{\rm d}\tilde{l}}{\tilde{l}^2} \,
    \int_{\tilde{R}_{\rm in}}^{\tilde{R}_{\rm out}} \frac{{\rm d}\tilde{R}}{\tilde{R}^{5/4}} \,
    \frac{\left[\varphi(\tilde{R})\right]^{1/4}}{\left(1 + \frac{\tilde{R}^2}{\tilde{l}^2}\right)^{3/2}}
    \left[\frac{\sigma_{\gamma \gamma}(s)}{\pi r_e^2}\right] (1 - \mu).
\end{equation}

As the author notice:
$$ \frac{\tau_{\gamma \gamma}}{M_8} = \text{function of } \xi \text{ and } r$$

where $\xi = \left(\frac{l_{\rm Edd}}{\eta M_8}\right)^{1/4}$.

Reorganising the formula:

\begin{equation}
\tau_{\gamma \gamma}(\hat{\nu}_1) = 3 \times 10^6 M_8 \xi^3
    \int_{\tilde{r}}^{\infty} \frac{{\rm d}\tilde{l}}{\tilde{l}^2} \,
    \int_{\tilde{R}_{\rm in}}^{\tilde{R}_{\rm out}} \frac{{\rm d}\tilde{R}}{\tilde{R}^{5/4}} \,
    \frac{\left[\varphi(\tilde{R})\right]^{1/4}}{\left(1 + \frac{\tilde{R}^2}{\tilde{l}^2}\right)^{3/2}}
    \left[\frac{\sigma_{\gamma \gamma}(s)}{\pi r_e^2}\right] (1 - \mu).
\end{equation}

```python
def evaluate_tau_dermer_2009(
    E,
    z,
    M_8,
    csi,
    R_in_tilde,
    R_out_tilde,
    r_tilde,
    R_tilde_size=100,
    l_tilde_size=50,
):
    """expression of the disk absorption copied from Dermer 2009"""
    R_tilde = np.linspace(R_in_tilde, R_out_tilde, R_tilde_size)
    l_tilde = np.logspace(0, 6, l_tilde_size) * r_tilde
    epsilon_1 = (E / (m_e * c**2)).to_value("") * (1 + z)
    _R_tilde, _l_tilde, _epsilon_1 = axes_reshaper(R_tilde, l_tilde, epsilon_1)
    _epsilon = 2.7 * 1e-4 * csi * _R_tilde ** (-3 / 4)
    _phi_disk = 1 - (R_in_tilde / _R_tilde) ** (1 / 2)
    _mu = 1 / (1 + _R_tilde ** 2 / _l_tilde ** 2) ** (1 / 2)
    _s = _epsilon_1 * _epsilon * (1 - _mu) / 2
    integrand = (
        1
        / _l_tilde ** 2
        / _R_tilde ** (5 / 4)
        / (1 + _R_tilde ** 2 / _l_tilde ** 2) ** (3 / 2)
        * _phi_disk ** (1 / 4)
        * (sigma(_s) / (np.pi * r_e ** 2)).to_value("")
        * (1 - _mu)
    )
    integral_R_tilde = np.trapz(integrand, R_tilde, axis=0)
    integral = np.trapz(integral_R_tilde, l_tilde, axis=0)
    prefactor = 3 * 1e6 * M_8 * csi ** 3
    return prefactor * integral
```

```python
# let us try to reproduce Figure 7 in Dermer et al 2009
Image("figures/figure_7_dermer_et_al_2009.png", width = 600, height = 400)
```

```python
z = 1
M_8 = 1
csi = [1, 10]
r_tilde = [10, 1e2, 1e3]

E = np.logspace(-3, 2, 50) * u.TeV

for color, _csi in zip(["red", "green"], csi):
    for ls, _r_tilde in zip(["-", ":", "--"], r_tilde):
        tau = evaluate_tau_dermer_2009(E, z=z, M_8=M_8, csi=_csi, R_in_tilde=6, R_out_tilde=200, r_tilde=_r_tilde)
        plt.loglog(E, tau, color=color, ls=ls, label=f"csi={_csi}, r = {_r_tilde} R_g")
plt.xlim([1e-3, 1e2])
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5))
plt.ylabel(r"$\tau_{\gamma\gamma}$")
plt.xlabel("E / GeV")
plt.show()
```

## Check Finke 2016 vs Dermer 2009
Let us check the two formulas implementations against each other.
Let us use the same parameters of the accretion disk we defined above.

```python
print(f"distance from BH: {r:.2e}")
print("accretion disk considered:")
print(disk)
```

    distance from BH: 1.10e+16 cm
    accretion disk considered:
    * Shakura Sunyaev accretion disk:
     - M_BH (central black hole mass): 2.39e+42 g
     - L_disk (disk luminosity): 2.00e+46 erg / s
     - eta (accretion efficiency): 8.33e-02
     - dot(m) (mass accretion rate): 2.67e+26 g / s
     - R_in (disk inner radius): 1.06e+15 cm
     - R_out (disk inner radius): 3.54e+16 cm

```python
nu = E.to("Hz", equivalencies = u.spectral())
tau_finke = evaluate_tau_disk_finke_2016(nu, z, disk.M_BH, disk.L_disk, disk.eta, disk.R_in, disk.R_out, r)
# calculate csi to use Dermer's formula
csi = (disk.l_Edd / (disk.M_8 * disk.eta))**(1/4)
r_tilde = (r / disk.R_g).to_value("")
tau_dermer = evaluate_tau_dermer_2009(E, z, disk.M_8, csi, disk.R_in_tilde, disk.R_in_tilde, r_tilde)

plt.loglog(E, tau_finke, label="Finke 2016")
plt.loglog(E, tau_finke, label="Dermer 2009", ls=":")
plt.ylabel(r"$\tau_{\gamma\gamma}$")
plt.xlabel("E / GeV")
plt.legend()
plt.show()
```


## Principal Component Analysis
---

[Image Source](http://www.sthda.com/english/sthda-upload/figures/principal-component-methods/006-principal-component-analysis-color-individuals-by-groups-and-variables-by-contributions-1.png)

>- Understanding the need for PCA
>- The statistics of PCA
>- The mathematics of PCA
>- A basic example

---

Dimensionality reduction by creating new  uncorrelated variables that are linear combination of original variables and successively maximize variance (minimizing information loss) and also provide interpretability and visulization aid.

```python
from __future__ import print_function
from __future__ import division

import pandas as pd
import numpy as np
from scipy.sparse.linalg import svds as sparse_svd
from sklearn.decomposition import PCA

from sklearn.preprocessing import StandardScaler

from sklearn.datasets import load_boston,load_iris

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sbn

```

### The statistics of PCA
---

[Correlation Coefficients](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Correlation_examples2.svg/400px-Correlation_examples2.svg.png)

#### Variance (Resource: See Wikipedia for revisiting variance and covariance.)

$$
\begin{align}
\operatorname{var}(X) &= \operatorname{E}\left[(X - \mu)^2 \right]
= \operatorname{E}\left[(X - \operatorname{E}[X])^2\right] \\[4pt]
&= \operatorname{E}\left[X^2 - 2X\operatorname{E}[X] + \operatorname{E}[X]^2\right] \\[4pt]
&= \operatorname{E}\left[X^2\right] - 2\operatorname{E}[X]\operatorname{E}[X] + \operatorname{E}[X]^2 \\[4pt]
&= \operatorname{E}\left[X^2 \right] - \operatorname{E}[X]^2
\end{align}
$$

#### Covariance
$$\large
\operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]}
$$
In other words,
$$
\begin{align}
\operatorname{cov}(X, Y)
&= \operatorname{E}\left[\left(X - \operatorname{E}\left[X\right]\right) \left(Y - \operatorname{E}\left[Y\right]\right)\right] \\
&= \operatorname{E}\left[X Y - X \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] Y + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right]\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right],
\end{align}
$$

Note that $\operatorname{var}(X) = \operatorname{cov}(X, X).$

Note that two random variables $X$ and $Y$ are uncorrelated when
$$\operatorname{cov}(X, Y) = 0 \Leftrightarrow E[XY] = E[X]E[Y]$$

#### Correlation coefficient
$$
\rho(X,Y) = \frac{Cov(X,Y)}{\sigma_X \sigma_Y}
$$

#### Basic Properties of Variance

>- $ \operatorname{var}(X)\ge 0.$

>- $ \operatorname{var}(a) = 0.$

>- $ \operatorname{var}(X)= 0 \iff \exists a : P(X=a) = 1.$

>- $ \operatorname{var}(X+a)=\operatorname{var}(X).$

>- $ \operatorname{var}(aX)=a^2\operatorname{var}(X).$

>- $\operatorname{var}(aX+bY)=a^2\operatorname{var}(X)+b^2\operatorname{var}(Y)+2ab\, \operatorname{cov}(X,Y),$

>- $ \operatorname{var}(aX-bY)=a^2\operatorname{var}(X)+b^2\operatorname{var}(Y)-2ab\, \operatorname{cov}(X,Y).$

In general, for the sum of $n$ random variables $\{X_1,\dots,X_n\}$:
$$ \operatorname{var}\left(\sum_{i=1}^n X_i\right)=\sum_{i,j=1}^n\operatorname{cov}(X_i,X_j)=\sum_{i=1}^n\operatorname{var}(X_i)+\sum_{i\ne j}\operatorname{cov}(X_i,X_j).$$

Thus the  variance of a linear combination random variables :

$$
\begin{align}
\operatorname{var}\left( \sum_{i=1}^n a_iX_i\right)
&=\sum_{i,j=1}^{n} a_ia_j\operatorname{cov}(X_i,X_j) \\
&=\sum_{i=1}^n a_i^2\operatorname{var}(X_i)+\sum_{i\not=j}a_ia_j\operatorname{cov}(X_i,X_j)\\
& =\sum_{i=1}^n a_i^2\operatorname{var}(X_i)+2\sum_{1\le i<j\le n}a_ia_j\operatorname{cov}(X_i,X_j) = a^T S a.
\end{align}
$$

Where  $S$ is the covariance matrix.

$$\Large
\color{blue}{\operatorname{var}(X a) = a^T S\,a.}
$$

The random variables $X_1,\dots,X_n$ are said to be uncorrelated  if

$$\operatorname{cov}(X_i,X_j)=0\ ,\ \forall\ (i\ne j)$$

It follows immediately from the expression given earlier that if the random variables $X_1,\dots,X_n$ are uncorrelated, then

$$\operatorname{var}\left(\sum_{i=1}^n X_i\right)=\sum_{i=1}^n\operatorname{var}(X_i).$$

**PROBLEM**: Prove that independence implies uncorrelatedness and provide a counterexample to show that uncorrelatedness does not imply independence.

#### Population covariance
$$\large
\sigma_{jk}=\frac 1 n \sum_{i=1}^n \left(  X_{ij} - \operatorname{E}(X_j)\right)  \left( X_{ik} - \operatorname{E}(X_k)\right)
$$

#### Sample covariance
$$\large
\sigma_{jk}=\frac{1}{n-1}\sum_{i=1}^{n}\left(  X_{ij}-\bar{X}_j \right)  \left( X_{ik}-\bar{X}_k \right),
$$

#### Covariance matrix
$$\Large
\color{blue}{ S = [\sigma_{jk}] = \frac{1}{n-1} [(X-\mu_X)^T(X-\mu_X)]}
$$

where $\mu_X$ is the vector of means of $m$ random vectors denoting the $m$ features of the dataset.

### Mathematics of PCA
---
 First Principal Component:  Our aim is to find a well-defined linear combination of features that has maximum variance. This boils down to

$$\large \color{blue}{ \operatorname{maximize}\ \  \operatorname{Var}\left( \sum_{i=1}^N a_iX_i\right) = a^T S\, a,   \quad \textrm{such that }\ \ {a^Ta=1}.
}
$$

Or equivalently, by using   Lagrange's optimization method (find stationary points)

$$ \large\color{green}{
\mathcal{L}(a,\lambda)  = a^T S\, a - \lambda(a^T a - 1)  \quad \textrm{such that }\ \ {a^Ta=1}.}
$$

where $\lambda$ is the lagrange multiplier.

---

#### Eigen Decomposition based PCA

[See spectral theorem for orthonormal eigen-decomposition](https://en.wikipedia.org/wiki/Spectral_theorem#Hermitian_maps_and_Hermitian_matrices)

**PROBLEM**: Prove that the vector $a$ could be given by the eigenvector associated with the largest eigenvalue of $S$.

$$\large
\nabla_{a} \mathcal{L}(a,\lambda) = 2Sa - 2\lambda  a = 0
$$

 This implies that $(a,\lambda)$ from an eigen-pair of the covariance matrix. In this case the variance is maximized as

 $$\large
 \operatorname{var} (Xa) = a^T S a = a^T \lambda a = \lambda a^T a = \lambda
 $$

 by taking $a$ to be the eigenvector corresponding to the largest eigenvalue (which happens to be the variance).

  Second Principal Component:
Next, we plan to find another linear combination $Xb$ that is uncorrelated with $Xa$ and has the largest variance. This need the additional condition that

$$
\large\operatorname{cov}(Xa, Xb)=b^T S a =b^T(\lambda a) =\lambda b^T a=0,
$$

implying that $a$ and $b$ be mutually orthogonal. This means that $b$ could be chosen to be the eigenvector associated with the second eigenvalue in the eigenvector decomposition of $S$.

  Remaining Principal Components:  We can continue with the above process to find the $k$-th PCs $X a_k$ where $a_k$ is the eigevector associated with the $k$-th largest eigenvalue of $S$  in the spectral-decomposition of the covariance matrix $S$.

 $$\large
 \operatorname{var} (X a_k) = a_k^T S a_k = a_k^T \lambda_k a_k = \lambda_k a_k^T a_k = \lambda_k.
 $$

Geometrically PCA creates a new set of orthogonal bsed on the maximization of the variation in the data where the first axis (PC1) accounts for the maximum variance and so on.

#### SVD Based PCA

**PROBLEM**: Verify that the eigenvectors of $S$ are   the right singular vectors of the column centered data matrix  $\tilde{X}$

$$
\begin{align}
(n-1)S &= \tilde{X}^T\tilde{X} \\
&= (U\Sigma V^T)^T (U\Sigma V^T)\\
&= V \Sigma^2 V^T\\
\Leftrightarrow (n-1)S V &= V \Sigma^2
\end{align}
$$

 PCA of X is equivalent to SVD of column centered $\tilde{X}$. The variance in the two approaches have the following relations:

$$
\operatorname{var}(Xa_k) = \lambda_k = \sigma_k^2/(n-1).
$$

### PCA Terminology
---
>- **Principal Components**: The linear combinations $Xa_k$ are called PCs.

>- **PC Loadings**: Elements of the eigenvector $a_k$ are called PC loadings.

>- **PC Scores**:Elemets of $Xa_k$ are called PC scores.

>- **Explained variance ratio**: of a PC $Xa_k$ is
$$
\pi_k = \frac{\lambda_k}{\sum \lambda_i } = \frac{\lambda_k}{trace(S) }
$$

>- **Total explained variance ratio**: of the first $k$ PCs is
$$
\sum_{i=1}^k \pi_i = \frac{\lambda_1+ \cdots+ \lambda_k}{\sum \lambda_i }
$$

#### Cautions
>- Only the relative magnitude and sign patterns of the loadings (and scores) are meaningful.

### Examples: PCA of diabetes data

```python
## PIMA Indian dataset: Has been removed from UCI now but available on Kaggle
## It is available with scikitlearn.datasets
#url = "https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data"
col_names = ['preg', 'gtt', 'bp', 'skin', 'test', 'bmi', 'pedi', 'age', 'class'] # Short Names
#df_pima = pd.read_csv("./pima_diabetes.csv", header=0)
df_pima = pd.read_csv("./pima_diabetes.csv", skiprows=1, names=col_names)
```

```python
df_pima.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 768 entries, 0 to 767
    Data columns (total 9 columns):
     #   Column  Non-Null Count  Dtype
    ---  ------  --------------  -----
     0   preg    768 non-null    int64
     1   gtt     768 non-null    int64
     2   bp      768 non-null    int64
     3   skin    768 non-null    int64
     4   test    768 non-null    int64
     5   bmi     768 non-null    float64
     6   pedi    768 non-null    float64
     7   age     768 non-null    int64
     8   class   768 non-null    int64
    dtypes: float64(2), int64(7)
    memory usage: 54.1 KB

```python
df_pima.head()
```

      preg
      gtt
      bp
      skin
      test
      bmi
      pedi
      age
      class

      0
      6
      148
      72
      35
      0
      33.6
      0.627
      50
      1

      1
      1
      85
      66
      29
      0
      26.6
      0.351
      31
      0

      2
      8
      183
      64
      0
      0
      23.3
      0.672
      32
      1

      3
      1
      89
      66
      23
      94
      28.1
      0.167
      21
      0

      4
      0
      137
      40
      35
      168
      43.1
      2.288
      33
      1

```python
# Correlation between features
pima_corr =df_pima.corr()
corr =pima_corr.values
```

```python
# Correlation between features

mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(8, 8))

# Generate a custom diverging colormap
cmap = sbn.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sbn.heatmap(pima_corr, mask=mask, cmap=cmap, vmax=1.0,annot=True, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
```

```python
data_mat = df_pima.values
X = data_mat[:,0:8]
Y = data_mat[:,8]
```

```python
np.set_printoptions(precision=2, suppress=True)
sigmaX = np.std(X,axis=0)
# X = X/sigmaX # If Correlation PCA is required
muX = np.mean(X, axis=0)
n = X.shape[0]-1
print("The size of data set is:", n)
S = np.dot((X - muX).T, (X - muX)) / (n-1)
print('Covariance matrix: \n:')
print(S)
```

    The size of data set is: 767
    Covariance matrix:
    :
    [[   11.37    13.97     9.23    -4.4    -28.59     0.47    -0.04    21.6 ]
     [   13.97  1023.58    94.55    29.28  1222.53    55.8      1.46    99.21]
     [    9.23    94.55   375.14    64.11   198.64    43.06     0.26    54.59]
     [   -4.4     29.28    64.11   254.81   804.03    49.44     0.97   -21.41]
     [  -28.59  1222.53   198.64   804.03 13298.52   180.01     7.08   -57.22]
     [    0.47    55.8     43.06    49.44   180.01    62.24     0.37     3.36]
     [   -0.04     1.46     0.26     0.97     7.08     0.37     0.11     0.13]
     [   21.6     99.21    54.59   -21.41   -57.22     3.36     0.13   138.48]]

```python
# Eigenvalue Decomposition of S
w, v = np.linalg.eig(S)

#print('Eigenvectors \n%s' %v)
print('\nEigenvalues \n%s' %w)
print('\n Explained Ratio: \n%s' %(w/np.sum(w)))
```

    Eigenvalues
    [13474.14   933.98   391.09   198.44   112.84    45.89     7.77     0.1 ]

     Explained Ratio:
    [0.89 0.06 0.03 0.01 0.01 0.   0.   0.  ]

```python
#SVD based approach
muX = np.mean(X, axis=0)
n = X.shape[0]-1
print("The size of data set is:", n)
U,s,V = np.linalg.svd(X-muX)
#print('Covariance matrix: \n:')
#print(S)
s2 = s**2;
print('\n Singularvalues \n%s' %s)
print('\n Singularvalues *(n-1) \n {}'.format(s**2/(n-1)))
print('\n Explained Ratio: \n%s' %(s2/np.sum(s2)))
```

    The size of data set is: 767

     Singularvalues
    [3212.66  845.83  547.33  389.88  293.99  187.49   77.15    8.88]

     Singularvalues *(n-1)
     [13474.14   933.98   391.09   198.44   112.84    45.89     7.77     0.1 ]

     Explained Ratio:
    [0.89 0.06 0.03 0.01 0.01 0.   0.   0.  ]

```python
# Instantiate PCA
pca = PCA(n_components=3)
pca.fit((X-muX).T) # Features are the rows in Standard PCA by convention, so we transform

Xpca = pca.transform(X.T)
#print("PCA components: %s") # pca.explained_variance_
#print(pca.components_)
print ("Explained Variance Ratio")
print (pca.explained_variance_ratio_)
```

    Explained Variance Ratio
    [0.89 0.06 0.02]

#### Correlation PCA
---
**Question**: Why is standardization of features desirable? What are its effects on PCA?

Correlation PCA is performed on the correlation matrix in place of the covariance matrix.

Correlation PCA is thus the covariance PCA on standardized data where column $X_j$ is standardized as

$$\large
Z_{ij} = \frac{X_{ij} - \bar{X_j}}{s_j}
$$
where $s_j$ is the standard deviation of the j-th column.

NOTE that the PCs and explained variance in correlation PCs are not the same as in covariance PCs.

```python
# Instantiate PCA
pca = PCA(n_components=8)
muX = np.mean(X, axis=0)
sigmaX = np.std(X,axis=0)
Xn = (X-muX)/sigmaX # One can use the StandardScalar available with ScikitLearn
pca.fit(Xn.T) # Features are the rows, so we transform

Xpca = pca.transform(Xn.T)

#print("PCA components: %s") # pca.explained_variance_
#print(pca.components_)
print ("Explained Variance Ratio")
print (pca.explained_variance_ratio_)
```

    Explained Variance Ratio
    [0.29 0.17 0.15 0.13 0.11 0.07 0.07 0.  ]

#### Some Findamental Issues
---
   >- Correlation-based vs. Covariance based PCA
   >- Biplots
   >- Centering
   >- $n
- Functional PCA
>- Simplified PCA
>- Robust PCA

### Robust PCA
---
**Question**: Do you agree that finding the first-$k$ PCs is equivalent to solving the following problem? Provide your reasoning.

$$\large
\operatorname{minimize}\ \|X-L\|_F,  \textrm{ subject to }\ \operatorname{rank}(L) \leq k.
$$

Note: PCA is sensitive to presence of outliers and errors in dataset.

 Robust Principal Component Analysis : Decomposition as $X \approx L+S$

$$\large
\operatorname{minimize}\ \|L\|_* + \lambda\, \|S\|_1
$$

where $\|L\|_* = \sigma_1 + \cdots \sigma_r$ is called the nuclear norm of $L$ and $\lambda$ is a penalty parameter.
>- In some applications such as video surveillance, we are interested in 'noise'.

>- Choice of $\lambda = 1/ \sqrt{max(n,\,m)}$ works well, so we can avoid parameter tuning.

>- Principal component pursuit is a popular method for this optimization. Under certain non-stringent conditions it is possible to find the low-rank and sparse components with high probability.

[Reading 2](https://arxiv.org/abs/0912.3599)

#### Question: How does $L_1$ norm induce sparsity? LASSO vs Ridge.

[Image Source: WikiMedia](https://upload.wikimedia.org/wikipedia/commons/f/f8/L1_and_L2_balls.svg)

**READING: Augmented Lagrangian Method** for solving constrained optimization problems.

[Wikipedia](https://en.wikipedia.org/wiki/Augmented_Lagrangian_method)

#### Example: PCA on Iris data-set
---

Adapted from the following sources:[Iris Data Wikipedia](https://en.wikipedia.org/wiki/Iris_flower_data_set)
[1](https://ostwalprasad.github.io/machine-learning/PCA-using-python.html), [2](https://stackoverflow.com/questions/39216897/plot-pca-loadings-and-loading-in-biplot-in-sklearn-like-rs-autoplot)

```python
iris= load_iris()
df_iris = pd.DataFrame(iris.data, columns=iris.feature_names)
df_iris.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 150 entries, 0 to 149
    Data columns (total 4 columns):
    sepal length (cm)    150 non-null float64
    sepal width (cm)     150 non-null float64
    petal length (cm)    150 non-null float64
    petal width (cm)     150 non-null float64
    dtypes: float64(4)
    memory usage: 4.8 KB

```python
X_iris = StandardScaler().fit_transform(iris.data)
```

```python
pca_iris = PCA(n_components=4)
pc_iris = pca_iris.fit_transform(X_iris)
pc_iris.shape
```

    (150, 4)

```python
pca_iris.explained_variance_
```

    array([ 2.93035378,  0.92740362,  0.14834223,  0.02074601])

```python
pca_iris.explained_variance_ratio_
```

    array([ 0.72770452,  0.23030523,  0.03683832,  0.00515193])

```python
plt.bar(range(1,len(pca_iris.explained_variance_ratio_ )+1),pca_iris.explained_variance_ratio_ )
plt.ylabel('Explained variance')
plt.xlabel('PCs')
plt.plot(range(1,len(pca_iris.explained_variance_ )+1),
         np.cumsum(pca_iris.explained_variance_),
         c='red',
         label="Cumulative Explained Variance")
plt.legend(loc='upper left')
```

```python

```

```python
# Scree-plot: #comp vs eigenvalue
plt.plot(pca_iris.explained_variance_ratio_)
plt.xlabel('# of components')
plt.ylabel('explained variance ratio')
plt.show()
```

```python
plt.scatter(pc_iris[:, 0], pc_iris[:, 1])
```

```python
ax = sns.heatmap(pca_iris.components_,
                 cmap='YlGnBu',
                 yticklabels=[ "PCA"+str(x) for x in range(1,pcamodel.n_components_+1)],
                 xticklabels=list(iris.feature_names),
                 cbar_kws={"orientation": "horizontal"})
ax.set_aspect("equal")
```

```python
# Biplot

def myplot(score,coeff,labels=None):
    xs = score[:,0]
    ys = score[:,1]
    n = coeff.shape[0]
    scalex = 1.0/(xs.max() - xs.min())
    scaley = 1.0/(ys.max() - ys.min())
    plt.scatter(xs * scalex,ys * scaley,s=5)
    for i in range(n):
        plt.arrow(0, 0, coeff[i,0], coeff[i,1],color = 'r',alpha = 0.5)
        if labels is None:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, "Var"+str(i+1), color = 'green', ha = 'center', va = 'center')
        else:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, labels[i], color = 'g', ha = 'center', va = 'center')

    plt.xlabel("PC{}".format(1))
    plt.ylabel("PC{}".format(2))
    plt.grid()

## Never mind, I have been trying this from different sources
def myplot2(score,coeff,labels=None):
    xs = score[:,0]
    ys = score[:,1]
    n = coeff.shape[0]
    scalex = 1.0/(xs.max() - xs.min())
    scaley = 1.0/(ys.max() - ys.min())
    plt.scatter(xs * scalex,ys * scaley, c = ys)
    for i in range(n):
        plt.arrow(0, 0, coeff[i,0], coeff[i,1],color = 'r',alpha = 0.5)
        if labels is None:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, "Var"+str(i+1), color = 'g', ha = 'center', va = 'center')
        else:
            plt.text(coeff[i,0]* 1.15, coeff[i,1] * 1.15, labels[i], color = 'g', ha = 'center', va = 'center')
    plt.xlim(-1,1)
    plt.ylim(-1,1)
    plt.xlabel("PC{}".format(1))
    plt.ylabel("PC{}".format(2))
    plt.grid()

myplot(pc_iris[:,0:2],np.transpose(pca_iris.components_[0:2, :]),list(iris.feature_names))
plt.show()
```

```python
# Still trying to show the different classes in different colors as in the given picture.
df_pca_iris = pd.DataFrame(pc_iris,
                           columns = ["PC1", "PC2","PC3","PC4"])
color_list = [{0:"r",1:"g",2:"b"}[x] for x in iris.target_names]

fig, ax = plt.subplots()
ax.scatter(x=pc_iris["PC1"], y=pc_iris["PC2"], color=color_list)
```

#### Exercise: Perform PCA on automobile data from UCI
---

```python
from sklearn.datasets import load_breast_cancer
```

```python
bcd = load_breast_cancer()
```

```python
import pandas as pd
```

```python
bcd_df = pd.DataFrame(bcd.data,columns=bcd.feature_names)
bcd_df.head()
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      ...
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      ...
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      ...
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      ...
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      ...
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678

5 rows × 30 columns

```python
ss = StandardScaler()
data_scaled = ss.fit_transform(bcd_df)
```

```python
pca = PCA(n_components=5)
```

```python
data_pca = pca.fit_transform(data_scaled)
```

```python
pca.explained_variance_ratio_
```

    array([0.44, 0.19, 0.09, 0.07, 0.05])

```python
plt.figure(figsize=(8,6))
plt.scatter(data_pca[:,0],data_pca[:,1],c=bcd.target)
plt.legend()
plt.xlabel("PC1")
plt.ylabel("PC2")
```

```python
bcd.target_names
```

    array(['malignant', 'benign'], dtype='<U9')

```python

```

```python

```

```python
%%html
<style>
.eqnbox{
    margin:auto;width:500px;padding:20px;
    border: 3px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}
.eqnbox2{
    margin:auto;width:500px;padding:20px;background-color:#c6d6b4;
    border: 1px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}
</style>
```

```python

```


# Crash course on Nilearn and MVPA (Multi-Variate Pattern Analysis)
In this lesson, we will cover some popular methods in MVPA in the context of brain imaging. Namely,
* Searchlight [Nikolaus et al. 2006 (PNAS)]
* Space-Net [Dohmatob et al. 2014 (PRNI)]

Load Haxby 2001 data

```python
from nilearn.datasets import fetch_haxby
data_files = fetch_haxby()
```

```python
# Load Target labels
import numpy as np
labels = np.recfromcsv(data_files.session_target[0], delimiter=" ")

```

```python
labels
```

    rec.array([('rest', 0), ('rest', 0), ('rest', 0), ..., ('rest', 11), ('rest', 11),
     ('rest', 11)],
              dtype=[('labels', 'S12'), ('chunks', '<i8')])

Restrict to face and house conditions

```python
target = labels['labels']
condition_mask = np.logical_or(target == b"face", target == b"house")
```

Split data into train and test pools

```python
condition_mask_train = np.logical_and(condition_mask, labels['chunks'] <= 6)
condition_mask_test = np.logical_and(condition_mask, labels['chunks'] > 6)
from nilearn.image import index_img
from nilearn.masking import compute_epi_mask
func_filenames = data_files.func[0]
X_train = index_img(func_filenames, condition_mask_train)
X_test = index_img(func_filenames, condition_mask_test)
y_train = target[condition_mask_train]
y_test = target[condition_mask_test]
```

Compute brain mask and background EPI image

```python
from nilearn.image import mean_img
mask_img = compute_epi_mask(func_filenames)
background_img = mean_img(func_filenames)
background_img.to_filename("bg_img.nii.gz")
```

Searchlight
============

Prepare masks

* mask_img is the original mask
* process_mask_img is a subset of mask_img, it contains the voxels that should be processed (we only keep the slice z = 26 and the back of the brain to speed up computation)

```python
from nilearn.decoding import SearchLight
from nilearn.image import new_img_like
process_mask = mask_img.get_data().astype(np.int)
picked_slice = 27
process_mask[..., (picked_slice + 1):] = 0
process_mask[..., :picked_slice] = 0
process_mask[:, 30:] = 0
process_mask_img = new_img_like(mask_img, process_mask)

```

Now, fit a Searchlight model on your data

```python
searchlight = SearchLight(mask_img, process_mask_img=process_mask_img, radius=5.6, verbose=1)
searchlight.fit(X_train, y_train)
```

    [Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    8.4s finished

    SearchLight(cv=None, estimator='svc',
          mask_img=<nibabel.nifti1.Nifti1Image object at 0x7f809d75af10>,
          n_jobs=1,
          process_mask_img=<nibabel.nifti1.Nifti1Image object at 0x7f8070e40f90>,
          radius=5.6, scoring=None, verbose=1)

Plot the model coefficients to see whats really going on.

```python
%matplotlib inline
```

```python
from nilearn.plotting import plot_stat_map, show
coef_img = new_img_like(background_img, searchlight.scores_)
plot_stat_map(coef_img, background_img,
              title="Search Light",
              cut_coords=(-34, -16), display_mode="yz")
```

SpaceNet
========
* This is a family of priors which simultaneously impose sparsity and structure on the maps of regressor coefficients.
* It includes GraphNet [Grosenick et al 2013], TV-L1 [Gramfort et al. 2013 (PRNI)], etc.

Formally, the model is
\begin{equation}
\text{minimize}_{\bf{w} \in \mathbb R^p}\mathcal L(y, X\bf{w}) + \alpha (\rho \|\bf{w}\|_1 + (1-\rho) \Omega(\bf{w}))
\end{equation}
where
* $\bf{w} \in \mathbb R^p$ is the vector of model coefficients and $p$ is the number of voxels in the brain mask (say, $p \sim 10^6$ in a full brain mask!)
* $\mathcal L(y, X\bf{w})$ is the loss term. For example $\mathcal L(y, X\bf{w}) = \frac{1}{2}\|X\bf{w} - y\|_2^2$ in regression settings, equals the logistic loss in classification settings, etc.
* $\alpha > 0$ is the regularization parameter and controls and controls the relative importance of the penalty and loss terms. $\rho \in [0, 1]$ is a mixing parameter and controls the tradeoff between sparsity and structure. For example  $\rho = 1$ corresponds to the classical LASSO.
* The penalty term $\|\bf{w}\|_1$ induces sparsity on the model coefficients $\bf{w}$: all coefficients are forced to zero, except the most "predictive" ones!
* $\Omega(\bf{w})$ is a structure inducing penalty on the model coefficients:

For example \begin{equation}\Omega(\bf{w}) = \frac{1}{2}\|\nabla \bf{w}\|_F^2 := \sum_{j=1}^p|(\nabla_x \bf{w})_j|^2 + |(\nabla_y \bf{w})_j|^2 + |(\nabla_z \bf{w})_j|^2\end{equation} in GraphNet / S-Lasso and for TV-L1, we have
\begin{equation}\Omega(\bf{w}) = \|\bf{w}\|_{TV} := \sum_{j=1}^p\sqrt{|(\nabla_x \bf{w})_j|^2 + |(\nabla_y \bf{w})_j|^2 + |(\nabla_z \bf{w})_j|^2}\end{equation}

* GraphNet is Tikhonov regularization on the spatial gradient of the model coefficients $\bf{w}$. It will produce smooth blobs, but no sharp edges.
* TV semi-norm promotes smooth blobs with sharp edges (i.e has a segmentation property, which is a very good thing). However, the resulting model is much harder to optimize than GraphNet.

**GraphNet / S-Lasso**

```python
from nilearn.decoding import SpaceNetClassifier

```

```python
decoder = SpaceNetClassifier(memory="nilearn_cache", memory_level=2, penalty='graph-net', verbose=1, loss="mse")
decoder.fit(X_train, y_train)
y_pred = decoder.predict(X_test)
accuracy = (y_pred == y_test).mean() * 100.
print("Graph-net classification accuracy : %g%%" % accuracy)
```

    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)

    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  8, ..., 125]), array([ 0,  1,  2,  3,  4,  5,  6,  7,  9, 10, 11, 12, 13, 14, 15, 16]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 0), screening_percentile=30.929715190250057, is_classif=False)

    .......

    ______________________________________________________path_scores - 4.7s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 8, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 1), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 4.9s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([25, 26, 34, 35, 36, 37, 38, 39, 40, 41, 45, 46, 47, 48, 49, 50]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 2), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 8.5s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([42, 43, 44, 51, 52, 53, 54, 55, 56, 57, 58, 63, 64, 65, 66, 67]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 3), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 6.1s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([59, 60, 61, 62, 68, 69, 70, 71, 72, 73, 74, 75, 81, 82, 83, 84]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 4), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 5.2s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 76,  77,  78,  79,  80,  85,  86,  87,  88,  89,  90,  91,  92,
            99, 100, 101]),
    { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 5), screening_percentile=30.929715190250057, is_classif=False)

    .......

    ______________________________________________________path_scores - 9.8s, 0.2min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 93,  94,  95,  96,  97,  98, 102, 103, 104, 105, 106, 107, 108,
           109, 117, 118]),
    { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 6), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 8.5s, 0.1min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<function _graph_net_squared_loss at 0x7f806c7780c8>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 118]), array([110, 111, 112, 113, 114, 115, 116, 119, 120, 121, 122, 123, 124, 125]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 7), screening_percentile=30.929715190250057, is_classif=False)

    ........

    ______________________________________________________path_scores - 4.7s, 0.1min
    Time Elapsed: 53.5496 seconds, 0 minutes.

    [Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:   52.6s finished
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)

    Graph-net classification accuracy : 97.7778%

```python

```

```python
from nilearn.plotting import plot_stat_map, show
coef_img = decoder.coef_img_
plot_stat_map(coef_img, background_img,
              title="graph-net: accuracy %g%%" % accuracy,
              cut_coords=(-34, -16), display_mode="yz")

```

**TV-L1**

```python
decoder = SpaceNetClassifier(memory="nilearn_cache", memory_level=2, penalty='tv-l1', loss="mse", verbose=1)
```

```python
decoder.fit(X_train, y_train)
y_pred = decoder.predict(X_test)
accuracy = (y_pred == y_test).mean() * 100.
print("Graph-net classification accuracy : %g%%" % accuracy)
```

    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)

    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  8, ..., 125]), array([ 0,  1,  2,  3,  4,  5,  6,  7,  9, 10, 11, 12, 13, 14, 15, 16]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 0), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    _____________________________________________________path_scores - 15.1s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 8, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 1), screening_percentile=30.929715190250057, is_classif=False)

    .........

    _____________________________________________________path_scores - 17.7s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([25, 26, 34, 35, 36, 37, 38, 39, 40, 41, 45, 46, 47, 48, 49, 50]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 2), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    _____________________________________________________path_scores - 20.0s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([42, 43, 44, 51, 52, 53, 54, 55, 56, 57, 58, 63, 64, 65, 66, 67]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 3), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    ____________________________________________________path_scores - 110.4s, 1.8min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([59, 60, 61, 62, 68, 69, 70, 71, 72, 73, 74, 75, 81, 82, 83, 84]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 4), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    _____________________________________________________path_scores - 18.9s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 76,  77,  78,  79,  80,  85,  86,  87,  88,  89,  90,  91,  92,
            99, 100, 101]),
    { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 5), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    _____________________________________________________path_scores - 20.4s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 125]), array([ 93,  94,  95,  96,  97,  98, 102, 103, 104, 105, 106, 107, 108,
           109, 117, 118]),
    { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 6), screening_percentile=30.929715190250057, is_classif=False)

    ..........

    _____________________________________________________path_scores - 18.6s, 0.3min
    ________________________________________________________________________________
    [Memory] Calling nilearn.decoding.space_net.path_scores...
    path_scores(<functools.partial object at 0x7f806a1ece10>, array([[ 1.280958, ...,  0.069205],
           ...,
           [-0.421694, ..., -2.454092]], dtype=float32),
    array([-1, ...,  1]), array([[[False, ..., False],
            ...,
            [False, ..., False]],

           ...,
           [[False, ..., False],
            ...,
            [False, ..., False]]], dtype=bool),
    None, [0.5], array([  0, ..., 118]), array([110, 111, 112, 113, 114, 115, 116, 119, 120, 121, 122, 123, 124, 125]), { 'max_iter': 1000, 'tol': 0.0001}, debias=False, verbose=1, n_alphas=10, eps=0.001, key=(0, 7), screening_percentile=30.929715190250057, is_classif=False)

    .........

    _____________________________________________________path_scores - 20.1s, 0.3min
    Time Elapsed: 242.163 seconds, 4 minutes.

    [Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:  4.0min finished
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)
    /home/elvis/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/hashing.py:197: DeprecationWarning: Changing the shape of non-C contiguous array by
    descriptor assignment is deprecated. To maintain
    the Fortran contiguity of a multidimensional Fortran
    array, use 'a.T.view(...).T' instead
      obj_bytes_view = obj.view(self.np.uint8)

    Graph-net classification accuracy : 97.7778%

```python
from nilearn.plotting import plot_stat_map, show
coef_img = decoder.coef_img_
plot_stat_map(coef_img, background_img,
              title="graph-net: accuracy %g%%" % accuracy,
              cut_coords=(-34, -16), display_mode="yz")
```

```python

```