#  Final Project - Glassman Yair, Vanelli Martina

  CS236605: Deep Learning

##  Our project

As anticipated, the aim of our project is to study the performance of the model and the successfulness of the adversarial attacks according to the graph structure.

Our main problem was then to create realistic datasets with different graph structures to be tested with Nettack.

  The original datasets

We decided to start from the datasets analyzed in the article: `citeseer.npz` and `cora.npz`. They both contain information about papers.

The Cora dataset contains machine-learning papers divided into one of 7 classes (Case Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule Learning, Theory) while the CiteSeer data set has 6 class labels (Agents, AI, DB, IR, ML, HCI). For both datasets, they performed stemming and stop word removal beside removing the words with document frequency less than 10.

The final corpus has 2708 documents (nodes), 1433 distinct words in the vocabulary (node features) and 5429 links (graph structure) in the case of Cora; and 3312 documents, 3703 distinct words in the vocabulary and 4732 links in the case of CiteSeer.

 Note:  In the original datasets, the graphs are directed. Since the implemented model takes as input an indirected graph, the authors change the data structure at the beginning of the code by making every link undirected (either if $A$ cites $B$ or $B$ cites $A$ or they both cite each other, they simply register that there is a connection between $A$ and $B$). We decided to use the same approach.

  The Stochastic Block Model and our new datasets

For simplicity, we decided to create our datasets starting from the ones already used in the article. We decided to keep the features and the labels of the old datasets and to change the relationship among the nodes.

We used the random graphs theory to change the graph structure and to generate different graphs.
 In particolar, we chose a specific random graph model: the Stochastic Block Model.
The stochastic block model takes the following parameters:
* The number $n$ of vertices;
* a partition of the vertex set $\{ 1 ,\dots , n \}$ into disjoint subsets $C_1 ,\dots , C_r$, called communities;
* a symmetric $r \times r$  matrix $P$ of edge probabilities.

The edge set is then sampled at random as follows: any two vertices $u \in C_i$ and $v \in C_j$ are connected by an edge independently with probability $P_{ij}$.

This model does not really fit the original dataset since the citations between articles are not independent from each other, but it is very useful for the scope of our project that is to study the importance of communities. The SBM produces indeed random graphs containing communities, that are subsets characterized by being connected with one another with particular edge densities.

We assumed that the labels represent our communities. In this way, we already had some features related to the communities.

  Parameters setting: the $\hat{P}$ matrix

Once we made the previous assumptions, it was straightforward to set the vertex set with the nodes of the largest connected component of the original dataset and to partition it by labels.

The main problem was to find a consistent choice of the matrix $P$ of edge probabilities. For this reason, we computed some statistics over the original dataset in order to find some estimates of these probabilities.

In particular, the $\hat{P}$ matrix, based on the original dataset and given as input of the SBM, is computed in the following way:
* $\hat{P}_{ii}=\frac{|\text{ links between articles with label }i\ |}{|\text{ possible links between articles with label }i\ |}\overset{*}{=}
 \frac{|\text{ links between articles with label }i\ |}{\binom{n_i}{2}}= \frac{2|\text{ links bewteeen articles with label }i\ |}{n_i(n_i-1)}$
*  $\hat{P}_{ij}=\hat{P}_{ji}=\frac{|\text{ links between articles with labels respectively }i \text{ and }j\ |}{|\text{ possible links between articles with labels respectively }i \text{ and } j\ |}=\frac{|\text{ links between articles with labels respectively }i \text{ and }j\ |}{n_i n_j }$

 Note 1:  $n_i$ is the number of node in the $i^{th}$ community, that is the number of articles that are labeled with the $i^{th}$ class.
 Note 2:  Each indirected link is counted once and the possible links are counted as unordered couples of nodes. The implementation of the SBM takes this into account and generates just the upper part of the matrix, that is then copied in the lower part leading to an indirected graph.

 Note*:  We observed that in the original datasets there are few self loops (2% of the total nodes in citeseer). In this case, the total number of possible links should consider also the $n$ possible self loops. Since we did not understand the meaning of the self loops in our datasets, we decided to delete them from the original datasets and then quantify the total number of possible links within a community $i$ as $\binom{n_i}{2}$ (no self loops).

####  Observations
Once we have the $\hat{P}$ matrix, we have all the parameters needed to sample a random graph and to build a dataset that is in some way consistent with the original one.

We recall that, although they have similar degree averages, the 2 datasets have underlying different graph structures due to the assumptions that we made on the link distribution.
 Later on, once we generated the random graphs, we computed some statistics in order to have a look on the behavior of our graphs compared to the original ones. The average degrees converge to the same values, while the max degree and min degree values are very different. This result was expectable: the SBM assumes that all the nodes in the same community have the same degree distribution, while in the original dataset there are very different behaviors among the nodes. Since we do not consider the variance, our model tend to "flatten" the degree distribution around the mean.

We also observed that in general the SBM dataset seems easier to attack but this is hard to say since, even though the features of each node remain the same, it is quite meaningless to compare the results of the attack on the same node in the original dataset and in the randomly generated one. From the structural point of view, there is not any relationship between nodes in the original one and the new ones except for the community belonging. Nodes in the same community are considered anonymus in the SBM while they were not in the original dataset.

On the other side, we also observed that, for all the classes, the average probability of having a link within a community is higher than all the other averages. This observation confirmed that there is a community behavior in both the original datasets and that part of our assumptions is valid.

  The $\tilde{P}$ matrix

At this point, in order to fulfill our analysis and to make meaningful comparisons, we alter the entries of $\hat{P}$ that is to modify the edge probabilities within and between communities. In particular, we aim to either increase or decrease the edge density within communities and then modify the other densities according to it.

Since the effectiveness of the attack changes according to the degree, we decided to alterate the edge probabilities trying to keep the total expected degree constant.

The expected degree of a member $u$ of the community $i$, that is $u \in C_i$, is:
$$
\mathbb{E}[\text{deg}(u)]=\hat{P}_{ii}(n_i-1) + \sum_{j\neq i } \hat{P}_{ij}n_j
$$
and this holds since, in a SBM, the number of neighbors of community $i$ of $u\in C_i$ is distributed as a $\text{Bin}(n_i-1, \hat{P}_{ii})$ while the number of neighbors of community $j$ is distributed as $\text{Bin}(n_j, \hat{P}_{ij})$.

Given $N$, we did the following trick to increase the edge density within communities (we assume $n_i>1$ $\forall i$):
$$
\begin{align}
\mathbb{E}[\text{deg}(u)]&=\hat{P}_{ii}(n_i-1) +N-N +\sum_{j\neq i } \hat{P}_{ij}n_j=\hat{P}_{ii}(n_i-1) + \frac{N (n_i-1)}{(n_i-1)} -N\frac{n_1+\dots+n_{i-1}+n_{i+1}+\dots+n_{C_r}}{\sum_{j\neq i} n_j} +\sum_{j\neq i } \hat{P}_{ij}n_j=\\
&=\left(\hat{P}_{ii}+\frac{N}{(n_i-1)}\right)(n_i-1) +\sum_{j\neq i } \left(\hat{P}_{ij}- \frac{N}{\sum_{j\neq i} n_j}\right)n_j\overset{\text{def}}{=}\tilde{P}_{ii}(n_i-1) +\sum_{j\neq i }\tilde{P}_{ij}n_j
\end{align}
$$
One can think of this trick as a way to change $N$ edges "between communities" to edges "within communities".

Let us note that, with this definition, $\tilde{P}_{ij}$ can be negative. We solve this problem by setting all the negative probabilities to 0. In this last case, the expected degree increases.

Thus, we obtain:
$$
\begin{align}
\tilde{P}_{ii}&\overset{\text{def}}{=}\hat{P}_{ii}+\frac{N}{(n_i-1)}\\
\tilde{P}_{ij}=\tilde{P}_{ji}&\overset{\text{def}}{=}\text{max}\{\hat{P}_{ij}- \frac{N}{\sum_{j\neq i} n_j},0\}
\end{align}
$$
To decrease the edge density within communities, we simply change the sign:
$$
\begin{align}
\tilde{P}_{ii}&\overset{\text{def}}{=}\text{max}\{\hat{P}_{ii}-\frac{N}{(n_i-1)},0\}\\
\tilde{P}_{ij}=\tilde{P}_{ji}&\overset{\text{def}}{=}\hat{P}_{ij}+ \frac{N}{\sum_{j\neq i} n_j}
\end{align}
$$

In the experiments, we set $N=1$ or $N=2$ since the average degree is very low. The boolean parameter $strong$ controls the sign: if it is `True` we increase the edge density within community, while if it is `False` we decrease it. In our experiments, we usually change just one community at a time (usually the one that has the lowest or highest "community behavior" in $\hat{P}$, that is if it has the highest or lowest diagonal entry in respect to the others and the total number of nodes). Anyway, we treat the "community" parameter as a list, in order to try also the case when more than one community behavior is changed (for example, all the communities are weakened or strengthened).

  The SBM generation: the coupling

We now have 2 matrices $\hat{P}$ and $\tilde{P}$ with different edges probabilities: the first one is directly obtained  from the original dataset while the second one has slightly changed community distributions. At this point, the two SBM probability spaces ( SBM($\hat{P}$) and SBM($\tilde{P}$) ) are completely defined and we can sample from them.

In order to make meaningful comparison between them, we decided to sample the random graphs through the following coupling.

Recall that $n$ is the number of nodes and $C_1 ,\dots , C_r$ the community subsets. We will call $[n]=\{1,\dots,n\}$.

For every indirected edge $e=\{v,w\}$, $v\in C_i$ and $ w\in C_j$, let $u_{\{v,w\}}$ be i.i.d. $\mathcal{U}[0,1]$. We consider the graphs $G_1$ and $G_2$ with edge sets to be
$$
\begin{align}
E(G_1)=&\{\{v,w\} \in [n]^{(2)}, v \in C_i, w \in C_j : u_{\{v,w\}}\leq \hat{P}_{ij} \} \\
E(G_2)=&\{\{v,w\} \in [n]^{(2)}, v \in C_i, w \in C_j  : u_{\{v,w\}}\leq \tilde{P}_{ij}\}
\end{align}
$$
In other words, for every unordered couple of nodes  $v \in C_i, w \in C_j$, we generate a uniform random number $u\in [0,1]$. We then put an edge in the first graph if $u \leq \hat{P}_{ij}$ and an edge in the second graph  if  $u \leq \tilde{P}_{ij}$. The main point is that the $u$ is the same for both graphs and in this way we can compare the results for each node.

  The experiments

Once we generated the 2 random graphs, we can study their properties. First, as anticipated, we compute few statistics in order to compare them with the original datasets. Then, we try the GCN and the Nettack on them.

Since we want to study the performance according to the structure, we performed structure attacks. In particular, we focused on direct evasion structure attacks.

In any case, we observed that, since  in   CiteSeer  and  Cora  the average degrees are respectively around 3.5 and 4 while the number of features per node are 3703 and 1433, the Nettack always prefer to perform structure attacks when it is not constrained. This is due to the fact that the authors set the same budget for both the features and structure attacks. Moreover, in the article, the authors set $\Delta=d_{v_0}+2$, where $d_{v_0}$ is the degree of the node $v_0$, that is a low budget for a feature attack (for these datasets) but it is quite high for a structure attacks since it usually allows to overturn the neighborhood. Since we agreed on the fact that effectiveness of the attack depends on the degree of the attacked node, we decided to set  $\Delta=\frac{d_{v_0}}{2}$ for the majority of the experiments. We also tried the effect of strong attack by trying $\Delta=d_{v_0}$. In the experiments, one can tell which budget was used from the term $factor$ and set it as he prefers ($\Delta=\frac{d_{v_0}}{factor}$). As anticipated, the main other parameters are $N$ ($N=1$ or $N=2$), $strong$ (`True` or `False`) and the $communities$ list. The experiments can be repeated for `cora` and `citeseer`.

Once we set all the parameters, we model both the datasets and we try to predict all the nodes in the "changed" communitiesthat have at least degree=2 (in order to have at least one perturbation in the attack). If the prediction is correct, we attack the node and see if the attack is successfull. We record the number of nodes correctly classified nodes and the number of successfully attacked nodes in the 2 datasets.

At the end of the experiments, the results are print on a file with the format `dataset_n#_community#_strong#_factor#`.

  The results

The results are contained in the `results` folder. Although we did not manage to run all the possible cases, we observed some different behaviors.

By running some experiments, we found that a higher “within-community” edges probability usually increases the accuracy of the model. It also descreases the effectiveness of the attack, but just under some assumptions. Results are indeed very affected by the number of nodes in the community, by the starting accuracy and by the budget of the attack.

In our experiments, we tried some parameter settings and we observed some behaviors, but, in order to make general conclusions, many more experiments should be tried.

The results are similar for the 2 datasets.

Below, some example of results:

##  The code

#### Requirements:
* `numpy`
* `scipy`
* `scikit-learn`
* `matplotlib`
* `tensorflow`
* `numba`

```python
from matplotlib import pyplot as plt
%matplotlib inline
from nettack import utils, GCN
from nettack import nettack as ntk
import numpy as np
gpu_id = None # set this to your desired GPU ID if you want to use GPU computations (only for the GCN/surrogate training)
from function_plot import plot

#from importlib import reload
import random
from nettack import sbm
import scipy.sparse as sp
```

### Load network, basic setup

To begin with, we load the data from one of the dataset present in the data folder (`citeseer.npz` and `cora.npz`).

```python
#load the data
#_A_obs: adjacency matrix for links (structure)
#_X_obs: features per node
#_z_obs: label per node
#One can also repeat the experiment with the dataset cora.
dataset='cora'
A_obs, X_obs, z_obs = utils.load_npz('data/'+dataset+'.npz')

#make the graph indirected, that is make the adjacency matrix _A_obs symmetric
A_obs = A_obs + A_obs.T
A_obs[A_obs > 1] = 1

print_matrix=False
if(print_matrix):
    print(_A_obs)

#number of vertices
print("Number of nodes:", A_obs.shape[0])
print("Number of features:",X_obs.shape[1])
```

    Number of nodes: 2708
    Number of features: 1433

### Data preprocessing

```python
#we select the largest connected component
lcc = utils.largest_connected_components(A_obs)
A_obs = A_obs[lcc][:,lcc]
X_obs = X_obs[lcc].astype('float32')
z_obs = z_obs[lcc]

#number of vertices
N = A_obs.shape[0]
print("Number of nodes in the largest connected component:",N)
#avoid self loops
A_obs=A_obs- sp.eye(N)
print("Number of self loops in the dataset:",int(N-np.sum(A_obs[A_obs<0]*-1)))
A_obs[A_obs < 0] = 0

assert np.abs(A_obs - A_obs.T).sum() == 0, "Input graph is not symmetric"
assert A_obs.max() == 1 and len(np.unique(A_obs[A_obs.nonzero()].A1)) == 1, "Graph must be unweighted"
assert A_obs.sum(0).A1.min() > 0, "Graph contains singleton nodes"
```

    Selecting 1 largest connected components
    Number of nodes in the largest connected component: 2485
    Number of self loops in the dataset: 0

```python
#number of labels
_K = z_obs.max()+1
print("Number of labels:", _K)
```

    Number of labels: 7

```python
#degrees and neighbors
neighbors=[]
degrees=np.zeros(N)
for i in range(N):
    neighbors.append(A_obs[i].nonzero()[1])
    degrees[i]=len(neighbors[i])
```

### The $\hat{P}$ matrix

```python
#number of communities/labels
num_communities=_K
```

```python
vertices=np.array(range(N))
#vector that stores the # of nodes for each community
dim_communities=np.zeros(num_communities)
#computes the values and fills the vector
for i in range(num_communities):
    dim_communities[i]=len(vertices[z_obs==i])

#upper triangular matrix of _A_obs,used to compute the ammount of links among members of
#the same community
temp=sp.triu(A_obs,k=0).todense()
#one can put norm=False is he wants to print the total number of links instead of the
#normalized value (without the norm, the result is meaningless for the creation of the random graph)
norm=True
#p_hat=matrix that stores the estimated probabilities of presence of an edge that links members of
#specified communities, that is, for each pair of communities i and j, the entry i,j of the matrix stores
#the average value of the number of edges present in the graphs that link members of the communities i and j
#respectively over the number of all possible edges among the 2 communities (that in the case of 2
#different communities is equal to dim_comm_i*dim_comm_j).
#The average is computed also for the entries i,i that represent an estimation of the probability of an edge that link
#2 members of community i.
p_hat=np.zeros([num_communities,num_communities])
#for each community
for i in range(num_communities):
    #total number of edges that link members of community i
    p_hat[i][i]=temp[z_obs==i][:,z_obs==i].sum()
    #this value has to be normalized over the total number of possible edges that link
    #members of the same community, that is dim_i(dim_i-1)/2= (dim_i)(dim_i-1)/2
    if norm:
        p_hat[i][i]=2*p_hat[i][i]/(dim_communities[i]*(dim_communities[i]-1))
    #for all the other communities
    for j in range(i+1, num_communities):
        #total number of edges that link a member of community i and a member of community j
        p_hat[i][j]=A_obs[z_obs==i][:,z_obs==j].sum()
        #over all the possible links, that is dim_i*dim_j
        if norm:
            p_hat[i][j]=p_hat[i][j]/(dim_communities[i]*dim_communities[j])
        p_hat[j][i]=p_hat[i][j]

print_avg=True
if(print_avg):
    print(p_hat)
```

    [[1.01309612e-02 2.24699680e-04 2.60983036e-04 1.75901495e-04
      4.59091654e-04 1.20530333e-03 7.64993880e-04]
     [2.24699680e-04 9.99817552e-03 1.79809741e-04 1.29976474e-05
      7.13595138e-04 3.76038807e-05 1.64680949e-04]
     [2.60983036e-04 1.79809741e-04 4.21392609e-03 4.83365679e-04
      4.31245334e-04 1.68233340e-04 6.44660132e-04]
     [1.75901495e-04 1.29976474e-05 4.83365679e-04 8.52982647e-03
      2.34261337e-04 1.20848355e-04 6.74970854e-04]
     [4.59091654e-04 7.13595138e-04 4.31245334e-04 2.34261337e-04
      1.79018033e-02 7.13419419e-05 4.34688111e-04]
     [1.20530333e-03 3.76038807e-05 1.68233340e-04 1.20848355e-04
      7.13419419e-05 2.18438050e-02 1.77525297e-03]
     [7.64993880e-04 1.64680949e-04 6.44660132e-04 6.74970854e-04
      4.34688111e-04 1.77525297e-03 8.98365991e-03]]

### The $\tilde{P}$ matrix
In order to study the importance and the effects of edge densities and community structure, we either strengthen or weaken the ties within communities.

```python
#we will call strongest_comm the community that has the highest probability to have an edges
#that link members of the same community

strongest_comm=np.argmax(np.diag(p_hat))
temp=(p_hat-np.diag(np.diag(p_hat))).sum(axis=1)
#print(temp)
strongest_comm_2=np.argmax(np.diag(p_hat)-temp)
#stongest_comm=np.argmax(np.diag(p_hat))
#weakest_comm has the lowest probability
weakest_comm=np.argmin(np.diag(p_hat))
weakest_comm_2=np.argmin(np.diag(p_hat)-temp)
print("The community with the  relative highest within-communities probability is: ")
print(strongest_comm, strongest_comm_2)
print("The community with the  relative highest between-communities probability is:")
print(weakest_comm, weakest_comm_2)
```

    The community with the  relative highest within-communities probability is:
    5 5
    The community with the  relative highest between-communities probability is:
    2 2

```python
#community=range(num_communities)
community=[2]
strong=True
n=1
```

```python
p_hat_2=np.copy(p_hat)

for i in community:
    #n=sum(_z_obs==i)/4
    if strong:
        p_hat_2[i][i]=p_hat_2[i][i]+n/(sum(z_obs==i)-1)
        sum_not_i=sum(z_obs!=i)
        for j in range(num_communities):
            if j!=i:
                p_hat_2[i][j]=p_hat_2[i][j]-n/sum_not_i
                if p_hat_2[i][j]<0:
                    p_hat_2[i][j]=0
                p_hat_2[j][i]=p_hat_2[i][j]
    else:
        p_hat_2[i][i]=p_hat_2[i][i]-n/(sum(z_obs==i)-1)
        if p_hat_2[i][i]<0:
            p_hat_2[i][i]=0
        sum_not_i=sum(z_obs!=i)
        for j in range(num_communities):
            if j!=i:
                p_hat_2[i][j]=p_hat_2[i][j]+n/sum_not_i
                p_hat_2[j][i]=p_hat_2[i][j]

print_changes=True
np.set_printoptions(suppress=True)
if print_changes:
    print("p_hat matrix for the " + dataset+" dataset:")
    print(p_hat)
    print("Modified matrix:")
    print(p_hat_2)

```

    p_hat matrix for the cora dataset:
    [[0.01013096 0.0002247  0.00026098 0.0001759  0.00045909 0.0012053
      0.00076499]
     [0.0002247  0.00999818 0.00017981 0.000013   0.0007136  0.0000376
      0.00016468]
     [0.00026098 0.00017981 0.00421393 0.00048337 0.00043125 0.00016823
      0.00064466]
     [0.0001759  0.000013   0.00048337 0.00852983 0.00023426 0.00012085
      0.00067497]
     [0.00045909 0.0007136  0.00043125 0.00023426 0.0179018  0.00007134
      0.00043469]
     [0.0012053  0.0000376  0.00016823 0.00012085 0.00007134 0.02184381
      0.00177525]
     [0.00076499 0.00016468 0.00064466 0.00067497 0.00043469 0.00177525
      0.00898366]]
    Modified matrix:
    [[0.01013096 0.0002247  0.         0.0001759  0.00045909 0.0012053
      0.00076499]
     [0.0002247  0.00999818 0.         0.000013   0.0007136  0.0000376
      0.00016468]
     [0.         0.         0.00559324 0.         0.         0.
      0.00007616]
     [0.0001759  0.000013   0.         0.00852983 0.00023426 0.00012085
      0.00067497]
     [0.00045909 0.0007136  0.         0.00023426 0.0179018  0.00007134
      0.00043469]
     [0.0012053  0.0000376  0.         0.00012085 0.00007134 0.02184381
      0.00177525]
     [0.00076499 0.00016468 0.00007616 0.00067497 0.00043469 0.00177525
      0.00898366]]

### The Stochastic Block Models generation
Once we have  p_hat_2 , we can generate a second SBM with a comparable average degree where edges more common within communities than between communities (or less common, depending on the boolean parameter  strong )

```python
degrees_hat_2=np.zeros(N)
degrees_hat=np.zeros(N)
n_matrices=1
for i in range(n_matrices):
    #generate a SBM based on the p_hat we just computed and the node labels/community membership
    rg=sbm.SBM(N, _K, z_obs, p_hat, p_hat_2)
    A_obs_hat=sp.csr_matrix(rg.block_matrix)
    A_obs_hat_2=sp.csr_matrix(rg.block_matrix_2)

    #update/compute statistics
    temp_neighbors=[]
    temp_degrees=np.zeros(N)
    for i in range(N):
        temp_neighbors.append(A_obs_hat[i].nonzero()[1])
        temp_degrees[i]=len(temp_neighbors[i])
        degrees_hat[i]+=temp_degrees[i]

    temp_neighbors_2=[]
    temp_degrees_2=np.zeros(N)
    for i in range(N):
        temp_neighbors_2.append(A_obs_hat_2[i].nonzero()[1])
        temp_degrees_2[i]=len(temp_neighbors_2[i])
        degrees_hat_2[i]+=temp_degrees_2[i]

degrees_hat=degrees_hat/n_matrices
degrees_hat_2=degrees_hat_2/n_matrices
```

### A comparison

In this section, we compute some statistics over the original dataset and the random graphs that we just  in order to compare them. Considering how they both were built, it is not surprising that the  One can decide if to print them or not through the boolean variable print_stats.

```python
print_stats=True
if(print_stats):
    print("STATISTICS OVER THE INPUT DATA")
    for i in range(4):
        print("# nodes with",i,"degree:",(degrees==i).sum())
    print("# nodes with degree less than average:",(degrees<sum(degrees)/len(degrees)).sum())
    print("# nodes with degree greater than average:",(degrees>sum(degrees)/len(degrees)).sum())
    print("max degree:",max(degrees))
    #print(neighbors[np.argmax(degrees)])
    #print(np.argmax(degrees))
    print("average degree:",sum(degrees)/len(degrees))
    for i in range(num_communities):
        print("degree average for community",i,":",sum(degrees[vertices[z_obs==i]])/len(degrees[vertices[z_obs==i]]))

print_stats_rg=True
if(print_stats_rg):
    print("\nSTATISTICS OVER THE RANDOM GRAPH")
    print("# nodes with 0 degree:",(degrees_hat<1).sum())
    for i in range(1,4):
        print("# nodes with",i,"degree:", (degrees_hat<i+1).sum()-(degrees_hat<i).sum())

    print("# nodes with degree less than average:",(degrees_hat<sum(degrees_hat)/len(degrees_hat)).sum())
    j=10
    print("# nodes with degree greater than average:",(degrees_hat>sum(degrees_hat)/len(degrees_hat)).sum())
    print("max degree:",(max(degrees_hat)))
    print("average degree:",(sum(degrees_hat)/len(degrees_hat)))
    for i in range(num_communities):
        print("average degree for community",i,":",sum(degrees_hat[vertices[z_obs==i]])/len(degrees_hat[vertices[z_obs==i]]))

print_stats_rg_2=True
if(print_stats_rg_2):
    print("\nSTATISTICS OVER THE MODIFIED RANDOM GRAPH")
    print("# nodes with 0 degree:",(degrees_hat_2<1).sum())
    for i in range(1,4):
        print("# nodes with",i,"degree:", (degrees_hat_2<i+1).sum()-(degrees_hat_2<i).sum())

    print("# nodes with degree less than average:",(degrees_hat_2<sum(degrees_hat_2)/len(degrees_hat_2)).sum())
    j=10
    print("# nodes with degree greater than average:",(degrees_hat_2>sum(degrees_hat_2)/len(degrees_hat_2)).sum())
    print("max degree:",(max(degrees_hat_2)))
    print("average degree:",(sum(degrees_hat_2)/len(degrees_hat_2)))
    for i in range(num_communities):
        print("average degree for community",i,":",sum(degrees_hat_2[vertices[z_obs==i]])/len(degrees_hat_2[vertices[z_obs==i]]))

```

    STATISTICS OVER THE INPUT DATA
    # nodes with 0 degree: 0
    # nodes with 1 degree: 354
    # nodes with 2 degree: 545
    # nodes with 3 degree: 525
    # nodes with degree less than average: 1799
    # nodes with degree greater than average: 686
    max degree: 168.0
    average degree: 4.0796780684104625
    degree average for community 0 : 3.743859649122807
    degree average for community 1 : 4.463054187192118
    degree average for community 2 : 3.721763085399449
    degree average for community 3 : 3.928759894459103
    degree average for community 4 : 4.794392523364486
    degree average for community 5 : 3.9923664122137406
    degree average for community 6 : 4.415697674418604

    STATISTICS OVER THE RANDOM GRAPH
    # nodes with 0 degree: 51
    # nodes with 1 degree: 190
    # nodes with 2 degree: 363
    # nodes with 3 degree: 472
    # nodes with degree less than average: 1532
    # nodes with degree greater than average: 953
    max degree: 12.0
    average degree: 4.015291750503018
    average degree for community 0 : 3.912280701754386
    average degree for community 1 : 4.539408866995074
    average degree for community 2 : 3.5702479338842976
    average degree for community 3 : 3.825857519788918
    average degree for community 4 : 4.700934579439252
    average degree for community 5 : 3.8931297709923665
    average degree for community 6 : 4.25

    STATISTICS OVER THE MODIFIED RANDOM GRAPH
    # nodes with 0 degree: 44
    # nodes with 1 degree: 193
    # nodes with 2 degree: 378
    # nodes with 3 degree: 485
    # nodes with degree less than average: 1100
    # nodes with degree greater than average: 1385
    max degree: 12.0
    average degree: 3.933199195171026
    average degree for community 0 : 3.754385964912281
    average degree for community 1 : 4.426108374384237
    average degree for community 2 : 3.9077134986225897
    average degree for community 3 : 3.496042216358839
    average degree for community 4 : 4.373831775700935
    average degree for community 5 : 3.732824427480916
    average degree for community 6 : 3.8372093023255816

### Data preprocessing

```python
def intersection(lst1, lst2):
    lst3 = [value for value in lst1 if value in lst2]
    return lst3
```

```python
#we select just the largest connected component
lcc_1= utils.largest_connected_components(A_obs_hat)
lcc_2= utils.largest_connected_components(A_obs_hat_2)
lcc=intersection(lcc_1,lcc_2)
_A_obs_hat = A_obs_hat[lcc][:,lcc]
_A_obs_hat_2 = A_obs_hat_2[lcc][:,lcc]
_X_obs_hat = X_obs[lcc].astype('float32')
_z_obs_hat = z_obs[lcc]

#number of vertices
_N = _A_obs_hat.shape[0]
print("Number of nodes in the intersection of the 2 largerst connected components:",_N)
```

    Selecting 1 largest connected components
    Selecting 1 largest connected components
    Number of nodes in the intersection of the 2 largerst connected components: 2418

```python
seed = 15
unlabeled_share = 0.8
val_share = 0.1
train_share = 1 - unlabeled_share - val_share
np.random.seed(seed)
split_train, split_val, split_unlabeled = utils.train_val_test_split_tabular(np.arange(_N),
                                                                       train_size=train_share,
                                                                       val_size=val_share,
                                                                       test_size=unlabeled_share,
                                                                       stratify=_z_obs_hat)
```

### Choose the node to attack

```python
set_deg=5

#degrees and neighbors
neighbors_hat=[]
_degrees_hat=np.zeros(_N)
for i in range(_N):
    neighbors_hat.append(_A_obs_hat[i].nonzero()[1])
    _degrees_hat[i]=len(neighbors_hat[i])

#degrees and neighbors
neighbors_hat_2=[]
_degrees_hat_2=np.zeros(_N)
for i in range(_N):
    neighbors_hat_2.append(_A_obs_hat_2[i].nonzero()[1])
    _degrees_hat_2[i]=len(neighbors_hat_2[i])

_vertices=np.array(range(_N))
#u = random.choice(vertices[degrees_hat==set_deg]) # node to attack
indices= _z_obs_hat==community[0]
for c in community:
    indices= [ indices[i] or _z_obs_hat[i]==c for i in range(len(_z_obs_hat))]

u = random.choice(_vertices[np.multiply(indices ,_degrees_hat>set_deg)])
assert u in split_unlabeled
deg=degrees_hat[u]
print("Neighbors of", u, "in the non-modified SBM:",neighbors_hat[u])
print("Labels of the neighbors:",_z_obs_hat[neighbors_hat[u]])
print("Neighbors of", u, "in the modified SBM:",neighbors_hat_2[u])
print("Labels of the neighbors:",_z_obs_hat[neighbors_hat_2[u]])

train_1=False
train_2=False
```

    Neighbors of 1688 in the non-modified SBM: [  76  707  871 1383 1825 2202 2300]
    Labels of the neighbors: [2 2 2 6 1 2 2]
    Neighbors of 1688 in the modified SBM: [  76  659  707  871 2202 2300]
    Labels of the neighbors: [2 2 2 2 2 2]

### Data preprocessing e general nettack set up

```python
_Z_obs_hat = np.eye(_K)[_z_obs_hat]
sizes = [16, _K]
```

```python
direct_attack = True
n_influencers = 1 if direct_attack else 5
perturb_features = False
perturb_structure = True
```

```python
verbose = False
retrain_iters=5
```

### Attack of u on SBM($\hat{P}$)

```python
n_perturbations_1=int(_degrees_hat[u]/2)
_An_hat = utils.preprocess_graph(_A_obs_hat)
```

```python
train_1=True

#Train surrogate model (i.e. GCN without nonlinear activation)
surrogate_model_1 = GCN.GCN(sizes, _An_hat, _X_obs_hat, with_relu=False, name="surrogate", gpu_id=gpu_id)
surrogate_model_1.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
W1_1 =surrogate_model_1.W1.eval(session=surrogate_model_1.session)
W2_1 =surrogate_model_1.W2.eval(session=surrogate_model_1.session)

#Setup Nettack
nettack_1 = ntk.Nettack(_A_obs_hat, _X_obs_hat, _z_obs_hat, W1_1, W2_1, u, verbose)

#Poison the data
nettack_1.reset()
nettack_1.attack_surrogate(n_perturbations_1, perturb_structure=perturb_structure, perturb_features=perturb_features, direct=direct_attack, n_influencers=n_influencers)

#Print perturbations
if verbose:
    print("Structure perturbations:",nettack_1.structure_perturbations)
    print("Features perturbations:", nettack_1.feature_perturbations)

#Train GCN without perturbations
if verbose:
    print("Train GCN without perturbations")

classification_margins_clean_1 = []
class_distrs_clean_1 = []
gcn_before_1 = GCN.GCN(sizes, _An_hat, _X_obs_hat, "gcn_orig", gpu_id=gpu_id)
for _ in range(retrain_iters):
    if verbose:
        print("... {}/{} ".format(_+1, retrain_iters))
    gcn_before_1.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
    probs_before_attack_1 = gcn_before_1.predictions.eval(session=gcn_before_1.session,feed_dict={gcn_before_1.node_ids: [nettack_1.u]})[0]
    class_distrs_clean_1.append(probs_before_attack_1)
    #best_second_class_before_1 = (probs_before_attack_1 - 1000*_Z_obs_hat[nettack_1.u]).argmax()
    #margin_before_1 = probs_before_attack_1[_z_obs_hat[nettack_1.u]] - probs_before_attack_1[best_second_class_before_1]
    #classification_margins_clean_1.append(margin_before_1)

class_distrs_clean_1 = np.array(class_distrs_clean_1)

#Train GCN with perturbations
if verbose:
    print("Train GCN with perturbations")
classification_margins_corrupted_1 = []
class_distrs_retrain_1 = []
gcn_retrain_1 = GCN.GCN(sizes, nettack_1.adj_preprocessed, nettack_1.X_obs.tocsr(), "gcn_retrain", gpu_id=gpu_id)
for _ in range(retrain_iters):
    if verbose:
        print("... {}/{} ".format(_+1, retrain_iters))
    gcn_retrain_1.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
    probs_after_attack_1 = gcn_retrain_1.predictions.eval(session=gcn_retrain_1.session,feed_dict={gcn_retrain_1.node_ids: [nettack_1.u]})[0]
    #best_second_class_after_1 = (probs_after_attack_1 - 1000*_Z_obs_hat[nettack_1.u]).argmax()
    #margin_after_1 = probs_after_attack_1[_z_obs_hat[nettack_1.u]] - probs_after_attack_1[best_second_class_after_1]
    class_distrs_retrain_1.append(probs_after_attack_1)
    #classification_margins_corrupted_1.append(margin_after_1)
class_distrs_retrain_1 = np.array(class_distrs_retrain_1)
```

    C:\Users\Martina\Anaconda3\envs\project\lib\site-packages\tensorflow\python\client\session.py:1702: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).
      warnings.warn('An interactive session is already active. This can '
    C:\Users\Martina\Anaconda3\envs\project\lib\site-packages\sklearn\metrics\classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

### Attack of u on SBM($\tilde{P}$)

```python
n_perturbations_2=int(_degrees_hat_2[u]/2)
_An_hat_2 = utils.preprocess_graph(_A_obs_hat_2)
```

```python
train_2=True
#Train surrogate model (i.e. GCN without nonlinear activation)
surrogate_model_2 = GCN.GCN(sizes, _An_hat_2, _X_obs_hat, with_relu=False, name="surrogate", gpu_id=gpu_id)
surrogate_model_2.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
W1_2 =surrogate_model_2.W1.eval(session=surrogate_model_2.session)
W2_2 =surrogate_model_2.W2.eval(session=surrogate_model_2.session)

#Setup Nettack
nettack_2 = ntk.Nettack(_A_obs_hat_2, _X_obs_hat, _z_obs_hat, W1_2, W2_2, u, verbose)

#Poison the data
nettack_2.reset()
nettack_2.attack_surrogate(n_perturbations_2, perturb_structure=perturb_structure, perturb_features=perturb_features, direct=direct_attack, n_influencers=n_influencers)

#Print perturbations
if verbose:
    print("Structure perturbations:",nettack_2.structure_perturbations)
    print("Features perturbations:", nettack_2.feature_perturbations)

#Train GCN without perturbations
if verbose:
    print("Train GCN without perturbations")

classification_margins_clean_2 = []
class_distrs_clean_2 = []
gcn_before_2 = GCN.GCN(sizes, _An_hat_2, _X_obs_hat, "gcn_orig", gpu_id=gpu_id)
for _ in range(retrain_iters):
    if verbose:
        print("... {}/{} ".format(_+1, retrain_iters))
    gcn_before_2.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
    probs_before_attack_2 = gcn_before_2.predictions.eval(session=gcn_before_2.session,feed_dict={gcn_before_2.node_ids: [nettack_2.u]})[0]
    class_distrs_clean_2.append(probs_before_attack_2)
    #best_second_class_before_2 = (probs_before_attack_2 - 1000*_Z_obs_hat[nettack_2.u]).argmax()
    #margin_before_2 = probs_before_attack_2[_z_obs_hat[nettack_1.u]] - probs_before_attack_2[best_second_class_before_2]
    #classification_margins_clean_2.append(margin_before_2)

class_distrs_clean_2 = np.array(class_distrs_clean_2)

#Train GCN with perturbations
if verbose:
    print("Train GCN with perturbations")
classification_margins_corrupted_2 = []
class_distrs_retrain_2 = []
gcn_retrain_2 = GCN.GCN(sizes, nettack_2.adj_preprocessed, nettack_2.X_obs.tocsr(), "gcn_retrain", gpu_id=gpu_id)
for _ in range(retrain_iters):
    if verbose:
        print("... {}/{} ".format(_+1, retrain_iters))
    gcn_retrain_2.train(split_train, split_val, _Z_obs_hat, print_info=verbose)
    probs_after_attack_2 = gcn_retrain_2.predictions.eval(session=gcn_retrain_2.session,feed_dict={gcn_retrain_2.node_ids: [nettack_2.u]})[0]
    #best_second_class_after_2 = (probs_after_attack_2 - 1000*_Z_obs_hat[nettack_2.u]).argmax()
    #margin_after_2 = probs_after_attack_2[_z_obs_hat[nettack_2.u]] - probs_after_attack_2[best_second_class_after_2]
    class_distrs_retrain_2.append(probs_after_attack_2)
    #classification_margins_corrupted_2.append(margin_after_2)
class_distrs_retrain_2 = np.array(class_distrs_retrain_2)
```

### Visualize and compare the results

```python
plot(class_distrs_clean_1, class_distrs_retrain_1, retrain_iters, nettack_1.u, nettack_1.label_u, _K, n_perturbations_1)
plot(class_distrs_clean_2, class_distrs_retrain_2, retrain_iters, nettack_2.u, nettack_2.label_u, _K, n_perturbations_2)
```

```python
#u_hat_2=random.choice(vertices[np.multiply(degrees_hat_2==deg,_z_obs==_z_obs[u])])
#n_perturbations=int(degrees_hat_2[u]/2)
#[clean, attacked]=test_nettack(_A_obs_hat_2, _X_obs_hat, _z_obs_hat, u,  n_perturbations, perturb_structure, perturb_features, direct_attack, n_influencers,
#             split_train, split_val, split_unlabeled, verbose=True)
```

### Experiments

```python
indices= _z_obs_hat==community[0]
attacked_nodes = _vertices[[indices[i]  and _degrees_hat[i]>1 and _degrees_hat_2[i]>1 for i in range(_N)]]
tot=len(attacked_nodes)

```

```python
 #DATA PREPROCESSING
#_An_1=_An_hat
#_An_2=_An_hat_2
_An_1 = utils.preprocess_graph(_A_obs_hat)
_An_2 = utils.preprocess_graph(_A_obs_hat_2)
```

```python
#surrogate model of SBM 1 - needed for nettack

if not train_1:
    surrogate_model_1 = GCN.GCN(sizes, _An_1, _X_obs_hat, with_relu=False, name="surrogate", gpu_id=gpu_id)
    surrogate_model_1.train(split_train, split_val, _Z_obs_hat, print_info=False)
    W1_1 =surrogate_model_1.W1.eval(session=surrogate_model_1.session)
    W2_1 =surrogate_model_1.W2.eval(session=surrogate_model_1.session)

    #Train GCN without perturbations
    gcn_before_1 = GCN.GCN(sizes, _An_1, _X_obs_hat, "gcn_orig", gpu_id=gpu_id)
    gcn_before_1.train(split_train, split_val, _Z_obs_hat, print_info=False)

if not train_2:
    #surrogate model of SBM 2 - needed for nettack
    surrogate_model_2 = GCN.GCN(sizes, _An_2, _X_obs_hat, with_relu=False, name="surrogate", gpu_id=gpu_id)
    surrogate_model_2.train(split_train, split_val, _Z_obs_hat, print_info=False)
    W1_2 =surrogate_model_2.W1.eval(session=surrogate_model_2.session)
    W2_2 =surrogate_model_2.W2.eval(session=surrogate_model_2.session)

    #Train GCN without perturbations
    gcn_before_2 = GCN.GCN(sizes, _An_2, _X_obs_hat, "gcn_orig", gpu_id=gpu_id)
    gcn_before_2.train(split_train, split_val, _Z_obs_hat, print_info=False)

```

```python
stamp=False
correct=0
mod_correct=0
mod_mis=0
mis=0
succ_att=0
mod_succ_att=0
i=0
limit=500

if strong:
    print("More edges within communities\n")
else:
    print("More edges between communities\n")

for u in attacked_nodes:
    if i ==limit:
        break
    if stamp:
        print("Iteration:",i,"/", tot)
    i+=1

    ###### AVERAGE COMMUNITIES ########
    n_perturbations=int(_degrees_hat[u]/2)
    if stamp:
        print("Testing nettack on the clean random graph (", n_perturbations, "perturbations)")
        print("Attacked node:",u,", class:",_z_obs_hat[u],", neighbors:", neighbors_hat[u], "with class", _z_obs_hat[neighbors_hat[u]])

    clean_1 = gcn_before_1.predictions.eval(session=gcn_before_1.session,feed_dict={gcn_before_1.node_ids: [u]})[0]
       #[clean_1, attacked_1]=test_nettack(_A_obs_hat, _X_obs_hat, _z_obs_hat, u,  n_perturbations, perturb_structure, perturb_features, direct_attack, n_influencers,
    #         split_train, split_val, split_unlabeled, verbose=False, plot=False)

    #if(np.argmax(np.sum(clean_1,axis=1))==_z_obs_hat[u]):
    if(np.argmax(clean_1)==_z_obs_hat[u]):
        correct+=1
        print("Iteration:",i,"/", tot)
        #Set up nettack
        #NOTE: W1_1 and W_2 are the same for all the attacks and are computed in the previous step
        nettack_1 = ntk.Nettack(_A_obs_hat, _X_obs_hat, _z_obs_hat, W1_1, W2_1, u,verbose=False)
        #Poison the data
        nettack_1.reset()
        nettack_1.attack_surrogate(n_perturbations, perturb_structure=perturb_structure,
                               perturb_features=perturb_features, direct=direct_attack, n_influencers=n_influencers)
        gcn_retrain_1 = GCN.GCN(sizes, nettack_1.adj_preprocessed, nettack_1.X_obs.tocsr(),
                                "gcn_retrain", gpu_id=gpu_id)
        gcn_retrain_1.train(split_train, split_val, _Z_obs_hat, print_info=False)
        attacked_1 = gcn_retrain_1.predictions.eval(session=gcn_retrain_1.session,
                                                    feed_dict={gcn_retrain_1.node_ids: [nettack_1.u]})[0]
        #if(np.argmax(np.sum(attacked_1,axis=1))!=_z_obs_hat[u]):
        if(np.argmax(attacked_1)!=_z_obs_hat[u]):
            succ_att+=1
    else:
        mis+=1

    if stamp:
        if(np.argmax(clean_1)==_z_obs_hat[u]):
            print("Correct classified")
            if(np.argmax(attacked_1)!=_z_obs_hat[u]):
                print("Successful attack\n")
            else:
                print("Unsuccessful attack\n")
        else:
            print("Misclassified")

    ######## MODIFIED COMMUNITIES ##########
    n_perturbations=int(_degrees_hat_2[u]/2)
    if stamp:
        print("Testing nettack on the modified random graph (", n_perturbations, "perturbations)")
        print("Attacked node:",u, ", class:",_z_obs_hat[u],", neighbors:", neighbors_hat_2[u], "with class",_z_obs_hat[neighbors_hat_2[u]])

    clean_2 = gcn_before_2.predictions.eval(session=gcn_before_2.session,feed_dict={gcn_before_2.node_ids: [u]})[0]

    #if(np.argmax(np.sum(clean_2,axis=1))==_z_obs_hat[u]):
    if(np.argmax(clean_2)==_z_obs_hat[u]):
        mod_correct+=1

        #Set up nettack
        #NOTE: W1_1 and W_2 are the same for all the attacks and are computed in the previous step
        nettack_2 = ntk.Nettack(_A_obs_hat_2, _X_obs_hat, _z_obs_hat, W1_2, W2_2, u, verbose=False)
        #Poison the data
        nettack_2.reset()
        nettack_2.attack_surrogate(n_perturbations, perturb_structure=perturb_structure,
                                   perturb_features=perturb_features, direct=direct_attack,
                                   n_influencers=n_influencers)

        gcn_retrain_2 = GCN.GCN(sizes, nettack_2.adj_preprocessed, nettack_2.X_obs.tocsr(), "gcn_retrain",
                                gpu_id=gpu_id)
        gcn_retrain_2.train(split_train, split_val, _Z_obs_hat, print_info=False)
        attacked_2 = gcn_retrain_2.predictions.eval(session=gcn_retrain_2.session,
                                                feed_dict={gcn_retrain_2.node_ids: [nettack_2.u]})[0]
        #if(np.argmax(np.sum(attacked_2,axis=1))!=_z_obs_hat[u]):
        if(np.argmax(attacked_2)!=_z_obs_hat[u]):
            mod_succ_att+=1
    else:
        mod_mis+=1

    if stamp:
        if(np.argmax(clean_2)==_z_obs_hat[u]):
            print("Correct classified")
            if(np.argmax(attacked_2)!=_z_obs_hat[u]):
                print("Successful attack\n")
            else:
                print("Unsuccessful attack\n")
        else:
            print("Misclassified")

print("# of correct classified nodes:",correct)
print("# of correct classified nodes in the second graph", mod_correct)
print("# of successful attacks ", succ_att)
print("# of successful attacks in the second graph", mod_succ_att)
```

```python
print("#### CASE N =", n, ", n_perturbations=degree/2, strong=", strong, " ####" )
print("# of tested nodes:", tot)
print("# of correct classified nodes in the first dataset:",correct, ", accuracy:", correct/tot)
print("# of correct classified nodes in the second dataset:", mod_correct, ", accuracy:", mod_correct/tot)
print("# of successful attacks in the first dataset:", succ_att, ", accuracy:", succ_att/correct)
print("# of successful attacks in the second dataset:", mod_succ_att, ", accuracy:", mod_succ_att/mod_correct)
```

```python
name="Experiment"+str(3)
f=open(name,"w+")
f.write("#### CASE N="+ str(n)+ ", n_perturbations=degree/2, strong=" + str(strong)+ " on " +dataset+" ####\n" )
f.write("# of tested nodes:"+str(tot)+ "\n")
f.write("# of correct classified nodes in the first dataset:"+str(correct)+ ", accuracy:"+str(correct/tot)+ "\n")
f.write("# of correct classified nodes in the second dataset:"+str(mod_correct)+ ", accuracy:"+str(mod_correct/tot)+ "\n")
f.write("# of successful attacks in the first dataset:" +str(succ_att)+ ", accuracy:"+ str(succ_att/correct)+ "\n")
f.write("# of successful attacks in the second dataset:"+str(mod_succ_att)+ ", accuracy:" + str(mod_succ_att/mod_correct)+ "\n")
f.close()
```

```python

```


```python
import sympy
from phasor.utilities.ipynb.displays import *
from phasor.utilities.ipynb.ipy_sympy import *
import scipy.linalg

import numpy.testing as np_test
import declarative

from test_SVD import SVD_gen_check, gen_rand_unitary
from phasor.system import DAG_algorithm
from phasor.system import SRE_matrix_algorithms
from phasor.system import scisparse_algorithm

import timeit

asavefig.org_subfolder = 'plots'
```

    Populating the interactive namespace from numpy and matplotlib
    Sympy version:  1.0

```python
from functools import reduce
def SVD_genmatrix(
    N = 10,
    length = 1,
):
    U = gen_rand_unitary(N = N, length = length)
    V = gen_rand_unitary(N = N, length = length)

    seq = dict()
    req = dict()
    edge_map = dict()
    S_diags = []
    for idx in range(N):
        s_diag = 10**(-5 + 10 * np.random.random(length))
        edge_map[idx, idx] = s_diag
        S_diags.append(s_diag)
        seq[idx] = set([idx])
        req[idx] = set([idx])
    S = seq, req, edge_map
    condition = reduce(np.maximum, S_diags) / reduce(np.minimum, S_diags)

    M = SRE_matrix_algorithms.matrix_mult_sre(
        SRE_matrix_algorithms.matrix_mult_sre(U, S), V
    )

    SRE_matrix_algorithms.check_sre(M)
    sparsity = SRE_matrix_algorithms.SRE_count_sparsity(M)
    edge_map = M[2]

    m = scisparse.dok_matrix((N, N), dtype = complex)
    for (k1, k2), e in edge_map.items():
        m[k1, k2] = e
    mtrx = scisparse.csc_matrix(m)

    b = declarative.Bunch(
        edge_map = edge_map,
        csc = mtrx,
    )
    b.update(sparsity)
    return b
```

```python
mat = SVD_genmatrix(N = 300)
import scipy.sparse as scisparse
```

```python

```

    <300x300 sparse matrix of type '<class 'numpy.complex128'>'
    	with 974 stored elements in Compressed Sparse Column format>

```python
#imshow(abs(mtrx.toarray()) != 0)
import scipy.io
```

```python
scipy.io.savemat(
    'mat_tests.mat',
     mdict=dict(
         test_100 = SVD_genmatrix(N = 100).csc,
         test_300 = SVD_genmatrix(N = 300).csc,
         test_1000 = SVD_genmatrix(N = 1000).csc,
     ),
)
```

```python

```


# Covector
> ### $ [V^{*},\; \mathbb{R},\; e^m,\; 4]$
>> ### $
[V, \mathbb{R}, e_m, 4] \rightarrow [V^{*},\; \mathbb{R},\; e^m,\; 4] \\
\qquad \quad \nwarrow \qquad  \swarrow \\
\qquad [V^{**},\; \mathbb{R},\; e,\; 4]
$
> ### vector
>> ### $[V, \mathbb{R}, e_m, 4] \iff A^{m}e_m$
> ### covector
>> ### $[V{*}, \mathbb{R}, e^m, 4] \iff A_{m}e^{m}$
> ### $ <e^i,e_j> = \delta^{i}_{j}$
> ### $ <B_{\mu}e^{\mu},A^{\nu}e_{\nu}> $
>> ### $\because \text{ linear map} \\
= B_{\mu}A^{\nu}<e^{\mu},e_{\nu}>\;\;
= \;B_{\mu}A^{\nu}\delta^{\mu}_{\nu}\;
=\; B_{\mu}A^{\mu}\\
= \sum_{i=1}^{\mu}B_iA^i = B \cdot A$

# covector
> ## row vector(행벡터)
>> ## $ \begin{bmatrix} 2 & 1\end{bmatrix}$
>> column vector(열벡터)를 입력으로 받아 스칼라를 출력해준다.(내적)
>>> ## $\therefore$ row vector is a function
>>>> ### $ f: \vec{V} \mapsto \mathbb{R} $
>> row vector는 선형 연산자 이다.
>>> operator란 입력값으로 받은 집합에 원소로 결과를 출력해준다.
>>> ## $ nf(\vec{v} + \vec{w}) = f(n\vec{v}) + f(n\vec{w}) \\
\begin{cases}
f(\vec{v} + \vec{w}) = f(\vec{v}) + f(\vec{w})\\
nf(\vec{v}) = f(n\vec{v})
\end{cases}$

# subspace
> ### 전체벡터의 부분으로만 만들었는데 그 부분만으로도 벡터인 공간되는 부분벡터들의 집합
>> ### row space
>>> #### row vectors들의 선형 결합(덧셈 스칼라곱)으로 구성되는 space
>> ### column space
>>> #### column vectors들의 선형 결합(덧셈 스칼라곱)으로 구성되는 space
>> ### null space
>>> #### row space와 직교인 벡터들의 부분 space
>>> ### 즉 row vector(covector)와 내적하면 0 이 되는 벡터들

```python
import matplotlib.pyplot as plt
%matplotlib widget
import numpy as np
import sympy as sm
import sympy.vector
import sympy.physics.vector
N = sm.vector.CoordSys3D('N')
#N = sm.physics.vector.ReferenceFrame('N')
# covector = row vector
# [2,1]
v = 2*N.i + 1*N.j

x,y = sm.symbols('x y')
(A*sm.Matrix([x,y]))[0]
(v.to_matrix(N)).T

```

$\displaystyle \left[\begin{matrix}2 & 1 & 0\end{matrix}\right]$

```python
fig = plt.figure()
ax = fig.add_subplot()
ax.set_aspect('equal')
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
ax.grid()
x = np.linspace(-3,3,100)
x,y = np.meshgrid(x,x)
contour = ax.contour(x,y,2*x + y,[0,1,2,3,4,5],linestyles='dotted')
ax.clabel(contour,colors='r')
ax.quiver(0,0,2,1,scale_units='xy',scale=1,color='r')
ax.quiver(0,0,1,2,scale_units='xy',scale=1,color='k')
ax.quiver(0,0,1.5,(4-1.5*2),scale_units='xy',scale=1,color='k')
ax.quiver(0,0,1.25,(4-1.25*2),scale_units='xy',scale=1,color='k')
```

    <matplotlib.quiver.Quiver at 0x7f14c585dee0>

        Figure

```python
# plane equation
# direction vector = partial differencial

import matplotlib.pyplot as plt
import numpy as np
%matplotlib widget
fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
ax.set_xlim(-3,3)
ax.set_ylim(-3,3)
ax.set_zlim(-3,3)
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
xi = np.linspace(-1,3,100)
x,y = np.meshgrid(xi,xi)

ax.quiver(0,0,0,2,1,0,color='r')
ax.contour(x, y, 2*(x-1)+(y-1), np.linspace(-3,3,100),alpha=0.5)
ax.plot_surface(x, y, 2*x+y,alpha=0.5)
```

    <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7f14c34900a0>

        Figure

```python
# addtion of row vector
import matplotlib.pyplot as plt
import numpy as np
%matplotlib widget
fig = plt.figure()
ax = fig.add_subplot()
ax.set_xlim(-3,3)
ax.set_ylim(-3,3)
ax.set_aspect('equal')
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
x = np.linspace(-3,3,100)
x,y = np.meshgrid(x,x)
ax.contour(x,y,2*x,[-3,-2,-1,0,1,2,3],
                     colors='k',
                     linestyles='--',
                     alpha=0.2)

ax.contour(x,y,y,[-3,-2,-1,0,1,2,3],
           colors='k',
           linestyles='--',
           alpha=0.2)
ax.clabel(ax.contour(x,y,2*x+y,[0,1,2,3,4,5,6],colors='g'),colors='r')
ax.quiver(0,0,2,1,scale_units='xy',scale=1)
ax.quiver(0,0,2,0,scale_units='xy',scale=1)
ax.quiver(0,0,0,1,scale_units='xy',scale=1)
```

    <matplotlib.quiver.Quiver at 0x7f14c0d26130>

        Figure

## [$V^{*}$ : row space = dual space ](https://angeloyeo.github.io/2020/11/17/four_fundamental_subspaces.html)
> ## column vector를 인수로 받아 내적을 시키는 row vector는 선형범함수(functional)이며 그 결과가 스칼라를 출력한다.
>> ## 함수 이지만 벡터의 정의에 일치되게 행동함으로(덧셈 스칼라곱) 벡터라고 할 수 있다.
> ## sets from addtions and scalar multiplies of row vectors
>> ## row vector is a function
>>> ## $\therefore$ row vector functions are is vector

> ## $ V^{*} = \Big\{
f: V \mapsto \mathbb{R} | f(c\vec{a} + \vec{b}) \quad {}^{\forall} \vec{a},\vec{b} \in V \Big\}$

```python

```


# Assignment 4
*Name (ID)*: Vinh Nguyen (1029531)
*Workshop*: Friday 04:15 PM

---

## References
[Factoextra R Package: Easy Multivariate Data Analyses and Elegant Visualization](http://www.sthda.com/english/wiki/factoextra-r-package-easy-multivariate-data-analyses-and-elegant-visualization)
[Articles - Principal Component Methods in R: Practical Guide](http://www.sthda.com/english/articles/31-principal-component-methods-in-r-practical-guide/118-principal-component-analysis-in-r-prcomp-vs-princomp/)
[Plotting PCA (Principal Component Analysis)](https://cran.r-project.org/web/packages/ggfortify/vignettes/plot_pca.html)
[ggfortify : Extension to ggplot2 to handle some popular packages - R software and data visualization](http://www.sthda.com/english/wiki/ggfortify-extension-to-ggplot2-to-handle-some-popular-packages-r-software-and-data-visualization)

## Setup

```
import sys
from pathlib import Path

# Notebook Library
url = "https://github.com/Data-Science-and-Data-Analytics-Courses/Notebook-Library"
repo = Path("/nblib")
!git clone "{url}" "{repo}"
if repo.parent.as_posix() not in sys.path:
  sys.path.append(repo.parent.as_posix())
%run "{repo}/.Importable.ipynb"

from nblib.imports.Basic import *
from nblib import Git

# Remote
URL = "https://github.com/Data-Science-and-Data-Analytics-Courses/UniMelb-A-First-Course-In-Statistical-Learning-MAST90104_2019_SM2"
REPO = Git.clone(URL, dest="/content")
if REPO.as_posix() not in sys.path:
  sys.path.append(REPO.as_posix())

# Working directory
os.chdir(REPO/"assignment4")

from nblib.imports.RPy import *
from setup.Setup import *

clear_output()
```

```r
%%R

install.packages(c("ggfortify", "factoextra")) # takes time

library(MASS)
library(nnet)
library(cluster)
library(factoextra)
library(ggplot2)
library(ggfortify)

CONFLEV = 0.95 # confidence interval
```

## Problem 1
1. College juniors (undergraduates) at US universities were asked if they were “unlikely”, “somewhat
likely”, or “very likely” to apply to graduate school. Data on whether the parents have graduate
education (pared), whether the undergraduate institution is public (public), and current GPA
(gpa), were also collected.

```r
%%R

load("ologit.Rdata")
print(head(dat))

print(ftable(xtabs(~ public + apply + pared, data = dat)))
```

                apply pared public  gpa
    1     very likely     0      0 3.26
    2 somewhat likely     1      0 3.21
    3        unlikely     1      1 3.94
    4 somewhat likely     0      0 2.81
    5 somewhat likely     0      0 2.53
    6        unlikely     0      1 2.59
                           pared   0   1
    public apply
    0      unlikely              175  14
           somewhat likely        98  26
           very likely            20  10
    1      unlikely               25   6
           somewhat likely        12   4
           very likely             7   3

(a) Fit a multinomial model to predict the probabilities for graduate application by category,
“unlikely”, “somewhat likely”, or “very likely”.

```r
%%R

# Categorical model with AIC step-wise selection
model.full <- multinom(apply ~ pared * public * gpa, data = dat) # full interactive
model.cat <- stepAIC(model.full, direction = "both", trace = FALSE) # simplified
print(summary(model.cat))

# Compare simplified to full interactive
av <- anova(model.cat, model.full)
print(av)
pval <- av[2, "Pr(Chi)"]
sprintf("Reject Null (similar models) at %s confidence level? %s", CONFLEV, pval < 1 - CONFLEV)
```

    # weights:  27 (16 variable)
    initial  value 439.444915
    iter  10 value 356.729468
    iter  20 value 350.788487
    iter  30 value 350.601860
    iter  40 value 350.590868
    iter  50 value 350.583458
    iter  60 value 350.582596
    final  value 350.582408
    converged
    # weights:  24 (14 variable)
    initial  value 439.444915
    iter  10 value 352.518781
    iter  20 value 351.033095
    iter  30 value 351.031368
    iter  30 value 351.031368
    final  value 351.031368
    converged
    # weights:  24 (14 variable)
    initial  value 439.444915
    iter  10 value 352.518781
    iter  20 value 351.033095
    iter  30 value 351.031368
    iter  30 value 351.031368
    final  value 351.031368
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 353.710062
    iter  20 value 352.220325
    final  value 352.218611
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 355.635036
    iter  20 value 354.982611
    final  value 354.982547
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 352.762475
    iter  20 value 352.000062
    final  value 351.999744
    converged
    # weights:  27 (16 variable)
    initial  value 439.444915
    iter  10 value 356.729468
    iter  20 value 350.788487
    iter  30 value 350.601860
    iter  40 value 350.590868
    iter  50 value 350.583458
    iter  60 value 350.582596
    final  value 350.582408
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 352.762475
    iter  20 value 352.000062
    final  value 351.999744
    converged
    # weights:  18 (10 variable)
    initial  value 439.444915
    iter  10 value 354.101438
    iter  20 value 353.441487
    final  value 353.441474
    converged
    # weights:  18 (10 variable)
    initial  value 439.444915
    iter  10 value 355.913353
    final  value 355.865699
    converged
    # weights:  24 (14 variable)
    initial  value 439.444915
    iter  10 value 352.518781
    iter  20 value 351.033095
    iter  30 value 351.031368
    iter  30 value 351.031368
    final  value 351.031368
    converged
    # weights:  18 (10 variable)
    initial  value 439.444915
    iter  10 value 354.101438
    iter  20 value 353.441487
    final  value 353.441474
    converged
    # weights:  15 (8 variable)
    initial  value 439.444915
    iter  10 value 354.812899
    final  value 354.741801
    converged
    # weights:  15 (8 variable)
    initial  value 439.444915
    iter  10 value 357.012275
    final  value 356.996982
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 352.762475
    iter  20 value 352.000062
    final  value 351.999744
    converged
    # weights:  21 (12 variable)
    initial  value 439.444915
    iter  10 value 353.710062
    iter  20 value 352.220325
    final  value 352.218611
    converged
    # weights:  15 (8 variable)
    initial  value 439.444915
    iter  10 value 354.812899
    final  value 354.741801
    converged
    # weights:  12 (6 variable)
    initial  value 439.444915
    iter  10 value 358.535353
    final  value 358.535259
    converged
    # weights:  18 (10 variable)
    initial  value 439.444915
    iter  10 value 354.101438
    iter  20 value 353.441487
    final  value 353.441474
    converged
    Call:
    multinom(formula = apply ~ pared + gpa + pared:gpa, data = dat)

    Coefficients:
                    (Intercept)     pared       gpa pared:gpa
    somewhat likely   -2.470841  5.780411 0.6293256 -1.559901
    very likely       -3.804172 -3.033925 0.6057269  1.332739

    Std. Errors:
                    (Intercept)    pared       gpa pared:gpa
    somewhat likely   0.9319653 2.644279 0.3091534 0.8425412
    very likely       1.6081085 4.122495 0.5300470 1.2495708

    Residual Deviance: 709.4836
    AIC: 725.4836
    Likelihood ratio tests of Multinomial Models

    Response: apply
                        Model Resid. df Resid. Dev   Test    Df LR stat.   Pr(Chi)
    1 pared + gpa + pared:gpa       792   709.4836
    2    pared * public * gpa       784   701.1648 1 vs 2     8 8.318787 0.4029658
    [1] "Reject Null (similar models) at 0.95 confidence level? FALSE"

The simplified model stands the Likelihood Ratio test, suggesting that it may be a valid model.

(b) Repeat the analysis with an ordinal model. Comment on any differences.

```r
%%R

# Ordinal model with AIC step-wise selection
model.full <- polr(apply ~ pared * public * gpa, data = dat, Hess=TRUE) # full interactive
model.ord <- stepAIC(model.full, direction = "both", trace = FALSE) # simplified
print(summary(model.ord))

# Compare simplified to full interactive
av <- anova(model.ord, model.full)
print(av)
pval <- av[2, "Pr(Chi)"]
sprintf("Reject Null (similar models) at %s confidence level? %s", CONFLEV, pval < 1 - CONFLEV)
```

    Call:
    polr(formula = apply ~ pared + gpa, data = dat, Hess = TRUE)

    Coefficients:
           Value Std. Error t value
    pared 1.0457     0.2656   3.937
    gpa   0.6042     0.2539   2.379

    Intercepts:
                                Value  Std. Error t value
    unlikely|somewhat likely    2.1763 0.7671     2.8370
    somewhat likely|very likely 4.2716 0.7922     5.3924

    Residual Deviance: 717.0638
    AIC: 725.0638
    Likelihood ratio tests of ordinal regression models

    Response: apply
                     Model Resid. df Resid. Dev   Test    Df LR stat.   Pr(Chi)
    1          pared + gpa       396   717.0638
    2 pared * public * gpa       391   714.3552 1 vs 2     5 2.708574 0.7448068
    [1] "Reject Null (similar models) at 0.95 confidence level? FALSE"

The simplified model stands the Likelihood Ratio test, suggesting that it may be a valid model.
The ordinal model differs mainly from the categorical model in that the ordinal model does not include the interaction term between parents education and GPA (*pared:gpa*). That is, the ordinal model is simpler than the categorical model. Additionally, the fact that the p-value of the Likelihood Ratio test of the ordinal model is much higher than that of categorical model (0.7 vs 0.4) indicates that the former tends to better fit the data than the latter, when compared to their corresponding full forms. Furthermore, the ordinal model seems to better agree with the common sense when it treats the responses (*apply*) as having a defined order. The results of these preliminary analyses seem to indicate that the ordinal model may appear to outweigh the categorical model.

(c) Under the ordinal model, what is the fitted probability that a student whose parents have
graduate degrees and who went to a public university, with a gpa of 3.0, is very likely to
undertake graduate study?

```r
%%R

newdata <- data.frame(pared=1, gpa=3.0)
pred <- predict(model.ord, newdata = newdata, type="probs")
print(pred["very likely"])
```

    very likely
      0.1957398

## Problem 2
The t(3) distribution has pdf p(x) = √
2
3π

1 + x
2
3
−2
, −∞ < x < ∞, and the Cauchy(
√
3, 0)
distribution has pdf g(y) = √
1
3π

1 + y
2
3
−1
, −∞ < y < ∞.

(a) Suppose U
d= U(0, 1), and Y =
√
3 tan(π(U −
1
2
)). Show that Y
d= Cauchy(
√
3, 0).

CDF of $Y \sim \text{Cauchy}(\text{location}=0, \text{scale}=\sqrt{3})$:
$$
\begin{align}
F_Y(y) = \frac{1}{\pi} \text{arctan}\left(\frac{y}{\sqrt{3}}\right) + \frac{1}{2} \\
\end{align}
$$

Inverse of $F_Y(y)$:
$$
\begin{align}
\text{arctan}\left(\frac{y}{\sqrt{3}}\right) &= \pi \left(F_Y(y) - \frac{1}{2}\right) \\
\frac{y}{\sqrt{3}} &= \text{tan} \left( \pi \left(F_Y(y) - \frac{1}{2}\right)\right) \\
y &= \sqrt{3} \text{tan} \left( \pi \left(F_Y(y) - \frac{1}{2}\right)\right) \\
\Rightarrow F_Y^{-1}(u) &= \sqrt{3} \text{tan} \left( \pi \left(u - \frac{1}{2}\right)\right) \\
\end{align}
$$

Hence, $Y = \sqrt{3} \text{tan} \left( \pi \left(u - \frac{1}{2}\right)\right) \sim \text{Cauchy}(\text{location}=0, \text{scale}=\sqrt{3})$

(b) Construct an Acceptance-Rejection (A-R) sampling algorithm (or a mixture of A-R and transformation algorithm) for generating random numbers from t(3) by using the result of (a).

Let us assume that $T \sim t(3)$ and $Y \sim \text{Cauchy}(\text{location}=0, \text{scale}=\sqrt{3})$ are close enough to each other so that the ratio of their distributions $f_T(x)/f_Y(x)$ is bounded by a constant $c > 0$.
$$
\begin{align}
\frac{f_T(x)}{f_Y(x)} &= \frac{\frac{2}{\sqrt{3}\pi}\left(1 + \frac{x^2}{3}\right)^{-2}}{\frac{1}{\sqrt{3}\pi}\left(1 + \frac{x^2}{3}\right)^{-1}} \\
&= 2 \left(1 + \frac{x^2}{3}\right)^{-1} \\
&\leq 2 \left(1 + \frac{0}{3}\right)^{-1} & (x^2 = 0) \\
&\leq 2 \\
\Rightarrow c &= 2 \\
\end{align}
$$

Acceptance-Rejection algorithm for generating $T$ distributed as $t(3)$:
1. Generate $U' \sim \text{Uniform}(0, 1)$, then set $Y = \sqrt{3} \text{tan} \left( \pi \left(U' - \frac{1}{2}\right)\right)$ $\Rightarrow$ $Y$ distributed as $\text{Cauchy}(0, \sqrt{3})$
2. Generate another uniform $U \sim \text{Uniform}(0, 1)$.
3. If $U ≤ \frac{f_T(Y)}{cf_Y(Y)}$, then set T = Y (“accept”) ; otherwise go back to 1 (“reject”).

(c) Write an R function to implement (b). Then use it to generate a sample of 1000 numbers
from t(3), and compare the sample pdf curve with the actual t(3) curve.

According to the graph below, the generated samples are close to the actual t(3) distribution.

```r
%%R

pt3 <- function(x) {
  # Calculate theoretical pdf of t(3)

  2 / (sqrt(3) * pi) * (1 + x^2/3)^(-2)
}

c = 2
rt3 <- function() {
  # Simulate t(3) using acceptance-rejection method
  success <- FALSE
  while (!success) {
    # Cauchy distribution
    uc <- runif(1, min = 0, max = 1)
    y <- sqrt(3) * tan(pi * (uc - 1/2))

    # Uniform distribution
    u <- runif(1, min = 0, max = 1)

    # Accept/Reject
    if (u <= 1/c * 2 * (1 + y^2/3)^(-1)) {
        return(y)
    }
  }
}

# Compare the sample pdf curve with the actual t(3) curve
set.seed(1)
n <- 1000
t3.sim <- replicate(n, rt3()) # random t(3) by acceptance-rejection method
hist(t3.sim, breaks=40, freq=F, xlab="t", ylab="pdf f(t)",
     main="theoretical and simulated t(3) density") # simulated pdf
t <- seq(min(t3.sim), max(t3.sim), .1)
lines(t, pt3(t)) # theoretical pdf
```

## Problem 3
The Dirichlet distribution is a multivariate generalisation of the beta distribution. It takes values
in {x = (x1, . . . , xd) : xi ∈ [0, 1],
P
i
xi = 1}, and, for α = (α1, . . . , αd) ∈ R
d
+, has density
f(x) = 1
B(α)
Y
d
i=1
x
αi−1
i
, where B(α) =
Q
i Γ(αi)
Γ (P
i αi)
.

(a) Prove that if X = (X1, . . . , Xd) has a Dirichlet distribution with parameter α (we write
X ∼ Dir(α)), then EXi = αi/
P
i αi
. Hint: R
· · · R
f(x)dx1 · · · dxd = 1.

Calculate $EX_1$:
$$
\begin{align}
EX_1 &= \int \dots \int x_1 f(x) dx & (dx = dx_1 \dots dx_d) \\
&= \int \dots \int x_1 \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \prod_{i} x_i^{\alpha_i-1} dx & \left(A = \sum_{i} \alpha_i\right) \\
&= \int \dots \int \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} x_1^{(\alpha_1 + 1) - 1} \prod_{i \neq 1} x_i^{\alpha_i-1} dx \\
&= \int \dots \int \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \prod_{i} x_i^{\alpha'_i-1} dx & (\alpha'_i = \alpha_i \text{ for } i \neq 1; \alpha'_i = \alpha_i + 1 \text{ otherwise}) \\
&= \int \dots \int \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \frac{\prod_{i} \Gamma(\alpha'_i)}{\Gamma(A')} \frac{\Gamma(A')}{\prod_{i} \Gamma(\alpha'_i)} \prod_{i} x_i^{\alpha'_i-1} dx & \left(A' = \sum_{i} \alpha'_i\right) \\
&= \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \frac{\prod_{i} \Gamma(\alpha'_i)}{\Gamma(A')} \int \dots \int \frac{\Gamma(A')}{\prod_{i} \Gamma(\alpha'_i)} \prod_{i} x_i^{\alpha'_i-1} dx \\
&= \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \frac{\prod_{i} \Gamma(\alpha'_i)}{\Gamma(A')} \int \dots \int f'(x) dx & \left(f'(x) \text{ PDF of } \text{Dir}\left(\alpha' = \left(\alpha'_1, \dots, \alpha'_d\right)\right)\right) \\
&= \frac{\Gamma(A)}{\prod_{i} \Gamma(\alpha_i)} \frac{\prod_{i} \Gamma(\alpha'_i)}{\Gamma(A')} & \left(\int \dots \int f'(x) dx = 1\right) \\
&= \frac{\Gamma(A)}{\Gamma(A')} \frac{\prod_{i} \Gamma(\alpha'_i)}{\prod_{i} \Gamma(\alpha_i)} & \left(\Gamma(\alpha'_i) = \Gamma(\alpha_i) \text{ for } i \neq 1\right) \\
&= \frac{\Gamma(A)}{\Gamma(A + 1)} \frac{\Gamma(\alpha'_1)}{\Gamma(\alpha_1)} \\
&= \frac{\Gamma(A)}{\Gamma(A + 1)} \frac{\Gamma(\alpha_1 + 1)}{\Gamma(\alpha_1)} \\
&= \frac{\alpha_1}{A} & \left(\Gamma(z + 1) = z\Gamma(z)\right) \\
\end{align}
$$

Without loss of generality, the steps above can be applied to calculate any $EX_i$:
$$
\begin{align}
EX_i &= \frac{\alpha_i}{A} = \frac{\alpha_i}{\sum_{i} \alpha_i} \\
\end{align}
$$
as required.

(b) Show that the Dirichlet distribution is the conjugate prior for the multinomial. That is, if
p ∼ Dir(α) and X|p ∼ multinomial(n, p), then p|{X = x} ∼ Dir(β), where β depends on α
and x.

PDF of prior Dirichlet is proportional to:
$$
\begin{align}
\prod_{i} p_i^{\alpha_i-1} \\
\end{align}
$$

PMF of conditional Multinomial is proportional to:
$$
\begin{align}
\prod_{i} p_i^{x_i} \\
\end{align}
$$

According to Bayes' rule, PDF of posterior is proportional to:
$$
\begin{align}
\prod_{i} p_i^{\alpha_i-1} \prod_{i} p_i^{x_i} = \prod_{i} p_i^{\alpha_i + x_i -1} \\
\end{align}
$$
which is also proportional to PDF of $\text{Dirichlet}(\alpha+x)$.
Thus, Dirichlet distribution is the conjugate prior for the multinomial.

(c) In 2003 Briggs, Ades and Price reported on a trial for the treatment of asthma. Patients
received one of two treatments (seretide or fluticasone), and their status was monitored from
week to week. Possible states were
STW Successfully treated week
UTW Unsuccessfully treated week
HEX Hospital managed exacerbation
PEX Primary-care managed exacerbation
TF Treatment failure (treatment ceased and patient removed from the trial)
For patients on seretide, the number of transitions from one state to another were
To
STW UTW HEX PEX TF
From Total
STW 210 60 0 1 1 272
UTW 88 641 0 4 13 746
HEX 0 0 0 0 0 0
PEX 1 0 0 0 1 2
TF 0 0 0 0 81 81
The rows of this table can be considered as observations from independent multinomial random
variables. Using Dir(1,. . . ,1) priors, give Bayesian estimates (posterior means) for
pij = P( state changes from i to j )
for i = STW, . . . , PEX and j = STW, . . . , TF.
What prior would be appropriate for the transitions from state TF?

Since Dirichlet is conjugate prior to Multinomial, the posterior is also Dirichlet, with parameters:
> From STW: $\alpha = (211, 61, 1, 2, 2)$
> From UTW: $\alpha = (89, 642, 1, 5, 14)$
> From HEX: $\alpha = (1, 1, 1, 1, 1)$
> From PEX: $\alpha = (2, 1, 1, 1, 2)$

Hence, posterior means are:
> From STW: $Ep = [211, 61, 1, 2, 2] / 277$
> From UTW: $Ep = [89, 642, 1, 5, 14] / 751$
> From HEX: $Ep = [1, 1, 1, 1, 1] / 5$
> From PEX: $Ep = [2, 1, 1, 1, 2] / 7$

```r
%%R

print("Posterior means")
alpha <- matrix(c(211,61,1,2,2,
                  89,642,1,5,14,
                  1,1,1,1,1,
                  2,1,1,1,2), nrow=4, byrow=TRUE)
print(alpha / rowSums(alpha))
```

    [1] "Posterior means"
              [,1]      [,2]        [,3]        [,4]        [,5]
    [1,] 0.7617329 0.2202166 0.003610108 0.007220217 0.007220217
    [2,] 0.1185087 0.8548602 0.001331558 0.006657790 0.018641811
    [3,] 0.2000000 0.2000000 0.200000000 0.200000000 0.200000000
    [4,] 0.2857143 0.1428571 0.142857143 0.142857143 0.285714286

For the transitions from state TF, there is no possibilities other than to state TF. Thus, the prior should be such that $p_{\text{TF}-j} = 0 \text{ for } j \neq \text{TF} \text{ and } p_{\text{TF-TF}} = 1$. That is, the prior follows $\text{MVN}(\mu=[0, 0, 0, 0, 1], \Sigma=0)$

## Problem 4
There is a famous data set from RA Fisher’s 1936 paper on The use of multiple measurements in
taxonomic problems which consists of 50 samples from each of three species of Iris (Iris setosa, Iris
virginica and Iris versicolor). Four features were measured from each sample: the length and the
width of the sepals and petals, in centimetres.
The data is available in R in the data set iris.
Perform a principal components and cluster analysis on the four variables using the R commands
prcomp and kmeans. Use two separate methods to determine the number of clusters.
Use the ggplot2 and ggfortify libraries to do plots of the first two principal components that
indicate where the directions of the original variables and the positions of the cluster means.
Comment on your results.

```r
%%R

# Data
print(str(iris))

# Separate measurements and species
measures <- subset(iris, select=-Species)
measures.std <- scale(measures) # standardized
species <- iris[, "Species"]
```

    'data.frame':	150 obs. of  5 variables:
     $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
     $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
     $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
     $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
     $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
    NULL

### Principal Component Analysis

```r
%%R

pca <- prcomp(measures.std, scale=TRUE)

# Importance of components
print(summary(pca))
print(fviz_eig(pca, addlabels=TRUE))
```

```r
%%R

# Contribution of variables
print(pca)
print(fviz_contrib(pca, choic="var", axes=1, top=10))
print(fviz_contrib(pca, choic="var", axes=2, top=10))
```

**Comments**:
From *Cumulative Proportion*, the first 2 principal components sufficiently capture over 95% of variance in data.
From *rotation*, *Sepal.Width* is the least significant contributing factor to first principal component (PC1), while *Petal.Length* and *Petal.Width* are the least significant contributing factors to second principal component (PC2).

### K-means

```r
%%R

# Run kmeans over a range of different numbers of clusters
print(fviz_nbclust(measures.std, kmeans, method="wss")) # Within Sum of Squares
print(fviz_nbclust(measures.std, kmeans, method="silhouette"))
```

**Comments**:
Both methods (Within Sum of Squares and Silhouette) happen to agree on the same optimal number of clusters $K = 2$ (Within Sum of Squares leveling off and Silhouette topping at same point $K = 2$).

Let us proceed to cluster the data accordingly.

```r
%%R

# Cluster
k <- 2
km <- kmeans(measures.std, k, nstart=10)
centers.pca <- predict(pca, newdata=km$centers)[, c("PC1", "PC2")] # principal components of centers

# Plot
autoplot(km, data=measures.std, scale=FALSE, frame=TRUE, loadings=TRUE, loadings.label=TRUE,
         loadings.colour="blue", loadings.label.colour="blue") + # clusters and directions
geom_point(data=centers.pca, colour=as.numeric(row.names(centers.pca))*2.5, size=3) + # centers
geom_point(aes(shape=species)) + scale_shape(solid=FALSE) # species
```

From the graph, it can be realized that:

*   Variable *Sepal.Width* has the most significant influence in PC1-PC2 coordinates.
*   The 2 clusters are well separated.
*   Although there are 3 classes in our dataset, two of them (*versicolor* and *virginica*) are close to each other, thus clustered into the same group.