# **CS224W - Colab 4**

In Colab 2 we constructed GNN models by using PyTorch Geometric's built in GCN layer, `GCNConv`. In Colab 3 we implemented the  **GraphSAGE** ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)) layer. In this colab you'll use what you've learned and implement a more powerful layer: **GAT** ([Veličković et al. (2018)](https://arxiv.org/abs/1710.10903)). Then we will run our models on the CORA dataset, which is a standard citation network benchmark dataset.

**Note**: Make sure to **sequentially run all the cells in each section** so that the intermediate variables / packages will carry over to the next cell

Have fun and good luck on Colab 4 :)

# Device
We recommend using a GPU for this Colab.

Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**.

## Installation

```python
# Install torch geometric
import os
if 'IS_GRADESCOPE_ENV' not in os.environ:
  !pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
  !pip install torch-sparse -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
  !pip install torch-geometric
  !pip install -q git+https://github.com/snap-stanford/deepsnap.git
```

```python
import torch_geometric
torch_geometric.__version__
```

# 1) GNN Layers

## Implementing Layer Modules

In Colab 2, we implemented a GCN model for node and graph classification tasks. However, for that notebook we took advantage of PyG's built in GCN module. For Colabs 3 and 4, we provide a build upon a general Graph Neural Network Stack, into which we will be able to plugin our own module implementations: GraphSAGE and GAT.

We will then use our layer implemenations to complete node classification on the CORA dataset, a standard citation network benchmark. In this dataset, nodes correspond to documents and edges correspond to undirected citations. Each node or document in the graph is assigned a class label and features based on the documents binarized bag-of-words representation. Specifically, the Cora graph has 2708 nodes, 5429 edges, 7 prediction classes, and 1433 features per node.

## GNN Stack Module

Below is the implementation of a general GNN stack, where we can plugin any GNN layer, such as **GraphSage**, **GAT**, etc. This module is provided for you. Your implementations of the **GraphSage** and **GAT** layers will function as components in the GNNStack Module.

```python
import torch
import torch_scatter
import torch.nn as nn
import torch.nn.functional as F

import torch_geometric.nn as pyg_nn
import torch_geometric.utils as pyg_utils

from torch import Tensor
from typing import Union, Tuple, Optional
from torch_geometric.typing import (OptPairTensor, Adj, Size, NoneType,
                                    OptTensor)

from torch.nn import Parameter, Linear
from torch_sparse import SparseTensor, set_diag
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.utils import remove_self_loops, add_self_loops, softmax, degree

class GNNStack(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, args, emb=False):
        super(GNNStack, self).__init__()
        conv_model = self.build_conv_model(args.model_type)
        self.convs = nn.ModuleList()
        self.convs.append(conv_model(input_dim, hidden_dim))
        assert (args.num_layers >= 1), 'Number of layers is not >=1'
        for l in range(args.num_layers-1):
            self.convs.append(conv_model(args.heads * hidden_dim, hidden_dim))

        # post-message-passing
        self.post_mp = nn.Sequential(
            nn.Linear(args.heads * hidden_dim, hidden_dim), nn.Dropout(args.dropout),
            nn.Linear(hidden_dim, output_dim))

        self.dropout = args.dropout
        self.num_layers = args.num_layers

        self.emb = emb

    def build_conv_model(self, model_type):
        if model_type == 'GraphSage':
            return GraphSage
        elif model_type == 'GAT':
            # When applying GAT with num heads > 1, you need to modify the
            # input and output dimension of the conv layers (self.convs),
            # to ensure that the input dim of the next layer is num heads
            # multiplied by the output dim of the previous layer.
            # HINT: In case you want to play with multiheads, you need to change the for-loop that builds up self.convs to be
            # self.convs.append(conv_model(hidden_dim * num_heads, hidden_dim)),
            # and also the first nn.Linear(hidden_dim * num_heads, hidden_dim) in post-message-passing.
            return GAT

    def forward(self, data):
        x, edge_index, batch = data.x, data.edge_index, data.batch

        for i in range(self.num_layers):
            x = self.convs[i](x, edge_index)
            x = F.relu(x)
            x = F.dropout(x, p=self.dropout,training=self.training)

        x = self.post_mp(x)

        if self.emb == True:
            return x

        return F.log_softmax(x, dim=1)

    def loss(self, pred, label):
        return F.nll_loss(pred, label)
```

## Creating Our Own Message Passing Layer

Now let's start implementing our own message passing layers! Working through this part will help us become acutely familiar with the behind the scenes work of implementing Pytorch Message Passing Layers, allowing us to build our own GNN models. To do so, we will work with and implement 3 critcal functions needed to define a PyG Message Passing Layer: `forward`, `message`, and `aggregate`.

Before diving head first into the coding details, let us quickly review the key components of the message passing process. To do so, we will focus on a single round of messsage passing with respect to a single central node $x$. Before message passing, $x$ is associated with a feature vector $x^{l-1}$, and the goal of message passing is to update this feature vector as $x^l$. To do so, we implement the following steps: 1) each neighboring node $v$ passes its current message $v^{l-1}$ across the edge $(x, v)$ - 2) for the node $x$, we aggregate all of the messages of the neighboring nodes (for example through a sum or mean) - and 3) we transform the aggregated information by for example applying linear and non-linear transformations. Altogether, the message passing process is applied such that every node $u$ in our graph updates its embedding by acting as the central node $x$ in step 1-3 described above.

Now, we extending this process to that of a single message passing layer, the job of a message passing layer is to update the current feature representation or embedding of each node in a graph by propagating and transforming information within the graph. Overall, the general paradigm of a message passing layers is: 1) pre-processing -> 2) **message passing** / propagation -> 3) post-processing.

The `forward` fuction that we will implement for our message passing layer captures this execution logic. Namely, the `forward` function handles the pre and post-processing of node features / embeddings, as well as initiates message passing by calling the `propagate` function.

The `propagate` function encapsulates the message passing process! It does so by calling three important functions: 1) `message`, 2) `aggregate`, and 3) `update`. Our implementation will vary slightly from this, as we will not explicitly implement `update`, but instead place the logic for updating node embeddings after message passing and within the `forward` function. To be more specific, after information is propagated (message passing), we can further transform the node embeddings outputed by `propagate`. Therefore, the output of `forward` is exactly the node embeddings after one GNN layer.

Lastly, before starting to implement our own layer, let us dig a bit deeper into each of the functions described above:

1.

```
def propagate(edge_index, x=(x_i, x_j), extra=(extra_i, extra_j), size=size):
```
Calling `propagate` initiates the message passing process. Looking at the function parameters, we highlight a couple of key parameters.

  - `edge_index` is passed to the forward function and captures the edge structure of the graph.
  - `x=(x_i, x_j)` represents the node features that will be used in message passing. In order to explain why we pass the tuple `(x_i, x_j)`, we first look at how our edges are represented. For every edge $(i, j) \in \mathcal{E}$, we can differentiate $i$ as the source or central node ($x_{central}$) and j as the neighboring node ($x_{neighbor}$).

    Taking the example of message passing above, for a central node $u$ we will aggregate and transform all of the messages associated with the nodes $v$ s.t. $(u, v) \in \mathcal{E}$ (i.e. $v \in \mathcal{N}_{u}$). Thus we see, the subscripts `_i` and `_j` allow us to specifcally differenciate features associated with central nodes (i.e. nodes  recieving message information) and neighboring nodes (i.e. nodes passing messages).

    This is definitely a somewhat confusing concept; however, one key thing to remember / wrap your head around is that depending on the perspective, a node $x$ acts as a central node or a neighboring node. In fact, in undirected graphs we store both edge directions (i.e. $(i, j)$ and $(j, i)$). From the central node perspective, `x_i`, x is collecting neighboring information to update its embedding. From a neighboring node perspective, `x_j`, x is passing its message information along the edge connecting it to a different central node.

  - `extra=(extra_i, extra_j)` represents additional information that we can associate with each node beyond its current feature embedding. In fact, we can include as many additional parameters of the form `param=(param_i, param_j)` as we would like. Again, we highlight that indexing with `_i` and `_j` allows us to differentiate central and neighboring nodes.

  The output of the `propagate` function is a matrix of node embeddings after the message passing process and has shape $[N, d]$.

2.
```
def message(x_j, ...):
```
The `message` function is called by propagate and constructs the messages from
neighboring nodes $j$ to central nodes $i$ for each edge $(i, j)$ in *edge_index*. This function can take any argument that was initially passed to `propagate`. Furthermore, we can again differentiate central nodes and neighboring nodes by appending `_i` or `_j` to the variable name, .e.g. `x_i` and `x_j`. Looking more specifically at the variables, we have:

  - `x_j` represents a matrix of feature embeddings for all neighboring nodes passing their messages along their respective edge (i.e. all nodes $j$ for edges $(i, j) \in \mathcal{E}$). Thus, its shape is $[|\mathcal{E}|, d]$!
  - In implementing GAT we will see how to access additional variables passed to propagate

  Critically, we see that the output of the `message` function is a matrix of neighboring node embeddings ready to be aggregated, having shape $[|\mathcal{E}|, d]$.

3.
```
def aggregate(self, inputs, index, dim_size = None):
```
Lastly, the `aggregate` function is used to aggregate the messages from neighboring nodes. Looking at the parameters we highlight:

  - `inputs` represents a matrix of the messages passed from neighboring nodes (i.e. the output of the `message` function).
  - `index` has the same shape as `inputs` and tells us the central node that corresponding to each of the rows / messages $j$ in the `inputs` matrix. Thus, `index` tells us which rows / messages to aggregate for each central node.

  The output of `aggregate` is of shape $[N, d]$.

For additional resources refer to the PyG documentation for implementing custom message passing layers: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html

## GAT Implementation

Attention mechanisms have become the state-of-the-art in many sequence-based tasks such as machine translation and learning sentence representations. One of the major benefits of attention-based mechanisms is their ability to focus on the most relevant parts of the input to make decisions. In this problem, we will see how attention mechanisms can be used to perform node classification over graph-structured data through the usage of Graph Attention Networks (GATs) ([Veličković et al. (2018)](https://arxiv.org/abs/1710.10903)).

The building block of the Graph Attention Network is the graph attention layer, which is a variant of the aggregation function. Let $N$ be the number of nodes and $F$ be the dimension of the feature vector for each node. The input to each graph attentional layer is a set of node features: $\mathbf{h} = \{\overrightarrow{h_1}, \overrightarrow{h_2}, \dots, \overrightarrow{h_N}$\}, $\overrightarrow{h_i} \in R^F$. The output of each graph attentional layer is a new set of node features, which may have a new dimension $F'$: $\mathbf{h'} = \{\overrightarrow{h_1'}, \overrightarrow{h_2'}, \dots, \overrightarrow{h_N'}\}$, with $\overrightarrow{h_i'} \in \mathbb{R}^{F'}$.

We will now describe how this transformation is performed for each graph attention layer. First, a shared linear transformation parametrized by the weight matrix $\mathbf{W} \in \mathbb{R}^{F' \times F}$ is applied to every node.

Next, we perform self-attention on the nodes. We use a shared attention function $a$:
\begin{equation}
a : \mathbb{R}^{F'} \times \mathbb{R}^{F'} \rightarrow \mathbb{R}.
\end{equation}

that computes the attention coefficients capturing the importance of node $j$'s features to node $i$:
\begin{equation}
e_{ij} = a(\mathbf{W_l}\overrightarrow{h_i}, \mathbf{W_r} \overrightarrow{h_j})
\end{equation}

The most general formulation of self-attention allows every node to attend to all other nodes which drops all structural information. However, to utilize graph structure in the attention mechanisms, we use **masked attention**. In masked attention, we only compute attention coefficients $e_{ij}$ for nodes $j \in \mathcal{N}_i$ where $\mathcal{N}_i$ is some neighborhood of node $i$ in the graph.

To easily compare coefficients across different nodes, we normalize the coefficients across $j$ using a softmax function:
\begin{equation}
\alpha_{ij} = \text{softmax}_j(e_{ij}) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_i} \exp(e_{ik})}
\end{equation}

For this problem, our attention mechanism $a$ will be a single-layer feedforward neural network parametrized by a weight vectors $\overrightarrow{a_l} \in \mathbb{R}^{F'}$ and $\overrightarrow{a_r} \in \mathbb{R}^{F'}$, followed by a LeakyReLU nonlinearity (with negative input slope 0.2). Let $\cdot^T$ represent transposition and $||$ represent concatenation. The coefficients computed by our attention mechanism may be expressed as:

\begin{equation}
\alpha_{ij} = \frac{\exp\Big(\text{LeakyReLU}\Big(\overrightarrow{a_l}^T \mathbf{W_l} \overrightarrow{h_i} + \overrightarrow{a_r}^T\mathbf{W_r}\overrightarrow{h_j}\Big)\Big)}{\sum_{k\in \mathcal{N}_i} \exp\Big(\text{LeakyReLU}\Big(\overrightarrow{a_l}^T \mathbf{W_l} \overrightarrow{h_i} + \overrightarrow{a_r}^T\mathbf{W_r}\overrightarrow{h_k}\Big)\Big)}
\end{equation}

For the following questions, we denote `alpha_l` = $\alpha_l = [...,\overrightarrow{a_l}^T \mathbf{W_l} \overrightarrow{h_i},...] \in \mathcal{R}^n$ and `alpha_r` = $\alpha_r = [..., \overrightarrow{a_r}^T \mathbf{W_r} \overrightarrow{h_j}, ...] \in \mathcal{R}^n$.

At every layer of GAT, after the attention coefficients are computed for that layer, the aggregation function can be computed by a weighted sum of neighborhood messages, where weights are specified by $\alpha_{ij}$.

Now, we use the normalized attention coefficients to compute a linear combination of the features corresponding to them. These aggregated features will serve as the final output features for every node.

\begin{equation}
h_i' = \sum_{j \in \mathcal{N}_i} \alpha_{ij} \mathbf{W_r} \overrightarrow{h_j}.
\end{equation}

At this point, we have covered a lot of information! Before reading further about multi-head attention, we encourage you to go again through the excersize of thinking about what components of the attention mechanism correspond with the different functions: 1) `forward`, 2) `message`, and 3 `aggregate`.

- Hint 1: Our aggregation is very similar to that of GraphSage except now we are using sum aggregation
- Hint 2: The terms we aggregate over again represent the individual message that each neighbor node j sends. Thus, we see that $\alpha_{ij}$ is part of the message each node sends and is thus computed during the message step. This makes sense since an attention weight is associated with each edge in the graph.
- Hint 3: Look at the terms in the definition of $\alpha_{ij}$. What values do we want to pre-process and pass as parameters to the `propagate` function. The parameters of `message(..., x_j, alpha_j, alpha_i, ...)` should give a good hint.

### Multi-Head Attention
To stabilize the learning process of self-attention, we use multi-head attention. To do this we use $K$ independent attention mechanisms, or ``heads'' compute output features as in the above equations. Then, we concatenate these output feature representations:

\begin{equation}
    \overrightarrow{h_i}' = ||_{k=1}^K \Big(\sum_{j \in \mathcal{N}_i} \alpha_{ij}^{(k)} \mathbf{W_r}^{(k)} \overrightarrow{h_j}\Big)
\end{equation}

where $||$ is concentation, $\alpha_{ij}^{(k)}$ are the normalized attention coefficients computed by the $k$-th attention mechanism $(a^k)$, and $\mathbf{W}^{(k)}$ is the corresponding input linear transformation's weight matrix. Note that for this setting, $\mathbf{h'} \in \mathbb{R}^{KF'}$.

```python
class GAT(MessagePassing):

    def __init__(self, in_channels, out_channels, heads = 2,
                 negative_slope = 0.2, dropout = 0., **kwargs):
        super(GAT, self).__init__(node_dim=0, **kwargs)

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.heads = heads
        self.negative_slope = negative_slope
        self.dropout = dropout

        self.lin_l = None
        self.lin_r = None
        self.att_l = None
        self.att_r = None

        ############################################################################
        # TODO: Your code here!
        # Define the layers needed for the message functions below.
        # self.lin_l is the linear transformation that you apply to embeddings
        # BEFORE message passing.
        #
        # Pay attention to dimensions of the linear layers, since we're using
        # multi-head attention.
        # Our implementation is ~1 lines, but don't worry if you deviate from this.

        ############################################################################

        self.lin_r = self.lin_l

        ############################################################################
        # TODO: Your code here!
        # Define the attention parameters \overrightarrow{a_l/r}^T in the above intro.
        # You have to deal with multi-head scenarios.
        # Use nn.Parameter instead of nn.Linear
        # Our implementation is ~2 lines, but don't worry if you deviate from this.

        ############################################################################

        self.reset_parameters()

    def reset_parameters(self):
        nn.init.xavier_uniform_(self.lin_l.weight)
        nn.init.xavier_uniform_(self.lin_r.weight)
        nn.init.xavier_uniform_(self.att_l)
        nn.init.xavier_uniform_(self.att_r)

    def forward(self, x, edge_index, size = None):

        H, C = self.heads, self.out_channels

        ############################################################################
        # TODO: Your code here!
        # Implement message passing, as well as any pre- and post-processing (our update rule).
        # 1. First apply linear transformation to node embeddings, and split that
        #    into multiple heads. We use the same representations for source and
        #    target nodes, but apply different linear weights (W_l and W_r)
        # 2. Calculate alpha vectors for central nodes (alpha_l) and neighbor nodes (alpha_r).
        # 3. Call propagate function to conduct the message passing.
        #    3.1 Remember to pass alpha = (alpha_l, alpha_r) as a parameter.
        #    3.2 See there for more information: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html
        # 4. Transform the output back to the shape of [N, H * C].
        # Our implementation is ~5 lines, but don't worry if you deviate from this.

        ############################################################################

        return out

    def message(self, x_j, alpha_j, alpha_i, index, ptr, size_i):

        ############################################################################
        # TODO: Your code here!
        # Implement your message function. Putting the attention in message
        # instead of in update is a little tricky.
        # 1. Calculate the final attention weights using alpha_i and alpha_j,
        #    and apply leaky Relu.
        # 2. Calculate softmax over the neighbor nodes for all the nodes. Use
        #    torch_geometric.utils.softmax instead of the one in Pytorch.
        # 3. Apply dropout to attention weights (alpha).
        # 4. Multiply embeddings and attention weights. As a sanity check, the output
        #    should be of shape [E, H, C].
        # 5. ptr (LongTensor, optional): If given, computes the softmax based on
        #    sorted inputs in CSR representation. You can simply pass it to softmax.
        # Our implementation is ~4-5 lines, but don't worry if you deviate from this.

        ############################################################################

        return out

    def aggregate(self, inputs, index, dim_size = None):

        ############################################################################
        # TODO: Your code here!
        # Implement your aggregate function here.
        # See here as how to use torch_scatter.scatter: https://pytorch-scatter.readthedocs.io/en/latest/_modules/torch_scatter/scatter.html
        # Pay attention to "reduce" parameter is different from that in GraphSage.
        # Our implementation is ~1 lines, but don't worry if you deviate from this.

        ############################################################################

        return out
```

## Building Optimizers

This function has been implemented for you. **For grading purposes please use the default Adam optimizer**, but feel free to play with other types of optimizers on your own.

```python
import torch.optim as optim

def build_optimizer(args, params):
    weight_decay = args.weight_decay
    filter_fn = filter(lambda p : p.requires_grad, params)
    if args.opt == 'adam':
        optimizer = optim.Adam(filter_fn, lr=args.lr, weight_decay=weight_decay)
    elif args.opt == 'sgd':
        optimizer = optim.SGD(filter_fn, lr=args.lr, momentum=0.95, weight_decay=weight_decay)
    elif args.opt == 'rmsprop':
        optimizer = optim.RMSprop(filter_fn, lr=args.lr, weight_decay=weight_decay)
    elif args.opt == 'adagrad':
        optimizer = optim.Adagrad(filter_fn, lr=args.lr, weight_decay=weight_decay)
    if args.opt_scheduler == 'none':
        return None, optimizer
    elif args.opt_scheduler == 'step':
        scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=args.opt_decay_step, gamma=args.opt_decay_rate)
    elif args.opt_scheduler == 'cos':
        scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=args.opt_restart)
    return scheduler, optimizer
```

## Training and Testing

Here we provide you with the functions to train and test. **Please do not modify this part for grading purposes.**

```python
import time

import networkx as nx
import numpy as np
import torch
import torch.optim as optim
from tqdm import trange
import pandas as pd
import copy

from torch_geometric.datasets import TUDataset
from torch_geometric.datasets import Planetoid
from torch_geometric.data import DataLoader

import torch_geometric.nn as pyg_nn

import matplotlib.pyplot as plt

def train(dataset, args):

    print("Node task. test set size:", np.sum(dataset[0]['test_mask'].numpy()))
    print()
    test_loader = loader = DataLoader(dataset, batch_size=args.batch_size, shuffle=False)

    # build model
    model = GNNStack(dataset.num_node_features, args.hidden_dim, dataset.num_classes,
                            args)
    scheduler, opt = build_optimizer(args, model.parameters())

    # train
    losses = []
    test_accs = []
    best_acc = 0
    best_model = None
    for epoch in trange(args.epochs, desc="Training", unit="Epochs"):
        total_loss = 0
        model.train()
        for batch in loader:
            opt.zero_grad()
            pred = model(batch)
            label = batch.y
            pred = pred[batch.train_mask]
            label = label[batch.train_mask]
            loss = model.loss(pred, label)
            loss.backward()
            opt.step()
            total_loss += loss.item() * batch.num_graphs
        total_loss /= len(loader.dataset)
        losses.append(total_loss)

        if epoch % 10 == 0:
          test_acc = test(test_loader, model)
          test_accs.append(test_acc)
          if test_acc > best_acc:
            best_acc = test_acc
            best_model = copy.deepcopy(model)
        else:
          test_accs.append(test_accs[-1])

    return test_accs, losses, best_model, best_acc, test_loader

def test(loader, test_model, is_validation=False, save_model_preds=False, model_type=None):
    test_model.eval()

    correct = 0
    # Note that Cora is only one graph!
    for data in loader:
        with torch.no_grad():
            # max(dim=1) returns values, indices tuple; only need indices
            pred = test_model(data).max(dim=1)[1]
            label = data.y

        mask = data.val_mask if is_validation else data.test_mask
        # node classification: only evaluate on nodes in test set
        pred = pred[mask]
        label = label[mask]

        if save_model_preds:
          print ("Saving Model Predictions for Model Type", model_type)

          data = {}
          data['pred'] = pred.view(-1).cpu().detach().numpy()
          data['label'] = label.view(-1).cpu().detach().numpy()

          df = pd.DataFrame(data=data)
          # Save locally as csv
          df.to_csv('CORA-Node-' + model_type + '.csv', sep=',', index=False)

        correct += pred.eq(label).sum().item()

    total = 0
    for data in loader.dataset:
        total += torch.sum(data.val_mask if is_validation else data.test_mask).item()

    return correct / total

class objectview(object):
    def __init__(self, d):
        self.__dict__ = d

```

## Let's Start the Training!

We will be working on the CORA dataset on node-level classification.

This part is implemented for you. **For grading purposes, please do not modify the default parameters.** However, feel free to play with different configurations just for fun!

**Submit your best accuracy and loss on Gradescope.**

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
    for args in [
        {'model_type': 'GAT', 'dataset': 'cora', 'num_layers': 2, 'heads': 1, 'batch_size': 32, 'hidden_dim': 32, 'dropout': 0.5, 'epochs': 500, 'opt': 'adam', 'opt_scheduler': 'none', 'opt_restart': 0, 'weight_decay': 5e-3, 'lr': 0.01},
    ]:
        args = objectview(args)
        for model in ['GAT']:
            args.model_type = model

            # Match the dimension.
            if model == 'GAT':
              args.heads = 2
            else:
              args.heads = 1

            if args.dataset == 'cora':
                dataset = Planetoid(root='/tmp/cora', name='Cora')
            else:
                raise NotImplementedError("Unknown dataset")
            test_accs, losses, best_model, best_acc, test_loader = train(dataset, args)

            print("Maximum test set accuracy: {0}".format(max(test_accs)))
            print("Minimum loss: {0}".format(min(losses)))

            # Run test for our best model to save the predictions!
            test(test_loader, best_model, is_validation=False, save_model_preds=True, model_type=model)
            print()

            plt.title(dataset.name)
            plt.plot(losses, label="training loss" + " - " + args.model_type)
            plt.plot(test_accs, label="test accuracy" + " - " + args.model_type)
        plt.legend()
        plt.show()

```

## Question 1: What is the maximum accuracy obtained on test set for GAT? (10 points)

Running the training cell above will also save your best GAT model predictions as *CORA-Node-GAT.csv*.

When you sumbit your assignment, you will have to download this file and attatch it to your submission. As with the other colabs, please zip this file (DON'T CHANGE ITS NAME) and the .csv file that's generated!


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

# Precalculus

## Week 1

**Topics:** Function definition, vertical line test, piecewise defined functions, domain and range, symmetry.

**Problems:** 3, 7, 8, 9, 31, 32, 37, 38, 44, 46, 47, 48, 63, 69, 70, 73, 74, 77, additional problem set G.

### Function definition

### Vertical line test

### Piecewise defined functions

A piecewise defined function is defined as:

$$\begin{align}f(x)=\begin{cases}2x\quad &, x > 1\\
-x\quad&, x \leq 1\end{cases}\end{align}$$

Now we can define the **absolute value** function $|\ x \ |$:

$$\begin{align}f(x) = |\ x \ |=\begin{cases}x\quad &, x > 0\\
-x\quad&, x < 0\end{cases}\end{align}$$

### Domain and range

### Symmetry

## Week 2

**Topics:** Polynomials, power functions, rational functions, trigonometric functions, exponential functions.

**Problems:** 1, 2, 3, 6, 7, 10, 14, 17, 20, 23, 27, 32, additional problem set B.

### Polynomials

### Power functions

### Rational functions

### Trigonometric functions

### Exponential functions

## Week 3

**Topics:** Transformations, composite functions, solving power equations.

**Problems:** 9, 10, 11, 12, 13, 16, 31, 32, 33, 34, 36.

### Transformations

### Composite functions

### Solving power equations

## Week 4

**Topics:** Exponential and logarithmic functions (Basisvaardigheden H10), solving power equations.

**Problems:** 1, 3, 4, 6, 7, 10, 11, 12, 15.

### Exponential functions

Exponential functions come in the form of $y=2^x$. Notice that the exponent is a variable, which defines this as an exponential function unlike $y=x^2$ which is a power function.

Graphing these functions yields:

```python
points = np.arange(-5,5.1,0.1)
plot(points, [2**x for x in points], c='b', lw=2)
plot(points, [(1/2)**x for x in points], c='r', lw=2)
grid()
legend(['2^x', '(1/2)^x']);
```

We can transform $2^x$:

* Moving up by adding $5$ gives $2^x+5$.
* Moving to the left by adding $5$ to the exponent gives $2^{x+5}$.

The effect can be seen in the plot:

```python
points = np.arange(-2, 2.1, 0.1)
plot(points, [2**x for x in points], c='b', lw=2)
plot(points, [2**x+5 for x in points], c='g', lw=2)
plot(points, [2**(x+5) for x in points], c='r', lw=2)
grid()
legend(['2^x', '2^x+5', '2^(x+5)']);
```

**Y-intercept**

To find the y-intercept we know that it happens where $x=0$, filling in $x=0$ into the equations in the example above gives:

* $2^0=1$
* $2^0+5=6$
* $2^{(0+5)}=32$

```python
print(2**0)
print(2**0+5)
print(2**(0+5))
```

    1
    6
    32

**Horizontal asymptote**

To find the horizontal asymptote is rather easy. For any exponential function which is not shifted up or down this will be at $y=0$. For the above given examples we get:

* $2^x$ at $y=0$.
* $2^x+5$ at $y=5$.
* $2^{(x+5)}$ at $y=0$.

### Logarithmic functions

The logarithms is the inverse of the exponential function. We can use logarithms to solve for $x$ in $2^x=32$ for example.

**Definition**

The logarithm is defined as $\log_a(b)=c \iff a^c=b$, where $a$ is called the base, and $c$ is the exponent.

```python
points = np.arange(0.01,3.1,0.01)
plot(points, [2**x for x in points])
plot(points, [math.log(x,2) for x in points])
grid()
```

**Rules for logarithms**

* $g^{\log(a)}=a$
* $^g\log(g^a)=a$
* $^g\log(a)+\ ^g\log(b)=\ ^g\log(ab)$
* $^g\log(a)-\ ^g\log(b)=\ ^g\log(\frac{a}{b})$
* $^g\log(a^p)=p\cdot \ ^g\log(a)$
* $^g\log(a) = \frac{^b\log(a)}{^b\log(g)}$ where $b$ is any base.

```python

```


```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## Regressão Linear

A Regressão Linear é usada entender a relação entre variáveis que irão explicar determinado comportamento.

Pode ser usada para:

* Predição de demandas para produtos
* Ajudar a entender o efeitos de certos tratamentos em um paciente
* Entender quais fatores influenciam em uma economia equilibrada

Existem vários tipos de regressão, tais como:

* Regressão Linear Simples
* Regressão Linear Múltipla
* Regressão Logística
* Regressão Polinomial

## 1. Regressão Linear Simples

A regressão linear simples irá mostrar a relação entre duas variáveis. Como a variação na variável independente podem influenciar no valor da dependente.

Pode ser entendida como uma equação da reta, onde existe uma variável Y (dependente) cujo valor está relacionado à X (independente). O objetivo dessa abordagem é chegar ao valor de Y a partir de X.

\begin{equation}
\label{eq:linear_regression}
y_1 = \beta_01 + \beta_1x_1
\end{equation}

* $y_n$  = Variável dependente
* $x_n$  = Variável independente

Por exemplo, a variável dependente é o resultado da predição de demandas de produtos em uma loja.

### 2. O que signfica ser linear?

Todos os termos são uma constante ou um parâmetro multiplicado por uma variável independente.

\begin{equation}
\label{eq:linear_regression}
y_1 = \beta_01 + \beta_1x_1
\end{equation}

* $\beta_0$  = Variável que intercepta o eixo Y
* $\beta_1$  = Variável que que indica a inclinação da reta

De maneira geral, o objetivo do algoritmo é gerar uma reta que faça a divisão entre os pontos, por exemplo, do gráfico abaixo.

```python
np.random.seed(42)
x = np.linspace(-10,10, 20)
y = x * .1 + 3 + np.random.normal(0,2,size=20)

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('Regressão Linear Simples')

ax1.scatter(x, y)
ax2.scatter(x, y)
ax2.scatter(1, 1 * .1 + 3, c='C4', s=100)
ax2.plot(x, x * .1 + 3, c='C1')
```

### 3. Como saber o quão boa é a linha gerada?

A melhor linha é aquela onde todos os pontos possuem a menor distância até a reta gerada.

Uma maneira de se obter o melhor fit, é por meio do cálculo das funções de custo. As mais comuns são:

* Erro absoluto médio
* Erro quadrático médio

É necessário desenvolver um algoritmo para minimizar essas funções.

### 3.1 Gradiente Descendente

O gradiente descendente é como se fosse uma bússola apontando para a direção do mínimo.

A taxa de aprendizagem indica a aceleração.

### 3.2  Erro absoluto médio

Considere que o eixo y representa o valor a ser predito, considerando o valor da variável x. Na figura abaixo, o ponto a ser predito está localizado na coordenada A(x,y) e o ponto identificado pelo algoritmo, aquele que intercepta a reta, é identificado por B(x, ŷ). Logo, o erro para essa predição é a distância vertical do ponto (x,y) a (x, ŷ), ou seja, a subtração dos valores y-ŷ.

Dessa forma, o cálculo do erro absoluto será a média de todos os pontos acima e abaixo da reta.

\begin{equation}
Error=\sum_{i=1}^m |y-ŷ|
\end{equation}

Por que usamos o valor absoluto?

Porque queremos que os pontos positivos não sejam anulados pelos pontos negativos durante o somatório, por isso utilizados |y-ŷ|.

* Se o ponto estiver acima da reta, a distância será y-ŷ
* Caso contrário, será ŷ-y

Sistemas lineares muito complexos com o número de observações maior que a dimensão não possuem resolução, normalmente. É o caso do contexto atual.

Considerando também que

```python
X = [(2, -2), (5, 6), (-4, -4), (-7, 1), (8, 14)]

sum_abs = 0
m = len(X)
for x, y in X:
    y_hat = 1.2 * x + 2
    sum_abs+= abs(y-y_hat)

mae = sum_abs/m
mae
```

    3.88

### 3.3 Erro quadrático médio

O  erro quadrático médio  segue o mesmo processo do  erro absoluto médio , entretanto, ao invés de realizar a distância vertical, é traçado um quadrado.

A área do quadrado será (y-ŷ)², um valor que sempre será positivo. O erro quadrático médio será a média entre as séries de quadrados. À medida que o algoritmo vai diminuindo sua percentagem de erro, a área do quadrado também diminui.

\begin{equation}
Error=\sum_{i=1}^m |y-ŷ|²
\end{equation}

```python
X = [(2, -2), (5, 6), (-4, -4), (-7, 1), (8, 14)]

sum_abs = 0
m = len(X)
for x, y in X:
    y_hat = 1.2 * x + 2
    sum_abs+= abs(y-y_hat)**2

mse = sum_abs/(2*m)
mse
```

    10.692000000000002

## Referências

[Regressão Linear Simples](https://www.ime.usp.br/~fmachado/MAE229/AULA10.pdf)

[Regressão Linear Simples II](https://edisciplinas.usp.br/pluginfile.php/1479289/mod_resource/content/0/regr_lin.pdf)

[Análise de Regressão](https://www.ime.unicamp.br/~nancy/Cursos/me104/regressao.pdf)


# TPSC approximation
Author: [Niklas Witt](mailto:niklas.witt@physik.uni-hamburg.de)

## Theory of TPSC

The Two-Particle Self-Consistent (TPSC) approximation is a non-perturbative semi-analytical method that was first introduced by Vilk and Tremblay {cite:p}`Vilk1997`. TPSC can be used to study magnetic fluctuations, while it also obeys the Mermin-Wagner theorem in two dimensions, i.e., a phase transtition at finite temperatures is prohibited. In addition, the TPSC method satisfies several conservation laws, sum rules and the Pauli principle (actually, it is constructed in a way to fulfill these, since they are used to determine model parameters self-consistently). TPSC is applicable in the weak to intermediate coupling regime, but it breaks down in the strong coupling regime and it cannot describe the Mott transition unlike other non-perturbative methods like Dynamical Mean-Field Theory (DMFT).

For a (pedagogical) review, please have a look at {cite:p}`Allen2004,Tremblay2012` for the single-orbital case implemented here and {cite:p}`Zantout2021` for the more complex multi-orbital theory.

#### Set of TPSC equations
We review the set of equations that need to be solved in the TPSC approximation assuming a one-band Hubbard model with interaction $U$ (it is not so easy to extend TPSC to models with more parameters, since sum rules to determine the additional parameters self-consistently need to be found) in the paramagnetic phase (SU(2) symmetric), i.e., $n = \langle n\rangle = 2n_{\sigma}$ for the electron filling $n$. TPSC is constructed in a way to fulfill certain sum rules and the Pauli principle in the form $\langle n^2\rangle = \langle n\rangle$. The control quantitites are spin and charge correlation function (susceptibilities) which are evaluated in a Random-Phase-Approximation (RPA) like fashion

$$ \chi_{\mathrm{sp}}^{\mathrm{RPA}}(i\nu_m, \boldsymbol{q}) = \frac{\chi_0(i\nu_m, \boldsymbol{q})}{1-U\chi_0(i\nu_m, \boldsymbol{q})}\;,\quad \chi_{\mathrm{ch}}^{\mathrm{RPA}}(i\nu_m, \boldsymbol{q}) = \frac{\chi_0(i\nu_m, \boldsymbol{q})}{1+U\chi_0(i\nu_m, \boldsymbol{q})} $$

with the irreducible susceptibility ("bubble diagram")

$$ \chi_0(i\nu_m, \boldsymbol{q}) = - \frac{T}{N_{\boldsymbol{k}}} \sum_{n,\boldsymbol{k}} G(i\omega_n + i\nu_m, \boldsymbol{k} + \boldsymbol{q})G(i\omega_n, \boldsymbol{k})\;,$$

where $\nu_m = 2n\pi T$ [$\omega_n=(2n+1)\pi T$] and $\boldsymbol{q}$ [$\boldsymbol{k}$] are bosonic [fermionic] Matsubara frequencies and momentum at temperature $T$, $N_{\boldsymbol{k}}$ denotes the number of $\boldsymbol{k}$-points, and $G_0(i\omega_n,\boldsymbol{k}) = [i\omega_n - (\varepsilon_{\boldsymbol{k}}-\mu)]^{-1}$ is the bare (non-interacting) Green function with with single-particle dispersion $\varepsilon_{\boldsymbol{k}}$ and chemical potential $\mu$. Please note that sometimes a factor of 2 is included in the definition of $\chi_0$ leading to slightly different factors in all equations given here. The convolution sum to calculate $\chi_0$ can be easily evaluated by Fourier transforming to imaginary-time and real space, resulting in a simple multiplication

$$ \chi_0(\tau, \boldsymbol{r}) = - G(\tau, \boldsymbol{r})G(-\tau,-\boldsymbol{r}) = G(\tau, \boldsymbol{r})G(\beta-\tau,\boldsymbol{r})\;.$$

In our practical implementation, we will perform this step using the `sparse-ir` package. A similar calculation is necessary to set the chemical potential $\mu$ for fixed electron density $n$, as

$$ n = 2n_{\sigma} = 2 - \frac{2}{N_{\boldsymbol{k}}} \sum_{\boldsymbol{k}} G(\tau=0^+, \boldsymbol{k}) $$

with a factor 2 from spin degeneracy and $0^+ = \lim_{\eta\to 0+} \eta$ needs to be solved by using some root finding algorithm like bisection method or Brent's method. The Fourier transformation to $\tau=0^+$ can be easily performed with the `sparse-ir` package.

The above RPA definition of spin and charge susceptibility violate the Pauli principle. In TPSC, we overcome this problem by introducing two effective, renormalized interactions ("irreducible vertices") $U_{\mathrm{sp}}$ and $U_{\mathrm{ch}}$ that enter spin and charge correlation functions as

$$ \chi_{\mathrm{sp}}(i\nu_m, \boldsymbol{q}) = \frac{\chi_0(i\nu_m, \boldsymbol{q})}{1-U_{\mathrm{sp}}\chi_0(i\nu_m, \boldsymbol{q})}\;,\quad \chi_{\mathrm{ch}}(i\nu_m, \boldsymbol{q}) = \frac{\chi_0(i\nu_m, \boldsymbol{q})}{1+U_{\mathrm{ch}}\chi_0(i\nu_m, \boldsymbol{q})}\,. $$

These two effetive interactions are determined by the two local sum rules

$$
\begin{align}
    2 \frac{T}{N_{\boldsymbol{k}}} \sum_{m,\boldsymbol{q}} \chi_{\mathrm{sp}} &= \left\langle (n_{\uparrow} - n_{\downarrow})^2\right\rangle = n - 2\langle n_{\uparrow}n_{\downarrow}\rangle\;,\\
    2 \frac{T}{N_{\boldsymbol{k}}} \sum_{m,\boldsymbol{q}} \chi_{\mathrm{ch}} &= \left\langle (n_{\uparrow} + n_{\downarrow})^2\right\rangle - \left\langle n_{\uparrow} + n_{\downarrow}\right\rangle^2 =  n + 2\langle n_{\uparrow}n_{\downarrow}\rangle - n^2\;.
\end{align}
$$

Both sum rules can be exactly derived from the Pauli principle ($\langle n^2\rangle = \langle n\rangle$). In principle, we could now determine $U_{\mathrm{sp}}$ and $U_{\mathrm{ch}}$ from local-spin and local-charge sum rule if we knew the double occupancy $\langle n_{\uparrow}n_{\downarrow}\rangle$. TPSC makes the ansatz

$$ U_{\mathrm{sp}}\langle n_{\uparrow}\rangle\langle n_{\downarrow}\rangle = U_{\mathrm{sp}}\frac{n^2}{4} = U\langle n_{\uparrow}n_{\downarrow}\rangle\;,$$

which reproduces Kanamori-Brueckner type screening. The four equations above form a set of self-consistent equations for either $U_{\mathrm{sp}}$ or equivalently $\langle n_{\uparrow}n_{\downarrow}\rangle$. In practice, we treat $U_{\mathrm{sp}}$ as the parameter to be determined self-consistently by inserting the ansatz in the local-spin sum rule. Effectively, we then need to find the root of the function

$$ f(U_{\mathrm{sp}}) = 2\frac{T}{N_{\boldsymbol{k}}} \sum_{m,\boldsymbol{q}}\chi_{\mathrm{sp}}(U_{\mathrm{sp}}) - n + \frac{U_{\mathrm{sp}}}{2U}n^2\;. $$

Afterwards we can calculate the double occupancy $\langle n_{\uparrow}n_{\downarrow}\rangle = \frac{U_{\mathrm{sp}}}{4U} n^2$ and then perform a similar root finding for $U_{\mathrm{ch}}$ from the function

$$ g(U_{\mathrm{ch}}) = 2 \frac{T}{N_{\boldsymbol{k}}} \sum_{m,\boldsymbol{q}} \chi_{\mathrm{ch}}(U_{\mathrm{ch}}) - n - 2\langle n_{\uparrow}n_{\downarrow}\rangle^2 + n^2\;. $$

In TPSC, a self-energy $\Sigma$ can be derived that is calculated from the interaction {cite:p}`Moukouri2000`

$$ V(i\nu_m, \boldsymbol{q}) = \frac{U}{4} \left(3 U_{\mathrm{sp}} \chi_{\mathrm{sp}}(i\nu_m, \boldsymbol{q}) + U_{\mathrm{ch}} \chi_{\mathrm{ch}}(i\nu_m, \boldsymbol{q})\right) + U\;. $$

The self-energy itself is given by a convolution in $(i\omega_n, \boldsymbol{k})$ space

$$ \Sigma(i\omega_n, \boldsymbol{k}) = \frac{T}{N_{\boldsymbol{k}}} \sum_{m,\boldsymbol{q}} V(i\nu_m, \boldsymbol{q}) G(i\omega_n - i\nu_m, \boldsymbol{k} - \boldsymbol{q}) $$

which Fourier transformed to $(\tau,\boldsymbol{r})$ space takes the form

$$ \Sigma(\tau, \boldsymbol{r}) = V(\tau, \boldsymbol{r}) G(\tau, \boldsymbol{r})\;. $$

The interacting Green function is determined by the Dyson equation

$$
\begin{align}
G(i\omega_n,\boldsymbol{k}) &= [G_0^{-1}(i\omega_n,\boldsymbol{k}) - \Sigma(i\omega_n,\boldsymbol{k})]^{-1} \\
& = [i\omega_n - (\varepsilon_{\boldsymbol{k}}-\mu) - \Sigma(i\omega_n,\boldsymbol{k})]^{-1}.
\end{align}
$$

#### Notes on practical implementation
When implementing the TPSC, a few points need to be treated carefully which we adress in the following:

* The constant Hartree term $V_{\mathrm{H}} = U$ in the interaction $V$ and respective self-energy term $\Sigma_H = U n_{\sigma} = U\frac{n}{2}$ can be absorbed into the definition of the chemical potential $\mu$. Otherwise we would have to treat them separately.
* An upper bound for the renormalized spin vertex $U_{\mathrm{sp}}$ exists. Since the denominator spin susceptibility $\chi_{\mathrm{sp}}$ should not diverge, the upper bound is given by the RPA critical interaction value $U_{\mathrm{crit}} = 1/\mathrm{max}\{\chi^0\}$. Mathematically, the function $f(U_{\mathrm{sp}}) = 2\sum \chi_{\mathrm{sp}}(U_{\mathrm{sp}}) - n + \frac{U_{\mathrm{sp}}}{2U}n^2$, from which $U_{\mathrm{sp}}$ is determined, turns unstable for $U_{\mathrm{sp}} \geq U_{\mathrm{crit}}$ (try plotting $f(U_{\mathrm{sp}})$!). At this point, TPSC is not applicable and, e.g., the temperature $T$ is too low or the (unrenormalized) interaction $U$ too large.
* An internal accuracy check $\frac{1}{2}\mathrm{Tr}(\Sigma G) = U \langle n_{\uparrow} n_{\downarrow}\rangle$ can be employed to test the validity of TPSC (not done here).

## Code implementation
We are implementing TPSC for the simple case of a square lattice model with dispersion $\varepsilon_{\boldsymbol{k}} = -2t\,[\cos(k_x) + \cos(k_y)]$ with nearest-neighbor hopping $t$ which sets the energy scale of our system (bandwidth $W = 8t$). First, we load all necessary basic modules that we are going to need in implementing TPSC and visualizing results:

```
import numpy as np
import scipy as sc
import scipy.optimize
from warnings import warn
import sparse_ir
%matplotlib inline
import matplotlib.pyplot as plt
```

#### Parameter setting

```
### System parameters
t    = 1      # hopping amplitude
W    = 8*t    # bandwidth
wmax = 10     # set wmax >= W

T    = 0.1    # temperature
beta = 1/T    # inverse temperature
n    = 0.85   # electron filling, here per spin per lattice site (n=1: half filling)
U    = 4      # Hubbard interaction

### Numerical parameters
nk1, nk2  = 24, 24    # number of k_points along one repiprocal crystal lattice direction k1 = kx, k2 = ky
nk        = nk1*nk2
IR_tol    = 1e-10     # accuary for l-cutoff of IR basis functions
```

#### Generating meshes
We need to generate a $\boldsymbol{k}$-mesh as well as set up the IR basis functions on a sparse $\tau$ and $i\omega_n$ grid. Then we can calculate the dispersion on this mesh.
In addition, we set calculation routines to Fourier transform $k\leftrightarrow r$ and $\tau\leftrightarrow i\omega_n$ (via IR basis).

```
#### Initiate fermionic and bosonic IR basis objects
IR_basis_set = sparse_ir.FiniteTempBasisSet(beta, wmax, eps=IR_tol)

class Mesh:
    """
    Holding class for k-mesh and sparsely sampled imaginary time 'tau' / Matsubara frequency 'iw_n' grids.
    Additionally it defines the Fourier transform routines 'r <-> k'  and 'tau <-> l <-> wn'.
    """
    def __init__(self,IR_basis_set,nk1,nk2):
        self.IR_basis_set = IR_basis_set

        # generate k-mesh and dispersion
        self.nk1, self.nk2, self.nk = nk1, nk2, nk1*nk2
        self.k1, self.k2 = np.meshgrid(np.arange(self.nk1)/self.nk1, np.arange(self.nk2)/self.nk2)
        self.ek = -2*t*( np.cos(2*np.pi*self.k1) + np.cos(2*np.pi*self.k2) ).reshape(nk)

        # lowest Matsubara frequency index
        self.iw0_f = np.where(self.IR_basis_set.wn_f == 1)[0][0]
        self.iw0_b = np.where(self.IR_basis_set.wn_b == 0)[0][0]

        ### Generate a frequency-momentum grid for iw_n and ek (in preparation for calculating the Green function)
        # frequency mesh (for Green function)
        self.iwn_f = 1j * self.IR_basis_set.wn_f * np.pi * T
        self.iwn_f_ = np.tensordot(self.iwn_f, np.ones(nk), axes=0)

        # ek mesh
        self.ek_ = np.tensordot(np.ones(len(self.iwn_f)), self.ek, axes=0)

    def smpl_obj(self, statistics):
        """ Return sampling object for given statistic """
        smpl_tau = {'F': self.IR_basis_set.smpl_tau_f, 'B': self.IR_basis_set.smpl_tau_b}[statistics]
        smpl_wn  = {'F': self.IR_basis_set.smpl_wn_f,  'B': self.IR_basis_set.smpl_wn_b }[statistics]
        return smpl_tau, smpl_wn

    def tau_to_wn(self, statistics, obj_tau):
        """ Fourier transform from tau to iw_n via IR basis """
        smpl_tau, smpl_wn = self.smpl_obj(statistics)

        obj_tau = obj_tau.reshape((smpl_tau.tau.size, self.nk1, self.nk2))
        obj_l   = smpl_tau.fit(obj_tau, axis=0)
        obj_wn  = smpl_wn.evaluate(obj_l, axis=0).reshape((smpl_wn.wn.size, self.nk))
        return obj_wn

    def wn_to_tau(self, statistics, obj_wn):
        """ Fourier transform from tau to iw_n via IR basis """
        smpl_tau, smpl_wn = self.smpl_obj(statistics)

        obj_wn  = obj_wn.reshape((smpl_wn.wn.size, self.nk1, self.nk2))
        obj_l   = smpl_wn.fit(obj_wn, axis=0)
        obj_tau = smpl_tau.evaluate(obj_l, axis=0).reshape((smpl_tau.tau.size, self.nk))
        return obj_tau

    def k_to_r(self,obj_k):
        """ Fourier transform from k-space to real space """
        obj_k = obj_k.reshape(-1, self.nk1, self.nk2)
        obj_r = np.fft.fftn(obj_k,axes=(1,2))
        obj_r = obj_r.reshape(-1, self.nk)
        return obj_r

    def r_to_k(self,obj_r):
        """ Fourier transform from real space to k-space """
        obj_r = obj_r.reshape(-1, self.nk1, self.nk2)
        obj_k = np.fft.ifftn(obj_r,axes=(1,2))/self.nk
        obj_k = obj_k.reshape(-1, self.nk)
        return obj_k
```

#### TPSC solver
We wrap the calculation steps of TPSC (i.e. determining $U_{\mathrm{sp}},U_{\mathrm{ch}}$) in the `TPSCSolver` class. We use the `Mesh` class defined above to perform calculation steps.

```
class TPSCSolver:
    def __init__(self, mesh, U, n, U_sfc_tol=1e-12, verbose=True):
        """
        Solver class to calculate the TPSC method.
        After initializing the Solver by `solver = TPSCSolver(mesh, U, n, **kwargs)` it
        can be run by `solver.solve()`.
        """
        ## set internal parameters for the solve
        self.U = U
        self.n = n
        self.mesh = mesh
        self.U_sfc_tol = U_sfc_tol
        self.verbose = verbose

        ## set initial Green function and irreducible susceptibility
        # NOT running the TPSCSolver.solve instance corresponds to staying on RPA level
        self.sigma = 0

        self.mu = 0
        self.mu_calc()

        self.gkio_calc(self.mu)
        self.grit_calc()
        self.ckio_calc()

        # determine critical U_crit = 1/max(chi0) as an upper bound to U_sp
        self.U_crit = 1/np.amax(self.ckio.real)

    #%%%%%%%%%%% Solving instance
    def solve(self):
        """
        Determine spin and charge vertex self-consistently from sum rules and calculate self-energy.
        """
        # determine spin vertex U_sp
        self.spin_vertex_calc()

        # set double occupancy from Kanamori-Bruckner screening
        self.docc_calc()

        # determine charge vertex U_ch
        self.charge_vertex_calc()

        # set spin and charge susceptibility
        self.chi_spin   = self.RPA_term_calc( self.U_sp)
        self.chi_charge = self.RPA_term_calc(-self.U_ch)

        # calculate interaction, self-energy and interacting Green function
        self.V_calc()
        self.sigma_calc()
        self.mu_calc()
        self.gkio_calc(self.mu)

    #%%%%%%%%%%% Calculation steps for self.energy
    def gkio_calc(self, mu):
        """ Calculate Green function G(iw,k) """
        self.gkio = (self.mesh.iwn_f_ - (self.mesh.ek_ - mu) - self.sigma)**(-1)

    def grit_calc(self):
        """ Calculate real space Green function G(tau,r) [for calculating chi0 and sigma] """
        # Fourier transform
        grit = self.mesh.k_to_r(self.gkio)
        self.grit = self.mesh.wn_to_tau('F', grit)

    def ckio_calc(self):
        """ Calculate irreducible susciptibility chi0(iv,q) """
        ckio = self.grit * self.grit[::-1, :]

        # Fourier transform
        ckio = self.mesh.r_to_k(ckio)
        self.ckio = self.mesh.tau_to_wn('B', ckio)

    def V_calc(self):
        """ Calculate interaction V(tau,r) from RPA-like spin and charge susceptibility """
        V = self.U/4 * (3*self.U_sp*self.chi_spin + self.U_ch*self.chi_charge)
        # Constant Hartree Term V ~ U needs to be treated extra, since they cannot be modeled by the IR basis.
        # In the single-band case, the Hartree term can be absorbed into the chemical potential.

        # Fourier transform
        V = self.mesh.k_to_r(V)
        self.V = self.mesh.wn_to_tau('B', V)

    def sigma_calc(self):
        """ Calculate self-energy Sigma(iwn,k) """
        sigma = self.V * self.grit

        # Fourier transform
        sigma = self.mesh.r_to_k(sigma)
        self.sigma = self.mesh.tau_to_wn('F', sigma)

    #%%%%%%%%%%% Determining spin and charge vertex
    def RPA_term_calc(self, U):
        """ Set RPA-like susceptibility """
        chi_RPA = self.ckio / (1 - U*self.ckio)
        return chi_RPA

    def chi_qtrace_calc(self, U):
        """ Calculate (iv_m, q) trace of chi_RPA term """
        # chi_qtrace = sum_(m,q) chi(iv_m,q)
        chi_RPA = self.RPA_term_calc(U)
        chi_trace = np.sum(chi_RPA, axis=1)/self.mesh.nk
        chi_trace_l  = self.mesh.IR_basis_set.smpl_wn_b.fit(chi_trace)
        chi_trace = self.mesh.IR_basis_set.basis_b.u(0)@chi_trace_l
        return chi_trace.real

    def docc_calc(self):
        """ Calculate double occupancy from Kanamori-Bruckner type screening """
        self.docc = 0.25 * self.U_sp/self.U * self.n**2

    def spin_vertex_calc(self):
        """ Determine self-consistently from sum rule """
        # interval [U_a, U_b] for root finding
        U_a = 0
        U_b = np.floor(self.U_crit*100)/100

        chi_trace = self.chi_qtrace_calc
        sfc_eq = lambda U_sp : 2*chi_trace(U_sp) - self.n + 0.5*(U_sp/self.U)*self.n**2

        if sfc_eq(U_b) > 0:
            self.U_sp = sc.optimize.brentq(sfc_eq, U_a, U_b, rtol = self.U_sfc_tol)
        else:
            warn("System underwent phase transition, U^sp > U_crit = {}! U is too large or T too low for given doping.".format(self.U_crit))

    def charge_vertex_calc(self):
        """ Determine self-consistently from sum rule """
        # interval [U_a, U_b] for root finding
        U_a = 0
        U_b = 100

        chi_trace = self.chi_qtrace_calc
        sfc_eq = lambda U_ch : 2*chi_trace(-U_ch) - self.n + (1 - 2*self.docc)*self.n**2

        self.U_ch = sc.optimize.brentq(sfc_eq, U_a, U_b, rtol = self.U_sfc_tol)

    #%%%%%%%%%%% Setting chemical potential mu
    def calc_electron_density(self, mu):
        """ Calculate chemical potential mu from Green function """
        self.gkio_calc(mu)
        gio  = np.sum(self.gkio,axis=1)/self.mesh.nk
        g_l  = self.mesh.IR_basis_set.smpl_wn_f.fit(gio)
        g_tau0 = self.mesh.IR_basis_set.basis_f.u(0)@g_l

        n  = 1 + np.real(g_tau0)
        n  = 2*n #for spin
        return n

    def mu_calc(self):
        """ Find chemical potential for a given filling n0 via brentq root finding algorithm """
        n_calc = self.calc_electron_density
        n0 = self.n
        f  = lambda mu : n_calc(mu) - n0

        self.mu = sc.optimize.brentq(f, np.amax(self.mesh.ek)*3, np.amin(self.mesh.ek)*3)
```

### Execute TPSC solver

```
# initialize calculation
IR_basis_set = sparse_ir.FiniteTempBasisSet(beta, wmax, eps=IR_tol)
mesh = Mesh(IR_basis_set, nk1, nk2)
solver = TPSCSolver(mesh, U, n)

# perform TPSC calculations
solver.solve()
```

#### Visualize results

```
# plot 2D k-dependence of lowest Matsubara frequency of e.g. green function
plt.pcolormesh(2*mesh.k1.reshape(nk1,nk2), 2*mesh.k2.reshape(nk1,nk2), np.real(solver.gkio[mesh.iw0_f].reshape(mesh.nk1,mesh.nk2)), shading='auto')
ax = plt.gca()
ax.set_xlabel('$k_x/\pi$')
ax.set_xlim([0,2])
ax.set_ylabel('$k_y/\pi$')
ax.set_ylim([0,2])
ax.set_aspect('equal')
ax.set_title('Re $G(k,i\omega_0)$')
plt.colorbar()
plt.show()
```

```
# plot 2D k-dependence of lowest Matsubara frequency of e.g. self-energy
plt.pcolormesh(2*mesh.k1.reshape(nk1,nk2), 2*mesh.k2.reshape(nk1,nk2), np.imag(solver.sigma[mesh.iw0_f].reshape(mesh.nk1,mesh.nk2)), shading='auto')
ax = plt.gca()
ax.set_xlabel('$k_x/\pi$')
ax.set_xlim([0,2])
ax.set_ylabel('$k_y/\pi$')
ax.set_ylim([0,2])
ax.set_aspect('equal')
ax.set_title('Im $\Sigma(k,i\omega_0)$')
plt.colorbar()
plt.show()
```

```
# plot 2D k-dependence of lowest Matsubara frequency of e.g. chi_spin
plt.pcolormesh(2*mesh.k1.reshape(nk1,nk2), 2*mesh.k2.reshape(nk1,nk2), np.real(solver.chi_spin[mesh.iw0_b].reshape(mesh.nk1,mesh.nk2)), shading='auto')
ax = plt.gca()
ax.set_xlabel('$k_x/\pi$')
ax.set_xlim([0,2])
ax.set_ylabel('$k_y/\pi$')
ax.set_ylim([0,2])
ax.set_aspect('equal')
ax.set_title('$\chi_{\mathrm{sp}}(k,i\nu_0)$')
plt.colorbar()
plt.show()
```

## Example: Interaction dependent renormalization
As a simple example demonstration of our `sparse-ir` TPSC code developed above, we will reproduce Fig. 2 of {cite:p}`Vilk1997`. It shows the $U$ dependence of renormalized/effective spin and charge interactions $U_{\mathrm{sp}}$ and $U_{\mathrm{ch}}$ (irreducible vertices) at half filling $n=1$ and $T>T_{\mathrm{crit}}$ for all considered $U$ (i.e. $U_{\mathrm{sp}}<U_{\mathrm{crit}}$ is ensured).

You can simply execute the following two code blocks which will first perform the calculation and then generate a figure like in the reference above.

```
#%%%%%%%%%%%%%%% Parameter settings
print('Initialization...')
# system parameters
t = 1         # hopping amplitude
n = 1         # electron filling, here per spin per lattice site (n=1: half filling)
T = 0.4       # temperature
beta =1/T
U_array = np.linspace(1e-10,5,51)  # Hubbard interaction

W    = 8*t    # bandwidth
wmax = 10     # set wmax >= W

# numerical parameters
nk1, nk2  = 24, 24    # k-mesh sufficiently dense!
nk        = nk1*nk2
IR_tol    = 1e-8      # accuary for l-cutoff of IR basis functions

# initialize meshes
IR_basis_set = sparse_ir.FiniteTempBasisSet(beta, wmax, eps=IR_tol)
mesh = Mesh(IR_basis_set, nk1, nk2)

# set initial self_energy - will be set to previous calculation step afterwards
sigma_init = 0

# empty arrays for results later
U_sp_array = np.empty((len(U_array)))
U_ch_array = np.empty((len(U_array)))

#%%%%%%%%%%%%%%% Calculations for different U values
print("Start TPSC loop...")
for U_it, U in enumerate(U_array):
    #print("Now: U = {:.1f}".format(U))

    # TPSC solver
    solver = TPSCSolver(mesh, U, n, verbose=False)
    solver.solve()

    # save data for plotting
    U_sp_array[U_it] = solver.U_sp
    U_ch_array[U_it] = solver.U_ch
print("Finished. Plotting now.")

#%%%%%%%%%%%%%%%% Plot results
plt.plot(U_array, U_ch_array, '-', label='$U_{\mathrm{ch}}$')
plt.plot(U_array, U_sp_array, '-', label='$U_{\mathrm{sp}}$')
ax = plt.gca()
ax.set_xlabel('$U$', fontsize=12)
ax.set_xlim([0,5])
ax.set_ylabel('Interaction', fontsize=12)
ax.set_ylim([0,20])
ax.legend(frameon=False, fontsize=12)
```

```

```


+ This notebook is part of lecture 17 *Orthogonal matrices and Gram-Scmidt* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, symbols, Matrix, sin, cos, sqrt, Rational, GramSchmidt
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

```python
theta = symbols('theta')
```

# Orthogonal basis
# Orthogonal matrix
# Gram-Schmidt

## Orthogonal basis

* Here we mean vectors *q*1,*q*2,...,*q*n

* We actually mean orthonormal vectors (for orthogonal or perpendicular and of unit length / normalized)
* Vectors that are orthogonal have a dot product equal to zero
    * If they are orthogonal
$$ {q}_{i}^{T}{q}_{j}={0} $$
    * If they are not
$$ {q}_{i}^{T}{q}_{j}\neq{0} $$

## Orthogonal matrix

* We can now put these (column) basis vectors into a matrix Q

* This brings about
$$ {Q}^{T}{Q}={I} $$

* In the case of the matrix Q being square the word *orthogonal matrix* is used
* When it is square we can calculate the inverse making
$$ {Q}^{T}={Q}^{-1} $$

* Consider the following permutation matrix with orthonormal column vectors

```python
Q = Matrix([[0, 0, 1], [1, 0, 0], [0, 1, 0]])
Q, Q.transpose()
```

$$\begin{pmatrix}\left[\begin{matrix}0 & 0 & 1\\1 & 0 & 0\\0 & 1 & 0\end{matrix}\right], & \left[\begin{matrix}0 & 1 & 0\\0 & 0 & 1\\1 & 0 & 0\end{matrix}\right]\end{pmatrix}$$

* In this example the transpose also contains orthonormal column vectors
* Multiplication gives the identity matrix

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

* Consider this example

```python
Q = Matrix([[cos(theta), -sin(theta)], [sin(theta), cos(theta)]])
Q, Q.transpose()
```

$$\begin{pmatrix}\left[\begin{matrix}\cos{\left (\theta \right )} & - \sin{\left (\theta \right )}\\\sin{\left (\theta \right )} & \cos{\left (\theta \right )}\end{matrix}\right], & \left[\begin{matrix}\cos{\left (\theta \right )} & \sin{\left (\theta \right )}\\- \sin{\left (\theta \right )} & \cos{\left (\theta \right )}\end{matrix}\right]\end{pmatrix}$$

* The two column vectors are orthogonal and the length of each column vector is 1
* It is thus an orthogonal matrix

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}\sin^{2}{\left (\theta \right )} + \cos^{2}{\left (\theta \right )} & 0\\0 & \sin^{2}{\left (\theta \right )} + \cos^{2}{\left (\theta \right )}\end{matrix}\right]$$

* The example below certainly has orthogonal column vectors, but they are not of unit length
$$ {Q}=\begin{bmatrix} 1 & 1 \\ 1 & {-1} \end{bmatrix} $$

* Well, we can change them into unit vectors by dividing each component by the length of that vector
$$ \sqrt { { \left( 1 \right)  }^{ 2 }+{ \left( 1 \right)  }^{ 2 } } =\sqrt { 2 } \\ \sqrt { { \left( 1 \right)  }^{ 2 }+{ \left( -1 \right)  }^{ 2 } } =\sqrt { 2 }  $$
$$ Q=\frac { 1 }{ \sqrt { 2 }  } \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} $$

* As it stands QTQ is not the identity matrix

```python
Q = Matrix([[1, 1], [1, -1]])
Q.transpose() * Q
```

$$\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right]$$

* Turning it into an orthogonal matrix

```python
Q = (1 / sqrt(2)) * Matrix([[1, 1], [1, -1]])
Q
```

$$\left[\begin{matrix}\frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\\\frac{\sqrt{2}}{2} & - \frac{\sqrt{2}}{2}\end{matrix}\right]$$

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

* Consider this example with orthogonal (but not orthonormal) column vectors

```python
Q = Matrix([[1, 1, 1, 1], [1, -1, 1, -1], [1, 1, -1, -1], [1, -1, -1, 1]])
Q
```

$$\left[\begin{matrix}1 & 1 & 1 & 1\\1 & -1 & 1 & -1\\1 & 1 & -1 & -1\\1 & -1 & -1 & 1\end{matrix}\right]$$

* Again, as it stands QTQ is not the identity matrix

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}4 & 0 & 0 & 0\\0 & 4 & 0 & 0\\0 & 0 & 4 & 0\\0 & 0 & 0 & 4\end{matrix}\right]$$

* But turning it into an orthogonal matrix works

```python
Q = Rational(1, 2) * Matrix([[1, 1, 1, 1], [1, -1, 1, -1], [1, 1, -1, -1], [1, -1, -1, 1]])
# Rational() creates a mathematical fraction instead of a decimal
Q
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & \frac{1}{2} & \frac{1}{2}\\\frac{1}{2} & - \frac{1}{2} & \frac{1}{2} & - \frac{1}{2}\\\frac{1}{2} & \frac{1}{2} & - \frac{1}{2} & - \frac{1}{2}\\\frac{1}{2} & - \frac{1}{2} & - \frac{1}{2} & \frac{1}{2}\end{matrix}\right]$$

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$$

* Consider this matrix Q with orthogonal column vectors, but that is not square

```python
Q = Rational(1, 3) * Matrix([[1, -2], [2, -1], [2, 2]])
Q
```

$$\left[\begin{matrix}\frac{1}{3} & - \frac{2}{3}\\\frac{2}{3} & - \frac{1}{3}\\\frac{2}{3} & \frac{2}{3}\end{matrix}\right]$$

* We now have a matrix with two column vectors that are normalized and orthogonal to each other and they form a basis for a plane (subspace) in ℝ3

* There must be a third column matrix of unit length, orthogonal to the other two so we end up with an orthogonal matrix

```python
Q = Rational(1, 3) * Matrix([[1, -2, 2], [2, -1, -2], [2, 2, 1]])
Q
```

$$\left[\begin{matrix}\frac{1}{3} & - \frac{2}{3} & \frac{2}{3}\\\frac{2}{3} & - \frac{1}{3} & - \frac{2}{3}\\\frac{2}{3} & \frac{2}{3} & \frac{1}{3}\end{matrix}\right]$$

```python
Q.transpose() * Q
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

* Let's make use of these matrices with orthonormal columns (which we will always denote with a letter Q) and project them onto their columnspace
* What would the projection matrix be?
$$ Q\underline{x}={b} \\ {P}={Q}{\left({Q}^{T}{Q}\right)}^{-1}{Q}^{T} $$

* Remember, though that for matrices with orthonormal column vectors we have QTQ is the identity matrix and we have
$$ {P}={Q}{Q}^{T} $$

* If additionally, Q is square, then we have independent columns and the columnspace contain the whole space ℝn and the projection matrix is the identity matrix in *n*
    * Remember QT = Q-1 in these cases making it easy to see that we get the identity matrix
    * Remember also that the projection matrix is symmetric
    * Lastly the projection matrix has the property of squaring it leaves us in the same spot, so here we will have (QQT)2=QQT

* All of this has the final consequence that
$$ {Q}^{T}Q\hat{x}={Q}^{T}\underline{b} \\ \hat{x}={Q}^{T}\underline{b} \\ \hat{x}_{i}={q}_{i}^{T}{b} $$

## Gram-Schmidt

* All of the above makes things quite easy, so we should try and create orthogonal matrices

* Good, let's start with two independent vectors **a** and **b** and try and create two orthogonal vectors **A** and **B** and then create two orthonormal vectors
$$ { q }_{ 1 }=\frac { A }{ \left\| A \right\|  } \\ { q }_{ 2 }=\frac { B }{ \left\| B \right\|  }  $$

* We can choose one of them as our initial vector, say **a** = **A**, so we have to get an orthogonal projection (to **a**) for **B**
* This is what we previously called the error vector **e**
$$ \underline{e}=\underline{b}-\underline{p} $$
* Remembering how to get **p** we have the following
$$ B=\underline { b } -\frac { { A }^{ T }\underline { b }  }{ { A }^{ T }A } A $$

* Let's do an example

```python
a = Matrix([1, 1, 1])
b = Matrix([1, 0, 2])
a, b
```

$$\begin{pmatrix}\left[\begin{matrix}1\\1\\1\end{matrix}\right], & \left[\begin{matrix}1\\0\\2\end{matrix}\right]\end{pmatrix}$$

```python
A = a
A
```

$$\left[\begin{matrix}1\\1\\1\end{matrix}\right]$$

```python
A.transpose() * b
```

$$\left[\begin{matrix}3\end{matrix}\right]$$

```python
A.transpose() * A
```

$$\left[\begin{matrix}3\end{matrix}\right]$$

```python
B = b - A
B
```

$$\left[\begin{matrix}0\\-1\\1\end{matrix}\right]$$

* Checking that they are perpendicular

```python
A.transpose() * B
```

$$\left[\begin{matrix}0\end{matrix}\right]$$

* Now we have to create Q by turning **A** and **B** into unit vectors and place them in the same matrix

```python
A.normalized() # Easy way no normalize a matrix
```

$$\left[\begin{matrix}\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\end{matrix}\right]$$

```python
B.normalized()
```

$$\left[\begin{matrix}0\\- \frac{\sqrt{2}}{2}\\\frac{\sqrt{2}}{2}\end{matrix}\right]$$

```python
Q = Matrix([[sqrt(3) / 3, 0], [sqrt(3) / 3, -sqrt(2) / 2], [sqrt(3) / 3, sqrt(2) / 2]])
Q
```

$$\left[\begin{matrix}\frac{\sqrt{3}}{3} & 0\\\frac{\sqrt{3}}{3} & - \frac{\sqrt{2}}{2}\\\frac{\sqrt{3}}{3} & \frac{\sqrt{2}}{2}\end{matrix}\right]$$

* The columnspace of the original matrix (of two column vectors) and Q are the same

* In python™ we can use the following code

```python
# The column matrices (independant orthogonal column vectors) are entered indivisually inside square bracket []
A = [Matrix([1, 1, 1]), Matrix([1, 0, 2])]
A
```

$$\begin{bmatrix}\left[\begin{matrix}1\\1\\1\end{matrix}\right], & \left[\begin{matrix}1\\0\\2\end{matrix}\right]\end{bmatrix}$$

```python
Q = GramSchmidt(A, True) # The True argument normalizes the columns
Q
```

$$\begin{bmatrix}\left[\begin{matrix}\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\end{matrix}\right], & \left[\begin{matrix}0\\- \frac{\sqrt{2}}{2}\\\frac{\sqrt{2}}{2}\end{matrix}\right]\end{bmatrix}$$

## Example problems

### Example problem 1

* Create an orthogonal matrix from the following matrix
$$ \begin{bmatrix} 1 & 2 & 4 \\ 0 & 0 & 5 \\ 0 & 3 & 6 \end{bmatrix} $$

#### Solution

```python
A = [Matrix([1, 0, 0]), Matrix([2, 0, 3]), Matrix([4, 5, 6])]
A
```

$$\begin{bmatrix}\left[\begin{matrix}1\\0\\0\end{matrix}\right], & \left[\begin{matrix}2\\0\\3\end{matrix}\right], & \left[\begin{matrix}4\\5\\6\end{matrix}\right]\end{bmatrix}$$

```python
Q = GramSchmidt(A, True)
Q
```

$$\begin{bmatrix}\left[\begin{matrix}1\\0\\0\end{matrix}\right], & \left[\begin{matrix}0\\0\\1\end{matrix}\right], & \left[\begin{matrix}0\\1\\0\end{matrix}\right]\end{bmatrix}$$

* We can also consider QR-factorization

```python
from sympy.mpmath import matrix, qr
```

```python
A = matrix([[1, 2, 4], [0, 0, 5], [0, 3, 6]])
print(A)
```

    [1.0  2.0  4.0]
    [0.0  0.0  5.0]
    [0.0  3.0  6.0]

```python
Q, R = qr(A)
```

```python
print(Q)
```

    [1.0   0.0   0.0]
    [0.0   0.0  -1.0]
    [0.0  -1.0   0.0]

```python
print(R)
```

    [1.0   2.0   4.0]
    [0.0  -3.0  -6.0]
    [0.0   0.0  -5.0]

```python

```


```python
%matplotlib inline

import os
import unicodedata
import pandas as pd

import numpy
import sympy

import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import matplotlib.animation as animation

from scipy.interpolate import make_interp_spline, BSpline
from scipy.ndimage.filters import gaussian_filter1d

from IPython.display import HTML
from IPython import display

s = sympy.symbols
```

```python
graphs = []
log_path = "/home/datta/lab/Rethinking-Binarized-Neural-Network-Optimization/hyperparameter_logs/acc/"
files = ["pit6g2","pit6g3","pit6g4"]
for file in os.listdir(log_path):
    df = pd.read_json(log_path+file)#+".json")
    df = df[[1, 2]]
    df[2] = gaussian_filter1d(df[2], sigma=2)

    df['label'] = file.split(".json")[0]
    graphs.append(df)

def best_acc(df):
    return df[2].max()

sorted_graphs = sorted(graphs, key = best_acc, reverse=True)
```

```python
%matplotlib inline
fig, ax = plt.subplots(1, 1)
_ = plt.rcParams['figure.figsize'] = (15, 5)

labels = []
for df in sorted_graphs:
    _ = ax.plot(df[1]/909, df[2])
    labels.append(df['label'][0])

#     print(df['label'][0])
# _ = plt.ylim(0.8, 1)

_ = plt.grid(True)
_ = plt.xlabel("Number of Epochs")
_ = plt.ylabel(r'$\pi$'+ " - Flip Ratio")

labels = [
    r'$\gamma = 10^{-2}$',
    r'$\gamma = 10^{-3}$',
    r'$\gamma = 10^{-4}$'
]
#03 632 53
# _ = ax.legend(labels, loc=4)

```

```python
# fig.savefig((str(time)+".png")
```

    1573058620.5257473

```python
df
```

      1
      2
      label

      0
      4500
      0.1570
      va3

      1
      9000
      0.2194
      va3

      2
      13500
      0.2478
      va3

      3
      18000
      0.2610
      va3

      4
      22500
      0.1246
      va3

      5
      27000
      0.2058
      va3

      6
      31500
      0.1958
      va3

      7
      36000
      0.2268
      va3

      8
      40500
      0.1138
      va3

      9
      45000
      0.2076
      va3

      10
      49500
      0.1542
      va3

      11
      54000
      0.1430
      va3

      12
      58500
      0.1264
      va3

      13
      63000
      0.0976
      va3

      14
      67500
      0.1486
      va3

      15
      72000
      0.1246
      va3

      16
      76500
      0.1770
      va3

      17
      81000
      0.0976
      va3

      18
      85500
      0.1788
      va3

      19
      90000
      0.2134
      va3

```python

```