# **CS224W - Colab 5**

In this Colab, we will shift our focus from homogenous graphs to heterogeneous graphs. Heterogeneous graphs extend the traditional homogenous graphs that we have been working with by incorperating different node and edge types. This additional information allows us to extend the graph neural nework models that we have worked with before. Namely, we can apply heterogenous message passing, where different message types now exist between different node and edge type relationships.

In this notebook, we will first learn how to transform NetworkX graphs into DeepSNAP representations. Then, we will dive deeper into how DeepSNAP stores and represents heterogeneous graphs as PyTorch Tensors.

Lastly, we will build our own heterogenous graph neural netowrk models using PyTorch Geometric and DeepSNAP. We will then apply our models for a node property prediction task; specifically, we will evaluate these models on the heterogeneous ACM node prediction dataset.

**Note**: Make sure to **sequentially run all the cells in each section**, so that the intermediate variables / packages will carry over to the next cell

Have fun and good luck on Colab 5 :)

# Device
You might need to use GPU for this Colab.

Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**.

# Installation

```python
# Install torch geometric
import os
if 'IS_GRADESCOPE_ENV' not in os.environ:
  !pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu111.html
  !pip install torch-sparse -f https://data.pyg.org/whl/torch-1.10.0+cu111.html
  !pip install torch-geometric
  !pip install -q git+https://github.com/snap-stanford/deepsnap.git
  !pip install -U -q PyDrive
```

    Looking in links: https://data.pyg.org/whl/torch-1.10.0+cu111.html
    Collecting torch-scatter
      Downloading https://data.pyg.org/whl/torch-1.10.0%2Bcu113/torch_scatter-2.0.9-cp37-cp37m-linux_x86_64.whl (7.9 MB)
         |████████████████████████████████| 7.9 MB 5.2 MB/s
    Installing collected packages: torch-scatter
    Successfully installed torch-scatter-2.0.9
    Looking in links: https://data.pyg.org/whl/torch-1.10.0+cu111.html
    Collecting torch-sparse
      Downloading https://data.pyg.org/whl/torch-1.10.0%2Bcu113/torch_sparse-0.6.12-cp37-cp37m-linux_x86_64.whl (3.5 MB)
         |████████████████████████████████| 3.5 MB 4.4 MB/s
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from torch-sparse) (1.4.1)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scipy->torch-sparse) (1.19.5)
    Installing collected packages: torch-sparse
    Successfully installed torch-sparse-0.6.12
    Collecting torch-geometric
      Downloading torch_geometric-2.0.3.tar.gz (370 kB)
         |████████████████████████████████| 370 kB 4.2 MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.19.5)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (4.62.3)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.4.1)
    Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.6.3)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.0.2)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.23.0)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (1.1.5)
    Collecting rdflib
      Downloading rdflib-6.1.1-py3-none-any.whl (482 kB)
         |████████████████████████████████| 482 kB 50.2 MB/s
    Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (0.4)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (2.11.3)
    Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (3.0.6)
    Collecting yacs
      Downloading yacs-0.1.8-py3-none-any.whl (14 kB)
    Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from torch-geometric) (3.13)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->torch-geometric) (2.0.1)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->torch-geometric) (2.8.2)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->torch-geometric) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->torch-geometric) (1.15.0)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from rdflib->torch-geometric) (4.10.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from rdflib->torch-geometric) (57.4.0)
    Collecting isodate
      Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
         |████████████████████████████████| 41 kB 553 kB/s
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->rdflib->torch-geometric) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->rdflib->torch-geometric) (3.10.0.2)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torch-geometric) (3.0.4)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->torch-geometric) (1.1.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->torch-geometric) (3.0.0)
    Building wheels for collected packages: torch-geometric
      Building wheel for torch-geometric (setup.py) ... done
      Created wheel for torch-geometric: filename=torch_geometric-2.0.3-py3-none-any.whl size=581968 sha256=c63ac66b814ba4e357e9d6ff9dce6849bf7b4b8c3b6ea929ff85a09fa80b0d9c
      Stored in directory: /root/.cache/pip/wheels/c3/2a/58/87ce0508964d4def1aafb92750c4f3ac77038efd1b9a89dcf5
    Successfully built torch-geometric
    Installing collected packages: isodate, yacs, rdflib, torch-geometric
    Successfully installed isodate-0.6.1 rdflib-6.1.1 torch-geometric-2.0.3 yacs-0.1.8
      Building wheel for deepsnap (setup.py) ... done

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  !nvcc --version
  !python -c "import torch; print(torch.version.cuda)"
```

    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2020 NVIDIA Corporation
    Built on Mon_Oct_12_20:09:46_PDT_2020
    Cuda compilation tools, release 11.1, V11.1.105
    Build cuda_11.1.TC455_06.29190527_0
    11.1

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  import torch
  print(torch.__version__)
  import torch_geometric
  print(torch_geometric.__version__)
```

    1.10.0+cu111
    2.0.3

# DeepSNAP Basics

In previous Colabs we used both of graph class (NetworkX) and tensor (PyG) representations of graphs separately. The graph class `nx.Graph` provides rich analysis and manipulation functionalities, such as the clustering coefficient and PageRank. To feed the graph into the model, we need to transform the graph into tensor representations including edge tensor `edge_index` and node attributes tensors `x` and `y`. But only using tensors (as the graphs formatted in PyG `datasets` and `data`) will make many graph manipulations and analysis less efficient and harder. So, in this Colab we will use DeepSNAP which combines both representations and offers a full pipeline for GNN training / validation / testing.

In general, [DeepSNAP](https://github.com/snap-stanford/deepsnap) is a Python library to assist efficient deep learning on graphs. DeepSNAP features in its support for flexible graph manipulation, standard pipeline, heterogeneous graphs and simple API.

1. DeepSNAP is easy to be used for the sophisticated graph manipulations, such as feature computation, pretraining, subgraph extraction etc. during/before the training.
2. In most frameworks, standard pipelines for node, edge, link, graph-level tasks under inductive or transductive settings are left to the user to code. In practice, there are additional design choices involved (such as how to split dataset for link prediction). DeepSNAP provides such a standard pipeline that greatly saves repetitive coding efforts, and enables fair comparision for models.
3. Many real-world graphs are heterogeneous graphs. But packages support for heterogeneous graphs, including data storage and flexible message passing, is lacking. DeepSNAP provides an efficient and flexible heterogeneous graph that supports both the node and edge heterogeneity.

[DeepSNAP](https://github.com/snap-stanford/deepsnap) is a newly released project and it is still under development. If you find any bugs or have any improvement ideas, feel free to raise issues or create pull requests on the GitHub directly :)

In this Colab, we will focus on learning using Heterogeneous Graphs. Not many libraries are able to handle heterogeneous graphs, but DeepSNAP handles them quite elegantly, which is why we're introducing it here!

# 1) DeepSNAP Heterogeneous Graph

First, we will explore how to transform a NetworkX graph into the format supported by DeepSNAP.

In DeepSNAP we have three levels of attributes. We can have **node level** attributes including `node_feature` and `node_label`. The other two levels of attributes are graph and edge attributes. The usage is similar to the node level one except that the feature becomes `edge_feature` or `graph_feature` and label becomes `edge_label` or `graph_label` etc.

DeepSNAP extends its traditional graph representation to include heterogeneous graphs by including the following graph property features:
* `node_feature`: The feature of each node (`torch.tensor`)
* `edge_feature`: The feature of each edge (`torch.tensor`)
* `node_label`: The label of each node (`int`)
* `node_type`: The node type of each node (`string`)
* `edge_type`: The edge type of each edge (`string`)

where the key **new** features we add are `node_type` and `edge_type`, which enables us to perform heterogenous message passing.

For this first question we will work with the familiar [karate club graph](https://networkx.github.io/documentation/stable/auto_examples/graph/plot_karate_club.html) seen in Colab 1. To start, since each node in the graph belongs to one of two clubs (club "Mr. Hi" or club "Officer"), we will treat the club as the `node_type`. The code below demonstrates how to differentiate the nodes in the NetworkX graph.

```python
import networkx as nx
import matplotlib.pyplot as plt
import copy

from networkx.algorithms.community import greedy_modularity_communities

if 'IS_GRADESCOPE_ENV' not in os.environ:

  from pylab import show

  G = nx.karate_club_graph()

  # Community map is a dict that associated the id of each node to one of the
  # two classes/types
  community_map = {}
  for node in G.nodes(data=True):
    if node[1]["club"] == "Mr. Hi":
      community_map[node[0]] = 0
    else:
      community_map[node[0]] = 1

  print(community_map)

  # Create a list fo define color of nodes
  node_color = []

  # Create a color mapping function
  color_map = {0: 0, 1: 1}

  # Get the color of each node given its type
  node_color = [color_map[community_map[node]] for node in G.nodes()]

  # Get position of Graph to obtain spring layout
  pos = nx.spring_layout(G)

  # Create a figure
  plt.figure(figsize=(7, 7))
  nx.draw(G, pos=pos, cmap=plt.get_cmap('coolwarm'), node_color=node_color)
  show()

```

### Question 1.1: Assigning Node Type and Node Features

Using the `community_map` dictionary and graph `G` from above, add node attributes `node_type` and `node_label` to the graph G. Namely, for `node_type` assign nodes in the "Mr. Hi" club to a node type `n0` and nodes in club "Officer" a node type `n1`. Note: the node type should be a `string` property.

Then for `node_label`, assign nodes in "Mr. Hi" club to a `node_label` `0` and nodes in club "Officer" a `node_label` of `1`.

Lastly, assign every node the *tensor* feature vector $[1, 1, 1, 1, 1]$.

**Hint**: Look at the NetworkX function `nx.classes.function.set_node_attributes`.

**Note**: This question is not specifically graded but is important for later questions.

```python
import torch

def assign_node_types(G, community_map):
  # TODO: Implement a function that takes in a NetworkX graph
  # G and community map assignment (mapping node id --> 0/1 label)
  # and adds 'node_type' as a node_attribute in G.

  ############# Your code here ############
  ## (~2 line of code)
  ## Note
  ## 1. Look up NetworkX `nx.classes.function.set_node_attributes`
  ## 2. Look above for the two node type values!
  renamed_community_map = {id: ('n0' if label == 0 else 'n1') for id, label in community_map.items()}
  nx.set_node_attributes(G, renamed_community_map, name='node_type')

  #########################################

def assign_node_labels(G, community_map):
  # TODO: Implement a function that takes in a NetworkX graph
  # G and community map assignment (mapping node id --> 0/1 label)
  # and adds 'node_label' as a node_attribute in G.

  ############# Your code here ############
  ## (~2 line of code)
  ## Note
  ## 1. Look up NetworkX `nx.classes.function.set_node_attributes`
  nx.set_node_attributes(G, community_map, name='node_label')

  #########################################

def assign_node_features(G):
  # TODO: Implement a function that takes in a NetworkX graph
  # G and adds 'node_feature' as a node_attribute in G. Each node
  # in the graph has the same feature vector - a torchtensor with
  # data [1., 1., 1., 1., 1.]

  ############# Your code here ############
  ## (~2 line of code)
  ## Note
  ## 1. Look up NetworkX `nx.classes.function.set_node_attributes`
  nx.set_node_attributes(G, torch.ones(5), name='node_feature')

  pass

  #########################################

if 'IS_GRADESCOPE_ENV' not in os.environ:
  assign_node_types(G, community_map)
  assign_node_labels(G, community_map)
  assign_node_features(G)

  # Explore node properties for the node with id: 20
  node_id = 20
  print (f"Node {node_id} has properties:", G.nodes(data=True)[node_id])

  print(G.nodes(data=True))
```

    Node 20 has properties: {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}
    [(0, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (1, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (2, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (3, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (4, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (5, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (6, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (7, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (8, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (9, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (10, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (11, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (12, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (13, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (14, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (15, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (16, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (17, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (18, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (19, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (20, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (21, {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (22, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (23, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (24, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (25, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (26, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (27, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (28, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (29, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (30, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (31, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (32, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}), (33, {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])})]

### Question 1.2: Assigning Edge Types

Next, we will assign three different `edge_types`:
* Edges within club "Mr. Hi": `e0`
* Edges within club "Officer": `e1`
* Edges between the two clubs: `e2`

**Hint**: Use the `community_map` from before and `nx.classes.function.set_edge_attributes`

```python
def assign_edge_types(G, community_map):
  # TODO: Implement a function that takes in a NetworkX graph
  # G and community map assignment (mapping node id --> 0/1 label)
  # and adds 'edge_type' as a edge_attribute in G.

  ############# Your code here ############
  ## (~5 line of code)
  ## Note
  ## 1. Create an edge assignment dict following rules above
  ## 2. Look up NetworkX `nx.classes.function.set_edge_attributes`

  edge_map = {}

  # Go through all the edges
  for edge in G.edges():
    if community_map[edge[0]] == community_map[edge[1]]:
      edge_map[edge] = 'e0' if community_map[edge[0]] == 0 else 'e1'
    else:
      edge_map[edge] = 'e2'

  nx.set_edge_attributes(G, edge_map, name='edge_type')

  #########################################

if 'IS_GRADESCOPE_ENV' not in os.environ:
  assign_edge_types(G, community_map)

  # Explore edge properties for a sampled edge and check the corresponding
  # node types
  edge_idx = 15
  n1 = 0
  n2 = 31
  edge = list(G.edges(data=True))[edge_idx]
  print (f"Edge ({edge[0]}, {edge[1]}) has properties:", edge[2])
  print (f"Node {n1} has properties:", G.nodes(data=True)[n1])
  print (f"Node {n2} has properties:", G.nodes(data=True)[n2])

  #print(G.edges(data=True))
```

    Edge (0, 31) has properties: {'edge_type': 'e2'}
    Node 0 has properties: {'club': 'Mr. Hi', 'node_type': 'n0', 'node_label': 0, 'node_feature': tensor([1., 1., 1., 1., 1.])}
    Node 31 has properties: {'club': 'Officer', 'node_type': 'n1', 'node_label': 1, 'node_feature': tensor([1., 1., 1., 1., 1.])}

## Heterogeneous Graph Visualization

Now we can visualize the Heterogeneous Graph we have generated.

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:

  # Prepare to save colors
  edge_color = {}

  for edge in G.edges():

    # Unpacking nodes
    n1, n2 = edge

    # I commented this line is useless
    #edge_color[edge] = community_map[n1] if community_map[n1] == community_map[n2] else 2

    if community_map[n1] == community_map[n2] and community_map[n1] == 0:
      edge_color[edge] = 'blue'

    elif community_map[n1] == community_map[n2] and community_map[n1] == 1:
      edge_color[edge] = 'red'

    else:
      edge_color[edge] = 'green'

  # Create a copy
  G_orig = copy.deepcopy(G)

  # Associate edge_color attribute
  nx.classes.function.set_edge_attributes(G, edge_color, name='color')

  # Get the colors
  colors = nx.get_edge_attributes(G,'color').values()

  # Get the labels
  labels = nx.get_node_attributes(G, 'node_type')

  plt.figure(figsize=(8, 8))
  nx.draw(G, pos=pos, cmap=plt.get_cmap('coolwarm'), node_color=node_color, edge_color=colors, labels=labels, font_color='white')
  show()
```

where we differentiate edges within each clubs (2 types) and edges between the two clubs (1 type). Different types of nodes and edges are visualized in different colors. The NetworkX object `G` in following code can be transformed into `deepsnap.hetero_graph.HeteroGraph` directly.

## Transforming to DeepSNAP representation

We will now work through transforming the NetworkX object `G` into a `deepsnap.hetero_graph.HeteroGraph`.

```python
from deepsnap.hetero_graph import HeteroGraph

if 'IS_GRADESCOPE_ENV' not in os.environ:
  hete = HeteroGraph(G_orig)  # done :) -- this is because we have created the necessary stuff: node_type, node_label, node_features, edge_type

  print(type(hete))
```

    <class 'deepsnap.hetero_graph.HeteroGraph'>

## Question 1.3: How many nodes are of each type (10 Points)

```python
def get_nodes_per_type(hete):
  # TODO: Implement a function that takes a DeepSNAP dataset object
  # and return the number of nodes per `node_type`.

  num_nodes_n0 = 0
  num_nodes_n1 = 0

  ############# Your code here ############
  ## (~2 line of code)
  ## Note
  ## 1. Colab autocomplete functionality might be useful.
  num_nodes_n0 = hete.num_nodes()['n0']
  num_nodes_n1 = hete.num_nodes()['n1']
  pass

  #########################################

  return num_nodes_n0, num_nodes_n1

if 'IS_GRADESCOPE_ENV' not in os.environ:
  num_nodes_n0, num_nodes_n1 = get_nodes_per_type(hete)
  print("Node type n0 has {} nodes".format(num_nodes_n0))
  print("Node type n1 has {} nodes".format(num_nodes_n1))
```

    Node type n0 has 17 nodes
    Node type n1 has 17 nodes

## Question 1.4: Message Types - How many edges are of each message type (10 Points)

When working with heterogenous graphs, as we have discussed before, we now work with heterogenous message types (i.e. different message types for the different `node_type` and `edge_type` combinations). For example, an edge of type `e0` connecting two nodes in club "Mr. HI" would have a message type of (`n0`, `e0`, `n0`). In this problem we will analyze how many edges in our graph are of each message type.

**Hint**: If you want to learn more about what the different message types are try the call `hete.message_types`

```python
def get_num_message_edges(hete):
  # TODO: Implement this function that takes a DeepSNAP dataset object
  # and return the number of edges for each message type.
  # You should return a list of tuples as
  # (message_type, num_edge)

  message_type_edges = []

  ############# Your code here ############
  ## (~2 line of code)
  ## Note
  ## 1. Colab autocomplete functionality might be useful.
  message_type_edges = hete.num_edges().items()

  #########################################

  return message_type_edges

if 'IS_GRADESCOPE_ENV' not in os.environ:
  message_type_edges = get_num_message_edges(hete)
  for (message_type, num_edges) in message_type_edges:
    print("Message type {} has {} edges".format(message_type, num_edges))

  print(hete.message_types)
```

    Message type ('n0', 'e0', 'n0') has 35 edges
    Message type ('n0', 'e2', 'n1') has 11 edges
    Message type ('n1', 'e1', 'n1') has 32 edges
    [('n0', 'e0', 'n0'), ('n0', 'e2', 'n1'), ('n1', 'e1', 'n1')]

## Question 1.5: Dataset Splitting - How many nodes are in each dataset split? (10 Points)

DeepSNAP has built in Dataset creation and splitting methods for heterogeneous graphs. Here we will create train, validation, and test datasets for a node prediction task and inspect the resulting subgraphs. Specifically, write a function that computes the number of nodes with a known label in each dataset split.

```python
from deepsnap.dataset import GraphDataset

def compute_dataset_split_counts(datasets):
  # TODO: Implement a function that takes a dict of datasets in the form
  # {'train': dataset_train, 'val': dataset_val, 'test': dataset_test}
  # and returns a dict mapping dataset names to the number of labeled
  # nodes used for supervision in that respective dataset.

  data_set_splits = {}

  ############# Your code here ############
  ## (~3 line of code)
  ## Note
  ## 1. The DeepSNAP `node_label_index` dictionary will be helpful.
  ## 2. Remember to count both node_types
  ## 3. Remember each dataset only has one graph that we need to access
  ##    (i.e. dataset[0])
  for phase, dataset in datasets.items():

    num_labels = 0

    # Not pythonic, but..
    for value in dataset[0].node_label_index.values():
      num_labels += value.shape[0]

    data_set_splits[phase] = num_labels

  #########################################

  return data_set_splits

if 'IS_GRADESCOPE_ENV' not in os.environ:

  # Create a Dataset from HeteroGraph
  dataset = GraphDataset([hete], task='node')

  # Splitting the dataset
  dataset_train, dataset_val, dataset_test = dataset.split(transductive=True, split_ratio=[0.4, 0.3, 0.3])
  datasets = {'train': dataset_train, 'val': dataset_val, 'test': dataset_test}

  data_set_splits = compute_dataset_split_counts(datasets)
  for dataset_name, num_nodes in data_set_splits.items():
    print("{} dataset has {} nodes".format(dataset_name, num_nodes))
```

    train dataset has 12 nodes
    val dataset has 10 nodes
    test dataset has 12 nodes

## DeepSNAP Dataset Visualization

We can now visualize the different nodes and edges used in each graph dataset split.

```python
from deepsnap.dataset import GraphDataset

if 'IS_GRADESCOPE_ENV' not in os.environ:
  dataset = GraphDataset([hete], task='node')

  # Splitting the dataset
  dataset_train, dataset_val, dataset_test = dataset.split(transductive=True, split_ratio=[0.4, 0.3, 0.3])
  titles = ['Train', 'Validation', 'Test']

  for i, dataset in enumerate([dataset_train, dataset_val, dataset_test]):

    # We need to re-index the id of the nodes to the original graph, because
    # the way that the ids of each node type is stored in HeteroGraph starts to
    # count for each one of the types
    n0 = hete._convert_to_graph_index(dataset[0].node_label_index['n0'], 'n0').tolist()
    n1 = hete._convert_to_graph_index(dataset[0].node_label_index['n1'], 'n1').tolist()

    #print(dataset[0].node_label_index['n0'])
    #print(n0)

    plt.figure(figsize=(7, 7))
    plt.title(titles[i])

    nx.draw(G_orig, pos=pos, node_color="grey", edge_color=colors, labels=labels, font_color='white')
    nx.draw_networkx_nodes(G_orig.subgraph(n0), pos=pos, node_color="blue")
    nx.draw_networkx_nodes(G_orig.subgraph(n1), pos=pos, node_color="red")

    show()
```

# 2) Heterogeneous Graph Node Property Prediction

Now, we will use PyTorch Geometric and DeepSNAP to implement a GNN model for heterogeneous graph node property prediction (node classification). We will draw upon our understanding of heterogeneous graphs from lecture and previous work in implementing GNN layers using PyG (introduced in Colab 3).

First let's take a look at the general structure of a heterogeneous GNN layer by working through an example:

Let's assume we have a graph $G$, which contains two node types $a$ and $b$, and three message types $m_1=(a, r_1, a)$, $m_2=(a, r_2, b)$ and $m_3=(a, r_3, b)$. Note: during message passing we view each message as (src, relation, dst), where messages "flow" from src to dst node types. For example, during message passing, updating node type $b$ relies on two different message types $m_2$ and $m_3$.

When applying message passing in heterogenous graphs, we seperately apply message passing over each message type. Therefore, for the graph $G$, a heterogeneous GNN layer contains three seperate Heterogeneous Message Passing layers (`HeteroGNNConv` in this Colab), where each `HeteroGNNConv` layer performs message passing and aggregation with respect to *only one message type*. Since a message type is viewed as (src, relation, dst) and messages "flow" from src to dst, each `HeteroGNNConv` layer only computes embeddings for the *dst* nodes of a given message type. For example, the `HeteroGNNConv` layer for message type $m_2$ outputs updated embedding representations *only* for node's with type b.

---

An overview of the heterogeneous layer we will create is shown below:

where we highlight the following notation:

- $H_a^{(l)[m_1]}$ is the intermediate matrix of of node embeddings for node type $a$, generated by the $l$th `HeteroGNNConv` layer for message type $m_1$.
- $H_a^{(l)}$ is the matrix with current embeddings for nodes of type $a$ after the $l$th layer of our Heterogeneous GNN model. Note that these embeddings can rely on one or more intermediate `HeteroGNNConv` layer embeddings(i.e. $H_b^{(l)}$ combines $H_b^{(l)[m_2]}$ and $H_b^{(l)[m_3]}$).

Since each `HeteroGNNConv` is only applied over a single message type, we additionally define a Heterogeneous GNN Wrapper layer (`HeteroGNNWrapperConv`). This wrapper manages and combines the output of each `HeteroGNNConv` layer in order to generate the complete updated node embeddings for each node type in layer $l$ of our model. More specifically, the $l^{th}$ `HeteroGNNWrapperConv` layer takes as input the node embeddings computed for each message type and node type (e.g. $H_b^{(l)[m_2]}$ and $H_b^{(l)[m_3]}$) and aggregates across message types with the same $dst$ node type. The resulting output of the $l^{th}$ `HeteroGNNWrapperConv` layer is the updated embedding matrix $H_i^{(l)}$ for each node type i.

Continuing on our example above, to compute the node embeddings $H_b^{(l)}$ the wrapper layer aggregates output embeddings from the `HeteroGNNConv` layers associated with message types $m_2$ and $m_3$ (i.e. $H_b^{(l)[m_2]}$ and $H_b^{(l)[m_3]}$).

---

With the `HeteroGNNWrapperConv` module, we can now draw a "simplified" heterogeneous layer structure as follows:

<br/>
<center>

</center>
<br/>

---
**NOTE**:
As reference, it may be helpful to additionally read through PyG's introduciton to heterogeneous graph representations and buidling heterogeneous GNN models: https://pytorch-geometric.readthedocs.io/en/latest/notes/heterogeneous.html

<font color='red'>Looking ahead, we recommend you implement the heterogeneous GNN model in following steps:</font>

1. Implement `HeteroGNNConv`.
2. Implement **just** `mean` aggregation within `HeteroGNNWrapperConv`.
3. Implement `generate_convs`.
4. Implement the `HeteroGNN` model and the `train` function.
5. Train the model with `mean` aggregation and test your model to make sure your model has reasonable performance.
6. Once you are confident in your mean aggregation model, implement `attn` aggregation in `HeteroGNNWrapperConv`.
7. Train the model with `attn` aggregation and test your model to make sure your model has reasonable performance.

Note: The key point of advice is to work completely through implementing the mean aggregation heterogeneous GNN model before diving into the more difficult attention based model.

## Setup

```python
import copy
import torch
import deepsnap
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch_geometric.nn as pyg_nn

from sklearn.metrics import f1_score
from deepsnap.hetero_gnn import forward_op
from deepsnap.hetero_graph import HeteroGraph
from torch_sparse import SparseTensor, matmul
```

## Dataset

You need to login to your Google account and enter the verification code below.

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  from pydrive.auth import GoogleAuth
  from pydrive.drive import GoogleDrive
  from google.colab import auth
  from oauth2client.client import GoogleCredentials

  # Authenticate and create the PyDrive client
  auth.authenticate_user()
  gauth = GoogleAuth()
  gauth.credentials = GoogleCredentials.get_application_default()
  drive = GoogleDrive(gauth)
```

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  id='1ivlxd6lJMcZ9taS44TMGG72x2V1GeVvk'
  downloaded = drive.CreateFile({'id': id})
  downloaded.GetContentFile('acm.pkl')
```

## Implementing `HeteroGNNConv`

Now let's start working on our own implementation of the heterogeneous message passing layer (`HeteroGNNConv`)! Just as in Colabs 3 and 4, we will implement the layer using PyTorch Geometric.

At a high level, the `HeteroGNNConv` layer is equivalent to the homogenous GNN layers we implemented in Colab 3, but now applied to an individual heterogeous message type. Moreover, our heterogeneous GNN layer draws directly from the **GraphSAGE** message passing model ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)).

We begin by defining the `HeteroGNNConv` layer with respect to message type $m$:

\begin{equation}
m =(s, r, d)
\end{equation}

where each message type is a tuple containing three elements: $s$ - the source node type, $r$ - the edge (relation) type, and $d$ - the destination node type.

The message passing update rule that we implement is very similar to that of GraphSAGE, except we now need to include the node types and the edge relation type. The update rule for message type $m$ is described below:

\begin{equation}
h_v^{(l)[m]} = W^{(l)[m]} \cdot \text{CONCAT} \Big( W_d^{(l)[m]} \cdot h_v^{(l-1)}, W_s^{(l)[m]} \cdot AGG(\{h_u^{(l-1)}, \forall u \in N_{m}(v) \})\Big)
\end{equation}

where we compute $h_v^{(l)[m]}$, the node embedding representation for node $v$ after `HeteroGNNConv` layer $l$ with respect message type $m$. Further unpacking the forumla we have:
- $W_s^{(l)[m]}$ - linear transformation matrix for the messages of neighboring source nodes of type $s$ along message type $m$.
- $W_d^{(l)[m]}$ - linear transformation matrix for the message from the node $v$ itself of type $d$.
- $W^{(l)[m]}$ - linear transformation matrix for the concatenated messages from neighboring node's and the central node.
- $h_u^{(l-1)}$ - the hidden embedding representation for node $u$ after the $(l-1)$th `HeteroGNNWrapperConv` layer. Note, that this embedding is not associated with a particular message type (see layer diagrams above).
- $N_{m}(v)$ - the set of neighbor source nodes $s$ for the node v that we are embedding along message type $m = (s, r, d)$.

**NOTE**: We emphasize that each weight matrix is associated with a specific message type $[m]$ and additionally, the weight matrices applied to node messages are differentiated by node type (i.e. $W_s$ and $W_d$).

Lastly, for simplicity, we use mean aggregations for $AGG$ where:

\begin{equation}
AGG(\{h_u^{(l-1)}, \forall u \in N_{m}(v) \}) = \frac{1}{|N_{m}(v)|} \sum_{u\in N_{m}(v)} h_u^{(l-1)}
\end{equation}

```python
class HeteroGNNConv(pyg_nn.MessagePassing):

  def __init__(
      self,
      in_channels_src,
      in_channels_dst,
      out_channels
  ):
      """Initializes a heterogeneous graph neural network.

      Arguments
      ---------

      self : class HeteroGNNConv
        Instantiation of HeteroGNNConv.

      in_channels_src : int
        Input dimension for source features.

      in_channels_dsf : int
        Input dimension for destination features.

      out_channels : int
        Final feature dimension after GNN layer.
      """

      # Parentazing
      super(HeteroGNNConv, self).__init__(aggr="mean")

      # Storing
      self.in_channels_src = in_channels_src
      self.in_channels_dst = in_channels_dst
      self.out_channels = out_channels

      # To simplify implementation, please initialize both self.lin_dst
      # and self.lin_src out_features to out_channels
      self.lin_dst = None
      self.lin_src = None

      self.lin_update = None

      ############# Your code here #############
      ## (~3 lines of code)
      ## Note:
      ## 1. Initialize the 3 linear layers.
      ## 2. Think through the connection between the mathematical
      ##    definition of the update rule and torch linear layers!

      # W_d and W_s in the formulation above
      self.lin_dst = nn.Linear(self.in_channels_dst, self.out_channels)
      self.lin_src = nn.Linear(self.in_channels_src, self.out_channels)

      # Last linear layer muliply the concatenated vector [out_channels +
      # out_channels]
      self.lin_update = nn.Linear(2*self.out_channels, self.out_channels)

      ##########################################

  def forward(
      self,
      node_feature_src,
      node_feature_dst,
      edge_index,
      size=None
  ):

    """Implements a heterogeneous GNN layer.

    Parameters
    ----------

    node_feature_src : tensor of shape (num_nodes_src_type, in_channels_src)
      Features of source nodes.

    node_feature_src : tensor of shape (num_nodes_dst_type, in_channels_dst)
      Features of destination nodes.

    edge_index : SparseTensor of shape (num_nodes_src_type, num_nodes_dst_type)
      Stores adjacency matrix as a sparse matrix.

    size : tuple with shape
      See: https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/nn/conv/message_passing.py

    Return
    ------

    out : tensor of shape (num_nodes_dst_type, out_channels)
      Embedded features of destination nodes.

    """

    ############# Your code here #############
    ## (~1 line of code)
    ## Note:
    ## 1. Unlike Colabs 3 and 4, we just need to call self.propagate with
    ## proper/custom arguments.
    out = self.propagate(edge_index, node_feature_src=node_feature_src,
                         node_feature_dst=node_feature_dst, size=size)

    ##########################################
    return out

  def message_and_aggregate(self, edge_index, node_feature_src):
    """Implements message and aggregation steps of message passing together.

    Parameters
    ----------

    edge_index : SparseTensor of shape (num_nodes_src_type, num_nodes_dst_type)
      Stores adjacency matrix as a sparse matrix.

    node_feature_src : tensor of shape (num_nodes_src_type, in_channels_src)
      Features of source nodes.

    Returns
    -------

    out : tensor of shape (num_nodes_dst_type, in_channels_src)
      Aggregated source features (messages) for each destination node.

    """

    out = None
    ############# Your code here #############
    ## (~1 line of code)
    ## Note:
    ## 1. Different from what we implemented in Colabs 3 and 4, we use message_and_aggregate
    ##    to combine the previously seperate message and aggregate functions.
    ##    The benefit is that we can avoid materializing x_i and x_j
    ##    to make the implementation more efficient.
    ## 2. To implement efficiently, refer to PyG documentation for message_and_aggregate
    ##    and sparse-matrix multiplication:
    ##    https://pytorch-geometric.readthedocs.io/en/latest/notes/sparse_tensor.html
    ## 3. Here edge_index is torch_sparse SparseTensor. Although interesting, you
    ##    do not need to deeply understand SparseTensor represenations!
    ## 4. Conceptually, think through how the message passing and aggregation
    ##    expressed mathematically can be expressed through matrix multiplication.

    # This is cool, we are multiplying a sparse stored version of the adjacency
    # matrix directly with the feature vector.. meaning that we do not need to
    # store the messages to do aggregation; we can do both together and just
    # store the result of the aggregation
    out = matmul(edge_index, node_feature_src, reduce='mean')

    ##########################################

    return out

  def update(self, aggr_out, node_feature_dst):
    """Update step of message passing.

    Parameters
    ----------

    aggr_out : tensor of shape (num_nodes_dst_type, in_channels_src)
      Aggregated source features (messages) for each destination node.

    node_feature_src : tensor of shape (num_nodes_dst_type, in_channels_dst)
      Features of destination nodes.

    Returns
    -------

    out : tensor of shape (num_nodes_dst_type, out_channels)
      Updated features of destination nodes.

    """

    ############# Your code here #############
    ## (~4 lines of code)
    ## Note:
    ## 1. The update function is called after message_and_aggregate
    ## 2. Think through the one-one connection between the mathematical update
    ##    rule and the 3 linear layers defined in the constructor.

    # Apply linear layer to self part
    self_loop = self.lin_dst(node_feature_dst)

    # Apply linear layer to aggr part
    aggr_out = self.lin_src(aggr_out)

    # Concatenate both vectors
    cat = torch.cat((self_loop, aggr_out), dim=-1)

    # Aggregate them together via another linear layer
    out = self.lin_update(cat)
    ##########################################

    return out
```

## Heterogeneous GNN Wrapper Layer

After implementing the `HeteroGNNConv` layer for each message type, we need to manage and aggregate the node embedding results (with respect to each message types). Here we will implement two types of message type level aggregation.

The first one is simply mean aggregation over message types:

\begin{equation}
h_v^{(l)} = \frac{1}{M}\sum_{m=1}^{M}h_v^{(l)[m]}
\end{equation}

where node $v$ has node type $d$ and we sum over the $M$ message types that have destination node type $d$. From our original example, for a node v of type $b$ we aggregate v's `HeteroGNNConv` embeddings for message types $m_2$ and $m_3$ (i.e. $h_v^{(l)[m_2]}$ and $h_v^{(l)[m_3]}$).

The second method we implement is the semantic level attention introduced in **HAN** ([Wang et al. (2019)](https://arxiv.org/abs/1903.07293)). Instead of directly averaging on the message type aggregation results, we use attention to learn which message type result is more important, then aggregate across all the message types. Below are the equations for semantic level attention:

\begin{equation}
e_{m} = \frac{1}{|V_{d}|} \sum_{v \in V_{d}} q_{attn}^T \cdot tanh \Big( W_{attn}^{(l)} \cdot h_v^{(l)[m]} + b \Big)
\end{equation}

where $m$ is the message type and $d$ refers to the destination node type for that message ($m = (s, r, d)$). Additionally, $V_{d}$ refers to the set of nodes v with type $d$. Lastly, the unormalized attention weight $e_m$ is a scaler computed for each message type $m$.

Next, we can compute the normalized attention weights and update $h_v^{(l)}$:

\begin{equation}
\alpha_{m} = \frac{\exp(e_{m})}{\sum_{m=1}^M \exp(e_{m})}
\end{equation}

\begin{equation}
h_v^{(l)} = \sum_{m=1}^{M} \alpha_{m} \cdot h_v^{(l)[m]}
\end{equation}

, where we emphasize that $M$ here is the number of message types associated with the destination node type $d$.

**Note**: The implementation of the attention aggregation is tricky and nuanced. We strongly recommend working carefully through the math equations to undersatnd exactly what each notation refers to and how all the pieces fit together. If you can, try to connect the math to our original example, focusing on node type $b$, which depends on two different message types!

**_We've implemented most of this for you but you'll need to initialize self.attn_proj in the initializer_**

```python
class HeteroGNNWrapperConv(deepsnap.hetero_gnn.HeteroConv):

    def __init__(
        self,
        convs,
        args,
        aggr="mean"
        ):

        """
        Initializes a heterogeneous GNN wrapper.

        Parameters
        ----------

        convs : dict
          Dictionary where keys are (src, msg, dst) and the values are
          HeteroGNNConv layers.

        args : dict
          Dictionary with additional configurations.

        aggr : str
          Choice on how to perform the aggreagation over different message types.

        """
        super(HeteroGNNWrapperConv, self).__init__(convs, None)

        # Store aggregation
        self.aggr = aggr

        # Map the index and message type
        self.mapping = {}

        # A numpy array that stores the final attention probability

        # Get required sizes from the args dictionary
        self.hidden_size = args['hidden_size']
        self.attn_size = args['attn_size']

        self.alpha = None

        self.attn_proj = None

        if self.aggr == "attn":
            ############# Your code here #############
            ## (~1 line of code)
            ## Note:
            ## 1. Initialize self.attn_proj, where self.attn_proj should include
            ##    two linear layers. Note, make sure you understand
            ##    which part of the equation self.attn_proj captures.
            ## 2. You should use nn.Sequential for self.attn_proj
            ## 3. nn.Linear and nn.Tanh are useful.
            ## 4. You can model a weight vector (rather than matrix) by using:
            ##    nn.Linear(some_size, 1, bias=False).
            ## 5. The first linear layer should have out_features as args['attn_size']
            ## 6. You can assume we only have one "head" for the attention.
            ## 7. We recommend you to implement the mean aggregation first. After
            ##    the mean aggregation works well in the training, then you can
            ##    implement this part.
            self.attn_proj = nn.Sequential(
                nn.Linear(self.hidden_size, self.attn_size, bias=True),  # this is for Wattn h + b
                nn.Tanh(),  # this is for the non-linear part
                nn.Linear(self.attn_size, 1, bias=False)  # this is for the qattnT
            )

            ##########################################

    def reset_parameters(self):
        super(HeteroConvWrapper, self).reset_parameters()

        if self.aggr == "attn":
            for layer in self.attn_proj.children():
                layer.reset_parameters()

    def forward(self, node_features, edge_indices):
        """Implements hetero GNN wrapper.

        Parameters
        ----------

        node_features : dict
          Dictionary where keys are 'node_types' and values are tensors of shape
          (num_node_type, in_channels_node_type).

        edge_indices : dict
          Dictionary where keys are 'message types' (src, msg, dst) and values
          are SparseTensors of shape (num_nodes_src_type, num_nodes_dst_type).

        Returns
        -------

        node_emb : dict
          Dictionary where keys are 'node_types' and values are tensors of shape
          (num_node_type, out_channels).

        """
        # Prepare to store the embedding for each message type
        message_type_emb = {}

        # Go through all message types and their adjacency matrices
        for message_key, message_type in edge_indices.items():

            # Extract current message type
            src_type, edge_type, dst_type = message_key

            # Get tensors with features according to the node type
            node_feature_src = node_features[src_type]
            node_feature_dst = node_features[dst_type]

            # Get current adjacency matrix
            edge_index = edge_indices[message_key]

            # Store embedding of the message after passing through a Hetero
            # GNN layer for example
            message_type_emb[message_key] = (
                self.convs[message_key](
                    node_feature_src,
                    node_feature_dst,
                    edge_index,
                )
            )

        # We now need to aggregate the information through messages

        # Initialize a dictionary where keys are node types (strings) of
        # destination nodes
        node_emb = {dst: [] for _, _, dst in message_type_emb.keys()}

        # Initialize another dictionary to figure out mapping
        mapping = {}

        # Go through all message types
        for (src, edge_type, dst), item in message_type_emb.items():

            # Create an index 0, 1,.. based on the number of times that the
            # particular node type was a destination node
            mapping[len(node_emb[dst])] = (src, edge_type, dst)

            # Store in the node_emb disctionary a list of tensors with the
            # embeddings
            node_emb[dst].append(item)

        # Store mapping
        self.mapping = mapping

        # Perform aggregation. Recall that:
        #   node_type is a dict where keys are node types and values are a list
        #   of tensors containing the embeddings
        #   embs is a list of tensors for each message where the node type was
        #   the destination
        for node_type, embs in node_emb.items():
            if len(embs) == 1:
                node_emb[node_type] = embs[0]
            else:
                node_emb[node_type] = self.aggregate(embs)

        return node_emb

    def aggregate(self, xs):
        """
        Aggregate results of message passings occuring in parallel for a node
        type that plays as a dst node more than once in (src, msg, dst)
        combinations.

        Parameters
        ----------

        xs : list
          List of tensors of shape [num_node_type, out_channels].

        Returns
        -------

        out : tensor of shape (num_node_type, out_channels)
          Aggregated embeddings.

        """

        # TODO: Implement this function that aggregates all message type results.
        # Here, xs is a list of tensors (embeddings) with respect to message
        # type aggregation results.

        if self.aggr == "mean":

            ############# Your code here #############
            ## (~2 lines of code)
            ## Note:
            ## 1. Explore the function parameter `xs`!
            out = torch.mean(torch.cat(xs), dim=0)

            ##########################################

            return out

        elif self.aggr == "attn":

            # Number of nodes for that node type
            N = xs[0].shape[0]

            # Number of message types for that node type
            M = len(xs)

            # Concatenate the list of tensors over M
            x = torch.cat(xs, dim=0).view(M, N, -1) # shape of (M, N, D)

            # Apply attention layer over x
            z = self.attn_proj(x).view(M, N) # shape of (M, N, 1)

            # Take the average over N -- all the nodes with same type
            z = z.mean(1) # shape of (M, 1) -- this is in fact denoted as e_m above

            # Normalize using softmax
            alpha = torch.softmax(z, dim=0) # shape of (M, 1)

            # Store the attention result to self.alpha as np array
            self.alpha = alpha.view(-1).data.cpu().numpy()

            # Add a third dimension to perform multiplication with x
            alpha = alpha.view(M, 1, 1)

            # Compute weigthed version of the feature vector
            x = x * alpha

            # Weighted average
            out = x.sum(dim=0)

            return out
```

## Initialize Heterogeneous GNN Layers

Now let's put it all together and initialize the Heterogeneous GNN Layers. Different from the homogeneous graph case, heterogeneous graphs can be a little bit complex.

In general, we need to create a dictionary of `HeteroGNNConv` layers where the keys are message types.

* To get all message types, `deepsnap.hetero_graph.HeteroGraph.message_types` is useful.
* If we are initializing the first conv layers, we need to get the feature dimension of each node type. Using `deepsnap.hetero_graph.HeteroGraph.num_node_features(node_type)` will return the node feature dimension of `node_type`. In this function, we will set each `HeteroGNNConv` `out_channels` to be `hidden_size`.
* If we are not initializing the first conv layers, all node types will have the smae embedding dimension `hidden_size` and we still set `HeteroGNNConv` `out_channels` to be `hidden_size` for simplicity.

```python
def generate_convs(hetero_graph, conv, hidden_size, first_layer=False):
    """
    Creates a conv layer for each message type in the hetero_graph.

    Parameters
    ----------

    hetero_graph : class HeteroGraph
      Heterogeneous graph.

    conv : class
      Some layer that we want to apply over each message type. In our case, we
      want to apply HeteroGNNConv.

    hidden_size : int
      Size of the hiddent layer.

    first_layer : bool
      First layer needs to consider the input feature dimensions.

    Returns
    -------

    convs : dictionary
      A dictionary containing a conv layer for each message type in hetero_graph.

    """
    # TODO: Implement this function that returns a dictionary of `HeteroGNNConv`
    # layers where the keys are message types. `hetero_graph` is deepsnap `HeteroGraph`
    # object and the `conv` is the `HeteroGNNConv`.

    # Initialize the dictionary
    convs = {}

    ############# Your code here #############
    ## (~9 lines of code)
    ## Note:
    ## 1. See the hints above!
    ## 2. conv is of type `HeteroGNNConv`

    # "To get all message types, deepsnap.hetero_graph.HeteroGraph.message_types
    # is useful.""
    message_types = hetero_graph.message_types

    # Go through all message types
    for (src, edge_type, dst) in message_types:

      # "If we are initializing the first conv layers, ..."
      if first_layer:

        # Extracting feature dimensions
        in_channels_src = hetero_graph.num_node_features(src)
        in_channels_dst = hetero_graph.num_node_features(dst)

      else:

        # "... all node types will have the smae embedding dimension hidden_size
        # ..."
        in_channels_src = in_channels_dst = hidden_size

      # Out_channels will always be the hidden_size
      out_channels = hidden_size

      # Now we just initialize the conv layer for the respective message type
      convs[(src, edge_type, dst)] = conv(in_channels_src, in_channels_dst, out_channels)

    ##########################################

    return convs
```

## HeteroGNN

Now we will make a simple HeteroGNN model which contains only two `HeteroGNNWrapperConv` layers.

For the forward function in `HeteroGNN`, the model is going to be run as following:

$\text{self.convs1} \rightarrow \text{self.bns1} \rightarrow \text{self.relus1} \rightarrow \text{self.convs2} \rightarrow \text{self.bns2} \rightarrow \text{self.relus2} \rightarrow \text{self.post_mps}$

```python
class HeteroGNN(torch.nn.Module):

  def __init__(self, hetero_graph, args, aggr="mean"):
    """Initializes our HeteroGNN model.

    Parameters
    ----------

    hetero_graph : class HeteroGraph
      Heterogeneous graph.

    args : dict
      Dictionary with additional configurations.

    aggr : str
      Choice on how to perform the aggreagation over different message types.

    """
    super(HeteroGNN, self).__init__()

    self.aggr = aggr
    self.hidden_size = args['hidden_size']

    self.convs1 = None  # in_channels to hidden
    self.convs2 = None  # hidden to hidden

    self.bns1 = nn.ModuleDict()
    self.bns2 = nn.ModuleDict()

    self.relus1 = nn.ModuleDict()
    self.relus2 = nn.ModuleDict()

    self.post_mps = nn.ModuleDict()

    ############# Your code here #############
    ## (~10 lines of code)
    ## Note:
    ## 1. For self.convs1 and self.convs2, call generate_convs at first and then
    ##    pass the returned dictionary of `HeteroGNNConv` to `HeteroGNNWrapperConv`.
    ## 2. For self.bns, self.relus and self.post_mps, the keys are node_types.
    ##    `deepsnap.hetero_graph.HeteroGraph.node_types` will be helpful.
    ## 3. Initialize all batchnorms to torch.nn.BatchNorm1d(hidden_size, eps=1).
    ## 4. Initialize all relus to nn.LeakyReLU().
    ## 5. For self.post_mps, each value in the ModuleDict is a linear layer
    ##    where the `out_features` is the number of classes for that node type.
    ##    `deepsnap.hetero_graph.HeteroGraph.num_node_labels(node_type)` will be
    ##    useful.

    # 1. For self.convs1 and self.convs2, call generate_convs at first and then
    #    pass the returned dictionary of `HeteroGNNConv` to `HeteroGNNWrapperConv`.
    convs1 = generate_convs(hetero_graph, HeteroGNNConv, self.hidden_size, first_layer=True)
    convs2 = generate_convs(hetero_graph, HeteroGNNConv, self.hidden_size, first_layer=False)

    self.convs1 = HeteroGNNWrapperConv(convs1, args, self.aggr)
    self.convs2 = HeteroGNNWrapperConv(convs2, args, self.aggr)

    # 2. For self.bns, self.relus and self.post_mps, the keys are node_types.
    #   `deepsnap.hetero_graph.HeteroGraph.node_types` will be helpful.

    # We need to create a dictionary (nn.ModuleDict) that contains a hidden
    # layer of the chosen type for each node type
    for node_type in hetero_graph.node_types:

      # 3. Initialize all batchnorms to torch.nn.BatchNorm1d(hidden_size, eps=1).
      self.bns1[node_type] = torch.nn.BatchNorm1d(self.hidden_size, eps=1)
      self.bns2[node_type] = torch.nn.BatchNorm1d(self.hidden_size, eps=1)

      # 4. Initialize all relus to nn.LeakyReLU().
      self.relus1[node_type] = nn.LeakyReLU()
      self.relus2[node_type] = nn.LeakyReLU()

      # 5. For self.post_mps, each value in the ModuleDict is a linear layer
      #    where the `out_features` is the number of classes for that node type.
      #    `deepsnap.hetero_graph.HeteroGraph.num_node_labels(node_type)` will be
      #    useful.
      self.post_mps[node_type] = nn.Linear(in_features=self.hidden_size, out_features=hetero_graph.num_node_labels(node_type))

        ##########################################

  def forward(self, node_features, edge_indices):
      """Implements our HeteroGNN model.

      Parameters
      ----------
        node_features : dict
          Dictionary where keys are 'node_types' and values are tensors of shape
          (num_node_type, in_channels_node_type).

        edge_indices : dict
          Dictionary where keys are 'message types' (src, msg, dst) and values
          are SparseTensors of shape (num_nodes_src_type, num_nodes_dst_type).

      Returns
      -------
        x : dict
        Dictionary where keys are 'node_types' and values are tensors of shape
        (num_node_type, out_channels).

      """
      # TODO: Implement the forward function. Notice that `node_feature` is
      # a dictionary of tensors where keys are node types and values are
      # corresponding feature tensors. The `edge_index` is a dictionary of
      # tensors where keys are message types and values are corresponding
      # edge index tensors (with respect to each message type).

      x = node_features

      ############# Your code here #############
      ## (~7 lines of code)
      ## Note:
      ## 1. `deepsnap.hetero_gnn.forward_op` can be helpful.

      # See: https://snap.stanford.edu/deepsnap/modules/hetero_gnn.html?highlight=forward_op#deepsnap.hetero_gnn.forward_op
      x = self.convs1(x, edge_indices)

      # Since some layers are only dependent on the node type, irrespective
      # of the message, we need to apply them over all possible messages.
      # forward_op does that for us. If we did not had this function, we should
      # a generate function as generate_convs for each layer.

      x = forward_op(x, self.bns1)
      x = forward_op(x, self.relus1)

      x = self.convs2(x, edge_indices)

      x = forward_op(x, self.bns2)
      x = forward_op(x, self.relus2)

      x = forward_op(x, self.post_mps)

      ##########################################

      return x

  def loss(self, preds, y, train_idx):
    """
    Compute loss.

    Parameters
    ----------

    self : HeteroGNN class
      Instantiation of HeretoGNN.

    preds : dict
      Dictionary of predicted labels keyed by node type.
              preds[node_type] = model_predictions
      model_predictions has shape (num_nodes_node_type, num_node_labels_node_type)
      because of the post_mps layer

    y : dict
      Dictionary of true labels keyed by node type.

    train_idx : dict
      Dictionary of indices keyed by node type, containing the mask of nodes
      used for training.

    Returns
    -------

    loss : float
      Sum of the losses over all node types.

    """
    loss = 0
    loss_func = F.cross_entropy

    ############# Your code here #############
    ## (~3 lines of code)
    ## Note:
    ## 1. For each node type in preds, accumulate computed loss to `loss`
    ## 2. Loss need to be computed with respect to the given index
    ## 3. preds is a dictionary of model predictions keyed by node_type.
    ## 4. indeces is a dictionary of labeled supervision nodes keyed
    ##    by node_type

    # 1. For each node type in preds, accumulate computed loss to `loss`
    for node_type in preds:

      # We need to get the training indexes of the nodes
      idx = train_idx[node_type]

      # Compute loss
      loss += loss_func(preds[node_type][idx], y[node_type][idx])

    ##########################################

    return loss
```

## Training and Testing

Here we provide you with the functions to train and test. You only need to implement one line of code here.

**Please do not modify other parts in `train` and `test` for grading purposes.**

```python
import pandas as pd

def train(model, optimizer, hetero_hetero_graph, train_idx):
    """Call training of the model.

    Parameters
    ----------

    model : class torch.nn.Module
      Our model.

    optimizer : class torch.optim
      Our optimizer.

    hetero_graph : class HeteroGraph
      Our graph.

    train_idx : dict
      Dictionary keyed by node types containing the indexes of the nodes used for
      training.

    Return
    ------
    loss : float
      Loss aggregated over all node types.

    """

    # Instantiate training
    model.train()

    # Zero grads
    optimizer.zero_grad()

    # Training phase
    preds = model(hetero_graph.node_feature, hetero_graph.edge_index)

    # Initialize loss
    loss = None

    ############# Your code here #############
    ## Note:
    ## 1. Compute the loss here
    ## 2. `deepsnap.hetero_graph.HeteroGraph.node_label` is useful
    loss = model.loss(preds, hetero_graph.node_label, train_idx)

    ##########################################

    loss.backward()
    optimizer.step()

    return loss.item()

def test(model, hetero_graph, indices, best_model=None, best_val=0, save_preds=False, agg_type=None):
    """Test our model.

    Parameters
    ----------

    model : class torch.nn.Module
      Our model.

    hetero_graph : class HeteroGraph
      Our hetero_graph.

    indices : list
      List cointaing the indexes of each phase [train, validation, test]

    best_model : class torch.nn.Module
      Best model based on best validation.

    best_val : float
      Best validation metric.

    save_preds : bool
      Save or not the predictions.

    agg_type : str
      String with the aggreagation type used during wrapping.

    Returns
    -------

    accs : list
      List of tuples. 0, 1 ,2 stards for test, val, train; tuple is (micro, macro);

    best_model : class torch.nn.Module
      Best model based on best validation.

    best_val : float
      Best validation metric.

    """
    model.eval()

    # Get predictions
    preds = model(hetero_graph.node_feature, hetero_graph.edge_index)

    # Prepare to save accuracies
    accs = []

    # Go through each phase
    for i, index in enumerate(indices):

        # preds = model(hetero_graph.node_feature, hetero_graph.edge_index) # I placed this outside, no reason why this is here

        num_node_types = 0
        micro = 0
        macro = 0

        for node_type in preds:
            idx = index[node_type]
            pred = preds[node_type][idx]
            pred = pred.max(1)[1]
            label_np = hetero_graph.node_label[node_type][idx].cpu().numpy()
            pred_np = pred.cpu().numpy()
            micro = f1_score(label_np, pred_np, average='micro')
            macro = f1_score(label_np, pred_np, average='macro')
            num_node_types += 1

        # Averaging f1 score might not make sense, but in our example we only
        # have one node type
        micro /= num_node_types
        macro /= num_node_types
        accs.append((micro, macro))

        # Only save the test set predictions and labels!
        if save_preds and i == 2:
          print ("Saving Heterogeneous Node Prediction Model Predictions with Agg:", agg_type)
          print()

          data = {}
          data['pred'] = pred_np
          data['label'] = label_np

          df = pd.DataFrame(data=data)
          # Save locally as csv
          df.to_csv('ACM-Node-' + agg_type + 'Agg.csv', sep=',', index=False)

    if accs[1][0] > best_val:
        best_val = accs[1][0]
        best_model = copy.deepcopy(model)
    return accs, best_model, best_val
```

```python
# Please do not change the following parameters
args = {
    'device': torch.device('cuda' if torch.cuda.is_available() else 'cpu'),
    'hidden_size': 64,
    'epochs': 500,
    'weight_decay': 1e-5,
    'lr': 0.003,
    'attn_size': 32,
}
```

## Dataset and Preprocessing

In the next, we will load the data and create a tensor backend (without a NetworkX graph) `deepsnap.hetero_graph.HeteroGraph` object.

We will use the `ACM(3025)` dataset in our node property prediction task, which is proposed in **HAN** ([Wang et al. (2019)](https://arxiv.org/abs/1903.07293)) and our dataset is extracted from [DGL](https://www.dgl.ai/)'s [ACM.mat](https://data.dgl.ai/dataset/ACM.mat).

The original ACM dataset has three node types and two edge (relation) types. For simplicity, we simplify the heterogeneous graph to one node type and two edge types (shown below). This means that in our heterogeneous graph, we have one node type (paper) and two message types (paper, author, paper) and (paper, subject, paper).

<br/>
<center>

</center>

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:

  print("Device: {}".format(args['device']))

  # Load the data
  data = torch.load("acm.pkl")

  # Message types
  message_type_1 = ("paper", "author", "paper")
  message_type_2 = ("paper", "subject", "paper")

  # Dictionary of edge indices
  edge_index = {}
  edge_index[message_type_1] = data['pap']
  edge_index[message_type_2] = data['psp']

  # Dictionary of node features
  node_feature = {}
  node_feature["paper"] = data['feature']

  # Dictionary of node labels
  node_label = {}
  node_label["paper"] = data['label']

  # Load the train, validation and test indices
  train_idx = {"paper": data['train_idx'].to(args['device'])}
  val_idx = {"paper": data['val_idx'].to(args['device'])}
  test_idx = {"paper": data['test_idx'].to(args['device'])}

  # Construct a deepsnap tensor backend HeteroGraph
  hetero_graph = HeteroGraph(
      node_feature=node_feature,
      node_label=node_label,
      edge_index=edge_index,
      directed=True
  )

  print(f"ACM heterogeneous graph: {hetero_graph.num_nodes()} nodes, {hetero_graph.num_edges()} edges")

  # Node feature and node label to device
  for key in hetero_graph.node_feature:
      hetero_graph.node_feature[key] = hetero_graph.node_feature[key].to(args['device'])
  for key in hetero_graph.node_label:
      hetero_graph.node_label[key] = hetero_graph.node_label[key].to(args['device'])

  # Edge_index to sparse tensor and to device
  for key in hetero_graph.edge_index:
      edge_index = hetero_graph.edge_index[key]
      adj = SparseTensor(row=edge_index[0], col=edge_index[1], sparse_sizes=(hetero_graph.num_nodes('paper'), hetero_graph.num_nodes('paper')))
      hetero_graph.edge_index[key] = adj.t().to(args['device'])

  print("Message Type 1:")
  print(hetero_graph.edge_index[message_type_1])
  print()
  print("Message Type 2:")
  print(hetero_graph.edge_index[message_type_2])
```

    Device: cpu
    ACM heterogeneous graph: {'paper': 3025} nodes, {('paper', 'author', 'paper'): 26256, ('paper', 'subject', 'paper'): 2207736} edges
    Message Type 1:
    SparseTensor(row=tensor([   0,    0,    0,  ..., 3024, 3024, 3024]),
                 col=tensor([   8,   20,   51,  ..., 2948, 2983, 2991]),
                 size=(3025, 3025), nnz=26256, density=0.29%)

    Message Type 2:
    SparseTensor(row=tensor([   0,    0,    0,  ..., 3024, 3024, 3024]),
                 col=tensor([  75,  434,  534,  ..., 3020, 3021, 3022]),
                 size=(3025, 3025), nnz=2207736, density=24.13%)

## Start Training!

Now lets start training!

## Training the Mean Aggregation

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  best_model = None
  best_val = 0

  model = HeteroGNN(hetero_graph, args, aggr="mean").to(args['device'])
  optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'], weight_decay=args['weight_decay'])

  for epoch in range(args['epochs']):
      loss = train(model, optimizer, hetero_graph, train_idx)
      accs, best_model, best_val = test(model, hetero_graph, [train_idx, val_idx, test_idx], best_model, best_val)
      print(
          f"Epoch {epoch + 1}: loss {round(loss, 5)}, "
          f"train micro {round(accs[0][0] * 100, 2)}%, train macro {round(accs[0][1] * 100, 2)}%, "
          f"valid micro {round(accs[1][0] * 100, 2)}%, valid macro {round(accs[1][1] * 100, 2)}%, "
          f"test micro {round(accs[2][0] * 100, 2)}%, test macro {round(accs[2][1] * 100, 2)}%"
      )
  best_accs, _, _ = test(best_model, hetero_graph, [train_idx, val_idx, test_idx])
  print(
      f"Best model: "
      f"train micro {round(best_accs[0][0] * 100, 2)}%, train macro {round(best_accs[0][1] * 100, 2)}%, "
      f"valid micro {round(best_accs[1][0] * 100, 2)}%, valid macro {round(best_accs[1][1] * 100, 2)}%, "
      f"test micro {round(best_accs[2][0] * 100, 2)}%, test macro {round(best_accs[2][1] * 100, 2)}%"
  )
```

    Epoch 1: loss 1.09996, train micro 33.33%, train macro 16.67%, valid micro 33.33%, valid macro 16.67%, test micro 31.29%, test macro 15.89%
    Epoch 2: loss 1.09203, train micro 57.83%, train macro 48.66%, valid micro 53.33%, valid macro 44.61%, test micro 50.59%, test macro 41.47%
    Epoch 3: loss 1.06677, train micro 66.5%, train macro 53.62%, valid micro 65.67%, valid macro 52.71%, test micro 65.46%, test macro 53.44%
    Epoch 4: loss 1.01573, train micro 66.67%, train macro 56.4%, valid micro 64.33%, valid macro 53.62%, test micro 64.28%, test macro 53.84%
    Epoch 5: loss 0.92693, train micro 65.67%, train macro 56.13%, valid micro 64.0%, valid macro 54.07%, test micro 63.67%, test macro 53.48%
    Epoch 6: loss 0.79406, train micro 66.5%, train macro 57.74%, valid micro 64.0%, valid macro 54.07%, test micro 63.86%, test macro 53.7%
    Epoch 7: loss 0.62822, train micro 68.17%, train macro 59.61%, valid micro 65.0%, valid macro 55.74%, test micro 64.38%, test macro 54.07%
    Epoch 8: loss 0.46084, train micro 69.33%, train macro 61.55%, valid micro 67.0%, valid macro 58.12%, test micro 64.85%, test macro 54.41%
    Epoch 9: loss 0.32139, train micro 71.83%, train macro 65.76%, valid micro 70.33%, valid macro 63.38%, test micro 65.88%, test macro 55.49%
    Epoch 10: loss 0.21974, train micro 76.83%, train macro 73.48%, valid micro 74.0%, valid macro 69.74%, test micro 66.64%, test macro 56.8%
    Epoch 11: loss 0.15064, train micro 82.67%, train macro 81.11%, valid micro 82.33%, valid macro 81.0%, test micro 68.47%, test macro 60.2%
    Epoch 12: loss 0.10394, train micro 90.83%, train macro 90.44%, valid micro 87.0%, valid macro 86.35%, test micro 71.01%, test macro 64.55%
    Epoch 13: loss 0.07211, train micro 94.5%, train macro 94.37%, valid micro 89.67%, valid macro 89.18%, test micro 73.18%, test macro 68.0%
    Epoch 14: loss 0.05103, train micro 95.67%, train macro 95.6%, valid micro 90.67%, valid macro 90.33%, test micro 74.45%, test macro 70.33%
    Epoch 15: loss 0.0379, train micro 96.67%, train macro 96.64%, valid micro 91.67%, valid macro 91.47%, test micro 75.29%, test macro 71.91%
    Epoch 16: loss 0.02955, train micro 97.83%, train macro 97.83%, valid micro 92.33%, valid macro 92.25%, test micro 75.72%, test macro 72.94%
    Epoch 17: loss 0.024, train micro 98.67%, train macro 98.66%, valid micro 92.67%, valid macro 92.64%, test micro 75.91%, test macro 73.52%
    Epoch 18: loss 0.02027, train micro 99.0%, train macro 99.0%, valid micro 93.33%, valid macro 93.32%, test micro 76.61%, test macro 74.55%
    Epoch 19: loss 0.01751, train micro 99.0%, train macro 99.0%, valid micro 94.33%, valid macro 94.33%, test micro 77.46%, test macro 75.67%
    Epoch 20: loss 0.01523, train micro 99.5%, train macro 99.5%, valid micro 95.0%, valid macro 95.0%, test micro 78.21%, test macro 76.54%
    Epoch 21: loss 0.01327, train micro 99.67%, train macro 99.67%, valid micro 95.67%, valid macro 95.66%, test micro 78.96%, test macro 77.44%
    Epoch 22: loss 0.01156, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.98%, test micro 79.76%, test macro 78.26%
    Epoch 23: loss 0.01008, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.97%, test micro 80.47%, test macro 79.05%
    Epoch 24: loss 0.00877, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.97%, test micro 81.22%, test macro 79.86%
    Epoch 25: loss 0.00766, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.65%, test micro 81.6%, test macro 80.31%
    Epoch 26: loss 0.00676, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.65%, test micro 81.79%, test macro 80.59%
    Epoch 27: loss 0.006, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 82.12%, test macro 81.04%
    Epoch 28: loss 0.00525, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 82.45%, test macro 81.5%
    Epoch 29: loss 0.00452, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 96.98%, test micro 82.31%, test macro 81.36%
    Epoch 30: loss 0.00388, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 82.59%, test macro 81.7%
    Epoch 31: loss 0.00333, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 82.59%, test macro 81.7%
    Epoch 32: loss 0.00287, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 82.73%, test macro 81.92%
    Epoch 33: loss 0.00249, train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.66%, test micro 82.73%, test macro 81.95%
    Epoch 34: loss 0.00217, train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.67%, test micro 82.73%, test macro 81.98%
    Epoch 35: loss 0.00191, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 82.87%, test macro 82.15%
    Epoch 36: loss 0.0017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 82.78%, test macro 82.05%
    Epoch 37: loss 0.00152, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 82.87%, test macro 82.18%
    Epoch 38: loss 0.00137, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 82.82%, test macro 82.15%
    Epoch 39: loss 0.00125, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 82.87%, test macro 82.22%
    Epoch 40: loss 0.00115, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 83.01%, test macro 82.41%
    Epoch 41: loss 0.00106, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.92%, test macro 82.31%
    Epoch 42: loss 0.00099, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.01%, test macro 82.42%
    Epoch 43: loss 0.00093, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.96%, test macro 82.4%
    Epoch 44: loss 0.00087, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.96%, test macro 82.43%
    Epoch 45: loss 0.00082, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.06%, test macro 82.56%
    Epoch 46: loss 0.00078, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.92%, test macro 82.43%
    Epoch 47: loss 0.00075, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 83.06%, test macro 82.61%
    Epoch 48: loss 0.00071, train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.67%, test micro 83.25%, test macro 82.84%
    Epoch 49: loss 0.00068, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.29%, test macro 82.9%
    Epoch 50: loss 0.00066, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.44%, test macro 83.07%
    Epoch 51: loss 0.00063, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.25%, test macro 82.88%
    Epoch 52: loss 0.00061, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.25%, test macro 82.88%
    Epoch 53: loss 0.00059, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.29%, test macro 82.94%
    Epoch 54: loss 0.00057, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.06%, test macro 82.72%
    Epoch 55: loss 0.00056, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.01%, test macro 82.68%
    Epoch 56: loss 0.00054, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.11%, test macro 82.79%
    Epoch 57: loss 0.00053, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 83.06%, test macro 82.74%
    Epoch 58: loss 0.00051, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.11%, test macro 82.8%
    Epoch 59: loss 0.0005, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.15%, test macro 82.85%
    Epoch 60: loss 0.00049, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.15%, test macro 82.85%
    Epoch 61: loss 0.00048, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.15%, test macro 82.86%
    Epoch 62: loss 0.00047, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.11%, test macro 82.83%
    Epoch 63: loss 0.00046, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.06%, test macro 82.78%
    Epoch 64: loss 0.00045, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 83.06%, test macro 82.78%
    Epoch 65: loss 0.00044, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.96%, test macro 82.7%
    Epoch 66: loss 0.00043, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.92%, test macro 82.66%
    Epoch 67: loss 0.00043, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.82%, test macro 82.58%
    Epoch 68: loss 0.00042, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.78%, test macro 82.54%
    Epoch 69: loss 0.00041, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.87%, test macro 82.64%
    Epoch 70: loss 0.00041, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.82%, test macro 82.59%
    Epoch 71: loss 0.0004, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.96%, test macro 82.74%
    Epoch 72: loss 0.0004, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.66%, test micro 82.96%, test macro 82.74%
    Epoch 73: loss 0.00039, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.66%, test micro 82.92%, test macro 82.7%
    Epoch 74: loss 0.00038, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.82%, test macro 82.6%
    Epoch 75: loss 0.00038, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.78%, test macro 82.56%
    Epoch 76: loss 0.00037, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.52%
    Epoch 77: loss 0.00037, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.78%, test macro 82.57%
    Epoch 78: loss 0.00037, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.68%, test macro 82.48%
    Epoch 79: loss 0.00036, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.68%, test macro 82.48%
    Epoch 80: loss 0.00036, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.54%
    Epoch 81: loss 0.00035, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.53%
    Epoch 82: loss 0.00035, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.53%
    Epoch 83: loss 0.00035, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.53%
    Epoch 84: loss 0.00034, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.68%, test macro 82.49%
    Epoch 85: loss 0.00034, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.54%
    Epoch 86: loss 0.00033, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.73%, test macro 82.54%
    Epoch 87: loss 0.00033, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.68%, test macro 82.49%
    Epoch 88: loss 0.00033, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.68%, test macro 82.49%
    Epoch 89: loss 0.00032, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.64%, test macro 82.44%
    Epoch 90: loss 0.00032, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.64%, test macro 82.44%
    Epoch 91: loss 0.00032, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.64%, test macro 82.44%
    Epoch 92: loss 0.00032, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.59%, test macro 82.39%
    Epoch 93: loss 0.00031, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.59%, test macro 82.4%
    Epoch 94: loss 0.00031, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.59%, test macro 82.4%
    Epoch 95: loss 0.00031, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.59%, test macro 82.41%
    Epoch 96: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.54%, test macro 82.36%
    Epoch 97: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.49%, test macro 82.31%
    Epoch 98: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.49%, test macro 82.31%
    Epoch 99: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.49%, test macro 82.31%
    Epoch 100: loss 0.00029, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.22%
    Epoch 101: loss 0.00029, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.22%
    Epoch 102: loss 0.00029, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.35%, test macro 82.17%
    Epoch 103: loss 0.00029, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 104: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 105: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 106: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 107: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 108: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 109: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.12%
    Epoch 110: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.13%
    Epoch 111: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.23%
    Epoch 112: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.23%
    Epoch 113: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.23%
    Epoch 114: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 82.23%
    Epoch 115: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.45%, test macro 82.28%
    Epoch 116: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.45%, test macro 82.28%
    Epoch 117: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.45%, test macro 82.28%
    Epoch 118: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.45%, test macro 82.28%
    Epoch 119: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.45%, test macro 82.28%
    Epoch 120: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.35%, test macro 82.18%
    Epoch 121: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.35%, test macro 82.18%
    Epoch 122: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.35%, test macro 82.18%
    Epoch 123: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.31%, test macro 82.14%
    Epoch 124: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.26%, test macro 82.09%
    Epoch 125: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.26%, test macro 82.09%
    Epoch 126: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.21%, test macro 82.05%
    Epoch 127: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.21%, test macro 82.05%
    Epoch 128: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.16%, test macro 82.0%
    Epoch 129: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.12%, test macro 81.95%
    Epoch 130: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.16%, test macro 82.0%
    Epoch 131: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.16%, test macro 82.0%
    Epoch 132: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.16%, test macro 82.0%
    Epoch 133: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.07%, test macro 81.91%
    Epoch 134: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.07%, test macro 81.91%
    Epoch 135: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.07%, test macro 81.91%
    Epoch 136: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.02%, test macro 81.86%
    Epoch 137: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.02%, test macro 81.86%
    Epoch 138: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 139: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 140: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 141: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 142: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 143: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.88%, test macro 81.71%
    Epoch 144: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 145: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 146: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 147: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 148: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 149: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 150: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.71%
    Epoch 151: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.84%, test macro 81.66%
    Epoch 152: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.88%, test macro 81.7%
    Epoch 153: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.84%, test macro 81.65%
    Epoch 154: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.84%, test macro 81.65%
    Epoch 155: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.84%, test macro 81.65%
    Epoch 156: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.79%, test macro 81.61%
    Epoch 157: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.74%, test macro 81.57%
    Epoch 158: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 159: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 160: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 161: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 162: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 163: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 164: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 165: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 166: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 167: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.51%
    Epoch 168: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.65%, test macro 81.47%
    Epoch 169: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.65%, test macro 81.47%
    Epoch 170: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.65%, test macro 81.47%
    Epoch 171: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.6%, test macro 81.42%
    Epoch 172: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.65%, test macro 81.48%
    Epoch 173: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.69%, test macro 81.52%
    Epoch 174: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.33%
    Epoch 175: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.33%
    Epoch 176: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.55%, test macro 81.38%
    Epoch 177: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.55%, test macro 81.38%
    Epoch 178: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.55%, test macro 81.38%
    Epoch 179: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 180: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.46%, test macro 81.29%
    Epoch 181: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.46%, test macro 81.29%
    Epoch 182: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.46%, test macro 81.29%
    Epoch 183: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.46%, test macro 81.29%
    Epoch 184: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 185: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 186: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 187: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 188: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 189: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 190: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 191: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 192: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 193: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 194: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 81.51%, test macro 81.34%
    Epoch 195: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.55%, test macro 81.39%
    Epoch 196: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.55%, test macro 81.39%
    Epoch 197: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.51%, test macro 81.34%
    Epoch 198: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.51%, test macro 81.34%
    Epoch 199: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.51%, test macro 81.34%
    Epoch 200: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.41%, test macro 81.25%
    Epoch 201: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.41%, test macro 81.26%
    Epoch 202: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.21%
    Epoch 203: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 204: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 205: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 206: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 207: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 208: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 209: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 210: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.22%
    Epoch 211: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.41%, test macro 81.27%
    Epoch 212: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.41%, test macro 81.27%
    Epoch 213: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.41%, test macro 81.27%
    Epoch 214: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 215: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 216: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 217: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 218: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 219: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.36%, test macro 81.23%
    Epoch 220: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.14%
    Epoch 221: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.32%, test macro 81.19%
    Epoch 222: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.14%
    Epoch 223: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.22%, test macro 81.09%
    Epoch 224: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.22%, test macro 81.09%
    Epoch 225: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.14%
    Epoch 226: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.14%
    Epoch 227: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.15%
    Epoch 228: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.15%
    Epoch 229: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.15%
    Epoch 230: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.27%, test macro 81.15%
    Epoch 231: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.22%, test macro 81.1%
    Epoch 232: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.18%, test macro 81.06%
    Epoch 233: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.18%, test macro 81.06%
    Epoch 234: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.18%, test macro 81.06%
    Epoch 235: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.18%, test macro 81.06%
    Epoch 236: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.13%, test macro 81.01%
    Epoch 237: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.08%, test macro 80.96%
    Epoch 238: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.91%
    Epoch 239: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.91%
    Epoch 240: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.91%
    Epoch 241: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 242: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 243: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 244: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 245: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 246: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 247: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.87%
    Epoch 248: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.92%
    Epoch 249: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.92%
    Epoch 250: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.08%, test macro 80.97%
    Epoch 251: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.92%
    Epoch 252: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 81.04%, test macro 80.92%
    Epoch 253: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.88%
    Epoch 254: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.88%
    Epoch 255: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.88%
    Epoch 256: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.99%, test macro 80.88%
    Epoch 257: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.94%, test macro 80.83%
    Epoch 258: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.94%, test macro 80.83%
    Epoch 259: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.67%, test micro 80.94%, test macro 80.83%
    Epoch 260: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.94%, test macro 80.83%
    Epoch 261: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.94%, test macro 80.83%
    Epoch 262: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.94%, test macro 80.83%
    Epoch 263: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.94%, test macro 80.82%
    Epoch 264: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.85%, test macro 80.73%
    Epoch 265: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.75%, test macro 80.64%
    Epoch 266: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.68%
    Epoch 267: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 268: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.85%, test macro 80.73%
    Epoch 269: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 270: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 271: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 272: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 273: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 274: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 95.99%, test micro 80.8%, test macro 80.69%
    Epoch 275: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.73%
    Epoch 276: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.73%
    Epoch 277: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.8%, test macro 80.69%
    Epoch 278: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.8%, test macro 80.69%
    Epoch 279: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 280: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.89%, test macro 80.78%
    Epoch 281: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.89%, test macro 80.78%
    Epoch 282: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.89%, test macro 80.78%
    Epoch 283: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.8%, test macro 80.7%
    Epoch 284: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 285: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 286: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 287: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 288: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.85%, test macro 80.74%
    Epoch 289: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.8%, test macro 80.69%
    Epoch 290: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.8%, test macro 80.69%
    Epoch 291: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.75%, test macro 80.64%
    Epoch 292: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.75%, test macro 80.64%
    Epoch 293: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.54%
    Epoch 294: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 295: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 296: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 297: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 298: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.54%
    Epoch 299: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.54%
    Epoch 300: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.49%
    Epoch 301: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 302: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 303: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 304: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 305: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 306: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 307: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.53%
    Epoch 308: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.49%
    Epoch 309: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.49%
    Epoch 310: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 311: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 312: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 313: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 314: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 315: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 316: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 317: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 318: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.52%, test macro 80.41%
    Epoch 319: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 320: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 321: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 322: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 323: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.45%
    Epoch 324: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 325: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 326: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 327: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 328: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 329: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 330: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.56%, test macro 80.46%
    Epoch 331: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 332: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 333: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 334: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 335: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.61%, test macro 80.5%
    Epoch 336: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.55%
    Epoch 337: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.55%
    Epoch 338: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.55%
    Epoch 339: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.66%, test macro 80.55%
    Epoch 340: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 341: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 342: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.59%
    Epoch 343: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.58%
    Epoch 344: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.66%, test micro 80.71%, test macro 80.58%
    Epoch 345: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.71%, test macro 80.58%
    Epoch 346: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.71%, test macro 80.58%
    Epoch 347: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 348: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 349: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 350: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 351: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 352: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 353: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 354: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 355: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.55%
    Epoch 356: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 357: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 358: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 359: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 360: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 361: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 362: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 363: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 364: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.71%, test macro 80.59%
    Epoch 365: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.71%, test macro 80.59%
    Epoch 366: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.71%, test macro 80.59%
    Epoch 367: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.66%, test macro 80.54%
    Epoch 368: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 369: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 370: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.49%
    Epoch 371: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.5%
    Epoch 372: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.5%
    Epoch 373: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.61%, test macro 80.5%
    Epoch 374: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 375: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 376: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 377: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 378: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 379: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 380: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.46%
    Epoch 381: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.52%, test macro 80.41%
    Epoch 382: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.52%, test macro 80.41%
    Epoch 383: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.52%, test macro 80.41%
    Epoch 384: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.52%, test macro 80.41%
    Epoch 385: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.45%
    Epoch 386: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.56%, test macro 80.45%
    Epoch 387: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.47%, test macro 80.36%
    Epoch 388: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.47%, test macro 80.36%
    Epoch 389: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.47%, test macro 80.35%
    Epoch 390: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.47%, test macro 80.35%
    Epoch 391: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 392: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 393: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.38%, test macro 80.26%
    Epoch 394: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.38%, test macro 80.26%
    Epoch 395: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.38%, test macro 80.26%
    Epoch 396: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.38%, test macro 80.26%
    Epoch 397: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.38%, test macro 80.26%
    Epoch 398: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 399: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 400: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 401: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 402: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 403: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 404: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 405: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.33%, test micro 80.42%, test macro 80.3%
    Epoch 406: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.38%, test macro 80.26%
    Epoch 407: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.38%, test macro 80.26%
    Epoch 408: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.38%, test macro 80.26%
    Epoch 409: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.38%, test macro 80.26%
    Epoch 410: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.28%, test macro 80.16%
    Epoch 411: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.28%, test macro 80.16%
    Epoch 412: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.28%, test macro 80.16%
    Epoch 413: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.28%, test macro 80.16%
    Epoch 414: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.28%, test macro 80.16%
    Epoch 415: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.24%, test macro 80.11%
    Epoch 416: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 94.99%, test micro 80.19%, test macro 80.06%
    Epoch 417: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 418: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 419: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 420: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 421: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 422: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 423: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.06%
    Epoch 424: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 425: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.07%
    Epoch 426: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.24%, test macro 80.11%
    Epoch 427: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.06%
    Epoch 428: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.06%
    Epoch 429: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 430: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 431: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 432: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.06%
    Epoch 433: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.19%, test macro 80.06%
    Epoch 434: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 435: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 436: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 437: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 438: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 439: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 440: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 441: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 442: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.14%, test macro 80.01%
    Epoch 443: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.09%, test macro 79.97%
    Epoch 444: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.88%
    Epoch 445: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.87%
    Epoch 446: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.87%
    Epoch 447: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.87%
    Epoch 448: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.95%, test macro 79.82%
    Epoch 449: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.87%
    Epoch 450: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.87%
    Epoch 451: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.86%
    Epoch 452: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.86%
    Epoch 453: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.86%
    Epoch 454: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 80.0%, test macro 79.86%
    Epoch 455: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.95%, test macro 79.81%
    Epoch 456: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.95%, test macro 79.81%
    Epoch 457: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.91%, test macro 79.77%
    Epoch 458: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.91%, test macro 79.77%
    Epoch 459: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.91%, test macro 79.77%
    Epoch 460: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.91%, test macro 79.77%
    Epoch 461: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.86%, test macro 79.73%
    Epoch 462: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.76%, test macro 79.62%
    Epoch 463: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.81%, test macro 79.67%
    Epoch 464: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.76%, test macro 79.62%
    Epoch 465: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.72%, test macro 79.58%
    Epoch 466: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.67%, test macro 79.52%
    Epoch 467: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.67%, test macro 79.52%
    Epoch 468: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.62%, test macro 79.47%
    Epoch 469: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.62%, test macro 79.47%
    Epoch 470: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.62%, test macro 79.47%
    Epoch 471: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 472: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 473: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 474: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 475: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 476: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 477: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.43%
    Epoch 478: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.42%
    Epoch 479: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.62%, test macro 79.47%
    Epoch 480: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.58%, test macro 79.42%
    Epoch 481: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.48%, test macro 79.33%
    Epoch 482: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.48%, test macro 79.33%
    Epoch 483: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.53%, test macro 79.38%
    Epoch 484: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.48%, test macro 79.34%
    Epoch 485: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.44%, test macro 79.29%
    Epoch 486: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.34%, test macro 79.2%
    Epoch 487: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.34%, test macro 79.2%
    Epoch 488: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.34%, test macro 79.2%
    Epoch 489: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.29%, test macro 79.15%
    Epoch 490: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.25%, test macro 79.1%
    Epoch 491: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.2%, test macro 79.05%
    Epoch 492: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.2%, test macro 79.05%
    Epoch 493: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.2%, test macro 79.05%
    Epoch 494: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.2%, test macro 79.05%
    Epoch 495: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.2%, test macro 79.05%
    Epoch 496: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.11%, test macro 78.96%
    Epoch 497: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.01%, test macro 78.86%
    Epoch 498: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.01%, test macro 78.86%
    Epoch 499: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.01%, test macro 78.86%
    Epoch 500: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 79.01%, test macro 78.86%
    Best model: train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.66%, test micro 82.73%, test macro 81.95%

## Question 2.1: What is your maximum test set **micro** F1 score for the best_model when using mean aggregation? (10 points)

## Question 2.2: What is your maximum test set **macro** F1 score for the best_model when using the mean aggregation? (10 points)

## Training the Attention Aggregation

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  best_model = None
  best_val = 0

  output_size = hetero_graph.num_node_labels('paper')
  model = HeteroGNN(hetero_graph, args, aggr="attn").to(args['device'])
  optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'], weight_decay=args['weight_decay'])

  for epoch in range(args['epochs']):
      loss = train(model, optimizer, hetero_graph, train_idx)
      accs, best_model, best_val = test(model, hetero_graph, [train_idx, val_idx, test_idx], best_model, best_val)
      print(
          f"Epoch {epoch + 1}: loss {round(loss, 5)}, "
          f"train micro {round(accs[0][0] * 100, 2)}%, train macro {round(accs[0][1] * 100, 2)}%, "
          f"valid micro {round(accs[1][0] * 100, 2)}%, valid macro {round(accs[1][1] * 100, 2)}%, "
          f"test micro {round(accs[2][0] * 100, 2)}%, test macro {round(accs[2][1] * 100, 2)}%"
      )
  best_accs, _, _ = test(best_model, hetero_graph, [train_idx, val_idx, test_idx], save_preds=True, agg_type="Attention")
  print(
      f"Best model: "
      f"train micro {round(best_accs[0][0] * 100, 2)}%, train macro {round(best_accs[0][1] * 100, 2)}%, "
      f"valid micro {round(best_accs[1][0] * 100, 2)}%, valid macro {round(best_accs[1][1] * 100, 2)}%, "
      f"test micro {round(best_accs[2][0] * 100, 2)}%, test macro {round(best_accs[2][1] * 100, 2)}%"
  )
```

    Epoch 1: loss 1.09888, train micro 33.33%, train macro 16.67%, valid micro 33.33%, valid macro 16.67%, test micro 31.29%, test macro 15.89%
    Epoch 2: loss 1.08958, train micro 33.33%, train macro 16.67%, valid micro 33.33%, valid macro 16.67%, test micro 31.29%, test macro 15.89%
    Epoch 3: loss 1.05758, train micro 54.33%, train macro 44.95%, valid micro 55.33%, valid macro 46.27%, test micro 49.18%, test macro 39.67%
    Epoch 4: loss 0.99157, train micro 66.17%, train macro 54.84%, valid micro 64.67%, valid macro 52.22%, test micro 62.73%, test macro 50.21%
    Epoch 5: loss 0.87943, train micro 67.67%, train macro 56.55%, valid micro 66.0%, valid macro 53.41%, test micro 64.19%, test macro 51.81%
    Epoch 6: loss 0.72531, train micro 68.33%, train macro 57.82%, valid micro 66.67%, valid macro 54.6%, test micro 64.47%, test macro 52.23%
    Epoch 7: loss 0.56074, train micro 69.5%, train macro 60.56%, valid micro 67.0%, valid macro 55.78%, test micro 65.32%, test macro 53.33%
    Epoch 8: loss 0.41621, train micro 70.83%, train macro 63.16%, valid micro 68.0%, valid macro 57.69%, test micro 65.55%, test macro 53.84%
    Epoch 9: loss 0.30229, train micro 73.17%, train macro 67.37%, valid micro 69.33%, valid macro 60.54%, test micro 65.84%, test macro 54.55%
    Epoch 10: loss 0.21979, train micro 78.0%, train macro 74.73%, valid micro 72.67%, valid macro 66.63%, test micro 66.54%, test macro 56.11%
    Epoch 11: loss 0.16181, train micro 84.33%, train macro 83.0%, valid micro 80.33%, valid macro 77.96%, test micro 67.81%, test macro 58.66%
    Epoch 12: loss 0.12026, train micro 90.83%, train macro 90.42%, valid micro 85.33%, valid macro 84.2%, test micro 70.21%, test macro 63.26%
    Epoch 13: loss 0.08934, train micro 94.67%, train macro 94.54%, valid micro 87.67%, valid macro 86.9%, test micro 72.8%, test macro 67.59%
    Epoch 14: loss 0.06547, train micro 97.0%, train macro 96.96%, valid micro 89.0%, valid macro 88.47%, test micro 74.12%, test macro 69.91%
    Epoch 15: loss 0.0475, train micro 97.5%, train macro 97.48%, valid micro 89.0%, valid macro 88.54%, test micro 75.06%, test macro 71.49%
    Epoch 16: loss 0.03503, train micro 98.0%, train macro 97.99%, valid micro 90.33%, valid macro 90.05%, test micro 75.53%, test macro 72.42%
    Epoch 17: loss 0.02675, train micro 98.17%, train macro 98.16%, valid micro 90.33%, valid macro 90.15%, test micro 75.86%, test macro 73.08%
    Epoch 18: loss 0.02102, train micro 98.83%, train macro 98.83%, valid micro 91.0%, valid macro 90.9%, test micro 75.95%, test macro 73.4%
    Epoch 19: loss 0.01665, train micro 98.83%, train macro 98.83%, valid micro 91.0%, valid macro 90.9%, test micro 76.09%, test macro 73.73%
    Epoch 20: loss 0.01322, train micro 99.17%, train macro 99.17%, valid micro 92.0%, valid macro 91.99%, test micro 76.47%, test macro 74.27%
    Epoch 21: loss 0.01052, train micro 99.17%, train macro 99.17%, valid micro 93.33%, valid macro 93.36%, test micro 77.04%, test macro 74.96%
    Epoch 22: loss 0.00845, train micro 99.5%, train macro 99.5%, valid micro 94.0%, valid macro 94.03%, test micro 77.55%, test macro 75.69%
    Epoch 23: loss 0.00678, train micro 99.67%, train macro 99.67%, valid micro 94.0%, valid macro 94.03%, test micro 78.21%, test macro 76.48%
    Epoch 24: loss 0.00538, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.68%, test micro 79.06%, test macro 77.48%
    Epoch 25: loss 0.00426, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.68%, test micro 79.91%, test macro 78.5%
    Epoch 26: loss 0.00339, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.68%, test micro 80.42%, test macro 79.12%
    Epoch 27: loss 0.00273, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.68%, test micro 81.08%, test macro 79.88%
    Epoch 28: loss 0.00225, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.0%, test micro 81.46%, test macro 80.31%
    Epoch 29: loss 0.00189, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.0%, test micro 81.79%, test macro 80.69%
    Epoch 30: loss 0.00162, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 81.93%, test macro 80.84%
    Epoch 31: loss 0.00141, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.33%, test micro 82.4%, test macro 81.41%
    Epoch 32: loss 0.00124, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.66%, test micro 82.45%, test macro 81.47%
    Epoch 33: loss 0.00111, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.73%, test macro 81.8%
    Epoch 34: loss 0.001, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.92%, test macro 82.02%
    Epoch 35: loss 0.00091, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.87%, test macro 81.98%
    Epoch 36: loss 0.00083, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 83.06%, test macro 82.23%
    Epoch 37: loss 0.00077, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 83.34%, test macro 82.59%
    Epoch 38: loss 0.00071, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 96.99%, test micro 83.25%, test macro 82.54%
    Epoch 39: loss 0.00066, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 96.99%, test micro 83.01%, test macro 82.31%
    Epoch 40: loss 0.00062, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.66%, test micro 83.25%, test macro 82.59%
    Epoch 41: loss 0.00058, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 83.48%, test macro 82.85%
    Epoch 42: loss 0.00055, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 83.62%, test macro 83.02%
    Epoch 43: loss 0.00052, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.32%, test micro 83.48%, test macro 82.9%
    Epoch 44: loss 0.00049, train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.66%, test micro 83.34%, test macro 82.78%
    Epoch 45: loss 0.00047, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.44%, test macro 82.92%
    Epoch 46: loss 0.00045, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.48%, test macro 82.99%
    Epoch 47: loss 0.00043, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.48%, test macro 83.01%
    Epoch 48: loss 0.00041, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.2%, test macro 82.73%
    Epoch 49: loss 0.00039, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.29%, test macro 82.86%
    Epoch 50: loss 0.00038, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.25%, test macro 82.85%
    Epoch 51: loss 0.00036, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.25%, test macro 82.85%
    Epoch 52: loss 0.00035, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.33%, test micro 83.06%, test macro 82.66%
    Epoch 53: loss 0.00034, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.92%, test macro 82.53%
    Epoch 54: loss 0.00033, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 82.92%, test macro 82.55%
    Epoch 55: loss 0.00032, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 82.92%, test macro 82.58%
    Epoch 56: loss 0.00031, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 82.59%, test macro 82.26%
    Epoch 57: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 82.64%, test macro 82.32%
    Epoch 58: loss 0.0003, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.67%, test micro 82.59%, test macro 82.28%
    Epoch 59: loss 0.00029, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.45%, test macro 82.15%
    Epoch 60: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.4%, test macro 82.11%
    Epoch 61: loss 0.00028, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.26%, test macro 81.97%
    Epoch 62: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.31%, test macro 82.02%
    Epoch 63: loss 0.00027, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.31%, test macro 82.03%
    Epoch 64: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.21%, test macro 81.94%
    Epoch 65: loss 0.00026, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.12%, test macro 81.85%
    Epoch 66: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 82.02%, test macro 81.79%
    Epoch 67: loss 0.00025, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 81.93%, test macro 81.69%
    Epoch 68: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 81.93%, test macro 81.7%
    Epoch 69: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 81.93%, test macro 81.7%
    Epoch 70: loss 0.00024, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.0%, test micro 81.88%, test macro 81.65%
    Epoch 71: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.84%, test macro 81.6%
    Epoch 72: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.56%
    Epoch 73: loss 0.00023, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.56%
    Epoch 74: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.56%
    Epoch 75: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.72%
    Epoch 76: loss 0.00022, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.72%
    Epoch 77: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.74%
    Epoch 78: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.74%
    Epoch 79: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.74%
    Epoch 80: loss 0.00021, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.93%, test macro 81.74%
    Epoch 81: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.84%, test macro 81.66%
    Epoch 82: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.84%, test macro 81.66%
    Epoch 83: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.61%
    Epoch 84: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.74%, test macro 81.57%
    Epoch 85: loss 0.0002, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.69%, test macro 81.52%
    Epoch 86: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.62%
    Epoch 87: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.61%
    Epoch 88: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.84%, test macro 81.66%
    Epoch 89: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.84%, test macro 81.66%
    Epoch 90: loss 0.00019, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.79%, test macro 81.64%
    Epoch 91: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.74%, test macro 81.59%
    Epoch 92: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.69%, test macro 81.54%
    Epoch 93: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.69%, test macro 81.54%
    Epoch 94: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.65%, test macro 81.48%
    Epoch 95: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.65%, test macro 81.48%
    Epoch 96: loss 0.00018, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.55%, test macro 81.39%
    Epoch 97: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 98: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 99: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 100: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 101: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 102: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.34%
    Epoch 103: loss 0.00017, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.35%
    Epoch 104: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.35%
    Epoch 105: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.3%
    Epoch 106: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.35%
    Epoch 107: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.3%
    Epoch 108: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 109: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 110: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 111: loss 0.00016, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 112: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.31%
    Epoch 113: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.36%
    Epoch 114: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.55%, test macro 81.4%
    Epoch 115: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.6%, test macro 81.44%
    Epoch 116: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.51%, test macro 81.36%
    Epoch 117: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.31%
    Epoch 118: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.31%
    Epoch 119: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.46%, test macro 81.31%
    Epoch 120: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.27%
    Epoch 121: loss 0.00015, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.27%
    Epoch 122: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 123: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 124: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 125: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 126: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 127: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 128: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 129: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.22%
    Epoch 130: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.27%, test macro 81.11%
    Epoch 131: loss 0.00014, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.27%, test macro 81.11%
    Epoch 132: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.27%, test macro 81.11%
    Epoch 133: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.27%, test macro 81.11%
    Epoch 134: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 135: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 136: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 137: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 138: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 139: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 140: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.32%, test macro 81.16%
    Epoch 141: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.21%
    Epoch 142: loss 0.00013, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.36%, test macro 81.21%
    Epoch 143: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 144: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 145: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 146: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.33%, valid macro 97.34%, test micro 81.41%, test macro 81.26%
    Epoch 147: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.01%, test micro 81.41%, test macro 81.26%
    Epoch 148: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.01%, test micro 81.46%, test macro 81.31%
    Epoch 149: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.01%, test micro 81.46%, test macro 81.31%
    Epoch 150: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.01%, test micro 81.41%, test macro 81.26%
    Epoch 151: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 97.0%, valid macro 97.01%, test micro 81.41%, test macro 81.26%
    Epoch 152: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 153: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 154: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 155: loss 0.00012, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 156: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 157: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 158: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 159: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 160: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 161: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 162: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 163: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 164: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.67%, valid macro 96.68%, test micro 81.32%, test macro 81.17%
    Epoch 165: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.32%, test macro 81.17%
    Epoch 166: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.32%, test macro 81.17%
    Epoch 167: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.13%
    Epoch 168: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.08%
    Epoch 169: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.13%
    Epoch 170: loss 0.00011, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.14%
    Epoch 171: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.14%
    Epoch 172: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.1%
    Epoch 173: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.1%
    Epoch 174: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.1%
    Epoch 175: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.1%
    Epoch 176: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.15%
    Epoch 177: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.15%
    Epoch 178: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.32%, test macro 81.2%
    Epoch 179: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.27%, test macro 81.14%
    Epoch 180: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 181: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 182: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 183: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 184: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 185: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 186: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 187: loss 0.0001, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 188: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.09%
    Epoch 189: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.18%, test macro 81.04%
    Epoch 190: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 191: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 192: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 193: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 194: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.18%, test macro 81.04%
    Epoch 195: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 196: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 197: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 198: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 199: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.0%
    Epoch 200: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.08%, test macro 80.96%
    Epoch 201: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.08%, test macro 80.96%
    Epoch 202: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.13%, test macro 81.01%
    Epoch 203: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.11%
    Epoch 204: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.18%, test macro 81.05%
    Epoch 205: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.18%, test macro 81.06%
    Epoch 206: loss 9e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.11%
    Epoch 207: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.22%, test macro 81.11%
    Epoch 208: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.18%, test macro 81.06%
    Epoch 209: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.08%, test macro 80.98%
    Epoch 210: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.04%, test macro 80.93%
    Epoch 211: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.33%, valid macro 96.35%, test micro 81.04%, test macro 80.93%
    Epoch 212: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.93%
    Epoch 213: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.93%
    Epoch 214: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.93%
    Epoch 215: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.93%
    Epoch 216: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.99%, test macro 80.89%
    Epoch 217: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.94%
    Epoch 218: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 81.04%, test macro 80.94%
    Epoch 219: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.94%, test macro 80.85%
    Epoch 220: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.94%, test macro 80.85%
    Epoch 221: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.8%
    Epoch 222: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.8%
    Epoch 223: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.8%
    Epoch 224: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.94%, test macro 80.84%
    Epoch 225: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.79%
    Epoch 226: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.79%
    Epoch 227: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.79%
    Epoch 228: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.85%, test macro 80.75%
    Epoch 229: loss 8e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.85%, test macro 80.75%
    Epoch 230: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.8%
    Epoch 231: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.89%, test macro 80.8%
    Epoch 232: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.85%, test macro 80.75%
    Epoch 233: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.75%, test macro 80.65%
    Epoch 234: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.75%, test macro 80.65%
    Epoch 235: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.75%, test macro 80.65%
    Epoch 236: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.75%, test macro 80.65%
    Epoch 237: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.71%, test macro 80.6%
    Epoch 238: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.71%, test macro 80.6%
    Epoch 239: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.71%, test macro 80.6%
    Epoch 240: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.61%, test macro 80.51%
    Epoch 241: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.56%, test macro 80.46%
    Epoch 242: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.56%, test macro 80.46%
    Epoch 243: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.56%, test macro 80.46%
    Epoch 244: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.42%, test macro 80.33%
    Epoch 245: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.42%, test macro 80.33%
    Epoch 246: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.47%, test macro 80.37%
    Epoch 247: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.47%, test macro 80.37%
    Epoch 248: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.42%, test macro 80.33%
    Epoch 249: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.52%, test macro 80.42%
    Epoch 250: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.52%, test macro 80.42%
    Epoch 251: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.52%, test macro 80.42%
    Epoch 252: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.52%, test macro 80.42%
    Epoch 253: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.61%, test macro 80.52%
    Epoch 254: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.71%, test macro 80.62%
    Epoch 255: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.75%, test macro 80.67%
    Epoch 256: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.75%, test macro 80.67%
    Epoch 257: loss 7e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.61%, test macro 80.52%
    Epoch 258: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.61%, test macro 80.52%
    Epoch 259: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.61%, test macro 80.52%
    Epoch 260: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.61%, test macro 80.52%
    Epoch 261: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.66%, test macro 80.57%
    Epoch 262: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.66%, test macro 80.57%
    Epoch 263: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.66%, test macro 80.57%
    Epoch 264: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.56%, test macro 80.48%
    Epoch 265: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.52%, test macro 80.43%
    Epoch 266: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 267: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 268: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 269: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 270: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 271: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 272: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.47%, test macro 80.39%
    Epoch 273: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.38%, test macro 80.29%
    Epoch 274: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.33%, test macro 80.25%
    Epoch 275: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.2%
    Epoch 276: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.33%, test macro 80.24%
    Epoch 277: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.19%
    Epoch 278: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.19%
    Epoch 279: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 280: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 281: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 282: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 283: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.18%
    Epoch 284: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.18%
    Epoch 285: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.33%, test macro 80.22%
    Epoch 286: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.33%, test macro 80.22%
    Epoch 287: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.33%, test macro 80.22%
    Epoch 288: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.28%, test macro 80.18%
    Epoch 289: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 290: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.24%, test macro 80.13%
    Epoch 291: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.14%, test macro 80.04%
    Epoch 292: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.09%, test macro 79.99%
    Epoch 293: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.09%, test macro 79.99%
    Epoch 294: loss 6e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.09%, test macro 79.99%
    Epoch 295: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 96.0%, valid macro 96.02%, test micro 80.09%, test macro 79.99%
    Epoch 296: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.09%, test macro 79.99%
    Epoch 297: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.09%, test macro 79.98%
    Epoch 298: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.05%, test macro 79.93%
    Epoch 299: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.88%
    Epoch 300: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.88%
    Epoch 301: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.88%
    Epoch 302: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.89%
    Epoch 303: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.89%
    Epoch 304: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.67%, valid macro 95.69%, test micro 80.0%, test macro 79.89%
    Epoch 305: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.95%, test macro 79.84%
    Epoch 306: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.91%, test macro 79.8%
    Epoch 307: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.86%, test macro 79.75%
    Epoch 308: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.86%, test macro 79.75%
    Epoch 309: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 95.03%, test micro 79.95%, test macro 79.85%
    Epoch 310: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 95.03%, test micro 79.86%, test macro 79.76%
    Epoch 311: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 95.03%, test micro 79.86%, test macro 79.76%
    Epoch 312: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.91%, test macro 79.8%
    Epoch 313: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.86%, test macro 79.76%
    Epoch 314: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.81%, test macro 79.7%
    Epoch 315: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.81%, test macro 79.7%
    Epoch 316: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.35%, test micro 79.81%, test macro 79.7%
    Epoch 317: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 318: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 319: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 320: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 321: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 322: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.76%, test macro 79.65%
    Epoch 323: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.72%, test macro 79.6%
    Epoch 324: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.72%, test macro 79.6%
    Epoch 325: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.67%, test macro 79.55%
    Epoch 326: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.62%, test macro 79.5%
    Epoch 327: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.62%, test macro 79.5%
    Epoch 328: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.58%, test macro 79.46%
    Epoch 329: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.62%, test macro 79.5%
    Epoch 330: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.62%, test macro 79.5%
    Epoch 331: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.67%, test macro 79.55%
    Epoch 332: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.72%, test macro 79.59%
    Epoch 333: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.67%, test macro 79.54%
    Epoch 334: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.67%, test macro 79.55%
    Epoch 335: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.58%, test macro 79.46%
    Epoch 336: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.53%, test macro 79.42%
    Epoch 337: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.58%, test macro 79.47%
    Epoch 338: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.58%, test macro 79.47%
    Epoch 339: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.48%, test macro 79.37%
    Epoch 340: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.44%, test macro 79.33%
    Epoch 341: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.44%, test macro 79.33%
    Epoch 342: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.39%, test macro 79.29%
    Epoch 343: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.39%, test macro 79.29%
    Epoch 344: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.39%, test macro 79.29%
    Epoch 345: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.34%, test macro 79.24%
    Epoch 346: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.25%, test macro 79.14%
    Epoch 347: loss 5e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.29%, test macro 79.18%
    Epoch 348: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.25%, test macro 79.14%
    Epoch 349: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.25%, test macro 79.14%
    Epoch 350: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.25%, test macro 79.14%
    Epoch 351: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.2%, test macro 79.09%
    Epoch 352: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.2%, test macro 79.09%
    Epoch 353: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.11%, test macro 79.0%
    Epoch 354: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.06%, test macro 78.96%
    Epoch 355: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.06%, test macro 78.96%
    Epoch 356: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.01%, test macro 78.9%
    Epoch 357: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.33%, valid macro 95.34%, test micro 79.01%, test macro 78.9%
    Epoch 358: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 95.01%, test micro 78.96%, test macro 78.85%
    Epoch 359: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 95.0%, valid macro 95.01%, test micro 78.96%, test macro 78.85%
    Epoch 360: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.92%, test macro 78.8%
    Epoch 361: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.87%, test macro 78.75%
    Epoch 362: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.92%, test macro 78.79%
    Epoch 363: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.92%, test macro 78.79%
    Epoch 364: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.92%, test macro 78.79%
    Epoch 365: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.87%, test macro 78.74%
    Epoch 366: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.87%, test macro 78.74%
    Epoch 367: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.82%, test macro 78.69%
    Epoch 368: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.78%, test macro 78.65%
    Epoch 369: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.78%, test macro 78.65%
    Epoch 370: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.73%, test macro 78.6%
    Epoch 371: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.73%, test macro 78.6%
    Epoch 372: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.73%, test macro 78.6%
    Epoch 373: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.54%, test macro 78.41%
    Epoch 374: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.54%, test macro 78.41%
    Epoch 375: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 376: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.42%
    Epoch 377: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.42%
    Epoch 378: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.49%, test macro 78.37%
    Epoch 379: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.49%, test macro 78.37%
    Epoch 380: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.49%, test macro 78.36%
    Epoch 381: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.49%, test macro 78.36%
    Epoch 382: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.59%, test macro 78.46%
    Epoch 383: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.59%, test macro 78.46%
    Epoch 384: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 385: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 386: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.59%, test macro 78.46%
    Epoch 387: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.59%, test macro 78.46%
    Epoch 388: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 389: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 390: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 391: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 392: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 393: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 394: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.54%, test macro 78.41%
    Epoch 395: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.49%, test macro 78.37%
    Epoch 396: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.45%, test macro 78.31%
    Epoch 397: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.45%, test macro 78.31%
    Epoch 398: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.45%, test macro 78.31%
    Epoch 399: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.67%, valid macro 94.67%, test micro 78.4%, test macro 78.27%
    Epoch 400: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.35%, test macro 78.23%
    Epoch 401: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.31%, test macro 78.18%
    Epoch 402: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.26%, test macro 78.14%
    Epoch 403: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.16%, test macro 78.05%
    Epoch 404: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.16%, test macro 78.04%
    Epoch 405: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.16%, test macro 78.04%
    Epoch 406: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.16%, test macro 78.04%
    Epoch 407: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.02%, test macro 77.9%
    Epoch 408: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.93%, test macro 77.8%
    Epoch 409: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.98%, test macro 77.84%
    Epoch 410: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 78.02%, test macro 77.89%
    Epoch 411: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.98%, test macro 77.84%
    Epoch 412: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.88%, test macro 77.74%
    Epoch 413: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.84%, test macro 77.69%
    Epoch 414: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.79%, test macro 77.64%
    Epoch 415: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.74%, test macro 77.59%
    Epoch 416: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.69%, test macro 77.55%
    Epoch 417: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.69%, test macro 77.55%
    Epoch 418: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.6%, test macro 77.45%
    Epoch 419: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.6%, test macro 77.45%
    Epoch 420: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.65%, test macro 77.51%
    Epoch 421: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.65%, test macro 77.51%
    Epoch 422: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.6%, test macro 77.46%
    Epoch 423: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.6%, test macro 77.46%
    Epoch 424: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.33%, valid macro 94.34%, test micro 77.55%, test macro 77.42%
    Epoch 425: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.51%, test macro 77.38%
    Epoch 426: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.51%, test macro 77.38%
    Epoch 427: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.46%, test macro 77.33%
    Epoch 428: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.46%, test macro 77.33%
    Epoch 429: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.46%, test macro 77.33%
    Epoch 430: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.41%, test macro 77.28%
    Epoch 431: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.36%, test macro 77.23%
    Epoch 432: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.36%, test macro 77.23%
    Epoch 433: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 434: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 435: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 436: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.13%, test macro 76.98%
    Epoch 437: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.13%, test macro 76.98%
    Epoch 438: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.13%, test macro 76.98%
    Epoch 439: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 440: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 441: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.18%, test macro 77.02%
    Epoch 442: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.04%, test macro 76.88%
    Epoch 443: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.08%, test macro 76.93%
    Epoch 444: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 77.08%, test macro 76.93%
    Epoch 445: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.99%, test macro 76.85%
    Epoch 446: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.99%, test macro 76.85%
    Epoch 447: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.99%, test macro 76.85%
    Epoch 448: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.99%, test macro 76.85%
    Epoch 449: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.85%, test macro 76.71%
    Epoch 450: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.85%, test macro 76.71%
    Epoch 451: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.85%, test macro 76.71%
    Epoch 452: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 453: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 454: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 455: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.75%, test macro 76.6%
    Epoch 456: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.75%, test macro 76.6%
    Epoch 457: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 458: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 459: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.8%, test macro 76.65%
    Epoch 460: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 461: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 462: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 463: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 464: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 465: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.8%, test macro 76.65%
    Epoch 466: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.85%, test macro 76.69%
    Epoch 467: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.85%, test macro 76.69%
    Epoch 468: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.75%, test macro 76.59%
    Epoch 469: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.71%, test macro 76.54%
    Epoch 470: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.66%, test macro 76.49%
    Epoch 471: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.66%, test macro 76.49%
    Epoch 472: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 94.0%, valid macro 94.0%, test micro 76.71%, test macro 76.53%
    Epoch 473: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.66%, test macro 76.49%
    Epoch 474: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.61%, test macro 76.44%
    Epoch 475: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.61%, test macro 76.43%
    Epoch 476: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.61%, test macro 76.43%
    Epoch 477: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.67%, test micro 76.61%, test macro 76.43%
    Epoch 478: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.52%, test macro 76.34%
    Epoch 479: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.47%, test macro 76.29%
    Epoch 480: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.47%, test macro 76.29%
    Epoch 481: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.47%, test macro 76.29%
    Epoch 482: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.42%, test macro 76.25%
    Epoch 483: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.42%, test macro 76.25%
    Epoch 484: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.38%, test macro 76.21%
    Epoch 485: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.38%, test macro 76.21%
    Epoch 486: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.24%, test macro 76.07%
    Epoch 487: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.28%, test macro 76.11%
    Epoch 488: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.28%, test macro 76.11%
    Epoch 489: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.24%, test macro 76.06%
    Epoch 490: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.67%, valid macro 93.66%, test micro 76.19%, test macro 76.02%
    Epoch 491: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 76.19%, test macro 76.02%
    Epoch 492: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 76.05%, test macro 75.89%
    Epoch 493: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 76.0%, test macro 75.84%
    Epoch 494: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.95%, test macro 75.8%
    Epoch 495: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.95%, test macro 75.8%
    Epoch 496: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.95%, test macro 75.8%
    Epoch 497: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.95%, test macro 75.8%
    Epoch 498: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.91%, test macro 75.74%
    Epoch 499: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.91%, test macro 75.74%
    Epoch 500: loss 4e-05, train micro 100.0%, train macro 100.0%, valid micro 93.33%, valid macro 93.32%, test micro 75.91%, test macro 75.74%
    Saving Heterogeneous Node Prediction Model Predictions with Agg: Attention

    Best model: train micro 100.0%, train macro 100.0%, valid micro 97.67%, valid macro 97.66%, test micro 83.34%, test macro 82.78%

## Question 2.3: What is your maximum test set **micro** F1 score for the best_model when using the attention aggregation? (4 points)

## Question 2.4: What is your maximum test set **macro** F1 score for the best_model when using the attention aggregation? (4 points)

## Attention for each Message Type

Through message type level attention we can learn which message type is more important to which layer.

Here we will print out and show that each layer pay how much attention on each message type.

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:
  if model.convs1.alpha is not None and model.convs2.alpha is not None:
      for idx, message_type in model.convs1.mapping.items():
          print(f"Layer 1 has attention {model.convs1.alpha[idx]} on message type {message_type}")
      for idx, message_type in model.convs2.mapping.items():
          print(f"Layer 2 has attention {model.convs2.alpha[idx]} on message type {message_type}")
```

    Layer 1 has attention 0.032038453966379166 on message type ('paper', 'author', 'paper')
    Layer 1 has attention 0.9679614901542664 on message type ('paper', 'subject', 'paper')
    Layer 2 has attention 0.7597429752349854 on message type ('paper', 'author', 'paper')
    Layer 2 has attention 0.24025702476501465 on message type ('paper', 'subject', 'paper')

# Submission

You will need to submit three files on Gradescope to complete this notebook.

1.   Your completed *CS224W_Colab5.ipynb*. From the "File" menu select "Download .ipynb" to save a local copy of your completed Colab.
2.  *ACM-Node-MeanAgg.csv*
3.  *ACM-Node-AttentionAgg.csv*

Download the csv files by selecting the *Folder* icon on the left panel.

To submit your work, zip the files downloaded in steps 1-3 above and submit to gradescope. **NOTE:** DO NOT rename any of the downloaded files.