# Graph Neural Networks

The biggest difficulty for deep learning with molecules is the choice and computation of "descriptors". Graph neural networks (GNNs) are a category of deep neural networks whose inputs are graphs and provide a way around the choice of descriptors. A GNN can take a molecule directly as input.

As usual, they are composed of specific layers that input a graph and those layers are what we're interested in. You can find reviews of GNNs in Dwivedi *et al.*{cite}`dwivedi2020benchmarking`, Bronstein *et al.*{cite}`bronstein2017geometric`, and  Wu *et al.*{cite}`wu2020comprehensive`. GNNs can be used for everything from coarse-grained molecular dynamics {cite}`li2020graph` to predicting NMR chemical shifts {cite}`yang2020predicting` to modeling dynamics of solids {cite}`xie2019graph`. Before we dive too deep into them, we must first understand how a graph is represented and how molecules are converted into graphs.

You can find an interactive introductory article on graphs and graph neural networks at [distill.pub](https://distill.pub/2021/gnn-intro/) {cite}`sanchez-lengeling2021a`.

## Representing a Graph

A graph $\mathbf{G}$ is a set of nodes $\mathbf{V}$ and edges $\mathbf{E}$. In our setting, node $i$ is defined by a vector $\vec{v}_i$, so that the set of nodes can be written as a rank 2 tensor. The edges can be represented as an adjacency matrix $\mathbf{E}$, where if $e_{ij} = 1$ then nodes $i$ and $j$ are connected by an edge. In many fields, graphs are often immediately simplified to be directed and acyclic, which simplifies things. Molecules are instead undirected and have cycles (rings). Thus, our adjacency matrices are always symmetric $e_{ij} = e_{ji}$. Often our edges themselves have features, so that $e_{ij}$ is itself a vector. Then the adjacency matrix becomes a rank 3 tensor. Examples of edge features might be covalent bond order or distance between two nodes.

```{figure} ./methanol.jpg
----
name: methanol
width: 400px
----
Methanol with atoms numbered so that we can convert it to a graph.
```

Let's see how a graph can be constructed from a molecule. Consider methanol, shown in {numref}`methanol`. I've numbered the atoms so that we have an order for defining the nodes/edges. First, the node features. You can use anything for node features, but often we'll begin with one-hot encoded feature vectors:

| Node | C  | H  | O  |
|:-----|----|----|---:|
| 1    | 0  | 1  |  0 |
| 2    | 0  | 1  |  0 |
| 3    | 0  | 1  |  0 |
| 4    | 1  | 0  |  0 |
| 5    | 0  | 0  |  1 |
| 6    | 0  | 1  |  0 |

$\mathbf{V}$ will be the combined feature vectors of these nodes. The adjacency matrix $\mathbf{E}$ will look like:

|    | 1  | 2  | 3  | 4  | 5  | 6  |
|:---|----|----|----|----|----|---:|
| 1  | 0  | 0  | 0  | 1  | 0  |  0 |
| 2  | 0  | 0  | 0  | 1  | 0  |  0 |
| 3  | 0  | 0  | 0  | 1  | 0  |  0 |
| 4  | 1  | 1  | 1  | 0  | 1  |  0 |
| 5  | 0  | 0  | 0  | 1  | 0  |  1 |
| 6  | 0  | 0  | 0  | 0  | 1  |  0 |

Take a moment to understand these two. For example, notice that rows 1, 2, and 3 only have the 4th column as non-zero. That's because atoms 1-3 are bonded only to carbon (atom 4). Also, the diagonal is always 0 because atoms cannot be bonded with themselves.

You can find a similar process for converting crystals into graphs in Xie et al. {cite}`Xie2018Crystal`. We'll now begin with a function which can convert a smiles string into this representation.

## Running This Notebook

Click the  <i aria-label="Launch interactive content" class="fas fa-rocket"></i>  above to launch this page as an interactive Google Colab. See details below on installing packages, either on your own environment or on Google Colab

````{tip} My title
:class: dropdown
To install packages, execute this code in a new cell
```
!pip install jupyter-book matplotlib numpy tensorflow pydot seaborn Pillow rdkit-pypi
```

````

```python
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib as mpl
import numpy as np
import tensorflow as tf
import warnings
import pandas as pd
import rdkit, rdkit.Chem, rdkit.Chem.rdDepictor, rdkit.Chem.Draw
import networkx as nx

warnings.filterwarnings("ignore")
sns.set_context("notebook")
sns.set_style(
    "dark",
    {
        "xtick.bottom": True,
        "ytick.left": True,
        "xtick.color": "#666666",
        "ytick.color": "#666666",
        "axes.edgecolor": "#666666",
        "axes.linewidth": 0.8,
        "figure.dpi": 300,
    },
)
color_cycle = ["#1BBC9B", "#F06060", "#5C4B51", "#F3B562", "#6e5687"]
mpl.rcParams["axes.prop_cycle"] = mpl.cycler(color=color_cycle)
# soldata = pd.read_csv('https://dataverse.harvard.edu/api/access/datafile/3407241?format=original&gbrecs=true')
# had to rehost because dataverse isn't reliable
soldata = pd.read_csv(
    "https://github.com/whitead/dmol-book/raw/master/data/curated-solubility-dataset.csv"
)
np.random.seed(0)
my_elements = {6: "C", 8: "O", 1: "H"}
```

The hidden cell below defines our function `smiles2graph`. This creates one-hot node feature vectors for the element C, H, and O. It also creates an adjacency tensor with one-hot bond order being the feature vector.

```python
def smiles2graph(sml):
    """Argument for the RD2NX function should be a valid SMILES sequence
    returns: the graph
    """
    m = rdkit.Chem.MolFromSmiles(sml)
    m = rdkit.Chem.AddHs(m)
    order_string = {
        rdkit.Chem.rdchem.BondType.SINGLE: 1,
        rdkit.Chem.rdchem.BondType.DOUBLE: 2,
        rdkit.Chem.rdchem.BondType.TRIPLE: 3,
        rdkit.Chem.rdchem.BondType.AROMATIC: 4,
    }
    N = len(list(m.GetAtoms()))
    nodes = np.zeros((N, len(my_elements)))
    lookup = list(my_elements.keys())
    for i in m.GetAtoms():
        nodes[i.GetIdx(), lookup.index(i.GetAtomicNum())] = 1

    adj = np.zeros((N, N, 5))
    for j in m.GetBonds():
        u = min(j.GetBeginAtomIdx(), j.GetEndAtomIdx())
        v = max(j.GetBeginAtomIdx(), j.GetEndAtomIdx())
        order = j.GetBondType()
        if order in order_string:
            order = order_string[order]
        else:
            raise Warning("Ignoring bond order" + order)
        adj[u, v, order] = 1
        adj[v, u, order] = 1
    return nodes, adj
```

```python
nodes, adj = smiles2graph("CO")
nodes
```

## A Graph Neural Network

A graph neural network (GNN) is a neural network with two defining attributes:

1. It's input is a graph
2. It's output is permutation invariant

We can understand clearly the first point. Here, a graph permutation means re-ordering our nodes. In our methanol example above, we could have easily made the carbon be atom 1 instead of atom 4. Our new adjacency matrix would then be:

|    | 1  | 2  | 3  | 4  | 5  | 6  |
|:---|----|----|----|----|----|---:|
| 1  | 0  | 1  | 1  | 1  | 1  |  0 |
| 2  | 1  | 0  | 0  | 0  | 0  |  0 |
| 3  | 1  | 0  | 0  | 0  | 0  |  0 |
| 4  | 1  | 0  | 0  | 0  | 1  |  0 |
| 5  | 1  | 0  | 0  | 0  | 0  |  1 |
| 6  | 0  | 0  | 0  | 0  | 1  |  0 |

```{margin}
Ok, so technically we might want our GNN to be permutation *equivariant*. If our GNN outputs per-node features, then obviously if we swap the node order of input, we want our per-node output to swap.
```

A GNN is permutation invariant if the output is insensitive to these kind of exchanges. Of course, there may exist GNNs out there which are not permutation invariant, especially if they are for trees where it is possible to deterministically order all nodes. Yet all the GNNs used in chemistry and most of the deep learning work is concerned with GNNs that are permutation invariant.

### A simple GNN

We will often mention a GNN when we really mean a layer from a GNN. Most GNNs implement a specific layer that can deal with graphs, and so usually we are only concerned with this layer. Let's see an example of a simple layer for a GNN:

\begin{equation}
f_k = \sigma\left( \sum_i \sum_j v_{ij}w_{jk}  \right)
\end{equation}

This equation shows that we first multiply every node feature by trainable weights $w_{jk}$, sum over all node features, and then apply an activation. This will yield a single feature vector for the graph. Is this equation permutation invariant? Yes, because the node index in our expression is index $i$ which can be re-ordered without affecting the output.

Let's see an example that is similar, but not permutation invariant:

\begin{equation}
f_k = \sigma\left( \sum_i v_{ij}w_{ik}  \right)
\end{equation}

This is a small change. We have one weight vector per node now. This makes the trainable weights depend on the ordering of the nodes. Then if we swap the node ordering, our weights will no longer align. So if we were to input two methanol molecules, which should have the same output, but we switched two atom numbers, we would get different answers. These simple examples differ from real GNNs in two important ways: (i) they give a single feature vector output, which throws away per-node information, and (ii) they do not use the adjacency matrix. Let's see a real GNN that has these properties while maintaining permutation invariance.

## Kipf & Welling GCN

One of the first popular GNNs was the Kipf & Welling graph convolutional network (GCN) {cite}`kipf2016semi`. Although some people consider GCNs to be a broad class of GNNs, we'll use GCNs to refer specifically the Kipf & Welling GCN.
Thomas Kipf has written an [excellent article introducing the GCN](https://tkipf.github.io/graph-convolutional-networks/). I will not repeat this article, so please take a look at it.

The input to a GCN layer is $\mathbf{V}$, $\mathbf{E}$ and it outputs an updated $\mathbf{V}'$. Each node feature vector is updated. The way it updates a node feature vector is by averaging the feature vectors of its neighbors, as determined by $\mathbf{E}$. The choice of averaging over neighbors is what makes a GCN layer permutation invariant. Averaging over neighbors is not trainable, so we must add trainable parameters. We multiply the neighbor features by a trainable matrix before the averaging, which gives the GCN the ability to learn. In Einstein notation, this process is:

\begin{equation}
v_{il} = \sigma\left(\frac{1}{d_i}e_{ij}v_{jk}w_{lk}\right)
\end{equation}

where $i$ is the node we're considering, $j$ is the neighbor index, $k$ is the node input feature, $l$ is the output node feature, $d_i$ is the degree of node i (which makes it an average instead of sum), $e_{ij}$ isolates neighbors so that all non-neighbor $v_{jk}$s are zero, $\sigma$ is our activation, and $w_{lk}$ is the trainable weights. This equation is a mouthful, but it truly just is the average over neighbors with a trainable matrix thrown in. One common modification is to make all nodes neighbors of themselves. This is so that the output node features $v_{il}$ depends on the input features $v_{ik}$. We do not need to change our equation, just make the adjacency matrix have $1$s on the diagonal instead of $0$ by adding the identity matrix during pre-processing.

Building understanding about the GCN is important for understanding other GNNs. You can view the GCN layer as a way to "communicate" between a node and its neighbors. The output for node $i$ will depend only on its immediate neighbors. For chemistry, this is not satisfactory. You can stack multiple layers though. If you have two layers, the output for node $i$ will include information about node $i$'s neighbors' neighbors. Another important detail to understand in GCNs is that the averaging procedure accomplishes two goals: (i) it gives permutation invariance by removing the effect of neighbor order and (ii) it prevents a change in magnitude in node features. A sum would accomplish (i) but would cause the magnitude of the node features to grow after each layer. Of course, you could ad-hoc put a batch normalization layer after each GCN layer to keep output magnitudes stable but averaging is easy.

```python
# THIS CELL IS USED TO GENERATE A FIGURE
# AND NOT RELATED TO CHAPTER
# YOU CAN SKIP IT
from myst_nb import glue
from moviepy.editor import VideoClip
from moviepy.video.io.bindings import mplfig_to_npimage

def draw_vector(x, y, s, v, ax, cmap, **kwargs):
    x += s / 2
    y += s / 2
    for vi in v:
        if cmap is not None:
            ax.add_patch(
                mpl.patches.Rectangle((x, y), s * 1.5, s, facecolor=cmap(vi), **kwargs)
            )
        else:
            ax.add_patch(
                mpl.patches.Rectangle(
                    (x, y), s * 1.5, s, facecolor="#FFF", edgecolor="#333", **kwargs
                )
            )
        ax.text(
            x + s * 1.5 / 2,
            y + s / 2,
            "{:.2f}".format(vi),
            verticalalignment="center",
            horizontalalignment="center",
        )
        y += s

def draw_key(x, y, s, v, ax, cmap, **kwargs):
    x += s / 2
    y += s / 2
    for vi in v:
        ax.add_patch(
            mpl.patches.Rectangle((x, y), s * 1.5, s, facecolor=cmap(1.0), **kwargs)
        )
        ax.text(
            x + s * 1.5 / 2,
            y + s / 2,
            vi,
            verticalalignment="center",
            horizontalalignment="center",
        )
        y += s
    ax.text(
        x, y + s / 2, "Key:", verticalalignment="center", horizontalalignment="left"
    )

def draw(
    nodes, adj, ax, highlight=None, key=False, labels=None, mask=None, draw_nodes=None
):
    G = nx.Graph()
    for i in range(adj.shape[0]):
        for j in range(adj.shape[0]):
            if np.any(adj[i, j]):
                G.add_edge(i, j)
    if mask is None:
        mask = [True] * len(G)
    if draw_nodes is None:
        draw_nodes = nodes
    # go from atomic number to element
    elements = np.argmax(draw_nodes, axis=-1)
    el_labels = {i: list(my_elements.values())[e] for i, e in enumerate(elements)}
    pos = nx.nx_agraph.graphviz_layout(G, prog="sfdp")
    pos = nx.rescale_layout_dict(pos)
    c = ["white"] * len(G)
    all_h = []
    if highlight is not None:
        for i, h in enumerate(highlight):
            for hj in h:
                c[hj] = "C{}".format(i)
                all_h.append(hj)
    nx.draw(G, ax=ax, pos=pos, labels=el_labels, node_size=700, node_color=c)
    cmap = plt.get_cmap("Wistia")
    for i in range(len(G)):
        if not mask[i]:
            continue
        if i in all_h:
            draw_vector(*pos[i], 0.15, nodes[i], ax, cmap)
        else:
            draw_vector(*pos[i], 0.15, nodes[i], ax, None)
    if key:
        draw_key(-1, -1, 0.15, my_elements.values(), ax, cmap)
    if labels is not None:
        legend_elements = []
        for i, l in enumerate(labels):
            p = mpl.lines.Line2D(
                [0], [0], marker="o", color="C{}".format(i), label=l, markersize=15
            )
            legend_elements.append(p)
        ax.legend(handles=legend_elements)
    ax.set_xlim(-1.2, 1.2)
    ax.set_ylim(-1.2, 1.2)

plt.figure()
draw(nodes, adj, plt.gca(), highlight=[[1], [5, 0]], labels=["center", "neighbors"])
glue("dframe", plt.gcf(), display=False)
```

```python
# THIS CELL IS USED TO GENERATE A FIGURE
# AND NOT RELATED TO CHAPTER
# YOU CAN SKIP IT
fig, axs = plt.subplots(1, 2, squeeze=True, figsize=(12, 4))
order = [5, 1, 0, 2, 3, 4]
time_per_node = 2
last_layer = [0]
layers = 2
input_nodes = np.copy(nodes)

def make_frame(t):
    axs[0].clear()
    axs[1].clear()

    layer_i = int(t / (time_per_node * len(order)))
    axs[0].set_title(f"Layer {layer_i + 1} Input")
    axs[1].set_title(f"Layer {layer_i + 1} Output")

    flat_adj = np.sum(adj, axis=-1)
    out_nodes = np.einsum(
        "i,ij,jk->ik",
        1 / (np.sum(flat_adj, axis=1) + 1),
        flat_adj + np.eye(*flat_adj.shape),
        nodes,
    )

    if last_layer[0] != layer_i:
        print("recomputing")
        nodes[:] = out_nodes
        last_layer[0] = layer_i

    t -= layer_i * time_per_node * len(order)
    i = order[int(t / time_per_node)]
    print(last_layer, layer_i, i, t)
    mask = [False] * nodes.shape[0]
    for j in order[: int(t / time_per_node) + 1]:
        mask[j] = True
    print(mask, i)
    neighs = list(np.where(adj[i])[0])
    if (t - int(t / time_per_node) * time_per_node) >= time_per_node / 4:
        draw(
            nodes,
            adj,
            axs[0],
            highlight=[[i], neighs],
            labels=["center", "neighbors"],
            draw_nodes=input_nodes,
        )
    else:
        draw(
            nodes,
            adj,
            axs[0],
            highlight=[[i]],
            labels=["center", "neighbors"],
            draw_nodes=input_nodes,
        )
    if (t - int(t / time_per_node) * time_per_node) < time_per_node / 2:
        mask[j] = False
    draw(
        out_nodes,
        adj,
        axs[1],
        highlight=[[i]],
        key=True,
        mask=mask,
        draw_nodes=input_nodes,
    )
    return mplfig_to_npimage(fig)

animation = VideoClip(make_frame, duration=time_per_node * nodes.shape[0] * layers)

animation.write_gif("../_static/images/gcn.gif", fps=2)
```

```{glue:figure} dframe
----
name: dframe
----
Intermediate step of the graph convolution layer. The center node is being updated by averaging its neighbors features.
```

```{figure} ../_static/images/gcn.gif
----
name: gcnanim
----
Animation of the graph convolution layer. The left is input, right is output node features. Note that two layers are shown (see title change).
```

To help understand the GCN layer, look at {numref}`dframe`. It shows an intermediate step of the GCN layer. Each node feature is represented here as a one-hot encoded vector at input. The animation in {numref}`gcnanim` shows the averaging process over neighbor features.  To make this animation easy to follow, the trainable weights and activation functions are not considered. Note that the animation repeats for a second layer. Watch how the "information" about there being an oxygen atom in the molecule is propagated only after two layers to each atom. All GNNs operate with similar approaches, so try to understand how this animation works.

### GCN Implementation

Let's now create a tensor implementation of the GCN. We'll skip the activation and trainable weights for now.
We must first compute our rank 2 adjacency matrix. The `smiles2graph` code above computes an adjacency tensor with feature vectors. We can fix that with a simple reduction and add the identity at the same time

```python
nodes, adj = smiles2graph("CO")
adj_mat = np.sum(adj, axis=-1) + np.eye(adj.shape[0])
adj_mat
```

To compute degree of each node, we can do another reduction:

```python
degree = np.sum(adj_mat, axis=-1)
degree
```

Now we can put all these pieces together into the Einstein equation

```python
print(nodes[0])
# note to divide by degree, make the input 1 / degree
new_nodes = np.einsum("i,ij,jk->ik", 1 / degree, adj_mat, nodes)
print(new_nodes[0])
```

To now implement this as a layer in Keras, we must put this code above into a new Layer subclass. The code is relatively straightforward, but you can read-up on the function names and Layer class in [this tutorial](https://keras.io/guides/making_new_layers_and_models_via_subclassing/). The three main changes are that we create trainable parameters `self.w` and use them in the {obj}`tf.einsum`, we use an activation `self.activation`, and we output both our new node features and the adjacency matrix. The reason to output the adjacency matrix is so that we can stack multiple GCN layers without having to pass the adjacency matrix each time.

```python
class GCNLayer(tf.keras.layers.Layer):
    """Implementation of GCN as layer"""

    def __init__(self, activation=None, **kwargs):
        # constructor, which just calls super constructor
        # and turns requested activation into a callable function
        super(GCNLayer, self).__init__(**kwargs)
        self.activation = tf.keras.activations.get(activation)

    def build(self, input_shape):
        # create trainable weights
        node_shape, adj_shape = input_shape
        self.w = self.add_weight(shape=(node_shape[2], node_shape[2]), name="w")

    def call(self, inputs):
        # split input into nodes, adj
        nodes, adj = inputs
        # compute degree
        degree = tf.reduce_sum(adj, axis=-1)
        # GCN equation
        new_nodes = tf.einsum("bi,bij,bjk,kl->bil", 1 / degree, adj, nodes, self.w)
        out = self.activation(new_nodes)
        return out, adj
```

We can now try our layer:

```python
gcnlayer = GCNLayer("relu")
# we insert a batch axis here
gcnlayer((nodes[np.newaxis, ...], adj_mat[np.newaxis, ...]))
```

It outputs (1) the new node features and (2) the adjacency matrix. Let's make sure we can stack these and apply the GCN multiple times

```python
x = (nodes[np.newaxis, ...], adj_mat[np.newaxis, ...])
for i in range(2):
    x = gcnlayer(x)
x
```

It works! Why do we see zeros though? Probably because we had negative numbers that were removed by our ReLU activation. This will be solved by training and increasing our dimension number.

## Solubility Example

We'll now revisit predicting solubility with GCNs. Remember before that we used the features included with the dataset. Now we can use the molecular structures directly. Our GCN layer outputs node-level features. To predict solubility, we need to get a graph-level feature. We'll see later how to be more sophisticated in this process, but for now let's just take the average over all node features after our GCN layers. This is simple, permutation invariant, and gets us from node-level to graph level. Here's an implementation of this

```python
class GRLayer(tf.keras.layers.Layer):
    """A GNN layer that computes average over all node features"""

    def __init__(self, name="GRLayer", **kwargs):
        super(GRLayer, self).__init__(name=name, **kwargs)

    def call(self, inputs):
        nodes, adj = inputs
        reduction = tf.reduce_mean(nodes, axis=1)
        return reduction
```

To complete our deep solubility predictor, we can add some dense layers and make sure we have a single-output without activation since we're doing regression. Note this model is defined using the [Keras functional API](https://keras.io/guides/functional_api/) which is necessary when you have multiple inputs.

```python
ninput = tf.keras.Input(
    (
        None,
        100,
    )
)
ainput = tf.keras.Input(
    (
        None,
        None,
    )
)
# GCN block
x = GCNLayer("relu")([ninput, ainput])
x = GCNLayer("relu")(x)
x = GCNLayer("relu")(x)
x = GCNLayer("relu")(x)
# reduce to graph features
x = GRLayer()(x)
# standard layers (the readout)
x = tf.keras.layers.Dense(16, "tanh")(x)
x = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs=(ninput, ainput), outputs=x)
```

where does the 100 come from? Well, this dataset has lots of elements so we cannot use our size 3 one-hot encodings because we'll have more than 3 unique elements. We previously only had C, H and O. This is a good time to update our `smiles2graph` function to deal with this.

```python
def gen_smiles2graph(sml):
    """Argument for the RD2NX function should be a valid SMILES sequence
    returns: the graph
    """
    m = rdkit.Chem.MolFromSmiles(sml)
    m = rdkit.Chem.AddHs(m)
    order_string = {
        rdkit.Chem.rdchem.BondType.SINGLE: 1,
        rdkit.Chem.rdchem.BondType.DOUBLE: 2,
        rdkit.Chem.rdchem.BondType.TRIPLE: 3,
        rdkit.Chem.rdchem.BondType.AROMATIC: 4,
    }
    N = len(list(m.GetAtoms()))
    nodes = np.zeros((N, 100))
    for i in m.GetAtoms():
        nodes[i.GetIdx(), i.GetAtomicNum()] = 1

    adj = np.zeros((N, N))
    for j in m.GetBonds():
        u = min(j.GetBeginAtomIdx(), j.GetEndAtomIdx())
        v = max(j.GetBeginAtomIdx(), j.GetEndAtomIdx())
        order = j.GetBondType()
        if order in order_string:
            order = order_string[order]
        else:
            raise Warning("Ignoring bond order" + order)
        adj[u, v] = 1
        adj[v, u] = 1
    adj += np.eye(N)
    return nodes, adj
```

```python
nodes, adj = gen_smiles2graph("CO")
model((nodes[np.newaxis, ...], adj_mat[np.newaxis, ...]))
```

```{margin}
We have switched from adjacency tensor to matrix only because a GCN cannot use edge features. Other architectures though can.
```
It outputs one number! That's always nice to have. Now we need to do some work to get a trainable dataset. Our dataset is a little bit complex because our features are tuples of tensors($\mathbf{V}, \mathbf{E}$) so that our dataset is a tuple of tuples: $\left((\mathbf{V}, \mathbf{E}), y\right)$. We use a **generator**, which is just a python function that can return multiple times. Our function returns once for every training example. Then we have to pass it to the `from_generator` {obj}`tf.data.Dataset` constructor which requires explicit declaration of the shapes of these examples.

```python
def example():
    for i in range(len(soldata)):
        graph = gen_smiles2graph(soldata.SMILES[i])
        sol = soldata.Solubility[i]
        yield graph, sol

data = tf.data.Dataset.from_generator(
    example,
    output_types=((tf.float32, tf.float32), tf.float32),
    output_shapes=(
        (tf.TensorShape([None, 100]), tf.TensorShape([None, None])),
        tf.TensorShape([]),
    ),
)
```

Whew, that's a lot. Now we can do our usual splitting of the dataset.

```python
test_data = data.take(200)
val_data = data.skip(200).take(200)
train_data = data.skip(400)
```

And finally, time to train.

```python
model.compile("adam", loss="mean_squared_error")
result = model.fit(
    train_data.batch(1), validation_data=val_data.batch(1), epochs=20, verbose=0
)
```

```python
plt.plot(result.history["loss"], label="training")
plt.plot(result.history["val_loss"], label="validation")
plt.legend()
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.show()
```

This model is definitely underfit. One reason is that our batch size is 1. This is a side-effect of making the number of atoms variable and then Keras/tensorflow has trouble batching together our data if there are two unknown dimensions. You can fix this by manually batching or padding all molecules to have as many atoms as the one with the max. In any case, this example shows how to use GCN layers in a complete model.

## Message Passing Viewpoint

One way to more broadly view a GCN layer is that it is a kind of "message-passing" layer. You first compute a message coming from each neighboring node:

\begin{equation}
\vec{e}_{{s_i}j} = \vec{v}_{{s_i}j} \mathbf{W}
\end{equation}

where $v_{{s_i}j}$ means the $j$th neighbor of node $i$. The $s_i$ means senders to $i$. This is how a GCN computes the messages, it's just a weight matrix times each neighbor node features. After getting the messages that will go to node $i$, $\vec{e}_{{s_i}j}$, we aggregate them using a function which is permutation invariant to the order of neighbors:

\begin{equation}
\vec{e}_{i} = \frac{1}{|\vec{e}_{{s_i}j}|}\sum \vec{e}_{{s_i}j}
\end{equation}

In the GCN this aggregation is just a mean, but it can be any permutation invariant (possibly trainable) function. Finally, we update our node using the aggregated message in the GCN:

\begin{equation}
\vec{v}^{'}_{i} = \sigma(\vec{e}_i)
\end{equation}

where $v^{'}$ indicates the new node features. This is simply the activated aggregated message. Writing it out this way, you can see how it is possible to make small changes. One important paper by Gilmer et al. explored some of these choices and described how this general idea of message passing layers does well in learning to predict molecular energies from quantum mechanics {cite}`gilmer2017neural`. Examples of changes to the above GCN equations are to include edge information when computing the neighbor messages or use a dense neural network layer in place of $\sigma$. You can think of the GCN as one type of a broader class of message passing graph neural networks, sometimes abbreviated as MPNN.

## Gated Graph Neural Network

One common variant of the message passing layer is the **gated graph neural network** (GGN) {cite}`li2015gated`. It replaces the last equation, the node update, with

\begin{equation}
\vec{v}^{'}_{i} = \textrm{GRU}(\vec{v}_i, \vec{e}_i)
\end{equation}

where the $\textrm{GRU}(\cdot, \cdot)$ is a gated recurrent unit{cite}`chung2014empirical`. The interesting property of a GRU relative to a GCN is that it has trainable parameters in the node update, giving the model a bit more flexibility, but the GRU parameters do not change as you stack more layers. A GRU is usually used for modeling sequences of undetermined length, like a sentence. What's nice about this is that you can stack infinite GGN layers without increasing the number of trainable parameters (assuming you make $\mathbf{W}$ the same at each layer). Thus GGNs are suited for large graphs, like a large protein or large unit cell.

```{margin}
You'll often see the prefix "gated" on GNNs and that means that the nodes are updated according to a GRU.
```

## Pooling

Within the message passing viewpoint, and in general for GNNS, the way that messages from neighbors are combined is a key step. This is sometimes called **pooling**, since it's similar to the pooling layer used in convolutional neural networks. Just like in pooling for convolutional neural networks, there are multiple reduction operations you can use. Typically you see a sum or mean reduction in GNNs, but you can be quite sophisticated like in the Graph Isomorphism Networks {cite}`xu2018powerful`. We'll see an example in our attention chapter of using self-attention, which can also be used for pooling. It can be tempting to focus on this step, but it's been empirically found that the choice of pooling is not so important{cite}`luzhnica2019graph,mesquita2020rethinking`. The key property of the pooling is permutation *invariance* - we want the aggregation operation to not depend on order of nodes (or edges if pooling over them). You can find a recent review of pooling methods in Grattarola et al. {cite}`grattarola2021understanding`.

You can see a more visual comparison and overview of the various pooling strategies in this distill article by Daigavane et al. {cite}`daigavane2021understanding`.

## Readout Function

GNNs output a graph by design. It is rare that our labels are graphs -- typically we have node labels or a single graph label. An example of a node label is partial charge of atoms. An example of a graph label would be the energy of the molecule. The process of converting the graph output from the GNN into our predicted node labels or graph label is called the **readout**. If we have node labels, we can simply discard the edges and use our output node feature vectors from the GNN as the prediction, perhaps with a few dense layers before our predicted output label.

If we're trying to predict a graph-level label like energy of the molecule or net charge, we need to be careful when converting from node/edge features to a graph label. If we simply put the node features into a dense layer to get to the desired shape graph label, we will lose permutation equivariance (technically it's permutation invariance now since our output is graph label, not node labels). The readout we did above in the solubility example was a reduction over the node features to get a graph feature. Then we used this graph feature in dense layers. It turns out this is the only way {cite}`zaheer2017deep` to do a graph feature readout: a reduction over nodes to get graph feature and then dense layers to get predicted graph label from those graph features. You can also do some dense layers on the node features individually, but that already happens in GNN so I do not recommend it. This readout is sometimes called DeepSets because it is the same form as the DeepSets architecture, which is a permutation invariant architecture for features that are sets{cite}`zaheer2017deep`.

You may notice that the pooling and readouts both use permutation invariant functions. Thus, DeepSets can be used for pooling and attention could be used for readouts.

### Intensive vs Extensive

One important consideration of a readout in regression is if your labels are **intensive** or **extensive**. An intensive label is one whose value is independent of the number of nodes (or atoms). For example, the index of refraction or solubility are intensive. The readout for an intensive label should (generally) be independent of the number of a nodes/atoms. So the reduction in the readout could be a mean or max, but not a sum. In contrast, an extensive label should (generally) use a sum for the reduction in the readout. An example of an extensive molecular property is enthalpy of formation.

## Battaglia General Equations

As you can see, message passing layers is a general way to view GNN layers. Battaglia *et al.* {cite}`battaglia2018relational` went further and created a general set of equations which captures nearly all GNNs. They broke the GNN layer equations down into 3 update equations, like the node update equation we saw in the message passing layer equations, and 3 aggregation equations (6 total equations). There is a new concept in these equations: graph feature vectors. Instead of having two parts to your network (GNN then readout), a graph level feature is updated at every GNN layer. The graph feature vector is a set of features which represent the whole graph or molecule. For example, when computing solubility it may have been useful to build up a per-molecule feature vector that is eventually used to compute solubility instead of having the readout. Any kind of per-molecule quantity, like energy, should be predicted with the graph-level feature vector.

The first step in these equations is updating the edge feature vectors, written as $\vec{e}_k$, which we haven't seen yet:

\begin{equation}
\vec{e}^{'}_k = \phi^e\left( \vec{e}_k, \vec{v}_{rk}, \vec{v}_{sk}, \vec{u}\right)
\end{equation}

where $\vec{e}_k$ is the feature vector of edge $k$, $\vec{v}_{rk}$ is the receiving node feature vector for edge $k$, $\vec{v}_{sk}$ is the sending node feature vector for edge $k$, $\vec{u}$ is the graph feature vector, and $\phi^e$ is one of the three update functions that the define the GNN layer. Note that these are meant to be general expressions and you define $\phi^e$ for your specific GNN layer.

Our molecular graphs are undirected, so how do we decide which node is receiving $\vec{v}_{rk}$ and which node is sending $\vec{v}_{sk}$? The individual $\vec{e}^{'}_k$ are aggregated in the next step as all the inputs into node $v_{rk}$. In our molecular graph, all bonds are both "inputs" and "outputs" from an atom (how else could it be?), so it makes sense to just view every bond as two directed edges: a C-H bond has an edge from C to H and an edge from H to C. In fact, our adjacency matrices already reflect that. There are two non-zero elements in them for each bond: one for C to H and one for H to C. Back to the original question, what is $\vec{v}_{rk}$ and $\vec{v}_{sk}$? We consider every element in the adjacency matrix (every $k$) and when we're on element $k = \{ij\}$, which is $A_{ij}$, then the receiving node is $j$ and the sending node is $i$. When we consider the companion edge $A_{ji}$, the receiving node is $i$ and the sending node is $j$.

$\vec{e}^{'}_k$ is like the message from the GCN. Except it's more general: it can depend on the receiving node and the graph feature vector $\vec{u}$. The metaphor of a "message" doesn't quite apply, since a message cannot be affected by the receiver. Anyway, the new edge updates are then aggregated with the first aggregation function:

\begin{equation}
\bar{e}^{'}_i = \rho^{e\rightarrow v}\left( E_i^{'}\right)
\end{equation}

where $\rho^{e\rightarrow v}$ is our defined function and $E_i^{'}$ represents stacking all $\vec{e}^{'}_k$ from edges  **into** node i. Having our aggregated edges, we can compute the node update:

\begin{equation}
\vec{v}^{'}_i = \phi^v\left( \bar{e}^{'}_i, \vec{v}_i, \vec{u}\right)
\end{equation}

This concludes the usual steps of a GNN layer because we have new nodes and new edges. If you are updating the graph features ($\vec{u}$), the following additional steps may be defined:

\begin{equation}
\bar{e}^{'} = \rho^{e\rightarrow u}\left( E^{'}\right)
\end{equation}

This equation aggregates all messages/aggregated edges across the whole graph. Then we can aggregate the new nodes across the whole graph:

\begin{equation}
\bar{v}^{'} = \rho^{v\rightarrow u}\left( V^{'}\right)
\end{equation}

Finally, we can compute the update to the graph feature vector as:
\begin{equation}
\vec{u}^{'} = \phi^u\left( \bar{e}^{'},\bar{v}^{'}, \vec{u}\right)
\end{equation}

### Reformulating GCN into Battaglia equations

Let's see how the GCN is presented in this form. We first compute our neighbor messages for all possible neighbors. Remember in the GCN, messages only depend on the senders.

\begin{equation}
\vec{e}^{'}_k = \phi^e\left( \vec{e}_k, \vec{v}_{rk}, \vec{v}_{sk}, \vec{u}\right) = \vec{v}_{sk} \mathbf{W}
\end{equation}

To aggregate our messages coming into node $i$, we average them.

\begin{equation}
\bar{e}^{'}_i = \rho^{e\rightarrow v}\left( E_i^{'}\right) = \frac{1}{|E_i^{'}|}\sum E_i^{'}
\end{equation}

Our node update is then the activation:

\begin{equation}
\vec{v}^{'}_i = \phi^v\left( \bar{e}^{'}_i, \vec{v}_i, \vec{u}\right) = \sigma(\bar{e}^{'}_i)
\end{equation}

we could include the self-loop above using $\sigma(\bar{e}^{'}_i + \vec{v}_i)$. The other functions are not used in a GCN, so those three completely define the GCN.

## Nodes vs Edges

You'll find that most GNNs use the node-update equation in the Battaglia equations but do not update edges. For example, the GCN will update nodes at each layer but the edges are constant. Some recent work has shown that updating edges can be important for learning when the edges have geometric information, like if the input graph is a molecule and the edges are distance between the atoms {cite}`klicpera2019directional`. As we'll see in the chapter on equivariances ({doc}`../dl/data`), one of the key properties of neural networks with geometric data (i.e., Cartesian xyz coordinates) is to have rotation equivariance. {cite}`klicpera2019directional` showed that you can achieve this if you do edge updates and encode the edge vectors using a rotation equivariant basis set with spherical harmonics and Bessel functions. These kind of edge updating GNNs can be used to predict protein structure {cite}`jing2020learning`.

## Common Architecture Motifs and Comparisons

We've now seen message passing layer GNNs, GCNs, GGNs, and the generalized Battaglia equations. You'll find common motifs in the architectures, like gating, attention, and pooling strategies. For example, Gated GNNS (GGNs) can be combined with attention pooling to create Gated Attention GNNs (GAANs){cite}`zhang2018gaan`. GraphSAGE is a similar to a GCN but it samples when pooling, making the neighbor-updates of fixed dimension{cite}`hamilton2017inductive`. So you'll see the suffix "sage" when you sample over neighbors while pooling. These can all be represented in the Battaglia equations, but you should be aware of these names.

The enormous variety of architectures has led to work on identifying the "best" or most general GNN architecture {cite}`dwivedi2020benchmarking,errica2019fair,shchur2018pitfalls`. Unfortunately, the question of which GNN architecture is best is as difficult as "what benchmark problems are best?" Thus there are no agreed-upon conclusions on the best architecture. However, those papers are great resources on training, hyperparameters, and reasonable starting guesses and I highly recommend reading them before designing your own GNN. There has been some theoretical work to show that simple architectures, like GCNs, cannot distinguish between certain simple graphs {cite}`xu2018powerful`. How much this practically matters depends on your data. Ultimately, there is so much variety in hyperparameters, data equivariances, and training decisions that you should think carefully about how much the GNN architecture matters before exploring it with too much depth.

## Do we need graphs?

It is possible to convert a graph into a string if you're working with a rather "basic" adjacency matrix. Molecules specifically can be converted into a string. This means you can use layers for sequences/strings (e.g., recurrent neural networks or 1D convolutions) and avoid the complexities of a graph neural network. SMILES is one way to convert molecular graphs into strings. With SMILES, you cannot predict a per-atom quantity and thus a graph neural network is required for atom/bond labels. However, the choice is less clear for per-molecule properties like toxicity or solubility. There is no consensus about if a graph or string/SMILES representation is better. SMILES can exceed certain graph neural networks in accuracy on some tasks. SMILES and SELFIES are better on generative tasks. Graphs obviously beat SMILES in label representations, because they have granularity of bonds/edges, but SMILES has some chemistry "intuition" baked-in. SMILES and SELFIES were built as a "compression" of molecular graphs, so you can intuit learning might be easier. But others argue that SMILES and SELFIES creates artificial long-range interactions because rings and branches are spread across sequence. We'll look at this more in {doc}`nlp`, but this is still an open question.

```{margin} novel molecules
 Some of the early work on using SMILES focused on teaching generative models (e.g., VAEs) to learn to make valid SMILES. Then, hilariously, someone realized you could just create a new way of converting molecules into strings that was surjective leading to SELFIES {cite}`krenn2020self`. In SELFIES then you can trivially generate molecules.
```

## Relevant Videos

### Intro to GNNs

### Overview of GNN with Molecule, Compiler Examples

## Chapter Summary

* Molecules can be represented by graphs by using one-hot encoded feature vectors that show the elemental identity of each node (atom) and an adjacency matrix that show immediate neighbors (bonded atoms).
* Graph neural networks are a category of deep neural networks that have graphs as inputs.
* One of the early GNNs is the Kipf & Welling GCN. The input to the GCN is the node feature vector and the adjacency matrix, and returns the updated node feature vector. The GCN is permutation invariant because it averages over the neighbors.
* A GCN can be viewed as a message-passing layer, in which we have senders and receivers. Messages are computed from neighboring nodes, which when aggregated update that node.
* A gated graph neural network is a variant of the message passing layer, for which the nodes are updated according to a gated recurrent unit function.
* The aggregation of messages is sometimes called pooling, for which there are multiple reduction operations.
* GNNs output a graph. To get a per-atom or per-molecule property, use a readout function. The readout depends on if your property is intensive vs extensive
* The Battaglia equations encompasses almost all GNNs into a set of 6 update and aggregation equations.

## Cited References

```{bibliography}
:style: unsrtalpha
:filter: docname in docnames
```


# メモ

離散数学への入門
わかりやすい離散数学

小倉久和

を読む。

目次

1. 論理と証明
2. 離散集合
3. 写像・関数
4. 帰納法
5. 離散関係
6. 整数演算
7. 離散代数系
8. 順序集合と束
9. 離散グラフ
10. 木グラフ

メモ $\quad$ 目次の章立てが古い本のせいか普通と違うと思う。 集合から始まっていない。 行列、線形代数が特に章立てされていない。 集合は離散集合 2章で扱っているし、行列は離散関係 5章で扱っているので、扱っていないわけではない。

楽しみかも。

# 論理と証明

命題 $\quad$ 文で表された真偽の明確な言明 statement を命題 proposition と言う。

$p,q$ を命題記号と言う。

素命題、基本命題 atom とも言う。

cf. 複合命題

真・偽を真理値 truth value と言い、$\mathrm{T,F}$ で表す。

$\mathrm{val} (p) = \mathrm T$

簡単のため $p = \mathrm T$ と書く。

変数に要素を代入すると真偽が決まる変数付きの言明を述語 predicate と言う。

述語は述語記号 $p$ に 引数 argument をつけて表す。 $p(x)$ 。

変数の変域をドメイン domain と言う。

述語は命題関数 propositional function と呼ばれる。

変数を含む言明であっても、$\forall$ や $\exists$ がつくと命題になる。 全称記号や存在記号を限量子 quantifier と言う。

$p = \forall x \; p(x) \quad$ 任意の $x$ に対して $p(x)$ である。

$q = \exists y \; q(4,y) \quad$ ある整数 $y$ が存在して $q(4,y)$ である。

ドメインは次のように明示することもある。

$p = \forall x \in D \ p(x)$

$q = \exists y \in \mathbf Z \ q(4,y)$

反例 counter example があれば命題は偽となる。

全称記号、存在記号の順序によって意味が異なる。

$p = \forall x \; \exists y \; (x-y)^2=4 \quad$ True

$p = \exists y \; \forall x \; (x-y)^2=4 \quad$ False

一般に存在記号で修飾された変数は、その左側にある全称記号で修飾された変数に依存してよい。 全称記号で修飾された変数がなければ、依存する変数はなく定数となる。

例 1.1 p.4 次の表現の意味を説明し、真偽を示す。 ドメインは整数。

1. $\exists y \; \forall x \; \ [x+y=x]$

ある $y$ が存在しすべての $x$ について、$x+y=x$ が成立する。 $y$ が 0 ならば成立するので真。

2. $\forall x \; \exists y \; [x+y=0]$

任意の $x$ について $x+y=0$ となる $y$ が存在する。 $-x$ が存在するので真。

否定は $\neg p$ または $\mathrm{not}\  p$ で表す。

否定の真理値表 truth table

p | not p
:-: | :-:
T | F
F | T

二重否定、対合律 (たいごうりつ) involution law が成立する。

$\neg (\neg p) = p$

一般に、限量記号 (全称記号、存在記号) を伴った命題の否定についてまとめると次のようになる。

$\neg (\forall x \; p(x)) = \exists x \; \neg p(x))\quad$ $\neg p(x)$ となる $x$ が存在する

$\neg (\exists x \; p(x)) = \forall x \; \neg p(x) \quad$ 任意の $x$ について $\neg p(x)$ である。

選言 $\quad p \lor q \quad (p \;\mathrm{or}\: q)$

連言 $\quad p \land q \quad (p \;\mathrm{and}\; q)$

ド・モルガン律 p.6

$\neg (p \lor q) = \neg p \land \neg q$

$\neg (p \land q) = \neg p \lor \neg q$

排他的選言 (排他的論理和 exlusive or) $\quad (p\;\mathrm{xor}\; q)$

同値 $\quad (p\;\mathrm{equiv}\; q)$

$p \rightarrow q$ 含意 (がんい) $\quad (p\;\mathrm{imp}\; q)$

排他的選言 (exlusive or) は排他的論理和とも言う。

$\quad p\;\mathrm{xor}\; q = (p \land \neg q) \lor (\neg p \land q)$

これは見方を変えると $p$ と $q$ の真理値が一致しないときに真、一致するときに偽、とみなすことができる。 排他的選言の否定は $p$ と $q$ の真理値が一致するときに真、一致しないときに偽となる。

この論理演算を同値 equivalence と言い、 $p \;\mathrm{equiv}\; q$ と表す。

含意 p.7,8

一般に、条件付き命題は、条件の命題 $p$ と結論の命題 $q$ からなる複合命題で、含意 implication と言う。 演算記号として $\rightarrow$ を使う。

含意 $\quad p \;\rightarrow\; q\quad (p\;\mathrm{imp}\; q)$

含意は $p$ が真、 $q$ が偽のときのみ偽となり、 $p$ が偽のときは常に真である。

含意について次の性質は重要である。

$p \;\rightarrow\; q \;=\; \neg q \;\rightarrow\; \neg q$

$p \;\rightarrow\; q \;=\; \neg p \lor q$

含意の否定は次のようになる。

$\neg (p \;\rightarrow\; q) \;=\; (p \land \neg q)$

証明について p.9,10

$a,b,c$ を実数として、$a+b+c=0$ ならば、$a^3+b^3+c^3=3abc$ であることを証明する。

```python
from sympy.abc import *
from sympy import *
# a+b+c=0 より
a = - (b + c)
# 左辺 - 右辺
display(a**3 + b**3 + c**3 - 3*a*b*c)
display(simplify(a**3 + b**3 + c**3 - 3*a*b*c))
```

証明の形式

この証明は

$p = (a+b+c=0),\quad q= (a^3+b^3+c^3=3abc)$

として、$p$ ならば $q$、$ p \rightarrow q$ の形式の条件付き命題である。

条件 $p = \mathrm T$ から $q = \mathrm T$ を導いている。 条件 $p$ が成立しない場合、$p=\mathrm F$ の場合は、$q$ の真偽によらず、$ p \rightarrow q$ であるから、条件付き命題である含意命題 $ p \rightarrow q$ が常に真 (恒真性) であることを証明している。

この例では $p$ も $q$ も任意の実数 $a,b,c$ を含むから、実際には、

$p(a,b,c) = (a+b+c=0),\quad q(a,b,c) = (a^3+b^3+c^3=3abc)$

であって、次の含意命題 (条件付き命題)

$\forall a\;\forall b\;\forall c\;p(a,b,c) \rightarrow q(a,b,c)$

の恒真性を証明する、という問題である。

三段論法

実際の証明では、いくつかの中間結果を導き、それを利用して多段に証明を行う。 例えば $p=\mathrm T $ から $q = \mathrm T$ が証明でき、さらに  $q = \mathrm T$ から $r = \mathrm T$ が証明できれば、 $p=\mathrm T $ から $r = \mathrm T$ が証明できることになる。

これは三段論法の 1 つである。

数学に重要な形式は前提に含意命題を含む仮言的三段論法である。

$\quad p \rightarrow q = \mathrm T$ と $p = \mathrm T$ から $q = \mathrm T$ を示す $\qquad$ (1.28)

$\quad p \rightarrow q = \mathrm T$ と $q \rightarrow r = \mathrm T$ から $p \rightarrow q = \mathrm T$ を示す $\qquad$ (1.29)

式 (1.28) はモダス・ポネンス、ラテン語で method of affirming (肯定式) の意味、式 (1.29) は全仮言的三段論法でシロギズム syllogism と呼ばれる。 その形から連鎖律 chain rule とも呼ばれる。

対偶問題による証明

例 1.5 $\qquad a,b,c$ を実数として、$a3+b^3+c^3 \neq 3abc$ ならば $a+b+c \neq 0$ であることを証明せよ。

解 1 $$\qquad a^{3} + b^{3} + c^{3} - 3 a b c = \left(a + b + c\right) \left(a^{2} + b^{2} + c^{2} - a b - a c - b c\right)$$

条件により、$\qquad a^{3} + b^{3} + c^{3} - 3 a b c \neq 0$ なので  $a+b+c \neq 0$

```python
from sympy.abc import *
from sympy import *
expr = a**3 +b**3 + c**3 - 3*a*b*c
display(factor(expr))
display(latex(expr))
display(latex(factor(expr)))
```

$\displaystyle \left(a + b + c\right) \left(a^{2} - a b - a c + b^{2} - b c + c^{2}\right)$

    'a^{3} - 3 a b c + b^{3} + c^{3}'

    '\\left(a + b + c\\right) \\left(a^{2} - a b - a c + b^{2} - b c + c^{2}\\right)'

対偶を証明すれば、もとの命題を証明したことになるので、

解 2 $\qquad$ この問題の対偶、$a+b+c=0$ ならば、$a^3+b^3+c^3=3abc$ であることを証明する。

$a+b+c=0$ より $c = - (b+c)$ を $a^3+b^3+c^3$ に代入すると $3abc$ になる。

解 1 より 解 2 の方が証明法としては容易である。 解 1 は直接法、解 2 は間接法の一種である。

逆・裏・対偶

命題 proposition $\qquad$ $p \rightarrow q$

逆 converse $\qquad$ $q \rightarrow p$

裏 reverse  $\qquad$ $\neg p \rightarrow \neg q$

対偶 contraposition  $\qquad$ $\neg q \rightarrow \neg p$

それぞれ、順命題、逆命題、裏命題、対偶命題と呼ばれる。逆命題の対偶は裏命題であり、裏命題の逆命題は対偶命題である。

メモ $\qquad$ そう言えば、逆問題を解く、という言葉があったね!!!!

世の中の課題は結果を見て原因を探るので、逆問題を解く、なのかと思っていたが数学的には解が $p$ で 方程式が $q$ で、$q$ が与えられて $p$ を求めるという意味なのだよ。

必要十分条件の記号 $\iff$ は ` $\iff$` で書く。 if and only then の意味。

メモ $\quad$ 思うに、このテキスト 小倉久和 の言葉による説明はわたし向きではないかもしれない。 背理法のところと演習問題を飛ばす。 風が吹けば桶屋がもうかる、を論理式で示すのを例にするのはよくないと思う。 まあいいか。

# 離散集合
p. 18

* ドメイン domain
* 集合 set
* 要素 element 元

集合は $\{1,2,3,45\}$ と言うふうに表す。同じものは含まない。

同じ要素を含む集合は多重集合 multiple set と言う。

集合は $A,B$ で表し、要素は $a,b$ で表す。

$a$ が $A$ に属することを $a \in A$ と書き、要素でないことを $a \not \in A$ と書く。

要素が等しいとき、集合が等しいと言い、$A=B$ と書く。

* 有限集合 finite set
* 無限集合 infinite set
* 空集合 $\phi, \varnothing, \emptyset$

このテキストは $\phi$ ファイを使っている。

* 外延記法 denotation, 枚挙法 $\{1,2,3,45\}$

この記法 denotation では空集合は $\phi = \{\}$ と表せる。

* 内包記法 connotation 集合を次のように表現する方法を内包記法と言う。

メモ $\quad$ 普通、set comprehension と言うのではないか?

$$ A = \{n \;|\; n は 10 以下の自然数\}$$

メモ $\quad$ あと、数式に文字がはいると latex がめんどうになるので、なるべく次のようにしたい。

$$ A = \{n \;|\; n \leq 10,\; n \in \mathbb N\}$$

一般形は

$$ A = \{x \;|\; P(x) \}$$

である。

# 離散集合

離散集合 discrete set

集合 $A$ の要素数は

$\qquad n(A)$ あるいは $|A|$

と表現する。

空集合 $\phi$ の大きさは

$\qquad |\phi| = 0$

である。

自然数の集合 $\boldsymbol N$ は無限集合であるが、離散集合である。

図 2.1 数直線

---
問題 $\qquad$ 無理数の稠密性 ちゅうみつせい を証明せよ。

---
番号付けできる集合を可付番集合 enumerable set あるいは可算集合 countable set と言う。

自然数の集合 $\boldsymbol N$ は可算集合である。

有理数は可算集合であるが、稠密 dense である。 なので離散集合ではない。

離散集合は有限集合であるか、互いに離れた要素からなる可算集合である。

# 集合演算

部分集合 subset

集合 $A$ が $B$ の部分集合であるということは、集合 $A$ の任意の元が $B$ の要素である。

$A \subset B \iff [x \in A \implies x \in B]$

集合 $A$ と $B$ が互いに部分集合となっているとき $A$ と $B$ は等しい。

$A = B \iff [A \subset B \land B \subset A]$

$A \subset B \land A \neq B$ のとき $A$ は $B$ の部分集合である。

定義より
* $P \subset P$
* $\phi \subset P$

集合の包含関係の性質

* 反射性 $\quad$ $A \subset A$
* 反対称性 $\quad A \subset B \land B \subset A \implies A = B$
* 推移性 $\quad A \subset B \land B \subset C \implies A \subset C$

---
例 2.1 p.23

集合 $A = \{0, 1\}$ の部分集合すべてを挙げ、要素が部分集合に含まれているかどうかを示す所属表を作成する。

部分集合 $\quad \phi, \{0\}, \{1\}, \{0,1\}$

要素 $\quad 0, 1$

部分集合 | 0 | 1
--- | --- | ---
$\phi$ | 0 | 0
$\{0\}$ | 1 | 0
$\{1\}$ | 0 | 1
$\{0,1\}$ | 1 | 1

---
集合 $B = \{a,b,c\}$ の部分集合すべてを挙げ、要素が部分集合に含まれているかどうかを示す所属表を作成する。

部分集合 $\quad \phi, \{a\}, \{b\}, \{c\}, \{a,b\}, \{a,c\}, \{b,c\}, \{a,b,c\}$

要素 $\quad a,b,c$

部分集合 | a | b | c
---- | --- | --- | ---
$\phi$ | 0 | 0 | 0
$\{a\}$ | 1 | 0 | 0
$\{b\}$ | 0 | 1 | 0
$\{c\}$ | 0 | 0 | 1
$\{a,b\}$ | 1 | 1 | 0
$\{a,c\}$ | 1 | 0 | 1
$\{b,c\}$ | 0 | 1 | 1
$\{ a,b,c$$\}$| 1 | 1 | 1

べき集合

集合の集まりを集合族 set family と言う。

集合は集合を要素とする集合である。

集合 $A$ のすべての部分集合からなる集合属は集合であり、べき集合 power set と言う。

$\mathcal P (A)$, $2^A$ などと書く。

要素の数は

$|\mathcal P (A)| = 2^{|A|} = 2^n$

証明 元のそれぞれについて選ぶ選ばないの 2 通りなので、$2^n$ になる。

無限集合のべき集合も定義できて、無限集合である。

集合と述語


```python
from collections import ChainMap
from configparser import ConfigParser
import logging
from math import pi as PI
import numpy
import pickle
from pathlib import Path
from pprint import pformat
import sys
```

```python
from anglr import Angle

import matplotlib.path
import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d
from sympy import *
init_printing()

```

```python
# from environment import *
# from ncprogram import Block, NCProgram

exec(Path("py/charconsts.py").read_text())
exec(Path("py/enumerations.py").read_text())
exec(Path("py/environment.py").read_text())
exec(Path("py/ncprogram.py").read_text())
```

```python
class Viewer():
    """ .

    """
    def __init__(self, settings=None):
        # self.settings = settings
#         Program.__init__(self, settings)
#         Driver.__init__(self, settings)
        print("initializing viewer")
        self.settings = settings
        if not self.settings:
            self.settings = { "verbose": True, "debug": False }
        if self.settings["verbose"]:
            print()
            print("Settings:")
            print()
            print(pformat([f"{k}: {v}" for k, v in self.settings.items()]).lstrip('[').rstrip(']'))
            print()
        self.env = Environment(DEFAULT_VARS)
        # programs should be loaded from a directory located by a configuration file setting
        self.programs = list()
        self.fig = plt.figure()
        self.ax = self.fig.add_subplot(111, projection='3d', xlim=(0, 6), ylim=(0, 6))
        # self.ax.plot([0.0, 1.0], [0.2, 0.7], 0.4)
        self.current_position = Point()
#         plt.show()
        self.current_program = NCProgram()

    def code_group(self, n):
        for k in gCodeGroups.keys():
            if n in gCodeGroups[k]:
                return gCodeGroupsDict[k]
        return None

    def make_gcodes_dict(self, gcodes):
        """
        """
        result = dict()
        result[NonModal] = list()
        for n in gcodes:
            klass = self.code_group(n)
            if klass == NonModal:
                result[NonModal].append(n)
            else:
                result.update({klass : klass(n)})
        return result

    def execute(self, blk):
        result = dict()
        # validate the block and exit on error
        # extract the `G` codes
        result = self.env["G"]
        print()
        print("Current G Code Settings:")
        print()
        pp(result)
        result[NonModal] = list()
        result.update(self.make_gcodes_dict(blk.pop_G_codes()))
        result = { "G" : result }
        print()
        print("New G Code Settings:")
        print()
        pp(result)
        print()
        # process the `G` codes
        # extract the `M` code
#         result.update({"G" : self.make_gcodes_dict(gcodes)})
        result.update(blk.execute())
        print("New Result Environment:")
        print()
        pp(result)
        print()
        self.env = self.env.new_child(result)
        if "O" in self.env.maps[-1]:
            self.load_program(self.env.maps[-1]["O"])
        self.move()
        if blk.mCode():
            pass
            # process the `M` code

    def load_program(self, n):
        """ Find the given number after 'O' and followed by an up-to-5-digit
            number, the last of which match `n` as the first line of a file in
            the `gcode` directory designated in `config.ini`.
        """

#     def run(self):
#         print("Running `ncview` ...")
#         if self.settings["verbose"]:
#             print(sys.version)
#         # self.get_input()
#         self.process_args()

#         self.cmdloop()

    def process_file(self, path):
        """There should be at most one filename given on the command line.
           Since we're processing a file, it has to have been on the command line,
           and so this function has to have been called from `Program.init`.
           `path` should be a Python 3.5+ `Path`, having been converted by
           `Program.process_fname`.
        """
        # DEPRECATED
        # self.programs.append(NCProgram(path.read_text().split('\n')))
        VERBOSE = self.settings['verbose']
        if VERBOSE:
            print(f"processing file: {path=}")
        with path.open() as f:
            self.current_program.extend([Block(line) for line in f])

#     def process_args(self):
#         super().process_args()
#         # for p in self.programs:
#         #     p.run()

    def is_valid_gcode(self, s):
        try:
            b = Block(s)
        except ValueError:
            return False
        return True

#     def default(self, args):
#         if self.is_valid_gcode(args): self.execute(Block(args))
#         else:
#             super.default(args)
#         # self.ax.plot([1.0, 3.0], [0.2, 0.7], 0.4)

#     def emptyline(self):
#         pass

    def do_step(self, args):
        if self.current_program:
            self.env = self.env.new_child(self.current_program.step())
            if self.current_program.state != NCProgram.State.END:
                if self.move():
                    return False
        else:
            print("ERROR: step: No program is loaded.")
            return False

    def move(self):
        P = self.current_position
        if self.settings["verbose"]:
            print("Current position:", P)
        cur_map = self.env.maps[0]
        INCREMENTAL = self.env["G"][PositionMode] == PositionMode.INCREMENTAL
        Q = Point(*[cur_map["X"] if "X" in cur_map.keys() else (0.0 if INCREMENTAL else self.env["X"]),
                    cur_map["Y"] if "Y" in cur_map.keys() else (0.0 if INCREMENTAL else self.env["Y"]),
                    cur_map["Z"] if "Z" in cur_map.keys() else (0.0 if INCREMENTAL else self.env["Z"])
                   ])
        if self.settings["verbose"]:
            print("X, Y, Z:", Q)
        if self.env["G"][PositionMode] == PositionMode.INCREMENTAL:
            Q = P + Q
        if P != Q:
            if self.settings["verbose"]: print("Moving the tool...")
            if self.settings["verbose"]: print("Motion mode:", self.env["G"][MotionMode])
            if self.env["G"][MotionMode] in { MotionMode.RAPID, MotionMode.LINEAR }:
                if self.settings["verbose"]: print("Motion mode:", self.env["G"][MotionMode])
                self.ax.plot([P.x, Q.x], [P.y, Q.y], [P.z, Q.z])
            elif self.env["G"][MotionMode] in { MotionMode.CLOCKWISE, MotionMode.COUNTER }:
                # Draw an arc of some sort.
                if self.settings["verbose"]:
                    print("Drawing an arc...")
                V = Point ( cur_map["I"] if "I" in cur_map.keys() else 0.0,
                            cur_map["J"] if "J" in cur_map.keys() else 0.0,
                            cur_map["K"] if "K" in cur_map.keys() else 0.0
                          )
                if self.settings["verbose"]:
                    print(V)
                K = {"I", "J", "K"}
                if K.intersection(cur_map.keys()):
                    C = P + V
                elif "R" in cur_map.keys():
                    C = self.find_arc_center(P, Q, cur_map["R"])
                if self.settings['verbose']:
                    print("Arc center:", C.evalf())
                R = cur_map["R"] if "R" in cur_map.keys() else abs(V)
                if self.settings["verbose"]:
                    print("R:", R)
                P1 = P.translate(*-C)
                P2 = Q.translate(*-C)
                if self.settings["verbose"]:
                    print("P1:", P1)
                    print("P2:", P2)
                YZ_PLANE = self.env["G"][CircularPlane] == CircularPlane.YZ
                XY_PLANE = self.env["G"][CircularPlane] == CircularPlane.XY
                if self.settings["verbose"]:
                    print( CircularPlane.XY if XY_PLANE
                               else ( CircularPlane.YZ if YZ_PLANE
                                          else CircularPlane.XZ
                                    ))
                x = P1.y if YZ_PLANE else P1.x
                y = P1.y if XY_PLANE else P1.z
                if self.settings["verbose"]:
                    print(f"(x, y): ({x, y})")
                t1 = Angle(atan(y / x).evalf())
                if self.settings["verbose"]:
                    print("t1:", t1)
                x = P2.y if YZ_PLANE else P2.x
                y = P2.y if XY_PLANE else P2.z
                t2 = Angle(atan(y / x).evalf())
                if self.settings["verbose"]:
                    print("t2:", t2)
                if t2 < t1:
                    t1, t2 = t2, t1
                    if self.settings["verbose"]:
                        print(f"t1: {t1}\nt2: {t2}")
                step = t1.degrees - t2.degrees / 10
                p = matplotlib.path.Path.arc(t1.degrees, t2.degrees, n=10).vertices
                if self.settings["verbose"]:
                    print("Path Vertices:")
                    print("--------------")
                    print(p)
                p = p.transpose()
                for point in p:
                    if Angle(atan(point[1]/point[1])) > t2:
                        del(point)
                if self.settings["verbose"]:
                    print()
                    print(type(p))
                # p[0] = [x + C.x for x in p[0]]
                # p[1] = [x + C.y for x in p[1]]
                if self.settings["verbose"]:
                    print()
                    print("ARC:")
                    print(p)
                    print()
                # p[0] = [x + C.x for x in p[0]]
                # p[1] = [x + C.x for x in p[1]]
                if YZ_PLANE:
                    x = numpy.full(len(p[0]), P.x)
                    y = p[0]
                    z = p[1]
                elif XY_PLANE:
                    x = p[0]
                    y = p[1]
                    z = numpy.full(len(p[0]), P.z)
                else:
                    x = p[0]
                    y = numpy.full(len(p[0]), P.y)
                    z = p[1]
                if self.settings["verbose"]:
                    print("x:", x)
                    print("y:", y)
                    print("z:", z)
                    print()
                x = R * x + C.x
                y = R * y + C.y
                z = R * z + C.z
                self.ax.plot(x, y, z)

            else:
                raise ValueError("Motion Mode is out of range!")

        else: # check for moves that don't require `X`, `Y` or `Z`

            # circular mill `G12`, `G13`
            pass

        self.current_position = Q
        return True

    def find_arc_center(self, P0, P, R0):
        if self.settings["verbose"]:
            print("Finding the arc center...")
        x, y, z, X, X0, Y, Y0, Z, Z0, R = symbols("x y z X X0 Y Y0 Z Z0 R")
        e1 = (x-X0)**2 + (y-Y0)**2
        e2 = (x-X)**2 + (y-Y)**2
        xa = (X - X0) / 2
        ya = (Y - Y0) / 2
        x0, y0 = X0 + xa, Y0 + ya
        a = sqrt(xa**2 + ya**2)
        b = sqrt(R**2 - a**2)
        x3 = x0 + b * ya / a
        x4 = x0 - b * ya / a
        y3 = y0 - b * xa / a
        y4 = y0 + b * xa / a
        if self.settings["verbose"]:
            print()
            print("x3:")
            print(x3)
            print()
            print(y3)
            print()
            print("x4:")
            print(x4)
            print("y3:")
            print()
            print("y4:")
            print(y4)
            print()

        YZ_PLANE = self.env["G"][CircularPlane] == CircularPlane.YZ
        XY_PLANE = self.env["G"][CircularPlane] == CircularPlane.XY
        params = dict(zip([X0, Y0, X, Y, R],
                          [ P0.y if YZ_PLANE else P0.x,
                            P0.y if XY_PLANE else P0.z,
                            P.y if YZ_PLANE else P.x,
                            P.y if XY_PLANE else P.z,
                            R0
                          ]))
        if self.settings["verbose"]:
            print("Parameters: ", params)
        x3 = x3.subs(params)
        y3 = y3.subs(params)
        x4 = x4.subs(params)
        y4 = y4.subs(params)

        if self.settings["verbose"]:
            print("x3:", x3)
            print("y3:", y3)
            print("x4:", x4)
            print("y4:", y4)

        args1 = [-x3, -y3]
        args2 = [ 1/params[R]] * 3
        Q0 = Point(params[X0], params[Y0], 0).translate(*args1).evalf()
        Q = Point(params[X], params[Y], 0).translate(*args1).evalf()
        theta1 = Angle(atan(Q0.y / Q0.x))
        theta2 = Angle(atan(Q.y / Q.x))
        abc = theta1.angle_between_clockwise(theta2)
        if self.settings["verbose"]:
            print("Q0:", Q0)
            print("Q:", Q)
            print("theta1:", theta1)
            print("theta2:", theta2)
            print("abc", abc)
        if abc.radians > PI and self.env["G"][MotionMode] == MotionMode.CLOCKWISE:
            rx, ry = x3, y3
        else:
            rx, ry = x4, y4
        if XY_PLANE:
            return Point(rx, ry, P0.z)
        elif XZ_PLANE:
            return Point(rx, P0.y, ry)
        else:
            return Point(P0.x, xr, ry)

    def do_run(self):
        while self.current_program.state != NCProgram.State.END and self.do_step(None):
            pass

```

```python
viewer = Viewer()
```

```python
viewer.process_file(Path('../../data/gcode/00001.gcode'))
```

    processing file: path=WindowsPath('../../data/gcode/00001.gcode')

```python
viewer.do_run()
```

    Executing block:  G90 G80 G40 G17 G0;
    Current position: Point3D(0, 0, 0)
    X, Y, Z: Point3D(0, 0, 0)

```python

```