```python
from IPython.display import HTML
HTML("""
<br><br>
<a href=http://wwwgong.pythonanywhere.com/cuspea/default/list_talks target=new>
<font size=+3 color=blue>CUSPEA Talks</font>
</a>
<br><br>
<br>
""")
```

CUSPEA Talks

```python

```

# Fun with MyPETS

## Table of Contents

* [Motivation](#hid_why)
* [Introduction](#hid_intro)
* [Problem Statement](#hid_problem)
* [Import packages](#hid_pkg)

* [History of Open Source Movement](#hid_open_src)
* [How to learn STEM (or MyPETS)](#hid_stem)

* [References](#hid_ref)
* [Contributors](#hid_author)
* [Appendix](#hid_apend)

## Motivation

* Current Choice

* A New Option

> The __Jupyter Notebook__ is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.

Useful for many tasks

* Programming
* Blogging
* Learning
* Research
* Documenting work
* Collaborating
* Communicating
* Publishing results

or even

* Doing homework as a student

```python
HTML("")
```

## Introduction

## Problem Statement

## Import packages

```python
# math function
import math

# create np array
import numpy as np

# pandas for data analysis
import pandas as pd

# plotting
import matplotlib.pyplot as plt
%matplotlib inline

# symbolic math
import sympy as sy

# html5
from IPython.display import HTML, SVG, YouTubeVideo

# widgets
from collections import OrderedDict
from IPython.display import display, clear_output
from ipywidgets import Dropdown

# csv file
import csv
```

## History of Open Source Movement

```python
with open('../dataset/open_src_move_v2_1.csv') as csvfile:
    reader = csv.DictReader(csvfile)
    table_str = '<table>'
    table_row = """
    <tr><td>{year}</td>
        <td></td>
        <td><table>
            <tr><td>{person}</td></tr>
            <tr><td><a target=new href={subject_url}>{subject}</a></td></tr>
            <tr><td>{history}</td></tr>
            </table>
        </td>
    </tr>
    """
    for row in reader:
        table_str = table_str + table_row.format(year=row['Year'], \
                subject=row['Subject'],\
                subject_url=row['SubjectURL'],\
                person=row['Person'],\
                picture=row['Picture'],\
                history=row['History'])
    table_str = table_str + '</table>'

HTML(table_str)
```

1983

Richard Stallman

GNU Project : gcc, Emacs, gdb

Launch of the free software movement and founder of Free Software Foundation

1984

X.Org

X Window Systerm

basic framework for a GUI environment

1985

Richard Stallman

GNU Manifesto: GNU's Not Unix

disagreement between Stallman and Symbolics, Inc. over MIT's access to updates Symbolics had made to its Lisp machine, which was based on MIT code

1987

Larry Wall

Perl programming lang

Common Gateway Interface (CGI )

1989

Guido van Rossum

Python programming lang

creator of popular Python scripting lang

1990

Tim Berners-Lee

HyperText Transfer Protocol (HTTP)

 creator of the World-Wide-Web at CERN

1991

Linus Torvalds

Linux

creator of freely modifiable Linux kernel source code

1993

Ian Murdock

Debian GNU/Linux

Debian Social Contract

1995

Rob McCool

Apache HTTP Server

a web server that powers the digital world

1995

David Axmark and Michael 'Monty' Widenius

MySQL

popular RDMS acquired by Oracle

1996

Jonathan Ian Schwartz

Java / JDK

general computing platform

1997

Eric S. Raymond

The Cathedral and the Bazaar

open-source software advocate and author

1998

Marc Lowell Andreessen

Mosaic

Netscape opened source codes of web browser

1999

Brian_Behlendorf

Apache Software Foundation

Open source software incubator

1999

Richard Baraniuk

OpenStax CNX

formerly called Connexions, is a global repository of educational content provided by volunteers

2000

Sun Microsystems

StarOffice

office suite to counter evil empire Microsoft

2001

Fernando Perez

IPython Notebook

active documents that contain live code, equations, visualizations and explanatory text

2001

Wikimedia Foundation

Wikipedia

the free encyclopedia

2003

Massimo Banzi

Arduino

open-source electronics platform based on easy-to-use hardware and software

2005

Linus Torvalds

Git

version control system

2005

Steve Chen

YouTube

Video Hosting Service

2006

Salman Khan

Khan Academy

learn anything -
For free. For everyone. Forever

2007

Google

Android

mobile operating system

2008

Satoshi Nakamoto

Blockchain / bitcoin

an open, distributed ledger that can record transactions between two parties efficiently and in a verifiable and permanent way.

2008

Tom Preston-Werner

GitHub

Open source project repository

2015

Google

Tensorflow

An open-source software library for Machine Intelligence

2017

Elon Musk

Open AI

promote and develop friendly AI in such a way as to benefit, rather than harm, humanity as a whole

## How to learn STEM

```python
HTML("Wen calls it -<br><br><br> <font color=red size=+4>M</font><font color=purple>y</font><font color=blue size=+3>P</font><font color=blue size=+4>E</font><font color=green size=+4>T</font><font color=magenta size=+3>S</font><br>")
```

Wen calls it -

 MyPETS

### Math

* [Awesome Math](https://github.com/rossant/awesome-math)

$$ e^{i \pi} + 1 = 0 $$

see more [MathJax](https://www.mathjax.org/)  equations [here](https://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html#Maxwell's-Equations)

### Science

#### Physics

*  [Computational Physics, 3rd Ed - Problem Solving with Python by Rubin Landau](http://physics.oregonstate.edu/~landaur/Books/CPbook/index.html)

### Engineering

* [How To Be A Programmer](https://github.com/braydie/HowToBeAProgrammer)

### Technology

* [Deep Learning for Self-Driving Cars](http://selfdrivingcars.mit.edu/)  @MIT
* [Deep Learning for Natural Language Processing](http://cs224d.stanford.edu/) @Stanford

## References

### Websites

* [DataCamp - Jupyter Notebook Tutorial](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook#gs.ClmI4Jc)

* http://docs.python.org

It goes without saying that Pythonâ€™s own online documentation is an excellent resource if you need to delve into the finer details of the language and modules. Just make sure youâ€™re looking at the documentation for Python 3 and not earlier versions.

### Books

### Other Resources

* Idea
    - [Google Search](http://www.google.com)
* Text
    - [Wikipedia](https://www.wikipedia.org/)
* Image
    - [Google Images](https://www.google.com/imghp)
* Video
    - [YouTube](https://www.youtube.com/)

## Contributors

* wen.gong@oracle.com (first created on 2017-03-09)

## Appendix

```python

```

```python

```


## Set up the environment

```
!nvidia-smi
```

    Sun Jun 13 02:41:42 2021
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   55C    P8    11W /  70W |      0MiB / 15109MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

```
#!pip install pymatgen==2020.12.31
!pip install pymatgen==2019.11.11
!pip install --pre graphdot
!pip install gdown
```

    Requirement already satisfied: pymatgen==2019.11.11 in /usr/local/lib/python3.7/dist-packages (2019.11.11)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.1.5)
    Requirement already satisfied: matplotlib>=1.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (3.2.2)
    Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.4.1)
    Requirement already satisfied: pydispatcher>=2.0.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2.0.5)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (0.8.9)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (0.17.9)
    Requirement already satisfied: spglib>=1.9.9.44 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.16.1)
    Requirement already satisfied: monty>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2021.6.10)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2.5.1)
    Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.19.5)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2.23.0)
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (3.3.0)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.7.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->pymatgen==2019.11.11) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->pymatgen==2019.11.11) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (0.10.0)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml>=0.15.6->pymatgen==2019.11.11) (0.2.2)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.2->pymatgen==2019.11.11) (4.4.2)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (2020.12.5)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (3.0.4)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pymatgen==2019.11.11) (1.2.1)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->pymatgen==2019.11.11) (1.15.0)
    Requirement already satisfied: graphdot in /usr/local/lib/python3.7/dist-packages (0.8a17)
    Requirement already satisfied: tqdm>=4.55 in /usr/local/lib/python3.7/dist-packages (from graphdot) (4.61.1)
    Requirement already satisfied: numba>=0.51.0 in /usr/local/lib/python3.7/dist-packages (from graphdot) (0.51.2)
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.19.5)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.7.1)
    Requirement already satisfied: pandas>=0.24 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.1.5)
    Requirement already satisfied: scipy>=1.3.0 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.4.1)
    Requirement already satisfied: pymatgen==2019.11.11 in /usr/local/lib/python3.7/dist-packages (from graphdot) (2019.11.11)
    Requirement already satisfied: networkx>=2.4 in /usr/local/lib/python3.7/dist-packages (from graphdot) (2.5.1)
    Requirement already satisfied: pycuda>=2019 in /usr/local/lib/python3.7/dist-packages (from graphdot) (2021.1)
    Requirement already satisfied: mendeleev>=0.7 in /usr/local/lib/python3.7/dist-packages (from graphdot) (0.7.0)
    Requirement already satisfied: treelib>=1.6.1 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.6.1)
    Requirement already satisfied: kahypar>=1.1.4 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.1.6)
    Requirement already satisfied: ase>=3.17 in /usr/local/lib/python3.7/dist-packages (from graphdot) (3.21.1)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.51.0->graphdot) (57.0.0)
    Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.51.0->graphdot) (0.34.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->graphdot) (1.2.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->graphdot) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->graphdot) (2.8.1)
    Requirement already satisfied: matplotlib>=1.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (3.2.2)
    Requirement already satisfied: spglib>=1.9.9.44 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (1.16.1)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (0.8.9)
    Requirement already satisfied: monty>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2021.6.10)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2.23.0)
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (3.3.0)
    Requirement already satisfied: pydispatcher>=2.0.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2.0.5)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (0.17.9)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.4->graphdot) (4.4.2)
    Requirement already satisfied: appdirs>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from pycuda>=2019->graphdot) (1.4.4)
    Requirement already satisfied: pytools>=2011.2 in /usr/local/lib/python3.7/dist-packages (from pycuda>=2019->graphdot) (2021.2.7)
    Requirement already satisfied: mako in /usr/local/lib/python3.7/dist-packages (from pycuda>=2019->graphdot) (1.1.4)
    Requirement already satisfied: pyfiglet<0.9,>=0.8.post1 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (0.8.post1)
    Requirement already satisfied: Pygments<3.0.0,>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (2.9.0)
    Requirement already satisfied: colorama<0.5.0,>=0.4.4 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (0.4.4)
    Requirement already satisfied: SQLAlchemy<2.0.0,>=1.3.23 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (1.4.15)
    Requirement already satisfied: six<2.0.0,>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (1.15.0)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from treelib>=1.6.1->graphdot) (0.16.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11->graphdot) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11->graphdot) (2.4.7)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11->graphdot) (1.3.1)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (2020.12.5)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (2.10)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml>=0.15.6->pymatgen==2019.11.11->graphdot) (0.2.2)
    Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from mako->pycuda>=2019->graphdot) (2.0.1)
    Requirement already satisfied: greenlet!=0.4.17; python_version >= "3" in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (1.1.0)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (4.0.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (3.4.1)
    Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (3.7.4.3)
    Requirement already satisfied: gdown in /usr/local/lib/python3.7/dist-packages (3.6.4)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from gdown) (4.61.1)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from gdown) (2.23.0)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from gdown) (1.15.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (2020.12.5)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (3.0.4)

```
%matplotlib inline
import io
import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')
import os
import urllib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import graphdot
from graphdot import Graph
from graphdot.graph.adjacency import AtomicAdjacency
from graphdot.graph.reorder import rcm
from graphdot.kernel.marginalized import MarginalizedGraphKernel # https://graphdot.readthedocs.io/en/latest/apidoc/graphdot.kernel.marginalized.html
from graphdot.kernel.marginalized.starting_probability import Uniform
from graphdot.model.gaussian_process import (
    GaussianProcessRegressor,
    LowRankApproximateGPR
)
from graphdot.kernel.fix import Normalization
import graphdot.microkernel as uX
import ase.io

# for getting all file names into a list under a directory
from os import listdir
# for getting file names that match certain pattern
import glob
import time
```

    /usr/local/lib/python3.7/dist-packages/graphdot/graph/__init__.py:24: UserWarning: Cannot import RDKit, `graph.from_rdkit()` will be unavailable.

      'Cannot import RDKit, `graph.from_rdkit()` will be unavailable.\n'

```
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
```

    Mounted at /content/gdrive

```
#cd gdrive/MyDrive/Google\ Colab/Covid-Data
%cd gdrive/MyDrive/Covid-Data/
```

    /content/gdrive/.shortcut-targets-by-id/1wtzMcocuK8kPsz8K0ktjCZPkv567W6M2/Covid-Data

```
!pwd
```

    /content/gdrive/.shortcut-targets-by-id/1wtzMcocuK8kPsz8K0ktjCZPkv567W6M2/Covid-Data

```
!mkdir /content/pkls
```

## load the data

```
files  = ['uncharged_NSP15_6W01_A_3_H.Orderable_zinc_db_enaHLL.2col.csv.1.xz']
dataset = pd.read_pickle(files[0]) # length of each csv file is 100000
```

```
target = 'energy'
batch_size = 1000
batch_num_train = 0.8*len(dataset)//batch_size # batch number of training
num_test = 0.2*len(dataset)
train_data = dataset.iloc[:int(batch_num_train)*batch_size]
test_data = dataset.iloc[int(batch_num_train)*batch_size:len(dataset)]
#print(len(train_data))
#print(len(test_data))
```

```
gpr = GaussianProcessRegressor(
    # kernel is the covariance function of the gaussian process (GP)
    kernel=Normalization( # kernel equals to normalization -> normalizes a kernel using the cosine of angle formula, k_normalized(x,y) = k(x,y)/sqrt(k(x,x)*k(y,y))
        # graphdot.kernel.fix.Normalization(kernel), set kernel as marginalized graph kernel, which is used to calculate the similarity between 2 graphs
        # implement the random walk-based graph similarity kernel as Kashima, H., Tsuda, K., & Inokuchi, A. (2003). Marginalized kernels between labeled graphs. ICML
        MarginalizedGraphKernel(
            # node_kernel - A kernelet that computes the similarity between individual nodes
            # uX - graphdot.microkernel - microkernels are positive-semidefinite functions between individual nodes and edges of graphs
            node_kernel=uX.Additive( # addition of kernal matrices: sum of k_a(X_a, Y_a) cross for a in features
                # uX.Constant - a kernel that returns a constant value, always mutlipled with other microkernels as an adjustable weight
                #   c, the first input arg. as 0.5, (0.01, 10) the lower and upper bounds of c that is allowed to vary during hyperpara. optimizartion
                # uX.KroneckerDelta - a kronecker delta returns 1 when two features are equal and return h (the first input arg here, which is 0.5 in this case) otherwise
                #   (0.1, 0.9) the lower and upper bounds that h is allowed to vary during hyperpara. optimization
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 2nd element of graphdot.graph.Graph.nodes
                atomic_number=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 3rd element of graphdot.graph.Graph.nodes
                # uX.SquareExponential - Equ. 26 in the paper
                # input arg. length_sacle is a float32, set as 1 in this case, which correspond to approx. 1 of the kernal value.
                # This is used to determins how quicklys should the kernel decay to zero.
                charge=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0), # the 4th element of graphdot.graph.Graph.nodes
                chiral=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 5th element of graphdot.graph.Graph.nodes
                hcount=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0), # the 6th element of graphdot.graph.Graph.nodes
                hybridization=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 7th element of graphdot.graph.Graph.nodes
                # uX.Convolution - a convolutional microkernel which averages evaluations of a base microkernel between pairs pf elememts of two variable-length feature sequences
                # uX.KroneckerDelta as the base kernel
                ring_list=uX.Constant(0.5, (0.01, 100.0)) * uX.Convolution(uX.KroneckerDelta(0.5,(0.1, 0.9))) # the 8th element of graphdot.graph.Graph.nodes
            ).normalized,
            # edge_kernel - A kernelet that computes the similarity between individual edge
            edge_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 3rd element of graphdot.graph.Graph.nodes
                conjugated=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 4th element of graphdot.graph.Graph.nodes
                order=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 5th element of graphdot.graph.Graph.nodes
                ring_stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 6th element of graphdot.graph.Graph.nodes
                stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)) # the 7th element of graphdot.graph.Graph.nodes
            ).normalized,
            p=Uniform(1.0, p_bounds='fixed'), # the strating probability of the random walk on each node
            q=0.05 # the probability for the random walk to stop during each step
        )
    ),
    alpha=1e-4, # value added to the diagonal of the kernel matrix during fitting
    optimizer=True, # default optimizer of L-BFGS-B based on scipy.optimize.minimize
    normalize_y=True, # normalize the y values so taht the means and variance is 0 and 1, repsectively. Will be reversed when predicions are returned
    regularization='+', # alpha (1e-4 in this case) is added to the diagonals of the kernal matrix
)
```

```
def find_repeat_batch():
  lower_batch = 0
  lower_repeat = 0
  for f in listdir('/content/pkls'):
    f = f.split('_')
    f[3] = f[3].split('.')

    if int(f[1]) > lower_batch:
        lower_batch = f[1]
    if int(f[3][0]) > lower_repeat:
        lower_repeat = f[3][0]

  return lower_batch, lower_repeat

def train_pipeline_batch(model, train_dataset, test_dataset, target, repeats, batch_num_train, batch_size, verbose = True, print_batch = True, print_repeat = True):

  start_time = time.time()
  lower_batch, lower_repeat = find_repeat_batch()
  assert(lower_batch <= int(batch_num_train)-1)
  assert(lower_repeat <= repeats)

  for repeat in range(lower_repeat, repeats):
    for batch in range(lower_batch int(batch_num_train)-1):
      batch_dataset = dataset.iloc[batch*batch_size:(batch+1)*batch_size] # divide the training data into different batches
      np.random.seed(0)
      if batch == 0 and repeat == 0:
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename='batch_0_repeat_0.pkl', overwrite=True)
      elif batch == 0 and repeat !=0:
        filename_load = 'batch_0_repeat_'+str(repeat-1)+'.pkl'
        filename_save = 'batch_0_repeat_'+str(repeat)+'.pkl'
        model.load(path="/content/pkls", filename=filename_load)
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename=filename_save, overwrite=True)
      else:
        filename_load = 'batch_'+str(batch-1)+'_repeat_'+str(repeat)+'.pkl'
        filename_save = 'batch_'+str(batch)+'_repeat_'+str(repeat)+'.pkl'
        model.load(path="/content/pkls", filename=filename_load)
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename=filename_save, overwrite=True)

      if print_batch:
        mu = gpr.predict(batch_dataset.graphs)
        print('   Training set of repeat '+str(repeat)+' and batch '+str(batch))
        print('   MAE '+str(repeat)+' and batch '+str(batch)+' '+str(np.mean(np.abs(batch_dataset[target] - mu))))
        print('   RMSE '+str(repeat)+' and batch '+str(batch)+' '+str(np.std(batch_dataset[target] - mu)))

    if print_repeat:
      mu = gpr.predict(train_dataset.graphs)
      print('Training set of repeat '+str(repeat))
      print('MAE '+str(repeat)+' '+str(np.mean(np.abs(train_dataset[target] - mu))))
      print('RMSE '+str(repeat)+' '+str(np.std(train_dataset[target] - mu)))

      mu = gpr.predict(test_dataset.graphs)
      print('Test set of repeat '+str(repeat))
      print('MAE '+str(repeat)+' '+str(np.mean(np.abs(test_dataset[target] - mu))))
      print('RMSE '+str(repeat)+' '+str(np.std(test_dataset[target] - mu)))

  end_time = time.time()
  print("the total time consumption is " + str(end_time - start_time) + ".")
```

```
train_pipeline_batch(gpr, train_data, test_data, target, repeats=2, batch_num_train=batch_num_train, batch_size=batch_size, print_batch = True, print_repeat = True)
```

```
!zip -r /content/pkls.zip /content/pkls
```

      adding: content/pkls/ (stored 0%)
      adding: content/pkls/batch_34_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_27_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_3_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_24_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_16_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_19_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_14_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_18_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_29_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_33_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_17_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_13_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_12_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_32_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_23_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_21_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_2_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_6_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_9_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_31_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_7_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_28_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_20_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_0_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_5_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_30_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_25_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_10_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_1_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_8_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_22_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_26_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_4_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_11_repeat_0.pkl (deflated 35%)
      adding: content/pkls/batch_15_repeat_0.pkl (deflated 35%)

```
from google.colab import files
files.download("/content/pkls.zip")
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```
gpr.kernel.hyperparameters
```

    starting_probability : Uniform
    	p : 1.0
    stopping_probability : 0.00010000000000000009
    node_kernel : Composite
    	aromatic : Multiply
    		lhs : Constant
    			c : 0.02343420535068002
    		rhs : KroneckerDelta
    			h : 0.18860200789175988
    	atomic_number : Multiply
    		lhs : Constant
    			c : 5.400776905055444
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	charge : Multiply
    		lhs : Constant
    			c : 0.0145590563621646
    		rhs : SquareExponential
    			length_scale : 1.0
    	chiral : Multiply
    		lhs : Constant
    			c : 0.05149063984443944
    		rhs : KroneckerDelta
    			h : 0.12853164901299938
    	hcount : Multiply
    		lhs : Constant
    			c : 10.000000000000002
    		rhs : SquareExponential
    			length_scale : 0.07795674396128768
    	hybridization : Multiply
    		lhs : Constant
    			c : 0.02783411902444354
    		rhs : KroneckerDelta
    			h : 0.17414808329497075
    	ring_list : Multiply
    		lhs : Constant
    			c : 6.550981640628747
    		rhs : Convolution
    			base : KroneckerDelta
    				h : 0.10000000000000002
    edge_kernel : Composite
    	aromatic : Multiply
    		lhs : Constant
    			c : 2.331300736214766
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	conjugated : Multiply
    		lhs : Constant
    			c : 3.166641607522053
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	order : Multiply
    		lhs : Constant
    			c : 0.05549109210250382
    		rhs : KroneckerDelta
    			h : 0.25478481015091287
    	ring_stereo : Multiply
    		lhs : Constant
    			c : 0.033323606027544916
    		rhs : KroneckerDelta
    			h : 0.8
    	stereo : Multiply
    		lhs : Constant
    			c : 0.1324351226762118
    		rhs : KroneckerDelta
    			h : 0.14521546104851712

```
mu = gpr.predict(train.graphs)
```

```
plt.scatter(train[target], mu)
plt.show()
```

```
print('Training set')
print('MAE:', np.mean(np.abs(train[target] - mu)))
print('RMSE:', np.std(train[target] - mu))
```

    Training set
    MAE: 0.2074369392067814
    RMSE: 0.3195501910711287

```
mu_test = gpr.predict(test.graphs)
```

```
plt.scatter(test[target], mu_test)
plt.show()
```

```
print('Test set')
print('MAE:', np.mean(np.abs(test[target] - mu_test)))
print('RMSE:', np.std(test[target] - mu_test))
```

    Test set
    MAE: 1.2461703788060545
    RMSE: 1.6040210737240486

Workon the kernel. Find a kernel that trains and predicts well.

```
gpr2 = GaussianProcessRegressor(
    kernel=Normalization(
        MarginalizedGraphKernel(
            node_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                atomic_number=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                charge=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0),
                chiral=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                hcount=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0),
                hybridization=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                ring_list=uX.Constant(0.5, (0.01, 100.0)) * uX.Convolution(uX.KroneckerDelta(0.5,(0.1, 0.9)))
            ).normalized,
            edge_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                conjugated=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                order=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                ring_stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9))
            ).normalized,
            p=Uniform(1.0, p_bounds='fixed'),
            q=0.05
        )
    ),
    alpha=1e-2, #different from gpr in alpha where gpr's alpha is 1e-4
    optimizer=True,
    normalize_y=True,
    regularization='+',
)
```

```
#gpr2.fit(train.graphs, train[target], repeat=3, verbose=True)
gpr2.fit(train.graphs, train[target], repeat=1, verbose=True)
```

    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |   1.667e+05|   1.928e+05|  1.7773e+05|      -11034|  2.2742e+05|   1.3e+02|       1.7|
    |  1.5585e+05|  2.2171e+05|  1.6671e+05|      -10857|  2.2218e+05|   1.3e+02|       1.7|
    |  1.3711e+05|  2.4873e+05|  1.4756e+05|      -10447|  2.0803e+05|   1.2e+02|       1.8|
    |  1.3286e+05|  2.3627e+05|   1.432e+05|      -10338|  2.0336e+05|   1.1e+02|       1.7|
    |  1.2977e+05|  2.2744e+05|  1.4002e+05|      -10249|  1.9714e+05|   1.1e+02|       1.7|
    |  1.3006e+05|  2.1429e+05|  1.4032e+05|      -10258|  2.0119e+05|     1e+02|       1.7|
    |  1.2923e+05|  2.2103e+05|  1.3946e+05|      -10238|  1.9906e+05|   1.1e+02|       1.7|
    |  1.2888e+05|  2.2097e+05|  1.3911e+05|      -10228|  1.9865e+05|   1.1e+02|       1.7|
    |  1.2843e+05|  2.2338e+05|  1.3864e+05|      -10211|  1.9582e+05|   1.1e+02|       1.7|
    |  1.2837e+05|  2.2244e+05|  1.3858e+05|      -10212|   1.969e+05|   1.1e+02|       1.7|
    |  1.2836e+05|  2.2225e+05|  1.3857e+05|      -10212|  1.9696e+05|   1.1e+02|       1.7|
    |  1.2831e+05|  2.2155e+05|  1.3852e+05|      -10211|  1.9715e+05|   1.1e+02|       1.7|
    |  1.2826e+05|  2.2184e+05|  1.3847e+05|      -10209|  1.9704e+05|   1.1e+02|       1.7|
    |  1.2821e+05|  2.2181e+05|  1.3841e+05|      -10207|  1.9671e+05|   1.1e+02|       1.7|
    |  1.2819e+05|  2.2165e+05|   1.384e+05|      -10207|  1.9664e+05|   1.1e+02|       1.8|
    |  1.2818e+05|  2.2125e+05|  1.3838e+05|      -10207|  1.9671e+05|   1.1e+02|       1.8|
    |  1.2817e+05|  2.2114e+05|  1.3838e+05|      -10207|  1.9673e+05|   1.1e+02|       1.7|
    |  1.2815e+05|  2.2077e+05|  1.3836e+05|      -10207|  1.9677e+05|   1.1e+02|       1.7|
    |  1.2812e+05|   2.185e+05|  1.3833e+05|      -10209|  1.9695e+05|   1.1e+02|       1.7|
    |  1.2809e+05|   2.178e+05|   1.383e+05|      -10209|    1.97e+05|   1.1e+02|       1.7|
    |  1.2807e+05|  2.1764e+05|  1.3828e+05|      -10208|  1.9695e+05|   1.1e+02|       1.7|
    |  1.2805e+05|  2.1597e+05|  1.3826e+05|      -10209|  1.9693e+05|   1.1e+02|       1.7|
    |  1.2805e+05|   2.172e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1693e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.8|
    |  1.2804e+05|  2.1674e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1691e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2805e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.8|
    |  1.2805e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    Optimization result:
          fun: 128045.20193517115
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 21.67035135,   5.42797631,   0.91584794,   6.62684845,
           636.88297304,  10.22203408,   0.        ,   6.13155297,
             0.80318432, -45.79780911,   0.        ,   9.32446636,
             2.06137402,   7.94609186, 767.483899  ,  -8.19180354,
            92.93654292,   7.92503084, 118.62629465,   3.36719643,
            38.71995733,   7.96834618,   0.        , -11.08475724,
             7.2860016 ])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 31
          nit: 21
       status: 0
      success: True
            x: array([-9.21034037, -4.51710969, -1.98267626,  1.72521606, -2.30258509,
           -4.60517019,  0.        , -4.60517019, -1.8070945 ,  2.30258509,
           -2.88031384, -3.6019383 , -2.30258509,  1.91177723, -2.30258509,
            0.28746812, -2.30258509,  0.54884256, -2.30258509, -0.5686784 ,
           -2.30258509, -4.35637808, -0.22314355, -2.55717239, -2.21179129])

    <graphdot.model.gaussian_process.gpr.GaussianProcessRegressor at 0x7f6bf00630d0>

```
mu = gpr2.predict(train.graphs)
plt.scatter(train[target], mu)
plt.show()
```

```
print('Training set')
print('MAE:', np.mean(np.abs(train[target] - mu)))
print('RMSE:', np.std(train[target] - mu))
```

    Training set
    MAE: 0.7238389142657052
    RMSE: 0.9634943206325994

```
mu_test = gpr2.predict(test.graphs)
plt.scatter(test[target], mu_test)
plt.show()
```

```
print('Test set')
print('MAE:', np.mean(np.abs(test[target] - mu_test)))
print('RMSE:', np.std(test[target] - mu_test))
```

    Test set
    MAE: 0.9561539409612109
    RMSE: 1.2284268143181998


# Twitter Tweets

In this homework, we are going to play with Twitter data.

The data is represented as rows of of [JSON](https://en.wikipedia.org/wiki/JSON#Example) strings.
It consists of [tweets](https://dev.twitter.com/overview/api/tweets), [messages](https://dev.twitter.com/streaming/overview/messages-types), and a small amount of broken data (cannot be parsed as JSON).

For this homework, we will only focus on tweets and ignore all other messages.

## Tweets

A tweet consists of many data fields. [Here is an example](https://gist.github.com/arapat/03d02c9b327e6ff3f6c3c5c602eeaf8b). You can learn all about them in the Twitter API doc. We are going to briefly introduce only the data fields that will be used in this homework.

* `created_at`: Posted time of this tweet (time zone is included)
* `id_str`: Tweet ID - we recommend using `id_str` over using `id` as Tweet IDs, becauase `id` is an integer and may bring some overflow problems.
* `text`: Tweet content
* `user`: A JSON object for information about the author of the tweet
    * `id_str`: User ID
    * `name`: User name (may contain spaces)
    * `screen_name`: User screen name (no spaces)
* `retweeted_status`: A JSON object for information about the retweeted tweet (i.e. this tweet is not original but retweeteed some other tweet)
    * All data fields of a tweet except `retweeted_status`
* `entities`: A JSON object for all entities in this tweet
    * `hashtags`: An array for all the hashtags that are mentioned in this tweet
    * `urls`: An array for all the URLs that are mentioned in this tweet

## Data source

All tweets are collected using the [Twitter Streaming API](https://dev.twitter.com/streaming/overview).

## Users partition

Besides the original tweets, we will provide you with a Pickle file, which contains a partition over 452,743 Twitter users. It contains a Python dictionary `{user_id: partition_id}`. The users are partitioned into 7 groups.

# Part 0: Load data to a RDD

The tweets data is stored on AWS S3. We have in total a little over 1 TB of tweets. We provide 10 MB of tweets for your local development. For the testing and grading on the homework server, we will use different data.

## Testing on the homework server
In the Playground, we provide input size of 4 GB. To test, read file list from `../../Data/hw6-files-4gb.txt`. More input files will be added soon.

For final submission, make sure to read files list from `../../Data/hw6-files-final.txt`. Otherwise your program will receive no points.

## Local test

For local testing, read files list from `../../Data/hw6-files-10mb.txt`.
Now let's see how many lines there are in the input files.

1. Make RDD from the list of files in `hw6-files-10mb.txt`.
2. Mark the RDD to be cached (so in next operation data will be loaded in memory)
3. call the `print_count` method to print number of lines in all these files

It should print
```
Number of elements: 2193
```

```python
def print_count(rdd):
    print 'Number of elements:', rdd.count()
```

```python
# Your code here
import findspark
findspark.init()

from pyspark import SparkContext
sc = SparkContext(master="local[4]")

filename_list = []
with open("../../Data/hw6-files-10mb.txt") as f:
    filename_list = [filename.strip() for filename in f]
rdd = sc.textFile(",".join(filename_list)).cache()

print_count(rdd)
```

    Number of elements: 2193

# Part 1: Parse JSON strings to JSON objects

Python has built-in support for JSON.

```python
import json

json_example = '''
{
    "id": 1,
    "name": "A green door",
    "price": 12.50,
    "tags": ["home", "green"]
}
'''

json_obj = json.loads(json_example)
json_obj
```

    {u'id': 1,
     u'name': u'A green door',
     u'price': 12.5,
     u'tags': [u'home', u'green']}

## Broken tweets and irrelevant messages

The data of this assignment may contain broken tweets (invalid JSON strings). So make sure that your code is robust for such cases.

In addition, some lines in the input file might not be tweets, but messages that the Twitter server sent to the developer (such as [limit notices](https://dev.twitter.com/streaming/overview/messages-types#limit_notices)). Your program should also ignore these messages.

*Hint:* [Catch the ValueError](http://stackoverflow.com/questions/11294535/verify-if-a-string-is-json-in-python)

(1) Parse raw JSON tweets to obtain valid JSON objects. From all valid tweets, construct a pair RDD of `(user_id, text)`, where `user_id` is the `id_str` data field of the `user` dictionary (read [Tweets](#Tweets) section above), `text` is the `text` data field.

```python
import json

def safe_parse(raw_json):
    try:
        tweet = json.loads(raw_json)
        pair = [tweet["user"]["id_str"], tweet["text"].encode("utf-8")]
        return pair
    except:
        return None

user_text_rdd = rdd.map(safe_parse).filter(lambda p: p is not None).cache()
```

(2) Count the number of different users in all valid tweets (hint: [the `distinct()` method](https://spark.apache.org/docs/latest/programming-guide.html#transformations)).

It should print
```
The number of unique users is: 2083
```

```python
def print_users_count(count):
    print 'The number of unique users is:', count
```

```python
# your code here
count = user_text_rdd.map(lambda p: p[0]).distinct().count()
print_users_count(count)
```

    The number of unique users is: 2083

# Part 2: Number of posts from each user partition

Load the Pickle file `../../Data/users-partition.pickle`, you will get a dictionary which represents a partition over 452,743 Twitter users, `{user_id: partition_id}`. The users are partitioned into 7 groups. For example, if the dictionary is loaded into a variable named `partition`, the partition ID of the user `59458445` is `partition["59458445"]`. These users are partitioned into 7 groups. The partition ID is an integer between 0-6.

Note that the user partition we provide doesn't cover all users appear in the input data.

(1) Load the pickle file.

```python
# your code here
import pickle

partition_dict = None
with open("../../Data/users-partition.pickle") as f:
    partition_dict = pickle.load(f)
```

(2) Count the number of posts from each user partition

Count the number of posts from group 0, 1, ..., 6, plus the number of posts from users who are not in any partition. Assign users who are not in any partition to the group 7.

Put the results of this step into a pair RDD `(group_id, count)` that is sorted by key.

```python
# your code here
def get_group_id(user_id):
    return partition_dict[user_id] if user_id in partition_dict else 7

from operator import add

group_count_rdd = user_text_rdd.map(lambda p: [get_group_id(p[0]), 1]).reduceByKey(add).sortByKey()
```

(3) Print the post count using the `print_post_count` function we provided.

It should print

```
Group 0 posted 81 tweets
Group 1 posted 199 tweets
Group 2 posted 45 tweets
Group 3 posted 313 tweets
Group 4 posted 86 tweets
Group 5 posted 221 tweets
Group 6 posted 400 tweets
Group 7 posted 798 tweets
```

```python
def print_post_count(counts):
    for group_id, count in counts:
        print 'Group %d posted %d tweets' % (group_id, count)
```

```python
# your code here
print_post_count(group_count_rdd.collect())
```

    Group 0 posted 81 tweets
    Group 1 posted 199 tweets
    Group 2 posted 45 tweets
    Group 3 posted 313 tweets
    Group 4 posted 86 tweets
    Group 5 posted 221 tweets
    Group 6 posted 400 tweets
    Group 7 posted 798 tweets

# Part 3:  Tokens that are relatively popular in each user partition

In this step, we are going to find tokens that are relatively popular in each user partition.

We define the number of mentions of a token $t$ in a specific user partition $k$ as the number of users from the user partition $k$ that ever mentioned the token $t$ in their tweets. Note that even if some users might mention a token $t$ multiple times or in multiple tweets, a user will contribute at most 1 to the counter of the token $t$.

Please make sure that the number of mentions of a token is equal to the number of users who mentioned this token but NOT the number of tweets that mentioned this token.

Let $N_t^k$ be the number of mentions of the token $t$ in the user partition $k$. Let $N_t^{all} = \sum_{i=0}^7 N_t^{i}$ be the number of total mentions of the token $t$.

We define the relative popularity of a token $t$ in a user partition $k$ as the log ratio between $N_t^k$ and $N_t^{all}$, i.e.

\begin{equation}
p_t^k = \log \frac{C_t^k}{C_t^{all}}.
\end{equation}

You can compute the relative popularity by calling the function `get_rel_popularity`.

(0) Load the tweet tokenizer.

```python
# %load happyfuntokenizing.py
#!/usr/bin/env python

"""
This code implements a basic, Twitter-aware tokenizer.

A tokenizer is a function that splits a string of text into words. In
Python terms, we map string and unicode objects into lists of unicode
objects.

There is not a single right way to do tokenizing. The best method
depends on the application.  This tokenizer is designed to be flexible
and this easy to adapt to new domains and tasks.  The basic logic is
this:

1. The tuple regex_strings defines a list of regular expression
   strings.

2. The regex_strings strings are put, in order, into a compiled
   regular expression object called word_re.

3. The tokenization is done by word_re.findall(s), where s is the
   user-supplied string, inside the tokenize() method of the class
   Tokenizer.

4. When instantiating Tokenizer objects, there is a single option:
   preserve_case.  By default, it is set to True. If it is set to
   False, then the tokenizer will downcase everything except for
   emoticons.

The __main__ method illustrates by tokenizing a few examples.

I've also included a Tokenizer method tokenize_random_tweet(). If the
twitter library is installed (http://code.google.com/p/python-twitter/)
and Twitter is cooperating, then it should tokenize a random
English-language tweet.

Julaiti Alafate:
  I modified the regex strings to extract URLs in tweets.
"""

__author__ = "Christopher Potts"
__copyright__ = "Copyright 2011, Christopher Potts"
__credits__ = []
__license__ = "Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-nc-sa/3.0/"
__version__ = "1.0"
__maintainer__ = "Christopher Potts"
__email__ = "See the author's website"

######################################################################

import re
import htmlentitydefs

######################################################################
# The following strings are components in the regular expression
# that is used for tokenizing. It's important that phone_number
# appears first in the final regex (since it can contain whitespace).
# It also could matter that tags comes after emoticons, due to the
# possibility of having text like
#
#     <:| and some text >:)
#
# Most imporatantly, the final element should always be last, since it
# does a last ditch whitespace-based tokenization of whatever is left.

# This particular element is used in a couple ways, so we define it
# with a name:
emoticon_string = r"""
    (?:
      [<>]?
      [:;=8]                     # eyes
      [\-o\*\']?                 # optional nose
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      |
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      [\-o\*\']?                 # optional nose
      [:;=8]                     # eyes
      [<>]?
    )"""

# The components of the tokenizer:
regex_strings = (
    # Phone numbers:
    r"""
    (?:
      (?:            # (international)
        \+?[01]
        [\-\s.]*
      )?
      (?:            # (area code)
        [\(]?
        \d{3}
        [\-\s.\)]*
      )?
      \d{3}          # exchange
      [\-\s.]*
      \d{4}          # base
    )"""
    ,
    # URLs:
    r"""http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"""
    ,
    # Emoticons:
    emoticon_string
    ,
    # HTML tags:
     r"""<[^>]+>"""
    ,
    # Twitter username:
    r"""(?:@[\w_]+)"""
    ,
    # Twitter hashtags:
    r"""(?:\#+[\w_]+[\w\'_\-]*[\w_]+)"""
    ,
    # Remaining word types:
    r"""
    (?:[a-z][a-z'\-_]+[a-z])       # Words with apostrophes or dashes.
    |
    (?:[+\-]?\d+[,/.:-]\d+[+\-]?)  # Numbers, including fractions, decimals.
    |
    (?:[\w_]+)                     # Words without apostrophes or dashes.
    |
    (?:\.(?:\s*\.){1,})            # Ellipsis dots.
    |
    (?:\S)                         # Everything else that isn't whitespace.
    """
    )

######################################################################
# This is the core tokenizing regex:

word_re = re.compile(r"""(%s)""" % "|".join(regex_strings), re.VERBOSE | re.I | re.UNICODE)

# The emoticon string gets its own regex so that we can preserve case for them as needed:
emoticon_re = re.compile(regex_strings[1], re.VERBOSE | re.I | re.UNICODE)

# These are for regularizing HTML entities to Unicode:
html_entity_digit_re = re.compile(r"&#\d+;")
html_entity_alpha_re = re.compile(r"&\w+;")
amp = "&"

######################################################################

class Tokenizer:
    def __init__(self, preserve_case=False):
        self.preserve_case = preserve_case

    def tokenize(self, s):
        """
        Argument: s -- any string or unicode object
        Value: a tokenize list of strings; conatenating this list returns the original string if preserve_case=False
        """
        # Try to ensure unicode:
        try:
            s = unicode(s)
        except UnicodeDecodeError:
            s = str(s).encode('string_escape')
            s = unicode(s)
        # Fix HTML character entitites:
        s = self.__html2unicode(s)
        # Tokenize:
        words = word_re.findall(s)
        # Possible alter the case, but avoid changing emoticons like :D into :d:
        if not self.preserve_case:
            words = map((lambda x : x if emoticon_re.search(x) else x.lower()), words)
        return words

    def tokenize_random_tweet(self):
        """
        If the twitter library is installed and a twitter connection
        can be established, then tokenize a random tweet.
        """
        try:
            import twitter
        except ImportError:
            print "Apologies. The random tweet functionality requires the Python twitter library: http://code.google.com/p/python-twitter/"
        from random import shuffle
        api = twitter.Api()
        tweets = api.GetPublicTimeline()
        if tweets:
            for tweet in tweets:
                if tweet.user.lang == 'en':
                    return self.tokenize(tweet.text)
        else:
            raise Exception("Apologies. I couldn't get Twitter to give me a public English-language tweet. Perhaps try again")

    def __html2unicode(self, s):
        """
        Internal metod that seeks to replace all the HTML entities in
        s with their corresponding unicode characters.
        """
        # First the digits:
        ents = set(html_entity_digit_re.findall(s))
        if len(ents) > 0:
            for ent in ents:
                entnum = ent[2:-1]
                try:
                    entnum = int(entnum)
                    s = s.replace(ent, unichr(entnum))
                except:
                    pass
        # Now the alpha versions:
        ents = set(html_entity_alpha_re.findall(s))
        ents = filter((lambda x : x != amp), ents)
        for ent in ents:
            entname = ent[1:-1]
            try:
                s = s.replace(ent, unichr(htmlentitydefs.name2codepoint[entname]))
            except:
                pass
            s = s.replace(amp, " and ")
        return s
```

```python
from math import log

tok = Tokenizer(preserve_case=False)

def get_rel_popularity(c_k, c_all):
    return log(1.0 * c_k / c_all) / log(2)

def print_tokens(tokens, gid = None):
    group_name = "overall"
    if gid is not None:
        group_name = "group %d" % gid
    print '=' * 5 + ' ' + group_name + ' ' + '=' * 5
    for t, n in tokens:
        print "%s\t%.4f" % (t, n)
    print
```

(1) Tokenize the tweets using the tokenizer we provided above named `tok`. Count the number of mentions for each tokens regardless of specific user group.

Call `print_count` function to show how many different tokens we have.

It should print
```
Number of elements: 8949
```

```python
# your code here
tokenizer = Tokenizer()
token_user_rdd = user_text_rdd.flatMap(lambda p: [[word, [p[0]]] for word in set(tokenizer.tokenize(p[1]))]) \
                              .reduceByKey(add)\
                              .map(lambda p: [p[0], list(set(p[1]))]).cache()
print_count(token_user_rdd)
```

    Number of elements: 8979

(2) Tokens that are mentioned by too few users are usually not very interesting. So we want to only keep tokens that are mentioned by at least 100 users. Please filter out tokens that don't meet this requirement.

Call `print_count` function to show how many different tokens we have after the filtering.

Call `print_tokens` function to show top 20 most frequent tokens.

It should print
```
Number of elements: 44
===== overall =====
:	1388.0000
rt	1237.0000
.	826.0000
…	673.0000
the	623.0000
trump	582.0000
to	499.0000
,	489.0000
a	404.0000
is	376.0000
in	297.0000
of	292.0000
and	288.0000
for	281.0000
!	269.0000
?	210.0000
on	195.0000
i	192.0000
you	191.0000
this	190.0000
```

```python
# your code here
frequent_token_rdd = token_user_rdd.map(lambda p: [p[0], len(p[1])])\
                                   .filter(lambda p: p[1] >= 100)\
                                   .sortBy(lambda p: p[1], ascending = False)
print_count(frequent_token_rdd)
print_tokens(frequent_token_rdd.take(20))
```

    Number of elements: 52
    ===== overall =====
    :	1386.0000
    rt	1237.0000
    .	865.0000
    \	745.0000
    the	621.0000
    trump	595.0000
    x80	545.0000
    xe2	543.0000
    to	499.0000
    ,	489.0000
    xa6	457.0000
    a	403.0000
    is	376.0000
    in	296.0000
    '	294.0000
    of	292.0000
    and	287.0000
    for	280.0000
    !	269.0000
    ?	210.0000

(3) For all tokens that are mentioned by at least 100 users, compute their relative popularity in each user group. Then print the top 10 tokens with highest relative popularity in each user group. In case two tokens have same relative popularity, break the tie by printing the alphabetically smaller one.

**Hint:** Let the relative popularity of a token $t$ be $p$. The order of the items will be satisfied by sorting them using (-p, t) as the key.

It should print
```
===== group 0 =====
...	-3.5648
at	-3.5983
hillary	-4.0875
i	-4.1255
bernie	-4.1699
not	-4.2479
https	-4.2695
he	-4.2801
in	-4.3074
are	-4.3646

===== group 1 =====
#demdebate	-2.4391
-	-2.6202
&	-2.7472
amp	-2.7472
clinton	-2.7570
;	-2.7980
sanders	-2.8838
?	-2.9069
in	-2.9664
if	-3.0138

===== group 2 =====
are	-4.6865
and	-4.7105
bernie	-4.7549
at	-4.7682
sanders	-4.9542
that	-5.0224
in	-5.0444
donald	-5.0618
a	-5.0732
#demdebate	-5.1396

===== group 3 =====
#demdebate	-1.3847
bernie	-1.8480
sanders	-2.1887
of	-2.2356
that	-2.3785
the	-2.4376
…	-2.4403
clinton	-2.4467
hillary	-2.4594
be	-2.5465

===== group 4 =====
hillary	-3.7395
sanders	-3.9542
of	-4.0199
clinton	-4.0790
at	-4.1832
in	-4.2143
a	-4.2659
on	-4.2854
.	-4.3681
the	-4.4251

===== group 5 =====
cruz	-2.3861
he	-2.6280
are	-2.7796
will	-2.7829
the	-2.8568
is	-2.8822
for	-2.9250
that	-2.9349
of	-2.9804
this	-2.9849

===== group 6 =====
@realdonaldtrump	-1.1520
cruz	-1.4532
https	-1.5222
!	-1.5479
not	-1.8904
…	-1.9269
will	-2.0124
it	-2.0345
this	-2.1104
to	-2.1685

===== group 7 =====
donald	-0.6422
...	-0.7922
sanders	-1.0282
trump	-1.1296
bernie	-1.2106
-	-1.2253
you	-1.2376
clinton	-1.2511
if	-1.2880
i	-1.2996
```

```python
# your code here
def get_relpop_vec(users):
    count_vec = [0 for k in range(8)]
    for user in users:
        count_vec[get_group_id(user)] += 1
    c_all = sum(count_vec)
    relpop_vec = [get_rel_popularity(count_vec[k], c_all) if count_vec[k] > 0 else float('-inf')
                      for k in range(8)]
    return relpop_vec

token_relpop_rdd = token_user_rdd.filter(lambda p: len(p[1]) >= 100)\
                                 .map(lambda p : [p[0].encode("utf-8"), get_relpop_vec(p[1])])\
                                 .cache()

for k in range(8):
    group_token_relpop_rdd = token_relpop_rdd.map(lambda p: [p[0], p[1][k]])\
                                             .sortBy(lambda p: (-p[1], p[0]))
    print_tokens(group_token_relpop_rdd.take(10), k)
```

    ===== group 0 =====
    ...	-3.5648
    at	-3.5983
    hillary	-4.0484
    bernie	-4.1430
    not	-4.2479
    he	-4.2574
    i	-4.2854
    s	-4.3309
    are	-4.3646
    in	-4.4021

    ===== group 1 =====
    #demdebate	-2.4391
    -	-2.6202
    clinton	-2.7174
    &	-2.7472
    amp	-2.7472
    ;	-2.7980
    sanders	-2.8745
    ?	-2.9069
    in	-2.9615
    if	-2.9861

    ===== group 2 =====
    are	-4.6865
    and	-4.7055
    bernie	-4.7279
    at	-4.7682
    sanders	-4.9449
    in	-5.0395
    donald	-5.0531
    a	-5.0697
    #demdebate	-5.1396
    that	-5.1599

    ===== group 3 =====
    #demdebate	-1.3847
    bernie	-1.8535
    sanders	-2.1793
    of	-2.2356
    t	-2.2675
    clinton	-2.4179
    hillary	-2.4203
    the	-2.4330
    xa6	-2.4962
    that	-2.5160

    ===== group 4 =====
    hillary	-3.8074
    sanders	-3.9449
    of	-4.0199
    what	-4.0875
    clinton	-4.0959
    at	-4.1832
    in	-4.2095
    a	-4.2623
    on	-4.2854
    '	-4.2928

    ===== group 5 =====
    cruz	-2.3344
    he	-2.6724
    will	-2.7705
    are	-2.7796
    the	-2.8522
    is	-2.8822
    that	-2.9119
    this	-2.9542
    for	-2.9594
    of	-2.9804

    ===== group 6 =====
    @realdonaldtrump	-1.1520
    cruz	-1.4657
    n	-1.4877
    !	-1.5479
    not	-1.8904
    xa6	-1.9172
    xe2	-1.9973
    /	-2.0238
    x80	-2.0240
    it	-2.0506

    ===== group 7 =====
    donald	-0.6471
    ...	-0.7922
    sanders	-1.0380
    what	-1.1178
    trump	-1.1293
    bernie	-1.2044
    you	-1.2099
    -	-1.2253
    if	-1.2602
    clinton	-1.2681

(4) (optional, not for grading) The users partition is generated by a machine learning algorithm that tries to group the users by their political preferences. Three of the user groups are showing supports to Bernie Sanders, Ted Cruz, and Donald Trump.

If your program looks okay on the local test data, you can try it on the larger input by submitting your program to the homework server. Observe the output of your program to larger input files, can you guess the partition IDs of the three groups mentioned above based on your output?

```python
# Change the values of the following three items to your guesses
users_support = [
    (-1, "Bernie Sanders"),
    (-1, "Ted Cruz"),
    (-1, "Donald Trump")
]

for gid, candidate in users_support:
    print "Users from group %d are most likely to support %s." % (gid, candidate)
```

    Users from group -1 are most likely to support Bernie Sanders.
    Users from group -1 are most likely to support Ted Cruz.
    Users from group -1 are most likely to support Donald Trump.


October 12, 2017: presentation time

October 8, 2017: second draft

October 3, 2017: first draft

Abstract

This presentation is a summary of the work I've done during the second year

of my PhD course in Computer Science, XXXI cycle @ UniFI

```python
from IPython.display import Markdown, Image, Latex

from collections import defaultdict

from muk.core import *
from muk.ext import *

from sympy import IndexedBase, symbols, latex, init_printing, Eq, Matrix
init_printing()

toc = ["", "exams & courses & conferences", "what I've done", "what I'm working on"]#, "thesis arguments"]
toc_iter = iter(toc[1:])

def reference_to_this_talk():
    src = '<a href="{href}">{href}</a>'
    return Markdown(src.format(href=r'http://massimo-nocentini.github.io/PhD/second-year-summary/talk.html'))

def table_of_contents():
    src = r'# TOC'
    return Markdown('\n- '.join(toc))

def greetings(smiley=True):
    return Markdown("<h1>{greet} {smile}</h1>".format(
        greet="Thanks for coming ", smile=":)" if smiley else ""))

def next_topic():
    return Markdown("# {topic}".format(topic=next(toc_iter)))
```

```python
__AUTHOR__ = ("Massimo Nocentini",
              "massimo.nocentini@unifi.it",
              "https://github.com/massimo-nocentini/")

__SUPERVISOR__ = ("Donatella Merlini ",
                  "donatella.merlini@unifi.it")

__ACKNOWLEDGEMENT__ = {"Giovanni Marchetti",
                       "Marco Maggesi",
                       "Antonio Bernini",
                       "Cecilia Verri"}

reference_to_this_talk()
```

http://massimo-nocentini.github.io/PhD/second-year-summary/talk.html

```python
table_of_contents()
```

- exams & courses & conferences
- what I've done
- what I'm working on

```python
next_topic()
```

# exams & courses & conferences

### Analysis of algorithms and data structures through Riordan arrays

taught by professor Donatella Merlini, 2 CFU:
- we studied enumeration of _binary words avoiding patterns_
- submitted a related paper __Algebraic generating functions for languages avoiding Riordan patterns__
    - received a review in July, just resubmitted with fixes
- more on this in a little while...

### Advanced programming techniques

taught by professor Lorenzo Bettini, 2 CFU:
- hard practice about _TDD, BDD_ and _CI_, the Pythonic way of course
- reworked on my own the _bootstrap of a TDD framework_ inspired by [Kent Beck][beck]
- more on this [here][repo]

[repo]:https://github.com/massimo-nocentini/advanced-programming-techniques-course
[beck]:https://www.amazon.it/Test-Driven-Development-Example-Kent-Beck/dp/0321146530/ref=sr_1_1?s=english-books&ie=UTF8&qid=1507040431&sr=1-1&keywords=beck+test+driven+development

### Bayesian methods for high-dimensional data

taught by professor Francesco Stingo, 2 CFU:
- new topics for me: _bayesian model selection for GLM_ and _high-dimension graphical model_
- worked through _Evolutionary Stochastic Search_, Gibbs and Metropolis-Hastings samplers
- more on this [here][repo]

[repo]:https://github.com/massimo-nocentini/statistical-models/tree/master/bayesian-inference

### Statistical learning

taught by professor Giovanni Maria Marchetti, 3 CFU:
- new topics for me: _graphical models_, $d$-connection, _categorical data analysis_
- worked through _Decomposition by clique separators_, theoretically
- more on this [here][repo]

[repo]:https://github.com/massimo-nocentini/statistical-models/tree/master/clique-separators

## Present and Future exams

### Numerical methods for simulation

taught by professor Luigi Brugnano, 3 CFU: _functions of matrices_.

### Graph mining

taught by professor Pierluigi Crescenzi, 3 CFU: _OEIS sequences networks_.

## Conferences

Last July, in Rimini, __EUROPYTHON__, https://ep2017.europython.eu/en/
- attended almost _pydata_ and _core language_ sessions
- met _Alex Martelli_

Last September, in Oxford, __ICFP2017__, https://icfp17.sigplan.org/
- served as _student volounteer_
- attended all ICFP talks

## SKKU open school

Given two talks (the first one [recorded][youtube]) at open school _Combinatorial methods in the analysis of algorithms and data structures_ ([link][os]) February 20-24, Suwon, Korea.

[youtube]:https://www.youtube.com/watch?v=6w8oi2VeFDM
[os]:http://shb.skku.edu/aorc/conference_info/new.jsp

```python
next_topic()
```

# what I've done

## in a nutshell

theoretical stuff:
- submitted a paper with Donatella for my master thesis about _binary words enumerations_
- application of the framework of _functions matrices_ to the _Riordan group_

coding stuff:
 - 70% Python
   - rewritten a set of tools to mine the [OEIS][oeis]
   - written a _complete_ port of _microKanren_
 - 20% Lisp
   - [MiniKanren][mk] under the hood
 - 10% OCaml
   - begin a _certified_ implementation of microKanren into _HOL Light_, joint work with professor _Marco Maggesi_

[mk]:http://minikanren.org/

[oeis]:http://oeis.org/

## theoretical time

## practical time

meet _microKanren_
- a DSL for relational programming, in Scheme
- purely functional of [_miniKanren_][mk]
- *explicit streams* of satisfying states, _goal-based_ approach
- _unification_ instead of _SLD-NF resolution_
- complete, _unfair_ search strategy

my contribution
- _Pythonic_ [implementation][mkpy]: functional at the core, objective at the interface
- generators subsume _countably_-satisfiable relations; complete, _fair_ [search][dovetail]
- _The Reasoned Schemer_ fully tested via [Travis CI][travis]; moreover, [read the docs][rtfd]
- case studies: Smullyan puzzles and combinatorics
- tweaking HOL Light for _certified deductions_, [wip][klight]

[mk]:http://minikanren.org/
[travis]:https://travis-ci.org/massimo-nocentini/microkanrenpy
[rtfd]:http://microkanrenpy.readthedocs.io/en/latest/
[klight]:https://github.com/massimo-nocentini/kanren-light
[mkpy]:https://github.com/massimo-nocentini/microkanrenpy
[dovetail]:http://microkanrenpy.readthedocs.io/en/latest/under_the_hood.html#muk.core.mplus

```python
rvar(0) # a logic variable
```

    ▢₀

```python
run(succeed) # a goal that always succeeds
```

    [Tautology]

```python
run(fail) # a goal that always fails
```

```python
run(fresh(lambda q: succeed)) # a free variable without association
```

    [▢₀]

```python
run(unify(3, 3)) # unification of ground values
```

    [Tautology]

```python
run(fresh(lambda q: unify(3, q))) # unification of a variable
```

```python
run(fresh(lambda q: unify([[2, 3], 1, 2, 3], [q, 1] + q))) # list unification
```

```python
run(fresh(lambda q, z: unify(q, z) & unify(z, 3))) # co-sharing
```

```python
run(fresh(lambda q: unify(q, False) | unify(q, True))) # disjunction
```

    [False, True]

```python
run(fresh(lambda q:
          fresh(lambda q: unify(q, False)) &
          unify(q, True))) # conjunction
```

    [True]

```python
def father(p, s):
    return conde([unify(p, 'paul'), unify(s, 'jason')],
                 [unify(p, 'john'), unify(s, 'henry')],
                 [unify(p, 'jason'), unify(s, 'tom')],
                 [unify(p, 'peter'), unify(s, 'brian')],
                 [unify(p, 'tom'), unify(s, 'peter')])

def grand_father(g, s):
    return fresh(lambda p: father(g, p) & father(p, s))

run(fresh(lambda rel, p, s: grand_father(p, s) & unify([p, s], rel)))
```

    [['paul', 'tom'], ['jason', 'peter'], ['tom', 'brian']]

```python
def fives(x):
    return unify(5, x) | fives(x)

try:
    run(fresh(lambda x: fives(x)))
except RecursionError:
    pass
```

```python
def fives(x):
    return unify(5, x) | fresh(lambda y: fives(y))

run(fresh(fives), n=5)
```

    [5, ▢₀, ▢₀, ▢₀, ▢₀]

```python
g = fresh(lambda x: fives(x))
states = g(emptystate())
[next(states) for i in range(5)]
```

    [state(sub={x₀: 5}, next_index=1),
     state(sub={y₁: 5}, next_index=2),
     state(sub={y₂: 5}, next_index=3),
     state(sub={y₃: 5}, next_index=4),
     state(sub={y₄: 5}, next_index=5)]

```python
def fives(x):
    return unify(5, x) | fresh(lambda: fives(x))

run(fresh(fives), n=5)
```

```python
g = fresh(lambda x: fives(x))
states = g(emptystate())
[next(states) for i in range(5)]
```

    [state(sub={x₀: 5}, next_index=1),
     state(sub={x₀: 5}, next_index=1),
     state(sub={x₀: 5}, next_index=1),
     state(sub={x₀: 5}, next_index=1),
     state(sub={x₀: 5}, next_index=1)]

```python
def nats(x, n=0):
    return unify(n, x) | fresh(lambda: nats(x, n+1))

run(fresh(lambda x: nats(x)), n=10)
```

```python
def nullo(l):
    return unify([], l)

def appendo(r, s, out):
    def A(r, out):
        return conde([nullo(r), unify(s, out)],
                     else_clause=fresh(lambda a, d, res:
                                           unify([a]+d, r) &
                                           unify([a]+res, out) &
                                           fresh(lambda: A(d, res))))
    return A(r, out)
```

```python
run(fresh(lambda l, q: appendo([1,2,3]+q, [4,5,6], l)), n=5)
```

    [[1, 2, 3, 4, 5, 6],
     [1, 2, 3, ▢₀, 4, 5, 6],
     [1, 2, 3, ▢₀, ▢₁, 4, 5, 6],
     [1, 2, 3, ▢₀, ▢₁, ▢₂, 4, 5, 6],
     [1, 2, 3, ▢₀, ▢₁, ▢₂, ▢₃, 4, 5, 6]]

```python
run(fresh(lambda r, x, y:
          appendo(x, y, ['cake', 'with', 'ice', 'd', 't']) &
          unify([x, y], r)))
```

    [[[], ['cake', 'with', 'ice', 'd', 't']],
     [['cake'], ['with', 'ice', 'd', 't']],
     [['cake', 'with'], ['ice', 'd', 't']],
     [['cake', 'with', 'ice'], ['d', 't']],
     [['cake', 'with', 'ice', 'd'], ['t']],
     [['cake', 'with', 'ice', 'd', 't'], []]]

```python
def dycko(α):
    return conde([nullo(α), succeed],
                  else_clause=fresh(lambda β, γ:
                                    appendo(['(']+β, [')']+γ, α) @
                                            (dycko(β) @ dycko(γ))))

paths = run(fresh(lambda α: dycko(α)), n=80)
D = defaultdict(list)
for α in map(lambda α: ''.join(α), paths):
    D[len(α)//2].append(α)
```

```python
from collections import namedtuple

dyck = namedtuple('dyck', ['paths', 'count'])
```

```python
[dyck(paths, len(paths)) for i in range(5) for paths in [D[i]]]
```

    [dyck(paths=[''], count=1),
     dyck(paths=['()'], count=1),
     dyck(paths=['()()', '(())'], count=2),
     dyck(paths=['()()()', '(())()', '()(())', '(()())', '((()))'], count=5),
     dyck(paths=['(())()()', '()()()()', '(())(())', '()(())()', '(()()())', '(()())()', '()()(())', '(()(()))', '((()))()', '()(()())', '((())())', '()((()))', '((()()))', '(((())))'], count=14)]

```python
P = IndexedBase('P')
n, m = symbols('n m')

def pascalo(depth, r, c, α):
    if not depth: return unify([P[r,c]], α)
    return fresh(lambda β, γ: (pascalo(depth-1, r-1, c-1, β) @
                               pascalo(depth-1, r-1, c, γ) @
                               appendo(β, γ, α)))

unfoldings = {d:sum(addends)
              for d in range(6)
              for addends in run(fresh(lambda α: pascalo(d, n, m, α)))}
```

```python
Matrix(5, 1, lambda i, j: unfoldings[i+1])
```

```python
next_topic()
```

# what I'm working on

- *recursively defined structures*, both *inductively* and *co*-inductively
- _matrices functions_ and _companion matrices_ applied to Riordan arrays
- `(Y Y)` works! you do know `Y`, didn't you?
    - *quine generation* with microKanren
    - _disequality_ constraints support
- *reasoning about Smullyan machines*, especially those that talk about themselves

```python
try:
    next_topic()
except StopIteration:
    import this
```

    The Zen of Python, by Tim Peters

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren't special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one-- and preferably only one --obvious way to do it.
    Although that way may not be obvious at first unless you're Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it's a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea -- let's do more of those!

```python
greetings(smiley=True)
```

Thanks for coming  :)

---

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.