# Polynomial Unconstrained Boolean Optimization Problem in MBQC

<em> Copyright (c) 2021 Institute for Quantum Computing, Baidu Inc. All Rights Reserved. </em>

In the tutorial [Measurement-based Quantum Approximate Optimization Algorithm](QAOA_EN.ipynb), we give a brief introduction to the **polynomial unconstrained boolean optimization (PUBO) problem** and propose the **measurement-based quantum approximate optimization algorithm (MB-QAOA)** to solve it. For interested readers, please refer to the previous tutorial for more information. In this tutorial, we will showcase two specific examples as practical demonstrations of MB-QAOA. The first one is a concrete PUBO problem, while the second one is a **maximum cut (MaxCut)** problem.

## Example: PUBO Problem

Let us first briefly review what a PUBO problem is. Consider a polynomial of $n$ variables $x = \{x_1,\cdots,x_n\}$,

$$
C(x) = \sum_{\lambda \in \Lambda } \alpha_{\lambda} \prod_{j \in \lambda} x_j,\tag{1}
$$

where $x_i \in \{0,1\}$ is a boolean variable, $\underset{j \in \lambda}{\prod} x_j$ is a monomial, $\lambda \subseteq [n]:= \{1, 2, ..., n\}$ is a set of indexes, $\Lambda$ is the set of index sets, $\alpha_\lambda$ is the real coefficient of monomial. In PUBO, $C(x)$ is called the objective polynomial. We hope to find an optimal solution $x^* = \{x_1^*, x_2^*, ..., x_n^*\} $ maximizing the value of objective polynomial. That is to find

$$
x^* = \underset{x}{\text{argmax}} \ C(x).\tag{2}
$$

For code implementation, we require that a standard polynomial is input as a list whose first item is the number of variables and the second item is a dictionary of all the monomials ('cons' stands for the constant item). In the dictionary, we make monomial variables split with ',' as keys and the corresponding coefficients as values. For example, suppose we want to input a polynomial $x_1 + x_2 - x_3 + x_1 x_2 - x_1 x_2 x_3 + 0.5$, we need to code as follows:

```python
# Number of variables
var_num = 3
# Polynomial as a dictionary
poly_dict = {'x_1': 1, 'x_2': 1, 'x_3': -1, 'x_1,x_2': 1, 'x_1,x_2,x_3': -1, 'cons':0.5}
# Construct the list required
polynomial = [var_num, poly_dict]
```

**Note:** As the variables are boolean, the power of variables in a monomial should be no greater than 1. That is, each variable should appear at most once in a key of the dictionary. For instance, it is not a valid to input something like {'x_1,x_1,x_2': 1}. Also, we set variable subscripts by consecutive numbers starting from '1' to be consistent with math conventions. A polynomial like $x_1 x_2 + x_6$ will raise an error automatically. A valid polynomial should be like $x_1x_2 + x_3$.

For convenience, we provide a function `is_poly_valid` in `pubo` to check the validity of the user's input. If the polynomial is valid, it will print a statement "The polynomial is valid.". Otherwise, an error will be raised.

```python
from paddle_quantum.mbqc.QAOA.pubo import is_poly_valid
```
We also provide a function `random_poly` to generate a random boolean polynomial with a given number of variables.

```python
from paddle_quantum.mbqc.QAOA.pubo import random_poly
```
**Note:** The randomly generated polynomial is not always valid and we still need to check the validity before calculation.

### Code implementation to slove PUBO problem

```python
# Import time module
from time import perf_counter
# Import sympy module for syntax calculation
from sympy import symbols
# Import paddle module
from paddle import seed, optimizer
# Import pubo module
from paddle_quantum.mbqc.QAOA.pubo import dict_to_symbol,is_poly_valid,brute_force_search
# Import qaoa module
from paddle_quantum.mbqc.QAOA.qaoa import MBQC_QAOA_Net, get_solution_string
```

We define a function ``mbqc_pubo`` which takes in the objective polynomial and returns an optimal solution. **The core part of ``mbqc_pubo`` is the ``MBQC_QAOA_Net`` class**, which integrates MB-QAOA and the optimization net. Please refer to [Measurement-Based Quantum Approximate Optimization Algorithm](QAOA_EN.ipynb) for more details. Here we directly call the function.

```python
# Define the PUBO main function
def mbqc_pubo(OBJ_POLY, DEPTH, SEED, LR, ITR, EPOCH, SHOTS=1024):

    # Symbolize the polynomial
    obj_poly = dict_to_symbol(OBJ_POLY)
    var_num, poly_symbol = obj_poly

    # Print the QAOA depth
    print("QAOA depth is:", DEPTH)

    #  Start timing
    start_time = perf_counter()

    # Instaniate a MB-QAOA traning net
    seed(SEED)
    mbqc_net = MBQC_QAOA_Net(DEPTH)

    # Choose Adams optimizer (or SGD optimizer)
    opt = optimizer.Adam(learning_rate=LR, parameters=mbqc_net.parameters())

    # Start training
    for epoch in range(EPOCH):
        # Update parameters for each iter
        for itr in range(1, ITR + 1):
            # Train with mbqc_net and return the loss
            loss, state_out = mbqc_net(poly=obj_poly)
            # Propagate loss backwards and optimize the parameters
            loss.backward()
            opt.minimize(loss)
            opt.clear_grad()
            if itr % 10 == 0:
                print("iter:", itr, "  loss_MBQC:", "%.4f" % loss.numpy())

    # Stop timing and print the running time
    end_time = perf_counter()
    print("MBQC running time is: ", end_time - start_time)

    # Print the optimization parameters
    print("Optimal parameter gamma: ", mbqc_net.gamma.numpy())
    print("Optimal parameter beta: ", mbqc_net.beta.numpy())

    # Decode the solution from the quantum state
    solution_str = get_solution_string(state_out, SHOTS)

    # Evaluate the corresponding value
    relation = {symbols('x_' + str(j + 1)): int(solution_str[j]) for j in range(var_num)}
    value = poly_symbol.evalf(subs=relation)

    # Return the solution and its corresponding value
    opt = [solution_str, value]

    return opt
```

To check the correctness of the training result, we provide a `brute_force_search` function in `pubo` that finds a global optimal value by brute force search. We can compare the training result with the optimal one.

```python
from paddle_quantum.mbqc.QAOA.pubo import brute_force_search
```

### Main function

After defining the main function, let's input the parameters to run the code!

```python
# Define the main function
def main():

    # Choose the example x_1 + x_2 - x_3 + x_1*x_2 -x_1*x_2*x_3 + 0.5
    var_num = 3
    poly_dict = {'x_1': 1, 'x_2': 1, 'x_3': -1, 'x_1,x_2': 1, 'x_1,x_2,x_3': -1, 'cons':0.5}
    polynomial = [var_num, poly_dict]

    # Print the input polynomial
    print("The input polynomial is: ", polynomial)

    # We can also randomly generate an objective function
    # polynomial = random_poly(var_num)

    # Check the validity of the input polynomial
    is_poly_valid(polynomial)

    # Starting training and obtain the result
    mbqc_result = mbqc_pubo(
        OBJ_POLY=polynomial,  # Objective Function
        DEPTH=6,  # QAOA Depth
        SEED=1024,  # Plant Seed
        LR=0.1,  # Learning Rate
        ITR=120,  # Training Iters
        EPOCH=1  # Epoch Times
    )

    # Print the result from MBQC model
    print("Optimal solution by MBQC: ", mbqc_result[0])
    print("Optimal value by MBQC: ", mbqc_result[1])

    # Compute the optimal result by brute-force search and print the result
    brute_result = brute_force_search(polynomial)
    print("Optimal solution by brute force search: ", brute_result[0])
    print("Optimal value by brute force search: ", brute_result[1])

    # Compare the training result with the optimal one
    print("Difference between optimal values from MBQC and brute force search: ", mbqc_result[1] - brute_result[1])

if __name__ == '__main__':
    main()
```

## Example: MaxCut

### Graph and cut

Maximum cut problem（MaxCut Problem）is a combinatorial optimization problem in graph theory, with plenty of applications in e.g. statistic physics and circuit design.

In graph theory, a graph is represented by $G = (V, E)$, where $V$ is a set of vertices and $E$ is a set of edges. For example, a square can be characterized by the graph $G = (V,E)$ with $V = [1,2,3,4]$ and $E = [(1,2),(2,3),(3,4),(1,4)]$.

For code implementation, we can use the `plot_graph` function in `maxcut` to plot a graph.

```python
from paddle_quantum.mbqc.QAOA.maxcut import plot_graph
V = [1,2,3,4]
E = [(1,2),(2,3),(3,4),(1,4)]
G = [V, E]
plot_graph(G,"A square")
```

A cut in the graph is a partition separating the vertices set $V$ into two complementary subsets $S_0$ and $S_1$. If two vertices of an edge in the graph are separated into different subsets, we score a goal. The size of a cut is defined by the total scores that we get. Then the MaxCut problem is to find a cut of graph with maximal size.

As for the above square $G$, one of the optimal solutions to the MaxCut problem is to put $1$ and $3$ into subset $S_0$ and put $2$ and $4$ into subset $S_1$.

### Transformation to a PUBO problem

A MaxCut problem can be transformed into a PUBO problem. Assume the graph to be cut $G = (V, E)$ has $n=|V|$ vertices and $m =|E|$ edges, we can transform the MaxCut problem into a PUBO problem of $n$ variables. Each variable $x_v$ corresponds to a vertex $v \in V$ in the graph $G$, with its domain $x_v \in \{0,1\}$ corresponding to its belonging to subset $S_0$ or subset $S_1$. So, each value of the string $x = \{x_1,\cdots,x_n\}$ corresponds to a cut. As a valid edge to score a goal is the one whose vertices $u$ and $v$ belong to different subsets, given a cut $x$, its size can be defined as:

$$
C(x) = \sum_{(u,v) \in E} (x_u \oplus x_v),\tag{3}
$$

where $\oplus$ represents XOR operation. Then the MaxCut problem is equivalent to solve the optimization $\underset{x}{\max} \ C(x)$. Since $C(x)$ can be written as a polynomial:

$$
C(x) = \sum_{(u, v) \in E} (x_u + x_v - 2 x_u x_v).\tag{4}
$$

this optimization is essentially a quadratic PUBO problem of $n$ variables. We hope to find an optimal solution $x^{*}$ maximizing the value of objective polynomial, that is,

$$
x^* = \underset{x}{\text{argmax}} \left( \sum_{(u, v) \in E} (x_u + x_v - 2 x_u x_v) \right).\tag{5}
$$

We provide a function `graph_to_poly` in `maxcut` which takes in the graph to be cut and returns the equivalent objective polynomial in PUBO.

```python
# Import maxcut module
from paddle_quantum.mbqc.QAOA.maxcut import graph_to_poly

# Input the vertices and edges
V = [1,2,3,4]
E = [(1,2),(2,3),(3,4),(1,4)]
# Construct the graph to be cut
G = [V, E]

# Transform the graph to the equivalent polynomial
poly = graph_to_poly(G)
print("The equivalent objective polynomial is：\n", poly)
```

### Code implementation to solve MaxCut problem

Once obtaining the objective polynomial, we can follow the same process as the previous example and solve the MaxCut problem as a special case of PUBO. The complete code implementation is as follows:

```python
# Import symbol calculaion module
from sympy import symbols
# Import paddle module
from paddle import seed, optimizer
# Import qaoa module
from paddle_quantum.mbqc.QAOA.qaoa import MBQC_QAOA_Net, get_solution_string
# Import maxcut module
from paddle_quantum.mbqc.QAOA.maxcut import plot_graph, graph_to_poly, plot_solution
```

We define the main function for MaxCut that takes in the graph to be cut and returns the optimal training results.

```python
# Define the MaxCut main function
def mbqc_maxcut(GRAPH, DEPTH, SEED, LR, ITR, EPOCH, SHOTS=1024):

    # Plot the graph to be cut
    plot_graph(graph=GRAPH, title="Graph to be cut")

    # Obtain the objective polynomial
    polynomial = graph_to_poly(GRAPH)
    print("Corresponding objective polynomial of the graph is:", polynomial[1])

    # Start timing
    start_time = perf_counter()

    # Instantiate a MB-QAOA training net
    seed(SEED)
    mbqc_net = MBQC_QAOA_Net(DEPTH)

    # Choose Adams optimizer (or SGD optimizer)
    opt = optimizer.Adam(learning_rate=LR, parameters=mbqc_net.parameters())

    # Start training
    for epoch in range(EPOCH):
        # Update parameters for each iter
        for itr in range(1, ITR + 1):
            # Train with mbqc_net and return the loss
            loss, state_out = mbqc_net(poly=polynomial)
            # Propagate loss backwards and optimize the parameters
            loss.backward()
            opt.minimize(loss)
            opt.clear_grad()
            if itr % 10 == 0:
                print("iter:", itr, "  loss_MBQC:", "%.4f" % loss.numpy())

    # Stop timing and print the running time
    end_time = perf_counter()
    print("MBQC running time: ", end_time - start_time)

    # Print the optimized parameters
    print("Optimal parameter gamma: ", mbqc_net.gamma.numpy())
    print("Optimal parameter beta: ", mbqc_net.beta.numpy())

    # Decode the MaxCut solution from the final state
    mbqc_solution = get_solution_string(state_out, SHOTS)
    # Plot the MaxCut solution
    plot_solution(GRAPH, mbqc_solution)

    # Evaluate the number of cuts
    var_num, poly_symbol = polynomial
    relation = {symbols('x_' + str(j + 1)): int(mbqc_solution[j]) for j in range(var_num)}

    mbqc_value = int(poly_symbol.evalf(subs=relation))
    mbqc_opt = [mbqc_solution, mbqc_value]

    return mbqc_opt
```

### Main function

After defining the main function, let's input the parameters to run the code!

```python
def main():
    #  A graph to be cut
    V = [1, 2, 3, 4]
    E = [(1, 2), (2, 3), (3, 4), (4, 1)]
    G = [V, E]

    # MaxCut under MBQC
    mbqc_result = mbqc_maxcut(
        GRAPH=G,  # Graph to be cut
        DEPTH=6,  # Depth
        SEED=1024,  # Plant Seed
        LR=0.1,  # Learning Rate
        ITR=120,  # Training Iters
        EPOCH=1,  # Epoch Times
        SHOTS=1024  # Shots for decoding the solution
    )

    # Print the result from MBQC model
    print("Optimal solution by MBQC: ", mbqc_result[0])
    print("Optimal value by MBQC: ", mbqc_result[1])

if __name__ == '__main__':
    main()
```

Now, we have completed the demonstration of two examples. The implementation of MB-QAOA indicates a great potential of MBQC in the field of quantum machine learning. Apparently, MBQC model can realize quantities of algorithms far beyond QAOA. We therefore are
looking forward to exploring more on this and to show some unparalleled advantages in practice.

---

## References

[1] Farhi, Edward, et al. "A quantum approximate optimization algorithm." [arXiv preprint arXiv:1411.4028 (2014).](https://arxiv.org/abs/1411.4028)


# Assignment 3
*Name (ID)*: Vinh Nguyen (1029531)
*Workshop*: Friday 04:15 PM

---

## Setup

```
import sys
from pathlib import Path

# Notebook Library
url = "https://github.com/Data-Science-and-Data-Analytics-Courses/Notebook-Library"
repo = Path("/nblib")
!git clone "{url}" "{repo}"
if repo.parent.as_posix() not in sys.path:
  sys.path.append(repo.parent.as_posix())
%run "{repo}/.Importable.ipynb"

from nblib.imports.Basic import *
from nblib import Git

# Remote
URL = "https://github.com/Data-Science-and-Data-Analytics-Courses/UniMelb-A-First-Course-In-Statistical-Learning-MAST90104_2019_SM2"
REPO = Git.clone(URL, dest="/content")
if REPO.as_posix() not in sys.path:
  sys.path.append(REPO.as_posix())

# Working directory
os.chdir(REPO/"assignment3")

from nblib.imports.RPy import *
from setup.Setup import *

clear_output()
```

```r
%%R

library(ggplot2)
```

## Problem 1
Skipped

## Problem 2
You may not use theR glm command for this question. Fit a binomial regression model
to the O-rings data from the Challenger disaster, available in "orings.csv", using a complementary
log-log link.
Your solution should include the following:

(a) parameter estimates

Complementary log-log:
$$
\begin{align}
p = 1 - e^{-e^{\eta}}
\end{align}
$$

Log likelihood:
$$
\begin{align}
l(\theta) &= c + \sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_i}}) + (6-y_i) \text{log} \left(e^{-e^{\eta_i}}\right)\right) \\
&= c + \sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_i}}) - (6-y_i) e^{\eta_i}\right)\end{align}
$$

```r
%%R

# Data
df <- read.csv("orings.csv", header=TRUE)
print(head(df))
y <- df$orings.damage
x <- matrix(c(rep(1, length(df$orings.temp)), df$orings.temp), ncol=2)
print(head(x))

# cloglog link function
cloglog <- function(eta) {
    1 - exp(-exp(eta))
}

# Log-likelihood function
loglike <- function(beta, y, x) {
    eta <- x %*% beta
    return(sum(y*log(1 - exp(-exp(eta))) - (6-y)*exp(eta)))
}

print("Parameters")
bhat <- optim(c(10, -0.1), loglike, y = y, x = x,
                 control = list(fnscale = -1,reltol=1e-16))$par # fnscale= -1 -> maximize
print(bhat)
```

      X orings.temp orings.damage
    1 1          53             5
    2 2          57             1
    3 3          58             1
    4 4          63             1
    5 5          66             0
    6 6          67             0
         [,1] [,2]
    [1,]    1   53
    [2,]    1   57
    [3,]    1   58
    [4,]    1   63
    [5,]    1   66
    [6,]    1   67
    [1] "Parameters"
    [1] 10.8639010 -0.2055249

(b) 95% CIs for the parameter estimates

[Generalized Linear Models](https://www.uio.no/studier/emner/matnat/math/STK3100/h14/lectures/lecture3.pdf) (see Expected information matrix)

Score contribution:
$$
\begin{align}
s_{ij} &= \frac{∂l_i}{∂β_j} = \frac{∂l_i}{∂θ_i} \frac{∂θ_i}{∂\mu_i} \frac{∂\mu_i}{∂η_i} \frac{∂η_i}{∂β_j} \\
\end{align}
$$
where:
$$
\begin{align}
\frac{∂l_i}{∂θ_i} &= \frac{∂}{∂θ_i}\left[\frac{Y_i θ_i−b(θ_i)}{a(\phi_i)} + c(Y_i, \phi_i)\right] = \frac{Y_i − b'(θ_i)}{a(\phi_i)} = \frac{Y_i − \mu_i}{a(\phi_i)} & (\text{E}Y = b'(θ_i) = \mu_i) \\
\frac{∂θ_i}{∂\mu_i} &= \frac{1}{\frac{∂\mu_i}{∂θ_i}} = \frac{1}{\frac{∂b'(θ_i)}{∂θ_i}} = \frac{1}{b''(θ_i)} = \frac{1}{v(\mu_i)} \\
\frac{∂\mu_i}{∂η_i} &= \frac{1}{\frac{∂η_i}{∂\mu_i}} = \frac{1}{\frac{∂h(\mu_i)}{∂\mu_i}} = \frac{1}{h'(\mu_i)} & \left(η_i = g(p_i) = \text{ln}(−\text{ln}(1 − p_i))) = \text{ln}(−\text{ln}(1 − \frac{\mu_i}{6})) = h(\mu_i)\right) \\
\frac{∂η_i}{∂β_j} &= x_{ij} \\
\end{align}
$$
$$
\begin{align}
s_{ij} &= \frac{1}{a(\phi_i)} x_{ij} \frac{Y_i − \mu_i}{h'(\mu_i) v(\mu_i)} \\
\end{align}
$$

Score function:
$$
\begin{align}
s_{j} &= \frac{∂l}{∂β_j} = \sum_{i} \left(\frac{1}{a(\phi_i)} x_{ij} \frac{Y_i − \mu_i}{h'(\mu_i) v(\mu_i)}\right) \\
\end{align}
$$

From *'Week 9 Workshop - question 9'*, under regularity conditions, such that there is an interchangeability between integration and differentiation, the log likelihood function carries the following property (expectation w.r.t $Y$):
$$
\begin{align}
-\text{E} \frac{∂^2l}{∂β_j ∂β_k} &= \text{E}\left(\frac{∂l}{∂β_j} \frac{∂l}{∂β_k}\right) \\
&= \text{E}\left(\sum_{i} \left(\frac{1}{a(\phi_i)} x_{ij} \frac{Y_i − \mu_i}{h'(\mu_i) v(\mu_i)}\right) \sum_{i'} \left(\frac{1}{a(\phi_{i'})} x_{i'k} \frac{Y_{i'} − \mu_{i'}}{∂h'(\mu_{i'}) v(\mu_{i'})}\right)\right) \\
&= \text{E}\sum_{i} \sum_{i'} \left(\frac{1}{a(\phi_i) a(\phi_{i'})} x_{ij} x_{i'k} \frac{1}{h'(\mu_i) ∂h'(\mu_{i'}) v(\mu_i) v(\mu_{i'})} (Y_i − \mu_i) (Y_{i'} − \mu_{i'})\right) \\
&= \sum_{i} \sum_{i'} \left(\frac{1}{a(\phi_i) a(\phi_{i'})} x_{ij} x_{i'k} \frac{1}{h'(\mu_i) ∂h'(\mu_{i'}) v(\mu_i) v(\mu_{i'})} \text{E}\left[(Y_i − \mu_i) (Y_{i'} − \mu_{i'})\right]\right) \\
&= \sum_{i} \left(\frac{1}{a^2(\phi_i)} x_{ij} x_{ik} \frac{1}{h'^2(\mu_i) v^2(\mu_i)} \text{E}\left[(Y_i − \mu_i)^2\right]\right) & \left(\text{E}\left[(Y_i − \mu_i) (Y_{i'} − \mu_{i'})\right] = 0, \forall i \neq i'\right) \\
&= \sum_{i} \left(\frac{1}{a^2(\phi_i)} x_{ij} x_{ik} \frac{1}{h'^2(\mu_i) v^2(\mu_i)} \text{var}(Y_i)\right) \\
&= \sum_{i} \left(\frac{1}{a^2(\phi_i)} x_{ij} x_{ik} \frac{1}{h'^2(\mu_i) v^2(\mu_i)} v(\mu_i)a(\phi_i)\right) & (\text{var}(Y_i) = v(\mu_i)a(\phi_i)) \\
&= \sum_{i} \left(\frac{1}{a(\phi_i)} x_{ij} x_{ik} \frac{1}{h'^2(\mu_i) v(\mu_i)}\right) \\
\end{align}
$$
where:
$$
\begin{align}
a(\phi_i) &= 1 \\
h'(\mu_i) &= -\frac{1}{6(1-p_i) \text{ln}(1-p_i)} \\
v(\mu_i) &= 6p_i(1 − p_i) \\
\end{align}
$$

Fisher Information matrix:
$$
\begin{align}
I_{jk} &= -\text{E} \frac{∂^2l}{∂β_j ∂β_k} \\
&= \sum_{i} \left(x_{ij} x_{ik} \frac{1}{\left(\frac{1}{6(1-p_i) \text{ln}(1-p_i)}\right)^2 6p_i(1 − p_i)}\right) \\
&= \sum_{i} \left(x_{ij} x_{ik} \frac{\left(6(1-p_i) \text{ln}(1-p_i)\right)^2}{6p_i(1 − p_i)}\right) \\
&= \sum_{i} \left(x_{ij} x_{ik} \frac{6 (1-p_i) \text{ln}^2(1-p_i)}{p_i}\right) \\
&= \sum_{i} \left(6 x_{ij} x_{ik} \frac{(1-p_i) \text{ln}^2(1-p_i)}{p_i}\right) \\
\end{align}
$$

```r
%%R

# Fisher Information
ehat <- x %*% bhat
phat <- cloglog(ehat)
pterm <- (1-phat) * (log(1-phat))^2 / phat # term involving p
I11 <- 6*sum(x[, 1] * x[, 1] * pterm)
I12 <- 6*sum(x[, 1] * x[, 2] * pterm)
I22 <- 6*sum(x[, 2] * x[, 2] * pterm)
fisher <- matrix(c(I11, I12, I12, I22), nrow=2, byrow=TRUE)
fisher.inv <- solve(fisher)
print(fisher.inv)

conflev <- 0.95 # confidence level
print(sprintf("%s CIs for the parameter estimates", conflev))
print(bhat + rbind(c(-1, 1), c(-1, 1)) * qnorm(1-(1-conflev)/2) * sqrt(diag(fisher.inv)))
```

               [,1]         [,2]
    [1,]  7.4894280 -0.124053108
    [2,] -0.1240531  0.002080062
    [1] "0.95 CIs for the parameter estimates"
               [,1]       [,2]
    [1,]  5.5001030 16.2276990
    [2,] -0.2949143 -0.1161354

(c) a likelihood ratio test for the significance of the temperature coefficient

Hypotheses:
$$
\begin{align}
\text{H$_0$}: \beta_1 = 0 \\
\text{H$_1$}: \beta_1 \neq 0
\end{align}
$$

Models:
$$
\begin{align}
\text{Reduced} &: \beta_r = \begin{bmatrix}
\beta_{r0} \\
\end{bmatrix} \\
\text{Full} &: \beta = \begin{bmatrix}
\beta_0 \\
\beta_1 \\
\end{bmatrix} \\
\end{align}
$$

Likelihood ratio (subscript $r$ denotes reduced model):
$$
\begin{align}
\text{LR} &= -2 \text{ln} \frac{L_r(\hat{\beta}_{r})}{L(\hat{\beta})} & (\text{$L$ likelihood function}) \\
&= -2 \left(l_r(\hat{\beta}_{r}) - l(\hat{\beta})\right) & (\text{$l$ log-likelihood function}) \\
&= -2 \left(\left(c + \sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_{ri}}}) - (6-y_i) e^{\eta_{ri}}\right)\right) - \left(c + \sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_{i}}}) - (6-y_i) e^{\eta_{i}}\right)\right)\right) \\
&= -2 \left(\sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_{ri}}}) - (6-y_i) e^{\eta_{ri}}\right) - \sum_{i} \left(y_i \text{log}(1 - e^{-e^{\eta_{i}}}) - (6-y_i) e^{\eta_{i}}\right)\right) \\
&\sim χ^2_{2-1=1} \\
\end{align}
$$
where: $\eta_{ri} = x_{ri}^T \hat{\beta}_r, \quad \eta_{i} = x_i^T \hat{\beta}$

```r
%%R

print("Reduced parameters")
bhat.r <- optim(10, loglike, y = y, x = matrix(x[, 1]),
                control = list(fnscale = -1,reltol=1e-16))$par # fnscale= -1 -> maximize
print(bhat.r)

print("Likelihood ratio")
eta.r <- matrix(x[, 1]) %*% bhat.r # reduced
eta <- x %*% bhat # full
lr <- -2 * (sum(y*log(1 - exp(-exp(eta.r))) - (6-y)*exp(eta.r)) - sum(y*log(1 - exp(-exp(eta))) - (6-y)*exp(eta)))
print(lr)

print("p-value")
pval <- pchisq(lr, df=1, lower.tail=FALSE)
print(pval)

conflev <- 0.95 # confidence level
print(sprintf("Reject null at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    [1] "Reduced parameters"
    [1] -2.488098
    [1] "Likelihood ratio"
    [1] 22.86909
    [1] "p-value"
    [1] 1.734185e-06
    [1] "Reject null at 0.95 confidence level?"
    [1] TRUE

(d) an estimate of the probability of damage when the temperature equals 29 Fahrenheit together
with a 95% CI

```r
%%R

# log-log link inverse function
loglog.inv <- function(eta) {
    1 - exp(-exp(eta))
}

new <- c(1, 29) # new predictor
conflev <- 0.95 # confidence level
print("estimate of η")
enew <- new %*% bhat
print(enew)

print(sprintf("%s CI for η", conflev))
confint.eta <- enew + c(-1, 1) * qnorm(1-(1-conflev)/2) * sqrt(new %*% fisher.inv %*% new)
print(confint.eta)

print("estimate of probability")
probnew <- loglog.inv(enew)
print(probnew)

print(sprintf("%s CI for probability", conflev))
confint.prob <- loglog.inv(confint.eta)
print(confint.prob)
```

    [1] "estimate of η"
            [,1]
    [1,] 4.90368
    [1] "0.95 CI for η"
    [1] 2.101768 7.705592
    [1] "estimate of probability"
         [,1]
    [1,]    1
    [1] "0.95 CI for probability"
    [1] 0.99972 1.00000

(e) a plot comparing the fitted c-log-log model to the fitted logit model.

[Regression diagnostic plots](https://www.andrew.cmu.edu/user/achoulde/94842/homework/regression_diagnostics.html)
The graph below suggests that there is no significant difference between the two fitted models, the c-log-log and the logit.

```r
%%R

# Fit logit model
loglike.logit <- function(beta, y, x) {
    eta <- x %*% beta
    return(sum(y*eta - 6*log(1 + exp(eta))))
}
print("Parameters (logit model)")
bhat.logit <- optim(c(10, -0.1), loglike.logit, y = y, x = x,
                 control = list(fnscale = -1,reltol=1e-16))$par # fnscale= -1 -> maximize
print(bhat.logit)

# Logit link function
logit.inv <- function(eta) {
    1 / (1 + exp(-eta))
}

# Plot
xsorted <- x[order(x[, 2]),] # x in ascending order of temperature (column 2)
plot(y/6 ~ x[, 2], main = "Fitted c-log-log model vs. the fitted logit model",
     xlab="temperature", ylab="damage/6") # data points
lines(xsorted[, 2], loglog.inv(xsorted %*% bhat), col="red") # probabilities of cloglog model
lines(xsorted[, 2], logit.inv(xsorted %*% bhat.logit), col="blue") # probabilities of logit model
legend("topright", title="Probabilites (p)", legend=c("cloglog", "logit"), col=c("red", "blue"), lty=1)
```

## Problem 3
Suppose the Y comes from an exponential family with pdf or pmf f of the form
$$
\begin{align}
f(y; θ, φ) = \text{exp} \left[\frac{yθ − b(θ)}{a(φ)} + c(y, φ)\right] \\
\end{align}
$$

a) Show that EY = b'(θ)

Log likelihood function:
$$
\begin{align}
l(\theta, \phi; y) &= \text{ln} f(y; \theta, \phi) = \frac{yθ − b(θ)}{a(\phi)} + c(y, \phi)
\end{align}
$$

Under regularity conditions, such that there is an interchangeability between integration and differentiation, the log likelihood function carries the following property (expectation w.r.t $Y$):
$$
\begin{align}
\text{E} \frac{\partial l(\theta, \phi; Y)}{\partial \theta} &= 0 \\
\Rightarrow \text{E} \frac{\partial}{\partial \theta} \left(\frac{Yθ − b(θ)}{a(\phi)} + c(Y, \phi)\right) &= 0 \\
\text{E} \left(\frac{1}{a(\phi)} \left(Y − b'(θ)\right)\right) &= 0 \\
\frac{1}{a(\phi)} \left(\text{E}Y − b'(θ)\right) &= 0 \\
\text{E}Y − b'(θ) &= 0 \\
\text{E}Y &= b'(θ) \\
\end{align}
$$
which is as required.

(b) Show that Var Y = b''(θ)a(φ).

Second order derivative of log likelihood function:
$$
\begin{align}
\frac{\partial^2 l(\theta, \phi; y)}{\partial \theta^2} &= \frac{\partial^2}{\partial \theta^2} \left(\frac{yθ − b(θ)}{a(\phi)} + c(y, \phi)\right) \\
&= \frac{\partial}{\partial \theta} \left(\frac{1}{a(\phi)} \left(Y − b'(θ)\right)\right) \\
&= \frac{1}{a(\phi)} \frac{\partial}{\partial \theta} \left(Y − b'(θ)\right) \\
&= \frac{1}{a(\phi)} \left(− b''(θ)\right) \\
&= -\frac{b''(θ)}{a(\phi)} \\
\end{align}
$$

Again, under regularity conditions, the log likelihood function carries the following property (expectation w.r.t $Y$):
$$
\begin{align}
\text{E} \frac{\partial^2 l(\theta, \phi; Y)}{\partial \theta^2} &= -\text{E} \left[\left(\frac{\partial l(\theta, \phi; Y)}{\partial \theta}\right)^2\right] \\
&= -\text{var} \left(\frac{\partial l(\theta, \phi; Y)}{\partial \theta}\right) \\
&= -\text{var} \left(\frac{1}{a(\phi)} \left(Y − b'(θ)\right)\right) \\
&= - \frac{1}{a^2(\phi)} \text{var} \left(Y − b'(θ)\right) \\
&= - \frac{1}{a^2(\phi)} \text{var}(Y) \\
\end{align}
$$

Combining the above results:
$$
\begin{align}
\text{E} \frac{\partial^2 l(\theta, \phi; Y)}{\partial \theta^2} = \text{E} \left(-\frac{b''(θ)}{a(\phi)}\right) &= - \frac{1}{a^2(\phi)} \text{var}(Y) \\
-\frac{b''(θ)}{a(\phi)} &= -\frac{1}{a^2(\phi)} \text{var}(Y) \\
b''(θ) &= \frac{1}{a(\phi)} \text{var}(Y) \\
\text{var}(Y) &= b''(θ) a(\phi) \\
\end{align}
$$
which is as required.

## Problem 4
Steve Sahyun of the University of Winsconsin has a website http://sahyun.net/neutron.php in
which he explains neutron activation as a means of producing radioactive metals from standard
metals. It includes data on samples of Silver (Ag), Aluminium (Al) and Copper (Cu) which have
been subject to neutron activation. For each of the elements, two samples have been tested. The
data file **Radioactive.csv** contains radioactive counts at various times after the initial measurement (with the square of time also recorded) as well as the name of the element and whether the
count is for the first or second sample.

(a) Use qplot (in package ggplot2 ) to plot the log of counts versus time with different colours
for the different elements and different shapes for samples one and two. Your answer should
include the plot and your command to get it.

[Plotting in Color with qplot](https://www.theanalysisfactor.com/r-programming-plotting-color-part-2-qplot/)

```r
%%R

data <- read.csv("Radioactive.csv", header=TRUE)
print(head(data))
print(str(data))

# Plot
qplot(x=Time, y=log(Counts), data = data, main="Log of counts vs. time",
      geom = c("point", "line"), color = Material, shape = Sample) +
scale_shape_manual(values = c(1, 15)) +
scale_colour_manual(values = c("red","blue", "green"))
```

(b) Standard theory on radioactive emissions suggests that counts of them should be Poisson
distributed with a varying rate as the radioactivity decays. This suggests a Poisson model for
the counts with a log link. Comment on this in the light of the plot in (a). Fit the Poisson
model with the log the mean having a linear dependence on time and including factors for
both Material and Sample. Do diagnostic plots using plot for the model. Comment and re-fit,
if needed, omitting some observations. Use stepwise AIC selection to see if some variables can
be omitted. What does the final residual deviance indicate?

[Poisson Regression in R](https://www.dataquest.io/blog/tutorial-poisson-regression-in-r/)

*Comment on the use of Poisson model for the counts with a log link:*
The graph in (a) shows that log(Counts) and time do not necessarily have a linear relationship with each other (especially for Cu and Al). However, the data points for each material and each sample tend to follow a low-order polynomial curve (not extremely far away from a straight line). Thus, it is reasonable to fit the data using a Poisson model, where the logarithm of expected values (mean) of Counts can assume a linear form of some unknown parameters.

```r
%%R

# Fit model
model.pois <- glm(Counts ~ Time + Material + Sample, data=data, family = poisson(link = "log"))
print(summary(model.pois))

# Diagnostic plots
par(mfrow = c(2, 2))
plot(model.pois)
```

*Comment on fitted model*:
According to the diagnostic plots, the model does not seem to be a good fit for the data. For example, the 'Residuals vs Fitted' graph, with the smoothing red line not horizontal at 0, indicating that the residuals are not randomly scattered around 0. Additionally, the 'Residuals vs Leverage' graph shows a number of points with high Cook's distances (> 1) and high standardized residuals in magnitude (> 20), such as points 41, 42, 81. Since the presence of such outliers may have a significant effect on the model, one may try to improve the fitness of the model by removing these outliers.

```r
%%R

# Improve model (outliers removed)
model.impr <- glm(Counts ~ Time + Material + Sample, data=data, subset=-c(41, 42, 81), family = poisson(link = "log"))
print(summary(model.impr))

# Stepwise AIC selection
model.impr <- step(model.impr, scope=~., steps=1) # stop

# Assess fitness
print("p-value")
dev <- summary(model.impr)$deviance # residual deviance
df <- summary(model.impr)$df.residual # degrees of freedom
pval <- pchisq(dev, df, lower.tail = FALSE)
print(pval)

conflev <- 0.95
print(sprintf("Reject model at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    Call:
    glm(formula = Counts ~ Time + Material + Sample, family = poisson(link = "log"),
        data = data, subset = -c(41, 42, 81))

    Deviance Residuals:
        Min       1Q   Median       3Q      Max
    -13.099   -5.461   -2.070    2.971   26.817

    Coefficients:
                  Estimate Std. Error  z value Pr(>|z|)
    (Intercept)  7.0642234  0.0083516  845.857   <2e-16 ***
    Time        -0.1447225  0.0007509 -192.736   <2e-16 ***
    MaterialAl  -0.1052690  0.0108855   -9.671   <2e-16 ***
    MaterialCu   0.4495237  0.0091410   49.176   <2e-16 ***
    SampleTwo   -0.6427469  0.0081638  -78.732   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for poisson family taken to be 1)

        Null deviance: 69453  on 222  degrees of freedom
    Residual deviance: 12768  on 218  degrees of freedom
    AIC: 14303

    Number of Fisher Scoring iterations: 5

    Start:  AIC=14303.49
    Counts ~ Time + Material + Sample

               Df Deviance   AIC
    <none>           12768 14303
    - Material  2    16443 17975
    - Sample    1    19030 20564
    - Time      1    65968 67502
    [1] "p-value"
    [1] 0
    [1] "Reject model at 0.95 confidence level?"
    [1] TRUE

The re-fit model remains the same after stepwise AIC selection.

Residual deviance (ref. [Interpreting Residual and Null Deviance in GLM R](https://stats.stackexchange.com/a/113022)):
A low residual deviance implies that the model is a good fit, a high residual deviance indicates the otherwise. In this case, the residual deviance is quite high for a $\chi_{218}^2$. Our model, thus, cannot be considered to have well captured our data.

(c) Try re-fitting the model including TimeSquare as an extra variable to Time but still including
Material and Sample. Carry out the additional steps in part (b). Comment on the contrast
and comparison with (b).

```r
%%R

# Fit model with additional TimeSquare
model2.pois <- glm(Counts ~ Time + TimeSquare + Material + Sample, data=data, family = poisson(link = "log"))
print(summary(model2.pois))

# Diagnostic plots
par(mfrow = c(2, 2))
plot(model2.pois)

print("Contrasts")
print(contrasts(data$Material))
print(contrasts(data$Sample))
```

*Contrasts*:
[Using and interpreting different contrasts in linear models in R](https://biologyforfun.wordpress.com/2015/01/13/using-and-interpreting-different-contrasts-in-linear-models-in-r/)
MaterialAl (-1.721e-01): on average, estimated log(Counts) of Al is lower than that of Ag by 0.17.
MaterialCu (2.771e-01): on average, estimated log(Counts) of Cu is higher than that of Ag by 0.27.
SampleTwo (-5.231e-01): on average, estimated log(Counts) of Sample Two is lower than that of Sample One by 0.52.

*Comment on fitted model with additional TimeSquare*:
According to the diagnostic plots, there is evidence that the new model seems to be a good fit for the data (e.g, there is no obvious pattern in 'Residuals vs Fitted' graph). However, there is a few points with high Cook's distances and high standardized residuals (graph 'Residuals vs Leverage'), such as points 41, 42. Let us remove these outliers in an effort to improve the new model.

```r
%%R

# Improve model
model2.impr <- glm(Counts ~ Time + TimeSquare + Material + Sample, data=data, subset=-c(41, 42), family = poisson(link = "log"))
print(summary(model2.impr))

# Stepwise AIC selection
model2.impr <- step(model2.impr, scope=~., steps=1) # stop

# Assess fitness
print("p-value")
dev <- summary(model2.impr)$deviance # residual deviance
df <- summary(model2.impr)$df.residual # degrees of freedom
pval <- pchisq(dev, df, lower.tail = FALSE)
print(pval)

conflev <- 0.95
print(sprintf("Reject model at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    Call:
    glm(formula = Counts ~ Time + TimeSquare + Material + Sample,
        family = poisson(link = "log"), data = data, subset = -c(41,
            42))

    Deviance Residuals:
        Min       1Q   Median       3Q      Max
    -16.261   -4.752   -1.069    3.479   20.286

    Coefficients:
                  Estimate Std. Error  z value Pr(>|z|)
    (Intercept)  7.351e+00  8.693e-03  845.551  < 2e-16 ***
    Time        -2.405e-01  1.410e-03 -170.528  < 2e-16 ***
    TimeSquare   4.289e-03  4.915e-05   87.269  < 2e-16 ***
    MaterialAl  -8.113e-02  1.045e-02   -7.766 8.12e-15 ***
    MaterialCu   3.695e-01  9.297e-03   39.745  < 2e-16 ***
    SampleTwo   -6.036e-01  8.154e-03  -74.030  < 2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for poisson family taken to be 1)

        Null deviance: 73590.0  on 223  degrees of freedom
    Residual deviance:  7944.2  on 218  degrees of freedom
    AIC: 9491.5

    Number of Fisher Scoring iterations: 4

    Start:  AIC=9491.51
    Counts ~ Time + TimeSquare + Material + Sample

                 Df Deviance   AIC
    <none>              7944  9492
    - Material    2    10397 11940
    - TimeSquare  1    13299 14844
    - Sample      1    13486 15031
    - Time        1    31696 33242
    [1] "p-value"
    [1] 0
    [1] "Reject model at 0.95 confidence level?"
    [1] TRUE

The re-fit model with additional TimeSquare remains the same after stepwise AIC selection.

Residual deviance is significantly lower than the previous model (with time only) for the same $\chi_{218}^2$, although it is still considerably high. This model is yet to be a good fit for the data.

(d) An alternative approach would be to run the same analyses just using the data for one material
at a time. Do this. Comment on the comparison with (b) and (c).

```r
%%R

# Model with additional TimeSquare for Ag
modelAg <- glm(Counts ~ Time + TimeSquare + Sample, data=data, subset=Material=="Ag", family = poisson(link = "log"))
print(summary(modelAg))

# Diagnostic plots
par(mfrow = c(2, 2))
plot(modelAg)
```

*Comment on fitted model for Ag*:
The diagnostic plots show that the fitted model for Ag is a relatively good fit for the data (e.g, not only there is not obvious pattern in residuals but also no outrageous outliers either. In this case, it is not necessary to remove outliers from the dataset.

```r
%%R

# Stepwise AIC selection
modelAg.impr <- step(modelAg, scope=~., steps=1) # dropped Sample
modelAg.impr <- step(modelAg.impr, scope=~. + Sample, steps=1) # stop
print(summary(modelAg.impr))

# Assess fitness
print("p-value")
dev <- summary(modelAg.impr)$deviance # residual deviance
df <- summary(modelAg.impr)$df.residual # degrees of freedom
pval <- pchisq(dev, df, lower.tail = FALSE)
print(pval)

conflev <- 0.95
print(sprintf("Reject model at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    Start:  AIC=615.83
    Counts ~ Time + TimeSquare + Sample

                 Df Deviance    AIC
    - Sample      1     74.5  613.9
    <none>              74.4  615.8
    - TimeSquare  1    245.1  784.5
    - Time        1   5000.8 5540.2

    Step:  AIC=613.86
    Counts ~ Time + TimeSquare

    Start:  AIC=613.86
    Counts ~ Time + TimeSquare

                 Df Deviance    AIC
    <none>              74.5  613.9
    + Sample      1     74.4  615.8
    - TimeSquare  1    245.1  782.5
    - Time        1   5000.8 5538.2

    Call:
    glm(formula = Counts ~ Time + TimeSquare, family = poisson(link = "log"),
        data = data, subset = Material == "Ag")

    Deviance Residuals:
        Min       1Q   Median       3Q      Max
    -2.4101  -0.5840  -0.1237   0.5222   2.2347

    Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
    (Intercept)  7.5411682  0.0102361  736.73   <2e-16 ***
    Time        -0.3338025  0.0044822  -74.47   <2e-16 ***
    TimeSquare   0.0043694  0.0003232   13.52   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for poisson family taken to be 1)

        Null deviance: 39933.421  on 79  degrees of freedom
    Residual deviance:    74.457  on 77  degrees of freedom
    AIC: 613.86

    Number of Fisher Scoring iterations: 4

    [1] "p-value"
    [1] 0.5609214
    [1] "Reject model at 0.95 confidence level?"
    [1] FALSE

Although the stepwise AIC selection excluded 'SampleNumber' from the fitted model for Ag, the difference in AIC value is not significant (614 and 616). This is due to the similar radioactive counts that both samples of Ag have at various times (plot in (a) shows almost identical graphs for Ag). In general, however, this may not always be the case, that is, 'Sample' can still play an important role in distinguishing between different samples.

In comparison to the models in b) and c), the model for Ag has a much lower residual deviance. In fact, such residual deviance value does result in a high p-value, indicating that this model is a good fit for our data.

```r
%%R

# Model with additional TimeSquare for Al
modelAl <- glm(Counts ~ Time + TimeSquare + Sample, data=data, subset=Material=="Al", family = poisson(link = "log"))
print(summary(modelAl))

# Diagnostic plots
par(mfrow = c(2, 2))
plot(modelAl)
```

*Comment on fitted model for Al*:
The diagnostic plots show that the fitted model for Al does not seem to be a good fit for the data. Although the model exposes no outliers with Cook's distance higher than 1, it does suffer highly biased residuals.

```r
%%R

# Stepwise AIC selection
modelAl.impr <- step(modelAl, scope=~., steps=1) # stop
print(summary(modelAl.impr))

# Assess fitness
print("p-value")
dev <- summary(modelAl.impr)$deviance # residual deviance
df <- summary(modelAl.impr)$df.residual # degrees of freedom
pval <- pchisq(dev, df, lower.tail = FALSE)
print(pval)

conflev <- 0.95
print(sprintf("Reject model at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    Start:  AIC=581.56
    Counts ~ Time + TimeSquare + Sample

                 Df Deviance    AIC
    <none>             143.6  581.6
    - TimeSquare  1    750.1 1186.1
    - Sample      1   4190.7 4626.7
    - Time        1   4198.6 4634.6

    Call:
    glm(formula = Counts ~ Time + TimeSquare + Sample, family = poisson(link = "log"),
        data = data, subset = Material == "Al")

    Deviance Residuals:
        Min       1Q   Median       3Q      Max
    -2.8912  -1.4151  -0.5218   1.1357   3.0303

    Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
    (Intercept)  7.6064459  0.0142322  534.45   <2e-16 ***
    Time        -0.2914879  0.0043830  -66.50   <2e-16 ***
    TimeSquare   0.0061823  0.0002374   26.05   <2e-16 ***
    SampleTwo   -1.0218532  0.0162752  -62.79   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for poisson family taken to be 1)

        Null deviance: 21337.43  on 64  degrees of freedom
    Residual deviance:   143.55  on 61  degrees of freedom
    AIC: 581.56

    Number of Fisher Scoring iterations: 4

    [1] "p-value"
    [1] 1.316297e-08
    [1] "Reject model at 0.95 confidence level?"
    [1] TRUE

The model for Al remains the same after stepwise AIC selection.

In comparison to the models in b) amd c), the model for Al has a somewhat lower residual deviance. However, the p-value is still insignificant that this model cannot be regarded as a seemingly good fit.

```r
%%R

# Model with additional TimeSquare for Cu
modelCu <- glm(Counts ~ Time + TimeSquare + Sample, data=data, subset=Material=="Cu", family = poisson(link = "log"))
print(summary(modelCu))

# Diagnostic plots
par(mfrow = c(2, 2))
plot(modelCu)
```

*Comment on fitted model for Cu*:
The diagnostic plots show that the fitted model for Cu could be a good fit for the data, in terms of pattern-free residuals and no remarkable outliers.

```r
%%R

# Stepwise AIC selection
modelCu.impr <- step(modelCu, scope=~., steps=1) # stop
print(summary(modelCu.impr))

# Assess fitness
print("p-value")
dev <- summary(modelCu.impr)$deviance # residual deviance
df <- summary(modelCu.impr)$df.residual # degrees of freedom
pval <- pchisq(dev, df, lower.tail = FALSE)
print(pval)

conflev <- 0.95
print(sprintf("Reject model at %s confidence level?", conflev))
print(pval < 1 - conflev)
```

    Start:  AIC=720.35
    Counts ~ Time + TimeSquare + Sample

                 Df Deviance    AIC
    <none>             121.8  720.3
    - TimeSquare  1    808.1 1404.7
    - Sample      1   3424.3 4020.8
    - Time        1   4843.0 5439.5

    Call:
    glm(formula = Counts ~ Time + TimeSquare + Sample, family = poisson(link = "log"),
        data = data, subset = Material == "Cu")

    Deviance Residuals:
        Min       1Q   Median       3Q      Max
    -2.5090  -0.8127   0.1135   0.8406   2.6154

    Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
    (Intercept)  7.386e+00  1.235e-02  597.97   <2e-16 ***
    Time        -1.443e-01  2.021e-03  -71.38   <2e-16 ***
    TimeSquare   1.765e-03  6.418e-05   27.50   <2e-16 ***
    SampleTwo   -7.327e-01  1.272e-02  -57.61   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for poisson family taken to be 1)

        Null deviance: 18202.06  on 80  degrees of freedom
    Residual deviance:   121.83  on 77  degrees of freedom
    AIC: 720.35

    Number of Fisher Scoring iterations: 4

    [1] "p-value"
    [1] 0.0008630902
    [1] "Reject model at 0.95 confidence level?"
    [1] TRUE

The model for Cu remains the same after stepwise AIC selection.

In comparison to the models in b) amd c), the model for Cu has a lower residual deviance. Despite the improved residual deviance value, the model fails to stand against the fitness test due to an insignificant p-value.

In general, the alternative approach, in which the Material is considered separately, has proved to result in better fitness of the models, measured by lower residual deviance values. This improvement may be a reflection of the fact that different materials have different physical properties, thus react differently to radiation. Hence, it is plausible to build a distinct model for each material.


\title{Digital Arithmetic Cells with myHDL}
\author{Steven K Armour}
\maketitle

Table of Contents

1  Compression of Number System Values

2  Half Adder

3  The full adder

```python
from myhdl import *
from myhdlpeek import Peeker
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from sympy import *
init_printing()

import random

#https://github.com/jrjohansson/version_information
%load_ext version_information
%version_information myhdl, myhdlpeek, numpy, pandas, matplotlib, sympy, random
```

    The version_information extension is already loaded. To reload it, use:
      %reload_ext version_information

Software	Version

Python	3.6.4 64bit [GCC 7.2.0]

IPython	6.2.1

OS	Linux 4.13.0 45 generic x86_64 with debian stretch sid

myhdl	0.10

myhdlpeek	0.0.6

numpy	1.13.3

pandas	0.21.1

matplotlib	2.1.1

sympy	1.1.1

random	The 'random' distribution was not found and is required by the application

Fri Jun 29 23:35:01 2018 MDT

```python
#helper  functions to read in the .v and .vhd generated files into python
def VerilogTextReader(loc, printresult=True):
    with open(f'{loc}.v', 'r') as vText:
        VerilogText=vText.read()
    if printresult:
        print(f'***Verilog modual from {loc}.v***\n\n', VerilogText)
    return VerilogText

def VHDLTextReader(loc, printresult=True):
    with open(f'{loc}.vhd', 'r') as vText:
        VerilogText=vText.read()
    if printresult:
        print(f'***VHDL modual from {loc}.vhd***\n\n', VerilogText)
    return VerilogText
```

# Compression of Number System Values

```python
ConversionTable=pd.DataFrame()
ConversionTable['Decimal']=np.arange(0, 21)
ConversionTable['Binary']=[bin(i, 3) for i in np.arange(0, 21)]
ConversionTable['hex']=[hex(i) for i in np.arange(0, 21)]
ConversionTable['oct']=[oct(i) for i in np.arange(0, 21)]
ConversionTable
```

      Decimal
      Binary
      hex
      oct

      0
      0
      000
      0x0
      0o0

      1
      1
      001
      0x1
      0o1

      2
      2
      010
      0x2
      0o2

      3
      3
      011
      0x3
      0o3

      4
      4
      100
      0x4
      0o4

      5
      5
      101
      0x5
      0o5

      6
      6
      110
      0x6
      0o6

      7
      7
      111
      0x7
      0o7

      8
      8
      1000
      0x8
      0o10

      9
      9
      1001
      0x9
      0o11

      10
      10
      1010
      0xa
      0o12

      11
      11
      1011
      0xb
      0o13

      12
      12
      1100
      0xc
      0o14

      13
      13
      1101
      0xd
      0o15

      14
      14
      1110
      0xe
      0o16

      15
      15
      1111
      0xf
      0o17

      16
      16
      10000
      0x10
      0o20

      17
      17
      10001
      0x11
      0o21

      18
      18
      10010
      0x12
      0o22

      19
      19
      10011
      0x13
      0o23

      20
      20
      10100
      0x14
      0o24

```python
binarySum=lambda a, b, bits=2: np.binary_repr(a+b, bits)
```

```python
for i in [[0,0], [0,1], [1,0], [1,1]]:
    print(f'{i[0]} + {i[1]} yields  {binarySum(*i)}_2, {int(binarySum(*i), 2)}_10')
```

    0 + 0 yields  00_2, 0_10
    0 + 1 yields  01_2, 1_10
    1 + 0 yields  01_2, 1_10
    1 + 1 yields  10_2, 2_10

Notice that when we add $1_2+1_2$ we get instead of just a single bit we also get a new bit in the next binary column which we call the carry bit. This yields the following truth table for what is called the Two Bit or Half adder

# Half Adder

```python
TwoBitAdderTT=pd.DataFrame()
TwoBitAdderTT['x2']=[0,0,1,1]
TwoBitAdderTT['x1']=[0,1,0,1]
TwoBitAdderTT['Sum']=[0,1,1,0]
TwoBitAdderTT['Carry']=[0,0,0,1]
TwoBitAdderTT
```

      x2
      x1
      Sum
      Carry

      0
      0
      0
      0
      0

      1
      0
      1
      1
      0

      2
      1
      0
      1
      0

      3
      1
      1
      0
      1

Looking at this truth table we can surmise the following

```python
x1, x2=symbols('x_1, x_2')
Sum, Carry=symbols(r'\text{Sum}, \text{Carry}')
HASumDef=x1^x2; HACarryDef=x1 & x2
HASumEq=Eq(Sum, HASumDef); HACarryDef=Eq(Carry, HACarryDef)
HASumEq, HACarryDef
```

$$\left ( \text{Sum} = x_{1} \veebar x_{2}, \quad \text{Carry} = x_{1} \wedge x_{2}\right )$$

We can thus generate the following myHDL

```python
@block
def HalfAdder(x1, x2, Sum, Carry):
    """
    Half adder in myHDL
    I/O:
        x1 (bool): x1 input
        x2 (bool): x2 input
        Sum (bool): the sum Half Adder ouput
        Carry (bool): the carry Half Adder ouput
    """
    @always_comb
    def logic():
        Sum.next=x1^x2
        Carry.next=x1 & x2
    return logic
```

```python
Peeker.clear()
x1=Signal(bool(0)); Peeker(x1, 'x1')
x2=Signal(bool(0)); Peeker(x2, 'x2')
Sum=Signal(bool(0)); Peeker(Sum, 'Sum')
Carry=Signal(bool(0)); Peeker(Carry, 'Carry')

DUT=HalfAdder(x1, x2, Sum, Carry)

def HalfAdder_TB():
    """
    Half Adder Testbench for use in python only
    """
    @instance
    def Stimules():
        for _, row in TwoBitAdderTT.iterrows():
            x1.next=row['x1']
            x2.next=row['x2']
            yield delay(1)
        raise StopSimulation()

    return instances()

sim = Simulation(DUT, HalfAdder_TB(), *Peeker.instances()).run()
```

```python
Peeker.to_wavedrom('x2 x1 | Sum Carry', title='Half Adder Wave Form', tock=True)
```

```python
HARes=Peeker.to_dataframe()
HARes=HARes[['x2', 'x1','Sum', 'Carry']]
HARes
```

      x2
      x1
      Sum
      Carry

      0
      0
      0
      0
      0

      1
      0
      1
      1
      0

      2
      1
      0
      1
      0

      3
      1
      1
      0
      1

Wich, we can then confirm via these results to the expected results via

```python
TwoBitAdderTT==HARes
```

      x2
      x1
      Sum
      Carry

      0
      True
      True
      True
      True

      1
      True
      True
      True
      True

      2
      True
      True
      True
      True

      3
      True
      True
      True
      True

```python
DUT.convert()
VerilogTextReader('HalfAdder');
```

    ***Verilog modual from HalfAdder.v***

     // File: HalfAdder.v
    // Generated by MyHDL 0.10
    // Date: Fri Jun 29 23:13:48 2018

    `timescale 1ns/10ps

    module HalfAdder (
        x1,
        x2,
        Sum,
        Carry
    );
    // Half adder in myHDL
    // I/O:
    //     x1 (bool): x1 input
    //     x2 (bool): x2 input
    //     Sum (bool): the sum Half Adder ouput
    //     Carry (bool): the carry Half Adder ouput

    input x1;
    input x2;
    output Sum;
    wire Sum;
    output Carry;
    wire Carry;

    assign Sum = (x1 ^ x2);
    assign Carry = (x1 & x2);

    endmodule

**HalfAdder RTL**

**HalfAdder Synthesis**

```python
@block
def HalfAdder_TBV():
    """
    Half Adder Testbench for use in Verilog
    """
    x1 = Signal(bool(0))
    x2 = Signal(bool(0))
    Sum = Signal(bool(0))
    Carry = Signal(bool(0))

    DUT = HalfAdder(x1, x2, Sum, Carry)

    testx1=Signal(intbv(int("".join([str(i) for i in TwoBitAdderTT['x1'].tolist()]), 2))[4:])
    testx2=Signal(intbv(int("".join([str(i) for i in TwoBitAdderTT['x2'].tolist()]), 2))[4:])

    @instance
    def Stimulus():
        for i in range(len(testx1)):
            x1.next = testx1[i]
            x2.next = testx2[i]
            yield delay(1)

        raise StopSimulation()

    @always_comb
    def print_data():
        print(x1, x2, Sum, Carry)

    return instances()

# create instaince of TB
TB = HalfAdder_TBV()
# convert to verilog with reintilzed values
TB.convert(hdl="verilog", initial_values=True)
VerilogTextReader('HalfAdder_TBV');
```

    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    ***Verilog modual from HalfAdder_TBV.v***

     // File: HalfAdder_TBV.v
    // Generated by MyHDL 0.10
    // Date: Fri Jun 29 23:16:26 2018

    `timescale 1ns/10ps

    module HalfAdder_TBV (

    );
    // Half Adder Testbench for use in Verilog

    wire Sum;
    reg x1 = 0;
    reg x2 = 0;
    wire Carry;
    wire [3:0] testx1;
    wire [3:0] testx2;

    assign testx1 = 4'd5;
    assign testx2 = 4'd3;

    assign Sum = (x1 ^ x2);
    assign Carry = (x1 & x2);

    initial begin: HALFADDER_TBV_STIMULUS
        integer i;
        for (i=0; i<4; i=i+1) begin
            x1 <= testx1[i];
            x2 <= testx2[i];
            # 1;
        end
        $finish;
    end

    always @(Carry, x2, Sum, x1) begin: HALFADDER_TBV_PRINT_DATA
        $write("%h", x1);
        $write(" ");
        $write("%h", x2);
        $write(" ");
        $write("%h", Sum);
        $write(" ");
        $write("%h", Carry);
        $write("\n");
    end

    endmodule

    /home/iridium/anaconda3/lib/python3.6/site-packages/myhdl/conversion/_toVerilog.py:349: ToVerilogWarning: Signal is not driven: testx1
      category=ToVerilogWarning
    /home/iridium/anaconda3/lib/python3.6/site-packages/myhdl/conversion/_toVerilog.py:349: ToVerilogWarning: Signal is not driven: testx2
      category=ToVerilogWarning

Where the above warning can be ignored since we are using `testx1` and `testx2` as stores for the binay set of inputs to apply to `x1` and `x2`

# The full adder
If we assume that a carry is an input we can then extend the truth table for the two-bit adder to a three-bit adder yielding

```python
ThreeBitAdderTT=pd.DataFrame()
ThreeBitAdderTT['c1']=[0,0,0,0,1,1,1,1]
ThreeBitAdderTT['x2']=[0,0,1,1,0,0,1,1]
ThreeBitAdderTT['x1']=[0,1,0,1,0,1,0,1]
ThreeBitAdderTT['Sum']=[0,1,1,0,1,0,0,1]
ThreeBitAdderTT['Carry']=[0,0,0,1,0,1,1,1]
ThreeBitAdderTT
```

      c1
      x2
      x1
      Sum
      Carry

      0
      0
      0
      0
      0
      0

      1
      0
      0
      1
      1
      0

      2
      0
      1
      0
      1
      0

      3
      0
      1
      1
      0
      1

      4
      1
      0
      0
      1
      0

      5
      1
      0
      1
      0
      1

      6
      1
      1
      0
      0
      1

      7
      1
      1
      1
      1
      1

! need a way to make K-Maps in python

whereupon making and reducing the maps from the three-bit adder truth table we get the following equations

```python
c1, x1, x2=symbols('c_1,x_1, x_2')
Sum, Carry=symbols(r'\text{Sum}, \text{Carry}')
FASumDef=x1^x2^c1; FACarryDef=x1&x2 | x1&c1 | x2&c1
FASumEq=Eq(Sum, FASumDef); FACarryDef=Eq(Carry, FACarryDef)
FASumEq, FACarryDef
```

$$\left ( \text{Sum} = c_{1} \veebar x_{1} \veebar x_{2}, \quad \text{Carry} = \left(c_{1} \wedge x_{1}\right) \vee \left(c_{1} \wedge x_{2}\right) \vee \left(x_{1} \wedge x_{2}\right)\right )$$

yielding the following next myHDL module

```python
@block
def FullAdder(x1, x2, c1, Sum, Carry):
    """
    Full adder in myHDL
    I/O:
        x1 (bool): x1 input
        x2 (bool): x2 input
        c1 (bool): carry input
        Sum (bool): the sum Full Adder ouput
        Carry (bool): the carry Full Adder ouput

    Note!:
        There something wrong on the HDL side at the moment
    """

    @always_comb
    def logic():
        Sum.next=x1 ^ x2 ^c1
        Carry.next=(x1 & x2) | (x1 & c1) | (x2 & c1)
    return logic
```

```python
Peeker.clear()
x1=Signal(bool(0)); Peeker(x1, 'x1')
x2=Signal(bool(0)); Peeker(x2, 'x2')
c1=Signal(bool(0)); Peeker(c1, 'c1')
Sum=Signal(bool(0)); Peeker(Sum, 'Sum')
Carry=Signal(bool(0)); Peeker(Carry, 'Carry')

DUT=FullAdder(x1, x2, c1, Sum, Carry)

def FullAdder_TB():

    @instance
    def Stimules():
        for _, row in ThreeBitAdderTT.iterrows():
            x1.next=row['x1']
            x2.next=row['x2']
            c1.next=row['c1']
            yield delay(1)
        raise StopSimulation()

    return instances()

sim = Simulation(DUT, HalfAdder_TB(), *Peeker.instances()).run()
```

```python
Peeker.to_wavedrom('c1 x2 x1 | Sum Carry', title='Full Adder Wave Form', tock=True)
```

```python
FARes=Peeker.to_dataframe()
FARes=FARes[['c1, x2','x1','Sum', 'Carry']]
FARes
```

```python
ThreeBitAdderTT==FARes
```

```python
DUT.convert()
VerilogTextReader('FullAdder');
```

    ***Verilog modual from FullAdder.v***

     // File: FullAdder.v
    // Generated by MyHDL 0.10
    // Date: Fri Jun 29 23:15:35 2018

    `timescale 1ns/10ps

    module FullAdder (
        x1,
        x2,
        c1,
        Sum,
        Carry
    );
    // Full adder in myHDL
    // I/O:
    //     x1 (bool): x1 input
    //     x2 (bool): x2 input
    //     c1 (bool): carry input
    //     Sum (bool): the sum Full Adder ouput
    //     Carry (bool): the carry Full Adder ouput

    input x1;
    input x2;
    input c1;
    output Sum;
    wire Sum;
    output Carry;
    wire Carry;

    assign Sum = ((x1 ^ x2) ^ c1);
    assign Carry = (((x1 & x2) | (x1 & c1)) | (x2 & c1));

    endmodule

**FullAdder RTL**

**FullAdder Synthesis**

```python
@block
def FullAdder_TBV():
    """
    Full Adder Testbench for use in Verilog
    """
    x1=Signal(bool(0))
    x2=Signal(bool(0))
    c1=Signal(bool(0))
    Sum=Signal(bool(0))
    Carry=Signal(bool(0))

    DUT = FullAdder(x1, x2, c1, Sum, Carry)

    testx1=Signal(intbv(int("".join([str(i) for i in ThreeBitAdderTT['x1'].tolist()]), 2))[4:])
    testx2=Signal(intbv(int("".join([str(i) for i in ThreeBitAdderTT['x2'].tolist()]), 2))[4:])
    testc1=Signal(intbv(int("".join([str(i) for i in ThreeBitAdderTT['c1'].tolist()]), 2))[4:])

    @instance
    def Stimulus():
        for i in range(len(testx1)):
            x1.next=testx1[i]
            x2.next=testx2[i]
            c1.next=testc1[i]
            yield delay(1)

        raise StopSimulation()

    @always_comb
    def print_data():
        print(x1, x2, c1, Sum, Carry)

    return instances()

# create instaince of TB
TB = FullAdder_TBV()
# convert to verilog with reintilzed values
TB.convert(hdl="verilog", initial_values=True)
VerilogTextReader('FullAdder_TBV');
```

    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    <class 'myhdl._Signal._Signal'> <class '_ast.Name'>
    ***Verilog modual from FullAdder_TBV.v***

     // File: FullAdder_TBV.v
    // Generated by MyHDL 0.10
    // Date: Fri Jun 29 23:24:09 2018

    `timescale 1ns/10ps

    module FullAdder_TBV (

    );
    // Full Adder Testbench for use in Verilog

    wire Sum;
    reg x1 = 0;
    reg x2 = 0;
    wire Carry;
    reg c1 = 0;
    wire [3:0] testx1;
    wire [3:0] testx2;
    wire [3:0] testc1;

    assign testx1 = 4'd5;
    assign testx2 = 4'd3;
    assign testc1 = 4'd15;

    assign Sum = ((x1 ^ x2) ^ c1);
    assign Carry = (((x1 & x2) | (x1 & c1)) | (x2 & c1));

    initial begin: FULLADDER_TBV_STIMULUS
        integer i;
        for (i=0; i<4; i=i+1) begin
            x1 <= testx1[i];
            x2 <= testx2[i];
            c1 <= testc1[i];
            # 1;
        end
        $finish;
    end

    always @(Carry, x1, c1, x2, Sum) begin: FULLADDER_TBV_PRINT_DATA
        $write("%h", x1);
        $write(" ");
        $write("%h", x2);
        $write(" ");
        $write("%h", c1);
        $write(" ");
        $write("%h", Sum);
        $write(" ");
        $write("%h", Carry);
        $write("\n");
    end

    endmodule

    /home/iridium/anaconda3/lib/python3.6/site-packages/myhdl/conversion/_toVerilog.py:349: ToVerilogWarning: Signal is not driven: testx1
      category=ToVerilogWarning
    /home/iridium/anaconda3/lib/python3.6/site-packages/myhdl/conversion/_toVerilog.py:349: ToVerilogWarning: Signal is not driven: testx2
      category=ToVerilogWarning
    /home/iridium/anaconda3/lib/python3.6/site-packages/myhdl/conversion/_toVerilog.py:349: ToVerilogWarning: Signal is not driven: testc1
      category=ToVerilogWarning

Where the above warning can be ignored since we are using `testx1`, `testx2`, and `testc1` as stores for the binay set of inputs to apply to `x1`, `x2`, and `c1`

```python

```