# Fundamentals of Data Science
Winter Semester 2021

## Prof. Fabio Galasso, Guido D'Amely, Alessandro Flaborea, Luca Franco, Muhammad Rameez Ur Rahman and Alessio Sampieri
<galasso@di.uniroma1.it>, <damely@di.uniroma1.it>, <flaborea@di.uniroma1.it>, <franco@diag.uniroma1.it>, <rahman@di.uniroma1.it>, <alessiosampieri27@gmail.com>

## Exercise 2: Classification

In Exercise 2, you will re-derive and implement logistic regression and optimize the parameters with Gradient Descent and with the Newton's method. Also, in this exercise you will re-derive and implement Gassian Discriminant Analysis.
We will use datasets generated from the make_classification function from the SkLearn library. Its first output contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

The completed exercise should be handed in as a single notebook file. Use Markdown to provide equations. Use the code sections to provide your scripts and the corresponding plots.
Submit it by sending an email to galasso@di.uniroma1.it, flaborea@di.uniroma1.it, franco@diag.uniroma1.it and alessiosampieri27@gmail.com  by Wednesday November 17th 2021, 23:59.

## Notation

- $x^i$ is the $i^{th}$ feature vector
- $y^i$ is the expected outcome for the $i^{th}$ training example
- $m$ is the number of training examples
- $n$ is the number of features

Let's start by setting up our Python environment and importing the required libraries:

```python
%matplotlib inline
import numpy as np # imports a fast numerical programming library
import scipy as sp # imports stats functions, amongst other things
import matplotlib as mpl # this actually imports matplotlib
import matplotlib.cm as cm # allows us easy access to colormaps
import matplotlib.pyplot as plt # sets up plotting under plt
import pandas as pd # lets us handle data as dataframes
from sklearn.datasets import make_classification
import seaborn as sns

# sets up pandas table display
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

import seaborn as sns # sets up styles and gives us more plotting options
```

## Question 1: Logistic Regression with Gradient Ascent **(10 Points)**

### Code and Theory

#### Exercise 1.a **(3 Points)** Equations for the log likelihood, its gradient, and the gradient ascent update rule.

Write and simplify the likelihood $L(\theta)$ and log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall the probabilistic interpretation of the hypothesis $h_\theta(x)= P(y=1|x;\theta)$ and that $h_\theta(x)=\frac{1}{1+\exp(-\theta^T x)}$.

Also derive the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ and write the gradient update equation.

Question: Are we looking for a local minimum or a local maximum using the gradient ascent rule?

################# Do not write above this line #################

Your equations and answers here.

- $P(Y=1|x; \theta) = h_{\theta}(x)$

- $P(Y=0|x; \theta) = 1 - h_{\theta}(x)$

Given these two points, we can say that in general we have:

$P(y|x;\theta)=h_{\theta}(x)^{y} (1+h_{\theta}(x))^{1-y}$

We must observe that for $y=1$, we get only $h_{\theta}(x)^{y}$, instead for $y=0$, we just get $(1+h_{\theta}(x))^{1-y}$.

The Likelihood $L(\theta)$ is given by:

$$L(\theta) = P(\vec{y}|x; \theta)  \\
 = \prod_{i=1}^{m} P(y^{(i)}|x^{(i)};\theta)  \\
 = \prod_{i=1}^{m} h_{\theta}(x^{(i)})^{y^{(i)}}(1-h_{\theta}(x^{(i)}))^{1-y^{(i)}}  \\
 = \prod_{i=1}^{m} \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{y^{(i)}}\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{1-y^{(i)}}
$$

Instead, in order to find the **Log-likelihood $l(\theta)$**, we must apply the natural logarithm function to the likelihood function previously found, and we get:
$$
l(\theta) = \log L(\theta)
 = \sum_{i=1}^{m} y^{(i)}\log h_{\theta}(x^{(i)})+(1-y^{(i)})\log(1-h_{\theta}(x^{(i)}))  \\
 = \sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)
$$

We proceed deriving the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ as follows:
$$
\frac{\delta l(\theta)}{\delta \theta_j} = \frac{\delta}{\delta \theta_j} \Bigg(\sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg)\\
= \Bigg[y^{(i)}-\bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg]x^{(i)}
$$

The procedure of the _gradient ascent rule_ leads us to a local maximum of our (differentiable) function.

#### Exercise 1.b **(7 Points)** Implementation of logistic regression with Gradient Ascent

Code up the equations above to learn the logistic regression parameters. The dataset used here is created using the make_classification function present in the SkLearn library. $x^{(i)}_1$ and $x^{(i)}_2$ represent the two features for the $i$-th data sample $x^{(i)}$ and $y^{(i)}$ is its ground truth label.

```python
X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

Adding a column of 1's to $X$ to take into account the zero intercept

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
[x[:5,:],x[-5:,:]] # Plot the first and last 5 lines of x, now containing features x0 (constant=1), x1 and x2
```

    [array([[ 1.        ,  2.25698215, -1.34710915],
            [ 1.        ,  1.43699308,  1.28420453],
            [ 1.        ,  0.57927295,  0.23690172],
            [ 1.        ,  0.42538132, -0.24611145],
            [ 1.        ,  1.13485101, -0.61162683]]),
     array([[ 1.        ,  1.56638944,  0.81749944],
            [ 1.        , -1.94913831, -1.90601147],
            [ 1.        ,  1.53440506, -0.11687238],
            [ 1.        , -0.39243599,  1.39209018],
            [ 1.        , -0.11881249,  0.96973739]])]

```python
[y[:5],y[-5:]] # Plot the first and last 5 lines of y
```

    [array([1, 1, 1, 0, 1]), array([1, 0, 0, 0, 1])]

Define the sigmoid function "sigmoid", the function to compute the gradient of the log likelihood  "grad_l" and the gradient ascent algorithm.

################# Do not write above this line #################

```python
def sigmoid(x):
    '''
    Function to compute the sigmoid of a given input x.

    Input:
    x: it's the input data matrix. The shape is (N, H)

    Output:
    g: The sigmoid of the input x
    '''

    g =  1 / (1 + np.exp(-x))

    return g

def log_likelihood(theta,features,target):
    '''
    Function to compute the log likehood of theta according to data x and label y

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)
    target: the label array

    Output:
    log_g: the log likehood of theta according to data x and label y
    '''

    t = features.dot(theta)
    log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]

    return log_l

def predictions(features, theta):
    '''
    Function to compute the predictions for the input features

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)

    Output:
    preds: the predictions of the input features
    '''

    preds = sigmoid(features.dot(theta))

    return preds

def update_theta(theta, target, preds, features, lr):
    '''
    Function to compute the gradient of the log likelihood
    and then return the updated weights

    Input:
    theta: the model parameter matrix.
    target: the label array
    preds: the predictions of the input features
    features: it's the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the updated model parameter matrix.
    '''

    prediction = predictions(features, theta)
    theta = theta + ((lr/features.shape[0]) * (features.T).dot(target-prediction))
    return theta

def gradient_ascent(theta, features, target, lr, num_steps):
    '''
    Function to execute the gradient ascent algorithm

    Input:
    theta: the model parameter matrix.
    target: the label array
    num_steps: the number of iterations
    features: the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the final model parameter matrix.
    log_likelihood_history: the values of the log likelihood during the process
    '''

    log_likelihood_history = np.zeros(num_steps)

    m = len(target)

    for it in range(num_steps):

        prediction = np.dot(features,theta)

        theta = update_theta(theta, target, prediction, features, lr)

        log_likelihood_history[it]  = log_likelihood(theta,features,target)

    return theta, log_likelihood_history

```

################# Do not write below this line #################

Check your grad_l implementation:
grad_l applied to the theta_test (defined below) should provide a value for log_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = -1.630501731599431

output_test  = log_likelihood(np.array([-7,4,1]),x,y)
error_test=np.abs(output_test-target_value)

print("{:f}".format(error_test))
```

    0.000000

Let's now apply the function gradient_ascent and print the final theta as well as theta_history

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Gradient Ascent method
n_iter=1000
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=0.5,num_steps=n_iter)
print(theta_final)

```

    [-0.46097042  2.90036399  0.23146846]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

Plot the data and the decision boundary:

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(X[:,0].min(),X[:,1].max(),2)

# Plot raw data
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X)

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red")
plt.ylim(X[:,1].min()-1,X[:,1].max()+1)
# Save the theta_final value for later comparisons
theta_GA = theta_final.copy()
```

################# Do not write above this line #################

Discuss these two points:
1. You have implemented the gradient ascent rule. Could we have also used gradient descent instead for the proposed problem? Why/Why not?

- Since we know that we can derive the cost function from the log-likelihood simply multiplying it by $-1$:
$J(\theta)=-\frac{1}{m}L(\theta)$ ,
we could have turned a maximization problem into a minimization one, then using the gradient descend.

2. Let's deeply analyze how the learning rate $\alpha$ and the number of iterations affect the final results. Run the algorithm you have written for different values of $\alpha$ and the number of iterations and look at the outputs you get. Is the decision boundary influenced by these parameters change? Why do you think these parameters are affecting/not affecting the results?

- We analyzed how the learning rate $\alpha$ and the number of iterations affect the final results:

1) if we don't change lr's value but we consider a lower number of iterations, theta_final doesn't change considerably;

2) the same thing happens if we take the same number of iterations but we increase the learning rate $\alpha$ or we increase both the learning rate and the number of iterations;

3) a little difference in our results appears if we decrease at the same time both the learning rate and the number of iterations, but still not so considerably;

4) we notice an abnormal behaviour if we take a very large value of $\alpha$ (with the same number of iterations).

- We can justify our results because generally, a large value of the learning rate makes the learning faster, at the cost of a less optimal result, instead a smaller learning rate may allow the model to learn a more optimal or even globally optimal result but may take significantly longer time to execute.

```python
#EXAMPLE with a too big value of alpha (alpha = 300)
# that gives us a wrong result

# Run Gradient Ascent method
n_iter=1500
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=300,num_steps=n_iter)
print(theta_final)
```

    [-45.43579893 158.26593918  23.81026057]

    <ipython-input-10-bfb1284f7296>:30: RuntimeWarning: divide by zero encountered in log
      log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]
    <ipython-input-10-bfb1284f7296>:30: RuntimeWarning: invalid value encountered in multiply
      log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]
    <ipython-input-10-bfb1284f7296>:12: RuntimeWarning: overflow encountered in exp
      g =  1 / (1 + np.exp(-x))

################# Do not write below this line #################

## Question 2: Logistic Regression with non linear boundaries (7 points)

#### Exercise 2.a **(4 Points)** Polynomial features for logistic regression

Define new features, e.g. of 2nd and 3rd degrees, and learn a logistic regression classifier by using the new features, by using the gradient ascent optimization algorithm you defined in Question 1.

In particular, we would consider a polynomial boundary with equation:

$f(x_1, x_2) = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_1^2 + c_4 x_2^2 + c_5 x_1 x_2 + c_6 x_1^3 + c_7 x_2^3 + c_8 x_1^2 x_2 + c_9 x_1 x_2^2$

We would therefore compute 7 new features: 3 new ones for the quadratic terms and 4 new ones for the cubic terms.

Create new arrays by stacking x and the new 7 features (in the order x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2). In particular create x_new_quad by additionally stacking with x the quadratic features, and x_new_cubic by additionally stacking with x the quadratic and the cubic features.

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

```python
# First extract features x1 and x2 from x and reshape them to x1 vector arrays
x1 = x[:,1]
x2 = x[:,2]
x1 = x1.reshape(x1.shape[0], 1)
x2 = x2.reshape(x2.shape[0], 1)
print(x[:5,:]) # For visualization of the first 5 values
print(x1[:5,:]) # For visualization of the first 5 values
print(x2[:5,:]) # For visualization of the first 5 values
```

    [[ 1.          2.25698215 -1.34710915]
     [ 1.          1.43699308  1.28420453]
     [ 1.          0.57927295  0.23690172]
     [ 1.          0.42538132 -0.24611145]
     [ 1.          1.13485101 -0.61162683]]
    [[2.25698215]
     [1.43699308]
     [0.57927295]
     [0.42538132]
     [1.13485101]]
    [[-1.34710915]
     [ 1.28420453]
     [ 0.23690172]
     [-0.24611145]
     [-0.61162683]]

################# Do not write above this line #################

Your code here

```python
def is_in(label, cur_labels):
    na, nb = label.count("a"), label.count("b")
    for l in cur_labels:
        na_l, nb_l = l.count("a"), l.count("b")
        if na_l == na and nb_l == nb:
            return True
    return False

def new_features(x, degree=2):
    '''
    Function to create n-degree features from the input

    Input:
    x: the initial features
    degree: the maximum degree you wantthe features

    Output:
    features: the final features.
              2nd degree features must have the order [x, x1x1, x1x2, x2x2]
              2nd degree features must have the order [x, x1x1, x1x2, x2x2, x1x1x1, x1x1x2, x1x2x2, x2x2x2]
    '''

    #features = np.ones(x[:,1].shape[0])

     # 3nd degree features must have the order [x, x1x1, x1x2, x2x2, x1x1x1, x1x1x2, x1x2x2, x2x2x2]
    features = []
    # initialize grades to degree 0
    for triple in x:
        c_features = []
        a, b = triple[1], triple[2]
        cur_factors, cur_labels = [a, b], ["a", "b"]
        for degrees in range(2, degree + 1):
            # factors list
            new_factors, new_labels = [], []
            for index in range(len(cur_factors)):
                r_a, ra_label = a * cur_factors[index], "a" + cur_labels[index]
                r_b, rb_label = b * cur_factors[index], "b" + cur_labels[index]
                if not is_in(ra_label, new_labels):
                    new_factors.append(r_a)
                    new_labels.append(ra_label)
                if not is_in(rb_label, new_labels):
                    new_factors.append(r_b)
                    new_labels.append(rb_label)
            c_features += new_factors
            cur_factors, cur_labels = new_factors, new_labels
        features.append([*triple, *c_features])
    features = np.array(features, np.float64)

    return features

```

################# Do not write below this line #################

```python
x_new_quad = new_features(x, degree=2)
x_new_cubic = new_features(x, degree=3)

#reordering output features
temp = np.copy(x_new_quad[:, -1])
x_new_quad[:, -1] = x_new_quad[:, -2]
x_new_quad[:, -2] = temp

temp = np.copy(x_new_cubic[:, -1])
x_new_cubic[:, -1] = x_new_cubic[:, -2]
x_new_cubic[:, -2] = x_new_cubic[:, -3]
x_new_cubic[:, -3] = temp

```

Now use the gradient ascent optimization algorithm to learn theta by maximizing the log-likelihood, both for the case of x_new_quad and x_new_cubic.

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

theta_final_quad, log_l_history_quad = gradient_ascent(theta0_quad,x_new_quad,y,lr=0.5,num_steps=n_iter)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, log_l_history_cubic = gradient_ascent(theta0_cubic,x_new_cubic,y,lr=0.5,num_steps=n_iter)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [ 0.07605952  3.33058375  0.27310376 -0.52398852 -0.34168459 -0.05180065]
    [ 0.84882395  2.50116735  1.74388569 -1.26222374 -0.34310518 -0.99094786
      0.37330203 -0.66243284  0.98447591  1.40258057]

```python
# Plot the log likelihood values in the optimization iterations, in one of the two cases.
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history_quad)),log_l_history_quad,'b.')
```

#### Exercise 2.b **(3 Points)** Plot the computed non-linear boundary and discuss the questions

First, define a boundary_function to compute the boundary equation for the input feature vectors $x_1$ and $x_2$, according to estimated parameters theta, both in the case of quadratic (theta_final_quad) and of quadratic and cubic features (theta_final_cubic). Refer for the equation to the introductory part of Question 2.

################# Do not write above this line #################

Your code here

```python
def boundary_function(x1_vec, x2_vec, theta_final):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)
    if len(theta_final) == 6:
        # boundary function value for features up to quadratic
        c_0, c_1, c_2, c_3, c_4, c_5 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec

    elif len(theta_final) == 10:
        # boundary function value for features up to cubic
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec + c_6*(x1_vec**3) + c_7*(x2_vec**3) + c_8*(x1_vec**2)*x2_vec + c_9*(x2_vec**2)*x1_vec

    else:
        raise("Number of Parameters is not correct")

    return x1_vec, x2_vec, f
```

################# Do not write below this line #################

Now plot the decision boundaries corresponding to the theta_final_quad and theta_final_cubic solutions.

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

#### Confusion Matrix

Here you can see the confusion matrices related to the three models you've implemented.

```python
from sklearn.metrics import confusion_matrix
```

```python
## logistic regression with linear buondary

z = np.dot(x,theta_final)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[226,  27],
           [ 31, 216]], dtype=int64)

```python
## logistic regression with non linear buondary - quadratic

z = np.dot(x_new_quad,theta_final_quad)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[220,  33],
           [ 15, 232]], dtype=int64)

```python
## logistic regression with non linear buondary - cubic

z = np.dot(x_new_cubic,theta_final_cubic)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[225,  28],
           [ 11, 236]], dtype=int64)

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
1. Look back at the plots you have generated. What can you say about the differences between the linear, quadratic, and cubic decision boundaries? Can you say if the model is improving in performances, increasing the degree of the polynomial? Do you think you can incur in underfitting increasing more and more the degree?

- The _linear decision boundary_ shown in the first plot is an example of underfitting, in fact the model is not capable of capturing the underlying structure of the data.
- In the same way we can say that the _cubic decision boundary_ overfits the data, in fact the model captures too scrupulously the behaviour of these particular observations. This means that it may fail to fit additional data or predict future observations reliably.
- The _quadratic decision boundary_ fits the data in the most balanced way.

- We can notice that, the more the degree grows, the more the decision boundaries fit the data. This leads us to an improvement if we move from the _linear function_ to the _quadratic_ one. The model does not improve in performances if we move from the _quadratic function_ to the _cubic_ one, instead.

2. Let's now delve into some quantitative analysis. The three tables you have generated represent the confusion matrix for the model you have implemented in the first two questions. What can you say about actual performances? Does the increase of the degree have a high effect on the results?

- As it is possible to see from the displayed results the parameters effectively increase even if slightly:
about 0.02 each time. It is possible to affirm that the increase of the degree actually impacts on all of the parameters: Accuracy, Precision, Recall which proportionally increase through the different cases.

| Confusion Matrix | Accuracy | Precision | Recall |
| --- | --- | --- | --- |
|[218,  35],[ 22, 225]| 0.886 | 0.865384 | 0.910931 |
|[225,  28],[ 11, 236]| 0.904 | 0.875471 | 0.939271 |
|[225,  28],[ 11, 236]| 0.922 | 0.893939 | 0.955465 |

################# Do not write below this line #################

## Question 3: Multinomial Classification (Softmax Regression) **(13 Points)**

### Code and Theory **(10 Points)**
### Report **(3 Points)**

#### Exercise 3.a **(4 Points)**

In the multinomial classification we generally have $K>2$ classes. So the label for the $i$-th sample $X_i$ is $y_i\in\{1,...,K\}$, where $i=1,...,N$. The output class for each sample is estimated by returning a score $s_i$ for each of the K classes. This results in a vector of scores of dimension K.
In this exercise we'll use the *Softmax Regression* model, which is the natural extension of *Logistic Regression* for the case of more than 2 classes. The score array is given by the linear model:

\begin{equation}
s_i =  X_i \theta
\end{equation}

Scores may be interpreted probabilistically, upon application of the function *softmax*. The position in the vector with the highest probability will be predicted as the output class. The probability of the class k for the $i$-th data sample is:

\begin{equation}
p_{ik} = \frac{\exp(X_i \theta_k)}{\sum_{j=1}^K(X_i \theta_j))}
\end{equation}

We will adopt the *Cross Entropy* loss and optimize the model via *Gradient Descent*.
In the first of this exercise we have to:
-    Write the equations of the Cross Entropy loss for the Softmax regression model;
-    Compute the equation for the gradient of the Cross Entropy loss for the model, in order to use it in the gradient descent algorithm.

#### A bit of notation

*  N: is the number of samples
*  K: is the number of classes
*  X: is the input dataset and it has shape (N, H) where H is the number of features
*  y: is the output array with the labels; it has shape (N, 1)
*  $\theta$: is the parameter matrix of the model; it has shape (H, K)

################# Do not write above this line #################

Your equations here.

\begin{equation}
L(\theta) = -\frac{1}{N} \sum_{i=1}^{N}\sum_{j=1}^{K} \textbf{1} \{y_{ij} = 1\} \log{\frac{\exp(X_i \theta_k)}{\sum_{j=1}^K\exp(X_i \theta_j)}}
\end{equation}

\begin{equation}
\nabla_{\theta_k} L(\theta) = -\frac{1}{N} \left[ X \left( y - \frac{\exp(X_i \theta_k)}{\sum_{j=1}^K\exp(X_i \theta_j)} \right) \right]
\end{equation}

################# Do not write below this line #################

#### Exercise 3.b **(4 Points)**

Now we will implement the code for the equations. Let's implement the functions:
-  softmax
-  CELoss
-  CELoss gradient
-  gradient descent

We generate a toy dataset with *sklearn* library. Do not change anything outside the parts provided of your own code (else the provided checkpoint will not work).

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X.shape, y.shape
```

    ((300, 7), (300,))

As a hint for the implementations of your functions: consider the labels $y$ as one-hot vector. This will allow matrix operations (element-wise multiplication and summation).

```python
import scipy
import numpy as np

def class2OneHot(vec):
    out_sparse = scipy.sparse.csr_matrix((np.ones(vec.shape[0]), (vec, np.array(range(vec.shape[0])))))
    out_onehot = np.array(out_sparse.todense()).T
    return out_onehot

y_onehot = class2OneHot(y)
```

Let's visualize the generated dataset. We use as visualizzation method the *Principal Component Analysis* (PCA). PCA summarize the high-dimensional feature vectors of each sample into 2  features, which we can illustrate with a 2D plot. Look at the following plot, the 3 generated classes do not seem separable.

```python
from sklearn.decomposition import PCA
import pandas as pd

pca = PCA(n_components=2)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data = principalComponents, columns = ['pc1', 'pc2'])
finalDf = pd.concat([principalDf, pd.DataFrame(y, columns = ['target'])], axis = 1)
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x='pc1', y='pc2', hue='target', data=finalDf);
```

################# Do not write above this line #################

```python
def softmax(theta, X):
    '''
    Function to compute associated probability for each sample and each class.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)

    Output:
    softmax: it's the matrix containing probability for each sample and each class. The shape is (N, K)
    '''

    # Create the matrix for the dot products between theta and X
    dot = np.zeros((len(X), len(theta[0])), dtype = float)

    # Iterate for each column of theta and for each row of X
    for k in range(len(theta[0])):
        for i in range(len(X)):
            # Compute the dot product
            dot[i][k] = np.dot(X[i], theta[:,k])

    # Subtract the maximum to normalize the results
    dot -= np.max(dot)

    # Compute the softmax matrix
    softmax = ( np.exp(dot).T / np.sum(np.exp(dot), axis = 1) ).T

    return softmax

def CELoss(theta, X, y_onehot):
    '''
    Function to compute softmax regression model and Cross Entropy loss.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    loss: The scalar that is the mean error for each sample.
    '''

    # Call the softmax function
    s = softmax(theta, X)

    # Compute loss
    l = []
    for k in range(len(s[0])):
        for i in range(len(s)):
            if y_onehot[i][k] == 1:
                l.append(-(y_onehot[i][k]) * np.log(s[i][k]))

    loss = np.mean(l)

    return loss

def CELoss_jacobian(theta, X, y_onehot):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    jacobian: A matrix with the partial derivatives of the loss. The shape is (H, K)
    '''

    # Call the softmax function
    s = softmax(theta, X)

    # Compute the jacobian matrix
    jacobian = (-1/len(X)) * (np.dot(X.T, (y_onehot - s)))

    return jacobian

def gradient_descent(theta, X, y_onehot, alpha=0.01, iterations=100):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)
    alpha: it's the learning rate, so it determines the speed of each step of the GD algorithm
    iterations: it's the total number of step the algorithm performs

    Output:
    theta: it's the updated matrix of the parameters after all the iterations of the optimization algorithm. The shape is (H, K)
    loss_history: it's an array with the computed loss after each iteration
    '''

    # We initialize an empty array to be filled with loss value after each iteration
    loss_history = np.zeros(iterations)

    # With a for loop we compute the steps of GD algo
    for it in range(iterations):
        theta = theta - alpha * CELoss_jacobian(theta, X, y_onehot)
        loss_history[it] = CELoss(theta, X, y_onehot)

    return theta, loss_history
```

################# Do not write below this line #################

```python
# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 1000
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.01, iterations=n_iter)
```

    Initial Loss with initialized theta is: 1.2646726695234938

```python
theta_final
```

    array([[ 0.36863817,  0.43665456,  0.44932945],
           [ 0.61032416,  0.46997052,  0.57767904],
           [ 0.44826049,  0.46877418,  1.10659579],
           [ 0.48684583,  0.15780697,  1.26364796],
           [ 0.70460402,  0.1696857 ,  0.21803725],
           [ 0.90348791,  0.89975282,  0.69354729],
           [ 0.99991163, -0.06843561,  1.21751273]])

```python
loss = CELoss(theta_final, X, y_onehot)
loss
```

    0.5852421930515843

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

#### Exercise 3.c **(2 Points)**

Let's now evaluate the goodness of the learnt based on accuracy:

\begin{equation}
Accuracy = \frac{Number\ of\ correct\ predictions}{Total\ number\ of\ predictions}
\end{equation}

Implement the compute_accuracy function. You may compare the accuracy achieved with learnt model Vs. a random model (random $\Theta$) or one based on $\Theta$'s filled with zeros.

################# Do not write above this line #################

```python
def compute_accuracy(theta, X, y):
    '''
    Function to compute accuracy metrics of the softmax regression model.

    Input:
    theta: it's the final parameter matrix. The one we learned after all the iterations of the GD algorithm. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y: it's the label array. The shape is (N, 1)

    Output:
    accuracy: Score of the accuracy.
    '''

    # Call the softmax function
    s = softmax(theta, X)

    # Get the predicted classes
    predict = np.argmax(s, axis = 1)

    # Get the number of correct predictions comparing them with the label array
    # Divide by the total number of predictions (num of rows of softmax matrix)
    accuracy = sum(predict == y)/(float(len(s)))

    return accuracy
```

################# Do not write below this line #################

```python
compute_accuracy(theta_final, X, y)
```

    0.7966666666666666

```python
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))
compute_accuracy(theta0, X, y)
```

    0.4766666666666667

```python
compute_accuracy(np.zeros((X.shape[1], len(np.unique(y)))), X, y)
```

    0.3333333333333333

### Report **(3 Points)**

Experiment with different values for the learning rate $\alpha$ and the number of iterations. Look how the loss plot changes the convergence rate and the resulting accuracy metric. Report also execution time of each run. For this last step you could you %%time at the beginning of the cell to display time needed for the algorithm.

```python
%%time

# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 100
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.001, iterations=n_iter)
```

    Initial Loss with initialized theta is: 1.748475150783264
    Wall time: 697 ms

**Write your Report here**

```python
# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))
```

Learning rate $\alpha = 0.01$

Iterations = $100$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.01, iterations = 100)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.01, 'Iter:', 100)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.01$

Iterations = $500$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.01, iterations = 500)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.01, 'Iter:', 500)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.01$

Iterations = $1,000$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.01, iterations = 1000)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.01, 'Iter:', 1000)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.005$

Iterations = $100$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.005, iterations = 100)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.005, 'Iter:', 100)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.005$

Iterations = $500$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.005, iterations = 500)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.005, 'Iter:', 500)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.005$

Iterations = $1,000$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.005, iterations = 1000)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.005, 'Iter:', 1000)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.001$

Iterations = $100$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.001, iterations = 100)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.001, 'Iter:', 100)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.001$

Iterations = $500$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.001, iterations = 500)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.001, 'Iter:', 500)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.001$

Iterations = $1,000$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.001, iterations = 1000)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.001, 'Iter:', 1000)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

Learning rate $\alpha = 0.001$

Iterations = $10,000$

```python
%%time

# Run Gradient descent method
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha = 0.001, iterations = 10000)

# Compute the accuracy
acc = compute_accuracy(theta_final, X, y)

print('LR:', 0.001, 'Iter:', 10000)
print('Accuracy:', acc)

# Loss plot
fig,ax = plt.subplots(num=2)
ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

| LR | Iter | Accuracy | Time |
|---|---|---|---|
| 0.01 | 100 |0.616  | 780 ms |
| 0.01 | 500 | 0.78 | 3.4 s |
| 0.01 |1,000 | 0.79 | 6.57 s |
| 0.005 | 100 | 0.553 | 767 ms |
| 0.005 | 500 | 0.736 | 3.47 s |
| 0.005 | 1,000 | 0.78 | 6.63 s |
| 0.001 | 100 | 0.456 | 679 ms |
| 0.001 | 500 | 0.553 | 3.33 s |
| 0.001 | 1,000 | 0.616 | 6.61 s |
| 0.001 | 10,000 | 0.79 | 1 min 10 s |

From data we can see that we get the best accuracy score ($0.79$) with $\alpha = 0.01$ e $1,000$ iterations.

We observe that with the same number of iterations, in terms of accuracy, we have better results with higher learning rates.
Infact, with lower learning rates we need more iterations to reach the minimum loss, as shown by the plots.
To confirm this hypothesis, we tried with $\alpha = 0.001$ and $10,000$ iterations and we get the same accuracy of $\alpha = 0.01$ e $1,000$ iterations. So, we can say that if we divide by $n$ our learning rate, to get the same accuracy level, we need to multiply our number of iterations by the same $n$.

In addition, it seems that with the same number of iterations, changing the learning rates doesn't affect so much the time needed for the execution of the algorithm. Obviously, the time depends mainly on the number of iterations and increase with it.

```python
# Set the learning rates
lr = ['0.01', '0.005', '0.001']

# Get the plot for each different number of iterations
iterations = [100, 500, 1000]
accuracy = np.asarray([[0.616, 0.553, 0.456],
                       [0.78, 0.736, 0.553],
                       [0.79, 0.78, 0.616]])

for i in range(len(iterations)):
    fig, ax = plt.subplots()
    b = ax.bar(lr, accuracy[i], width = 0.5)
    ax.set_ylim(0, 1)
    ax.set_ylabel('Accuracy')
    ax.set_xlabel('Learning Rates')
    ax.set_title(f'Iterations: {iterations[i]}')
    ax.grid(True, linewidth = 0.5)
    plt.show()
```

## Question 4: Multinomial Naive Bayes **(6 Points)**

### Code and Theory

The Naive Bayes classifier is a probabilistic machine learning model often used for classification tasks, e.g. document classification problems.

In the multinomial Naive Bayes classification you generally have $K>2$ classes, and the features are assumed to be generated from a multinomial distribution.

##### __*Example Data*__
General models consider input data as values. In the case of MultinomialNB, being used mainly in the field of document classification, these data consider how many features $X_i$ are present in the sample. Basically, it is a count of features within each document.

Taking into account $D=3$ documents and a vocabulary consisting of $N=4$ words, the data are considered as follows.

|  | $w_1$ | $w_2$ | $w_3$ | $w_4$ |
|---|---|---|---|---|
| $d_1$  | 3 | 0 | 1 | 1 |
| $d_2$ | 2 | 1 | 3 |0|
| $d_3$ | 2 | 2 | 0 |2|

By randomly generating the class to which each document belongs we have $y=[1,0,1]$

##### __*A bit of notation*__
- $Y =\{y_1, y_2, ... , y_{|Y|}\}$: set of classes

- $V =\{w_1, w_2, ... , w_{|V|}\}$: set of vocabulary

-  $D =\{d_1, d_2, ... , d_{|D|}\}$: set of documents

-  $N_{yi}$:  count of a specific word $w_i$ in each unique class, e.g. for $y=1$ you select $D_1$ and $D_3$, then for third column  you have $N_{y,3}=1$
-  $N_y$: total count of features for a specific class, e.g. for $y=1$ you sum all rows values which the correspondent label is 1, so $N_y=11$

-  $n$: total number of features (words in vocabulary)
-  $\alpha$: smoothing parameters

##### __*Task*__
Find the class $y$ to which the document is most likely to belong given the words $w$.
Use the Bayes formula and the posterior probability for it.

Bayes Formula:
\begin{equation}
P(A|B) = \frac{P(A)*P(B|A)}{P(B)}
\end{equation}

Where:
- P(A): Prior probability of A
- P(B): Prior probability of B
- P(B|A): Likelihood, multiplying posterior probability, that is multinomial Naive Bayes is:
\begin{equation}
P(B|A) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

**Reminder: do not change any part of this notebook outside the assigned work spaces**

#### Generate random dataset

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X = np.floor(X)-np.min(np.floor(X))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train.shape, y_train.shape
```

    ((240, 7), (240,))

#### Step0: $N_y$ and $N_{yi}$

```python
def feature_count(X, y, classes, n_classes, n_features):
        '''
        Function to compute the count of a specific word in each unique class and the total count.

        Input:
        X: it's the input data matrix.
        y: label array
        classes: unique values of y
        n_classes: number of classes
        n_features: it's the number of word in Vocabulary.

        Output:
        N_yi:   count of a specific word $w_i$ in each unique class
        N_y: total count of features for a specific class
        '''
        N_yi = np.zeros((n_classes, n_features)) # feature count
        N_y = np.zeros((n_classes)) # total count

        for i in range(len(y)):
            # Compute N_y counting the features for each specific class
            N_y[y[i]] += np.sum(X[i])

            # Compute N_yi adding counting the specific words in each class
            N_yi[y[i]] += (X[i])

        return N_yi, N_y
```

```python
n_samples_train, n_features = X_train.shape
classes = np.unique(y_train)
n_classes = 3
alpha = 0.1

N_yi, N_y = feature_count(X_train, y_train, classes, n_classes, n_features)
```

#### Step1: Prior Probability
The probability of a document being in a specific category from the given set of documents.

######################################

Your equations here:
\begin{equation}
P(y_j) = \frac{\sum_{i=1}^{n\_samples} \textbf{1} \{ y^{(i)}=1 \} } {n\_samples}
\end{equation}

######################################

```python
def prior_(X, y, n_classes, n_samples):
        """
        Calculates prior for each unique class in y.

        Input:
        X: it's the input data matrix.
        y: label array
        n_classes: number of classes
        n_samples: number of documents

        Output:
        P: prior probability for each class. Shape: (, n_classes)
        """
        classes = np.unique(y)
        P = np.zeros(n_classes)

        # Implement Prior Probability P(A)
        for j in classes:
            P[j] = np.count_nonzero(y == j)/(n_samples)

        return P
```

```python
prior_prob = prior_(X_train, y_train, n_classes, n_samples_train)
print(prior_prob)
```

    [0.3        0.34583333 0.35416667]

#### Step2
Posterior Probability: The conditional probability of a word occurring in a document given that the document belongs to a particular category.

\begin{equation}
P(w_i|y_j) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

Likelihood for a single document:

######################################

Your equations here:
\begin{equation}
P(w|y_j) = \prod_{i=1}^{n} \left( \frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features} \right)^{X_{doc,i}}
\end{equation}

######################################

```python
def posterior_(x_i, i, h, N_y, N_yi, n_features, alpha):
        """
        Calculates posterior probability. aka P(w_i|y_j) using equation in the notebook.

        Input:
        x_i: feature x_i
        i: feature index.
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        posterior: P(xi | y). Float.
        """

        # Implement Posterior Probability
        posterior = float(((N_yi[h][i] + alpha)/(N_y[h] + alpha * n_features))**x_i)

        return posterior

def likelihood_(x, h, N_y, N_yi, n_features, alpha):
        """
        Calculates Likelihood P(w|j_i).

        Input:
        x: a row of test data. Shape(n_features,)
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        likelihood: Float.
        """

        tmp = []
        for i in range(x.shape[0]):
            tmp.append(posterior_(x[i], i, h, N_y, N_yi, n_features, alpha))

        # Implement Likelihood
        likelihood = float(np.prod(tmp))

        return likelihood
```

```python
# Example of likelihood for first document
likelihood_(X_test[0], 0, N_y, N_yi, n_features, alpha)
```

    2.7754694679413126e-53

#### Step3
Joint Likelihood that, given the words, the documents belongs to specific class

######################################

Your equations here:
\begin{equation}
P(y_i|w) = P(y_i)*P(w|y_i)
\end{equation}

######################################

Finally, from the probability that the document is in that class given the words, take the argument correspond to max value.

\begin{equation}
y(D) = argmax_{y \in Y} \frac{P(y|w)}{\sum_{j}P(y_j|w)}
\end{equation}

```python
def joint_likelihood(X, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha):
        """
        Calculates the joint probability P(y_i|w) for each class and makes it probability.
        Then take the argmax.

        Input:
        X: test data
        prior_prob:
        classes:
        n_classes:
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        predicted_class: Predicted class of the documents. Int. Shape: (,#documents)
        """
        samples, features = X.shape
        predict_proba = np.zeros((samples,n_classes))

        # Calculate Joint Likelihood of each row for each class, then normalize in order to make them probabilities
        # Finally take the argmax to have the predicted class for each document

        for i in range(samples):
            for j in range(n_classes):
                l = likelihood_(X[i], j, N_y, N_yi, n_features, alpha)
                predict_proba[i][j] = (prior_prob[j]*l)

        predicted_class = []
        for i in range(samples):
            predicted_class.append(np.argmax(predict_proba[i]/np.sum(predict_proba[i])))

        return predicted_class
```

```python
yhat = joint_likelihood(X_test, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha)
```

#### Step4: Calculate the Accuracy Score

```python
print('Accuracy: ', np.round(accuracy_score(yhat, y_test),3))
```

    Accuracy:  0.717

**Sanity Check**

Here we use a function from the sklearn library, one of the most widely used in machine learning. MultinomialNB() implements the required algorithm, so the result of your implementation should be equal to the output of the following function.

```python
from sklearn import naive_bayes
clf = naive_bayes.MultinomialNB(alpha=0.1)
clf.fit(X_train,y_train)
sk_y = clf.predict(X_test)
print('Accuracy: ', np.round(accuracy_score(sk_y, y_test),3))
```

    Accuracy:  0.717


# Acoustic radiation force on two & three sound-hard spheres

### Arnold D. Kim
*Department of Applied Mathematics, University of California, Merced*

### Nicholas St. Clair
*Department of Physics, University of California, Merced*

```python
import  time
print( 'Last updated: %s' %time.strftime('%d/%m/%Y') )
```

    Last updated: 13/08/2021

```python
# defaults for the codes below

import numpy as np
import scipy.special as sp
import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D
```

## 1 Problem description

NEEDS UPDATING

We compute the solution of the scattering problem for the scalar potential, $\phi_{1}$:
\begin{gather*}
   \varDelta \phi_{1} + k^{2} \phi_{1} = 0, \quad \text{in $r > a$},\\
   \partial_{r} \phi_{1} = 0 \quad \text{on $r = a$}.
\end{gather*}
The incident field is given by a plane wave,
\begin{equation}
   \phi_{1}^{\text{inc}} = e^{\mathrm{i} k z}.
\end{equation}

We solve this scattering problem using two methods. The first uses the analytical representation as an expansion in Legendre polynomials. The second method uses the method of fundamental solutions (MFS). Through a comparison of these two results, we validate the use of the MFS to solve this problem.

Using the results from this scattering problem, we compute the acoustic radiation force defined according to (see Eq. (10) of H. Bruus, "Acoustofluidics 7: The acoustic radiation force on small particles," Lab Chip **12**, 1014 -- 1021 (2012))
\begin{equation}
   \vec{F} = -\int_{\partial \Omega} \left\{ \left[ \frac{1}{2} \kappa_{0} \langle p_{1}^{2} \rangle
   - \frac{1}{2} \rho_{0} \langle v_{1}^{2} \rangle \right] \hat{n} + \rho_{0} \langle ( \hat{n} \cdot \vec{v}_{1} )
   \vec{v}_{1} \rangle \right\} \mathrm{d}S.
\end{equation}
Here $p_{1}$ is the pressure field defined according to
\begin{equation}
   p_{1} = \mathrm{i} \rho_{0} \omega \phi_{1}
\end{equation}
$v_{1}$ is the velocity field defined according to
\begin{equation}
   v_{1} = \nabla \phi_{1},
\end{equation}
and the notation $\langle \cdot \rangle$ denotes a time-averaging operation.

Note that due to the sound-hard assumption, $\hat{n} \cdot \vec{v}_{1} = 0$ identically. Therefore, this expression for the force reduces to
\begin{equation}
   \vec{F} = -\int_{\partial \Omega} \left\{ \left[ \frac{1}{2} \kappa_{0} \langle p_{1}^{2} \rangle
   - \frac{1}{2} \rho_{0} \langle v_{1}^{2} \rangle \right] \hat{n} \right\} \mathrm{d}S.
\end{equation}

## 2 The Method of Fundamental Solutions (MFS)

Here, we compute the solution of the scattering problem using the Method of Fundamental Solutions (MFS).

### 2.1 Compute the MFS points

We compute the Fibonnaci lattice (see A. Gonzalez, "Measurement of areas using Fibonnaci and latitude-longitude lattices," Math. Geosci. **42**, 49 (2010)) and use the resulting points on the sphere for the MFS.

```python
def ComputeMFSReferencePoints( a, N ):
    """
    This function computes the set of points for a reference sphere that is to be used in the method of
    fundamental solutions (MFS).

    In particular, given the radius of a sphere, a, and the number of points N, this function computes
    randomly distributed points on the unit sphere and stores them as the unit normal vectors ν. Using
    ν, we then compute ρ_bdy = a * ν, and ρ_src = ( a - ℓ ) * ν.

    This function outputs three vectors: ν, ρ_bdy, and ρ_src.
    """
    # allocate memory for the Fibonacci lattice points on the unit sphere

    ν = np.full( ( N, 3 ), float( 'nan' ) )

    # compute the "golden angle"

    golden_angle = np.pi * ( 3.0 - np.sqrt( 5.0 ) )

    # compute the points on the unit sphere

    ν[:,2] = ( 1 - 1 / N ) * ( 1 - 2 * np.arange( 0, N ) / ( N - 1 ) )

    ρ = np.sqrt( 1.0 - ν[:,2] ** 2 )
    θ = golden_angle * np.arange( 0, N )

    ν[:,0] = ρ * np.cos( θ )
    ν[:,1] = ρ * np.sin( θ )

    # compute the boundary points, interior points, and exterior points

    ℓ = 0.4 * a

    ρ_bdy = a * ν
    ρ_src = ( a - ℓ ) * ν

    return ν, ρ_bdy, ρ_src;
```

It would be interesting to see if $l$ has any effect on the double body simulation.

### 2.2 Compute the fundamental solution and its normal derivative

The functions below compute Green's function in a medium with wavenumber $k$ and its normal derivative.

```python
# function to compute Green's function

def ComputeG( k, R ):
    """
    This function computes the whole space Green's function given a wavenumber k and a distance R.

    The output of this function is the evaluation of the whole space Green's function.
    """

    # compute Green's function

    G = np.exp( 1j * k * R ) / ( 4 * np.pi * R )

    return G;

# function to compute the normal derivative of Green's function

def ComputeDνG( k, R, CosΘ ):
    """
    This function computes the normal derivative of the whole space Green's function
    given a wavenumber k and distance R, and the cosine of the angle made between the
    difference vector and the unit normal.

    The output of this function is the evaluation of the normal derivative of the whole
    space Green's function.
    """

    # compute Green's function

    G = ComputeG( k, R )

    # compute the normal derivative of Green's function

    DνG = CosΘ * ( 1j * k - 1 / R ) * G

    return DνG;
```

### 2.3 Compute the MFS expansion coefficients

The function below constructs and solves the linear system for the MFS expansion coefficients. This linear system results from requiring that the sound-hard boundary condition on the two spheres is satisfied exactly on the Fibonnaci lattice points.

```python
def ComputeMFSExpansionCoefficients1Sphere( k, ν, ρ_bdy, ρ_sca, N ):
    """
    This function solves the linear system of equations for the MFS expansion coefficients.

    This code requires a wavenumber k, the results from ComputeMFSPoints, namely ν, ρ_bdy,
    and ρ_ext in addition to the number of MFS points, N, and the number of scattering
    objects, Nobjs.

    The output from this code are the N x Nobs matrix, c, corresponding to the MFS expansion
    coefficients for the scattered field.
    """

    # compute a mesh grid of indices to compute the matrices below

    indx, jndx = np.mgrid[ 0 : N , 0 : N ]

    # allocate memory for the linear system

    b = np.full( ( N, 1 ), 'nan', dtype = 'complex' )
    A = np.full( ( N, N ), 'nan', dtype = 'complex' )

    # compute the incident field (normal derivative of two, counter-propagating plane waves)

    b = -1j * k * ν[:,2] * np.exp( 1j * k * ρ_bdy[:,2] )

    # compute the distance from scatter points on object q to boundary points on object p

    R = np.sqrt( ( ( ρ_bdy[indx,:] - ρ_sca[jndx,:] ) ** 2 ).sum( axis = 2 ) )

    # compute the cosine of the angle made between the normal on object p and R

    CosΘ = ( ν[indx,:] * ( ρ_bdy[indx,:] - ρ_sca[jndx,:] ) ).sum( axis = 2 ) / R

    # compute the matrix sub-block

    A = ComputeDνG( k, R, CosΘ )

    # solve the linear system

    c = np.linalg.solve( A, b )

    return c;
```

```python
def ComputeMFSExpansionCoefficients2Spheres( k, δx, δy, ν, ρ_bdy, ρ_sca, N ):
    """
    This function solves the linear system of equations for the MFS expansion coefficients.

    This linear system is constructed by requiring that the MFS approximations exactly satisfy
    the interface conditions for each sphere on the collocation points.

    The result is a set of expansion coefficients for the fields scattered by each of the spheres.

    This method has been written out expressly for the purpose of ease of understanding, however writing out each coefficient
    by hand quickly becomes intractible for a larger number of particles. The MFS class on the github repo (__init.py__ document)
    is written to take care of any general number of particles.
    """
    # compute a mesh grid of indices to compute the matrices below

    indx, jndx = np.mgrid[ 0 : N , 0 : N ]

    # compute boundary and scattering points

    ρ_bdy1 = ρ_bdy
    ρ_sca1 = ρ_sca

    ρ_bdy2 = [ δx, δy, 0 ] + ρ_bdy
    ρ_sca2 = [ δx, δy, 0 ] + ρ_sca

    # allocate memory for the linear system

    #b1 = np.full( ( N, 1 ), 'nan', dtype = 'complex' )
    #b2 = np.full( ( N, 1 ), 'nan', dtype = 'complex' )

    A1 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A2 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A3 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A4 = np.full( ( N, N ), 'nan', dtype = 'complex' )

    # compute the incident field (normal derivative of a plane wave)

    b1 = -1j * k * ν[:,2] * np.exp( 1j * k * ρ_bdy1[:,2] )
    b2 = -1j * k * ν[:,2] * np.exp( 1j * k * ρ_bdy2[:,2] )

    # compute the distance from scatter points on object q to boundary points on object p

    R1 = np.sqrt( ( ( ρ_bdy1[indx,:] - ρ_sca1[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R2 = np.sqrt( ( ( ρ_bdy1[indx,:] - ρ_sca2[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R3 = np.sqrt( ( ( ρ_bdy2[indx,:] - ρ_sca1[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R4 = np.sqrt( ( ( ρ_bdy2[indx,:] - ρ_sca2[jndx,:] ) ** 2 ).sum( axis = 2 ) )

    # compute the cosine of the angle made between the normal on object p and R

    CosΘ1 = ( ν[indx,:] * ( ρ_bdy1[indx,:] - ρ_sca1[jndx,:] ) ).sum( axis = 2 ) / R1
    CosΘ2 = ( ν[indx,:] * ( ρ_bdy1[indx,:] - ρ_sca2[jndx,:] ) ).sum( axis = 2 ) / R2
    CosΘ3 = ( ν[indx,:] * ( ρ_bdy2[indx,:] - ρ_sca1[jndx,:] ) ).sum( axis = 2 ) / R3
    CosΘ4 = ( ν[indx,:] * ( ρ_bdy2[indx,:] - ρ_sca2[jndx,:] ) ).sum( axis = 2 ) / R4

    # compute the matrix sub-block

    A1 = ComputeDνG( k, R1, CosΘ1 )
    A2 = ComputeDνG( k, R2, CosΘ2 )
    A3 = ComputeDνG( k, R3, CosΘ3 )
    A4 = ComputeDνG( k, R4, CosΘ4 )

    # solve the linear system

    c = np.linalg.solve( np.block( [ [ A1, A2 ], [ A3, A4 ] ] ), np.block( [ b1, b2 ] ) )

    # parse the solution

    c1 = c[:N]
    c2 = c[N:]

    return c1, c2;
```

```python
def ComputeMFSExpansionCoefficients3Spheres( k, δ2, δ3, ν, ρ_bdy, ρ_sca, N ):
    """
    This function solves the linear system of equations for the MFS expansion coefficients.

    This linear system is constructed by requiring that the MFS approximations exactly satisfy
    the interface conditions for each sphere on the collocation points.

    The result is a set of expansion coefficients for the fields scattered by each of the spheres.
    This method has been written out expressly for the purpose of ease of understanding, however writing out each coefficient
    by hand quickly becomes intractible for a larger number of particles. The MFS class on the github repo (__init.py__ document)
    is written to take care of any general number of particles.
    """
    # compute a mesh grid of indices to compute the matrices below

    indx, jndx = np.mgrid[ 0 : N , 0 : N ]

    # compute boundary and scattering points

    ρ_bdy1 = ρ_bdy
    ρ_sca1 = ρ_sca

    ρ_bdy2 = δ2 + ρ_bdy
    ρ_sca2 = δ2 + ρ_sca

    ρ_bdy3 = δ3 + ρ_bdy
    ρ_sca3 = δ3 + ρ_sca

    # allocate memory for the linear system

    #b1 = np.full( ( N, 1 ), 'nan', dtype = 'complex' )
    #b2 = np.full( ( N, 1 ), 'nan', dtype = 'complex' )

    A1 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A2 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A3 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A4 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A5 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A6 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A7 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A8 = np.full( ( N, N ), 'nan', dtype = 'complex' )
    A9 = np.full( ( N, N ), 'nan', dtype = 'complex' )

    # compute the incident field (normal derivative of a plane wave)

    b1 = -1j * k * ν[:,2] *  np.exp( 1j * k * ρ_bdy1[:,2] )
    b2 = -1j * k * ν[:,2] *  np.exp( 1j * k * ρ_bdy2[:,2] )
    b3 = -1j * k * ν[:,2] *  np.exp( 1j * k * ρ_bdy3[:,2] )

    # compute the distance from scatter points on object q to boundary points on object p

    R1 = np.sqrt( ( ( ρ_bdy1[indx,:] - ρ_sca1[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R2 = np.sqrt( ( ( ρ_bdy1[indx,:] - ρ_sca2[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R3 = np.sqrt( ( ( ρ_bdy1[indx,:] - ρ_sca3[jndx,:] ) ** 2 ).sum( axis = 2 ) )

    R4 = np.sqrt( ( ( ρ_bdy2[indx,:] - ρ_sca1[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R5 = np.sqrt( ( ( ρ_bdy2[indx,:] - ρ_sca2[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R6 = np.sqrt( ( ( ρ_bdy2[indx,:] - ρ_sca3[jndx,:] ) ** 2 ).sum( axis = 2 ) )

    R7 = np.sqrt( ( ( ρ_bdy3[indx,:] - ρ_sca1[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R8 = np.sqrt( ( ( ρ_bdy3[indx,:] - ρ_sca2[jndx,:] ) ** 2 ).sum( axis = 2 ) )
    R9 = np.sqrt( ( ( ρ_bdy3[indx,:] - ρ_sca3[jndx,:] ) ** 2 ).sum( axis = 2 ) )

    # compute the cosine of the angle made between the normal on object p and R

    CosΘ1 = ( ν[indx,:] * ( ρ_bdy1[indx,:] - ρ_sca1[jndx,:] ) ).sum( axis = 2 ) / R1
    CosΘ2 = ( ν[indx,:] * ( ρ_bdy1[indx,:] - ρ_sca2[jndx,:] ) ).sum( axis = 2 ) / R2
    CosΘ3 = ( ν[indx,:] * ( ρ_bdy1[indx,:] - ρ_sca3[jndx,:] ) ).sum( axis = 2 ) / R3

    CosΘ4 = ( ν[indx,:] * ( ρ_bdy2[indx,:] - ρ_sca1[jndx,:] ) ).sum( axis = 2 ) / R4
    CosΘ5 = ( ν[indx,:] * ( ρ_bdy2[indx,:] - ρ_sca2[jndx,:] ) ).sum( axis = 2 ) / R5
    CosΘ6 = ( ν[indx,:] * ( ρ_bdy2[indx,:] - ρ_sca3[jndx,:] ) ).sum( axis = 2 ) / R6

    CosΘ7 = ( ν[indx,:] * ( ρ_bdy3[indx,:] - ρ_sca1[jndx,:] ) ).sum( axis = 2 ) / R7
    CosΘ8 = ( ν[indx,:] * ( ρ_bdy3[indx,:] - ρ_sca2[jndx,:] ) ).sum( axis = 2 ) / R8
    CosΘ9 = ( ν[indx,:] * ( ρ_bdy3[indx,:] - ρ_sca3[jndx,:] ) ).sum( axis = 2 ) / R9

    # compute the matrix sub-block

    A1 = ComputeDνG( k, R1, CosΘ1 )
    A2 = ComputeDνG( k, R2, CosΘ2 )
    A3 = ComputeDνG( k, R3, CosΘ3 )
    A4 = ComputeDνG( k, R4, CosΘ4 )
    A5 = ComputeDνG( k, R5, CosΘ5 )
    A6 = ComputeDνG( k, R6, CosΘ6 )
    A7 = ComputeDνG( k, R7, CosΘ7 )
    A8 = ComputeDνG( k, R8, CosΘ8 )
    A9 = ComputeDνG( k, R9, CosΘ9 )

    # solve the linear system

    c = np.linalg.solve( np.block( [ [ A1, A2, A3 ], [ A4, A5, A6 ], [ A7, A8, A9 ] ] ), np.block( [ b1, b2, b3 ] ) )

    # parse the solution

    c1 = c[:N]
    c2 = c[N:2*N]
    c3 = c[2*N:3*N]

    return c1, c2, c3;
```

## 3 Compute the acoustic radiation force

The function below computes the $x$-, $y$-, and $z$-components of the acoustic radiation force using the MFS result for the scalar potential.

```python
def ComputeMFSForce1Sphere( k, a, ρ0, κ0, ω, Nμ, N ):
    """
    This function computes the radiation force using the MFS solution for the scattered
    scalar potential φ1.
    """

    # compute the MFS points for the reference sphere

    ν, ρ_bdy, ρ_src = ComputeMFSReferencePoints( a, N )

    # solve the linear system of equations for the MFS expansion coefficients

    c = ComputeMFSExpansionCoefficients1Sphere( k, ν, ρ_bdy, ρ_src, N )

    # compute the Gauss-Legendre quadrature rule points and weights

    μ, w = np.polynomial.legendre.leggauss( Nμ )

    # evaluate the solution for a fixed azimuthal angle φ

    ϕ = np.pi / Nμ * np.arange( 2 * Nμ )

    # compute a meshgrid of the angle points

    indx, jndx = np.mgrid[ 0 : Nμ, 0 : 2 * Nμ ]

    # compute the components of the normal vector

    nx = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.cos( ϕ[jndx] ) ).flatten()
    ny = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.sin( ϕ[jndx] ) ).flatten()
    nz = ( μ[indx] ).flatten()

    # compute the spatial points

    X = a * nx
    Y = a * ny
    Z = a * nz

    # compute the quadrature weights

    weights = ( np.pi / Nμ * w[indx] ).flatten()

    ii, jj = np.mgrid[ 0:2*Nμ*Nμ, 0:N ]

    # compute the distance from source points to the evaluation point

    R = np.sqrt( ( X[ii] - ρ_src[jj,0] ) ** 2  \
                + ( Y[ii] - ρ_src[jj,1] ) ** 2 \
                + ( Z[ii] - ρ_src[jj,2] ) ** 2 )

    # compute Green's function

    G  = np.exp( 1j * k * R ) / ( 4.0 * np.pi * R )

    # compute the total field

    φ1 = np.exp( 1j * k * Z ) + G @ c

    # compute components of the gradient of the total field

    φ1x = ( ( X[ii] - ρ_src[jj,0] ) / R * ( 1j * k - 1.0 / R ) * G ) @ c

    φ1y = ( ( Y[ii] - ρ_src[jj,1] ) / R * ( 1j * k - 1.0 / R ) * G ) @ c

    φ1z = 1j * k * np.exp( 1j * k * Z ) \
        + ( ( Z[ii] - ρ_src[jj,2] ) / R * ( 1j * k - 1.0 / R ) * G ) @ c

    # compute the time-averaged pressure field

    p1_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ1 ) ** 2

    # compute the time-averaged velocity field

    v1_ave = 0.5 * np.abs( φ1x ) ** 2 + 0.5 * np.abs( φ1y ) ** 2 + 0.5 * np.abs( φ1z ) ** 2

    # compute the function to be integrated

    F = - 0.5 * κ0 * p1_ave + 0.5 * ρ0 * v1_ave

    # compute integrals

    Fx = np.sum( F * nx * weights )
    Fy = np.sum( F * ny * weights )
    Fz = np.sum( F * nz * weights )

    return Fx, Fy, Fz;
```

```python
def ComputeMFSForce2Spheres( k, a, δx, δy, ρ0, κ0, ω, Nμ, N ):
    """
    This function computes the radiation force using the MFS solution for the scattered
    scalar potential φ1.
    """

    # compute the MFS points for the reference sphere

    ν, ρ_bdy, ρ_src = ComputeMFSReferencePoints( a, N )

    # solve the linear system of equations for the MFS expansion coefficients

    c1, c2 = ComputeMFSExpansionCoefficients2Spheres( k, δx, δy, ν, ρ_bdy, ρ_src, N )

    # compute the scattering points for the 2 spheres

    ρ_src1 = ρ_src
    ρ_src2 = [ δx, δy, 0 ] + ρ_src

    # compute the Gauss-Legendre quadrature rule points and weights

    μ, w = np.polynomial.legendre.leggauss( Nμ )

    # evaluate the solution for a fixed azimuthal angle φ

    ϕ = np.pi / Nμ * np.arange( 2 * Nμ )

    # compute a meshgrid of the angle points

    indx, jndx = np.mgrid[ 0 : Nμ, 0 : 2 * Nμ ]

    # compute the components of the normal vector

    nx = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.cos( ϕ[jndx] ) ).flatten()
    ny = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.sin( ϕ[jndx] ) ).flatten()
    nz = ( μ[indx] ).flatten()

    # compute the spatial points

    X1 = a * nx
    Y1 = a * ny
    Z1 = a * nz

    X2 = δx + a * nx
    Y2 = δy + a * ny
    Z2 = a * nz

    # compute the quadrature weights

    weights = ( np.pi / Nμ * w[indx] ).flatten()

    ii, jj = np.mgrid[ 0:2*Nμ*Nμ, 0:N ]

    # compute the distance from source points to the evaluation point

    R11 = np.sqrt( ( X1[ii] - ρ_src1[jj,0] ) ** 2  \
                + ( Y1[ii] - ρ_src1[jj,1] ) ** 2 \
                + ( Z1[ii] - ρ_src1[jj,2] ) ** 2 )

    R12 = np.sqrt( ( X1[ii] - ρ_src2[jj,0] ) ** 2  \
                + ( Y1[ii] - ρ_src2[jj,1] ) ** 2 \
                + ( Z1[ii] - ρ_src2[jj,2] ) ** 2 )

    R21 = np.sqrt( ( X2[ii] - ρ_src1[jj,0] ) ** 2  \
                + ( Y2[ii] - ρ_src1[jj,1] ) ** 2 \
                + ( Z2[ii] - ρ_src1[jj,2] ) ** 2 )

    R22 = np.sqrt( ( X2[ii] - ρ_src2[jj,0] ) ** 2  \
                + ( Y2[ii] - ρ_src2[jj,1] ) ** 2 \
                + ( Z2[ii] - ρ_src2[jj,2] ) ** 2 )

    # compute Green's function

    G11 = np.exp( 1j * k * R11 ) / ( 4.0 * np.pi * R11 )
    G12 = np.exp( 1j * k * R12 ) / ( 4.0 * np.pi * R12 )
    G21 = np.exp( 1j * k * R21 ) / ( 4.0 * np.pi * R21 )
    G22 = np.exp( 1j * k * R22 ) / ( 4.0 * np.pi * R22 )

    # compute the total field

    φ1 = np.exp( 1j * k * Z1 ) + G11 @ c1 + G12 @ c2
    φ2 = np.exp( 1j * k * Z2 ) + G21 @ c1 + G22 @ c2

    # compute components of the gradient of the total field

    φ1x = ( ( X1[ii] - ρ_src1[jj,0] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( X1[ii] - ρ_src2[jj,0] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2

    φ1y = ( ( Y1[ii] - ρ_src1[jj,1] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( Y1[ii] - ρ_src2[jj,1] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2

    φ1z = 1j * k * np.exp( 1j * k * Z1 ) \
        + ( ( Z1[ii] - ρ_src1[jj,2] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( Z1[ii] - ρ_src2[jj,2] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2

    φ2x = ( ( X2[ii] - ρ_src1[jj,0] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( X2[ii] - ρ_src2[jj,0] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2

    φ2y = ( ( Y2[ii] - ρ_src1[jj,1] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( Y2[ii] - ρ_src2[jj,1] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2

    φ2z = 1j * k * np.exp( 1j * k * Z2 ) \
        + ( ( Z2[ii] - ρ_src1[jj,2] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( Z2[ii] - ρ_src2[jj,2] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2

    # compute the time-averaged pressure fields

    p1_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ1 ) ** 2
    p2_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ2 ) ** 2

    # compute the time-averaged velocity field

    v1_ave = 0.5 * np.abs( φ1x ) ** 2 + 0.5 * np.abs( φ1y ) ** 2 + 0.5 * np.abs( φ1z ) ** 2
    v2_ave = 0.5 * np.abs( φ2x ) ** 2 + 0.5 * np.abs( φ2y ) ** 2 + 0.5 * np.abs( φ2z ) ** 2

    # compute the function to be integrated

    F1 = - 0.5 * κ0 * p1_ave + 0.5 * ρ0 * v1_ave
    F2 = - 0.5 * κ0 * p2_ave + 0.5 * ρ0 * v2_ave

    # compute integrals

    F1x = np.sum( F1 * nx * weights )
    F1y = np.sum( F1 * ny * weights )
    F1z = np.sum( F1 * nz * weights )

    F2x = np.sum( F2 * nx * weights )
    F2y = np.sum( F2 * ny * weights )
    F2z = np.sum( F2 * nz * weights )

    return F1x, F1y, F1z;
```

```python
def ComputeMFSForce3Spheres( k, a, δ2, δ3, ρ0, κ0, ω, Nμ, N ):
    """
    This function computes the radiation force using the MFS solution for the scattered
    scalar potential φ1.
    """

    # compute the MFS points for the reference sphere

    ν, ρ_bdy, ρ_src = ComputeMFSReferencePoints( a, N )

    # solve the linear system of equations for the MFS expansion coefficients

    c1, c2, c3 = ComputeMFSExpansionCoefficients3Spheres( k, δ2, δ3, ν, ρ_bdy, ρ_src, N )

    # compute the scattering points for the 2 spheres

    ρ_src1 = ρ_src
    ρ_src2 = δ2 + ρ_src
    ρ_src3 = δ3 + ρ_src

    # compute the Gauss-Legendre quadrature rule points and weights

    μ, w = np.polynomial.legendre.leggauss( Nμ )

    # evaluate the solution for a fixed azimuthal angle φ

    ϕ = np.pi / Nμ * np.arange( 2 * Nμ )

    # compute a meshgrid of the angle points

    indx, jndx = np.mgrid[ 0 : Nμ, 0 : 2 * Nμ ]

    # compute the components of the normal vector

    nx = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.cos( ϕ[jndx] ) ).flatten()
    ny = ( np.sqrt( 1.0 - μ[indx] ** 2 ) * np.sin( ϕ[jndx] ) ).flatten()
    nz = ( μ[indx] ).flatten()

    # compute the spatial points

    X1 = a * nx
    Y1 = a * ny
    Z1 = a * nz

    X2 = δ2[0] + a * nx
    Y2 = δ2[1] + a * ny
    Z2 = δ2[2] + a * nz

    X3 = δ3[0] + a * nx
    Y3 = δ3[1] + a * ny
    Z3 = δ3[2] + a * nz

    # compute the quadrature weights

    weights = ( np.pi / Nμ * w[indx] ).flatten()

    ii, jj = np.mgrid[ 0:2*Nμ*Nμ, 0:N ]

    # compute the distance from source points to the evaluation point

    R11 = np.sqrt( ( X1[ii] - ρ_src1[jj,0] ) ** 2  \
                + ( Y1[ii] - ρ_src1[jj,1] ) ** 2 \
                + ( Z1[ii] - ρ_src1[jj,2] ) ** 2 )

    R12 = np.sqrt( ( X1[ii] - ρ_src2[jj,0] ) ** 2  \
                + ( Y1[ii] - ρ_src2[jj,1] ) ** 2 \
                + ( Z1[ii] - ρ_src2[jj,2] ) ** 2 )

    R13 = np.sqrt( ( X1[ii] - ρ_src3[jj,0] ) ** 2  \
                + ( Y1[ii] - ρ_src3[jj,1] ) ** 2 \
                + ( Z1[ii] - ρ_src3[jj,2] ) ** 2 )

    R21 = np.sqrt( ( X2[ii] - ρ_src1[jj,0] ) ** 2  \
                + ( Y2[ii] - ρ_src1[jj,1] ) ** 2 \
                + ( Z2[ii] - ρ_src1[jj,2] ) ** 2 )

    R22 = np.sqrt( ( X2[ii] - ρ_src2[jj,0] ) ** 2  \
                + ( Y2[ii] - ρ_src2[jj,1] ) ** 2 \
                + ( Z2[ii] - ρ_src2[jj,2] ) ** 2 )

    R23 = np.sqrt( ( X2[ii] - ρ_src3[jj,0] ) ** 2  \
                + ( Y2[ii] - ρ_src3[jj,1] ) ** 2 \
                + ( Z2[ii] - ρ_src3[jj,2] ) ** 2 )

    R31 = np.sqrt( ( X3[ii] - ρ_src1[jj,0] ) ** 2  \
                + ( Y3[ii] - ρ_src1[jj,1] ) ** 2 \
                + ( Z3[ii] - ρ_src1[jj,2] ) ** 2 )

    R32 = np.sqrt( ( X3[ii] - ρ_src2[jj,0] ) ** 2  \
                + ( Y3[ii] - ρ_src2[jj,1] ) ** 2 \
                + ( Z3[ii] - ρ_src2[jj,2] ) ** 2 )

    R33 = np.sqrt( ( X3[ii] - ρ_src3[jj,0] ) ** 2  \
                + ( Y3[ii] - ρ_src3[jj,1] ) ** 2 \
                + ( Z3[ii] - ρ_src3[jj,2] ) ** 2 )

    # compute Green's function

    G11 = np.exp( 1j * k * R11 ) / ( 4.0 * np.pi * R11 )
    G12 = np.exp( 1j * k * R12 ) / ( 4.0 * np.pi * R12 )
    G13 = np.exp( 1j * k * R13 ) / ( 4.0 * np.pi * R13 )

    G21 = np.exp( 1j * k * R21 ) / ( 4.0 * np.pi * R21 )
    G22 = np.exp( 1j * k * R22 ) / ( 4.0 * np.pi * R22 )
    G23 = np.exp( 1j * k * R23 ) / ( 4.0 * np.pi * R23 )

    G31 = np.exp( 1j * k * R31 ) / ( 4.0 * np.pi * R31 )
    G32 = np.exp( 1j * k * R32 ) / ( 4.0 * np.pi * R32 )
    G33 = np.exp( 1j * k * R33 ) / ( 4.0 * np.pi * R33 )

    # compute the scattered field on particle 3

    φ1 = np.exp( 1j * k * Z1 ) + G11 @ c1 + G12 @ c2 + G13 @ c3
    φ2 = np.exp( 1j * k * Z2 ) + G21 @ c1 + G22 @ c2 + G23 @ c3
    φ3 = np.exp( 1j * k * Z3 )  + G31 @ c1 + G32 @ c2 + G33 @ c3

    # compute components of the gradient of the total field

    φ1x = ( ( X1[ii] - ρ_src1[jj,0] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( X1[ii] - ρ_src2[jj,0] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2 \
        + ( ( X1[ii] - ρ_src3[jj,0] ) / R13 * ( 1j * k - 1.0 / R13 ) * G13 ) @ c3

    φ1y = ( ( Y1[ii] - ρ_src1[jj,1] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( Y1[ii] - ρ_src2[jj,1] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2 \
        + ( ( Y1[ii] - ρ_src3[jj,1] ) / R13 * ( 1j * k - 1.0 / R13 ) * G13 ) @ c3

    φ1z = 1j * k * ( np.exp( 1j * k * Z1 ) ) \
        + ( ( Z1[ii] - ρ_src1[jj,2] ) / R11 * ( 1j * k - 1.0 / R11 ) * G11 ) @ c1 \
        + ( ( Z1[ii] - ρ_src2[jj,2] ) / R12 * ( 1j * k - 1.0 / R12 ) * G12 ) @ c2 \
        + ( ( Z1[ii] - ρ_src3[jj,2] ) / R13 * ( 1j * k - 1.0 / R13 ) * G13 ) @ c3

    φ2x = ( ( X2[ii] - ρ_src1[jj,0] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( X2[ii] - ρ_src2[jj,0] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2 \
        + ( ( X2[ii] - ρ_src3[jj,0] ) / R23 * ( 1j * k - 1.0 / R23 ) * G23 ) @ c3

    φ2y = ( ( Y2[ii] - ρ_src1[jj,1] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( Y2[ii] - ρ_src2[jj,1] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2 \
        + ( ( Y2[ii] - ρ_src3[jj,1] ) / R23 * ( 1j * k - 1.0 / R23 ) * G23 ) @ c3

    φ2z = 1j * k * ( np.exp( 1j * k * Z2 ) ) \
        + ( ( Z2[ii] - ρ_src1[jj,2] ) / R21 * ( 1j * k - 1.0 / R21 ) * G21 ) @ c1 \
        + ( ( Z2[ii] - ρ_src2[jj,2] ) / R22 * ( 1j * k - 1.0 / R22 ) * G22 ) @ c2 \
        + ( ( Z2[ii] - ρ_src3[jj,2] ) / R23 * ( 1j * k - 1.0 / R23 ) * G23 ) @ c3

    φ3x = ( ( X3[ii] - ρ_src1[jj,0] ) / R31 * ( 1j * k - 1.0 / R31 ) * G31 ) @ c1 \
        + ( ( X3[ii] - ρ_src2[jj,0] ) / R32 * ( 1j * k - 1.0 / R32 ) * G32 ) @ c2 \
        + ( ( X3[ii] - ρ_src3[jj,0] ) / R33 * ( 1j * k - 1.0 / R33 ) * G33 ) @ c3

    φ3y = ( ( Y3[ii] - ρ_src1[jj,1] ) / R31 * ( 1j * k - 1.0 / R31 ) * G31 ) @ c1 \
        + ( ( Y3[ii] - ρ_src2[jj,1] ) / R32 * ( 1j * k - 1.0 / R32 ) * G32 ) @ c2 \
        + ( ( Y3[ii] - ρ_src3[jj,1] ) / R33 * ( 1j * k - 1.0 / R33 ) * G33 ) @ c3

    φ3z =  1j * k * ( np.exp( 1j * k * Z3 ) )  \
        + ( ( Z3[ii] - ρ_src1[jj,2] ) / R31 * ( 1j * k - 1.0 / R31 ) * G31 ) @ c1 \
        + ( ( Z3[ii] - ρ_src2[jj,2] ) / R32 * ( 1j * k - 1.0 / R32 ) * G32 ) @ c2 \
        + ( ( Z3[ii] - ρ_src3[jj,2] ) / R33 * ( 1j * k - 1.0 / R33 ) * G33 ) @ c3

    # compute the time-averaged pressure fields

    p1_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ1 ) ** 2
    p2_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ2 ) ** 2
    p3_ave = 0.5 * np.abs( 1j * ρ0 * ω * φ3 ) ** 2

    # compute the time-averaged velocity field

    v1_ave = 0.5 * np.abs( φ1x ) ** 2 + 0.5 * np.abs( φ1y ) ** 2 + 0.5 * np.abs( φ1z ) ** 2
    v2_ave = 0.5 * np.abs( φ2x ) ** 2 + 0.5 * np.abs( φ2y ) ** 2 + 0.5 * np.abs( φ2z ) ** 2
    v3_ave = 0.5 * np.abs( φ3x ) ** 2 + 0.5 * np.abs( φ3y ) ** 2 + 0.5 * np.abs( φ3z ) ** 2

    # compute the function to be integrated

    F1 = - 0.5 * κ0 * p1_ave - 0.5 * ρ0 * v1_ave
    F2 = - 0.5 * κ0 * p2_ave - 0.5 * ρ0 * v2_ave
    F3 = - 0.5 * κ0 * p3_ave + 0.5 * ρ0 * v3_ave

    # compute integrals

    F1x = np.sum( F1 * nx * weights )
    F1y = np.sum( F1 * ny * weights )
    F1z = np.sum( F1 * nz * weights )

    F2x = np.sum( F2 * nx * weights )
    F2y = np.sum( F2 * ny * weights )
    F2z = np.sum( F2 * nz * weights )

    F3x = np.sum( F3 * nx * weights )
    F3y = np.sum( F3 * ny * weights )
    F3z = np.sum( F3 * nz * weights )

    return F1x, F1y, F1z;
```

## 4 Specific example

In the codes that follow, we compute the MFS for the scattered field for a specific set of physical parameters.

```python
# set the frequency of the acoustic waves (kHz)

f = 40.0

# set the wave speed (cm kHz)

c0 = 34.3

# set the density of the spheres (g/cm^3)

ρ0 = 0.001225

# set the compressibility

κ0 = 1.0 / ( ρ0 * c0 ** 2 )

# compute the circular frequency

ω = 2.0 * np.pi * f

# compute the wavenumber (cm^-1)

k = ω / c0

#wavelength of incident field
λ = 2 * np.pi / k

# set the nondimensional size parameter ka

ka = 1

# compute the radius of the sphere from ka

a = ka / k
```

```python
# set the order of the quadrature rule

Nμ = 8

# set the number of MFS points

N = 128

# set the number of locations over which to compute forces, and set values of δ (separation distance between particles)
#In the case of forces between 3 particles, the configuration is set up in the shape of an isoceles triangles, with the
#edge length of the triangle corresponding to δ

Nδ = 201
δ  = np.linspace( 2.01 * a, 25.0 * a, Nδ )

# allocate memory for force components

F3x = np.full( Nδ, 'nan', dtype = 'float' )
F3y = np.full( Nδ, 'nan', dtype = 'float' )
F3z = np.full( Nδ, 'nan', dtype = 'float' )

# loop over all δ values

for n in range( Nδ ):

    # compute the acoustic raditaion force for one sphere

    #Fx[n], Fy[n], Fz[n] = ComputeMFSForce1Sphere( k, a, ρ0, κ0, ω, Nμ, N )

    # compute the acoustic raditaion force for two spheres

    #F1x[n], F1y[n], F1z[n] = ComputeMFSForce2Spheres( k, a, -δ[n], 0, ρ0, κ0, ω, Nμ, N )

    # compute the total acoustic radiation force for three spheres in a regular triangle

    F3x[n], F3y[n], F3z[n] = ComputeMFSForce3Spheres( k, a, [ -δ[n] * np.sqrt(3)/2, δ[n] / 2, 0 ],\
                                                    [ -δ[n] * np.sqrt(3)/2, -δ[n] / 2, 0 ], ρ0, κ0, ω, Nμ, N )

```

```python
# plot the x-component of the force as a function of δ

plt.rcParams['figure.figsize'] = [8,6]

plt.plot( δ / λ, F3x, 'b-' )
plt.grid()
plt.xlabel( r'separation ($\delta / \lambda$)', fontsize = 14 )
plt.ylabel( r'$F_{x}$', fontsize = 14 )
plt.title(f'ka = {ka}')
plt.show()
```

```python
# plot the z-component of the force as a function of δ

plt.rcParams['figure.figsize'] = [8,6]

plt.plot( δ / λ, F3z, 'b-' )
plt.grid()
plt.xlabel( r'separation ($\delta / \lambda$)', fontsize = 14 )
plt.ylabel( r'$F_{z}$', fontsize = 14 )
plt.title(f'ka = {ka}')

plt.show()
```

```python

```

Jupyter Notebooks/Acoustic radiation force on multiple sound-hard spheres.ipynb