# HW2 Multinomial Bayes Classfier

#### Name: Haochen (Harry) Wang

```python
from run import *
import matplotlib.pyplot as plt
```

    [nltk_data] Downloading package stopwords to
    [nltk_data]     /Users/harrywanghc/nltk_data...
    [nltk_data]   Package stopwords is already up-to-date!

### Q.1 (18 Points)

##### You will first run an experiment to evaluate how the performance of the Naive Bayes algorithm is affected depending on whether (i) you classify instances by computing the pos- terior probability, Pr(yi | Doc), according to the standard equation (Eq. (1)); or (ii) you classify instances by performing the log-transformation trick discussed in class and compare log-probabilities, log(Pr(yi | Doc)), instead of probabilities. Recall, in particular, that estimating the posterior probability using Eq. (1) might cause numerical issues/instability since it requires computing the product of (possibly) hundreds of small terms—which makes this probability estimate rapidly approach zero. To tackle this problem, we often compare not Pr(y1 | Doc) and Pr(y2 | Doc), but the corresponding log-probabilities: log(Pr(y1 | Doc)) and log(Pr(y2 | Doc)). Taking the logarithm of such probabilities transforms the product of hundreds of terms into the sum of hundreds of terms—which avoids numerical issues. Importantly, it does not change which class is more likely according to the trained model. When classifying a new instance, the log-probabilities that should be compared, for each class yi, are as follows:
$$ log(Pr(y_i|doc)) = log\left( Pr(y_i) \prod_{k=1}^{len(Doc)} Pr(w_k|y_i)\right) $$
$$ log(Pr(y_i|doc)) = log\left( Pr(y_i) +  \sum_{k=1}^{len(Doc)} Pr(w_k|y_i)\right) $$
##### In this experiment, you should use 20% of the training set and 20% of the test set; i.e., call the dataset-loading functions by passing 0.2 as their parameters. First, perform the classification of the instances in the test set by comparing posterior probabilities, Pr(yi | Doc), according to Eq. (1), for both classes. Then, report (i) the accuracy of your model; (ii) its precision; (iii) its recall; and (iv) the confusion matrix resulting from this experiment. Now repeat the same experiment above but classify the instances in the test set by comparing log-probabilities, log(Pr(yi | Doc)), according to Eq. (5), for both classes. Report the same quantities as before. Discuss whether classifying instances by computing log-probabilities, instead of probabilities, affects the model’s performance. Assuming that this transformation does have an impact on performance, does it affect more strongly the model’s accuracy, precision, or recall? Why do you think that is the case?

#### Q1 Answer:

```python
# Find the value using equation (1), with no log and no laplace smoothing
tpq1_1, tnq1_1, fpq1_1, fnq1_1 = naive_bayes(0.2,0.2,0.2,0.2,laplacesmooth=False,logbool=False)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  4.82 sec
    Testing Positive, at time  6.93 sec
    Testing Negative, at time  7.34 sec
    Total Time Cost is  7.72 sec
    Accuarcy  is:  0.370622
    Precision is:  0.362732
    Recall    is:  0.334263
    F-Score   is:  0.347916

```python
# Now use q1_1 to implement confusion matrix
confusionmatrix(tpq1_1, tnq1_1, fpq1_1, fnq1_1,"Question 1 no log")
```

Accuarcy  is:  0.387914
Precision  is:  0.384452
Recall       is:  0.348613

Here is the one of the Accuarcy, Precision, Recall and Matrix I have for using equation (1) without using log for the Probability. The value might differ when I export and run the code again.

```python
# Repeat the experiment, find the value using equation (5), with no log and no laplace smoothing
tpq1_2, tnq1_2, fpq1_2, fnq1_2 = naive_bayes(0.2,0.2,0.2,0.2,laplacesmooth=False,logbool=True)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  4.76 sec
    Testing Positive, at time  7.09 sec
    Testing Negative, at time  7.49 sec
    Total Time Cost is  7.88 sec
    Accuarcy  is:  0.572148
    Precision is:  0.583810
    Recall    is:  0.495354
    F-Score   is:  0.535956

```python
# Confusion Matrix
confusionmatrix(tpq1_2, tnq1_2, fpq1_2, fnq1_2, "Question 1 with log")
```

Accuarcy  is:  0.573660
Precision  is:  0.592018
Recall        is:  0.519253

Here is the accuarcy, precision, recall and the confusion matrix after implementing log for probability using equation (5). The data might differ since I will run the code again when exporting.

The computaion of summing log-probabilities instead of the 'prod_probabilities' increase the accuracy, precision, and recall of the algorithm.

Taking accuracy as an example, before using the log-probibiliies, the accuracy was about 0.38. After using lofg for prob, we have accuracy about 0.59, which is a huge improvment: we now correctly classify more than have of the comments (finally better than random!). And for precision and recall, there are also high improvement: from 0.38 to 0.59; from 0.34 to 0.52.

I think the reason for this is that in equation (1), we are taking the product of a list of probabilities that are small, for instance, (6.4e-18, 2.3e-28, 1.8e-24, ...) and the length of one comment would generally be more than 100 words (count repeated as one) appeared, so the product would easily exceed the smallest float in python, which is 2.2251e-308. Hence the classifier is inaccurate since there are many zeros. (Also I set the classifier to > and < instead of <= and >=. If the no-log-prob classifier was with <= and >=, the accuracy was about 0.70). After using the log-probilities, we change the product to the sum of log-probilities, and that follows:

$$a = b$$
$$log(a) = (b) $$

So, for probability,

$$ Prob(a) < Prob(b) $$
$$ log(Prob(a)) < log(Prob(b)) $$

And since

$$ log(a*b) = log(a)+log(b) $$

So we solve the problem of exceeding 2.2251e-308 in equation (1). Hence the equation (5) have a better performance.

### Q.2 (18 Points)
##### An issue with the original Naive Bayes formulation is that if a test instance contains a word that is not present in the vocabulary identified during training, then Pr(word|label) = 0. To mitigate this issue, one solution is to employ Laplace Smoothing. To do so, as discussed in class, we replace the standard way of estimating the probability of a word wk, given a class yi, with the following equation:
$$ Pr(w_k|y_i) = \frac{n(w_k,y_i)+1}{\sum_{s=1}^{|V|} n(w_s,y_i)+|V|} $$
##### More generally, Laplace Smoothing can be performed according to a parametric equation, where instead of adding 1 to the numerator, we adjust the probability of a word belong to a class by adding a user-defined parameter α to the numerator, as follows:
$$ Pr(w_k|y_i) = \frac{n(w_k,y_i)+\alpha}{\sum_{s=1}^{|V|} n(w_s,y_i)+\alpha|V|} $$
##### Intuitively, setting α = 0 results in the standard formulation of Naive Bayes—which does not tackle the problem of words that do not appear in the training set. Suppose, alternatively, that we set α = 4. This is equivalent to adding 4 “fake” occurrences of that word to the training set, in order to avoid the zero-frequency problem. Using α = 1000, on the other hand, is equivalent to pretending we have seen that word 1000 times in the training set—even though we may have seen it, say, only 8 times. Although this solves the problem of zero-frequency words, it also strongly biases the model to “believe” that that word appears much more frequently than it actually does; and this could make the predictions made by the system less accurate. For these reasons, although it is important/necessary to perform Laplace Smoothing, we have to carefully pick the value of α that works best for our dataset. Using α = 1 is common, but other values might result in better performance, depending on the dataset being analyzed.
##### In this experiment, you should use 20% of the training set and 20% of the test set; i.e., call the dataset-loading functions by passing 0.2 as their parameters. You should first report the confusion matrix, precision, recall, and accuracy of your classifier (when evaluated on the test set) when using α = 1. Now, vary the value of α from 0.0001 to 1000, by multiplying α with 10 each time. That is, try values of α equal to 0.0001, 0.001, 0.01, 0.1, 1.0, 100, and 1000. For each value, record the accuracy of the resulting model when evaluated on the test set. Then, create a plot of the model’s accuracy on the test set (shown on the y-axis) as a function of the value of α (shown on the x-axis). The x-axis should represent α values and use a log scale. Analyze this graph and discuss why do you think the accuracy suffers when α is too high or too low.

#### Q2 Answer:

```python
tpq2, tnq2, fpq2, fnq2 = naive_bayes(0.2,0.2,0.2,0.2,laplacesmooth=True,logbool=True,smoothconst=1)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  4.62 sec
    Testing Positive, at time  6.89 sec
    Testing Negative, at time  7.32 sec
    Total Time Cost is  7.75 sec
    Accuarcy  is:  0.817056
    Precision is:  0.856054
    Recall    is:  0.766975
    F-Score   is:  0.809070

```python
confusionmatrix(tpq2, tnq2, fpq2, fnq2,"Question2 alpha = 1")
```

Accuarcy  is:  0.812151
Precision  is:  0.868593
Recall        is:  0.738218

We can see with laplace transform with $\alpha = 1$ , the accuracy, precision and recall all improved with the one without using laplace smoothing: I think that's due to the count in of data in instance/comment with words that's not in the positive/negative 'database' or 'dictionary'.

Without Laplace smoothing, when we see a word that never appear in positive dictionary, we just ignore the word, which is incorrect: ignoring the word make the probability of that word 1 (the prob doesn't change after this word); however, what really should happen is we would have a probability count in, and the probability should be close to zero. So laplace smoothing solve the problem of word not in dictionary.

```python
# Now have fixed data for q2

positrainq2, negatrainq2, vocabq2 = load_training_set(0.2,0.2)
positestq2, negatestq2 = load_test_set(0.2,0.2)
```

```python
alpha = 0.0001
alphalist = []
accuracylist = []
while alpha <= 1000:
    tpq2_2, tnq2_2, fpq2_2, fnq2_2 = fixdatabayes(positrainq2, negatrainq2, positestq2, negatestq2, vocabq2, True, True, smoothconst=alpha)
    accuracylist.append(accuracy(tpq2_2, tnq2_2, fpq2_2, fnq2_2))
    alphalist.append(alpha)
    alpha*=10
    #print(alpha)
```

```python
plt.xscale("log")
plt.xlabel("Alpha Value")
plt.ylabel("Accuracy")
plt.title("Accuracy vs. Laplace Smoothing Constant ALPHA")
plt.plot(alphalist,accuracylist)
plt.show
```

From the graph, we can find out that it follows the shape of a rollercoaster: is a peak around $\alpha = 10^1 $ to $10^2$ and the accuracy is low around very small and very big $\alpha $ value.

I think that's because when the $\alpha $ is too small, the laplace transform's affect would be countless. And when $\alpha $ is too big, the normal probility's contribution would be decreased. Consider two extreme condition:

 $$lim_{\alpha \to 0}\frac{n+\alpha}{N_{sum}+\alpha |V|} = \frac{n}{N_{sum}}$$

$$lim_{\alpha \to \infty}\frac{n+\alpha}{N_{sum}+\alpha |V|} = \frac{1}{|V|}$$

where $n $ is $n(w_k,y_i)$ and $N_{sum} $ is $\sum_{s=1}^{|V|}n(w_s,y_i)$

What we have in the $\alpha $ as 1000 and 0.0001 fit in two of those condition above: too big and too small.

### Q.3 (18 Points)
##### Now you will investigate the impact of the training set size on the performance of the model. The classification of new instances, here, should be done by comparing the posterior log-probabilities, log(Pr(yi | Doc)), according to Eq. (5), for both classes. You should use the value of α that resulted in the highest accuracy according to your experiments in the previous question. In this question, you should use 100% of the training set and 100% of the test set; i.e., call the dataset-loading functions by passing 1.0 as their parameters. Then, report (i) the accuracy of your model; (ii) its precision; (iii) its recall; and (iv) the confusion matrix resulting from this experiment.

#### Q3 Answer:

```python
tpq3, tnq3, fpq3, fnq3 = naive_bayes(1,1,1,1,laplacesmooth=True,logbool=True,smoothconst=10)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  40.89 sec
    Testing Positive, at time  119.61 sec
    Testing Negative, at time  129.43 sec
    Total Time Cost is  138.25 sec
    Accuarcy  is:  0.839720
    Precision is:  0.870194
    Recall    is:  0.798560
    F-Score   is:  0.832840

```python
confusionmatrix(tpq3, tnq3, fpq3, fnq3,'question3')
```

Accuarcy  is:  0.839720
Precision  is:  0.870194
Recall        is:  0.798560
F-Score     is:  0.832840

The data might change when I run everything again when export the document.

### Q.4 (18 Points)
##### Now repeat the experiment above but use only 50% of the training instances; that is, load the training set by calling load training set(0.5, 0.5). The entire test set should be used. Report the same quantities as in the previous question. Discuss whether using such a smaller training set had any impact on the performance your learned model. Analyze the confusion matrices (of this question and the previous one) and discuss whether one particular class was more affected by changing the size of the training set.

#### Q4 Answer:

```python
tpq4, tnq4, fpq4, fnq4 = naive_bayes(0.5,0.5,1,1,laplacesmooth=True,logbool=True,smoothconst=10)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  17.64 sec
    Testing Positive, at time  35.01 sec
    Testing Negative, at time  39.46 sec
    Total Time Cost is  43.88 sec
    Accuarcy  is:  0.838400
    Precision is:  0.870403
    Recall    is:  0.795200
    F-Score   is:  0.831104

```python
confusionmatrix(tpq4, tnq4, fpq4, fnq4,'question4')
```

For Question 4,

Accuarcy  is:  0.832160
Precision  is:  0.875611
Recall        is:  0.774320  re

as we can compare to question 3,

Accuarcy  is:  0.839720
Precision  is:  0.870194
Recall        is:  0.798560

We can find out that the A,P,R (Accuarcy, Precision, Recall) for 1.0 testing data and 0.5 testing data is actually very close, except for the recall:

the recall of question 4 drop from 0.799 to 9.774. Which is due to the increase in falseNegative.

But overall, as we can see, there isn't a greate change in the result. I think that would be due to the information gain ratio to the increase in training set. Think about len(vocab) as kind of info gain, the graph of

$$[len-vocab-gain]\  vs. \  [trainingset-loaded]$$

what we should have would be a log graph, when we load training set from from 0.0001 to 0.001, we would have a really rapid len(vocab) gain, and as the training-set-portion getting larger, the increase in the len(vocab) decreases. Hence a 0.5 training & a 1.0 training wouldn't have a big difference compare to 0.01 training & a 0.3 training.

As shown in the graph below.

```python
# Code reference
trainportion = 0.0001
portionlist =[]
voclist = []
while trainportion<=0.7:
    voc = load_training_set(trainportion,trainportion)[2]
    voclist.append(len(voc))
    portionlist.append(trainportion)
    trainportion*=1.5
```

```python
plt.xlabel("training Portion")
plt.ylabel("len(Vocab)/VocabGain")
plt.title("len(vocab) vs. train_portion")
plt.plot(portionlist,voclist)
plt.show()
```

### Q.5 (10 Points)
##### In this application (i.e., accurately classifying movie reviews), would you say that it is more important to have high accuracy, high precision, or high recall? Justify your opinion.

#### Q5 Answer:

To be honest, I think for the application of classifying movie reviews, it doesn't matter a lot with the accuracy, since it's unlike classifying tumor, covid or other more significant issues.

But try to consider three:
High Accuracy: Find out how's the classification overall
High Precision: Find out the how many percent of predict+ is real+
High Recall: Find out the how many percent of real+ is predicted as +

Since I suppose one of the application of the imdb comment is to find out whether like the movie or not, so they can further recommend other movie to users. So if we know a certain user A is positive about movie X, we can still recommend type X or other Movie, but for certain user that's negative about movie X, we don't want to give him type X movie.

So what's important here is to don't let true negative see movie X again. So we'd like a 'higher recall if consider negative_comment as positive', hence that would be higher precision in normal 'degree of freedom'.

Or we can suppose the application as show positive review in the front (The director pay us 1 million dollar), we don't want to show the falsePositive review/comment in the front. So here, also a higher precision would be better.

### Q.6 (18 Points)
##### Finally, you will study how the performance of the learned model is affected by training it using an unbalanced dataset (i.e., a dataset with significantly more examples of one of the classes). The classification of new instances, here, should be done by comparing the posterior log-probabilities, log(Pr(yi | Doc)), according to Eq. (5), for both classes. You should use the value of α that resulted in the highest accuracy according to your experiments in the previous questions. You will now conduct an experiment where you use only 10% of the available positive training instances and that uses 50% of the available negative training instances. That is, use load training set(0.1, 0.5). The entire test set should be used. Show the confusion matrix of your trained model, as well as its accuracy, precision, and recall. Compare this model’s performance to the performance (according to these same metrics) of the model trained in question Q.4—that is, a model that was trained under a balanced dataset. Discuss how training under an unbalanced dataset affected each of these performance metrics.

#### Q6 Answer:

```python
tpq6, tnq6, fpq6, fnq6 = naive_bayes(0.1,0.5,1,1,laplacesmooth=True,logbool=True,smoothconst=1)
```

    Loading Data, at time 0.00 sec
    Learning Training Data, at time  11.68 sec
    Testing Positive, at time  22.60 sec
    Testing Negative, at time  25.47 sec
    Total Time Cost is  28.28 sec
    Accuarcy  is:  0.667440
    Precision is:  0.878071
    Recall    is:  0.388880
    F-Score   is:  0.539033

```python
confusionmatrix(tpq6, tnq6, fpq6, fnq6, "question 6")
```

(The data might be different when running again while exporting)

Accuarcy  is:  0.667440
Precision  is:  0.878071
Recall        is:  0.388880
F-Score    is:  0.539033

The unbalanced training dataset create a very biased result compared to what we have in question 4: We have a very low recall and high precision.

Consider (0.1 postrain, 0.5 neg_train) as example, we are predicting more of the testing as negative because the lack of pos_train data would result in lack of pos_train_dictionary. Hence there would be many word get 'laplace smoothed' because it nevered appeared in the dictionary, and those 'smoothed' word might be a 'high-frequency pos word'. So that result lots of the realpositive data predicted as negative. So does the real negative data, more of them are predicted as negative. So that make the precision really high.

Also, when I started with this question, I leave $\alpha = 10 $ as the questions above, however, that's very bad decision because that make the prediction in positve vey close to zero, which is more biased, and accuracy end around 50%. That's because the vocablist is very large and with smooth constant 10, that make the original n and N_sum part very neglectable.

 $\frac{n+\alpha}{N_{sum}+\alpha |V|} = \frac{1}{|V|}$, when $\alpha |V| >> N_{sum}$

$n $ is $n(w_k,y_i)$ and $N_{sum} $ is $\sum_{s=1}^{|V|}n(w_s,y_i)$

So in this case, all pos_train probability factor become really close to the laplace smoothing factor, hence, result in worse accuracy.

#### APPENDIX: utils.py

```python
from cgi import test
import re
import os
import glob
import random
from nltk.corpus import stopwords
import nltk
import string
from collections import Counter
from pprint import pprint
import math
import numpy as np
import matplotlib.pyplot as plt

from sympy import true

REPLACE_NO_SPACE = re.compile("[._–;:!`¦\'?,\"()\[\]]") # add one '–'
REPLACE_WITH_SPACE = re.compile("(<br\s*/><br\s*/>)|(\-)|(\/)")
nltk.download('stopwords')

def preprocess_text(text):
	stop_words = set(stopwords.words('english'))
	text = REPLACE_NO_SPACE.sub("", text)
	text = REPLACE_WITH_SPACE.sub(" ", text)
	text = re.sub(r'\d+', '', text)
	text = text.lower()
	text="".join([i for i in text if i not in string.punctuation])
	words = text.split()
	return [w for w in words if w not in stop_words]

def load_training_set(percentage_positives, percentage_negatives):
	vocab = set() # I returned list later
	positive_instances = []
	negative_instances = []
	for filename in glob.glob('train/pos/*.txt'):
		if random.random() > percentage_positives:
			continue
		with open(os.path.join(os.getcwd(), filename), 'r', encoding="utf-8") as f:
			contents = f.read()
			contents = preprocess_text(contents)
			positive_instances.append(contents)
			vocab = vocab.union(set(contents))
	for filename in glob.glob('train/neg/*.txt'):
		if random.random() > percentage_negatives:
			continue
		with open(os.path.join(os.getcwd(), filename), 'r', encoding="utf-8") as f:
			contents = f.read()
			contents = preprocess_text(contents)
			negative_instances.append(contents)
			vocab = vocab.union(set(contents))
	return positive_instances, negative_instances, list(vocab)

def load_test_set(percentage_positives, percentage_negatives):
	positive_instances = []
	negative_instances = []
	for filename in glob.glob('test/pos/*.txt'):
		#print(filename)
		if random.random() > percentage_positives:
			continue
		with open(os.path.join(os.getcwd(), filename), 'r', encoding="utf-8") as f:
			contents = f.read()
			contents = preprocess_text(contents)
			positive_instances.append(contents)
	for filename in glob.glob('test/neg/*.txt'):
		if random.random() > percentage_negatives:
			continue
		with open(os.path.join(os.getcwd(), filename), 'r', encoding="utf-8") as f:
			contents = f.read()
			contents = preprocess_text(contents)
			negative_instances.append(contents)
	return positive_instances, negative_instances

def train(positivedata, negativedata):
    posidict = dict(Counter(sum(positivedata,[])))
    # print(len(sum(positivedata,[])))
    # print(len(sum(negativedata,[])))
    negadict = dict(Counter(sum(negativedata,[])))
    return posidict, negadict

def probilityof(positivetrain, negativetrain, positivedict, negativedict, vocab, category: string, instance, log: bool, laplacesmooth: bool, smoothconstant = 1):

    posinum = len(positivetrain)
    neganum = len(negativetrain)
    suminstance = neganum+posinum

    if category == "posi" or category == "positive":
        num = posinum
        trainset = positivetrain
        dictuse = positivedict
    else:
        num = neganum
        trainset = negativetrain
        dictuse = negativedict

    plist = []
    p0 = num/suminstance
    plist.append(p0)
    denominator = sum([len(i) for i in trainset])
    # print(sum(dictuse.values())-denominator)
    vsize = len(vocab)
    # vsize = len(dictuse)

    for i in instance:
        if (not laplacesmooth):
            if i in dictuse:
                probinstance = (dictuse[i]/denominator)
            else:
                continue
        else: # Smooth
            if i in dictuse:
                probinstance = ((dictuse[i]+smoothconstant)/(denominator+smoothconstant*vsize))
            else:
                probinstance = ((smoothconstant)/(smoothconstant*vsize))
        plist.append(probinstance)

    if log:
        loglist = list(map(math.log10, plist))
        return sum(loglist)

    # if 0 not in plist:
    #     print("prob is not zero")

    return math.prod(plist)

def toCounterDictList(trainingset):
	return [dict(Counter(i)) for i in trainingset]

def accuracy(truePosi, trueNega, falsePosi, falseNega): # Count of all four
	return (truePosi+trueNega)/(truePosi+trueNega+falseNega+falsePosi)

def precision(truePosi, trueNega, falsePosi, falseNega):
	if (truePosi+falsePosi) == 0:
		return 0
	preposi = truePosi/(truePosi+falsePosi)
	prenega = trueNega/(trueNega+falseNega)
	return preposi

def recall(truePosi, trueNega, falsePosi, falseNega):
	if (truePosi+falseNega)== 0:
		return 0
	recposi = truePosi/(truePosi+falseNega)
	recnega = trueNega/(trueNega+falsePosi)
	return recposi

def fscore(truePosi, trueNega, falsePosi, falseNega, beta: 1):
	pre = precision(truePosi, trueNega, falsePosi, falseNega)
	rec = recall(truePosi, trueNega, falsePosi, falseNega)
	if (pre*(beta**2)+rec) == 0:
		return 0
	f = (1+beta**2)*((pre*rec)/(pre*(beta**2)+rec))
	return f

def confusionmatrix(truePosi, trueNega, falsePosi, falseNega, title=""):
	fig = plt.figure()
	plt.title(title)
	col_labels = ['Predict:+', 'Predict:-']
	row_labels = ['Real:+', 'Real:-']
	table_vals = [[truePosi, falseNega], [falsePosi, trueNega]]
	the_table = plt.table(cellText=table_vals,
                      colWidths=[0.1] * 3,
                      rowLabels=row_labels,
                      colLabels=col_labels,
                      loc='center')
	the_table.auto_set_font_size(False)
	the_table.set_fontsize(24)
	the_table.scale(4, 4)
	plt.tick_params(axis='x', which='both', bottom=False, top=False, labelbottom=False)
	plt.tick_params(axis='y', which='both', right=False, left=False, labelleft=False)

	for pos in ['right','top','bottom','left']:
		plt.gca().spines[pos].set_visible(False)

	plt.show()

	return

```

#### APPENDIX: run.py

```python
from utils import *
from pprint import pprint
import time

def naive_bayes(ptrain:float=0.1,ntrain:float=0.1,ptest:float=0.1,ntest:test=0.1,laplacesmooth:bool=True, logbool:bool=True, smoothconst:float=1):

	t0 = time.time()

	percentage_positive_instances_train = ptrain
	percentage_negative_instances_train = ntrain

	percentage_positive_instances_test  = ptest
	percentage_negative_instances_test  = ntest

	print("Loading Data, at time 0.00 sec")

	(pos_train, neg_train, vocab) = load_training_set(percentage_positive_instances_train, percentage_negative_instances_train)
	(pos_test,  neg_test)         = load_test_set(percentage_positive_instances_test, percentage_negative_instances_test)

	# print("Number of positive training instances:", len(pos_train))
	# print("Number of negative training instances:", len(neg_train))
	# print("Number of positive test instances:", len(pos_test))
	# print("Number of negative test instances:", len(neg_test))

	with open('vocab.txt','w') as f: # modified so it don't print a empty line in the last.
		f.write("%s" % vocab[0])
		for word in vocab[1:]:
			f.write("\n%s" % word)
	# print("Vocabulary (training set):", len(vocab))

	print("Learning Training Data, at time ", format((time.time()-t0),".2f"), "sec")

	posidict, negadict = train(pos_train, neg_train)

	positest = toCounterDictList(pos_test)
	negatest = toCounterDictList(neg_test)

	truePositive = 0
	trueNegative = 0
	falsePositive = 0
	falseNegative = 0

	# i = 0
	print("Testing Positive, at time ", format((time.time()-t0),".2f"), "sec")
	for posiinstance in positest:
		ppositive = probilityof(pos_train,neg_train,posidict,negadict,vocab,'positive',posiinstance,logbool,laplacesmooth,smoothconst)
		pnegative = probilityof(pos_train,neg_train,posidict,negadict,vocab,'negative',posiinstance,logbool,laplacesmooth,smoothconst)
		if ppositive > pnegative:
			truePositive += 1
		else:
			falseNegative += 1

	print("Testing Negative, at time ", format((time.time()-t0),".2f"), "sec")
	for negainstance in negatest:
		ppositive = probilityof(pos_train,neg_train,posidict,negadict,vocab,'positive',negainstance,logbool,laplacesmooth,smoothconst)
		pnegative = probilityof(pos_train,neg_train,posidict,negadict,vocab,'negative',negainstance,logbool,laplacesmooth,smoothconst)
		if ppositive < pnegative:
			trueNegative += 1
		else:
			falsePositive += 1

	acc = accuracy(truePositive,trueNegative,falsePositive,falseNegative)
	pre = precision(truePositive,trueNegative,falsePositive,falseNegative)
	rec = recall(truePositive,trueNegative,falsePositive,falseNegative)
	f = fscore(truePositive,trueNegative,falsePositive,falseNegative,1)

	print("Total Time Cost is ", format((time.time()-t0),".2f"), "sec")
	print("Accuarcy  is: ", format(acc,".6f"))
	print("Precision is: ",format(pre,".6f"))
	print("Recall    is: ",format(rec,".6f"))
	print("F-Score   is: ", format(f,".6f"))

	return truePositive,trueNegative,falsePositive,falseNegative

def fixdatabayes(pos_train, neg_train, pos_test, neg_test, vocab, laplacesmooth:bool=True, logbool:bool=True, smoothconst:float=1):

	t0 = time.time()

	# print("Learning Training Data, at time 0.00 sec")

	posidict, negadict = train(pos_train, neg_train)

	positest = toCounterDictList(pos_test)
	negatest = toCounterDictList(neg_test)

	truePositive = 0
	trueNegative = 0
	falsePositive = 0
	falseNegative = 0

	# i = 0
	# print("Testing Positive, at time ", format((time.time()-t0),".2f"), "sec")
	for posiinstance in positest:
		ppositive = probilityof(pos_train,neg_train,posidict,negadict,vocab,'positive',posiinstance,logbool,laplacesmooth,smoothconst)
		pnegative = probilityof(pos_train,neg_train,posidict,negadict,vocab,'negative',posiinstance,logbool,laplacesmooth,smoothconst)
		if ppositive > pnegative:
			truePositive += 1
		else:
			falseNegative += 1

	# print("Testing Negative, at time ", format((time.time()-t0),".2f"), "sec")
	for negainstance in negatest:
		ppositive = probilityof(pos_train,neg_train,posidict,negadict,vocab,'positive',negainstance,logbool,laplacesmooth,smoothconst)
		pnegative = probilityof(pos_train,neg_train,posidict,negadict,vocab,'negative',negainstance,logbool,laplacesmooth,smoothconst)
		if ppositive < pnegative:
			trueNegative += 1
		else:
			falsePositive += 1

	acc = accuracy(truePositive,trueNegative,falsePositive,falseNegative)
	pre = precision(truePositive,trueNegative,falsePositive,falseNegative)
	rec = recall(truePositive,trueNegative,falsePositive,falseNegative)
	f = fscore(truePositive,trueNegative,falsePositive,falseNegative,1)

	# print("Total Time Cost is ", format((time.time()-t0),".2f"), "sec")
	# print("Accuarcy  is: ", format(acc,".6f"))
	# print("Precision is: ",format(pre,".6f"))
	# print("Recall    is: ",format(rec,".6f"))
	# print("F-Score   is: ", format(f,".6f"))

	return truePositive,trueNegative,falsePositive,falseNegative

if __name__=="__main__":
	naive_bayes(0.1,0.1,0.03,0.03)

```


```python
import os
import sys
import glob
import operator as op
import itertools as it
from functools import reduce, partial
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context("notebook", font_scale=1.5)
%matplotlib inline

from sympy import symbols, hessian, Function, N
```

## Univariate Optimization

### Interval elimination

#### Bracketing

To bracket a minimum, we need 3 points such that $a \le b \le c$ and $f(b) \le f(a)$, $f(b \le f(c)$. Interval elimination is the minimization analog of bisection. One famous method we will describe in class is Golden Section search.

#### Golden section search

Let the relative length of $b$ from $a$ be $w$. That is

$$
b - a  = w \\
c - a  = 1
$$

We choose a new point $x$ between $a$ and $b$ - Suppose $x$ lies a distance $z$ beyond $b$.

If $f(x) > 4(b)$, the next bracket is $(a, b, x)$; otherwise the next bracket is $(b, x, c)$.

How do we choose the distance $z$?

There are two possible values for the next bracket length $s$, $s = w + z$ or $s = 1 - w$.

Since we have no information, we make them equal giving $z = 1 - 2w$.

By a self-similarity argument, we also know that $\frac{z}{1-w} = w$.

Eliminating $z$ from these equations gives $w^2 - 3w + 1 = 0$.

#### Interpolation

Near a minimum, Taylor series tells us that the function will look like a quadratic. So quadratic interpolation is a reasonable approach. For a quadratic $ax^2 + bx + c = 0$, the minimum is given by $\frac{-b}{2a}$. Given 3 points on the function, we can solve for $a, b, c$ as a linear system, and set the next point to be $x = \frac{-b}{2a}$.

#### Newton method

Differentiating the Taylor series with only linear terms for $f$ and setting $\frac{df}{dx} = 0$, we get

$$
x_{k+1} = x_{k} - \frac{f'(x)}{f''(x)}
$$

Derivatives can be expensive to calculate, so they may be replaced by finite approximations - these are called Quasi-Newton methods.

Alternatively, the regula falsi method replaces the second derivative with the secant approximation of the first derivative, where the two points are chosen so that the first derivative have different signs - i.e. they bracket a critical point.

# Algorithms for Optimization and Root Finding for Multivariate Problems

**Note**: much of the following notes are taken from Nodecal, J., and S. J. Wright. "Numerical optimization." (2006). It is available online via the library.

## Convexity

A subset $A\subset \mathbb{R}^n$ is *convex* if for any two points $x,y\in A$, the line segment:

$$tx + (1-t)y \;\;\;\;\;\; t\in [0,1]$$
is also in $A$

A function $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is *convex* if its domain $D$ is a convex set and for any two points $x,y\in D$, the graph of $f$ (a subset of $\mathbb{R}^{n+1})$ lies below the line:

$$tf(x) + (1-t)f(y)\;\;\;\;\;t\in [0,1]$$

i.e.
$$f(tx+(1-t)y) \leq tf(x) + (1-t)f(y)\;\;\;\;\;t\in [0,1]$$

#### Convexity guarantees that if an optimizer converges, it converges to the global minimum.

Luckily, we often encounter convex problems in statistics.

## Line Search Methods

There are essentially two classes of multivariate optimization methods. We'll cover line search methods, but refer the reader to Nodecal and Wright for discussion of 'trust region methods'. We should note that all of these methods require that we are 'close' to the minimum (maximum) we are seeking, and that 'noisy' functions or ill-behaved functions are beyond our scope.

A line search method is exactly as it sounds - we search on a line (in $n$ dimensional space) and try to find a minimum.  We start with an initial point, and use an iterative method:

$$x_{k+1} = x_k + \alpha_k p_k$$

where $\alpha_k$ is the *step size* and $p_k$ is the search direction. These are the critical choices that change the behavior of the search.

#### Step Size

Ideally, (given a choice of direction, $p_k$) we would want to minimize:

$$\varphi(\alpha) = f(x_k + \alpha p_k)$$

with respect to $\alpha$. This is usually computationally intensive, so in practice, a sequence of $\alpha$ candidates are generated, and then the 'best' is chosen according to some 'conditions' (e.g. see Wolfe conditions below). We won't be going into detail regarding these. The important thing to know is that they ensure that $f$ decreases sufficiently, according to some conditions. Interested students should see Nodecal

##### Wolfe conditions

- Condition 1: $f(x_k + \alpha_k p) \le f(x_k) + c_1 p_k^T \nabla f(x_k)$

Function is decreased sufficiently after step.

- Condition 2: $p_k^T \nabla f(x_k + \alpha_k p) > c_2 p_k \nabla f(x_k)$$

Slope is decreased sufficiently after step.

## Steepest Descent

In steepest descent, one chooses $p_k=\nabla f_k = \nabla f(x_k)$. It is so named, because the gradient points in the direction of steepest ascent, thus, $-\nabla f_k$ will point in the direction of steepest descent. We'll consider this method in its ideal case, that of a quadratic:

$$f(x) = \frac12 x^TQx - b^Tx$$

where $Q$ is positive-definite and symmetric. Note that:

$$\nabla f = Qx -b$$

so the minimum occurs at $x$ such that

$$Qx= b$$

Clearly, we can solve this easily, but let's walk through the algorithm and first find the (ideal) step length:

$$f(x_k - \alpha \nabla f_k) = \frac12\left(x_k - \alpha \nabla f_k\right)^TQ\left(x_k - \alpha \nabla f_k\right) - b^T \left(x_k - \alpha \nabla f_k\right) $$

If we differentiate this with respect to $\alpha$ and find the zero, we obtain:

$\alpha_k = \frac{\nabla f_k^T\nabla f_k}{\nabla f_k^TQ\nabla f_k}$

Thus,

$$x_{k+1} = x_k - \frac{\nabla f_k^T\nabla f_k}{\nabla f_k^TQ\nabla f_k} \nabla f_k$$

But we know that $\nabla f_k = Qx_k -b$, so we have a closed form solution for $x_{k+1}$. This allows us to compute an error bound. Again, details can be found in the text, but here is the result:

$$||x_{k+1} - x^*||_Q^2 \leq \left(\frac{\lambda_n - \lambda_1}{\lambda_n+\lambda_1}\right)^2 ||x_{k} - x^*||_Q^2$$

where $0<\lambda_1\leq ... \leq \lambda_n$ and $x^*$ denotes the minimizer.

Now, if $\lambda_1=...=\lambda_n = \lambda$, then $Q=\lambda I$, the algorithm converges in one step. Geometrically, the contours are ellipsoids, the value of $\frac{\lambda_n}{\lambda_1}$ elongates the axes and causes the steps to 'zig-zag'. Because of this, convergence slows as $\frac{\lambda_n}{\lambda_1}$ increases.

## Newton's Method

Newton's method is another line-search, and here

$$p_k = -H^{-1}\nabla f_k$$

Note that if the Hessian is not positive definite, this may not always be a descent direction.

In the neighborhood of a local minimum, the Hessian *will* be positive definite. Now, if $x_0$ is 'close enough' to the minimizer $x^*$, the step size $\alpha_k =1$ gives quadratic convergence.

The advantage of multiplying the gradient by the inverse of the Hessian is that the gradient is corrected for curvature, and the new direction points toward the minimum.

```python

#def Quad(x):
#    return (x[1:])*np.sin(x[:-1])**2.0)

#def DQuad(x,y):
#    return (np.array([np.cos(x)*np.sin(y)**2.0,2.0*np.sin(x)*np.cos(y)**2.0]))

def Quad(x):
    return ((x[1:])**2.0 + 5*(x[:-1])**2.0)

def DQuad(x,y):
    return (np.array([2.0*x,10.0*y]))
```

```python
x = np.linspace(-20,20, 100)
y = np.linspace(-20,20, 100)
X, Y = np.meshgrid(x, y)
Z = Quad(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
Hinv=-np.array([[0.5,0],[0,0.1]])
```

```python
plt.figure(figsize=(12,4))
plt.subplot(121)

plt.contour(X,Y,Z);
plt.title("Steepest Descent");
step=-0.25
X0 = 10.0
Y0 = 1.0

Ngrad=Hinv.dot(DQuad(X0,Y0))

sgrad = step*DQuad(X0,Y0)
plt.quiver(X0,Y0,sgrad[0],sgrad[1],color='red',angles='xy',scale_units='xy',scale=1);
X1 = X0 + sgrad[0]
Y1 = Y0 + sgrad[1]
sgrad = step*DQuad(X1,Y1)
plt.quiver(X1,Y1,sgrad[0],sgrad[1],color='green',angles='xy',scale_units='xy',scale=1);
X2 = X1 + sgrad[0]
Y2 = Y1 + sgrad[1]
sgrad = step*DQuad(X2,Y2)
plt.quiver(X2,Y2,sgrad[0],sgrad[1],color='purple',angles='xy',scale_units='xy',scale=1);

plt.subplot(122)
plt.contour(X,Y,Z);
plt.title("Newton's Method")
plt.quiver(X0,Y0,Ngrad[0],Ngrad[1],color='purple',angles='xy',scale_units='xy',scale=1);

#Compute Hessian and plot again.

```

## Coordinate Descent

Another method is called 'coordinate' descent, and it involves searching along coordinate directions (cyclically), i.e.:

$$p_{mk} = e_{k} \;\;\;\;\;\; k=1,...,n$$

where $m$ is the number of steps.

The main advantage is that $\nabla f$ is not required. It can behave reasonably well, if coordinates are not tightly coupled.

### Newton CG Algorithm

Features:

* Minimizes a 'true' quadratic on $\mathbb{R}^n$ in $n$ steps
* Does NOT require storage or inversion of an $n \times n$ matrix.

We begin with $:\mathbb{R}^n\rightarrow \mathbb{R}$.  Take a quadratic approximation to $f$:

$$f(x) \approx \frac12 x^T H x + b^Tx + c$$

Note that in the neighborhood of a minimum, $H$ will be positive-definite (and symmetric). (If we are maximizing, just consider $-H$).

This reduces the optimization problem to finding the zeros of

$$Hx = -b$$

This is a linear problem, which is nice.  The dimension $n$ may be very large - which is not so nice.

#### General Inner Product

Recall the axiomatic definition of an inner product $<,>_A$:

* For any two vectors $v,w$ we have
$$<v,w>_A = <w,v>_A$$
* For any vector $v$
$$<v,v>_A \;\geq 0$$
   with equality $\iff$ $v=0$.
* For $c\in\mathbb{R}$ and $u,v,w\in\mathbb{R}^n$, we have
$$<cv+w,u> = c<v,u> + <w,u>$$

These properties are known as symmetric, positive definite and bilinear, respectively.

Fact: If we denote the standard inner product on $\mathbb{R}^n$ as $<,>$ (this is the 'dot product'), any symmetric, positive definite $n\times n$ matrix $A$ defines an inner product on $\mathbb{R}^n$ via:

$$<v,w>_A \; = <v,Aw> = v^TAw$$

Just as with the standard inner product, general inner products define for us a notion of 'orthogonality'. Recall that with respect to the standard product, 2 vectors are orthogonal if their product vanishes. The same applies to $<,>_A$:

$$<v,w>_A = 0 $$

means that $v$ and $w$ are orthogonal under the inner product induced by $A$.  Equivalently, if $v,w$ are orthogonal under $A$, we have:

$$v^TAw = 0$$

This is also called *conjugate* (thus the name of the method).

#### Conjugate Vectors

Suppose we have a set of $n$ vectors $p_1,...,p_n$ that are mutually conjugate.  These vectors form a basis of $\mathbb{R}^n$.  Getting back to the problem at hand, this means that our solution vector $x$ to the linear problem may be written as follows:

$$x = \sum\limits_{i=1}^n \alpha_i p_i$$

So, finding $x$ reduces to finding a conjugate basis and the coefficients for $x$ in that basis.

If we let $A=H$,note that:

$${p}_k^{T} {-b}={p}_k^{T} {A}{x}$$

and because $x = \sum\limits_{i=1}^n \alpha_i p_i$, we have:
$$p^TAx = \sum\limits_{i=1}^n \alpha_i p^TA p_i$$

we can solve for $\alpha_k$:

$$\alpha_k = \frac{{p}_k^{T}{(-b)}}{{p}_k^{T} {A}{p}_k} = -\frac{\langle {p}_k, {b}\rangle}{\,\,\,\langle {p}_k,  {p}_k\rangle_{A}} = -\frac{\langle{p}_k, {b}\rangle}{\,\,\,\|{p}_k\|_{A}^2}.$$

Now, all we need are the $p_k$'s.

A nice initial guess would be the gradient at some initial point $x_1$.  So, we set $p_1 = \nabla f(x_1)$. Then set:

$$x_2 = x_1 + \alpha_1p_1$$

This should look familiar.  In fact, it is gradient descent.  For $p_2$, we want $p_1$ and $p_2$ to be conjugate (under $A$).  That just means orthogonal under the inner product induced by $A$.  We set

$$p_2 = \nabla f(x_2) - \frac{p_1^TA\nabla f(x_2)}{{p}_1^{T}{A}{p}_1} {p}_1$$

I.e. We take the gradient at $x_1$ and subtract its projection onto $p_1$.  This is the same as Gram-Schmidt orthogonalization.

The $k^{th}$ conjugate vector is:

$$p_{k} = \nabla f(x_k) - \sum\limits_{i=1}^{k-1}\frac{p_i^T A \nabla f(x_k)}{p_i^TAp_i} p_i$$

The 'trick' is that in general, we do not need all $n$ conjugate vectors. In fact, it turns out that $\nabla f(x_k) = b-Ax_k$ is conjugate to all the $p_i$ for $i=1,...,k-2$. Therefore, we need only the last term in the sum.

Convergence rate is dependent on sparsity and condition number of $A$.  Worst case is $n^2$.

### BFGS - Broyden–Fletcher–Goldfarb–Shanno

BFGS is a 'quasi' Newton method of optimization.  Such methods are variants of the Newton method, where the Hessian $H$ is replaced by some approximation.  We we wish to solve the equation:

$$B_k{p}_k = -\nabla f({x}_k)$$

for $p_k$. This gives our search direction, and the next candidate point is given by:

$$x_{k+1} = x_k + \alpha_k p_k$$.

where $\alpha_k$ is a step size.

At each step, we require that the new approximate $H$ meets the secant condition:

$$B_{k+1}(x_{k+1}-x_k) = \nabla f(x_{k+1}) -\nabla f(x_k)$$

There is a unique, rank one update that satisfies the above:

$$B_{k+1} = B_k + c_k v_kv_k^T$$

where

$$ c_k = -\frac{1}{\left(B_k(x_{k+1}-x_k) - (\nabla f(x_{k+1})-\nabla f(x_k)\right)^T (x_{k+1}-x_k) }$$

and

$$v_k = B_k(x_{k+1}-x_k) - (\nabla f(x_{k+1})-\nabla f(x_k))$$

Note that the update does NOT preserve positive definiteness if $c_k<0$.  In this case, there are several options for the rank one correction, but we will not address them here.  Instead, we will describe the BFGS method, which almost always guarantees a positive-definite correction.  Specifically:

$$B_{k+1} = B_k + b_k g_k g_k^T + c_k B_k d_k d_k^TB_k$$

where we have introduced the shorthand:

$$g_k = \nabla f(x_{k+1}) - \nabla f(x_k) \;\;\;\;\;\;\;\ \mathrm{ and }\;\;\;\;\;\;\; d_k = x_{k+1} - x_k$$

If we set:

$$b_k = \frac{1}{g_k^Td_k} \;\;\;\;\; \mathrm{ and } \;\;\;\;\; c_k = \frac{1}{d_k^TB_kd_k}$$

we satisfy the secant condition.

### Nelder-Mead Simplex

While Newton's method is considered a 'second order method' (requires the second derivative), and quasi-Newton methods are first order (require only first derivatives), Nelder-Mead is a zero-order method.  I.e. NM requires only the function itself - no derivatives.

For $f:\mathbb{R}^n\rightarrow \mathbb{R}$, the algorithm computes the values of the function on a simplex of dimension $n$, constructed from $n+1$ vertices.  For a univariate function, the simplex is a line segment.  In two dimensions, the simplex is a triangle, in 3D, a tetrahedral solid, and so on.

The algorithm begins with $n+1$ starting points and then the follwing steps are repeated until convergence:

* Compute the function at each of the points
* Sort the function values so that
$$f(x_1)\leq ...\leq f(x_{n+1})$$
* Compute the centroid $x_c$ of the n-dimensional region defined by $x_1,...,x_n$
* Reflect $x_{n+1}$ about the centroid to get $x_r$
$$x_r = x_c + \alpha (x_c - x_{n+1})$$
* Create a new simplex according to the following rules:
    - If $f(x_1)\leq f(x_r) < f(x_n)$, replace $x_{n+1}$ with $x_r$
    - If $f(x_r)<f(x_1)$, expand the simplex through $x_r$:
    $$x_e = x_c + \gamma (x_c - x_{n+1})$$
      If $f(x_e)<f(x_r)$, replace $x_{n+1}$ with $x_e$, otherwise, replace $x_{n+1}$ with $x_r$
    - If $f({x}_{r}) \geq f({x}_{n})$, compute $x_p = x_c + \rho(x_c - x_{n+1})$.  If $f({x}_{p}) < f({x}_{n+1})$, replace $x_{n+1}$ with $x_p$
    - If all else fails, replace *all* points except $x_1$ according to
    $$x_i = {x}_{1} + \sigma({x}_{i} - {x}_{1})$$

The default values of $\alpha, \gamma,\rho$ and $\sigma$ in scipy are not listed in the documentation, nor are they inputs to the function.

### Powell's Method

Powell's method is another derivative-free optimization method that is similar to conjugate-gradient.  The algorithm steps are as follows:

Begin with a point $p_0$ (an initial guess) and a set of vectors $\xi_1,...,\xi_n$, initially the standard basis of $\mathbb{R}^n$.

- Compute for $i=1,...,n$, find $\lambda_i$ that minimizes $f(p_{i-1} +\lambda_i \xi_i)$ and set $p_i = p_{i-1} + \lambda_i\xi_i$
- For $i=1,...,n-1$, replace $\xi_{i}$ with $\xi_{i+1}$ and then replace $\xi_n$ with $p_n - p_0$
- Choose $\lambda$ so that $f(p_0 + \lambda(p_n-p_0)$ is minimum and replace $p_0$ with $p_0 + \lambda(p_n-p_0)$

Essentially, the algorithm performs line searches and tries to find fruitful directions to search.

## Solvers

### Levenberg-Marquardt (Damped Least Squares)

Recall the least squares problem:

Given a set of data points $(x_i, y_i)$ where $x_i$'s are independent variables (in $\mathbb{R}^n$ and the $y_i$'s are response variables (in $\mathbb{R}$), find the parameter values of $\beta$ for the model $f(x;\beta)$ so that

$$S(\beta) = \sum\limits_{i=1}^m \left(y_i - f(x_i;\beta)\right)^2$$
is minimized.

If we were to use Newton's method, our update step would look like:

$$\beta_{k+1} = \beta_k - H^{-1}\nabla S(\beta_k)$$

Gradient descent, on the other hand, would yield:

$$\beta_{k+1} = \beta_k - \gamma\nabla S(\beta_k)$$

Levenberg-Marquardt adaptively switches between Newton's method and gradient descent.

$$\beta_{k+1} = \beta_k - (H + \lambda I)^{-1}\nabla S(\beta_k)$$

When $\lambda$ is small, the update is essentially Newton-Gauss, while for $\lambda$ large, the update is gradient descent.

### Newton-Krylov

The notion of a Krylov space comes from the Cayley-Hamilton theorem (CH).  CH states that a matrix $A$ satisfies its characteristic polynomial.  A direct corollary is that $A^{-1}$ may be written as a linear combination of powers of the matrix (where the highest power is $n-1$).

The Krylov space of order $r$ generated by an $n\times n$ matrix $A$ and an $n$-dimensional vector $b$ is given by:

$$\mathcal{K}_r(A,b) = \operatorname{span} \, \{ b, Ab, A^2b, \ldots, A^{r-1}b \}$$

These are actually the subspaces spanned by the conjugate vectors we mentioned in Newton-CG, so, technically speaking, Newton-CG is a Krylov method.

Now, the scipy.optimize newton-krylov solver is what is known as a 'Jacobian Free Newton Krylov'.  It is a very efficient algorithm for solving *large* $n\times n$ non-linear systems.  We won't go into detail of the algorithm's steps, as this is really more applicable to problems in physics and non-linear dynamics.

## GLM Estimation and IRLS

Recall generalized linear models are models with the following components:

* A linear predictor $\eta = X\beta$
* A response variable with distribution in the exponential family
* An invertible 'link' function $g$ such that
$$E(Y) = \mu = g^{-1}(\eta)$$

We may write the log-likelihood:

$$\ell(\eta) = \sum\limits_{i=1}^m (y_i \log(\eta_i) + (\eta_i - y_i)\log(1-\eta_i) $$

where $\eta_i = \eta(x_i,\beta)$.

Differentiating, we obtain:

$$\frac{\partial L}{\partial \beta} = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta} = 0$$

Written slightly differently than we have in the previous sections, the Newton update to find $\beta$ would be:

$$-\frac{\partial^2 L}{\partial \beta \beta^T} \left(\beta_{k+1} -\beta_k\right) = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta}$$

Now, if we compute:

$$-\frac{\partial^2 L}{\partial \beta \beta^T} = \sum \frac{\partial L}{\partial \eta_i}\frac{\partial^2 \eta_i}{\partial \beta \beta^T} - \frac{\partial \eta}{\partial \beta}^T \frac{\partial^2 L}{\partial \eta \eta^T}  \frac{\partial \eta}{\partial \beta}$$

Taking expected values on the right hand side and noting:

$$E\left(\frac{\partial L}{\partial \eta_i} \right) = 0$$

and

$$E\left(-\frac{\partial^2 L}{\partial \eta \eta^T} \right) = E\left(\frac{\partial L}{\partial \eta}\frac{\partial L}{\partial \eta}^T\right) \equiv A$$

So if we replace the Hessian in Newton's method with its expected value, we obtain:

$$\frac{\partial \eta}{\partial \beta}^TA\frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right) = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta} $$

Now, these actually have the form of the normal equations for a weighted least squares problem.

$$\min_{\beta_{k+1}}\left(A^{-1}\frac{\partial L}{\partial \eta} + \frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right)\right)^T A \left(A^{-1}\frac{\partial L}{\partial \eta} + \frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right)\right)$$

$A$ is a weight matrix, and changes with iteration - thus this technique is *iteratively reweighted least squares*.

### Constrained Optimization and Lagrange Multipliers

Often, we want to optimize a function subject to a constraint or multiple constraints.  The most common analytical technique for this is called 'Lagrange multipliers'.  The theory is based on the following:

If we wish to optimize a function $f(x,y)$ subject to the constraint $g(x,y)=c$, we are really looking for points at which the gradient of $f$ and the gradient of $g$ are in the same direction.  This amounts to:

$$\nabla_{(x,y)}f = \lambda \nabla_{(x,y)}g$$

(often, this is written with a (-) sign in front of $\lambda$). The 2-d problem above defines two equations in three unknowns.  The original constraint, $g(x,y)=c$ yields a third equation.  Additional constraints are handled by finding:

$$\nabla_{(x,y)}f = \lambda_1 \nabla_{(x,y)}g_1 + ... + \lambda_k \nabla_{(x,y)}g_k$$

The generalization to functions on $\mathbb{R}^n$ is also trivial:

$$\nabla_{x}f = \lambda \nabla_{x}g$$

```python

```


```python
import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interactive
import sympy as sp
import math
import sys
from matplotlib.animation import FuncAnimation
from IPython import display
```

```python
def trajectory(theta,velo,height,acceleration):

  theta=theta*np.pi/180
  u=velo
  h=height
  accel=acceleration

  d= sp.Symbol('d')
  k=d*np.tan(theta)-accel*d**2/(2*u**2*(np.cos(theta))**2)+h
  m=sp.solve(k)
  m[0]=float(m[0])
  m[1]=float(m[1])

  xlim=0
  if len(m)==2:
    if m[1]>1:
       xlim=np.ceil(m[1])
    else:
       y=round(m[1],3)
       xlim=y+0.01*y
  else:
    xlim=np.ceil(m[0])

  x=np.linspace(0,1.1*xlim,100)

  f=lambda theta,u,h,accel:x*np.tan(theta)-accel*x**2/(2*u**2*(np.cos(theta))**2)+h
  t=(u*np.sin(theta)+np.sqrt((u*np.sin(theta))**2+2*accel*h))/accel

  x_1=np.tan(theta)*(2*u**2*(np.cos(theta))**2)/(2*accel)

  if x_1<0:
    x_1=0

  if theta>=0:
      h_max=(u*np.sin(theta))**2/(2*accel)+h
  else:
      h_max=h+0.0
  y_1=h_max

  time=round(t,3)

  if len(m)==2:
     range=round(m[1],3)
  else:
     range=round(m[0],3)
  height=round(h_max,3)
  ylim=h_max+0.1*h_max

  fig2=plt.figure(figsize=(10,10),facecolor='white')
  plt.plot(x,f(theta,u,h,accel),'k--')
  plt.grid()
  plt.ylabel("Height [m]")
  plt.xlabel("Range [m]")
  plt.axis('scaled')
  plt.ylim(top=int(ylim),bottom=0)
  plt.xlim(0,int(xlim))
  #plt.xticks(np.arange(0,int(xlim),step=1.0))
  #plt.yticks(np.arange(0,int(ylim),step=1.0))

  points=plt.plot([],'ro')
  point=points[0]
  time_text=plt.title('',fontsize=18,loc='left')

  def marker(b):
    return(b*np.tan(theta)-accel*b**2/(2*u**2*(np.cos(theta))**2)+h)

  def animate(frame):
    if frame<int(np.ceil((t)*50)):
      time=frame*20/1000
      time_text.set_text("Time[s] ={}".format(time))
      g=u*np.cos(theta)*time
      #print("frame {}".format(frame))
      point.set_data((g,marker(g)))

  print("Number of frames {}".format(int(np.ceil((t+1)*50))))
  k_y=input("Type 'yes' to animate (for frames>1000 it can take anywhere from several minutes to hours for animation to render): ")
  if k_y=="yes":
    anim =FuncAnimation(fig2,animate,frames=int(np.ceil((t+1)*50)),interval=20)
    video=anim.to_html5_video()
    html=display.HTML(video)
    display.display(html)

    plt.close()
```

```python
v_2=float(input("Enter angle of projection in degrees:"))
v_3=float(input("Enter projection velocity in m/s:"))
v_4=float(input("Enter initial elevation in m:"))
v_5=float(input("Enter downward acceleration in m/s^2:"))
trajectory(v_2,v_3,v_4,v_5)
```


## RKR Computation
All Code and Markdown written by Gary Zeri, Chapman University Student and member of the LaRue Cat Lab

All equations and information within this notebook originated from The Computation of RKR Potential Energy Curves of Diatomic Molecules using Matematica, written by Peter Senn.

The RKR method is a procedure used to determine the potential energy curves of diatomic molecules by calculating the classical turning points, $r_-$ and $r_+$ from the molecule's diatomic constants.

The turning points, $r_{\pm}$, can be computed using the following equation:
$$r_{\pm} = \frac{f(v)}{2} \cdot \left[\sqrt{1 + \frac{1}{f(v)g(v)}} \pm 1\right]$$

It should be noted that in the above equation the fraction in the square root has a one in the numerator while in the paper by Peter Senn the numerator has a four. The four is not present in the Matematica code in Senn's paper and in Lecture #21: Construction of Potential Curves by the Rydberg-Klein-Rees Method (RKR) from MIT OpenCourseWare, the four is not included as well, leading to the omission of the four in the equation presented here to provide an exact equation to that used within the below code.

$f(v)$ and $g(v)$ are defined as follows:
$$f(v) = \int^v_{\frac{-1}{2}}{ \left[ \frac{dv^{'}}{\sqrt{E(v)-E(v^{'})}{}} \right] }$$

$$g(v) = \int_{\frac{-1}{2}}^v{\left[ \frac{\frac{\partial E}{\partial J}}{\sqrt{E(v) - E(v^{'})}}\right]dv^{'}}$$

In the original paper by Senn, there are constants in front of the integrals that are omitted here as they are used to scale the values to a desired unit system. Since this RKR code will use wavenumbers for energy and angstroms for bond distance, the constant of $\frac{8.211609}{\sqrt{\mu}}$ will be multiplied by the final answers to perform the conversion to Hartrees as an energy value.

For vibronic states with no rotation and $J=0$, $E(v)$ and $\frac{\partial E}{\partial J}$ can be represented as:
$$E(v) = \omega_e(v+ \frac{1}{2}) - \omega_ex_e(v+\frac{1}{2})^2 + \omega_ey_e(v+\frac{1}{2})^3 + \omega_ez_e(v+\frac{1}{2})^4 + \dots$$

$$\frac{\partial E}{\partial J} = B(v) = B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2 + \dots$$

An important computational issue to note with the RKR method is that the integrand term $\frac{dv^{'}}{\sqrt{E(v) - E(v^{'})}}$ will become one over zero since the integration is from $\frac{-1}{2}$ to $v$, thus the integrand will become $\frac{1}{\sqrt{E(v) - E(v)}}$ when $v^{'} = v$ at the end of the integration.

In order to deal with this issue, the integral is changed from $\int^{v}_{\frac{-1}{2}}$ to $\int^{v-\delta}_{\frac{-1}{2}}$, where $\delta$ is some extremely small value. A correction term is then added to both $f(v)$ and $g(v)$ to account for the missing endpoint of the integration resulting in new and more easily computable versions of $f(v)$ and $g(v)$:

$$f(v) = \int^{v-\delta}_{\frac{-1}{2}}{\frac{dv^{'}}{\sqrt{E(v) - E(v^{'})}} + \sqrt{\frac{\delta}{Q_v}}}$$

$$g(v) = \int^{v-\delta}_{\frac{-1}{2}}{\frac{B(v^{'})}{\sqrt{E(v)-E(v^{'})}}dv^{'} } + 2B(v)\sqrt{\frac{\delta}{Q_v}} $$

Where $Q_v$ is the following series:
$$Q_v = w_e-2\omega_ex_e(v+\frac{1}{2}) + 3\omega_ey_e(v+\frac{1}{2})^2 + 4\omega_ez_e(v+\frac{1}{2})^3 + \dots$$

The following code only uses the terms listed above in the computation for all series listed.

## RKR With Excited States

In order to allow for RKR computations with excited states, $E(v)$ and $\frac{\partial E}{\partial J}$ must include the $J$ term. The full $E(v)$ equation is included below:
\begin{align}
    E(v) &= \omega_e(v+\frac{1}{2}) - \omega_ex_e(v+\frac{1}{2})^2 + \omega_ey_e(v+\frac{1}{2})^3 + \omega_ez_e(v+\frac{1}{2}) + \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot J(J+1)-D_eJ^2(J+1)^2 \\
    E(v) &= c(v) + \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot J(J+1)-D_eJ^2(J+1)^2 \\
    E(v) &= c(v) + f(v, J) - g(J)
\end{align}

\begin{align}
    \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}\left[ c(v) + f(v, J) - g(J)\right] \\
    \frac{\partial E}{\partial J} &= 0 + \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
    \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
\end{align}

\begin{align}
    \frac{\partial }{\partial J}f(v, J) &= \frac{\partial }{\partial J}\left[d(v)\cdot h(J)\right] \\
    &= d(v) \frac{\partial}{\partial J}h(v) \\
    &= d(v) \frac{\partial}{\partial J}\left[J(J+1)\right] \\
    &= d(v) \cdot \left(2J + 1\right) \\
    &= \left[B_e-\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] \cdot \left(2J + 1\right) \\
    &= B_e \cdot \left(2J + 1\right) - \alpha_e(v+\frac{1}{2})\left(2J + 1\right) + y_e(v+\frac{1}{2})^2 \left(2J + 1\right) \\
    &= 2JB_e + B_e -2J\alpha_e(v+\frac{1}{2}) -2\alpha_e(v+\frac{1}{2} ) + 2Jy_e(v+\frac{1}{2})^2 + y_e(v+\frac{1}{2})^2 \\
    &= 2J\left[B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right] + B_e - \alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2 \\
    &= 2Jd(v) + d(v) \\
    \frac{\partial}{\partial J}f(v, J) &= d(v)\left(2J + 1\right)
\end{align}

\begin{align}
    \frac{\partial }{\partial J}g(J) &= \frac{\partial}{\partial J} \left[D_e J^2(J+1)^2 \right] \\
        &= D_e \frac{\partial}{\partial J}\left[ J^2(J+1)^2\right] \\
        &= D_e \left[ \frac{\partial}{\partial J}[J^2] \cdot (J+1)^2 + J^2 \cdot \frac{\partial}{\partial J}[(J+1)^2]\right] \\
        &= D_e \left[ 2J \cdot (J+1)^2 + J^2 \cdot 2(J+1)\right] \\
        &= 2J(J+1)D_e \left[ (J+1) + J\right] \\
        &= 2J(J+1)D_e\left[ 2J + 1 \right] \\
    \frac{\partial}{\partial J}g(J) &= 2D_eJ(J+1)(2J+1)
\end{align}

\begin{align}
        \frac{\partial E}{\partial J} &= \frac{\partial}{\partial J}f(v, J) - \frac{\partial}{\partial J}g(J) \\
       \frac{\partial E}{\partial J} &= d(v)(2J + 1) - 2D_eJ(J+1)(2J+1) \\
       \frac{\partial E}{\partial J} &= \left[B_e -\alpha_e(v+\frac{1}{2}) + y_e(v+\frac{1}{2})^2\right](2J+1) - 2D_eJ(J+1)(2J+1)
\end{align}

```python
#Import all needed classes here and create all global variables
#Allow Notebook to Import from Comp_Chem_Package
import sys
sys.path.append("..\\Comp_Chem_Package")

from compChemGlobal import *
import numpy as np
from tqdm import tqdm
from scipy.integrate import quad as integrate

from plotly.offline import iplot, init_notebook_mode
init_notebook_mode(connected=True)

#Should be a very small value
#Is the distance from v that the integration stops at
delta = pow(10, -3)

#Reduced Molecular Mass
#In Non Hartree Atomic Units, each proton has an amu of 1
#u = (12 * 16) / (12+16)
#u = (14 * 16) / (14 + 16)
u = 0.5
#Diatomic Constants, must be in Wavenumbers (1 / cm)
#Currently Has Data for COA
'''
COA
we = 1518.28
wxe = 19.4
wye = 0
wze = 0
Be = 1.6115
alphae = 0.02325
ye = 0
De = 0.00000733
'''
'''
#NOX
we = 1904.20
wxe = 14.075
wye = 0
wze = 0
Be = 1.67195
alphae = 0.0171
ye = 0
De = 0.54 * pow(10, -6)
'''
'''
#NOA
we = 2374.31
wxe = 10.106
wye = -0.0465
wze = 0
Be = 1.9965
alphae = 0.01915
ye = 0
De = 5.4 * pow(10, -6)
'''

#H2X
we = 4401.21
wxe = 121.33
wye = 0
wze = 0
Be = 60.853
alphae = 3.062
ye = 0
De = 0.0471
re = .74144
```

```python
#Allow Notebook to Import from Comp_Chem_Package
import sys
sys.path.append("..\\Comp_Chem_Package")

from compChemGlobal import *
from nistScraper import getDiatomicConstants
import numpy as np
from tqdm import tqdm

#Global variables to be modified by the user

#Distance from v that the integration should stop at
#Should be a very very small number strictly greater than 0
delta = pow(10, -5)
```

```python
#Define All Functions Here

def E(v, J=0):
    term = v + 0.5
    d = (we * term) - (wxe*pow(term, 2)) + (wye*pow(term, 3)) + (wze*pow(term,4))
    return d * ( 2*J + 1) - 2*De*J*(J+1)*(2*J+1)

def B(v):
    term = v + 0.5
    return Be - (alphae * term)  + ye*pow(term, 2)

#Used in the correctionFactor calculation
def Q(v):
    term = v + 0.5
    return we - (2*wxe*term) + (3*wye*pow(term, 2)) + (4*wze*pow(term,3))

#Used to correct integrals that stop delta away from v
def correctionFactor(v):
    return 2 * np.sqrt(delta / Q(v))

def integralRadical(v, vPrime):
    return np.sqrt(E(v) - E(vPrime))

def f(v):
    integrand = lambda vPrime: 1 / integralRadical(v, vPrime)
    return integrate(integrand, -0.5, v-delta) + correctionFactor(v)

def g(v):
    integrand = lambda vPrime : B(vPrime) / integralRadical(v, vPrime)
    return integrate(integrand, -0.5, v-delta) + (B(v)*correctionFactor(v))

#v refers to a float energy level for the potential well
#returns the tuple (r+, r-)
def RKR(v):

    c0 = (8.211609 * f(v) ) / (2 * np.sqrt(u))

    radicand = 1 / (f(v) * g(v))
    c1 = np.sqrt(1 + radicand)

    return c0 * (c1 + 1), c0 * (c1 - 1)
```

```python
#Lists to hold data
x = []
y = []
ddX = []
ddX2 = []
ddx = []
ddx2 = []
leftAsympCutOff = False

x.append(re)
y.append(0)

#Loop to generate all data to graph
print("Building RKR Potential")
for v in tqdm(np.arange(-.499, 17, .01)):

    energy = E(v, 0)
    xPoints = RKR(v)

    if(len(x) != 0):
        #Compute First Derivative
        ddX.append( (min(xPoints) + min(x[-2:])) / 2 )
        slope = (energy-y[-1]) / ( min(xPoints) - min(x[-2:]) )
        ddx.append( slope )

        if(len(ddx) > 1):
            #Compute 2nd Derivative
            ddX2.append( (ddX[-2] + ddX[-1]) / 2 )
            ddx2.append( (ddx[-1] - ddx[-2]) / (ddX[-1] - ddX[-2]) )

            #Determine if Cutoff should be used
            if(ddx2[-1] <= 0):
                leftAsympCutOff = True

    if(not leftAsympCutOff):
        x.extend( xPoints )
        y.extend( [energy] * 2 )
    else:
        x.append(max(xPoints))
        y.append(energy)

print("Graphing Potential")
#Prepare data for graphing
figure = {
    "data":[
        {
            "type":"scatter",
            "x":x,
            "y":y,
            "connectgaps":True,
            "mode":"markers",
            "name":"RKR Potential",
        }
    ],

    "layout":{
        "xaxis":{"title":"Bond Distance non-Hartree Atomic Units"},
        "yaxis":{"title":"Energy in Wavenumbers"},
        "title":{"text":"RKR Generated Potential Energy Curve"}
    }
}

figure["data"].append(
    {
        "type":"scatter",
        "x":ddX,
        "y":ddx,
        "name":"1st Derivative"
    }
)

figure["data"].append(
    {
        "type":"scatter",
        "x":[ddX[ddx.index(min(ddx))]] * 2,
        "y":[0, pow(10, 4) * 2],
        "name":"1st Derivative Turning Point"
    }
)

figure["data"].append(
    {
        "type":"scatter",
        "x":ddX2,
        "y":ddx2,
        "name":"2nd Derivative"
    }
)

iplot(figure)
```

```python
#Test for the RKR Class
from rkr import rkr as rkrClass

rkr = rkrClass()

rkr.setDiatomicConstants(alphae, Be, we, wxe, wye, wze, ye)
rkr.setReducedMass(0.5)
rkr.setDelta(delta)

x, y = rkr.graphData()

figure = {

    "data":[
        {
            "type":"scatter",
            "x":x,
            "y":y,
            "connectgaps":True,
            "mode":"markers",
            "name":"RKR Potential",
        }
    ],

    "layout":{
        "xaxis":{"title":"Bond Distance non-Hartree Atomic Units"},
        "yaxis":{"title":"Energy in Wavenumbers"},
        "title":{"text":"RKR Generated Potential Energy Curve"}
    }
}

iplot(figure)
```

```python
E(900)
```

```python

```


# Polynomial minimization

This examples shows how to solve a constrained polynomial minimization problem. Given p(x) and g(x), consider the problem in `x`

    min  p(x)
    s.t. g(x) >= 0

We can lower bound the optimal value by solving the following problem in `gamma, sigma(x)`

    max  gamma
    s.t. p(x) - gamma - sigma(x) g(x) >= 0                (1)
         sigma(x) >= 0

```python
from sympy.abc import x, y
from posipoly import *

tot_deg = 4        # overall degree of problem
n = 2              # number of variables
pp_cone = 'psd'    # can be 'psd' or 'sdd'

p = Polynomial.from_sympy(-x**2 - y**2 + x, [x,y])
g = Polynomial.from_sympy(1 - x**2 - y**2, [x,y])

sigma_deg = tot_deg - g.d   # degree of sigma

prob = PPP()
prob.add_var('gamma', n, 0, 'coef')
prob.add_var('sigma', n, sigma_deg, 'pp')
prob.add_constraint({'gamma': -PTrans.eye(n, 0, n, tot_deg),
                     'sigma': -PTrans.mul_pol(n, sigma_deg, g)},
                     -p, 'pp')
prob.set_objective({'gamma': [-1]})

_, status = prob.solve(pp_cone)
opt_gamma = prob.get_poly('gamma')(0,0)

print('a lower bound is {:.2f}'.format(opt_gamma))
```

    optimizing...
    solved in 0.02s
    solsta.optimal
    a lower bound is -2.00

```python
# plot polynomial for comparison
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

XX, YY = np.meshgrid(np.linspace(-2, 2, 100), np.linspace(-2, 2, 100))
XX = XX.flatten()
YY = YY.flatten()

VV = np.vectorize(p)(XX, YY)
VV_control = opt_gamma * np.ones(VV.shape)

mask = np.vectorize(g)(XX, YY) >= 0

ax = plt.figure().gca(projection='3d')
ax.plot_trisurf(XX[mask], YY[mask], VV[mask])
ax.plot_trisurf(XX[mask], YY[mask], VV_control[mask])

plt.show()
```

    <IPython.core.display.Javascript object>


[Table of Contents](./table-of-contents.ipynb)

## K-Means
The k-means clustering algorithm
is one of the most popular unsupervised clustering algorithms in machine learning for partitioning data into representative clusters. In this section we first review the k-means problem and  continue with the k-means algorithm as an iterative method to approximate to the solution.
### Story behind:  K-means Problem

The $k$-means algorithm aims to find optimal centroids for different clusters of data that lie in an Euclidean space for a desired number of partitions. We can reformulate the $k$-means problem as an optimization problem as follows:

Given a finite set $\mathbb{X}\subset\mathbb{R}^d$ and a positive integer $k\in \mathbb{Z^+}$, we need to find a function $w\colon \mathbb{X}\to \mathbb{R}^d$ such that $|im(w)|=k$ and the following cost function is minimized:
\begin{equation}
\sum_{x\in \mathbb{X}} \|x-w(x)\|^2.
\end{equation}
Our search space for an optimal solution is the set of all functions $w\colon \mathbb{X}\to \mathbb{R}^d$ with $|im(w)|=k$, i.e. we need

\begin{equation*}
\arg\min_{w} \sum_{x\in \mathbb{X}} \|x-w(x)\|^2.
\end{equation*}

Since the sum of the squared Euclidean distances is minimized when $w$ assigns the mean of the cluster that the data point belongs to, this problem now is equivalent to finding cluster centroids and writing the function $w(x)$ that assigns the centroid of the cluster to each point $x\in\mathbb{X}$.

### Solution: K-means Clustering

The $k$-means clustering algorithm is a multi-pass clustering algorithm, and it requires access to all of the data points multiple times. The algorithm groups the data based on their mutual euclidean distances, and therefore closer data points are more likely to be in the same group. The algorithm takes the mean value of a group as the similarity parameter and  forms clusters by assigning data points based on the closest mean.

Given a data set $\mathbb{X}=\{x^{t}\}_{t=1}^{N}$, and a positive integer $k$, the algorithm starts by randomly selecting $k$ points as the initial cluster centroids.
Then it assigns each data point to its closest cluster centroid.  Let $W=\{w_{1}, \ldots, w_{k} \}$ be the initial cluster centroids then  each data point $x^{t} $ is assigned
 to a cluster based on:

\begin{equation*}
w(x) = \arg\min_{w_i \in W} \|w_i - x^{t}\|^2 .
\end{equation*}

Then the algorithm recalculates the cluster centroids $\{w_j\}_{j=1}^{k}$ as the mean of the data points in the current clusters.

Let $S_i$ be the set of data point assignments for the $i^{th}$ cluster.
Let us define

\begin{equation}
 h_i^{t} =
 \begin{cases}
      1 & \text{ if } i = \arg\min_\ell{\|x^t- w_\ell\|} \\
      0 & \text{ otherwise }
   \end{cases}
\end{equation}

The batch algorithm, $k$-means, updates the centroids as

\begin{equation} \label{eqkmeans:1.3}
w_i
= \frac{\sum_t h_i^{t}x^t}{\sum_t h_i^{t}}
=  \frac{1}{|S_i|} \sum_{x^t \in S_i} x^t
\end{equation}

Data points are reassigned to their new clusters with respect to the new cluster centroids and centroids are updated until the cluster centroids stabilize.

```python
import numpy as np
from sklearn.datasets.samples_generator import make_blobs
import matplotlib.pyplot as plt

#synthetical data generation
points, labels = make_blobs(n_samples=600, centers=2, n_features=2,
                       cluster_std=0.7, random_state=123)

```

```python
def kmeans(points, k, epochs=100, epsilon=1e-9):
    # initialize k centroids randomly from data points
    centroids = points[np.random.choice(np.arange(len(points)), k), :]
    # keep track of the centroids
    trajectories = [centroids]
    for epoch in range(epochs):
        distances = np.sqrt(((points - centroids[:, np.newaxis])**2).sum(axis=2))
        closest = np.argmin(distances, axis=0)
        # assign each data point to the cluster with the closest centroid
        # update the centroid to be the mean of its cluster’s data points
        centroids = np.array([points[closest==k].mean(axis=0) for k in range(centroids.shape[0])])
        trajectories.append(centroids)

        # if centroids do not change then break
        if np.abs((trajectories[epoch]-trajectories[epoch-1])).all() < epsilon:
            break
    return centroids, trajectories

centroids, trj = kmeans(points, k=2)
```

```python
def plot_trajectories(trajectories):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    plt.scatter(points[:, 0], points[:, 1], color='yellow',alpha=0.3,  s=50, cmap='viridis')
    alpha_2 = 0.3
    for i in trj:
        plt.scatter(i[:,0],i[:,1], c='k', alpha=alpha_2, marker='x')
        alpha_2 += 0.1
    plt.title('Trajectories')
    plt.show()

plot_trajectories(trj)
```

#### Convergence
The cost function of $k$-means algorithm is the squared objective function:

\begin{equation}
J(k,W) =\frac{1}{2}
\sum_{t=1}^{N} \sum_{i=1}^{k} h_i^{t}\|w_i -x^{t}\|^2
\end{equation}

The $k$-means algorithm is guaranteed to converge because $J$  is monotonically decreasing, and $J$ is bounded below. However, since $J$ is a non-convex function, it is not guaranteed to converge to its global minimum. In other words, $k$-means can stuck on a local minimum. In other words,
random initialization of the cluster centroids might give different results. In batch processing, a simple strategy to avoid this situation is to run $k$-means many times using different random initial values for the cluster centroids $w_j$, and then pick the one that gives the lowest value for $J$.

## Competitive Neural Networks
Competitive learning is a form of *Hebbian learning* which states if two neurons fire simultaneously then the connections between the neurons selectively strengthened i.e *neurons that fire together, wire together.* In a competitive neural network, the neurons "compete" with each other as the one with the highest activation "wins" and moved towards the incoming data. These networks are mainly used for the online clustering problems.

The input layer contains the input vector $x$, outputs are fully connected and there is no bias unit. In this model, only binary inputs and outputs are considered. Figure below shows the basic structure of a competitive neural network of $k$ perceptrons with recurrent connections at the output. Each unit at the output reinforces its value and tries to suppress the other outputs. Under a suitable arrangement of recurrent weights, the maximum suppresses all the others.

The values of the output units $O_i$ are perceptrons:

\begin{equation*}
O_i = w_i^{T}\cdot x
\end{equation*}

The activation, of the $i^{th}$ output node is just the inner product of the weight vector of the node and the current input vector:

\begin{equation}
O_i=\sum_{j=1}^{d} w_{ij} \cdot x_{j}^{t} =w_i^{T}\cdot x^t = \|w_i\| \cdot \|x^t\| \cdot cos{\theta}
\end{equation}

which is closely related to the difference between the two vectors:

\begin{equation*}
\|w_i -x^t\|^2 = \|w_i\|^2 + \|x^t\|^2 - 2\|w_i\| \cdot \|x^t\|\cdot cos\theta
\end{equation*}

where $\theta$ is the angle between the two vectors.

In a competitive network, only one output unit, called *the winner unit*, can be become active at a time. The winner unit $i^*$ is the only node that fires and it has the largest net input. Hence,  we must have

\begin{equation}
w_{i^*}\cdot x \geq w_{i}\cdot x, \text{ for all } i
\end{equation}

where the winning unit $i^*$ has $O_{i^*}=1$. If we normalize all the weights to $\|w_i\|=1$ for every $i$, since we have

\begin{equation}
\|x-y\|^2=\|x\|^2 - 2 \left<x,y\right> + \|y\|^2
\end{equation}

We see that the winner unit is the one that minimizes the squared distance

\begin{equation}
i^* =  \arg\min_{i=1,\ldots,k} \|w_i - x\|
\end{equation}

where $k$ is the number nodes in the layer.  This means the unit with the minimum Euclidean distance is the same as the one with the maximum dot product. In other words, the winner is the unit with normalized $w$ closest to the input vector $x$.

Assume that we would like to group spherical data points into three clusters that are given as:

```python
import pandas as pd
import math
import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d.axes3d as axes3d
from matplotlib import cm
from sklearn.preprocessing import normalize

def unit_sphere_data_generator(M):
    np.random.seed(56)
    u = np.random.uniform(0,1,M)
    v = np.random.uniform(0,1,M)
    theta = 2 * np.pi * u
    phi = np.arccos((2*v - 1.0))
    c = np.zeros((M,3))
    c[:,0] = np.cos(theta) * np.sin(phi)
    c[:,1] = np.sin(theta) * np.sin(phi)
    c[:,2] = np.cos(phi)
    sphere_points = pd.DataFrame(c)
    # filter if there are any duplicates
    sphere_points_no_duplicates = sphere_points.drop_duplicates()
    generators = sphere_points_no_duplicates.as_matrix()
    return generators

def generate_clusters(num_samples=800):
    points_on_sphere = unit_sphere_data_generator(num_samples)

    distinct_data1 = []
    distinct_data2 = []
    distinct_data3 = []

    for i in points_on_sphere:
        #If all coordinates are positive and above 0.05
        if np.all(i > 0.05):
            distinct_data1.append(i)
        #If all coordinates are negative and belove -0.05
        elif np.all(i < -0.05):
            distinct_data2.append(i)
        #If x-axis negative and the others positive
        #and between provided values
        elif i[0] < 0 and i[1] > 0 and i[1] < 0.8 and i[2] > 0 and i[2] < 0.5:
            distinct_data3.append(i)

    all_data_points = np.vstack((distinct_data1, distinct_data2, distinct_data3))
    return all_data_points

all_data_points = generate_clusters(num_samples=800)
```

    /Users/melodi/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:21: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.

```python
def plot_unit_sphere():
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    plt3d = plt.subplot(projection='3d')
    #plot unit sphere
    u = np.linspace(0,2*np.pi, 100)
    v = np.linspace(0, np.pi, 100)
    x = np.outer(np.cos(u),np.cos(v))
    y = np.outer(np.cos(u),np.sin(v))
    z = np.outer(np.sin(u),np.ones(np.size(v)))
    plt3d.plot_surface(x, y, z, color='w', alpha=0.3)
    return plt3d, (x, y, z)

def plot_data(data):
    # plot unit sphere
    plt3d, _ = plot_unit_sphere()
    #plot data points on the unit sphere
    plt3d.scatter(data[:,0], data[:,1],
    all_data_points[:,2], c='k')
    plt3d.set_xlabel('X axis')
    plt3d.set_ylabel('Y axis')
    plt3d.set_zlabel('Z axis')
    plt3d.view_init(0,-90)
    plt.title('Before Clustering')
    plt.show()

plot_data(all_data_points)
```

We start by assigning three points randomly selected from the data. As we select data points one by one and feed into the network, only one of the weight outputs one and the winner unit updates its value to that point.

```python
def competitive_network(X, k, epochs, eta=0.01):
    """
    X: Input data points
    epochs: number of epochs
    eta: learning rate
    """

    # number of data points
    N = X.shape[0]

    # it's convenient to choose
    # initial k weights randomly from data points
    W = X[np.random.choice(np.arange(len(X)),k ), :]

    # store the weights to plot weights trajectories
    trj = [W]
    X2 = (X**2).sum(axis=1)[:,np.newaxis]

    for epoch in range(epochs):
        assigned_labels = []
        for i in range(N):
            distance = X2[i:i+1].T-2*np.dot(W,X[i:i+1,:].T)
            +(W**2).sum(axis=1)[:,np.newaxis]
            output = (distance == distance.min(axis=0)[np.newaxis,:]).T
            output = output.astype("int")
            assigned_labels.append([X[i:i+1,:],output])
            # output=[1,0,0] if first class is winner and so on.
            # So multiplication with "output" in below,
            # provides update for the winner weight only.

            # weight update
            W += eta*(np.dot(output.T,X[i:i+1,:])
                     -output.sum(axis=0)[:,np.newaxis]*W)

            # normalize the weights
            W = normalize(W, norm='l2', axis=1)

        trj.append(W)
        # if weights doesn't change then break
        if (trj[epoch] == trj[epoch-1]).all():
            break

    return W, trj, assigned_labels

centers, trajectories, labelled_data = competitive_network(all_data_points, k=3, epochs=100, eta=0.01)
```

Shows the weight trajectories formed following each iteration. When the trajectory gets stabilized we have the regions split into three areas as given below.

```python
def plot_trajectories(trajectories):
    # plot unit sphere
    plt3d, (x,y,z) = plot_unit_sphere()
    plt3d.plot_surface(x, y, z, color='w', alpha=0.3)
    # plot weight trajectories
    for i in trajectories:
        plt3d.scatter(i[:,0],i[:,1],i[:,2], c='k', marker='x')
        plt3d.view_init(0,-90)
    plt.title('Weight Trajectories')
    plt.show()

plot_trajectories(trajectories)
```

After this separation, we can use these centers as a decision rule for incoming data and test our results.

```python
def color_classes(labelled_data):
    pink = []
    blue = []
    green = []
    for i in labelled_data:
        if np.all(i[1][0] == [1,0,0]):
            pink.append(i[0][0])
        elif np.all(i[1][0] == [0,1,0]):
            blue.append(i[0][0])
        elif np.all(i[1][0] == [0,0,1]):
            green.append(i[0][0])
    return np.array(pink), np.array(blue), np.array(green)
pink, blue, green = color_classes(labelled_data)

def plot_clusters(pink, blue, green, centers):
    #plot resulted clusters with the last weights
    plt3d, _ = plot_unit_sphere()
    #plot data points
    plt3d.scatter(pink[:,0],pink[:,1],pink[:,2], c='m')
    plt3d.scatter(green[:,0],green[:,1],green[:,2], c='g')
    plt3d.scatter(blue[:,0],blue[:,1],blue[:,2], c='b')
    #plot centers
    plt3d.scatter(centers[:,0],centers[:,1],centers[:,2], c='k', marker='x')
    plt3d.set_xlabel('X axis')
    plt3d.set_ylabel('Y axis')
    plt3d.set_zlabel('Z axis')
    plt3d.view_init(0,-90)
    plt.title('Clustered data with their centers')
    plt.show()

plot_clusters(pink, blue, green, centers)
```


## Document-image binarization using modified Sauvola's algorithm

This binarization technique is an improvement over Sauvola's binarization technique. In this work, we improve the existing Sauvola's binarization technique by preserving more foreground information in the binarized document-images. In order to achieve this, we introduce a confidence score for the background pixels.

In Sauvola's algorithm, a threshold is computed for each pixel using the Eq $\ref{eq:sauvola}$, where, for an input image $I$,  $R=\frac{\textrm{max}(I) - \textrm{min}(I)}{2}$.
\begin{equation}
T_W (p)= m_{W}^{p} \times [1+k\times (\frac{s_{W}^{p}}{R} - 1)]
\label{eq:sauvola} \tag{1}
\end{equation}
The threshold $T$ is computed for each pixel ($p$) based on a window $W$ of size $n \times n$ surrounding it, where $m_{W}^{p}, s_{W}^{p}$ respectively represent mean and standard deviation of $W$ around pixel $p$, and  $k$ lies between $0 \le k \le 1$.

We define a confidence value, $C$, for each pixel $p$ as Eq \ref{eq:conf}, where max($I$) represent maximum value of any pixel of an input image $I$.
\begin{equation}
  C_W(p)  = \left\{
  \begin{array}{l l}
    \frac{I(p) - T_W(p)}{\textrm{max}(I) - T_W(p)} & \quad \textrm{if \textit{$I(p)$} $>$ \textit{$T_W(p)$}}\\
    0 & \quad  \textrm{otherwise}\\
  \end{array} \right.
\label{eq:conf} \tag{2}
\end{equation}

A new image $I_{c}$ corresponding to an input image $I$ is generated using the confidence value given by the  Eq $\ref{eq:conf}$. Later, Eq $\ref{eq:sauvola}$ is used to compute thresholds for the image $I_c$. These thresholds are further used to generate the new binary image.

For details refer to this [paper](https://drive.google.com/file/d/1D3CyI5vtodPJeZaD2UV5wdcaIMtkBbdZ/view?usp=sharing)

This notebook provides an example of how to utilize modified sauvola algorithm for image binarization task.

```python
import numpy as np
import cv2
from matplotlib import pyplot as plt
from skimage.filters import threshold_sauvola
import ModifiedSauvola_Binarization as MSB
```

# Sample Usage

```python

filename = "test_images/image_165.png"
in_color = cv2.imread(filename,1)

plt.imshow(cv2.cvtColor(in_color, cv2.COLOR_BGR2RGB))
plt.title('input')
plt.show()

##########################################################################################
# Executing Sauvola for binarization

win_size = int(0.05*min(in_color.shape[0],in_color.shape[1]))
if (win_size%2==0):
    win_size = win_size + 1
k = 0.5
gray = cv2.cvtColor(in_color, cv2.COLOR_BGR2GRAY)
T = threshold_sauvola(gray, window_size=win_size,k=k)
binary = np.copy(gray)
binary[gray <= T] = 0
binary[gray > T] = 255

plt.imshow(gray,cmap = 'gray')
plt.title('gray')
plt.show()

plt.imshow(binary,cmap = 'gray')
plt.title('sauvola')
plt.show()
##########################################################################################
# Executing modified sauvola for binarization using deafult parameters

binary_default = MSB.SauvolaModBinarization(in_color)

plt.imshow(binary_default,cmap = 'gray')
plt.title('sauvola_mod_default')
plt.show()

##########################################################################################
# Executing modified sauvola for binarization using custom parameters

k1=0.5
k2=0.5
n1 = int(0.04*min(in_color.shape[0],in_color.shape[1]))
if (n1%2==0):
    n1 = n1+1
n2 = int(0.09*min(in_color.shape[0],in_color.shape[1]))
if (n2%2==0):
    n2 = n2+1
binary_nondefault = MSB.SauvolaModBinarization(in_color,n1,n2,k1,k2,default=False)

plt.imshow(binary_nondefault,cmap = 'gray')
plt.title('sauvola_mod_nondefault')
plt.show()

##########################################################################################
```

```python

```


# Analyzing the Solow model

## Introduction

In this project we will examine the standard Solow model with technological growth (A). We solve for steady state and use interactive figures to examine how different parameter values affects the steady state capital. Later, we extend the model by adding the exogenous variable, land, to our model. Here we examine how parameter values affects the steady steate capital-output ratio.

Imports and set magics:

```python
from __future__ import print_function
import numpy as np
from scipy import optimize
import sympy as sm
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2
```

# The standard Solow model

We consider the standard Solow model with a Cobb-Douglas production function:
$$ Y_t=K_t^{\alpha}(A_tL_t)^{1-\alpha}, 0<\alpha<1 $$

* $Y_t$ is GDP
* $K_t$ is capital
* $L_t$ is labor
 * growing with a constant rate of $n$: $L_{t+1}=(1+n)L_t, n\geq0$
* $A_t$ is technology
 * growing with a constant rate of $g$: $A_{t+1}=(1+g)A_t, g\geq0$

Saving is a constant fraction of GDP

$$
S_t = sY_t, 0<s<1
$$

such that capital accumulation is savings and undepreciated capital:

$$
K_{t+1}=S_t+(1-\delta)K_t, 0<\delta<1
$$

We define the technology adjusted capital-labour ratio as

$$\tilde{k_{t+1}}=\frac{K_{t+1}}{L_{t+1}A_{t+1}} $$

The transition equation becomes:
$$
\tilde{k_{t+1}}=\frac{1}{(1+n)(1+g)}*(s\tilde{k_t^\alpha}+(1-\delta)\tilde{k_t})
$$

We know that steady state requires $k_{t+1}=k_t$

## Solving steady state analytical

We will now solve for the techonolgy adjusted capital intensity steady state using sympy

We define our symbols:

```python
k = sm.symbols('\tilde{k_t}')
k1 = sm.symbols('\tilde{k_{t+1}}')
kstar = sm.symbols("k^*")
g = sm.symbols('g')
n = sm.symbols('n')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
s = sm.symbols('s')
```

To find steady state we set
$\tilde{k_{t+1}}=\tilde{k_t}$
in the **Solow** equation

```python
SS_capital = sm.solve((s*kstar**(alpha)-(n+g+delta+n*g)*kstar)/((1+n)*(1+g)),kstar)
sm.Eq(kstar, SS_capital[0])
```

$\displaystyle k^{*} = \left(\frac{\delta + g n + g + n}{s}\right)^{\frac{1}{\alpha - 1}}$

We will now examine the **Solow** diagram and illustrate what happens, when we change the parameters.

## Graphical analysis

First we plot the **Solow** diagram with fixed parameter values

```python
# a. create function for generating solow diagram
def solow(c,alpha,delta,s,n,g):
    '''Function for creating the solow diagram

    input:
    c = capital
    alpha = share of capital
    delta = deprication rate
    s = savings rate
    n = population growth rate
    g = technologic growth rate

    output:
    Depriciation of capital
    Kapital growth
    '''

    #i. create empty lists
    Deprication = []
    Capital = []

    # ii. create loop which adds values for deprication of capital
    for c in range(1,c):
        Deprication.append((n+g+delta+n*g)*c)

    # iii. create loop which adds values for growth of capital
    for c in range(1,c):
        Capital.append(s*(c**alpha))

    # iv. plot in figure
    plt.style.use('seaborn')
    plt.figure(figsize=(5,5))
    plt.plot(Deprication, label = '$(n+g+\delta+n+ng)k_t$', color = 'g')
    plt.plot(Capital, label = '$sk_t^\\alpha$', color = 'r')
    plt.grid(True)
    plt.ylim(0,5)
    plt.xlim(0,c)
    plt.xlabel('$k_t$')
    plt.legend()
    plt.legend()
    return plt.show()

# # b. call function with fixed parameters
solow(200,1/3,0.05,0.101,0.025,0.02)
```

By using the interactive figure below **you** can examine the steady state capital for different values of depreciation, saving, population growth and techology growth.

```python
widgets.interact(solow,
    c = widgets.fixed(200),
    alpha = widgets.fixed(1/3),
    delta = widgets.FloatSlider(description = 'd' , min = 0.01, max = 0.1, step=0.01, value=0.05),
    s = widgets.FloatSlider(description = 's', min=0.1, max=0.9, step = 0.1, value= 0.1),
    n = widgets.FloatSlider(description = 'n', min = 0.01, max = 0.1, step = 0.01, value = 0.025),
    g = widgets.FloatSlider(description = 'g', min = 0.01, max = 0.1, step = 0.01, value = 0.02))
```

    interactive(children=(FloatSlider(value=0.05, description='d', max=0.1, min=0.01, step=0.01), FloatSlider(valu…

    <function __main__.solow(c, alpha, delta, s, n, g)>

The figure shows, that an increase in the savings rate, s, results in an increase in the SS level. With a higher savings rate it is possible to accumulate more capital, since capital next period is equal to the amount of capital last period and the savings from the present period.

From an increase in N, we find a lower SS level. This is due to and increase in the population wears the capital, and thus lower the SS level.

# Model extension

We consider the Solow model with land.

We assume the production function is Cobb-Douglas:
$$Y_t=K_t^{\alpha}(A_tL_t)^{\beta}X^{\kappa}, \alpha>0, \beta>0, \kappa>0, \alpha+\beta+\kappa=1$$
where:
* $Y_t$ is GDP
* $K_t$ is capital
* $L_t$ is labor
 * growing with a constant rate of $n$: $L_{t+1}=(1+n)L_t, n\geq0$
* $A_t$ is technology
 * growing with a constant rate of $g$: $A_{t+1}=(1+g)A_t, g\geq0$
* $X$ is land
 * constant

Saving is a constant fraction of GDP:

$$
S_t = sY_t,\,s\in(0,1)
$$

such that capital accumulation is savings and undepreciated capital:

$$
K_{t+1}=S_t+(1-\delta)K_t, 0<\delta<1
$$

We assume that economy converges towards a steady state in which the capital-output ratio is constant:
$$
z_t=\frac{K_t}{Y_t}=\frac{k_t}{y_t}
$$

The trasition equation is:
$$
z_{t+1}=\frac{1}{((1+g)(1+n))^\beta} [s+z_t(1-\delta)]^{1-\alpha}z_t^{\alpha}
$$

## Steady state
We solve the steady state capital-output ratio **analytical** and **numerical**.

We define our symbols:

```python
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
beta = sm.symbols('beta')
kappa = sm.symbols('kappa')
s = sm.symbols('s')
g = sm.symbols('g')
n = sm.symbols('n')
z = sm.symbols('z')
```

### Analytical solution
The steady state solution is:
$$
z^*=\frac{s}{[(1+n)(1+g)]^{\frac{\beta}{\beta+\kappa}}-(1-\delta)}
$$

We turn the solution into a sympy equation:

```python
ss = sm.Eq(z, s/((((1+n)*(1+g))**(beta/(beta+kappa)))-(1-delta))) #Turns SS into a sympy equation
ss_capital_ratio = sm.solve(ss,z)[0] #Solve it by sympy
ss_func = sm.lambdify((s,g,n,beta,kappa,delta),ss_capital_ratio) # Turns it into a Python function
```

We solve for steady state using the analytical expression and inserting plausible parameter values:

```python
print(f'The steady state for capital-output ratio is: {ss_func(0.2,0.02,0.01,0.6,0.2,0.05):.3f}')
```

    The steady state for capital-output ratio is: 2.756

### Numerical analysis
We can also solve with numerical optimization:

If we rewrite the transition equation it clearly becomes a root-finding problem:

$$
0=\frac{1}{((1+g)(1+n))^\beta} [s+z^*(1-\delta)]^{1-\alpha}z^{*\alpha}-z^*
$$

We state the same parameter values:

```python
alpha = 0.2
beta = 0.6
kappa = 0.2
delta = 0.05
g = 0.02
n = 0.01
s = 0.2
```

To solve we need an objective function:

```python
obj_zss = lambda z: z - ((s + z*(1-delta))**(1-alpha)*z**alpha)/(((1+n)*(1+g))**beta)
interval = [0.1,100]
```

We choose to apply the Brentq method as it is seen as one of most safe rootfinding methods in the scipy package. The method simply finds a zero of the function given an interval $[a,b]$. Using Brentq we find the optimal level of capital-output ratio for our parameters:

```python
res  = optimize.root_scalar(obj_zss,bracket=interval,method='brentq')
print(res)
```

    converged: True
               flag: 'converged'
     function_calls: 7
         iterations: 6
               root: 2.7561286078149587

We can see it took 6 iterations and 7 function calls, which is relatively quick.

We call the two solutions and compare:

```python
print(f'analytical solution is: {ss_func(s,g,n,beta,kappa,delta):.3f}')
print(f'numerical solution is: {res.root:.3f}')
```

    analytical solution is: 2.756
    numerical solution is: 2.756

As we can see the solutions are identical (3 decimals).

## Graphical analysis

We want to analyze capital-ratio output over time so we create the following function:

```python
# a. generate function for transition equation for the capital-output ratio
def trans_eq(Z0,alpha,beta,delta,s,g,n,t):

    '''Function creates a graph based on the transition equation

    input:
    Z0 = capital-ratio output in first period
    alpha = share of capital
    beta = share of effective labor
    delta = deprication rate
    s = savings rate
    g = technology growth rate
    n = population growth rate
    t = periods

    output:
    Z_t1 = capital-ratio output in next period

    '''

     # i. create two empty list
    Z_t = [Z0]
    Z_t1 = []

    i = 1

    # ii. create a loop which generates two lists for both z_t1 and z_t
    while i <= t:
        Z_t.append(((s+Z_t[-1]*(1-delta))**(1-alpha)*Z_t[-1]**alpha)/(((1+n)*(1+g))**beta)) # Adds to list for z_t
        Z_t1.append(((s+Z_t[-1]*(1-delta))**(1-alpha)*Z_t[-1]**alpha)/(((1+n)*(1+g))**beta)) # Adds to list for z_t1

        i += 1

    # iii. generate plot
    plt.style.use('seaborn')
    fig = plt.figure(figsize=(5,5))
    ax = fig.add_subplot(1,1,1)
    ax.plot(Z_t1)
    ax.set_title('Growth in capital-output ratio over time')
    ax.set_xlabel('time')
    ax.set_ylabel('$z_{t+1}$')
    return plt.show()
```

```python
# a. call the function for 200 periods
trans_eq(1,alpha,beta,delta,s,g,n,200)
```

We want to include an interactive element to our graph so we create the following:

```python
widgets.interact(trans_eq,
    Z0 = widgets.fixed(1),
    alpha = widgets.fixed(0.2),
    beta = widgets.fixed(0.6),
    delta = widgets.FloatSlider(description = 'd' , min = 0.01, max = 0.1, step=0.01, value=0.05),
    s = widgets.FloatSlider(description = 's', min=0.1, max=0.9, step = 0.1, value= 0.2),
    g = widgets.FloatSlider(description = 'g', min=0.01, max=0.1, step= 0.01, value=0.02),
    n = widgets.FloatSlider(description = 'n', min = 0.01, max = 0.1, step = 0.01, value = 0.01),
    t = widgets.fixed(200))
```

    interactive(children=(FloatSlider(value=0.05, description='d', max=0.1, min=0.01, step=0.01), FloatSlider(valu…

    <function __main__.trans_eq(Z0, alpha, beta, delta, s, g, n, t)>

An increase in s is displacing the SS upwards, since the capital/output ratio is increasing.
A decrease in n results in less thinning out of capital, thus capital/worker is increasing and thereby the capital/output relationship, which also increases the SS level.

These effects are plottet in the figures below.

```python
# a. creates a list for savings rate and a list for population growth
s_list = np.linspace(0.1,1,num=10, endpoint=True)
results_s = []

n_list = np.linspace(0.01,0.05,num=10, endpoint=True)
results_n = []

# b. create two loops which makes a steady state for each value of savings rate and population growth
for s in s_list:
    res  = optimize.root_scalar(obj_zss,bracket=interval,method='brentq')
    results_s.append(res.root)

for n in n_list:
    res  = optimize.root_scalar(obj_zss,bracket=interval,method='brentq')
    results_n.append(res.root)

# c. create figure for both results
plt.style.use('seaborn')
fig = plt.figure(figsize=(10,4))

# d. figure based on savings
ax_s = fig.add_subplot(1,2,1)
ax_s.scatter(s_list,results_s)
ax_s.set_title('Steady state based on rate of savings')
ax_s.set_xlabel('s')
ax_s.set_ylabel('$z^*$')
ax_s.grid(True)

# e. figure based on population growth
ax_n = fig.add_subplot(1,2,2)
ax_n.scatter(n_list,results_n)
ax_n.set_title('Steady state based on population growth')
ax_n.set_xlabel('n')
ax_n.set_ylabel('$z^*$')
ax_n.grid(True)
```

The figures clearly shows how an increase in the saving rate results in a larger capital-output ratio, where an incresase in the population growth results in a lower capital-output ratio, all other things held equal.

# Conclusion

The project examined the standard Solow model with technological growth and the extended Solow model with land. Here we saw how different parameter values affected the steady state level of capital (model 1) and capital-output ratio (model 2). Futhermore, we showed how the numerical solution and the analytical solution were identical, which means that our numerical optimization worked as it should.

```python

```


```python
from sympy import *

init_session()
```

    IPython console for SymPy 1.8 (Python 3.8.12-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.8/

```python
f = exp(-x**2)*sin(3*x)
```

```python
fxxx = diff(f, x, 3)
```

The third derivative of $f$ is {{ fxxx }} which is ... a jerk move