# ***Jupyter-Notebook to Practice Physical Constants Ratios***

## Dr Francis M. Sanchez's Physical Constants Ratios in Cosmology and MicroPhysics

### Physical Constants Ratios

Introduction
------------

`Jupyter-Notebook Practice Physical Constants Ratios` provides common physical constants ratios.

Installation
------------

Run the Jupyter-notebook in [Mybinder.org](https://mybinder.org/v2/gh/LaGuer/Jupyter-Notebook-Practice-Physical-Constants-Ratios/master) will be available for Julia 0.7 and later versions, and can be
installed with
[Github.com source code](https://github.com/LaGuer/Jupyter-Notebook-Practice-Physical-Constants-Ratios/blob/master/Jupyter-Notebook%20Practice%20Physical%20Constants%20Ratios.ipynb).
In a terminal session run the command

```
jupyter-notebook

```

### Updated universal constants and particle properties ( thanks to Jean Maruani / Francis M. Sanchez)
* [The_Dirac_Electron_From_Quantum_Chemistry_to_Holistic_Cosmology @ researchgate](https://www.researchgate.net/publication/287808070_The_Dirac_Electron_From_Quantum_Chemistry_to_Holistic_Cosmology)
* [Dirac_Electron_From_Quantum_Chemistry_to_Holistic_Cosmology @ Wiley](https://onlinelibrary.wiley.com/doi/abs/10.1002/jccs.201500374)

| Symbol  | Name                                | Formula                  | Dimension      | Value              | Unit            |
| ------  | ----                                | -------                  | ---------      | -----              | ----            |
| `π`     | pi                                  | -                        | -              | 3.1415926535897932 | `pure number`   |
| `e`     | e                                   | -                        | -              | 2.7182817284590452 | `pure number`   |
| `γ`     | gamma                               | -                        | -              | 0.5772156649015329 | `pure number`   |
| `26SGP` | 26 sporadic group Product           | -                        | -              | exp(841.2869278)   | `pure number`   |
| `HFP`   | Happy Family product                | -                        | -              | exp(674.5210288)   | `pure number`   |
| `O_M`   | Monster Group Cardinal              | -                        | -              | 8.080174248 × 1053 | `pure number`   |
| `O_B`   | Baby Monster Group Cardinal         | -                        | -              | 4.15478148 × 1033  | `pure number`   |
| `c`     | Speed of light in vacuum            | -                        | -              | 2.99792458e8       | `m.s^-1`        |
| `ħ`     | Reduced Planck constant             | ħ=h/2pi                  | -              | 1.054571800e-34    | `m^2 kg s^-1`   |
| `G'`    | constant of gravitation CODATA2014  | -                        | M^-1L^3T^-2    | 6.67408(31)e-11    | `m^3 kg^-1 s^-2`|
| `G`     | F.M.Sanchez constant of gravitation | F_gr=Gmm'/ d^2           | M^-1L^3T^-2    | 6.675453818e-11    | `m^3 kg^-1 s^-2`|
| `a`     | Electric constant                   | a=α^-1  F_qq=ħc/ad^2     | dimensionless  | 137.035999139^-1   | `pure number`   |
| `R_U`   | Universe Hubble radius (Sz)         | R_U=2G.M_U/c)^2          | L              | 1.3065e26          | `m`             |
| `G_F`   | Fermi Coupling Constant (Sz)        | G_F=(ħc)^3/E_F^2         |   ML^5T^-2     | 1.4358509(7)e-62   | `J.m^3`         |
| `a_G`   |Gravit Sanchez Coupling Constant     | a_G=ħc/Gm_pm_H           | -              | 1.691936465e38     | `pure number`   |
| `M_U`   | Universe Sanchez Mass               | M_U=(ħc/G)^2/m_e.m_p.m_H | M              | 8.7936e52          | `kg`            |
| `t_cc`  | Kotov Cosmic Periodicity            | -                        | s              | 9600.061(2)        | `s`             |
|`r_H(0)` | Bare Hydrogen radius                |    aħ/m_ec               | L              | 5.291772103e-11    | `m`             |
| `H`     |Hydrogen-electron mass ratio         |    m_H/m_e               | dimensionless  | 1837.152645        | `pure number`   |
| `p`     |Proton-electron mass ratio           |    m_p/m_e               | dimensionless  | 1836.152672        | `pure number`   |
| `n`     |Neutron-electron mass ratio          |    m_n/m_e               | dimensionless  | 1838.683659        | `pure number`   |
| `m_e`   |electron mass                        | -                        | M              |9.10938356(11)e-31  | `kg`            |
| `μ'`    |Muon electron mass ratio CODATA2014  | -                        | dimensionless  |206.7682826(41)     | `pure number`   |
| `μ`     |Muon mass F.M.SANCHEZ  (Sz)          | -                        | dimensionless  |206.7682869         | `pure number`   |
| `f'`    |Strg Nuc Coupling Constant CODATA2014| -                        | dimensionless  |1/0.1181(11)        | `pure number`   |
| `f`     |Strg Nuc Coupling Constant C.Bizouard|    F^2/2pi(pH)^3/2       | dimensionless  |8.434502892         | `pure number`   |
| `F'`    |Fermi Constant reduced mass CODAT2014|   E_F / m_ec^2           | dimensionless  |573007.33(25)       | `pure number`   |
| `F`     |Fermi Constant reduc mass F.M.SANCHEZ| -                        | dimensionless  |573007.3652         | `pure number`   |
| `d_e`   | electron magnetic moment excess     | -                        | dimensionless  |1.00115965218091(26)| `pure number`   |
| `θ'`    |Cosmic Temperature CODATA2014        | -                        | -              |  2.7255(6) K       | `K`             |
| `θ`     |Cosmic Temperature F.M.Sanchez       | -                        | -              |  2.725820831 K     | `K`             |
| `k_B`   |Boltzman Energy-temperature Convers° | -                        | -              | 1.3806488e10-23    | `J K^-1`        |
| `W`     |Intermediate vector boson Red. Mass  | -                        | dimensionless  | 157298(23)         | `pure number`   |
| `Z`     |Intermediate vector boson Red. Mass  | -                        | dimensionless  | 178450.0(41)       | `pure number`   |
| `Π±`    |Intermediate vector boson Red. Mass  | -                        | dimensionless  | 273.13203(68)      | `pure number`   |
| `Π0`    |Intermediate vector boson Red. Mass  | -                        | dimensionless  | 264.1426(18)       | `pure number`   |

## Latex and notations

- Put your LaTeX math inside ```$$ $$```

- Example of notation for:

- $$\sqrt{a_W} = \frac{E_F}{m_e \cdot c^2}$$ type ```$$\sqrt{a_W} = \frac{E_F}{m_e \cdot c^2}$$```

- Write your equation in a dedicated cell with latex

Exercice of notation for:

- $$\sqrt{G'G_F} = \frac{\frac{ƛ_e^2}{t_cc} \cdot ħ}{\sqrt{m_p \cdot m_H}}$$ ```$$\sqrt{G'G_F} = \frac{\frac{ƛ_e^2}{t_cc} \cdot ħ}{\sqrt{m_p \cdot m_H}}$$```

- This translates ```\begin{equation}
  \frac{\pi_Pt}{\pi_Bb}\ = 6\cdot\frac{\pi^5}{p_H^9}
  \label{eq:sample}
\end{equation}
```

- to this:
    \begin{equation}
  \frac{\pi_Pt}{\pi_Bb}\ = 6\cdot\frac{\pi^5}{p_H^9}
  \label{eq:sample}
\end{equation}

```
%%latex
\begin{align}
\R_U = \frac{2\cdot G\cdot M_U}{c^2} \\
\M_U = \frac{({ħc}{G})^2}{m_em_pm_h} \\
\r_H/ƛ_e = frac{\sum\frac{1}{n}{\sum(\frac{1}{n^2})}\\
\end{align}
```

- $$\e^{i\pi} = -1$$ type ```$$\e^{i\pi} = -1$$```
```

- to this:
- $$\sum{n}(\frac{1}{n})\ = \frac{137}{60} = 2 \times (\pi-2)$$ type ```$$\sum{n}(\frac{1}{n})\ = \frac{137}{60} = 2 \times (\pi-2)$$```

##

```python
%%timeit
# %matplotlib inline
%matplotlib notebook
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

# using PhysicalConstants.CODATA2014
## m_p/m_e
## R_U=2G.M_U/c)^2
```

```latex
%%latex
\begin{align}
\frac{m_p^4}{m_p\cdot m_H\cdot\ m_e} = M \\
\end{align}
```

\begin{align}
\frac{π_P_t}{π_B_b} = {6π^5}{p_H^9} \\
\frac{\pi_Pt}{\pi_Bb} = 6 \cdot \frac{\pi^5}{p_H^9}
\end{align}

```python
# %%timeit
# Inclusions
from math import *
values = []
exponents = []

########################################################
#     Physical Constants Ratios                        #
########################################################

progression_display = True # True / False
exponent_minmax = 3
precision = 0.00000001 # add or remove 0 for precision's accuracy
number_min_Zero = 1

# exercice 2
#exponent_minmax = 4
#precision = 0.00000002
#values.append(['a/137',log(137.0359991)-log(137)])
#values.append(['pg',log(1831.531308)-log(1836.152672)*0])
#values.append(['pK',log(1842.604127)-log(1836.118109)*0])
#values.append(['H',log(1837.152645)-log(1836.118109)*0])
#values.append(['p',log(1836.152672)+log(1836.118109)*0])
#values.append(['n',log(1838.683659)])
#values.append(['1sH-p',-log(0.999973)])
#values.append(['d',log(1.001159652)])

# exercice 2
values.append(['pi',1.144729886])
values.append(['a',4.920243658])
values.append(['137',4.919980926])
values.append(['F',13.25865385])
values.append(['H/p',0.00054445428])
values.append(['x',0.002792343634])
values.append(['n',7.516805193])
values.append(['8',2.079441542])
values.append(['d',0.001158980123])
values.append(['Ggamma',3.22597446])

########################################################
#     The Program itself                               #
########################################################

print ("Starting CosmHology...")
length = len(values)
precedent=-exponent_minmax

# Min exponent set for each value
for i in range (0,length):
    exponents.append(-exponent_minmax)
    print(values[i][0],end='\t')

print("Zeros",end='\t');
print("Precision");

nbSolutions = 0
#<= 0 Do not make twice the same calculation with the inverses
while exponents[length-1]<=0:
    # Calculate and process the result
    result= 0
    j=0
    while j<length:
        result=result + (values[j][1])*(exponents[j])
        j+=1
    #if result<0:
    #    result*=(-1)

    # Display or not the results
    if result<=precision and result>=-precision:
        # counting zeros
        nbzero=0
        for j in range (0,length):
            if (exponents[j] == 0):
                nbzero+=1
        if nbzero >= number_min_Zero:
            for j in range (0,length):
                print(exponents[j],end='\t')
            print(nbzero,end='\t')
            print(int(result*1000000000))
            nbSolutions = nbSolutions + 1

    # Add + Exponents Rotation
    exponents[0]+=1
    for j in range (0,length-1):
        if exponents[j]>exponent_minmax:
            exponents[j]=-exponent_minmax
            exponents[j+1]+=1
    if (progression_display):
        if (exponents[length-2]!=precedent):
            precedent=exponents[length-2]
            #print("Status:",end=' ')

print(exponents[j+1]);
print ("All results lead to nbSolutions =", nbSolutions)
print("Precision", precision)
print("Used Values are:", values)

```

    Starting CosmHology...
    a/137	pg	pK	H	p	n	1sH-p	d	Zeros	Precision
    1	2	2	1	-3	-2	-1	0	1	-16
    0	0	0	0	0	0	0	0	8	0
    -1	-2	-2	-1	3	2	1	0	1	16
    All results lead to nbSolutions = 3
    Precision 2e-08
    Used Values are: [['a/137', 0.00026273263604270625], ['pg', 7.51290767625786], ['pK', 7.518935136442214], ['H', 7.515972176482602], ['p', 7.515427722407373], ['n', 7.5168051918630026], ['1sH-p', 2.7000364506560378e-05], ['d', 0.0011589801229981887]]

In the following example, one can write the equation with Latex:

```python
from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\frac{m_p^4}{m_p\cdot m_H\cdot\ m_e} = M  \\
\R_U = \frac{2\cdot G\cdot M_U}{c^2} \\
\M_U = \frac{({ħc}{G})^2}{m_em_pm_h} \\
\frac{r_H}{ƛ_e} = \frac{\sum\frac{1}{n}}{\sum\frac{1}{n^2}}\\
\end{eqnarray}""")

```

\begin{eqnarray}
\frac{m_p^4}{m_p\cdot m_H\cdot\ m_e} = M  \\
\R_U = \frac{2\cdot G\cdot M_U}{c^2} \\
\M_U = \frac{({ħc}{G})^2}{m_em_pm_h} \\
\frac{r_H}{ƛ_e} = \frac{\sum\frac{1}{n}}{\sum\frac{1}{n^2}}\\
\end{eqnarray}

:star:

# Docs

This project aims at providing a repository dedicated to Dr Francis Michel Sanchez' work in holography and cosmology; it is just a mere continuation of what was initiated and published on a couple of other blogs.

## Blog
* [cosmhology blog in english](https://cosmhology.blogspot.com/)
* [cosmhologie blog in french](https://cosmhologie.blogspot.com/)
* [Concorde Physiques in russian ](http://concorde-physiques.blogspot.com/)

## Publications
* Please, cite this paper (http://vixra.org/pdf/1811.0146v8.pdf) if you employ this package in your research work.
* [Vixra Publications](https://vixra.org/author/francis_m_sanchez)
* [A coherent resonant cosmology approach and its implications in microphysics and Biophysics Pages 375-407 of Quantum Systems in Physics, Chemistry](https://www.springer.com/fr/book/9783319502540?token=gbgen&wt_mc=GoogleBooks.GoogleBooks.3.EN#)
* [From Coherent Cosmic Oscillations to the Steady-State Cyclic Ultrafast](https://rxiv.org/abs/1401.0228)
* [Evidence for a Steady-State, Holographic, Tachyonic and Super-Symmetric Cosmology](https://www.researchgate.net/publication/267430152_Evidence_for_a_Steady-State_Holographic_Tachyonic_and_Super-Symmetric_Cosmology)
* [academia.edu Correcting Five Historical Faults in Modern Cosmology](https://www.academia.edu/30843365/Correcting_Five_Historical_Faults_in_Modern_Cosmology)

## Videos

* [L'enigme des grands nombres Part1](https://www.youtube.com/watch?v=ev_eIqLtK8s)
* [L'enigme des grands nombres Part2](https://www.youtube.com/watch?v=i4w392nYqns)
* [L'enigme des grands nombres Part3](https://www.youtube.com/watch?v=b4WvI0yVXnE)
* [Exposés de Francis Sanchez](https://www.youtube.com/playlist?list=PLQXUNrzoAeFbmfHo3jgWzzExlVoBn2JUw)
* [Khelif/Torgoz/Sanchez' Debate on LE FORMALISME EN PHYSIQUE ET EN SCIENCE](https://www.youtube.com/watch?v=TEK2hCI2we0)

* https://www.youtube.com/watch?v=dyUlUahgHGg
* https://www.youtube.com/watch?v=NmvvIQjxBw8
* https://www.youtube.com/watch?v=dyUlUahgHGg
* https://www.youtube.com/watch?v=zUt1hxTdiuA
* https://www.youtube.com/watch?v=05g416VWjnw

## Books

* [Quantum Systems in Physics, Chemistry, Springer Ed](https://www.amazon.com/gp/search?index=books&linkCode=qs&keywords=9783319502557)
* [The Worldwide List of Alternative Theories and Critics](https://books.google.com/books?isbn=2902425171)
* [The Dirac Electron: From Quantum Chemistry to Holistic Cosmology by J.Maruani](https://onlinelibrary.wiley.com/doi/abs/10.1002/jccs.201500374#)
* [solar 22 years cycle](https://link.springer.com/article/10.1007%2Fs10509-016-2985-8)
* [Current Issues in Cosmology](https://www.cambridge.org/gb/academic/subjects/physics/cosmology-relativity-and-gravitation/current-issues-cosmology?format=HB&isbn=9780521858984)
* ```@inproceedings{Sanchez2015RemarkablePO,
  title=Remarkable Properties of the Eddington Number 137 and Electric Parameter 137 . 036 excluding the Multiverse Hypothesis},
  author={Francis M. Sanchez and Valery A. Kotov,
  year=2015```

## Papers

* [Discussions sur les oscillations cosmiques, les nombres sans dimension et les periodicites en microphysique et cosmologie - College de France 2004](https://lempel.pagesperso-orange.fr/occ_cdf.pdf)

More to come soon stay tuned.../.:stars:.

```python

    #Write your equation in a dedicated cell with latex

from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\frac{m_p^4}{m_p\cdot m_H\cdot\ m_e} = M \\
\frac{\pi_Pt}{\pi_Bb} = 6 \cdot \frac{\pi^5}{p_H^9} \\
\end{eqnarray}""")

```

Jupyter-Notebook Practice Physical Constants Ratios.ipynb


# Classical Support Vector Machines

This notebook will serve as a summary of some of the resources below and is not meant to be used as a stand-alone reading material for Classical Support Vector Machines. We encourage you to complete reading the resources below before going forward with the notebook.

### Resources:
1. MIT Open Courseware lecture: https://youtu.be/_PwhiWxHK8o
3. MIT lecture slides: http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
4. SVM Wikipedia page : https://en.wikipedia.org/wiki/Support_vector_machine
2. SVM tutorial using sklearn: https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html

## Contents

1. [Introduction](#intro)
2. [SVMs as Linear Classifiers](#linear)
3. [Lagrange Multipliers and the Primal and Dual form](#primal)
4. [Class Prediction For a New Datapoint](#pred)
5. [Classifying Linearly Separable Data](#class-linear)
6. [Dealing With Non-Linearly Separable Data](#non-linear)
7. [Feature Map and Kernel](#kernel)
8. [Additional Resources](#add)

## Introduction

```python
# installing a few dependencies
!pip install --upgrade seaborn==0.10.1
!pip install --upgrade scipy==1.4.1
!pip install --upgrade scikit-learn==0.23.1
!pip install --upgrade matplotlib==3.2.0

# the output will be cleared after installation
from IPython.display import clear_output
clear_output()
```

Suppose you are a Botanist trying to distinguish which one of the **three species** a flower belongs to just by looking at **four features** of a flower - the length and the width of the sepals and petals. As part of your research you create a **dataset** of these features for a set of flowers for which the **species is already known**, where each **datapoint** of this dataset corresponds to a single flower. Now, your colleague brings in a new flower and asks you which species it belongs to. You could go into the lab and do the necessary tests to figure out what species it is, however, the lab is under renovation. So, left with no other choice you pull up the dataset that you created earlier and after a few minutes of trying to find a pattern you realise that this new flower has a petal width and sepal length similar to all the flowers of species 1. Thus, you **predict** this new flower to be of the species 1. This process of assigning a new datapoint to one of the known **classes** (flower species) is called **classfication**. And, as we used a dataset where we knew the classes corresponding to the datapoints before-hand, thus, this classification procedure comes under the umbrella of [**supervised learning**](https://en.wikipedia.org/wiki/Supervised_learning).

Support Vector Machines (SVMs) are **supervised learning models** that are mainly used for **classification** and **regression** tasks. In the context of classification, which is the topic of discussion, we use SVMs to find a **linear decision boundary with maximum width** splitting the space such that datapoints belonging to different classes are on either side of the boundary. Classification takes place based on which side of the decision boundary a new datapoint lands.

Before we try to understand how SVMs work, let's take a look at the [Iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) which was the dataset mentioned in the first paragraph.

```python
# importing the iris dataset
from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()

print("Number of datapoints: {}".format(iris['data'].shape[0]))
print("Number of features: {}".format(iris['data'].shape[1]))
print("Sample of the dataset:")
print(iris['data'][:5])
print("Unique species : {}".format(np.unique(iris['target'])))
```

    Number of datapoints: 150
    Number of features: 4
    Sample of the dataset:
    [[5.1 3.5 1.4 0.2]
     [4.9 3.  1.4 0.2]
     [4.7 3.2 1.3 0.2]
     [4.6 3.1 1.5 0.2]
     [5.  3.6 1.4 0.2]]
    Unique species : [0 1 2]

Looking at the first 5 datapoints of the Iris dataset we realize that each datapoint is an array with four features. The number of features in a dataset is called the **dimension of the dataset**. Further, there are three unique species, which implies, three **classes** in the dataset. It's important to note that SVMs are natively binary classification algorithms, i.e, can classify between only 2 classes. However, there are methods to convert a binary classifier to a multi-class classifier, mentioned [here](https://datascience.stackexchange.com/questions/46514/how-to-convert-binary-classifier-to-multiclass-classifier). Let us now dig deeper into the mathematics of how SVMs work.

**Reminder:** Read the resources provided above to understand the next section with greater degree of clarity.

## SVMs as Linear Classifiers

Source: [wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)

Our input dataset is of the form $(\vec{x}_{1}, y_{1}), ..., (\vec{x}_{n}, y_{n})$,
where, $\vec{x}$ is a $d$ dimensional vector where $d$ is the number of features and $y_{i}$'s are the labels $ y_{i} \in {-1, +1}$ as it is a binary classification problem.

$\vec{w}$ is a vector perpendicular to the **decision boundary** (hyperplane that cuts the space into two parts and is the result of classification). Let $\vec{u}$ be a vector representing a point on our feature space. Now, to understand whether a point is on the +ve side or the -ve side we'll have to project the point $\vec{u}$ onto $\vec{w}$, which will give us a scaled version of $\vec{u}$'s projection in the direction perpendicular to the decision boundary. Depending on the value of this quantity we'll have the point either on the +ve side or the -ve side. This can be represented mathematically as

$$\begin{equation} \vec{w}\cdot\vec{x}_{+} + b \geq 1 \label{eq:vector_ray} \tag{1}\end{equation}$$
$$\begin{equation} \vec{w}\cdot\vec{x}_{-} + b \leq -1  \tag{2}\end{equation}$$

where, $\vec{x}_{+}$ is a datapoint with label $y_{i} = +1$,

$\vec{x}_{-}$ is a datapoint with label $y_{i} = -1$ and

      b is parameter that has to be learnt

These two lines are separated by a distance of $\frac{2}{||{\vec{w}}||}$. The line in the middle of both of these, i.e,

$$\begin{equation} \vec{w}\cdot\vec{u} + b = 0  \tag{3}\end{equation}$$

is the equation of the hyperplane denoting our decision boundary. Together, the space between (1) and (2) forms what is usually know as the **street** or the **gutter**.

equation (1) and (2) can be conveniently combined to give

$$y_{i}(\vec{w}\cdot\vec{x}_{i} + b) \geq 1\tag{4}$$

And the limiting case would be

$$y_{i}(\vec{w}\cdot\vec{x}_{i} + b) -1 = 0 \tag{5}$$

Which is attained when the points lie on the edges of the street, i.e, on (1) or (2). These points are responsible for the change in the width of the street and are called **support vectors**. Once the support vectors are calculated in the training phase we only need these vectors to classify new datapoints during the prediction phase, hence reducing the computational load significantly. Equation (4) is a constraint in the optimization process of maximizing the street width $\frac{2}{||{\vec{w}}||}$. In the next section let us see how we can combine the optimization problem and the constraints together into a single optimization equation using the concept of  lagrange multipliers.

## Lagrange Multipliers and the Primal and Dual form

Support Vector Machines are trying to solve the optimization problem of maximizing the street width $\frac{2}{||{\vec{w}}||}$ (which is equivalent to minimizing $\frac{||w||^2}{2}$) with the contraint $y_{i}(\vec{w}\cdot\vec{x}_{i} + b) \geq 1$. This can be elegantly written in a single equation with the help of [lagrange multipliers](https://en.wikipedia.org/wiki/Lagrange_multiplier). The resulting equation to be minimized is called the **primal form** (6).

**Primal form:** $$ L_{p} = \frac{||w||^2}{2} - \sum_{i}{\alpha_{i}[y_{i}(\vec{w}\cdot\vec{x}_{i} + b) -1]}\tag{6}$$

$$\frac{{\partial L}}{\partial \vec{w}} = \vec{w} - \sum_{i}{\alpha_{i}y_{i}\vec{x_{i}}}$$

equating $\frac{{\partial L}}{\partial \vec{w}}$ to 0 we get,

$$ \vec{w} = \sum_{i}{\alpha_{i}y_{i}\vec{x_{i}}}\tag{7}$$

$$\frac{{\partial L}}{\partial \vec{b}} = \sum_{i}{\alpha_{i}y_{i}}$$

and equating $\frac{{\partial L}}{\partial \vec{b}}$ to 0 we convert the primal form to the dual form,

$$\sum_{i}{\alpha_{i}y_{i}} = 0\tag{8}$$

$$L = \frac{1}{2}(\sum_{i}{\alpha_{i}y_{i}\vec{x_{i}}})(\sum_{j}{\alpha_{j}y_{j}\vec{x_{j}}}) - (\sum_{i}{\alpha_{i}y_{i}\vec{x_{i}}})(\sum_{j}{\alpha_{j}y_{j}\vec{x_{j}}}) - \sum_{i}{\alpha_{i}y_{i}b} + \sum_{i}{\alpha_{i}}$$

**Dual form:** $$L_{d} = \sum_{i}{\alpha_{i}} - \frac{1}{2}\sum_{i}\sum_{j}\alpha_{i}\alpha_{j}y_{i}y_{j}(\vec{x}_{i}\cdot\vec{x}_{j})\tag{9}$$
subject to: $$\sum_{i}{\alpha_{i}y_{i}} = 0$$

Taking a closer look at the dual form $L_{d}$ we can see that it is a function quadratic in the lagrange multipler terms which can be solved efficiently on a classical computer using [quadratic programming](https://en.wikipedia.org/wiki/Quadratic_programming) techniques. However, Note that finding the dot product $\vec{x}_{i}\cdot\vec{x}_{j}$ becomes computationally expensive as the dimension of our data increases. In the days to come we'll learn how a quantum computer could be used to classify a classical dataset using an algorithm called the Variational Quantum Classifier (VQC) Algorithm as given in [this paper](https://arxiv.org/abs/1804.11326). Understanding of Classical SVM may not be required, however, some of the concepts such as kernels and feature maps will be crucial in understanding the VQC algorithm.

## Class Prediction for a New Datapoint

The output of the training step are values of lagrange multipliers. Now, when a new datapoint $\vec z$ is given lets see how we can find the classification result corresponding to it:

* Step 1: Use the obtained values of lagrange multipliers to calculate the value of $\vec{w}$ using $(7)$.
* Step 2: Substitute the value of $\vec{w}$ in equation $(5)$ and substitute a support vector in the place of $\vec{x}_{i}$ to find the value of $b$.
* Step 3: Find the value of $\vec{w}\cdot\vec{z} + b$. If it $>0$ then assign $\vec{z}$ a label $y_{z} = 1$ and $y_{z} = -1$ if the obtained value is $< 0$.

## Classifying Linearly Separable Data

Lets switch gears and look at how we can use scikit-learn's Support Vector Classifier method to draw a decision boundary on a linearly separable dataset. This section of the notebook is a recap of resource \[4\] and thus we recommend reading it before going forward. The code used in this section is from the corresponding Github [repo](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/05.07-Support-Vector-Machines.ipynb).

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# use seaborn plotting defaults
import seaborn as sns; sns.set()
```

### Importing the dataset

```python
# we are importing the make_blobs dataset as it can be clearly seen to be linearly separable
from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=50, centers=2,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn');
```

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear', C=1E10)
model.fit(X, y)
```

    SVC(C=10000000000.0, kernel='linear')

```python
# helper plotting function
def plot_svc_decision_function(model, ax=None, plot_support=True):
    if ax is None:
        ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()

    # create grid to evaluate model
    x = np.linspace(xlim[0], xlim[1], 30)
    y = np.linspace(ylim[0], ylim[1], 30)
    Y, X = np.meshgrid(y, x)
    xy = np.vstack([X.ravel(), Y.ravel()]).T
    P = model.decision_function(xy).reshape(X.shape)

    # plot decision boundary and margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

    # plot support vectors
    if plot_support:
        ax.scatter(model.support_vectors_[:, 0],
                   model.support_vectors_[:, 1],
                   s=300, linewidth=1, facecolors='none', edgecolors='b');
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model);
```

As we can see SVM works quite well when it deals with linearly separable datasets. The points which lie on the dotted lines denoted by $y_{i}(\vec{w}\cdot\vec{x}_{i} + b) = \pm1$ are the **support vectors**. Part of the reason why SVMs are popular are because, during the classification step only support vectors are used to classify a new point. This reduces the computational load significantly. The reason for this is because values of the lagrange multipliers turn out to be zero for vectors which are not support vectors.

```python
model.support_vectors_
```

    array([[0.44359863, 3.11530945],
           [2.33812285, 3.43116792],
           [2.06156753, 1.96918596]])

## Dealing With Non-Linearly Separable Data

In the previous example we've seen how we can find a model to classify linearly separable data. Lets look at an example and see if SVMs can find a solution when the data is non-linear.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(clf, plot_support=False);
```

When the data is circular, like in the example above, SVM fails to find a satisfactory linear classification model. However, if we cleverly introduce a new parameter $r$ such that $r = e^{-x^{2}}$ and using that as our third parameter construct a new dataset (see picture below), we'll observe that a plane can be drawn horizontally passing through, say, $r=0.7$ to classify the dataset! This method in which we are mapping our dataset into a higher dimension to be able to find a linear boundary in the higher dimension is called a **feature map**.

```python
r = np.exp(-(X ** 2).sum(1))
```

```python
from mpl_toolkits import mplot3d
# from ipywidgets import interact, fixed

def plot_3D(elev=30, azim=30, X=X, y=y):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap='autumn')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('r')

plot_3D()
# interact(plot_3D, elev=[-90, 90], azip=(-180, 180),
#          X=fixed(X), y=fixed(y));
```

## Feature Map and Kernel

As we have seen earlier a **feature map** maps our (non-linearly separable) input data to a higher dimensional  **feature space** where our data is now linearly separable. This helps circumvent the problem of dealing with non-linearly separable data, however, a new problem arises. As we keep increasing the dimension of our data, computing the coordinates of our data and the dot product $\phi(\vec{x}_{i})\cdot\phi(\vec{x}_{j})$ in this higher dimentional feature space becomes computationally expensive. This is where the idea of the [Kernel functions](https://en.wikipedia.org/wiki/Kernel_method) comes in.

Kernel functions allow us to deal with our data in the higher dimensional feature space (where our data is linearly separable) without ever having to compute the dot product in that space.

if $\phi(\vec{x})$ is the feature map, then the corresponding kernel function is the dot product $\phi(\vec{x}_{i})\cdot\phi(\vec{x}_{j})$, therefore, the kernel function $k$ is

$$k(x_{i},x_{j}) = \phi(\vec{x}_{i})\cdot\phi(\vec{x}_{j})$$

Therefore, the corresponding transformed optimization problem can we written as,

**Primal form:** $$ L_{p} = \frac{||w||^2}{2} - \sum_{i}{\alpha_{i}[y_{i}(\vec{w}\cdot\phi(\vec{x}_{i}) + b) -1]}\tag{6}$$

**Dual form:** $$L_{d} = \sum_{i}{\alpha_{i}} - \frac{1}{2}\sum_{i}\sum_{j}\alpha_{i}\alpha_{j}y_{i}y_{j}(\phi(\vec{x}_{i})\cdot\phi(\vec{x}_{j}))$$
or $$L_{d} = \sum_{i}{\alpha_{i}} - \frac{1}{2}\sum_{i}\sum_{j}\alpha_{i}\alpha_{j}y_{i}y_{j}k(x_{i},x_{j})$$
subject to: $$\sum_{i}{\alpha_{i}y_{i}} = 0$$
where $$ \vec{w} = \sum_{i}{\alpha_{i}y_{i}\phi(\vec{x_{i}})}$$

To understand why Kernel functions are useful lets look at an example using the Radial Basis Function (rbf) Kernel.

the rbf kernel is written as,

$$k(x_{i},x_{j}) = exp(-||x_{i} - x_{j}||^{2}/2\sigma^{2}) $$

where $\sigma$ is a tunable parameter

What we should understand here is that the rbf kernel projects our data into an infinite dimensional feature space, however, the computational power required to compute the kernel function's value is quite negligible! As you see, we don't have to compute the dot product of the infinite dimensional vectors. This is how kernels help SVMs tackle non-linearly separable data.

Rbf kernel in action:

```python
clf = SVC(kernel='rbf', C=1E6)
clf.fit(X, y)
```

    SVC(C=1000000.0)

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=300, lw=1, facecolors='none');
```

In the next notebook we will learn how to use Quantum Computers to do the same task of classification and why it may be advantageous in the future.

## Additional Resources

1. Andrew NG notes: http://cs229.stanford.edu/notes/cs229-notes3.pdf
2. Andrew NG lecture: https://youtu.be/lDwow4aOrtg

Day 5/Classical Support Vector Machines.ipynb


```python
import numpy as np
import scipy as sp
import numpy.linalg
import matplotlib.pyplot as plt
import matplotlib as mpl
import sympy as sy
```

```python
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

```python
def round_expr(expr, num_digits):
    return expr.xreplace({n : round(n, num_digits) for n in expr.atoms(sy.Number)})
```

#  Diagonalization of Symmetric Matrices

The  first  theorem of symmetric matrix:

If $A$ is symmetric, i.e. $A = A^T$, then any two eigenvectors from different eigenspaces are orthogonal.

$$\begin{aligned}
\lambda_{1} \mathbf{v}_{1} \cdot \mathbf{v}_{2} &=\left(\lambda_{1} \mathbf{v}_{1}\right)^{T} \mathbf{v}_{2}=\left(A \mathbf{v}_{1}\right)^{T} \mathbf{v}_{2} \\
&=\left(\mathbf{v}_{1}^{T} A^{T}\right) \mathbf{v}_{2}=\mathbf{v}_{1}^{T}\left(A \mathbf{v}_{2}\right) \\
&=\mathbf{v}_{1}^{T}\left(\lambda_{2} \mathbf{v}_{2}\right) \\
&=\lambda_{2} \mathbf{v}_{1}^{T} \mathbf{v}_{2}=\lambda_{2} \mathbf{v}_{1} \cdot \mathbf{v}_{2}
\end{aligned}$$

Because $\lambda_1 \neq \lambda_2$, so only condition which makes the equation holds is

$$
 \mathbf{v}_{1} \cdot \mathbf{v}_{2}=0
$$

With the help of this theorem, we can conclude that if symmetric matrix $A$ has different eigenvalues, its corresponding eigenvectors must be mutually orthogonal.

The diagonalization of $A$ is

$$
A = PDP^T = PDP^{-1}
$$

where $P$ is an orthonormal matrix with all eigenvectors of $A$.

The  second theorem of symmetric matrix:

 An $n \times n$ matrix $A$ is orthogonally diagonalizable if and only if $A$ is a symmetric matrix: $A^{T}=\left(P D P^{T}\right)^{T}=P^{T T} D^{T} P^{T}=P D P^{T}=A$.

##  An Example

Create a random symmetric matrix.

```python
A = np.round(2*np.random.rand(3, 3)); A
```

    array([[0., 2., 2.],
           [1., 0., 2.],
           [1., 2., 1.]])

```python
B = A@A.T; B # generate a symmetric matrix
```

    array([[8., 4., 6.],
           [4., 5., 3.],
           [6., 3., 6.]])

Perform diagonalization with ```np.linalg.eig()```.

```python
D, P = np.linalg.eig(B); P
```

    array([[ 0.709,  0.699,  0.089],
           [ 0.399, -0.294, -0.868],
           [ 0.581, -0.651,  0.488]])

```python
D = np.diag(D); D
```

    array([[11.787,  0.   ,  0.   ],
           [ 0.   ,  0.082,  0.   ],
           [ 0.   ,  0.   ,  4.131]])

Check the norm of all eigenvectors.

```python
for i in [0, 1, 2]:
    print(np.linalg.norm(P[:,i]))
```

    0.9999999999999998
    0.9999999999999999
    1.0

Check the orthogonality of eigenvectors, see if $PP^T=I$

```python
P@P.T
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

##  The Spectral Theorem

An $n \times n$ symmetric matrix $A$ has the following properties:
1. $A$ has $n$ real eigenvalues, counting multiplicities.
2. The dimension of the eigenspace for each eigenvalue $\lambda$ equals the multiplicity of $\lambda$ as a root of the characteristic equation.
3. The eigenspaces are mutually orthogonal, in the sense that eigenvectors corresponding to different eigenvalues are orthogonal.
4. $A$ is orthogonally diagonalizable.

All these properties are obvious without proof, as the example above shows.However the purpose of the theorem  is not reiterating last section, it paves the way for spectral decomposition.

Write diagonalization explicitly, we get the representation of spectral decomposition

$$
\begin{aligned}
A &=P D P^{T}=\left[\begin{array}{lll}
\mathbf{u}_{1} & \cdots & \mathbf{u}_{n}
\end{array}\right]\left[\begin{array}{ccc}
\lambda_{1} & & 0 \\
& \ddots & \\
0 & & \lambda_{n}
\end{array}\right]\left[\begin{array}{c}
\mathbf{u}_{1}^{T} \\
\vdots \\
\mathbf{u}_{n}^{T}
\end{array}\right] \\
&=\left[\begin{array}{lll}
\lambda_{1} \mathbf{u}_{1} & \cdots & \lambda_{n} \mathbf{u}_{n}
\end{array}\right]\left[\begin{array}{c}
\mathbf{u}_{1}^{T} \\
\vdots \\
\mathbf{u}_{n}^{T}
\end{array}\right]\\
&= \lambda_{1} \mathbf{u}_{1} \mathbf{u}_{1}^{T}+\lambda_{2} \mathbf{u}_{2} \mathbf{u}_{2}^{T}+\cdots+\lambda_{n} \mathbf{u}_{n} \mathbf{u}_{n}^{T}
\end{aligned}
$$

$ \mathbf{u}_{i} \mathbf{u}_{i}^{T}$ are rank $1$ symmetric matrices, because all rows of $ \mathbf{u}_{i} \mathbf{u}_{i}^{T}$ are multiples of $\mathbf{u}_{i}^{T}$.

Following the example above, we demonstrate in SymPy.

```python
lamb0,lamb1,lamb2 = D[0,0], D[1,1], D[2,2]
u0,u1,u2 = P[:,0], P[:,1], P[:,2]
```

Check rank of $ \mathbf{u}_{i} \mathbf{u}_{i}^{T}$ by ```np.linalg.matrix_rank()```.

```python
np.linalg.matrix_rank(np.outer(u0,u0))
```

    1

Use spectral theorem to recover $A$:

```python
specDecomp = lamb0 * np.outer(u0,u0) + lamb1 * np.outer(u1,u1) + lamb2 * np.outer(u2,u2)
specDecomp
```

    array([[2., 3., 2.],
           [3., 6., 4.],
           [2., 4., 8.]])

#  Quadratic Form

A  quadratic form is a function with form $Q(\mathbf{x})=\mathbf{x}^TA\mathbf{x}$, where $A$ is an $n\times n$ symmetric matrix, which is called the  the matrix of the quadratic form.

Consider a matrix of quadratic form

$$
A =
\left[
\begin{matrix}
3 & 2 & 0\\
2 & -1 & 4\\
0 & 4 & -2
\end{matrix}
\right]
$$

construct the quadratic form $\mathbf{x}^TA\mathbf{x}$.

\begin{align}
\mathbf{x}^TA\mathbf{x}&=
\left[
\begin{matrix}
x_1 & x_2 & x_3
\end{matrix}
\right]
\left[
\begin{matrix}
3 & 2 & 0\\
2 & -1 & 4\\
0 & 4 & -2
\end{matrix}
\right]
\left[
\begin{matrix}
x_1 \\ x_2\\ x_3
\end{matrix}
\right]\\
& =\left[
\begin{matrix}
x_1 & x_2 & x_3
\end{matrix}
\right]
\left[
\begin{matrix}
3x_1+2x_2 \\ 2x_1-x_2+4x_3 \\ 4x_2-2x_3
\end{matrix}
\right]\\
& =
x_1(3x_1+2x_2)+x_2(2x_1-x_2+4x_3)+x_3(4x_2-2x_3)\\
& = 3x_1^2+4x_1x_2-x_2^2+8x_2x_3-2x_3^2
\end{align}

Fortunately, there is an easier way to calculate quadratic form.

Notice that coefficients of $x_i^2$ is on the principal diagonal and coefficients of $x_ix_j$ are be split evenly between $(i,j)-$ and $(j, i)-$entries in $A$.

##  Example

Consider another example,

$$
A =
\left[
\begin{matrix}
3 & 2 & 0 & 5\\
2 & -1 & 4 & -3\\
0 & 4 & -2 & -4\\
5 & -3 & -4 & 7
\end{matrix}
\right]
$$

All $x_i^2$'s terms are

$$
3x_1^2-x_2^2-2x_3^2+7x_4^2
$$

whose coefficients are from principal diagonal.

All $x_ix_j$'s terms are

$$
4x_1x_2+0x_1x_3+10x_1x_4+8x_2x_3-6x_2x_4-8x_3x_4
$$

Add up together then quadratic form is

$$
3x_1^2-x_2^2-2x_3^2+7x_4^2+4x_1x_2+0x_1x_3+10x_1x_4+8x_2x_3-6x_2x_4-8x_3x_4
$$

Let's verify in SymPy.

```python
x1, x2, x3, x4 = sy.symbols('x_1 x_2 x_3 x_4')
A = sy.Matrix([[3,2,0,5],[2,-1,4,-3],[0,4,-2,-4],[5,-3,-4,7]])
x = sy.Matrix([x1, x2, x3, x4])
```

```python
sy.expand(x.T*A*x)
```

$\displaystyle \left[\begin{matrix}3 x_{1}^{2} + 4 x_{1} x_{2} + 10 x_{1} x_{4} - x_{2}^{2} + 8 x_{2} x_{3} - 6 x_{2} x_{4} - 2 x_{3}^{2} - 8 x_{3} x_{4} + 7 x_{4}^{2}\end{matrix}\right]$

The results is exactly the same as we derived.

##  Change of Variable in Quadratic Forms

To convert a matrix of quadratic form into diagonal matrix can save us same troubles, that is to say, no cross products terms.

Since $A$ is symmetric, there is an orthonormal $P$ that

$$
PDP^T = A \qquad \text{and}\qquad PP^T = I
$$

We can show that

$$
\mathbf{x}^TA\mathbf{x}=\mathbf{x}^TIAI\mathbf{x}=\mathbf{x}^TPP^TAPP^T\mathbf{x}=\mathbf{x}^TPDP^T\mathbf{x}=(P^T\mathbf{x})^TDP^T\mathbf{x}=\mathbf{y}^T D \mathbf{y}$$

where $P^T$ defined a coordinate transformation and $\mathbf{y} = P^T\mathbf{x}$.

Consider $A$

$$
A =
\left[
\begin{matrix}
3 & 2 & 0\\
2 & -1 & 4\\
0 & 4 & -2
\end{matrix}
\right]
$$

Find eigenvalue and eigenvectors.

```python
A = np.array([[3,2,0],[2,-1,4],[0,4,-2]]); A
```

    array([[ 3,  2,  0],
           [ 2, -1,  4],
           [ 0,  4, -2]])

```python
D, P = np.linalg.eig(A)
D = np.diag(D); D
```

    array([[ 4.388,  0.   ,  0.   ],
           [ 0.   ,  1.35 ,  0.   ],
           [ 0.   ,  0.   , -5.738]])

Test if $P$ is normalized.

```python
P.T@P
```

    array([[ 1., -0., -0.],
           [-0.,  1., -0.],
           [-0., -0.,  1.]])

We can compute $\mathbf{y}= P^T\mathbf{x}$

```python
x1, x2, x3 = sy.symbols('x1 x2 x3')
x = sy.Matrix([[x1], [x2], [x3]])
x
```

$\displaystyle \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\end{matrix}\right]$

```python
P = round_expr(sy.Matrix(P), 4); P
```

$\displaystyle \left[\begin{matrix}0.7738 & -0.6143 & 0.1544\\0.5369 & 0.5067 & -0.6746\\0.3362 & 0.6049 & 0.7219\end{matrix}\right]$

So the $\mathbf{y} = P^T \mathbf{x}$ is

$$
\left[\begin{matrix}0.7738 x_{1} + 0.5369 x_{2} + 0.3362 x_{3}\\- 0.6143 x_{1} + 0.5067 x_{2} + 0.6049 x_{3}\\0.1544 x_{1} - 0.6746 x_{2} + 0.7219 x_{3}\end{matrix}\right]
$$

The transformed quadratic form $\mathbf{y}^T D \mathbf{y}$ is

```python
D =  round_expr(sy.Matrix(D),4);D
```

$\displaystyle \left[\begin{matrix}4.3876 & 0.0 & 0.0\\0.0 & 1.3505 & 0.0\\0.0 & 0.0 & -5.7381\end{matrix}\right]$

```python
y1, y2, y3 = sy.symbols('y1 y2 y3')
y = sy.Matrix([[y1], [y2], [y3]]);y
```

$\displaystyle \left[\begin{matrix}y_{1}\\y_{2}\\y_{3}\end{matrix}\right]$

```python
y.T*D*y
```

$\displaystyle \left[\begin{matrix}4.3876 y_{1}^{2} + 1.3505 y_{2}^{2} - 5.7381 y_{3}^{2}\end{matrix}\right]$

##  Visualize the Quadratic Form

The codes are exceedingly lengthy, but intuitive.

```python
%matplotlib notebook
k = 6
x = np.linspace(-k, k)
y = np.linspace(-k, k)
X, Y = np.meshgrid(x, y)

fig = plt.figure(figsize = (7, 7))

########################### xAx 1 ############################
Z = 3*X**2 + 7*Y**2
ax = fig.add_subplot(221, projection='3d')
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')
ax.set_title('$z = 3x^2+7y^2$')

xarrow = np.array([[-5, 0, 0, 10, 0, 0]])
X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)
ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

yarrow = np.array([[0, -5, 0, 0, 10, 0]])
X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)
ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

zarrow = np.array([[0, 0, -3, 0, 0, 300]])
X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)
ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black',
alpha = .6, arrow_length_ratio = .001, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

########################### xAx 2 ############################
Z = 3*X**2
ax = fig.add_subplot(222, projection='3d')
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')
ax.set_title('$z = 3x^2$')
xarrow = np.array([[-5, 0, 0, 10, 0, 0]])
X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)
ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

yarrow = np.array([[0, -5, 0, 0, 10, 0]])
X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)
ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

zarrow = np.array([[0, 0, -3, 0, 0, 800]])
X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)
ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

########################### xAx 3 ############################
Z = 3*X**2 - 7*Y**2
ax = fig.add_subplot(223, projection='3d')
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')
ax.set_title('$z = 3x^2-7y^2$')
xarrow = np.array([[-5, 0, 0, 10, 0, 0]])
X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)
ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

yarrow = np.array([[0, -5, 0, 0, 10, 0]])
X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)
ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

zarrow = np.array([[0, 0, -150, 0, 0, 300]])
X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)
ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

########################### xAx 4 ############################
Z = -3*X**2 - 7*Y**2
ax = fig.add_subplot(224, projection='3d')
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')
ax.set_title('$z = -3x^2-7y^2$')
xarrow = np.array([[-5, 0, 0, 10, 0, 0]])
X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)
ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

yarrow = np.array([[0, -5, 0, 0, 10, 0]])
X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)
ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)

zarrow = np.array([[0, 0, -300, 0, 0, 330]])
X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)
ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black',
          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',
          linestyles = 'solid',linewidths = 2)
plt.show()
```

    <IPython.core.display.Javascript object>

Now there are some important terms to define, a quadratic form $Q$ is:
1. positive definite if $Q(\mathbf{x})>0$ for all $\mathbf{x} \neq \mathbf{0}$
2. negative definite if $Q(\mathbf{x})<0$ for all $\mathbf{x} \neq \mathbf{0}$
3. positive semidefinite if $Q(\mathbf{x})\geq0$ for all $\mathbf{x} \neq \mathbf{0}$
4. negative semidefinite if $Q(\mathbf{x})\leq0$ for all $\mathbf{x} \neq \mathbf{0}$
5. indefinite if $Q(\mathbf{x})$ assumes both positive and negative values.

We have a theorem for quadratic forms and eigenvalues:

Let $A$ be an $n \times n$ symmetric matrix. Then a quadratic form $\mathbf{x}^{T} A \mathbf{x}$ is:

1. positive definite if and only if the eigenvalues of $A$ are all positive
2. negative definite if and only if the eigenvalues of $A$ are all negative
3. indefinite if and only if $A$ has both positive and negative eigenvalues

With the help of this theorem, we can immediate tell if a quadratic form has a maximum, minimum or saddle point after calculating the eigenvalues.

##  Positive Definite Matrix

Symmetric matrices are one of most important kinds in linear algrebra, we will show they are always positive definite.

${A}$ is a symmetric matrix, premultiplying ${A}\mathbf{x}=\lambda \mathbf{x}$ by $\mathbf{x}^T$

$$
\mathbf{x}^T{A}\mathbf{x} = \lambda \mathbf{x}^T\mathbf{x} = \lambda \|\mathbf{x}\|^2
$$

$\mathbf{x}^T{A}\mathbf{x}$ must be postive, since we defined so, then $\lambda$ must be larger than $0$.

Try asking the other way around: if all eigenvalues are positive, is $A_{n\times n}$ positive definite? Yes.

Here is the Principal Axes Theorem which employs the orthogonal change of variable $\mathbf{x}=P\mathbf{y}$:

$$
Q(\mathbf{x})=\mathbf{x}^{T} A \mathbf{x}=\mathbf{y}^{T} D \mathbf{y}=\lambda_{1} y_{1}^{2}+\lambda_{2} y_{2}^{2}+\cdots+\lambda_{n} y_{n}^{2}
$$

If all of $\lambda$'s are positive, $\mathbf{x}^{T} A \mathbf{x}$ is also positive.

##  Cholesky Decomposition

Cholesky decomposition is modification of $LU$ decomposition. And it is more efficient than $LU$ algorithm.

If $A$ is positive definite matrix, i.e. $\mathbf{x}^{T} A \mathbf{x}>0$ or every eigenvalue is strictly positive. A positive definite matrix can be decomposed into a multiplication of lower triagnluar matrix and its transpose.

$$\begin{aligned}
{A}={L} {L}^{T} &=\left[\begin{array}{ccc}
l_{11} & 0 & 0 \\
l_{21} & l_{22} & 0 \\
l_{31} & l_{32} & l_{33}
\end{array}\right]\left[\begin{array}{ccc}
l_{11} & l_{21} & l_{31} \\
0 & l_{22} & l_{32} \\
0 & 0 & l_{33}
\end{array}\right] \\
\left[\begin{array}{ccc}
a_{11} & a_{21} & a_{31} \\
a_{21} & a_{22} & a_{32} \\
a_{31} & a_{32} & a_{33}
\end{array}\right]
&=\left[\begin{array}{ccc}
l_{11}^{2} &l_{21} l_{11} & l_{31} l_{11} \\
l_{21} l_{11} & l_{21}^{2}+l_{22}^{2} & l_{31} l_{21}+l_{32} l_{22} \\
l_{31} l_{11} & l_{31} l_{21}+l_{32} l_{22} & l_{31}^{2}+l_{32}^{2}+l_{33}^{2}
\end{array}\right]
\end{aligned}$$

We will show this with NumPy.

```python
A = np.array([[16, -8, -4], [-8, 29, 12], [-4, 12, 41]]); A
```

    array([[16, -8, -4],
           [-8, 29, 12],
           [-4, 12, 41]])

```python
L = sp.linalg.cholesky(A, lower = True); L
```

    array([[ 4.,  0.,  0.],
           [-2.,  5.,  0.],
           [-1.,  2.,  6.]])

Check if $LL^T=A$

```python
L@L.T
```

    array([[16., -8., -4.],
           [-8., 29., 12.],
           [-4., 12., 41.]])

#  Some Facts of Symmetric Matrices

## Rank and Positive Definiteness

If a symmetric matrix $A$ does not have full rank, which means there must be a non-trivial vector $\mathbf{v}$ satisfies

$$
A\mathbf{v} = \mathbf{0}
$$

which also means the quadratic form equals zero $\mathbf{v}^TA\mathbf{v} = \mathbf{0}$. Thus $A$ can not be a positive definite matrix if it does not have full rank.

Contrarily, a matrix to be positive definite must have full rank.

Chapter 17 - Symmetric Matrices , Quadratic Form and Cholesky Decomposition.ipynb


# Sample Coding Exercise : Interpolation
- https://www.hackerrank.com/contests/intro-to-statistics/challenges/temperature-predictions/problem
- Take care with 2-D: you may need to use the correlation in the variables to improve the fit!\

```python
%matplotlib inline
from IPython.core.display import display, HTML
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from pylab import rcParams
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pandas_profiling
from pylab import rcParams
rcParams['figure.figsize'] = 10, 6
plt.rc("font", size=14)
```

    /anaconda3/lib/python3.6/site-packages/pandas_profiling/plot.py:15: UserWarning:
    This call to matplotlib.use() has no effect because the backend has already
    been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
    or matplotlib.backends is imported for the first time.

    The backend was *originally* set to 'module://ipykernel.pylab.backend_inline' by the following code:
      File "/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
        app.launch_new_instance()
      File "/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
        app.start()
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 486, in start
        self.io_loop.start()
      File "/anaconda3/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 127, in start
        self.asyncio_loop.run_forever()
      File "/anaconda3/lib/python3.6/asyncio/base_events.py", line 422, in run_forever
        self._run_once()
      File "/anaconda3/lib/python3.6/asyncio/base_events.py", line 1432, in _run_once
        handle._run()
      File "/anaconda3/lib/python3.6/asyncio/events.py", line 145, in _run
        self._callback(*self._args)
      File "/anaconda3/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 117, in _handle_events
        handler_func(fileobj, events)
      File "/anaconda3/lib/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper
        return fn(*args, **kwargs)
      File "/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
        self._handle_recv()
      File "/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
        self._run_callback(callback, msg)
      File "/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
        callback(*args, **kwargs)
      File "/anaconda3/lib/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper
        return fn(*args, **kwargs)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
        return self.dispatch_shell(stream, msg)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
        handler(stream, idents, msg)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
        user_expressions, allow_stdin)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
        res = shell.run_cell(code, store_history=store_history, silent=silent)
      File "/anaconda3/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2662, in run_cell
        raw_cell, store_history, silent, shell_futures)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2785, in _run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2903, in run_ast_nodes
        if self.run_code(code, result):
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-1-183fefe6eb41>", line 1, in <module>
        get_ipython().run_line_magic('matplotlib', 'inline')
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2131, in run_line_magic
        result = fn(*args,**kwargs)
      File "<decorator-gen-107>", line 2, in matplotlib
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/magic.py", line 187, in <lambda>
        call = lambda f, *a, **k: f(*a, **k)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/magics/pylab.py", line 99, in matplotlib
        gui, backend = self.shell.enable_matplotlib(args.gui)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3051, in enable_matplotlib
        pt.activate_matplotlib(backend)
      File "/anaconda3/lib/python3.6/site-packages/IPython/core/pylabtools.py", line 311, in activate_matplotlib
        matplotlib.pyplot.switch_backend(backend)
      File "/anaconda3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 231, in switch_backend
        matplotlib.use(newbackend, warn=False, force=True)
      File "/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 1410, in use
        reload(sys.modules['matplotlib.backends'])
      File "/anaconda3/lib/python3.6/importlib/__init__.py", line 166, in reload
        _bootstrap._exec(spec, module)
      File "/anaconda3/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 16, in <module>
        line for line in traceback.format_stack()

      matplotlib.use(BACKEND)

```python
import os, sys, re
import calendar
import collections
from collections import defaultdict, OrderedDict

from scipy.stats import linregress

from datetime import datetime
from dateutil.relativedelta import *
import itertools
from dateutil import parser

import pandas as pd
pd.set_option('display.max_columns', 100)

import numpy as np
import scipy
import statsmodels
import statsmodels.api as sm
import statsmodels.formula.api as smf
import statsmodels.tsa.api as smt

import sympy
import requests
from bs4 import BeautifulSoup
from scipy.stats import mode
from scipy import interp

from sklearn import preprocessing, linear_model, metrics
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.cross_validation import cross_val_score
from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LinearRegression

from sklearn.model_selection import GridSearchCV
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, f1_score, classification_report, roc_curve, auc
from sklearn.pipeline import Pipeline, FeatureUnion
```

    /anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)

```python
## Data I/O Test Data Provided
- Input: I am reading in the data from copy paste from the website
- Output: ordered list printed to terminal

    #! note copy from keyboard will turn columns to strings
```

```python
df = pd.read_clipboard(header = 0)
display(df)
```

      yyyy
      month
      tmax
      tmin

      0
      1908
      January
      5.0
      -1.4

      1
      1908
      February
      7.3
      1.9

      2
      1908
      March
      6.2
      0.3

      3
      1908
      April
      Missing_1
      2.1

      4
      1908
      May
      Missing_2
      7.7

      5
      1908
      June
      17.7
      8.7

      6
      1908
      July
      Missing_3
      11.0

      7
      1908
      August
      17.5
      9.7

      8
      1908
      September
      16.3
      8.4

      9
      1908
      October
      14.6
      8.0

      10
      1908
      November
      9.6
      3.4

      11
      1908
      December
      5.8
      Missing_4

      12
      1909
      January
      5.0
      0.1

      13
      1909
      February
      5.5
      -0.3

      14
      1909
      March
      5.6
      -0.3

      15
      1909
      April
      12.2
      3.3

      16
      1909
      May
      14.7
      4.8

      17
      1909
      June
      15.0
      7.5

      18
      1909
      July
      17.3
      10.8

      19
      1909
      August
      18.8
      10.7

```python
df_answer = pd.read_clipboard(header = None)
df_answer = pd.to_numeric(df_answer[0])
df_answer = df_answer.to_frame("truth")
display(df_answer)
```

      truth

      0
      8.6

      1
      15.8

      2
      18.9

      3
      0.0

## Treat missing values in a standard way

```python
df2 = df.copy(deep = True)
df2[["tmax_clean", "tmin_clean"]] = df2[["tmax", "tmin"]].replace(to_replace= r'(?i)missing', value=np.nan, regex= True)

df2["tmax_clean"] = df2["tmax_clean"].apply(pd.to_numeric)
df2["tmin_clean"] = df2["tmin_clean"].apply(pd.to_numeric)
df2.head(15)
```

      yyyy
      month
      tmax
      tmin
      tmax_clean
      tmin_clean

      0
      1908
      January
      5.0
      -1.4
      5.0
      -1.4

      1
      1908
      February
      7.3
      1.9
      7.3
      1.9

      2
      1908
      March
      6.2
      0.3
      6.2
      0.3

      3
      1908
      April
      Missing_1
      2.1
      NaN
      2.1

      4
      1908
      May
      Missing_2
      7.7
      NaN
      7.7

      5
      1908
      June
      17.7
      8.7
      17.7
      8.7

      6
      1908
      July
      Missing_3
      11.0
      NaN
      11.0

      7
      1908
      August
      17.5
      9.7
      17.5
      9.7

      8
      1908
      September
      16.3
      8.4
      16.3
      8.4

      9
      1908
      October
      14.6
      8.0
      14.6
      8.0

      10
      1908
      November
      9.6
      3.4
      9.6
      3.4

      11
      1908
      December
      5.8
      Missing_4
      5.8
      NaN

      12
      1909
      January
      5.0
      0.1
      5.0
      0.1

      13
      1909
      February
      5.5
      -0.3
      5.5
      -0.3

      14
      1909
      March
      5.6
      -0.3
      5.6
      -0.3

### Convert to datetime index

```python
d = {"month", dict(zip(pd.date_range('2000-01-01', freq='M', periods=12).strftime('%B'), range(1,13)))}

# df2["month_number"] = df2["month"].replace(d)  Does not work in pandas 0.19
for idx,row in df2.iterrows():
    df2.loc[idx, "month_number"] = d[row["month"]]

df2["yyyy"] = df2["yyyy"].map(str)
df2["date_time"] = df2['month'] + "-" + df2["yyyy"]
df2["date_time"]  = df2["date_time"].apply(lambda x: pd.to_datetime(x,format = '%B-%Y'))
df2.set_index("date_time", inplace = True)

#pandas_profiling.ProfileReport(df2[["tmax_clean", "tmin_clean", "month_number"]])
```

# Correlation among the Temperature Min and Max Values

```python
df2.plot(x='tmin_clean', y='tmax_clean', style='o')
```

# Perform Linear interpolation  [tmin,tmax]
- leverage the correlation in the data

```python
df_answer = df_answer["truth"]
```

```python
x = df2.dropna(how='any',subset= ["tmin_clean", "tmax_clean"]).tmin_clean.values
y = df2.dropna(how='any',subset= ["tmin_clean", "tmax_clean"]).tmax_clean.values

stats = linregress(x, y)

m = stats.slope
b = stats.intercept
print(m,b)

fig2, ax2 = plt.subplots(figsize=(10,6))
plt.scatter(x, y)
plt.plot(x, m * x + b, color="red")   # I've added a color argument here

ax2.set_title("Temperature Correlation (Dropouts Removed)")
ax2.set_ylabel("Temp_Max")
ax2.set_xlabel("Temp_Min")
plt.tight_layout()
plt.savefig("TempCorrelation.png")
plt.show()

my_dict = OrderedDict()
for idx, row in df2.iterrows():
    if (("Missing" in row["tmin"]) & (not "Missing" in row["tmax"])):
        my_dict[row["tmin"]] = 1/float(m)*(row["tmax_clean"] - b)
    if (("Missing" in row["tmax"]) & (not "Missing" in row["tmin"])):
        my_dict[row["tmax"]] = m * row["tmin_clean"] + b

print(my_dict)
my_list = list(my_dict.values())
print()
for elem in my_list:
    print(elem)

df_answer = pd.concat([df_answer, pd.DataFrame(my_list, columns= ["answer_lreg",])], axis = 1)
df_answer["delta_lreg2"] = df_answer["truth"] - df_answer["answer_lreg"]

df_answer
```

## SciKit Learn Fit based on [month_number, tmin, tmax] ignoring the year.
 - Use data without Nan's as the training set
 - Use the tmin = nan as those to predict based on [month_number, tmax]
 - Use the tmax = nan as those to predict based on [month_number, tmin]

```python
df_train = df2.dropna(how='any',subset= ["tmin_clean", "tmax_clean"])
df_train = df_train[["month_number", "tmax_clean", "tmin_clean"]]

df_test = df2[df2[["tmin_clean", "tmax_clean"]].isnull().any(axis=1)]
df_test = df_test[["month_number", "tmax_clean", "tmin_clean"]]

X_train = df_train[["month_number", "tmax_clean"]].values
Y_train = df_train["tmin_clean"].values
X_mintest = df_test[df_test["tmin_clean"].isnull()][["month_number", "tmax_clean"]].values

reg = LinearRegression()
model = reg.fit(X_train, Y_train)
tmin_predict = model.predict(X_mintest)

X_train = df_train[["month_number", "tmin_clean"]].values
Y_train = df_train["tmax_clean"].values
X_maxtest = df_test[df_test["tmax_clean"].isnull()][["month_number", "tmin_clean"]].values

reg = LinearRegression()
model = reg.fit(X_train, Y_train)

tmax_predict = model.predict(X_maxtest)

df_sklearn = df2.copy(deep = True)
df_sklearn["tmax_hat"] = df_sklearn["tmax_clean"]
df_sklearn["tmin_hat"] = df_sklearn["tmin_clean"]
df_sklearn.loc[df_sklearn["tmax_clean"].isnull(),"tmax_hat"] = tmax_predict
df_sklearn.loc[df_sklearn["tmin_clean"].isnull(),"tmin_hat"] = tmin_predict

my_dict = OrderedDict()
for idx, row in df_sklearn.iterrows():
    if "Missing" in row["tmin"]:
        my_dict[row["tmin"]] = row["tmin_hat"]
    if "Missing" in row["tmax"]:
        my_dict[row["tmax"]] = row["tmax_hat"]

my_list = list(my_dict.values())
print()
for elem in my_list:
    print(elem)

df_answer = pd.concat([df_answer, pd.DataFrame(my_list, columns= ["answer_scikitreg",])], axis = 1)
df_answer["delta_scikitreg"] = df_answer["truth"] - df_answer["answer_scikitreg"]
df_answer
```

    8.662087950728417
    15.401943733483362
    19.291694067282894
    1.3625249040725382

# Apply Pandas built in interpolation methods
- https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.interpolate.html

Types of missing data:
- if upsampling is required:
      upsampled = df.series.resample('D')
- if the dates are missing
      df = df.reindex(pd.date_range("2011-01-01", "2011-10-31"), fill_value="NaN")
- if the data contains duplicates:
      df.drop_duplicates(keep = 'first', inplace = True)

- forward fill copies values forward.  Limit will impact how big a gap you will fill
https://chrisalbon.com/machine_learning/preprocessing_dates_and_times/handling_missing_values_in_time_series/
https://chrisalbon.com/python/data_wrangling/pandas_missing_data/

- methods: {‘linear’, ‘time’, ‘index’, ‘values’, ‘nearest’, ‘zero’, 'slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’,
            ‘krogh’, ‘polynomial’, ‘spline’, ‘piecewise_polynomial’, ‘from_derivatives’, ‘pchip’, ‘akima’}

      - method='quadratic'  if you are dealing with a time series that is growing at an increasing rate.
      - method='pchip' if you have values approximating a cumulative distribution function.
      - method='akima': to fill missing values with goal of smooth plotting.

```python
df_interp = df2.copy(deep = True)

df_interp["tmin_hat"] = df_interp["tmin_clean"].interpolate(axis=0, method='time',\
                         limit=None, inplace=False, limit_direction='forward', limit_area=None, downcast=None).ffill().bfill()
df_interp["tmax_hat"] = df_interp["tmax_clean"].interpolate(axis=0, method='time',\
                        limit=None, inplace=False, limit_direction='forward', limit_area=None, downcast=None).ffill().bfill()

# Print the missing values
df_pandas = df_interp[df_interp['tmin'].str.startswith("Missing") | df_interp['tmax'].str.startswith("Missing")]

my_dict = OrderedDict()
for idx, row in df_pandas.iterrows():
    if "Missing" in row["tmin"]:
        my_dict[row["tmin"]] = row["tmin_hat"]
    if "Missing" in row["tmax"]:
        my_dict[row["tmax"]] = row["tmax_hat"]
#print(my_dict)
my_list = list(my_dict.values())
print()
for elem in my_list:
    print(elem)

df_answer = pd.concat([df_answer, pd.DataFrame(my_list, columns= ["answer_pandasreg",])], axis = 1)
df_answer["delta_pandasreg"] = df_answer["truth"] - df_answer["answer_pandasreg"]
df_answer
```

    10.075
    13.825
    17.601639344262296
    1.777049180327869

      truth
      answer_lreg
      delta_lreg2
      answer_scikitreg
      delta_scikitreg
      answer_pandasreg
      delta_pandasreg

      0
      8.6
      8.668500
      -0.068500
      8.662088
      -0.062088
      10.075000
      -1.475000

      1
      15.8
      15.282367
      0.517633
      15.401944
      0.398056
      13.825000
      1.975000

      2
      18.9
      19.179825
      -0.279825
      19.291694
      -0.391694
      17.601639
      1.298361

      3
      0.0
      -0.328775
      0.328775
      1.362525
      -1.362525
      1.777049
      -1.777049

### Varia/tion on Pandas interpolation method

```python
df_interp2 = df2.copy(deep = True)

df_interp2["tmin_hat"] = df_interp2["tmin_clean"].interpolate(method='polynomial', order=2).ffill().bfill()
df_interp2["tmax_hat"] = df_interp2["tmax_clean"].interpolate(method='polynomial', order=2).ffill().bfill()

# Print the missing values
df_pandas2 = df_interp2[df_interp2['tmin'].str.startswith("Missing") | df_interp2['tmax'].str.startswith("Missing")]

my_dict = OrderedDict()
for idx, row in df_pandas2.iterrows():
    if "Missing" in row["tmin"]:
        my_dict[row["tmin"]] = row["tmin_hat"]
    if "Missing" in row["tmax"]:
        my_dict[row["tmax"]] = row["tmax_hat"]
#print(my_dict)
my_list = list(my_dict.values())
print()
for elem in my_list:
    print(elem)

df_answer = pd.concat([df_answer, pd.DataFrame(my_list, columns= ["answer_pdPolyreg",])], axis = 1)
df_answer["delta_pdPolyreg"] = df_answer["truth"] - df_answer["answer_pdPolyreg"]
df_answer
```

    8.381624067909202
    13.975349515190798
    18.423150274632224
    0.8619823925615726

      truth
      answer_lreg
      delta_lreg2
      answer_scikitreg
      delta_scikitreg
      answer_pandasreg
      delta_pandasreg
      answer_pdPolyreg
      delta_pdPolyreg

      0
      8.6
      8.668500
      -0.068500
      8.662088
      -0.062088
      10.075000
      -1.475000
      8.381624
      0.218376

      1
      15.8
      15.282367
      0.517633
      15.401944
      0.398056
      13.825000
      1.975000
      13.975350
      1.824650

      2
      18.9
      19.179825
      -0.279825
      19.291694
      -0.391694
      17.601639
      1.298361
      18.423150
      0.476850

      3
      0.0
      -0.328775
      0.328775
      1.362525
      -1.362525
      1.777049
      -1.777049
      0.861982
      -0.861982

# SCIKIT Learn is the Winner!

## Look at the Fit Constraints
 -  1908 <=time <= 2013
 -  -75 <= Tmax/Tmin <= 75

```python
df_sklearn["temp_constraint_v"] = df_sklearn["tmax_hat"]/df_sklearn["tmin_hat"]
df_sklearn[abs(df_sklearn["temp_constraint_v"]) > 75]
```

      yyyy
      month
      tmax
      tmin
      tmax_clean
      tmin_clean
      month_number
      tmax_hat
      tmin_hat
      temp_constraint_v

      date_time

# Check the Output by making some Residual Plots

```python
df_sklearn[['tmin', 'tmin_hat']].plot(figsize=(12, 8))
plt.show()

df_sklearn[['tmax', 'tmax_hat']].plot(figsize=(12, 8))
plt.show()

df_sklearn["min_resid"] = df_sklearn['tmin_clean'] - df_sklearn['tmin_hat']
df_sklearn["min_resid"].plot(figsize=(12, 8))
plt.show()

df_sklearn["max_resid"] = df_sklearn['tmax_clean'] - df_sklearn['tmax_hat']
df_sklearn["max_resid"].plot(figsize=(12, 8))
plt.show()
```

# SUBMITTED CODE

```python
import os, sys, re
import calendar
import collections
from collections import defaultdict, OrderedDict

from scipy.stats import linregress

from datetime import datetime
from dateutil.relativedelta import *
import itertools
from dateutil import parser

import pandas as pd
pd.set_option('display.max_columns', 100)

import numpy as np
import scipy
import statsmodels
import statsmodels.api as sm
import statsmodels.formula.api as smf
import statsmodels.tsa.api as smt

import sympy
import requests
from bs4 import BeautifulSoup
from scipy.stats import mode
from scipy import interp

from sklearn import preprocessing, linear_model, metrics
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.cross_validation import cross_val_score
from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LinearRegression

from sklearn.model_selection import GridSearchCV
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, f1_score, classification_report, roc_curve, auc
from sklearn.pipeline import Pipeline, FeatureUnion

if __name__ == "__main__":
    # Read Data from STDIN
    t=int(sys.stdin.readline())
    my_header = sys.stdin.readline().split()

    data = sys.stdin.read().splitlines()
    data = [re.split(r'\t', l) for l in data]
    df = pd.DataFrame(data, columns= my_header)

    # PreProcess data
    df2 = df.copy(deep = True)
    df2[["tmax_clean", "tmin_clean"]] = df2[["tmax", "tmin"]].\
                                          replace(to_replace= r'(?i)missing', value=np.nan, regex= True)
    #df2["tmax_clean"] =  = df["tmax"].replace(to_replace= r'(?i)missing', value=np.nan, regex= True)
    #df2["tmin_clean"] = df["tmin"].replace(to_replace= r'(?i)missing', value=np.nan, regex= True)

    df2["tmax_clean"] = df2["tmax_clean"].apply(pd.to_numeric)
    df2["tmin_clean"] = df2["tmin_clean"].apply(pd.to_numeric)

    # ### Convert to datetime index
    d = dict(zip(pd.date_range('2000-01-01', freq='M', periods=12).strftime('%B'), range(1,13)))
    for idx,row in df2.iterrows():
        df2.loc[idx, "month_number"] = d[row["month"]]

    df2["yyyy"] = df2["yyyy"].map(str)
    df2["date_time"] = df2['month'] + "-" + df2["yyyy"]
    df2["date_time"]  = df2["date_time"].apply(lambda x: pd.to_datetime(x,format = '%B-%Y'))
    df2.set_index("date_time", inplace = True)

    # # SciKit Learn interpolation methods

    df_train = df2.dropna(how='any',subset= ["tmin_clean", "tmax_clean"])
    df_train = df_train[["month_number", "tmax_clean", "tmin_clean"]]

    df_test = df2[df2[["tmin_clean", "tmax_clean"]].isnull().any(axis=1)]
    df_test = df_test[["month_number", "tmax_clean", "tmin_clean"]]

    X_train = df_train[["month_number", "tmax_clean"]].values
    Y_train = df_train["tmin_clean"].values
    X_mintest = df_test[(df_test["tmin_clean"].isnull()) &\
                        (df_test["tmax_clean"].notnull())][["month_number", "tmax_clean"]].values
    reg = LinearRegression()
    model = reg.fit(X_train, Y_train)
    tmin_predict = model.predict(X_mintest)

    X_train = df_train[["month_number", "tmin_clean"]].values
    Y_train = df_train["tmax_clean"].values
    X_maxtest = df_test[(df_test["tmax_clean"].isnull()) &\
                                (df_test["tmin_clean"].notnull())][["month_number", "tmin_clean"]].values

    reg = LinearRegression()
    model = reg.fit(X_train, Y_train)

    tmax_predict = model.predict(X_maxtest)

    df_sklearn = df2.copy(deep = True)
    df_sklearn["tmax_hat"] = df_sklearn["tmax_clean"]
    df_sklearn["tmin_hat"] = df_sklearn["tmin_clean"]

    df_sklearn.loc[((df_sklearn["tmax_clean"].isnull()) &\
                    (df_sklearn["tmin_clean"].notnull())), "tmax_hat"] = tmax_predict
    df_sklearn.loc[((df_sklearn["tmin_clean"].isnull()) &\
                    (df_sklearn["tmax_clean"].notnull())), "tmin_hat"] = tmin_predict

    my_dict = OrderedDict()
    for idx, row in df_sklearn.iterrows():
        if "Missing" in row["tmin"]:
            my_dict[row["tmin"]] = row["tmin_hat"]
        if "Missing" in row["tmax"]:
            my_dict[row["tmax"]] = row["tmax_hat"]

    my_list = list(my_dict.values())
    print()
    for elem in my_list:
        print(elem)
```

# PostMortem
 - The second test failed initially b/c I did not know how to get it to read the STD input correctly.
 - Below, I downloaded that file and ran it locally here

```python
df_case2 = pd.read_csv("test_case#2.tsv", sep = '\t', header = 1)
df_case2answer= pd.read_csv("answer_case#2", sep = '\t', header = None, names = ["truth"])
```

```python
if __name__ == "__main__":

    df2 = df_case2.copy(deep = True)
    df2[["tmax_clean", "tmin_clean"]] = df2[["tmax", "tmin"]].\
                                          replace(to_replace= r'(?i)missing', value=np.nan, regex= True)
    #df2["tmax_clean"] =  = df["tmax"].replace(to_replace= r'(?i)missing', value=np.nan, regex= True)
    #df2["tmin_clean"] = df["tmin"].replace(to_replace= r'(?i)missing', value=np.nan, regex= True)

    df2["tmax_clean"] = df2["tmax_clean"].apply(pd.to_numeric)
    df2["tmin_clean"] = df2["tmin_clean"].apply(pd.to_numeric)

    # ### Convert to datetime index
    d = dict(zip(pd.date_range('2000-01-01', freq='M', periods=12).strftime('%B'), range(1,13)))
    for idx,row in df2.iterrows():
        df2.loc[idx, "month_number"] = d[row["month"]]

    df2["yyyy"] = df2["yyyy"].map(str)
    df2["date_time"] = df2['month'] + "-" + df2["yyyy"]
    df2["date_time"]  = df2["date_time"].apply(lambda x: pd.to_datetime(x,format = '%B-%Y'))
    df2.set_index("date_time", inplace = True)

    # # SciKit Learn interpolation methods

    df_train = df2.dropna(how='any',subset= ["tmin_clean", "tmax_clean"])
    df_train = df_train[["month_number", "tmax_clean", "tmin_clean"]]

    df_test = df2[df2[["tmin_clean", "tmax_clean"]].isnull().any(axis=1)]
    df_test = df_test[["month_number", "tmax_clean", "tmin_clean"]]

    X_train = df_train[["month_number", "tmax_clean"]].values
    Y_train = df_train["tmin_clean"].values
    X_mintest = df_test[(df_test["tmin_clean"].isnull()) &\
                        (df_test["tmax_clean"].notnull())][["month_number", "tmax_clean"]].values
    reg = LinearRegression()
    model = reg.fit(X_train, Y_train)
    tmin_predict = model.predict(X_mintest)

    X_train = df_train[["month_number", "tmin_clean"]].values
    Y_train = df_train["tmax_clean"].values
    X_maxtest = df_test[(df_test["tmax_clean"].isnull()) &\
                                (df_test["tmin_clean"].notnull())][["month_number", "tmin_clean"]].values

    reg = LinearRegression()
    model = reg.fit(X_train, Y_train)

    tmax_predict = model.predict(X_maxtest)

    df_sklearn = df2.copy(deep = True)
    df_sklearn["tmax_hat"] = df_sklearn["tmax_clean"]
    df_sklearn["tmin_hat"] = df_sklearn["tmin_clean"]

    df_sklearn.loc[((df_sklearn["tmax_clean"].isnull()) &\
                    (df_sklearn["tmin_clean"].notnull())), "tmax_hat"] = tmax_predict
    df_sklearn.loc[((df_sklearn["tmin_clean"].isnull()) &\
                    (df_sklearn["tmax_clean"].notnull())), "tmin_hat"] = tmin_predict

    my_dict = OrderedDict()
    for idx, row in df_sklearn.iterrows():
        if "Missing" in row["tmin"]:
            my_dict[row["tmin"]] = row["tmin_hat"]
        if "Missing" in row["tmax"]:
            my_dict[row["tmax"]] = row["tmax_hat"]

    my_list = list(my_dict.values())
    print()
    for elem in my_list:
        print(elem)
```

    8.023567576598913
    14.956071972588784
    18.876006747611
    1.4060334342946659
    7.502582149144462
    2.940294910002235
    7.074080962002098
    0.2394289909028684
    12.897675794501763
    5.857332293815556
    0.9488187331023301
    9.90315408171065
    13.571314830987102
    17.608417930900735
    3.1853997123960625
    7.124921110525822
    10.559753472400955
    6.397686123397952
    8.770170965854728
    13.740795283767309
    3.7887345051962704
    17.99479765750084
    11.097168013436823
    5.953237861611667
    -0.6042969976989871
    19.00189376048388
    15.998042827497684
    12.356038142165625
    0.22528834240129836
    16.978982866536334
    4.165818750696401
    9.030663679581954
    8.914723440701952
    18.2465716832466
    19.262386474211105
    14.990946724514643
    5.857332293815556
    0.1498714933012728
    3.8005183504017506
    9.473279484701361
    15.242720750260407
    1.4460985765032852
    2.2309051678027734
    3.543629702802444
    4.856354237802115
    19.388273487083985
    -0.514739500097392
    8.657834793102644
    17.11358856739068
    10.845393987691065
    15.594225724934653
    11.58495125950209
    8.177052401499349
    5.072028771501506
    -0.10701715429803382
    3.4988509540016457
    17.634573994845127
    11.0535745735295
    11.735784957702142
    9.609972534399844
    14.461242609078731
    3.813158642772704
    6.864428396798598
    7.706937353850524
    6.755978759888647
    9.699530032001439
    10.635170321500981
    3.3362334105961144
    10.163646795437874
    17.374081281117903
    5.942828225501708
    3.109982863296035
    6.8731470847800615
    10.967475818200311
    6.096018726997849
    9.217581508325086
    19.262386474211105
    3.4870671087961655
    0.4256143959943275
    8.023567576598913
    11.238505114301187
    2.8837323159959585
    9.291807686202082
    19.640047512829746
    1.7477659729033883
    10.541307834056514
    7.601998905702279
    3.600084750085178
    -0.4841013997981638
    6.630091747015767
    5.565743980001578
    10.107252377800798
    13.740795283767309
    -2.4449394763988406
    6.864428396798598
    6.369599033288542
    3.260816561496088
    -1.3443248401976784
    4.750299288402861
    10.861420868801057
    6.063023823402531
    19.631328824848282
    11.60943475290981
    -1.4197416892977044
    11.170742898420915
    12.438331715131183
    18.884725435592465
    9.156550692454834
    16.375703866116325
    12.985473206530026
    -0.10701715429803382
    18.381177384100944
    2.6127030198950827
    17.23075689228209
    6.881865772761527
    8.778889653836194
    9.72984824779807
    -1.3136867398984502
    7.888961875744568
    15.368607763133284
    6.096018726997849
    9.383721987099767
    8.965807482579326
    5.0071879360021665
    3.109982863296035
    1.8585345288948218
    0.1498714933012728
    10.4536985230017
    1.099652431302383
    3.1665454573023144
    17.986078969519372
    8.89605797872761
    8.73325164220267
    11.78056370650294
    3.4870671087961655
    9.744308780802237
    0.9181806328031028
    0.12394699959422262
    -3.531413464098433
    9.895142479002288
    15.872155814624808
    7.502582149144462
    15.585507036953185
    17.23075689228209
    16.753364904734966
    6.834446951721827
    -1.1628530416983982
    3.0157117591022615
    18.62423272186524
    17.508686981972247
    9.469355534070846
    8.653002640963313
    3.1971835576015426
    18.363740008138013
    1.250486129502435
    8.140735901490329
    0.11923339300204461
    18.381177384100944
    7.086220977467589
    19.37955479910252
    11.087671416101136
    8.509678252127504
    10.724727819102576
    6.322269274297927
    1.2952648783032323
    10.163646795437874
    7.3003315093021754
    13.199343190901866
    8.478719797899453
    13.740795283767309
    3.109982863296035
    3.24196230640234
    9.85036373020149
    9.956418679600745
    7.679772558098396
    17.104869879409215
    7.1518546143982125
    5.227897229451155
    5.6105227288023745
    11.163088265201162
    2.940294910002235
    17.104869879409215
    12.596008398101658
    17.608417930900735
    13.992569309513069
    15.468338712061769
    18.758838422719585
    -3.576192212899231
    10.121393026302366
    10.529115372101726
    13.363134245148668
    6.213857521602584
    3.109982863296035
    9.774946881101467
    14.244343335258826
    8.73325164220267
    3.7698802501025224
    6.6969967165019675
    14.838903647697371
    20.89019895357708
    6.486767358179957
    8.887339290746144
    3.1665454573023144
    19.25366778622964
    2.46186932169503
    -0.026886698605829373
    11.841839907101397
    20.26076388921268
    17.508686981972247
    9.390887342237665
    10.163646795437874
    1.3259029786024605
    12.037452354102248
    9.847016572689487
    6.171435576097876
    8.280750547602514
    9.139113316491905
    6.5485198215980045
    3.2160378126952898
    9.508055667129081
    10.65847615894793
    7.752832603902332
    18.138122046336647
    4.014985052496348
    20.27820126517561
    9.60396123492519
    13.562596143005639
    9.685389383499869
    10.263377744366363
    14.830184959715904
    11.615589359801318
    7.378105161698292
    21.393747005068604
    18.884725435592465
    6.699353519798057
    0.31484584000289484
    2.3817388660028254
    16.853095853663454
    11.20786701400196
    5.579884628503146

```python
df_case2answer = pd.concat([df_case2answer, pd.DataFrame(my_list, columns= ["answer_scikit2",])], axis = 1)
df_case2answer["delta_scikit2"] = df_case2answer["truth"] - df_case2answer["answer_scikit2"]
df_case2answer
```

      truth
      answer_scikit2
      delta_scikit2

      0
      8.6
      8.023568
      0.576432

      1
      15.8
      14.956072
      0.843928

      2
      18.9
      18.876007
      0.023993

      3
      0.0
      1.406033
      -1.406033

      4
      7.0
      7.502582
      -0.502582

      5
      2.2
      2.940295
      -0.740295

      6
      6.0
      7.074081
      -1.074081

      7
      1.8
      0.239429
      1.560571

      8
      12.5
      12.897676
      -0.397676

      9
      4.8
      5.857332
      -1.057332

      10
      1.3
      0.948819
      0.351181

      11
      9.0
      9.903154
      -0.903154

      12
      14.9
      13.571315
      1.328685

      13
      16.7
      17.608418
      -0.908418

      14
      3.8
      3.185400
      0.614600

      15
      7.2
      7.124921
      0.075079

      16
      10.1
      10.559753
      -0.459753

      17
      7.3
      6.397686
      0.902314

      18
      8.5
      8.770171
      -0.270171

      19
      12.5
      13.740795
      -1.240795

      20
      3.5
      3.788735
      -0.288735

      21
      17.5
      17.994798
      -0.494798

      22
      11.2
      11.097168
      0.102832

      23
      6.6
      5.953238
      0.646762

      24
      0.3
      -0.604297
      0.904297

      25
      18.4
      19.001894
      -0.601894

      26
      15.8
      15.998043
      -0.198043

      27
      12.5
      12.356038
      0.143962

      28
      -0.9
      0.225288
      -1.125288

      29
      18.7
      16.978983
      1.721017

      ...
      ...
      ...
      ...

      210
      11.3
      10.163647
      1.136353

      211
      0.1
      1.325903
      -1.225903

      212
      12.4
      12.037452
      0.362548

      213
      10.5
      9.847017
      0.652983

      214
      5.9
      6.171436
      -0.271436

      215
      8.1
      8.280751
      -0.180751

      216
      7.9
      9.139113
      -1.239113

      217
      7.1
      6.548520
      0.551480

      218
      2.6
      3.216038
      -0.616038

      219
      8.8
      9.508056
      -0.708056

      220
      9.7
      10.658476
      -0.958476

      221
      6.6
      7.752833
      -1.152833

      222
      18.3
      18.138122
      0.161878

      223
      2.9
      4.014985
      -1.114985

      224
      19.6
      20.278201
      -0.678201

      225
      8.5
      9.603961
      -1.103961

      226
      15.1
      13.562596
      1.537404

      227
      9.8
      9.685389
      0.114611

      228
      8.9
      10.263378
      -1.363378

      229
      15.4
      14.830185
      0.569815

      230
      12.8
      11.615589
      1.184411

      231
      7.6
      7.378105
      0.221895

      232
      20.0
      21.393747
      -1.393747

      233
      18.8
      18.884725
      -0.084725

      234
      6.9
      6.699354
      0.200646

      235
      1.0
      0.314846
      0.685154

      236
      3.6
      2.381739
      1.218261

      237
      18.2
      16.853096
      1.346904

      238
      10.5
      11.207867
      -0.707867

      239
      4.6
      5.579885
      -0.979885

240 rows × 3 columns


```python
def f_demo5(x):
    return x[0]**2+x[1]**2+x[2]**3+(1-x[3])**2,[],[x[0]**2+x[1]**2-1,x[0]**2+x[2]**2-1]
```

# Exercise 5, answers

## Problem 1

```python
import numpy as np
import ad

#if k=0, returns the gradient of lagrangian, if k=1, returns the hessian
def diff_L(f,x,m,k):
    #Define the lagrangian for given m and f
    L = lambda x_: f(x_)[0] + (np.matrix(f(x_)[2])*np.matrix(m).transpose())[0,0]
    return ad.gh(L)[k](x)
#Returns the gradients of the equality constraints
def grad_h(f,x):
    return  [ad.gh(lambda y:
            f(y)[2][i])[0](x) for i in range(len(f(x)[2]))]

#Solves the quadratic problem inside the SQP method
def solve_QP(f,x,m):
    left_side_first_row = np.concatenate((
    np.matrix(diff_L(f,x,m,1)),
    np.matrix(grad_h(f,x)).transpose()),axis=1)
    left_side_second_row = np.concatenate((
    np.matrix(grad_h(f,x)),
    np.matrix(np.zeros((len(f(x)[2]),len(f(x)[2]))))),axis=1)
    right_hand_side = np.concatenate((
    -1*np.matrix(diff_L(f,x,m,0)).transpose(),
    -np.matrix(f(x)[2]).transpose()),axis = 0)
    left_hand_side = np.concatenate((
                                    left_side_first_row,
                                    left_side_second_row),axis = 0)
    temp = np.linalg.solve(left_hand_side,right_hand_side)
    return temp[:len(x)],temp[len(x):]

def SQP(f,start,precision):
    x = start
    m = np.ones(len(f(x)[2]))
    f_old = float('inf')
    f_new = f(x)[0]
    while abs(f_old-f_new)>precision:
        print x
        f_old = f_new
        (p,v) = solve_QP(f,x,m)
        x = x+np.array(p.transpose())[0]
        m = m+v
        f_new = f(x)[0]
    return x
```

```python
SQP(f_demo5,[0.1,0.1,0.1,1.],0.00001)
```

    [0.1, 0.1, 0.1, 1.0]
    [ 2.66904762  2.43095238  2.43095238  1.        ]
    [ 1.43316532  1.31285412  1.31285412  1.        ]
    [ 0.89811129  0.8391126   0.8391126   1.        ]
    [ 0.7232535   0.72194696  0.72194696  1.        ]
    [ 0.5275908   0.88728073  0.88728073  1.        ]
    [ 3.07171309 -0.66247127 -0.66247127  1.        ]
    [ 1.69814965 -0.33347293 -0.33347293  1.        ]
    [ 1.13944518 -0.18745083 -0.18745083  1.        ]
    [ 1.00162637 -0.13570611 -0.13570611  1.        ]
    [ 0.993672   -0.11456882 -0.11456882  1.        ]
    [ 0.9951009  -0.09994974 -0.09994974  1.        ]
    [ 0.99614324 -0.08849291 -0.08849291  1.        ]
    [ 0.99689678 -0.07926269 -0.07926269  1.        ]
    [ 0.99745626 -0.07168508 -0.07168508  1.        ]
    [ 0.99788143 -0.06536638 -0.06536638  1.        ]
    [ 0.99821115 -0.06002606 -0.06002606  1.        ]
    [ 0.99847142 -0.05545938 -0.05545938  1.        ]
    [ 0.9986801  -0.05151381 -0.05151381  1.        ]
    [ 0.99884973 -0.04807369 -0.04807369  1.        ]
    [ 0.99898932 -0.04504989 -0.04504989  1.        ]
    [ 0.99910547 -0.04237271 -0.04237271  1.        ]
    [ 0.99920306 -0.03998695 -0.03998695  1.        ]

    array([ 0.99928579, -0.03784837, -0.03784837,  1.        ])

### What is going on in here?

* Many feasible solutions actually satisfy KKT conditions, but are not global optima

* In addition, the problem has multiple local but not global optima

## Problem 2

```python
def augmented_langrangian(f,x,mu,c):
    second_term = float(numpy.matrix(mu)*numpy.matrix(f(x)[2]).
                    transpose())
    third_term = 0.5*c*numpy.linalg.norm(f(x)[2])**2
    return f(x)[0]-second_term+third_term
```

```python
from scipy.optimize import minimize
import numpy
def augmented_langrangian_method(f,start,mu0,c0):
    x_old = [float('inf')]*4
    x_new = start
    mu = mu0
    c = c0
    while numpy.linalg.norm(f(x_new)[2])>0.000001:
        res = minimize(lambda x:augmented_langrangian(f,x,mu,c),x_new)
        x_old = x_new
        mu = mu-c*numpy.matrix(f(res.x)[2])
        x_new = res.x
        c=2*c
    return x_new,c
```

```python
%pdb
```

    Automatic pdb calling has been turned ON

```python
augmented_langrangian_method(f_demo5,[1.,1.,1,1.],[1,1],1)
```

    (array([  1.93341446e-07,   1.00000001e+00,  -1.00000002e+00,
              1.00000002e+00]), 256)

## Problem 3

Now, the stability rule becomes
$$
(2x_1,2x_2)-\mu(1,1) = (0,0),
$$
and completementary rule becomes
$$
\mu(x_1+x_2-1) = 0.
$$
Thus, we need to have
$$
\left\{
\begin{align}
2x_1-\mu=0\\
2x_2-\mu=0\\
\mu(x_1+x_2-1) = 0.
\end{align}
\right.
$$
Now deducting equation (2), from equation (1) gives $2x_1-2x_2=0$, thus $x_1=x_2$.  Now if $\mu= 0$, then $x_1=x_2=0$. However, this solution is not feasible. Thus, $\mu\neq0$, which implies $x_1+x_2-1=0$, which gives $x_1=x_2=\frac12$ and, thus, $\mu=1$. These variables satisfy KKT conditions.

Because the problem is quadratic, it has an optimal solution. Since only solution satisfies KKT conditions, this solution optimal.

## Problem 4

Now,
$$
\begin{align}
\nabla_x L_c(x^*,\mu^*)& = \nabla f(x^*)+\sum_{k=1}^K \mu^*_k\nabla h_k(x^*)+c\nabla(\sum_{k=1}^Kh_k(x^*)^2)\\
&=\nabla f(x^*)+\sum_{k=1}^K \mu^*_k\nabla h_k(x^*)+2c\sum_{k=1}^Kh_k(x^*)\nabla h_k(x^*)\\
&=0+2c\sum_{k=1}^K0\nabla h_k(x^*)=0.
\end{align}
$$
The first zero is given by the KKT conditions and the second zero is due to the solution being feasible.