# Classification using Gaussian distributions

For this weeks assignment, you will implement a *classification* algorithm, which is the other main type of *supervised learning* algorithm (as opposed to *regression*, from last week). Specificially, you will build a *Bayes classifier* using *Gaussian distributions* to estimate the likelihood of continuous variables.

For this we will work with a classic dataset from machine learning; Fisher's Iris dataset. The dataset contains the measurements of *length* and *width* of the *sepals* and *petals* of 150 flowers.

Using these 4 attributes (*length* and *width* of both *sepals* and *petals*), the flowers should then be classified as as one of 3 species of Iris flower:

* Iris setosa
* Iris versicolor
* Iris virginica

This dataset is such a classic example that is even included in machine learning libraries. To get started we will install the [scikit-learn](http://scikit-learn.org/stable/index.html) library. For now, we will only use it to load in the dataset, but in later assignments we will use more of its capabilities. Install the library with

    sudo pip3 install scikit-learn

and also install the plotting library [seaborn](https://seaborn.pydata.org/) with

    sudo pip3 install seaborn

Now the code below should store and show the dataset as the variable `iris`

```python
import math
import matplotlib.pylab as plt
import numpy as np

from sklearn import datasets

iris = datasets.load_iris()
print(iris)
```

    {'data': array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2],
           [5.4, 3.9, 1.7, 0.4],
           [4.6, 3.4, 1.4, 0.3],
           [5. , 3.4, 1.5, 0.2],
           [4.4, 2.9, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.1],
           [5.4, 3.7, 1.5, 0.2],
           [4.8, 3.4, 1.6, 0.2],
           [4.8, 3. , 1.4, 0.1],
           [4.3, 3. , 1.1, 0.1],
           [5.8, 4. , 1.2, 0.2],
           [5.7, 4.4, 1.5, 0.4],
           [5.4, 3.9, 1.3, 0.4],
           [5.1, 3.5, 1.4, 0.3],
           [5.7, 3.8, 1.7, 0.3],
           [5.1, 3.8, 1.5, 0.3],
           [5.4, 3.4, 1.7, 0.2],
           [5.1, 3.7, 1.5, 0.4],
           [4.6, 3.6, 1. , 0.2],
           [5.1, 3.3, 1.7, 0.5],
           [4.8, 3.4, 1.9, 0.2],
           [5. , 3. , 1.6, 0.2],
           [5. , 3.4, 1.6, 0.4],
           [5.2, 3.5, 1.5, 0.2],
           [5.2, 3.4, 1.4, 0.2],
           [4.7, 3.2, 1.6, 0.2],
           [4.8, 3.1, 1.6, 0.2],
           [5.4, 3.4, 1.5, 0.4],
           [5.2, 4.1, 1.5, 0.1],
           [5.5, 4.2, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.2],
           [5. , 3.2, 1.2, 0.2],
           [5.5, 3.5, 1.3, 0.2],
           [4.9, 3.6, 1.4, 0.1],
           [4.4, 3. , 1.3, 0.2],
           [5.1, 3.4, 1.5, 0.2],
           [5. , 3.5, 1.3, 0.3],
           [4.5, 2.3, 1.3, 0.3],
           [4.4, 3.2, 1.3, 0.2],
           [5. , 3.5, 1.6, 0.6],
           [5.1, 3.8, 1.9, 0.4],
           [4.8, 3. , 1.4, 0.3],
           [5.1, 3.8, 1.6, 0.2],
           [4.6, 3.2, 1.4, 0.2],
           [5.3, 3.7, 1.5, 0.2],
           [5. , 3.3, 1.4, 0.2],
           [7. , 3.2, 4.7, 1.4],
           [6.4, 3.2, 4.5, 1.5],
           [6.9, 3.1, 4.9, 1.5],
           [5.5, 2.3, 4. , 1.3],
           [6.5, 2.8, 4.6, 1.5],
           [5.7, 2.8, 4.5, 1.3],
           [6.3, 3.3, 4.7, 1.6],
           [4.9, 2.4, 3.3, 1. ],
           [6.6, 2.9, 4.6, 1.3],
           [5.2, 2.7, 3.9, 1.4],
           [5. , 2. , 3.5, 1. ],
           [5.9, 3. , 4.2, 1.5],
           [6. , 2.2, 4. , 1. ],
           [6.1, 2.9, 4.7, 1.4],
           [5.6, 2.9, 3.6, 1.3],
           [6.7, 3.1, 4.4, 1.4],
           [5.6, 3. , 4.5, 1.5],
           [5.8, 2.7, 4.1, 1. ],
           [6.2, 2.2, 4.5, 1.5],
           [5.6, 2.5, 3.9, 1.1],
           [5.9, 3.2, 4.8, 1.8],
           [6.1, 2.8, 4. , 1.3],
           [6.3, 2.5, 4.9, 1.5],
           [6.1, 2.8, 4.7, 1.2],
           [6.4, 2.9, 4.3, 1.3],
           [6.6, 3. , 4.4, 1.4],
           [6.8, 2.8, 4.8, 1.4],
           [6.7, 3. , 5. , 1.7],
           [6. , 2.9, 4.5, 1.5],
           [5.7, 2.6, 3.5, 1. ],
           [5.5, 2.4, 3.8, 1.1],
           [5.5, 2.4, 3.7, 1. ],
           [5.8, 2.7, 3.9, 1.2],
           [6. , 2.7, 5.1, 1.6],
           [5.4, 3. , 4.5, 1.5],
           [6. , 3.4, 4.5, 1.6],
           [6.7, 3.1, 4.7, 1.5],
           [6.3, 2.3, 4.4, 1.3],
           [5.6, 3. , 4.1, 1.3],
           [5.5, 2.5, 4. , 1.3],
           [5.5, 2.6, 4.4, 1.2],
           [6.1, 3. , 4.6, 1.4],
           [5.8, 2.6, 4. , 1.2],
           [5. , 2.3, 3.3, 1. ],
           [5.6, 2.7, 4.2, 1.3],
           [5.7, 3. , 4.2, 1.2],
           [5.7, 2.9, 4.2, 1.3],
           [6.2, 2.9, 4.3, 1.3],
           [5.1, 2.5, 3. , 1.1],
           [5.7, 2.8, 4.1, 1.3],
           [6.3, 3.3, 6. , 2.5],
           [5.8, 2.7, 5.1, 1.9],
           [7.1, 3. , 5.9, 2.1],
           [6.3, 2.9, 5.6, 1.8],
           [6.5, 3. , 5.8, 2.2],
           [7.6, 3. , 6.6, 2.1],
           [4.9, 2.5, 4.5, 1.7],
           [7.3, 2.9, 6.3, 1.8],
           [6.7, 2.5, 5.8, 1.8],
           [7.2, 3.6, 6.1, 2.5],
           [6.5, 3.2, 5.1, 2. ],
           [6.4, 2.7, 5.3, 1.9],
           [6.8, 3. , 5.5, 2.1],
           [5.7, 2.5, 5. , 2. ],
           [5.8, 2.8, 5.1, 2.4],
           [6.4, 3.2, 5.3, 2.3],
           [6.5, 3. , 5.5, 1.8],
           [7.7, 3.8, 6.7, 2.2],
           [7.7, 2.6, 6.9, 2.3],
           [6. , 2.2, 5. , 1.5],
           [6.9, 3.2, 5.7, 2.3],
           [5.6, 2.8, 4.9, 2. ],
           [7.7, 2.8, 6.7, 2. ],
           [6.3, 2.7, 4.9, 1.8],
           [6.7, 3.3, 5.7, 2.1],
           [7.2, 3.2, 6. , 1.8],
           [6.2, 2.8, 4.8, 1.8],
           [6.1, 3. , 4.9, 1.8],
           [6.4, 2.8, 5.6, 2.1],
           [7.2, 3. , 5.8, 1.6],
           [7.4, 2.8, 6.1, 1.9],
           [7.9, 3.8, 6.4, 2. ],
           [6.4, 2.8, 5.6, 2.2],
           [6.3, 2.8, 5.1, 1.5],
           [6.1, 2.6, 5.6, 1.4],
           [7.7, 3. , 6.1, 2.3],
           [6.3, 3.4, 5.6, 2.4],
           [6.4, 3.1, 5.5, 1.8],
           [6. , 3. , 4.8, 1.8],
           [6.9, 3.1, 5.4, 2.1],
           [6.7, 3.1, 5.6, 2.4],
           [6.9, 3.1, 5.1, 2.3],
           [5.8, 2.7, 5.1, 1.9],
           [6.8, 3.2, 5.9, 2.3],
           [6.7, 3.3, 5.7, 2.5],
           [6.7, 3. , 5.2, 2.3],
           [6.3, 2.5, 5. , 1.9],
           [6.5, 3. , 5.2, 2. ],
           [6.2, 3.4, 5.4, 2.3],
           [5.9, 3. , 5.1, 1.8]]), 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]), 'target_names': array(['setosa', 'versicolor', 'virginica'], dtype='<U10'), 'DESCR': '.. _iris_dataset:\n\nIris plants dataset\n--------------------\n\n**Data Set Characteristics:**\n\n    :Number of Instances: 150 (50 in each of three classes)\n    :Number of Attributes: 4 numeric, predictive attributes and the class\n    :Attribute Information:\n        - sepal length in cm\n        - sepal width in cm\n        - petal length in cm\n        - petal width in cm\n        - class:\n                - Iris-Setosa\n                - Iris-Versicolour\n                - Iris-Virginica\n                \n    :Summary Statistics:\n\n    ============== ==== ==== ======= ===== ====================\n                    Min  Max   Mean    SD   Class Correlation\n    ============== ==== ==== ======= ===== ====================\n    sepal length:   4.3  7.9   5.84   0.83    0.7826\n    sepal width:    2.0  4.4   3.05   0.43   -0.4194\n    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)\n    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)\n    ============== ==== ==== ======= ===== ====================\n\n    :Missing Attribute Values: None\n    :Class Distribution: 33.3% for each of 3 classes.\n    :Creator: R.A. Fisher\n    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n    :Date: July, 1988\n\nThe famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\nfrom Fisher\'s paper. Note that it\'s the same as in R, but not as in the UCI\nMachine Learning Repository, which has two wrong data points.\n\nThis is perhaps the best known database to be found in the\npattern recognition literature.  Fisher\'s paper is a classic in the field and\nis referenced frequently to this day.  (See Duda & Hart, for example.)  The\ndata set contains 3 classes of 50 instances each, where each class refers to a\ntype of iris plant.  One class is linearly separable from the other 2; the\nlatter are NOT linearly separable from each other.\n\n.. topic:: References\n\n   - Fisher, R.A. "The use of multiple measurements in taxonomic problems"\n     Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to\n     Mathematical Statistics" (John Wiley, NY, 1950).\n   - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.\n   - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System\n     Structure and Classification Rule for Recognition in Partially Exposed\n     Environments".  IEEE Transactions on Pattern Analysis and Machine\n     Intelligence, Vol. PAMI-2, No. 1, 67-71.\n   - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions\n     on Information Theory, May 1972, 431-433.\n   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II\n     conceptual clustering system finds 3 classes in the data.\n   - Many, many more ...', 'feature_names': ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)'], 'filename': '/anaconda3/lib/python3.7/site-packages/sklearn/datasets/data/iris.csv'}

## Separate and plot the data [1 pt]

The `iris` variable now contains an object with all sorts of information about the dataset. The 2 most important atributes are *target* and *data* containing respectively the labels and the data points for the 150 flowers. An attribute of this data object can be accessed with **.** as shown below.

Use this to create the matrix $X$ and the vector $R$. Then divide the matrix $X$ into 3 parts, $C_0$, $C_1$ and $C_2$, each containing the data for a different class, as labeled by the vector $R$. *Hint: Some clever [indexing](https://docs.scipy.org/doc/numpy/user/basics.indexing.html#boolean-or-mask-index-arrays) might help here.*

```python
# YOUR SOLUTION HERE
X = iris.data
R = iris.target
C0 = iris.data[iris.target == 0]
C1 = iris.data[iris.target == 1]
C2 = iris.data[iris.target == 2]
```

Lets plot this data, so we can see what it looks like. We have prepared some code below that plots a so-called scatter matrix. This is a matrix of scatter plots that can be used to easily identify correlations and see the seperability of the dataset. The diagonals of this plot show how the data is distributed for every *single* feature, while the non-diagonals show scatter plots using *two* of the features as the $x$ and $y$ axis, with all the different possible configurations shown in the matrix.

The different colors here correspond to the different classes of Iris, which is the same separation you just made, dividing the data points into $C_0$, $C_1$ and $C_2$.

```python
import seaborn as sns

df_iris = sns.load_dataset('iris')
sns.pairplot(df_iris, hue='species')
plt.show()
```

Looking at these plots, it should be quite clear that the Setosa species is more easily separated than Versicolor and Virginica. Also, it seems that quite a few of the variable combinations are correlated with each other.

## Inspect and estimate the correlations [1pt]

One measure of how variables are correlated is the covariance. The covariance between two jointly distributed real-valued random variables $X$ and $Y$  is defined as the expected product of their deviations from their individual expected values:

\begin{align}
\operatorname{cov}(X, Y)
&= \operatorname{E}\left[\left(X - \operatorname{E}\left[X\right]\right) \left(Y - \operatorname{E}\left[Y\right]\right)\right] \\
&= \operatorname{E}\left[X Y - X \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] Y + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right]\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right],
\end{align}

When covariance is positive, this means that the variables are positively correlated: i.e. when one variable increases in value, it is probable that the other variable also increases. When covariance is negative, this means that the variables are negatively correlated: i.e. when one variable increases in value, it is probable that the other variable decreases. When covariance is near zero, there is no correlation between the variables.

In the cell below we have provided you with a small bit of code that should help you to fit a first Gaussian distribution to the Versicolor part of the data. The code generates some samples from a 2 dimensional Gaussian distribution, using a *mean vector* and a *covariance matrix* and plots those together with the data. The mean vector and both the standard deviations have already been provided, but to complete the covariance matrix, the covariance between the two features should also be set.

Together, this is enough to start generating samples, and the more the different distributions overlap, the better the fit of the generated distribution. Manually change the variable `covar` such that the distribution of the generated samples overlaps as much as possible with the real data.

```python
target_class = 1
x_feature = 0
y_feature = 2

std_x = 0.25
std_y = 0.22

covar = 0.18 # Change me

cov = np.array([[std_x, covar], [covar, std_y]])

mean = [np.mean(C1[:, x_feature]), np.mean(C1[:, y_feature])]

samples = np.random.multivariate_normal(mean, cov, size=100)

plt.scatter(C1[:, x_feature], C1[:, y_feature], label='Real data')
plt.scatter(*zip(*samples), label='Generated samples')
plt.title(iris.target_names[target_class])
plt.xlabel(iris.feature_names[x_feature])
plt.ylabel(iris.feature_names[y_feature])
plt.legend()
plt.show()

print(cov)
```

**How do the values of `std_x` and `std_y` affect the shape of the resulting distribution?**

The greater `std_x` is, the more spread out the distribution is in the direction of the x axis;
the greater `std_y` is, the more spread out the distribution is in the direction of the y axis;

**How are the sepal length and petal length correlated for the Versicolor class?**

When I set `cover` to 0.18, the generated samples overlaps a lot with the real data. So the sepal length and petal length are positively correlated.

## Validation sets [1 pt]

Now that we have an idea what our dataset looks like, our goal is to create a model that will predict the class (in this case iris species) based on the other variables. In order to evaluate how well the model fits, we will also need a validation set where we can test some of our predictions. For this we will again split the data into a training and a validation set. Fill in the function below, you can use your function from last week as inspiration. Don't forget to randomize your data before splitting.

Now use this function on the data from each of the 3 classes, using a ratio of $0.7$ for each. This should result in 3 different training sets and 3 different validation sets. Be sure to name them all differently, so you can still use them all later in the assignment.

```python
def validation_split(data, ratio):
    # YOUR SOLUTION HERE
    np.random.shuffle(data)
    n = int(ratio*len(data))
    training_set = data[:n, :]
    validation_set = data[n:, :]
    return training_set, validation_set
# YOUR SOLUTION HERE
train0, val0 = validation_split(C0, 0.7)
train1, val1 = validation_split(C1, 0.7)
train2, val2 = validation_split(C2, 0.7)
```

## Univariate model [1 pt]

Looking at the scatter matrix of the data from the previous section, you might conclude that separating the different classes would be a lot easier based on the petal data (3rd and 4th variable) than on the sepal data (1st and 2nd variable), as it easier to distinguish the different clusters in that plot. In fact, for now we will only focus on 1 variable, the petal length (3rd variable), as it looks like it might be useful just on its own and this will simplify the model a lot.

Using the training data from each of 3 classes, compute the mean and standard deviation for the *petal length* attribute for each class. The Maximum Likelihood Estimators for these are given by

(4.8a) $$m = \frac{\sum_{t=1}^Nx^t}{N}$$

(4.8b) $$s = \sqrt{\frac{\sum_{t=1}^N(x^t - m)^2}{N}}$$

You can also use the built-in *Numpy* functions for [mean](https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html) and [standard deviation](https://docs.scipy.org/doc/numpy/reference/generated/numpy.std.html). Print the results for each class and make sure they make sense in comparison to the plot.

```python
# YOUR SOLUTION HERE
m0 = np.mean(C0[:,2])
s0 = np.std(C0[:,2])
m1 = np.mean(C1[:,2])
s1 = np.std(C1[:,2])
m2 = np.mean(C2[:,2])
s2 = np.std(C2[:,2])
print('The petal length for Setosa: \tmean: ' + str(m0)  + '\tstandard deviation: ' + str(s0))
print('The petal length for Versicolour: \tmean: ' + str(m1) + '\t\t\t\tstandard deviation: ' + str(s1))
print('The petal length for Virginica: \tmean: ' + str(m2) + '\tstandard deviation: ' + str(s2))
```

    The petal length for Setosa: 	mean: 1.462	standard deviation: 0.17191858538273283
    The petal length for Versicolour: 	mean: 4.26				standard deviation: 0.4651881339845203
    The petal length for Virginica: 	mean: 5.5520000000000005	standard deviation: 0.546347874526844

## Probability density function [2 pts]

The probability density function for a Gaussian distribution is defined as

(4.7) $$p(x|\mu, \sigma)=\frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x - \mu)^2}{2\sigma^2}}$$

where $X$ is Gaussian (normal) distributed with mean $\mu$ and variance $\sigma^2$, denoted as $\mathcal{N}(\mu,
\sigma^2$).

That means that if we have estimates for $\mu$ and $\sigma$, we can compute the probability density for a specific value $x$. Implement this in the function below.

We already made estimates for $\mu$ and $\sigma$ for the *petal length* for each of the 3 classes, so we can now define PDFs for each separate class. Plot the 3 functions using [linspace](https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.linspace.html) for a range of x-values (like in last weeks assignment) and apply the PDF functions to obtain the y-values. Use the previous plots to estimate a sensible range for *linspace* and make sure that the plotted densities correspond with your expectations.

```python
def PDF(x, m, s):
    # YOUR SOLUTION HERE
    return np.exp(-(x-m)**2/(2*s**2))/(np.sqrt(2*np.pi)*s)
# YOUR SOLUTION HERE
x = np.linspace(0, 8, 100)
y0 = PDF(x, m0, s0)
y1 = PDF(x, m1, s1)
y2 = PDF(x, m2, s2)
plt.plot(x, y0)
plt.plot(x, y1)
plt.plot(x, y2)
plt.legend(['setosa', 'versicolour', 'virginica'])
plt.show()
```

## Posterior probabilities [2 pts]

The plot above shows the probability densities for a value $x$, provided that you know the parameters for a specific class $C_i$, i.e. $p(x | \mu_i, \sigma_i)$. Because the mean and standard deviation are suffient statistics for normally distributed data, this is equivalent to the probability density given that specific class $p(x | C_i)$. However, what would be useful for classification, is the posterior probabilities of the classes given the data, i.e. $P(C_i | x)$. We can use Bayes' rule for this

(3.5) $$P(C_i | x) =  \frac{p(x | C_i) P(C_i)}{p(x)} = \frac{p(x | C_i) P(C_i)}{\sum_{k=1}^K p(x | C_k) P(C_k)}$$

Because here we have no prior knowledge of the distribution of the different classes, we can just assume all prior class probabilities $P(C_i)$ to be equal. For our 3 class problem, that would mean a probability of $\frac{1}{3}$ for each class, but we can also just factor the common prior out of the equation and simplify to

$$P(C_i | x) = \frac{p(x | C_i)}{\sum_{k=1}^K p(x | C_k)}$$

In order to make the code a little easier to write, make a vector $m$ with all values of $m_i$ and vector $s$ with all values of $s_i$, where $m_i$ and $s_i$ are the estimates for the class $C_i$. You can then just use those 2 vectors and a parameter $i$ to indicating which class you want to compute the posterior for. Write the function for the `posteriors`. It should return the posterior for each of the three classes as a list. Plot the posterior probabilities for all 3 classes. Does the plot of these 3 posteriors make sense based on the data?

```python
def posteriors(x, m, s):
    # YOUR SOLUTION HERE
    return [PDF(x, m[i], s[i]) for i in range(len(m))]
# YOUR SOLUTION HERE
x = np.linspace(0, 8, 100)
m = [m0, m1, m2]
s = [s0, s1, s2]
y = np.array([posteriors(xi, m, s)/np.sum(posteriors(xi, m, s)) for xi in x])
#y = posteriors(x, m, s)
for i in range(y.shape[1]):
    plt.plot(x, y[:, i])
plt.legend(['setosa', 'versicolour', 'virginica'])
plt.show()
```

## Bayes Classifier [2 pts]

Now that we can compute the posteriors for every class, constructing a classifier is easy. The Bayes classifier is defined as

(3.6) $$choose\ C_i\ if\ P (C_i |x) = max_k\ P(C_k |x)$$

Write the code for the `classify` function. It should classify a single data point $x$ as one of the 3 classes, returning $0$, $1$ or $2$ based on the which class the flower is most likely to belong to. The other arguments of the function should therefore be the vector of mean estimates $m$ and the vector of standard deviation estimates $s$.

Next complete the `validate` function below. It should take a validation set, the expected class for all data points in that set and the vectors $m$ and $s$ with which to make the classifications. Based on this, it should return the percentage of elements in the validation set that were classified correctly.

Remember that the mean and standand deviation estimates we have computed are only based on the *petal length* variable, while a data point in the validation set consists of a complete row, i.e. all 4 variables describing the dimensions of a flower instance. So, for now, you will only need to use the *petal length* variable from each data point to attempt to classify it.

Apply the `validate` function to all 3 validation sets and report the percentage correct for each class. Also include the average correct percentage for all 3 classes.

```python
def classify(x, m, s):
    # YOUR SOLUTION HERE
    return np.argmax(posteriors(x, m, s), axis = 0)
def validate(val, expected, m, s):
    # YOUR SOLUTION HERE
    pred = classify(val[:, 2], m, s)
    return len([i for i in pred if i == expected]) / len(pred)
# YOUR SOLUTION HERE
print('Correct percentage for class 0: ' + str(validate(val0, 0, m, s)))
print('Correct percentage for class 1: ' + str(validate(val1, 1, m, s)))
print('Correct percentage for class 2: ' + str(validate(val2, 2, m, s)))
print('Average correct percentage: ' + str(np.mean([validate(val0, 0, m, s), validate(val1, 1, m, s), validate(val2, 2, m, s)])))
```

    Correct percentage for class 0: 1.0
    Correct percentage for class 1: 0.9333333333333333
    Correct percentage for class 2: 0.9333333333333333
    Average correct percentage: 0.9555555555555556

# Multivariate distributions

You should already get pretty good results using just the *petal length* variable (depending on the splits that were made for the validation sets, the exact percentage might vary a little), but maybe we can do better still. Looking at the original plots of the data, you might conclude that there could be more information for the classification in the other variables, so a multi-variate approach would make sense. This means we will need a multi-variate version of the probability density function for normal distributions, which is defined as

(5.9) $$p(\boldsymbol{x}|\boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{\frac{d}{2}}|\boldsymbol{\Sigma}|^{\frac{1}{2}}} e^{-\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\boldsymbol{x} - \boldsymbol{\mu})}$$

where $\boldsymbol{x}$ is a $d$-dimensional vector drawn from the normal distribution $\mathcal{N}_d(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, with mean vector $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$.

If we used all 4 variables in the dataset to compute the vector of means and the covariance matrix for each class, then we could use the above equation to compute the probability densities for each class, with those the posteriors for each class and finally use the same Bayes classifier to select the most probable class for a data point. However if the number of variables is large, then the number parameters for the covariance matrix (sized $d \times d$) can become too large to properly fit to the amount of data we have. It is therefore common to make some simplifying assumptions, to decrease the complexity of the model. One well know variant is *Naive Bayes*, where the assumption is that all variables are independent. This means the model only uses the variance for each variable and no covariances are included, which corresponds to a covariance matrix with only entries on the diagonal.

## Naive bivariate distributions [4 pts]

We will start with a bi-variate (i.e. 2 variables) distribution, as this will be easier to plot, visualize, and thus reason about. Looking at the original 2 plots of the data, the second plot using the *petal lenght* and *petal width* seems to result in nice clusters, so will use those 2 variables.  We will use the *Naive Bayes* model for our bivariate data. Even if computing the covariance might not be problematic in this case, it will still be useful to see the effect of such a simplifying assumping.

This assumption of independence might be *naive* using real world data, you might even expect *length* and *width* of flower petals to be strongly correlated, but it turns out *Naive Bayes* can still make an effective classifier. The assumption of independence also simplifies the equation a lot: Instead of writing out the full version using the covariance matrices, we can write a multi-variate distribution as a product of one-dimensional distributions for each variable. So for our 2 variable case:

$$p(\boldsymbol{x}| C_i) = p(x_1 | C_i) p(x_2 | C_i)$$

Where $p(x_1 | C_i)$ is the old univariate distribution using the $\mu_1$ and $\sigma_1$ parameters of the first variable for class $C_i$ and $p(x_2 | C_i)$ is the same equation using the $\mu_2$ and $\sigma_2$ parameters of the second variable for class $C_i$.

Start by writing this *naive* version of the probability density function for the bivariate case. You will also have to compute the 3 means (for each of the 3 classes) and 3 standard deviations for the additional *petal width* variable. You should already have the code to compute these for the *petal length* variable from the univariate distributions.

Now we will plot the density for each class to see what they look like. These plots will be a 3-dimensional; 2 dimensions for the values of *petal length* and *petal width*, and 1 dimension for the probability density belonging to that combination of variables. There are several ways to make 3d plots in *matplotlib*; we will use contour plots, as that is also what is used most in the book (see figure 5.3, 5.4, 5.5 and 5.6). Start with a simple plot of the point values of the 2 variables using a different color for each class (like at the start of the assignment) and overlay the 3 contours on top of them.

Construct a mesh of X and Y values using [linspace](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html) for each dimension and combining them with [meshgrid](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html). Then compute the Z values for a probability density function of 1 class and use the [contour](https://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.contour) function to plot those Z values. Repeat this for the other 2 classes and show the final plot. You can also look at some demo uses of contour [here](http://matplotlib.org/examples/pylab_examples/contour_demo.html).

```python
def PDF_naive(x1, x2, m1, m2, s1, s2):
    # YOUR SOLUTION HERE
    return PDF(x1, m1, s1)*PDF(x2, m2, s2)
# YOUR SOLUTION HERE
x = np.linspace(0, 8, 320)
y = np.linspace(0, 3, 120)
X, Y = np.meshgrid(x, y)
Z0 = PDF_naive(X, Y, np.mean(C0[:, 2]), np.mean(C0[:, 3]), np.std(C0[:, 2]), np.std(C0[:, 3]))
Z1 = PDF_naive(X, Y, np.mean(C1[:, 2]), np.mean(C1[:, 3]), np.std(C1[:, 2]), np.std(C1[:, 3]))
Z2 = PDF_naive(X, Y, np.mean(C2[:, 2]), np.mean(C2[:, 3]), np.std(C2[:, 2]), np.std(C2[:, 3]))
plt.contour(X, Y, Z0)
plt.contour(X, Y, Z1)
plt.contour(X, Y, Z2)
plt.show()
```

## Naive classifier [4 pts]

Now that you have the naive bivariate version of the `PDF` fuctions, constructing the *Naive Bayes* classifier will be straightforward. Simply adapt your previous `posteriors`, `classify` and `validate` functions to work with your `PDF_naive` function.

Show the validation results for each of the 3 classes and the average over all 3 classes.

```python
def posteriors_naive(x1, x2, m1, m2, s1, s2):
    # YOUR SOLUTION HERE
    return [PDF_naive(x1, x2, m1[i], m2[i], s1[i], s2[i]) for i in range(len(m1))]
def classify_naive(x1, x2, m1, m2, s1, s2):
    # YOUR SOLUTION HERE
    return np.argmax(posteriors_naive(x1, x2, m1, m2, s1, s2), axis = 0)
def validate_naive(val, expected, m1, m2, s1, s2):
    # YOUR SOLUTION HERE
    pred = classify_naive(val[:, 2], val[:, 3], m1, m2, s1, s2)
    return len([i for i in pred if i == expected]) / len(pred)
# YOUR SOLUTION HERE
m1 = [np.mean(C0[:, 2]), np.mean(C1[:, 2]), np.mean(C2[:, 2])]
m2 = [np.mean(C0[:, 3]), np.mean(C1[:, 3]), np.mean(C2[:, 3])]
s1 = [np.std(C0[:, 2]), np.std(C1[:, 2]), np.std(C2[:, 2])]
s2 = [np.std(C0[:, 3]), np.std(C1[:, 3]), np.std(C2[:, 3])]
print('Correct percentage for class 0: ' + str(validate_naive(val0, 0, m1, m2, s1, s2)))
print('Correct percentage for class 1: ' + str(validate_naive(val1, 1, m1, m2, s1, s2)))
print('Correct percentage for class 2: ' + str(validate_naive(val2, 2, m1, m2, s1, s2)))
print('Average correct percentage: ' + str(np.mean([validate_naive(val0, 0, m1, m2, s1, s2), validate_naive(val1, 1, m1, m2, s1, s2), validate_naive(val2, 2, m1, m2, s1, s2)])))
```

    Correct percentage for class 0: 1.0
    Correct percentage for class 1: 1.0
    Correct percentage for class 2: 0.9333333333333333
    Average correct percentage: 0.9777777777777779

## Decision boundaries [1 pt]

In *Alpaydin* the posterior equations are simplified as much as possible into a discriminant function, while still retaining the property

$$g_i(x) > g_j(x) \iff P(x|C_i) > P(x|C_j)$$

This can simplify the computation, or help gain insight in the actual distinguishing elements of the particular model. With some algebra, they can also be written into equations for the decision boundaries (the line where $g_i(x) = g_j(x)$). As we have a computer at our disposal, we won't worry too much about how many computations are needed and just let the machine do the work. If you make a contour plot of the complete posterior, these contours should also meet at the decision boundary, while being a lot less work to program.

For the previous cell you already wrote the `posteriors_naive` function, so all that is left is to make contour plots just like for the `PDF_naive` function. Show the points for the 3 classes again and overlay the 3 posterior contours on top. Do the decision regions look like you would expect?

```python
# YOUR SOLUTION HERE
plt.scatter(C0[:, 2], C0[:, 3])
plt.scatter(C1[:, 2], C1[:, 3])
plt.scatter(C2[:, 2], C2[:, 3])
x = np.linspace(0, 8, 320)
y = np.linspace(0, 3, 120)
X, Y = np.meshgrid(x, y)
plt.contour(X, Y, posteriors_naive(X, Y, m1, m2, s1, s2)[0])
plt.contour(X, Y, posteriors_naive(X, Y, m1, m2, s1, s2)[1])
plt.contour(X, Y, posteriors_naive(X, Y, m1, m2, s1, s2)[2])
plt.show()
```

## Full bivariate distribution [3 pts]

Finally, we will compare this naive bivariate distribution with a full bivariate distribution, using the complete covariance matrix. In order to skip programming out the complete equations for this, you may use a built-in function, namely [multivariate normal distribution](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.multivariate_normal.html) from the *SciPy* stats module. With this we can create a distribution based on a mean vector and a covariance matrix. This distribution can then be used to compute the probability density for a vector of x values, like so:

    distribution = multivariate_normal(mean=mean_vector, cov=covariance_matrix)
    prob_density = distribution.pdf(x_vector)

The mean vector and covariance matrix for a class can be computed using the *Numpy* functions [mean](https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html) and [covariance](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cov.html).

Create 3 distributions, one for each class, using the functions linked above. Write the `PDF_full` function, which takes a `distr` argument that should correspond to one distribution for a class. Use that distribution to compute the probability density of the pair `x1`, `x2`.

Again plot the points for the 3 classes, using different colors for each class and overlay the contours of the probability densities for each of the 3 classes using a meshgrid. This should be the same as your contour plot for the naive bivariate distribution, only now using the full bivariate distribution to compute the densities.

**What is the difference between these 2 plots? Why?**

The plot of Naive bivariate distributions is symmetric with respect to the x-axis and y-axis because of the assumption of independence; The plot of bivariate distribution is not because of the existence of the covariance.

```python
from scipy.stats import multivariate_normal

def PDF_full(x1, x2, distr):
    # YOUR SOLUTION HERE
    pos = np.dstack((x1, x2))
    return distr.pdf(pos)
# YOUR SOLUTION HERE
plt.scatter(C0[:, 2], C0[:, 3])
plt.scatter(C1[:, 2], C1[:, 3])
plt.scatter(C2[:, 2], C2[:, 3])
x = np.linspace(0, 8, 320)
y = np.linspace(0, 3, 120)
X, Y = np.meshgrid(x, y)
distr0 = multivariate_normal([np.mean(C0[:, 2]), np.mean(C0[:, 3])], np.cov(C0[:, 2], C0[:, 3]))
distr1 = multivariate_normal([np.mean(C1[:, 2]), np.mean(C1[:, 3])], np.cov(C1[:, 2], C1[:, 3]))
distr2 = multivariate_normal([np.mean(C2[:, 2]), np.mean(C2[:, 3])], np.cov(C2[:, 2], C2[:, 3]))
Z0 = PDF_full(X, Y, distr0)
Z1 = PDF_full(X, Y, distr1)
Z2 = PDF_full(X, Y, distr2)
plt.contour(X, Y, Z0)
plt.contour(X, Y, Z1)
plt.contour(X, Y, Z2)
plt.show()
```

## Full bivariate classifier [3 pts]

Adapt your `posteriors`, `classify` and `validate` functions to work your `PDF_full` function. Each function should take a `distrs` argument, which should be a list containing the 3 `multivariate_normal` distributions for the 3 classes.

Show the validation results for each of the 3 classes and the average over all 3 classes.

```python
def posteriors_full(x1, x2, distrs):
    # YOUR SOLUTION HERE
    li = []
    for i in range(len(distrs)):
        li.append(PDF_full(x1, x2, distrs[i]))
    li = li / np.sum(li)
    return li
def classify_full(x1, x2, distrs):
    # YOUR SOLUTION HERE
    return np.argmax(posteriors_full(x1, x2, distrs), axis = 0)
def validate_full(val, expected, distrs):
    # YOUR SOLUTION HERE
    pred = classify_full(val[:, 2], val[:, 3], distrs)
    return len([i for i in pred if i == expected]) / len(pred)
# YOUR SOLUTION HERE
distrs = [distr0, distr1, distr2]
print('Correct percentage for class 0: ' + str(validate_full(val0, 0, distrs)))
print('Correct percentage for class 1: ' + str(validate_full(val1, 1, distrs)))
print('Correct percentage for class 2: ' + str(validate_full(val2, 2, distrs)))
print('Average correct percentage: ' + str(np.mean([validate_full(val0, 0, distrs), validate_full(val1, 1, distrs), validate_full(val2, 2, distrs)])))
```

    Correct percentage for class 0: 1.0
    Correct percentage for class 1: 1.0
    Correct percentage for class 2: 0.9333333333333333
    Average correct percentage: 0.9777777777777779

```python

```


# Two photon emission rate of  hydrogen atom in the free space

This program calculates the two photon emission rates for hydrogen atom. The details of the physics is given in ref. [1]. This code uses lots of the libraries developed by https://arc-alkali-rydberg-calculator.readthedocs.io/en/latest/detailed_doc.html [2]. The TPE rates are then compared with the literature [3]

[1]. Generating heralded high-dimensional hyper-entangled photons using Rydberg atoms,...

[2]. N. Sibalic, J.D. Pritchard, C.S. Adams, and K.J.Weatherill, Arc: An open-source library for calculating properties of alkali rydberg atoms, Computer Physics Communications 220, 319-331 (2017).

[3]. J. Chluba and R. A. Sunyaev, “Two-photon transitions in hydrogen and cosmological recombination,” Astronomy and Astrophysics 480, 629–645 (2008).

[4]. L. D. Landau and E. M. Lifshitz, Quantum Mechanics - Non relativistic theory, Ed. 3 (Pergamon, London,1977).

The TPE rate for hydrogen atom in the free space is given as:
\begin{align}
\Gamma   &= \frac{3^2 Z^{10}}{2^{11}} R_H \alpha^6 c \Big( \frac{k_{fi}}{k_0}\Big)^5 \int_{y=0}^{1} y^3 (1-y)^3 dy  \Big| \sum_m d_{fm}d_{mi} \Big( \frac{1}{y - y_{im}} + \frac{1}{1 - y - y_{im}} \Big) \Big|^2
\end{align}

Please check the Appendix B of ref. [1] for better understanding of the formula above.

```python
#This program is to connect with the library

# Configure the matplotlib graphics library and configure it to show
# show figures inline in the notebook
%matplotlib inline
import matplotlib.pyplot as plt  # Import library for direct plotting functions
import numpy as np               # Import Numerical Python
from IPython.core.display import display, HTML #Import HTML for formatting output

# NOTE: Uncomment following lines ONLY if you are not using installation via pip
import sys,os
rootDir = '/home/sutapa/ARC-Alkali-Rydberg-Calculator-2.0.5/' # e.g. '/Users/Username/Desktop/ARC-Alkali-Rydberg-Calculator'
sys.path.insert(0,rootDir)

from arc import *                 #Import ARC (Alkali Rydberg Calculator)
```

The dipole moment contains the contribution from both the bound states and the continuum states. Now we are calculating the continuous states of H-atom. This wavefunction is for E>0 and energy eigenvalues are highly degenerate. The analytical expression for the continuum state for hydrogenic atom is written as:

\begin{align*}
R^{cont}_{pl} &= \frac{C_{pl}}{(2l+1)!} (2pr)^l e^{-ipr} F(iZ/p+l+1,2l+2,2ipr)\\
C_{pl} &= 2p e^{\pi Z/2p} |\Gamma(l+1+iZ/p)|
\end{align*}

where F represents a first order hypergeometric function and $p=\sqrt{2E}$ represents the continuous momentum states in atomic units. For TPE rate, we calculate the overlap wavefunction of the initial and final bound state with the continuum state for p-state, $l=1$.

In this code we calculate the continuous state for radius, r and save it so that we do not have to calculate them again. Later these states are called in the other part of the program.

```python
from mpmath import *
mp.dps = 5; mp.pretty = True
from scipy.special import factorial, gamma, assoc_laguerre
from scipy import integrate
import array as arr

l=1
b=2*l+2
coef_1=1/factorial(2*l+1)
k=np.arange(0.01,2,0.01)
r=np.arange(0,105000,1)
R_kl=np.zeros((len(k),len(r)), dtype=np.float64 )
A_kl=arr.array('f',k)
for j in range(0,len(k)):
    a=(1j/k[j])+l+1
    coef=2*k[j]*np.exp(np.pi/(2*k[j]))*abs(gamma(l+1-1j/k[j]))
    for i in range(0,len(r)):
        R_kl[j][i]= np.real(coef*coef_1*np.exp(-1j*k[j]*r[i])*((2*k[j]*r[i])**l)*hyp1f1(a,b,2*1j*k[j]*r[i]))

#np.savetxt('R_kl_105000_1.dat', R_kl)
```

The variable A_kl1 is an overlap integral between the final transition state and all the continuous states of the hydrogen atom. Make sure that the radius, r is defined same as for the calculation of the continuous states. Because we will be calculating the integral with the $R_{kl}$

```python
#Calculation of matrix element
from mpmath import *
mp.dps = 5; mp.pretty = True
from scipy.special import factorial, gamma, assoc_laguerre
from scipy import integrate
import array as arr

atom=Hydrogen()
n1=1
l1=0
j1=0.5
l=1

step=0.01
a1,b1 = atom.radialWavefunction(l1,0.5,j1,atom.getEnergy(n1, l1, j1)/27.211,atom.alphaC**(1/3.0),200, step)
b1=(-1)**(n1-1)*(-1)**l1*b1
#plt.plot(a1,b1)

b=2*l+2
coef_1=1/factorial(2*l+1)
k=np.arange(0.01,2,0.01)
r=a1
R_kl=np.zeros((len(k),len(r)), dtype=np.float64 )
A_kl1=arr.array('f',k)
for j in range(0,len(k)):
    a=(1j/k[j])+l+1
    coef=2*k[j]*np.exp(np.pi/(2*k[j]))*abs(gamma(l+1-1j/k[j]))
    sum_c=0.0
    for i in range(1,len(r)):
        R_kl[j][i]= np.real(coef*coef_1*np.exp(-1j*k[j]*r[i])*((2*k[j]*r[i])**l)*hyp1f1(a,b,2*1j*k[j]*r[i]))
        sum_c+= R_kl[j][i]*b1[i]*(r[i]**2)*abs(r[i-1]-r[i])
    A_kl1[j]=sum_c/((2*np.pi)**0.5)

print(A_kl1)
#np.savetxt('A_kl1_6S_200_0.01.dat', A_kl1)
```

This will calculate the overlap of the wavefuction, A_kl2 of continuous state with the initial state ($n_i$ $l_i$) of the two photon transition. This part of the code need to be change everytime depending on the initial state. Again, here make sure that the radius, r is defined same as for the calculation of the continuous states. Because we will be calculating the integral with the $R_{kl}$

```python
#Calculation of matrix element
from mpmath import *
mp.dps = 5; mp.pretty = True
from scipy.special import factorial, gamma, assoc_laguerre
from scipy import integrate
import array as arr

atom=Hydrogen()
n2=60
l2=0
j2=0.5
l=1

step=0.01
a1,b1 = atom.radialWavefunction(l2,0.5,j2,atom.getEnergy(n2, l2, j2)/27.211,atom.alphaC**(1/3.0),15000, step)
b1=(-1)**(n2-1)*(-1)**l2*b1
#plt.plot(a1,b1)

b=2*l+2
coef_1=1/factorial(2*l+1)
k=np.arange(0.01,2,0.01)
r=a1
R_kl=np.zeros((len(k),len(r)), dtype=np.float64 )
A_kl2=arr.array('f',k)
for j in range(0,len(k)):
    a=(1j/k[j])+l+1
    coef=2*k[j]*np.exp(np.pi/(2*k[j]))*abs(gamma(l+1-1j/k[j]))
    sum_c=0.0
    for i in range(1,len(r)):
        R_kl[j][i]= np.real(coef*coef_1*np.exp(-1j*k[j]*r[i])*((2*k[j]*r[i])**l)*hyp1f1(a,b,2*1j*k[j]*r[i]))
        sum_c+= R_kl[j][i]*b1[i]*(r[i]**2)*abs(r[i]-r[i-1])
    A_kl2[j]=sum_c/((2*np.pi)**0.5)

print(A_kl2)
#np.savetxt('A_kl2_60s_15000_0.01.dat', A_kl2)
```

This code calls the saved variables A_kl1 and A_kl2.

```python
#R_kl= np.loadtxt('R_kl_100000_1.dat')
A_kl1= np.loadtxt('A_kl1_6S_200_0.01.dat')
A_kl2= np.loadtxt('A_kl2_60s_15000_0.01.dat')
```

Finally we will calculate the two photon rates below for the H-atom for a transition from $n_i l_i \rightarrow 1S$ state and will compare with the literature for various values of the initial state $n_i l_i$

```python
from scipy.special import assoc_laguerre, factorial,gamma
from scipy import integrate
import array as arr

def R(r,n,l):
    Z=1
    R1= ((2*Z/n)**3 * factorial(n-l-1)/(2*n*factorial(n+l)))**0.5 * np.exp(-Z*r/n)*(2*Z*r/n)**l*assoc_laguerre(2*Z*r/n,n-l-1,2*l+1)
    return(R1)

def kappa(n1,l1,n2,l2,m,lm):
    r=np.arange(0,10000,0.1)
    dip1=R(r,n2,l2)*r**3*R(r,m,lm)
    Adip1=integrate.simps(dip1,r)
    dip2=R(r,m,lm)*r**3*R(r,n1,l1)
    Adip2=integrate.simps(dip2,r)
    return(Adip1*Adip2)

def Mnr(y,ym):
    fm= 1/(y+ym)-(1/(y-1-ym))
    return(fm)

atom=Hydrogen()
n1 = 60
l1 = 0
j1=0.5
n2=1
l2=0
j2=0.5
lm = 1
jm=1.5

alpha= 1/137.036
c = 2.99792458*pow(10,8)
Rh = 10973731.5685
#coefficient=9*pow(alpha,6)*Rh*c/pow(2,10)*pow(1-(1/pow(n1,2)),5)/pow((1-(1/pow(2,2))),5)
coefficient=9*pow(alpha,6)*Rh*c/2**10*(abs((atom.getEnergy(n2,l2,j2)-atom.getEnergy(n1,l1,j1))/(atom.getEnergy(2,0,0.5)-atom.getEnergy(1,0,0.5))))**5

sumnr=0.0;
k=np.arange(0.01,2,0.01)
y= np.arange(0.001,1,0.001)

for m in range(n1,n1+100):
    ym=(atom.getEnergy(n1,l1,j1)-atom.getEnergy(m,lm,jm))/(atom.getEnergy(n2,l2,j2)-atom.getEnergy(n1,l1,j1))
    kap=kappa(n1,l1,n2,l2,m,lm)
    sumnr=sumnr+(Mnr(y,ym)*kap)

sum_c=arr.array('f',y)
for i in range(0,len(y)):
    for j in range(0,len(k)):
        yc=(1/n1**2+k[j]**2)/(1/n2**2-1/n1**2)
        sum_c[i]= sum_c[i]+(A_kl1[j]*A_kl2[j]*(1/(y[i]+yc)-(1/(y[i]-1-yc))))
    sumnr[i]+=sum_c[i]*(k[1]-k[0])

phi= coefficient*pow(y,3)*pow((1-y),3)*(sumnr*sumnr)
phi1=phi/max(phi)
plt.plot(y,phi1)
#np.savetxt('phi_50S.dat', phi1)

Anr=0.5*integrate.simps(phi,y)
print(Anr)
```

```python

```

```python

```

```python

```


# Synchornization in complete networks of neurons
Here is the equation of neurons potentails dynamics:
$$
\dot{v_i}=a_i - v_i - \frac{g}{N} \sum_{n|t_n<t} S_{i,l(n)} \delta(t - t_n - t_d)
$$

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm
```

## Constants and parameters:

```python
num_neurons = 1000
g = 5
random_input_span = (1.2,2.8)

total_time = 1000
time_step = 0.01
total_steps = int(total_time/time_step)
time_span = np.arange(0,total_time,time_step)

delay_time = 0.1
delay_step = int(delay_time/time_step)

potentail_arr = np.zeros(num_neurons,dtype = float)
random_input = np.random.uniform(*random_input_span,size = num_neurons)
```

```python
spiking_records = np.array( np.zeros((total_steps,num_neurons)), dtype = bool )
```

### Useful functions:

```python
def retarded_spikes_record(step):
    """
    This is the function which take cares of response delay between spikes and their effects.
    """
    if step > delay_step:
        return np.sum(spiking_records[step - delay_step])
    else:
        return 0
```

## Solving the differential equation

```python
for i in tqdm( range(total_steps) ):
    potentail_arr = potentail_arr + (random_input - potentail_arr)*time_step - (g/num_neurons)*retarded_spikes_record(i)
    spiking_records[i] = potentail_arr > 1
    reset_func = lambda v: v-1 if v>1 else v
    vfunc = np.vectorize(reset_func)
    potentail_arr = vfunc(potentail_arr)
```

    100%|████████████████████████████████████████████████████████████████████████| 100000/100000 [00:53<00:00, 1863.38it/s]

```python

```

```python
plt.hist(potentail_arr)
```

```python

```

# Measuring E (effective field):
we can measure if the total network is synchronized. let's define E as following:

$$
\ddot{E}+ 2\alpha \dot{E}+\alpha^{2}E =\frac{\alpha^{2}}{N} \sum_{n|tـn<t} \delta(t - t_n - t_d)
$$

To do so, we define $M$ as $M = \dot{E} + \alpha E$. As a result we wil have the following systems of equation:
\begin{equation}
    \begin{cases}
      \dot{E} = M - \alpha E\\
      \dot{M} = -  \alpha M + \frac{ \alpha^{2} }{N} \sum_{n|tـn<t} \delta(t - t_n - t_d)
    \end{cases}\,.
\end{equation}

```python
alpha = 20
m_arr = np.zeros(total_steps)
e_arr = np.zeros(total_steps)
```

```python
for i in tqdm(range(0,total_steps-1)):
    m_arr[i+1] = m_arr[i] + time_step*( -alpha*m_arr[i] ) + ( (alpha**2)/num_neurons ) *retarded_spikes_record(i)
    e_arr[i+1] = e_arr[i] + time_step*( m_arr[i] - alpha*e_arr[i] )
```

    100%|█████████████████████████████████████████████████████████████████████████| 99999/99999 [00:04<00:00, 23065.10it/s]

```python
plt.plot(time_span,e_arr)
plt.title('E')
```

```python
sigma = np.std( e_arr )
sigma
```

    0.25254336999893046

```python

```

# Rasterplot

```python
plt.imshow( spiking_records,aspect= 'auto')
plt.xlabel('Neurons')
plt.ylabel('Time')
plt.title('Rasterplot')
plt.savefig('raster_plot_N{}.png'.format(num_neurons))
```

```python
spiking_records
```

    array([[False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           ...,
           [False, False,  True, ..., False, False,  True],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False]])

```python
plt.plot(np.sum( spiking_records, axis = 1))
```

```python

```


**1**. Replicate the contents below by creating a new (MarkDown) cell below this one (using keyboard shortcuts).

## Euler's formula

The function $e^{i \phi}$ traces the unit circle in the complex plane.

$$
e^{i\phi} = \cos \phi + i \sin \phi
$$

This implies that

\begin{align}
\cos x &= \Re(e^{i x}) &= \frac{e^{ix} + e^{-ix}}{2} \\
\sin x &= \Im(e^{i x}) &= \frac{e^{ix} - e^{-ix}}{2}
\end{align}

In Python, we can express the second function as
```python
def sin(x):
    """The sine function ."""
    import numpy as np
    return 0.5*(np.exp(x*1j) - np.exp(-x*1j))
```

**1.5**. Delete this cell using a keyboard shortcut.

**2**. Write a program to create a list of the first 100 Fizz buzz numbers. Any number divisible by three is replaced by the word fizz and any divisible by five by the word buzz. Numbers divisible by both become fizz buzz. Otherwise the original number is retained.

For example, the first Fizz buzz numbers are  `1, 2, Fizz, 4, Buzz, Fizz, 7, 8, Fizz, Buzz, 11, Fizz, 13, 14, Fizz Buzz, 16, 17, Fizz, 19, Buzz, Fizz, 22, 23, Fizz, Buzz, 26, Fizz, 28, 29, Fizz Buzz, 31, 32, Fizz, 34, Buzz, Fizz, ...`.

```python
fb = []
for i in range(100):
    if i % 15 == 0:
        fb.append("fizz buzz")
    elif i % 5 == 0:
        fb.append("buzz")
    elif i % 3 == 0:
        fb.append("fizz")
    else:
        fb.append(i)

fb
```

    ['fizz buzz',
     1,
     2,
     'fizz',
     4,
     'buzz',
     'fizz',
     7,
     8,
     'fizz',
     'buzz',
     11,
     'fizz',
     13,
     14,
     'fizz buzz',
     16,
     17,
     'fizz',
     19,
     'buzz',
     'fizz',
     22,
     23,
     'fizz',
     'buzz',
     26,
     'fizz',
     28,
     29,
     'fizz buzz',
     31,
     32,
     'fizz',
     34,
     'buzz',
     'fizz',
     37,
     38,
     'fizz',
     'buzz',
     41,
     'fizz',
     43,
     44,
     'fizz buzz',
     46,
     47,
     'fizz',
     49,
     'buzz',
     'fizz',
     52,
     53,
     'fizz',
     'buzz',
     56,
     'fizz',
     58,
     59,
     'fizz buzz',
     61,
     62,
     'fizz',
     64,
     'buzz',
     'fizz',
     67,
     68,
     'fizz',
     'buzz',
     71,
     'fizz',
     73,
     74,
     'fizz buzz',
     76,
     77,
     'fizz',
     79,
     'buzz',
     'fizz',
     82,
     83,
     'fizz',
     'buzz',
     86,
     'fizz',
     88,
     89,
     'fizz buzz',
     91,
     92,
     'fizz',
     94,
     'buzz',
     'fizz',
     97,
     98,
     'fizz']

**3**. A Pythagorean triplet is a set of 3 integers $a, b, c$ such that $a^2 + b^2 = c^2$. For example, the tuple (3,4,5) is a  Pythagorean triplet. Find all Pythagorean triplets where $c < 100$.

```python
for a in range(1,100):
    for b in range(a, 100):
        for c in range(b, 100):
            if a**2 + b**2 == c**2:
                print(a, b, c)
```

    3 4 5
    5 12 13
    6 8 10
    7 24 25
    8 15 17
    9 12 15
    9 40 41
    10 24 26
    11 60 61
    12 16 20
    12 35 37
    13 84 85
    14 48 50
    15 20 25
    15 36 39
    16 30 34
    16 63 65
    18 24 30
    18 80 82
    20 21 29
    20 48 52
    21 28 35
    21 72 75
    24 32 40
    24 45 51
    24 70 74
    25 60 65
    27 36 45
    28 45 53
    30 40 50
    30 72 78
    32 60 68
    33 44 55
    33 56 65
    35 84 91
    36 48 60
    36 77 85
    39 52 65
    39 80 89
    40 42 58
    40 75 85
    42 56 70
    45 60 75
    48 55 73
    48 64 80
    51 68 85
    54 72 90
    57 76 95
    60 63 87
    65 72 97

**4**. Write a program using a dictionary to find the top 3 most frequently occurring words and how often they occur in the passage given below. Remove punctuation and convert words to lower case. We consider a word to be any continuous sequence of characters or numbers separated by whitespace.

> Write a program to create a list of the first 100 Fizz buzz numbers. Any number divisible by three is replaced by the word fizz and any divisible by five by the word buzz. Numbers divisible by both become fizz buzz. Otherwise the original number is retained.
For example, the first Fizz buzz numbers are  1, 2, Fizz, 4, Buzz, Fizz, 7, 8, Fizz, Buzz, 11, Fizz, 13, 14, Fizz Buzz, 16, 17, Fizz, 19, Buzz, Fizz, 22, 23, Fizz, Buzz, 26, Fizz, 28, 29, Fizz Buzz, 31, 32, Fizz, 34, Buzz, Fizz, ....

```python
text = 'Write a program to create a list of the first 100 Fizz buzz numbers. Any number divisible by three is replaced by the word fizz and any divisible by five by the word buzz. Numbers divisible by both become fizz buzz. Otherwise the original number is retained. For example, the first Fizz buzz numbers are 1, 2, Fizz, 4, Buzz, Fizz, 7, 8, Fizz, Buzz, 11, Fizz, 13, 14, Fizz Buzz, 16, 17, Fizz, 19, Buzz, Fizz, 22, 23, Fizz, Buzz, 26, Fizz, 28, 29, Fizz Buzz, 31, 32, Fizz, 34, Buzz, Fizz, ....'
```

```python
from string import punctuation
```

```python
table = dict.fromkeys(ord(c) for c in punctuation)
```

```python
text = text.translate(table)
```

```python
words = text.strip().lower().split()
```

```python
counter = {}
for word in words:
    counter[word] = counter.get(word, 0) + 1
```

```python
sorted(counter.items(), key=lambda x: x[1], reverse=True)[:3]
```

    [('fizz', 16), ('buzz', 11), ('by', 5)]


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## Graded Homework 5

For this homework, we'll focus on:
- using Maclaurin and Taylor series
- performing integrals (e.g. practice with u-substitution and integration by parts)
---

As a reminder, you do not need to use Python to solve the problems. If you want, you can use other methods, just put your answers in the appropriate places.

To turn in, either download as Notebook (.ipynb) or Print to PDF and upload to Gradescope.

Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE", as well as your name and collaborators (i.e., anyone you discussed this with) below:

```python
NAME = ""
COLLABORATORS = ""
```

### Maclaurin & Taylor Series

As a reminder, we can use Maclaurin series (around x=0) and Taylor series to derive approximations to more complicated functions. This is particularly useful if the function is not completely known, or hard to take derivatives / integrals.

### Hyperbolic Sine

Derive the 3rd order (i.e., up to $x^3$) series for the *hyperbolic sine* function:

$$
\sinh (x)=\frac{\mathrm{e}^{x}-\mathrm{e}^{-x}}{2}
$$

Show work... (You can do this with Sympy, Wolfram Alpha, Pen-and-Paper.. whatever works best.)
(You can create extra cells / work area in the notebook, but please put the answer where indicated.

```python
from sympy import init_session
init_session()
```

```python
# if you have an error, make sure you run the cell above this
hsin = (exp(x) - exp(-x))/2

# write the Maclaurin series here
```

```python
# use numpy and matplotlib
# to plot the true function
# and your Maclaurin series

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

x_plot = np.arange(-2.5, 2.5, 0.1)
y_plot = (np.exp(x_plot) - np.exp(-x_plot))/2
plt.plot(x_plot, y_plot)

# make sure to plot your Maclaurin series here

plt.show()
```

### Taylor Series in Integration

Okay, what do you do if you need to integrate an "impossible" case, e.g.:

$$
\int \sin \left(x^{2}\right) d x
$$

One option is to use numeric integration. We'll do that later.

For this case, work out the first 3 terms in a Taylor series expansion for $\sin (x^2)$. You will need to go higher than the $x^3$ (i.e., you'll want to see three non-zero terms).

Once we have the power series, we can integrate each term in the power series to approximate the integral we want.

```python
# your work here

```

### Integration via Substitution

A few u-substitution problems - evaluate each of the following integrals by u-substitution. You can check your work via Sympy or Wolfram Alpha, etc. but please show work.

$$
\int(\tan x) d x
$$

$$
\int \frac{4 x^{3}}{x^{4}+7} d x
$$

$$
\int \frac{(\ln x)^{2}}{x} d x=
$$

```python
# Your work here
int_tan =
fraction =
lnx2_x =
```

### Integration by Parts

One great thing about teaching is that you learn new techniques.

To help pick the "u" for integration, there's a priority list ["LIATE"](https://en.wikipedia.org/wiki/Integration_by_parts#LIATE_rule):
1 Logarithm
2 Inverse trigonometric
3 Algebraic
4 Trigonometric
5 Exponential

In other words, "u" comes earlier in the list, and "dv" comes later in the list.

There's also a useful technique for repeated integration-by-parts using [tabular integration](https://en.wikipedia.org/wiki/Integration_by_parts#Tabular_integration_by_parts) I'll show off on Monday.

In the meantime, here are some useful practice problems:

$$
\int x \sin (x) d x
$$

$$
\int \sin (x) e^{ax} d x
$$

$$
\int w^{2} \sin (10 w) d w
$$

Much like above, you can use sympy or Wolfram Alpha or other tools, but try to work by yourself. Please show work.

```python
# your work here
xsinx =
sin_eax =
w2_sin10w =
```


# Entrenamiento del Modelo de Asignación Latente de Dirichlet

> Ya que comprendimos el modelo de asignación latente de Dirichlet, veamos como entrenarlo.

> **Objetivos:**
> - Usar la inferencia variacional para entrenar el modelo de asignación latente de Dirichlet.
> - Comprender el uso de la asignación latente de Dirichlet mediante un ejemplo.

> **Referencias:**
> - Bayesian Methods for Machine Learning course, HSE University, Coursera.

## 1. Preliminares

Tenemos entonces que la función de verosimilitud conjunta es:

$$
p(\Theta, Z, X | \Phi) = \prod_{d=1}^{D} p(\theta_d) \prod_{n=1}^{N_d} p(z_{dn} | \theta_d) p(x_{dn} | z_{dn}),
$$

de donde la log-verosimilitud es:

\begin{align}
\log p(\Theta, Z, X | \Phi) = \sum_{d=1}^{D} \left[\log p(\theta_d) + \sum_{n=1}^{N_d} \left( \log p(z_{dn} | \theta_d) + \log p(x_{dn} | z_{dn}) \right) \right]
\end{align}

Ahora, dado que

- $p(\theta_d) = \text{Dir}(\theta | \alpha)$
- $p(z_{dn} | \theta_d) = \theta_{d z_{dn}}$
- $p(x_{dn} | z_{dn}) = \phi_{z_{dn} x_{dn}}$

Es posible notar que:

- $\log p(\theta_d) = \sum_{t=1}^{T} (\alpha_t - 1) \log \theta_{dt} + const.$
- $\log p(z_{dn} | \theta_d) = \log \theta_{d z_{dn}}$
- $\log p(x_{dn} | z_{dn}) = \log \phi_{z_{dn} x_{dn}}$

Con lo cual

\begin{align}
\log p(\Theta, Z, X | \Phi) = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} + \log \phi_{t x_{dn}} \right) \right] + const.
\end{align}

Para entrenar los parámetros del modelo $\Phi$, podemos maximizar la verosimilitud marginal:

$$
\max_{\Phi} p(X | \Phi),
$$

para lo cual usamos el algoritmo de maximización de la esperanza variancional:

**E-step variacional**

$$
\min_{q(\Theta), q(Z)}\mathcal{KL}(q(\Theta) q(Z) || p(\Theta, Z | X)).
$$

**M-Step**

$$
\max_{\Phi} \mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi)
$$

## 2. E-step variacional

$$
\min_{q(\Theta), q(Z)}\mathcal{KL}(q(\Theta) q(Z) || p(\Theta, Z | X)).
$$

### Con respecto a $q(\Theta)$

Usando las ecuaciones de campo media:

$$
\begin{align}
\log q(\Theta) & = \mathbb{E}_{q(Z)} \log \underbrace{p(\Theta, Z | X)}_{\frac{p(\Theta, Z, X)}{p(X)}} + const. \\
               & = \mathbb{E}_{q(Z)} \log p(\Theta, Z, X) + const. \\
               & = \mathbb{E}_{q(Z)} \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T \underbrace{\mathbb{E}_{q(Z)}[\delta(z_{dn} = t)]}_{\mathbb{E}_{q(z_{dn})}[\delta(z_{dn} = t)]} \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \left[\sum_{t=1}^T (\alpha_t-1) \log \theta_{dt} + \sum_{n=1}^{N_d} \sum_{t=1}^T q(z_{dn} = t) \left( \log \theta_{dt} \right) \right] + const. \\
               & = \sum_{d=1}^{D} \sum_{t=1}^T \left[ (\alpha_t-1) + \sum_{n=1}^{N_d} q(z_{dn} = t) \right] \log \theta_{dt} + const.
\end{align}
$$

Con lo cual tenemos que:

$$
\begin{align}
q(\Theta) & \propto \prod_{d=1}^{D} \prod_{t=1}^T \theta_{dt}^{\sum_{n=1}^{N_d} q(z_{dn} = t) + \alpha_t - 1} + const. \\
           & = \prod_{d=1}^{D} \text{Dir}\left(\theta_d \left\lvert \begin{array}{c} ... \\ \alpha_t + \sum_{n=1}^{N_d} q(z_{dn} = t) \\ ... \end{array}\right. \right)
\end{align}
$$

De manera que para encontrar $q(\Theta)$, necesitamos calcular $q(z_{dn} = t)$.

### Con respecto a $q(Z)$

Usando las ecuaciones de campo media:

$$
\begin{align}
\log q(Z) & = \mathbb{E}_{q(\Theta)} \log p(\Theta, Z, X) + const. \\
          & = \mathbb{E}_{q(\Theta)} \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left( \log \theta_{dt} + \log \phi_{t x_{dn}} \right) + const. \\
          & = \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t) \left(\mathbb{E}_{q(\Theta)} \log \theta_{dt} + \log \phi_{t x_{dn}} \right) + const.
\end{align}
$$

Por tanto:

$$
\begin{align}
q(Z) & = \prod_{d=1}^{D} \prod_{n=1}^{N_d} q(z_{dn}),
\end{align}
$$

donde

$$
q(z_{dn} = t) = \frac{\phi_{t x_{dn}} \exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)}{\sum_{t=1}^T \phi_{t x_{dn}} \exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)}.
$$

El valor de

$$
\exp\left(\mathbb{E}_{q(\Theta)} \log \theta_{dt}\right)
$$

se puede encontrar en:

https://en.wikipedia.org/wiki/Dirichlet_distribution

**¿Recuerdan el modelo de Iising?**

Debemos iterar para que estas distribuciones converjan.

## 3. M-step

$$
\max_{\Phi} \mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi)
$$

Conservamos solo los términos que dependen de $\Phi$:

$$
\begin{align}
\mathbb{E}_{q(\Theta)q(Z)}\log p(\Theta, Z, X | \Phi) & = \mathbb{E}_{q(\Theta)q(Z)}\sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t)  \log \phi_{t x_{dn}} + const.
\end{align}
$$

Debemos maximizar esta expresión teniendo en cuenta las restricciones:

- $\phi_{tx} \geq 0, \quad \forall t, x$.
- $\sum_{x} \phi_{tx} = 1, \quad \forall t$

La condición de no negatividad se satisface ya que los términos $\phi$ están dentro del $\log$.

Definimos el lagrangiano:

$$
\begin{align}
L(\Phi, \lambda) & = \mathbb{E}_{q(\Theta)q(Z)}\sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T \delta(z_{dn} = t)  \log \phi_{t x_{dn}} + \sum_{t=1}^T \lambda_t \left( \sum_{x} \phi_{tx} - 1 \right) \\
                 & = \sum_{d=1}^{D} \sum_{n=1}^{N_d} \sum_{t=1}^T q(z_{dn} = t)  \log \phi_{t x_{dn}} - \sum_{t=1}^T \lambda_t \left( \sum_{x} \phi_{tx} - 1 \right)
\end{align}
$$

Entonces

$$
\frac{\partial L(\Phi, \lambda)}{\partial \phi_{tx}} = \sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \frac{1}{\phi_{t x_{dn}}} \delta(x_{dn} = x) - \lambda_t = 0
$$

si y solo si:

$$
\phi_{tx} = \frac{\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}{\lambda_t}
$$

Ahora, imponiendo la restricción $\sum_{x} \phi_{tx} = 1$, nos damos cuenta que:

$$
\lambda_t = \sum_{x}\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)
$$

Con lo cual:

$$
\phi_{tx} = \frac{\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}{\sum_{x}\sum_{d=1}^{D} \sum_{n=1}^{N_d} q(z_{dn} = t) \delta(x_{dn} = x)}.
$$

## 4. Comentarios

Hemos desarrollado las fórmulas para entrenar nuestro modelo. Recordamos que

1. Hay que iterar varias veces el E-step y el M-step hasta la convergencia de los parámetros en la matriz $\Phi$.
   - La matriz $\Phi$ contiene la distribución tópico-palabras. De acá podemos identificar qué palabras son más importantes para cada tópico y así inferir el tópico.

2. Dentro del E-step en sí, hay que iterar hasta la convergencia de las distribuciones $q(\Theta), q(Z)$.

3. Al final, la distribución $q(\Theta)$ modelará las distribuciones de documentos-tópicos. Podremos extraer de los parámetros $\alpha$ la representación de los documentos en los tópicos.

5. Si quisiéramos predecir para un nuevo documento, podemos reutilizar las ecuaciones del E-step.

## 5. Aplicación

Para finalizar todas estas matemáticas, veamos un ejemplo para aterrizar todo lo que hemos visto en este tema.

Para trabajar, utilizaremos [the 20 newsgroups dataset](http://qwone.com/~jason/20Newsgroups/)

```python
# Importamos librerías
# Para obtener los datos
from sklearn.datasets import fetch_20newsgroups
# Para procesar el texto
from sklearn.feature_extraction.text import CountVectorizer
# Latent Dirichlet Allocation
from sklearn.decomposition import LatentDirichletAllocation
# Para graficar
from matplotlib import pyplot as plt
```

```python
# Obtenemos datos
data, _ = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'),
                             return_X_y=True)
```

```python
# Visualizamos datos
data[:5]
```

    ["Well i'm not sure about the story nad it did seem biased. What\nI disagree with is your statement that the U.S. Media is out to\nruin Israels reputation. That is rediculous. The U.S. media is\nthe most pro-israeli media in the world. Having lived in Europe\nI realize that incidences such as the one described in the\nletter have occured. The U.S. media as a whole seem to try to\nignore them. The U.S. is subsidizing Israels existance and the\nEuropeans are not (at least not to the same degree). So I think\nthat might be a reason they report more clearly on the\natrocities.\n\tWhat is a shame is that in Austria, daily reports of\nthe inhuman acts commited by Israeli soldiers and the blessing\nreceived from the Government makes some of the Holocaust guilt\ngo away. After all, look how the Jews are treating other races\nwhen they got power. It is unfortunate.\n",
     "\n\n\n\n\n\n\nYeah, do you expect people to read the FAQ, etc. and actually accept hard\natheism?  No, you need a little leap of faith, Jimmy.  Your logic runs out\nof steam!\n\n\n\n\n\n\n\nJim,\n\nSorry I can't pity you, Jim.  And I'm sorry that you have these feelings of\ndenial about the faith you need to get by.  Oh well, just pretend that it will\nall end happily ever after anyway.  Maybe if you start a new newsgroup,\nalt.atheist.hard, you won't be bummin' so much?\n\n\n\n\n\n\nBye-Bye, Big Jim.  Don't forget your Flintstone's Chewables!  :) \n--\nBake Timmons, III",
     "Although I realize that principle is not one of your strongest\npoints, I would still like to know why do do not ask any question\nof this sort about the Arab countries.\n\n   If you want to continue this think tank charade of yours, your\nfixation on Israel must stop.  You might have to start asking the\nsame sort of questions of Arab countries as well.  You realize it\nwould not work, as the Arab countries' treatment of Jews over the\nlast several decades is so bad that your fixation on Israel would\nbegin to look like the biased attack that it is.\n\n   Everyone in this group recognizes that your stupid 'Center for\nPolicy Research' is nothing more than a fancy name for some bigot\nwho hates Israel.",
     'Notwithstanding all the legitimate fuss about this proposal, how much\nof a change is it?  ATT\'s last product in this area (a) was priced over\n$1000, as I suspect \'clipper\' phones will be; (b) came to the customer \nwith the key automatically preregistered with government authorities. Thus,\naside from attempting to further legitimize and solidify the fed\'s posture,\nClipper seems to be "more of the same", rather than a new direction.\n   Yes, technology will eventually drive the cost down and thereby promote\nmore widespread use- but at present, the man on the street is not going\nto purchase a $1000 crypto telephone, especially when the guy on the other\nend probably doesn\'t have one anyway.  Am I missing something?\n   The real question is what the gov will do in a year or two when air-\ntight voice privacy on a phone line is as close as your nearest pc.  That\nhas got to a problematic scenario for them, even if the extent of usage\nnever surpasses the \'underground\' stature of PGP.',
     "Well, I will have to change the scoring on my playoff pool.  Unfortunately\nI don't have time right now, but I will certainly post the new scoring\nrules by tomorrow.  Does it matter?  No, you'll enter anyway!!!  Good!\n\n--\n    Keith Keller\t\t\t\tLET'S GO RANGERS!!!!!\n\t\t\t\t\t\tLET'S GO QUAKERS!!!!!\n\tkkeller@mail.sas.upenn.edu\t\tIVY LEAGUE CHAMPS!!!!"]

```python
# ¿Cuántos documentos tenemos?
len(data)
```

    11314

```python
# Definamos algunas cosas
# Número de documentos
n_docs = 3000
# Número de tópicos
n_topics = 10
# Número de palabras en el vocabulario
n_words = 1000
```

```python
# Vectorizamos el texto por medio de cuentas crudas de palabras
vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                             max_features=n_words,
                             stop_words='english')
corpus = vectorizer.fit_transform(data[:n_docs])
```

```python
# Matriz de documentos vectorizados
corpus
```

    <3000x1000 sparse matrix of type '<class 'numpy.int64'>'
    	with 77074 stored elements in Compressed Sparse Row format>

```python
def plot_top_words(model, feature_names, n_top_words, title):
    # Malla para graficar 2 filas 5 columnas
    fig, axes = plt.subplots(2, 5, figsize=(30, 15), sharex=True)
    axes = axes.flatten()
    # model.components_ hace referencia a la matriz phi
    for topic_idx, topic in enumerate(model.components_):
        # La hacemos distribución
        topic /= topic.sum()
        # Seleccionamos los índices más relevantes del tópico
        top_features_ind = topic.argsort()[:-n_top_words - 1:-1]
        # Obtenemos las palabras
        top_features = [feature_names[i] for i in top_features_ind]
        # Obtenemos la ponderación
        weights = topic[top_features_ind]

        # Graficamos barras horizontales por tópicos con las palabras más relevantes
        ax = axes[topic_idx]
        ax.barh(top_features, weights, height=0.7)
        ax.set_title(f'Topic {topic_idx +1}',
                     fontdict={'fontsize': 30})
        ax.invert_yaxis()
        ax.tick_params(axis='both', which='major', labelsize=20)
        for i in 'top right left'.split():
            ax.spines[i].set_visible(False)
        fig.suptitle(title, fontsize=40)

    plt.subplots_adjust(top=0.90, bottom=0.05, wspace=0.90, hspace=0.3)
    plt.show()
```

```python
lda = LatentDirichletAllocation(n_components=n_topics)
```

```python
lda.fit(corpus)
```

    LatentDirichletAllocation(batch_size=128, doc_topic_prior=None,
                              evaluate_every=-1, learning_decay=0.7,
                              learning_method='batch', learning_offset=10.0,
                              max_doc_update_iter=100, max_iter=10,
                              mean_change_tol=0.001, n_components=10, n_jobs=None,
                              perp_tol=0.1, random_state=None,
                              topic_word_prior=None, total_samples=1000000.0,
                              verbose=0)

```python
plot_top_words(lda, vectorizer.get_feature_names(), 10, 'Topics in LDA model')
```

#### Veamos algunos ejemplos de textos y su categorización en tópicos

```python
print(data[0])
lda.transform(corpus[0])
```

    Well i'm not sure about the story nad it did seem biased. What
    I disagree with is your statement that the U.S. Media is out to
    ruin Israels reputation. That is rediculous. The U.S. media is
    the most pro-israeli media in the world. Having lived in Europe
    I realize that incidences such as the one described in the
    letter have occured. The U.S. media as a whole seem to try to
    ignore them. The U.S. is subsidizing Israels existance and the
    Europeans are not (at least not to the same degree). So I think
    that might be a reason they report more clearly on the
    atrocities.
    	What is a shame is that in Austria, daily reports of
    the inhuman acts commited by Israeli soldiers and the blessing
    received from the Government makes some of the Holocaust guilt
    go away. After all, look how the Jews are treating other races
    when they got power. It is unfortunate.

    array([[0.00344828, 0.00344858, 0.00344904, 0.44832181, 0.00344966,
            0.00344847, 0.00344942, 0.00344902, 0.00344941, 0.5240863 ]])

```python
print(data[101])
lda.transform(corpus[101])
```

    The Fluke 87 beeps at you if you try to take it out of a current measuring
    range while there is still a lead plugged in to the current sense socket.

    ... this should solve your problem - unless you are in a noisy environment.

    JohnH

    ----------------------------------------------------------------------------

          |  _  |_   _   |_|  _   _|  _|              Electronics Department
        |_| (_) | | | |  | | (_| (_| (_| \/           School of MPCE
        ---------------------------------/-           Macquarie University
                                                      Sydney, AUSTRALIA 2109

        Email: johnh@mpce.mq.edu.au, Ph: +61 2 805 8959, Fax: +61 2 805 8983

    array([[0.00625   , 0.00625245, 0.00625198, 0.00625242, 0.00625087,
            0.00625038, 0.00625223, 0.5504139 , 0.39957521, 0.00625056]])

```python
print(data[251])
lda.transform(corpus[251])
```

    A while back someone had several equations which could be used for changing 3 f
    iltered grey scale images into one true color image.  This is possible because
    it's the same theory used by most color scanners.  I am not looking for the obv
    ious solution which is to buy a color scanner but what I do need is those equat
    ions becasue I am starting to write software which will automate the conversion
     process.  I would really appreciate it if someone would repost the 3 equations
    /3 unknowns.   Thanks for the help!!!

    array([[0.18916581, 0.00500063, 0.00500003, 0.005     , 0.00500067,
            0.00500084, 0.00500044, 0.0050012 , 0.77082958, 0.0050008 ]])

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez.
</footer>


# 12 ODE Applications (Projectile motion)
Let's apply our ODE solvers to some problems involving balls and projectiles.

The `integrators.py` file from [Lesson 10](https://asu-compmethodsphysics-phy494.github.io/ASU-PHY494/2019/02/19/10_ODEs/) is used here (and named [`ode.py`](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/tree/master/12_ODE_applications/ode.py)).

## Contents

1. Projectile with linear air-resistance (theory and full code)
2. Baseball physics (theory), see [baseball_solution](baseball_solution.ipynb) for the notebook

```python
import numpy as np
import ode
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.matplotlib.style.use('ggplot')
```

## Projectile with linear air-resistance
Linear drag force

$$
\mathbf{F}_1 = -b_1 \mathbf{v}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -\frac{b_1}{m} \mathbf{v}
\end{align}

Bring into standard ODE form for

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$

as

$$
\mathbf{y} = \begin{pmatrix}
x\\
y\\
v_x\\
v_y
\end{pmatrix}, \quad
\mathbf{f} = \begin{pmatrix}
v_x\\
v_y\\
-\frac{b_1}{m} v_x\\
-g -\frac{b_1}{m} v_y
\end{pmatrix}
$$

(Based on Wang 2016, Ch 3.3.1)

```python
def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):

    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

```python
r = simulate(initial_v(200, 30), h=0.01, b1=1)
```

```python
plt.plot(r[:, 1], r[:, 2])
plt.xlabel(r"distance $x$ (m)")
plt.ylabel(r"height $y$ (m)");
```

```python
for angle in (5, 7.5, 10, 20, 30, 45):
    r = simulate(initial_v(200, angle), h=0.01, b1=1)
    plt.plot(r[:, 1], r[:, 2], label=r"$\theta = {}^\circ$".format(angle))
plt.legend(loc="best")
plt.xlabel(r"distance $x$ (m)")
plt.ylabel(r"height $y$ (m)");
```

## Simple Baseball physics

- quadratic air resistance (with velocity-dependent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v \mathbf{v}
$$

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}\cdot\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y \mathbf{v} -\frac{b_2}{m} v \mathbf{v} + \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included.)

### Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

We will live-code the baseball simulation in class (i.e., build it from scratch), but if you want to work on this problem on your own and need some starter code, see below.

#### Baseball  code
Full solution will be posted as [`baseball_solution.ipynb`](baseball_solution.ipynb).

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`ipympl` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib ipympl
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], 'o', label="no spin")
# ...

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
```

## Reynolds number

$$
\text{Re} = \frac{\rho v L}{\mu}\\
\text{Re} > 2300\quad\text{flow turbulent}
$$

* density $\rho$: air  1.275 kg/m^3 (kilograms per cubic meter)
* fluid viscosity $\mu$: air 1.845×10^-5 Pa s (pascal seconds) (at 25 °C)

(from Wolfram Alpha)

```python
rho_air = 1.275    # kg/m^3
mu_air = 1.845e-5  # Pa s
```

```python
L = 0.05  # m
v = 200   # m/s
```

```python
def ReynoldsNumber(v, L, rho=rho_air, mu=mu_air):
    return rho*v*L/mu
```

```python
ReynoldsNumber(v, L)
```

    691056.9105691056

This means that we really should be using quadratic air resistance for the projectile.

```python

```


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x_1 = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python
y_1 = x_1.lower()
```

```python
print(y_1)
```

    diclorod  ifeni  ltriclor oetano  dichlorod ipheny ltrichl oroethane

```python
type(y_1)
```

    str

```python
y_1[0:len(y_1)].replace(" ", "")
```

    'diclorodifeniltricloroetanodichlorodiphenyltrichloroethane'

```python
print(y_1[0].replace("d", "D")+y_1[1:len(y_1)].replace(" ", ""))
```

    Diclorodifeniltricloroetanodichlorodiphenyltrichloroethane

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
a,b,c,d = 'word',15,3.14,'list'
```

```python
mySeparator = " "
```

```python
c = str(c)
```

```python
b = str(b)
```

```python
type(a)
```

    str

```python
type(b)
```

    str

```python
type(c)
```

    str

```python
type(d)
```

    str

```python
print(a, b, c, d)
```

    word 15 3.14 list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
wh = input("Enter your weight and height ")
```

    Enter your weight and height 68.4 1.71

```python
print(wh)
```

    68.4 1.71

```python
wh = wh.split(" ", 1)
```

```python
type(wh)
```

    list

```python
w = float(wh[0])
```

```python
h = float(wh[1])
```

```python
type(w)
```

    float

```python
type(h)
```

    float

```python
BMI = w/(h**2)
```

```python
print(BMI)
```

    23.39181286549708

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<center><\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
if(BMI<18.5): print("Underweight")
elif BMI >= 18.5 and BMI<25.0: print("Normal weight")
elif BMI >= 25.0 and BMI<29.9: print("Pre-obesity")
elif BMI >= 30.0 and BMI<34.9: print("Obesity class I")
elif BMI >= 35.0 and BMI<39.9: print("Obesity class II")
else: print("Obesity class III")
```

    Normal weight

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
number = input("Give me a number to calculate the factorial: ")
```

    Give me a number to calculate the factorial: 8

```python
number = int(number)
```

```python
fac = 1
```

```python
for i in range(1, number+1): fac = fac*i
```

```python
print(fac)
```

    40320

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
a = 0
```

```python
Sum = 0
```

```python
endless_int = [ ]
```

```python
while(a >= 0):
    endless_int.insert(a, input("Enter as many integers as you like: "))
    a = a+1
    if endless_int[a] == -1: break
```

```python
for n in range(0, len(endless_int)): Sum = sum(endless_int[n])
```

```python
res = Sum - endless_int[len(endless_int) - 1]
```

```python
print(res)
```

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
name, hours, salary_hour = "Bob", 20.5, 40.9
```

```python
total_salary = hours * salary_hour
```

```python
format_float = "{:.2f}".format(total_salary)
```

```python
print(total_salary)
```

    800

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
values = input("Give me three numbers: ")
```

    Give me three numbers: 17.5 6.3 9.7

```python
values = values.split()
```

```python
A = float(values[0])
```

```python
B = float(values[1])
```

```python
C = float(values[2])
```

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
triangle_area = 0.5 * A * C
```

```python
print(triangle_area)
```

    84.875

    b) the area of the circle of radius C. (pi = 3.14159)

```python
circle_area = 3.14159 * C * C
```

```python
print(circle_area)
```

    295.59220309999995

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
trapezoid_area = 0.5 * C * (A+B)
```

```python
print(trapezoid_area)
```

    115.42999999999999

    d) the area of the square that has side B.

```python
square_area = B * B
```

```python
print(square_area)
```

    39.69

    e) the area of the rectangle that has sides A and B.

```python
rectangle_area = A * B
```

```python
print(rectangle_area)
```

    110.25

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
val = input("Three numbers, please: ")
```

    Three numbers, please: 23.4 13.5 53.8

```python
val = val.split()
```

```python
a = float(val[0])
```

```python
b = float(val[1])
```

```python
c = float(val[2])
```

```python
import math
```

```python
root1 = -1 * b + (math.sqrt(math.pow(b, 2) - 4*a*c))/(2*a)
```

```python
root2 = -1 * b - (math.sqrt(math.pow(b, 2) - 4*a*c))/(2*a)
```

```python
if math.isnan(root1) == True and math.isnan(root2) == True: print("There are no real roots")
else: print(root1, root2)
```

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
coordinates = input("Enter 4 float numbers: ")
```

    Enter 4 float numbers: 56.78 -17.82 23.57 0.38

```python
coordinates = coordinates.split()
```

```python
x1 = float(coordinates[0])
```

```python
y1 = float(coordinates[1])
```

```python
x2 = float(coordinates[2])
```

```python
y2 = float(coordinates[3])
```

```python
d = math.sqrt((math.pow(x1 - x2, 2) + math.pow(y1 - y2, 2)))
```

```python
print(d)
```

    37.87009506193508

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
coord = input("Input two numbers: ")
```

    Input two numbers: 4.2 -3.7

```python
coord = coord.split()
```

```python
x = float(coord[0])
```

```python
y = float(coord[1])
```

```python
if(x == 0.0 and y == 0.0): print("You are at the origin")
elif(y == 0.0 and (x > 0.0 or x < 0.0)): print("You are on the x axis")
elif(x == 0.0 and (y > 0.0 or y < 0.0)): print("You are on the y axis")
elif(x > 0.0 and y > 0.0): print("First quadrant")
elif(x > 0.0 and y < 0.0): print("Second quadrant")
elif(x < 0.0 and y < 0.0): print("Third quadrant")
else: print("Fourth quadrant")
```

    Second quadrant

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
def func_nb():
 df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
 df = df.iloc[:,:2]
 df.head(20)
 return df

df = func_nb()
```

```python
phone_code = input("Enter a phone code: ")
```

```python
phone_code = int(phone_code)
```

```python
df.set_index("Country calling code")
```

```python
df[df["Country calling code"] == phone_code]

```

```python

```

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
six_numbers = input("Give me 6 numbers: ")
```

    Give me 6 numbers: 4.2 7.5 -1.2 -5.6 8.9 3.5

```python
six_numbers = six_numbers.split()
```

```python
pos = 0
```

```python
for i in range(0,5):
    if float(six_numbers[i]) > 0.0:
        pos = pos + 1
    else: continue
```

```python
print(pos)
```

    3

```python
average = 0.0
```

```python
average = (float(six_numbers[0]) + float(six_numbers[1]) + float(six_numbers[2]) + float(six_numbers[3]) + float(six_numbers[4]) + float(six_numbers[5]))/6
print(format(average, ".1f"))

```

    2.9

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
N = input("Input an integer: ")
```

    Input an integer: 6

```python
N = int(N)
```

```python
for num in range(1, N + 1):

    # checking condition
    if num % 2 == 0:
        print(num, num*num)
```

    2 4
    4 16
    6 36

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
Number = input("Input an integer: ")
```

    Input an integer: 34

```python
Number = int(Number)
```

```python
if Number % 2 == 0 and Number > 0:
    print("even", "positive")
elif Number % 2 == 0 and Number < 0:
    print("even", "negative")
elif Number % 2 != 0 and Number > 0:
    print("odd", "positive")
elif Number % 2 != 0 and Number < 0:
    print("odd", "negative")
else: print("null")
```

    even positive

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python

```