# Hypothesis Testing

## Single-Sample, One-Sided Tests
Our students have completed their school year, and been asked to rate their statistics class on a scale between -5 (terrible) and 5 (fantastic). The statistics class is taught online to tens of thousands of students, so to assess its success, we'll take a random sample of 50 ratings.

Run the following code to draw 50 samples.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

np.random.seed(123)
lo = np.random.randint(-5, -1, 6)
mid = np.random.randint(0, 3, 38)
hi = np.random.randint(4, 6, 6)
sample = np.append(lo,np.append(mid, hi))
print("Min:" + str(sample.min()))
print("Max:" + str(sample.max()))
print("Mean:" + str(sample.mean()))

plt.hist(sample)
plt.show()
```

A question we might immediately ask is: "how do students tend to like the class"? In this case, possible ratings were between -5 and 5, with a "neutral" score of 0. In other words, if our average score is above zero, then students tend to enjoy the course.

In the sample above, the mean score is above 0 (in other words, people liked the class in this data). If you had actually run this course and saw this data, it might lead you to believe that the overall mean rating for this class (i.e., not just the sample) is likely to be positive.

There is an important point to be made, though: this is just a sample, and you want to make a statement not just about your sample but the whole population from which it came. In other words, you want to know how the class was received overall, but you only have access to a limited set of data. This often the case when analyzing data.

So, how can you test your belief that your positive looking *sample* reflects the fact that the course does tend to get good evaluations, that your *population* mean (not just your sample mean) is positive?

We start by defining two hypotheses:

* The *null* hypothesis (**H0**) is that the population mean for all of the ratings is *not* higher than 0, and the fact that our sample mean is higher than this is due to random chance in our sample selection.
* The *alternative* hypothesis (**H1**) is that the population mean is actually higher than 0, and the fact that our sample mean is higher than this means that our sample correctly detected this trend.

You can write these as mutually exclusive expressions like this:

\begin{equation}H_{0}: \mu \le 0 \\ H_{1}: \mu > 0 \end{equation}

So how do we test these hypotheses? Because they are mutually exclusive, if we can show the null is probably not true, then we are safe to reject it and conclude that people really do like our online course. But how do we do that?

Well, if the *null* hypothesis is true, the sampling distribution for ratings with a sample size of 50 will be a normal distribution with a mean of 0. Run the following code to visualize this, with the mean of 0 shown as a yellow dashed line.

*(The code just generates a normal distribution with a mean of 0 and a standard deviation that makes it approximate a sampling distribution of 50 random means between -5 and 5 - don't worry too much about the actual values, it's just to illustrate the key points!)*

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

pop = np.random.normal(0, 1.15, 100000)
plt.hist(pop, bins=100)
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
plt.show()
```

This illustrates all the *sample* results you could get if the null hypothesis was true (that is, the rating population mean is actually 0). Note that if the null hypothesis is true, it's still *possible* to get a sample with a mean ranging from just over -5 to just under 5. The question is how *probable* is it to get a sample with a mean as high we did for our 50-rating sample under the null hypothesis? And how improbable would it *need* to be for us to conclude that the null is, in fact, a poor explanation for our data?

Well, we measure distance from the mean in standard deviations, so we need to find out how many standard deviations above the null-hypothesized population mean of 0 our sample mean is, and measure the area under the distribution curve from this point on - that will give us the probability of observing a mean that is *at least* as high as our sample mean. We call the number of standard deviations above the mean where our sample mean is found the *test statistic* (or sometimes just *t-statistic*), and we call the area under the curve from this point (representing the probability of observing a sample mean this high or greater) the *p-value*.

So the p-value tells us how probable our sample mean is when the null is true, but we need to set a threshold under which we consider this to be too improbable to be explained by random chance alone. We call this threshold our *critical value*, and we usually indicate it using the Greek letter alpha (**α**). You can use any value you think is appropriate for **α** - commonly a value of 0.05 (5%) is used, but there's nothing special about this value.

We calculate the t-statistic by performing a statistical test. Technically, when the standard deviation of the population is known, we call it a *z-test* (because a *normal* distribution is often called a *z-distribution* and we measure variance from the mean in multiples of standard deviation known as *z-scores*). When the standard deviation of the population is not known, the test is referred to as a *t-test* and based on an adjusted version of a normal distribution called a *student's t distribution*, in which the distribution is "flattened" to allow for more sample variation depending on the sample size. Generally, with a sample size of 30 or more, a t-test is approximately equivalent to a z-test.

Specifically, in this case we're performing a *single sample* test (we're comparing the mean from a single sample of ratings against the hypothesized population mean), and it's a *one-tailed* test (we're checking to see if the sample mean is *greater than* the null-hypothesized population mean - in other words, in the *right* tail of the distribution).

The general formula for one-tailed, single-sample t-test is:

\begin{equation}t = \frac{\bar{x} - \mu}{s \div \sqrt{n}} \end{equation}

In this formula, **x̄** is the sample mean, **μ** is the population mean, **s** is the standard deviation, and **n** is the sample size. You can think of the numerator of this equation (the expression at the top of the fraction) as a *signal*, and the denominator (the expression at the bottom of the fraction) as being *noise*. The signal measures the difference between the statistic and the null-hypothesized value, and the noise represents the random variance in the data in the form of standard deviation (or standard error). The t-statistic is the ratio of signal to noise, and measures the number of standard errors between the null-hypothesized value and the observed sample mean. A large value tells you that your "result" or "signal" was much larger than you would typically expect by chance.

Fortunately, most programming languages used for statistical analysis include functions to perform a t-test, so you rarely need to manually calculate the results using the formula.

Run the code below to run a single-sample t-test comparing our sample mean for ratings to a hypothesized population mean of 0, and visualize the resulting t-statistic on the normal distribution for the null hypothesis.

```python
from scipy import stats
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# T-Test
t,p = stats.ttest_1samp(sample, 0)
# ttest_1samp is 2-tailed, so half the resulting p-value to get a 1-tailed p-value
p1 = '%f' % (p/2)
print ("t-statistic:" + str(t))
print("p-value:" + str(p1))

# calculate a 90% confidence interval. 10% of the probability is outside this, 5% in each tail
ci = stats.norm.interval(0.90, 0, 1.15)
plt.hist(pop, bins=100)
# show the hypothesized population mean
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
# show the right-tail confidence interval threshold - 5% of propbability is under the curve to the right of this.
plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)
# show the t-statistic - the p-value is the area under the curve to the right of this
plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

In the plot produced by the code above, the yellow line shows the population mean for the null hypothesis. The area under the curve to the right of the red line represents the critical value of 0.05 (or 5%). The magenta line indicates how much higher the sample mean is compared to the hypothesized population mean. This is calculated as the t-statistic (which is printed above the plot) multiplied by the standard deviation. The area under the curve to the right of this encapsulates the p-value calculated by the test (which is also printed above the plot).

So what should we conclude from these results?

Well, if the p-value is smaller than our critical value of 0.05, that means that under the null hypothesis, the probability of observing a sample mean as high as we did by random chance is low. That's a good sign for us, because it means that our sample is unlikely under the null, and therefore the null is a poor explanation for the data. We can safely *reject* the null hypothesis in favor of the alternative hypothesis - there's enough evidence to suggest that the population mean for our class ratings is greater than 0.

Conversely, if the p-value is greater than the critical value, we *fail to reject the null hypothesis* and conclude that the mean rating is not greater than 0. Note that we never actually *accept* the null hypothesis, we just conclude that there isn't enough evidence to reject it!

## Two-Tailed Tests
The previous test was an example of a one-tailed test in which the p-value represents the area under one tail of the distribution curve. In this case, the area in question is under the right tail because the alternative hypothesis we were trying to show was that the true population mean is *greater than* the mean of the null hypothesis scenario.

Suppose we restated our hypotheses like this:
* The *null* hypothesis (**H0**) is that the population mean for all of the ratings is 0, and the fact that our sample mean is higher or lower than this can be explained by random chance in our sample selection.
* The *alternative* hypothesis (**H1**) is that the population mean is not equal to 0.

We can write these as mutually exclusive expressions like this:

\begin{equation}H_{0}: \mu = 0 \\ H_{1}: \mu \neq 0 \end{equation}

Why would we do this? Well, in the test we performed earlier, we could only reject the null hypothesis if we had really *positive* ratings, but what if our sample data looked really *negative*? It would be a mistake to turn around and run a one-tailed test the other way, for negative ratings. Instead, we conduct a test designed for such a question: a two-tailed test.

In a two-tailed test, we are willing to reject the null hypothesis if the result is significantly *greater* or *lower* than the null hypothesis. Our critical value (5%) is therefore split in two: the top 2.5% of the curve and the bottom 2.5% of the curve. As long as our test statistic is in that region, we are in the extreme 5% of values (p < .05) and we reject the null hypothesis. In other words, our p-value now needs to be below .025, but it can be in either tail of the distribution. For convenience, we usually "double" the p-value in a two-tailed test so that we don't have to remember this rule and still compare against .05 (this is known as a "two-tailed p-value"). In fact, it is assumed this has been done in all statistical analyses unless stated otherwise.

The following code shows the results of a two-tailed, single sample test of our class ratings. Note that the ***ttest_1samp*** function in the ***stats*** library returns a 2-tailed p-value by default (which is why we halved it in the previous example).

```python
from scipy import stats
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# T-Test
t,p = stats.ttest_1samp(sample, 0)
print ("t-statistic:" + str(t))
# ttest_1samp is 2-tailed
print("p-value:" + '%f' % p)
# calculate a 95% confidence interval. 50% of the probability is outside this, 2.5% in each tail
ci = stats.norm.interval(0.95, 0, 1.15)
plt.hist(pop, bins=100)
# show the hypothesized population mean
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
# show the confidence interval thresholds - 5% of propbability is under the curve outside these.
plt.axvline(ci[0], color='red', linestyle='dashed', linewidth=2)
plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)
# show the t-statistic thresholds - the p-value is the area under the curve outside these
plt.axvline(pop.mean() - t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

Here we see that our 2-tailed p-value was clearly less than 0.05; so We reject the null hypothesis.

You may note that doubling the p-value in a two-tailed test makes it harder to reject the null. This is true; we require more evidence because we are asking a more complicated question.

## Two-Sample Tests
In both of the previous examples, we compared a statistic from a single data sample to a null-hypothesized population parameter. Sometimes you might want to compare two samples against one another.

For example, let's suppose that some of the students who took the statistics course had previously studied mathematics, while other students had no previous math experience. You might hypothesize that the grades of students who had previously studied math are significantly higher than the grades of students who had not.

* The *null* hypothesis (**H0**) is that the population mean grade for students with previous math studies is not greater than the population mean grade for students without any math experience, and the fact that our sample mean for math students is higher than our sample mean for non-math students can be explained by random chance in our sample selection.
* The *alternative* hypothesis (**H1**) is that the population mean grade for students with previous math studies is greater than the population mean grade for students without any math experience.

We can write these as mutually exclusive expressions like this:

\begin{equation}H_{0}: \mu_{1} \le \mu_{2} \\ H_{1}: \mu_{1} > \mu_{2} \end{equation}

This is a one-sided test that compares two samples. To perform this test, we'll take two samples. One sample contains 100 grades for students who have previously studied math, and the other sample contains 100 grades for students with no math experience.

We won't go into the test-statistic formula here, but it essentially the same as the one above, adapted to include information from both samples. We can easily test this in most software packages using the command for an "independent samples" t-test:

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
%matplotlib inline

np.random.seed(123)
nonMath = np.random.normal(66.0, 1.5, 100)
math = np.random.normal(66.55, 1.5, 100)
print("non-math sample mean:" + str(nonMath.mean()))
print("math sample mean:" + str(math.mean()))

# Independent T-Test
t,p = stats.ttest_ind(math, nonMath)
# ttest_ind is 2-tailed, so half the resulting p-value to get a 1-tailed p-value
p1 = '%f' % (p/2)
print("t-statistic:" + str(t))
print("p-value:" + str(p1))

pop = np.random.normal(nonMath.mean(), nonMath.std(), 100000)
# calculate a 90% confidence interval. 10% of the probability is outside this, 5% in each tail
ci = stats.norm.interval(0.90, nonMath.mean(), nonMath.std())
plt.hist(pop, bins=100)
# show the hypothesized population mean
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
# show the right-tail confidence interval threshold - 5% of propbability is under the curve to the right of this.
plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)
# show the t-statistic - the p-value is the area under the curve to the right of this
plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

You can interpret the results of this test the same way as for the previous single-sample, one-tailed test. If the p-value (the area under the curve to the right of the magenta line) is smaller than our critical value (**α**) of 0.05 (the area under the curve to the right of the red line), then the difference can't be explained by chance alone; so we can reject the null hypothesis and conclude that students with previous math experience perform better on average than students without.

Alternatively, you could always compare two groups and *not* specify a direction (i.e., two-tailed). If you did this, as above, you could simply double the p-value (now .001), and you would see you could still reject the null hypothesis.

## Paired Tests
In the two-sample test we conduced previously, the samples were independent; in other words there was no relatioship between the observations in the first sample and the observations in the second sample. Sometimes you might want to compare statistical differences between related observations before and after some change that you believe might influence the data.

For example, suppose our students took a mid-term exam, and later took and end-of-term exam. You might hypothesise that the students will improve their grades in the end-of-term exam, after they've undertaken additional study. We could test for a general improvement on average across all students with a two-sample independent test, but a more appropriate test would be to compare the two test scores for each individual student.

To accomplish this, we need to create two samples; one for scores in the mid-term, exam, the other for scores in the end-of-term exam. Then we need to compare the samples in such a way that each pair of observations for the same student are compared to one another.

This is known as a paired-samples t-test or a dependent-samples t-test. Technically, it tests whether the *changes* tend to be in the positive or negative direction.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
%matplotlib inline

np.random.seed(123)
midTerm = np.random.normal(59.45, 1.5, 100)
endTerm = np.random.normal(60.05, 1.5, 100)

# Paired (related) test
t,p = stats.ttest_rel(endTerm, midTerm)
# ttest_rel is 2-tailed, so half the resulting p-value to get a 1-tailed p-value
p1 = '%f' % (p/2)
print("t-statistic:" + str(t))
print("p-value:" + str(p1))

pop = np.random.normal(midTerm.mean(), midTerm.std(), 100000)
# calculate a 90% confidence interval. 10% of the probability is outside this, 5% in each tail
ci = stats.norm.interval(0.90, midTerm.mean(), midTerm.std())
plt.hist(pop, bins=100)
# show the hypothesized population mean
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
# show the right-tail confidence interval threshold - 5% of propbability is under the curve to the right of this.
plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)
# show the t-statistic - the p-value is the area under the curve to the right of this
plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

In our sample, we see that scores did in fact improve, so we can we reject the null hypothesis.

```python

```

Statistics and Probability by Hiren/04-06-Hypothesis Testing.ipynb


# Hierarchical Clustering

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets, preprocessing
from sklearn.pipeline import make_pipeline
from sklearn.cluster import AgglomerativeClustering
# produce vector inline graphics
from IPython.display import set_matplotlib_formats, Code
set_matplotlib_formats('svg')
from ipywidgets import interact
import ipywidgets as widgets
from util import plot_dendrogram
from scipy.optimize import linear_sum_assignment
```

In this notebook, we will continue to cluster the instances in the iris dataset. We will try to use the hierarchical clustering algorithm to identify the different iris species without looking at the class attribute.

## Clustering using Weka

Similar to the last tutorial, use the `Cluster` panel to cluster the `iris.2D` dataset:

1. Using the `Preprocess` panel, load `iris.2D.arff` from the Weka data folder.
1. Using the `Cluster` panel, choose the `Clusterer` as `HierarchicalClusterer`, which implements the agglomerative nesting (AGNES) for different cluster distance measures.
1. The default number of clusters is \(k=2\). Change it to \(k=3\) instead, i.e., set `numClusters` to 3.
1. Select `Classes to clusters evaluation` as the `cluster mode`. Unlike other cluster modes, the class attribute is ignored automatically so there is NO need to click `ignore attributes` button below the `Cluster mode` to ignore `class`.
1. Click `Start` to run the clustering algorithm.

Alternatively, you may also use the [python-weka-wrapper](https://github.com/fracpete/python-weka-wrapper-examples/tree/master/src/wekaexamples/clusterers) to obtain the answer.

### Single-linkage

The default `linkType` is `SINGLE`, which measures the distance between two clusters, say $C$ and $C'$, by

\begin{align}
\min_{\boldsymbol{p}\in C, \boldsymbol{p'}\in C'} \operatorname{dist}(\boldsymbol{p}, \boldsymbol{p'}),
\end{align}

which is the minimum distance between points in different clusters. This gives the [single-linkage clustering solution](https://en.wikipedia.org/wiki/Single-linkage_clustering).

**Exercise**  Assign to `error_rate` the fraction (NOT percentage) of incorrectly clustered instances.

```python
# YOUR CODE HERE
raise NotImplementedError()
error_rate
```

```python
# tests
```

**Exercise** Right click your result and click `visualize cluster assignments`. (If the item is disabled, check the `Store clusters for visualization` option and rerun the clustering.) From the visualization of the errors, explain why the performance is better/worse than that of the $k$-means algorithm.

The distance between the closest points of Iris-versicolor and Iris-virginica is too small for the single-linkage algorithm to distinguish the two classes. The dendrogram suggests there are two clearly separated clusters only.

### Complete-linkage

Repeat the hierarchical clustering procedure but with the complete linkage algorithm by setting the `linkType` to COMPLETE. The distance between two clusters is measured by

\begin{align}
\max_{\boldsymbol{p}\in C, \boldsymbol{p'}\in C'} \operatorname{dist}(\boldsymbol{p}, \boldsymbol{p'}),
\end{align}

which is the maximum distance between points in different clusters. This gives the [complete-linkage clustering solution](https://en.wikipedia.org/wiki/Complete_linkage).

**Exercise** Assign to `error_rate` the fraction (NOT percentage) of incorrectly clustered instances.

```python
# YOUR CODE HERE
raise NotImplementedError()
error_rate
```

```python
# tests
```

**Exercise** Visualize the clustering assignments again and explain whether the clustering solution better than the single-linkage solution.

YOUR ANSWER HERE

### Dendrogram

To visualize the dendrogram, right click your last clustering result and select `Visualize tree`. (If the item is disabled, check the `Store clusters for visualization` option and rerun the clustering.) The following is dendrogram for complete linkage.

There are two issues with the dendrogram:

1. The dendrogram is incomplete and contains nodes for the first cluster only. The agglomerative clustering stopped immediately after having `numClusters = 3` clusters.
2. The leaf nodes have names that are fractions. By default, the leaf names are taken from the last attribute other than the ignored class attribute. In this case, the name such as 0.2 is a `petalwidth`, which is not useful in identifying the samples or evaluating the clustering solution.

To fix the above issues:

1. Set `numClusters` to 1 so that the agglomerative clustering continues to merge all nodes into a single clusters.
1. In the preprocess panel, create a new attribute `ID` as the last attribute to name the leaf node uniquely. You can copy the following configuration into the `Filter` panel:
```
weka.filters.MultiFilter -F "weka.filters.unsupervised.attribute.AddID -C last -N ID" -F "weka.filters.unsupervised.attribute.NumericToNominal -R last" -F "weka.filters.unsupervised.attribute.NominalToString -C last"
```

The above uses `MultiFilter` to compose three filters sequentially into one meta-filter.
1. `AddID` create a unique numeric ID for each sample.
1. `NumericToNominal` followed by `NominalToString` converts the ID to a string. The clustering algorithm use it for leaf name but not in the calculation of cluster distances.

After applying the filter and re-running the clustering algorithm with `numClusters = 1` and `class` attribute ignored, the resulting dendrogram is as follows:

**Exercise** Put the complete dendrogram for single-linkage method in the following cell. Each leaf name should be a unique ID.

YOUR ANSWER HERE

**Exercise** By comparing the dendrogram with single linkage vs that with complete linkage, explain why complete linkage gives a better solution for the iris dataset.

*Hint: Consider the range of cophenetic distance that gives 3 clusters instead of 2.*

YOUR ANSWER HERE

### Dendrogram in text format

The dendrogram is quite hard to read especially when the data set is large. By default, the dendrogram is printed as text in the result buffer:
```
Cluster 0
(((((((((((1:0,2:0):0,29:0):0,(5:0,9:0):0):0,((34:0,48:0):0,50:0):0):0.01695,((3:0,43:0):0,(37:0,39:0):0):0.01695):0.01695,((4:0,(8:0,11:0):0):0,(28:0,(40:0,49:0):0):0):0.0339):0.01982,(((7:0,(18:0,46:0):0):0.01695,(41:0,42:0):0.01695):0.01695,20:0.0339):0.01982):0.03625,(((10:0,38:0):0,(33:0,35:0):0):0.01695,13:0.01695):0.07301):0.01746,(14:0.04498,((15:0,36:0):0.0339,23:0.0339):0.01108):0.06245):0.11748,(((((6:0.01695,27:0.01695):0.01695,((16:0,22:0):0,32:0):0.0339):0.0339,17:0.0678):0.02982,(24:0.04498,44:0.04498):0.05264):0.07663,((((((((12:0,26:0):0,30:0):0,31:0):0,47:0):0.01695,21:0.01695):0.02803,19:0.04498):0.00873,25:0.05371):0.04391,45:0.09762):0.07663):0.05066):1.11921,(((((((((51:0,64:0):0.01695,77:0.01695):0.06263,(((56:0.01695,59:0.01695):0.01695,88:0.0339):0.01982,((66:0,76:0):0.0339,92:0.0339):0.01982):0.02586):0.01804,74:0.09762):0.04423,(((((54:0,72:0):0,90:0):0.01695,(89:0,100:0):0.01695):0.0339,((75:0,98:0):0.01695,(95:0,97:0):0.01695):0.0339):0.02873,(91:0.0339,96:0.0339):0.04568):0.06227):0.10672,(((((((((52:0,(79:0,85:0):0):0,67:0):0,69:0):0.01695,55:0.01695):0.01695,87:0.0339):0.01982,(57:0.0339,86:0.0339):0.01982):0.03625,107:0.08996):0.00766,62:0.09762):0.06153,(((53:0,73:0):0.0339,(120:0.01695,134:0.01695):0.01695):0.05114,(78:0.04498,84:0.04498):0.04006):0.07411):0.08942):0.14931,(((((71:0,(127:0,139:0):0):0.01695,(124:0,128:0):0.01695):0.04879,(((102:0,143:0):0.01695,147:0.01695):0.02803,150:0.04498):0.02076):0.04169,(((111:0.01695,114:0.01695):0.01695,122:0.0339):0.04568,(112:0.04498,148:0.04498):0.03459):0.02785):0.1693,(((104:0.01695,(117:0,138:0):0.01695):0.0678,(109:0.0339,126:0.0339):0.05085):0.09518,(130:0.08996,135:0.08996):0.08996):0.0968):0.12116):0.1883,((((58:0,94:0):0.0339,(61:0,80:0):0.0339):0.06054,99:0.09443):0.10278,((60:0.06574,65:0.06574):0.10434,(((63:0.01695,68:0.01695):0.05085,82:0.0678):0.02982,((70:0.01695,81:0.01695):0.03676,(83:0.01695,93:0.01695):0.03676):0.04391):0.07246):0.02714):0.38897):0.24263,(((((101:0.01695,110:0.01695):0.07301,(136:0.0339,144:0.0339):0.05607):0.01746,((137:0,141:0):0.04498,145:0.04498):0.06245):0.09715,((((103:0.0339,125:0.0339):0.05085,((113:0.01695,129:0.01695):0.01695,140:0.0339):0.05085):0.01288,((105:0.0339,133:0.0339):0.01108,121:0.04498):0.05264):0.0868,(115:0.06574,(((116:0.01695,146:0.01695):0.01695,142:0.0339):0.01695,149:0.05085):0.01489):0.11868):0.02016):0.1177,(((106:0.04498,118:0.04498):0.05264,119:0.09762):0.13421,((108:0.05371,131:0.05371):0.05619,(123:0.05085,132:0.05085):0.05905):0.12193):0.09046):0.50654):0.5153)
```
which is in [Newick format](https://en.wikipedia.org/wiki/Newick_format#Examples) with distances and leaf names. E.g., `(1:0,2:0)` means node `1` and `2` are merged to an internal node at cophenetic distance `0`.

**Exercise** From the following dendrogram in Newick format, assign to `cluster0` and `cluster1` the two clusters merged last, and assign to `cluster_distance` the distance of the two clusters.
```
((((((1:0,2:0):0,29:0):0,(5:0,9:0):0):0,((34:0,48:0):0,50:0):0):0.01695,((3:0,43:0):0,(37:0,39:0):0):0.01695):0.01695)
```
Your answer may look like
```Python
cluster0 = {1, 2, ...}
cluster1 = {...}
cluster_distance = ___
```

```python
# YOUR CODE HERE
raise NotImplementedError()
cluster0, cluster1, cluster_distance
```

```python
# tests
```

## Clustering using scikit-learn

We first import the iris dataset from `sklearn.datasets` and store it as a DataFrame.

```python
# load the dataset from sklearn
dataset = datasets.load_iris()

# create a DataFrame to help further analysis
df = pd.DataFrame(data=dataset.data, columns=dataset.feature_names)
df['target'] = dataset.target
df.target = df.target.astype('category')
df.target.cat.categories = dataset.target_names
df  # display an overview of the data
```

To normalize the features followed by [agglomerative clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html), we create a pipeline as follows:

```python
from sklearn.cluster import AgglomerativeClustering

agnes_minmax_normalized = make_pipeline(
    preprocessing.MinMaxScaler(),
    AgglomerativeClustering(n_clusters=3,
                            linkage='complete',
                            memory='private',
                            compute_distances=True))
agnes_minmax_normalized
```

In the above, we configured the agglomerative clustering algorithm to use complete-linkage to return 3 clusters. Furthermore:
- By setting the parameter `memory`, the solution will be cached to the specified folder `private`.
- By setting compute_distances to `True`, the cluster distances are computer for dendrogram visualization later on.

To cluster based on the two input features `petal length (cm)` and `petal width (cm)`:

```python
feature1, feature2 = 'petal length (cm)', 'petal width (cm)'
cluster_labels = agnes_minmax_normalized.fit_predict(df[[feature1, feature2]])

plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.scatter(df[feature1], df[feature2], c=cluster_labels)
plt.title("Cluster assignment")
plt.xlabel(feature1)
plt.ylabel(feature2)
plt.subplot(122)
plt.scatter(df[feature1], df[feature2], c=dataset['target'])
plt.title("Class (ground truth)")
plt.xlabel(feature1)
plt.show()
```

The [`fit_predict`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering.fit_predict) method fits and returns the cluster labels for the given data.

**Exercise** `AgglomerateClustering` does not implement [a separate `predict` method, unlike `KMeans`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans.predict). Why?

The `predict` method of `KMeans` take samples that are possibly not used for training and returns the closest cluster the samples belong to by comparing . This is not possible for `AgglomerativeClustering` as it can only cluster samples used for training.

### Dendrogram

Further details of a clustering solution can be obtained from the fitted properties (those with a [trailing underscore](https://github.com/rasbt/python-machine-learning-book/blob/master/faq/underscore-convention.md)):

```python
agnes = agnes_minmax_normalized['agglomerativeclustering']
print('Cluster labels for each sample:', *agnes.labels_)
print('Children of each non-leaf node:', *agnes.children_)
print('Distances between nodes:', *agnes.distances_.round(3))
print('Number of leaves:', agnes.n_leaves_)
print('Number of connected components:', agnes.n_connected_components_)
```

To plot the dendrogram, we will use the function `plot_dendrogram` below:

```python
Code('util.py')
```

import numpy as np
from scipy.cluster.hierarchy import dendrogram
import matplotlib.pyplot as plt

def plot_dendrogram(model, ax=None, **kwargs):
    '''Plot the dendrogram a clusterer on a 2D dataset.

    Parameters
    ----------
    model (sklearn): An AgglomerativeClustering object with cluster distances computed,
      e.g., by setting the parameter compute_distances=True and applying fit method to samples.
    ax (axis): axis to plot the dendrogram.
    **kwargs: additional parameters to be passed to scipy.cluster.hierarchy.dendrogram

    Return
    ------
    axis: axis for the plot of the decision boundaries

    See also
    --------
    scipy.cluster.hierarchy.dendrogram

    Reference
    ---------
    https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html
    '''
    if ax is None:
        ax = plt.gca()

    # create the counts of samples under each node
    counts = np.zeros(model.children_.shape[0])
    n_samples = len(model.labels_)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_samples:
                current_count += 1  # leaf node
            else:
                current_count += counts[child_idx - n_samples]
        counts[i] = current_count

    # Create linkage matrix and then plot the dendrogram
    linkage_matrix = np.column_stack([model.children_, model.distances_,
                                      counts]).astype(float)

    # Plot the corresponding dendrogram
    dendrogram(linkage_matrix, ax=ax, **kwargs)
    return ax

The code is adopted from [an example in `sklearn`](https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html), which uses [`dendrogram` from `scipy.cluster.hierarchy`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html). To generate the dendrogram:

```python
from util import plot_dendrogram

plt.figure(figsize=(10, 5))
plot_dendrogram(agnes, leaf_font_size=4, distance_sort=True)
plt.title('Dendrogram for complete-linkage method on iris dataset')
plt.ylabel('cophenetic distance')
plt.xlabel("instance index")
plt.show()
```

You can improve the visualization using the [additional parameters](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html) passed to `scipy.cluster.hierarchy.dendrogram`.

**Exercise** Complete the following code to assign to `agnes_minmax_normalized` a pipelined agglomerative clusterer that
- min-max normalizes the features, and
- clusters the instances of features specified by `feature1` and `feature2` into `k` clusters according to the linkage method specified by `linkage`.

Observe the change in the dendrogram and clustering solution when you change the linkage, features, and number of clusters.

```python
@interact(linkage=['ward', 'complete', 'average', 'single'],
          feature1=dataset.feature_names,
          feature2=dataset.feature_names)
def analyze_agnes(
        linkage,
        feature1=dataset.feature_names[2],
        feature2=dataset.feature_names[3],
        k=widgets.IntSlider(3, 1, 4, continuous_update=False),
):
    # YOUR CODE HERE
    raise NotImplementedError()
    agnes = agnes_minmax_normalized['agglomerativeclustering']
    plt.figure(figsize=(10, 10))

    plt.subplot(211)
    plot_dendrogram(agnes)
    plt.title(f'Dendrogram for {linkage}-linkage method on iris dataset')
    plt.ylabel('cophenetic distance')
    plt.xlabel("instance index")

    plt.subplot(223)
    plt.scatter(df[feature1], df[feature2], c=agnes.labels_)
    plt.title("Cluster assignment")
    plt.ylabel(feature2)
    plt.xlabel(feature1)

    plt.subplot(224)
    plt.scatter(df[feature1], df[feature2], c=dataset['target'])
    plt.title("Class (ground truth)")
    plt.xlabel(feature1)

    plt.show()
```


# Notebook 4: Linear Regression (Ising)

## Learning Goal
Let us now apply linear regression to an example that is familiar from Statistical Mechanics: the Ising model. The goal of this notebook is to revisit the concepts of in-sample and out-of-sample errors, as well as $L2$- and $L1$-regularization, in an example that is more intuitive to physicists.

## Overview
Consider the 1D Ising model with nearest-neighbor interactions

$$H[\boldsymbol{S}]=-J\sum_{j=1}^L S_{j}S_{j+1}$$

on a chain of length $L$ with periodic boundary conditions and $S_j=\pm 1$ Ising spin variables. In one dimension, this paradigmatic model has no phase transition at finite temperature.

### Exercises (optional): ###
We invite the reader who is unfamiliar with the property of the Ising model to solve the following problems.

 Compute the partition function of the Ising model in one dimension at inverse temperature $\beta$ when $L\rightarrow\infty$ (thermodynamic limit):
    $$Z=\sum_S \exp(-\beta H[S]).$$
Here the sum is carried over all $2^L$ spin configurations.

 Compute the model's magnetization $M=\langle\sum_i S_i\rangle$ in the same limit ($L\rightarrow\infty$). The expectation is taken with respect to the Boltzmann distribution:
    $$p(S)=\frac{\exp(-\beta H[S])}{Z}$$

 How does $M$ behave as a function of the temperature $T=\beta^{-1}$?

For a more detailed introduction we refer the reader to consult one of the many textbooks on the subject (see for instance Goldenfeld, Lubensky, Baxter , etc.).

### Learning the Ising model ###

Suppose your boss set $J=1$, drew a large number of spin configurations, and computed their Ising energies. Then, without telling you about the above Hamiltonian, he or she handed you a data set of $i=1\ldots n$ points of the form $\{(H[\boldsymbol{S}^i],\boldsymbol{S}^i)\}$. Your task is to learn the Hamiltonian using Linear regression techniques.

```python
import numpy as np
import scipy.sparse as sp
np.random.seed(12)

import warnings
# Comment this to turn on warnings
warnings.filterwarnings('ignore')

### define Ising model aprams
# system size
L=40

# create 10000 random Ising states
states=np.random.choice([-1, 1], size=(10000,L))

def ising_energies(states):
    """
    This function calculates the energies of the states in the nn Ising Hamiltonian
    """
    L = states.shape[1]
    J = np.zeros((L, L),)
    for i in range(L):
        J[i,(i+1)%L]=-1.0 # interaction between nearest-neighbors

    # compute energies
    E = np.einsum('...i,ij,...j->...',states,J,states)

    return E
# calculate Ising energies
energies=ising_energies(states)
```

## Recasting the problem as a Linear Regression
First of all, we have to decide on a model class (possible Hamiltonians) we use to fit the data. In the absence of any prior knowledge, one sensible choice is the all-to-all Ising model

$$
H_\mathrm{model}[\boldsymbol{S}^i] = - \sum_{j=1}^L \sum_{k=1}^L J_{j,k}S_{j}^iS_{k}^i.
$$
Notice that this model is uniquely defined by the non-local coupling strengths $J_{jk}$ which we want to learn. Importantly, this model is linear in ${\mathbf J}$ which makes it possible to use linear regression.

To apply linear regression, we would like to recast this model in the form
$$
H_\mathrm{model}^i \equiv \mathbf{X}^i \cdot \mathbf{J},
$$

where the vectors $\mathbf{X}^i$ represent all two-body interactions $\{S_{j}^iS_{k}^i \}_{j,k=1}^L$, and the index $i$ runs over the samples in the data set. To make the analogy complete, we can also represent the dot product by a single index $p = \{j,k\}$, i.e. $\mathbf{X}^i \cdot \mathbf{J}=X^i_pJ_p$. Note that the regression model does not include the minus sign, so we expect to learn negative $J$'s.

```python
# reshape Ising states into RL samples: S_iS_j --> X_p
states=np.einsum('...i,...j->...ij', states, states)
shape=states.shape
states=states.reshape((shape[0],shape[1]*shape[2]))
# build final data set
Data=[states,energies]
```

## Numerical Experiments

As we already mentioned a few times in the review, learning is not fitting: the subtle difference is that once we fit the data to obtain a candidate model, we expect it to generalize to unseen data not used for the fitting procedure. For this reason, we begin by specifying a training and test data sets

```python
# define number of samples
n_samples=400
# define train and test data sets
X_train=Data[0][:n_samples]
Y_train=Data[1][:n_samples] #+ np.random.normal(0,4.0,size=X_train.shape[0])
X_test=Data[0][n_samples:3*n_samples//2]
Y_test=Data[1][n_samples:3*n_samples//2] #+ np.random.normal(0,4.0,size=X_test.shape[0])
```

# Evaluating the performance:  coefficient of determination $R^2$

In what follows the model performance (in-sample and out-of-sample) is evaluated using the so-called coefficient of determination, which is given by:
\begin{align}
R^2 &= \left(1-\frac{u}{v}\right),\\
u&=(y_{pred}-y_{true})^2\\
v&=(y_{true}-\langle y_{true}\rangle)^2
\end{align}

The best possible score is 1.0 but it can also be negative. A constant model that always predicts the expected value of $y$, $\langle y_{true}\rangle$, disregarding the input features, would get a $R^2$ score of 0.

## Applying OLS, Ridge regression and LASSO:

```python
from sklearn import linear_model
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import seaborn
%matplotlib inline

# set up Lasso and Ridge Regression models
leastsq=linear_model.LinearRegression()
ridge=linear_model.Ridge()
lasso = linear_model.Lasso()

# define error lists
train_errors_leastsq = []
test_errors_leastsq = []

train_errors_ridge = []
test_errors_ridge = []

train_errors_lasso = []
test_errors_lasso = []

# set regularisation strength values
lmbdas = np.logspace(-4, 5, 10)

#Initialize coeffficients for ridge regression and Lasso
coefs_leastsq = []
coefs_ridge = []
coefs_lasso=[]

for lmbda in lmbdas:

    ### ordinary least squares
    leastsq.fit(X_train, Y_train) # fit model
    coefs_leastsq.append(leastsq.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_leastsq.append(leastsq.score(X_train, Y_train))
    test_errors_leastsq.append(leastsq.score(X_test,Y_test))

    ### apply RIDGE regression
    ridge.set_params(alpha=lmbda) # set regularisation parameter
    ridge.fit(X_train, Y_train) # fit model
    coefs_ridge.append(ridge.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_ridge.append(ridge.score(X_train, Y_train))
    test_errors_ridge.append(ridge.score(X_test,Y_test))

    ### apply LASSO regression
    lasso.set_params(alpha=lmbda) # set regularisation parameter
    lasso.fit(X_train, Y_train) # fit model
    coefs_lasso.append(lasso.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_lasso.append(lasso.score(X_train, Y_train))
    test_errors_lasso.append(lasso.score(X_test,Y_test))

    ### plot Ising interaction J
    J_leastsq=np.array(leastsq.coef_).reshape((L,L))
    J_ridge=np.array(ridge.coef_).reshape((L,L))
    J_lasso=np.array(lasso.coef_).reshape((L,L))

    cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')

    fig, axarr = plt.subplots(nrows=1, ncols=3)

    axarr[0].imshow(J_leastsq,**cmap_args)
    axarr[0].set_title('OLS \n Train$=%.3f$, Test$=%.3f$'%(train_errors_leastsq[-1], test_errors_leastsq[-1]),fontsize=16)
    axarr[0].tick_params(labelsize=16)

    axarr[1].imshow(J_ridge,**cmap_args)
    axarr[1].set_title('Ridge $\lambda=%.4f$\n Train$=%.3f$, Test$=%.3f$' %(lmbda,train_errors_ridge[-1],test_errors_ridge[-1]),fontsize=16)
    axarr[1].tick_params(labelsize=16)

    im=axarr[2].imshow(J_lasso,**cmap_args)
    axarr[2].set_title('LASSO $\lambda=%.4f$\n Train$=%.3f$, Test$=%.3f$' %(lmbda,train_errors_lasso[-1],test_errors_lasso[-1]),fontsize=16)
    axarr[2].tick_params(labelsize=16)

    divider = make_axes_locatable(axarr[2])
    cax = divider.append_axes("right", size="5%", pad=0.05, add_to_figure=True)
    cbar=fig.colorbar(im, cax=cax)

    cbar.ax.set_yticklabels(np.arange(-1.0, 1.0+0.25, 0.25),fontsize=14)
    cbar.set_label('$J_{i,j}$',labelpad=15, y=0.5,fontsize=20,rotation=0)

    fig.subplots_adjust(right=2.0)

    plt.show()
```

To quantify learning, we also plot the in-sample and out-of-sample errors

```python
# Plot our performance on both the training and test data
plt.semilogx(lmbdas, train_errors_leastsq, 'b',label='Train (OLS)')
plt.semilogx(lmbdas, test_errors_leastsq,'--b',label='Test (OLS)')
plt.semilogx(lmbdas, train_errors_ridge,'r',label='Train (Ridge)',linewidth=1)
plt.semilogx(lmbdas, test_errors_ridge,'--r',label='Test (Ridge)',linewidth=1)
plt.semilogx(lmbdas, train_errors_lasso, 'g',label='Train (LASSO)')
plt.semilogx(lmbdas, test_errors_lasso, '--g',label='Test (LASSO)')

fig = plt.gcf()
fig.set_size_inches(10.0, 6.0)

#plt.vlines(alpha_optim, plt.ylim()[0], np.max(test_errors), color='k',
#           linewidth=3, label='Optimum on test')
plt.legend(loc='lower left',fontsize=16)
plt.ylim([-0.1, 1.1])
plt.xlim([min(lmbdas), max(lmbdas)])
plt.xlabel(r'$\lambda$',fontsize=16)
plt.ylabel('Performance',fontsize=16)
plt.tick_params(labelsize=16)
plt.show()
```

## Understanding the results

Let us make a few remarks:

(i) the inverse (see [Scikit documentation](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model)) regularization parameter $\lambda$ affects the Ridge and LASSO regressions at scales, separated by a few orders of magnitude. Notice that this is different for the data considered in Notebook 3 __Section VI: Linear Regression (Diabetes)__. Therefore, it is considered good practice to always check the performance for the given model and data with $\lambda$ over multiple decades.

(ii) at $\lambda\to 0$ and $\lambda\to\infty$, all three models overfit the data, as can be seen from the deviation of the test errors from unity (dashed lines), while the training curves stay at unity.

(iii) While the OLS and Ridge regression test curves are monotonic, the LASSO test curve is not -- suggesting the optimal LASSO regularization parameter is $\lambda\approx 10^{-2}$. At this sweet spot, the Ising interaction weights ${\bf J}$ contain only nearest-neighbor terms (as did the model the data was generated from).

__Gauge degrees of freedom__: recall that the uniform nearest-neighbor interactions strength $J_{j,k}=J$ which we used to generate the data was set to unity, $J=1$. Moreover, $J_{j,k}$ was NOT defined to be symmetric (we only used the $J_{j,j+1}$ but never the $J_{j,j-1}$ elements). The colorbar on the matrix elements plot above suggest that the OLS and Ridge regression learn uniform symmetric weights $J=-0.5$. There is no mystery since this amounts to taking into account both the $J_{j,j+1}$ and the $J_{j,j-1}$ terms, and the weights are distributed symmetrically between them. LASSO, on the other hand, can break this symmetry (see matrix elements plots for $\lambda=0.001$ and $\lambda=0.01$). Thus, we see how different regularization schemes can lead to learning equivalent models but in different gauges. Any information we have about the symmetry of the unknown model that generated the data has to be reflected in the definition of the model and the regularization chosen.

### Exercises: ###

 Plot a histogram of the distribution of the components of ${\bf J}$ at different values of the number of training sample (one can go up to $2\times 10^4$). What happens with the sampling noise as the number of samples is increased/decreased for the three types of regression considered? How do the matrix elements plots above change?

 Try to learn the underlying model of the data, assuming it lies within the class of one-body Hamiltonians, i.e. make the ansatz
$$H_\mathrm{model}[\boldsymbol{S}^i] = \sum_{j=1}^L h_jS_{j}$$
for some unknown field $h_j$. How well can you explain the data? How well does the model generalize? Study these problems by playing with the size of the data set. Try out all three regression models and determine which one does the best. What is the relationship to Mean-Field Theory of this model?

```python

```


# Crecimiento ilimitado de la población

Un modelo elemental del crecimiento de una población se basa en la hipótesis de que:

## La velocidad de crecimiento de la población es proporcional al tamaño de la población

1. Observe que la razón de cambio de una población sólo depende del tamaño de ésta.

- En particular, las limitaciones del espacio o recursos no tienen efecto.

- Esta hipótesis es razonable para pequeñas poblaciones en grandes entornos, por ejemplo, los primeros brotes de moho en una pieza de pan o los primeros colonizadores de Estados Unidos.

Como la hipótesis es tan simple, esperamos que el modelo también lo sea, Las cantidades implicadas son

- $t$ = tiempo (variable inependiente).
- $P$ = población (variable dependiente)
- $k$ = constante de proporcionalidad (parámetro) entre la tasa de crecimiento de la población y el tamaño de ésta.

** El parámetro $k$ suele llamarse "coeficiente de velocidad de crecimiento".**

Las unidades para esas cantidades dependen de la aplicación.

- Si estamos modelando el crecimiento de moho en el pan, entonces $t$ podría medirse en días y $P(t)$ sería el área cubierta por el moho o bien el peso del moho.

- Si estamos hablando de la población europea en Estados Unidos, entonces $t$ probablemente se medirá en años y $P(t)$ en millones de personas. En este caso haríamos corresponder $t=0$ a cualquier tiempo que quisiéramos. El año 1790 (el año del primer censo) es una opción conveniente.

## Expresemos ahora nuestra hipótesis usando esta notación.

La tasa de crecimiento de la población $P$ es la derivada $\frac{dP}{dt}$. Puesto que ésta es proporcional a la población, se expresa como el producto, $kP$, de la población $P$ y la constante $k$ de proporcionalidad.

Por consiguiente, nuestra hipótesis se expresa por la ecuación diferencial

$$ \frac{dP}{dt} = kP $$

En otras palabras, la razón de cambio de $P$ es proporcional a $P$.
** Éste es nuestro primer ejemplo de una ecuación diferencial.**

Asociada con ella hay varios adjetivos que describen su tipo. En particula, se trata de una ecuación de **primer orden** porque contiene sólo primeras derivadas de la variable dependiente, y es una **ecuación differencial ordinaria** porque no contiene derivadas parciales.

Hemos escrito esta ecuación diferencial usando la notación de Leibniz, es decir, $\frac{dP}{dt}$, y es la que tenderemos a usar. Sin embargo, hay muchas otras maneras de expresar la misma ecuación diferencial.

En particula, también podríamos escribirla como $P'=kP$ o como $\dot{P} = kP$. la notación "punto" suele utilizarse cuando la variable independiente es el tiempo $t$.

# ¿Qué predice el modelo?

Más importante que los adjetivos o cómo se escribe la ecuación es preguntar qué nos dice acerca de la situación que se está modelando. Como $\frac{dP}{dt}=kP$ para alguna constante $k$ y $\frac{dP}{dt}=0$ si $P=0$.

Entonces la función constante $P(t)=0$ es una solución de la ecuacioón diferencial. A este tipo especial se le denomina **solución de equilibrio ** porque es constante para siempre. En términos del modelo de población, corresponde a una especie que es no existente.

Si $P(t_0) \neq 0$ en algún tiempo $t_0$, entonces en el tiempo $t=t_0$

$$ \frac{dP}{dt} = kP(t_0) \neq 0 $$

En consecuencia, la población no es constante. Si $k>0$ y $P(t_0)>0$, tenemos

$$ \frac{dP}{dt}= kP >0$$

en el tiempo $t=t_0$ y la población está creciendo (como era de esperarse).

Conforme $t$ crece, $P(t)$ se vuelve mayor, por lo que $ \frac{dP}{dt} $ aumenta. A su vez, $P(t)$ crece aún más rápidamente.

Es decir, la velocidad de crecimiento crece en relación directa con la población.

El valor de $P(t)$ en  $t=0 $ se llama una **condición inicial **.

Si comenzamos con una condición inicial diferente obtenemos una función $P(t)$ distinta,

```python

```

# Solucion analitica de la ecuación diferencial

Si, por otra parte, conocemos el valor exacto de $P_0$ de $P(0)$ y queremos predecir el valor de $P(10)$ o $P(100)$, entonces necesitamos información más precisa sobre la función $P(t)$. El par de ecuaciónes

## $$ \frac{dP}{dt}= kP \ \ \ \ P(0) = P_0$$

se llama **problema de valor inicial. ** Y una ** solución** al problema de valor inicial es una función $P(t)$ que satisface ambas ecuaciones. Es decir

## $$ \frac{dP}{dt}= kP \ \ y \ \ P(0) = P_0$$

En consecuencia, para solucionar esta ecuación diferencial debemos hallar una función $P(t)$ cuya derivada sea el producto de $k$ con $P(t)$.

# Separación de variables

## $$ \frac{dP}{P}= kdt$$

## $$\int \frac{dP}{P}= \int kdt$$

## $$ \ln P + c_1= k t +c_2$$
## $$ e^{\ln P} = e^{k t +c}$$
## $$ P(t) = e^{k t} e^{c}$$
## $$ P(t) = ce^{k t} $$

**Existe un número infinito de soluciones** para la ecuación diferencial, **una para cada valor de $c$.**

Para determinar cuál de ésas es la correcta para la situación considerada, usamos la condición inicial dada.

## $$P_0= P(0) = c \cdot e^{k\cdot 0} = c \cdot e^0 = c \cdot 1 = c $$

En consecuencia, debemos escoger $c= P_0$, por lo que una solución del problema del valor inicial es

## $$ P(t) = P_0e^{k t} $$

La función $P(t)$ se llama solución al problema del valor inicial así como **solución particular** de la ecuación diferencial.

El conjunto de funciones $P(t) = ce^{kt}$ se llama ** solucion general** de la ecuación general, porque podemos usarla para encontrar la respuesta particular correspondiente a cualquier problema de valor inicial.

# Ejemplo: Método de Euler

$$ \frac{dP}{dt}= 2P \ \ \ \ P(0) = 10 $$

$$ \frac{P_{k+1}-P_{k}}{h}= 2P_k $$

$$ P_{k+1}=P_{k} + h2P_k $$

Es evidente que la ordenada $y_1$ calculada de esta manera no es igual a $F(x1)$ pues existe un pequeño error. No obstante, el valor $y_1$ sive para aproximar $F'(x)$ en el punto $P=(x_1,y_1)$ y repeteri el procedimiento anterior a fin de generar la sucesión de aproximaciones siguiente

$$ y_1 = y_0 + h f(x_0,y_0)$$
$$ y_2 = y_1 + h f(x_1,y_1)$$
$$\vdots $$
$$ y_{i+1} = y_i + h f(x_i,y_i)$$
$$\vdots $$
$$ y_{n} = y_{n-1} + h f(x_{n-1},y_{n-1})$$

```python

```

# Ejemplo:  Solución analitica de la ecuación diferencial

$$ \frac{dP}{dt}= 2P \ \ \ \ P(0) = 10, \ \ \ k=2$$

## Separación de variables

$$ \frac{dP}{P}= 2dt$$

$$\int \frac{dP}{P}= \int 2dt$$

$$ \ln P + c_1= 2 t +c_2$$
$$ e^{\ln P} = e^{2 t +c}$$
$$ P(t) = e^{2 t} e^{c}$$
$$ P(t) = ce^{2 t} $$

$$P_0= P(0) = c \cdot e^{2\cdot 0} = c \cdot e^0 = c \cdot 1 = c $$

Solución:
$$ P(t)=  10e^{2t}$$

- Ecuación diferencial
$$ \frac{dP}{dt}= 2P \ \ \ \ P(0) = 10 $$
- Método de Euler
$$ P_{k+1}=P_{k} + h2P_k $$
- Solución particular de la ecuación diferencial.
$$ P(t) = P_0e^{2 t} $$

# Ejemplo

```python
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import matplotlib as mpl
label_size = 14
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
```

```python

```

```python
P = lambda t: 10*np.exp(2*t)

def dP (Pk,h):
    Pk1= Pk + h*2*Pk
    return Pk1

def dP_dt (P,t):
    dP= 2*P
    return dP

tfin =3
h=.1

#Solucion exacta
t = np.arange(0,tfin+.1,.1)
plt.plot(t,P(t),"b")

#Utilizando metodo de euler
Pk=np.arange(0,tfin+h,h)
Pk[0]=10

for k in np.arange(0,Pk.size-1) :
    Pk[k+1] = dP(Pk[k],h)

tk=np.arange(0,tfin+h,h)
plt.plot(tk,Pk,"g")

#Utilizando odeint
t = np.arange(0,3,.1);
x = odeint(dP_dt,10,t)
plt.plot(t,x,'r');
```

```python

```

# La población de Estados Unidos

Consideremos las cifras de los censos de Estados Unidos desde 1790.

- Medimos el tiempo en años y la población $P(t)$ en millones de personas.

- Hacemos que $t=0$ sea el año $1790$, por lo que la condición inicial es $P(0) = 3.9$

## $$ \frac{dP}{dt}= kP \ \ \ \ P(0) = 3.9$$

## tiene $ P(t) = 3.9 e^{k t} $ como solución.

Pero no podemos usar este modelo para hacer predicciones porque **no conocemos el valor de $k$**.

Sin embargo, sabemos que la población en el año $1800$ era de $5.3$ millones y podemos usar este valor para determinar $k$. Si hacemos

## $$5.3 = P(10) = 3.9 e^{k\cdot 10 }$$

tenemos entonces

## $$  e^{k\cdot 10}  = \frac{5.3}{3.9} $$

## $$  10 k   = \ln \frac{5.3}{3.9} $$

## $$ k   \approx 0.03067. $$

## $$ P(t) = 3.9 e^{ 0.03067 \cdot t} $$

```python
año = np.arange(1790,2000,10)
t = np.arange(0,210,10)
real = np.array([3.9,5.3,7.2,9.6,12,17,23,31,38,50,62,75,91,105,122,131,151,179,203,226,249])

Pt = 3.9*np.exp(0.03067*t)

plt.plot(t,real)
plt.plot(t,Pt)

_=plt.xticks(t, año, rotation='vertical')
plt.margins(0.05)
plt.xlabel("t")
plt.ylabel("P(t)")
```

Nuestro modelo es bastante bueno siempre que la población sea relativamente pequeña.

Sin embargo, con el paso del tiempo el modelo predice que la población continuará creciendo sin límite, y obviamente esto no sucede en el mundo real.

En consecuencia, si queremos un modelo que sea exacto sobre una escala grande de tiempo, debemos tomar en cuenta el hecho de que la poblaciones existen en una cantidad finita de espacio y con recursos limitados.

```python

```

# Ejercicio: La población de Estados Unidos

## $$ \frac{dP}{dt}= kP \ \ \ \ P(0) = 3.9$$

## $$ k   \approx 0.03067. $$

```python
def df (P,k,h):

    return P + h*k*P

h=.5
Pk1=np.arange(0,200+h,h)
t = np.arange(0,Pk1.size-1)
k=0.03067
Pk1[0]= 3.9

for i in  t:
    Pk1[i+1] = df(Pk1[i],k,h)

x= int (1/h)

t = np.arange(0,200+h,h)
plt.plot(t,Pk1,"r")

año = np.arange(1790,2000,10)
t = np.arange(0,210,10)
real = np.array([3.9,5.3,7.2,9.6,12,17,23,31,38,50,62,75,91,105,122,131,151,179,203,226,249])

Pt = 3.9*np.exp(0.03067*t)

plt.plot(t,real,"g")
plt.plot(t,Pt,"b")

_=plt.xticks(t, año, rotation='vertical')
plt.margins(0.05)
plt.xlabel("t")
plt.ylabel("P(t)")
```

```python

```

```python

```

# Modelo logístico de la población

Para ajustar el modelo de crecimiento exponencial de la población que tome en cuenta un entorno y recursos limitados, agregamos la hipótesis:

## Si la población es pequeña, la razón de crecimiento de la población es proporcional a su tamaño.

## Si la población es demasiado grande para ser soportada por su entorno y recursos. la  población disminuirá. Es decir, la razón de crecimiento es negativa.

Para este modelo usamos de nuevo

- $t$ = tiempo (variable inependiente).
- $P$ = población (variable dependiente)
- $k$ = constante de proporcionalidad (parámetro) entre la tasa de crecimiento de la población y el tamaño de ésta.

Sin embargo, nuestra hipótesis acerca de recursos limitados introduce otra cantidad, el tamaño de la población que corresponde a ser **"demasiado grande"**. Esta cantidad es un segundo parámetro, denotado por $N$, que llamamos la **"capacidad de soporte"** del entorno.

En términos de la capacidad de soporte, estamos suponiendo que $P(t)$ crece si $P(t)<N$. No obstante, si $P(t)>N$, suponemos que $P(t)$ está decreciendo.

Usando esta notación, podemos reescribir nuestras hipótesis como:

- $\frac{dP}{dt} \approx kP$ si $P$ es pequeña (primera hipótesis).

- Si $P>N,\  \frac{dP}{dt} <0 $ (segunda hipótesis).

Queremos también que el modelo sea **"algebraicamente simple"** o por lo menos tan simple como sea posible, por lo que tratamos de modificar el modelo exponencial lo menos posible.

Por ejemplo, podríamos intentar una expresión de la forma

$$\frac{dP}{dt} = k (algo) P $$

Queremos que el factor **"algo"** sea cercano a $1$ si $P$ es pequeña, pero si $P>N$, queremos que **"algo"** sea negativo. La expresión más simple que tienen estas propiedades es la función

$$ (algo) = \left(1 - \frac{P}{N} \right)$$

Note que esta expresión es igual a $1$ si $P=0$ y es negativa si $P>N$. Nuestro modelo es entonces

$$\frac{dP}{dt} = k \left(1 - \frac{P}{N} \right) P $$

Éste se llama el **modelo logístico de la población ** con velocidad de crecimiento $k$ y capacidad $N$ de soporte.

Se trata de otra ecuación diferencial de primer orden. Se dice que esta ecuación es **no lineal ** por que su lado derecho no es una función lineal de $P$ como lo era en el modelo de crecimiento exponencial.

```python
def df (P,k,h,N):
    return P + h*k*(1-P/N)*P

h=.01
N=203

Pk1=np.arange(0,200+h,h)
t = np.arange(0,Pk1.size-1)

k=0.03067

Pk1[0]= 3.9

for i in  t:
    Pk1[i+1] = df(Pk1[i],k,h,N)

x= int (100/h)
print (Pk1[x],t[x])

t = np.arange(0,200+h,h)
plt.plot(t,Pk1,"r",label="Modelo")

año = np.arange(1790,2000,10)
t = np.arange(0,210,10)
real = np.array([3.9,5.3,7.2,9.6,12,17,23,31,38,50,62,75,91,105,122,131,151,179,203,226,249])

Pt = 3.9*np.exp(0.03067*t)

plt.plot(t,real,"g",label="Real")

#plt.plot(t,Pt,"b")

_=plt.xticks(t, año, rotation='vertical')
plt.margins(0.05)
plt.xlabel("t")
plt.ylabel("P(t)")

plt.legend(loc = 'best', prop={'size': 14})
```

# Ecuaciones de Volterra: modelo presa depredador

Se trata de un sistema de dos ecuaciones diferenciales de primer orden, acopladas, autónomas y no lineales:

$$ \displaystyle \frac{dx}{dt} = \alpha x - \beta x y $$

$$ \displaystyle \frac{dy}{dt} = -\gamma y + \delta y x $$

donde x es el número de presas (cebras en nuestro caso) e y es el número de depredadores (leones). Los parámetros son constantes positivas que representan:

- $\alpha$: tasa de crecimiento de las presas.
- $\beta$: éxito en la caza del depredador.
- $\gamma$: tasa de decrecimiento de los depredadores.
- $\delta$: éxito en la caza y cuánto alimenta cazar una presa al depredador.

Aunque este modelo puede parecer relativamente simple, ha sido la base de algunos interesantes estudios ecológicos. En particular, Volterra y D'Ancona usaron con éxito el modelo para explicar el incremento en la población de tiburones en el mar Mediterráneo durante la Primera Guerra Mundial, cuando la pesca de las especies **"presa"** decreció. El modelo puede también usarse como base para el estudio de los efectos de los pesticidas en la población de insectos depredadores e insectos presas.

```python
from scipy.integrate import odeint

def df_dt(x, t, a, b, c, d):
    """Función del sistema en forma canónica"""
    dx = a * x[0] - b * x[0] * x[1]
    dy = - c * x[1] + d * x[0] * x[1]
    return np.array([dx, dy])

# Parámetros
a = 0.1
b = 0.02
c = 0.3
d = 0.01

# Condiciones iniciales
x0 = 40   # Presas
y0 = 9    # Depredadores
conds_iniciales = np.array([x0, y0])

# Condiciones para integración
tf = 200
N = 800
t = np.linspace(0, tf, N)

solucion = odeint(df_dt, conds_iniciales, t, args=(a, b, c, d))
#plt.plot(t, solucion[:, 0], label='presa')
#plt.plot(t, solucion[:, 1], label='depredador')
plt.plot(solucion[:, 0], solucion[:, 1])
plt.xlabel('Presas', fontsize = 14)
plt.ylabel('Depredadores', fontsize = 14)
```

## Ecuaciones de Volterra: modelo presa depredador

$$
\begin{align}
\dot{x}_1 &=-x_1 +x_1x_2,\\
\dot{x}_2 &=x_2 -x_1x_2
\end{align}
$$

```python
def Voltera (x,t):

    return [-x[0] + x[0]*x[1],x[1]-x[0]*x[1]]

t = np.arange(0,200,.1)
x = odeint(Voltera,[1.2,1.2],t)
plt.plot(x[:,0],x[:,1])
```

```python
t = np.arange(0,3,.1)

x1 = np.arange(-2,2,.2)
x2=x1

N= len(x1)

for xr in x1:
    for xc in x2:
        x0 = [xr,xc]

        x = odeint(Voltera,x0,t)
        plt.plot(x[:,0],x[:,1])
        plt.hold(True)

plt.axis([-5,5,-5,5])
```

# Teorema 1

La ecuación diferencial lineal de orden $n$

$$x^{(n)} + a_1x^{(n-1)}+ a_2x^{(n-2)} + \ldots + a_{n-1}x' + a_nx =f(t) $$

puede expresarse como un sistema de $n$ ecuaciónes lineales de primer orden.

*Demostración* Defina $x_1=x ,x_2=x',x_3=x'', \ldots, x_n=x^{n-1}$ Entonces tenemos

$$ \dot{x}_1 = x_2$$
$$ \dot{x}_2 = x_3$$
$$ \vdots$$
$$ \dot{x}_{n-1} = x_n$$
$$ \dot{x}_n = -a_nx_1 -a_{n-1}x_2 - \ldots - a_1x_n +f$$

# Ejemplo

$$t^3x''' + 4t^2x'' - 8tx' + 8x =0  $$

$$x(0)=3,x'(0)=-6,x''(0)=14 $$.

Definiendo $x_1=x,x_2=x',x_3=x''$, obtenemos el sistema

$$\dot{x}_1 = x_2 $$
$$\dot{x}_2 = x_3 $$
$$\dot{x}_3 =  -\frac{8}{t^3}x_1 +\frac{8}{t^2}x_2 - \frac{4}{t}x_3  $$

```python
def df_dt2(x, t):
    """Función del sistema en forma canónica"""
    dx1 = x[1];
    dx2 = x[2]
    dx3 = - 8/t**3 *x[0] + 8/t**2 *x[1] - 4/t *x[2]
    return np.array([dx1, dx2 , dx3])

# Condiciones iniciales
x0 = 3
x1 = -6
x2 = 14
conds_iniciales = np.array([x0,x1,x2])

# Condiciones para integración
tf = 10
N = 800
t = np.linspace(0.1, tf, N)

solucion = odeint(df_dt2, conds_iniciales, t)
plt.plot(t, solucion[:, 0])
plt.plot(t, solucion[:, 1])
plt.plot(t, solucion[:, 2])
#plt.plot(solucion[:, 0], solucion[:, 1])
```

# Un Modelo de ahorro

Supongamos que depositamos $ \$5000 $ en una cuenta de ahorro con interés incrementado a una tasa de $ 5\% $ compuesto en forma continua. Si $ A(t)$ denota la cantidad de dinero en la cuenta en el tiempo $ t$,

entonces la ecuación diferencial para $A$ es

$$ \frac{dA}{dt} = 0.05A $$

Como vimos en la sección previa, la solución general para esta ecuación es la función exponencial

$$A(t) = ce^{0.05t} $$

donde $c = A(0).$ Entonces

$$ A(t) = 5000e^{0.05t}$$
es nuestra solución particular.

Suponiendo que la tasa de interés nunca cambian, después de 10 años tendremos

$$ A(10) = 5000 e^{0.5} \approx  8244  $$

Pesos en esta cuenta.

# Simula la solución analitica de la ecuación diferencial

```python
def fuc(t):
    return 5000*np.exp(0.05*t)

t = np.arange(0,11,1)

A=fuc(t)

plt.plot(t,A)
```

```python
fuc(10)
```

    8243.6063535006415

# Simula usando el Método de Euler

$$ \frac{dA}{dt} = 0.05A $$

$$ A(0) = 5000$$

$$ \frac{A_{k+1} - A_k}{T} = 0.05A $$
$$ A_{k+1} =A_k + T (0.05A) $$

```python
T=1;
A = np.arange(0,11,T)
K = np.arange(0,A.size-1,1)
A[0] = 5000

for k in K:
    A[k+1] = A[k] + T* (0.05*A[k])

t = np.arange(0,11,T)
plt.plot(t,A)
```

```python
A[int (10/T) ]
```

    8138

# Simula usando la funcion odeint de scipy

# $ \frac{dx}{dt} $

# $ \frac{dA}{dt} $

```python
from scipy.integrate import odeint

def dA_dt(A,t):
    dA =(0.05*A)
    return dA

A0 = 5000
T=1
t = np.arange(0,11,T)
solucion = odeint(dA_dt,A0,t)

plt.plot(t,solucion)
print (solucion[int (10/T)])
```

# Compara los tres resultados para A(10) = ?

```python

```


# DC 2 layer foundation

- [**Questions**](https://www.dropbox.com/s/uizpgz3eyt3urim/DC-2-layer-foundation.pdf?dl=0)

In this notebook, we use widgets to explore the physical principals governing DC resistivity. For a half-space and a 2-layer resistivity model, we will learn about the behavior of the *currents*, *electric fields* and *electric potentials* that are produced when electric currents are injected into the Earth.

In the DC resistivity experiment, we measure the different in electric potential between two locations; also known as a *voltage*. Using these voltage measurements, we can get information about the resistivity of the Earth. Here, be begin to understand how these measurements depend on the electrode locations.

**DC resistivity over a 2 layered Earth**

## Background: Computing Apparent Resistivity

In practice we cannot measure the electric potentials everywhere. We are instead limited to a set of locations where we have placed potential electrodes. For each source (current electrode pair) many potential differences are measured between M and N electrode pairs to characterize the overall distribution of potentials. The widget below allows you to visualize the potentials, electric fields, and current densities from a dipole source in a simple model with 2 layers. For different electrode configurations you can measure the potential differences and see the calculated apparent resistivities.

In a uniform halfspace the potential differences can be computed by summing up the potentials at each measurement point from the different current sources based on the following equations:

\begin{align}
    V_M = \frac{\rho I}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} \right] \\
    V_N = \frac{\rho I}{2 \pi} \left[ \frac{1}{AN} - \frac{1}{NB} \right]
\end{align}
where $AM$, $MB$, $AN$, and $NB$ are the distances between the corresponding electrodes.

The potential difference $\Delta V_{MN}$ in a dipole-dipole survey can therefore be expressed as follows,

\begin{equation}
    \Delta V_{MN} = V_M - V_N = \rho I \underbrace{\frac{1}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} - \frac{1}{AN} + \frac{1}{NB} \right]}_{G}
\end{equation}

and the resistivity of the halfspace $\rho$ is equal to,

$$
    \rho = \frac{\Delta V_{MN}}{IG}
$$

In this equation $G$ is often referred to as the geometric factor.

In the case where we are not in a uniform halfspace the above equation is used to compute the apparent resistivity ($\rho_a$) which is the resistivity of the uniform halfspace which best reproduces the measured potential difference.

In the top plot the location of the A electrode is marked by the red +, the B electrode is marked by the blue -, and the M/N potential electrodes are marked by the black dots. The $V_M$ and $V_N$ potentials are printed just above and to the right of the black dots. The calculted apparent resistivity is shown in the grey box to the right. The bottom plot can show the resistivity model, the electric fields (e), potentials, or current densities (j) depending on which toggle button is selected. Some patience may be required for the plots to update after parameters have been changed.

## Import Packages

```python
from utils import DCLayers
from IPython.display import display
%matplotlib inline
from matplotlib import rcParams
rcParams['font.size'] = 14
```

## User Defined Parameters for the App

Below are the parameters that can be changed by the user:

 - **A**: (+) Current electrode location
 - **B**: (-) Current electrode location
 - **M**: (+) Potential electrode location
 - **N**: (-) Potential electrode location
 - **$\rho_1$**: Resistivity of the first layer
 - **$\rho_2$**: Resistivity of the second layer
 - **h**: Thickness of the first layer
 - **Plot**: Choice of 2D plot (Model, Potential, Electric field, Currents)

## Run the App

```python
out = DCLayers.plot_layer_potentials_app()
display(out)
```

    MyApp(children=(ToggleButtons(description='survey', options=('Dipole-Dipole', 'Dipole-Pole', 'Pole-Dipole', 'P…

```python

```


#   Copyright statement:
#### The initial code was provided by Dr. Sandoval (Assitant professor of Electrical Engineering, New Mexico State University) as a starting point for project. https://web.nmsu.edu/~spsandov/
#### The algorithm in current form is developed by Sina Zamen.

#   Objective:
## In this problem, for finding minimum value of a Rosenbrock's cost function, “Gradient Decent” method is implemented.

### Loading Required Modules:

```python
import math
import numpy as np
import sympy as sym
from PIL import Image
from numpy import array
from matplotlib import style
from numpy.linalg import inv
import matplotlib.pyplot as plt
style.use('ggplot')
```

### Setting up countour map:

```python
HX ,HY  = 50,50 #number of x,y  points for countour
xmin,xmax = -15,15
ymin,ymax = -12,12
x1 = np.linspace(xmin,xmax,HX)
x2 = np.linspace(ymin,ymax,HY)
X1,X2 = np.meshgrid(x1,x2) # genertate mesh grid
w1=sym.Symbol('w1') # define symbols
w2=sym.Symbol('w2')
```

```python
bowl = 'bowl'
case2= ''
case = bowl #case select
if(case==bowl):
    j=((1-w1)**2 + 100*(w2-w1**2)**2)# define equation
elif(case==case2):
    #define other surfaces here
    pass
else:
    print('case not recognized')
```

```python
#compute gradient
j_grad1=sym.diff(j,w1)
j_grad2=sym.diff(j,w2)
#compute hessian
hess11=sym.diff(j_grad1,w1)
hess12=sym.diff(j_grad1,w2)
hess21=sym.diff(j_grad2,w1)
hess22=sym.diff(j_grad2,w2)

#generate contour map
ConMap=np.zeros((HX,HY))
for i in range(HX):
    for k in range(HY):
        ConMap[i,k]=j.subs({w1:x1[i],w2:x2[k]})

w=[-0.5,1] # starting point
wStar=[1,1] # ending point
ew=array([7])
jw=array([100])
eta = 0.004 # learning rate
lim=200 # number of iterations
count= 0
line=[]
h = ew.astype(int)
h = ew[-1:].astype(int)
while (h > 0.05):
    h = ew[-1:].astype(int)
    #compute gradient matrix and hessian matrix
    g= np.array([float(j_grad1.subs({w1:w[0],w2:w[1]})),float(j_grad2.subs({w1:w[0],w2:w[1]}))])
    H= np.array([[float(hess11.subs({w1:w[0],w2:w[1]})),float(hess12.subs({w1:w[0],w2:w[1]}))],                 [float(hess21.subs({w1:w[0],w2:w[1]})),float(hess22.subs({w1:w[0],w2:w[1]}))]])

    wnew =  w-eta*g
    #loop check
    if( count>lim ):
        print('Count Break')
        break
    elif(np.isnan(g).any()):
        print('nan break')
        break
    else:
        count=count +1
        wprev=w.copy()

        line.append(w)
        w=wnew.copy()

        ew1 = (np.linalg.norm(w-wStar))
        jw1 = (j.subs({w1:w[0],w2:w[1]}))
        ew = np.append(ew, ew1)
        jw = np.append(jw, jw1)
line=np.array(line)
```

```python
print("Number of Iterations:")
print(count)
```

    Number of Iterations:
    3

```python
plt.figure(figsize=(10,10))
plt.contour(X1,X2,ConMap)
plt.plot(line[:,1],line[:,0])
plt.scatter(line[:,1],line[:,0])
plt.plot(wStar[1],wStar[0],'go')
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('J(w) vs iterations')
plt.ylabel('J(w)')
plt.xlabel('iteration')
plt.plot(range(len(jw)),jw)
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('E(w) vs iterations')
plt.ylabel('E(w)')
plt.xlabel('iteration')
plt.plot(range(len(ew)),ew)
plt.show()
```

Gradient Decent/Gradient Decent With a Rosenbrock's Cost Function.ipynb


Introduction
--------------

In these pages, I have tried to distill and illustrate the keys concepts needed in statistical signal processing, and in this section, we will cover the most fundamental statistical result that underpins statistical signal processing. The last sections on [conditional expectation](http://python-for-signal-processing.blogspot.com/2012/11/conditional-expectation-and-mean.html) and [projection](http://python-for-signal-processing.blogspot.com/2012/11/the-projection-concept.html) are prerequisites for what follows. Please review those before continuing.

## Inner Product for Random Variables

From our previous work on projection for vectors in $\mathbb{R}^n$, we have a good geometric grasp on how projection is related to minimum mean squared error (MMSE). It turns out by one abstract step, we can carry all of our geometric interpretations to the space of random variables.

For example, we previously noted that at the point of projection, we had

$$ ( \mathbf{y} - \mathbf{v}_{opt} )^T \mathbf{v} = 0$$

which by noting the inner product slightly more abstractly as $\langle\mathbf{x},\mathbf{y} \rangle = \mathbf{x}^T \mathbf{y}$, we can express as

$$\langle \mathbf{y} - \mathbf{v}_{opt},\mathbf{v} \rangle = 0  $$

and, in fact, by defining the inner product for the random variables $X$ and $Y$ as

$$ \langle X,Y \rangle = \mathbb{E}(X Y)$$

we remarkably have the same relationship:

$$\langle X-h_{opt}(Y),Y \rangle = 0  $$

which holds not for vectors in $\mathbb{R}^n$, but for random variables $X$ and $Y$ and functions of those random variables. Exactly why this is true is technical, but it turns out that one can build up the **entire theory of probability** this way (see Nelson,1987), by using the expectation as an inner product.

Furthermore, by abstracting out the inner product concept, we have drawn a clean line between MMSE optimization problems, geometry, and random variables. That's  a lot of mileage to get a out of an abstraction and it is key to everything we pursue in statistical signal processing because now we can shift between these interpretations to address real problems. Soon, we'll see how to do this with some examples, but first we will collect one staggering result that flows naturally from this abstraction.

## Conditional Expectation as Projection

[Previously](http://python-for-signal-processing.blogspot.com/2012/11/conditional-expectation-and-mean.html), we noted that the conditional expectation is the minimum mean squared error (MMSE) solution to the following problem:

$$ \min_h \int_{\mathbb{R}} (x - h(y) )^2 dx $$

with the minimizing $h_{opt}(Y) $ as

$$ h_{opt}(Y) = \mathbb{E}(X|Y) $$

which is another way of saying that among all possible functions $h(Y)$, the one that minimizes the MSE is $ \mathbb{E}(X|Y)$ (see appendix for a quick proof). From our discussion on [projection](http://python-for-signal-processing.blogspot.com/2012/11/the-projection-concept.html), we noted that these MMSE solutions can be thought of as projections onto a subspace that characterizes $Y$. For example, we previously noted that at the point of projection, we have

$$\langle X-h_{opt}(Y),Y \rangle = 0 $$

but since we know that the MMSE solution

$$ h_{opt}(Y) = \mathbb{E}(X|Y) $$

we have by direct substitution,

$$ \mathbb{E}( X-\mathbb{E}(X|Y), Y) = 0$$

That last step seems pretty innocuous, but it is the step that ties MMSE to conditional expectation to the inner project abstraction, and in so doing, reveals the conditional expectation to be a projection operator for random variables. Before we develop this further, let's grab some quick dividends:

From the previous equation, by linearity of the expectation, we may obtain,

$$  \mathbb{E}(X Y) =  \mathbb{E}(Y \mathbb{E}(X|Y))$$

which we could have found by using the formal definition of conditional expectation,

\begin{equation}
\mathbb{E}(X|Y) = \int_{\mathbb{R}^2} x \frac{f_{X,Y}(x,y)}{f_Y(y)} dx dy
\end{equation}

and direct integration,

$$ \mathbb{E}(Y \mathbb{E}(X|Y))= \int_{\mathbb{R}} y \int_{\mathbb{R}} x \frac{f_{X,Y}(x,y)}{f_Y(y)}  f_Y(y) dx dy =\int_{\mathbb{R}^2} x y f_{X,Y}(x,y) dx dy =\mathbb{E}( X Y) $$

which is good to know, but not very geometrically intuitive. And this lack of geometric intuition makes it hard to apply these concepts and keep track of these relationships.

We can keep pursuing this analogy and obtain the length of the error term as

$$ \langle X-h_{opt}(Y),X-h_{opt}(Y)  \rangle = \langle X,X  \rangle - \langle h_{opt}(Y),h_{opt}(Y)  \rangle  $$

and then by substituting all the notation we obtain

\begin{equation}
\mathbb{E}(X-  \mathbb{E}(X|Y))^2 = \mathbb{E}(X)^2 - \mathbb{E}(\mathbb{E}(X|Y) )^2
\end{equation}

which would be tough to compute by direct integration.

We recognize that $\mathbb{E}(X|Y)$ *is* in fact **a projection operator**. Recall previously that we noted that the projection operator is idempotent, which means that once we project something onto a subspace, further projections essentially do nothing. Well, in the space of random variables, $\mathbb{E}(X|\cdot$) is the idempotent projection as we can show by noting that

$$ h_{opt} = \mathbb{E}(X|Y)$$

is purely a function of $Y$, so that

$$ \mathbb{E}(h_{opt}(Y)|Y) = h_{opt}(Y) $$

since $Y$ is fixed and this  is the statement of idempotency. Thus, conditional expectation is the corresponding projection operator in this space of random variables. With this happy fact, we can continue to carry over our geometric interpretations of projections for vectors ($\mathbf{v}$) into random variables ( $X$ ).

Now that we have just stuffed our toolbox, let's consider some example conditional expectations obtained by using brute force to find the optimal MMSE function $h_{opt}$ as well as by using the definition of the conditional expectation.

Example
---------

Suppose we have a random variable, $X$, then what constant is closest to $X$ in the mean-squared-sense (MSE)? In other words, which $c$ minimizes the following:

$$ J = \mathbb{E}( X - c )^2 $$

we can work this out as

$$ \mathbb{E}( X - c )^2 = \mathbb{E}(c^2 - 2 c X + X^2) = c^2-2 c \mathbb{E}(X) + \mathbb{E}(X^2) $$

and then take the first derivative with respect to $c$ and solve:

$$ c_{opt}=\mathbb{E}(X) $$

Remember that $X$ can take on all kinds of values, but this says that the closest number to $X$ in the MSE sense is $\mathbb{E}(X)$.

Coming at this same problem using our inner product, we know that at the point of projection

$$ \mathbb{E}((X-c_{opt}) 1) = 0$$

where the $1$ represents the space of constants (i.e. $c \cdot 1 $)  we are projecting on. This, by linearity of the expectation, gives

$$ c_{opt}=\mathbb{E}(X) $$

Because $\mathbb{E}(X|Y)$ is the projection operator, with $Y=\Omega$ (the entire underlying probability space), we have, using the definintion of conditional expectation:

$$  \mathbb{E}(X|Y=\Omega) = \mathbb{E}(X)  $$

Thus, we just worked the same problem three ways (optimization, inner product, projection).

Example
-----------

Let's consider the following example with probability density $f_{X,Y}= x + y $ where $(x,y) \in [0,1]^2$ and compute the conditional expectation straight from the definition:

$$ \mathbb{ E}(X|Y) = \int_0^1 x \frac{f_{X,Y}(x,y)}{f_Y(y)} dx=  \int_0^1 x \frac{x+y}{y+1/2} dx =\frac{3 y + 2}{6 y + 3} $$

That was pretty easy because the density function was so simple. Now, let's do it the hard way by going directly for the MMSE solution $h(Y)$. Then,

$$ \min_h \int_0^1\int_0^1 (x - h(y) )^2 f_{X,Y}(x,y)dx dy = \min_h \int_0^1  y h^2 {\left (y \right )} - y h{\left (y \right )} + \frac{1}{3} y + \frac{1}{2} h^{2}{\left (y \right )} - \frac{2}{3} h{\left (y \right )} + \frac{1}{4} dy $$

Now we have to find a function $h$ that is going to minimize this. Solving for a function, as opposed to solving for a number, is generally very, very hard, but because we are integrating over a finite interval, we can use the Euler-Lagrange method from variational calculus to take the derivative of the integrand with respect to the function $h(y)$ and set it to zero. Euler-Lagrange methods will be the topic of a later section, but for now we just want the result, namely,

$$ 2 y h{\left (y \right )} - y + h{\left (y \right )} - \frac{2}{3} =0 $$

Solving this gives

$$ h_{opt}(y)= \frac{3 y + 2}{6 y + 3} $$

Finally, we can try solving this using our inner product as

$$ \mathbb{E}( (X - h(Y)) Y ) = 0$$

Writing this out gives,

$$ \int_0^1 \int_0^1  (x-h(y))(x+y) dx dy = \int_0^1 \frac{y\,\left(2 + 3\,y -  3\,\left( 1 + 2\,y \right) \,h(y)\right) }{6} dy = 0$$

and if this is zero everywhere, then the integrand must be zero,

$$ 2 y + 3 y^2 - 3 y h(y) - 6 y^2 h(y)=0 $$

and solving this for $h(y)$ gives the same solution:

$$ h_{opt}(y)= \frac{3 y + 2}{6 y + 3} $$

Thus, doing it by the definition, optimization, or inner product gives us the same answer; but, in general, no method is necessarily easier because they both involve potentially difficult or impossible integration, optimization, or functional equation solving. The point is that now that we have a deep toolbox, we can pick and choose which tools we want to apply for different problems.

Before we leave this example, let's use `sympy` to verify the length of the error function we found earlier:

\begin{equation}
\mathbb{E}(X-  \mathbb{E}(X|Y))^2 = \mathbb{E}(X)^2 - \mathbb{E}(\mathbb{E}(X|Y) )^2
\end{equation}

that is based on the Pythagorean theorem.

```
from sympy.abc import y,x
from sympy import integrate, simplify

fxy = x + y                 # joint density
fy = integrate(fxy,(x,0,1)) # marginal density
fx = integrate(fxy,(y,0,1)) # marginal density

h = (3*y+2)/(6*y+3) # conditional expectation
LHS=integrate((x - h)**2 *fxy, (x,0,1),(y,0,1)) # from the definition
RHS=integrate( (x)**2 *fx, (x,0,1)) - integrate( (h)**2 *fy, (y,0,1)) # using Pythagorean theorem
print simplify(LHS-RHS)==0
```

    True

Summary
--------

In this section, we have pulled together all the projection and least-squares optimization ideas from the previous posts to draw a clean line between our geometric notions of projection from vectors in $\mathbb{R}^n$ to general random variables. This resulted in the remarkable realization that the conditional expectation is in fact a projection operator for random variables. The key idea is that because we have these relationships, we can approach difficult problems in multiple ways, depending on which way is more intuitive or tractable in a particular situation. In these pages, we will again and again come back to these intuitions because they form the backbone of statistical signal processing.

In the next section, we will have a lot of fun with these ideas working out some examples that are usually solved using more general tools from measure theory.

Note that the book by Mikosch (1998) has some excellent sections covering much of this material in more detail. Mikosch has a very geometric view of the material as well.

Appendix
---------

#### Proof of MMSE of Conditional expectation by Optimization

$$ \min_h \int_\mathbb{R^2} | X - h(Y) |^2 f_{x,y}(X,Y) dx dy $$

$$ \min_h \int_\mathbb{R^2} | X |^2 f_{x,y}(X,Y) dx dy + \int_\mathbb{R^2} | h(Y) |^2 f_{x,y}(X,Y) dx dy - \int_\mathbb{R^2} 2 X h(Y) f_{x,y}(X,Y) dx dy $$

Now, we want to maximize the following:

$$ \max_h \int_\mathbb{R^2}  X h(Y) f_{x,y}(X,Y) dx dy  $$

Breaking up the integral using the definition of conditional expectation

$$ \max_h \int_\mathbb{R}   \left(\int_\mathbb{R} X  f_{x|y}(X|Y) dx \right)h(Y) f_Y(Y)   dy  $$

$$ \max_h \int_\mathbb{R} \mathbb{E}(X|Y) h(Y)f_Y(Y)   dy  $$

From properties of the Cauchy-Schwarz inequality, we know that the maximum happens when $h_{opt}(Y) = \mathbb{E}(X|Y)$, so we have found the optimal $h(Y)$ function as :

$$ h_{opt}(Y) = \mathbb{E}(X|Y)$$

which shows that the optimal function is the conditional expectation.

### References

This post was created using the [nbconvert](https://github.com/ipython/nbconvert) utility from the source [IPython Notebook](www.ipython.org) which is available for [download](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Conditional_Expectation_Projection.ipynb) from the main github [site](https://github.com/unpingco/Python-for-Signal-Processing) for this blog.

### Bibliography

* Nelson, Edward. Radically Elementary Probability Theory.(AM-117). Vol. 117. Princeton University Press, 1987.

* Mikosch, Thomas. Elementary stochastic calculus with finance in view. Vol. 6. World Scientific Publishing Company Incorporated, 1998.


```python
import scipy.io as sio
import numpy as np
import math
import matplotlib.pyplot as plt
from scipy.sparse import csc_matrix
data = sio.loadmat('heightWeight.mat')
hwdata = data['heightWeightData']
y = hwdata[:,0]
x = hwdata[:,1:]
maleNdx = np.where(y==1)
femaleNdx = np.where(y==2)
```

### Gaussian Models
The probability density function for an Multivariate Normal in D dimensions is defined as follows,
$$\mathcal{N}(\mathbf{x}|\mathbf{\mu},\mathbf{\Sigma})\triangleq \frac{1}{(2\pi)^{\frac{D}{2}}|\mathbf{\Sigma}|^{\frac{1}{2}}}\exp\bigg[-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^\top\mathbf{\Sigma}^{-1}(\mathbf{x}-\mathbf{\mu})\bigg]$$

The expression inside the exponent is the **Mahalanobis distance** between a data vector $\mathbf{x}$ and the mean vector $\mathbf{\mu}$. To gain a better understanding of this quantity by performing an **eigendecomposition** of $\mathbf{\Sigma}=\mathbf{U\Lambda U^\top}$, where $\mathbf{U}$ is an orthonormal matrix of eigenvectors satisfying $\mathbf{U^\top U=I}$, and $\mathbf{\Lambda}$ is a diagonal matrix of eigenvalues. That is,

$$\mathbf{\Sigma}^{-1}=\mathbf{U^{-\top}\Lambda^{-1}U^{-1}}=\mathbf{U\Sigma^{-1}U^\top}=\sum^D_{i=1}\frac{1}{\lambda_i}\mathbf{u_iu_i^\top}$$
where $\mathbf{u_i}$ is the $i^{th}$ column of $\mathbf{U}$, containing the $i^{th}$ eigenvector. Hence, the Mahalanobis distance can be rewritten as,

\begin{align}
(\mathbf{x}-\mathbf{\mu})^\top\mathbf{\Sigma}^{-1}(\mathbf{x}-\mathbf{\mu})&= (\mathbf{x}-\mathbf{\mu})^\top\sum^D_{i=1}\frac{1}{\lambda_i}\mathbf{u_iu_i^\top}(\mathbf{x}-\mathbf{\mu}) \\
&= \sum^D_{i=1}\frac{1}{\lambda}(\mathbf{x}-\mathbf{\mu})^\top \mathbf{u_i}\mathbf{u_i}^\top(\mathbf{x}-\mathbf{\mu}) = \sum^D_{i=1}\frac{y_i^2}{\lambda_i}
\end{align}
where $y_i\triangleq \mathbf{u_i}^\top(\mathbf{x}-\mathbf{\mu})$. This matches the elliptical equation in 2d,

$$\frac{y_1^2}{\lambda_1}+\frac{y_2^2}{\lambda_2}=1$$

#### MLE for an MVN
The parameters are given by,

\begin{align}
\mathbf{\mu} &= \frac{1}{N}\sum^N_{i=1}\mathbf{x}_i \triangleq \bar{\mathbf{x}} \\
\mathbf{\Sigma} &= \frac{1}{N}\sum^N_{i=1}(\mathbf{x}_i-\bar{\mathbf{x}})(\mathbf{x}_i-\bar{\mathbf{x}})^\top = \frac{1}{N}\sum^N_{i=1}\mathbf{x_i}\mathbf{x_i}^\top - \bar{\mathbf{x}}\bar{\mathbf{x}}^\top
\end{align}

```python
mx = x[maleNdx,:]
mm = np.mean(mx,axis=1); mv = np.cov(np.transpose(mx.squeeze()))
fx = x[femaleNdx,:]
fm = np.mean(fx,axis=1); fv = np.cov(fx.squeeze().T)
```

```python
def gaussPlot2d(mu,Sigma):
    n = 100; mu = mu.T; k = np.sqrt(6)
    d,u = np.linalg.eig(Sigma)
    t = np.linspace(0,2*math.pi,n)
    xy = np.vstack((np.cos(t),np.sin(t)))
    w = np.matmul((k*np.matmul(u,np.sqrt(np.diag(d)))),xy)
    z = np.tile(mu,(1,n))+w
    plt.plot(z[0,:],z[1,:])
```

```python
plt.scatter(x[maleNdx,0],x[maleNdx,1],c='b')
gaussPlot2d(mm,mv)
plt.scatter(x[femaleNdx,0],x[femaleNdx,1],c='r')
gaussPlot2d(fm,fv)
```

### Proof
The log-likelihood is

$$\mathcal{l}(\mathbf{\mu},\mathbf{\Sigma}) = \log p(\mathcal{D}|\mathbf{\mu},\mathbf{\Sigma}) = \frac{N}{2}\log|\mathbf{\Lambda}|-\frac{1}{2}\sum^N_{i=1}(\mathbf{x_i}-\mathbf{\mu})^\top \mathbf{\Lambda}(\mathbf{x_i}-\mathbf{\mu})$$
where $\mathbf{\Lambda}=\mathbf{\Sigma}^{-1}$ is the precision matrix.

Using the substitution $\mathbf{y_i}=\mathbf{x_i}-\mathbf{\mu}$,

\begin{align}
\frac{\partial}{\partial \mathbf{\mu}}(\mathbf{x_i}-\mathbf{\mu})^\top\mathbf{\Sigma}^{-1}(\mathbf{x_i}-\mathbf{\mu})=\frac{\partial}{\partial\mathbf{\mu}}\mathbf{y_i}^\top\mathbf{\Sigma}^{-1}\mathbf{y_i}
\end{align}

With $\frac{\partial}{\partial \mathbf{a}}(\mathbf{a}^\top\mathbf{A}\mathbf{a})=(\mathbf{A}+\mathbf{A}^\top)\mathbf{a}$, and chain rule $\frac{\partial}{\partial \mathbf{\mu}}=\frac{\partial}{\partial\mathbf{y_i}}\frac{\partial\mathbf{y_i}}{\partial\mathbf{\mu}}$,

$$\frac{\partial}{\partial\mathbf{\mu}}\mathbf{y_i}^\top\mathbf{\Sigma^{-1}}\mathbf{y_i} = \frac{\partial \mathbf{y_i}}{\partial\mathbf{\mu}}\frac{\partial}{\partial \mathbf{y_i}}\mathbf{y_i}^\top\mathbf{\Sigma}^{-1}\mathbf{y_i}=-(\mathbf{\Sigma}^{-1}+\mathbf{\Sigma}^{-\top})\mathbf{y_i}=-2\mathbf{\Sigma}^{-1}\mathbf{y_i}$$

Hence

\begin{align}
\frac{\partial}{\partial\mathbf{\mu}}\mathcal{l}(\mathbf{\mu},\mathbf{\Sigma})&=-\frac{1}{2}\sum^N_{i=1}-2\mathbf{\Sigma}^{-1}(\mathbf{x_i}-\mathbf{\mu})=\mathbf{\Sigma}^{-1}\sum^N_{i=1}(\mathbf{x_i}-\mathbf{\mu})=0 \\
\hat{\mathbf{\mu}}\sum^N_{i=1} &= N\hat{\mathbf{\mu}} = \sum^N_{i=1}\mathbf{x_i} \Longrightarrow\hat{\mu} = \frac{1}{N}\sum^N_{i=1}\mathbf{x_i}=\mathbf{\bar{x}}
\end{align}

Given the trace-trick $\mathbf{x^\top Ax}=Tr(\mathbf{x^\top Ax})=Tr(\mathbf{xx^\top A})=Tr(\mathbf{Axx^\top})$,

\begin{align}
\mathcal{l}(\mathbf{\Lambda})&=\frac{N}{2}\log|\mathbf{\Lambda}|-\frac{1}{2}\sum_i Tr[(\mathbf{x_i}-\mathbf{\mu})(\mathbf{x_i}-\mathbf{\mu})^\top\mathbf{\Lambda}] \\
&= \frac{N}{2}\log|\mathbf{\Lambda}|-\frac{1}{2}Tr|\mathbf{S_\mu}\mathbf{\Lambda}|
\end{align}
where $$\mathbf{S_\mu} \triangleq \sum^N_{i=1}(\mathbf{x_i}-\mathbf{\mu})(\mathbf{x_i}-\mathbf{\mu})^\top$$ is the scatter matrix centered on $\mathbf{\mu}$.

Given $\frac{\partial}{\partial\mathbf{A}}\log|\mathbf{A}| = \mathbf{A}^{-\top}$ and $\frac{\partial}{\partial\mathbf{A}}Tr(\mathbf{BA})=\mathbf{B}^\top$, taking derivatives of this expression with respect to $\mathbf{\Lambda}$ yields,

\begin{align}
\frac{\partial\mathcal{l}(\mathbf{\Lambda})}{\partial\mathbf{\Lambda}}&=\frac{N}{2}\frac{\partial}{\partial\mathbf{\Lambda}}\log|\mathbf{\Lambda}|-\frac{1}{2}\frac{\partial}{\partial\mathbf{\Lambda}}Tr|\mathbf{S_\mu\Lambda}| = \frac{N}{2}\mathbf{\Lambda}^{-\top}-\frac{1}{2}\mathbf{S_\mu^\top} = 0 \\
\mathbf{\Lambda}^{-\top} &= \mathbf{\Lambda}^{-1} = \mathbf{\Sigma}=\frac{1}{N}\mathbf{S_\mu}
\end{align}

so
$$\mathbf{\hat{\Sigma}}=\frac{1}{N}\sum^N_{i=1}(\mathbf{x_i}-\mathbf{\mu})(\mathbf{x_i}-\mathbf{\mu})^\top$$

Machine Learning A Probabilistic Perspective/4GaussianModels/F4.2/.ipynb_checkpoints/4.2gaussHeightWeight-checkpoint.ipynb