# Non-linear least Squares Fitting in Ecology and Evolution

## Introduction and preliminaries

This chapter assumes that you have already seen the [NLLS lecture](https://github.com/mhasoba/TheMulQuaBio/blob/master/Lectures/NLLS/Pawar_NLLS.pdf) and/or its [recording](https://imperial.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=2aa5bbb4-e1d0-44a1-adea-a87000aab72f).

We will work with several NLLS examples here.

> *You should build a separate R scripts for each of these examples using the principles of good coding you learnt in the R week*. In particular, create separate `code`, `data`, `results` directories, and `setwd()` to `code`.
>
> For example, for the Allometry example below, create a new R script file called `Allometry_nlls.R` and save it to your `code` directory. Then build your script for tackling this particular example in that file.

You may work in RStudio or any other Code editor you prefer.

You will need the `nls.lm` R package, which you can install using the standard method (linux users, launch R in `sudo` mode first):

```r
> install.packages("minpack.lm")
```

Now, load the necessary packages, and clear all variables and graphic devices:

```R
rm(list = ls())
graphics.off()

library("minpack.lm") # for Levenberg-Marquardt nls fitting
library("ggplot2")
```

*You will need to repeat these commands at the start of every new nlls script file you develop below.*

### Why use the nls.lm package?

The standard NLLS function in R, cals `nls` uses a less robust algorithm called the Gauss-Newton algorithm. Therefore, `nls` will often fail to fit your model to the data if you start off at starting values for the parameters that are too far from what the optimal vaues would be, especially if the "parameter space" is weirdly shaped, i.e., the model has a mathematical form that makes it hard to find parameter combinations that minimize the NLLS. If this does not makes ense, don't worry about it- just go with `nls_LM` from the `nls.lm` package instead of `nls`! If you are really curious, try substituting `nls` for `nls_LM` in the examples below and compare the results.

## Allometric scaling

Let's start with a common and reasonably simple example from biology: [allometric scaling](https://en.wikipedia.org/wiki/Allometry). We will look at allometric scaling of body weight vs. total body length in dragonflies and damselfiles.

Allometric relationships take the form:

\begin{equation} \label{eq:allom}
y = a x^b
\end{equation}

where $x$ and $y$ are morphological measures (body length and body weight respectively, in our current example), the constant is the value of $y$ at body length $x = 1$ unit, and $b$ is the scaling "exponent". This is also called a power-law, because $y$ relates to $x$ through a simple power.

First create a function object for the power law model:

```R
powMod <- function(x, a, b) {
    return(a * x^b)
}
```

Now read in the [data](https://raw.githubusercontent.com/mhasoba/TheMulQuaBio/master/Data/GenomeSize.csv) (first click on link and use "Save as" or `Ctrl+S` to download it as a csv):

```R
MyData <- read.csv("../Data/GenomeSize.csv")

head(MyData)
```

Suborder	Family	Species	GenomeSize	GenomeSE	GenomeN	BodyWeight	TotalLength	HeadLength	ThoraxLength	AdbdomenLength	ForewingLength	HindwingLength	ForewingArea	HindwingArea	MorphologyN

Anisoptera          	Aeshnidae           	Aeshna canadensis   	2.20                	  NA                	1                   	0.159               	67.58               	6.83                	11.81               	48.94               	45.47               	45.40               	369.57              	483.61              	2

Anisoptera          	Aeshnidae           	Aeshna constricta   	1.76                	0.06                	4                   	0.228               	71.97               	6.84                	10.72               	54.41               	46.00               	45.48               	411.15              	517.38              	3

Anisoptera          	Aeshnidae           	Aeshna eremita      	1.85                	  NA                	1                   	0.312               	78.80               	6.27                	16.19               	56.33               	51.24               	49.47               	460.72              	574.33              	1

Anisoptera          	Aeshnidae           	Aeshna tuberculifera	1.78                	0.10                	2                   	0.218               	72.44               	6.62                	12.53               	53.29               	49.84               	48.82               	468.74              	591.42              	2

Anisoptera          	Aeshnidae           	Aeshna umbrosa      	2.00                	  NA                	1                   	0.207               	73.05               	4.92                	11.11               	57.03               	46.51               	45.97               	382.48              	481.44              	1

Anisoptera          	Aeshnidae           	Aeshna verticalis   	1.59                	  NA                	1                   	0.220               	66.25               	6.48                	11.64               	48.13               	45.91               	44.91               	400.40              	486.97              	1

[Anisoptera](https://en.wikipedia.org/wiki/Dragonfly) are dragonflies, and [Zygoptera](https://en.wikipedia.org/wiki/Damselfly) are Damselflies. The variables of interest are `BodyWeight` and `TotalLength`.  Let's use the dragonflies data subset.

So subset the data accordingly and remove NAs:

```R
Data2Fit <- subset(MyData,Suborder == "Anisoptera")

Data2Fit <- Data2Fit[!is.na(Data2Fit$TotalLength),] # remove NA's
```

Plot it:

```R
plot(Data2Fit$TotalLength, Data2Fit$BodyWeight)
```

```R
ggplot(Data2Fit, aes(x = TotalLength, y = BodyWeight)) + geom_point() # or using ggplot!
```

Now fit the model to the data using NLLS:

```R
PowFit <- nlsLM(BodyWeight ~ powMod(TotalLength, a, b), data = Data2Fit, start = list(a = .1, b = .1))
```

We can use `summary()` just like we would for a `lm()` fit object.

```R
summary(PowFit)
```

    Formula: BodyWeight ~ powMod(TotalLength, a, b)

    Parameters:
       Estimate Std. Error t value Pr(>|t|)
    a 3.941e-06  2.234e-06   1.764    0.083 .
    b 2.585e+00  1.348e-01  19.174   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.02807 on 58 degrees of freedom

    Number of iterations to convergence: 39
    Achieved convergence tolerance: 1.49e-08

Most of the output is analogous to the output of an `lm()`. However, further statistucal inference here cannot be done using Analysis of Variance (ANOVA) (), because the mdoel is not a Linear Model. Try `anova(PowFit)`, and see what happens. The `Number of iterations to convergence: 39`, and `Achieved convergence tolerance: 1.49e-08` stem from the fact that NLLS requires computer simulations; revisit the [Lecture](https://github.com/mhasoba/TheMulQuaBio/blob/master/Lectures/NLLS/Pawar_NLLS.pdf) for an explanation of this.

Now let's visualize the fit. For this, first we need to generate a vector of body lengths (the x-axis variable) for plotting:

```R
Lengths <- seq(min(Data2Fit$TotalLength),max(Data2Fit$TotalLength),len=200)
```

Next, calculate the predicted line. For this, we will need to extract the coefficient from the model fit object using the `coef()`command.

```R
coef(PowFit)["a"]
coef(PowFit)["b"]
```

a: 3.94068491030517e-06

b: 2.58504796772587

So, we can do the following:

```R
Predic2PlotPow <- powMod(Lengths,coef(PowFit)["a"],coef(PowFit)["b"])
```

Now plot the data and the fitted model line:

```R
plot(Data2Fit$TotalLength, Data2Fit$BodyWeight)
lines(Lengths, Predic2PlotPow, col = 'blue', lwd = 2.5)
```

We can claculate the confidence intervals on the estimated parameters as we would in OLS fitting used for Linear Models:

```R
confint(PowFit)
```

    Waiting for profiling to be done...

	2.5%	97.5%

a	1.171935e-06	1.205273e-05

b	2.318292e+00	2.872287e+00

As you learnt before, a coefficient's CI should not include zero for it to be statistically significant (different from zero).

#### Exercises

(a) Make the same plot as above, fitted line and all, in `ggplot`, and add (display) the equation you estimated to your new (ggplot) plot. The equation is: $\text{Weight} = 3.94 \times 10^{-06} \times \text{Length}^{2.59}$

(b) Try playing with the starting values, and see if you can "break" the model fit -- that is the NLLS fitting does not converge on a solution.

(c) Repeat the model fitting (incuding a-b above) using the Zygoptera data subset.

(d) There is an alternative (and in fact, more commonly-used) approach for fitting the allometric model to data: using Oridinary Least Squares on bi-logarithamically transformed data. That is, if you take a log of both sides of the [allometric equation](#eq:allom) we get,

$$
\log(y) = \log(a) + b \log(x)
$$

This is a straight line equation of the form $c = d + b z $, where $c = \log(c)$, $d = \log(a)$, $z = \log(x)$, and $b$ is now the slope parameter. So you can use Ordinary Least Squares and the linear models framework (with `lm()`) in R to estimate the parameters of the allometric equation.

In this exercise, try comparing the NLLS vs OLS methods to se how much difference you get in the parameter estimates between them. For example, see the methods used in this paper by [Cohen et al 2012](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3465447/).

(e) The allometry between Body weight and Length is not the end of the story, you have a number of other linear morphological measurements (`HeadLength`, `ThoraxLength`, `AdbdomenLength`, `ForewingLength`, `HindwingLength`, `ForewingArea`, and `HindwingArea`) that can also be investigated. In this exercise, you will try two lines of investigation (again, repeated separately for Dragonflies and Damselfiles):

    (i) How do each of these measurs allometrically scale with Body length (obtain estimates of scaling constant and exponent)? (Hint: you may want to use the `pairs()` command in R to get an overview of all the pairs of potential scaling relationships.

    (ii) Do any of the linear morphological measurements other than body length better predict Body weight? That is, does body weight scale more tightly with a linear morphological measurement other than total body length?

### Comparing two models

*How do we know that there isn't a better or alternative model that adequately explains the pattern in your dataset?*

This is important consideration in all data analyses (and more generally, the scientific method!), so you must aim to compare your NLLS model with an one or more alternatives for a more extensive and reliable investigation of the problem.

Let's use  model comparison to investigate whether the relationship between body weight and length we found above is indeed allometric. For this, we need an alternative model that can be fitted to the same data. Let's try a quadratic curve, which is of the form:

$$y = a + b x + c x^2$$

This can also capture curvature in data, just as the [allometric equation](#eq:allom). Note that this mode is linear in its parameters (a linear model), which You can fit to the simply data using your favorite `lm()` function:

```R
QuaFit <- lm(BodyWeight ~ poly(TotalLength,2), data = Data2Fit)
```

And like before, we obtain the predicted values (but this time using the `predict.lm` function):

```R
Predic2PlotQua <- predict.lm(QuaFit, data.frame(TotalLength = Lengths))
```

Now let's plot the two fitted models together:

```R
plot(Data2Fit$TotalLength, Data2Fit$BodyWeight)
lines(Lengths, Predic2PlotPow, col = 'blue', lwd = 2.5)
lines(Lengths, Predic2PlotQua, col = 'red', lwd = 2.5)
```

Very similar fits, except that the quadratic model seems to get it wrong at the lower end of the data range. Let's do a proper/formal model comparison now to check which model better-fits the data.

Let's first calculate the R$^2$ values of the two fitted models:

```R
RSS_Pow <- sum(residuals(PowFit)^2)  # Residual sum of squares
TSS_Pow <- sum((Data2Fit$BodyWeight - mean(Data2Fit$BodyWeight))^2)  # Total sum of squares
RSq_Pow <- 1 - (RSS_Pow/TSS_Pow)  # R-squared value

RSS_Qua <- sum(residuals(QuaFit)^2)  # Residual sum of squares
TSS_Qua <- sum((Data2Fit$BodyWeight - mean(Data2Fit$BodyWeight))^2)  # Total sum of squares
RSq_Qua <- 1 - (RSS_Qua/TSS_Qua)  # R-squared value

RSq_Pow
RSq_Qua
```

0.90054752976309

0.900302864503218

Not very useful. In general, R$^2$ is a good measure of model fit, but cannot be used for model selection -- epecially not here, given the tiby difference in the R$^2$'s.

Instead, as discussed in the lecture, we can use the Akaike Information Criterion (AIC):

```R
n <- nrow(Data2Fit) #set sample size
kPow <- length(coef(PowFit)) # get number of parameters in power law model
kQua <- length(coef(QuaFit)) # get number of parameters in quadratic model

AIC_Pow <- n * log((2 * pi) / n) + n + 2 + n * log(RSS_Pow) + 2 * kPow
AIC_Qua <- n * log((2 * pi) / n) + n + 2 + n * log(RSS_Qua) + 2 * kQua
AIC_Pow - AIC_Qua
```

-2.1474260812509

Of course, as you might have suspected, we can do this using an in-built function in R!

```R
AIC(PowFit) - AIC(QuaFit)
```

-2.1474260812509

*So which model wins?* As we had dicussed in the NLLS lecture, a rule of thumb is that a AIC value difference (typically denoted as $\Delta$AIC) > 2 is a acceptable cutoff for calling a winner. So the power law (allometric model) is a better fit here. Read the [Johnson & Omland paper](https://github.com/mhasoba/TheMulQuaBio/blob/master/Readings/Modelling/JohnsonOmland2004.pdf) for more on model selection in Ecology and Evolution.

#### Exercises

(a) Calculate the Bayesian Information Criterion (BIC), also know as the Schwarz Criterion (see your Lecture notes and the [Johnson & Omland paper](https://github.com/mhasoba/TheMulQuaBio/blob/master/Readings/Modelling/JohnsonOmland2004.pdf), and use $\Delta$BIC to select the better fitting model.

(b) Fit a straight line to the same data and compare with the allometric and quadratic models.

(c) Repeat the model comparison (incuding 1-2 above) using the Damselflies (Zygoptera) data subset -- does the allometric mdoel still win?

(d) Repeat exercise (e)(i) and (ii) from the [above set](#Allom_Exercises), biut with model comparison (e.g., again using a quadratic as a alternative model) to establish that the relationships are indeed allometric.

### Fitting growth rate models

Let's up the ante and tackle a more complicated NLLS problem - Fitting population growth rates to data. In general, a population grows exponentially while its abundance is low and resources are not limiting. This growth then slows and eventually stops as resources become limiting. There may also be a time lag before the population growth really takes off at the start.

We will work with some Bacterial growth data that [Tom Smith](https://mhasoba.pythonanywhere.com/pawarlab/default/people), a PhD student at Silwood, has been generating as part of his Dissertation research.  Bacterial growth in batch culture follows a distinct set of phases; lag phase, exponential phase and stationary phase. During the lag phase a suite of transcriptional machinery is activated, including genes involved in nutrient uptake and metabolic changes, as bacteria prepare for growth. During the exponential growth phase, bacteria divide at a constant rate, the population doubling with each generation. When the carrying capacity of the media is reached, growth slows and the number of cells in the culture stabilises, beginning the stationary phase.

Traditionally, growth rate can be measured by plotting out cell numbers or culture density against time on a semi-log graph and fitting a straight line through the exponential growth phase - the slope of the line gives the maximum growth rate ($r_{max}$). Models have since been devised which we can use to describe the whole sigmoidal bacterial growth curve.

Let's first generate some "data" on the number of bacterial cells as a function of time taht we can play with:

```R
time <- c(0, 2, 4, 6, 8, 10, 12, 16, 20, 24) # timepoints, in hours
log_cells <- c(3.62, 3.62, 3.63, 4.14, 5.23, 6.27, 7.57, 8.38, 8.70, 8.69) # logged cell counts - more on this below

data <- data.frame(time, log_cells) + rnorm(length(time),sd=.1) # add some random error

names(data) <- c("t", "LogN")
```

We have added a vector of normally distributed errors to emulate random "sampling errors". Note also that the the assumption of normality of these errors underlies the statistical analyses of Ordinary NLLS fits just as it underies Ordinary Least Squares (your standard linear modelling).

Plot the data:

```R
ggplot(data, aes(x = t, y = LogN)) + geom_point()
```

We will fit three growth models, all of which are known to fit such population growth data, especially in microbes. These are a modified Gompertz model (Zwietering et. al., 1990), the Baranyi model (Baranyi, 1993) and the Buchanan model (or three-phase logistic model; Buchanan, 1997). Given a set of cell numbers (N) and times (t), each growth model can be described in terms of:

$N_0$:  Initial cell culture (Population) density (number of cells per unit volume)

$N_{max}$: Maximum culture density (aka "carrying capacity")

$r_{max}$: Maximum growth rate

$t_{lag}$: Duration of the lag phase before the population starts growing exponentially

First let's specify the model functions:

```R
baranyi_model <- function(t, r_max, N_max, N_0, t_lag){  # Baranyi model (Baranyi 1993)
return(N_max + log10((-1+exp(r_max*t_lag) + exp(r_max*t))/(exp(r_max*t) - 1 + exp(r_max*t_lag) * 10^(N_max-N_0))))
}

buchanan_model <- function(t, r_max, N_max, N_0, t_lag){ # Buchanan model - three phase logistic (buchanan 1997)
  return(N_0 + (t >= t_lag) * (t <= (t_lag + (N_max - N_0) * log(10)/r_max)) * r_max * (t - t_lag)/log(10) +
           (t >= t_lag) * (t > (t_lag + (N_max - N_0) * log(10)/r_max)) * (N_max - N_0))
}

gompertz_model <- function(t, r_max, N_max, N_0, t_lag){  # Modified gompertz growth model (Zwietering 1990)
  return(N_0 + (N_max - N_0) * exp(-exp(r_max * exp(1) * (t_lag - t)/((N_max - N_0) * log(10)) + 1)))
}

```

It is important to note that we have written the funcions in log (to the base 10 - can also be base 2 or natural log) scale. This is because NLLS fitting often converges better in log scale. The interpretation of each of the the estimated/fitted paramters does not change if we take a log of the model's equation.

Now let's generate some starting values for the NLLS fitting. We did not pay much attention to what starting values we used in the above example on fitting an allometric model because the power-law model is easy to fit using NLLS, and starting far from the optimal parameters does not matter too much. Here, we derive the starting values by using the actual data:

```R
N_0_start <- min(data$LogN)
N_max_start <- max(data$LogN)
t_lag_start <- data$t[which.max(diff(diff(data$LogN)))]
r_max_start <- max(diff(data$LogN))/mean(diff(data$t))
```

Now fit the models:

```R
fit_baranyi <- nlsLM(LogN ~ baranyi_model(t = t, r_max, N_max, N_0, t_lag), data,
              list(t_lag=t_lag_start, r_max=r_max_start, N_0 = N_0_start, N_max = N_max_start))

fit_buchanan <- nlsLM(LogN ~ buchanan_model(t = t, r_max, N_max, N_0, t_lag), data,
                        list(t_lag=t_lag_start, r_max=r_max_start, N_0 = N_0_start, N_max = N_max_start))

fit_gompertz <- nlsLM(LogN ~ gompertz_model(t = t, r_max, N_max, N_0, t_lag), data,
                      list(t_lag=t_lag_start, r_max=r_max_start, N_0 = N_0_start, N_max = N_max_start))
```

    Warning message in baranyi_model(t = t, r_max, N_max, N_0, t_lag):
    “NaNs produced”Warning message in baranyi_model(t = t, r_max, N_max, N_0, t_lag):
    “NaNs produced”Warning message in baranyi_model(t = t, r_max, N_max, N_0, t_lag):
    “NaNs produced”

You might get a warning message such as:

`Warning message in baranyi_model(t = t, r_max, N_max, N_0, t_lag):
“NaNs produced”`

This just means that the Baranyi model generated some NaNs during the fitting procedure for the given data. You can ignore it in this case (but not always - sometimes these NaNs mean that the equation is wrongly written, or that it generates NaNs across the whole range of the x-values, in which case the model is inappropriate for these data).

Get the model summaries:

```R
summary(fit_baranyi)
summary(fit_buchanan)
summary(fit_gompertz)
```

    Formula: LogN ~ baranyi_model(t = t, r_max, N_max, N_0, t_lag)

    Parameters:
          Estimate Std. Error t value Pr(>|t|)
    t_lag  5.23445    0.39510   13.25 1.14e-05 ***
    r_max  1.34149    0.09418   14.24 7.48e-06 ***
    N_0    3.54887    0.09488   37.40 2.44e-08 ***
    N_max  8.65373    0.09067   95.45 8.91e-11 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.153 on 6 degrees of freedom

    Number of iterations to convergence: 17
    Achieved convergence tolerance: 1.49e-08

    Formula: LogN ~ buchanan_model(t = t, r_max, N_max, N_0, t_lag)

    Parameters:
          Estimate Std. Error t value Pr(>|t|)
    t_lag  4.13715    0.73469   5.631  0.00134 **
    r_max  0.99942    0.08768  11.398 2.73e-05 ***
    N_0    3.57192    0.16900  21.136 7.31e-07 ***
    N_max  8.78296    0.20698  42.433 1.15e-08 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.2927 on 6 degrees of freedom

    Number of iterations to convergence: 6
    Achieved convergence tolerance: 1.49e-08

    Formula: LogN ~ gompertz_model(t = t, r_max, N_max, N_0, t_lag)

    Parameters:
          Estimate Std. Error t value Pr(>|t|)
    t_lag  5.47660    0.23460   23.34 4.05e-07 ***
    r_max  1.46198    0.07087   20.63 8.44e-07 ***
    N_0    3.55900    0.05991   59.41 1.53e-09 ***
    N_max  8.84733    0.07520  117.64 2.54e-11 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.09735 on 6 degrees of freedom

    Number of iterations to convergence: 9
    Achieved convergence tolerance: 1.49e-08

And see how the fits look:

```R
timepoints <- seq(0, 24, 0.1)
baranyi_points <- baranyi_model(t = timepoints, r_max = coef(fit_baranyi)["r_max"], N_max = coef(fit_baranyi)["N_max"],
                                N_0 = coef(fit_baranyi)["N_0"], t_lag = coef(fit_baranyi)["t_lag"])
buchanan_points <- buchanan_model(t = timepoints, r_max = coef(fit_buchanan)["r_max"], N_max = coef(fit_buchanan)["N_max"],
                                N_0 = coef(fit_buchanan)["N_0"], t_lag = coef(fit_buchanan)["t_lag"])
gompertz_points <- gompertz_model(t = timepoints, r_max = coef(fit_gompertz)["r_max"], N_max = coef(fit_gompertz)["N_max"],
                                N_0 = coef(fit_gompertz)["N_0"], t_lag = coef(fit_gompertz)["t_lag"])

df1 <- data.frame(timepoints, baranyi_points)
df1$model <- "Baranyi"
names(df1) <- c("t", "LogN", "model")

df2 <- data.frame(timepoints, buchanan_points)
df2$model <- "Buchanan"
names(df2) <- c("t", "LogN", "model")

df3 <- data.frame(timepoints, gompertz_points)
df3$model <- "Gompertz"
names(df3) <- c("t", "LogN", "model")

model_frame <- rbind(df1, df2, df3)

ggplot(data, aes(x = t, y = LogN)) +
  geom_point(size = 3) +
  geom_line(data = model_frame, aes(x = t, y = LogN, col = model), size = 1)
```

#### Exercises

(a) Calculate the confidence intervals on the parameters of each of the three fitted models, and use model selection (using AIC and/or BIC) as you did before to find the best-fitting model of the three

(b) Repeat the model comparison exercise 1000 times (You will have to write a loop), and determine whether one model gerally wins more often than the others. Note that each rub will generate a slightly different dataset, because we are adding a vector of random errors every time the "data" are generated.

(c) Repeat (2), but increase the error by increasing the standard deviation of the normal error distributon, and see if there are differences in the robustness of the models to sampling/experimental errors. You may also want to try chaning the distribution of the errors to some non-normal distribtion as see what happens.

(d) Fit some real data to these models! Import the [following dataset on bacterial growth rates](https://github.com/mhasoba/TheMulQuaBio/blob/master/Data/example_growth_data.csv) into R:

```R
BacData <- read.csv("../Data/example_growth_data.csv")

head(BacData)
tail(BacData)
```

ID	bacterial_genus	replicate	trait_name	trait_value	hour

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	5.301030      	 0

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	5.301030      	 5

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	6.991226      	10

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	8.094820      	15

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	8.358316      	20

Sch_AE103_02  	Flavobacterium	1             	Log(cells/mL) 	8.460296      	25

	ID	bacterial_genus	replicate	trait_name	trait_value	hour

963	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.086360     	25

964	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.322219     	30

965	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.361728     	35

966	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.322219     	40

967	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.260071     	45

968	Wil_SP04_02  	Bacillus     	4            	Log(cells/mL)	7.292256     	50

The column `trait_value` and `hour` are your variables of interest (log cell density and time), respectively. Note that the `ID` column will tell you which rows represent one separate growth experiment.  Make sure you have a good look at the data first by plotting them up (idealy, in a loop).

That's the end of this NLLS tutorial. Hope you found it useful, and that it gives you a toe-hold for learning how to fit non-linear models to your own data.

Also, note that similar to linear models, you can use Non-linear Mixed Effects statistical methods to fit nonlinear models to data (e.g., using the `lme4` and/or `nlme` packages in R).

## Readings and Resources

* Motulsky, Harvey, and Arthur Christopoulos. Fitting models to biological data using linear and nonlinear regression: a practical guide to curve fitting. OUP USA, 2004.
* Johnson, J. B. & Omland, K. S. 2004 Model selection in ecology and evolution. Trends Ecol. Evol. 19, 101–108.
* The [NCEAS non-linear modelling working group](https://groups.nceas.ucsb.edu/non-linear-modeling/projects/OrangeTree)
* [Mixed-Effects Models in S and S-PLUS](https://link.springer.com/book/10.1007/b98882)


# Project 1

The Lorenz equations show deterministic non-periodic behavior, or chaos.  One aspect of this behavior is the system's sensitive dependence on initial conditions, which has been termed "the butterfly effect" and has been discussed
and used in many books and movies.  The Lorenz equations are:

\begin{align}
   \frac{{\rm d}x(t)}{{\rm d}t} &= \sigma (y(t) -x(t)) \\
   \frac{{\rm d}y(t)}{{\rm d}t} &= x(t) (\rho - z(t)) - y(t) \\
   \frac{{\rm d}z(t)}{{\rm d}t} &= x(t)y(t) - \beta z(t)
\end{align}

where $\sigma, \rho$ and $\beta$ are constants, $x, y$, and $z$ are dependent variables and $t$ is time.  In this problem you will integrate the Lorenz equations in time for different values of $\rho$ and different initial conditions in order to see the butterfly effect for yourself.

1. Complete the Python class `Lorenz()` to integrate the Lorenz equations with different initial conditions.  The values of $\sigma$, $\beta$, and $\rho$ should be set as arguments to the class and the initial conditions vector to $[x_0, 1, 0]$ where $x_0$ is given as an argument to the class function `solve()` which should return the solution.

    **Hint**: Google `scipy.integrate.odeint`.

2. Set $\sigma=10$, $\beta=8/3$, and $\rho =14$ and integrate the Lorenz equation twice, with $x_0 = 0$, and then with $x_0 = 1 \times 10^{-5}$.  Use subscript $1$ for the result when $x_0 = 0$ and subscript $2$ for the result when $x_0 = 1 \times 10^{-5}$ .  For this value of $\rho$, the behavior of the system is called a stable limit cycle.

   Using `matplotlib`, make three plots:

     a. $x_1(t)$ and $x_2(t)$ as a function of $t$

     b. the absolute value of the difference $x_1(t) - x_2(t)$ as a function of $t$

     c. $z_1$ as a function of $x_1$, which is called a phase plot

1. Repeat Steps 2 and 3 above with $\rho = 28$.  For the value of $\rho$, the behavior of the system is called a strange attractor.  The second plot of the absolute value of the difference of $x_1(t) - x_2(t)$ demonstrates the butterfly effect.

1. Complete the function `plot()` so that 2 and 3 above automatically (without any arguments) create the following plot.

```python
import numpy as np
import scipy.integrate

import matplotlib
import matplotlib.pyplot as plt

class Lorenz(object):

    def __init__(self, sigma, beta, rho):

        self.sigma = sigma
        self.beta = beta
        self.rho = rho

        self.t = np.linspace(0,100,num=10000)

    def solve(self, x0):

        return

    def plot(self, figname=None):

        if figname is not None:
            plt.savefig(figname, bbox_inches='tight')
```

```python
#%matplotlib inline
#l = Lorenz(0,0,0)
#l.plot('lorenz_gold.png')
```


# Taylor problem 1.39

last revised: 07-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

The goal of this notebook is to practice Python while considering some visualizations of problem 1.39 to see how they might help check the results, interpret the behavior, or suggest new ideas on how to extend the problem.  Suggestions are very welcome! We'll use the notebook Matplotlib backend for Jupyter (`%matplotlib notebook`) to allow for reading points off the plot.

**Problem statement:** A ball is thrown with initial speed $v_0$ up an inclined plane.  The plane is inclined at an angle $\phi$ above the horizontal, and the ball's initial velocity is at an angle $\theta$ above the plane.  Choose axes with $x$ measured up the slope, $y$ normal to the slope, and $z$ across it.  Write down Newton's second law using these axes and find the ball's position as a function of time.  Show that the ball lands a distance

$
\begin{align}
    R = \frac{2 v_0^2 \sin\theta \cos(\theta+\phi)}{g \cos^2\phi}
\end{align}
$

from its launch point.  Show that for given $v_0$ and $\phi$, the maximum possible range up the inclined plane is

$
\begin{align}
   R_{\textrm{max}} = \frac{v_0^2}{g(1+\sin\phi)}  \;.
\end{align}
$

### Gameplan:
* Accept the formulas for $R$ and $R_\textrm{max}$ as given and make plots.  Interpret.
* Use a fixed value of phi first, then loop through phi.
* Add formatting to plots (and widgets!).
* Check the formulas using sympy.
* Solve the ODE numerically and make some spot comparisons to the formulas.

### Make plots of $R$ and $R_\textrm{max}$

We are given expressions for $R$ and $R_\textrm{max}$, so we'll first turn them into functions.  Then make some plots.

```python
import numpy as np
from numpy import pi
```

```python
def Range_R(theta, phi, g=9.8, v0=1.0):
    """
    Distance R up the plane given angles theta and phi.
    Given in Taylor problem 1.39.
    """
    return (2.*v0**2 * np.sin(theta) * np.cos(theta+phi))/(g*np.cos(phi)**2)
```

```python
def Rmax(phi, g=9.8, v0=1.0):
    """
    Maximum distance up the plane for fixed angle phi.
    Given in Taylor problem 1.39.
    """
    return v0**2/(g*(1. + np.sin(phi)))
```

```python
# set up for plotting but now using the notebook backend
%matplotlib notebook
import matplotlib.pyplot as plt
import matplotlib as mpl

# The following can be uncommented if we want to adjust the font size.
plt.rcParams.update({'font.size': 10})
```

```python
# Make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8.]
theta = np.arange(0,pi/2.,0.01)

# Label the figure and axis with _R to distinguish it
fig_R = plt.figure(figsize=(3,3))
ax_R = fig_R.add_subplot(1,1,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

# start with phi=0 to see if it looks right
ax_R.plot(theta, Range_R(theta,phi=0.))
fig_R.tight_layout()  # make the spacing of subplots nicer

```

Point to the curve to read off coordinates and try out the various controls (hover over a button to reveal a tooltip).

Remember that this is not a trajectory; each trajectory is just one point on this curve, specified by $\theta$ and $\phi=0$.  Does the shape and features of the curve (e.g., symmetry about the peak) agree with your intuition?  Can you prove (or verify) it should be that way from the analytic formula?

It's awkward to work in radians when our intuition (mine, at least) is better in degrees.  Let's define some functions to convert (no doubt there are built-in functions, but these are easy and good practice for us).

```python
def rad_to_deg(theta_rad):
    """Take as input an angle in radians and return it in degrees."""
    return 180./np.pi * theta_rad

def deg_to_rad(theta_deg):
    """Take as input an angle in degrees and return it in radians."""
    return np.pi/180. * theta_deg

```

Now make a plot with $\theta$ in degrees with a list of $\phi$ values, specified in radians but converted to degrees for the plot legend.

```python
# Now make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8., 0.99*pi/2]
theta = np.arange(0,pi/2.,0.01)
theta_deg = rad_to_deg(theta)

fig_R = plt.figure(figsize=(4,4))
ax_R = fig_R.add_subplot(1,1,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$ (degrees)')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

#ax_R.plot(theta, Range_R(theta,phi=0.))

ax_R.set_ylim(0.,0.15)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    ax_R.plot(theta_deg, Range_R(theta,phi), label=label_phi)
ax_R.legend()

fig_R.tight_layout()  # make the spacing of subplots nicer

```

Hmmm, it would be nice to have a picture of the incline next to this.  How would we do that?

```python
# make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8., 0.99*pi/2]
theta = np.arange(0,pi/2.,0.01)
theta_deg = rad_to_deg(theta)

fig_R = plt.figure(figsize=(6,3))
ax_R = fig_R.add_subplot(1,2,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

ax_R.set_ylim(0.,0.15)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    ax_R.plot(theta_deg, Range_R(theta,phi), label=label_phi)
ax_R.legend()

# now add another subplot with the inclined plane.
ax_plane = fig_R.add_subplot(1,2,2)
ax_plane.set_title('Inclined plane')
ax_plane.set_xlim(0.,1.1)
ax_plane.set_ylim(0.,1.1)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    x_pts = [0., 1.]
    y_pts = [0., np.tan(phi)]
    ax_plane.plot(x_pts, y_pts, label=label_phi, lw=2)
ax_plane.axis('off')
ax_plane.legend()

fig_R.tight_layout()  # make the spacing of subplots nicer

```

Ok, now we can look at the $R(\theta,\phi)$ plot and make observations, which we can then try to back-up from the formula. **Try to answer these questions.**
* Can you generalize the argument about symmetry to any $\phi$?
* What are the constraints on $R$ from the figure?  Do they match the formula?
* Do the changes in the curves with increasing $\phi$ make sense?  Can they be extracted from the formula?

Now plot the $R_{max}$ formula. **Evaluate whether it is consistent with the plots of R.**

```python
# make a plot of R_max versus phi
fig_Rmax = plt.figure(figsize=(3,3), num='Taylor 1.39')
ax_Rmax = fig_Rmax.add_subplot(1,1,1)
ax_Rmax.set_title(r'Maximum Distance given $\phi$')
ax_Rmax.set_xlabel(r'$\phi$ (degrees)')
ax_Rmax.set_ylabel(r'$R_{max}$')

phi = np.arange(0,pi/2.,.1)
ax_Rmax.plot(rad_to_deg(phi), Rmax(phi))

fig_Rmax.tight_layout()  # make the spacing of subplots nicer
```

**Spotcheck some points on this curve against the previous graph of $R$.  What should you compare?**

### Widgetizing with ipywidgets and interact

The widgets don't seem to play well with the notebook back end, so we'll switch to `%matplotlib inline`.  You may need to restart the kernel.

```python
%matplotlib inline

from ipywidgets import interact, fixed
import ipywidgets as widgets

# A simple function evaluation (all in radians at this point).
interact(Range_R, theta=np.pi/4., phi=(0,np.pi/2), g=fixed(9.8));
```

```python
def plot_range_R_versus_theta(phi_deg=0):
    """Make a plot of range R versus theta for given phi values in degrees"""

    phi_rad = deg_to_rad(phi_deg)
    theta_deg = np.arange(0,90.,0.1)
    theta_rad = deg_to_rad(theta_deg)
    range_R = Range_R(theta_rad,phi_rad)

    fig_R = plt.figure(figsize=(6,3), num='Taylor 1.39')
    ax_R = fig_R.add_subplot(1,2,1)
    ax_R.set_title('Distance to landing')
    ax_R.set_xlabel(r'$\theta$')
    ax_R.set_ylabel(r'$R(\theta,\phi)$')

    label_phi = fr'$\phi = {phi_deg:.1f}\!^\circ$'
    ax_R.plot(theta_deg, range_R, label=label_phi)
    ax_R.set_ylim(bottom=0., top=1.1*range_R.max())  # set limit *after* plotting to get auto-scaling
    ax_R.legend()

    ax_plane = fig_R.add_subplot(1,2,2)
    ax_plane.set_title('Inclined plane')
    ax_plane.set_xlim(0.,1.1)
    ax_plane.set_ylim(0.,1.1)
    label_phi = fr'$\phi = {phi_deg:.1f}\!^\circ$'
    x_pts = [0., 1.]
    y_pts = [0., np.tan(phi_rad)]
    ax_plane.plot(x_pts, y_pts, label=label_phi, lw=2)
    ax_plane.axis('off')
    ax_plane.legend()

    fig_R.tight_layout()  # make the spacing of subplots nicer

interact(plot_range_R_versus_theta, phi_deg=(0.,90.));
```

```python
# to avoid the jiggling and do some formatting
phi_deg_widget = widgets.FloatSlider(min=0., max=90.0, step=0.1, value=0.,
                                     description=r'$\phi$ in degrees',
                                     readout_format='.0f',
                                     continuous_update=False
                                    )
interact(plot_range_R_versus_theta, phi_deg=phi_deg_widget);
```

### Ok, how about using sympy?

See if we can reproduce the algebra of solving for final t, x, and y.

```python
import sympy as sy
theta, phi, v0, m, g = sy.symbols('theta phi v0 m g')
x, y, t = sy.symbols('x y t')
half = sy.Rational(1,2)
```

```python
tf = sy.solve(v0*sy.sin(theta)*t - half*g*sy.cos(phi)*t**2, t)
tf
```

```python
x = v0*sy.cos(theta)*t - half*g*sy.sin(phi)*t**2
xf = x.subs(t,tf[1])
R = sy.trigsimp(xf)
R
```

```python
thetamax = sy.solve(sy.trigsimp(sy.diff(R,theta)),theta)
thetamax
```

```python
Rmax = R.subs(theta,thetamax[0])
sy.simplify(Rmax)
```

### Ok, now as a differential equation

We'll use odeint for simplicity.  Treat it as coupled 2nd-order differential equations for $x(t)$, $y(t)$, $v_x(t)$, and $v_y(t)$:

$
\begin{align}
   \frac{d}{dt}\left(\begin{array}{c}
                          \mathbf{x} \\
                          \mathbf{v}
                      \end{array}\right)
               = \left(\begin{array}{c}
                          \mathbf{v} \\
                          \mathbf{F}/m
                       \end{array}\right)
      \qquad \Longrightarrow \qquad
   \frac{d}{dt}\left(\begin{array}{c}
                          x \\
                          y \\
                          v_x \\
                          v_y
                      \end{array}\right)
               = \left(\begin{array}{c}
                          v_x \\
                          v_y \\
                          F_x/m \\
                          F_y/m
                       \end{array}\right)
\end{align}
$

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
```

```python
def ode_rhs(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [x, y, v_x, v_y] and params = [g, phi].
    """
    x, y, v_x, v_y = u_vec  # We don't actually use x or y here, but could!
    g, phi = params
    return [v_x, v_y, -g*np.sin(phi), -g*np.cos(phi)]
```

```python
g = 9.8
phi = np.pi/8.
theta = np.pi/4.
v0 = 1.

analytic_range = Range_R(theta,phi,g,v0)
print("Analytic range = {}".format(analytic_range))

u0_vec = [0, 0, v0*np.cos(theta), v0*np.sin(theta)]

t_max = 1.  # integration time
t_pts = np.arange(0, t_max, 0.01)

# absolute and relative tolerances for ode solver
abserr = 1.0e-8
relerr = 1.0e-6

# Integrate the differential equation
x, y, v_x, v_y = odeint(ode_rhs, u0_vec, t_pts, args=(g, phi),
              atol=abserr, rtol=relerr).T
```

We'll just check one case here, but you can do more!

```python
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(1,1,1)
ax.plot(x, y)
ax.set_ylim(0,.15)
ax.set_xlim(0,.15)
ax.set_xlabel('x: distance up incline')
ax.set_ylabel(r'y: distance $\perp$ to incline')
ax.axvline(analytic_range, color="red")
```

**Did it work?**

```python

```


```python

```

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

# Exercise 1

Let us consider the sequence $U_n$ given by
\begin{equation}\label{fib}
\left\lbrace
\begin{array}{ll}
U_0 &= 1,\\
U_1 &= 2,\\
U_{n} &=-3U_{n-1} +U_{n-2}, \;\; \forall\; n=2,3,4\cdots
\end{array}\right.
\end{equation}

Write a python function named  SeqTerms that takes as input an integer $n,\;\;n\geq 0$ and return an array of the first $n$ terms (i.e. $U_0, \cdots, U_{n-1}$) of the sequence \eqref{fib}.

```python
import numpy as np
def Seq(n):
    a=1
    b=2
    if n==0:
        return 1
    if n==1:
        return 2
    for i in range(2,n+1):
        c=-3*b+a
        a=b
        b=c
    return c
Seq(2)
def SeqTerms(n):
    l=[]
    g=np.vectorize(Seq)
    for i in range(n):
        l+=[Seq(i)]
    return l
SeqTerms(4)
```

    [1, 2, -5, 17]

# Exercise 2

Let $\{ x_k\}$ be a partition of $[a,b]$ such that $a=x_0<x_1<\cdots<x_{N-1}<x_N=b$ and $H$ be the length of the $k$-th subinterval ($H = x_k - x_{k-1}$),
then we have
$$\int_a^bf(x)dx \approx \sum_{k=1}^N \frac{f(x_{k-1})+f(x_k)}{2}H = A$$

1. Write a function named Trap that takes $a,b,N, f$ as inputs and return A

```python
def trap(a,b,N,f):
    C=np.linspace(a,b,N+1)
    g=np.vectorize(f)
    A=g(C)
    S=0
    for i in range(1,len(A)):
        S+=A[i]+A[i-1]
    K=1/2*S*((b-a)/N)
    return K
f= lambda x: x**3+7
trap(0,1,10**6,f)
```

    7.250000000000129

2. Write a Python code to compute and display an approximation $Aquad$ of  the integral bellow using the Python function $quad$
$$A = \int_{0}^{2} \dfrac{x^3+5x-20}{x^2+3}dx$$

```python
from scipy.integrate import quad
a = 0
b = 2
#f = lambda x: (x**3+5*x-20)/(x**2+3)
Aquad= quad(f, a, b)[0]
print(Aquad)
```

    -7.049316535735796

3. write a Python function ErrorTrap that takes $M$ as input and return an arrays $ErrorInt$ and $ListN$. Here, $ErrorInt$ contains the absolute errors between $Aquad$ and the approximation of the integral $A$ obtained using the function Trap for all positve intergers $N$ in $ListN$ the set of all multiples of 10 less or equal to $M$.

```python
def ErrorTrap(M):
    u= lambda x: abs(quad(f,0,2)[0]-trap(0,2,x,f))
    ListN=[]
    #ErrorInt=np.zeros(M)
    for i in range(1,M+1):
        if i%10==0:
            ListN+=[i]
    g=np.vectorize(u)
    ErrorInt=g(ListN)
    return ErrorInt, ListN
ErrorTrap(30)
```

    (array([0.0030795, 0.0007707, 0.0003426]), [10, 20, 30])

4. Plot the output $ErrorInt$ against $ListN$ for $M=200$

```python
𝐸𝑟𝑟𝑜𝑟𝐼𝑛𝑡 , 𝐿𝑖𝑠𝑡𝑁 = ErrorTrap(200)
print(𝐿𝑖𝑠𝑡𝑁)
print(ErrorInt)
```

    [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200]
    [3.07950054e-03 7.70701307e-04 3.42601551e-04 1.92726674e-04
     1.23349010e-04 8.56605200e-05 6.29349176e-05 4.81848732e-05
     3.80721757e-05 3.08385649e-05 2.54864800e-05 2.14157629e-05
     1.82477772e-05 1.57340710e-05 1.37061368e-05 1.20464185e-05
     1.06708827e-05 9.51816892e-06 8.54262634e-06 7.70972323e-06]

# Exercise 3

1. Write code to solve the following system of ordinary differential equations using the Python function odeint.

$$
\begin{cases}
\dfrac{dx_1}{dt}& = & -\dfrac{1}{2}x_1\\\\
\dfrac{dx_2}{dt}& = & \dfrac{1}{2}x_1-\dfrac{1}{4}x_2\\\\
\dfrac{dx_3}{dt}& = & \dfrac{1}{4}x_2-\dfrac{1}{6}x_3
\end{cases}, \text{ on } [0,4]
$$

Subject to the initial conditions $x_1(0) = 1, x_2(0) = 1, x_3(0) = 1$.

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# function that returns dz/dt
def model(z,t):
    x_1,x_2,x_3 = z
    dx_1dt = -1/2*x_1
    dx_2dt = 1/2*x_1 -1/4*x_2
    dx_3dt = 1/4*x_2-1/6*x_3
    return dx_1dt,dx_2dt,dx_3dt

# initial condition
z0 = [1,1,1]

# time points
a = 0
b = 4
N = 100
t = np.linspace(a,b,N+1)

# solve ODE
z = odeint(model,z0,t)

x_1 = z[:,0]
x_2 = z[:,1]
x_3=z[:,2]

plt.plot(t,x_1,'b-')
plt.plot(t,x_3,'r--')
plt.plot(t,x_2,'green')

```

2. The exact solution of the above system of ODEs is given by

$$
\begin{cases}
x_1(t)& = & e^{-t/2}\\
x_2(t)& = & -2e^{-t/2}+3e^{-t/4}\\
x_3(t)& = & \dfrac{3}{2}e^{-t/2} - 9e^{-t/4} + \dfrac{17}{2}e^{-t/6}
\end{cases}
$$

Use $Subplot$ to plot side by side

- each exact and approximate solution in the same window
- and their absolute error vs the time

```python
import numpy as np
import matplotlib.pyplot as plt

# x_1(t)=np.exp(-t/2)
# x_2(t)=-2*np.exp(-t/2)+3*np.exp(-t/4)
# x_3(t)=3/2*np.exp(-t/2)-9*np.exp(-t/4)+17/2*np.exp(-t/6)
# #plot results
plt.subplot(3,1,1)
plt.plot(t,np.exp(-t/2),'b')
plt.xlabel('time')
plt.ylabel('x_1(t)')
plt.show()

#plot results
plt.subplot(3,1,2)
plt.plot(t,-2*np.exp(-t/2)+3*np.exp(-t/4),'g-')
plt.xlabel('time')
plt.ylabel('x_2(t)')
plt.show()

plt.subplot(3,1,3)
plt.plot(t,3/2*np.exp(-t/2)-9*np.exp(-t/4)+17/2*np.exp(-t/6),'r-')
plt.xlabel('time')
plt.ylabel('x_3(t)')
plt.show()

#plot results
# plt.subplot(3,1,3)
# plt.plot(x,y)
# plt.xlabel('x')
# plt.ylabel('y')
# plt.show()

```

# Exercise 4

Let $\{ t_k\}$ be a partition of $[a,b]$ such that $a=t_1<t_2<\cdots<t_{N}=b$ and $H$ be the constant length of the $k$-th subinterval ($H = t_k - t_{k-1}$). Let us consider initial value problem

\begin{equation}\label{eul2}
  \begin{cases}
    \dfrac{dz}{dt} = f(z,t),      & \quad \text{on } [a, b]\\\\
    z(a) = c,
  \end{cases}
\end{equation}
where $z,f,c\in R^M$ i.e. $z = [x_1, x_2,\cdots, x_{M}]$, $c = [x_1(a), x_2(a),\cdots, x_{M}(a)]$ and $f = [f_1, f_2,\cdots, f_{M}]$. Note that \eqref{eul2} is a the general form of system of ODEs.

Let $t, z_k,Z$ defined as follows $$t=[t_1,t_2,\cdots,t_{N-1},t_{N}],\quad z_k = [x_1(t_k), x_2(t_k),\cdots, x_{M}(t_k)], \quad
Z =\begin{pmatrix}
x_1(t_1)& x_2(t_1)&\cdots& x_{M}(t_1)\\
x_1(t_2)& x_2(t_2)&\cdots& x_{M}(t_2)\\
\vdots& \vdots&\ddots& \vdots\\
x_1(t_{N})& x_2(t_{N})&\cdots& x_{M}(t_{N})
\end{pmatrix}
$$

1. Write a python function  EulerOdeSys  that takes $f,c,t$ and return the solution $Z$ of the initial value problem \eqref{eul2} using Euler method i.e.
$$ z_{k+1} = z_k + Hf(z_k,t_k) $$

```python
def EulerOdeSys(f,c,t):
    z_1=c[0]
    s=t[0]
    T=[s]
    Z=[z_1]
    for i in range(len(c)+1):
        z_1=z_1+(t[-1]-t[0])/len(t)*f(z_1,s)
        s=s+(t[-1]-t[0])/len(t)
        T.append(s)
        Z.append(z_1)
    return Z
f=lambda x,y: x+y
c=[0,4]
t=[0,2,4]
EulerOdeSys(f,c,t)
```

    [0, 0.0, 1.7777777777777777, 7.703703703703703]

2. Write a python function  RK4OdeSys  that takes $f,c,t$ and return the solution $Z$ of the initial value problem (1) using the fourth order Runge-Kutta method i.e.

\begin{equation}
\begin{cases}
k_1 = f(z_k,t_k),\\\\
k_2 = f(z_k+H\dfrac{k_1}{2}, t_k + \dfrac{H}{2}),\\\\
k_3 = f(z_k+H\dfrac{k_2}{2}, t_k + \dfrac{H}{2}),\\\\
k_4 = f(z_k+Hk_3, t_k + H),\\\\
z_{k+1} = z_k + \dfrac{H}{6}(k_1+2k_2+2k_3+k_4)
\end{cases}
\end{equation}

```python
def RK4OdeSys(f,c,t):
    k1=
```

    2

3. Solve the system of ODEs in $Exercise2$ using your function  EulerOdeSys  and  RK4OdeSys

```python

```

4. By plotting the absolute error in the approximate and exact solutions, tell us which function gives a more accurate solution of a system of ODEs.

```python

```

# Exercise 5

Let consider us consider the function  primes  that takes $n$ as input and return a list of primes less than $n$

```python
# This cell is only to import the labraries

import numpy as np
import time

def primes(n):
    """ Returns  a list of primes < n """
    sieve = [True] * (n//2)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = [False] * ((n-i*i-1)//(2*i)+1)
    return [2] + [2*i+1 for i in range(1,n//2) if sieve[i]]
primes(30)

```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]

For any integer $n>0$ and a prime number $p$, define $\nu_p(n)$ as the greatest integer $r$ such that $p^r$ divides $n$.
Define $$ D(n,m) = \sum_{p\; prime} \Bigl| \nu_p(n) - \nu_p(m)\Bigr| $$

For example $D(14,24)=4$.

Furthermore, define

$$S(N)  = \sum_{n=1}^{N}\sum_{m=1}^{N}D(n,m).$$

You are given $S(10)=210$.

1. Write an efficient python function, Func_S , that takes $N$ as input and return the value $S(N)$.

```python
def val(n,p):
    L=[]
    for i in range(n):
        if n%(p**i)==0:
            L+=[i]
    return L[-1]

def dump(n,m):
    list_prime=primes(100)
    S=0
    for i in list_prime:
        S+=abs(val(n,i)-val(m,i))
    return S

def Func_S(N):
    s=0
    for i in range(1,N+1):
        for j in range(1,N+1):
            s=s+dump(i,j)
    return s
Func_S(10)

```

    210

2. Compute $S(10)$ and display its computational time

```python
N = 10
time_start = time.perf_counter()
S = Func_S(N)
time_elapsed = (time.perf_counter() - time_start)
print('S({}) =  {}'.format(N,S))
print('computational Time = ', time_elapsed)
```

    S(10) =  210
    computational Time =  0.01053150900406763

3. Compute $S(100)$ and display its computational time

```python
N = 100
time_start = time.perf_counter()
S = Func_S(N)
time_elapsed = (time.perf_counter() - time_start)
print('S({}) =  {}'.format(N,S))
print('computational Time = ', time_elapsed)
```

    S(100) =  37018
    computational Time =  8.480332631006604

4. Compute $S(1000)$ and display its computational time

```python
N = 1000
time_start = time.perf_counter()
S = Func_S(N)
time_elapsed = (time.perf_counter() - time_start)
print('S({}) =  {}'.format(N,S))
print('computational Time = ', time_elapsed)
```

5. Compute $S(10000)$ and display its computational time

```python

```

6. Compute $S(100000)$ and display its computational time

```python

```

7. Compute $S(1000000)$ and display its computational time

```python

```

# Exercise 6
1. Import the necessary libraries and read the chipio and assign it to a variable called chipo (this dataset is available at the course website)

```python
import pandas as pd
import numpy as np
chipo=pd.read_csv('chipo.tsv',sep='\t',header=0)
chipo
```

      order_id
      quantity
      item_name
      choice_description
      item_price

      0
      1
      1
      Chips and Fresh Tomato Salsa
      NaN
      $2.39

      1
      1
      1
      Izze
      [Clementine]
      $3.39

      2
      1
      1
      Nantucket Nectar
      [Apple]
      $3.39

      3
      1
      1
      Chips and Tomatillo-Green Chili Salsa
      NaN
      $2.39

      4
      2
      2
      Chicken Bowl
      [Tomatillo-Red Chili Salsa (Hot), [Black Beans...
      $16.98

      ...
      ...
      ...
      ...
      ...
      ...

      4617
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Black Beans, Sour ...
      $11.75

      4618
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Sour Cream, Cheese...
      $11.75

      4619
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $11.25

      4620
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Lettu...
      $8.75

      4621
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $8.75

4622 rows × 5 columns

2. Show the first 10 entries

```python
chipo.iloc[0:10]
chipo[:10]
```

      order_id
      quantity
      item_name
      choice_description
      item_price

      0
      1
      1
      Chips and Fresh Tomato Salsa
      NaN
      $2.39

      1
      1
      1
      Izze
      [Clementine]
      $3.39

      2
      1
      1
      Nantucket Nectar
      [Apple]
      $3.39

      3
      1
      1
      Chips and Tomatillo-Green Chili Salsa
      NaN
      $2.39

      4
      2
      2
      Chicken Bowl
      [Tomatillo-Red Chili Salsa (Hot), [Black Beans...
      $16.98

      5
      3
      1
      Chicken Bowl
      [Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou...
      $10.98

      6
      3
      1
      Side of Chips
      NaN
      $1.69

      7
      4
      1
      Steak Burrito
      [Tomatillo Red Chili Salsa, [Fajita Vegetables...
      $11.75

      8
      4
      1
      Steak Soft Tacos
      [Tomatillo Green Chili Salsa, [Pinto Beans, Ch...
      $9.25

      9
      5
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Black Beans, Pinto...
      $9.25

3. How many observations in the dataset?

```python
chipo.describe()
```

      order_id
      quantity

      count
      4622.000000
      4622.000000

      mean
      927.254868
      1.075725

      std
      528.890796
      0.410186

      min
      1.000000
      1.000000

      25%
      477.250000
      1.000000

      50%
      926.000000
      1.000000

      75%
      1393.000000
      1.000000

      max
      1834.000000
      15.000000

4. How many columns in the dataset?

```python
col=chipo.shape[0]
print('the dataset have\t',col,"columns")
```

5. Print the names of all the columns

```python
chipo.columns
```

    Index(['order_id', 'quantity', 'item_name', 'choice_description', 'item_price',
           'revenue'],
          dtype='object')

```python
chipo
```

      order_id
      quantity
      item_name
      choice_description
      item_price

      0
      1
      1
      Chips and Fresh Tomato Salsa
      NaN
      $2.39

      1
      1
      1
      Izze
      [Clementine]
      $3.39

      2
      1
      1
      Nantucket Nectar
      [Apple]
      $3.39

      3
      1
      1
      Chips and Tomatillo-Green Chili Salsa
      NaN
      $2.39

      4
      2
      2
      Chicken Bowl
      [Tomatillo-Red Chili Salsa (Hot), [Black Beans...
      $16.98

      ...
      ...
      ...
      ...
      ...
      ...

      4617
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Black Beans, Sour ...
      $11.75

      4618
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Sour Cream, Cheese...
      $11.75

      4619
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $11.25

      4620
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Lettu...
      $8.75

      4621
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $8.75

4622 rows × 5 columns

6. How much was the revenue for the period in the dataset?

```python
kk=chipo.groupby('item_price').sum()
kk
chipo[['item_price','quantity']][:6]
chipo['revenue']=chipo['quantity']*chipo["item_price"]
chipo
```

      order_id
      quantity
      item_name
      choice_description
      item_price
      revenue

      0
      1
      1
      Chips and Fresh Tomato Salsa
      NaN
      $2.39
      $2.39

      1
      1
      1
      Izze
      [Clementine]
      $3.39
      $3.39

      2
      1
      1
      Nantucket Nectar
      [Apple]
      $3.39
      $3.39

      3
      1
      1
      Chips and Tomatillo-Green Chili Salsa
      NaN
      $2.39
      $2.39

      4
      2
      2
      Chicken Bowl
      [Tomatillo-Red Chili Salsa (Hot), [Black Beans...
      $16.98
      $16.98 $16.98

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      4617
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Black Beans, Sour ...
      $11.75
      $11.75

      4618
      1833
      1
      Steak Burrito
      [Fresh Tomato Salsa, [Rice, Sour Cream, Cheese...
      $11.75
      $11.75

      4619
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $11.25
      $11.25

      4620
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Lettu...
      $8.75
      $8.75

      4621
      1834
      1
      Chicken Salad Bowl
      [Fresh Tomato Salsa, [Fajita Vegetables, Pinto...
      $8.75
      $8.75

4622 rows × 6 columns

7. How many unique items were sold?

```python
s=np.unique(chipo.item_name)
len(s)
```

    50

8. How many orders were made in the period?

```python
for i in chipo[['item_price','quantity']][0:]:
    print(i)
```

    item_price
    quantity

9. How many missing values in the dataframe?

```python
chipo[['item_price','quantity']][0:1]
```

      item_price
      quantity

      0
      $2.39
      1

```python

```


```python
import sympy
```

```python
sympy.pi.evalf(100)
```

```python
s3=sympy.sqrt(3)
s3
```

```python
s3.evalf(100)
```

```python
s3*s3
```

```python
3**0.5 * 3**0.5
```

```python
sympy.pi
```

```python
x, y = sympy.symbols('x y')
expr = x + 2*y
expr
```

```python
expr-x  # sympy simplifies expressions
```

```python
x*expr
```

```python
sympy.expand(x*expr)
```

```python
# Differentiation
sympy.diff(sympy.sin(x)*sympy.exp(x), x)
```

```python
t=sympy.symbols('t')
y = sympy.Function('y')
sympy.dsolve(sympy.Eq(y(t).diff(t) , -y(t)), y(t) )
```

```python
a,b,c,d = sympy.symbols('a b c d')
sympy.Matrix([[a, b], [c, d]]).eigenvals()
```

```python
sympy.Matrix([[a, b], [c, d]]).inv()
```

```python
sympy.Matrix([[a, b], [c, d]]).det()
```

```python
a,b,c,d,e,f,g,h,i = sympy.symbols('a b c d e f g h i')
```

```python
sympy.Matrix([[a, b, c], [d, e, f], [g, h,i ]]).det()
```

```python
sympy.Matrix([[a, b, c], [d, e, f], [g, h,i ]]).inv()
```

```python
sympy.Matrix([[a, b, c], [d, e, f], [g, h,i ]]).eigenvals()
```

```python
sympy.Matrix([[a, b, c], [d, e, f], [g, h,i ]]).eigenvects()
```

```python

```


# 2017-01-05-RAL Inclusão de fórmulas LaTeX

## Escrevendo em LaTeX em célula Markdown
Equação em uma única linha utilizando \$\$
$$ a = \sum_i jˆi$$
Equação no meio da linha utilizando \$: $ a = \sum i $ como aqui

Utilizando

%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

## Usando %%latex em célula Markdown

Exemplo de equação

%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

# Inserindo fórmula LaTeX nos códigos

```python
from IPython.display import display, Math
display(Math(r'\sqrt{a^2 + b^2}'))

```

$$\sqrt{a^2 + b^2}$$


# Session 7: Introduction to Binary Classification

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

Today, we will start with the second block of the course: classification. Like regression, classification is another example of [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning).

Consider we have a **training** database of $N$ entries of the form $(\mathbf{x}^{(i)},y^{(i)})$, where $\mathbf{x}\in\mathbb{R}^L$ and the label $y$ only belongs to **two possible categories**: $y\in\{0,1\}$. Our goal in classificaiton is to propose an hipothesis function $h(\mathbf{x},y)$ that will be used to estimate the most likely class of a new point $\mathbf{x}^*$ as follows

\begin{align}
y* \approx \arg \max_{y\in\{0,1\}} h(\mathbf{x},y)
\end{align}

Today we will introduce two classification tools:
- K-NN classification
- Logistic Regression

First, lets import a toy dataset to play with.

### Importing Packages

```python
import numpy as np
import sklearn
import functions    #Some useful functions
import matplotlib.pyplot as plt
%matplotlib inline

```

### Loading the database

```python
data = np.loadtxt('ex2data1.txt', delimiter=',')
```

```python
## Dimensions
dims=np.shape(data)
N=dims[0]

## Create X and Y
X=data[:,0:2]
Y=data[:,2]

```

### Splitting in training and validation data sets

```python
np.random.seed(0)

X_train,X_test,Y_train,Y_test = functions.split_set(X,Y,0.8)

min1=np.min(X[:,0])
max1=np.max(X[:,0])
min2=np.min(X[:,1])
max2=np.max(X[:,1])

## Plot Training Data
index=(Y_train==1)
plt.plot(X_train[index,0],X_train[index,1],'s',label="Class 1")
index=(Y_train==0)
plt.plot(X_train[index,0],X_train[index,1],'ro',label="Class 0")
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.grid(True)
plt.legend()
plt.title('Training Data')

```

## K-NN Classifier

Lets start with a **nonparametric** classifier. The K-NN classifier is actually very similar to the K-NN regressor, we merely replace the average NN value by a mayority voting scheme.

Given a new point $\mathbf{x}^*$ the estimated target $y^*$ is computed as follows:

\begin{align}
P(y^*=1|\mathbf{x}^*) \approx \frac{1}{K}\sum_{k\in \mathcal{S}_{K}(\mathbf{x}^*)} \mathbb{I}[y^{(k)}==1],
\end{align}
where $\mathbb{I}[\cdot]$ is the indicator function that takes value $1$ if the condition within the brakets is met, and zero otherwise. Also, $\mathcal{S}_{K}$ is the set of the **$K$ training points that are closest to $\mathbf{x}^*$ according to a given distance metric $d(\mathbf{x}^{(i)},\mathbf{x}^*)$**. For instance, in real spaces the most common choice is the **ecludean distance**:

\begin{align}
d(\mathbf{x}^{(i)},\mathbf{x}^*) = \left|\left|\mathbf{x}^{(i)}-\mathbf{x}^*\right|\right|^2
\end{align}

There are many variants of the above expression. For instance, we can **weight differently the indicator terms according to the distance**.

Lets use the [*sklearn K-NN classifier implementation*](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) in our example.

```python
from sklearn.neighbors import KNeighborsClassifier

K=20

knnc = KNeighborsClassifier(n_neighbors=K)
knnc.fit(X_train,Y_train)

```

    KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
               metric_params=None, n_jobs=1, n_neighbors=20, p=2,
               weights='uniform')

Lets plot the decision boundary, looking for points with predicted label probability around 0.5

```python
#Plotting the decision boundary
x1,x2 = np.mgrid[min1:max1:(max1-min1)/50, min2:max2:(max2-min2)/50]
grid = np.transpose(np.row_stack([x1.ravel(), x2.ravel()]))
probs = knnc.predict_proba(grid)

## Plot Training Data
index=(Y_train==1)
plt.plot(X_train[index,0],X_train[index,1],'s',label="Class 1")
index=(Y_train==0)
plt.plot(X_train[index,0],X_train[index,1],'ro',label="Class 0")
plt.contour(x1,x2,np.reshape(probs[:,0],np.shape(x1)),[0.5])
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.grid(True)
plt.legend()
plt.title('Training Data')

```

### Excersise:

Plot in the above figure the validation points and compute the missclassification rate in both training and validation as a function of $K$.

## Logistic Regression

[Logistic Regression](https://onlinecourses.science.psu.edu/stat504/node/149) (LR) is a particular class of **generalized linear model** in which we use a **linear combination of the input features** to adjust the parameter of a particular probability distribution that explains the variable that we want to predict.

Given our **trainin database** of $N$ entries of the form $(\mathbf{x}^{(i)},y^{(i)})$, imagine that each of the input vectors is transformed to include **additional features** (such as polynomial expansions or any other transformation). We denote the **feature vector** as

\begin{align}
\phi(\mathbf{x}): \mathbb{R}^L \rightarrow \mathbb{R}^D,
\end{align}
where $D\geq L$. In LR, we interpolate the following probabilistic function:
\begin{align}
P(y=1|\phi(\mathbf{x})) =\frac{1}{1+\text{e}^{-\mathbf{\theta}^T\phi(\mathbf{x})}}=\sigma\left(\mathbf{\theta}^T\phi(\mathbf{x})\right),
\end{align}
where $\sigma(\cdot): \mathbb{R}\rightarrow [0,1]$ is the [sigmoid function](http://mathworld.wolfram.com/SigmoidFunction.html).

----

### The cross entropy objective function

While in standard regression the most common loss function is the MSE (mean squared error), in classification with probabilistic ouputs the most common choice is the [**cross entropy loss**](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/):
\begin{align}
L(\theta)&=\frac{1}{N}\left[-\sum_{i=0}^{N}y^{(i)}\log\left(\sigma(\mathbf{\theta}^T\phi(\mathbf{x}^{(i)})\right)+(1-y^{(i)})\log\left(1-\sigma(\mathbf{\theta}^T\phi(\mathbf{x}^{(i)})\right)\right].
\end{align}
Note that this loss is always positives and grows when $y^{(i)}=1$ and $\sigma(\mathbf{\theta}^T\phi(\mathbf{x}^{(i)})\rightarrow 0$ and viceversa. I.e., penalizes predicted probabilities far from the right label. As in linear regression, we can use $L_2$ or $L_1$ regularization to control overfitting:

\begin{align}
L(\theta,\lambda)&=\frac{1}{N}\left[-\sum_{i=0}^{N}y^{(i)}\log\left(\sigma(\mathbf{\theta}^T\phi(\mathbf{x}^{(i)})\right)+(1-y^{(i)})\left(1-\sigma(\mathbf{\theta}^T\phi(\mathbf{x}^{(i)})\right)+\lambda \mathbf{\theta}^T \mathbf{\theta}\right].
\end{align}

Unlike linear regression with MSE loss, minimizing the cross entropy loss function w.r.t. $\mathbf{\theta}$ **does not have closed form solution**. However, the function is still **convex** and can be easily optimized by **gradient descent**.

----

### The decision boundary in LR

Note that, given the LR predicted class probability
\begin{align}
P(y=1|\phi(\mathbf{x})) =\frac{1}{1+\text{e}^{-\mathbf{\theta}^T\phi(\mathbf{x})}}=\sigma\left(\mathbf{\theta}^T\phi(\mathbf{x})\right),
\end{align}
The decision boundary is computed by all points $\mathbf{x}$ in $\mathbb{R}^L$ such that $P(y=1|\phi(\mathbf{x}))=0.5$ or, equivalently, all points $\mathbf{x}$ in $\mathbb{R}^L$ such that

\begin{align}
\mathbf{\theta}^T\phi(\mathbf{x})=0,
\end{align}

which is the general equation of a [manifold](https://en.wikipedia.org/wiki/Manifold) in $\mathbb{R}^L$. Thus, the decision boundary is completely determined by the features in $\phi(\mathbf{x})$. For instance, consider $\mathbf{x}=[x_1, x_2]$. Then if

- $\phi(\mathbf{x})=[1, x_1, x_2] ~ \Rightarrow ~ \mathbf{\theta}^T\phi(\mathbf{x})=0$ is a straighline in the $\mathbb{R}^2$ space.

- $\phi(\mathbf{x})=[1, x_1, x_2, x_1^2, x_2^2, x_1x_2] ~ \Rightarrow ~ \mathbf{\theta}^T\phi(\mathbf{x})=0$ is an ellipsoid in the $\mathbb{R}^2$ space.

Lets use the [*sklearn LR classifier implementation*](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) in our example.

```python
from sklearn.linear_model import LogisticRegression as LR
```

```python
# C is the inverse of the regularization parameter (C=1/\lambda)
LR_class = LR(C=1e3, penalty='l2', tol=0.01,fit_intercept=True)
```

### Example 1: Linear features

$\phi(\mathbf{x})=[1, x_1, x_2]$

```python
X_tnorm,mx,stdx=functions.normalize(X_train,0,1,True)
X_tnorm = functions.add_interfit(X_tnorm)
```

```python
#Training the LR model with quadratic features
LR_class.fit(X_tnorm, Y_train)
```

    LogisticRegression(C=1000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,
              solver='liblinear', tol=0.01, verbose=0, warm_start=False)

```python
#Plotting the decision boundary
x1,x2 = np.mgrid[min1:max1:(max1-min1)/50, min2:max2:(max2-min2)/50]
grid = np.transpose(np.row_stack([x1.ravel(), x2.ravel()]))
#Normalizing data
grid_norm,_,_=functions.normalize(grid,mx,stdx,False)
grid_norm = functions.add_interfit(grid_norm)

#Estimating class probabilities
probs_LR=LR_class.predict_proba(grid_norm)

## Plot Training Data
index=(Y_train==1)
plt.plot(X_train[index,0],X_train[index,1],'s',label="Class 1")
index=(Y_train==0)
plt.plot(X_train[index,0],X_train[index,1],'ro',label="Class 0")
plt.contour(x1,x2,np.reshape(probs[:,0],np.shape(x1)),[0.5])
plt.contour(x1,x2,np.reshape(probs_LR[:,0],np.shape(x1)),[0.5],linestyles='dashed')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.grid(True)
plt.legend()
plt.title('Training Data')

```

```python
# Computing accuracy score

X_testnorm,_,_=functions.normalize(X_test,mx,stdx,False)
X_testnorm = functions.add_interfit(X_testnorm)

print ('Logistic Regression  - Train SCORE: ', LR_class.score(X_tnorm, Y_train))
print ('Logistic Regression  - Test SCORE: ', LR_class.score(X_testnorm, Y_test))
```

    Logistic Regression  - Train SCORE:  0.8375
    Logistic Regression  - Test SCORE:  0.894736842105

### Example 2: Quadratic Features

$\phi(\mathbf{x})=[1, x_1, x_2,x_1^2,x_2^2,x_1x_2]$

```python
# C is the inverse of the regularization parameter (C=1/\lambda)
LR_classq = LR(C=1e3, penalty='l2', tol=0.01,fit_intercept=True)
X_train_ext = np.copy(X_train)
X_train_ext = np.concatenate([X_train,X_train**2,np.reshape(X_train[:,0]*X_train[:,1],[-1,1])],1)
X_tnorm,mx,stdx=functions.normalize(X_train_ext,0,1,True)
X_tnorm = functions.add_interfit(X_tnorm)

X_test_ext = np.copy(X_test)
X_test_ext = np.concatenate([X_test,X_test**2,np.reshape(X_test[:,0]*X_test[:,1],[-1,1])],1)
X_testnorm,_,_=functions.normalize(X_test_ext,mx,stdx,False)
X_testnorm = functions.add_interfit(X_testnorm)
```

```python
#Training the LR model with quadratic features
LR_classq.fit(X_tnorm, Y_train)
```

    LogisticRegression(C=1000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,
              solver='liblinear', tol=0.01, verbose=0, warm_start=False)

```python
#Plotting the decision boundary
x1,x2 = np.mgrid[min1:max1:(max1-min1)/50, min2:max2:(max2-min2)/50]
grid = np.transpose(np.row_stack([x1.ravel(), x2.ravel()]))
grid_ext = np.concatenate([grid,grid**2,np.reshape(grid[:,0]*grid[:,1],[-1,1])],1)
#Normalizing data
grid_norm,_,_=functions.normalize(grid_ext,mx,stdx,False)
grid_norm = functions.add_interfit(grid_norm)

#Estimating class probabilities
probs_LR=LR_classq.predict_proba(grid_norm)

## Plot Training Data
index=(Y_train==1)
plt.plot(X_train[index,0],X_train[index,1],'s',label="Class 1")
index=(Y_train==0)
plt.plot(X_train[index,0],X_train[index,1],'ro',label="Class 0")
plt.contour(x1,x2,np.reshape(probs[:,0],np.shape(x1)),[0.5])
plt.contour(x1,x2,np.reshape(probs_LR[:,0],np.shape(x1)),[0.5],linestyles='dashed')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.grid(True)
plt.legend()
plt.title('Training Data')

```

```python
print ('Logistic Regression  - Train SCORE: ', LR_classq.score(X_tnorm, Y_train))
print ('Logistic Regression  - Test SCORE: ', LR_classq.score(X_testnorm, Y_test))
```

    Logistic Regression  - Train SCORE:  0.85
    Logistic Regression  - Test SCORE:  0.894736842105

### Exercise

Compare K-NN and LR for the following database. Compare both LR with linear and nonlinear features.

```python
data = np.loadtxt('ex2data2.txt', delimiter=',')
```

```python
## Dimensions
dims=np.shape(data)
N=dims[0]

## Create X and Y
X=data[:,0:2]
Y=data[:,2]

### Splitting in training and validation data sets

np.random.seed(0)

X_train,X_test,Y_train,Y_test = functions.split_set(X,Y,0.8)

min1=np.min(X[:,0])
max1=np.max(X[:,0])
min2=np.min(X[:,1])
max2=np.max(X[:,1])

## Plot Training Data
index=(Y_train==1)
plt.plot(X_train[index,0],X_train[index,1],'s',label="Class 1")
index=(Y_train==0)
plt.plot(X_train[index,0],X_train[index,1],'ro',label="Class 0")
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.grid(True)
plt.legend()
plt.title('Training Data')

```

```python
# Your code here
```

Notebooks/Session 7 Intro Classification/Session_7_Intro_Classification.ipynb