# Optimization

- [Least squares](#Least-squares)
- [Gradient descent](#Gradient-descent)
- [Constraint optimization](#Constraint-optimization)
- [Global optimization](#Global-optimization)

## Intro

Biological research uses optimization when performing many types of machine learning, or when it interfaces with engineering. A particular example is metabolic engineering. As a topic in itself, optimization is extremely complex and useful, so much so that it touches to the core of mathematics and computing.

An optimization problem complexity is dependent on several factors, such as:
- Do you intend a local or a global optimization?
- Is the function linear or nonlinear?
- Is the function convex or not?
- Can a gradient be computed?
- Can the Hessian matrix be computed?
- Do we perform optimization under constraints?
- Are those constraints integers?
- Is there a single objective or several?

Scipy does not cover all solvers efficiently but there are several Python packages specialized for certain classes of optimization problems. In general though, the heavier optimization tasks are solved with dedicated programs, many of whom have language bindings for Python.

## Least squares

In practical terms, the most basic application of optimization is computing the local vs global minima of functions. We will exemplify this with the method of least squares. This method is being used to fit the parameters of a function by performing an error minimization.

**Problem context** Having a set of $m$ data points, $(x_1, y_1), (x_2, y_2),\dots,(x_m, y_m)$ and a curve (model function) $y=f(x, \boldsymbol \beta)$ that in addition to the variable $x$ also depends on $n$ parameters, $\boldsymbol \beta = (\beta_1, \beta_2, \dots, \beta_n)$ with $m\ge n$.

It is desired to find the vector  $\boldsymbol \beta$ of parameters such that the curve fits best the given data in the least squares sense, that is, the sum of squares of the residuals is minimized:

$$ min \sum_{i=1}^{m}(y_i - f(x_i, \boldsymbol \beta))^2$$

Let us use a similar exercise as the basic linear regression performet in the statistics chapter, but fit a curve instead. That is to say, we are now performing a very basic form of nonlinear regression. While not strictly statistics related, this exercise can be useful for example if we want to decide how a probability distribution fits our data. We will use the least-square again, through the optimization module of scipy.

```python
%matplotlib inline
import numpy as np
import pylab as plt
from scipy import optimize

nsamp = 30
x = np.linspace(0,1,nsamp)
"""
y = -0.5*x**2 + 7*sin(x)
This is what we try to fit against. Suppose we know our function is generated
by this law and want to find the (-0.5, 7) parameters. Alternatively we might
not know anything about this dataset but just want to fit this curve to it.
"""

# define the normal function
f = lambda p, x: p[0]*x*x + p[1]*np.sin(x)

testp = (-0.5, 7)
print("True(unknown) parameter value:", testp)
y = f(testp,x)
yr = y + .5*np.random.normal(size=nsamp) # adding a small noise

# define the residual function
e = lambda p, x, y: (abs((f(p,x)-y))).sum()

p0 = (5, 20) # initial parameter value
print("Initial parameter value:", p0)

# uses the standard least squares algorithm
p_est1 = optimize.least_squares(e, p0, args=(x, yr))
print("Parameters estimated with least squares:",p_est1.x)
y_est1 = f(p_est1.x, x)

plt.plot(x,y_est1,'r-', x,yr,'o', x,y,'b-')
plt.show()

# uses a simplex algorithm
p_est2 = optimize.fmin(e, p0, args=(x,yr))
print("Parameters estimated with the simplex algorithm:",p_est2)
y_est2 = f(p_est2, x)

plt.plot(x,y_est2,'r-', x,yr,'o', x,y,'b-')
plt.show()
```

Exercises:
- Use a different nonlinear function.
- Define a normal Python function f() instead!
- Improve the LS fit by using nonstandart loss functions (soft_l1, cauchy)
- Improve the LS fit by using different methods {‘dogbox’, ‘lm’}

## Gradient descent

Note that least squares is not an optimization method per se, it is a method to frame linear regression in the terms of an optimization problem. Gradient descent is the basis optimizatin method for most of modern machine learning, and any processor software today is judged by its speed to compute gradient descent. On GPUs, it sits as the foundation for Deep Learning and Reinforcement Learning.

The method is making an iterative walk in the direction of the local gradient, until the step size becomes :
$$\mathbf{a}_{n+1} = \mathbf{a}_n-\gamma\nabla F(\mathbf{a}_n)$$

Here is the naive algorithm, adapted from Wikipedia:

```python
%matplotlib inline
import numpy as np
import pylab as plt

cur_x = 6 # The algorithm starts at x=6
gamma = 0.01 # step size multiplier
precision = 0.00001
previous_step_size = 1/precision; # some large value

f = lambda x: x**4 - 3 * x**3 + 2
df = lambda x: 4 * x**3 - 9 * x**2

x = np.linspace(-4,cur_x,100)
while previous_step_size > precision:
    prev_x = cur_x
    cur_x += -gamma * df(prev_x)
    previous_step_size = abs(cur_x - prev_x)

print("The local minimum occurs at %f" % cur_x)

plt.plot(x,f(x),'b-')
```

The naive implementations suffer from many downsides, from speed to oscilating accross a valey. Another typical fault of the naive approach is assuming that the functions to be optimized are smooth, with expected variation for small variations in their parameters. Here is an example of how to compute the gradient descent with scipy, for the rosenbrock function, a function known to be ill-conditioned.

But before, here is a [practical advice from Scipy](https://www.scipy-lectures.org/advanced/mathematical_optimization/index.html):

- Gradient not known:
> In general, prefer BFGS or L-BFGS, even if you have to approximate numerically gradients. These are also the default if you omit the parameter method - depending if the problem has constraints or bounds. On well-conditioned problems, Powell and Nelder-Mead, both gradient-free methods, work well in high dimension, but they collapse for ill-conditioned problems.
- With knowledge of the gradient:
> BFGS or L-BFGS. Computational overhead of BFGS is larger than that L-BFGS, itself larger than that of conjugate gradient. On the other side, BFGS usually needs less function evaluations than CG. Thus conjugate gradient method is better than BFGS at optimizing computationally cheap functions.
With the Hessian:
- If you can compute the Hessian, prefer the Newton method (Newton-CG or TCG).
- If you have noisy measurements: Use Nelder-Mead or Powell.

```python
import numpy as np
import scipy.optimize as optimize

def f(x):   # The rosenbrock function
    return .5*(1 - x[0])**2 + (x[1] - x[0]**2)**2
def fprime(x):
    return np.array((-2*.5*(1 - x[0]) - 4*x[0]*(x[1] - x[0]**2), 2*(x[1] - x[0]**2)))
print(optimize.fmin_ncg(f, [2, 2], fprime=fprime))

def hessian(x): # Computed with sympy
    return np.array(((1 - 4*x[1] + 12*x[0]**2, -4*x[0]), (-4*x[0], 2)))
print(optimize.fmin_ncg(f, [2, 2], fprime=fprime, fhess=hessian))

%matplotlib inline
from matplotlib import cm
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.gca(projection='3d')
X = np.arange(-1, 1, 0.005)
Y = np.arange(-1, 1, 0.005)
X, Y = np.meshgrid(X, Y)
Z = .5*(1 - X)**2 + (Y - X**2)**2
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm,
        linewidth=0, antialiased=False)
#ax.set_zlim(-1000.01, 1000.01)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

## Constraint optimization

Problem definition:

$$
\begin{array}{rcll}
\min &~& f(\mathbf{x}) & \\
\mathrm{subject~to} &~& g_i(\mathbf{x}) = c_i &\text{for } i=1,\ldots,n \quad \text{Equality constraints} \\
 &~& h_j(\mathbf{x}) \geqq d_j &\text{for } j=1,\ldots,m \quad \text{Inequality constraints}
\end{array}
$$

Let's take the particular case when the objective function and the constraints linear, such as in the cannonical form:

$$\begin{align}
& \text{maximize}   && \mathbf{c}^\mathrm{T} \mathbf{x}\\
& \text{subject to} && A \mathbf{x} \leq \mathbf{b} \\
& \text{and} && \mathbf{x} \ge \mathbf{0}
\end{align}$$

Scipy has methods for optimizing functions under constraints, including linear programming. Aditionally many (linear or nonlinear) constraint optimization problems can be turned into full optimization problems using Lagrange multipliers. We will learn how to run linear problems with PuLP.

```python
"""
maximize: 4x + 3y

x > 0
y >= 2

x + 2y <= 25
2x - 4y <= 8
-2x + y <= -5
"""

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

x = np.linspace(0, 20, 2000)
y1 = (x*0) + 2
y2 = (25-x)/2.0
y3 = (2*x-8)/4.0
y4 = 2 * x -5

# Make plot
plt.plot(x, y1, label=r'$y\geq2$')
plt.plot(x, y2, label=r'$2y\leq25-x$')
plt.plot(x, y3, label=r'$4y\geq 2x - 8$')
plt.plot(x, y4, label=r'$y\leq 2x-5$')
plt.xlim((0, 16))
plt.ylim((0, 11))
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')

# Fill feasible region
y5 = np.minimum(y2, y4)
y6 = np.maximum(y1, y3)
plt.fill_between(x, y5, y6, where=y5>y6, color='grey', alpha=0.5)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
```

The solution to an optimization problem with linear constraints lies somewhere on the gray area, and it is not necessarily unique. However for linear optimization functions the solution is on one of the vertices. Now let us frame it with PuLP:

```
conda install -c conda-forge pulp
```

```python
import pulp
my_lp_problem = pulp.LpProblem("My LP Problem", pulp.LpMaximize)

x = pulp.LpVariable('x', lowBound=0, cat='Continuous')
y = pulp.LpVariable('y', lowBound=2, cat='Continuous')

# Objective function
my_lp_problem += 4 * x + 3 * y, "Z"

# Constraints
my_lp_problem += 2 * y <= 25 - x
my_lp_problem += 4 * y >= 2 * x - 8
my_lp_problem += y <= 2 * x - 5

my_lp_problem
```

    My LP Problem:
    MAXIMIZE
    4*x + 3*y + 0
    SUBJECT TO
    _C1: x + 2 y <= 25

    _C2: - 2 x + 4 y >= -8

    _C3: - 2 x + y <= -5

    VARIABLES
    x Continuous
    2 <= y Continuous

```python
my_lp_problem.solve()
print(pulp.LpStatus[my_lp_problem.status])
for variable in my_lp_problem.variables():
    print("{} = {}".format(variable.name, variable.varValue))
print(pulp.value(my_lp_problem.objective))
```

    Optimal
    x = 14.5
    y = 5.25
    73.75

Further complications arise when some of the constraints need to be integers, in which case the problem becomes known as Mixed Integer Liniar Programming and it computationally more expensive. Yet such problems are quite frequent, for example in metabolic engineering, where you need to deal machines operating on discrete intervals, or when studying protein folding or DNA recombination. In such cases one can also install python packages that deal with the specific problem, such as [cobrapy](https://cobrapy.readthedocs.io/en/latest/) for metabolic engineering.

Further reading:
- For LP with PuLP, I recommend this tutorial, which also uses some real life problems, and has a github link for the notebooks: [http://benalexkeen.com/linear-programming-with-python-and-pulp/](http://benalexkeen.com/linear-programming-with-python-and-pulp/)
- For a great list of LP solvers check [https://stackoverflow.com/questions/26305704/python-mixed-integer-linear-programming](https://stackoverflow.com/questions/26305704/python-mixed-integer-linear-programming)

## Global optimization

The most computation efficient optimization methods can only find local optima, while using different heuristics when needing to access global optima. In the class of global optimization algorithms a few methods to mention are:

- Grid search: These methods belong to the class of brute force or greedy searches and check for solutions in multidimensional sollution spaces. Typicall employed when having to find optimal parameter combinations for machine learning problems (hyperparametrization)
- Branch and Bound: This method, belonging to the more general class called dynamical programming, uses an optimal rooted tree, thus breaking the problem into smaller local optima problems. It is used in LP/MILP for example, or by sequence alignment programs such as BLAST.
- Monte Carlo: This method belongs to the stochastic optimization class, that instead of looking for an exact fit is using random sampling and bayesian statistics. These methods are expected to gain more traction with the evolution of computing power.
- Heuristics: Many methods in this class are nature inspired, such as genetic programming, ant colony optimization etc.

```python
import sys
sys.version

```

    '3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 18:21:58) \n[GCC 7.2.0]'

```python
sys.path

```

    ['',
     '/home/sergiu/programs/miniconda3/envs/pycourse/lib/python36.zip',
     '/home/sergiu/programs/miniconda3/envs/pycourse/lib/python3.6',
     '/home/sergiu/programs/miniconda3/envs/pycourse/lib/python3.6/lib-dynload',
     '/home/sergiu/programs/miniconda3/envs/pycourse/lib/python3.6/site-packages',
     '/home/sergiu/programs/miniconda3/envs/pycourse/lib/python3.6/site-packages/IPython/extensions',
     '/home/sergiu/.ipython']

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
```

```python
import pandas as pd
```

# Motivation

I have downloaded prices of flats in Wrocław (the web-scraping code is in the notebook `03-scrape-house-prices`).

The scatter plot of the `price` vs the `area` of the flat reveals that there is a strong dependency between price and area. Linear regression finds a *linear* relation between them.

```python
house_prices = pd.read_csv('03-house-prices.csv',index_col=0)

house_prices.head()
```

      rooms
      area
      price

      0
      3
      66.80
      367.801

      1
      3
      55.03
      289.458

      2
      4
      70.80
      389.000

      3
      1
      29.02
      195.695

      4
      2
      45.76
      265.754

House area and price seem to be correlated:

```python
house_prices = house_prices[:100]

_ = scatter(house_prices.area, house_prices.price, alpha=0.3)
_ = xlabel('area m2')
_ = ylabel('price kPLN')
```

## Linear regression analytical solution

We will now find the linear relation between house area and price.

Our data forms a table:

```python
house_prices.head()
```

      rooms
      area
      price

      0
      3
      66.80
      367.801

      1
      3
      55.03
      289.458

      2
      4
      70.80
      389.000

      3
      1
      29.02
      195.695

      4
      2
      45.76
      265.754

## Overview of learning

We have gathered a dataset consisting of pairs $\{(x^{(i)}, y^{(i)}), i=1\ldots N \}$. We want to find the relation between $x$, the *independent* variable, area in our example and $y$, the *dependent* variable, price in the example.

We will define a family of *hypotheses* (functions) parameterized by a vector of parameters $\Theta$.

$$ y \approx h(x; \Theta). $$

To *learn* the relation between areas and prices, or in general between $x$ and $y$, we will search the space of hypotheses (by changing the parameters $\Theta$) to find the one whose predictions best match the true targets.

We will need a *cost function* to measure the goodness of fit between hypotheses and the data.

Finally, we will need a training (learning) algorithm that will perform the search for us.

## The hypothesis

We will start with a simple linear family of hypotheses:
$$ h(x;\Theta) = \Theta_0 + \Theta_1 x. $$

## The cost function

We will measure the discrepancy between predictions of a hypothesis and the true $y$ on the dataset by computing the sum of squares of errors:

$$ J(\Theta) = \sum_{i=1}^{N} (y^{(i)}-h(x^{(i)};\Theta))^2 $$

## The training algorithm

We will first find the optimal $\Theta$ by deriving the analytical solution. However we will first generalize the problem a little, by assuming that $x$ can be a vector of values, $x=[x_1,\ldots,x_m]$. We will also assume, that $x_0=1$. Therefore:

$$ h(x;\Theta) = \Theta{}_0 + \sum_{j=1}^m \Theta_j x_j = \sum_{j=0}^m \Theta_j x_j = \Theta^T x $$

The cost function then becomes

$$ J(\Theta) = \sum_{i=1}^N (y^{(i)} - \Theta^T x^{(i)})^2 $$

We will make even greater use of matrix calculus. If each data sample $x$ is a vector, we can stack them into a large matrix $X \in \mathbb{R}^{m\times N}$, such that the $i$-th column of $X$ is the sample $x^{(i)}$ likewise we will concatenate all targets $y^{(i)}$ into a matrix $Y\in\mathbb{R}^{1 \times N}$. Armed with this notation, we can compute the predictions on all data points as $\Theta^T X$ and compute the matrix of errors $E\in \mathbb{R}^{1 \times N}$ as:

$$ E = \Theta^T X - Y. $$

Finally, please note that the cost function becomes simply:

$$ J(\Theta) = E E^T = (\Theta^T X - Y)(\Theta^T X - Y)^T $$

We will compute the *gradient* of $J$ with respect to $\Theta$. By definition, we will assume that the gradient is also a vector of the same shape as $\Theta$. To quickly compute the gradient, we will use the matrix calculus relations described here: http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/calculus.html and here: http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3274/pdf/imm3274.pdf:

$$ \frac{\partial J(\Theta)}{\partial\Theta} = \frac{\partial}{\partial\Theta} (\Theta^T X - Y)(\Theta^T X - Y)^T =
\frac{\partial}{\partial\Theta} [\Theta^T X X^T \Theta - YX^T\Theta - \Theta^T X Y^T + YY^T] =
2 X X^T \Theta -2 X Y^T  = 2X(X^T\Theta - Y^T) = 2X E^T$$

Solving for the zero-crossing of the derivative we obtain:

$$
\begin{align}
2 X X^T \Theta &= 2 X Y^T \\
\Theta &= (X X^T) ^{-1} X Y^T
\end{align}
$$

Please note: this is also called the Moore-Penrose pseudo-inverse of matrix $X$.

## Example of operation

```python
X = np.vstack((np.ones_like(house_prices.area), house_prices.area))
X[:,:10]
```

    Shape: (2, 10)
    [[  1.     1.     1.     1.     1.     1.     1.     1.     1.     1.  ]
     [ 66.8   55.03  70.8   29.02  45.76  38.99  55.7   76.42  37.6   59.51]]

```python
Y = np.asarray(house_prices.price)[None,:]
Y
```

    Shape: (1, 100)
    [[ 367.801  289.458  389.     195.695  265.754  236.614  317.967  409.
       227.211  351.704  259.893  675.615  413.699  302.305  314.908  389.
       270.     527.     347.     390.     439.946  390.     895.     730.
       313.5    251.     289.9    384.     417.7    421.5    285.     374.
       455.     604.7    339.     359.9    545.     239.729  333.196  320.737
       285.     365.     240.     359.     410.     429.     315.     269.
       305.     270.5    245.     299.     183.     300.     489.9    595.
       207.928  238.     349.9    185.     323.4    279.     320.     245.2
       249.     257.     310.     494.     244.57   208.8    185.13   197.
       380.     340.     361.     425.     399.     499.     465.     188.
       276.85   340.     330.     295.     369.     444.     444.     494.
       499.     504.     385.     325.     424.427  650.76   210.     279.
       306.735  349.     466.5    524.064]]

Remember, $\Theta = (X X^T)^{-1} X Y^T$.

```python
Theta = np.linalg.inv(X.dot(X.T)).dot(X).dot(Y.T)
Theta
```

    Shape: (2, 1)
    [[ 53.89879917]
     [  4.72624117]]

```python
scatter(house_prices.area, house_prices.price, alpha=0.3, label='data')

test_areas = np.linspace(xlim()[0], xlim()[1], 512)
test_prices = Theta[0] + Theta[1]*test_areas

plot(test_areas, test_prices,'g', label='lin. regression')
xlim(test_areas[0], test_areas[-1])
xlabel('area m2')
ylabel('price kPLN')
text(150, 500, "$y=%.2fx+%.2f$" % (Theta[1], Theta[0]))
legend()
```

## Interpretation of the least squares criterion

We will now work out a *probabilistic interpretation* of our model. We will assume that the data is generated according to the following process:
1. $x$ is sampled
2. $y$ is computed as $y=\Theta^T x + \epsilon$, where $\epsilon \sim \mathcal{N}(\mu=0,\sigma)$

We assume that the dependent variable $y$ is produced by linearly transforming $x$ and adding a little bit of Gaussian noise. The intuition is that there is the main linear relation that we model, and all other phenomena add up as noise, which by the central limit theorem we assume to be normal.

We assume that for each sample in our dataset the noise was sampled independently from the same noise distribution (IID). This allows us to write the following conditional distribution of $y$ given $x$:

$$ p(y|x;\Theta) = \mathcal{N}(y; \mu=\Theta^T x, \sigma) $$

Due to the IID assumption for the noise variables $\epsilon^{(i)}$, we can also assume that the targets $y$ in our data aree conditionally independent given the $x$s. This allows us to easily compute the probability of observing the collected data under our data-generating hypothesis as:

$$ p(Data; \Theta) = \prod_{i=1}^N p(y^{(i)}|x^{(i)};\Theta) = \prod_{i=1}^N \mathcal{N}(y^{(i)}; \mu=\Theta^T x^{(i)}, \sigma) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(y^{(i)}-\Theta^Tx^{(i)})^2}{2\sigma^2}\right)$$

We will use the maximum likelihood principle to choose $\Theta$. More specifically, this time we will minimize the *negative log-likelihood*, written in short nll:

$$
\begin{align}
nll(\Theta) &= - \log p(Data; \Theta) \\
&= -\sum_{i=1}^{N}\log \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(y^{(i)}-\Theta^Tx^{(i)})^2}{2\sigma^2}\right)  \\
&= -\sum_{i=1}^{N}\log \frac{1}{\sqrt{2\pi}\sigma} - \frac{(y^{(i)}-\Theta^Tx^{(i)})^2}{2\sigma^2} \\
&= C + \frac{1}{2\sigma^2}\sum_{i=1}^{N}(y^{(i)}-\Theta^Tx^{(i)})^2
\end{align}
$$

Note that minimization of nll leads to maximization of the likelihood. Therefore the Gaussian noise assumption leads to the least squares cost function!

## Validity of the Gaussian noise assumption

Statisticians like to check if the residuals (the errors made by our model on the train set) follow indeed the assumed noise distribution. In the house prices example we can see that the residuals do not strictly follow the Gaussian distribution. Maybe we should take into account another house feature, such as neighborhood, age, whether it is a new or used one... Think how you would incorporate such new information. Is it still possible to use linear regression to fit a model dependent on many variables?

In the following example the Gaussian noise assumption will be satisfied. We will download data of people's height vs weight.

```python
# url = './03-SOCR_Data_Dinov_020108_HeightsWeights.html'
url = 'http://socr.ucla.edu/docs/resources/SOCR_Data/SOCR_Data_Dinov_020108_HeightsWeights.html'
height_weight = pd.read_html(url)[0]
```

```python
height_weight.head()
```

      0
      1
      2

      0
      Index
      Height(Inches)
      Weight(Pounds)

      1
      1
      65.78331
      112.9925

      2
      2
      71.51521
      136.4873

      3
      3
      69.39874
      153.0269

      4
      4
      68.2166
      142.3354

```python
inch_to_meter = 0.0254
height = np.asarray(height_weight.iloc[1:,1].astype(float))[None,:] * inch_to_meter #convert to meters
pound_to_kg = 0.453592
weight = np.asarray(height_weight.iloc[1:,2].astype(float))[None,:] * pound_to_kg # convert to kg
```

```python
scatter(height, weight, alpha=0.2, linewidths=0)
xlabel('height [m]')
ylabel('weight [kg]')
```

```python
X = np.vstack((np.ones_like(height), height))
X
```

    Shape: (2, 25000)
    [[ 1.          1.          1.         ...,  1.          1.          1.        ]
     [ 1.67089607  1.81648633  1.762728   ...,  1.64334317  1.71524117
       1.74949129]]

```python
Y = weight
Y
```

    Shape: (1, 25000)
    [[ 51.25249406  61.90954738  69.41177762 ...,  53.64428468  59.99579737
       56.64193813]]

```python
Theta = np.linalg.inv(X.dot(X.T)).dot(X).dot(Y.T)
Theta
```

    Shape: (2, 1)
    [[-37.45569645]
     [ 55.06457669]]

```python
scatter(height, weight, alpha=0.1)
test_heights = np.linspace(xlim()[0], xlim()[1], 512)
test_weights = Theta[0] + Theta[1]*test_heights

plot(test_heights, test_weights,'g')
xlim(test_heights[0], test_heights[-1])
xlabel('height [m]')
ylabel('weight [kg]')
```

```python
E = Theta.T.dot(X) - Y
E
```

    Shape: (1, 25000)
    [[ 3.29899449  0.65880721 -9.80360316 ..., -0.60998512 -3.00246477
       2.23736295]]

```python
scatter (X[1,:], E, alpha=0.1)
title("Scatter plot of residuals.")
```

```python
counts, bins, patches = hist(E.ravel(),100, normed=True)

mu = E.mean()
std = E.std()
e_range = np.linspace(bins[0], bins[-1], 100)
plot(e_range, 1/np.sqrt(2*np.pi)/std*np.exp(-e_range**2/2/std**2), 'r', linewidth=2)
title("Histogram of residuals\nNote the nice gaussian fit")
```

```python
# Analyze the histogram of errors along a few heights

slice_height = 1.85
selected = np.abs(X[1,:]-slice_height) < 0.05

selected_E = E[0,selected]
counts, bins, patches = hist(selected_E, 10, normed=True)

selected_mu = selected_E.mean()
selected_std = selected_E.std()
selected_e_range = np.linspace(bins[0], bins[-1], 100)
plot(selected_e_range, 1/np.sqrt(2*np.pi)/selected_std*np.exp(-(selected_mu-selected_e_range)**2/2/selected_std**2), 'r', linewidth=2)

title('Histogram of residuals for heights=%f +-1cm'
      'Slice residual mean: %f stdev: %f'% (slice_height,selected_mu, selected_std))
```

```python
# Are the errors homoscedastic?

err_stats = []

slice_width = 0.02

for slice_height in np.arange(height.min(), height.max(), slice_width):
    selected = np.abs(X[1,:]-slice_height) < slice_width/2.0

    selected_E = E[0,selected]

    selected_mu = selected_E.mean()
    selected_std = selected_E.std()
    err_stats.append((slice_height, selected_mu, selected_std))

err_stats = np.asarray(err_stats)

plot(err_stats[:,0], err_stats[:,1], label='mean of residuals in slice')
plot(err_stats[:,0], err_stats[:,2], label='stdev of residuals in slice')
legend(loc='lower right')
title("mean and standard deviation of residuals vs height")
xlabel("height")
```

```python
err_stats
```

    Shape: (19, 3)
    [[  1.53107034e+00   2.14386532e+00   5.23856017e+00]
     [  1.55107034e+00  -3.69963253e+00   2.29752329e+00]
     [  1.57107034e+00   5.59137546e+00   5.63231137e+00]
     [  1.59107034e+00  -1.42154057e-01   4.97655815e+00]
     [  1.61107034e+00  -3.35868391e-01   4.36489480e+00]
     [  1.63107034e+00  -9.43881477e-03   4.65754915e+00]
     [  1.65107034e+00  -2.56189507e-02   4.70251580e+00]
     [  1.67107034e+00  -1.13242658e-01   4.61848993e+00]
     [  1.69107034e+00   7.56820063e-02   4.52490018e+00]
     [  1.71107034e+00  -3.49821388e-03   4.58388038e+00]
     [  1.73107034e+00   1.24413620e-01   4.56959502e+00]
     [  1.75107034e+00  -1.38191096e-02   4.56081512e+00]
     [  1.77107034e+00  -1.83136847e-01   4.56016427e+00]
     [  1.79107034e+00   7.44721543e-02   4.52788329e+00]
     [  1.81107034e+00  -2.09913184e-01   4.50193216e+00]
     [  1.83107034e+00   4.32546349e-01   4.41835785e+00]
     [  1.85107034e+00   7.55783464e-04   4.54527322e+00]
     [  1.87107034e+00   3.93371931e-01   4.22798409e+00]
     [  1.89107034e+00   1.83819411e-02   4.74623216e+00]]

```python

```


# Double Pendulum #

In this notebook, we are going to develop the EoMs for Double Pendulum system. We are going to derive EoMs, simulate them and finally visualize them.

The code is loaded in the notebook using `%load` IPython magic, to avoid duplication of the code.

```python
# %load double_pendulum.py
from sympy import symbols
from sympy.physics.mechanics import *

q1, q2 = dynamicsymbols('q1 q2')
q1d, q2d = dynamicsymbols('q1 q2', 1)
u1, u2 = dynamicsymbols('u1 u2')
u1d, u2d = dynamicsymbols('u1 u2', 1)
l, m, g = symbols('l m g')

N = ReferenceFrame('N')
A = N.orientnew('A', 'Axis', [q1, N.z])
B = N.orientnew('B', 'Axis', [q2, N.z])

A.set_ang_vel(N, u1 * N.z)
B.set_ang_vel(N, u2 * N.z)

O = Point('O')
P = O.locatenew('P', l * A.x)
R = P.locatenew('R', l * B.x)

O.set_vel(N, 0)
P.v2pt_theory(O, N, A)
R.v2pt_theory(P, N, B)

ParP = Particle('ParP', P, m)
ParR = Particle('ParR', R, m)

kd = [q1d - u1, q2d - u2]
FL = [(P, m * g * N.x), (R, m * g * N.x)]
BL = [ParP, ParR]

KM = KanesMethod(N, q_ind=[q1, q2], u_ind=[u1, u2], kd_eqs=kd)

KM.kanes_equations(FL, BL)
#kdd = KM.kindiffdict()
#mass_matrix = KM.mass_matrix_full
#forcing_vector = KM.forcing_full
#qudots = mass_matrix.inv() * forcing_vector
#qudots = qudots.subs(kdd)
#qudots.simplify()
#

```

    (Matrix([
     [-2*g*l*m*sin(q1(t))],
     [  -g*l*m*sin(q2(t))]]), Matrix([
     [-l**2*m*(sin(q1(t))*sin(q2(t)) + cos(q1(t))*cos(q2(t)))*Derivative(u2(t), t) + l**2*m*(-sin(q1(t))*cos(q2(t)) + sin(q2(t))*cos(q1(t)))*u2(t)**2 - 2*l**2*m*Derivative(u1(t), t)],
     [   -l**2*m*(sin(q1(t))*sin(q2(t)) + cos(q1(t))*cos(q2(t)))*Derivative(u1(t), t) + l**2*m*(sin(q1(t))*cos(q2(t)) - sin(q2(t))*cos(q1(t)))*u1(t)**2 - l**2*m*Derivative(u2(t), t)]]))

```python
# %load simulate.py
"""
This file will use pydy.codegen to simulate the double pendulum.

"""

from numpy import linspace
from pydy.system import System

from double_pendulum import *

constants = {l: 10.0, m: 10.0, g: 9.81}

initial_conditions = {q1: 1.0, q2: 0.0, u1: 0.0, u2: 0.0}

sys = System(KM, constants=constants,
        initial_conditions=initial_conditions)

frames_per_sec = 60
final_time = 5.0

times = linspace(0.0, final_time, final_time * frames_per_sec)
sys.times = times
x = sys.integrate()

```

    /home/moorepants/miniconda3/envs/pydy-examples-py35/lib/python3.5/site-packages/ipykernel_launcher.py:22: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
# %load visualize.py
"""
This file will use pydy.viz to visualize the double pendulum.  Run this script
via a command line:

    $ python visualization.py

"""

from numpy import pi

from pydy.viz.shapes import Cylinder, Sphere
from pydy.viz.scene import Scene
from pydy.viz.visualization_frame import VisualizationFrame

from simulate import *

# Create geometry
# ===============

# Each link in the pendulum is visualized with a cylinder, and a sphere at its
# far end.
link = Cylinder(name='link', radius=0.5, length=l, color='red')
sphere = Sphere(name='sphere', radius=1.0)

# By default, Cylinders are drawn so that their center is at the origin of the
# VisualizationFrame, and their axis is the y axis of the VisualizationFrame.
# We want the end of the Cylinder to be at the origin of the
# VisualizationFrame, and we want the Cylinder's axis to be aligned with the x
# axis of the VisualizationFrame. For these reasons, we must use the
# 'orientnew' and 'locatenew' methods to create new frames/points.
linkP_frame = A.orientnew('frameP', 'Axis', [0.5 * pi, N.z])
linkP_origin = O.locatenew('originP', 0.5 * l * A.x)
linkP_viz_frame = VisualizationFrame('linkP', linkP_frame, linkP_origin, link)

linkR_frame = B.orientnew('frameR', 'Axis', [0.5 * pi, N.z])
linkR_origin = P.locatenew('originP', 0.5 * l * B.x)
linkR_viz_frame = VisualizationFrame('linkR', linkR_frame, linkR_origin, link)

sphereP_viz_frame = VisualizationFrame('sphereP', N, P, sphere)
sphereR_viz_frame = VisualizationFrame('sphereR', N, R, sphere)

# Construct the scene
# ===================

# We want gravity to be directed downwards in the visualization. Gravity is in
# the -x direction. By default, the visualization uses the xz plane as the
# ground plane. Thus, gravity is contained in the ground plane. However, we
# want gravity to point in the -y direction in the visualization. To achieve
# this, we create a world frame that is rotated +90 degrees about the N frame's
# z direction.
world_frame = N.orientnew('world', 'Axis', [0.5 * pi, N.z])
scene = Scene(world_frame, O,
        linkP_viz_frame, linkR_viz_frame, sphereP_viz_frame, sphereR_viz_frame)

# Create the visualization
# ========================

scene.generate_visualization_json_system(sys)

if __name__ == "__main__":
    try: #If called from inside notebook,
        scene.display_ipython()
    except:#If called from interpreter
        scene.display()

```

```python
%load_ext version_information
```

```python
%version_information numpy, scipy, pydy
```

Software	Version

Python	3.5.3 64bit [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

IPython	4.2.0

OS	Linux 4.10.0 21 generic x86_64 with debian stretch sid

numpy	1.12.1

scipy	0.19.0

pydy	0.4.0dev

Mon May 29 13:31:36 2017 PDT


# Programming Exercise 5:

# Regularized Linear Regression and Bias vs Variance

## Introduction

In this exercise, you will implement regularized linear regression and use it to study models with different bias-variance properties. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.

```python
# used for manipulating directory paths
import os

# Scientific and vector computation for python
import numpy as np

# Plotting library
from matplotlib import pyplot

# Optimization module in scipy
from scipy import optimize

# will be used to load MATLAB mat datafile format
from scipy.io import loadmat

# tells matplotlib to embed plots within the notebook
%matplotlib inline
```

## 1 Regularized Linear Regression

In the first half of the exercise, you will implement regularized linear regression to predict the amount of water flowing out of a dam using the change of water level in a reservoir. In the next half, you will go through some diagnostics of debugging learning algorithms and examine the effects of bias v.s. variance.

- A training set that your model will learn on: X, y
- A cross validation set for determining the regularization parameter: Xval, yval
- A test set for evaluating performance. These are “unseen” examples which your model did not see during training: Xtest, ytest

Run the next cell to plot the training data. In the following parts, you will implement linear regression and use that to fit a straight line to the data and plot learning curves. Following that, you will implement polynomial regression to find a better fit to the data.

### 1.1 Visualizing the dataset

```python
# Load from ex5data1.mat, where all variables will be store in a dictionary
data = loadmat(os.path.join('Data', 'ex5data1.mat'))

# Extract train, test, validation data from dictionary
# and also convert y's form 2-D matrix (MATLAB format) to a numpy vector
X, y = data['X'], data['y'][:, 0]
Xtest, ytest = data['Xtest'], data['ytest'][:, 0]
Xval, yval = data['Xval'], data['yval'][:, 0]

# m = Number of examples
m = y.size

# Plot training data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)');
```

### 1.2 Regularized linear regression cost function

Recall that regularized linear regression has the following cost function:

$$ J(\theta) = \frac{1}{2m} \left( \sum_{i=1}^m \left( h_\theta\left( x^{(i)} \right) - y^{(i)} \right)^2 \right) + \frac{\lambda}{2m} \left( \sum_{j=1}^n \theta_j^2 \right)$$

where $\lambda$ is a regularization parameter which controls the degree of regularization (thus, help preventing overfitting). The regularization term puts a penalty on the overall cost J. As the magnitudes of the model parameters $\theta_j$ increase, the penalty increases as well. Note that you should not regularize the $\theta_0$ term.

You should now complete the code in the function linearRegCostFunction in the next cell. Your task is to calculate the regularized linear regression cost function. If possible, try to vectorize your code and avoid writing loops.

```python
def linearRegCostFunction(X, y, theta, lambda_=0.0):
    """
    Compute cost and gradient for regularized linear regression
    with multiple variables. Computes the cost of using theta as
    the parameter for linear regression to fit the data points in X and y.

    Parameters
    ----------
    X : array_like
        The dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each datapoint. A vector of
        shape (m, ).

    theta : array_like
        The parameters for linear regression. A vector of shape (n+1,).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    J : float
        The computed cost function.

    grad : array_like
        The value of the cost function gradient w.r.t theta.
        A vector of shape (n+1, ).

    Instructions
    ------------
    Compute the cost and gradient of regularized linear regression for
    a particular choice of theta.
    You should set J to the cost and grad to the gradient.
    """
    # Initialize some useful values
    m = y.size # number of training examples

    # You need to return the following variables correctly
    J = 0
    grad = np.zeros(theta.shape)

    # ====================== YOUR CODE HERE ======================
    h = np.dot(X, theta)
    reg_term = lambda_/(2*m) * np.sum(np.square(theta[1:]))
    J = 1/(2*m) * np.sum(np.square(h - y)) + reg_term

    grad[0] = (1 / m) * np.dot(X[:, 0].T, (h - y))
    grad[1:] = ((1 / m) * np.dot(X[:, 1:].T, (h - y))) + ((lambda_ / m) * theta[1:])

    # ============================================================
    return J, grad
```

When you are finished, the next cell will run your cost function using theta initialized at [1, 1]. You should expect to see an output of 303.993.

```python
theta = np.array([1, 1])
J, _ = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Cost at theta = [1, 1]:\t   %f ' % J)
print('This value should be about 303.993192)\n' % J)
```

    Cost at theta = [1, 1]:	   303.993192
    This value should be about 303.993192)

### 1.3 Regularized linear regression gradient

Correspondingly, the partial derivative of the cost function for regularized linear regression is defined as:

$$
\begin{align}
& \frac{\partial J(\theta)}{\partial \theta_0} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left(x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} & \qquad \text{for } j = 0 \\
& \frac{\partial J(\theta)}{\partial \theta_j} = \left( \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left( x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} \right) + \frac{\lambda}{m} \theta_j & \qquad \text{for } j \ge 1
\end{align}
$$

In the function linearRegCostFunction above, add code to calculate the gradient, returning it in the variable grad.

When you are finished, use the next cell to run your gradient function using theta initialized at [1, 1]. You should expect to see a gradient of [-15.30, 598.250].

```python
theta = np.array([1, 1])
J, grad = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Gradient at theta = [1, 1]:  [{:.6f}, {:.6f}] '.format(*grad))
print(' (this value should be about [-15.303016, 598.250744])\n')
```

    Gradient at theta = [1, 1]:  [-15.303016, 598.250744]
     (this value should be about [-15.303016, 598.250744])

### Fitting linear regression

Once your cost function and gradient are working correctly, the next cell will run the code in trainLinearReg to compute the optimal values of $\theta$. This training function uses scipy's optimization module to minimize the cost function.

In this part, we set regularization parameter $\lambda$ to zero. Because our current implementation of linear regression is trying to fit a 2-dimensional $\theta$, regularization will not be incredibly helpful for a $\theta$ of such low dimension. In the later parts of the exercise, you will be using polynomial regression with regularization.

Finally, the code in the next cell should also plot the best fit line, which should look like the figure below.

```python
def trainLinearReg(linearRegCostFunction, X, y, lambda_=0.0, maxiter=200):
    """
    Trains linear regression using scipy's optimize.minimize.
    Parameters
    ----------
    X : array_like
        The dataset with shape (m x n+1). The bias term is assumed to be concatenated.
    y : array_like
        Function values at each datapoint. A vector of shape (m,).
    lambda_ : float, optional
        The regularization parameter.
    maxiter : int, optional
        Maximum number of iteration for the optimization algorithm.
    Returns
    -------
    theta : array_like
        The parameters for linear regression. This is a vector of shape (n+1,).
    """
    # Initialize Theta
    initial_theta = np.zeros(X.shape[1])

    # Create "short hand" for the cost function to be minimized
    costFunction = lambda t: linearRegCostFunction(X, y, t, lambda_)

    # Now, costFunction is a function that takes in only one argument
    options = {'maxiter': maxiter}

    # Minimize using scipy
    res = optimize.minimize(costFunction, initial_theta, jac=True, method='TNC', options=options)
    return res.x
```

```python
# add a columns of ones for the y-intercept
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
theta = trainLinearReg(linearRegCostFunction, X_aug, y, lambda_=0)

#  Plot fit over the data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1.5)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.plot(X, np.dot(X_aug, theta), '--', lw=2)
```

The best fit line tells us that the model is not a good fit to the data because the data has a non-linear pattern. While visualizing the best fit as shown is one possible way to debug your learning algorithm, it is not always easy to visualize the data and model. In the next section, you will implement a function to generate learning curves that can help you debug your learning algorithm even if it is not easy to visualize the data.

## 2 Bias-variance

An important concept in machine learning is the bias-variance tradeoff. Models with high bias are not complex enough for the data and tend to underfit, while models with high variance overfit to the training data.

In this part of the exercise, you will plot training and test errors on a learning curve to diagnose bias-variance problems.

### 2.1 Learning Curves

You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve plots training and cross validation error as a function of training set size. Your job is to fill in the function learningCurve in the next cell, so that it returns a vector of errors for the training set and cross validation set.

To plot the learning curve, we need a training and cross validation set error for different training set sizes. To obtain different training set sizes, you should use different subsets of the original training set X. Specifically, for a training set size of $i$, you should use the first  $i$ examples (i.e., X[:i, :] and y[:i]).

You can use the trainLinearReg function (by calling utils.trainLinearReg(...)) to find the $\theta$ parameters. Note that the lambda_ is passed as a parameter to the learningCurve function. After learning the $\theta$ parameters, you should compute the error on the training and cross validation sets. Recall that the training error for a dataset is defined as

$$ J_{\text{train}} = \frac{1}{2m} \left[ \sum_{i=1}^m \left(h_\theta \left( x^{(i)} \right) - y^{(i)} \right)^2 \right] $$

In particular, note that the training error does not include the regularization term. One way to compute the training error is to use your existing cost function and set $\lambda$ to 0 only when using it to compute the training error and cross validation error. When you are computing the training set error, make sure you compute it on the training subset (i.e., X[:n,:] and y[:n]) instead of the entire training set. However, for the cross validation error, you should compute it over the entire cross validation set. You should store the computed errors in the vectors error train and error val.

```python
def learningCurve(X, y, Xval, yval, lambda_=0):
    """
    Generates the train and cross validation set errors needed to plot a learning curve
    returns the train and cross validation set errors for a learning curve.

    In this function, you will compute the train and test errors for
    dataset sizes from 1 up to m. In practice, when working with larger
    datasets, you might want to do this in larger intervals.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    error_train : array_like
        A vector of shape m. error_train[i] contains the training error for
        i examples.
    error_val : array_like
        A vecotr of shape m. error_val[i] contains the validation error for
        i training examples.

    Instructions
    ------------
    Fill in this function to return training errors in error_train and the
    cross validation errors in error_val. i.e., error_train[i] and
    error_val[i] should give you the errors obtained after training on i examples.

    Notes
    -----
    - You should evaluate the training error on the first i training
      examples (i.e., X[:i, :] and y[:i]).

      For the cross-validation error, you should instead evaluate on
      the _entire_ cross validation set (Xval and yval).

    - If you are using your cost function (linearRegCostFunction) to compute
      the training and cross validation error, you should call the function with
      the lambda argument set to 0. Do note that you will still need to use
      lambda when running the training to obtain the theta parameters.

    Hint
    ----
    You can loop over the examples with the following:

           for i in range(1, m+1):
               # Compute train/cross validation errors using training examples
               # X[:i, :] and y[:i], storing the result in
               # error_train[i-1] and error_val[i-1]
               ....
    """
    # Number of training examples
    m = y.size
    mval = Xval.shape[0]

    # You need to return these values correctly
    error_train = np.zeros(m)
    error_val   = np.zeros(m)

    # ====================== YOUR CODE HERE ======================
    for i in range(1, m+1):
        Xtrain = X[:i, :]
        ytrain = y[:i]

        theta = trainLinearReg(linearRegCostFunction, Xtrain, ytrain, lambda_)

        error_train[i-1] = 1/(2*i) * np.sum(np.square(Xtrain.dot(theta) - ytrain))
        error_val[i-1] = 1/(2*mval) * np.sum(np.square(Xval.dot(theta) - yval))

    # =============================================================
    return error_train, error_val
```

When you are finished implementing the function learningCurve, executing the next cell prints the learning curves and produce a plot similar to the figure below.

```python
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
Xval_aug = np.concatenate([np.ones((yval.size, 1)), Xval], axis=1)
error_train, error_val = learningCurve(X_aug, y, Xval_aug, yval, lambda_=0)

pyplot.plot(np.arange(1, m+1), error_train, np.arange(1, m+1), error_val, lw=2)
pyplot.title('Learning curve for linear regression')
pyplot.legend(['Train', 'Cross Validation'])
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 150])

print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

## 3 Polynomial regression

The problem with our linear model was that it was too simple for the data and resulted in underfitting (high bias). In this part of the exercise, you will address this problem by adding more features. For polynomial regression, our hypothesis has the form:

$$
\begin{align}
h_\theta(x)  &= \theta_0 + \theta_1 \times (\text{waterLevel}) + \theta_2 \times (\text{waterLevel})^2 + \cdots + \theta_p \times (\text{waterLevel})^p \\
& = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_p x_p
\end{align}
$$

Notice that by defining $x_1 = (\text{waterLevel})$, $x_2 = (\text{waterLevel})^2$ , $\cdots$, $x_p =
(\text{waterLevel})^p$, we obtain a linear regression model where the features are the various powers of the original value (waterLevel).

Now, you will add more features using the higher powers of the existing feature $x$ in the dataset. Your task in this part is to complete the code in the function polyFeatures in the next cell. The function should map the original training set $X$ of size $m \times 1$ into its higher powers. Specifically, when a training set $X$ of size $m \times 1$ is passed into the function, the function should return a $m \times p$ matrix  X_poly, where column 1 holds the original values of X, column 2 holds the values of $X^2$, column 3 holds the values of $X^3$, and so on. Note that you don’t have to account for the zero-eth power in this function.

```python
def polyFeatures(X, p):
    """
    Maps X (1D vector) into the p-th power.

    Parameters
    ----------
    X : array_like
        A data vector of size m, where m is the number of examples.

    p : int
        The polynomial power to map the features.

    Returns
    -------
    X_poly : array_like
        A matrix of shape (m x p) where p is the polynomial
        power and m is the number of examples. That is:

        X_poly[i, :] = [X[i], X[i]**2, X[i]**3 ...  X[i]**p]

    Instructions
    ------------
    Given a vector X, return a matrix X_poly where the p-th column of
    X contains the values of X to the p-th power.
    """
    # You need to return the following variables correctly.
    X_poly = np.zeros((X.shape[0], p))

    # ====================== YOUR CODE HERE ======================
    for i in range(p):
        X_poly[:, i] = np.power(X, i + 1).ravel()

    # ============================================================
    return X_poly
```

```python
def featureNormalize(X):
    """
    Normalizes the features in X returns a normalized version of X where the mean value of each
    feature is 0 and the standard deviation is 1. This is often a good preprocessing step to do when
    working with learning algorithms.
    Parameters
    ----------
    X : array_like
        An dataset which is a (m x n) matrix, where m is the number of examples,
        and n is the number of dimensions for each example.
    Returns
    -------
    X_norm : array_like
        The normalized input dataset.
    mu : array_like
        A vector of size n corresponding to the mean for each dimension across all examples.
    sigma : array_like
        A vector of size n corresponding to the standard deviations for each dimension across
        all examples.
    """
    mu = np.mean(X, axis=0)
    X_norm = X - mu

    sigma = np.std(X_norm, axis=0, ddof=1)
    X_norm /= sigma
    return X_norm, mu, sigma
```

Now you have a function that will map features to a higher dimension. The next cell will apply it to the training set, the test set, and the cross validation set.

```python
p = 8

# Map X onto Polynomial Features and Normalize
X_poly = polyFeatures(X, p)
X_poly, mu, sigma = featureNormalize(X_poly)
X_poly = np.concatenate([np.ones((m, 1)), X_poly], axis=1)

# Map X_poly_test and normalize (using mu and sigma)
X_poly_test = polyFeatures(Xtest, p)
X_poly_test -= mu
X_poly_test /= sigma
X_poly_test = np.concatenate([np.ones((ytest.size, 1)), X_poly_test], axis=1)

# Map X_poly_val and normalize (using mu and sigma)
X_poly_val = polyFeatures(Xval, p)
X_poly_val -= mu
X_poly_val /= sigma
X_poly_val = np.concatenate([np.ones((yval.size, 1)), X_poly_val], axis=1)

print('Normalized Training Example 1:')
X_poly[0, :]
```

    Normalized Training Example 1:

    array([ 1.        , -0.36214078, -0.75508669,  0.18222588, -0.70618991,
            0.30661792, -0.59087767,  0.3445158 , -0.50848117])

### 3.1 Learning Polynomial Regression

After you have completed the function polyFeatures, we will proceed to train polynomial regression using your linear regression cost function.

Keep in mind that even though we have polynomial terms in our feature vector, we are still solving a linear regression optimization problem. The polynomial terms have simply turned into features that we can use for linear regression. We are using the same cost function and gradient that you wrote for the earlier part of this exercise.

For this part of the exercise, you will be using a polynomial of degree 8. It turns out that if we run the training directly on the projected data, will not work well as the features would be badly scaled (e.g., an example with $x = 40$ will now have a feature $x_8 = 40^8 = 6.5 \times 10^{12}$). Therefore, you will need to use feature normalization.

Before learning the parameters $\theta$ for the polynomial regression, we first call featureNormalize and normalize the features of the training set, storing the mu, sigma parameters separately. We have already implemented this function for you and it is the same function from the first exercise.

After learning the parameters $\theta$, you should see two plots generated for polynomial regression with $\lambda = 0$, which should be similar to the ones here:

```python
def plotFit(polyFeatures, min_x, max_x, mu, sigma, theta, p):
    """
    Plots a learned polynomial regression fit over an existing figure.
    Also works with linear regression.
    Plots the learned polynomial fit with power p and feature normalization (mu, sigma).
    Parameters
    ----------
    polyFeatures : func
        A function which generators polynomial features from a single feature.
    min_x : float
        The minimum value for the feature.
    max_x : float
        The maximum value for the feature.
    mu : float
        The mean feature value over the training dataset.
    sigma : float
        The feature standard deviation of the training dataset.
    theta : array_like
        The parameters for the trained polynomial linear regression.
    p : int
        The polynomial order.
    """
    # We plot a range slightly bigger than the min and max values to get
    # an idea of how the fit will vary outside the range of the data points
    x = np.arange(min_x - 15, max_x + 25, 0.05).reshape(-1, 1)

    # Map the X values
    X_poly = polyFeatures(x, p)
    X_poly -= mu
    X_poly /= sigma

    # Add ones
    X_poly = np.concatenate([np.ones((x.shape[0], 1)), X_poly], axis=1)

    # Plot
    pyplot.plot(x, np.dot(X_poly, theta), '--', lw=2)
```

```python
lambda_ = 0
theta = trainLinearReg(linearRegCostFunction, X_poly, y,
                             lambda_=lambda_, maxiter=55)

# Plot training data and fit
pyplot.plot(X, y, 'ro', ms=10, mew=1.5, mec='k')

plotFit(polyFeatures, np.min(X), np.max(X), mu, sigma, theta, p)

pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.title('Polynomial Regression Fit (lambda = %f)' % lambda_)
pyplot.ylim([-20, 50])

pyplot.figure()
error_train, error_val = learningCurve(X_poly, y, X_poly_val, yval, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

You should see that the polynomial fit is able to follow the datapoints very well, thus, obtaining a low training error. The figure on the right shows that the training error essentially stays zero for all numbers of training samples. However, the polynomial fit is very complex and even drops off at the extremes. This is an indicator that the polynomial regression model is overfitting the training data and will not generalize well.

To better understand the problems with the unregularized ($\lambda = 0$) model, you can see that the learning curve shows the same effect where the training error is low, but the cross validation error is high. There is a gap between the training and cross validation errors, indicating a high variance problem.

### 3.2 Optional (ungraded) exercise: Adjusting the regularization parameter

In this section, you will get to observe how the regularization parameter affects the bias-variance of regularized polynomial regression. You should now modify the the lambda parameter and try $\lambda = 1, 100$. For each of these values, the script should generate a polynomial fit to the data and also a learning curve.

For $\lambda = 1$, the generated plots should look like the the figure below. You should see a polynomial fit that follows the data trend well (left) and a learning curve (right) showing that both the cross validation and training error converge to a relatively low value. This shows the $\lambda = 1$ regularized polynomial regression model does not have the high-bias or high-variance problems. In effect, it achieves a good trade-off between bias and variance.

```python
lambda_ = 1
theta = trainLinearReg(linearRegCostFunction, X_poly, y,
                             lambda_=lambda_, maxiter=55)

# Plot training data and fit
pyplot.plot(X, y, 'ro', ms=10, mew=1.5, mec='k')

plotFit(polyFeatures, np.min(X), np.max(X), mu, sigma, theta, p)

pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.title('Polynomial Regression Fit (lambda = %f)' % lambda_)
pyplot.ylim([-20, 50])

pyplot.figure()
error_train, error_val = learningCurve(X_poly, y, X_poly_val, yval, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

For $\lambda = 100$, you should see a polynomial fit (figure below) that does not follow the data well. In this case, there is too much regularization and the model is unable to fit the training data.

```python
lambda_ = 100
theta = trainLinearReg(linearRegCostFunction, X_poly, y,
                             lambda_=lambda_, maxiter=55)

# Plot training data and fit
pyplot.plot(X, y, 'ro', ms=10, mew=1.5, mec='k')

plotFit(polyFeatures, np.min(X), np.max(X), mu, sigma, theta, p)

pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.title('Polynomial Regression Fit (lambda = %f)' % lambda_)
pyplot.ylim([-20, 50])

pyplot.figure()
error_train, error_val = learningCurve(X_poly, y, X_poly_val, yval, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

### 3.3 Selecting $\lambda$ using a cross validation set

From the previous parts of the exercise, you observed that the value of $\lambda$ can significantly affect the results of regularized polynomial regression on the training and cross validation set. In particular, a model without regularization ($\lambda = 0$) fits the training set well, but does not generalize. Conversely, a model with too much regularization ($\lambda = 100$) does not fit the training set and testing set well. A good choice of $\lambda$ (e.g., $\lambda = 1$) can provide a good fit to the data.

In this section, you will implement an automated method to select the $\lambda$ parameter. Concretely, you will use a cross validation set to evaluate how good each $\lambda$ value is. After selecting the best $\lambda$ value using the cross validation set, we can then evaluate the model on the test set to estimate how well the model will perform on actual unseen data.

Your task is to complete the code in the function validationCurve. Specifically, you should should use the utils.trainLinearReg function to train the model using different values of $\lambda$ and compute the training error and cross validation error. You should try $\lambda$ in the following range: {0, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10}.

```python
def validationCurve(X, y, Xval, yval):
    """
    Generate the train and validation errors needed to plot a validation
    curve that we can use to select lambda_.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n) where m is the
        total number of training examples, and n is the number of features
        including any polynomial features.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n) where m is the
        total number of validation examples, and n is the number of features
        including any polynomial features.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    Returns
    -------
    lambda_vec : list
        The values of the regularization parameters which were used in
        cross validation.

    error_train : list
        The training error computed at each value for the regularization
        parameter.

    error_val : list
        The validation error computed at each value for the regularization
        parameter.

    Instructions
    ------------
    Fill in this function to return training errors in `error_train` and
    the validation errors in `error_val`. The vector `lambda_vec` contains
    the different lambda parameters to use for each calculation of the
    errors, i.e, `error_train[i]`, and `error_val[i]` should give you the
    errors obtained after training with `lambda_ = lambda_vec[i]`.

    Note
    ----
    You can loop over lambda_vec with the following:

          for i in range(len(lambda_vec))
              lambda = lambda_vec[i]
              # Compute train / val errors when training linear
              # regression with regularization parameter lambda_
              # You should store the result in error_train[i]
              # and error_val[i]
              ....
    """
    # Selected values of lambda (you should not change this)
    lambda_vec = [0, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10]

    # You need to return these variables correctly.
    error_train = np.zeros(len(lambda_vec))
    error_val = np.zeros(len(lambda_vec))

    m = y.size
    mval = Xval.shape[0]

    # ====================== YOUR CODE HERE ======================
    for i in range(len(lambda_vec)):
        lambda_ = lambda_vec[i]
        theta = trainLinearReg(linearRegCostFunction, X, y, lambda_)
        error_train[i] = 1/(2*m) * np.sum(np.square(X.dot(theta) - y))
        error_val[i] = 1/(2*mval) * np.sum(np.square(Xval.dot(theta) - yval))

    # ============================================================
    return lambda_vec, error_train, error_val
```

After you have completed the code, the next cell will run your function and plot a cross validation curve of error v.s. $\lambda$ that allows you select which $\lambda$ parameter to use. You should see a plot similar to the figure below.

In this figure, we can see that the best value of $\lambda$ is around 3. Due to randomness in the training and validation splits of the dataset, the cross validation error can sometimes be lower than the training error.

```python
lambda_vec, error_train, error_val = validationCurve(X_poly, y, X_poly_val, yval)

pyplot.plot(lambda_vec, error_train, '-o', lambda_vec, error_val, '-o', lw=2)
pyplot.legend(['Train', 'Cross Validation'])
pyplot.xlabel('lambda')
pyplot.ylabel('Error')

print('lambda\t\tTrain Error\tValidation Error')
for i in range(len(lambda_vec)):
    print(' %f\t%f\t%f' % (lambda_vec[i], error_train[i], error_val[i]))
```

### 3.4 Optional (ungraded) exercise: Computing test set error

```python
lambda_ = 3
theta = trainLinearReg(linearRegCostFunction, X_poly_test, ytest,
                             lambda_=lambda_, maxiter=55)

error_train, error_test = learningCurve(X_poly, y, X_poly_test, ytest, lambda_)

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTest Error')
for i in range(m):
    print('  \t%d\t\t%f' % (i+1, error_test[i]))
```

    Polynomial Regression (lambda = 3.000000)

    # Training Examples	Test Error
      	1		164.709076
      	2		169.779712
      	3		11.376190
      	4		5.365247
      	5		4.894375
      	6		4.997470
      	7		4.456939
      	8		4.346039
      	9		4.755793
      	10		4.780203
      	11		4.888742
      	12		3.859909

```python

```


```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = "retina"
import matplotlib.pyplot as plt
plt.style.use("default")
plt.rcParams["savefig.dpi"] = 100
plt.rcParams["figure.dpi"] = 100
plt.rcParams["font.size"] = 16
plt.rcParams["font.family"] = "sans-serif"
plt.rcParams["font.sans-serif"] = ["Liberation Sans"]
plt.rcParams["mathtext.fontset"] = "custom"
```

```python
import sympy as sm

fB, fC, fD, a1, a2, N0, N1_a1, N1_a2, N2 = sm.symbols("fA, fB, fC, a1, a2, N0, N1_a1, N1_a2, N2")

fA = 1 - fB - fC - fD

P0 = fA + fB * (1 - 1/a1) + fC * (1 - 1/a1) + fD * (1 - 1/a2)
P1_a1 = fC / a1 + fB * (1/a1 - 1/a2)
P1_a2 = fD / a2
P2 = fB / a2

L = N0 * sm.log(P0) + N1_a1 * sm.log(P1_a1) + N1_a2 * sm.log(P1_a2) + N2 * sm.log(P2)

sys = [
    sm.Eq(sm.simplify(sm.diff(L, fB)), 0),
    sm.Eq(sm.simplify(sm.diff(L, fC)), 0),
    sm.Eq(sm.simplify(sm.diff(L, fD)), 0),
]
sm.simplify(sm.solve(sys, (fB, fC, fD)))
```

    [(N2*a2/(N0 + N1_a1 + N1_a2 + N2), (N1_a1*a1 + N2*a1 - N2*a2)/(N0 + N1_a1 + N1_a2 + N2), N1_a2*a2/(N0 + N1_a1 + N1_a2 + N2))]

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
K = 10000
a1 = 5
a2 = 12
a = np.array([a1, a2])

def simulate(fB_true=0.3, fC_true=0.1, fD_true=0.1):
    cosinc_sim = np.random.uniform(0, 1, K)
    b = a[:, None] * cosinc_sim[None, :]
    obs = (b < 1)

    f = np.random.rand(K)

    m = (fB_true + fC_true + fD_true) <= f
    obs[:, m] = False

    m = ((fB_true + fC_true) <= f) & (f < (fB_true + fC_true + fD_true))
    obs[0, m] = False

    m = (fB_true <= f) & (f < (fB_true + fC_true))
    obs[1, m] = False

    N_obs = np.sum(obs, axis=0)

    # Treat planets independently
    Na, Nb = np.sum(obs, axis=1)
    Pa = 1. / a1
    Pb = 1. / a2
    occ_a = (Na / Pa + Nb / Pb) / K

    # Deal with multiplicity
    bins = -0.5 + np.arange(4)
    N0, N1, N2 = np.histogram(N_obs, bins)[0]
    P0 = 1 - 1. / a1
    P1 = 1. / a1 - 1. / a2
    P2 = 1. / a2
    occ_b = (N1 + N2) / ((1 - P0) * K)

    # Fit for multiplicity
    N1_a1 = np.sum((N_obs == 1) & obs[0])
    N1_a2 = np.sum((N_obs == 1) & obs[1])
    fB = N2*a2/K
    fC = (N1_a1*a1 + N2*a1 - N2*a2)/K
    fD = N1_a2*a2/K
    occ_c = fB + fC + fD

    return N0, N1, N2, occ_a, occ_b, occ_c
```

```python
np.random.seed(1234)
sims = np.array([simulate() for k in range(5000)])
```

```python
plt.hist(sims[:, -3], density=True, histtype="step", label="A")
plt.hist(sims[:, -2], density=True, histtype="step", label="B")
plt.hist(sims[:, -1], density=True, histtype="step", label="C")
plt.axvline(0.5, color="k", linewidth=1, alpha=0.5)
plt.legend()
plt.xlabel("occurence rate")
plt.ylabel("fraction of simulations")
plt.yticks([])
plt.savefig("simulated.pdf", bbox_inches="tight");
```

```python

```


# Modelo dinámico de un circuito RLC

```python
import sympy as sym
sym.init_printing()
import matplotlib.pyplot as plt
%matplotlib inline
```

| Elemento | Dominio del tiempo  (EDO)|
|:---:|:---:|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

Considere el circuito que se muestra en la figura. Suponga que la entrada es el voltaje $x(t)$ y la salida es el voltaje $y(t)$. Suponga además que no hay energía almacenada en los elementos que almacenan energía en $t<0$. Además, considere que $x(t)=0 \forall t<0$.

El voltaje $x(t)$ es igual a los voltajes que caen en la red $RLC$.

\begin{align}
v_R(t) + v_L(t) + v_C(t) &= x(t) \\
R i(t) + L \frac{di}{dt} + \frac{1}{C} \int_{0}^{t} i(\tau) d\tau &= x(t) \\
\end{align}

La salida $y(t)$ es el voltaje del capacitor.

\begin{equation}
y(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau
\end{equation}

De donde se obtiene la corriente $i(t)$

\begin{equation}
C \frac{dy}{dt} = i(t)
\end{equation}

Incorporando las expresiones de la salida $y(t)$ y la variable interna $i(t)$ en la ecuación de la malla se obtiene

\begin{align}
R i(t) + L \frac{di}{dt} + \frac{1}{C} \int_{0}^{t} i(\tau) d\tau &= x(t) \\
C R \frac{d y(t)}{dt} + C L \frac{d^2 y(t)}{dt^2} + y(t)& = x(t) \\
\end{align}

-------------------

\begin{equation}
y(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau
\end{equation}

\begin{equation}
C \cdot y(t) = C \cdot \frac{1}{C} \int_{0}^{t} i(\tau) d\tau
\end{equation}

\begin{equation}
C \cdot y(t) =  \int_{0}^{t} i(\tau) d\tau
\end{equation}

\begin{equation}
\frac{d}{dt} (C \cdot y(t)) =  \frac{d}{dt}(\int_{0}^{t} i(\tau) d\tau)
\end{equation}

\begin{equation}
C \frac{dy}{dt} =  i(t)
\end{equation}

```python
t, L, R, C = sym.symbols('t L R C', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(C*R*y.diff(t) + C*L*y.diff(t, 2) +  y, x)
ode
```

Dada una entrada $x(t)$, se puede calcular la solución del sistema. Observe que al usar `dsolve` obtiene la solución general de la homogénea relacionada junto a la solución particular.

```python
u = sym.Heaviside(t)
# u = sym.Heaviside(t)*t
# sol = sym.dsolve( ode.subs(x,u))
sol = sym.dsolve( ode.subs(x,u))
sol
```

$\displaystyle y{\left(t \right)} = C_{1} e^{\frac{t \left(- R - \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} + C_{2} e^{\frac{t \left(- R + \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} - \frac{\left(\begin{cases} \begin{cases} \frac{2 C L}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\0 & \text{otherwise} \end{cases} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R + \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} + \frac{\left(\begin{cases} \begin{cases} \frac{2 C L}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\0 & \text{otherwise} \end{cases} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R - \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} + \frac{\left(\begin{cases} \begin{cases} \frac{2 C L e^{\frac{R t}{2 L} - \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\t & \text{otherwise} \end{cases} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L e^{\frac{R t}{2 L} - \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R + \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} - \frac{\left(\begin{cases} \begin{cases} \frac{2 C L e^{\frac{R t}{2 L} + \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\t & \text{otherwise} \end{cases} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L e^{\frac{R t}{2 L} + \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R - \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}}$

Las constantes $C_1$ y $C_2$ deben calcularse para satisfacer las condiciones iniciales  $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ para $t<0$.

```python
C_int = sym.solve((sol.rhs.limit(t, 0, '-'),
                   sol.rhs.diff(t).limit(t, 0, '-')),
                  ['C1', 'C2'])
C_int
```

```python
sol = sol.subs(C_int)
sol
```

$\displaystyle y{\left(t \right)} = - \frac{\left(\begin{cases} \begin{cases} \frac{2 C L}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\0 & \text{otherwise} \end{cases} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R + \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} + \frac{\left(\begin{cases} \begin{cases} \frac{2 C L}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\0 & \text{otherwise} \end{cases} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R - \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} + \frac{\left(\begin{cases} \begin{cases} \frac{2 C L e^{\frac{R t}{2 L} - \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\t & \text{otherwise} \end{cases} & \text{for}\: C R - \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L e^{\frac{R t}{2 L} - \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R - \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R + \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}} - \frac{\left(\begin{cases} \begin{cases} \frac{2 C L e^{\frac{R t}{2 L} + \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} \neq 0 \\t & \text{otherwise} \end{cases} & \text{for}\: C R + \sqrt{C^{2} R^{2} - 4 C L} = 0 \\\frac{2 C L e^{\frac{R t}{2 L} + \frac{t \sqrt{C^{2} R^{2} - 4 C L}}{2 C L}}}{C R + \sqrt{C^{2} R^{2} - 4 C L}} & \text{otherwise} \end{cases}\right) e^{\frac{t \left(- R - \frac{\sqrt{C \left(C R^{2} - 4 L\right)}}{C}\right)}{2 L}} \theta\left(t\right)}{\sqrt{C \left(C R^{2} - 4 L\right)}}$

```python
RLC = {R: 2,
       L: 0.05,
       C: .03}
RLC
```

```python
tlaps = (t,-10e-3,0.5*10e-1)

plt.rcParams['figure.figsize'] = 9, 3

sym.plot(sol.rhs.subs(RLC), tlaps, line_color='red',
         ylabel = '$y(t)$',);

sym.plot(u, tlaps, line_color='blue',
         ylabel = '$x(t)$',);

```

```python
sol.rhs.subs(RLC)
```

```python

```

```python

```