# Optimization in Python

You might have noticed that we didn't do anything related to sparsity with scikit-learn models. A lot of the work we covered in the machine learning class is very recent research, and as such is typically not implemented by the popular libraries.

If we want to do things like sparse regression, we're going to have to roll up our sleeves and do it ourselves. For that, we need to be able to solve optimization problems. In Julia, we did this with JuMP. In Python, we'll use a similar library called *pyomo*.

# Installing pyomo

You can run the following command to install pyomo if you haven't already.

```python
!pip install pyomo --user
```

# Intro to pyomo

Let's see how we translate a simple, 2 variable LP to pyomo code.

$$
\begin{align*}
\max_{x,y} \quad& x + 2y \\
\text{s.t.}\quad& x + y \leq 1 \\
& x, y \geq 0.
\end{align*}
$$

First thing is to import the pyomo functions:

```python
from pyomo.environ import *
from pyomo.opt import SolverFactory
```

Next, we construct a model object. This is a container for everything in our optimization problem: variables, constraints, solver options, etc.

```python
m = ConcreteModel()
```

Next, we define the two decision variables in our optimization problem. We use the ``Var`` function to create the variables. The `within` keyword is used to specify the bounds on the variables, or equivalently the `bounds` keyword. The variables are added to the model object with names `x` and `y`.

```python
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, float('inf')))
```

We now add the single constraint of our problem using the ``Constraint`` function. We write it algebraically, and save the result to the model.

```python
m.con = Constraint(expr=m.x + m.y <= 1)
```

We specify the objective function with the `Objective` function:

```python
m.obj = Objective(sense=maximize, expr=m.x + 2 * m.y)
```

We solve the optimization problem by first specifying a solver using `SolverFactory` and then using this solver to solve the model:

```python
solver = SolverFactory('gurobi')
solver.solve(m)
```

    {'Problem': [{'Name': 'x3', 'Lower bound': 2.0, 'Upper bound': 2.0, 'Number of objectives': 1, 'Number of constraints': 2, 'Number of variables': 3, 'Number of binary variables': 0, 'Number of integer variables': 0, 'Number of continuous variables': 3, 'Number of nonzeros': 3, 'Sense': 'maximize'}], 'Solver': [{'Status': 'ok', 'Return code': '0', 'Message': 'Model was solved to optimality (subject to tolerances), and an optimal solution is available.', 'Termination condition': 'optimal', 'Termination message': 'Model was solved to optimality (subject to tolerances), and an optimal solution is available.', 'Wall time': '0.06682205200195312', 'Error rc': 0, 'Time': 1.0039210319519043}], 'Solution': [OrderedDict([('number of solutions', 0), ('number of solutions displayed', 0)])]}

We can now inspect the solution values and optimal cost.

```python
m.obj()
```

    2.0

```python
m.x.value
```

    0.0

```python
m.y.value
```

    1.0

Let's put it all together to compare with Julia/JuMP

```python
# Create model
m = ConcreteModel()
# Add variables
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, float('inf')))
# Add constraint
m.con = Constraint(expr=m.x + m.y <= 1)
# Add objective
m.obj = Objective(sense=maximize, expr=m.x + 2 * m.y)
# Solve model
solver = SolverFactory('gurobi')
solver.solve(m)
# Inspect solution
print(m.obj())
print(m.x.value)
print(m.y.value)
```

    2.0
    0.0
    1.0

```julia
# Create model
m = Model(solver=GurobiSolver())
# Add variables
@variable(m, x >= 0)
@variable(m, y >= 0)
# Add constraint
@constraint(m, x + y <= 1)
# Add objective
@objective(m, Max, x + 2y)
# Solve model
solve(m)
# Inspect solution
@show getobjectivevalue(m)
@show getvalue(x)
@show getvalue(y)
```

### Exercise

Code and solve the following optimization problem:

$$
\begin{align*}
\min_{x,y} \quad& 3x - y \\
\text{s.t.}\quad& x + 2y \geq 1 \\
& x \geq 0 \\
& 0 \leq y \leq 1.
\end{align*}
$$

```python
# Create the model
m = ConcreteModel()
# Add the variables
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, 1))
# Add the constraint
m.con = Constraint(expr=m.x + 2 * m.y >= 1)
# Add the objective
m.obj = Objective(sense=minimize, expr=3 * m.x - m.y)

solver = SolverFactory('gurobi')
solver.solve(m)

print(m.x.value, m.y.value)
```

    0.0 1.0

```python
for v in m.component_data_objects(Var, active=True):
    print(v, value(v)) # doctest: +SKIP
```

    x 0.0
    y 1.0

```python
m.pprint()
```

    2 Var Declarations
        x : Size=1, Index=None
            Key  : Lower : Value : Upper : Fixed : Stale : Domain
            None :     0 :   0.0 :  None : False : False : NonNegativeReals
        y : Size=1, Index=None
            Key  : Lower : Value : Upper : Fixed : Stale : Domain
            None :     0 :   1.0 :     1 : False : False :  Reals

    1 Objective Declarations
        obj : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize : 3*x - y

    1 Constraint Declarations
        con : Size=1, Index=None, Active=True
            Key  : Lower : Body    : Upper : Active
            None :   1.0 : x + 2*y :  +Inf :   True

    4 Declarations: x y con obj

# Index sets

Let's now move to a more complicated problem. We'll look at a transportation problem:

$$
\begin{align}
\min & \sum\limits_{i = 1}^{m} \sum\limits_{j = 1}^{n} c_{ij} x_{ij}\\
& \sum\limits_{j = 1}^{n} x_{ij} \leq b_i && i = 1, \ldots, m\\
& \sum\limits_{i = 1}^{m} x_{ij} = d_j && j = 1, \ldots, n\\
& x_{ij} \ge 0 && i = 1, \ldots, m, j = 1, \ldots, n
\end{align}
$$

And with some data:

```python
import numpy as np

m = 2  # Number of supply nodes
n = 5  # Number of demand nodes
# Supplies
b = np.array([1000, 4000])
# Demands
d = np.array([500, 900, 1800, 200, 700])
# Costs
c = np.array([[2, 4, 5, 2, 1],
              [3, 1, 3, 2, 3]])
```

Now we can formulate the problem with pyomo

```python
model = ConcreteModel()
```

First step is adding variables. We can add variables with indices by passing the relevant index sets to the `Var` constructor. In this case, we need a $m$-by$n$ matrix of variables:

```python
model.x = Var(range(m), range(n), within=NonNegativeReals)
```

Now to add the constraints. We have to add one supply constraint for each factory, so we might try something like:

```python
for i in range(m):
    model.supply = Constraint(expr=sum(model.x[i, j] for j in range(n)) <= b[i])
```

    WARNING: Implicitly replacing the Component attribute supply (type=<class
        'pyomo.core.base.constraint.SimpleConstraint'>) on block unknown with a
        new Component (type=<class
        'pyomo.core.base.constraint.SimpleConstraint'>). This is usually
        indicative of a modelling error. To avoid this warning, use
        block.del_component() and block.add_component().

Can you see the problem? We are overwriting `model.supply` in each iteration of the loop, and so only the last constraint is applied.

Luckily, pyomo has a (not-so-easy) way to add multiple constraints at a time. We first define a *rule* that takes in the model and any required indices, and then returns the expression for the constraint:

```python
def supply_rule(model, i):
    return sum(model.x[i, j] for j in range(n)) <= b[i]
```

We then add the constraint by referencing this rule along with the index set we want the constraint to be defined over:

```python
model.supply2 = Constraint(range(m), rule=supply_rule)
```

We then apply the same approach for the demand constraints

```python
def demand_rule(model, j):
    return sum(model.x[i, j] for i in range(m)) == d[j]

model.demand = Constraint(range(n), rule=demand_rule)
```

Finally, we add the objective:

```python
model.obj = Objective(sense=minimize,
    expr=sum(c[i, j] * model.x[i, j]
    for i in range(m) for j in range(n)))
```

Now we can solve the problem

```python
solver = SolverFactory('gurobi')
solver.solve(model)
```

    {'Problem': [{'Name': 'x11', 'Lower bound': 8600.0, 'Upper bound': 8600.0, 'Number of objectives': 1, 'Number of constraints': 9, 'Number of variables': 11, 'Number of binary variables': 0, 'Number of integer variables': 0, 'Number of continuous variables': 11, 'Number of nonzeros': 26, 'Sense': 'minimize'}], 'Solver': [{'Status': 'ok', 'Return code': '0', 'Message': 'Model was solved to optimality (subject to tolerances), and an optimal solution is available.', 'Termination condition': 'optimal', 'Termination message': 'Model was solved to optimality (subject to tolerances), and an optimal solution is available.', 'Wall time': '0.04288291931152344', 'Error rc': 0, 'Time': 0.31615591049194336}], 'Solution': [OrderedDict([('number of solutions', 0), ('number of solutions displayed', 0)])]}

It solved, so we can extract the results

```python
flows = np.array([[model.x[i, j].value for j in range(n)] for i in range(m)])
flows
```

    array([[ 300.,    0.,    0.,    0.,  700.],
           [ 200.,  900., 1800.,  200.,    0.]])

We can also check the objective value for the cost of this flow

```python
model.obj()
```

    8600.0

For simplicity, here is the entire formulation and solving code together:

```python
model = ConcreteModel()
# Variables
model.x = Var(range(m), range(n), within=NonNegativeReals)
# Supply constraint
def supply_rule(model, i):
    return sum(model.x[i, j] for j in range(n)) <= b[i]
model.supply2 = Constraint(range(m), rule=supply_rule)
# Demand constraint
def demand_rule(model, j):
    return sum(model.x[i, j] for i in range(m)) == d[j]
model.demand = Constraint(range(n), rule=demand_rule)
# Objective
model.obj = Objective(sense=minimize,
    expr=sum(c[i, j] * model.x[i, j]
    for i in range(m) for j in range(n)))
# Solve
solver = SolverFactory('gurobi')
solver.solve(model)
# Get results
flows = np.array([[model.x[i, j].value for j in range(n)] for i in range(m)])
model.obj()
```

# Machine Learning

Now let's put our pyomo knowledge to use and implement some of the same methods we saw in the machine learning class

First, specify your solver executable location:

```python
executable='C:/Users/omars/.julia/v0.6/Ipopt/deps/usr/bin/ipopt.exe'
```

To use the version left over from Julia
### On MacOS and Linux

`executable="~/.julia/v0.6/Homebrew/deps/usr/Cellar/ipopt/3.12.4_1/bin/ipopt")`

### On Windows

The path is probably under WinRPM:

`executable='%HOME%/.julia/v0.6/WinRPM/...')")`

# Linear Regression

Let's just try a simple linear regression

```python
def linear_regression(X, y):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add constraints

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return [m.beta[j].value for j in range(p)]
```

Let's load up some data to test it out on:

```python
from sklearn.datasets import load_boston
data = load_boston()
X = data.data
y = data.target
```

Try our linear regression function:

```python
print(linear_regression(X, y))
```

    [-0.0928965170276332, 0.04871495518299193, -0.004059979574994615, 2.8539988199938313, -2.8684363704210067, 5.928147779053185, -0.007269334576040927, -0.9685141573949657, 0.1711511282943897, -0.009396215397156512, -0.39219092629505153, 0.014905610228206412, -0.4163044707374162]

We can compare with sklearn to make sure it's right:

```python
from sklearn.linear_model import LinearRegression
m = LinearRegression(fit_intercept=False)
m.fit(X, y)
m.coef_
```

    array([-9.28965170e-02,  4.87149552e-02, -4.05997958e-03,  2.85399882e+00,
           -2.86843637e+00,  5.92814778e+00, -7.26933458e-03, -9.68514157e-01,
            1.71151128e-01, -9.39621540e-03, -3.92190926e-01,  1.49056102e-02,
           -4.16304471e-01])

Just for reference, let's look back at how we do the same thing in JuMP!

```julia
using JuMP, Gurobi
function linear_regression(X, y)
    n, p = size(X)
    m = Model(solver=GurobiSolver())
    @variable(m, beta[1:p])
    @objective(m, Min, sum((y[i] - sum(X[i, j] * beta[j] for j = 1:p)) ^ 2 for i = 1:n))
    solve(m)
    getvalue(beta)
end
```

or even

```julia
using JuMP, Gurobi
function linear_regression(X, y)
    n, p = size(X)
    m = Model(solver=GurobiSolver())
    @variable(m, beta[1:p])
    @objective(m, Min, sum((y - X * beta) .^ 2))
    solve(m)
    getvalue(beta)
end
```

Much simpler!

### Exercise

Modify the linear regression formulation to include an intercept term, and compare to scikit-learn's LinearRegression with `fit_intercept=False` to make sure it's the same

```python
def linear_regression_intercept(X, y):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.b0 = Var()

    # Add constraints

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)) - m.b0, 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return [m.beta[j].value for j in range(p)]

linear_regression_intercept(X, y)
```

    [-0.10801135783681295,
     0.046420458366832826,
     0.020558626367267345,
     2.6867338193441053,
     -17.766611228344583,
     3.8098652068064345,
     0.0006922246403880231,
     -1.4755668456006032,
     0.30604947898554247,
     -0.012334593916578101,
     -0.9527472317092022,
     0.009311683273791104,
     -0.5247583778555439]

```python
m = LinearRegression(fit_intercept=True)
m.fit(X, y)
m.coef_
```

    array([-1.08011358e-01,  4.64204584e-02,  2.05586264e-02,  2.68673382e+00,
           -1.77666112e+01,  3.80986521e+00,  6.92224640e-04, -1.47556685e+00,
            3.06049479e-01, -1.23345939e-02, -9.52747232e-01,  9.31168327e-03,
           -5.24758378e-01])

# Robust Regression

We saw in the class that both ridge and lasso regression were robust versions of linear regression. Both of these are provided by `sklearn`, but we need to know how to implement them if we want to extend regression ourselves

```python
def ridge_regression(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)),2)
        for i in range(n)) + rho * sum(pow(m.beta[j], 2) for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
ridge_regression(X, y, 100000)
```

    [-0.03349704413300387,
     0.09109374003810565,
     -0.020781437073126584,
     0.001953623650442121,
     0.0010799179606038737,
     0.036595936096463166,
     0.031147076775253392,
     0.008145310191304656,
     0.000355681037898492,
     0.0012341682146102984,
     0.025195318040087757,
     0.05360061512165024,
     -0.09887567363356603]

```python
def lasso(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)),2)
        for i in range(n)) + rho * sum(pow(m.beta[j], 2) for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

### Exercise

Implement Lasso regression

```python
def lasso_regression(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.absb = Var(range(p))

    # Add constraints
    def absbeta1(m, j):
        return m.beta[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.beta[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)) + rho * sum(m.absb[j] for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
lasso_regression(X, y, 1000)
```

# Sparse Regression

```python
def sparse_regression(X, y, k):
    n, p = X.shape
    M = 1000

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.z = Var(range(p), within=Binary)

    # Add constraints
    def bigm1(m, j):
        return m.beta[j] <= M * m.z[j]
    m.bigm1 = Constraint(range(p), rule=bigm1)
    def bigm2(m, j):
        return m.beta[j] >= -M * m.z[j]
    m.bigm2 = Constraint(range(p), rule=bigm2)

    m.sparsity = Constraint(expr=sum(m.z[j] for j in range(p)) <= k)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
sparse_regression(X, y, 10)
```

    [-0.09289651702716464,
     0.048714955183037804,
     -0.004059979575834942,
     2.8539988201207356,
     -2.868436370310866,
     5.9281477790418435,
     -0.0072693345761560306,
     -0.9685141573945727,
     0.1711511282944914,
     -0.009396215397248317,
     -0.39219092628560964,
     0.014905610227998044,
     -0.4163044707410663]

```python
import numpy as np
l = np.array([1,2,3,4])

print(l**2)
print([sqrt(i) for i in l])
```

    [ 1  4  9 16]
    [1.0, 1.4142135623730951, 1.7320508075688772, 2.0]

### Exercise

Try implementing the algorithmic framework for linear regression:
- sparsity constraints
- lasso regularization
- restrict highly correlated pairs of features
- nonlinear transformations (just $\sqrt(x)$ and $x^2$)

```python
import numpy as np
from sklearn.preprocessing import normalize

def all_regression(X_orig, y, k, rho):
    n, p_orig = X_orig.shape
    M = 10

    X = np.concatenate(
        [X_orig, np.sqrt(X_orig), np.square(X_orig)], axis=1
    )
    p = X.shape[1]

    # Normalize data
    X = normalize(X, axis=0)
    y = (y - np.mean(y)) / np.linalg.norm(y)

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.z = Var(range(p), within=Binary)
    m.absb = Var(range(p))

    # Sparsity constraints
    def bigm1(m, j):
        return m.beta[j] <= M * m.z[j]
    m.bigm1 = Constraint(range(p), rule=bigm1)
    def bigm2(m, j):
        return m.beta[j] >= -M * m.z[j]
    m.bigm2 = Constraint(range(p), rule=bigm2)
    m.sparsity = Constraint(expr=sum(m.z[j] for j in range(p)) <= k)

    # Lasso constraints
    def absbeta1(m, j):
        return m.beta[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.beta[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Correlation constraints
    corX = np.corrcoef(np.transpose(X))
    def cor_rule(m, i, j):
        if i > j and abs(corX[i, j]) > 0.8:
            return (sum(m.z[k] for k in range(i, p, p_orig)) +
                    sum(m.z[k] for k in range(j, p, p_orig)) <= 1)
        else:
            return Constraint.Skip
    m.cor = Constraint(range(p_orig), range(p_orig), rule=cor_rule)

    # Nonlinear constraints
    def nl_rule(m, i):
        return sum(m.z[k] for k in range(i, p, p_orig)) <= 1
    m.nl = Constraint(range(p_orig), rule=nl_rule)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)) + rho * sum(m.absb[j] for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
#     results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return np.array([m.beta[j].value for j in range(p)]).reshape(-1, p_orig)
```

```python
all_regression(X, y, 6, 0)
```

    array([[-0.2436117 , -0.40635426,  1.287454  ,  0.00847825, -3.77070371,
            -5.52116145, -0.89579769,  1.8866262 , -1.2658388 , -0.27200984,
            -4.29661229,  1.08490846,  0.87517741],
           [ 0.08922823,  0.22757203, -1.40483302,  0.00847825,  6.06383002,
             1.71571313,  0.89614225, -3.03336942,  1.22670959, -0.27220395,
             5.06365681, -0.5261114 , -1.95285925],
           [ 0.1005534 ,  0.19449394, -0.33587296,  0.00847825,  0.1653508 ,
             2.76298931,  0.2188393 , -0.32862084,  0.51518994,  0.15346198,
             0.63958091, -0.5226175 , -0.10282404]])

# Logistic Regression

Like JuMP, we need to use a new solver for the nonlinear problem. We can use Ipopt as before, except we have to set it up manually. You'll need to download Ipopt and add it to the PATH.

On Mac, you can do this with Homebrew if you have it:

The other way is to download a copy of ipopt and specify the path to it exactly when creating the solver. For example, I have a copy of Ipopt left over from JuMP, which I can use by modifying the SolverFactory line as indicated below:

```python
def logistic_regression(X, y):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Set nonlinear objective function
    m.obj = Objective(sense=maximize, expr=-sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

Load up some data

```python
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X = data.data
y = data.target
```

```python
logistic_regression(X, y)
```

    ([4149.289184609701,
      -95.66597524418847,
      -122.72220946545622,
      -31.07342010773767,
      -30864.886098201634,
      36295.57939884905,
      -19052.20895802693,
      -19686.209843004974,
      14077.278909141973,
      -45229.07643132272,
      -2194.6637679242845,
      98.59470071343296,
      810.8129671962596,
      -71.48231452267014,
      65260.80657288916,
      -62374.925075090185,
      52121.749598051225,
      -203728.7743634409,
      62535.47114299024,
      488515.76125488814,
      -1086.5758594687723,
      -31.442744168114533,
      -43.120968967631356,
      7.303064300605229,
      3500.09658663806,
      5108.10478323704,
      -4412.61061910415,
      -4740.534608043277,
      -14164.859267037991,
      -35109.19922500868],
     -323.2327374749216)

### Exercise

Implement the regularized versions of logistic regression that scikit-learn provides:

```python
def logistic_regression_l1(X, y, C):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Lasso constraints
    m.absb = Var(range(p))
    def absbeta1(m, j):
        return m.b[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.b[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Set nonlinear objective function
    m.obj = Objective(sense=minimize, expr=sum(m.absb[j] for j in range(p)) + C * sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

```python
logistic_regression_l1(X, y, 100)
```

```python
def logistic_regression_l2(X, y, C):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Set nonlinear objective function
    m.obj = Objective(sense=minimize, expr=0.5 * sum(pow(m.b[j], 2) for j in range(p)) + C * sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

```python
logistic_regression_l2(X, y, 1000)
```

```python

```

```python

```

ML3 - Optimization Modeling (Complete).ipynb


Attribution:

   * Most material based on [Sam Roweis](https://www.cs.nyu.edu/home/people/in_memoriam/samroweis.html)' Linear Algebra Review

```python
%matplotlib notebook

# Note that this is the usual way that I import Numpy and Matplotlib
import numpy as np
import matplotlib.pyplot as plt
```

# Vectors and Matrices

Linear algebra is the study of vectors and matrices and how they can be manipulated to perform various calculations.

Consider functions which take several input arguments and produce several output arguments. If we stack up the input arguments into a vector $\mathbf{x}$ and the outputs into a vector $\mathbf{y}$, then a function $\mathbf{y} = f(\mathbf{x})$ is said to be *linear* if:

$$f\left(\alpha \mathbf{x} + \beta \mathbf{u} \right) = \alpha f\left( \mathbf{x} \right) + \beta f \left( \mathbf{u} \right)$$

for all scalars, $\alpha, \beta$ and all vectors $\mathbf{x}, \mathbf{u}$. In other words, *scaling the input scales the output* and *summing inputs sums their outputs*.

**Now for an amazing fact**:

All functions which are linear can be written in the form of a matrix $F$ which left multiplies the input argument $\mathbf{x}$:

$$\mathbf{y} = F \mathbf{x}$$

Furthermore, all matrix relations like the one above represent linear functions from their inputs to their outputs.

**Another interesting fact** is that the composition of two linear functions is still linear:

$$g\left(f\left(\mathbf{x}\right)\right) = GF\mathbf{x} = H \mathbf{x} = h(\mathbf{x})$$

*The entire study of multiple-input multiple-output linear functions can be reduced to the study of vectors and matrices.*

# Multiplication, Addition, Transposition

Adding up two vectors or two matrices is easy: just add their corresponding elements (of course the two must be the same shape!):

```python
a = np.array([1, 2, 3])
b = np.ones(3,)
print a
print b
print a + b
```

```python
A = np.array([[1, 2, 3], [4, 5, 6]])
B = np.arange(6).reshape(A.shape)
print A
print B
print A + B
```

Multiplying a vector or matrix by a scalar just multiplies each element by the scalar:

```python
print 2 * a
print 0.5 * A
```

## Matrix-vector and matrix-matrix multiplication

A good way to think of an $n \times m$ matrix $F$ is as a machine that eats $m$ sized vectors and spits out $n$ sized vectors. This conversion process is known as (left) multiplying by $F$ and has many similarities to scalar multiplication, but also a few differences. Most importantly, the machine only accepts inputs of the right size.

```python
# In Numpy, both matrix-vector and matrix-matrix multiplication is performed by np.dot
print A.shape
print a.shape
print np.dot(A, a)
print A.dot(a)
```

```python
print np.dot(A, a[:2])  # not compatible sizes
```

Like scalar multiplication, matrix multiplication is *distributive* and *associative*:

$$
\begin{aligned}
F(\mathbf{a} + \mathbf{b}) & = F\mathbf{a} + F\mathbf{b}\\
 G(F\mathbf{a}) & = (GF)\mathbf{a}
\end{aligned}
$$

One way to think of this is that the matrix product $GF$ is the equivalent linear operator you get if you compose the action of $F$ followed by the action of $G$.

Matrix-matrix multiplication can be thought of as a sequence of matrix-vector multiplications, one for each column, whose results get stacked beside each other in columns to form a new matrix. In general, we can think of column vectors of length $k$ as just $k \times 1$ and row vectors as $1 \times k$ matrices. This eliminates any distinction between matrix-matrix and matrix-vector multiplication.

```python
print np.dot(A, A.T)
```

Note that in the above, we flipped or "transposed" the matrix. This interchanges the rows and columns, and in the example above, made the shapes compatible for matrix-matrix multiplication.

```python
print A.shape
print A.T.shape
```

Unlike scalar multiplication, matrix multiplication is not *commutative*:

$$ F \mathbf{a} \neq \mathbf{a} F $$

Multiplying a vector by itself (transposed) gives a scalar $\mathbf{x}^T \mathbf{x}$ which is known as the (squared) *norm* or squared length of the vector and is written $||\mathbf{x}||^2$. This measure adds up the sum of the squares of the elements of the vector. For much more about norms, see [Goodfellow et al. Chapter 2](http://www.deeplearningbook.org/contents/linear_algebra.html).

```python
print np.dot(a.T, a)
print np.dot(a, a)  # note we didn't actually need the transpose - numpy automatically does dot product with two vector inputs
print np.linalg.norm(a)**2  # this is a more powerful function for computing general norms
```

For more information about efficiently computing norms (and also how to call [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) directly from Python) see [this blog post](http://fseoane.net/blog/2011/computing-the-vector-norm/).

The *Frobenius norm* of a matrix $||A||^2$ similarly adds up the squares of all the matrix elements.

```python
print np.sum(A * A)  # note * is element-wise multiplication, not matrix-matrix multiplication
print np.linalg.norm(A)**2  # by default, linalg.norm() computes Frobenius norm for matrix input
```

# Inverses and Determinants

First, let's consider the concept of reversing or undoing or *inverting* the function represented by a matrix $A$. For a function to be invertible, there needs to be a one-to-one relationship between inputs and outputs so that given the output you can always say exactly what the input was. In other words, we need a function which, when composed with $A$ gives back the original vector. Such a function -- if it exists -- is called the *inverse* of $A$ and the matrix is denoted $A^{-1}$.

In matrix terms, we seek a matrix that left multiplies $A$ to give the identity matrix:

$$A^{-1}A = I$$

The identity matrix, $I_{ij} = \delta_{ij}$ corresponds to the identity (do-nothing) function.

Only a few, special linear functions are invertible.

* They must have at least as many outputs as inputs
* They must not map any two inputs to the same output

Technically this means that they must have *full rank*, a concept which we will get to later.

Non-square matrices ($m$-by-$n$ matrices for which $m \neq n$) technically do not have an inverse. However, in some cases such a matrix may have a *left inverse* or *right inverse* (but not both). If A is $m$-by-$n$ and the rank of $A$ is equal to $n$, then $A$ has a left inverse: an $n$-by-$m$ matrix $B$ such that $BA = I$. If $A$ has rank $m$, then it has a right inverse: an $n$-by-$m$ matrix $B$ such that $AB = I$. [Goodfellow et al. Chapter 2](http://www.deeplearningbook.org/contents/linear_algebra.html) provides a lot more detail on inverses.

```python
C = A.dot(A.T)  # A trick to make an invertible matrix
C_inv = np.linalg.inv(C)
print C
print C_inv
print C_inv.dot(C)
```

The matrix *determinant* is a scalar quantity, normally denoted $|A|$ or $\text{det}(A)$ whose absolute value measures how much the matrix "stretches" or "squishes" volume as it transforms its inputs to outputs and whose sign indicates whether the transformation is [orientation preserving](http://en.wikipedia.org/wiki/Orientation_%28vector_space%29). Matrices with large determinants do (on average) a lot of stretching and those with small determinants do a lot of squishing.

Matrices with zero determinant have rank less than the number of rows and and actually collapse some of their input space into a line or hyperplane (pancake) in the output space, and this can be thought of doing "infinite squishing". Conventionally, the determinant is only defined for square matrices, but there is a natural extension to rectangular ones using the *singular value decomposition*.

```python
A = np.array([[2, 3, 0], [3, 2, 7], [2, 1, 6]])
print A
print np.linalg.det(A)
print np.linalg.det(np.linalg.inv(A))  # note 1/det(A)
B = np.array([[2, 3, 2], [1, 2, 4], [3, 5, 6]])
print np.linalg.matrix_rank(B)
print B
print np.linalg.det(B)  # not quite zero because of numerical instability
```

# Fundamental Matrix Equations

The two most important matrix equations are the system of linear equations:

$$A \mathbf{x} = \mathbf{b}$$

and the eigenvector equation:

$$A \mathbf{x} = \lambda \mathbf{x}$$

which between them cover a large number of optimization and constraint satisfaction problems. As we've written them above, $\mathbf{x}$ is a vector but these equations also have natural extensions to the case where there are many vectors simultaneously satisfying the equation $AX = B$ or $AX = \lambda X$.

# Systems of Linear Equations

A central problem in linear algebra is the solution of a system of linear equations like this:

$$
\begin{align}
3x + 4y + 2z &= 12\\
x + y + z &=5
\end{align}
$$

Typically, we express this system as a single *matrix equation* in the form: $A\mathbf{x} = \mathbf{b}$, where $A$ is an $m$-by-$n$ matrix, $\mathbf{x}$ is an $n$-dimensional column vector, and $\mathbf{b}$ is an $m$-dimensional column vector. The number of unknowns is $n$ and the number of equations or constraints is $m$. Here is another simple example:

$$
\left[
\begin{array}{rr}
2 & -1\\
1 & 1
\end{array}
\right]
\left[
\begin{array}{c}
x_1 \\
x_2
\end{array}
\right]
=
\left[
\begin{array}{c}
1\\
5
\end{array}
\right]
$$

How do we go about "solving" this system of equations?

* Finding $\mathbf{b}$ given $A$ and $\mathbf{x}$ is easy - just multiply
* For a single $\mathbf{x}$ and $\mathbf{b}$ there are usually a great many matrices $A$ which satisfy the equation
* The only interesting problem left is to find $\mathbf{x}$ given $A$ and $\mathbf{b}$!

This kind of equation is really a problem statement. It says "we applied the function $A$ and generated the output $\mathbf{b}$; what was the input $\mathbf{x}$?"

The matrix $A$ is dictated to us by our problem, and represents our model of how the system we are studying converts inputs to outputs. The vector $\mathbf{b}$ is the output that we observe (or desire) - we know it. The vector $\mathbf{x}$ is the set of inputs that we are trying to find.

There are two ways of thinking about this system of equations. One is *rowwise* as a set of $m$ equations, or constraints that correspond geometrically to $m$ intersecting constraint surfaces:

$$
\left[
\begin{array}{r}
2x_1 - x_2 &= 1\\
x_1 + x_2 &= 5
\end{array}
\right]
$$

The goal is to find the point(s), for example $(x_1, x_2)$ above, which are at the intersection of all the constraint surfaces. In the example above, these surfaces are two lines in a plane. If the lines intersect then there is a solution. If they are parallel, there is not. If they are coincident then there are infinite solutions. In higher dimensions there are more geometric solutions, but in general there can be no solutions, one solution, or infinite solutions.

The other way of thinking about this system is *columnwise* in which we think of the entire system as a single vector relation:

$$
x_1
\left[
\begin{array}{r}
2\\
1
\end{array}
\right]
+
x_2
\left[
\begin{array}{r}
-1\\
1
\end{array}
\right]
=
\left[
\begin{array}{r}
1\\
5
\end{array}
\right]
$$

The goal here is to discover which linear combination(s) $(x_1, x_2)$, if any, of the $n$ column vectors on the left will give the one on the right.

Either way, the matrix equation $A\mathbf{x} = \mathbf{b}$ is an almost ubiquitous problem whose solution comes up again and again in theoretical derivations and in practical data analysis problems. One of the most important applications is the minimization of quadratic energy functions: if $A$ is symmetric positive definite then the quadratic form $\mathbf{x}^TA\mathbf{x} - 2 \mathbf{x}^T\mathbf{b} + c$ is *minimized* at the point where $A\mathbf{x}=\mathbf{b}$. Such quadratic forms arise often in the study of linear models with Gaussian noise since the log likelihood of data under such models is always a matrix quadratic.

## Least squares: solving for $\mathbf{x}$

Let's first consider the case of a single $\mathbf{x}$. As noted above, geometrically we can think of the rows of the system as encoding constraint surfaces in which the solution vector $\mathbf{x}$ must lie and what we are looking for is the point(s) at which these surfaces intersect. Of course, there may be no solution satisfying the equation; then we typically need some way to pick the "best" approximate solution. The constraints may also intersect along an entire line or surface in which case there are an infinity of solutions; once again we would like to think about which one might be best.

Let's consider finding exact solutions first. The most naive thing we could do is just find the inverse of $A$ and solve the equations as follows:

$$
\begin{align}
A^{-1}A\mathbf{x} & = A^{-1} \mathbf{b}\\
I\mathbf{x} &= A^{-1}\mathbf{b}\\
\mathbf{x} &= A^{-1}\mathbf{b}
\end{align}
$$

which is known as *Cramer's rule*.

There are several problems with this approach. Most importantly, many interesting functions are not invertible. Beyond that, matrix inversion is a difficult and potentially numerically unstable operation. **Don't invert a matrix unless you really have to!**

In fact, there is a much better way to define what we want as a solution. We will say that we want a solution $\mathbf{x}^*$ which minimizes the error:

$$E = ||A\mathbf{x}^* - \mathbf{b}||^2 = \mathbf{x}^TA^TA\mathbf{x} - 2\mathbf{x}^TA^T\mathbf{b} + \mathbf{b}^T\mathbf{b}$$

This problem is known as *linear least squares*, for obvious reasons. If there is an exact solution (one giving zero error) it will certainly minimize the above cost, but if there is no solution, we can still find the best possible approximation. If we take the matrix derivative of this expression, we can find the best solution:

$$\mathbf{x}^* = (A^TA)^{-1}A^T\mathbf{b}$$

which takes advantage of the fact that even if $A$ is not invertible, $A^TA$ may be.

But what if the problem is degenerate? In other words, what if there is more than one exact solution or more than one inexact solution which all achieve the same minimum error. How can this occur?

Imagine an equation like this:

$$\left[
\begin{array}{rr}
1 &  -1
\end{array}
\right] \mathbf{x} = 4$$

in which $A = \left[ \begin{array}{rr}
1 &  -1
\end{array} \right]$. This equation constrains the difference between the two elements of $\mathbf{x}$ to be 4 but the sum of the two elements can be as large or small as we want.

We can take things one step further to get around this problem. The answer is to ask for the *minimum norm* vector $\mathbf{x}$ that still minimizes the above error. This breaks the degeneracies in both the exact and inexact cases. In terms of the cost function, this corresponds to adding an infinitesimal penalty on $\mathbf{x}^T\mathbf{x}$:

$$
E = \text{lim}_{\epsilon \rightarrow 0} \left[ \mathbf{x}^TA^TA\mathbf{x} - 2\mathbf{x}^TA^T\mathbf{b} + \mathbf{b}^T\mathbf{b} + \epsilon \mathbf{x}^T\mathbf{x} \right]
$$

and the optimal solution becomes

$$
\mathbf{x}^* = \lim_{\epsilon \rightarrow 0}  \left[ \left(A^TA + \epsilon I\right)^{-1}A^T\mathbf{b}\right]
$$

Now, of course actually computing these solutions efficiently and in a numerically stable way is the topic of much study in numerical methods. However, in Python you don't have to worry about any of this, you can just type ``np.linalg.solve(A, b)`` (if $A$ is square) or ``np.linalg.lstsq(A, b)`` (if $A$ is not square) and let someone else worry about it.

```python
A = np.array([[1,2],[3,4]])
print A
b = np.array([[5],[6]])
print b
x = np.linalg.inv(A).dot(b) #Cramer's rule, slow
print x
print A.dot(np.linalg.inv(A).dot(b))-b #check
x1 = np.linalg.solve(A,b) #fast; computes the "exact" solution
print x1
print A.dot(x1)-b #check
```

```python
A = np.array([[1, 2, 3],[4, 5, 6]])
print A
b = np.array([[5],[6]])
print b
# computes the least-squares solution to a linear matrix equation
# equation may be under-, well, or over- determined
x, residuals, rank, s =  np.linalg.lstsq(A, b)
print x
print A.dot(x)
```

## Linear regression: solving for $A$

Now consider what happens if we have many vectors $\mathbf{x}_i$ and $\mathbf{b}_i$, all of which we want to satisfy the equation $A \mathbf{x}_i = \mathbf{b}_i$. If we stack the vectors $\mathbf{x}_i$ beside each other as the columns of a large matrix $X$ and do the same for $\mathbf{b}_i$ to form $B$, we can write the problem as a large matrix equation:

$$AX = B$$

There are two things we could do here. If, as before, $A$ is known, we could find $X$ given $B$. To do this, we would just need to apply the techniques described above to solve the system $A \mathbf{x}_i = \mathbf{b}_i$ independently for each column $i$.

However, if we were given *both* $X$ and $B$ we could try to find a *single* $A$ which satisfies the equations. In essense, we are fitting a linear function given its inputs $X$ and corresponding outputs $B$. This problem is called *linear regression*. Usually we add a row of ones to $X$ to fit an affine function (i.e. one with an offset).

Again, there are only very few cases in which there exists an $A$ which exactly satisfies the equations (if there is, $X$ will be square and invertible). However, we can set things up the same way as before and ask for the *least-squares* $A$ which minimizes:

$$E = \sum_i ||A \mathbf{x}_i - \mathbf{b}_i||^2$$

Using matrix calculus, we can derive the optimal solution to this problem. The answer, one of the most famous formulas in all of mathematics, is known as the *discrete Wiener filter*:

$$ A^* = BX^T\left(XX^T\right)^{-1}$$

Once again, we might have invertibility problems in $XX^T$, this corresponds to having fewer equations than unknowns in our linear system (or duplicated equations), thus leaving some of the elements in $A$ unconstrained. We can get around this in the same way as with linear least squares by adding a small amount of penalty on the norm of the elements of $A$.

$$E = \sum_i ||A \mathbf{x}_i - \mathbf{b}_i||^2 + \epsilon ||A||^2$$

which means we are asking for the matrix of minimum norm which still minimizes the sum squared error on the outputs. Under this cost, the optimal solution is:

$$ A^* = BX^T\left(XX^T + \epsilon I\right)^{-1}$$

which is known as *ridge regression*. Often it is a good idea to use a small nonzero value of $\epsilon$ even if $XX^T$ is technically invertible, because this gives more stable solutions by penalizing large elements of $A$ that aren't doing much to reduce the error. In neural networks, this is known as *weight decay*, and in general, is known as *regularization*. You can also interpret it as having a Gaussian prior with mean zero and variance $1/2\epsilon$ on each element of $A$.

In Python, you should not actually invert the matrix, just type ``np.linalg.lstsq(X, b)``.

```python
N = 9  # how many data points
x1 = np.arange(0, N)
# note, here the data is stacked in columns
# this is not Python convention (convention is to stack data in rows)
# but it's the convention in this tutorial!
X = np.array([x1, np.ones(N)])
y = 4 * x1 + 3 + np.random.randn(N)  # note x1 is only 1-d here, but it is m-d in general
A, residuals, rank, s = np.linalg.lstsq(X.T, y)
line = A[0] * x1 + A[1] # regression line
plt.plot(x1, line, 'r-', x1, y, 'bo')
plt.xlabel('x')
plt.ylabel('y')
```


```python
import numpy as np
import sympy as sp
import scipy as sc
import matplotlib.pyplot as plt
from scipy.optimize import fsolve
import scipy.optimize as op
from mpl_toolkits.mplot3d import axes3d
%matplotlib notebook
```

$PROBLEM$ $1$

$Part$ $A$

```python
def vFunc(r):
 return sp.sin(r) + r ** 2 + 1
r=sp.Symbol('r')
#to find the expression of first, second and third differential
print(sp.diff(vFunc(r),r))
print(sp.diff(vFunc(r),r,r))
print(sp.diff(vFunc(r),r,r,r))
```

    2*r + cos(r)
    2 - sin(r)
    -cos(r)

$Part$ $B$

```python
r=np.arange(-np.pi,np.pi,0.02)
v=np.sin(r)+ r ** 2 + 1
plt.xlabel("r in radians")
plt.ylabel("v(r)")
plt.ylim(0,5)
plt.plot(r,v,'r')
plt.show()
plt.legend(['v(r)'])
```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x2aa833038e0>

$Part$ $C$

The Value of the "r" for which function "v" is minimun is approximately -0.48

$Part$ $D$

```python
r=np.arange(-np.pi,np.pi,0.05)
w=2*r + np.cos(r)
plt.xlabel("r in radians")
plt.ylim(0,7,0.5)
plt.plot(r,w,'b')
plt.legend(['dv(r)/dr'])
plt.show()

```

The Value of the "r" for which function "w" is 0 is approximately -0.46. This is equal with the result in Part C. Because the the dervateive of of a function solved at zero would give us the value at which the original function is minumum. In other words the Minimum values of a function can be found using by taking it's first dervative and equating it to zero.

$PROBLEM$ $2$

```python

ax=plt.axes(projection="3d")
def zFunc(x, y):
    return np.sin(x*y) + np.cos(x*y)
#x=np.linspace(-4,4,500)
#y=np.linspace(-1,1,500)
X, Y=np.meshgrid(x, y)
Z=zFunc(X, Y)
ax.plot_surface(X, Y, Z, rstride=20, cstride=20, cmap='winter', edgecolor='none')
ax.set_title('Surface', color='r');
ax.plot_wireframe(X, Y, Z, color='none')
ax.set_xlabel('X - axis', color='green')
ax.set_ylabel('Y - axis', color='purple')
ax.set_zlabel('Z - axis', color='blue')
plt.show()
```

    <IPython.core.display.Javascript object>

$PROBLEM$ $3$

$Part$ $A$

```python
x=np.arange(0,2*np.pi,0.05)
y=np.sin(x)+np.cos(x)
plt.xlabel("X in radians")
plt.ylabel("Y=sin(x)+cos(x)")
plt.plot(x,y)
plt.show()
```

    <IPython.core.display.Javascript object>

$Part$ $B$

```python
#Taylor Series approximation
x=np.arange(0,2*np.pi,0.05)
x0=0
yacc=np.sin(x)+np.cos(x)
y2=np.sin(x0)+np.cos(x0)+(x-x0)*(np.cos(x0)-np.sin(x0))
y3=y2+((x-x0)**2)*(-np.cos(x0)-np.sin(x0))/2
y4=y3+((x-x0)**3)*(-np.cos(x0)+np.sin(x0))/6
y5=y4+((x-x0)**4)*(np.cos(x0)+np.sin(x0))/24
y6=y5+((x-x0)**5)*(np.cos(x0)-np.sin(x0))/120
plt.plot(x,yacc,x,y2,x,y3,x,y4,x,y5,x,y6)
plt.show()
```

    <IPython.core.display.Javascript object>

$PROBLEM$ $4$

$Part$ $A$

```python
x=np.arange(-6*np.pi,6*np.pi,0.05)
f=(x**3)*np.cos(x)
plt.xlabel("X in rad")
plt.ylabel("F(x)=x^3(Cos(x))")
# plt.ylim(0,5)
plt.plot(x,f)
p=np.pi
xData = np.array([-5.5*p,-4.5*p,-3.5*p,-2.5*p,-1.5*p,-0.5*p,0,0.5*p,1.5*p,2.5*p,3.5*p,4.5*p,5.5*p])
yData = np.array([0,0,0,0,0,0,0,0,0,0,0,0,0])
plt.plot(x,np.zeros(len(x)))
plt.plot(xData, yData, 'ro')
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x22564979a00>]

$Part$ $B$

```python
#Let's define the function F(x)=x^3(cos(x))
def func(x):
    y = (x**3)*np.cos(x)
    return y

x=fsolve(func,-18)
print(x)
x=fsolve(func,-15)
print(x)
x=fsolve(func,-12)
print(x)
x=fsolve(func,-8)
print(x)
x=fsolve(func,-4)
print(x)
x=fsolve(func,-2)
print(x)
x=fsolve(func,1)
print(x)
x=fsolve(func,2)
print(x)
x=fsolve(func,5)
print(x)
x=fsolve(func,7)
print(x)
x=fsolve(func,11)
print(x)
x=fsolve(func,14)
print(x)
x=fsolve(func,16)
print(x)
print("We see that it has found approximated the 13 roots of the function")
```

    [-17.27875959]
    [-14.13716694]
    [-10.99557429]
    [-7.85398163]
    [-4.71238898]
    [-1.57079633]
    [1.09422902e-49]
    [1.57079633]
    [4.71238898]
    [7.85398163]
    [10.99557429]
    [14.13716694]
    [17.27875959]
    We see that it has found approximated the 13 roots of the function

$PROBLEM$ $5$

```python
def f(z):
    x=z[0]
    y=z[1]
    f=np.zeros(2)
    f[0]=y+2*x+4
    f[1]=y-x**2+4
    return f
zGuess=np.array([-1,1])
z=fsolve(f,zGuess)
print ("First set of Solutions")
print(z,"\n")

zGuess=np.array([0,4])
print ("Second set of Solutions")
z=fsolve(f,zGuess)
print(z)

```

    First set of Solutions
    [-2.00000000e+00  1.66094557e-16]

    Second set of Solutions
    [-1.61558713e-27 -4.00000000e+00]


# Statistics Fundamentals
Statistics is primarily about analyzing data samples, and that starts with udnerstanding the distribution of data in a sample.

## Analyzing Data Distribution
A great deal of statistical analysis is based on the way that data values are distributed within the dataset. In this section, we'll explore some statistics that you can use to tell you about the values in a dataset.

### Measures of Central Tendency
The term *measures of central tendency* sounds a bit grand, but really it's just a fancy way of saying that we're interested in knowing where the middle value in our data is. For example, suppose decide to conduct a study into the comparative salaries of people who graduated from the same school. You might record the results like this:

| Name     | Salary      |
|----------|-------------|
| Dan      | 50,000      |
| Joann    | 54,000      |
| Pedro    | 50,000      |
| Rosie    | 189,000     |
| Ethan    | 55,000      |
| Vicky    | 40,000      |
| Frederic | 59,000      |

Now, some of the former-students may earn a lot, and others may earn less; but what's the salary in the middle of the range of all salaries?

#### Mean
A common way to define the central value is to use the *mean*, often called the *average*. This is calculated as the sum of the values in the dataset, divided by the number of observations in the dataset. When the dataset consists of the full population, the mean is represented by the Greek symbol ***μ*** (*mu*), and the formula is written like this:

\begin{equation}\mu = \frac{\displaystyle\sum_{i=1}^{N}X_{i}}{N}\end{equation}

More commonly, when working with a sample, the mean is represented by ***x̄*** (*x-bar*), and the formula is written like this (note the lower case letters used to indicate values from a sample):

\begin{equation}\bar{x} = \frac{\displaystyle\sum_{i=1}^{n}x_{i}}{n}\end{equation}

In the case of our list of heights, this can be calculated as:

\begin{equation}\bar{x} = \frac{50000+54000+50000+189000+55000+40000+59000}{7}\end{equation}

Which is **71,000**.

>In technical terminology, ***x̄*** is a *statistic* (an estimate based on a sample of data) and ***μ*** is a *parameter* (a true value based on the entire population). A lot of the time, the parameters for the full population will be impossible (or at the very least, impractical) to measure; so we use statistics obtained from a representative sample to approximate them. In this case, we can use the sample mean of salary for our selection of surveyed students to try to estimate the actual average salary of all students who graduate from our school.

In Python, when working with data in a *pandas.dataframe*, you can use the ***mean*** function, like this:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mean())
```

So, is **71,000** really the central value? Or put another way, would it be reasonable for a graduate of this school to expect to earn $71,000? After all, that's the average salary of a graduate from this school.

If you look closely at the salaries, you can see that out of the seven former students, six earn less than the mean salary. The data is *skewed* by the fact that Rosie has clearly managed to find a much higher-paid job than her classmates.

#### Median
OK, let's see if we can find another definition for the central value that more closely reflects the expected earning potential of students attending our school. Another measure of central tendancy we can use is the *median*. To calculate the median, we need to sort the values into ascending order and then find the middle-most value. When there are an odd number of observations, you can find the position of the median value using this formula (where *n* is the number of observations):

\begin{equation}\frac{n+1}{2}\end{equation}

Remember that this formula returns the *position* of the median value in the sorted list; not the value itself.

If the number of observations is even, then things are a little (but not much) more complicated. In this case you calculate the median as the average of the two middle-most values, which are found like this:

\begin{equation}\frac{n}{2} \;\;\;\;and \;\;\;\; \frac{n}{2} + 1\end{equation}

So, for our graduate salaries; first lets sort the dataset:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

There's an odd number of observation (7), so the median value is at position (7 + 1) ÷ 2; in other words, position 4:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
|***>54,000*** |
| 55,000      |
| 59,000      |
| 189,000     |

So the median salary is **54,000**.

The *pandas.dataframe* class in Python has a ***median*** function to find the median:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].median())
```

#### Mode
Another related statistic is the *mode*, which indicates the most frequently occurring value. If you think about it, this is potentially a good indicator of how much a student might expect to earn when they graduate from the school; out of all the salaries that are being earned by former students, the mode is earned by more than any other.

Looking at our list of salaries, there are two instances of former students earning **50,000**, but only one instance each for all other salaries:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

The mode is therefore **50,000**.

As you might expect, the *pandas.dataframe* class has a ***mode*** function to return the mode:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mode())
```

##### Multimodal Data
It's not uncommon for a set of data to have more than one value as the mode. For example, suppose Ethan receives a raise that takes his salary to **59,000**:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
|***>59,000***|
|***>59,000***|
| 189,000     |

Now there are two values with the highest frequency. This dataset is *bimodal*. More generally, when there is more than one mode value, the data is considered *multimodal*.

The *pandas.dataframe.**mode*** function returns all of the modes:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,59000,40000,59000]})

print (df['Salary'].mode())
```

### Distribution and Density
Now we know something about finding the center, we can start to explore how the data is distributed around it. What we're interested in here is understanding the general "shape" of the data distribution so that we can begin to get a feel for what a 'typical' value might be expected to be.

We can start by finding the extremes - the minimum and maximum. In the case of our salary data, the lowest paid graduate from our school is Vicky, with a salary of **40,000**; and the highest-paid graduate is Rosie, with **189,000**.

The *pandas.dataframe* class has ***min*** and ***max*** functions to return these values.

Run the following code to compare the minimum and maximum salaries to the central measures we calculated previously:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print ('Min: ' + str(df['Salary'].min()))
print ('Mode: ' + str(df['Salary'].mode()[0]))
print ('Median: ' + str(df['Salary'].median()))
print ('Mean: ' + str(df['Salary'].mean()))
print ('Max: ' + str(df['Salary'].max()))
```

We can examine these values, and get a sense for how the data is distributed - for example, we can see that the *mean* is closer to the max than the *median*, and that both are closer to the *min* than to the *max*.

However, it's generally easier to get a sense of the distribution by visualizing the data. Let's start by creating a histogram of the salaries, highlighting the *mean* and *median* salaries (the *min*, *max* are fairly self-evident, and the *mode* is wherever the highest bar is):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
salary.plot.hist(title='Salary Distribution', color='lightblue', bins=25)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

The ***mean*** and ***median*** are shown as dashed lines. Note the following:
- *Salary* is a continuous data value - graduates could potentially earn any value along the scale, even down to a fraction of cent.
- The number of bins in the histogram determines the size of each salary band for which we're counting frequencies. Fewer bins means merging more individual salaries together to be counted as a group.
- The majority of the data is on the left side of the histogram, reflecting the fact that most graduates earn between 40,000 and 55,000
- The mean is a higher value than the median and mode.
- There are gaps in the histogram for salary bands that nobody earns.

The histogram shows the relative frequency of each salary band, based on the number of bins. It also gives us a sense of the *density* of the data for each point on the salary scale. With enough data points, and small enough bins, we could view this density as a line that shows the shape of the data distribution.

Run the following cell to show the density of the salary data as a line on top of the histogram:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
density = stats.gaussian_kde(salary)
n, x, _ = plt.hist(salary, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*5)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

Note that the density line takes the form of an asymmetric curve that has a "peak" on the left and a long tail on the right. We describe this sort of data distribution as being *skewed*; that is, the data is not distributed symmetrically but "bunched together" on one side. In this case, the data is bunched together on the left, creating a long tail on the right; and is described as being *right-skewed* because some infrequently occurring high values are pulling the *mean* to the right.

Let's take a look at another set of data. We know how much money our graduates make, but how many hours per week do they need to work to earn their salaries? Here's the data:

| Name     | Hours |
|----------|-------|
| Dan      | 41    |
| Joann    | 40    |
| Pedro    | 36    |
| Rosie    | 30    |
| Ethan    | 35    |
| Vicky    | 39    |
| Frederic | 40    |

Run the following code to show the distribution of the hours worked:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Hours':[41,40,36,30,35,39,40]})

hours = df['Hours']
density = stats.gaussian_kde(hours)
n, x, _ = plt.hist(hours, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7)
plt.axvline(hours.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(hours.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

Once again, the distribution is skewed, but this time it's **left-skewed**. Note that the curve is asymmetric with the ***mean*** to the left of the ***median*** and the *mode*; and the average weekly working hours skewed to the lower end.

Once again, Rosie seems to be getting the better of the deal. She earns more than her former classmates for working fewer hours. Maybe a look at the test scores the students achieved on their final grade at school might help explain her success:

| Name     | Grade |
|----------|-------|
| Dan      | 50    |
| Joann    | 50    |
| Pedro    | 46    |
| Rosie    | 95    |
| Ethan    | 50    |
| Vicky    | 5     |
| Frederic | 57    |

Let's take a look at the distribution of these grades:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Grade':[50,50,46,95,50,5,57]})

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7.5)
plt.axvline(grade.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(grade.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

This time, the distribution is symmetric, forming a "bell-shaped" curve. The ***mean***, ***median***, and mode are at the same location, and the data tails off evenly on both sides from a central peak.

Statisticians call this a *normal* distribution (or sometimes a *Gaussian* distribution), and it occurs quite commonly in many scenarios due to something called the *Central Limit Theorem*, which reflects the way continuous probability works - more about that later.

#### Skewness and Kurtosis
You can measure *skewness* (in which direction the data is skewed and to what degree) and kurtosis (how "peaked" the data is) to get an idea of the shape of the data distribution. In Python, you can use the ***skew*** and ***kurt*** functions to find this:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' skewness: ' + str(df[col].skew()))
    print(df[col].name + ' kurtosis: ' + str(df[col].kurt()))
    density = stats.gaussian_kde(df[col])
    n, x, _ = plt.hist(df[col], histtype='step', normed=True, bins=25)
    plt.plot(x, density(x)*6)
    plt.show()
    print('\n')
```

Now let's look at the distribution of a real dataset - let's see how the heights of the father's measured in Galton's study of parent and child heights are distributed:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

import statsmodels.api as sm

df = sm.datasets.get_rdataset('GaltonFamilies', package='HistData').data

fathers = df['father']
density = stats.gaussian_kde(fathers)
n, x, _ = plt.hist(fathers, histtype='step', normed=True, bins=50)
plt.plot(x, density(x)*2.5)
plt.axvline(fathers.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(fathers.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

As you can see, the father's height measurements are approximately normally distributed - in other words, they form a more or less *normal* distribution that is symmetric around the mean.

### Measures of Variance
We can see from the distribution plots of our data that the values in our dataset can vary quite widely. We can use various measures to quantify this variance.

#### Range
A simple way to quantify the variance in a dataset is to identify the difference between the lowest and highest values. This is called the *range*, and is calculated by subtracting the minimim value from the maximum value.

The following Python code creates a single Pandas dataframe for our school graduate data, and calculates the *range* for each of the numeric features:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' range: ' + str(df[col].max() - df[col].min()))
```

#### Percentiles and Quartiles
The range is easy to calculate, but it's not a particularly useful statistic. For example, a range of 149,000 between the lowest and highest salary does not tell us which value within that range a graduate is most likely to earn -  it doesn't tell us nothing about how the salaries are distributed around the mean within that range.  The range tells us very little about the comparative position of an individual value within the distribution - for example, Frederic scored 57 in his final grade at school; which is a pretty good score (it's more than all but one of his classmates); but this isn't immediately apparent from a score of 57 and range of 90.

##### Percentiles
A percentile tells us where a given value is ranked in the overall distribution. For example, 25% of the data in a distribution has a value lower than the 25th percentile; 75% of the data has a value lower than the 75th percentile, and so on. Note that half of the data has a value lower than the 50th percentile - so the 50th percentile is also the median!

Let's examine Frederic's grade using this approach. We know he scored 57, but how does he rank compared to his fellow students?

Well, there are seven students in total, and five of them scored less than Frederic; so we can calculate the percentile for Frederic's grade like this:

\begin{equation}\frac{5}{7} \times 100 \approx 71.4\end{equation}

So Frederic's score puts him at the 71.4th percentile in his class.

In Python, you can use the ***percentileofscore*** function in the *scipy.stats* package to calculate the percentile for a given value in a set of values:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'strict'))
```

We've used the strict definition of percentile; but sometimes it's calculated as being the percentage of values that are less than *or equal to* the value you're comparing. In this case, the calculation for Frederic's percentile would include his own score:

\begin{equation}\frac{6}{7} \times 100 \approx 85.7\end{equation}

You can calculate this way in Python by using the ***weak*** mode of the ***percentileofscore*** function:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'weak'))
```

We've considered the percentile of Frederic's grade, and used it to rank him compared to his fellow students. So what about Dan, Joann, and Ethan? How do they compare to the rest of the class? They scored the same grade (50), so in a sense they share a percentile.

To deal with this *grouped* scenario, we can average the percentage rankings for the matching scores. We treat half of the scores matching the one we're ranking as if they are below it, and half as if they are above it. In this case, there were three matching scores of 50, and for each of these we calculate the percentile as if 1 was below and 1 was above. So the calculation for a percentile for Joann based on scores being less than or equal to 50 is:

\begin{equation}(\frac{4}{7}) \times 100 \approx 57.14\end{equation}

The value of **4** consists of the two scores that are below Joann's score of 50, Joann's own score, and half of the scores that are the same as Joann's (of which there are two, so we count one).

In Python, the ***percentileofscore*** function has a ***rank*** function that calculates grouped percentiles like this:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 50, 'rank'))
```

##### Quartiles
Rather than using individual percentiles to compare data, we can consider the overall spread of the data by dividing those percentiles into four *quartiles*. The first quartile contains the values from the minimum to the 25th percentile, the second from the 25th percentile to the 50th percentile (which is the median), the third from the 50th percentile to the 75th percentile, and the fourth from the 75th percentile to the maximum.

In Python, you can use the ***quantile*** function of the *pandas.dataframe* class to find the threshold values at the 25th, 50th, and 75th percentiles (*quantile* is a generic term for a ranked position, such as a percentile or quartile).

Run the following code to find the quartile thresholds for the weekly hours worked by our former students:

```python
# Quartiles
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df['Hours'].quantile([0.25, 0.5, 0.75]))
```

Its usually easier to understand how data is distributed across the quartiles by visualizing it. You can use a histogram, but many data scientists use a kind of visualization called a *box plot* (or a *box and whiskers* plot).

Let's create a box plot for the weekly hours:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Hours'].plot(kind='box', title='Weekly Hours Distribution', figsize=(10,8))
plt.show()
```

The box plot consists of:
- A rectangular *box* that shows where the data between the 25th and 75th percentile (the second and third quartile) lie. This part of the distribution is often referred to as the *interquartile range* - it contains the middle 50 data values.
- *Whiskers* that extend from the box to the bottom of the first quartile and the top of the fourth quartile to show the full range of the data.
- A line in the box that shows that location of the median (the 50th percentile, which is also the threshold between the second and third quartile)

In this case, you can see that the interquartile range is between 35 and 40, with the median nearer the top of that range. The range of the first quartile is from around 30 to 35, and the fourth quartile is from 40 to 41.

#### Outliers
Let's take a look at another box plot - this time showing the distribution of the salaries earned by our former classmates:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8))
plt.show()
```

So what's going on here?

Well, as we've already noticed, Rosie earns significantly more than her former classmates. So much more in fact, that her salary has been identifed as an *outlier*. An outlier is a value that is so far from the center of the distribution compared to other values that it skews the distribution by affecting the mean. There are all sorts of reasons that you might have outliers in your data, including data entry errors, failures in sensors or data-generating equipment, or genuinely anomalous values.

So what should we do about it?

This really depends on the data, and what you're trying to use it for. In this case, let's assume we're trying to figure out what's a reasonable expectation of salary for a graduate of our school to earn. Ignoring for the moment that we have an extremly small dataset on which to base our judgement, it looks as if Rosie's salary could be either an error (maybe she mis-typed it in the form used to collect data) or a genuine anomaly (maybe she became a professional athelete or some other extremely highly paid job). Either way, it doesn't seem to represent a salary that a typical graduate might earn.

Let's see what the distribution of the data looks like without the outlier:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8), showfliers=False)
plt.show()
```

Now it looks like there's a more even distribution of salaries. It's still not quite symmetrical, but there's much less overall variance. There's potentially some cause here to disregard Rosie's salary data when we compare the salaries, as it is tending to skew the analysis.

So is that OK? Can we really just ignore a data value we don't like?

Again, it depends on what you're analyzing. Let's take a look at the distribution of final grades:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

Once again there are outliers, this time at both ends of the distribution. However, think about what this data represents. If we assume that the grade for the final test is based on a score out of 100, it seems reasonable to expect that some students will score very low (maybe even 0) and some will score very well (maybe even 100); but most will get a score somewhere in the middle.  The reason that the low and high scores here look like outliers might just be because we have so few data points. Let's see what happens if we include a few more students in our data:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

With more data, there are some more high and low scores; so we no longer consider the isolated cases to be outliers.

The key point to take away here is that you need to really understand the data and what you're trying to do with it, and you need to ensure that you have a reasonable sample size, before determining what to do with outlier values.

#### Variance and Standard Deviation
We've seen how to understand the *spread* of our data distribution using the range, percentiles, and quartiles; and we've seen the effect of outliers on the distribution. Now it's time to look at how to measure the amount of variance in the data.

##### Variance
Variance is measured as the average of the squared difference from the mean. For a full population, it's indicated by a squared Greek letter *sigma* (***σ2***) and calculated like this:

\begin{equation}\sigma^{2} = \frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}\end{equation}

For a sample, it's indicated as ***s2*** calculated like this:

\begin{equation}s^{2} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}\end{equation}

In both cases, we sum the difference between the individual data values and the mean and square the result. Then, for a full population we just divide by the number of data items to get the average. When using a sample, we divide by the total number of items **minus 1** to correct for sample bias.

Let's work this out for our student grades (assuming our data is a sample from the larger student population).

First, we need to calculate the mean grade:

\begin{equation}\bar{x} = \frac{50+50+46+95+50+5+57}{7}\approx 50.43\end{equation}

Then we can plug that into our formula for the variance:

\begin{equation}s^{2} = \frac{(50-50.43)^{2}+(50-50.43)^{2}+(46-50.43)^{2}+(95-50.43)^{2}+(50-50.43)^{2}+(5-50.43)^{2}+(57-50.43)^{2}}{7-1}\end{equation}

So:

\begin{equation}s^{2} = \frac{0.185+0.185+19.625+1986.485+0.185+2063.885+43.165}{6}\end{equation}

Which simplifies to:

\begin{equation}s^{2} = \frac{4113.715}{6}\end{equation}

Giving the result:

\begin{equation}s^{2} \approx 685.619\end{equation}

The higher the variance, the more spread your data is around the mean.

In Python, you can use the ***var*** function of the *pandas.dataframe* class to calculate the variance of a column in a dataframe:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].var())
```

##### Standard Deviation
To calculate the variance, we squared the difference of each value from the mean. If we hadn't done this, the numerator of our fraction would always end up being zero (because the mean is at the center of our values). However, this means that the variance is not in the same unit of measurement as our data - in our case, since we're calculating the variance for grade points, it's in grade points squared; which is not very helpful.

To get the measure of variance back into the same unit of measurement, we need to find its square root:

\begin{equation}s = \sqrt{685.619} \approx 26.184\end{equation}

So what does this value represent?

It's the *standard deviation* for our grades data. More formally, it's calculated like this for a full population:

\begin{equation}\sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}}\end{equation}

Or like this for a sample:

\begin{equation}s = \sqrt{\frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}}\end{equation}

Note that in both cases, it's just the square root of the corresponding variance forumla!

In Python, you can calculate it using the ***std*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].std())
```

#### Standard Deviation in a Normal Distribution

In statistics and data science, we spend a lot of time considering *normal* distributions; because they occur so frequently. The standard deviation has an important relationship to play in a normal distribution.

Run the following cell to show a histogram of a *standard normal* distribution (which is a distribution with a mean of 0 and a standard deviation of 1):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

# Create a random standard normal distribution
df = pd.DataFrame(np.random.randn(100000, 1), columns=['Grade'])

# Plot the distribution as a histogram with a density curve
grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, color='lightgrey', normed=True, bins=100)
plt.plot(x, density(x))

# Get the mean and standard deviation
s = df['Grade'].std()
m = df['Grade'].mean()

# Annotate 1 stdev
x1 = [m-s, m+s]
y1 = [0.25, 0.25]
plt.plot(x1,y1, color='magenta')
plt.annotate('1s (68.26%)', (x1[1],y1[1]))

# Annotate 2 stdevs
x2 = [m-(s*2), m+(s*2)]
y2 = [0.05, 0.05]
plt.plot(x2,y2, color='green')
plt.annotate('2s (95.45%)', (x2[1],y2[1]))

# Annotate 3 stdevs
x3 = [m-(s*3), m+(s*3)]
y3 = [0.005, 0.005]
plt.plot(x3,y3, color='orange')
plt.annotate('3s (99.73%)', (x3[1],y3[1]))

# Show the location of the mean
plt.axvline(grade.mean(), color='grey', linestyle='dashed', linewidth=1)

plt.show()
```

The horizontal colored lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

#### Z Score
So in a normal (or close to normal) distribution, standard deviation provides a way to evaluate how far from a mean a given range of values falls, allowing us to compare where a particular value lies within the distribution. For example, suppose Rosie tells you she was the highest scoring student among her friends - that doesn't really help us assess how well she scored. She may have scored only a fraction of a point above the second-highest scoring student. Even if we know she was in the top quartile; if we don't know how the rest of the grades are distributed it's still not clear how well she performed compared to her friends.

However, if she tells you how many standard deviations higher than the mean her score was, this will help you compare her score to that of her classmates.

So how do we know how many standard deviations above or below the mean a particular value is? We call this a *Z Score*, and it's calculated like this for a full population:

\begin{equation}Z = \frac{x - \mu}{\sigma}\end{equation}

or like this for a sample:

\begin{equation}Z = \frac{x - \bar{x}}{s}\end{equation}

So, let's examine Rosie's grade of 95. Now that we know the *mean* grade is 50.43 and the *standard deviation* is 26.184, we can calculate the Z Score for this grade like this:

\begin{equation}Z = \frac{95 - 50.43}{26.184} = 1.702\end{equation}.

So Rosie's grade is 1.702 standard deviations above the mean.

### Summarizing Data Distribution in Python
We've seen how to obtain individual statistics in Python, but you can also use the ***describe*** function to retrieve summary statistics for all numeric columns in a dataframe. These summary statistics include many of the statistics we've examined so far (though it's worth noting that the *median* is not included):

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df.describe())
```

Statistics and Probability by Hiren/04-02-Statistics Fundamentals.ipynb


```python
%matplotlib inline

import operator as op
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.model_selection import train_test_split
import itertools
from utils import *

```

# 1) Logistic Regression on simple dataset with two classes

Here we'll use a regularized logistic regression model to classify points that belong to two possible classes. Because we only have two classes here, the true labels of the data points are denoted by 0 or 1.

Given a training example $x$ and a parameter vector $\theta$, the prediction of our model is given by
\begin{equation}
\hat{y}(x)_{\theta} = \frac{1}{1 + e^{-\theta^T x}}
\end{equation}

This function is simply a sigmoid, meaning that our output value always lies beteen 0 and 1. The value of the prediction can be interpreted as the probability of the training sample $x$ to belong to the class '1'. Whenever $\theta^T x \geq 0$, we have $\hat{y}(x)_{\theta} \geq 0.5$ and the sample $x$ will be predicted to belong to class '1'.

To perform logistic regression, we need to specify a cost function. We will use the following cost function:
\begin{align}
J(\theta)  &= -\frac{1}{m} \Big[ \sum_{i =1}^m y^{(i)} \text{log}\,(\,\hat{y}_{\theta}(x^{(i)})\,) + (1 - y^{(i)})\, \text{log}\Big(1 - \hat{y}_{\theta}(x^{(i)})\Big) \Big] \nonumber \\
&+ \frac{\lambda}{2 m} \sum_{i = 1}^n \theta_i^2
\end{align}
where $\hat{y}_{\theta}(x^{(i)})$ and $y^{(i)}$ denote the predicted and true label of training example $x^{(i)}$, where $i = 1, \cdots m$ with $m$ the total number of training examples. If each training example consists of two features, we will have $x^{(i)} \in \mathbb{R}^{m\times2}$. The last term is a regularisation term.

When performing gradient descent to find the optimal parameter vector $\theta$, we also require the gradient $\partial_{\theta} J(\theta)$. Given this gradient, the parameters $\theta$ are updated using the rule
\begin{equation}
\theta_k \rightarrow \theta_k - \alpha \partial_{\theta} J(\theta)
\end{equation}
where $\alpha$ is the learning rate. In this way, we're updating each parameter such that the cost decreases. Evaluating $\partial_{\theta} J(\theta)$ gives:
\begin{equation}
\theta_k \rightarrow \theta_k\left(1 - \alpha \frac{\lambda}{m}\right) - \frac{\alpha}{m} \sum_{i = 1}^m (\,\hat{y}_{\theta}(x^{(i)}) - y^{(i)}) x_k^{(i)}
\end{equation}

```python
#some parameters
nb_c = 2 #number of classes
degree = 3
lr = 0.1
nIters = 1000
lamb = 1 #regularization parameter
```

```python
#data
data = np.genfromtxt('../Data/Data.txt', delimiter = ',')
X = data[:,:2]
X = (X - np.mean(X, axis = 0))/np.std(X, axis = 0) #normalize the data
y = data[:,2]; y = np.reshape(y, (len(y), 1)) #y is an array consisting of 0's, 1's, etc.. as labels for different classes

#create new features, polynomial of degree 'degree'.
X, nbFeatures = mapFeatures(X, degree)
```

## Visualise the data

To classify these data, a nonlinear decision boundary is required, which is why we will create additional features such as $\,x_1^2, x_2^2, x_1 x_2, x_1^3, x_2^3, x_1^2 x_2, \cdots$. We fill fit a polynomial and to prevent overfitting we use regularization.

```python
PlotData(X,y)
plt.xlabel('Feature 1'); plt.ylabel('Feature 2')
```

## Train model

```python
#initialize parameters
initial_theta = np.zeros((nbFeatures, 1))

#find final theta values using gradient descent.
finalTheta, costValues = GradientDescent(X, y, initial_theta, lr, lamb, nIters)

#Plot the decision boundary
fig, axs = plt.subplots(1,2, figsize = (10, 5)); axs = axs.ravel()
plt.axes(axs[0])
PlotData(X,y)
plotDecisionBoundary(finalTheta, X, y, degree)
plt.xlabel('Feature 1'); plt.ylabel('Feature 2'); plt.title('Decision boundary')

#Plot the cost as a function of the number of iterations of gradient descent
plt.axes(axs[1])
plt.plot(np.arange(nIters), costValues)
plt.xlabel('Iterations'); plt.ylabel('Cost'); plt.title('Cost during gradient descent')

```

# 2) Logistic regression with more classes

Use the iris dataset

```python
X, y =datasets.load_iris(return_X_y=True)
```

```python
#some parameters
nb_c = 3 #number of classes
degree = 2
lr = 0.1
nIters = 1000
lamb = 4 #regularization parameter

#data
X = (X - np.mean(X, axis = 0))/np.std(X, axis = 0)
y = np.reshape(y, (len(y), 1))
X, nbFeatures = mapFeatures(X, degree)
```

```python
#Split in training and test. Should of course do this multiple times with different
#random splits but for now let's just do it once.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)

#initialize parameters
initial_theta = np.zeros((nbFeatures, 1))
allTheta = np.zeros((nb_c, nbFeatures)) #to store the learned theta parameters

#for each class, find final theta values. We will use all-vs-one to predict the class
for c in range(nb_c):
    finalTheta, costValues = GradientDescent(X_train, y_train==c, initial_theta, lr, lamb, nIters)
    allTheta[c,:] = np.reshape(finalTheta, nbFeatures)
```

```python
#find out accuracy on training and test set
p_train = predictOneVsAll(allTheta, X_train)
pcorrect_train = FractionCorrect(p_train, y_train)

p_test = predictOneVsAll(allTheta, X_test)
pcorrect_test = FractionCorrect(p_test, y_test)

print('Accuracy on training set:\t{}').format(pcorrect_train)
print('Accuracy on test set:\t\t{}').format(pcorrect_test)
```

    Accuracy on training set:	0.980952380952
    Accuracy on test set:		0.911111111111

Now create multiple random splits of training and test set, for each calculate the
accuracy in the predictions of training and test, then take average

```python
#Get average errors using multiple random splits of data into training and test sets
pcor_train, pcor_test = PredictAccuracy(X, y, lamb, lr, nIters, nb_c)

#print(pcor_train)
#print(pcor_test)

print('Mean training correct fraction:\t{}').format(np.mean(pcor_train))
print('Mean test correct fraction:\t{}').format(np.mean(pcor_test))
```

    Mean training correct fraction:	0.96619047619
    Mean test correct fraction:	0.955555555556

### Look at some diagnostics


# One Step Methods
Given the first order equation
\begin{equation}
y^{'} = 2y-0.1y^2,
\end{equation}
with the initial condition
\begin{equation}
y(0) = 1.
\end{equation}

Aproximate the solution of the inital value problem for the interval $0 \leq t \leq 4$ using $N=8$.
Using the:
1. Euler method:
$$ w_{i+1}=w_i+hf(t_i,w_i);$$

2. Midpoint method:
$$ w_{i+1}=w_i+hf(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
where $k1=f(t_i,w_i)$;

3. 4th Order Runge Kutta method:
$$ w_{i+1}=w_i+\frac{h}{6}(k1+2k2+2k3+k4),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
$$k3=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k2),$$
$$k4=f(t_i+h,w_i+hk3);$$

4. Imposter-Butler method:
$$ w_{i+1}=w_i+\frac{h}{4}(k1+2k2),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+h,w_i+hk1).$$

What is wrong with the Imposter-Butler method?

### DECLARING LIBRARIES

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

### Outputting Table

```python
class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)
```

## Setting up the discrete time axis
$$ h=\frac{4-0}{N}=0.5 $$
$$t_i=t_0+ih$$
$$t_i=0+0.5i$$
for $i=0,1,2,...,8.$

```python
N=8
x_end=4.0
h=((x_end-0)/N)

INITIALCONDITION=1
time=np.zeros(N+1)

table = ListTable()
table.append(['time', 'Euler', 'Mid', 'RK','Imposter'])

```

### Defining the function
From the initial value problem the function
$$ f(t,y)= 2y-0.1y^2. $$

```python
def myfun(w):
    return 2*w-0.1*w*w
```

## NUMERICAL SOLUTION

```python
time[0]=0
```

## Euler Method
$$w_0=1$$
$$ w_{i+1}=w_i+h(2w_i-0.1w_i^2)$$
$$N=0,1,2,...7$$

```python
Euler=np.zeros(N+1)
Euler[0]=INITIALCONDITION

for i in range (0,N):
    # Euler Method
    Euler[i+1]=Euler[i]+h*myfun(Euler[i])
```

## Midpoint Method
$$w_0=1$$
$$k=2w_i-0.1w_i^2$$
$$ w_{i+1}=w_i+h(2(w_i+\frac{h}{2}k)-0.1(w_i+\frac{h}{2}k)^2)$$
$$N=0,1,2,...7$$

```python
Midpoint=np.zeros(N+1)
Midpoint[0]=INITIALCONDITION

for i in range (0,N):
    # Mid-point method
    Midpoint[i+1]=Midpoint[i]+h*myfun(Midpoint[i]+h/2*myfun(Midpoint[i]))
```

## 4th Order Runge Kutta Method
$$w_0=1$$
$$k1=2w_i-0.1w_i^2$$
$$k2=2(w_i+\frac{h}{2}k1)-0.1(w_i+\frac{h}{2}k1)^2$$
$$k3=2(w_i+\frac{h}{2}k2)-0.1(w_i+\frac{h}{2}k2)^2$$
$$k4=2(w_i+hk)-0.1(w_i+hk)^2$$
$$ w_{i+1}=w_i+\frac{h}{6}(k1+2k2+2k3+k4)$$
$$N=0,1,2,...7$$

```python
RK4=np.zeros(N+1)
RK4[0]=INITIALCONDITION

for i in range (0,N):
    # 4th Order Runge Kutta
    k1=myfun(RK4[i])
    k2=myfun(RK4[i]+h/2*k1)
    k3=myfun(RK4[i]+h/2*k2)
    k4=myfun(RK4[i]+h*k3)
    RK4[i+1]=RK4[i]+h/6*(k1+2*k2+2*k3+k4)

```

## Imposter Butler Method
$$w_0=1$$
$$k1=2w_i-0.1w_i^2$$
$$k2=2(w_i+hk1)-0.1(w_i+hk1)^2$$
$$ w_{i+1}=w_i+\frac{h}{4}(k1+2k2)$$
$$N=0,1,2,...7$$

```python
Imposter=np.zeros(N+1)
Imposter[0]=INITIALCONDITION
i=0
table.append([time[i], Euler[i], Midpoint[i], RK4[i],Imposter[i]])
for i in range (0,N):
    #Imposter method
    k1=myfun(Imposter[i])
    k2=myfun(Imposter[i]+h*k3)
    Imposter[i+1]=Imposter[i]+h/4*(k1+2*k2)
    time[i+1]=time[i]+h
    table.append([time[i+1], Euler[i+1], Midpoint[i+1], RK4[i+1],Imposter[i+1]])
```

## Plotting

```python
fig = plt.figure(figsize=(8,4))
plt.plot(time,Euler,'o:',color='red',label='Euler')
plt.plot(time,Midpoint,'o:',color='blue',label='Midpoint')
plt.plot(time,RK4,'o:',color='green',label='RK4')
plt.plot(time,Imposter,'o:',color='yellow',label='Imposter')
plt.legend(loc=4)
plt.title('Numerical Solutions of initial value problem.')
```

```python
v
```

time	Euler	Mid	RK	Imposter

0.0	1.0	1.0	1.0	1.0

0.5	1.95	2.36621875	2.4973140745034703	1.828354103622306

1.0	3.7098750000000003	5.19438758389418	5.581882795387722	3.1792956569283346

1.5	6.73159137421875	9.77881461292582	10.250670627504523	5.27210794647557

2.0	11.197466626964687	14.519443452104497	14.812884418161534	8.243922547632055

2.5	16.125770310829978	17.40121357626255	17.70715620549379	11.899511859117052

3.0	19.24951721577768	18.761663288245703	19.080300940495654	15.487688268593597

3.5	19.971838779529293	19.397552928361538	19.6459932243498	18.03499834285824

4.0	19.99996034728308	19.703035811592716	19.865924950313506	19.25729194968363

Chapter 01 - Euler Methods/.ipynb_checkpoints/Example Euler Method 2nd Order Equation-checkpoint.ipynb