Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \\ y = a x^3 + b x^2 + c x + d \\ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \\log(x)$$

Or even, more complicated such as :
$$ y = \\log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of **log()**: i.e. $$ y = \\log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\\begin{equation}
y = \\log(X)
\\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \\frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-Coursera/labs/Data_files/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2020-09-17 16:17:16 URL:https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-Coursera/labs/Data_files/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \\hat{Y} = \\frac1{1+e^{\\beta_1(X-\\beta_2)}}$$

$\\beta_1$: Controls the curve's steepness,

$\\beta_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447527, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# write your code here
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    /home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/scipy/optimize/minpack.py:829: OptimizeWarning: Covariance of the parameters could not be estimated
      category=OptimizeWarning)

    Mean absolute error: 0.17
    Residual sum of squares (MSE): 0.15
    R2-score: -92823079925823634226942355313786880.00

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

| Date (YYYY-MM-DD) | Version | Changed By | Change Description    |
| ----------------- | ------- | ---------- | --------------------- |
| 2020-08-04        | 0       | Nayef      | Upload file to Gitlab |
|                   |         |            |                       |

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

```python

```


Copyright (c) 2022 Valentin Hartmann. All rights reserved.
Released under Apache 2.0 license as described in the file LICENSE.

```python
import math
import scipy.integrate as integrate
import scipy.stats as stats
import numpy as np
import matplotlib.pyplot as plt
import warnings

plt.rcParams.update({'font.size': 14,
                     'pdf.fonttype': 42}) # TrueType font as requested by PETS
```

# SVT

Our goal in this section is to demonstrate the advantage of using ODP analysis over standard DP analysis for releasing a sparse vector via SVT. We consider a 100-dimensional vector and want to release the indices and values of those entries whose absolute value lies above the threshold $T=500$. For more details see Sec. 4.1 of the paper.

We start by defining some constants.

```python
Delta = 1
c = 20
eps_1_plus_eps_2 = 0.5
eps_3 = 0.5
```

For computing the optimal $\varepsilon_1,\varepsilon_2$ for a total budget of $\alpha = \varepsilon_1 + \varepsilon_2$ we solve the system of equations
$$
\begin{align}
\varepsilon_1+\varepsilon_2 &= \alpha\\
\frac{\varepsilon_1}{\varepsilon_2} &= \frac{1}{(2c)^{2/3}}.
\end{align}
$$
We get
$$
\begin{align}
\varepsilon_1 &= \frac{1}{1+\frac{1}{(2c)^{2/3}}} \frac{1}{(2c)^{2/3}} \alpha\\
\varepsilon_2 &= \alpha - \varepsilon_1.
\end{align}
$$

```python
def split_budget_eps_1_2(budget, c):
    eps_1 = (1/(1+(1/math.pow(2*c, 2/3)))) * (1/math.pow(2*c,2/3)) * budget
    eps_2 = budget - eps_1
    return eps_1, eps_2
```

Setting $\alpha=0.5$ and $c=20$ yields:

```python
eps_1, eps_2 = split_budget_eps_1_2(eps_1_plus_eps_2, c)
print(f'epsilon_1 = {eps_1}, epsilon_2 = {eps_2}')
```

    epsilon_1 = 0.039382262580038824, epsilon_2 = 0.46061773741996115

We will compute an upper bound on the expected noise added with an ODP analysis using Chebyshev's inequality. For this we need to compute the expected number of released vector entries and the variance of this count. We first compute the probability $\mathbb{P}(\nu_i - \rho \geq T_i - q_i(\mathbf{x}))$ that a given entry is released.
We have $\rho \sim \text{Lap}(\Delta/\varepsilon_1)$ and $\nu_i \sim \text{Lap}(2c\Delta/\varepsilon_2)$. Since the distribution of $\rho$ is symmetric around 0, $\nu_i - \rho$ follows the same distribution as $\nu_i + \rho$. Letting $b_1 = \Delta/\varepsilon_1$ and $b_2 = 2c\Delta/\varepsilon_2$, we need to integrate over the convolution of the two Laplace distributions:
$$
\mathbb{P}(\nu_i - \rho \geq T_i - q_i(\mathbf{x})) = \int_{T_i - q_i(\mathbf{x})}^{\infty} \int_{-\infty}^{\infty} \frac{1}{2b_1} \exp\left(-\frac{|\rho|}{b_1}\right) \frac{1}{2b_2} \exp\left(-\frac{|\nu - \rho|}{b_2}\right) d\rho d\nu.
$$

We compute $b_1$ and $b_2$:

```python
def compute_b_1(Delta, eps_1):
    return Delta / eps_1
```

```python
def compute_b_2(c, Delta, eps_2):
    return 2*c*Delta / eps_2
```

```python
b_1 = compute_b_1(Delta, eps_1)
b_2 = compute_b_2(c, Delta, eps_2)
print(f'b_1 = {b_1}, b_2 = {b_2}')
```

    b_1 = 25.392141905702925, b_2 = 86.8399037867068

Now we can (numerically) compute the above integral. We consider $T_i=500$ and for the values of $q_i(\mathbf{x})$ only the cases 0 and 1000. Let
$$
p_0(1) = \mathbb{P}(\nu_i - \rho \geq 500),\quad p_0(0) = 1 - p_0(1)
$$
and
$$
p_{1000}(1) = \mathbb{P}(\nu_i - \rho \geq -500),\quad p_{1000}(0) = 1 - p_{1000}(1)
$$
be the four probabilities that we are interested in.
Due to the symmetry of the Laplace distribution we can further simplify $p_{1000}(1)$ as
$$
p_{1000}(1) = 1 - p_0(1).
$$

```python
def integrand_single_all_args(b_1, b_2, rho, nu):
    fraction = 1 / (4*b_1*b_2)
    power = (-abs(rho)/b_1) - (abs(nu-rho)/b_2)
    return fraction * np.exp(power)
```

```python
integrand_single = lambda rho, nu: integrand_single_all_args(b_1, b_2, rho, nu)
```

```python
p_0_1, error_0_1 = integrate.nquad(integrand_single, [[-np.inf, np.inf], [500, np.inf]])
print(f'p_0(1) = {p_0_1}, error = {error_0_1}')
```

    p_0(1) = 0.0017267930924315283, error = 1.484769393467497e-08

```python
p_1000_1 = 1 - p_0_1
```

Let $n_0$, $n_{1000}$ be the number of small and the number of large queries, respectively, and assume w.l.o.g. the order $q_1(\mathbf{x})=,\dots,q_{n_0}(\mathbf{x})=0$, and $q_{n_0+1}(\mathbf{x})=\dots=q_{n_0+n_{1000}}(\mathbf{x}) = 0$. For $i=1,\dots,n_0$ let $X_i \sim \text{Ber}(p_0)$ be the Bernoulli random variable that is 1 if the corresponding noisy query with $q_i(\mathbf{x}) = 0$ exceeds the threshold, and likewise $X_i \sim \text{Ber}(p_{1000}),\ i=n_0,\dots,n_0+n_{1000},$ for the queries with $q_i(\mathbf{x}) = 1000$. We are interested in the expectation and the variance of the sum
$$
S = \sum_{k=1}^{n_0+n_{1000}} X_i.
$$
We can compute the expectation easily via its linearity from $p_0$ and $p_1$:
$$
\mathbb{E}(S) = n_0 p_0(1) + n_{1000} p_{1000}(1).
$$

```python
def compute_expectation_S(n_0, n_1000, p_0_1, p_1000_1):
    return n_0*p_0_1 + n_1000*p_1000_1
```

The variance can be computed as
\begin{align*}
\text{Var}(S) &= \sum_i \sum_j \text{Cov}(X_i, X_j)\\
&= n_0 \text{Var}(X_1) + n_{1000} \text{Var}(X_{n_0+1}) + n_0(n_0-1)\text{Cov}(X_1,X_2) + n_{1000}(n_{1000}-1)\text{Cov}(X_{n_0+1},X_{n_0+2}) + n_0 n_{1000} \text{Cov}(X_1,X_{n_0+1})
\end{align*}
We have
$$
\text{Var}(X_1) = p_0(1) (1-p_0(1)),\quad \text{Var}(X_{n_0+1}) = p_{1000}(1) (1-p_{1000}(1)).
$$
For the mix terms we need to compute, for $(i=1,j=2),\ (i=n_0+1, j=n_0+2),\ (i=1, j=n_0+1)$,
\begin{align*}
\text{Cov}(X_i,X_j) &= \mathbb{E}(X_i X_j) - \mathbb{E}(X_i) \mathbb{E}(X_j)\\
&= \mathbb{P}(\nu_i - \rho \geq T_i - q_i(\mathbf{x}), \nu_j - \rho \geq T_j - q_j(\mathbf{x})) - \mathbb{E}(X_i) \mathbb{E}(X_j)\\
&= \int_{T_i - q_i(\mathbf{x})}^{\infty} \int_{T_j - q_j(\mathbf{x})}^{\infty} \int_{-\infty}^{\infty} \frac{1}{2b_1} \exp\left(-\frac{|\rho|}{b_1}\right) \frac{1}{2b_2} \exp\left(-\frac{|\nu_j - \rho|}{b_2}\right) \frac{1}{2b_2} \exp\left(-\frac{|\nu_i - \rho|}{b_2}\right) d\rho d\nu_j d\nu_i - \mathbb{E}(X_i) \mathbb{E}(X_j).
\end{align*}

Let
$$
g([a_1, a_2], [b_1, b_2]) = \int_{a_1}^{a_2} \int_{b_1}^{b_2} \int_{-\infty}^{\infty} \frac{1}{2b_1} \exp\left(-\frac{|\rho|}{b_1}\right) \frac{1}{2b_2} \exp\left(-\frac{|\nu_j - \rho|}{b_2}\right) \frac{1}{2b_2} \exp\left(-\frac{|\nu_i - \rho|}{b_2}\right) d\rho d\nu_j d\nu_i.
$$
Note that $g$ is invariant w.r.t. swapping $[a_1, a_2]$ and $[b_1, b_2]$.
We have to compute $\text{Cov}(X_1,X_2)$, $\text{Cov}(X_{n_0+1},X_{n_0+2})$ and $\text{Cov}(X_1,X_{n_0+1})$, and thus the values $g([500,\infty], [500, \infty])$, $g([-500,\infty], [-500, \infty])$ and $g([500,\infty], [-500, \infty])$. To save computation and avoid the numerical problems that scipy has with computing $g([-500,\infty], [-500, \infty])$ directly, we rewrite $g([500,\infty], [-500, \infty])$ as
$$
g([500,\infty], [-500, \infty]) = g([500,\infty], [500, \infty]) + g([500,\infty], [-500, 500])
$$
and $g([-500,\infty], [-500, \infty])$ as
\begin{align*}
g([-500,\infty], [-500, \infty]) &= 1 - (g([-\infty,-500], [-500, \infty]) + g([-500, \infty], [-\infty,-500]) + g([-\infty,-500], [-\infty,-500]))\\
&= 1 - (2g([-\infty,-500], [-500, \infty]) + g([-\infty,-500], [-\infty,-500]))\\
&= 1 - (2g([-\infty,-500], [-500, \infty]) + g([500,\infty], [500, \infty]))\\
&= 1 - (2g([500,\infty], [-500, \infty]) + g([500,\infty], [500, \infty]))\\
&= 1 - (2(g([500,\infty], [500, \infty]) + g([500,\infty], [-500, 500])) + g([500,\infty], [500, \infty]))\\
&= 1 - (3g([500,\infty], [500, \infty]) + 2g([500,\infty], [-500, 500])),
\end{align*}
where we exploit the symmetry of the joint distribution $(\nu_i-\rho, \nu_j-\rho)$.
We hence only have to compute the values $g([500,\infty], [500, \infty])$ and $g([500,\infty], [-500, 500])$.

```python
def integrand_cov_i_j_all_args(b_1, b_2, rho, nu_j, nu_i):
    fraction = 1 / (8*b_1*b_2*b_2)
    power = (-abs(rho)/b_1) - (abs(nu_j-rho)/b_2) - (abs(nu_i-rho)/b_2)
    return fraction * np.exp(power)

integrand_cov_i_j = lambda rho, nu_j, nu_i: integrand_cov_i_j_all_args(b_1, b_2, rho, nu_j, nu_i)
```

```python
def compute_variance_X_i(p):
    return p*(1-p)
```

```python
def compute_integrals_for_covariance():
    # The order of the limits is reversed as compared to how g is defined, because nquad takes the innermost limits first.
    # But it also does not really matter, since we can change the order of integration.
    integration_limits_list = [[[500, np.inf], [500, np.inf]], [[-500, 500], [500, np.inf]]]
    integrals = []
    errors = []
    for integration_limits in integration_limits_list:
        integration_limits.insert(0, [-np.inf, np.inf]) # integration limits for rho
        curr_integral, error = integrate.nquad(integrand_cov_i_j, integration_limits)
        integrals.append(curr_integral)
        errors.append(error)
    print('Integral errors:')
    print(f'Integral 1: {errors[0]}, Integral 2: {errors[1]}')
    return integrals
```

```python
def compute_variance_S(n_0, n_1000, p_0_1, p_1000_1, integrals):
    variances = n_0*compute_variance_X_i(p_0_1) + n_1000*compute_variance_X_i(p_1000_1)

    covariances = 0
    covariances += n_0*(n_0-1) * (integrals[0] - p_0_1*p_0_1)
    covariances += n_1000*(n_1000-1) * ((1-(3*integrals[0] + 2*integrals[1])) - p_1000_1*p_1000_1)
    covariances += n_0*n_1000 * (integrals[0] + integrals[1] - p_0_1*p_1000_1)

    return variances + covariances
```

```python
with warnings.catch_warnings():
    warnings.filterwarnings("ignore", message="The maximum number of subdivisions")
    integrals_for_covariance = compute_integrals_for_covariance() # This may take a couple of minutes
```

    Integral errors:
    Integral 1: 1.4892516520505813e-08, Integral 2: 7.326226175516681e-07

We want to find an upper bound for the expected variance of the noise that needs to be added to the released vector entries according to the ODP analysis. This variance depends on the value of $S$ and it is a monotonically increasing function in $S$. Call this function $f$. We hence want to find an upper bound on
$$
\mathbb{E}(f(S)) = \sum_{i=0}^{c-1} f(i) \mathbb{P}(S=i) + f(c) \mathbb{P}(S\geq c).
$$

Let $\mu$ be the expectation and $\sigma^2$ the variance of $S$. Chebyshev's inequality states that, for any $r>0$,
$$
\mathbb{P}(\lvert S-\mu\rvert\geq r)\leq \frac{\sigma^2}{r^2}.
$$
Note that we can exchange the larger or equal inside the probability by a strictly smaller for all practical purposes, which we will use below.

Let $t = \lceil\mu\rceil-\mu$. Then the inequality implies, for $k=0,1,\dots$,
$$
\mathbb{P}(S\geq \lceil\mu\rceil + k)\leq \frac{\sigma^2}{(t+k)^2}
$$
and
$$
\mathbb{P}(S< \lceil\mu\rceil + k)\geq 1 - \frac{\sigma^2}{(t+k)^2}.
$$
Further,
\begin{equation}
f(\lceil\mu\rceil + k) \mathbb{P}(S\geq\lceil\mu\rceil + k) + f(\lceil\mu\rceil + k+1) \mathbb{P}(S\geq\lceil\mu\rceil + k+1) \leq f(\lceil\mu\rceil + k) \left(\frac{\sigma^2}{(t+k)^2} - \frac{\sigma^2}{(t+k+1)^2}\right) + f(\lceil\mu\rceil + k+1) \frac{\sigma^2}{(t+k+1)^2},\label{eq:estimate_sum_of_two}\tag{1}
\end{equation}
since due to the monotonicity of $f$ the value of the sum is maximized if we shift the maximal amount of probability mass to $\lceil\mu\rceil + k + 1$. The same bound holds when replacing $\mathbb{P}(S\geq\lceil\mu\rceil + k)$ by $\mathbb{P}(S=\lceil\mu\rceil + k)$, and when replacing both $\mathbb{P}(S\geq\lceil\mu\rceil + k)$ by $\mathbb{P}(S=\lceil\mu\rceil + k)$ and $\mathbb{P}(S\geq\lceil\mu\rceil + k+1)$ by $\mathbb{P}(S=\lceil\mu\rceil + k+1)$.

Let $K=c-\lceil\mu\rceil$. The repeated application of $\eqref{eq:estimate_sum_of_two}$ yields
\begin{align*}
\mathbb{E}(f(S)) &= \sum_{i=0}^{c-1} f(i) \mathbb{P}(S=i) + f(c) \mathbb{P}(S\geq c)\\
&\leq f(\lceil\mu\rceil - 1) \left(1 - \frac{\sigma^2}{t^2}\right) + \sum_{k=0}^{K-1} f(\lceil\mu\rceil + k) \left(\frac{\sigma^2}{(t+k)^2} - \frac{\sigma^2}{(t+k+1)^2}\right) + f(c) \frac{\sigma^2}{(t+K)^2}.\label{eq:final_E_S_bound}\tag{2}
\end{align*}
Note the different summation limits in the two lines. We bound all summands with $i\geq\lceil\mu\rceil$ in the first sum using the upper bound $\eqref{eq:estimate_sum_of_two}$ $K$ times. We put the remaining probability mass
$$
1 - \frac{\sigma^2}{t^2} = 1 - \left(\sum_{k=0}^{K-1} \left(\frac{\sigma^2}{(t+k)^2} - \frac{\sigma^2}{(t+k+1)^2}\right) + \frac{\sigma^2}{(t+K)^2}\right)
$$
towards the summands with $i<\lceil\mu\rceil$, and use the fact that $f(i)\leq f(\lceil\mu\rceil - 1)$ for $i=0,\dots,\lceil\mu\rceil - 1$.
We assume that $\mu$ is not an integer, so that $t\neq 0$ and we do not divide by 0. Otherwise we need to start the summation at $k=1$ instead of $k=0$ and have $f(\mu) \left(1 - \frac{\sigma^2}{(0+1)^2}\right)$ as the first summand in Eq. $\eqref{eq:final_E_S_bound}$ instead of $f(\lceil\mu\rceil - 1) \left(1 - \frac{\sigma^2}{t^2}\right)$.

If $\frac{\sigma^2}{t^2}>1$, we get an even better bound, since we only need to sum up probability bounds until we have reached a probability of 1. Let $k^*$ be the smallest non-negative integer such that $\frac{\sigma^2}{(t+k)^2}<1$. Then
$$
\mathbb{E}(f(S)) \leq f(\lceil\mu\rceil + k^* - 1) \left(1 - \frac{\sigma^2}{(t+k^*)^2}\right) + \sum_{k=k^*}^{K-1} f(\lceil\mu\rceil + k) \left(\frac{\sigma^2}{(t+k)^2} - \frac{\sigma^2}{(t+k+1)^2}\right) + f(c) \frac{\sigma^2}{(t+K)^2}.
$$

We already have functions that compute the expectation and variance of $S$. What remains is a function for computing $f$.
For a given per-entry budget $\varepsilon$, the standard deviation of the added Laplace noise is
$$
\sqrt{2}\frac{\Delta}{\varepsilon}.
$$
In the non-ODP case we have a budget of
$$
\varepsilon = \frac{\varepsilon_3}{c}
$$ per entry.

Let $c'$ be the number of released entries. Then in the ODP case we have a budget of
$$
\varepsilon = \frac{1}{c'}\left(\varepsilon_3 + \frac{c-c'}{c}\varepsilon_2\right)
$$
per entry (see paper).

```python
def std_dev_laplace(Delta, eps):
    return math.sqrt(2) * Delta / eps
```

```python
def non_odp_eps_per_entry(eps_3, c):
    return eps_3 / c
```

```python
def odp_eps_per_entry(eps_2, eps_3, c, c_prime):
    eps_per_entry = eps_3 + ((c-c_prime)/c)*eps_2
    eps_per_entry /= c_prime
    return eps_per_entry
```

Now the function $f$:

```python
def compute_variance_laplace_odp(Delta, eps_2, eps_3, c, c_prime):
    if c_prime == 0:
        return 0
    return std_dev_laplace(Delta, odp_eps_per_entry(eps_2, eps_3, c, c_prime))
```

And the upper bound for $\mathbb{E}(f(S))$:

```python
def compute_upper_bound_expected_variance_laplace(Delta, eps_2, eps_3, c, n_0, n_1000, p_0_1, p_1000_1, integrals):
    mu = compute_expectation_S(n_0, n_1000, p_0_1, p_1000_1)
    mu_int = math.ceil(mu)
    t = mu_int - mu
    sigma_2 = compute_variance_S(n_0, n_1000, p_0_1, p_1000_1, integrals)
    f = lambda c_prime: compute_variance_laplace_odp(Delta, eps_2, eps_3, c, c_prime)

    bound = 0
    K = c - mu_int
    k = 0
    for k in reversed(range(K)):
        if sigma_2/math.pow(t, 2) >= 1:
            k += 1 # set k = k^*
            break
        fraction_1 = sigma_2 / math.pow(t+k, 2)
        fraction_2 = sigma_2 / math.pow(t+k+1, 2)
        bound += f(mu_int + k) * (fraction_1 - fraction_2)
    bound += f(mu_int+k-1) * (1 - (sigma_2/math.pow(t, 2) + k))
    bound += f(c) * (sigma_2 / math.pow(t+K, 2))

    worst_case_bound = compute_variance_laplace_odp(Delta, eps_2, eps_3, c, c) # We never release more than c entries
    return min(bound, worst_case_bound)
```

```python
non_odp_std_dev = std_dev_laplace(Delta, non_odp_eps_per_entry(eps_3, c))
odp_std_dev_list = [compute_upper_bound_expected_variance_laplace(Delta, eps_2, eps_3, c, 100-num_large_entries, num_large_entries, p_0_1, p_1000_1, integrals_for_covariance) for num_large_entries in range(21)]
```

```python
plt.plot(range(0,21), np.ones(21)*non_odp_std_dev, label='without ODP')
plt.plot(range(0,21), odp_std_dev_list, label='with ODP (upper bound)')
plt.xlabel('Number of large entries')
plt.ylabel('Expected std deviation of noise')
plt.legend()
plt.tight_layout()
plt.savefig('svt.pdf')
```

# ERM

In this section we investigate the ODP variant of output perturbation for releasing a noisy logistic regression model, but only its error lies below a given threshold. We compute the amount of noise that needs to be added to the error before the comparison with the threshold. For more details see Sec. 6.1 of the paper.

We start by computing $a$ from line 2 of Alg. 5.

```python
def compute_a(n_train, n_test, Lambda):
    return max(2/n_test, 2*(np.exp(2/(n_train*Lambda)) - 1))
```

We want to plot the 95th percentile of the noise added to the model error.

```python
def score_noise_quantile(a, epsilon_2, F):
    '''
    Computes the F-quantile of the noise added to the model error.
    F must be >= 1/2.
    '''

    laplace_param = a/epsilon_2
    return -laplace_param * np.log(2-2*F)
```

We define the different settings that we consider. In all settings 70% of the data is used for training and 30% is used for testing.

```python
epsilons_2_s = [0.1, 0.2, 0.5]
dataset_sizes = range(1000, 5000)
fraction_test = 0.3 # fraction of test records
Lambda = 1
quantile = 0.95
```

```python
for epsilon_2 in epsilons_2_s:
    percentiles = [score_noise_quantile(compute_a((1-fraction_test)*size, fraction_test*size, Lambda), epsilon_2, quantile)
                   for size in dataset_sizes]
    plt.plot(dataset_sizes, percentiles, label=f'$\\varepsilon_2={epsilon_2}$')

plt.ylabel('95th percentile of error noise')
plt.xlabel('Total number of database records')
plt.legend()
plt.tight_layout()
plt.savefig('erm.pdf')
```


```python
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import numpy as np
import sympy as sym
```

```python
def f(X, n, q, b, c):
    Z = np.zeros(len(X))
    for i in range(len(X)):
        for j in range(int(n)):
            for k in range(int(n)):
                Z[i] += q[j][k]*X[i][j]*X[i][k]

        for j in range(int(n)):
            Z[i] += b[j]*X[i][j]
    return Z
```

```python
def f2(X, Y, n, q, b, c):
    Z = q[0][0]*X*X + q[0][1]*X*Y + q[1][0]*Y*X + q[1][1]*Y*Y + b[0]*X + b[1]*Y + int(c[0])
    return Z
```

```python
def f_mesh(X, Y, q, b, c):
    Z = np.zeros(len(X))
    Z = q[0][0]*X*X + q[0][1]*X*Y + q[1][0]*Y*X + q[1][1]*Y*Y + b[0]*X + b[1]*Y + c
    return Z
```

```python
def z_func(X_old, n, q, b, c):

    x, y, t = sym.symbols('x y t')

    X = sym.Matrix([[x, y]])

    T = sym.Matrix([[t]])

    df = sym.Matrix([[sym.diff(f2(x, y, n, q, b, c), x),
                      sym.diff(f2(x, y, n, q, b, c), y)]])
    z = X - t * df

    z = f2(z[0], z[1], n, q, b, c)
    z_diff = sym.diff(z, t)
    eqn = sym.Eq(z_diff)
    sol = sym.solve((eqn), (t))
    sym.expr = sol[0]
    sym.expr = sym.expr.subs([(x, X_old[0][0]), (y, X_old[0][1])])
    return sym.expr
```

```python
def steepest(X, Y, n, q, b, c, precision=0.0001, max_iter=200):
    X_old = np.zeros((1, 2))
    X_new = np.zeros((1, 2))
    dfr = np.zeros((1, 2))

    X_new[0][0] = 4.9
    X_new[0][1] = 4.9

    i = 0
    Xs = np.zeros((max_iter, 2))
    Ys = np.zeros(max_iter)

    x, y = sym.symbols('x y')
    df1 = sym.diff(f2(x, y, n, q, b, c), x)
    df2 = sym.diff(f2(x, y, n, q, b, c), y)

    while  np.all(abs(X_new - X_old)) > precision and max_iter > i:
        Xs[i] = X_new
        Ys[i] = f2(X_new[0][0], X_new[0][1], n, q, b, c)

        X_old = X_new

        dfr[0][0] = df1.evalf(subs={x: X_old[0][0], y: X_old[0][1]})
        dfr[0][1] = df2.evalf(subs={x: X_old[0][0], y: X_old[0][1]})

        eps = z_func(X_old, n, q, b, c)

        X_new = X_old - eps * dfr

        # print("X_new: {}, X_old: {}, eps:{}, dfr: {}".format(X_new, X_old, eps, dfr))
        i += 1

    print("Finished with {} step".format(i))
    if (i < max_iter):
        Xs[i] = X_new
        Ys[i] = f2(X_new[0][0], X_new[0][1], n, q, b, c)

        for j in range(max_iter - 1, i, -1):
            Xs = np.delete(Xs, j, axis=0)
            Ys = np.delete(Ys, j, axis=0)

    return Xs, Ys
```

```python
X1 = np.arange(-5, 5, 0.1)
Y1 = np.arange(-5, 5, 0.1)
Z1 = np.zeros(len(X1))

X_new = np.zeros((100,2))

for i in range(len(X1)):
    X_new[i][0] = X1[i]
    X_new[i][1] = Y1[i]

Z1 = f(X_new, n, q, b, c)

x_list, y_list = steepest(X_new, Z1, n, q, b, c)
```

    Finished with 10 step

```python
n = input("Enter power of your function: ")
q = []
for i in range(int(n)):
    q.append(input("Enter the function's coefficient matrix q's row q[{}]: ".format(i)).split())
b = input("Enter the function's coefficient b matrix: ").split()
c = input("Enter the function's constant c: ")

for i in range(int(n)):
    q[i] = list(map(float, q[i]))
b = list(map(float, b))
c = list(map(float, c))
```

    Enter power of your function: 2
    Enter the function's coefficient matrix q's row q[0]: 1 0
    Enter the function's coefficient matrix q's row q[1]: 0 2
    Enter the function's coefficient b matrix: 0 0
    Enter the function's constant c: 0

```python
X1, Y1 = np.meshgrid(X1, Y1)
Z1 = f_mesh(X1, Y1, q, b, c)

X, Y = zip(*x_list)
Z = y_list

ax = plt.subplots(nrows=1, ncols=1, figsize=(10,10))
cs = plt.contour(X1, Y1, Z1)
plt.clabel(cs, inline=1, fontsize=10)
colors = ['b', 'g', 'm', 'c', 'orange']
for j in range(1, len(X)):
    ax[1].annotate('', xy=(X[j], Y[j]), xytext=(X[j-1], Y[j-1]),
                arrowprops={'arrowstyle': '->', 'color': 'r', 'lw': 1},
                va='center', ha='center')
ax[1].scatter(X, Y, s=40, lw=0)
ax[1].set_xlabel('X')
ax[1].set_ylabel('Y')
ax[1].set_title('Minimizing function')
```

```python
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(X1, Y1, Z1, rstride=1, cstride=1,
                cmap='viridis', edgecolor='none')
ax.set_title('surface')
plt.show()
```

```python
plt.figure()
cs = plt.contour(X1, Y1, Z1)
plt.clabel(cs, inline=1, fontsize=10)
plt.title('Two-dimensional contour plot')
plt.show()
```

```python

```


```python
import os
os.chdir("..")

import liesym as ls
from sympy import Matrix, Rational
```

#### Cartan Matrix

```python

A3 = ls.A(3)
A3.cartan_matrix

```

$\displaystyle \left[\begin{matrix}2 & -1 & 0\\-1 & 2 & -1\\0 & -1 & 2\end{matrix}\right]$

#### Positive Roots

```python
A3.positive_roots

```

    [Matrix([[1, 0, 0, -1]]),
     Matrix([[0, 1, 0, -1]]),
     Matrix([[1, 0, -1, 0]]),
     Matrix([[0, 1, -1, 0]]),
     Matrix([[0, 0, 1, -1]]),
     Matrix([[1, -1, 0, 0]])]

#### Simple Roots

```python
A3.simple_roots
```

    [Matrix([[1, -1, 0, 0]]), Matrix([[0, 1, -1, 0]]), Matrix([[0, 0, 1, -1]])]

#### Fundamental Weights

```python
A3.fundamental_weights # Orthogonal Basis
```

    [Matrix([[3/4, -1/4, -1/4, -1/4]]),
     Matrix([[1/2, 1/2, -1/2, -1/2]]),
     Matrix([[1/4, 1/4, 1/4, -3/4]])]

#### Dimension of representation

```python

print("Dim | Rep (Omega)")
print("---------")
for i in A3.fundamental_weights:
    print(" ", A3.dim(i), "|", A3.to_omega(i))
```

    Dim | Rep (Omega)
    ---------
      4 | Matrix([[1, 0, 0]])
      6 | Matrix([[0, 1, 0]])
      4 | Matrix([[0, 0, 1]])

#### Name of rep

Commonly in literature (especially physics), names of the reps are the dimension rather than the matrix rep.

```python
A3.dim_name(Matrix([[0, 0, 1]]))
```

$\displaystyle \bar{4}$

```python
A3.irrep_lookup(r"\bar{4}")
```

$\displaystyle \left[\begin{matrix}0 & 0 & 1\end{matrix}\right]$

#### Tensor product decomps

The decomp of irreps from a product of irreps

```python
results = A3.tensor_product_decomposition([
    Matrix([[1,0,0]]),
    Matrix([[1,0,0]]),
])

for i in results:
    print("Rep:", A3.to_omega(i),"Dim Name:", A3.dim_name(i))
```

    Rep: Matrix([[0, 1, 0]]) Dim Name: 6
    Rep: Matrix([[2, 0, 0]]) Dim Name: \bar{10}

#### Lie Groups

Currently supports SU(N), SO(N), Sp(N)

```python
su2 = ls.SU(2)
su2.generators()
```

    [Matrix([
     [  0, 1/2],
     [1/2,   0]]),
     Matrix([
     [  0, -I/2],
     [I/2,    0]]),
     Matrix([
     [1/2,    0],
     [  0, -1/2]])]

Structure constants. SU(2) structure constants are $e_{ijk}$

```python
su2.structure_constants()
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}0 & 0 & 0\\0 & 0 & 1\\0 & -1 & 0\end{matrix}\right] & \left[\begin{matrix}0 & 0 & -1\\0 & 0 & 0\\1 & 0 & 0\end{matrix}\right] & \left[\begin{matrix}0 & 1 & 0\\-1 & 0 & 0\\0 & 0 & 0\end{matrix}\right]\end{matrix}\right]$

```python
A1 = ls.A(1)
[A1.to_omega(x) for x in A1.simple_roots]
```

    [Matrix([[2]])]

Quadratic Casimir

```python
s = ls.Sp(6)
r = s.algebra.fundamental_weights[0]
s.quadratic_casimir(r)
```

$\displaystyle \frac{7}{2}$

```python
from liesym.algebras._backend import _rust_wrapper
```


```python
import sympy as sym
x, L, C, D, c_0, c_1, = sym.symbols('x L C D c_0 c_1')

def model1(f, L, D):
    """Solve -u'' = f(x), u(0)=0, u(L)=D."""
    # Integrate twice
    u_x = - sym.integrate(f, (x, 0, x)) + c_0
    u = sym.integrate(u_x, (x, 0, x)) + c_1
    # Set up 2 equations from the 2 boundary conditions and solve
    # with respect to the integration constants c_0, c_1
    r = sym.solve([u.subs(x, 0)-0,  # x=0 condition
                   u.subs(x,L)-D],  # x=L condition
                  [c_0, c_1])       # unknowns
    # Substitute the integration constants in the solution
    u = u.subs(c_0, r[c_0]).subs(c_1, r[c_1])
    u = sym.simplify(sym.expand(u))
    return u

def model2(f, L, C, D):
    """Solve -u'' = f(x), u'(0)=C, u(L)=D."""
    u_x = - sym.integrate(f, (x, 0, x)) + c_0
    u = sym.integrate(u_x, (x, 0, x)) + c_1
    r = sym.solve([sym.diff(u,x).subs(x, 0)-C,  # x=0 cond.
                   u.subs(x,L)-D],              # x=L cond.
                  [c_0, c_1])
    u = u.subs(c_0, r[c_0]).subs(c_1, r[c_1])
    u = sym.simplify(sym.expand(u))
    return u

def model3(f, a, L, C, D):
    """Solve -(a*u')' = f(x), u(0)=C, u(L)=D."""
    au_x = - sym.integrate(f, (x, 0, x)) + c_0
    u = sym.integrate(au_x/a, (x, 0, x)) + c_1
    r = sym.solve([u.subs(x, 0)-C,
                   u.subs(x,L)-D],
                  [c_0, c_1])
    u = u.subs(c_0, r[c_0]).subs(c_1, r[c_1])
    u = sym.simplify(sym.expand(u))
    return u

def demo():
    f = 2
    u = model1(f, L, D)
    print(('model1:', u, u.subs(x, 0), u.subs(x, L)))
    print((sym.latex(u, mode='plain')))
    u = model2(f, L, C, D)
    #f = x
    #u = model2(f, L, C, D)
    print(('model2:', u, sym.diff(u, x).subs(x, 0), u.subs(x, L)))
    print((sym.latex(u, mode='plain')))
    u = model3(0, 1+x**2, L, C, D)
    print(('model3:', u, u.subs(x, 0), u.subs(x, L)))
    print((sym.latex(u, mode='plain')))

if __name__ == '__main__':
    demo()

```

    ('model1:', x*(D + L*(L - x))/L, 0, D)
    \frac{x \left(D + L \left(L - x\right)\right)}{L}
    ('model2:', -C*L + C*x + D + L**2 - x**2, C, D)
    - C L + C x + D + L^{2} - x^{2}
    ('model3:', (C*atan(L) - C*atan(x) + D*atan(x))/atan(L), C, D)
    \frac{C \operatorname{atan}{\left(L \right)} - C \operatorname{atan}{\left(x \right)} + D \operatorname{atan}{\left(x \right)}}{\operatorname{atan}{\left(L \right)}}

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/39_U_XX_F_SYMPY.ipynb


# Diffusion Theory

```python
# Our numerical workhorses
import numpy as np
import pandas as pd

# Import matplotlib stuff for plotting
import matplotlib.pyplot as plt
import matplotlib.cm as cm

# Seaborn, useful for graphics
import seaborn as sns

# favorite Seaborn settings for notebooks
rc={'lines.linewidth': 2,
    'axes.facecolor' : 'F4F3F6',
    'axes.edgecolor' : '000000',
    'axes.linewidth' : 1.2,
    'grid.color' : 'a6a6a6',
    'lines.linewidth': 2,
    'axes.labelsize': 18,
    'axes.titlesize': 20,
    'xtick.major' : 13,
    'xtick.labelsize': 'large',
    'ytick.labelsize': 13,
    'font.family': 'Lucida Sans Unicode',
    'grid.linestyle': ':',
    'grid.linewidth': 1.5,
    'mathtext.fontset': 'stixsans',
    'mathtext.sf': 'sans',
    'legend.frameon': True,
    'legend.fontsize': 13}

plt.rc('text.latex', preamble=r'\usepackage{sfmath}')
plt.rc('mathtext', fontset='stixsans', sf='sans')
sns.set_style('darkgrid', rc=rc)
sns.set_palette("colorblind", color_codes=True)
sns.set_context('notebook', rc=rc)

# Magic function to make matplotlib inline; other style specs must come AFTER
%matplotlib inline

# This enables SVG graphics inline (only use with static plots (non-Bokeh))
%config InlineBackend.figure_format = 'svg'
```

# Kolmogorov Forward Equation.

## Mutation and Selection.

The steady state distribution for mutation and selection is of the form
$$
P(f) \propto \left[ f(1-f) \right]^{2N\mu - 1},
\tag{1}
$$
where $f$ is the allele frequency $\mu$ is the mutation rate, and $N$ is the population size (effective population size can be).

Let's plot this for different values of $\mu$

```python
def mut_drift(f, N_mu):
    prob = (f * (1 - f))**(2 * N_mu - 1)
    idx = np.logical_and(prob >=0, prob!=np.inf) # to avoid indeterminations
    return f[idx], prob[idx]
```

```python
# Define parameters
N_mu = [0.1, 1 / 2, 10, 100]
freq = np.linspace(0, 1, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for nm in N_mu:
    f, prob = mut_drift(freq, nm)
    plt.plot(f, prob / np.sum(prob),
             label=str(nm))
ax.legend(title='$N \mu$')
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('mutation-drift')
ax.axes.get_yaxis().set_ticks([])
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/mutation_drift.png')
```

    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in power
      from ipykernel import kernelapp as app

## Selection and Drift

#### Haploid organisms

The steady state distribution of allele frequency for an allele $A$ with fitness $\omega_A = 1$ that is in a population with another allele $a$ with fitness $\omega_a = 1 - s$ is of the form
$$
P(f) \propto \frac{e^{-2Ns(1-f)}}{f(1-f)}
\tag{2}
$$
Let's plot it for different values of $s$.

```python
def sel_drift_hap(f, N_s):
    prob = np.exp(-2 * N_s * (1 - f)) / (f * (1 - f))
    idx = np.logical_and(prob >=0, prob!=np.inf) # to avoid indeterminations
    return f[idx], prob[idx]
```

```python
# Define parameters
N_s = [0, 0.1, 1, 10]
freq = np.linspace(0, 1, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for ns in N_s:
    f, prob = sel_drift_hap(freq, ns)
    plt.plot(f, prob, #/ np.sum(prob),
             label=str(ns))
ax.legend(loc='upper center', title='$N s$')
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('selection-drift (haploids)')
ax.axes.get_yaxis().set_ticks([])
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/sel_drift_haploid.png')
```

    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in true_divide
      from ipykernel import kernelapp as app

### Diploid organisms

#### Heterozygote advantage.
For the case of a diploid organism where the genotypes have fitness values
\begin{align}
\omega_{AA} = 1,\\
\omega_{Aa} = 1 + s,\\
\omega_{aa} = 1,
\end{align}
we have that the steady state distribution of alleles is of the form
$$
P(f) \propto \frac{e^{4Ns f(1 - f)}}{f(1 - f)}
\tag{3}
$$
Let's look at some distributions.

```python
def sel_drift_dip_hetero_advantage(f, N_s):
    prob = np.exp(4 * N_s * f * (1 - f)) / (f * (1 - f))
    idx = np.logical_and(prob >=0, prob!=np.inf) # to avoid indeterminations
    return f[idx], prob[idx]
```

```python
# Define parameters
N_s = [1, 10, 50]
freq = np.linspace(0, 1, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for ns in N_s:
    f, prob = sel_drift_dip_hetero_advantage(freq, ns)
    plt.plot(f, prob / np.sum(prob),
             label=str(ns))
ax.legend(loc=0, title='$N s$')
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('selection-drift (diploids heterozygote advantage)', fontsize=14)
ax.axes.get_yaxis().set_ticks([])
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/sel_drift_diploid_hetero_advantage.png')
```

    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in true_divide
      from ipykernel import kernelapp as app

#### Heterozygote Intermediate.
For the case of a diploid organism where the genotypes have fitness values
\begin{align}
\omega_{AA} = 1 + 2s,\\
\omega_{Aa} = 1 + s,\\
\omega_{aa} = 1,
\end{align}
we have that the steady state distribution of alleles is of the form
$$
P(f) \propto \frac{e^{4Ns f}}{f(1 - f)}
\tag{3}
$$
Let's look at some distributions.

```python
def sel_drift_dip_hetero_intermediate(f, N_s):
    prob = np.exp(4 * N_s * f ) / (f * (1 - f))
    idx = np.logical_and(prob >=0, prob!=np.inf) # to avoid indeterminations
    return f[idx], prob[idx]
```

```python
# Define parameters
N_s = [0.1, 1, 5]
freq = np.linspace(0, 0.95, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for ns in N_s:
    f, prob = sel_drift_dip_hetero_intermediate(freq, ns)
    plt.plot(f, prob / np.sum(prob),
             label=str(ns))
ax.legend(loc=0, title='$N s$')
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('selection-drift (diploids heterozygote intermediate)', fontsize=14)
ax.axes.get_yaxis().set_ticks([])
ax.set_xlim([0, 1])
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/sel_drift_diploid_hetero_intermediate.png')
```

    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in true_divide
      from ipykernel import kernelapp as app

### Mutation-Selection-Drift.
For the case of a diploid organism where the genotypes have fitness values
\begin{align}
\omega_{AA} = 1 - s,\\
\omega_{Aa} = 1,\\
\omega_{aa} = 1,
\end{align}
And the mutation rates are given by
\begin{align}
\mu_{A \rightarrow a} = 0\\
\mu_{a \rightarrow A} = \mu\\
\end{align}

we have that the steady state distribution of alleles is of the form
$$
P(f) \propto \frac{e^{-2Ns f^2} (1 - f)^{2N \mu}}{f(1 - f)}
\tag{3}
$$
Let's look at some distributions.

```python
def mut_sel_drift(f, N, s, mu):
    prob = np.exp(- 2 * N * s * f**2) * (1 - f)**(2 * N * mu) / (f * (1 - f))
    idx = np.logical_and(prob >=0, prob!=np.inf) # to avoid indeterminations
    return f[idx], prob[idx]
```

```python
# Define parameters
N = [10**2, 10**4]
s = 0.0001
mu = [10**-1, 10**-6]
```

```python
freq = np.linspace(0, 1, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for n in N:
    for m in mu:
        f, prob = mut_sel_drift(freq, n, s, m)
        plt.plot(f, prob / np.sum(prob),
                label='$N = {0:d}, \mu = {1:f}$'.format(n, m))
ax.legend(loc=0)
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('selection-drift (diploids heterozygote intermediate)', fontsize=14)
ax.axes.get_yaxis().set_ticks([])
ax.set_xlim([0, 1])
ax.margins(0.02)
plt.tight_layout()
# plt.savefig('fig/sel_drift_diploid_hetero_intermediate.png')
```

    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in true_divide
      from ipykernel import kernelapp as app
    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: invalid value encountered in true_divide
      from ipykernel import kernelapp as app
    /Users/razo/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:3: RuntimeWarning: invalid value encountered in greater_equal
      app.launch_new_instance()

```python
freq = np.linspace(1E-3, 0.02, 200)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for n in N:
    for m in mu:
        f, prob = mut_sel_drift(freq, n, s, m)
        plt.plot(f, prob / np.sum(prob),
                label='$N = {0:d}, \mu = {1:f}$'.format(n, m))
ax.legend(loc=0)
ax.set_xlabel('allele frequency $f$')
ax.set_ylabel('$\propto$ probability')
ax.set_title('selection-drift (diploids heterozygote intermediate)', fontsize=14)
ax.axes.get_yaxis().set_ticks([])
# ax.set_xlim([0, 1])
ax.margins(0.02)
plt.tight_layout()
# plt.savefig('fig/sel_drift_diploid_hetero_intermediate.png')
```

# Kolmogorov Backwards Equation.

## Probability of fixation (Haploid)

Using the Kolmogorov backwards equation we derived that the probability of an allele being fixed is of the form
$$
P(1, \infty \mid f_o) = \frac{1 - e^{-2N_e s f_o}}{1 - e^{-2N_e s}},
\tag{4}
$$
where $f_o$ is the initial frequency of the allele, $N_e$ is the effective population size, and $s$ is the selection coefficient.

Let's define a function to compute this quantity.

```python
def fix_prob_haploid(N, s, fo):
    return (1 - np.exp(-2 * N * s * fo)) / (1 - np.exp(-2 * N * s))
```

```python
N_array = 10**np.array([2, 3, 4])
s_array = np.linspace(-0.005, 0.005, 100)

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for n in N_array:
    prob = fix_prob_haploid(n, s_array, 1 / n)
    plt.plot(s_array, prob,
            label='$N = {0:d}$'.format(n))
ax.legend(loc=0)
ax.set_xlabel('selection coefficient $s$')
ax.set_ylabel('fixation probability')
ax.set_title('haploids', fontsize=14)
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/fix_prob_haploid.png')
```

## Probability of fixation (diploids)

For the diploid case we have a very similar equation for the probability of fixation of an allele
$$
P(1, \infty \mid f_o) = \frac{1 - e^{-4N_e h s f_o}}{1 - e^{-2N_e h s}},
\tag{5}
$$
where $f_o$ is the initial frequency of the allele, $N_e$ is the effective population size, $h$ is the dominance, and $s$ is the selection coefficient.

Let's define a function to compute this quantity.

```python
def fix_prob_diploid(N, h, s, fo):
    return (1 - np.exp(-4 * N * h * s * fo)) / (1 - np.exp(-4 * N * h * s))
```

```python
h_array = [0.05, 0.5, 1]
s_array = np.linspace(-0.005, 0.005, 100)
N = 1E3

fig, ax = plt.subplots(1, 1)
# Loop through mutation rates and plot the distribution
for h in h_array:
    prob = fix_prob_diploid(N, h, s_array, 1 / N)
    plt.plot(s_array, prob,
            label='$h = {0:.1f}$'.format(h))
ax.legend(loc=0)
ax.set_xlabel('selection coefficient $s$')
ax.set_ylabel('fixation probability')
ax.set_title('diploids', fontsize=14)
ax.margins(0.02)
plt.tight_layout()
plt.savefig('fig/fix_prob_diploids.png')
```

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth.

# Coding Assignment: Rocket

The equations of motion for a rocket in purely vertical flight are given by

\begin{align}
\frac{dh}{dt} &= v\\
(m_s+m_p) \frac{dv}{dt}& = -(m_s+m_p)g + \dot{m}_pv_e - \frac{1}{2}\rho v|v|AC_D
\end{align}

$h$ is the altitude of the rocket

$m_s = 50kg$ is the weight of the rocket shell

$g = 9.81 \frac{m}{s^2}$

$\rho = 1.091 \frac{kg}{m^3}$ is the average air density (assumed constant throughout flight)

$A = \pi r^2$ is the maximum cross sectional area of the rocket, where $r = 0.5 m$

$v_e = 325 \frac{m}{s}$ is the exhaust speed

$C_D = 0.15 $ is the drag coefficient

$m_{po} = 100 kg$ at time $t = 0$ is the initial weight of the rocket propellant

The mass of the remaining propellant is given by:

$$m_p = m_{po} - \int^t_0 \dot{m}_p d\tau$$

where $\dot{m}_p$ is the time-varying burn rate given by the following figure:

Propellant Burn Rate

Using Euler's method with a timestep of $\Delta t=0.1s$, create a Python script to calculate the altitude and velocity of the rocket from launch until crash down.

## Assessment:

To check your answers, you can register for [MAE 6286: Practical Numerical Methods with Python](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about).

1. At time $t=3.2s$, what is the mass (in kg) of rocket propellant remaining in the rocket?

2. What is the maximum speed of the rocket in $\frac{m}{s}$?
    At what time does this occur (in seconds)?
    What is the altitude at this time (in meters)?

3. What is the rocket's maximum altitude during flight (in meters)? At what time (in seconds) does this occur?

4. At what time (in seconds) does the rocket impact the ground? What is the velocity of the rocket (in $\frac{m}{s}$) at time of impact?

## Derivation of the rocket equations

In case you are kind of confused about the rocket equations, here we show how to get to them.

Newton's second law states that the acceleration of the vehicle times its mass is equal to all the forces acting on it. Therefore,

\begin{equation}
(m_s + m_p)\frac{d\bf{v}}{dt}=\sum {\bf F}.
\end{equation}
In the above formula we have assumed that the propellant inside the rocket and the rocket move at the same velocity (in other words, their relative velocity is negligible).

Two of the external forces acting on the rocket are,

\begin{align}
{\bf F}_g&= (m_s+m_p)\bf{g} \quad (\rm{Gravity}),\\
{\bf F}_d&= - \frac{1}{2} \rho_a \mathbf{v} |\mathbf{v}| A  C_D \quad (\rm{Drag}).
\end{align}

We also need to consider the force resulting from the ejection of the propellant. During an interval $dt$, the engine of the rocket ejects downwards a mass of propellant given by $\dot m_p dt$. Relative to the rocket, the speed of the ejected burning gas is assumed constant and equal to $v_e$ (the exhaust speed). The momentum variation induced on the exhaust gas by the engine during that interval is therefore, $d{\bf p}_{gas} = \dot m_p {\bf v}_e dt$. Again using Newton's second law we conclude that the force applied by the rocket on the gas is,

\begin{align}
{\bf F}_{rocket\rightarrow gas} = \frac{d{\bf p}_{gas}}{dt} = \dot m_p {\bf v}_e
\end{align}

Using Newton's third law (|action| = |reaction|), the force exerted by the exhaust gas on the rocket is then,

\begin{align}
{\bf F}_{gas\rightarrow rocket} = -{\bf F}_{rocket\rightarrow gas} = -\dot m_p {\bf v}_e
\end{align}

If we collect all the forces acting on the rocket we finally have:

\begin{align}
(m_s + m_p)\frac{d\bf{v}}{dt}=(m_s+m_p){\bf g}- \frac{1}{2} \rho_a \mathbf{v} |v| A  C_D -\dot m_p {\bf v}_e
\end{align}

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>

```python

```


## 함수

**함수(function)**는 입력 값을 출력 값으로 바꾸어 출력하는 관계(relationship)를 말한다.

**정의역(domain)** : 함수에서 입력변수가 가질 수 있는 값의 집합

**공역(range)** : 출력변수가 가질 수 있는 값의 집합

### 변수

**변수(variable)** : 어떤 숫자를 대표하는 기호.

**입력변수(input variable)** : 입력 값을 대표하는 변수

**출력변수(output variable)** : 출력 값을 대표하는 변수

### 불연속함수
- 데이터 분석에서 많이 사용되는 불연속함수들

### 부호함수

입력이 양수이면 1, 음수이면 -1, 0이면 0을 출력하는 $x=0$에서 불연속인 함수이다.

넘파이에서는 `sign()` 명령으로 구현

```python
np.sign(-0.01), np.sign(0), np.sign(0.01)
```

    (-1.0, 0, 1.0)

### 단위계단함수

단위계단함수(Heaviside step function)도 $x=0$에서 불연속인 함수이다. 넘파이 구현이 없으므로 직접 구현해야 한다.

### 지시함수

함수 이름에 아래 첨자로 미리 지정된 값이 들어오면 출력이 1이 되고 아니면 출력이 0이 된다.

지시함수는 데이터 중에서 특정한 데이터만 선택하여 갯수를 세는데 사용된다.

### 역함수
- 어떤 함수의 입력/출력 관계와 정반대의 입출력 관계를 갖는 함수

$
y = f(x), \;\;\; \rightarrow \;\;\; x = f^{-1}(y)
$

### 데이터 분석에서 많이 사용되는 함수들

### 다항식함수

거듭제곱 항의 선형조합으로 이루어진 함수다.

$
f(x) = c_0 + c_1 x + c_2 x^2 + \cdots + c_n x^n
$

### 최대함수와 최소함수

최대함수는 두 개의 인수 중에서 큰 값을 출력하는 함수이다.

$
\begin{align}
\max(x, y)
=
\begin{cases}
x & \text{ if } x \geq y \\
y & \text{ if } x < y
\end{cases}
\end{align}
$

최소함수는 최대함수와 반대로 두 개의 인수 중 작은 값을 출력하는 함수이다.

$
\begin{align}
\min(x, y)
=
\begin{cases}
x & \text{ if } x \leq y \\
y & \text{ if } x > y
\end{cases}
\end{align}
$

### 지수함수

밑을 오일러 수 $e$로 하여 거듭제곱을 하는 함수를 **지수함수(exponential function)**라고 한다.

$
y = e^x
$

$
y = \exp (x) =\exp x
$

```python
np.exp(-2), np.exp(0), np.exp(2)
```

    (0.1353352832366127, 1.0, 7.38905609893065)

#### 지수함수 특성

* 양수($e$)를 거듭제곱한 값이므로 항상 양수다.
* $x=0$일 때 1이 된다.
* $x$가 양의 무한대로 가면($x \rightarrow \infty$), 양의 무한대로 다가간다.
* $x$가 음의 무한대로 가면($x \rightarrow -\infty$), 0으로 다가간다.
* $x_1 > x_2$이면 $\exp{x_1} > \exp{x_2}$이다.

```python
np.exp(5 + 3), np.exp(5) * np.exp(3)
```

    (2980.9579870417283, 2980.9579870417283)

### 로지스틱함수
- 지수함수를 변형한 함수
- 회귀 분석이나 인공신경망에서 자주 사용

$
\begin{align}
\sigma(x) = \dfrac{1}{1 + \exp(-x)}
\end{align}
$

### 로그함수

```python
np.log(10)
```

    2.302585092994046

로그는 지수함수의 역함수이다.

$
y = \log x
$

#### 로그함수 특징

- $x$값, 즉 입력변수값이 양수이어야 한다. 0이거나 음수이면 정의되지 않는다.
- $x > 1$면 $y > 0$ (양수)
- $x = 1$이면 $y = 0$
- $0 < x < 1$면 $y < 0$ (음수)
- $x_1 > x_2$면 $\log{x_1} > \log{x_2}$이다.

#### 로그함수의 성질 1
- 로그함수는 곱하기를 더하기로 변환한다.

$
\log{(x_1 \cdot x_2)} = \log{x_1} + \log{x_2}
$

$
\log x^n = n \log x
$

#### 로그함수의 성질 2
- 어떤 함수에 로그를 적용해도 함수의 최고점, 최저점의 위치는 변하지 않는다.
- 로그를 취한 함수에 대해 최적화를 하는 경우가 많다.

#### 로그함수의 성질 3
- 0부터 1사이의 작은 값을 확대시켜 보여준다.
- 0부터 1사이의 구간을 음의 무한대부터 0까지로 확장시켜주기 때문에 0과 1사이에 있는 작은 값을 더 확실하게 비교할 수 있도록 한다.

### 소프트플러스함수
- 지수함수와 로그함수를 결합하여 만든 함수
- 최대함수(ReLU)와 비슷하지만 $x=0$ 근처에서 값이 부드럽게 변한다는 장점

$
\zeta(x) = \log( 1 + \exp(x))
$

### 다변수함수
- 복수의 입력변수를 가지는 함수

$
w = f(x, y)
$

### 함수의 평행이동
- 단변수함수를 그래프 상에서 오른쪽으로 $a$만큼 평행

$
f(x) \;\; \rightarrow \;\; f(x-a)
$

### 함수의 스케일링
- 단변수함수를 x축 방향으로 k배만큼 늘릴때

$
f(x) \;\; \rightarrow \;\; f\left(\frac{x}{k}\right)
$

- 단변수함수를 y축 방향으로 k배만큼 늘릴때

$
f(x) \;\; \rightarrow \;\; kf(x)
$


```python
try:
    from openmdao.utils.notebook_utils import notebook_mode
except ImportError:
    !python -m pip install openmdao[notebooks]
```

# Using the Case Viewer

The CaseViewer is a tool view interactive viewing of recorded cases data in Jupyter notebooks.
This is currently an experimental feature of OpenMDAO but it's useful enough in its current state that we've included it in the package.

## Example: Queen Dido's Problem

Queen Dido's problem is an optimization problem that is also a legend about the founding of Carthage.
Supposedly Queen Dido was allowed all the land she could enclose within the hide of an ox.
Cutting the hide into very thin strips provided her with a long, but finite, length of "fencing."
What arrangement of this fencing provides the greatest possible enclosed area?

If we assume a straight edge along one side of the fence (such as a riverbank or shoreline), how should the fencing be arranged to capture the greatest possible area?

The following component takes the distance of the river bank, $L$, as well as the distance of the "fence posts" from the riverbank, $y$, and computes the enclosed area and the perimeter.

```python
import openmdao.api as om
import numpy as np
```

```python
class Fence(om.ExplicitComponent):

    def initialize(self):
        self.options.declare('n', types=int, default=50, desc='number of fenceposts')

    def setup(self):
        n = self.options['n']
        self.add_input('L', val=10.0, units='m')
        self.add_input('y', val=np.zeros(n), units='m')

        self.add_output('perimeter', val=0.0, units='m')
        self.add_output('area', val=0.0, units='m**2')
        self.add_output('x', val=np.linspace(0, 100, n), units='m')

        self.declare_partials(of='*', wrt='*', method='cs')

    def compute(self, inputs, outputs):
        n = self.options['n']
        L = inputs['L']
        y = inputs['y']

        x = np.linspace(0, L, n)

        dx = x[1:] - x[:-1]
        dy = y[1:] - y[:-1]

        mean_y = 0.5 * (y[1:] + y[:-1])

        outputs['perimeter'] = np.sum(np.sqrt(dx**2+dy**2))
        outputs['area'] = np.sum(dx * mean_y)
        outputs['x'] = x
```

The optimization problem is to arrange these fence posts such that the perimeter $p$ does not exceed some value while maximizing the enclosed area $A$.

In addition, the first and last fenceposts should be placed along the river's edge.

\begin{align}
\mathrm{Minimize}\;J &= -A \\
\mathrm{Subject\;to:} \\
& 0 \le y_i \quad \mathrm{for} \quad i=1 ... N \\
& y[0] = 0 \; \mathrm{m}\\
& y[N] = 0 \; \mathrm{m}\\
& p = 300 \; \mathrm{m}
\end{align}

```python
import matplotlib.pyplot as plt

def plot_fence(p, idx=0, fig=None, axes=None):
    if axes is None:
        fig, axes = plt.subplots(3, 1, figsize=(12, 6), tight_layout=True, sharex=True)
    axes[idx].plot(p.get_val('fence.x'), p.get_val('fence.y'), 'ko-')
    axes[idx].fill_between(p.get_val('fence.x'), p.get_val('fence.y'), color='tab:green')
    axes[idx].fill_between([-1, 12], [-2, -2], color='tab:blue')
    A = p.get_val('fence.area', units='m**2')[0]
    perimeter = p.get_val('fence.perimeter', units='m')[0]
    L = p.get_val('fence.L', units='m')[0]
    axes[idx].annotate(f'p = {perimeter:6.2f} $m$', (4, 2.5), fontsize=16, color='white')
    axes[idx].annotate(f'A = {A:6.2f} $m^2$', (4, 0.5), fontsize=16, color='white')
    ht = -1
    w = L
    axes[idx].annotate('', xy=(0, ht), xytext=(w, ht), textcoords=axes[idx].transData, arrowprops=dict(arrowstyle='<->', color='white'))
    axes[idx].annotate('', xy=(0, ht), xytext=(w, ht), textcoords=axes[idx].transData, arrowprops=dict(arrowstyle='|-|', color='white'))
    bbox=dict(fc='tab:blue', ec='none')
    axes[idx].text(w/2, ht, f'L={L:6.2f} m', ha='center', va='center', color='white', fontsize=16, bbox=bbox)
    return fig, axes

```

```python
N = 4

prob = om.Problem()

prob.model.add_subsystem('fence', Fence(n=N))

prob.model.add_design_var('fence.L', lower=1)
prob.model.add_design_var('fence.y', lower=0, indices=om.slicer[1:-1])
prob.model.add_constraint('fence.perimeter', equals=100 * np.pi, ref=100 * np.pi)
prob.model.add_objective('fence.area', ref=-1000.0)

prob.driver = om.pyOptSparseDriver(optimizer='SLSQP')

prob.setup(force_alloc_complex=True)

prob.set_val('fence.L', val=10)

fig = None
axes = None

for i, inp in enumerate(((10, [0, 3, 5, 0]),
                         (11, [0, 4, 7, 0]),
                         (9, [0, 7, 7, 0]))):
    L, ys = inp
    prob.set_val('fence.L', val=L)
    prob.set_val('fence.y', val=ys)
    prob.run_model()
    fig, axes = plot_fence(prob, idx=i, fig=fig, axes=axes)

fig.suptitle('Potential arrangements of 4 Fence posts and the resulting perimeter and enclosed area.', fontsize=16)
plt.show()
```

## Solving the Optimization Problem and Recording the Driver Iterations

In the following code, we set up the optimziation problem to find the location of 20 fence posts which maximize area while limiting perimeter to $300$ meters.
A recorder is added to the driver that records the design variables, constraints, and objectives.

```python
N = 20
p = om.Problem()

p.model.add_subsystem('fence', Fence(n=N))

p.model.add_design_var('fence.L', lower=1)
p.model.add_design_var('fence.y', lower=0, indices=om.slicer[1:-1])
p.model.add_constraint('fence.perimeter', equals=100 * np.pi, ref=100 * np.pi)
p.model.add_objective('fence.area', ref=-1000.0)

p.driver = om.pyOptSparseDriver(optimizer='SLSQP')

p.driver.add_recorder(om.SqliteRecorder('driver_cases.db'))
p.driver.recording_options['includes'] = ['*']
p.driver.recording_options['record_desvars'] = True
p.driver.recording_options['record_constraints'] = True
p.driver.recording_options['record_objectives'] = True

p.setup(force_alloc_complex=True)

p.set_val('fence.y', val=np.zeros(N))

p.run_driver();
```

```python
from openmdao.utils.assert_utils import assert_near_equal

assert_near_equal(p.get_val('fence.area'), 821.85, tolerance=1.0E-2)
```

## Invoking the CaseViewer

In the following line we invoke the CaseViewer.
It will not be displayed in the static documentation here, but executing this notebook in Jupyter will provide a GUI like that shown in the figure below.

```python
om.CaseViewer('driver_cases.db');
```

## Plotting a scalar vs. the case index

To see how the value of a variable evolves over the iteration history, use `Case Index` as the X-Axis variable.

For instance, to see how the area (the objective function) changes over iterations, choose `outputs -> fence.area` as the Y-Axis variable.

## Plotting an array vs. the case index

When an array-valued variable is plotted against the Case Index, each value in the array is plotted at the X-Axis value corresponding to each case.
The color of each point represents its index in the array, with warmer colors being higher indices.

For instance, to see how the values of `fence.y` at each iteration, choose `inputs -> fence.y` as the Y-Axis variable.

If this type of plot is too noisy for your data, you can use the `Y Transform` options to retrieve things like the maximum value in the array, the minimum value in the array, or the norm of the array.

## Plotting one array vs. another

Sometimes it's useful to visualize the evolution of a solution as the plot of one array-shaped variable vs. another.
In our case, the shape of the fence is represented by plotting `outputs -> fence.x` on the X-Axis and `inputs -> fence.y` on the Y-Axis.

When plotting arrays, the X-Axis is required to be viewable as a flattened array.
The Y-Axis variable must have the same number of points as the X-Axis variable in its first dimension.
In our use-case here, `fence.x` and `fence.y` have the same shape.
If they did not, then the slice text boxes could be used to reshape the variable in the appropriate way.
The default value, `[...]`, plots the entire array.

When plotting one array vs. another, the CaseViewer will connect the points in each case using a line, and color that line based on the index of the case in the selected cases box at the top of the GUI.
Scrolling the slider will highlight a particular case.

## Current limitations of the CaseViewer

CaseViewer is currently limited to one instance per notebook.  Instantiating more than one will cause the plotting interface to fail.


```python
import numpy as np
import sympy as sym
import numba
from scipy.integrate import ode
from scipy import optimize
from collections import deque
import pydae.build as db
```

```python
R = 0.1  # Resistor (Ohm)
L = 0.01 # Inductor (L)

# Control design
T_pi = L/R    # PI time constant
K_p = L/0.01  # PI proportational gain

params= {'R' : 0.1, 'L':0.01, 'V_max':1.0, 'V_min':0.0, 'K_p':K_p, 'T_pi':T_pi, 'K_aw':1.0}
u_ini_dict = { 'i_ref':1.0}  # for the initialization problem
u_run_dict = { 'i_ref':1.0}  # for the running problem (here initialization and running problem are the same)

x_list = ['i','xi']    # [inductor current, PI integrator]
y_ini_list = ['y','v'] # for the initialization problem
y_run_list = ['y','v'] # for the running problem (here initialization and running problem are the same)

sys_vars = {'params':params,
            'u_list':u_run_dict,
            'x_list':x_list,
            'y_list':y_run_list}

```

```python
exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants
```

```python
epsilon = i_ref - i

di  = 1/L*(v-i*R)
dxi = epsilon - K_aw*(y - v)

g_1 = -y + K_p*(epsilon +1/T_pi*xi)
g_2 = -v + sym.Piecewise((V_min,y<V_min),(V_max,y>V_max),(y,True))

h_1 = v

x = sym.Matrix(x_list)

sys = {'name':'rl_ctrl_sat_aw',
       'params_dict':params,
       'f_list':[di,dxi],
       'g_list':[g_1,g_2],
       'x_list':x_list,
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'h_1':h_1}}

sys = db.system(sys)
db.sys2num(sys)
```

```python

```


Non Linear Regression Analysis

## RIHAD VARIAWA, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING
If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O _datasets/china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("_datasets/china_gdp.csv")
df.head(10)
```

    2019-01-22 08:45:15 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "_datasets/china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

```

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

Data Science Professional/6 - Machine Learning With Python/2.4 Non-linear Regression.ipynb


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

## Independence ##

Jointly distributed random variables $X$ and $Y$ are *independent* if
$$
P(X \in A, Y \in B) = P(X \in A)P(Y \in B)
$$
for all intervals $A$ and $B$.

Let $X$ have density $f_X$, let $Y$ have density $f_Y$, and suppose $X$ and $Y$ are independent. Then if $f$ is the joint density of $X$ and $Y$,

$$
\begin{align*}
f(x, y)dxdy &\sim P(X \in dx, Y \in dy) \\
&= P(X \in dx)P(Y \in dy) ~~~~~ \text{(independence)} \\
&= f_X(x)dx f_Y(y)dy \\
&= f_X(x)f_Y(y)dxdy
\end{align*}
$$

Thus if $X$ and $Y$ are independent then their joint density is given by

$$
f(x, y) = f_X(x)f_Y(y)
$$

This is the *product rule for densities*: the joint density of two independent random variables is the product of their densities.

### Independent Standard Normal Random Variables ###
Suppose $X$ and $Y$ are i.i.d. standard normal random variables. Then their joint density is given by

$$
f(x, y) = \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}x^2} \cdot \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}y^2}, ~~~~ -\infty < x, y < \infty
$$

Equivalently,
$$
f(x, y) = \frac{1}{2\pi} e^{-\frac{1}{2}(x^2 + y^2)}, ~~~~ -\infty < x, y < \infty
$$

Here is a graph of the joint density surface.

```python
def indep_standard_normals(x,y):
    return 1/(2*math.pi) * np.exp(-0.5*(x**2 + y**2))

Plot_3d((-4, 4), (-4, 4), indep_standard_normals, rstride=4, cstride=4)
```

Notice the circular symmetry of the surface. This is because the formula for the joint density involves the pair $(x, y)$ through the expression $x^2 + y^2$ which is symmetric in $x$ and $y$.

Notice also that $P(X = Y) = 0$, as the probability is the volume over a line. This is true of all pairs of independent random variables with a joint density: $P(X = Y) = 0$. So for example $P(X > Y) = P(X \ge Y)$. You don't have to worry about whether or not to the inequality should be strict.

### The Larger of Two Independent Exponential Random Variables ###
Let $X$ and $Y$ be independent random variables. Suppose $X$ has the exponential $(\lambda)$ distribution and $Y$ has the exponential $(\mu)$ distribution. The goal of this example is to find $P(Y > X)$.

By the product rule, the joint density of $X$ and $Y$ is given by

$$
f(x, y) ~ = ~ \lambda e^{-\lambda x} \mu e^{-\mu y}, ~~~~ x > 0, ~ y > 0
$$

The graph below shows the joint density surface in the case $\lambda = 0.5$ and $\mu = 0.25$, so that $E(X) = 2$ and $E(Y) = 4$.

```python
def independent_exp(x, y):
    return 0.5 * 0.25 * np.e**(-0.5*x - 0.25*y)

Plot_3d((0, 10), (0, 10), independent_exp)
```

To find $P(Y > X)$ we must integrate the joint density over the upper triangle of the first quadrant, a portion of which is shown below.

```python
# NO CODE
plt.axes().set_aspect('equal')
xx = np.arange(0, 10.1, 0.1)
yy = 10*np.ones(len(xx))
plt.fill_between(xx, xx, yy, alpha=0.3)
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$Y > X$ (portion of infinite region)');
```

The probability is therefore
$$
P(Y > X) ~ = ~ \int_0^\infty \int_x^\infty \lambda e^{-\lambda x} \mu e^{-\mu y} dy dx
$$

We can do this double integral without much calculus, just by using probability facts.

$$
\begin{align*}
P(Y > X) &= \int_0^\infty \int_x^\infty \lambda e^{-\lambda x} \mu e^{-\mu y} dy dx \\ \\
&= \int_0^\infty \lambda e^{-\lambda x} \big{(} \int_x^\infty \mu e^{-\mu y} dy\big{)} dx \\ \\
&= \int_0^\infty \lambda e^{-\lambda x} e^{-\mu x} dx ~~~~~~ \text{(survival function of } Y\text{, evaluated at } x \text{)} \\ \\
&= \frac{\lambda}{\lambda + \mu} \int_0^\infty (\lambda + \mu) e^{-(\lambda + \mu)x} dx \\ \\
&= \frac{\lambda}{\lambda + \mu} ~~~~~~~ \text{(total integral of exponential }
(\lambda + \mu) \text{ density is 1)}
\end{align*}
$$

Thus

$$
P(Y > X) ~ = ~ \frac{\lambda}{\lambda + \mu}
$$

Analogously,

$$
P(X > Y) ~ = ~ \frac{\mu}{\lambda + \mu}
$$

Notice that the two chances are proportional to the parameters. This is consistent with intuition if you think of $X$ and $Y$ as two lifetimes. If $\lambda$ is large, the corresponding lifetime $X$ is likely to be short, and therefore $Y$ is likely to be larger than $X$ as the formula implies.

If $\lambda = \mu$ then $P(Y > X) = 1/2$ which you can see by symmetry since $P(X = Y) = 0$.

If we had attempted the double integral in the other order – first $x$, then $y$ – we would have had to do more work. The integral is

$$
\int_0^\infty \int_0^y \lambda e^{-\lambda x} \mu e^{-\mu y} dx dy
$$

Let's take the easy way out by using `SymPy` to confirm that we will get the same answer.

```python
# Create the symbols; they are all positive

x = Symbol('x', positive=True)
y = Symbol('y', positive=True)
lamda = Symbol('lamda', positive=True)
mu = Symbol('mu', positive=True)
```

```python
# Construct the expression for the joint density

f_X = lamda * exp(-lamda * x)
f_Y = mu * exp(-mu * y)
joint_density = f_X * f_Y
joint_density
```

```python
# Display the integral – first x, then y

Integral(joint_density, (x, 0, y), (y, 0, oo))
```

```python
# Evaluate the integral

answer = Integral(joint_density, (x, 0, y), (y, 0, oo)).doit()
answer
```

```python
# Confirm that it is the same
# as what we got by integrating in the other order

simplify(answer)
```

```python

```


 $$\left\{ \begin{array}{lcc}
            \dot{x}_{1}=x_{1}-x_{2}-(x_{1}^{2}+\frac{3}{2}x_{2}^{2})x_{1} \\
             \\ \dot{x}_{2}=x_{1}+x_{2}-(x_{1}^{2}+\frac{1}{2}x_{2}^{2})x_{2}
             \end{array}
\right.$$

```python
import sympy as sym
```

```python
#Con esto las salidas van a ser en LaTeX
sym.init_printing(use_latex=True)
```

```python
x_1, x_2 ,theta = sym.symbols('x_1 x_2 theta')
```

```python
X = sym.Matrix([x_1, x_2])
X
```

```python
f_1 = x_1 - x_2 - (x_1**2 + sym.Rational(3,2)* x_2**2)*x_1
f_1
```

```python
f_2 = x_1 + x_2 - (x_1**2 + sym.Rational(1,2)* x_2**2)*x_2
```

```python
F = sym.Matrix([f_1,f_2])
F
```

```python
A = F.jacobian(X)
A
```

```python
A_1 = A.subs({x_1:0,x_2:0})
A_1
```

```python
A_1.eigenvals()
```

```python
sym.latex(A_1)
```

    '\\left[\\begin{matrix}1 & -1\\\\1 & 1\\end{matrix}\\right]'

```python
#sym.plot_implicit((f_1 )&(f_2))
```

```python
ine = (f_1 <= 0)&(f_2 <= 0)
```

```python
campo = 2*x_1*F[0] + 2*x_2*F[1]
campo = campo.simplify()
```

```python
sym.plot_implicit(campo < 0,xlabel=r'$x_1$',ylabel=r'$x_2$',title='Zona de existencia de ciclo limite')
```

```python
sym.exp(x_2)
```

```python
F.subs({x_1:2,x_2:-2})
```

```python
#sym.solve(F)
```

```python
expr = sym.cos(theta)**4 + sym.sin(theta)**4 - 5*sym.cos(theta)**2 * sym.sin(theta)**2
expr
```

```python
sym.plot(expr)
```

```python
expr = sym.diff(F[0],x_1) + sym.diff(F[1],x_2)
expr
```

```python

```