# Computing the distribution of mRNA using the solution of the chemical master equation for a two state promoter.

```python
import glob
# Our numerical workhorses
import numpy as np
import scipy.optimize
import scipy.special
import scipy.integrate
import mpmath
import pandas as pd
# Import libraries to parallelize processes
from joblib import Parallel, delayed

# Import matplotlib stuff for plotting
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib as mpl

# Seaborn, useful for graphics
import seaborn as sns

# Import the utils for this project
import chann_cap_utils as chann_cap

# Set PBoC plotting style
chann_cap.set_plotting_style()

# Magic function to make matplotlib inline; other style specs must come AFTER
%matplotlib inline

# This enables SVG graphics inline (only use with static plots (non-Bokeh))
%config InlineBackend.figure_format = 'svg'

figdir = '../../fig/chemical_master_analytic_mRNA/'
tmpdir = '../../tmp/'
```

#  Analytical solution of the chemical master equation for the two-state repressor mRNA distribution

[Sanchez and Kondev's paper](http://www.sciencedirect.com/science/article/pii/S1046202313000959) as well as [Shahrezaei and Swain](http://www.pnas.org/content/105/45/17256.short) present the analytical distribution for the mRNA copy number of a two-state promoter at steady state. We can list the possible "reactions" (state transitions in the Markovian chain) with their respective rates as
\begin{align}
\begin{array}{cc}
\text{reaction} & \text{rate}\\
\text{active} \rightarrow \text{inactive} & k_{off}\\
\text{inactive} \rightarrow \text{active} & k_{on}\\
m \rightarrow m+1 & r\\
m \rightarrow m-1 & m \cdot \gamma\\
\end{array}
\end{align}

With these states and transition rates we can write the distribution $P(m)$ where $ m $ is the number of mRNA molecules is given by
\begin{equation}
    P(m) = \frac{\Gamma \left( \frac{k_{on}}{\gamma} + m \right)}{\Gamma(m + 1) \Gamma\left( \frac{k_{off}+k_{on}}{\gamma} + m \right)} \frac{\Gamma\left( \frac{k_{off}+k_{on}}{\gamma} \right)}{\Gamma\left( \frac{k_{on}}{\gamma} \right)} \left( \frac{r}{\gamma} \right)^m F_1^1 \left( \frac{k_{on}}{\gamma} + m, \frac{k_{off} + k_{on}}{\gamma} + m, -\frac{r}{\gamma} \right),
\end{equation}
where $r$ is the production rate, $\gamma$ is the degradation rate, $k_{off}$ and $k_{on}$ are the transition rates between the inactive and the active state respectively and $\Gamma$ is the Gamma function.

The function $F^1_1$ is the so-called [confluent hypergeometric function of first kind](http://mathworld.wolfram.com/ConfluentHypergeometricFunctionoftheFirstKind.html) which is implemented in [SymPy](http://docs.sympy.org/0.7.1/modules/mpmath/functions/hypergeometric.html) and in [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.special.hyp1f1.html#scipy.special.hyp1f1)

**It is important to clarify that there is some inconsistent nomenclature between Sanchez and Kondev's paper and Jones and Brewster's paper. For Sanchez (and for this distribution) $k_{on}$ represents the rate at which the promoters goes to the active state. For Brewster that rate is given by $k_{off}$. So that is important to keep in mind when extracting the parameters from these papers**

<!---
But to have anlytical progress we can use its integral representation given by
\begin{equation}
F^1_1(a; b; z) = \frac{\Gamma(b)}{\Gamma(b - a) \Gamma(a)} \int^1_0 \mathbf{e}^{z t} t^{a - 1} (1 - t)^{b - a - 1} dt
\end{equation}
-->

Numerically it is better to use the log of the probability to facilitate the computation. Since scipy has a special function to compute the log of the gamma function we can express $ \ln P(m \mid C, R)$ as
\begin{equation}
    \ln P(m \mid C, R) = \ln \Gamma \left( \frac{k_{on}}{\gamma} + m \right)
    - \ln \Gamma \left( m + 1 \right)
    - \ln \Gamma \left( \frac{k_{off} + k_{on}}{\gamma} + m \right)
    + \ln \Gamma \left( \frac{k_{off} + k_{on}}{\gamma} \right)
    - \ln \Gamma \left( \frac{k_{on}}{\gamma} \right)
    + m \ln \left( \frac{r}{\gamma} \right)
    + \ln F^1_1 \left( \frac{k_{on}}{\gamma} + m,
    \frac{k_{off} + k_{on}}{\gamma} + m,
    -\frac{r}{\gamma} \right)
\end{equation}

### Including the effect of the inducer (lactose) in the model.

In order to compute the joint probability distribution of the gene expression level  given a substrate concentration $ C $ we will use the MWC model in the following way:
Since $k_{off}$ is a function of the number of **active** number of repressors (i.e. able to bind the DNA), we can multiply this rate by the probability of the repressors being active $p_{act}$ which would tell us the fraction of repressors that are in the active state on average.
For the *lac* repressor this probability is given by
\begin{equation}
    p_{act}(C) = \frac{\left(1 + \frac{C}{K_A} \right)^2}{\left(1 + \frac{C}{K_A} \right)^2 + \omega \left(1 + \frac{C}{K_I} \right)^2}
\end{equation}
where all the parameters were determined in Tal's project.

This means that the conditional probability we are looking for is given by
\begin{equation}
 P(m \mid C, R) = P \left(m \mid k_{off} = k_0 \cdot R \cdot p_{act}(C)\right)
\end{equation}

Now we can define the function to compute the log conditional probability
\begin{align}
    \ln P(m \mid C, R)
\end{align}

```python
# define a np.frompyfunc that allows us to evaluate the sympy.mp.math.hyp1f1
np_log_hyp= np.frompyfunc(lambda x, y, z: \
mpmath.ln(mpmath.hyp1f1(x, y, z, zeroprec=1000)), 3, 1)

def log_p_m_mid_C(C, mRNA, rep, ki, ka, epsilon, kon, k0, gamma, r_gamma,
                 logC=False):
    '''
    Computes the log conditional probability lnP(m|C,R),
    i.e. the probability of having m mRNA molecules given
    an inducer concentration C and a repressor copy number R.

    Parameters
    ----------
    C : float.
        Concentration at which evaluate the probability. if logC=True, then
        this array is defined as log10(C).
    mRNA : float.
        mRNA copy number at which evaluate the probability.
    repressor : float.
        repressor copy number per cell.
    ki, ka : float.
        dissociation constants for the inactive and active states respectively
        in the MWC model of the lac repressor.
    epsilon : float.
        energetic barrier between the inactive and the active state.
    kon : float.
        rate of activation of the promoter in the chemical master equation
    k0 : float.
        diffusion limited rate of a repressor binding the promoter
    gamma : float.
        half-life time for the mRNA.
    r_gamma : float.
        average number of mRNA in the unregulated promoter.
    logC : Bool.
        boolean indicating if the concentration is given in log scale

    Returns
    -------
    log probability lnP(m|c,R)
    '''
    # Convert C and mRNA into np.arrays
    C = np.array(C)
    mRNA = np.array(mRNA)
    if logC:
        C = 10**C

    # Calculate the off rate including the MWC model
    koff = k0 * rep * chann_cap.p_act(C, ka, ki, epsilon)

    # Compute the probability
    lnp = scipy.special.gammaln(kon / gamma + mRNA) \
    - scipy.special.gammaln(mRNA + 1) \
    - scipy.special.gammaln((koff + kon) / gamma + mRNA) \
    + scipy.special.gammaln((koff + kon) / gamma) \
    - scipy.special.gammaln(kon / gamma) \
    + mRNA * np.log(r_gamma) \
    + np_log_hyp(kon / gamma + mRNA,
            (koff + kon) / gamma + mRNA, -r_gamma)

    return lnp.astype(float)
```

# Determining the parameters for the function.

Let's compile all the parameters I think are true for this function to reproduce experimental results.

As mentioned above the parameter $k_{off}$ is given by a diffusion-limited rate $k_0$ times the concentration of active repressors $[R_{act}] = [R] \cdot p_{act}$. [Jones & Brewster](http://science.sciencemag.org/content/346/6216/1533)'s paper report using a value of
\begin{equation}
k_0 = 2.7 \times 10^{-3} (\text{s nM})^{-1},
\end{equation}
where to convert from repressor copy number to concentration we will use an *E. coli* volume of *1 fL*.

For the $k_{on}$ parameter we can use the fact that the mean of this distribution is given by
\begin{equation}
\left\langle m \right\rangle = \frac{r}{\gamma} \frac{k_{on}}{k_{on} + k_{off}}.
\end{equation}

When computing the fold-change with respect to the $\Delta lacI$ strain where  $k_{off} = 0$ we find that
\begin{align}
\text{fold-change} &= \frac{k_{on}}{k_{on} + k_{off}}\\
&= \frac{k_{on}}{k_{on} + k_0 \cdot p_{act} \cdot [R]}\\
&= \frac{1}{1 + \frac{k_0 \cdot p_{act} \cdot [R]}{k_{on}}}.
\end{align}

This last functional form of the fold-change is completely analogous to the one determined by [Garcia & Phillips](http://www.pnas.org/content/108/29/12173.short) using the language of statistical mechanics to compute the mean fold change
\begin{equation}
\text{fold-change} = \frac{1}{1 + \frac{2R \cdot p_{act}}{N_{NS}} e^{-\beta \Delta \epsilon_r}},
\end{equation}
where $R$ is the absolute copy number of repressors per cell, $N_{NS}$ is the number of non-specific binding sites for the repressors (in our case the entire *E. coli genome* $4.6 \times 10^6$), $\beta$ is the inverse temperature times the Boltzmann constant and $\Delta \epsilon_r$ is the binding energy of the repressor to the operator binding site. Our equation is different from the one used in the paper since we include the factor $p_{act}$ to account for the number of active repressors in the cell.

Equating both of this expressions and solving for $k_{on}$ we obtain
\begin{equation}
k_{on} = \frac{k_0 \cdot [R] \cdot N_{NS}}{2R}e^{\beta \Delta \epsilon_r}.
\end{equation}
For the units to cancel out properly the repressor concentration has to be given in nM. If we consider that the repressor concentration is equal to
\begin{equation}
[R] = \frac{R}{V_{cell}}\cdot \frac{1}{Av},
\end{equation}
where $R$ is the absolute repressor copy number per cell, $V_{cell}$ is the cell volume and $Av$ is Avogadro's number. We said that the cell volume is in the order of 1 fL = $10^{-15}$ L, and Avogadro's number is $6.022 \times 10^{23}$. if we further include the conversion factor to turn M into nM we find that
\begin{equation}
[R] = \frac{R}{10^{-15}L}\cdot \frac{1}{6.022 \times 10^{23}} \cdot \frac{10^9 nmol}{1 mol},
\end{equation}
which is equal to

```python
1E15 * 1E9 / 6.022E23
```

    1.6605778811026237

Therefore we conclude that $[R] \approx 1.66 \times R$ nM. This gives
\begin{equation}
k_{on} = \frac{1.66}{2} k_0 \cdot N_{NS} e^{\beta \Delta \epsilon_r}.
\end{equation}
Computing the values for the known operators we find

```python
# Define a simple lambda function to compute kon as a function of the binding
# energy
kon_fn = lambda epsilon, k0: 1.66 / 2 * k0 * 4.6E6 * np.exp(epsilon)
k0 = 2.7E-3
kon_array = [kon_fn(-17, k0), kon_fn(-15.3, k0),
             kon_fn(-13.9, k0), kon_fn(-9.7, k0)]

print("""
k_on values for the 4 operators
-------------------------------
k_on Oid : {0:.5f} s^-1
k_on O1  : {1:.4f} s^-1
k_on O2  : {2:.3f} s^-1
k_on O3  : {3:.3f} s^-1
""".format(kon_array[0], kon_array[1], kon_array[2], kon_array[3]))
```

    k_on values for the 4 operators
    -------------------------------
    k_on Oid : 0.00043 s^-1
    k_on O1  : 0.0023 s^-1
    k_on O2  : 0.009 s^-1
    k_on O3  : 0.632 s^-1

For the $\frac{r}{\gamma}$ parameter Jones & Brewster report having measured for the *LacUV5* promoter a mean copy number of *15.7 mRNA per cell*. For the parameter $\gamma$ we will use [this value](http://bionumbers.hms.harvard.edu/bionumber.aspx?&id=107514&ver=1&trm=mean%20mrna%20life%20e%20coli) from bionumbers of 352 seconds for the mean life.

As it is today the parameters for the MWC model are note entirely determined. For now we will use some preliminary values
\begin{align}
\Delta\varepsilon_{AI} &= 4.5\\
K_A &= 0.21 mM\\
K_I &= 0.00087 mM
\end{align}

# Testing the function with these parameters
Now lets plot this function with the parameters given in Daniel and Brewster's Science paper using the values for $O1$

```python
# Specify some parameters
k0 = 2.7E-3
params = dict(ka=0.21, ki=0.00087, epsilon=4.5, kon=kon_fn(-13.9, k0),
              k0=k0, gamma=0.00284, r_gamma=15.7)
```

Compute the probability distributions for different repressor copy numbers

```python
# Define the range of m on which evaluate the function
m_grid = np.arange(0,30)

# Define the repressor copy number for which evaluate the function
R_array = [0, 11, 130, 870]

# Compute the conditional probability for ech repressor copy number
p_mC_mat = np.empty([len(R_array), len(m_grid)])
# Define the colors for the plot
colors = sns.color_palette('Blues_r', len(R_array) + 1)[:-1]

for i, r in enumerate(R_array):
    params['kon'] = chann_cap.kon_fn(-9.7, k0)
    p_mC_mat[i, :] = np.exp(log_p_m_mid_C(C=0, mRNA=m_grid,
                                          rep=r * 1.66, **params))
```

```python
fig, ax = plt.subplots(2, 1, figsize=(6,5), sharex=True)
for i, R in enumerate(R_array):
    ax[0].plot(m_grid, p_mC_mat[i, :],
             label=r'${0:0.0f}$'.format(R * 2), drawstyle='steps',
              color=colors[i])
    ax[0].fill_between(m_grid, p_mC_mat[i, :], color=colors[i], alpha=0.1,
                      step='pre')
    ax[1].plot(m_grid, np.cumsum(p_mC_mat[i, :]), drawstyle='steps',
              color=colors[i])

ax[0].legend(loc=0, title='repressors / cell')
ax[0].set_ylabel('probability')
ax[0].margins(0.02)

ax[1].set_xlabel('number of mRNA')
ax[1].set_ylabel('CDF')
ax[1].margins(0.02)

plt.tight_layout()
plt.savefig(figdir + 'mRNA_dist_O3.pdf')
```

## Testing the effect of the inducer

As a test let's also plot for different number of repressors the mRNA distribution at different inducer concentrations.

```python
# Specify some parameters
k0 = 2.7E-3
params = dict(ka=0.21, ki=0.00087, epsilon=4.5, kon=kon_fn(-9.7, k0),
              k0=k0, gamma=0.00284, r_gamma=15.7)
```

Compute the probability distributions for different inducer concentrations

```python
# Define the range of m on which evaluate the function
m_grid = np.arange(0,30)

# Define the concentrations to test
concentrations = [0, 10, 50, 100, 500] # uM IPTG

# Compute the conditional probability for ech repressor copy number
p_mC_mat = np.empty([len(concentrations), len(m_grid)])
for i, c in enumerate(concentrations):
    p_mC_mat[i, :] = np.exp(log_p_m_mid_C(C=c / 1E3, mRNA=m_grid,
                                          rep=130 * 1.66, **params))
```

```python
# Define the colors for the plot
colors = sns.color_palette('Blues_r', len(concentrations) + 1)[:-1]

fig, ax = plt.subplots(2, 1, figsize=(7, 6), sharex=True)
for i, c in enumerate(concentrations):
    # Plot the PMF
    ax[0].plot(m_grid, p_mC_mat[i, :],
             label=r'{0:0.0f} $\mu$M'.format(c), drawstyle='steps',
              color=colors[i])
    ax[0].fill_between(m_grid, p_mC_mat[i, :], color=colors[i], alpha=0.1,
                      step='pre')
    # Plot the CDF
    ax[1].plot(m_grid, np.cumsum(p_mC_mat[i, :]), drawstyle='steps',
              color=colors[i])

ax[0].legend(loc=0, title='inducer concentration', ncol=5, fontsize=10)
ax[0].set_ylabel('PMF')
ax[0].set_ylim(top=0.2)
ax[0].margins(0.02)

ax[1].legend(loc=0)
ax[1].set_xlabel('number of mRNA')
ax[1].set_ylabel('CDF')
ax[1].margins(0.02)

plt.tight_layout()
plt.savefig(figdir + 'mRNA_dist_IPTG_RBS1027_O3.pdf')
```

    /Users/razo/anaconda/lib/python3.5/site-packages/matplotlib/axes/_axes.py:519: UserWarning: No labelled objects found. Use label='...' kwarg on individual plots.
      warnings.warn("No labelled objects found. "

## Reproducing the lacI titration curves with the chemical master equation.

As a simple sanity check it is important to confirm that the chemical master equation is indeed reproducing the "golden-standard" lacI titration curves from Hernan's PNAS paper.

What we need to do to reproduce this is to compute the full distribution as a function of the repressor copy number and then obtain the mean expression level by simply computing
\begin{equation}
\left\langle m \right\rangle = \sum_m P(m) \cdot m.
\end{equation}
Then we divide this mean expression level by the mean expression level of the strain with $R = 0$ to compute the fold change.

```python
# Define the repressor and mRNA array
R_array = np.logspace(0, 3, 100)
mRNA_grid = np.arange(0, 50)

# Define array with operators binding energies
operators = [-17, -15.3, -13.9, -9.7]
# Define a quick Lambda function to compute Hernan's fold change
fc_fn = lambda R, epsilon : 1 / (1 + 2 * R / 4.6E6 * np.exp(-epsilon))

k0 = 2.7E-3
params = dict(ka=0.21, ki=0.00087, epsilon=4.5, kon=kon_fn(-15.3, k0),
              k0=k0, gamma=0.00284, r_gamma=15.7)

# Compute the mean expression of the delta strain
dist_delta = np.exp(log_p_m_mid_C(C=0, mRNA=m_grid, rep=0, **params))
mean_m_delta = np.sum(m_grid * dist_delta)

# Iinitialize the array to save the fold-changes
fc_chem_master = np.zeros([len(operators), len(R_array)])
fc_stat_mech = np.zeros([len(operators), len(R_array)])
#loop through operators and repressors
for i, o in enumerate(operators):
    params['kon'] = kon_fn(o, k0)
    for j, r in enumerate(R_array):
        # compute the distribution. The factor of 1.66 is to convert from
        # absolute copy number to nM concentrations.
        dist = np.exp(log_p_m_mid_C(C=0, mRNA=m_grid, rep=r * 1.66, **params))
        # from the distribution obtain the mean
        mean_m = np.sum(m_grid * dist)
        # compute the fold change with respect to the ∆lacI strain
        fc_chem_master[i, j] = mean_m / mean_m_delta
        fc_stat_mech[i, j] = fc_fn(r, o)
```

```python
operators_name = ['Oid', 'O1', 'O2', 'O3']
# Plot the results of both theories
fig, ax = plt.subplots(1, 2, figsize=(9, 4))
for i, o in enumerate(operators):
    ax[0].plot(R_array, fc_chem_master[i,:],
              label=operators_name[i])
    ax[1].plot(R_array, fc_stat_mech[i,:],
              label=operators_name[i])
# Format the plots
ax[0].set_xscale('log')
ax[0].set_yscale('log')
ax[0].set_xlabel('repressor copy number')
ax[0].set_ylabel('fold-change')
ax[0].set_title('chemical master equation')
ax[0].legend(loc='lower left')
ax[0].set_ylim([1E-4, 1.2])

ax[1].set_xscale('log')
ax[1].set_yscale('log')
ax[1].set_xlabel('repressor copy number')
ax[1].set_ylabel('fold-change')
ax[1].set_title('statistical mechanics')
ax[1].set_ylim([1E-4, 1.2])

# Label plots with letters
plt.figtext(0.0, 0.90, 'A', fontsize=20)
plt.figtext(0.5, 0.90, 'B', fontsize=20)

plt.tight_layout()
plt.savefig(figdir + 'CME_statmech_comparison.pdf', bbox_inches='tight')
```

The curves look exactly the same. That means that by design our selected parameters are indeed in agreement with the previous work from the lab.

### Comparison with the parameters used by Jones, Brewster and Phillips, Science 2014

```python
# Define the repressor and mRNA array
R_array = np.logspace(0, 3, 100)
mRNA_grid = np.arange(0, 50)

# Define array with operators binding energies
operators = [-17, -15.3, -13.9, -9.7]
# Define array with Jones and Brewster's kon values (koff in their nomenclature)
kon_array = [0.0023, 0.0069, 0.091, 2.1]
# Define a quick Lambda function to compute Hernan's fold change
fc_fn = lambda R, epsilon : 1 / (1 + 2 * R / 4.6E6 * np.exp(-epsilon))

k0 = 2.7E-3
params = dict(ka=0.21, ki=0.00087, epsilon=4.5, kon=kon_fn(-15.3, k0),
              k0=k0, gamma=0.00284, r_gamma=15.7)

# Compute the mean expression of the delta strain
dist_delta = np.exp(log_p_m_mid_C(C=0, mRNA=m_grid, rep=0, **params))
mean_m_delta = np.sum(m_grid * dist_delta)

# Iinitialize the array to save the fold-changes
fc_chem_master = np.zeros([len(operators), len(R_array)])
fc_stat_mech = np.zeros([len(operators), len(R_array)])

#loop through operators and repressors
for i, o in enumerate(operators):
    params['kon'] = kon_array[i]
    for j, r in enumerate(R_array):
        # compute the distribution. The factor of 1.66 is to convert from
        # absolute copy number to nM concentrations.
        dist = np.exp(log_p_m_mid_C(C=0, mRNA=m_grid, rep=r * 1.66, **params))
        # from the distribution obtain the mean
        mean_m = np.sum(m_grid * dist)
        # compute the fold change with respect to the ∆lacI strain
        fc_chem_master[i, j] = mean_m / mean_m_delta
        fc_stat_mech[i, j] = fc_fn(r, o)
```

```python
operators_name = ['Oid', 'O1', 'O2', 'O3']
# Plot the results of both theories
fig, ax = plt.subplots(1, 2, figsize=(8, 4))
for i, o in enumerate(operators):
    ax[0].plot(R_array, fc_chem_master[i,:],
              label=operators_name[i])
    ax[1].plot(R_array, fc_stat_mech[i,:],
              label=operators_name[i])
# Format the plots
ax[0].set_xscale('log')
ax[0].set_yscale('log')
ax[0].set_xlabel('repressor copy number')
ax[0].set_ylabel('fold-change')
ax[0].set_title('DJ, RCB, RP, Science 2014')
ax[0].legend(loc='lower left')
ax[0].set_ylim([1E-4, 1.2])

ax[1].set_xscale('log')
ax[1].set_yscale('log')
ax[1].set_xlabel('repressor copy number')
ax[1].set_ylabel('fold-change')
ax[1].set_title('HG, RP, PNAS 2011')
ax[1].set_ylim([1E-4, 1.2])

# Label plots with letters
plt.figtext(0.0, 0.90, '(A)', fontsize=15)
plt.figtext(0.5, 0.90, '(B)', fontsize=15)

plt.tight_layout()
plt.savefig(figdir + 'CME_statmech_comparison_DJ_RCB.pdf')
```

```python
# Define operator names
operators_name = ['Oid', 'O1', 'O2', 'O3']
# Generate dictionary with colors
op_color = dict(zip(operators_name,
                sns.color_palette('colorblind', n_colors=len(operators_name))))
# Plot the results of both theories
fig, ax = plt.subplots(1, 1, figsize=(5.5,5))
for i, o in enumerate(operators):
    ax.plot(R_array, fc_chem_master[i,:],
              label=operators_name[i], color=op_color[operators_name[i]])
    ax.plot(R_array, fc_stat_mech[i,:],
             label='', linestyle='--',
             color=op_color[operators_name[i]])

# Add the first legend
first_legend = plt.legend(title='operator')
# Add secondary label to distinguish two- vs three-moment approximation
# Add the legend manually to the current Axes.
ax2 = ax.add_artist(first_legend)

# "plot" a solid and a dashed line for the legend
line1, = plt.plot([], [], color='black', label='HG & RP, 2011')
line2, = ax.plot([], [], color='black', linestyle='--',
                 label='DJ, RCB & RP, 2014')
# Create another legend.
plt.legend(handles=[line1, line2], loc='lower center', frameon=True)

ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel('repressors / cell')
ax.set_ylabel('fold-change')

plt.tight_layout()
plt.savefig(figdir + 'CME_statmech_single_comparison_DJ_RCB.pdf',
           bbox_inches='tight')
```

Suprisingly the parameters that Brewster, Jones and Phillips report on their Science, 2014 paper are not consistent with the statistical mechanics framework. The way they determined the $k_{off}$ parameter was by using an *in-vitro* measurement of this parameter using Oid and then they used Hernan's binding energies to take the ratio between the dissociation constants (calculated from the binding energies) and assumed that this was the same as the ratio of the rate constants.


Author: Adrian Swartz

Date: 3/29/2019

# Solutions to Andew Ng's coursera problems
## Example 2: Logistic Regression

Andrew Ng's course is designed for the solutions to be worked through in Octave (similar to matlab). However, I am going to do these problems in python (within a jupyter notebook) and demonstrate how various approaches can be implemented in python.

This notebook is structured as follows:

[Part 1 - Logistic Regression](#Part1)
1. [Exploratory Data Analysis](#eda)
2. [Developing Sigmoid, Cost, and Gradient Functions](#funcs)
3. [Finding the Optimal Model Parameters](#opt)
4. [Plotting the Decision Boundary](#plot)
5. [Model Evaluation](#metrics)
6. [Implementing Logistic Regression with sklearn](#sklearn)

[Part 2 - Regularized Logistic Regression with Non-linear Features](#Part2)
1. [Exploratory Data Analysis](#eda2)
2. [Polynomial Feature Generation](#fg)
3. [Adding Regularization to Cost and Gradient Functions](#reg)
4. [Finding the Optimal Model Parameters](#opt2)
5. [Non-linear Logistic Regression with sklearn](#sklearn2)
6. [Tuning the Regularization Strength](#rs)
7. [Polynomial Degree and Overfitting](#pdo)

***

For this exercise, we will import several python packages: NumPy (numerical python for matrix manupulations and advanced math), pandas (DataFrames offer a useful platform for importing data and performing exploratory data analysis), matplotlib and seaborn for data visualization.

```python
# import standard packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns
cp=sns.color_palette()

# import a few special packages we'll use for this example

import scipy.optimize as opt
from sympy.solvers import solve
from sympy import Symbol
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import PolynomialFeatures
```

# Part 1 - Logistic Regression

Prompt:

Build a logistic regression model to predict whether a student gets admitted into a university.

Suppose that you are the administrator of a university department and you want to determine each applicant’s chance of admission based on their results on two exams. You have historical data from previous applicants that you can use as a training set for logistic regression. For each training example, you have the applicant’s scores on two exams and the admissions decision.

Your task is to build a classification model that estimates an applicant’s probability of admission based the scores from those two exams.

***

To be clear, this is not really a "regression" problem. But rather we are faced with a binary classification problem: accept or reject the studen't application. The twist is that we are asked to determine each applicant's chance (probability) of acceptance. For this type of problem, a logistic classifier is actually a regression (the sigmoid function) between two cases (0 or 1 - reject or accept) and the crossover between these two cases gives us the probability associated with acceptance.

OK. Let's dive into the data and begin building the frameworks for logistic regression.

# Exploratory Data Analysis

```python
#Load the data into a pandas DataFrame
data = pd.read_csv('ex2/ex2data1.txt', header=None, names=['exam1', 'exam2', 'admitted'])
data.head()
```

      exam1
      exam2
      admitted

      0
      34.623660
      78.024693
      0

      1
      30.286711
      43.894998
      0

      2
      35.847409
      72.902198
      0

      3
      60.182599
      86.308552
      1

      4
      79.032736
      75.344376
      1

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 100 entries, 0 to 99
    Data columns (total 3 columns):
    exam1       100 non-null float64
    exam2       100 non-null float64
    admitted    100 non-null int64
    dtypes: float64(2), int64(1)
    memory usage: 2.4 KB

```python
data.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      exam1
      100.0
      65.644274
      19.458222
      30.058822
      50.919511
      67.032988
      80.212529
      99.827858

      exam2
      100.0
      66.221998
      18.582783
      30.603263
      48.179205
      67.682381
      79.360605
      98.869436

      admitted
      100.0
      0.600000
      0.492366
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000

We can see that 60% of the students were accepted

```python
#use boolean masking to look separate data of accepted and rejected applicants
data[data.admitted==0].describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      exam1
      40.0
      52.032301
      17.549050
      30.058822
      35.707085
      50.410642
      61.928323
      95.861555

      exam2
      40.0
      54.620392
      16.081591
      30.603263
      42.808798
      49.698756
      63.594638
      98.869436

      admitted
      40.0
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000

```python
#use boolean masking to look separate data of accepted and rejected applicants
data[data.admitted==1].describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      exam1
      60.0
      74.718923
      14.912420
      40.457551
      63.597244
      75.436657
      84.675745
      99.827858

      exam2
      60.0
      73.956402
      16.012403
      43.390602
      65.409676
      74.753748
      87.442625
      97.718692

      admitted
      60.0
      1.000000
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000

Not very shocking, but students who were accepted had on average, significantly higher exame scores.

```python
sns.pairplot(data)
```

That doesn't look as helpful does it? But actually, pay close attention to the exam1 vs. admitted scatter-plot. This looks almost exactly like a text book logistic regression problem. Let's look at the data a slightly different way

```python
accepted = data[data.admitted==1]
rejected = data[data.admitted==0]
```

```python
fig, ax = plt.subplots(figsize=(8,6))
ax.scatter(accepted.exam1, accepted.admitted, s=50, color=cp[0], label='Admitted')
ax.scatter(rejected.exam1, rejected.admitted, s=50, color=cp[3], marker='x', label='Rejected')
ax.set_xlabel('Admitted or Rejected')
ax.set_ylabel('Exam1 Score')
ax.legend()
```

```python
fig, ax = plt.subplots(figsize=(8,6))
ax.scatter(accepted.exam1, accepted.exam2, s=50, color=cp[0], label='Admitted')
ax.scatter(rejected.exam1, rejected.exam2, s=50, color=cp[3], marker='x', label='Rejected')
ax.set_xlabel('Exam1 Score')
ax.set_ylabel('Exam2 Score')
ax.legend()
```

We can actually do something slighly fancier with seaborn's pairpolot.

```python
#have to cast the data into a 'category' type to get it to plot with seaborn's pairplot
data.iloc[:,-1] = data.iloc[:,-1].astype('category')
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 100 entries, 0 to 99
    Data columns (total 3 columns):
    exam1       100 non-null float64
    exam2       100 non-null float64
    admitted    100 non-null category
    dtypes: category(1), float64(2)
    memory usage: 1.8 KB

```python
sns.pairplot(data, vars=["exam1", "exam2"], hue="admitted", \
             palette={0:cp[3], 1:cp[0]}, height=3, aspect=1.2, markers=["x", "o"])
```

```python
#have to cast the data back to int type
data.iloc[:,-1] = data.iloc[:,-1].astype('int')
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 100 entries, 0 to 99
    Data columns (total 3 columns):
    exam1       100 non-null float64
    exam2       100 non-null float64
    admitted    100 non-null int64
    dtypes: float64(2), int64(1)
    memory usage: 2.4 KB

# Developing Sigmoid, Cost, and Gradient Descent Functions
Let's start by looking at the sigmoid or logit function

### The Sigmoid Function

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
#notice that in this form it can be used on an array or on individual numbers
```

```python
#Let's plot what that looks like
x = np.linspace(-10,10,num=100)
sig = [sigmoid(i) for i in x] # isn't list comprehension nice?
plt.plot(x,sig)
```

Let's write our cost function next.

### The Cost Function For Logistic Regression

```python
def Cost(theta, X, y):
    """ Calculates the cost function for logistic regression
    input
    theta: a numpy array of length n; the model parameters
    X: an m x n matrix; instances x features
    y: an m x 1 matrix; target variable

    output
    cost: the sum of the ordinary least squares over all the data
    """
    theta = np.matrix(theta) # convert numpy array to matrix for matrix multiplications

    M = len(X)

    hyp = sigmoid(X * theta.T)

    c = np.multiply(-y, np.log(hyp)) - np.multiply((1 - y), np.log(1 - hyp))

    return np.sum(c) / (M)

```

We will need to make sure we keep careful track of the matrix shapes when we calculate the cost. We'll get into this in more detail later, but here we want the initialized $\theta$ to be a numpy array of length $n$. This is because the scipy optimizing functions wants an array not a matrix. So we need to convert $\theta$ to a matrix in our Cost and Gradient functions.

```python
# convert the features from the DataFrame into a numpy matrix
X = np.matrix(data.iloc[:,:-1])

# add the bias column
X = np.insert(X, 0, np.ones(len(X)), axis=1)

# convert the target variables from the DataFrame into a numpy matrix
y = np.matrix(data.iloc[:,-1]).T

#Initialize theta
theta = np.zeros(X.shape[1])

X.shape, y.shape, theta.shape
```

    ((100, 3), (100, 1), (3,))

```python
X[:5]
```

    matrix([[ 1.        , 34.62365962, 78.02469282],
            [ 1.        , 30.28671077, 43.89499752],
            [ 1.        , 35.84740877, 72.90219803],
            [ 1.        , 60.18259939, 86.3085521 ],
            [ 1.        , 79.03273605, 75.34437644]])

```python
Cost(theta, X, y)
```

    0.6931471805599453

Great! Cost function is working since we have matched the $X$, $y$, and $\theta$ matrix dimensions carefully with out matrix multiplications in the cost function.

### The Gradient Function For Logistic Regression

Next, we are asked to write the gradient function.  In the linear regression example, we manually implemented gradient descent. Here we are asked to use a function optimizer (`fminunc` in Octave). In that case, we need to simply write a function that calculates the gradient and then let the function optimizer (i.e. minimizer) do the heavy lifting. In python we'll use scipy.optimize API (see: [scipy's Optimization page](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html) and [scipy.optmize documentation](https://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize)). A key thing to remember here is that the parameters which we want to find the optimal values for (in this case theta) should be the __FIRST__ argument of the function. Notice that I wrote the cost function with the same convention.

```python
def Gradient(theta, X, y):
    """ Calculates the gradient of the logistic regression cost function
    input
    theta: a 1 x n matrix; the model parameters
    X: an m x n matrix; instances x features
    y: an m x 1 matrix; target variable

    output
    gradient: a numpy array of floats of lenth n
    """
    M=len(X)
    theta=np.matrix(theta)
    parameters = int(theta.shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):
        grad[i] = np.sum(np.multiply(error, X[:,i])) / M

    return grad
```

```python
Gradient(theta,X,y)
```

    array([ -0.1       , -12.00921659, -11.26284221])

# Finding the Optimal Model Parameters

Excellent! We have a working function that calculates the gradient. Now we need to call a function optimizer (minimizer) from scipy. For this problem, we'll use `fmin_trunc`. Here is a snippet from the `opt.fmin_tnc` documentation: `"Minimize a function with variables subject to bounds, using gradient information in a truncated Newton algorithm. ... The underlying algorithm is truncated Newton, also called Newton Conjugate-Gradient. The algorithm incorporates the bound constraints by determining the descent direction as in an unconstrained truncated Newton, but never taking a step-size large enough to leave the space of feasible x's. ..."`

Look carefully at the documentation arguments in more detail to get a better feel for why they are being called and why we wrote the Cost and Gradient functions the way we did.

```python
result = opt.fmin_tnc(func=Cost, x0=theta, fprime=Gradient, args=(X, y))
result
```

    (array([-25.16131866,   0.20623159,   0.20147149]), 36, 0)

```python
Cost(result[0], X, y)
```

    0.20349770158947444

Just like that we've found the optimal logistic regression parameters for this data set. -25.16 is the bias term, while 0.206 and 0.201 represent the linear components for exam1 and exam2, respectively.

Alternatively, we could have used the more flexible `optimize.minimize` function from scipy.

```python
result = opt.minimize(fun=Cost, x0=theta, args=(X,y), method='TNC', jac=Gradient)
result
```

         fun: 0.20349770158947444
         jac: array([9.01534233e-09, 8.69256070e-08, 4.80499281e-07])
     message: 'Local minimum reached (|pg| ~= 0)'
        nfev: 36
         nit: 17
      status: 0
     success: True
           x: array([-25.16131866,   0.20623159,   0.20147149])

```python
result.x
```

    array([-25.16131866,   0.20623159,   0.20147149])

```python
Cost(result.x, X, y)
```

    0.20349770158947444

Awesome! It may be fun for yourself to play around with different optimizers and see how long it takes for scipy to converge on a solution. With the logistic cost function some solvers won't perform as well and will spit out some warnings.

So it seems like we have the optimal model parameters, but it would be nice to visualize the boundary.

# Plotting the Decision Boundary

```python
def linear_boundary(weights,xmin,xmax):

    x1 = np.linspace(xmin,xmax,num=4)
    x2 = np.zeros(len(x1))
    x0 = np.ones(len(x1))

    x = Symbol('x')
    e = 2.71828
    c = weights[2]

    for index, value in enumerate(x1):
        a = weights[0]*x0[index]
        b = weights[1]*x1[index]
        result = solve(a + b + weights[2]*x, x)
        x2[index] = result[0]

    return x1, x2
```

```python
x1,x2 = linear_boundary(result.x,20,100)

fig, ax = plt.subplots(figsize=(8,6))
ax.scatter(accepted.exam1, accepted.exam2, s=50, color=cp[0], label='Admitted')
ax.scatter(rejected.exam1, rejected.exam2, s=50, color=cp[3], marker='x', label='Rejected')
ax.plot(x1,x2, color='red')
ax.set_xlabel('Exam1 Score')
ax.set_ylabel('Exam2 Score')
ax.legend()
```

Look's pretty good, maybe it would be better to have a non-linear boundary, but in logistic regression this would require generating non-linear features (feature engineering). But it "seems" like a linear boundary is doing OK here. But can we quantify it? Of course we can. We're talking about measures like accuracy, precision, recall, and f1 score. To get these, we first need to find the predicted results of our model.

# Model Evaluation

```python
def predict(theta, X):
    proba = sigmoid(X * theta.T)
    prediction = [1 if x >= 0.5 else 0 for x in proba]
    return prediction
```

```python
best_fit_params = np.matrix(result.x)
lr_predictions = predict(best_fit_params, X)
```

We could calculate our own accuracy, etc... metrics by calculating the number of true positives, etc..., but I'm lazy, and sklearn can do this for you!

```python
def get_metrics(y_data, y_pred):
    # true positives / (true positives+false positives)
    precision = precision_score(y_data, y_pred)

    # true positives / (true positives + false negatives)
    recall = recall_score(y_data, y_pred)

    # harmonic mean of precision and recall
    f1 = f1_score(y_data, y_pred)

    # true positives + true negatives/ total
    accuracy = accuracy_score(y_data, y_pred)
    return accuracy, precision, recall, f1
```

```python
acc, prec, rec, f1 = get_metrics(y, lr_predictions)
```

```python
print(' ')
print('Accuracy: {:.2}; Precision: {:.2}; Recall: {:.2}; F1 Score: {:.2}'.format(acc, prec, rec, f1))
print(' ')
```

    Accuracy: 0.89; Precision: 0.9; Recall: 0.92; F1 Score: 0.91

```python
conf = confusion_matrix(y, lr_predictions)
print(conf)
#OK. so we printed a confusion matrix. But it would be nicer to plot it
```

    [[34  6]
     [ 5 55]]

```python
# Seaborn has a nice heatmap plot that's great for confusion matrices.
df_cm = pd.DataFrame(conf.astype(int), index = ['TP','TN'], columns = ['PP','PN'])
plt.figure(figsize = (5,5))
sns.heatmap(df_cm, annot=True, cmap="YlGnBu")
```

The accuracy is 89%, which is not horrible and from looking at the confusion matrix, it seems that we are pretty well balanced between choosing false positives and false negatives. You can see this directly from how the logistic regression boundary slices through the data in the 2D scatter plot.

# Implementing Logistic Regression with sklearn

```python
# convert the features from the DataFrame into a numpy matrix
X = np.matrix(data.iloc[:,:-1])

# convert the target variables from the DataFrame into a numpy matrix
y = np.array(data.iloc[:,-1]).T

X.shape, y.shape
```

    ((100, 2), (100,))

```python
# create the LogisticRegression classifier object
clf = LogisticRegression(C=1000, solver='newton-cg').fit(X, y)
clf.intercept_, clf.coef_
```

    (array([-25.16109754]), array([[0.20622982, 0.20146969]]))

Here I used a very large inverse regulatization parameter (C=1000). The LogisticRegression algorithm has regularization built into it and the default parameter (C=1) provides some finite regularization (L2 with the newton-cg solver). Since we want to compare with the home-made solution above, we want to get rid of the regularization. Also, I set fit_intercept to False (the default it True) because we have already manually inserted the bias term into the $X$ matrix.

Next, let's get the predictions of the classifier on the data and then check the metrics. Since the classifier coefficients are nearly identical to our home-made solution, we should expect the same accuracy results.

```python
y_pred = clf.predict(X)
acc, prec, rec, f1 = get_metrics(y, y_pred)
print(' ')
print('Accuracy: {:.2}; Precision: {:.2}; Recall: {:.2}; F1 Score: {:.2}'.format(acc, prec, rec, f1))
print(' ')
```

    Accuracy: 0.89; Precision: 0.9; Recall: 0.92; F1 Score: 0.91

# Part 2 - Regularized Logistic Regression with Non-linear Features
Prompt:

Implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assurance (QA). During QA, each microchip goes through various tests to ensure it is functioning correctly.

Suppose you are the product manager of the factory and you have the test results for some microchips on two different tests. From these two tests, you would like to determine whether the microchips should be accepted or rejected. To help you make the decision, you have a dataset of test results on past microchips, from which you can build a logistic regression model.

# Exploratory Data Analysis

```python
#Load the data into a pandas DataFrame
data = pd.read_csv('ex2/ex2data2.txt', header=None, names=['test1', 'test2', 'accepted'])
data.head()
```

      test1
      test2
      accepted

      0
      0.051267
      0.69956
      1

      1
      -0.092742
      0.68494
      1

      2
      -0.213710
      0.69225
      1

      3
      -0.375000
      0.50219
      1

      4
      -0.513250
      0.46564
      1

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 118 entries, 0 to 117
    Data columns (total 3 columns):
    test1       118 non-null float64
    test2       118 non-null float64
    accepted    118 non-null int64
    dtypes: float64(2), int64(1)
    memory usage: 2.8 KB

```python
data.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      test1
      118.0
      0.054779
      0.496654
      -0.83007
      -0.372120
      -0.006336
      0.478970
      1.0709

      test2
      118.0
      0.183102
      0.519743
      -0.76974
      -0.254385
      0.213455
      0.646562
      1.1089

      accepted
      118.0
      0.491525
      0.502060
      0.00000
      0.000000
      0.000000
      1.000000
      1.0000

```python
#use boolean masking to look separate data of accepted and rejected applicants
accepted = data[data.accepted==1]
rejected = data[data.accepted==0]
```

```python
#have to cast the data into a 'category' type to get it to plot with seaborn's pairplot
data.iloc[:,-1] = data.iloc[:,-1].astype('category')

sns.pairplot(data, vars=["test1", "test2"], hue="accepted", \
             palette={0:cp[3], 1:cp[0]}, height=3, aspect=1.2, markers=["x", "o"])
```

```python
# cast the data back to int type
data.iloc[:,-1] = data.iloc[:,-1].astype('int')
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 118 entries, 0 to 117
    Data columns (total 3 columns):
    test1       118 non-null float64
    test2       118 non-null float64
    accepted    118 non-null int64
    dtypes: float64(2), int64(1)
    memory usage: 2.8 KB

The data clearly has more structure and we can see that a linear boundary between the two variables will not provide a good model. So we need to generate non-linear features (i.e test1$\times$test2, test1$^2$, and test2$^2$ at the very least). When generating non-linear features, it's ALWAYS a good idea to include regulatization.

# Polynomial Feature Generation

Conveniently, sklearn provides a function which automatically generates nonlinear features for you! Sensing a theme? sklearn is amazingly well-designed for machine learning problems.

```python

```

```python
X = np.matrix(data.iloc[:,:-1])

poly = PolynomialFeatures(6)
X = poly.fit_transform(X)

# convert the target variables from the DataFrame into a numpy matrix
# notice that for the Gradient and Cost functions, this NEEDS TO BE A MATRIX mx1
# (for sklearn, it needs to be an array of length m)
y = np.matrix(data.iloc[:,-1]).T

#Initialize theta
theta = np.zeros(X.shape[1])

X.shape, y.shape, theta.shape
```

    ((118, 28), (118, 1), (28,))

```python
#let's just look at it quickly.
pd.DataFrame(X).head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27

      0
      1.0
      0.051267
      0.69956
      0.002628
      0.035864
      0.489384
      0.000135
      0.001839
      0.025089
      0.342354
      ...
      0.000900
      0.012278
      0.167542
      1.815630e-08
      2.477505e-07
      0.000003
      0.000046
      0.000629
      0.008589
      0.117206

      1
      1.0
      -0.092742
      0.68494
      0.008601
      -0.063523
      0.469143
      -0.000798
      0.005891
      -0.043509
      0.321335
      ...
      0.002764
      -0.020412
      0.150752
      6.362953e-07
      -4.699318e-06
      0.000035
      -0.000256
      0.001893
      -0.013981
      0.103256

      2
      1.0
      -0.213710
      0.69225
      0.045672
      -0.147941
      0.479210
      -0.009761
      0.031616
      -0.102412
      0.331733
      ...
      0.015151
      -0.049077
      0.158970
      9.526844e-05
      -3.085938e-04
      0.001000
      -0.003238
      0.010488
      -0.033973
      0.110047

      3
      1.0
      -0.375000
      0.50219
      0.140625
      -0.188321
      0.252195
      -0.052734
      0.070620
      -0.094573
      0.126650
      ...
      0.017810
      -0.023851
      0.031940
      2.780914e-03
      -3.724126e-03
      0.004987
      -0.006679
      0.008944
      -0.011978
      0.016040

      4
      1.0
      -0.513250
      0.46564
      0.263426
      -0.238990
      0.216821
      -0.135203
      0.122661
      -0.111283
      0.100960
      ...
      0.026596
      -0.024128
      0.021890
      1.827990e-02
      -1.658422e-02
      0.015046
      -0.013650
      0.012384
      -0.011235
      0.010193

5 rows × 28 columns

# Adding Regularization to Cost and Gradient Functions

```python
def Regularized_Cost(theta, X, y, reg_scale):
    """ Calculates the regularized cost function for logistic regression
    input
    theta: a numpy array of length n; the model parameters
    X: an m x n matrix; instances x features
    y: an m x 1 matrix; target variable
    reg_scale: a float; strength of the regularization

    output
    cost: the sum of the ordinary least squares over all the data
    """
    theta = np.matrix(theta)
    M = len(X)
    hyp = sigmoid(X * theta.T)

    c = np.multiply(-y, np.log(hyp)) - np.multiply((1 - y), np.log(1 - hyp))
    reg = (reg_scale * np.sum(np.power(theta[:,1:theta.shape[1]], 2)) / (2 * M))
    return (np.sum(c) / M) + reg
```

```python
Regularized_Cost(theta, X, y, 1)
```

    0.6931471805599454

Agrees with `ex2.pdf`. However, when $\theta$ is initialized to zero, we can't tell if the regularization part of the cost function is working. Let's initialize $\theta$ to all ones and see how the cost function changes if we tune the regularization parameter

```python
theta = np.ones(X.shape[1])
```

```python
Regularized_Cost(theta, X, y, .001), Regularized_Cost(theta, X, y, 1), \
Regularized_Cost(theta, X, y, 100), Regularized_Cost(theta, X, y, 1000)
```

    (2.020555941784501, 2.134848314665857, 13.461119501106536, 116.42722119602179)

```python
def Regularized_Gradient(theta, X, y, reg_scale):
    """ Calculates the regularized gradient of the logistic regression cost function
    input
    theta: a 1 x n matrix; the model parameters
    X: an m x n matrix; instances x features
    y: an m x 1 matrix; target variable
    reg_scale: a float; strength of the regularization

    output
    gradient: a numpy array of floats of lenth n
    """
    M=len(X)
    theta=np.matrix(theta)
    parameters = int(theta.shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):

        if (i == 0):
            grad[i] = np.sum(np.multiply(error, np.matrix(X[:,i]).T)) / M
        else:
            grad[i] = (np.sum(np.multiply(error, np.matrix(X[:,i]).T)) / M)
            grad[i] += (reg_scale * theta[:,i] / M)

    return grad
```

I have included an if statement in the gradient calculation which avoids regularizing the bias term.

```python
Regularized_Gradient(theta, X, y, 1)
```

    array([0.34604507, 0.08508073, 0.11852457, 0.1505916 , 0.01591449,
           0.16811439, 0.06712094, 0.03217053, 0.02604321, 0.10719727,
           0.09725885, 0.01098433, 0.04195657, 0.00957212, 0.12367776,
           0.05895534, 0.01870409, 0.01729323, 0.02352665, 0.01513039,
           0.09858123, 0.07328323, 0.01051447, 0.02270567, 0.00904832,
           0.02563548, 0.00823079, 0.10601204])

# Finding the Optimal Model Parameters

```python
result = opt.minimize(fun=Regularized_Cost, x0=theta, args=(X,y,1), method='TNC', \
                      jac=Regularized_Gradient, options={'maxiter':1000})
result
```

         fun: 0.5290027297445772
         jac: array([ 8.45222178e-08, -9.29970655e-07, -2.06666060e-07,  3.21329701e-07,
           -1.38270147e-07, -2.07703990e-07, -4.23808363e-07,  1.55632651e-07,
           -4.53215445e-08, -2.41372575e-07,  3.03267104e-07, -5.04976970e-08,
            2.36283776e-07,  8.23622646e-11,  2.29393391e-08, -3.40841758e-07,
           -4.16539275e-09, -1.68027167e-08,  1.06755703e-07,  5.84578486e-08,
           -2.11921153e-07,  1.97399227e-07, -5.59518431e-08,  1.39789220e-07,
           -3.17202377e-08,  2.39144154e-08,  3.51117671e-08,  1.10228540e-07])
     message: 'Converged (|f_n-f_(n-1)| ~= 0)'
        nfev: 39
         nit: 8
      status: 1
     success: True
           x: array([ 1.27273509,  0.62525435,  1.18108521, -2.01994882, -0.91742555,
           -1.43167368,  0.12399628, -0.36552234, -0.35723208, -0.17514253,
           -1.4581339 , -0.05098852, -0.61553085, -0.27470069, -1.19280263,
           -0.24220871, -0.20601057, -0.04472766, -0.2777735 , -0.29536755,
           -0.45637086, -1.04318579,  0.02776829, -0.29241701,  0.01556523,
           -0.32737793, -0.14388044, -0.92463148])

```python
pred = predict(np.matrix(result.x), X)
np.mean(pred == y.flatten()) * 100
```

    83.05084745762711

```python
lr_predictions = predict(np.matrix(result.x), X)
acc, prec, rec, f1 = get_metrics(y, lr_predictions)
print(' ')
print('Accuracy: {:.2}; Precision: {:.2}; Recall: {:.2}; F1 Score: {:.2}'.format(acc, prec, rec, f1))
print(' ')
```

    Accuracy: 0.83; Precision: 0.78; Recall: 0.91; F1 Score: 0.84

# Non-linear Logistic Regression with sklearn

```python
X = np.matrix(data.iloc[:,:-1])

poly = PolynomialFeatures(6)
X = poly.fit_transform(X)

# convert the target variables from the DataFrame into a numpy array
# notice that for the Gradient and Cost functions, y is an mx1 matrix
# for sklearn, it needs to be an array of length m
y = np.array(data.iloc[:,-1]).T

#Initialize theta
theta = np.zeros(X.shape[1])

X.shape, y.shape, theta.shape
```

    ((118, 28), (118,), (28,))

```python
clf = LogisticRegression(C=1, solver='newton-cg', fit_intercept=False).fit(X, y)

clf.intercept_[0], clf.coef_
```

    (0.0, array([[ 1.14213575,  0.60132186,  1.16718326, -1.87174359, -0.91573503,
             -1.26953757,  0.12668235, -0.3687234 , -0.3451782 , -0.17377628,
             -1.42386177, -0.04856119, -0.6064216 , -0.26932201, -1.16315301,
             -0.24310264, -0.20706754, -0.0431884 , -0.28028106, -0.28695055,
             -0.46910709, -1.03618515,  0.02923411, -0.29262323,  0.01736154,
             -0.32897195, -0.13796124, -0.93198618]]))

```python
y_pred = clf.predict(X)
acc, prec, rec, f1 = get_metrics(y, y_pred)
print(' ')
print('Accuracy: {:.2}; Precision: {:.2}; Recall: {:.2}; F1 Score: {:.2}'.format(acc, prec, rec, f1))
print(' ')
```

    Accuracy: 0.82; Precision: 0.78; Recall: 0.9; F1 Score: 0.83

```python
clf.score(X,y)
```

    0.8220338983050848

# Tuning the Regularization Strength

```python
cs = np.logspace(-3,4,100)
accs = []

for c in cs:
    clf = LogisticRegression(C=c, solver='newton-cg', fit_intercept=False).fit(X, y)
    accs.append(clf.score(X, y))
```

```python
fig, ax = plt.subplots()
plt.plot(cs, accs)
ax.set_xscale('log')
```

In sklearn, $C$ is the inverse regularization rate (i.e. $C \sim 1/\lambda$). When $C$ is very small, this means that the regularization is actually very strong, this makes the bias term dominate and variance is heavily reduced (the bias/variance tradeoff). For that reason the accuracy is not great. In the opposite limit (large $C$), the regularization is reduced and the model overfits.

# Polynomial Degree and Overfitting

```python
X = np.matrix(data.iloc[:,:-1])

# convert the target variables from the DataFrame into a numpy matrix
# notice that for the Gradient and Cost functions, this NEEDS TO BE A MATRIX mx1
# (for sklearn, it needs to be an array of length m)
y = np.array(data.iloc[:,-1]).T

poly_orders = range(1,60)
c_strong = 10000 # weak regularization
c_weak = .01 # weak regularization

accs_strong = []
accs_weak = []

for p in poly_orders:
    poly = PolynomialFeatures(p)
    X_poly = poly.fit_transform(X)

    clf = LogisticRegression(C=c_strong, solver='newton-cg', fit_intercept=False).fit(X_poly, y)
    accs_strong.append(clf.score(X_poly, y))

    clf = LogisticRegression(C=c_weak, solver='newton-cg', fit_intercept=False).fit(X_poly, y)
    accs_weak.append(clf.score(X_poly, y))

```

```python
fig, ax = plt.subplots(figsize=(10,6))
plt.plot(poly_orders, accs_strong, '-o', label='C = 10000')
plt.plot(poly_orders, accs_weak, '-o', color=cp[3], label='C = 0.01')
ax.set_xlim((0,60))
ax.set_ylim((.3,1))
ax.legend()
ax.set_xlabel('Polynomial Feature Order')
ax.set_ylabel('Accuracy')
```


# Linear optimization

```python
# Import Gurobi package
import gurobipy as gb
```

## Exercise

\begin{align}
Min. 98x_1 + 100x_2\\
6x_1 + 3x_2 \geq 500\\
7x_2 \geq 500\\
2x_1 +4x_2 \geq 500\\
x_1,x_2 \geq 0
\end{align}

```python
# Create model
m = gb.Model()

# Add variables
x1 = m.addVar()
x2 = m.addVar()

# Define objective function
m.setObjective(98 * x1 + 100 * x2, sense=gb.GRB.MINIMIZE)

# Add constraints
m.addConstr(6 * x1 + 3 * x2 >= 500)
m.addConstr(7 * x2 >= 500)
m.addConstr(2 * x1 + 4 * x2 >= 500)

# Run optimization
m.optimize()
```

    Gurobi Optimizer version 9.0.1 build v9.0.1rc0 (win64)
    Optimize a model with 3 rows, 2 columns and 5 nonzeros
    Model fingerprint: 0x276de63b
    Coefficient statistics:
      Matrix range     [2e+00, 7e+00]
      Objective range  [1e+02, 1e+02]
      Bounds range     [0e+00, 0e+00]
      RHS range        [5e+02, 5e+02]
    Presolve removed 1 rows and 0 columns
    Presolve time: 0.00s
    Presolved: 2 rows, 2 columns, 4 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    1.1809524e+04   7.440476e+00   0.000000e+00      0s
           1    1.3833333e+04   0.000000e+00   0.000000e+00      0s

    Solved in 1 iterations and 0.00 seconds
    Optimal objective  1.383333333e+04

```python
# Get values of decision variables
print('x1 =', x1.X)
print('x2 =', x2.X)
print('Objective function value =', m.ObjVal)
```

    x1 = 27.77777777777778
    x2 = 111.11111111111111
    Objective function value = 13833.333333333334

```python

```

Basics of gurobi optimization/01 Linear optimization.ipynb


# Least Absolute Deviation (LAD) Regression

Linear regression is a supervised machine learning technique that produces a linear model predicting values of a dependent variable from known values of one or more independent variables. Linear regression has a long history dating back to at least the 19th century and is a mainstay of modern data analysis.

This notebook demonstrates a technique for linear regression based on linear programming to minimize a sum of absolute errors between the model prediction and data from a training set. The sum of absolute values of  errors is the $L_1$ norm which is known to have favorable robustness characteristics in practical use. We follow closely this [paper](https://www.jstor.org/stable/1402501).

```python
# Install Pyomo and solvers for Google Colab
import sys
if "google.colab" in sys.modules:
    !wget -N -q https://raw.githubusercontent.com/jckantor/MO-book/main/tools/install_on_colab.py
    %run install_on_colab.py
```

## Generate data

The Python [scikit learn](https://scikit-learn.org/stable/) library for machine learning provides a full-featured collection of tools for regression. The following cell uses `make_regression` from scikit learn to generate a synthetic data set for use in subsequent cells. The data consists of a numpy array `y` containing `n_samples` of one dependent variable $y$, and an array `X` containing `n_samples` observations of `n_features` independent explanatory variables.

```python
from sklearn.datasets import make_regression
import numpy as np

n_features = 1
n_samples = 1000
noise = 30

# generate regression dataset
np.random.seed(2020)
X, y = make_regression(n_samples=n_samples, n_features=n_features, noise=noise)
```

## Data Visualization

Before going further, it is generally useful to prepare an initial visualization of the data. The following cell presents a scatter plot of $y$ versus $x$ for the special case of one explanatory variable, and a histogram of the difference between $y$ and the mean value $\bar{y}$. This histogram will provide a reference against which to compare the residual error in $y$ after regression.

```python
import matplotlib.pyplot as plt

if n_features == 1:
    plt.scatter(X, y, alpha=0.3)
    plt.xlabel("X")
    plt.ylabel("y")
    plt.grid(True)

plt.figure()
plt.hist(y - np.mean(y), bins=int(np.sqrt(len(y))))
plt.title('histogram y - mean(y)')
plt.ylabel('counts')
```

## Model

Suppose we have a finite dataset consisting of $n$ points $\{({X}^{(i)}, y^{(i)})\}_{i=1,\dots,n}$ with ${X}^{(i)} \in \mathbb{R}^k$ and $y^{(i)} \in \mathbb{R}$. A linear regression model assumes the relationship between the vector of $k$ regressors ${X}$ and the dependent variable $y$ is linear. This relationship is modeled through an error or deviation term $e_i$, which quantifies how much each of the data points diverge from the model prediction and is defined as follows:

$$
\begin{equation}\label{eq:regression}
    e_i:= y^{(i)} - {m}^\top {X}^{(i)} - b = y^{(i)} - \sum_{j=1}^k X^{(i)}_j m_j - b,
\end{equation}
$$

for some real numbers $m_1,\dots,m_k$ and $b$.

The Least Absolute Deviation (LAD) is a possible statistical optimality criterion for such a linear regression. Similar to the well-known least-squares technique, it attempts to find a vector of linear coefficients ${m}=(m_1,\dots,m_k)$ and intercept $b$ so that the model closely approximates the given set of data. The method minimizes the sum of absolute errors, that is $\sum_{i=1}^n \left |e_i \right|$.

The LAD regression is formulated as an optimization problem with the intercept $b$, the coefficients $m_i$'s, and the errors $e_i$'s as decision variables, namely

$$
\begin{align}
    \min \quad & \sum_{i=1}^n |e_i| \\
    \text{s.t.} \quad & e_i = y^{(i)} - {m}^\top {X}^{(i)} - b & \forall\, i=1,\dots,n.
\end{align}
$$

In general, the appearance of an absolute value term indicates the problem is nonlinear and, worse, that the objective function is not differentiable when any $e_i = 0$. However, for this case where the objective is to minimize a sum of absolute errors, one can reformulate the decision variables to transform this into a linear problem. More specifically, introducing for every term $e_i$ two new variables $e_i^-, e_i^+ \geq 0$, we can rewrite the model as

$$
\begin{align}
    \min \quad & \sum_{i=1}^n ( e_i^+ + e_i^-) \\
    \text{s.t.} \quad & e_i^+ - e_i^- = y^{(i)} - {m}^\top {X}^{(i)}-b & \forall\, i=1, \dots, n \\
    & e_i^+, e_i^- \geq 0 & \forall\, i=1, \dots, n
\end{align}
$$

The following cell provides a direct implementation of LAD regression.

```python
import pyomo.environ as pyo

def lad_regression(X, y):

    m = pyo.ConcreteModel('LAD Regression')

    n, k = X.shape

    # note use of Python style zero based indexing
    m.I = pyo.RangeSet(0, n-1)
    m.J = pyo.RangeSet(0, k-1)

    m.ep = pyo.Var(m.I, domain=pyo.NonNegativeReals)
    m.em = pyo.Var(m.I, domain=pyo.NonNegativeReals)
    m.m = pyo.Var(m.J)
    m.b = pyo.Var()

    @m.Constraint(m.I)
    def residuals(m, i):
        return m.ep[i] - m.em[i] == y[i] - sum(X[i][j]*m.m[j] for j in m.J) - m.b

    @m.Objective(sense=pyo.minimize)
    def sum_of_abs_errors(m):
        return sum(m.ep[i] + m.em[i] for i in m.I)

    pyo.SolverFactory('glpk').solve(m)

    return m

m = lad_regression(X, y)
m.m.display()
m.b.display()
```

    m : Size=1, Index=J
        Key : Lower : Value            : Upper : Fixed : Stale : Domain
          0 :  None : 90.4819738904496 :  None : False : False :  Reals
    b : Size=1, Index=None
        Key  : Lower : Value              : Upper : Fixed : Stale : Domain
        None :  None : -0.187847121914792 :  None : False : False :  Reals

## Visualizing the Results

```python
y_fit = np.array([sum(x[j]*m.m[j]() for j in m.J) + m.b() for x in X])

if n_features == 1:
    plt.scatter(X, y, alpha=0.3, label="data")
    plt.plot(X, y_fit, 'r', label="y_fit")
    plt.xlabel("X")
    plt.ylabel("y")
    plt.grid(True)
    plt.legend()

plt.figure()
plt.hist(y - np.mean(y), bins=int(np.sqrt(len(y))), alpha=0.5, label="y - mean(y)")
plt.hist(y - y_fit, bins=int(np.sqrt(len(y))), color='r', alpha=0.8, label="y - y_fit")
plt.title('histogram of residuals')
plt.ylabel('counts')
plt.legend()
```

```python

```


# Проверка усвоения знаний

```python
from sympy import *

def rz(a):
    return Matrix([
        [cos(a), -sin(a), 0, 0],
        [sin(a), cos(a), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

def ry(a):
    return Matrix([
        [cos(a), 0, sin(a), 0],
        [0, 1, 0, 0],
        [-sin(a), 0, cos(a), 0],
        [0, 0, 0, 1]
    ])

def rx(a):
    return Matrix([
        [1, 0, 0, 0],
        [0, cos(a), -sin(a), 0],
        [0, sin(a), cos(a), 0],
        [0, 0, 0, 1]
    ])

def trs(x, y, z):
    return Matrix([
        [1, 0, 0, x],
        [0, 1, 0, y],
        [0, 0, 1, z],
        [0, 0, 0, 1]
    ])

def vec(x, y, z):
    return Matrix([
        [x],
        [y],
        [z],
        [1]
    ])

def dh(theta, d, alpha, a):
    return rz(theta) * trs(0, 0, d) * rx(alpha) * trs(a, 0, 0)
```

```python
def joint_spec(number):
    return [
        "вращение вокруг X",
        "вращение вокруг Y",
        "вращение вокруг Z",
        "перемещение вдоль X",
        "перемещение вдоль Y",
        "перемещение вдоль Z"
    ][number]

def bone_spec(number):
    return [
        "смещение вдоль X",
        "смещение вдоль Y",
        "смещение вдоль Z",
    ][number]
```

```python
def get_my_robot(my_id):
    my_id = my_id * 7919
    first_id = my_id
    second_id = my_id // 6
    third_id = second_id // 3
    fourth_id = third_id // 3
    fifth_id = fourth_id // 3
    sixth_id = fifth_id // 3

    first = first_id % 3
    second = (first + second_id + 1) % 6
    third = (second + third_id + 1) % 3
    fourth = (third + fourth_id + 1) % 3
    fifth = (fourth + fifth_id + 1) % 3
    sixth = (fifth + sixth_id + 1) % 3

    return "Мой робот состоит из следующих элементов:" +\
        "\n\t- " + bone_spec(third) +\
        "\n\t- " + joint_spec(first) +\
        "\n\t- " + bone_spec(fourth) +\
        "\n\t- " + joint_spec(second) +\
        "\n\t- " + bone_spec(fifth) +\
        "\n\t- " + joint_spec(sixth) +\
        "\n\t- " + bone_spec(2)
```

Введите номер вашего варианта:

```python
my_variant = 0
```

```python
print(
    get_my_robot(my_variant)
)
```

## Задание 0

Нарисуйте свой манипулятор в _нулевом_ положении: все обобщенные координаты равны $0$, все смещения совпадают с описанием.

## Задание 1

Для своего робота решите прямую задачу кинематики простым способом:

```python
q1, q2, q3 = symbols("q_1, q_2, q_3")
l1, l2, l3, l4, l5, l6, l7 = symbols("l_1, l_2, l_3, l_4, l_5, l_6, l_7")
```

```python
# FIX NEXT LINE
pos = trs(0, 0, l1) * rx(q1) * trs(l2, 0, 0) * ry(q2) * trs(0, l3, 0) * rz(q3) * trs(0, 0, l4)
```

```python
simplify(
    pos
)
```

```python
simplify(
    pos * vec(0, 0, 0)
)
```

## Задание 2

Воспользуйтесь DH-нотацией чтобы расставить системы координат, решите прямую задачу кинематики.
Помните что угол задается в радианах, например, как:
```
pi / 2
```

В данном примере в DH-нотации сначала идут поворот-смещение вокруг-вдоль $Z$, затем поворот-смещение вокруг-воль $X$.

```python
# FIX NEXT LINE, MAKE SURE IT IS FILLED WITH `dh` ONLY
pos_dh = dh(pi / 2, l1, pi / 2, 0) * dh(q1 + pi / 2, l2, pi / 2, 0) * dh(q2 + pi / 2, l3, pi / 2, 0) * dh(q3, l4, 0, 0)
```

```python
simplify(
    pos_dh
)
```

```python
simplify(
    pos_dh * vec(0, 0, 0)
)
```


### DEMSLV08

# Nonlinear complementarity problem methods

Solve nonlinear complementarity problem on $R^2$ using semismooth and minmax methods.

Function to be solved is

\begin{equation}
f(x,y) = \begin{bmatrix}
200x(y - x^2) + 1 - x \\ 100(x^2 - y)
\end{bmatrix}
\end{equation}

### Set up the problem
The class **MCP** is used to represent mixed-complementarity problems. To create one instance, we define the objective function and the boundaries $a$ and $b$ such that for $a \leq x \leq b$. Apart from the required parameters, we can specify options to be used when solving the problem.

```python
from demos.setup import np, tic, toc
from compecon import MCP
```

```python
def f(z):
    x, y = z
    fval = np.array([200 * x * (y - x ** 2) + 1 - x,
                     100 * (x ** 2 - y)])
    fjac = np.array([[200 * (y - x ** 2) - 400 * x ** 2 - 1, 200 * x],
                         [200 * x, -100]])

    return fval, fjac
```

### Generate problem test data

```python
z = 2 * np.random.randn(2, 2)
a = 1 + np.min(z, 0)
b = 1 + np.max(z, 0)

F = MCP(f, a, b, maxit=1500)
```

### Solve by applying Newton method
We'll use a random initial guess $x$

```python
F.x0 = np.random.randn(2)
```

### Check the Jacobian

```python
F.check_jacobian()
```

    All numerical derivatives differ from
    the user-provided ones by less than 8 decimal digits.

    The maximum error is 7.34e-09, for row 0 and column 0.

###  Using minmax formulation

```python
t0 = tic()
x1 = F.zero(transform='minmax')
t1 = 100 * toc(t0)
n1 = F.fnorm
```

### Using semismooth formulation

```python
t0 = tic()
x2 = F.zero(transform='ssmooth')
t2 = 100*toc(t0)
n2 = F.fnorm
```

###  Print results

```python
hdr = 'Hundreds of seconds required to solve nonlinear complementarity \n' +\
      'problem on R^2 using minmax and semismooth formulations, with \n'  +\
      'randomly generated bounds \n' +\
      '\ta = [{:4.2f}, {:4.2f}] \n' + \
      '\tb = [{:4.2f}, {:4.2f}]'

frm = '{:20} {:6.3f}  {:8.0e}     {:5.2f}  {:5.2f}'
prt = lambda method, t, n, x: print(frm.format(method, t, n, *x))

print(hdr.format(a[0], a[1], b[0], b[1]))
print('\nAlgorithm           Time      Norm        x1     x2\n{}'.format('-' * 56));
prt('Newton minmax',     t1, n1, x1)
prt('Newton semismooth', t2, n2, x2)
```

    Hundreds of seconds required to solve nonlinear complementarity
    problem on R^2 using minmax and semismooth formulations, with
    randomly generated bounds
    	a = [-1.95, -1.94]
    	b = [0.54, -0.73]

    Algorithm           Time      Norm        x1     x2
    --------------------------------------------------------
    Newton minmax         1.600     3e-10      1.00   1.00
    Newton semismooth     0.800     3e-10      1.00   1.00

notebooks/slv/08 Nonlinear complementarity problem methods.ipynb


#
Linear approximation of $\sqrt{1+x}$ where $x \in [0,1]$

---------------------------------------------------------------------

```python
import pandas as pd

import numpy as np
from numpy.linalg import inv

from matplotlib import pyplot as plt
plt.style.use(['ggplot'])
%matplotlib inline
fig = plt.figure(figsize = (16,8))

from sklearn.linear_model import LinearRegression as LR
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures

import tensorflow as tf
print('tensorflow versio: ', tf.__version__)

import warnings
warnings.filterwarnings('ignore')
```

    tensorflow versio:  1.14.0

```python
np.random.seed(22)
tf.set_random_seed(22)
```

```python
# Number of data points
n = 10000
```

```python
X = np.random.random(n).reshape((n, 1))
y = np.sqrt(1+X)
```

```python
plt.plot(X, y, 'ro', label ='Original data')
plt.title('Generated data')
plt.legend()
plt.show()
```

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=22)

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=22)
```

# 1. With closed-form:

For linear regression on a model of the form $y=X \beta$, where $X$ is a matrix with ***full column rank***, the least squares solution,

$\hat{\beta}=argmin||X \beta - y||_{2}$

is given by

$\hat{\beta}=(X^{T}X)^{-1}X^{T}y$

Here requires the inversion of $X^{T}X$. If this matrix is not of full rank (i.e. there are linearly dependent columns) or if it is poorly conditioned (i.e. eigen values which are very close to zero) then the inversion can either be impossible, or numerically unstable.

A solution is to add a small element to the diagonal such that the eigen-values (and the matrix) are better conditioned

$\hat{\beta} = (X^{T}X+ \lambda^{2} I_{n})^{-1}X^{T}y$

This is actually the solution to the following quadratic cost function:

$J(\beta)=(y - X \beta )^{T} (y - X \beta) + \lambda^{2} \beta ^{T} \beta$

This is proven as follows:

$J(\beta)=(y - X \beta )^{T} (y - X \beta) + \lambda^{2} \beta ^{T} \beta =
(y^{T} - \beta^{T} X^{T} ) (y - X \beta) + \lambda^{2} \beta ^{T} \beta =
y^{T}y - y^{T}X \beta - \beta^{T} X^{T}y + \beta^{T} X^{T}X \beta + \lambda^{2} \beta ^{T} \beta =
y^{T}y - 2\beta^{T} X^{T}y + \beta^{T} X^{T}X \beta + \lambda^{2} \beta ^{T} \beta$

Then Calculate gradient by $\beta$ and we will obtain above mentioned equation.

```python
# create vector of ones
int = np.ones(shape=y_train.shape[0])[..., None]

#and add to feature matrix
X_train_with_ones = np.concatenate((int, X_train), 1)
```

```python
# # calculate coefficients using closed-form solution
# coeffs = inv(X_train_with_ones.transpose().dot(X_train_with_ones)).dot(X_train_with_ones.transpose()).dot(y_train)
```

```python
def closed_form_reg_solution(X, y, lamda = 0.1):
    '''
    Closed form solution for ridge regression

    Parameters:
        X (list or nd.array) - actual inputs
        y (list or nd.array) - actual labels
        lambda(float)

    Returns:
        list (coefficients)
    '''
    m,n = X.shape
    I = np.eye((n))
    return (inv(X.transpose().dot(X) + lamda * I)).dot(X.transpose()).dot(y)[:,0]
```

```python
def mse(actual, predicted):
    """
    Calculate mean squared error

    Parameters:
        actual (nd.array) - actual values
        predicted (nd.array) - predicted values

    Return:
        float(mse)
    """
    return (np.square(actual - predicted)).mean()
```

```python
lambdas = [0.01, 0.1, 1, 10]
mse_s = []
for i in range(len(lambdas)):
    coeffs = closed_form_reg_solution(X_train_with_ones, y_train, lamda=lambdas[i])
    prediction = coeffs[0] * X_val + coeffs[1]
    mse_s.append(mse(y_val,prediction ))
```

```python
best_lambda_index = np.argmin(mse_s)
```

```python
best_lambda = lambdas[best_lambda_index]
```

```python
coeffs = closed_form_reg_solution(X_train_with_ones, y_train, lamda=best_lambda)
```

```python
intercept_closed_form = np.round(coeffs[0], 3)
slope_closed_form = np.round(coeffs[1], 3)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_closed_form, slope_closed_form))
print('Approximated function: ', intercept_closed_form, '+', slope_closed_form, '* X' )
```

    Intercept:       1.010,
    Slope:           0.414
    Approximated function:  1.01 + 0.414 * X

```python
prediction_closed_form = slope_closed_form * X_test + intercept_closed_form

print('MSE in Test dataset: ', mse(y_test, prediction_closed_form))

plt.plot(X_test, y_test, 'ro', label ='test data')
plt.plot(X_test, prediction_closed_form, label ='Fitted line', color='green')
plt.title('Closed form solution result')
plt.legend()
plt.show()
```

# 2. With sklearn

```python
lambdas = [0.01, 0.1, 1, 10]
mse_s = []

for i in range(len(lambdas)):

    regressor = Ridge(alpha=i, random_state=42)
    regressor.fit(X_train, y_train)
    prediction = regressor.predict(X_val)
    mse_s.append(mse(y_val, prediction ))
```

```python
best_lambda_index = np.argmin(mse_s)
```

```python
best_lambda = lambdas[best_lambda_index]
```

```python
regressor = Ridge(alpha=best_lambda, random_state=42)
regressor.fit(X_train, y_train)
```

    Ridge(alpha=0.01, copy_X=True, fit_intercept=True, max_iter=None,
       normalize=False, random_state=42, solver='auto', tol=0.001)

```python
intercept_ridge = np.round(regressor.intercept_[0], 3)
slope_ridge = np.round(regressor.coef_[0, 0], 3)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_ridge, slope_ridge))
print('Approximated function: ', intercept_ridge, '+', slope_ridge, '* X' )
```

    Intercept:       1.013,
    Slope:           0.412
    Approximated function:  1.013 + 0.412 * X

```python
prediction_ridge = regressor.predict(X_test)

print('MSE in Test dataset: ', mse(y_test, prediction_ridge))

plt.plot(X_test, y_test, 'ro', label ='Original data')
plt.plot(X_test, prediction_ridge, label ='Fitted line', color='green')
plt.title('Ridge Regression Result')
plt.legend()
plt.show()
```

```python
regressor_lr = LR()
regressor_lr.fit(X_train, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
intercept_lr = np.round(regressor_lr.intercept_[0], 3)
slope_lr = np.round(regressor_lr.coef_[0, 0], 2)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_lr, slope_lr))
print('Approximated function: ', intercept_lr, '+', slope_lr, '* X' )
```

    Intercept:       1.013,
    Slope:           0.410
    Approximated function:  1.013 + 0.41 * X

```python
prediction_lr = regressor_lr.predict(X_test)

print('MSE in Test dataset: ', mse(y_test, prediction_lr))

plt.plot(X_test, y_test, 'ro', label ='Original data')
plt.plot(X_test, prediction_lr, label ='Fitted line', color='green')
plt.title('Linear Regression Result')
plt.legend()
plt.show()
```

# 3. With Implemented gradient descent

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_{0} -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_{1} -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_{2} -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_{j} -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}

```python
def gradient_descent(X, y, theta, learning_rate=0.01, iterations=100):
    '''
    Parameters:
        X     = Matrix of X with added bias units
        y     = Vector of Y
        theta = Vector of thetas
        learning_rate
        iterations = no of iterations

    Returns:
        np.array - (Returns the final theta vector)
    '''
    m = len(y)
    for it in range(iterations):

        prediction = np.dot(X,theta)

        theta = theta -(1/m)*learning_rate*( X.T.dot(prediction - y))
    return theta
```

```python
lr =0.01
n_iter = 100000

theta = np.random.randn(2,1)

theta = gradient_descent(X_train_with_ones, y_train, theta, lr, n_iter)

intercept_gd = np.round(theta[0,0], 3)
slope_gd = np.round(theta[1,0], 3)

print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_gd, slope_gd))
print('Approximated function: ', intercept_gd, '+', slope_gd, '* X' )
```

    Intercept:       1.013,
    Slope:           0.412
    Approximated function:  1.013 + 0.412 * X

```python
prediction_gd = slope_gd * X_test + intercept_gd

print('MSE in Test dataset: ', mse(y_test, prediction_gd))

plt.plot(X_train, y_train, 'ro', label ='Original data')
plt.plot(X_test, prediction_gd, label ='Fitted line', color='green')
plt.title('Implemented gradient descent Result')
plt.legend()
plt.show()
```

```python
def gradient_descent_with_regularization(X, y, theta, learning_rate=0.01, iterations=100, lamda = 0.1):
    '''
    Gradient descent for ridge regression

    Parameters:
        X     = Matrix of X with added bias units
        y     = Vector of Y
        theta = Vector of thetas
        learning_rate
        iterations = no of iterations
        lamda = regularization parameter

    Returns:
        np.array - (Returns the final theta vector)

    '''
    m = len(y)
    for it in range(iterations):

        prediction = np.dot(X,theta)

        theta = theta -(1/m)*learning_rate*( X.T.dot(prediction - y) + lamda * theta)
    return theta
```

```python
lr =0.01
n_iter = 100000

lambdas = [0.01, 0.1, 1, 10]
mse_s = []
for i in range(len(lambdas)):
    theta = np.random.randn(2,1)

    theta = gradient_descent_with_regularization(X_train_with_ones, y_train, theta, lr, n_iter, i)

    prediction = theta[1,0] * X_val + theta[0,0]

    mse_s.append(mse(y_val, prediction ))
```

```python
best_lambda_index = np.argmin(mse_s)
```

```python
best_lambda = lambdas[best_lambda_index]
```

```python
lr =0.01
n_iter = 100000

theta = np.random.randn(2,1)

theta = gradient_descent_with_regularization(X_train_with_ones, y_train, theta, lr, n_iter, best_lambda)

intercept_rgd = np.round(theta[0,0], 3)
slope_rgd = np.round(theta[1,0], 3)

print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_rgd, slope_rgd))
print('Approximated function: ', intercept_rgd, '+', slope_rgd, '* X' )
```

    Intercept:       1.013,
    Slope:           0.412
    Approximated function:  1.013 + 0.412 * X

```python
prediction_rgd = slope_rgd * X_test + intercept_rgd

print('MSE in Test dataset: ', mse(y_test, prediction_rgd))

plt.plot(X_train, y_train, 'ro', label ='Original data')
plt.plot(X_test, prediction_rgd, label ='Fitted line', color='green')
plt.title('Implemented gradient descent with regularization Result')
plt.legend()
plt.show()
```

```python
def stocashtic_gradient_descent(X, y, theta, learning_rate=0.01, iterations=100):
    '''
    Stochastic gradient descent

    Parameters:
        X     = Matrix of X with added bias units
        y     = Vector of Y
        theta = Vector of thetas
        learning_rate
        iterations = no of iterations

    Returns:
        np.array - (Returns the final theta vector)

    '''
    m = len(y)

    for it in range(iterations):
        for i in range(m):
            rand_ind = np.random.randint(0,m)
            X_i = X[rand_ind,:].reshape(1,X.shape[1])
            y_i = y[rand_ind].reshape(1,1)
            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
    return theta
```

```python
## It takes a long time

# lr =0.01
# n_iter = 10000

# theta = np.random.randn(2,1)

# theta = stocashtic_gradient_descent(X_train_with_ones, y_train, theta, lr, n_iter)

# intercept_sgd = np.round(theta[0,0], 3)
# slope_sgd = np.round(theta[1,0], 3)

# print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_sgd, slope_sgd))
# print('Approximated function: ', intercept_sgd, '+', slope_sgd, '* X' )
```

```python
# prediction_sgd = slope_sgd * X_test + intercept_sgd

# print('MSE in Test dataset: ', mse(y_test, prediction_sgd))

# plt.plot(X_train, y_train, 'ro', label ='Original data')
# plt.plot(X_test, prediction_sgd, label ='Fitted line', color='green')
# plt.title('Implemented stochastic gradient descent Result')
# plt.legend()
# plt.show()
```

```python
import math
def minibatch_gradient_descent(X, y, theta, learning_rate=0.01, iterations=100, batch_size = 20):
    '''
    Minibatch gradient descent

    Parameters:
        X    = Matrix of X without added bias units
        y    = Vector of Y
        theta=Vector of thetas np.random.randn(j,1)
        learning_rate
        iterations = no of iterations
        batch_size

    Returns:
        np.array - (Returns the final theta vector)

    '''
    m = len(y)
    n_batches = math.floor(m/batch_size)
#     n_batches = int(m/batch_size)

    for it in range(iterations):
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            X_i = X[i:i+batch_size]
            y_i = y[i:i+batch_size]

            X_i = np.c_[np.ones(len(X_i)),X_i]

            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))

    return theta
```

```python
lr =0.01
n_iter = 10000
batch_size = 20

theta = np.random.randn(2,1)

theta = minibatch_gradient_descent(X_train, y_train, theta, lr, n_iter, batch_size)

intercept_mgd = np.round(theta[0,0], 3)
slope_mgd = np.round(theta[1,0], 3)

print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_mgd, slope_mgd))
print('Approximated function: ', intercept_mgd, '+', slope_mgd, '* X' )
```

    Intercept:       1.012,
    Slope:           0.413
    Approximated function:  1.012 + 0.413 * X

```python
prediction_mgd = slope_mgd * X_test + intercept_mgd

print('MSE in Test dataset: ', mse(y_test, prediction_mgd))

plt.plot(X_train, y_train, 'ro', label ='Original data')
plt.plot(X_test, prediction_mgd, label ='Fitted line', color='green')
plt.title('Implemented minibatch gradient descent Result')
plt.legend()
plt.show()
```

# 4. With Tensorflow

```python
X = tf.placeholder("float")
Y = tf.placeholder("float")
```

```python
W = tf.Variable(np.random.randn(), name = "W")
b = tf.Variable(np.random.randn(), name = "b")
```

```python
learning_rate = 0.01
training_epochs = 100
```

```python
y_pred = tf.add(tf.multiply(X, W), b)

# Mean Squared Error Cost Function
cost = tf.reduce_sum(tf.pow(y_pred-Y, 2)) / (2 * n)

# Gradient Descent Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()
```

    WARNING: Logging before flag parsing goes to stderr.
    W0924 23:14:20.509317 10128 deprecation.py:323] From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\math_grad.py:1205: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where

```python
with tf.Session() as sess:

    sess.run(init)

    for epoch in range(training_epochs):
        for (_x, _y) in zip(X_train, y_train):
            sess.run(optimizer, feed_dict = {X : _x, Y : _y})

        if (epoch + 1) % 50 == 0:
            c = sess.run(cost, feed_dict = {X : X_train, Y : y_train})
            print("Epoch", (epoch + 1), ": cost =", c, "W =", sess.run(W), "b =", sess.run(b))

    weight = sess.run(W)
    bias = sess.run(b)
```

    Epoch 50 : cost = 0.3669787 W = 0.3993049 b = -0.051624123
    Epoch 100 : cost = 0.16549797 W = 0.5393183 b = 0.23081736

```python
intercept_tf = bias
slope_tf = weight

print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_tf, slope_tf))
print('Approximated function: ', intercept_tf, '+', slope_tf, '* X' )
```

    Intercept:       0.231,
    Slope:           0.539
    Approximated function:  0.23081736 + 0.5393183 * X

```python
prediction_tf = slope_tf * X_test + intercept_tf

print('MSE in Test dataset: ', mse(y_test, prediction_tf))

plt.plot(X_train, y_train, 'ro', label ='Original data')
plt.plot(X_test, prediction_tf, label ='Fitted line', color='green')
plt.title('Linear regression with tensorflow Result')
plt.legend()
plt.show()
```

##  Compare all solutions with the first degree Taylor approximation of the function $\sqrt{x+1}$

First degree Taylor approximation of the function $\sqrt{x+1}$ around x=0 is $1 + 1/2 x$

```python
print('coefficients with closed form')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_closed_form, slope_closed_form))
print('\n')

print('coefficients with ridge regression')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_ridge, slope_ridge))
print('\n')

print('coefficients with lr')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_lr, slope_lr))
print('\n')

print('coefficients with gradient descent')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_gd, slope_gd))
print('\n')

print('coefficients with regularized gradient descent')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_rgd, slope_rgd))
print('\n')

print('coefficients with tensorflow')
print('-'*50)
print('Intercept:       {:0.3f},\nSlope:           {:0.3f}'.format(intercept_tf, slope_tf))
print('\n')
```

    coefficients with closed form
    --------------------------------------------------
    Intercept:       1.010,
    Slope:           0.414

    coefficients with ridge regression
    --------------------------------------------------
    Intercept:       1.013,
    Slope:           0.412

    coefficients with lr
    --------------------------------------------------
    Intercept:       1.013,
    Slope:           0.410

    coefficients with gradient descent
    --------------------------------------------------
    Intercept:       1.013,
    Slope:           0.412

    coefficients with regularized gradient descent
    --------------------------------------------------
    Intercept:       1.013,
    Slope:           0.412

    coefficients with tensorflow
    --------------------------------------------------
    Intercept:       0.231,
    Slope:           0.539

from the above printed coefficients values we can say, that closed form (with regularization) gives the best (close to the actual coefficients in taylor series) result among others.

## Can you implement the polynomial regression using linear regression?

```python
poly = PolynomialFeatures(degree = 4)
X_train_poly = poly.fit_transform(X_train)
```

```python
X_train_poly
```

    array([[1.        , 0.52453875, 0.27514091, 0.14432207, 0.07570252],
           [1.        , 0.54196234, 0.29372318, 0.1591869 , 0.08627331],
           [1.        , 0.93854611, 0.8808688 , 0.82673598, 0.77592984],
           ...,
           [1.        , 0.70716309, 0.50007963, 0.35363786, 0.25007964],
           [1.        , 0.30338642, 0.09204332, 0.02792469, 0.00847197],
           [1.        , 0.20287155, 0.04115686, 0.00834956, 0.00169389]])

```python
regressor_lr_poly = LR()
regressor_lr_poly.fit(X_train_poly, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
intercept_lr_poly = np.round(regressor_lr_poly.intercept_[0], 3)
slope_1_lr_poly = np.round(regressor_lr_poly.coef_[0,1], 3)
slope_2_lr_poly = np.round(regressor_lr_poly.coef_[0,2], 3)
slope_3_lr_poly = np.round(regressor_lr_poly.coef_[0,3], 3)
slope_4_lr_poly = np.round(regressor_lr_poly.coef_[0,4], 3)
print('Intercept:       {:0.3f},\nSlope 1:         {:0.3f},\nSlope 2:         {:0.3f},\nSlope 3:         {:0.3f},\nSlope 4:         {:0.3f}'.format(intercept_lr_poly, slope_1_lr_poly, slope_2_lr_poly, slope_3_lr_poly, slope_4_lr_poly))
print('Approximated function: ', intercept_lr_poly, '+', slope_1_lr_poly, '* X', '+(', slope_2_lr_poly, ')* X^2', '+(', slope_3_lr_poly, ')* X^3', '+(', slope_4_lr_poly, ')* X^4' )
```

    Intercept:       1.000,
    Slope 1:         0.499,
    Slope 2:         -0.119,
    Slope 3:         0.044,
    Slope 4:         -0.010
    Approximated function:  1.0 + 0.499 * X +( -0.119 )* X^2 +( 0.044 )* X^3 +( -0.01 )* X^4

```python
X_test_poly = poly.fit_transform(X_test)
prediction_poly = regressor_lr_poly.predict(X_test_poly)

print('MSE in Test dataset: ', mse(y_test, prediction_poly))

plt.plot(X_test, y_test, 'ro', label ='Original data')
plt.plot(X_test, prediction_poly, label ='Fitted line', color='green')
plt.title('Polinomeal Regression Result')
plt.legend()
plt.show()
```

# All approximations

```python
plt.figure(figsize = (15, 10))

plt.plot(X_test, y_test, 'ro', label ='Original data', alpha=0.1)

plt.plot(X_test, prediction_closed_form, label = 'Closed form')
plt.plot(X_test, prediction_ridge, label = 'Ridge Regression')
plt.plot(X_test, prediction_lr, label = 'Linear Regression')
plt.plot(X_test, prediction_gd, label = 'Gradient Descent')
plt.plot(X_test, prediction_rgd, label = 'Regularized Gradient Descent')
plt.plot(X_test, prediction_tf, label = 'Tensorflow')
plt.plot(X_test, prediction_poly, label = 'Polinomeal Regression', alpha = 0.5)

plt.title('All approximations')
plt.xlabel('Test data')
plt.ylabel('Predicted values')
plt.legend()
plt.show()
```

```python

```

```python

```


# Matriz de rigidez de una viga de Bernoulli-Euler

_Análisis de Estructuras. Grado en Ingeniería Civil._

Autor:

- Alejandro E. Martínez-Castro.
- Departamento de Mecánica de Estructuras e Ingeniería Hidráulica. Universidad de Granada.

Matriz de rigidez de viga de Bernoulli-Euler by Alejandro Martínez Castro is licensed under a Creative Commons Reconocimiento-NoComercial-CompartirIgual 4.0 Internacional License.

## Objetivos del cuaderno

En este cuaderno se va a obtener la matriz de rigidez para los términos de flexión de una viga de Bernoulli-Euler. La matriz se va a obtener a partir de la definición vista en clase. Esto permite explorar los siguientes objetivos:

- Generación de las funciones cúbicas, soluciones a problemas de flexión de vigas con cargas en los extremos (funciones de Hermite).
- Introducción de la metodología general para obtener las denominadas "funciones de forma", que más adelante se emplearán en el Método de los Elementos Finitos.
- Determinación de los términos de las columnas de la matriz de rigidez mediante la evaluación directa de los esfuerzos cortante y flector en los extremos, y sus relaciones con las fuerzas y momentos en los extremos de barra.

En primer lugar, cargaremos el paquete SymPy, para poder realizar cálculos simbólicos

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.9 (Python 3.8.12-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.9/

## Definición de matrices de base polinómica y su primera derivada

A continuación definiremos el símbolo para la longitud $L$, y definiremos la base de los polinomios cúbicos sobre la que se interpolará la solución.

```python
L = symbols('L')
base = Matrix([[1,x,x**2,x**3]])
base
```

Definamos igualmente una lista con la derivada primera de estas funciones de base, pues necesitaremos imponer condiciones en giros:

```python
deribase = diff(base,x)
deribase
```

## Construcción de la matriz de aplicaciones lineales asociadas a evaluar en los extremos del elemento viga.

A continuación, construiremos la matriz asociada a cuatro aplicaciones lineales (por filas). En cada fila, se construirán las siguientes matrices:
- En la primera fila, la aplicación para evaluar en el origen, $x=0$.
- En la segunda fila, la aplicación para evaluar la primera derivada en el origen $x=0$.
- En la tercera fila, la aplicación para evaluar la función en $x=L$.
- En la cuarta fila, la aplicación para evaluar la derivada de la función en $x=L$.

```python
matM = Matrix.vstack(base.subs(x, 0),
                     deribase.subs(x, 0),
                     base.subs(x, L),
                     deribase.subs(x, L))
matM
```

Según se vio en la teoría, la inversa de esta matriz, por columnas, contiene los cuatro coeficientes que multiplicarán a la base $[1,x,x^2,x^3]$ para construir las cuatro funciones $h1(x), h2(x), h3(x), h4(x)$.

```python
Inv_matM = matM.inv()
Inv_matM
```

## Construcción de las funciones de forma

Finalmente, se construyen las cuatro funciones cúbicas $hi(x)$ (también llamadas "polinomios de Hermite"), que representan las cuatro soluciones a una viga de Bernoulli-Euler, que incluyen un desplazamiento unidad según el grado de libertad $i$, manteniendo el resto de grados de libertado con valor nulo.

```python
h1 = base.dot(Inv_matM.col(0))
h2 = base.dot(Inv_matM.col(1))
h3 = base.dot(Inv_matM.col(2))
h4 = base.dot(Inv_matM.col(3))
```

Veamos una de ellas:

```python
h2
```

## Definición de las funciones "esfuerzo cortante" y "momento flector"

A continuación se definen las funciones que permiten obtener el esfuerzo cortante $\mathcal{V}(x)$ y el momento flector $\mathcal{M}(x)$. Estas funciones pueden construirse a partir de la solución para el desplazamiento $y(x)$, como:

\begin{eqnarray}
\mathcal{V}(x) & = & - EI\,y'''(x) \\
\mathcal{M}(x) & = & EI\,y''(x)
\end{eqnarray}

Nótese que estas funciones tienen signo. En clase se vio el criterio de signos positivo.

```python
EI = symbols('EI')
def cortante(f):
    return -EI*diff(f,x,3)

def flector(f):
    return EI*diff(f,x,2)
```

Podemos probar estas funciones. Por ejemplo, el momento flector en $x=0$ correspondiente a la función de desplazamiento $h1(x)$ es:

```python
flector(h1).subs(x,0)
```

## Construcción de la matriz de rigidez.

La columna $j$ de la matriz de rigidez, con $j\in[1,4]$ tiene 4 términos. En los cuatro términos de las filas, se ubican las fuerzas y momentos, según el criterio de signos para fuerzas en extremos de barras (no confundir con los signos para los momentos), que deben aplicarse para que la función de desplazamiento sea $hj(x)$.

Sus relaciones con las fuerzas y momentos en los extremos de barra, $F_i, M_i, F_j, M_j$, son las siguientes:

\begin{eqnarray}
F_i & = & - \mathcal{V}(0) \\
M_i & = & - \mathcal{M}(0) \\
F_j & = & \mathcal{V}(L) \\
M_j & = & \mathcal{M}(L)
\end{eqnarray}

Considerando estas relaciones, podemos construir la matriz de rigidez, evaluando en los extremos según la relación anterior, y teniendo en cuenta las soluciones ya calculadas para las funciones $h1(x), h2(x), h3(x), h4(x)$.

Definamos una función que nos devolverá la columna correspondiente a la matriz de rigidez, dada una función de desplazamiento genérica:

```python
def columna(f):
    return Matrix([[-cortante(f).subs(x,0)],
                   [-flector(f).subs(x,0)],
                   [cortante(f).subs(x,L)],
                   [flector(f).subs(x,L)]])
```

Veamos cómo funciona con una de las funciones polinómicas de Hermite:

```python
columna(h1)
```

Para construir la matriz de rigidez, simplemente "apilamos" las cuatro columnas, construidas consecutivamente para las cuatro funciones de Hermite:

```python
matriz_rigidez = Matrix.hstack(columna(h1),columna(h2),columna(h3),columna(h4))
matriz_rigidez

```