# Debiasing with Orthogonalization

Previously, we saw how to evaluate a causal model. By itself, that's a huge deed. Causal models estimates the elasticity $\frac{\delta y}{\delta t}$, which is an unseen quantity. Hence, since we can't see the ground truth of what our model is estimating, we had to be very creative in how we would go about evaluating them.

The technique shown on the previous chapter relied heavily on data where the treatment was randomly assigned. The idea was to estimate the elasticity $\frac{\delta y}{\delta t}$ as the coefficient of a single variable linear regression of `y ~ t`. However, this only works if the treatment is randomly assigned. If it isn't, we get into trouble due to omitted variable bias.

To workaround this, we need to make the data look as if the treatment is randomly assigned. I would say there are two main techniques to do this. One is using propensity score and the other using orthogonalization. We will cover the latter in this chapter.

One final word of caution before we continue. I would argue that probably the safest way out of non random data is to go out and do some sort of experiment to gather random data. I myself don't trust very much on debiasing techniques because you can never know if you've accounted for every confounder. Having said that, orthogonalization is still very much worth learning. It's an incredibly powerful technique that will be the foundation of many causal models to come.

## Linear Regression Reborn

The idea of orthogonalization is based on a theorem designed by three econometricians in 1933, Ragnar Frisch, Frederick V. Waugh, and Michael C. Lovell. Simply put, it states that you can decompose any multivariable linear regression model into three stages or models. Let's say that your features are in an $X$ matrix. Now, you partition that matrix in such a way that you get one part, $X_1$, with some of the features and another part, $X_2$, with the rest of the features.

In the first stage, we take the first set of features and estimate the following linear regression model

$$
y_i = \theta_0 + \pmb{\theta_1 X}_{1i} + e_i
$$

where $\pmb{\theta_1}$ is a vector of parameters. We then take the residuals of that model

$$
y^* = y_i - (\hat{\theta}_0 + \pmb{\hat{\theta}_1 X}_{1i})
$$

On the second stage, we take the first set of features again, but now we run a model where we estimate the second set of features

$$
\pmb{X}_{2i} = \gamma_0 + \pmb{\gamma_1 X}_{1i} + e_i
$$

Here, we are using the first set of features to predict the second set of features. Finally, we also take the residuals for this second stage.

$$
\pmb{X}^*_{2i} = \pmb{X}_{2i} - (\hat{\gamma}_0 + \pmb{\hat{\gamma}_1 X}_{1i})
$$

Lastly, we take the residuals from the first and second stage, and estimate the following model

$$
y_i^* = \beta_0 + \pmb{\beta_2 X}^*_{2i} + e_i
$$

The Frisch–Waugh–Lovell theorem states that the parameter estimate $\pmb{\hat{\beta}_2}$ from estimating this model is equivalent to the one we get by running the full regression, with all the features:

$$
y_i = \beta_0 + \pmb{\beta_1 X}_{1i} + \pmb{\beta_2 X}_{2i} + e_i
$$

OK. Let's unpack this a bit further. We know that regression is a very special model. Each of its parameters has the interpretation of a partial derivative: how much would $Y$ increase if I increase one feature **while holding all the others fixed**. This is very nice for causal inference, because it means we can control for variables in the analysis, even if those same variables have not been held fixed during the collection of the data.

We also know that if we omit variables from the regression, we get bias. Specifically, omitted variable bias (or confounding bias). Still, the Frisch–Waugh–Lovell is saying that I can break my regression model into two parts, neither of them containing the full feature set, and still get the same estimate I would get by running the entire regression. Not only that, this theorem also provides some insight into what linear regression is doing. To get the coefficient of one variable $X_k$, regression first uses all the other variables to predict $X_k$ and takes the residuals. This "cleans"  $X_k$ of any influence from those variables. That way, when we try to understand $X_k$'s impact on $Y$, it will be free from omitted variable bias. Second, regression uses all the other variables to predict $Y$ and takes the residuals. This "cleans" $Y$ from any influence from those variables, reducing the variance of $Y$ so that it is easier to see how $X_k$ impacts $Y$.

I know it can be hard to appreciate how awesome this is. But remember what linear regression is doing. It's estimating the impact of $X_2$ on $y$ while accounting for $X_1$. This is incredibly powerful for causal inference. It says that I can build a model that predicts my treatment $t$ using my features $X$, a model that predicts the outcome $y$ using the same features, take the residuals from both models and run a model that estimates how the residual of $t$ affects the residual of $y$. This last model will tell me how $t$ affects $y$ while controlling for $X$. In other words, the first two models are controlling for the confounding variables. They are generating data which is as good as random. This is debiasing my data. That's what we use in the final model to estimate the elasticity.

There is a (not so complicated) mathematical proof for why that is the case, but I think the intuition behind this theorem is so straightforward we can go directly into it.

## The Intuition Behind Orthogonalization

```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split

import statsmodels.formula.api as smf
import statsmodels.api as sm

from nb21 import cumulative_elast_curve_ci, elast, cumulative_gain_ci
```

Let's take our price data once again. But now, we will only take the sample where prices where **not** randomly assigned. Once again, we separate them into a training and a test set. Since we will use the test set to evaluate our causal model, let's see how we can use orthogonalization to debias it.

```python
prices = pd.read_csv("./data/ice_cream_sales.csv")

train, test = train_test_split(prices, test_size=0.5)
train.shape, test.shape
```

    ((5000, 5), (5000, 5))

If we show the correlations on the test set, we can see that price is positively correlated with sales, meaning that sales should go up as we increase prices. This is obviously nonsense. People don't buy more if ice cream is expensive. We probably have some sort of bias here.

```python
test.corr()
```

      temp
      weekday
      cost
      price
      sales

      temp
      1.000000
      0.003630
      0.006605
      -0.011977
      0.379108

      weekday
      0.003630
      1.000000
      0.011889
      0.002610
      0.004589

      cost
      0.006605
      0.011889
      1.000000
      0.388046
      -0.009410

      price
      -0.011977
      0.002610
      0.388046
      1.000000
      0.080040

      sales
      0.379108
      0.004589
      -0.009410
      0.080040
      1.000000

If we plot our data, we can see why this is happening. Weekends (Saturday and Sunday) have higher price but also higher sales. We can see that this is the case because the weekend cloud of points seems to be to the upper right part of the plot.

Weekend is probably playing an important role in the bias here. On the weekends, there are more ice cream sales because there is more demand. In response to that demand, prices go up. So it is not that the increase in price causes sales to go up. It is just that both sales and prices are high on weekends.

```python
np.random.seed(123)
sns.scatterplot(data=test.sample(1000), x="price", y="sales", hue="weekday");
```

To debias this dataset we will need two models. The first model, let's call it $M_t(X)$, predicts the treatment (price, in our case) using the confounders. It's the one of the stages we've seen above, on the Frisch–Waugh–Lovell theorem.

```python
m_t = smf.ols("price ~ cost + C(weekday) + temp", data=test).fit()
debiased_test = test.assign(**{"price-Mt(X)":test["price"] - m_t.predict(test)})
```

Once we have this model, we will construct the residuals

$$
\hat{t}_i = t_i - M_t(X_i)
$$

You can think of this residual as a version of the treatment that is unbiased or, better yet, that is impossible to predict from the confounders $X$. Since the confounders were already used to predict $t$, the residual is by definition, unpredictable with $X$. Another way of saying this is that the bias has been explained away by the model $M_t(X_i)$, prudicing $\hat{t}_i$ which is as good as randomly assigned. Of course this only works if we have in $X$ all the confounders that cause both $T$ and $Y$.

We can also plot this data to see what it looks like.

```python
np.random.seed(123)
sns.scatterplot(data=debiased_test.sample(1000), x="price-Mt(X)", y="sales", hue="weekday")
plt.vlines(0, debiased_test["sales"].min(), debiased_test["sales"].max(), linestyles='--', color="black");
```

We can see that the weekends are no longer to the upper right corner. They got pushed to the center. Moreover, we can no longer differentiate between different price levels (the treatment) using the weekdays. We can say that the residual $price-M_t(X)$, plotted on the x-axis, is a "random" or debiased version of the original treatment.

This alone is sufficient to debias the dataset. This new treatment we've created is as good as randomly assigned. But we can still do one other thing to make the debiased dataset even better. Namely, we can also construct residuals for the outcome.

$$
\hat{y}_i = y_i - M_y(X_i)
$$

This is another stage from the Frisch–Waugh–Lovell theorem. It doesn't make the set less biased, but it makes it easier to estimate the elasticity by reducing the variance in $y$. Once again, you can think about $\hat{y}_i$ as a version of $y_i$ that is unpredictable from $X$ or that had all its variances due to $X$ explained away. Think about it. We've already used $X$ to predict $y$ with $M_y(X_i)$. And $\hat{y}_i$ is the error of this prediction. So, by definition, it's not possible to predict it from $X$. All the information in $X$ to predict $y$ has already been used. If that is the case, the only thing left to explain $\hat{y}_i$ is something we didn't used to construct it (not included in $X$), which is only the treatment (again, assuming no unmeasured confounders).

```python
m_y = smf.ols("sales ~ cost + C(weekday) + temp", data=test).fit()

debiased_test = test.assign(**{"price-Mt(X)":test["price"] - m_t.predict(test),
                               "sales-My(X)":test["sales"] - m_y.predict(test)})
```

Once we do both transformations, not only does weekdays not predict the price residuals, but it also can't predict the residual of sales $\hat{y}$. The only thing left to predict these residuals is the treatment. Also, notice something interesting. In the plot above, it was hard to know the direction of the price elasticity. It looked like sales decreased as prices went up, but there was such a large variance in sales that it was hard to say that for sure.

Now, when we plot the two residuals, it becomes much clear that sales indeed causes prices to go down.

```python
np.random.seed(123)
sns.scatterplot(data=debiased_test.sample(1000), x="price-Mt(X)", y="sales-My(X)", hue="weekday")
plt.vlines(0, debiased_test["sales-My(X)"].min(), debiased_test["sales-My(X)"].max(), linestyles='--', color="black");
```

One small disadvantage of this debiased data is that the residuals have been shifted to a different scale. As a result, it's hard to interpret what they mean (what is a price residual of -3?). Still, I think this is a small price to pay for the convenience of building random data from data that was not initially random.

To summarize, by predicting the treatment, we've constructed $\hat{t}$ which works as an unbiased version of the treatment; by predicting the outcome, we've constructed $\hat{y}$ which is a version of the outcome that can only be further explained if we use the treatment. This data, where we replace $y$ by $\hat{y}$ and $t$ by $\hat{t}$ is the debiased data we wanted. We can use it to evaluate our causal model just like we deed previously using random data.

To see this, let's once again build a causal model for price elasticity using the training data.

```python
m3 = smf.ols(f"sales ~ price*cost + price*C(weekday) + price*temp", data=train).fit()
```

Then, we'll make elasticity predictions on the debiased test set.

```python
def predict_elast(model, price_df, h=0.01):
    return (model.predict(price_df.assign(price=price_df["price"]+h))
            - model.predict(price_df)) / h

debiased_test_pred = debiased_test.assign(**{
    "m3_pred": predict_elast(m3, debiased_test),
})

debiased_test_pred.head()
```

      temp
      weekday
      cost
      price
      sales
      price-Mt(X)
      sales-My(X)
      m3_pred

      7791
      20.8
      3
      1.5
      6.3
      187
      -0.201769
      1.441373
      -0.073317

      1764
      26.6
      3
      1.5
      6.3
      201
      -0.179506
      4.737748
      -2.139611

      5785
      24.0
      4
      1.0
      5.8
      186
      -0.215107
      -5.855171
      -0.549798

      3542
      20.9
      3
      1.5
      5.1
      180
      -1.401386
      -5.743172
      -0.108943

      9250
      26.7
      5
      1.0
      7.0
      201
      0.978382
      4.384885
      -1.427230

Now, when it comes to plotting the cumulative elasticity, we still order the dataset by the predictive elasticity, but now we use the debiased versions of the treatment and outcome to get this elasticity. This is equivalent to estimating $\beta_1$ in the following regression model

$$
\hat{y}_i = \beta_0 + \beta_1 \hat{t}_i + e_i
$$

where the residuals are like we've described before.

```python
plt.figure(figsize=(10,6))

cumm_elast = cumulative_elast_curve_ci(debiased_test_pred, "m3_pred", "sales-My(X)", "price-Mt(X)", min_periods=50, steps=200)
x = np.array(range(len(cumm_elast)))
plt.plot(x/x.max(), cumm_elast, color="C0")

plt.hlines(elast(debiased_test_pred, "sales-My(X)", "price-Mt(X)"), 0, 1, linestyles="--", color="black", label="Avg. Elast.")
plt.xlabel("% of Top Elast. Customers")
plt.ylabel("Elasticity of Top %")
plt.title("Cumulative Elasticity")
plt.legend();
```

We can do the same thing for the cumulative gain curve, of course.

```python
plt.figure(figsize=(10,6))

cumm_gain = cumulative_gain_ci(debiased_test_pred, "m3_pred", "sales-My(X)", "price-Mt(X)", min_periods=50, steps=200)
x = np.array(range(len(cumm_gain)))
plt.plot(x/x.max(), cumm_gain, color="C1")

plt.plot([0, 1], [0, elast(debiased_test_pred, "sales-My(X)", "price-Mt(X)")], linestyle="--", label="Random Model", color="black")

plt.xlabel("% of Top Elast. Customers")
plt.ylabel("Cumulative Gain")
plt.title("Cumulative Gain on Debiased Sample")
plt.legend();
```

Notice how similar these plots are to the ones in the previous chapter. This is some indication that the debiasing worked wonders here.

In contrast, let's see what the cumulative gain plot would look like if we used the original, biased data.

```python
plt.figure(figsize=(10,6))

cumm_gain = cumulative_gain_ci(debiased_test_pred, "m3_pred", "sales", "price", min_periods=50, steps=200)
x = np.array(range(len(cumm_gain)))
plt.plot(x/x.max(), cumm_gain, color="C1")

plt.plot([0, 1], [0, elast(debiased_test_pred, "sales", "price")], linestyle="--", label="Random Model", color="black")

plt.xlabel("% of Top Elast. Customers")
plt.title("Cumulative Gains on Biased Sample")
plt.ylabel("Cumulative Gains")
plt.legend();
```

First thing you should notice is that the average elasticity goes up, instead of down. We've seen this before. In the biased data, it looks like sales goes up as price increases. As a result, the final point in the cumulative gain plot is positive. This makes little sense, since we now people don't buy more as we increase ice cream prices. If the average price elasticity is already messed up, any ordering in it also makes little sense. The bottom line being that this data should not be used for model evaluation.

## Orthogonalization with Machine Learning

In a 2016 paper, Victor Chernozhukov *et all* showed that you can also do orthogonalization with machine learning models. This is obviously very recent science and we still have much to discover on what we can and can't do with ML models. Still, it's a very interesting idea to know about.

The nuts and bolts are pretty much the same to what we've already covered. The only difference is that now, we use machine learning models for the debiasing.

$$
\begin{align}
\hat{y}_i &= y_i - M_y(X_i) \\
\hat{t}_i &= t_i - M_t(X_i)
\end{align}
$$

There is a catch, though. As we know very well, machine learning models are so powerful that they can fit the data perfectly, or rather, overfit. Just by looking at the equations above, we can know what will happen in that case. If $M_y$ somehow overfitts, the residuals will all be very close to zero. If that happens, it will be hard to find how $t$ affects it. Similarly, if $M_t$ somehow overfitts, its residuals will also be close to zero. Hence, there won't be variation in the treatment residual to see how it can impact the outcome.

To account for that, we need to do sample splitting. That is, we estimate the model with one part of the dataset and we make predictions in the other part. The simplest way to do this is to split the test sample in half, make two models  in such a way that each one is estimated in one half of the dataset and makes predictions in the other half.

A slightly more elegant implementation uses K-fold cross validation. The advantage being that we can train all the models on a sample which is bigger than half the test set.

Fortunately, this sort of cross prediction is very easy to implement using Sklearn's `cross_val_predict` function.

```python
from sklearn.model_selection import cross_val_predict
from sklearn.ensemble import RandomForestRegressor

X = ["cost", "weekday", "temp"]
t = "price"
y = "sales"

folds = 5

np.random.seed(123)
m_t = RandomForestRegressor(n_estimators=100)
t_res = test[t] - cross_val_predict(m_t, test[X], test[t], cv=folds)

m_y = RandomForestRegressor(n_estimators=100)
y_res = test[y] - cross_val_predict(m_y, test[X], test[y], cv=folds)
```

Now that we have the residuals, let's store them as columns on a new dataset.

```python
ml_debiased_test = test.assign(**{
    "sales-ML_y(X)": y_res,
    "price-ML_t(X)": t_res,
})
ml_debiased_test.head()
```

      temp
      weekday
      cost
      price
      sales
      sales-ML_y(X)
      price-ML_t(X)

      7791
      20.8
      3
      1.5
      6.3
      187
      -3.150833
      -0.869267

      1764
      26.6
      3
      1.5
      6.3
      201
      -0.418857
      -0.192867

      5785
      24.0
      4
      1.0
      5.8
      186
      -2.515667
      0.790429

      3542
      20.9
      3
      1.5
      5.1
      180
      -11.718500
      -1.280460

      9250
      26.7
      5
      1.0
      7.0
      201
      -1.214167
      1.715117

Finally, we can plot the debiased dataset.

```python
np.random.seed(123)
sns.scatterplot(data=ml_debiased_test.sample(1000),
                x="price-ML_t(X)", y="sales-ML_y(X)", hue="weekday");
```

Once again, we've uncovered a negative price elasticity on sales. Actually, the plot is incredibly similar to the one we've got when using simple linear regression. But that's probably because this is a very simple dataset. The advantages of machine learning orthogonalization is that it can estimate more complicated functions. It can learn interactions and non linearities in a way that it's hard to encode into linear regression. Also, there is the advantage that some machine learning models (those bases on decision trees) are much simpler to run than linear regression. They can handle categorical data, outliers and even missing data, stuff that would require some attention if you are just using linear regression.

Finally, before we close, I just need to cover one final common mistake that data scientists often make when they are introduced to this idea (been there, done that). If the treatment or the outcome is binary, one might think it is better to replace the machine learning regression models for their classification versions. However, this does not work. The theory of orthogonalization only functions under regression models, similarly with what we've seen a long time ago when talking about Instrumental Variables. To be honest, it is not that the model will fail miserably if you replace regression by classification, but I would advise against it. If the theory doesn't justify it, why run the risk?

## Key Ideas

We've started the chapter by highlighting the necessity of random treatment assignment in order for our causal evaluation methods to work. This poses a problem in the case where random data is not available. To be clear, the safest solution in this case is to go and do some experiments in order to get random data. If that is out of the question, only then, we can rely on a clever alternative: transform our data to look as if the treatment has been randomly assigned.

Here, we've covered how to do that using the principles of orthogonalization. First, we've built a model that uses our features $X$ to predict the treatment $t$ and get it's residuals. The idea being that the treatment residuals is, by definition, independent of the features used to construct it. In other words, the treatment residuals are orthogonal to the features. We can see these residuals as a version of the treatment where all the confounding bias due to $X$ has been removed.

That alone is enough to make our data look as good as random. But we can go one step further. We can build a model that predicts the outcome $y$ using the features $X$ but not the treatment and also get its residuals. Again, the intuition is very similar. These outcome residuals is a version of the outcome where all the variance due to the features has been explained away. That will hopefully explay a lot of the variance, making it easier to see the treatment effect.

Here we are using orthogonalization with the goal of debiasing our data for model evaluation. However, this technique is also used for other purposes. Namely, lot's of causal inference models use orthogonalization as a first pre-processing step to ease the task of the causal inference model. We can say that orthogonalization makes the foundation of many modern causal inference algorithms.

## References

The things I've written here are mostly stuff from my head. I've learned them through experience. This means that they have **not** passed the academic scrutiny that good science often goes through. Instead, notice how I'm talking about things that work in practice, but I don't spend too much time explaining why that is the case. It's a sort of science from the streets, if you will. However, I am putting this up for public scrutiny, so, by all means, if you find something preposterous, open an issue and I'll address it to the best of my efforts.

This chapter is based on Victor Chernozhukov *et all* (2016), Double/Debiased Machine Learning for Treatment and Causal Parameters. You can also check Frisch, Ragnar; Waugh, Frederick V. (1933) original article, Partial Time Regressions as Compared with Individual Trends.

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers).

```python

```


</a>

# Modulo II: Vectores Palabra (Word Embeddings) y CBOW 02

Veremos cómo preparar los datos para aplicar:

- Propagación hacia adelante (Forward propagation).

- Pérdida de entropía cruzada (crosss-entropy loss).

- Retropropagación (Backpropagation).

- Descenso de gradiente (gradient descent).

```python
import numpy as np
```

## Forward propagation

<div style="width:image width px; font-size:100%; text-align:center;"> Figure 2 </div>

```python
N = 3
V = 5

# Inicializando los pesos de la red

```

```python

```

### Inicialización de pesos y bías

```python
W1 = np.array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
               [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
               [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])

W2 = np.array([[-0.22182064, -0.43008631,  0.13310965],
               [ 0.08476603,  0.08123194,  0.1772054 ],
               [ 0.1871551 , -0.06107263, -0.1790735 ],
               [ 0.07055222, -0.02015138,  0.36107434],
               [ 0.33480474, -0.39423389, -0.43959196]])

b1 = np.array([[ 0.09688219],
               [ 0.29239497],
               [-0.27364426]])

b2 = np.array([[ 0.0352008 ],
               [-0.36393384],
               [-0.12775555],
               [-0.34802326],
               [-0.07017815]])
```

Agregar las funciones vistas en los notebooks pasados

```python
def get_dict(data):
    words = sorted(list(set(data)))
    n = len(words)
    idx = 0
    # return these correctly
    word2Ind = {}
    Ind2word = {}
    for k in words:
        word2Ind[k] = idx
        Ind2word[idx] = k
        idx += 1
    return word2Ind, Ind2word

def get_windows(words, C):
    i = C
    while i < len(words) - C:
        center_word = words[i]
        context_words = words[(i - C):i] + words[(i+1):(i+C+1)]
        yield context_words, center_word
        i += 1

def word_to_one_hot_vector(word, word2Ind, V):
    one_hot_vector = np.zeros(V)
    one_hot_vector[word2Ind[word]] = 1
    return one_hot_vector

def context_words_to_vector(context_words, word2Ind, V):
    context_words_vectors = [word_to_one_hot_vector(w, word2Ind, V) for w in context_words]
    context_words_vectors = np.mean(context_words_vectors, axis=0)
    return context_words_vectors

def get_training_example(words, C, word2Ind, V):
    for context_words, center_word in get_windows(words, C):
        yield context_words_to_vector(context_words, word2Ind, V), word_to_one_hot_vector(center_word, word2Ind, V)

```

```python
words = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning']
```

```python

word2Ind, Ind2word = get_dict(words)

```

```python
Ind2word
```

    {0: 'am', 1: 'because', 2: 'happy', 3: 'i', 4: 'learning'}

```python
word2Ind
```

    {'am': 0, 'because': 1, 'happy': 2, 'i': 3, 'learning': 4}

## Datos de entrenamiento

```python
training_examples = get_training_example(words, 2, word2Ind, V)
```

```python
training_examples
```

    <generator object get_training_example at 0x7fc9f562b9e0>

```python
x_array, y_array = next(training_examples)
```

```python
x_array
```

    array([0.25, 0.25, 0.  , 0.5 , 0.  ])

```python
y_array
```

    array([0., 0., 1., 0., 0.])

```python
x = x_array.copy()
```

```python
x.reshape(V,1)
```

    array([[0.25],
           [0.25],
           [0.  ],
           [0.5 ],
           [0.  ]])

```python
x.shape=(V,1)
x
```

    array([[0.25],
           [0.25],
           [0.  ],
           [0.5 ],
           [0.  ]])

```python
y = y_array.copy()
```

```python
y.shape = (V,1)
```

```python
y
```

    array([[0.],
           [0.],
           [1.],
           [0.],
           [0.]])

```python
def relu(z):
    result = z.copy()
    result[result<0]=0
    return result

def softmax(z):
    e_z= np.exp(z)
    sum_ez = np.sum(e_z)
    return e_z / sum_ez
```

## Forward

### Valores de la capa oculta

\begin{align}
\mathbf{z_1} = \mathbf{W_1}\mathbf{x} + \mathbf{b1} \\
\mathbf{h} = \mathbf{ReLu}(\mathbf{z_1)} \\
\end{align}

```python
z1 = np.dot(W1, x) + b1
```

```python
z1
```

    array([[ 0.36483875],
           [ 0.63710329],
           [-0.3236647 ]])

```python
h = relu(z1)
h
```

    array([[0.36483875],
           [0.63710329],
           [0.        ]])

### Valores de la capa de salida

\begin{align}
\mathbf{z_2} = \mathbf{W_2}\mathbf{h} + \mathbf{b2} \\
\mathbf{\hat{y}} = \mathbf{softmax}(\mathbf{z_2)} \\
\end{align}

```python
z2 = np.dot(W2, h) + b2
```

```python
y_hat = softmax(z2)
```

```python
y_hat
```

    array([[0.18519074],
           [0.19245626],
           [0.23107446],
           [0.18236353],
           [0.20891502]])

```python

```

    2

```python
y
```

    array([[0.],
           [0.],
           [1.],
           [0.],
           [0.]])

```python
Ind2word[y_hat.argmax()]
```

    'happy'

### Cross-entropy loss

$$ J = -\sum\limits_{k=1}^{V} y_k \log{\hat{y}_k}$$

```python
def cross_entropy_loss(y_predicted, y_actual):
    loss = np.sum(-np.log(y_predicted)*y_actual)
    return loss
```

```python
cross_entropy_loss(y_hat, y)
```

    1.4650152923611106

```python

```

### Backpropagation

Las formulas que necesitamos para implementar el backpropagation son:

\begin{align}
 \frac{\partial J}{\partial \mathbf{W_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right )\mathbf{x}^\top \tag{7}\\
 \frac{\partial J}{\partial \mathbf{W_2}} &= (\mathbf{\hat{y}} - \mathbf{y})\mathbf{h^\top} \tag{8}\\
 \frac{\partial J}{\partial \mathbf{b_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right ) \tag{9}\\
 \frac{\partial J}{\partial \mathbf{b_2}} &= \mathbf{\hat{y}} - \mathbf{y} \tag{10}
\end{align}

Calcule la derivada parcial de la función de pérdida con respecto a $ \mathbf {b_2} $ y almacene el resultado en `grad_b2`.

$$\frac{\partial J}{\partial \mathbf{b_2}} = \mathbf{\hat{y}} - \mathbf{y} \tag{10}$$

```python
grad_b2 = y_hat - y
```

```python
grad_b2
```

    array([[ 0.18519074],
           [ 0.19245626],
           [-0.76892554],
           [ 0.18236353],
           [ 0.20891502]])

Calcular la derivada parcial de la función con respecto a $ \mathbf {w_2} $, y guardarlo en `grad_W2`

$$\frac{\partial J}{\partial \mathbf{W_2}} = (\mathbf{\hat{y}} - \mathbf{y})\mathbf{h^\top} \tag{8}$$

```python
grad_w2 = np.dot((y_hat - y), h.T)
grad_w2
```

    array([[ 0.06756476,  0.11798563,  0.        ],
           [ 0.0702155 ,  0.12261452,  0.        ],
           [-0.28053384, -0.48988499, -0.        ],
           [ 0.06653328,  0.1161844 ,  0.        ],
           [ 0.07622029,  0.13310045,  0.        ]])

**Ahora, calcule la derivada con respecto a $\mathbf{b_1}$ y guardar el resultado en `grad_b1`.**

$$\frac{\partial J}{\partial \mathbf{b_1}} = \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right ) \tag{9}$$

```python
grad_b1 = relu(np.dot(W2.T, (y_hat - y)))
grad_b1

```

    array([[0.        ],
           [0.        ],
           [0.17045858]])

**Finalmente, calcular la derivada parcial del loss con respecto a $\mathbf{W_1}$, y guardarlo en`grad_W1`.**

$$\frac{\partial J}{\partial \mathbf{W_1}} = \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right )\mathbf{x}^\top \tag{7}$$

```python
grad_w1 = np.dot(relu(np.dot(W2.T,grab_b2)),x.T)
grad_w1

```

    array([[0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.04261464, 0.04261464, 0.        , 0.08522929, 0.        ]])

Resultado esperado

    array([[0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.04261464, 0.04261464, 0.        , 0.08522929, 0.        ]])

## Gradiante descendente

Durante la fase del gradiante descendente, actualizará los pesos y los bías $ \alpha $ veces el gradiente de las matrices y vectores originales, utilizando las siguientes fórmulas.

\begin{align}
 \mathbf{W_1} &:= \mathbf{W_1} - \alpha \frac{\partial J}{\partial \mathbf{W_1}} \tag{11}\\
 \mathbf{W_2} &:= \mathbf{W_2} - \alpha \frac{\partial J}{\partial \mathbf{W_2}} \tag{12}\\
 \mathbf{b_1} &:= \mathbf{b_1} - \alpha \frac{\partial J}{\partial \mathbf{b_1}} \tag{13}\\
 \mathbf{b_2} &:= \mathbf{b_2} - \alpha \frac{\partial J}{\partial \mathbf{b_2}} \tag{14}\\
\end{align}

```python
alpha = 0.03

```

```python

```

**Ahora Calcule los nuevo valores de $\mathbf{W_2}$ (que serán guardados en `W2_new`), $\mathbf{b_1}$ (en `b1_new`), y $\mathbf{b_2}$ (en `b2_new`).**

\begin{align}
 \mathbf{W_2} &:= \mathbf{W_2} - \alpha \frac{\partial J}{\partial \mathbf{W_2}} \tag{12}\\
 \mathbf{b_1} &:= \mathbf{b_1} - \alpha \frac{\partial J}{\partial \mathbf{b_1}} \tag{13}\\
 \mathbf{b_2} &:= \mathbf{b_2} - \alpha \frac{\partial J}{\partial \mathbf{b_2}} \tag{14}\\
\end{align}

```python
# Actualizacion de pesos
w1_new = W1 - alpha*grad_w1
w2_new = W2 - alpha*grad_w2
b1_new = b1 - alpha*grad_b1
b2_new = b2 - alpha*grad_b2

```

```python
W1
```

    array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
           [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
           [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])

```python
w1_new
```

    array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
           [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
           [ 0.25359163, -0.25125325, -0.37770863, -0.13956325,  0.34008124]])

```python

```

## Opción 1: extraer los embeddings de W1

```python
w1_new
```

    array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
           [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
           [ 0.25359163, -0.25125325, -0.37770863, -0.13956325,  0.34008124]])

```python
for i in range(V):
    print(Ind2word[i])
```

    am
    because
    happy
    i
    learning

```python
word2Ind
```

    {'am': 0, 'because': 1, 'happy': 2, 'i': 3, 'learning': 4}

```python
print(f'{x} ----> {w1_new[]}')
```

## Opción 2: extraer los embeddings de W2

```python
w2_new
```

    array([[-0.24209007, -0.465482  ,  0.13310965],
           [ 0.06370138,  0.04444758,  0.1772054 ],
           [ 0.27131525,  0.08589287, -0.1790735 ],
           [ 0.05059224, -0.0550067 ,  0.36107434],
           [ 0.31193865, -0.43416402, -0.43959196]])

```python

```

## Opción 3: extraer los embeddings de W1 y W2

```python
W3 = (W1 + W2)/2

```

```python

```

```python
def gradient_descent(xtrain,ytrain, N, V, numiter, alpha):

    #inicicalizar el modelo
    W1,W2,b1,b2

    for i in range(numiter):

        z,h,yhat = forward(xtrain,ytrain,w1,w2,b1,b2)

        cost = funcion_costo(ytrain, yhat)

        grad_w1,grad_w2 ... = back_prop

        # actualizar los pesos
        W1 = W1 - alpha*grad_w1
        W2 = W2 - alpha*grad_w2
        b1 = b1 - alpha*grad_b1
        b2 = b2 - alpha*grad_b2

```


```python
import sympy as sm
import numpy as np
import matplotlib.pyplot as plt
from pyodesys.symbolic import SymbolicSys
sm.init_printing()
%matplotlib inline
```

```python
def mk_odesys(power=1):
    def rhs(t, y, p):
        k = t**power
        return [-k*y[0], k*y[0]]

    def analytic(tout, init_y):
        y0ref = init_y[0]*np.exp(-tout**(power+1)/(power+1))
        return np.array([y0ref, init_y[0] - y0ref + init_y[1]]).T

    odesys = SymbolicSys.from_callback(rhs, 2, 0, linear_invariants=[[1, 1]])
    return odesys, analytic
```

```python
odesys1, analytic1 = mk_odesys(1)
odesys1.dep, odesys1.exprs, odesys1.get_jac()
```

```python
def integrate_and_plot(odesys, analytic, axes=None, tout=3, init_y=(1,0)):
    result = odesys.integrate(tout, init_y, integrator='cvode')
    if axes is None:
        _fig, axes = plt.subplots(1, 3, figsize=(14, 4))
    result.plot(ax=axes[0])
    result.plot(ax=axes[1], y=result.yout - analytic(result.xout, result.yout[0, :]))
    result.plot_invariant_violations(ax=axes[2])
    print({k: v for k, v in result.info.items() if not k.startswith('internal')})
```

```python
integrate_and_plot(odesys1, analytic1)
```

```python
asys1 = odesys1.as_autonomous()
asys1.dep, asys1.exprs, asys1.get_jac()
```

```python
integrate_and_plot(asys1, analytic1)
```

```python
def compare(power, **kwargs):
    odesys, analytic = mk_odesys(power)
    autsys = odesys.as_autonomous()
    fig, axes = plt.subplots(2, 3, figsize=(14, 8))
    integrate_and_plot(odesys, analytic, axes=axes[0], **kwargs)
    integrate_and_plot(autsys, analytic, axes=axes[1], **kwargs)
```

```python
compare(2)
```

```python
compare(3, init_y=(3,1))
```

```python
compare(10, tout=1.5)
```


# 1. Numbers

##  Creating formulas

Write the following mathematical formula in Python:

\begin{align}
 result = 6a^3 - \frac{8b^2 }{4c} + 11
\end{align}

```python
a = 2
b = 3
c = 2
```

> write your formula as
```python
result =
```

It must verify
```python
>>> assert result == 50
```

## Floating point pitfalls

Show that `0.1 + 0.2 == 0.3`

# 2. Lists

## Fill the missing pieces

Fill the `____` parts of the code below.

```python
# Let's create an empty list
my_list = ____

# Let's add some values
my_list.____('Python')
my_list.____('is ok')
my_list.____('sometimes')

# Let's remove 'sometimes'
my_list.____('sometimes')

# Let's change the second item
my_list[____] = 'is neat'
```

Let's verify that it's correct
```python
>>> assert my_list == ['Python', 'is neat']
```

## Create a new list without modifiying the original one

```python
original = ['I', 'am', 'learning', 'hacking', 'in']
```

> Your implementation here
```python
modified =
```

```python
>>> assert original == ['I', 'am', 'learning', 'hacking', 'in']
>>> assert modified == ['I', 'am', 'learning', 'lists', 'in', 'Python']
```

## Create a merged sorted list

```python
list1 = [6, 12, 5]
list2 = [6.2, 0, 14, 1]
list3 = [0.9]
```

> Your implementation here
```python
my_list =
```

```python
>>> assert my_list == [14, 12, 6.2, 6, 5, 1, 0.9, 0]
```

# 3. Dictionaries

## Populating a dictionary

Create a dictionary by using all the given variables.

```python
first_name = 'John'
last_name = 'Doe'
favorite_hobby = 'Python'
sports_hobby = 'gym'
age = 82
```

> Your implementation
```python
my_dict =
```

```python
>>> assert my_dict == {
        'name': 'John Doe',
        'age': 82,
        'hobbies': ['Python', 'gym']
    }
```

## Accessing and merging dictionaries

Combine `dict1`, `dict2`, and `dict3` into `my_dict`. In addition, get the value of `special_key` from `my_dict` into a `special_value` variable. Note that original dictionaries should stay untouched and `special_key` should be removed from `my_dict`.

```python
dict1 = dict(key1='This is not that hard', key2='Python is still cool')
dict2 = {'key1': 123, 'special_key': 'secret'}
# This is also a away to initialize a dict (list of tuples)
dict3 = dict([('key2', 456), ('keyX', 'X')])
```

> Your impelementation
```python
my_dict =
special_value =
```

```python
>>> assert my_dict == {'key1': 123, 'key2': 456, 'keyX': 'X'}
>>> assert special_value == 'secret'

>>> # Let's check that the originals are untouched
>>> assert dict1 == {
        'key1': 'This is not that hard',
        'key2': 'Python is still cool'
    }
>>> assert dict2 == {'key1': 123, 'special_key': 'secret'}
>>> assert dict3 == {'key2': 456, 'keyX': 'X'}
```

# 4. Functions

## Fill the missing pieces of the `count_even_numbers` function

Fill `____` pieces of the `count_even_numbers` implemention in order to pass the assertions. You can assume that `numbers` argument is a list of integers.

```python
____ count_even_numbers(numbers):
    count = 0
    for num in ____:
        if ____ % 2 == ____:
            count += ____
    _____ _____
```

```python
>>> assert count_even_numbers([1, 2, 3, 4, 5, 6]) == 3
>>> assert count_even_numbers([1, 3, 5, 7]) == 0
>>> assert count_even_numbers([-2, 2, -10, 8]) == 4
```

## Searching for wanted people

Implement `find_wanted_people` function which takes a list of names (strings) as argument. The function should return a list of names which are present both in `WANTED_PEOPLE` and in the name list given as argument to the function.

```python
WANTED_PEOPLE = ['John Doe', 'Clint Eastwood', 'Chuck Norris']
```

> Your implementation here

```python
>>> people_to_check1 = ['Donald Duck', 'Clint Eastwood', 'John Doe', 'Barack Obama']
>>> wanted1 = find_wanted_people(people_to_check1)
>>> assert len(wanted1) == 2
>>> assert 'John Doe' in wanted1
>>> assert 'Clint Eastwood'in wanted1

>>> people_to_check2 = ['Donald Duck', 'Mickey Mouse', 'Zorro', 'Superman', 'Robin Hood']
>>> wanted2 = find_wanted_people(people_to_check2)
>>> assert wanted2 == []
```

## Counting average length of words in a sentence

Create a function `average_length_of_words` which takes a string as an argument and returns the average length of the words in the string. You can assume that there is a single space between each word and that the input does not have punctuation. The result should be rounded to one decimal place (hint: see [`round`](https://docs.python.org/3/library/functions.html#round)).

> Your implementation here

```python
>>> assert average_length_of_words('only four lett erwo rdss') == 4
>>> assert average_length_of_words('one two three') == 3.7
>>> assert average_length_of_words('one two three four') == 3.8
>>> assert average_length_of_words('') == 0
```

# 5. Strings

## Fill missing pieces

Fill `____` pieces below to have correct values for `lower_cased`, `stripped` and `stripped_lower_case` variables.

```python
original = ' Python strings are COOL! '
lower_cased = original._____
stripped = ____.strip()
stripped_lower_cased = original._____._____
```

Let's verify that the implementation is correct by running the cell below. `assert` will raise `AssertionError` if the statement is not true.

```python
>>> assert lower_cased == ' python strings are cool! '
>>> assert stripped == 'Python strings are COOL!'
>>> assert stripped_lower_cased == 'python strings are cool!'
```

## Prettify ugly string

Use `str` methods to convert `ugly` to wanted `pretty`.

```python
ugly = ' tiTle of MY new Book\n\n'
```

> Your implementation:
```python
pretty = 'TODO'
```

Let's make sure that it does what we want. `assert` raises [`AssertionError`](https://docs.python.org/3/library/exceptions.html#AssertionError) if the statement is not `True`.

```python
>>> print('pretty: {}'.format(pretty))
>>> assert pretty == 'Title Of My New Book'
```

## Format string based on existing variables

Create `sentence` by using `verb`, `language`, and `punctuation` and any other strings you may need.

```python
verb = 'is'
language = 'Python'
punctuation = '!'
```

> Your implementation:
```python
sentence = 'TODO'
```

```python
>>> print('sentence: {}'.format(sentence))
>>> assert sentence == 'Learning Python is fun!'
```

# 6. For Loops

## Fill the missing pieces

Fill the `____` parts in the code below.

```python
words = ['PYTHON', 'JOHN', 'chEEse', 'hAm', 'DOE', '123']
upper_case_words = []

for ____ in words:
    if ____.isupper():
        ____.append(____)

>>> assert upper_case_words == ['PYTHON', 'JOHN', 'DOE']
```

## Calculate the sum of dict values

Calculate the sum of the values in `magic_dict` by taking only into account numeric values (hint: see [isinstance](https://docs.python.org/3/library/functions.html#isinstance)).

```python
magic_dict = dict(val1=44, val2='secret value', val3=55.0, val4=1)
```

> Your implementation
```python
sum_of_values =
```

```python
>>> assert sum_of_values == 100
```

## Create a list of strings based on a list of numbers

The rules:
* If the number is a multiple of five and odd, the string should be `'five odd'`
* If the number is a multiple of five and even, the string should be `'five even'`
* If the number is odd, the string is `'odd'`
* If the number is even, the string is `'even'`

```python
numbers = [1, 3, 4, 6, 81, 80, 100, 95]
```

> Your implementation
```python
my_list =
```

```python
>>> assert my_list == ['odd', 'odd', 'even', 'even', 'odd', 'five even', 'five even', 'five odd']
```

# 7. Conditionals

## `if-elif-else`

Fill missing pieces (`____`) of the following code such that prints make sense.

```python
name = 'John Doe'

if ____:
    print('Name "{}" is more than 20 chars long'.format(name))
    length_description = 'long'
elif ____:
    print('Name "{}" is more than 15 chars long'.format(name))
    length_description = 'semi long'
elif ____:
    print('Name "{}" is more than 10 chars long'.format(name))
    length_description = 'semi long'
elif ____:
    print('Name "{}" is 8, 9 or 10 chars long'.format(name))
    length_description = 'semi short'
else:
    print('Name "{}" is a short name'.format(name))
    length_description = 'short'

>>> assert length_description == 'semi short'
```

# 8. Classes

## Fill the missing pieces of the `Calculator` class

Fill `____` pieces of the `Calculator` implemention in order to pass the assertions.

```python
class Calculator:
    def __init__(self, var1, ____):
        self.____ = var1
        self.____ = _____

    def calculate_power(self):
        return self.____ ** ____.____

    def calculate_sum(____, var3):
        return ____.____ + ____.____ + var3
```

```python
>>> calc = Calculator(2, 3)
>>> assert calc.calculate_power() == 8
>>> assert calc.calculate_sum(4) == 9
```

## Finalize `StringManipulator` class

Fill `____` pieces and create implementation for `stripped_title()`.

```python
class StringManipulator:
    """____"""

    category = ____

    def __init__(self, original):
        self.string = ____

    def reverse_words(self):
        words = self.string.____
        self.string = ' '.join(reversed(____))

    def make_title(self):
        # Create implementation for this

    def get_manipulated(____):
        return self._____
```

```python
>>> assert StringManipulator.__doc__ == 'Docstring of StringManipulator'
>>> assert StringManipulator.category == 'Manipulator'

>>> str_manip = StringManipulator('cOOL pyThON')

>>> str_manip.reverse_words()
>>> assert str_manip.get_manipulated() == 'pyThON cOOL'

>>> str_manip.make_title()
>>> assert str_manip.get_manipulated() == 'Python Cool'
```

## Create `Dog` class

Create `Dog` class which has the following specification:

* Dogs consume their energy by barking and gain energy by sleeping
* A fresh `Dog` instance has 10 units of energy
* `Dog` has a method `sleep` which gives 2 units of energy
* `Dog` has a method `bark` which consumes 1 unit of energy
* `Dog` has a method `get_energy` which returns the amount of energy left

```python
class Dog:
    # Your implementation here
```

```python
>>> doge = Dog()
>>> assert doge.get_energy() == 10

>>> doge.bark()
>>> doge.bark()
>>> doge.bark()
>>> assert doge.get_energy() == 7

>>> doge.sleep()
>>> assert doge.get_energy() == 9

>>> another_doge = Dog()
>>> assert another_doge.get_energy() == 10
```

# 9. Exceptions

# 10. File IO

# 11. Standard Library

## Collections

### NamedTuple

#### Creating a namedtuple

Create a namedtuple `Car` which has fields `price`, `mileage`, and `brand`.

> Your implemenation here

```python
>>> car1 = Car(25000, 2000, 'BMW')
>>> assert car1.price == 25000
>>> assert car1.mileage == 2000
>>> assert car1.brand == 'BMW'
>>> assert isinstance(car1, tuple)

>>> # Note that indexing works also!
>>> # This means that if you change a tuple into a namedtuple,
>>> # the change will be backwards compatible.
>>> assert car1[2] == 'BMW'

>>> print('All good!')
```

### urllib

#### example 1

Below you can see `get_wiki_article` function which is a very simple implementation for fetching an article from wikipedia. Your task is to mock it's implementation such that it's going to always return `'Python is cool!'`. However, note that you should be able to check which argument is given to `urlopen` when `get_wiki_article` is called.

**Note**: `get_content_of_url` uses [`urrlib`](https://docs.python.org/3/library/urllib.html#module-urllib), which is part of the Standard Library, for creating a HTTP request. Usually it's preferable to use [`requests`](http://docs.python-requests.org/en/master/) library (not part of the Standard Library) for such operations. Actually, `requests` uses `urllib` under the hood so it's good to know what's happening when you start using `requests` - or maybe you have already used it.

```python
from urllib.request import urlopen

def get_wiki_article(name):
    url = 'https://en.wikipedia.org/wiki/{}'.format(name)
    response = urlopen(url)
    content = str(response.read())
    return content
```

> Your implementation here

```python
>>> article = 'Python_(programming_language)'
>>> res = get_wiki_article(article)
>>> assert 'Guido van Rossum' not in res, 'Guido is still there!'
>>> assert res == 'Python is cool!'
>>> urlopen.assert_called_with('https://en.wikipedia.org/wiki/Python_(programming_language)')

>>> print('All good!')
```

```python
# css style
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


# One Step Methods
Given the first order equation
\begin{equation}
y^{'} = 2y-0.1y^2,
\end{equation}
with the initial condition
\begin{equation}
y(0) = 1.
\end{equation}

Aproximate the solution of the inital value problem for the interval $0 \leq t \leq 4$ using $N=8$.
Using the:
1. Euler method:
$$ w_{i+1}=w_i+hf(t_i,w_i);$$

2. Midpoint method:
$$ w_{i+1}=w_i+hf(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
where $k1=f(t_i,w_i)$;

3. 4th Order Runge Kutta method:
$$ w_{i+1}=w_i+\frac{h}{6}(k1+2k2+2k3+k4),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
$$k3=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k2),$$
$$k4=f(t_i+h,w_i+hk3);$$

4. Imposter-Butler method:
$$ w_{i+1}=w_i+\frac{h}{4}(k1+2k2),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+h,w_i+hk1).$$

4. Lepa-frog method:
$$ w_{i+1}=w_{i-1}+\frac{h}{4}(k1+2k2),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+h,w_i+hk1).$$
What is wrong with the Imposter-Butler method?

### DECLARING LIBRARIES

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

### Outputting Table

```python
class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)
```

## Setting up the discrete time axis
$$ h=\frac{4-0}{N}=0.5 $$
$$t_i=t_0+ih$$
$$t_i=0+0.5i$$
for $i=0,1,2,...,8.$

```python
N=8
x_end=4.0
h=((x_end-0)/N)

INITIALCONDITION=1
time=np.zeros(N+1)

table = ListTable()
table.append(['time', 'Euler', 'Mid', 'RK','Imposter'])

```

### Defining the function
From the initial value problem the function
$$ f(t,y)= 2y-0.1y^2. $$

```python
def myfun(w):
    return 2*w-0.1*w*w
```

## NUMERICAL SOLUTION

```python
time[0]=0
```

## Euler Method
$$w_0=1$$
$$ w_{i+1}=w_i+h(2w_i-0.1w_i^2)$$
$$N=0,1,2,...7$$

```python
Euler=np.zeros(N+1)
Euler[0]=INITIALCONDITION

for i in range (0,N):
    # Euler Method
    Euler[i+1]=Euler[i]+h*myfun(Euler[i])
```

## Midpoint Method
$$w_0=1$$
$$k=2w_i-0.1w_i^2$$
$$ w_{i+1}=w_i+h(2(w_i+\frac{h}{2}k)-0.1(w_i+\frac{h}{2}k)^2)$$
$$N=0,1,2,...7$$

```python
Midpoint=np.zeros(N+1)
Midpoint[0]=INITIALCONDITION

for i in range (0,N):
    # Mid-point method
    Midpoint[i+1]=Midpoint[i]+h*myfun(Midpoint[i]+h/2*myfun(Midpoint[i]))
```

## 4th Order Runge Kutta Method
$$w_0=1$$
$$k1=2w_i-0.1w_i^2$$
$$k2=2(w_i+\frac{h}{2}k1)-0.1(w_i+\frac{h}{2}k1)^2$$
$$k3=2(w_i+\frac{h}{2}k2)-0.1(w_i+\frac{h}{2}k2)^2$$
$$k4=2(w_i+hk)-0.1(w_i+hk)^2$$
$$ w_{i+1}=w_i+\frac{h}{6}(k1+2k2+2k3+k4)$$
$$N=0,1,2,...7$$

```python
RK4=np.zeros(N+1)
RK4[0]=INITIALCONDITION

for i in range (0,N):
    # 4th Order Runge Kutta
    k1=myfun(RK4[i])
    k2=myfun(RK4[i]+h/2*k1)
    k3=myfun(RK4[i]+h/2*k2)
    k4=myfun(RK4[i]+h*k3)
    RK4[i+1]=RK4[i]+h/6*(k1+2*k2+2*k3+k4)

```

## Imposter Butler Method
$$w_0=1$$
$$k1=2w_i-0.1w_i^2$$
$$k2=2(w_i+hk1)-0.1(w_i+hk1)^2$$
$$ w_{i+1}=w_i+\frac{h}{4}(k1+2k2)$$
$$N=0,1,2,...7$$

```python
Imposter=np.zeros(N+1)
Imposter[0]=INITIALCONDITION
i=0
table.append([time[i], Euler[i], Midpoint[i], RK4[i],Imposter[i]])
for i in range (0,N):
    #Imposter method
    k1=myfun(Imposter[i])
    k2=myfun(Imposter[i]+h*k3)
    Imposter[i+1]=Imposter[i]+h/4*(k1+2*k2)
    time[i+1]=time[i]+h
    table.append([time[i+1], Euler[i+1], Midpoint[i+1], RK4[i+1],Imposter[i+1]])
```

## Plotting

```python
fig = plt.figure(figsize=(8,4))
plt.plot(time,Euler,'o:',color='red',label='Euler')
plt.plot(time,Midpoint,'o:',color='blue',label='Midpoint')
plt.plot(time,RK4,'o:',color='green',label='RK4')
plt.plot(time,Imposter,'o:',color='yellow',label='Imposter')
plt.legend(loc=4)
plt.title('Numerical Solutions of initial value problem.')
plt.show()
```

```python
table
```

time	Euler	Mid	RK	Imposter

0.0	1.0	1.0	1.0	1.0

0.5	1.95	2.36621875	2.4973140745034703	1.828354103622306

1.0	3.7098750000000003	5.19438758389418	5.581882795387722	3.1792956569283346

1.5	6.73159137421875	9.77881461292582	10.250670627504523	5.27210794647557

2.0	11.197466626964687	14.519443452104497	14.812884418161534	8.243922547632055

2.5	16.125770310829978	17.40121357626255	17.70715620549379	11.899511859117052

3.0	19.24951721577768	18.761663288245703	19.080300940495654	15.487688268593597

3.5	19.971838779529293	19.397552928361538	19.6459932243498	18.03499834285824

4.0	19.99996034728308	19.703035811592716	19.865924950313506	19.25729194968363

```python

```

Chapter 05 - IVP Consistent Convergence Stability/.ipynb_checkpoints/05_One Step Methods Population growth-checkpoint.ipynb


13.

Computer Lab – Chemical Kinetics 1

Goal: The purpose of this lab is to become familiar with modeling a simple ligand binding reaction.

Introduction

Consider a binding reaction where ligand ($L$) binds to a protein receptor ($R$) to form a loosely bound encounter complex ($L\cdot R$), and finally a tightly bound product ($P$) with the forward and backward rate constants $k_{f}$ and $k_{b}$, respectively.1

$$L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P \tag{1}$$

First, we need to write out the elementary rate equations:

\begin{align*}
    \\
    \frac{d[L]}{dt} &= \frac{d[R]}{dt} = -k_{f1}[L][R] + k_{b1}[L\cdot R] \tag{1.1}\\
    \frac{d[L\cdot R]}{dt} &= k_{f1}[L][R] -k_{b1}[L\cdot R] - k_{f2}[L\cdot R] \tag{1.2}\\
    \frac{d[P]}{dt} &=  k_{f2}[L\cdot R] \tag{1.3}
    \\
\end{align*}

Next, we apply a steady state approximation on $d[L\cdot R]/dt$:

\begin{equation}
k_{b1}[L\cdot R] = k_{f1}[L][R] - k_{f2}[L\cdot R]
 \implies \boxed{[L\cdot R] = \left( \frac{k_{f1}}{k_{b1}+k_{f2}} [L][R]\right)}
\end{equation}

Now, we take $[L\cdot R]$ and substitute into equations ${1.1}$ and ${1.3}$.

\begin{align}
    \frac{d[L]}{dt} &= \frac{d[R]}{dt} = -k_{f1}[L][R] + k_{b1}\left(\frac{k_{f1}}{k_{b1}+k_{f2}} [L][R] \right) \tag{1.1a}\\
    \frac{d[P]}{dt} &=  k_{f2} \left( \frac{k_{f1}}{k_{b1}+k_{f2}} [L][R]\right) \tag{1.3a}
\end{align}

Note that this last kinetic step is “one-way”, which violates thermodynamics (!). Don’t worry about this yet (we’re getting to kinetics soon in the lecture). For now, suffice it to say that here we invoke the assumption that the back rate is very small compared to forward rate.↩

Let’s walk through the steps of how to build a model of this receptor-ligand reaction using Python.

Procedure

Part I.

Set initial concentration of ligand to 10.0 and receptor to 5.0

Set rate_eqs=ODE_eqs_part_I

set “Forward Rate Parameter” ($k_{f1}$) value to 0.02

set “Reverse Rate Parameter” ($k_{b1}$) value to 0.01

Run the simulation and save the graph.

Part II.

Repeat simulation for following values:

Add "k-prod" ($k_{f2}$) and set value to 0.05.

Set rate_eqs=ODE_eqs_part_II

Run the simulation and save the graph.

Does the reaction reach a steady state?

If not, increase simulation time to 100

Set rate_eqs=ODE_eqs_part_II_SS

Run the simulation. How does this plot/simulation differ from the last?

Repeat simulation for following values:

Set rate_eqs=ODE_eqs_part_II

“Forward Rate Parameter” ($k_{f1}$): 0.01 , “Reverse Rate Parameter” ($k_{b1}$): 0.02 , “k-prod” ($k_{f2}$): 0.05

“Forward Rate Parameter” ($k_{f1}$): 0.02, “Reverse Rate Parameter” ($k_{b1}$): 0.01 , “k-prod” ($k_{f2}$): 0.5

```python
import numpy as np
import pandas as pd
```

```python
def ODE_eqs_part_I(C, t, k):
    r"""
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R`

    Args:
        C(list): collection of concentrations for each species [L, R, LR]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1]
    """

    kf1, kb1 = k[0], k[1]
    dC1dt = -kf1*C[0]*C[1]+kb1*C[2]
    dC0dt = dC1dt
    dC2dt = -dC1dt
    return [dC0dt, dC1dt, dC2dt]

def ODE_eqs_part_II(C, t, k):
    r"""See eqs 1.1, 1.2 and 1.3
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P`

    Args:
        C(list): collection of concentrations for each species [L, R, LR, P]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1, kf2]
    """

    kf1, kb1, kf2 = k[0], k[1], k[2]
    dC1dt = -kf1*C[0]*C[1]+kb1*C[2]
    dC0dt = dC1dt
    dC2dt = k[0]*C[0]*C[1]-k[1]*C[2]-k[2]*C[2]
    dC3dt = kf2*C[2]
    return [dC0dt, dC1dt, dC2dt, dC3dt]

def ODE_eqs_part_II_SS(C, t, k):
    r"""See eqs 1.2, 1.1a and 1.3a
    Using the steady state approximation on dLR/dt.
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P`

    Args:
        C(list): collection of concentrations for each species [L, R, LR, P]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1, kf2]
    """

    kf1, kb1, kf2 = k[0], k[1], k[2]
    dC1dt = -kf1*C[0]*C[1]+kb1*(kf1/(kb1+kf2)*C[0]*C[1])
    dC0dt = dC1dt
    dC2dt = k[0]*C[0]*C[1]-k[1]*C[2]-k[2]*C[2]
    dC3dt = kf2*(kf1/(kb1+kf2)*C[0]*C[1])
    return [dC0dt, dC1dt, dC2dt, dC3dt]

def simulate(steps, simulation_time, parameters, rate_eqs):
    """Method for integrating the ODEs with SciPy odeint
    https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html

    Args:
        steps(int): number of iterations
        simulation_time(int): duration of simulation (units: s)
        parameters(dict): {'species': ['L', 'R', 'LR'],\
                          'k': [kf1, kb1, kf2, kb2, ...],\  # rate constants
                          'InitConc': [L, R, P, ...]}       # initial concentrations (units: M)
        rate_eqs(function): method containing the ordinary differential equations
    Returns:
        Pandas DataFrame, plot
    """

    from scipy.integrate import odeint
    print(f"Number of species: {len(parameters['InitConc'])}")
    time = np.linspace(0, simulation_time, steps)
    C = odeint(rate_eqs, parameters["InitConc"], time, (parameters["k"],)).transpose()
    data = {"time": time, **{"%s"%s: C[i] for i,s in enumerate(parameters["species"])}}
    df = pd.DataFrame(data)
    plot = df.plot(x="time", y=parameters["species"], figsize=(10, 4),
                   xlim=(data["time"][0], data["time"][-1]))
    return df,plot

```

```python
parameters = {"species": ["L","R","LR"],
             "InitConc": [10, 5, 0],
              #          [kf1, kb1, kf2]
             "k":        [0.02, 0.01, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=10,
                   parameters=parameters, rate_eqs=ODE_eqs_part_I)
plot = plot.set_title(r"$L + R ⥂ L\cdot R$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
parameters = {"species": ["L","R", "LR", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [0.02, 0.01, 0.05, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=100,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
#“Forward Rate Parameter” ( 𝑘𝑓1 ): 0.01 , “Reverse Rate Parameter” ( 𝑘𝑏1 ): 0.02 , “k-prod” ( 𝑘𝑓2 ): 0.05

```

```python
#“Forward Rate Parameter” ( 𝑘𝑓1 ): 0.02, “Reverse Rate Parameter” ( 𝑘𝑏1 ): 0.01 , “k-prod” ( 𝑘𝑓2 ): 0.5

```

Bonus

This same reaction can be used to describe Michaelis-Menten enzyme kinetics, in which substrate (S) binds a protein enzyme (E) to form an encounter complex (E∙S), and then chemically converts to product (E + P).

$$E + S \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} E\cdot S \stackrel{k_{cat}}{\to} E + P$$

Suppose an enzyme has kf1 = 107 s-1, kb1 = 106 s-1 and kcat = 104 s-1. By adjusting parameters in the function above, which of these rates constants would you say most sensitively controls the overall rate of catalysis?

```python

```


# 10. Time dependent problem in 1D

we consider the time dependent problem
\begin{align*}
  -k_{\perp}^2 \partial_t \phi &= \partial_s J
  \\
  \partial_t A + \mu \partial_t J &= \partial_s \left( n - \phi \right)
  \\
  \partial_t n &= \partial_s J
  \\
  \beta J &= k_{\perp}^2 A
%  \label{}
\end{align*}
where $s \in [-\pi, \pi]$, $\beta \sim 10^{-3}$, $\mu \sim 10^{-4}$ and $k_{\perp} \in [10^{-2},10^{1}]$.
\\
It is easy to check that the eigenvalues related to the previous system are $\{-V_a k_{\parallel}, 0, V_a k_{\parallel} \}$ with $V_a := \frac{1+k_{\perp}^2}{\beta + \mu k_{\perp}^2}$.

## Wave equation for $A$

Multiplying the equation on $n$ by $k_{\perp}^2$ then adding it to the equation on $\phi$, we get
$$
\left( \beta + \mu k_{\perp}^2 \right) \partial_{tt} A = \left( 1 + k_{\perp}^2 \right) \partial_{ss} A
$$
which leads to the wave equation
$$
\partial_{tt} A = \frac{1+k_{\perp}^2}{\beta + \mu k_{\perp}^2} \partial_{ss} A
$$

## Time discretization

Let's define $\gamma := \frac{k_{\perp}^2}{\beta}$, and replace $J$ in the equation on $A$. We get
\begin{align*}
  \partial_t \phi &= - \frac{1}{\beta} \partial_s A
  \\
  \partial_t A &= \frac{1}{1 + \mu \gamma} \partial_s \left( n - \phi \right)
  \\
  \partial_t n &= \gamma \partial_s A
%  \label{}
\end{align*}

Using a full implicit time scheme, we have,
\begin{align*}
   \frac{\phi^{k+1} - \phi^{k}}{\Delta t} &= - \frac{1}{\beta} \partial_s A^{k+1}
  \\
  \frac{A^{k+1} - A^{k}}{\Delta t} &= \frac{1}{1 + \mu \gamma} \partial_s \left( n^{k+1} - \phi^{k+1} \right)
  \\
  \frac{n^{k+1} - n^{k}}{\Delta t} &= \gamma \partial_s A^{k+1}
%  \label{}
\end{align*}

finally,

\begin{align*}
    \phi^{k+1} + \frac{\Delta t}{\beta} \partial_s A^{k+1}  &= \phi^k
    \\
    \frac{\Delta t}{1+\mu \gamma} \phi^{k+1} + A^{k+1} - \frac{\Delta t}{1+\mu \gamma} \partial_s n^{k+1} &= A^k
    \\
    -\Delta t \gamma \partial_s A^{k+1} + n^{k+1} &= n^k
\end{align*}

## Weak formulation

Let $v$ denote a test function, in a Finite Elements space $V \subset H^1(\Omega)$. Multiplying all the equations by $v$, then integrating over the whole domain, we get

\begin{align*}
    \langle \phi^{k+1}, v \rangle
    + \frac{\Delta t}{\beta} \langle \partial_s A^{k+1}, v \rangle  &= \langle \phi^k, v \rangle
    \\
    \frac{\Delta t}{1+\mu \gamma} \langle \phi^{k+1}, v \rangle + \langle A^{k+1}, v \rangle - \frac{\Delta t}{1+\mu \gamma} \langle \partial_s n^{k+1}, v \rangle &= \langle A^k, v \rangle
    \\
    -\Delta t \gamma \langle \partial_s A^{k+1}, v \rangle + \langle n^{k+1}, v \rangle &= \langle n^k, v \rangle
\end{align*}

We use a symmetrized weak formulation, where we assume having periodic boundary conditions:

\begin{align*}
    \langle \phi^{k+1}, v \rangle
    + \frac{\Delta t}{2 \beta} \langle \partial_s A^{k+1}, v \rangle
    - \frac{\Delta t}{2 \beta} \langle A^{k+1}, \partial_s v \rangle
    &= \langle \phi^k, v \rangle
    \\
    \frac{\Delta t}{1+\mu \gamma} \langle \phi^{k+1}, v \rangle
    + \langle A^{k+1}, v \rangle
    - \frac{\Delta t}{2+2\mu \gamma} \langle \partial_s n^{k+1}, v \rangle
    + \frac{\Delta t}{2+2\mu \gamma} \langle n^{k+1}, \partial_s v \rangle
    &= \langle A^k, v \rangle
    \\
    -\frac{\Delta t \gamma}{2} \langle \partial_s A^{k+1}, v \rangle
    +\frac{\Delta t \gamma}{2} \langle A^{k+1}, \partial_s v \rangle
    + \langle n^{k+1}, v \rangle
    &= \langle n^k, v \rangle
\end{align*}

In order to simplify the notation, we introduction the following bilinear form
$$
b( v,u ) := \frac{1}{2} \left( \langle \partial_s u, v \rangle - \langle u, \partial_s v \rangle \right)
$$
then our weak formulation writes

\begin{align*}
    \langle \phi^{k+1}, v \rangle
    + \frac{\Delta t}{\beta} b(v,A^{k+1})
    &= \langle \phi^k, v \rangle
    \\
    \frac{\Delta t}{1+\mu \gamma} \langle \phi^{k+1}, v \rangle
    + \langle A^{k+1}, v \rangle
    - \frac{\Delta t}{1+\mu \gamma} b(v,n^{k+1})
    &= \langle A^k, v \rangle
    \\
    -\Delta t \gamma b(v,A^{k+1})
    + \langle n^{k+1}, v \rangle
    &= \langle n^k, v \rangle
\end{align*}

Finally, let's introduce the weak formulation related to the mass matrix $a_m(v,u) := \langle u, v \rangle$,

\begin{align*}
    a_m(v, \phi^{k+1})
    + \frac{\Delta t}{\beta} b(v, A^{k+1})
    &= a_m(v, \phi^k)
    \\
    \frac{\Delta t}{1+\mu \gamma} a_m(v, \phi^{k+1})
    + a_m(v, A^{k+1})
    - \frac{\Delta t}{1+\mu \gamma} b(v,  n^{k+1})
    &= a_m(v, A^k)
    \\
    - \Delta t \gamma b(v, A^{k+1})
    + a_m(v, n^{k+1})
    &= a_m(v, n^k)
\end{align*}

```python
import numpy as np
from numpy import linspace, zeros, pi

from sympy.core.containers import Tuple
from sympy import symbols
from sympy import Symbol
from sympy import Lambda
from sympy import Function

from gelato.glt import glt_symbol
from gelato.calculus   import (Dot, Cross, Grad, Curl, Rot, Div, dx)
from gelato.calculus   import Constant
from gelato.fem.assembly import assemble_matrix
from gelato.fem.utils    import compile_kernel
from gelato.fem.utils    import compile_symbol

from spl.fem.splines import SplineSpace
from spl.fem.vector  import VectorFemSpace

from IPython.display import Math
from sympy import latex
```

```python
x = Symbol('x')

u = Symbol('u')
v = Symbol('v')

a_m = lambda v,u: u*v
b = lambda v,u: 0.5*(dx(u)*v - u*dx(v))
```

\begin{align*}
    a_m(v, \phi^{k+1})
    + \frac{\Delta t}{\beta} b(v, A^{k+1})
    &= a_m(v, \phi^k)
    \\
    \frac{\Delta t}{1+\mu \gamma} a_m(v, \phi^{k+1})
    + a_m(v, A^{k+1})
    - \frac{\Delta t}{1+\mu \gamma} b(v,  n^{k+1})
    &= a_m(v, A^k)
    \\
    - \Delta t \gamma b(v, A^{k+1})
    + a_m(v, n^{k+1})
    &= a_m(v, n^k)
\end{align*}

```python
phi, A, n = symbols('phi A n')
v0, v1, v2 = symbols('v0 v1 v2')

dt = Constant('dt')
beta = Constant('beta')
mu = Constant('mu')
gamma = Constant('gamma')

a = Lambda((x,v0,v1,v2,phi,A,n),
             a_m(v0, phi) + dt/beta * b(v0, A)
           + dt/(1+mu * gamma) * a_m(v1, phi)
           + a_m(v1, A)
           - dt/(1+mu * gamma) * b(v1, n)
           - dt * gamma * b(v2, A) + a_m(v2, n))
```

```python
# create a finite element space
p  = 3
ne = 64

grid = linspace(0., 1., ne+1)

W = SplineSpace(p, grid=grid)
V = VectorFemSpace(W, W, W)
```

```python
symbol = glt_symbol(a, space=V)
```

```python
Math(latex(symbol))
```

$$\left[\begin{matrix}m_{1} & \frac{1.0 a_{1}}{\beta} dt & 0\\\frac{dt m_{1}}{\gamma \mu + 1} & m_{1} & - \frac{1.0 dt a_{1}}{\gamma \mu + 1}\\0 & - 1.0 dt \gamma a_{1} & m_{1}\end{matrix}\right]$$

```python
eigen = symbol.eigenvals()
eigen = list(eigen.keys())
```

```python
Math(latex(eigen))
```

$$\left [ m_{1}, \quad \frac{1}{\beta \gamma \mu + \beta} \left(\beta \gamma \mu m_{1} + \beta m_{1} + dt \sqrt{\beta^{2} \gamma^{2} \mu a_{1}^{2} + \beta^{2} \gamma a_{1}^{2} + \beta \gamma \mu a_{1} m_{1} + \beta a_{1} m_{1}}\right), \quad \frac{1}{\beta \gamma \mu + \beta} \left(\beta \gamma \mu m_{1} + \beta m_{1} - dt \sqrt{\beta^{2} \gamma^{2} \mu a_{1}^{2} + \beta^{2} \gamma a_{1}^{2} + \beta \gamma \mu a_{1} m_{1} + \beta a_{1} m_{1}}\right)\right ]$$

```python
# compute the symbol of the mass
symbol_m = glt_symbol(Lambda((x,v,u), u*v), space=V)
symbol_a = glt_symbol(Lambda((x,v,u), dx(u)*v), space=V)
```

```python
eigen_normalized = [e/symbol_m for e in eigen]
```

```python
Math(latex(eigen_normalized))
```

$$\left [ 1, \quad \frac{1}{m_{1} \left(\beta \gamma \mu + \beta\right)} \left(\beta \gamma \mu m_{1} + \beta m_{1} - dt \sqrt{\beta^{2} \gamma^{2} \mu a_{1}^{2} + \beta^{2} \gamma a_{1}^{2} - \beta \gamma \mu a_{1} m_{1} - \beta a_{1} m_{1}}\right), \quad \frac{1}{m_{1} \left(\beta \gamma \mu + \beta\right)} \left(\beta \gamma \mu m_{1} + \beta m_{1} + dt \sqrt{\beta^{2} \gamma^{2} \mu a_{1}^{2} + \beta^{2} \gamma a_{1}^{2} - \beta \gamma \mu a_{1} m_{1} - \beta a_{1} m_{1}}\right)\right ]$$

```python
e = eigen_normalized[1]
```

```python
from sympy import simplify, cancel, collect, expand
```

```python
Math(latex(cancel(e-1)))
```

$$- \frac{dt}{\beta \gamma \mu m_{1} + \beta m_{1}} \sqrt{\beta^{2} \gamma^{2} \mu a_{1}^{2} + \beta^{2} \gamma a_{1}^{2} - \beta \gamma \mu a_{1} m_{1} - \beta a_{1} m_{1}}$$

```python
Math(latex(symbol_a))
```

$$a_{1}$$

```python
print(symbol_a.is_complex)
```

    None

```python

```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


## Exploring reference frame

```python
import sympy as sp
import sympy.physics.mechanics as me
```

```python
psi = me.dynamicsymbols('psi')
x0,y0 = me.dynamicsymbols('x0 y0')
x01d,y01d = me.dynamicsymbols('x0 y0',1)

u,v = me.dynamicsymbols('u v')
```

```python
N = me.ReferenceFrame('N')
B = N.orientnew('B','Axis',[psi,N.z])
```

```python
B.dcm(N)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\psi{\left(t \right)} \right)} & \sin{\left(\psi{\left(t \right)} \right)} & 0\\- \sin{\left(\psi{\left(t \right)} \right)} & \cos{\left(\psi{\left(t \right)} \right)} & 0\\0 & 0 & 1\end{matrix}\right]$

### V2pt_theory

```python
O = me.Point('O')
P = O.locatenew('P',10*B.x)
```

```python
O.set_vel(N,x0*N.x + y0*N.y)
```

```python
P.v2pt_theory(O,N,B)
```

$\displaystyle 10 \dot{\psi}\mathbf{\hat{b}_y} + x_{0}\mathbf{\hat{n}_x} + y_{0}\mathbf{\hat{n}_y}$

```python
P.vel(N).to_matrix(N)
```

$\displaystyle \left[\begin{matrix}\operatorname{x_{0}}{\left(t \right)} - 10 \sin{\left(\psi{\left(t \right)} \right)} \frac{d}{d t} \psi{\left(t \right)}\\\operatorname{y_{0}}{\left(t \right)} + 10 \cos{\left(\psi{\left(t \right)} \right)} \frac{d}{d t} \psi{\left(t \right)}\\0\end{matrix}\right]$

### V1pt_theory

```python
O = me.Point('O')
P = O.locatenew('P',0)
```

```python
O.set_vel(N,0)
P.set_vel(B,u*B.x + v*B.y)
```

```python
P.v1pt_theory(O,N,B)
```

$\displaystyle u\mathbf{\hat{b}_x} + v\mathbf{\hat{b}_y}$

```python
P.vel(N).to_matrix(N)
```

$\displaystyle \left[\begin{matrix}u{\left(t \right)} \cos{\left(\psi{\left(t \right)} \right)} - v{\left(t \right)} \sin{\left(\psi{\left(t \right)} \right)}\\u{\left(t \right)} \sin{\left(\psi{\left(t \right)} \right)} + v{\left(t \right)} \cos{\left(\psi{\left(t \right)} \right)}\\0\end{matrix}\right]$

## Bludder

```python
O = me.Point('O')
O.set_vel(N,0)

P = me.Point('P')
P.set_pos(O,x0*N.x + y0*N.y)
P.set_vel(B,u*B.x + v*B.y)

P.v1pt_theory(O,N,B)
```

$\displaystyle u\mathbf{\hat{b}_x} + v\mathbf{\hat{b}_y} -  y_{0} \dot{\psi}\mathbf{\hat{n}_x} + x_{0} \dot{\psi}\mathbf{\hat{n}_y}$

```python
P.vel(N).to_matrix(N)
```

$\displaystyle \left[\begin{matrix}u{\left(t \right)} \cos{\left(\psi{\left(t \right)} \right)} - v{\left(t \right)} \sin{\left(\psi{\left(t \right)} \right)} - \operatorname{y_{0}}{\left(t \right)} \frac{d}{d t} \psi{\left(t \right)}\\u{\left(t \right)} \sin{\left(\psi{\left(t \right)} \right)} + v{\left(t \right)} \cos{\left(\psi{\left(t \right)} \right)} + \operatorname{x_{0}}{\left(t \right)} \frac{d}{d t} \psi{\left(t \right)}\\0\end{matrix}\right]$

```python

```