# Project 5, deadline  December 14, 2018

**The Rossby wave equation in a basin**

Date: **Fall semester 2018**

**For this project you can collaborate with fellow students and you can  hand in a common report.**
This project (together with projects 3 and 4) counts 1/3 of the final mark.

## Theoretical background and description of the system

Large scale, time-dependent motion in the atmosphere and ocean is
often conceptualized in terms of *Rossby waves*, which have scales
of hundreds to thousands of kilometers. The waves are used, for
example, to understand the meandering of the atmospheric Jet Stream
and the adjustment of the ocean to changes in wind forcing. Rossby
waves exist because of the Coriolis acceleration, which acts
perpendicular to the velocity of a fluid parcel. The Coriolis
acceleration varies with latitude, being negative in the Southern
Hemisphere and positive in the Northern (and vanishes at the equator).
As a result of this variation, fluid parcels experience a change in
their spin or *vorticity* if they move to different latitudes.

We will use the shorthand notations

0

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

1

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\partial_{xx} = \frac{\partial^2 }{\partial x^2}
$$

etc.

The equation governing the vorticity is:

$$
\begin{equation}
\label{eq:Rossby1} \tag{1}
   \partial_t \zeta  + \beta \partial_x\psi = 0.
 \end{equation}
$$

where $\zeta$ is the vorticity and $\psi$ is the streamfunction, which determines the velocities:

$$
\begin{equation}
	u = -\partial_y \psi \quad,\quad v = \partial_x \psi,
\label{_auto1} \tag{2}
\end{equation}
$$

Here $u$ is the velocity component in the east-west direction ($x$)
and $v$ in the north-south direction ($y$). The vorticity is
defined:

$$
\begin{equation}
\label{eq:Rossby2} \tag{3}
	\zeta = \partial_x v -\partial_y u = \nabla_H^2 \psi.
\end{equation}
$$

Thus the vorticity is the Laplacian of the streamfunction.

Then there is the Coriolis parameter, defined as:

$$
\begin{equation}
f = 2 \Omega \sin{(\theta)}
\label{_auto2} \tag{4}
\end{equation}
$$

where $\theta$ is the latitude and $\Omega$ is the rotation
rate of the Earth, $\Omega=2 \pi/day$. We often approximate
$f$ as a linear function, centered on a latitude $\theta_0$:

$$
\begin{equation}
f \approx f_0 + \beta y
\label{_auto3} \tag{5}
\end{equation}
$$

where $f_0 = 2 \Omega \sin{(\theta_0)}$, $\beta = 2 \Omega
\cos{(\theta_0)}/R_e$ and $y = R_e (\theta - \theta_0)$, if $R_e$ is the
Earth's radius. This linear representation is called the
$\beta$-plane approximation, and accounts for the $\beta$ term in
([eq:Rossby1](#eq:Rossby1)).

Thus the vorticity equation can be written in terms of one variable,
the streamfunction, thus:

$$
\begin{equation}
\label{eq:Rossby} \tag{6}
\boxed{\partial_t \nabla_H^2 \psi  + \beta \partial_x\psi = 0.}
 \end{equation}
$$

This is the *barotropic Rossby wave equation*.

We will examine analytic and numerical solutions to the vorticity
equation in a *periodic domain* and a *closed domain*. A
periodic domain is one that "wraps around", like the atmosphere. A
closed domain has walls, like the continents bounding the oceans.

## Analytical solutions

### Project 5a): Solution to the Rossby equation in a periodic domain

Consider the vorticity equation ([eq:Rossby](#eq:Rossby)) in one dimension
(we'll add the second dimension later). Say the periodic domain
extends from $x=[0,L]$ (east-west). A suitable wave solution has the
form:

$$
\begin{equation}
\label{eq:Rossby5} \tag{7}
	\psi = A \cos{(\frac{2n\pi x}{L} - \omega t)},
\end{equation}
$$

where $n$ is an integer ($=1,2,\cdots$), and where $\omega$ is the
wave frequency and $A$ is the amplitude.

Show that the solution is the same at the two boundaries, so that it
satisfies the periodic boundary conditions. Then use this wave solution
in equation ([eq:Rossby](#eq:Rossby)) and solve for $\omega$. Use the
one-dimensional form for the vorticity:

$$
\begin{equation}
\label{eq:Rossby6} \tag{8}
	\zeta = \partial_x v = \partial_{xx} \psi
\end{equation}
$$

The result is known as the wave *dispersion relation*.

The *phase speed* is the speed at which the Rossby waves move. To see how this comes about, consider the phase of the wave:

$$
\begin{equation}
\label{eq:Rossby7} \tag{9}
	\theta = \frac{n\pi x}{L} - \omega t.
\end{equation}
$$

For instance, if $\theta=2\pi$, then $\psi = A\cos{(2\pi)} = A$. This is a maximum or "high pressure" point. To see how this moves, we solve ([eq:Rossby7](#eq:Rossby7)) for $x$:

$$
\begin{equation}
	x = \frac{\theta L}{2n\pi} + \frac{\omega tL}{2n\pi}.
\label{_auto4} \tag{10}
\end{equation}
$$

Taking a time derivative, we obtain

$$
\begin{equation}
	c = \frac{dx}{dt} = \frac{\omega L}{2n\pi}
\label{_auto5} \tag{11}
\end{equation}
$$

Calculate the phase speed for the Rossby wave, using the dispersion relation. Which direction is the wave moving?

### Project 5b): Solution to the Rossby equation with solid boundaries

More appropriate boundary conditions for the ocean are no flow at the
two end points. In the present problem, we can enforce this with
simple Dirichlet conditions: $\psi=0$ at $x=0$ and $x=L$. Use
a wave solution with the form:

$$
\begin{equation}
	\psi = A(x)\cos{(kx - \omega t)},
\label{_auto6} \tag{12}
\end{equation}
$$

where $k$ is the wavenumber. Substitute this into equation
([eq:Rossby](#eq:Rossby)). You'll obtain terms multiplied by $\sin{(kx - \omega
t)}$ and others multiplied by $\sin{(kx - \omega t)}$. Set each of these
expressions to zero. One will give you the dispersion relation for the
waves. The other will give you the form for $A(x)$, which must satisfy
the boundary conditions. You see that $k$ will be *quantized*:
only specific values will be allowed.

Waves like these are known as Rossby "basin modes". The phase speed
can be calculated as in the preceding problem. What is it? Discuss the
structure of the wave (which is unusual).

## Numerical solution
Now we consider solving equation ([eq:Rossby](#eq:Rossby)) numerically. This
involves two steps. If we know the velocity or streamfunction
initially, we can advance the vorticity in time to a new time. Then
the streamfunction at the new time is found by inverting
([eq:Rossby2](#eq:Rossby2)). Having done this we may advance the vorticity to
the next time level using ([eq:Rossby](#eq:Rossby)), and so forth. We note
that ([eq:Rossby](#eq:Rossby)) is a *prognostic equation* and may
therefore be solved numerically using a time stepping method. In
contrast, ([eq:Rossby2](#eq:Rossby2)) is a *diagnostic equation*. We will
solve this using an elliptic solver.

To discretize the equations, we assume a grid of equally-spaced
points:

$$
\begin{equation}
x_j = j\Delta x
\label{_auto7} \tag{13}
\end{equation}
$$

where $\Delta x$ is the grid spacing. We discretize time in a similar way:

$$
\begin{equation}
t^n = n\Delta t
\label{_auto8} \tag{14}
\end{equation}
$$

where $\Delta t$ is the time step. Thus $t^0=0$, $t^1=\Delta t$, and so on.

We then approximate the derivatives by finite differences. For
the spatial derivatives, we use centered-differences:

$$
\begin{equation}
	\partial_x\psi \approx \frac{\psi_{j+1}^{n} - \psi_{j-1}^{n}}{2\Delta x},
\label{_auto9} \tag{15}
\end{equation}
$$

$$
\begin{equation}
	\partial_{xx}\psi \approx \frac{\psi_{j+1}^{n} - 2\psi_{j}^{n} + \psi_{j-1}^{n}}{\Delta x^2},
\label{_auto10} \tag{16}
\end{equation}
$$

For the time stepping, we will test two methods. One involves a
forward difference:

$$
\begin{equation}
\label{eq:Rossby3} \tag{17}
  \partial_t\psi \approx \frac{\psi_{j}^{n+1} - \psi_{j}^{n}}{\Delta t} ,
\end{equation}
$$

while the second involves a centered difference:

$$
\begin{equation}
\label{eq:Rossby4} \tag{18}
	\partial_t\psi \approx \frac{\psi_{j}^{n+1} - \psi_{j}^{n-1}}{2\Delta t}.
\end{equation}
$$

### Project 5c): Setting up the algorithms

Now we will examine
numerical solutions in one dimension in the east-west
direction. When calculating these, it is convenient to use
*non-dimensional* forms of the equations:

$$
\begin{equation}
	\partial_t\zeta + \partial_x\psi = 0.
\label{_auto11} \tag{19}
\end{equation}
$$

$$
\begin{equation}
	\partial_{xx}\psi = \zeta.
\label{_auto12} \tag{20}
\end{equation}
$$

These are obtained by *scaling* each term by typical values.
Doing this, the domain extends from $x=0$ to $x=1$, which is
much simpler than worrying about realistic sizes.   For the latter equation you can use your code from project 1.

For the boundary conditions, we'll consider both the periodic
domain (that wraps around) and the bounded domain (with solid
walls). For the bounded domain, the streamfunction is zero at the
ends, while for the periodic one the boundary values can vary.

Write the algorithms for the two time-stepping methods and the
equations you need to implement for the one-dimensional case,
including the numerical analogues of the boundary conditions.

### Project 5d): Truncation errors and stability

Determine the truncation errors of the two time-stepping schemes
and investigate their stability properties. What do you conclude
about the two schemes? What is the stability criterion for the
stable algorithm.

### Project 5e): Implementation

Now we'll implement the model, using both time-stepping schemes.
Hereafter we'll use two (non-dimensional) initial
conditions, a sine wave:

$$
\begin{equation}
\psi(x,0) = \sin{\left( 4\pi x \right)}
\label{_auto13} \tag{21}
\end{equation}
$$

and a Gaussian:

$$
\begin{equation}
\psi = \exp{-\left(\frac{x-x0}{\sigma}\right)^2},
\label{_auto14} \tag{22}
\end{equation}
$$

Here $\sigma = 0.1$ the width of the Gaussian.

Compare the time-stepping routines, using the sine wave in the
periodic domain. Use $\Delta x=1/40$ and $\Delta t$ as determined from the
results of (5d). Store the results at three times, in addition to the
initial condition, e.g. $t=0$, $t=50$, $t=100$ and $t=150$.  Is one of the
time-stepping routines better than the other?

### Project 5f): Visualization and discussion

Now we'll examine a suite of solutions, using the stable time-stepping
algorithm. Consider the following:

a) Calculate the phase speed of the sine wave in the periodic domain.
Do this by using a *Hovmuller diagram*.  For this you construct a
matrix with $\psi(x,t)$ at different times. $t=0$ can be on the bottom
row, then $t=\Delta t$ in the next row, $t=2\Delta t$ in the next and
so. Contour the matrix and extract the phase speed. How does it
compare to the theoretical prediction in (5a)?

b) Now consider the sine wave in the bounded domain. How does this
evolve differently? Rationalize the results using the theoretical
prediction from (5b).

c) Now examine the Gaussian, in the periodic domain. How does the
Hovmuller diagram differ? Can you find a phase speed? Try varying
the width, $\sigma$. How does this affect the phase speed.

d) Lastly, describe the evolution of the Gaussian in the bounded
domain. How does this compare to the sine wave in the same domain?

### Project 5g): Moving to two dimensions

Extend the code you have developed here to two dimensions, that is,
$\psi = \psi(x,y,t)$ and $\zeta = \zeta(x,y,t)$. It means that we deal
with a $2+1$ dimensional problem. Our differential equations become

$$
\begin{equation}
\label{eq:Rossby10} \tag{23}
	\partial_t\zeta + \partial_x\psi = 0, \quad \textrm{$t > 0$ and} \quad x,y\in [0,1],
\end{equation}
$$

and

$$
\begin{equation}
\label{eq:Rossby11} \tag{24}
	\left( \partial_{xx}\psi + \partial_{yy}\psi \right) = \zeta, \quad \textrm{$t > 0$ and} \quad x,y\in [0,1],
\end{equation}
$$

where we now have made a model with a square lattice for $x$ and $y$. The last equation is just another example of Poisson's equation which can be solved by Jacobi's method, or the Gauss-Seidel method or the so-called Successive Over Relaxation method discussed in chapters 6 and 10 of the lecture notes.

How would you extend the boundary conditions from one dimension to two
dimensions? And can you find a closed form solution here as well?  It
is left to you to decide upon what kind of boundary conditions you
deem appropriate.

### Project 5h): Solving the two-dimensional equations numerically

Use an explicit scheme for ([eq:Rossby10](#eq:Rossby10)). To solve
([eq:Rossby11](#eq:Rossby11)) you need to use for example an iterative method
like Jacobi's or Gauss-Seidel method or the Successive Over Relaxation (SOR) method. These methods are described in the lecture notes, see chapters 6 and 10.

Outline the algorithm for solving the two-dimensional equations and
implement the scheme as function of $\Delta x$ (assuming $\Delta x =
\Delta y$) and $\Delta t$. Solve the equations numerically and give a
critical discussion of your results. Compare your results with the
closed-form answer if possible. Discuss the stability of the solution
as function of different values of $\Delta x$ and $\Delta t$.

### References

A very good reference is the textbook by [Winther and Tveito on partial differential equations](http://www.springer.com/us/book/9783540225515). It is available online [from the University library](https://vpn2.uio.no/+CSCO+0h756767633A2F2F797661782E66636576617472652E70627A++/book/10.1007/b138016/page/1).

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use Devilry to hand in your projects, log in  at  <http://devilry.ifi.uio.no> with your normal UiO username and password and choose either 'fys3150' or 'fys4150'. There you can load up the files within the deadline.

  * Upload **only** the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them.

  * In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parametxers.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. For this specific report you can  hand in a common report.


# Robinson Crusoe

With ``respy`` you are able to prototype a model similar to Keane and Wolpin (1997) in minutes. As economists love Robinsonades[1](#fn1), we will showcase the implementation of a the Robinson Crusoe economy as a discrete choice dynamic programming model. Throughout the notebook you find indented text which tell parts of Robinson's story and motivates the model. We will first set the scene with a broad introduction and then turn to the precise model specification. We continue by simulating the model and analyze its comparative statics. We then extend the model and showcase the estimation of the model parameters.

Just to be clear, don't misinterpret the fact that we explain ``respy`` using such a simplistic model. ``respy`` is not a toy and can just as well solve state-of-the-art structural models. It's just easier to explain respy in a situation where we don't have to explain a complicated model at the same time.

```python
%matplotlib inline

import io
import matplotlib.pyplot as plt
import pandas as pd
import respy as rp
import yaml
import seaborn as sns
import numpy as np

from pathlib import Path
from time import time

plt.style.use("../_static/respy.mplstyle")
```

## Introduction

> After setting sail against his parents' wishes, being captured by pirates, escaping from them, building a plantation, and setting sail again to capture slaves in Africa, [Robinson Crusoe](https://en.wikipedia.org/wiki/Robinson_Crusoe) stranded on a small island. He is alone with one dog, two cats, and only some supplies. He goes fishing to make ends meet and if he is too tired he will relax in his hammock. But, he cannot relax to often as storing food is not easy on a tropical island.

In the discrete choice dynamic programming model, Robinson chooses every period to either go fishing, $a = 0$, or spend the day in the hammock, $a = 1$, to maximize his expected sum of discounted lifetime utility. The utility of a choice, $U(s_t, a_t)$, depends on the state $s_t$, which contains information on the individual's characteristics, and the chosen alternative $a_t$. For working alternatives like fishing utility consists of two components, a wage and a non-pecuniary component.

$$
    U(s_t, a_t) = W(s_t, a_t) + N(s_t, a_t)
$$

For non-working alternatives like the hammock, $W(s_t, a_t) = 0$. The wage is defined as

$$\begin{align}
        W(s_t, a_t)  &= r_a \exp\{x^w_{at} \beta^w_a + \epsilon_{at}\}\\
    \ln(W(s_t, a_t)) &= \ln(r_a) + x^w_{at} \beta^w_a + \epsilon_{at}
\end{align}$$

where $r_a$ is normally a market rental price for the skill units generated in the exponential expression. Another interpretation is that $ln(r_a)$ is simply the constant in the skill units. The skill units are generated by two components. $x^w_{at}$ and $\beta^w_a$ are the choice- and time-dependent covariates and parameters related to the wage signaled by superscript $w$. The last term, $\epsilon_{at}$ is a random shock.

The non-pecuniary rewards for working alternatives are simply a vector dot product of covariates $x_t^w$ and parameters $\beta^w$. The superscript $w$ signals that the components belong to working alternatives.

$$
    N^w(s_t, a_t) = x_t^w\beta^w
$$

The non-pecuniary reward for non-working alternatives is very similar except that the shocks enter the equation additively. Superscript $n$ stands for non-pecuniary.

$$
    N^n(s_t, a_t) = x_t^n\beta^n + \epsilon_{at}
$$

Along with the lower triangular elements of the shock variance-covariance matrix of $\epsilon_t$, the utility parameters $\beta_a^w$, $\beta_a^n$ and $r_a$ form the main parameters of the model.

## Specification

How can we express the equations and parameters with ``respy``? The following cell contains the code to write a ``.csv`` file which is the cornerstone of a model as it contains all parameters and some other specifications. It is quickly written and easily loaded with ``pandas``.

```python
%%writefile robinson_crusoe_basic.csv

category, name, value
delta, delta, 0.95
wage_fishing, exp_fishing, 0.1
nonpec_fishing, constant, -1
nonpec_hammock, constant, 2.5
nonpec_hammock, not_fishing_last_period, -1
shocks_sdcorr, sd_fishing, 1
shocks_sdcorr, sd_hammock, 1
shocks_sdcorr, corr_hammock_fishing, -0.2
lagged_choice_1_hammock, constant, 1
```

    Overwriting robinson_crusoe_basic.csv

```python
params = pd.read_csv(
    "robinson_crusoe_basic.csv", index_col=["category", "name"]
)
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      nonpec_fishing
      constant
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      -0.20

      lagged_choice_1_hammock
      constant
      1.00

The parameters :class:`pd.DataFrame` contains a two-level :class:`pd.MultiIndex` to group parameters in categories. ``name`` should be uniquely assigned in each category or otherwise only the sum of identically named parameters is identified. ``value`` contains the value of the parameter. Note that we named Robinson's alternatives ``"fishing"`` and ``"hammock"`` and we have to use the names consistently. As long as you stick to lowercase letters separated by underscores, you can choose any name you want.

The parameter specification contains following entries:

- The first entry contains the discount factor of individuals.
- The second category ``"wage_fishing"`` contains the parameters of the log wage equation for fishing. The group contains only one name called ``"exp_fishing"`` where ``"exp_*"`` is an identifier in the model for experience accumulated in a certain alternative. ``respy`` requires that you respect those identifiers of which there are not many and reference your alternatives consistently with the same name. If you stick to lowercase letters possibly separated by underscores, you are fine.
- The third and fourth categories concern the non-pecuniary reward of fishing and relaxing in the hammock.
- ``"shocks_sdcorr"`` groups the lower triangular of the variance-covariance matrix of shocks.
- ``"lagged_choice_1_hammock"`` governs the distribution of previous choices at the begin of the model horizon.

``params`` is complemented with ``options`` which contains additional information. Here is short description:

- ``"n_periods"`` defines the number of periods for which decision rules are computed.
- ``"_seed"``: Seeds are used in every model component to ensure reproducibility.
- ``"estimation_draws"`` defines the number of draws used to simulate the choice probabilities with Monte Carlo integration.
- ``"estimation_tau"`` controls the smoothing to avoid zero-valued probabilities.
- ``"interpolation_points"`` controls how many states are used to approximate the value functions of others states in each period. ``-1`` turns the approximation off. The approximation is detailed in Keane and Wolpin (1994).
- ``"simulation_agents"`` defines how many agents are simulated.
- ``"solution_draws"`` defines the number of draws used to simulate the value functions.
- ``"covariates"`` is another dictionary where the key determines the covariate's name and the value is its definition. Here, we have to define what ``"constant"`` means. The covariate is created with :func:`pd.eval`.

```python
%%writefile robinson_crusoe_basic.yaml

n_periods: 10
estimation_draws: 200
estimation_seed: 500
estimation_tau: 0.001
interpolation_points: -1
simulation_agents: 1_000
simulation_seed: 132
solution_draws: 500
solution_seed: 456
covariates:
    constant: "1"
    not_fishing_last_period: "lagged_choice_1 != 'fishing'"
```

    Overwriting robinson_crusoe_basic.yaml

```python
options = yaml.safe_load(Path("robinson_crusoe_basic.yaml").read_text())
options
```

    {'n_periods': 10,
     'estimation_draws': 200,
     'estimation_seed': 500,
     'estimation_tau': 0.001,
     'interpolation_points': -1,
     'simulation_agents': 1000,
     'simulation_seed': 132,
     'solution_draws': 500,
     'solution_seed': 456,
     'covariates': {'constant': '1',
      'not_fishing_last_period': "lagged_choice_1 != 'fishing'"}}

## Simulation

We are now ready to simulate the model.

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

```python
df.head(15)
```

      Experience_Fishing
      Lagged_Choice_1
      Shock_Reward_Fishing
      Meas_Error_Wage_Fishing
      Shock_Reward_Hammock
      Meas_Error_Wage_Hammock
      Choice
      Wage
      Discount_Rate
      Nonpecuniary_Reward_Fishing
      Wage_Fishing
      Flow_Utility_Fishing
      Value_Function_Fishing
      Continuation_Value_Fishing
      Nonpecuniary_Reward_Hammock
      Wage_Hammock
      Flow_Utility_Hammock
      Value_Function_Hammock
      Continuation_Value_Hammock

      Identifier
      Period

      0
      0
      0
      hammock
      2.048628
      1
      0.866250
      1
      fishing
      2.048628
      0.95
      -1
      2.048628
      1.048628
      19.513202
      19.436393
      1.5
      NaN
      2.366250
      19.233744
      17.755256

      1
      1
      fishing
      0.147087
      1
      1.421523
      1
      hammock
      NaN
      0.95
      -1
      0.162556
      -0.837444
      16.864110
      18.633215
      2.5
      NaN
      3.921523
      20.032537
      16.958962

      2
      1
      hammock
      0.903027
      1
      -0.351595
      1
      fishing
      0.998000
      0.95
      -1
      0.998000
      -0.002000
      15.655857
      16.481955
      1.5
      NaN
      1.148405
      15.352842
      14.952039

      3
      2
      fishing
      0.339405
      1
      -0.930422
      1
      hammock
      NaN
      0.95
      -1
      0.414550
      -0.585450
      13.941609
      15.291641
      2.5
      NaN
      1.569578
      14.680286
      13.800745

      4
      2
      hammock
      2.822820
      1
      -0.420713
      1
      fishing
      3.447800
      0.95
      -1
      3.447800
      2.447800
      14.749573
      12.949235
      1.5
      NaN
      1.079287
      12.114100
      11.615592

      5
      3
      fishing
      2.015148
      1
      2.056790
      1
      hammock
      NaN
      0.95
      -1
      2.720165
      1.720165
      12.341990
      11.180868
      2.5
      NaN
      4.556790
      13.991270
      9.931031

      6
      3
      hammock
      5.802097
      1
      -0.090973
      1
      fishing
      7.832012
      0.95
      -1
      7.832012
      6.832012
      14.963155
      8.559098
      1.5
      NaN
      1.409027
      8.503160
      7.467509

      7
      4
      fishing
      0.429942
      1
      1.443708
      1
      hammock
      NaN
      0.95
      -1
      0.641398
      -0.358602
      5.609535
      6.282250
      2.5
      NaN
      3.943708
      9.014635
      5.337817

      8
      4
      hammock
      0.216153
      1
      -0.298857
      1
      hammock
      NaN
      0.95
      -1
      0.322463
      -0.677537
      2.612370
      3.463061
      1.5
      NaN
      1.201143
      3.663504
      2.591958

      9
      4
      hammock
      7.604617
      1
      -0.666748
      1
      fishing
      11.344756
      0.95
      -1
      11.344756
      10.344756
      10.344756
      0.000000
      1.5
      NaN
      0.833252
      0.833252
      0.000000

      1
      0
      0
      hammock
      1.247475
      1
      -2.189594
      1
      fishing
      1.247475
      0.95
      -1
      1.247475
      0.247475
      18.712048
      19.436393
      1.5
      NaN
      -0.689594
      16.177899
      17.755256

      1
      1
      fishing
      0.802887
      1
      0.931514
      1
      hammock
      NaN
      0.95
      -1
      0.887327
      -0.112673
      17.588881
      18.633215
      2.5
      NaN
      3.431514
      19.542528
      16.958962

      2
      1
      hammock
      0.213228
      1
      1.803811
      1
      hammock
      NaN
      0.95
      -1
      0.235653
      -0.764347
      14.893510
      16.481955
      1.5
      NaN
      3.303811
      17.508248
      14.952039

      3
      1
      hammock
      1.824770
      1
      0.998793
      1
      hammock
      NaN
      0.95
      -1
      2.016682
      1.016682
      14.671037
      14.373005
      1.5
      NaN
      2.498793
      14.820062
      12.969757

      4
      1
      hammock
      0.602507
      1
      0.905523
      1
      hammock
      NaN
      0.95
      -1
      0.665873
      -0.334127
      11.258946
      12.203234
      1.5
      NaN
      2.405523
      12.794276
      10.935529

We can inspect Robinson's decisions period by period.

```python
fig, ax = plt.subplots()

df.groupby("Period").Choice.value_counts(normalize=True).unstack().plot.bar(
    stacked=True, ax=ax
)

plt.xticks(rotation="horizontal")

plt.legend(loc="lower center", bbox_to_anchor=(0.5, -0.275), ncol=2)

plt.show()
plt.close()
```

We can also analyze the persistence in decisions.

```python
data = pd.crosstab(df.Lagged_Choice_1, df.Choice, normalize=True)
sns.heatmap(data, cmap="Blues", annot=True)
```

## Analysis

We now study how Robinson's behavior changes as we increase the returns to experience. We do so by plotting the average level of final experience in the sample under the different parameterizations.

This analysis of the comparative statics of the model is straightforward to implement. In models of educational choice, this type of analysis is often applied to evaluate the effect of alternative tuition policies on average educational attainment. See Keane & Wolpin (1997, 2001) for example. The basic structure of the analysis remains the same.

```python
# Specification of grid for evaluation
num_points = 15
grid_start = 0.0
grid_stop = 0.3

grid_points = np.linspace(grid_start, grid_stop, num_points)

rslts = list()
for value in grid_points:

    params.loc["wage_fishing", "exp_fishing"] = value

    df = simulate(params)

    stat = df.groupby("Identifier")["Experience_Fishing"].max().mean()
    rslts.append(stat)
```

We collected all results in `rslts` and are ready to create a basic visualization.

```python
fig, ax = plt.subplots()

ax.plot(grid_points, rslts)

ax.set_ylim([0, 10])
ax.set_xlabel("Return to experience")
ax.set_ylabel("Average final level of exerience")

plt.show()
plt.close()
```

In the absence of any returns to experience, Robinson still spends more than two periods fishing. This share then increases with the return. Starting at around 0.2, Robinson spends all his time fishing.

## Extension

Let us make the model more interesting!

> At some point Crusoe notices that a group of cannibals occasionally visits the island and celebrate one of their dark rituals. But then, a prisoner can escape and becomes Crusoe's new friend Friday whom he teaches English. In return Friday can share his knowledge once to help Robinson improve his fishing skills, but that is only possible after Robinson tried at least once to go fishing.

A common extension to structural models is to increase the choice set. Here, we want to add another choice called `"friday"` which affects the utility of fishing. The choice should be available once, starting with the third period, and only after Robinson has been fishing before.

Note that, we load the example models with the function, `rp.get_example_model`. The command for the former model is `params, options, df = rp.get_example_model("robinson_crusoe_basic")`. You can use `with_data=False` to suppress the automatic simulation of a sample with this parameterization.

```python
params, options = rp.get_example_model(
    "robinson_crusoe_extended", with_data=False
)
```

At first, take a look at the parameterization. There is a new positive parameter called `"contemplation_with_friday"` which enters the wage equation of fishing. The choice `"friday"` itself has a negative constant utility term which models the effort costs of learning and the food penalty. The variance-covariance matrix is also adjusted.

```python
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      contemplation_with_friday
      0.40

      nonpec_fishing
      constant
      -1.00

      nonpec_friday
      constant
      -1.00

      not_fishing_last_period
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_friday
      1.00

      sd_hammock
      1.00

      corr_friday_fishing
      0.00

      corr_hammock_fishing
      0.00

      corr_hammock_friday
      0.00

      lagged_choice_1_hammock
      constant
      1.00

Turning to the `options`, we can see that the new covariate `"contemplation_with_friday"` is only affecting utility if Robinson is experienced in fishing and only for one interaction with friday. This naturally limits the interaction with Friday. The key `"inadmissible_states"` can be used to restrict the choice Friday to the third and following periods. The first key matches a choice. The value of the key can be a list of strings. If the string evaluates to `True`, a utility penalty ensures that individuals will never choose the corresponding states. There exist some states in the state space which will never be reached because choices are mutually exclusive or are affected by other restrictions. Filters under `"core_state_space_filters"` can be used to purge those states from the state space, reducing runtime and memory consumption.

```python
options
```

    {'n_periods': 10,
     'estimation_draws': 200,
     'estimation_seed': 500,
     'estimation_tau': 0.001,
     'interpolation_points': -1,
     'simulation_agents': 1000,
     'simulation_seed': 132,
     'solution_draws': 500,
     'solution_seed': 456,
     'covariates': {'constant': '1',
      'contemplation_with_friday': 'exp_friday == 1 and exp_fishing >= 1',
      'not_fishing_last_period': "lagged_choice_1 != 'fishing'"},
     'inadmissible_states': {'friday': ['period < 2', 'exp_fishing == 0']},
     'core_state_space_filters': ["period > 0 and exp_fishing + exp_friday == period and lagged_choice_1 == 'hammock'",
      'period <= 2 and exp_friday != 0',
      'period >= 3 and period - exp_friday < 2',
      'exp_friday > 0 and exp_fishing == 0',
      "exp_friday > 0 and exp_fishing == 1 and lagged_choice_1 == 'fishing'",
      "period - exp_friday == 2 and lagged_choice_1 != 'friday' and period > 2",
      "exp_{i} == 0 and lagged_choice_1 == '{i}'"]}

Now, let us simulate a sample of the new model.

```python
simulate = rp.get_simulate_func(params, options)
```

```python
df = simulate(params)
```

```python
fig, ax = plt.subplots()

df.groupby("Period").Choice.value_counts(normalize=True).unstack().plot.bar(
    stacked=True, ax=ax, color=["C0", "C2", "C1"],
)

plt.xticks(rotation="horizontal")

plt.legend(loc="lower center", bbox_to_anchor=(0.5, -0.275), ncol=3)

plt.show()
plt.close()
```

## Estimation

To estimate model parameters via maximum likelihood, ``respy`` relies on [``estimagic``](https://github.com/OpenSourceEconomics/estimagic), an open-source tool to estimate structural models and more. That way, ``respy`` only has to implement the likelihood function of your model, the optimization and standard error calculation is done by ``estimagic``.

Unlike other optimization libraries, ``estimagic`` does not optimize over a simple vector of parameters, but instead stores parameters in a ``pd.DataFrame``, which makes it easier to parse them into the quantities we need, store lower and upper bounds together with parameters and express constraints on the parameters.

For ``estimagic``, we need to pass constraints on the parameters in a list containing dictionaries. Each dictionary is a constraint. A constraint includes two components: First, we need to tell ``estimagic`` which parameters we want to constrain. This is achieved by specifying an index location which will be passed to `df.loc`. Then, define the type of the constraint. Here, we only impose the constraint that the shock parameters have to be valid variances and correlations.

Optionally, we can add a column ``"group"`` which is identical to the category column. The ``estimagic`` dashboard will then contain one parameter convergence plot per group instead of plotting all parameters in the same figure. Since ``respy`` has quite many parameters, this will make the plots much more readable.

```python
from estimagic.optimization.optimize import maximize
```

```python
params["group"] = params.index.get_level_values("category")

crit_func = rp.get_crit_func(params, options, df)
crit_func(params)

constr = rp.get_parameter_constraints("robinson_crusoe")
```

```python
results, params = maximize(
    crit_func,
    params,
    "scipy_L-BFGS-B",
    algo_options={"maxfun": 1},
    constraints=constr,
    dashboard=False,
)
```

If we hadn't limited the optimization to just one function evaluation, ``params`` would contain the estimated parameters and results would contain additional information on the optimization. If we had set ``dashboard=True``, the call to maximize would have opened a browser window with the beautiful ``estimagic`` dashboard. Try it out if you run this notebook locally.

## Footnotes

1
    One of the earliest references of Robinsonades in economics can be found in Marx (1867). In the 37th footnote, he mentions that even Ricardo used the theme before him.

## References

> Keane, M. P. and Wolpin, K. I. (1997). [The Career Decisions of Young Men](https://doi.org/10.1086/262080). *Journal of Political Economy*, 105(3): 473-522.

> Marx, K. (1867). Das Kapital, Bd. 1. *MEW*, Bd, 23, 405


# **Introducción a Python**

- Una de las multiples maneras de acceder a python es mediante Anaconda, donde se puede obtener algunas interfaces para manejar este programa. [vinculo para descargar anaconda](https://www.anaconda.com/products/individual)

```python
import pandas as pd
import numpy as np
```

## **Instalación de paquetes**

- La instalación de paquetes en python se puede realizar de multiples maneras. La más sencilla de estas es a través del comando "pip install nombre_paquete".

## **Numpy (Manejo de marices)**

- Definición de vectores como numpy arrays

```python
dato=np.array([1,2,3]) ##definir un vector
dato1=np.arange(2, 10) ##definir un vector con números del 2 al 10
v2 = np.ones(3)
v2
```

    array([1., 1., 1.])

- Operaciones con vectores

```python
##sumas de vectores
a = np.array([2, 6, 9, 10])
b = np.array([2, 4, 9, 11])
c=a+b

```

```python
##resta de vectores
a-b
```

    array([ 0,  2,  0, -1])

```python
##Multiplicación de vectores por un escalar
3*b
```

    array([ 6, 12, 27, 33])

```python
##producto scalar
np.dot(a,b)
```

    219

- Definición de matrices como numpy arrays

```python
c = np.array([[5, 3],
              [1, 0]])

d = np.array([[1, 0],
              [7, 5]])
c
```

    array([[5, 3],
           [1, 0]])

```python
identidad=np.eye(3) ##crear matriz identidad 3*3
diagonal=np.diag([1, 2, 3]) ##definir matriz 3*3 con diagonal definida
ceros=np.zeros((2, 3)) ## matriz de ceros 2*3
```

- Operaciones de matrices

```python
##sumas de matrices
c+d
```

    array([[6, 3],
           [8, 5]])

```python
## Multiplicación de matrices
c.dot(d)
##np.dot(c,d)
```

    array([[26, 15],
           [ 1,  0]])

```python
## Imprimir dimensiones de una matriz
c.shape
```

    (2, 2)

```python
##elementos totales de una matriz
c.size
```

    4

```python
##inversa de una matriz
np.linalg.inv(c)
```

    array([[ 1.11022302e-17,  1.00000000e+00],
           [ 3.33333333e-01, -1.66666667e+00]])

```python
## Determinante de una matriz
np.linalg.det(d)
```

    4.999999999999998

```python
## Traspuesta de una matriz
##c.T
np.transpose(c)
```

    array([[5, 1],
           [3, 0]])

### **¿Cómo calcular MCO con numpy?**

- Para esto primero vamos a simular los datos de nuestras variables dependientes e independientes. Para esto debemos instalar la libreria random

```python
import random
```

```python
## Definimos una semilla para que nuestros resultados sean replicables
np.random.seed(9876789)
```

- Ahora vamos a simular las variables, de distribuciones normales con mu y sigmas definidos, y agregar un vector de unos para calcular el intercepto.

```python
X=np.random.normal(3, 2.5, size=(7000, 9))
Y=np.random.normal(40, 250, size=(7000, 1))
v = np.ones((7000, 1))
X1 = np.c_[v, X]
```

- Teniendo ya nuestra data simulada podemos aplicar la expresión analítica de los estimadores de MCO
\begin{equation}
\hat{\beta}=(X'X)^{-1}X'y
\end{equation}

```python
beta_estimados = (np.linalg.inv(X1.T.dot(X1))).dot(X1.T.dot(Y))
```

```python
beta_estimados
```

    array([[35.05108084],
           [-0.24294684],
           [ 0.65479819],
           [-0.17125465],
           [-1.07571486],
           [-1.64156742],
           [ 0.86446528],
           [ 0.93448394],
           [ 0.85726027],
           [ 0.21346173]])

- Podemos verificar este resultado con diferentes librerias que ya tienen dentro de ellas funciones para calcular los estimadores de MCO, en este caso lo haremos como la libreria [Spreg.ols](https://spreg.readthedocs.io/en/latest/generated/spreg.OLS.html)

```python
pip install spreg.OLS
```

    Note: you may need to restart the kernel to use updated packages.

    ERROR: Could not find a version that satisfies the requirement spreg.OLS (from versions: none)
    ERROR: No matching distribution found for spreg.OLS

```python
from spreg import OLS
ols=OLS(Y, X1)
ols.betas
```

    array([[35.05108084],
           [-0.24294684],
           [ 0.65479819],
           [-0.17125465],
           [-1.07571486],
           [-1.64156742],
           [ 0.86446528],
           [ 0.93448394],
           [ 0.85726027],
           [ 0.21346173]])

```python
print(ols.summary)
```

    REGRESSION
    ----------
    SUMMARY OF OUTPUT: ORDINARY LEAST SQUARES
    -----------------------------------------
    Data set            :     unknown
    Weights matrix      :        None
    Dependent Variable  :     dep_var                Number of Observations:        7000
    Mean dependent var  :     36.3666                Number of Variables   :          10
    S.D. dependent var  :    251.5773                Degrees of Freedom    :        6990
    R-squared           :      0.0007
    Adjusted R-squared  :     -0.0006
    Sum squared residual:442682355.377                F-statistic           :      0.5128
    Sigma-square        :   63330.809                Prob(F-statistic)     :      0.8665
    S.E. of regression  :     251.656                Log likelihood        :  -48624.011
    Sigma-square ML     :   63240.336                Akaike info criterion :   97268.023
    S.E of regression ML:    251.4763                Schwarz criterion     :   97336.559

    ------------------------------------------------------------------------------------
                Variable     Coefficient       Std.Error     t-Statistic     Probability
    ------------------------------------------------------------------------------------
                CONSTANT      35.0510808      11.2599339       3.1129029       0.0018600
                   var_1      -0.2429468       1.2156946      -0.1998420       0.8416100
                   var_2       0.6547982       1.2061414       0.5428868       0.5872251
                   var_3      -0.1712547       1.1872226      -0.1442481       0.8853087
                   var_4      -1.0757149       1.2044647      -0.8931062       0.3718311
                   var_5      -1.6415674       1.1981245      -1.3701142       0.1706953
                   var_6       0.8644653       1.2144026       0.7118441       0.4765851
                   var_7       0.9344839       1.1955210       0.7816541       0.4344443
                   var_8       0.8572603       1.1936583       0.7181789       0.4726710
                   var_9       0.2134617       1.1990665       0.1780233       0.8587099
    ------------------------------------------------------------------------------------
    Warning: One variable has been removed for being constant.

    REGRESSION DIAGNOSTICS
    MULTICOLLINEARITY CONDITION NUMBER           10.423

    TEST ON NORMALITY OF ERRORS
    TEST                             DF        VALUE           PROB
    Jarque-Bera                       2           1.696           0.4282

    DIAGNOSTICS FOR HETEROSKEDASTICITY
    RANDOM COEFFICIENTS
    TEST                             DF        VALUE           PROB
    Breusch-Pagan test                9          14.049           0.1206
    Koenker-Bassett test              9          13.538           0.1397
    ================================ END OF REPORT =====================================

## **Listas**

```python
ff=lista[2:4]
```

```python
##string
ff='456'
```

```python
lista=['Rafael','Ignacio','Eduard']
```

```python
lista=[1,3,4,5,3]
```

```python
lista=[1,3,4,5,3]
#lista['Rafael','andres'] lista compuesta de strings
```

```python
len(lista)
##lista[1:3]
##lista[-1] ultimo elemento de la lista
```

    3

```python
lista.remove(lista[1]) ## Es analogo a lista.pop(1)
```

```python
import random
```

```python
random.shuffle(lista) ##organiza aleatoriamente la lista
```

```python
lista
```

    ['Rafael', 'Eduard', 'Ignacio']

```python
lista.append('Camila')
```

```python
lista
```

    ['Rafael', 'Eduard', 'Ignacio', 'Camila']

```python
pd.DataFrame(lista)
```

      0

      0
      Rafael

      1
      Eduard

      2
      Ignacio

      3
      Camila

## **Pandas**

**Importar bases de datos**
- Datos sacados de [kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data?select=train.csv)

```python
data=pd.read_csv('train.csv')
##data=pd.read_excel('nombre archivo.xlsx', sheet_name='pagina1') para leer una página de un archivo de excel, si es solo una página se borra el "sheet_name"
##data=pd.read_pickle('nobre archivo.pkl') para leer archivos tipo pickle

```

**Visualización bases de datos**

```python
data.head() ## muestra las primeras 5 filas de la base de datos
##data.tail() muestra las ultimas 5 filas
##data.columns muestra el nombre de todas las columnas de la base de datos
```

      Id
      MSSubClass
      MSZoning
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      ...
      PoolArea
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice

      0
      1
      60
      RL
      65.0
      8450
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      2
      2008
      WD
      Normal
      208500

      1
      2
      20
      RL
      80.0
      9600
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      5
      2007
      WD
      Normal
      181500

      2
      3
      60
      RL
      68.0
      11250
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      9
      2008
      WD
      Normal
      223500

      3
      4
      70
      RL
      60.0
      9550
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      2
      2006
      WD
      Abnorml
      140000

      4
      5
      60
      RL
      84.0
      14260
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      12
      2008
      WD
      Normal
      250000

5 rows × 81 columns

**¿Cómo ver las dimensiones de la base de datos?**

```python
data.shape
```

    (1460, 81)

**Sintaxis para crear una nueva columna a partir de otras columnas**

```python
data['precio_m2']=data['SalePrice']/data['LotArea']
```

**Cortes en la base de datos**
1. Cortes por filas
2. Cortes por columnas
3. Cortes por una condición dada
4. Tomar un dato de una columna y fila específica

```python
data[20:100] ##muestra unicamente los primeros 100 registros
```

```python
data[['SalePrice','LotArea','precio_m2']]##muestra unicamente las columnas seleccionadas dentro de los corchetes
```

      SalePrice
      LotArea
      precio_m2

      0
      208500
      8450
      24.674556

      1
      181500
      9600
      18.906250

      2
      223500
      11250
      19.866667

      3
      140000
      9550
      14.659686

      4
      250000
      14260
      17.531557

      ...
      ...
      ...
      ...

      1455
      175000
      7917
      22.104332

      1456
      210000
      13175
      15.939279

      1457
      266500
      9042
      29.473568

      1458
      142125
      9717
      14.626428

      1459
      147500
      9937
      14.843514

1460 rows × 3 columns

```python
data[data['precio_m2']>data['precio_m2'].mean()]##muestra aquellos inmuebles donde el precio por m2 sea mayor al primedio
```

      Id
      MSSubClass
      MSZoning
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      ...
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice
      precio_m2

      0
      1
      60
      RL
      65.0
      8450
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      2
      2008
      WD
      Normal
      208500
      24.674556

      6
      7
      20
      RL
      75.0
      10084
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      8
      2007
      WD
      Normal
      307000
      30.444268

      8
      9
      50
      RM
      51.0
      6120
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      4
      2008
      WD
      Abnorml
      129900
      21.225490

      11
      12
      60
      RL
      85.0
      11924
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      7
      2006
      New
      Partial
      345000
      28.933244

      13
      14
      20
      RL
      91.0
      10652
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      8
      2007
      New
      Partial
      279500
      26.239204

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1451
      1452
      20
      RL
      78.0
      9262
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      5
      2009
      New
      Partial
      287090
      30.996545

      1452
      1453
      180
      RM
      35.0
      3675
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      5
      2006
      WD
      Normal
      145000
      39.455782

      1454
      1455
      20
      FV
      62.0
      7500
      Pave
      Pave
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      10
      2009
      WD
      Normal
      185000
      24.666667

      1455
      1456
      60
      RL
      62.0
      7917
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      8
      2007
      WD
      Normal
      175000
      22.104332

      1457
      1458
      70
      RL
      66.0
      9042
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      GdPrv
      Shed
      2500
      5
      2010
      WD
      Normal
      266500
      29.473568

509 rows × 82 columns

**Documentación de iloc** : [iloc](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html)

```python
data['precio_m2'].iloc[0]
```

    24.674556213017752

**Agrupación (groupby)**

```python
data.groupby('Neighborhood').agg({'precio_m2':'mean'})
##data.groupby('Neighborhood').agg({'precio_m2':'mean','SalePrice','median'})
##data.groupby('Neighborhood').agg({'id':'count'})
##data.groupby(['Neighborhood','SaleCondition']).agg({'precio_m2':'mean'})

```

      precio_m2

      Neighborhood

      Blmngtn
      57.643934

      Blueste
      86.410256

      BrDale
      58.323157

      BrkSide
      17.500370

      ClearCr
      13.780805

      CollgCr
      21.393421

      Crawfor
      21.043650

      Edwards
      15.475815

      Gilbert
      18.575884

      IDOTRR
      13.179700

      MeadowV
      45.394765

      Mitchel
      16.430881

      NAmes
      15.086940

      NPkVill
      49.577688

      NWAmes
      16.505557

      NoRidge
      25.668164

      NridgHt
      31.426916

      OldTown
      17.036698

      SWISU
      18.630789

      Sawyer
      13.966735

      SawyerW
      19.014902

      Somerst
      35.461897

      StoneBr
      36.074463

      Timber
      18.386003

      Veenker
      18.081011

1. **Renombrar columnas**
2. **Eliminar columnas**
3. **Eliminar filas donde hay datos nulos en columnas determinadas**
4. **Eliminar duplicados en una columna**

```python
data.rename(columns={'precio_m2':'price_m2'})
```

      Id
      MSSubClass
      MSZoning
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      ...
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice
      price_m2

      0
      1
      60
      RL
      65.0
      8450
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      2
      2008
      WD
      Normal
      208500
      24.674556

      1
      2
      20
      RL
      80.0
      9600
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      5
      2007
      WD
      Normal
      181500
      18.906250

      2
      3
      60
      RL
      68.0
      11250
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      9
      2008
      WD
      Normal
      223500
      19.866667

      3
      4
      70
      RL
      60.0
      9550
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      2
      2006
      WD
      Abnorml
      140000
      14.659686

      4
      5
      60
      RL
      84.0
      14260
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      12
      2008
      WD
      Normal
      250000
      17.531557

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1455
      1456
      60
      RL
      62.0
      7917
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      8
      2007
      WD
      Normal
      175000
      22.104332

      1456
      1457
      20
      RL
      85.0
      13175
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      MnPrv
      NaN
      0
      2
      2010
      WD
      Normal
      210000
      15.939279

      1457
      1458
      70
      RL
      66.0
      9042
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      GdPrv
      Shed
      2500
      5
      2010
      WD
      Normal
      266500
      29.473568

      1458
      1459
      20
      RL
      68.0
      9717
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      4
      2010
      WD
      Normal
      142125
      14.626428

      1459
      1460
      20
      RL
      75.0
      9937
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      6
      2008
      WD
      Normal
      147500
      14.843514

1460 rows × 82 columns

```python
data.drop(columns=['MSZoning','price_m2'])
```

```python
data=data.dropna(subset=['price_m2'])
```

      Id
      MSSubClass
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      LotConfig
      ...
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice
      precio_m2

      0
      1
      60
      65.0
      8450
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      NaN
      NaN
      0
      2
      2008
      WD
      Normal
      208500
      24.674556

      1
      2
      20
      80.0
      9600
      Pave
      NaN
      Reg
      Lvl
      AllPub
      FR2
      ...
      NaN
      NaN
      NaN
      0
      5
      2007
      WD
      Normal
      181500
      18.906250

      2
      3
      60
      68.0
      11250
      Pave
      NaN
      IR1
      Lvl
      AllPub
      Inside
      ...
      NaN
      NaN
      NaN
      0
      9
      2008
      WD
      Normal
      223500
      19.866667

      3
      4
      70
      60.0
      9550
      Pave
      NaN
      IR1
      Lvl
      AllPub
      Corner
      ...
      NaN
      NaN
      NaN
      0
      2
      2006
      WD
      Abnorml
      140000
      14.659686

      4
      5
      60
      84.0
      14260
      Pave
      NaN
      IR1
      Lvl
      AllPub
      FR2
      ...
      NaN
      NaN
      NaN
      0
      12
      2008
      WD
      Normal
      250000
      17.531557

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1455
      1456
      60
      62.0
      7917
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      NaN
      NaN
      0
      8
      2007
      WD
      Normal
      175000
      22.104332

      1456
      1457
      20
      85.0
      13175
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      MnPrv
      NaN
      0
      2
      2010
      WD
      Normal
      210000
      15.939279

      1457
      1458
      70
      66.0
      9042
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      GdPrv
      Shed
      2500
      5
      2010
      WD
      Normal
      266500
      29.473568

      1458
      1459
      20
      68.0
      9717
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      NaN
      NaN
      0
      4
      2010
      WD
      Normal
      142125
      14.626428

      1459
      1460
      20
      75.0
      9937
      Pave
      NaN
      Reg
      Lvl
      AllPub
      Inside
      ...
      NaN
      NaN
      NaN
      0
      6
      2008
      WD
      Normal
      147500
      14.843514

1460 rows × 81 columns

```python
data=data[data['Id'].duplicated()==False] ## Crea una base donde no hay duplicados en la variable Id
##data=data[data.duplicated()==False]
```

- Ver estadísticas descriptivas de variables continuas

```python
data[['SalePrice','LotFrontage','LotArea','FullBath','TotRmsAbvGrd','GarageCars']].describe()
```

      SalePrice
      LotFrontage
      LotArea
      FullBath
      TotRmsAbvGrd
      GarageCars

      count
      1460.000000
      1201.000000
      1460.000000
      1460.000000
      1460.000000
      1460.000000

      mean
      180921.195890
      70.049958
      10516.828082
      1.565068
      6.517808
      1.767123

      std
      79442.502883
      24.284752
      9981.264932
      0.550916
      1.625393
      0.747315

      min
      34900.000000
      21.000000
      1300.000000
      0.000000
      2.000000
      0.000000

      25%
      129975.000000
      59.000000
      7553.500000
      1.000000
      5.000000
      1.000000

      50%
      163000.000000
      69.000000
      9478.500000
      2.000000
      6.000000
      2.000000

      75%
      214000.000000
      80.000000
      11601.500000
      2.000000
      7.000000
      2.000000

      max
      755000.000000
      313.000000
      215245.000000
      3.000000
      14.000000
      4.000000

**A continuación vamos a ver como hacer algunas gráficas con ayuda de la libreria [matplotlib](https://matplotlib.org/)**

```python
##pip install matplotlib
import matplotlib.pyplot as plt
```

- El primer ejemplo que vamos a ver es como realizar histogramas, en este caso vamos a ver como se distribuye los precios de los inmuebles para distintos tamaños de lotes

```python
## &=y o=|
```

```python
b=data[(data['LotArea']<7553.500000)]
a=data[(data['LotArea']>7553.500000)&(data['LotArea']<9478.500000)]
c=data[(data['LotArea']>9478.500000)]
```

```python
c
```

      Id
      MSSubClass
      MSZoning
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      ...
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice
      precio_m2

      1
      2
      20
      RL
      80.0
      9600
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      5
      2007
      WD
      Normal
      181500
      18.906250

      2
      3
      60
      RL
      68.0
      11250
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      9
      2008
      WD
      Normal
      223500
      19.866667

      3
      4
      70
      RL
      60.0
      9550
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      2
      2006
      WD
      Abnorml
      140000
      14.659686

      4
      5
      60
      RL
      84.0
      14260
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      12
      2008
      WD
      Normal
      250000
      17.531557

      5
      6
      50
      RL
      85.0
      14115
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      NaN
      MnPrv
      Shed
      700
      10
      2009
      WD
      Normal
      143000
      10.131066

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1448
      1449
      50
      RL
      70.0
      11767
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      GdWo
      NaN
      0
      5
      2007
      WD
      Normal
      112000
      9.518144

      1453
      1454
      20
      RL
      90.0
      17217
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      7
      2006
      WD
      Abnorml
      84500
      4.907940

      1456
      1457
      20
      RL
      85.0
      13175
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      MnPrv
      NaN
      0
      2
      2010
      WD
      Normal
      210000
      15.939279

      1458
      1459
      20
      RL
      68.0
      9717
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      4
      2010
      WD
      Normal
      142125
      14.626428

      1459
      1460
      20
      RL
      75.0
      9937
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      NaN
      NaN
      NaN
      0
      6
      2008
      WD
      Normal
      147500
      14.843514

730 rows × 82 columns

```python
fig, ax = plt.subplots()
kwargs = dict(histtype='stepfilled', alpha=0.3,  bins=40)

ax.hist(b['SalePrice'], **kwargs)
ax.hist(c['SalePrice'], **kwargs)
ax.hist(a['SalePrice'], **kwargs)

fig.savefig("histogramas.png")
```

```python
fig, ax = plt.subplots()
ax.scatter(data['LotArea'],data['SalePrice'], c="g", alpha=0.5, marker=r'$\clubsuit$',
            label="Luck")
ax.set(xlabel='Area lote', ylabel='Precio Inmueble',
       title='Area Lote vs precio inmueble')
ax.grid()
fig.savefig("scatter.png")
```

**Guardar bases de datos**

```python
##data.to_excel('nombre_archivo.xlsx')
##data.to_csv('nombre_archivo.csv')
##data.to_pickle('nombre_archivo.pkl')
```

## **Functional forms and Box-Cox**

```python
data=pd.read_csv('train.csv')
```

- Supose we want to estimate a lin-lin linear regresion that relates the sale price of an aparment and its area and garage area

```python
from sklearn import linear_model
import statsmodels.api as sm
from scipy import stats
X = data[['LotArea','GarageArea']]
Y = data['SalePrice']
regr = linear_model.LinearRegression()
regr.fit(X, Y)
```

    LinearRegression()

```python
regr.intercept_
```

    63221.052205943124

```python
regr.coef_
```

    array([  1.24534492, 221.15741677])

- Supose you want to estimate this particular model but in a log-lin form. What would be this model interpretation?

```python
data['log_SalePrice'] = np.log2(data['SalePrice'])
```

```python
X = data[['LotArea','GarageArea']]
Y = data['log_SalePrice']
regr1 = linear_model.LinearRegression()
regr1.fit(X, Y)
```

    LinearRegression()

```python
regr1.coef_
```

    array([8.34899429e-06, 1.68408062e-03])

- What about the log-log form?. What would be this model interpretation?

```python
data=data.dropna(subset=['LotArea','GarageArea'])
data=data[(data['LotArea']!=0)&(data['GarageArea']!=0)]
```

```python
data['log_LotArea'] = np.log2(data['LotArea'])
data['log_GarageArea'] = np.log2(data['GarageArea'])
```

```python
X = data[['log_LotArea','log_GarageArea']]
Y = data['log_SalePrice']
regr2 = linear_model.LinearRegression()
regr2.fit(X, Y)
```

    LinearRegression()

```python
regr2.coef_
```

    array([0.16373529, 0.54881774])

- **Box Cox Transformation**: Let´s say you are suspicious about  the distribution of the variable `SalePrice`, so you want to use Box-Cox transformation to ensure a distribution that looks more like a normal distribution. Remeber that boxcox transformation finds the $\lambda$ that maximazes the log likelihood, and uses this lambda to perform the following transformation of a given variable:
$$ \begin{eqnarray}
x(\lambda) & = &  \frac{x^\lambda -1}{\lambda}  \,\,\,\,\, & \lambda \ne 0 \\
x(\lambda) & = & \mbox{ln}(x) & \lambda = 0 \, .
\end{eqnarray} $$

```python
data=data.dropna(subset=['SalePrice'])
data=data[(data['SalePrice']!=0)]
```

```python
t, Max_MLE = stats.boxcox(data['SalePrice'])
```

```python
Max_MLE
```

    -0.07692391328663316

```python
fig = plt.figure()
ax = fig.add_subplot(111)
prob = stats.boxcox_normplot(data['SalePrice'], -1, 1, plot=ax)
ax.axvline(Max_MLE, color='r')
```

```python
Y=data['SalePrice']**lmax_mle
X = data[['LotArea','GarageArea']]

regr3 = linear_model.LinearRegression()
regr3.fit(X, Y)
```

    LinearRegression()

```python
regr3.coef_
```

    array([-9.41114039e-08, -2.05785132e-05])

## While Statement, it's highly recommended to use recursion instead!

```python
u=0
la=1
p=0
while u==0:
    la+=p
    p+=1
    if p==10:
        u=1

```

## **For loops**
- Un for loop es un procedimiento donde se le aplica una función determinada a unos objetos determinados. Estos objetos pueden ser los objetos que constituyen una lista, un DataFrame, etc. Un ejemplo que permite entender claramente la dinámica de un for loop es una sumatoria.El siguiente ejemplo muestra como resolver mediante un loop la siguiente operación:
$ \sum \limits _{i=1} ^{10} i$

```python
a=0
for i in range(1,11):
    a=a+i
    print(a)

```

    1
    3
    6
    10
    15
    21
    28
    36
    45
    55

En el anterior ejemplo se ve como se puede usar el comando range para delimitar el rango por el cual se quiere iterar. El límite derecho de este rango no se toma. Ahora, supongamos que tenemos una lista de strings de donde queremos sacar un substring, en este caso tenemos string que contiene dentro un precio el cual queremos tener como un integrer. Este tipo de procedimientos es muy frecuente en la construcción de bases de datos

```python
from tqdm import tqdm
lista=['precio:12.222','precio:55.000','precio:77.223','cio:89.545','precio:123.656','fhdjs','precio:99.878']
precios=[]
for i in tqdm(lista):
    try:
        precios.append(int(i.split(':')[1].replace('.','')))
    except:pass
precios
```

    100%|██████████| 7/7 [00:00<?, ?it/s]

    [12222, 55000, 77223, 89545, 123656, 99878]

- En este ejemplo quise ejemplificar el uso del comando **tqdm**. Este comando permite ver cuanto tiempo se demora cada iteración. En el ejemplo anterior no es muy relevante pues las iteraciones son rapidas y son pocas, pero esto puede ser muy util a la hora de tener procesos largos para estimar el tiempo que estos tomaran. Por otro lado, también se muestra el uso del comando **try**. Este es usado cuando en una de las iteraciones no se puede llevar a cabo el procedimiento establecido, con el fin de que el proceso general no se detenga
- En el siguiente ejemplo quiero mostrar porque el uso de for loops no es muy eficiente, por lo cual debe intentarse buscar una solución alternativa de ser posible, aunque no siempre lo es. En los casos donde no lo es, recomiendo usar paralelismo, sobre todo trabajando bases de datos grandes, tema del que hablare levemente la clase que viene

```python
import random
```

```python
pp=[]
hh=[]
for i in range(0,100000):
    pp.append(random.uniform(200,1000))
    hh.append(random.uniform(200,1000))
```

-Aquí tienen un ejemplo de como convertir listas de la misma longitud en un DataFrame

```python
d={'pp':pp,'aa':hh}
data=pd.DataFrame(d)
```

```python
%%time
data['yy']=data['aa']+data['pp']
```

    Wall time: 8 ms

```python
%%time
ej=[]
for i in range(0,len(pp)):
    ej.append(pp[i]+hh[i])
```

    Wall time: 26.9 ms

El anterior ejemplo pretendía generar una lista que fuera el resultado de la suma de dos listas en cada posición. Como vemos, Pandas logra resolver este problema mucha más rápido, al convertirlo en una suma de vectores.

En algunos casos es más eficiente solucionar problemas para los cuales se quiera iterar por medio **recursión**. [Dejo este enlace para quien esté interesado](https://hackernoon.com/recursion-vs-looping-in-python-9261442f70a5)

## **Funciones**

```python
def Fahrenheit_Celsius(lista,x):
    traduccion=[]
    if x=='f':
        for i in lista:
            traduccion.append((i-32)* 5/9)
    elif x=='c':
        for i in lista:
            traduccion.append(( i* 9/5) + 32 )
    return traduccion

```

```python
l=[100,150,200]
a=Fahrenheit_Celsius(l,'c')

```

**Función para calcular los estimadores de MCO**

```python
def MCO(data, Y, X):
    la=X.append(Y)
    data=data.dropna(subset=X)
    Y1=data[Y].to_numpy()
    X1=data[X].to_numpy()
    l=Y1.shape[0]
    v = np.ones((l, 1))
    X2 = np.c_[v, X1]
    beta_estimados = np.linalg.inv(X2.T @ X2) @ X2.T @ Y1
    return beta_estimados
```

```python
a=MCO(data,'SalePrice',['LotFrontage','LotArea','FullBath','TotRmsAbvGrd','GarageCars'])
```

## **Diccionarios**

Los diccionarios son un tipo de dato con el cual se puede relacionar un **key** con su imagen. Son utiles cuando se quiere relacionar una palabra con un valor en específico. Por ejemplo, si se quiere relacionar algunas personas con su edad, se podría pensar en algo como lo siguiente.

```python
data={}
data['Rafael']=23
data['Daniela']=44
data['Jesus']=33
```

```python
data
```

    {'Rafael': 23, 'Daniela': 44, 'Jesus': 33}

```python
data['Rafael']
```

    23

Los diccionarios no permiten acceder a objetos dentro de ellos por medio de la posición de este objeto en el diccionario, como si ocurre con las listas. Tampoco permiten duplicados.

```python
data[0]
```

Los diccionarios son muy utiles para ciertos procesos en programación, como veremos más adelante. Un tip que puede ser de utilidad es que pandas tiene programado en si una manera de convertir una lista de diccinarios en un DataFrame. Para mostrar esto, generare aleatorioamente una serie de diccinarios con características de algunas personas

```python
%%time
datas=[]
a=0
for i in tqdm(range(0,10000)):
    a+=1
    data={}
    data['id']=a ##Es importante tener un identificador unico por cada individuo
    data['altura']=random.normalvariate(1.66, 0.3)##se genera una observación aleatoria de una distribución normal con media y varianza definidas
    data['peso']=random.normalvariate(56, 0.3)
    datas.append(data)

```

    100%|██████████| 10000/10000 [00:00<00:00, 357707.90it/s]

    Wall time: 30.9 ms

```python
data=pd.DataFrame(datas)
```

```python
data
```

      id
      altura
      peso

      0
      1
      1.232615
      55.430352

      1
      2
      1.684467
      55.915907

      2
      3
      1.458465
      55.582554

      3
      4
      1.949240
      55.967801

      4
      5
      1.331605
      56.000718

      ...
      ...
      ...
      ...

      9995
      9996
      1.582933
      56.500257

      9996
      9997
      1.694079
      55.741432

      9997
      9998
      2.000832
      56.054465

      9998
      9999
      2.077467
      56.059562

      9999
      10000
      1.404221
      56.326264

10000 rows × 3 columns

## **Progamación en paralelo**

```python
from multiprocessing.dummy import Pool
from multiprocessing import cpu_count
```

- Un ejemplo de cuando podría ser de utilidad un usar programación en paralelo es cuando se tiene un proceso iterativo, en el cual cada iteración no requiere la totalidad de la capacidad de procesamiento de la maquina en la cual corre. En este caso, pensar en poder realizar varias de estas iteraciones al mismo tiempo, o en paralelo, podría hacer los procesos mucho más eficientes. En esta clase vamos a ver una manera de aplicar programación el paralelo con la función pool.map() de la libreria multiprocessing

```python
##Este comando permite ver la cantidad de cpu´s de la máquina, la libreria recomienda que el número de cpu´s sea igual al número procesos paralelos
cpu_count()
```

    12

```python
strings=[]
a=0
li=random.uniform(200,1000)
for i in range(0,1000000):
    a+=0
    if a==li:
        li=random.uniform(200,1000)
        a=0
        strings.append('sdfasdf')
    else:
        strings.append('precio:'+' '+str(random.uniform(200,1000)))
```

- Retomemos el ejemplo de más arriba donde queriamos extraer un precio de un string que lo contiene, pero ahora tenemos 1 millón de observaciones, por lo cual iterar puede resultar ineficiente. Para realizar el procesamiento en paralelo en este caso se debe:1) construir una función que

```python
precio=[]
def extraer_precio(x):
    global strings,precio
    for i in strings[x:x+100000]:
        try:
            precio.append(int(i.split(':')[1].replace('.','')))
        except:pass
```

```python
f=[]
o=0
for i in range(0,10):
    f.append(0)
    o+=100000
pool=Pool(10)
paralelo=pool.map(extraer_precio,f)
```

- Let's say we want to estimate a linear regression with every row in our data set, and save the coeficients of every regression in a list

```python
data=pd.read_csv('train.csv')
```

```python
data=data.dropna(subset=['SalePrice','LotArea','GarageArea'])
```

```python
la=[]
for i in tqdm(data.index):
    b=data.iloc[[0]]
    Y=data['SalePrice']**lmax_mle
    X = data[['LotArea','GarageArea']]
    regr0 = linear_model.LinearRegression()
    regr0.fit(X, Y)
    la.append(regr0.coef_)

```

    100%|██████████| 1460/1460 [00:02<00:00, 496.32it/s]

- What if we were using "BigData"?

```python
data.groupby('LotShape').count()
```

```python
r=data.groupby('LotShape').count().index.to_list()
```

```python
def paralel2(string):
    global la,data
    data=data[data['LotShape']==str(string)]
    for i in tqdm(data.index):
        b=data.iloc[[0]]
        Y=data['SalePrice']**lmax_mle
        X = data[['LotArea','GarageArea']]
        regr0 = linear_model.LinearRegression()
        regr0.fit(X, Y)
        la.append(regr0.coef_)
```

```python
pool=Pool(4)
re=pool.map(paralel2,r)
```

      0%|          | 0/41 [00:00<?, ?it/s]
      0%|          | 0/484 [00:00<?, ?it/s]

      0%|          | 0/10 [00:00<?, ?it/s]

     44%|████▍     | 18/41 [00:00<00:00, 176.94it/s]
    100%|██████████| 10/10 [00:00<00:00, 97.35it/s]s]

     88%|████████▊ | 36/41 [00:00<00:00, 150.84it/s]]
    100%|██████████| 41/41 [00:00<00:00, 146.82it/s]]

      3%|▎         | 32/925 [00:00<00:05, 154.11it/s]
     11%|█▏        | 55/484 [00:00<00:02, 168.45it/s]

      5%|▌         | 49/925 [00:00<00:05, 159.24it/s]
     16%|█▌        | 78/484 [00:00<00:02, 190.97it/s]

      7%|▋         | 65/925 [00:00<00:05, 156.09it/s]
     21%|██        | 100/484 [00:00<00:01, 199.87it/s]

      9%|▉         | 87/925 [00:00<00:04, 178.01it/s]
     26%|██▌       | 124/484 [00:00<00:01, 212.20it/s]

     11%|█▏        | 105/925 [00:00<00:04, 175.74it/s]
     31%|███       | 150/484 [00:00<00:01, 225.63it/s]

     14%|█▎        | 125/925 [00:00<00:04, 182.38it/s]
     36%|███▌      | 174/484 [00:00<00:01, 228.34it/s]

     16%|█▌        | 147/925 [00:00<00:04, 193.20it/s]
     41%|████      | 197/484 [00:01<00:01, 192.02it/s]

     18%|█▊        | 167/925 [00:00<00:04, 178.00it/s]
     45%|████▌     | 218/484 [00:01<00:01, 190.85it/s]

     20%|██        | 188/925 [00:01<00:03, 186.21it/s]
     49%|████▉     | 238/484 [00:01<00:01, 187.53it/s]

     23%|██▎       | 209/925 [00:01<00:03, 192.29it/s]
     53%|█████▎    | 258/484 [00:01<00:01, 190.04it/s]

     25%|██▍       | 229/925 [00:01<00:03, 190.80it/s]
     58%|█████▊    | 281/484 [00:01<00:01, 200.81it/s]

     27%|██▋       | 249/925 [00:01<00:03, 189.77it/s]
     62%|██████▏   | 302/484 [00:01<00:00, 199.94it/s]

     29%|██▉       | 270/925 [00:01<00:03, 193.31it/s]
     67%|██████▋   | 323/484 [00:01<00:00, 197.34it/s]

     31%|███▏      | 291/925 [00:01<00:03, 196.59it/s]
     71%|███████   | 343/484 [00:01<00:00, 190.67it/s]

     34%|███▍      | 313/925 [00:01<00:03, 201.80it/s]
     75%|███████▌  | 364/484 [00:01<00:00, 194.07it/s]

     36%|███▌      | 334/925 [00:01<00:02, 202.59it/s]
     79%|███████▉  | 384/484 [00:01<00:00, 194.25it/s]

     38%|███▊      | 355/925 [00:01<00:02, 203.73it/s]

     41%|████      | 376/925 [00:01<00:02, 203.34it/s]
     83%|████████▎ | 404/484 [00:02<00:00, 189.01it/s]
     88%|████████▊ | 424/484 [00:02<00:00, 190.65it/s]

     43%|████▎     | 399/925 [00:02<00:02, 209.49it/s]
     92%|█████████▏| 445/484 [00:02<00:00, 195.79it/s]

     45%|████▌     | 420/925 [00:02<00:02, 207.97it/s]
     97%|█████████▋| 469/484 [00:02<00:00, 208.18it/s]

    100%|██████████| 484/484 [00:02<00:00, 195.08it/s]

     50%|████▉     | 462/925 [00:02<00:02, 199.84it/s]

     55%|█████▍    | 505/925 [00:02<00:01, 266.05it/s]

     58%|█████▊    | 532/925 [00:02<00:01, 237.08it/s]

     61%|██████    | 562/925 [00:02<00:01, 252.87it/s]

     64%|██████▍   | 596/925 [00:02<00:01, 276.46it/s]

     69%|██████▉   | 638/925 [00:02<00:00, 316.46it/s]

     73%|███████▎  | 678/925 [00:03<00:00, 338.80it/s]

     78%|███████▊  | 722/925 [00:03<00:00, 367.39it/s]

     83%|████████▎ | 765/925 [00:03<00:00, 383.77it/s]

     87%|████████▋ | 809/925 [00:03<00:00, 399.42it/s]

     92%|█████████▏| 853/925 [00:03<00:00, 409.32it/s]

    100%|██████████| 925/925 [00:03<00:00, 253.41it/s]

- ¿cómo podemos aplicar esto para correr la función de mco mostrada más arriba paralelamente, donde cada proceso paralelo corra la función para una parte de la data?

```python

```


# Due Dec 5, 11:59 PM
Programming Assignment
15 points
Shovon Bhowmik Nov 17 (Edited Nov 17)
Problems that has to be solved using python or matlab:

1.
 * Make an excel sheet of the datas given for the correlation example in class.
 * Calculate correlation coefficient and find the relationship between the datas.
 * Also compare the result with the defined function used by python or matlab for establishing correlation between datas.
 * Also visualize the graphical representation of the datas.

2.
 * Collect any dataset with two variables x and y from any site. **The dataset must be atleast of 100 items.**
 * Find the predicted y value based on x using **simple linear regression equation.**

 Then find

 1. SSR,
 2. SSE
 and
 3. Coefficient of determination by calculating difference between the actual y value and the predicted y value.
 4. Finally show the regression line based on your result graphically and find the accuracy of your model's performance.

N. B:

1) You all are requested to submit the problems in the following link:
https://drive.google.com/drive/folders/1qreLOzd0iN3ye4JfZ9S_O9O5qzTMnD-N?usp=sharing
Upload your zip file which will be named after your roll number and there will be two files inside the zip file. Both files will be in .py or .mat format

2) If I find same code, the code that will come second or next will be not accepted and he or she will get no marks in the assignment.

Please knock me if there is any question regarding the assignment.
Thank you.

```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

```

DATASET
> https://archive.ics.uci.edu/ml/datasets/student+performance

```python
df = pd.read_csv('student-mat.csv', delimiter=';')
```

```python
df.head()
```

      school
      sex
      age
      address
      famsize
      Pstatus
      Medu
      Fedu
      Mjob
      Fjob
      ...
      famrel
      freetime
      goout
      Dalc
      Walc
      health
      absences
      G1
      G2
      G3

      0
      GP
      F
      18
      U
      GT3
      A
      4
      4
      at_home
      teacher
      ...
      4
      3
      4
      1
      1
      3
      6
      5
      6
      6

      1
      GP
      F
      17
      U
      GT3
      T
      1
      1
      at_home
      other
      ...
      5
      3
      3
      1
      1
      3
      4
      5
      5
      6

      2
      GP
      F
      15
      U
      LE3
      T
      1
      1
      at_home
      other
      ...
      4
      3
      2
      2
      3
      3
      10
      7
      8
      10

      3
      GP
      F
      15
      U
      GT3
      T
      4
      2
      health
      services
      ...
      3
      2
      2
      1
      1
      5
      2
      15
      14
      15

      4
      GP
      F
      16
      U
      GT3
      T
      3
      3
      other
      other
      ...
      4
      3
      2
      1
      2
      5
      4
      6
      10
      10

5 rows × 33 columns

```python
mydf = df.iloc[ : , -3:-1]
```

```python
# G1 = Grade1
# G2 = Grade2
#
mydf.head()
```

      G1
      G2

      0
      5
      6

      1
      5
      5

      2
      7
      8

      3
      15
      14

      4
      6
      10

```python
dataX = mydf.iloc[ : , 0:1]
dataY = mydf.iloc[ : , 1:]
```

```python
dataX.head()
```

      G1

      0
      5

      1
      5

      2
      7

      3
      15

      4
      6

```python
dataY.head()
```

      G2

      0
      6

      1
      5

      2
      8

      3
      14

      4
      10

```python

```

```python

```

```python

```

```python

```

\begin{equation} r = ( sum((Xi - meanX) * (Yi - meanY) ) / sqrt( sum( squre( Xi - meanX) * squre(Yi - meanY) ) )) \end{equation}

```python
r = ( np.sum( (dataX.values - dataX.values.mean()) * (dataY.values - dataY.values.mean()) ) / np.sqrt( np.sum( np.square(dataX.values - dataX.values.mean()) ) * np.sum( np.square(dataY.values - dataY.values.mean()) ) ) )
```

```python
r
```

    0.8521180663648062

```python
meanX = dataX.values.mean()
```

```python
meanY = dataY.values.mean()
```

```python
sumOfMultiplyOfMeanXY = np.sum((dataX.values - meanX) * (dataY.values-meanY))
```

```python
meanX
```

    10.90886075949367

```python
meanY
```

    10.713924050632912

```python
sumOfMultiplyOfMeanXY
```

    4191.701265822785

```python
np.sum( (dataX.values - dataX.values.mean()) * (dataY.values - dataY.values.mean()))
```

    4191.701265822785

```python
np.sqrt( np.sum( np.square(dataX.values - dataX.values.mean()) ) * np.sum( np.square(dataY.values - dataY.values.mean()) ) )
```

    4919.15549180276

```python
sumOfSqureOfX = np.sum(np.square(dataX.values - dataX.values.mean()))
```

```python
sumOfSqureOfY = np.sum(np.square(dataY.values - dataY.values.mean()))
```

```python
sqrtOfSSXY = np.sqrt( sumOfSqureOfX * sumOfSqureOfY )
```

```python
sqrtOfSSXY
```

    4919.15549180276

```python
check_r = sumOfMultiplyOfMeanXY / sqrtOfSSXY
```

```python
check_r
```

    0.8521180663648062

# $$y = a + bx$$

## $$b = r * (sdY/sdX)$$

## $$a = meanY - b * meanX$$

```python
b = r * (np.std(dataY.values) / dataX.values.std())
```

```python
b
```

    0.9656698067869525

```python
np.std(dataY.values)
```

    3.756740243888906

```python
dataY.values.std()
```

    3.756740243888906

```python
a = dataY.values.mean() - ( b * dataX.values.mean())
```

```python
a
```

    0.1795665887468907

```python
def Yprediction(x):
    return (a + (b * x))

```

```python
# Calculating Predicted Y
predictedY = dataX.apply(Yprediction)

# print(predictedY.head())?
# print(dataY.head())

```

```python

```

```python

```

```python
r == check_r
```

    True

```python
print(type(dataY), type(predictedY))

```

    <class 'pandas.core.frame.DataFrame'> <class 'pandas.core.frame.DataFrame'>

```python
# >>> d = {'col1': [1, 2], 'col2': [3, 4]}
# >>> df = pd.DataFrame(data=d)
# >>> df

testdf = dataY
testdf['G2_Predicted'] = predictedY
```

```python

testdf.head()
```

      G2
      G2_Predicted

      0
      6
      5.007916

      1
      5
      5.007916

      2
      8
      6.939255

      3
      14
      14.664614

      4
      10
      5.973585

```python

```

```python

```

```python
# type(testdf['G2_Predicted'].values)
```

```python
# SSR = np.sum( testdf['G2_Predicted'].values, testdf['G2'].values.mean() )
# testdf['G2'].values.mean()
# testdf['G2_Predicted'].values
# np.sum( np.square( testdf['G2_Predicted'].values, testdf['G2'].values.mean() ) )

# np.sum(np.square(testdf['G2_Predicted'].values - testdf['G2'].values.mean()))
```

```python
SSR = np.sum(np.square( np.subtract(testdf['G2_Predicted'].values , testdf['G2'].values.mean()) ))
```

```python
SSE = np.sum(np.square( np.subtract(testdf['G2'].values , testdf['G2_Predicted'].values.mean()) ))
```

```python
CoeffOfDetermination = (SSR / SSE)
```

```python
CoeffOfDetermination
```

    0.7261051990252965

```python
np.sqrt(CoeffOfDetermination)
```

    0.8521180663648064

```python
r
```

    0.8521180663648062

```python

```

```python

```

```python

```

```python
plt.scatter(df.G1, df.G2)
plt.plot(df.G1, testdf[['G2_Predicted']])
plt.show()
```

```python
from sklearn import linear_model

reg = linear_model.LinearRegression()
```

```python
reg.fit(df[['G1']], df.G2)

```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
y_cap = reg.predict(df[['G1']])
reg.coef_
```

    array([0.96566981])

```python
# %matplotlib inline

plt.scatter(df.G1,df.G2,color='red', marker='o')
plt.plot(df.G1,y_cap)
plt.show()
```

```python

```

# Mean absolute error(MAE)

$$MAE = \frac{1}{n} \sum_{}\left\lvert{y- \hat{y}}\right\rvert$$

# Mean Squared Error

$$MSE = \frac{1}{n} \sum_{}{(y- \hat{y})^2}$$

# Root Mean Squared Error

$$RMSE = \sqrt{ \frac{1}{n} \sum_{}{(y- \hat{y})^2}}$$

```python
# testdf.G2 - testdf.G2_Predicted
```

```python
MAE = ( (1/len(testdf.G2)) * np.sum(abs(np.subtract(testdf.G2 , testdf.G2_Predicted))) )
```

```python
MAE
```

    1.3026637719467193

```python
YY = testdf.G2
YY.head()
YYpredict = testdf.G2_Predicted
YYpredict.head()
# ( (1 / len(YY)) * ( np.sum(np.abs(np.subtract(YY,YYp)))) )
MSE = ( (1 / len(YY)) * ( np.sum(np.square(np.subtract(YY,YYpredict)))) )
```

```python
MSE
```

    3.8655039651792578

```python
RMSE = np.sqrt(MSE)
```

```python
RMSE
```

    1.966088493730447

```python
from sklearn.metrics import mean_squared_error

from math import sqrt

rmse = sqrt(mean_squared_error(YY, YYpredict))

print(rmse)
```

    1.9660884937304468

```python

```

```python

```

```python

```

```python

```

```python

```


```python
from sympy.physics.continuum_mechanics.beam import Beam
from sympy import symbols
```

# Workbook 03

# Task A

1. Use the "file opening template" to open "beam_1.txt" with the `.read()` method and assign the file data to a variable `beam_data`
2. Turn the file data into a list of lines using `.split()`
3. Loop through each of the lines in the file and print them to the notebook

## Task A response

```python
with open("beam_1.txt", 'r') as file:
    beam_data = file.read()

beam_lines = beam_data.split("\n")

for line in beam_lines:
    print(line)
```

    4800, 19200, 1000000
    0, 3000
    100, 2000, 0

# Task B

```python
lines = [
    "4800, 19200, 1000000",
    "0, 3000",
    "100, 500, 0",
    "200, 3600, 0",
]
```

1. Use the "transforming" recipe to loop through all of the numbers in just the first line of `lines` (remember, you will need to split the line up first). Convert each number string into an `int` and save each in your accumulator.
2. Do the same thing with the second line, the beam support locations.
3. Now, use the "double loop" recipe to loop through 1) the lines of the loads, then 2) the items in each line (see Lesson 02 reference if you need a refresher on how to slice a list to start at the third line and go to the end). Convert each load parameter into an `int` and accumulate it into a list of lists of ints.
4. Print the results of each accumulator to the notebook so you can see if it was successful or not.

## Task B response

```python
l_e_i = []
for item in lines[0].split(", "):
    l_e_i.append(int(item))

supports = []
for support in lines[1].split(", "):
    supports.append(int(support))

outer_acc = []
for load_line in lines[2:]:
    inner_acc = []
    for param in load_line.split(", "):
        inner_acc.append(int(param))
    outer_acc.append(inner_acc)

print(l_e_i)
print(supports)
print(outer_acc)
```

    [4800, 19200, 1000000]
    [0, 3000]
    [[100, 500, 0], [200, 3600, 0]]

# Task C

```python
L_E_I = [4800, 19200, 80000]
supports = [0, 2000, 4800]
loads = [
    [100, 500, 0],
    [200, 3600, 0],
]
```

Using the extracted data above (`L_E_I`, `supports`, `loads`), perform the following:

_Hint_: Use the "unpacking" syntax to turn the elements of the list into the arguments of the `Beam` (see Lesson 03 Reference).

1. Create a new `Beam` object with the elements of `L_E_I`
2. Using the "transforming" recipe, loop through `loads` and add each load to the beam with `.apply_load()`.
3. Using the "enumeration" recipe, loop through `supports` and perform all of the steps required to add the supports to the beam:
    1. Create a sympy symbol to represent each reaction. Use the enumerator to create symbols that look like this, `"R0"`, `"R1"`, `"R2"`, etc.
    2. Add the reaction symbol to your accumulator
    3. Use `.apply_load()` to add the symbol and support location as a point load support
    4. The beam object has an _attribute_ that we have been using called `.bc_deflection`. This attribtue is an empty list until we put things into it. We can `.append()` to it just like an accumulator. Create the boundary condition as a tuple (like a list but with parentheses) with the support location and `0` to indicate no deflection.
    5. After the loop, call `.solve_for_reaction_loads()` with the reaction symbols in your accumulator passed as the arguments.

## Task C response

```python
beam_1 = Beam(*L_E_I)
reactions = []
for idx, support in enumerate(supports):
    R = symbols(f"R{idx}")
    beam_1.apply_load(R, support, -1)
    beam_1.bc_deflection.append((support, 0))
    reactions.append(R)

for load in loads:
    beam_1.apply_load(*load)

beam_1.solve_for_reaction_loads(*reactions)
print(beam_1.reaction_loads)
beam_1.plot_shear_force()
beam_1.plot_bending_moment()
```

```python

```

# Beam Analysis: Beam File Reader Script

Using the steps that you have practiced in **Task A**, **Task B**, and **Task C**, write a new script to read a beam file, transforms the string data into integers, and accumulates them into lists similar to those seen in **Task C**. Input the data accumulated into the lists into a new beam object. Solve for the reaction loads, print the reaction loads, and plot the shear, moment, and deflection diagrams.

## Task D response

```python
file_name = 'beam_2.txt'

with open(file_name, 'r') as file:
    b1_data = file.read()

b1_lines = b1_data.split("\n")

l_e_i = []
for item in b1_lines[0].split(", "):
    l_e_i.append(int(item))

supports = []
for item in b1_lines[1].split(", "):
    supports.append(int(item))

loads = []
for load in b1_lines[2:]:
    load_acc = []
    for load_param in load.split(", "):
        load_acc.append(int(load_param))
    loads.append(load_acc)

beam = Beam(*l_e_i)

for load in loads:
    beam.apply_load(*load)

reactions = []
for idx, support in enumerate(supports):
    R = symbols(f"R{idx}")
    reactions.append(R)
    beam.apply_load(R, support, -1)
    beam.bc_deflection.append((support, 0))

beam.solve_for_reaction_loads(*reactions)
print(beam.reaction_loads)
beam.plot_shear_force()
beam.plot_bending_moment()
```

# Task E

1. Create a new variable at the top of your **Task D** script called `file_name` that holds the string of the file name to analyze. Replace your `with open()...` line so that it opens the variable, `file_name` instead.
2. Change out the file names with `beam_1.txt`, `beam_2.txt`, and `beam_3.txt` to see the analysis results change.
4. Rename your **Task D** to **Beam Analysis: Beam File Reader Script**
3. Create a new `beam_4.txt` file and put in all new values representing a new beam to analyze. Note that you have to include values for `E` and `I` in the file format for your script to read it properly. What happens if you don't include `E` and `I` in the first line of the file? Try writing some more files and running them! Did any of your experiments "break" your script or cause an error? If so, why do you think that is?
4. You have now written a script that will read in a series of beam parameters from a text file, solve the beam, and plot the results. However, this script has some limitations, including:
    1. It can only read one beam per file
    2. It can only handle point load reactions
    3. It can only handle deflection boundary conditions

Do you have any ideas on what we would need to do in order for it to be able to read multiple beams in one file with any kind of support conditions?

That will be what we work on next week!