```python
from decodes.core import *
from decodes.io.jupyter_out import JupyterOut
out = JupyterOut.unit_square( )
```

# Coordinate Systems

Our experience as users of CAD has trained us to look at coordinate systems in a particular way. We understand that they are generally made up of **three axes**, and are visually represented by **three vectors** that are **mutually perpendicular**.

Most CAD packages maintain a single **world coordinate system** that defines the underlying space of the model, but also allows **local coordinate systems** defined at arbitrary positions and orientations within this world space which are typically positioned for convenient drafting or modeling.

Our experience in CAD suggests an intimate link between coordinate systems, vectors, and the space they inhabit. In this section, we establish the mathematical foundations that unpacks this relationship: by using two critical concepts:

* **span**: the spatial extent that may be “reached” using a set of vectors and a limited means of manipulation.

* **basis**: any minimal set of vectors which spans a given space.

## Basis and Coordinates

We already know that we may add and scalar multiply any set of vectors $\vec{v_{1}},\ \vec{v_{2}},\ \ldots \ , \vec{v_{n}}$, in any order of operation, and that these operations will result in another vector. If the scalars are known, then the vector

\begin{align}
\vec{w} = c_{1}\vec{v_{1}} \ + \ c_{2}\vec{v_{2}} \ + \ \ldots \ c_{2}\vec{v_{2}}
\end{align}

can be drawn easily using the “head-to-tail” method. This procedure is called a linear combination of $\vec{v_{1}},\ \vec{v_{2}},\ \ldots \ , \vec{v_{n}}$.

Consider for the moment **all possible linear combinations**. In other words, given some set of vectors, imagine all the locations that can be “reached” by vectors resulting from a combination of both addition and scalar multiplication. Mathematically, this is termed the **span** of $\vec{v_{1}},\ \vec{v_{2}},\ \ldots \ , \vec{v_{n}}$.

To develop our intuition for what the vector span looks like, we might consider three two-dimensional cases:

* The span of a single vector
* The span of any two (non-paralell) vectors
* The span of three (mutually non-paralell) vectors

The span of a single vector $\vec{v}$ is the set of scalar multiples $c\vec{v}$. Since the scalars can be positive or negative, we may scale this vector to reach out infinitely in either direction.

***The span of $\vec{v}$, then, is the entire line through the origin determined by $\vec{v}$***.

Assuming $\vec{v_{1}}$ and $\vec{v_{2}}$ are not parallel, we find that any linear combination $c2\vec{v_{1}} + c2\vec{v_{2}}$ may be produced by applying the parallelogram rule and adjusting the scalar multipliers.

The span of *any* two vectors in the plane allow us to reach any location in two dimensions, so we say that the span of this set is the ***entirety of the plane***.

The third case, more than two vectors, is not necessary. There is only one parallelogram with sides in the directions of v1 and v2 for which the far corner coincides with the desired vector.

This second case is special.

For any two non-parallel vectors:
* their span is the entire space
* any other vector in the space can be expressed as a unique linear combination of this set of vectors.

Any set of vectors that exhibits both of these properties is called a **basis** for the space, and ***the number of vectors*** in a basis is the **dimension** of the space.

The most standard of bases in R2 contains the vectors (1,0) and (0,1). Notice that any vector (x,y) is easily expressed as a linear combination of these basis vectors. Here we find a formal mathematical
account of a common instrument in CAD software, that of ***coordinates***.

### Frames

Just as two perpendicular unit vectors make up a special basis for two-dimensions, three mutually perpendicular unit vectors is a special basis in three dimensions. A coordinate system in three dimensions made up of such a basis, together with an origin, is called a **frame**.

Further refining this term, an **orthonormal basis** is one in which the contained vectors are unit vectors, and are mutually perpendicular. This is a particularly convenient format in computational geometry.

\begin{align}
\vec{w} = L_{1}\vec{u_{1}} + L_{2}\vec{u_{2}} + L_{3}\vec{u_{3}}
\end{align}

**span**

The spatial extent to which a given set of vectors may “reach” by application
of vector addition and scalar multiplication.

**basis**

A set of vectors that spans the entire space in which they are described,
with a number of vectors in the set that matches the dimension of the
space. A basis in R2 must contain two vectors, and a basis in R3 contains
three. A basis can be used to express any desired vector in the space as a
unique linear combination of basis vectors.

**orthonormal basis**

A special kind of basis comprised of a set of mutually perpendicular unit
vectors. Orthonormal bases are prevalent in computational geometry, as
a basis in this format makes the evaluation of coordinates particularly
convenient.

**frame**

A coordinate system in three dimensions made up of an orthonormal
basis (with three vectors) together with a position in space. Throughout
this text, we will refer to the mathematical concept as a frame, and the
implementation in code or in software as a coordinate system.

**coordinate system**

A frame implemented in software or in code. Frames implemented in these
contexts go by many names, including “construction plane” (Rhino), “user
coordinate system” (AutoCAD), “drawing axis” (SketchUp), and, fittingly,
“frame” (Grasshopper).

## CS Objects in Decodes

    *CS Members*

    `cs.origin`
    Point
    The local origin of this coordinate system.

    `cs.x_axis`
`cs.y_axis`
`cs.z_axis`
    Vec
    Vectors that represent the axes of this coordinate system. Constrained upon construction to ensure orthonormality.

```python
"""
Coordinate System Initialization
Shown here is an initialization by a Point and two Vecs which represent
the desired orientation of the resulting CS. The first of the given vectors
is assigned to the x-axis. The second influences the direction of the
y-axis, but is not used to set it directly as to ensure perpendicularity.
"""
class CS(Geometry):

    def __init__(self, pt, vec_a, vec_b):
        self.origin = pt
        # set the x-axis to the first given vector, normalized
        self.x_axis = vec_a.normalized()
        # set the z-axis to a vector perpendicular to both given vectors
        self.z_axis = self.x_axis.cross(vec_b).normalized()
        # set the y-axis to a vector perpendicular to the x- and z-axes
        self.y_axis = self.z_axis.cross(self.x_axis).normalized()
```

### Coordinate System Evaluation

```python
"""
CS Evaluation
Returns a Point in "world" space that corresponds to the given u,v,w
coordinates that are described in the "local" space of this CS.
"""
def eval(self,u,v,w):
    offset_vec = (self.x_axis*u) + (self.y_axis*v) + (self.z_axis*w)
    return Point(self.origin + offset_vec)

```

### Coordinate System Devaluation

```python
"""
CS Devaluation
Returns a Vec containing coordinates in the "local" space of this CS that
correspond with the given x,y,z coordinates that are described in "world"
space.
"""
def deval(self,x,y,z):
    pt = Point(x,y,z)
    # project the given point onto an axis line, store the distance
    xx = Line(self.origin,self.x_axis).near(pt)[1]
    yy = Line(self.origin,self.y_axis).near(pt)[1]
    zz = Line(self.origin,self.z_axis).near(pt)[1]
    return Vec(xx,yy,zz)

```

103 - Points, Vectors, and Coordinate Systems/100 - Coordinate Systems.ipynb


# Pumas.jl Workshop Exercises
### Chris Rackauckas, Vijay Ivaturi

These exercises teach common workflows which involve Pumas.jl. This exercise
worksheet is meant to be a living document leading new users through a deep dive
of the Pumas.jl feature set. If you have further suggestions or want to contribute
new problems, please open an issue or PR at the
[PumasTutorials.jl](https://github.com/PumasAI/PumasTutorials.jl) repository.

The exercises are described as follows:

- Exercise 1 takes you through simulating a compartmental system in Pumas. The
  user will gain experience in writing a model using the Pumas Domain Specific
  Language (DSL), setting up a dosing regimen and population, simulating the
  model into the designed population and finally plotting the results of the
  simulation.
- Exercise 2 takes the user through performing an Non-compartmental analysis
  (NCA). The user will get familiarized with reading data sets in a spread
  sheet format (e.g, .csv) or as the simulated output as in exercise 1 and
  perform a simple NCA analysis.
- Exercise 3 introduces the user to perform a non-linear mixed effects modeling
  estimation (NLME) on pharmacokinetic data. The user will learn how to read and
  set up the data for NLME estimation, fit the model, infer and inspect the
  results of the model fit.
- Exercise 4 introduces the user to various post-processing steps of model
  fitting, including using the final
  model fits to peform simulation into an alternate dosing regimen or populaion
  of interest.

# Problem 1: Simulate a first-order absorption model with linear elimination after a 100 mg oral dose in 24 subjects

In this poblem, we will walk through the basics of writing a model with Pumas.jl.
The pharmacokinetics after an oral dose are commonly described by a first-order
process. In our example we will use a linear one-compartmental system for
elimination of the drugs.

## Part 1: Understand the system

Let's understand the system by setting up the differential equation or the
analytical solution that describes the systems

$$\begin{align}
\frac{dDepot}{dt} &= -K_a.Depot\\
\frac{dCentral}{dt} &= K_a.Depot - (CL/V).Central\end{align}$$

The analytical form of this equation to calculate the concentration, `Cp` at
any given time can be written as

$$Cp = \frac{F \times Dose \times K_a}{V (K_a-K_{el})}(e^{-K_{el} t} - e^{-K_a t})$$

where the parameters are: `Ka = 1 hr-1`, `CL = 1 L/hr`, `V = 20 L/hr`.

## Part 2: Setup the population

Setting up a dosing regimen and population of subjects. Go through the section on [Generating and simulating populations](https://tutorials.pumas.ai/html/introduction/simulating_populations.html) in the
tutorials to set up a population of 24 subjects that receives 100 mg dose via the mouth and assign a random body weight to each subjects

## Part 3: Write the model

Write up a first-order absorption model where body weight is a covariate on `CL` and `V`.
understand the different steps of writing up a model by referring to the
documentation on [models](https://docs.pumas.ai/dev/basics/models/).

## Part 4: Simulate

Perform the simulation by using the `simobs` function. The details on the use of `simobs` can be seen in the documentation on [simulation](https://docs.pumas.ai/dev/basics/simulation/) and by
looking at one of the [tutorials](https://tutorials.pumas.ai/html/introduction/introduction.html). Finish by plotting the result of the simulation.

# Problem 2: Peform Non-compartmental analysis

Use the dataset generated from Problem 1 that is stored as a CSV file [here]()
to read in the data using the `read_nca` function and
generate a NCA report. You can know more about how to do this by looking at
one of the [NCA tutorials](https://tutorials.pumas.ai/html/nca/basic_nca.html)

# Problem 3: Estimate using Non-linear mixed effects

The same dataset that was read in for NCA analysis will be used for fitting a
NLME model. You can learn more on how to read the data in for NLME estimation
using `read_pumas` by following the tutorial in the
[readme](https://github.com/PumasAI/Pumas.jl/blob/master/README.md#fit) or the
[documentation](https://docs.pumas.ai/dev/basics/doses_subjects_populations/)

## Part 1: Read datasets for NLME estimation

Read the dataset and evaluate the Population

## Part 2: Perform a model fit

Fit the model using FOCEI() estimation

## Part 3: Infer the results

Infer the results of your `fit`

## Part 4: Inspect the results

Inspect the results of your `fit`


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Summary of Properties, Theorems and Transforms

The [properties](properties.ipynb), [theorems](theorems.ipynb) and transforms of the two-sided Laplace transform as derived in the previous sections are summarized in the following. The corresponding tables serve as a reference for the application of the Laplace transform in the theory of signals and systems. Please refer to the respective sections for details.

### Definition

The two-sided Laplace transform and its inverse are defined as

\begin{align}
X(s) &= \int_{-\infty}^{\infty} x(t) \, e^{- s t} \; dt \\
x(t) &= \frac{1}{2 \pi j} \int_{\sigma - j \infty}^{\sigma + j \infty} X(s) \, e^{s t} \; ds
\end{align}

where $s \in \text{ROC} \{ x(t) \}$.

### Properties and Theorems

The properties and theorems of the two-sided Laplace transform are given as

|  | $x(t) \qquad \qquad \qquad \qquad$ | $X(s) = \mathcal{L} \{ x(t) \} \qquad \qquad \qquad$ | ROC $\qquad \qquad \qquad \qquad \qquad \qquad \qquad$ |
|:---|:---:|:---:|:---|
| [Linearity](properties.ipynb#Linearity) | $A \, x_1(t) + B \, x_2(t)$ | $A \, X_1(s) + B \, X_2(s)$ | $\supseteq \text{ROC}\{x_1(t)\} \cap \text{ROC}\{x_2(t)\}$ |
| [Real-valued signal](properties.ipynb#Symmetry-for-Real-Valued-Signals) | $x(t) = x^*(t)$ | $X(s) = X^*(s^*)$ | |
| [Scaling](theorems.ipynb#Temporal-Scaling-Theorem) | $x(a t)$ | $\frac{1}{\lvert a \rvert} X\left( \frac{s}{a} \right)$ | $s: \frac{s}{a} \in \text{ROC}\{x(t)\}$ |
| [Convolution](theorems.ipynb#Convolution-Theorem) | $x(t) * h(t)$ | $X(s) \cdot H(s)$ | $\supseteq \text{ROC}\{x(t)\} \cap \text{ROC}\{h(t)\}$ |
| [Shift](theorems.ipynb#Temporal-Shift-Theorem) | $x(t - \tau)$ | $e^{-s \tau} \cdot X(s)$ | $\text{ROC}\{x(t)\}$ |
| [Differentiation](theorems.ipynb#Differentiation-Theorem) (causal signal) | $\frac{d}{dt} x(t)$ | $s \cdot X(s) - x(0+)$ | $\supseteq \text{ROC}\{x(t)\}$ |
| [Integration](theorems.ipynb#Integration-Theorem) | $\int_{-\infty}^{t} x(t) \; dt$ | $\frac{1}{s} \cdot X(s)$ | $\supseteq \text{ROC}\{x(t)\} \cap \{s: \Re \{s\} > 0 \}$ |
| [Modulation](theorems.ipynb#Modulation-Theorem) | $e^{s_0 t}\cdot x(t)$ | $X(s - s_0)$ | $s: s - \Re \{s_0\} \in \text{ROC}\{x(t)\}$ |

where $A, B, s_0 \in \mathbb{C}$, $a \in \mathbb{R} \setminus \{0\}$ and $\tau \in \mathbb{R}$

### Selected Transforms

Two-sided Laplace transforms which are frequently used are given as

| $x(t) \qquad \qquad \qquad \qquad$ | $X(s) = \mathcal{L} \{ x(t) \} \qquad \qquad \qquad$ | ROC $\qquad \qquad \qquad \qquad \qquad \qquad \qquad$ |
|:---:|:---:|:---|
| $\delta(t)$ | $1$ | $\mathbb{C}$ |
| $\epsilon(t)$ | $\frac{1}{s}$ | $\Re \{s\} > 0$ |
| $t \epsilon(t)$ | $\frac{1}{s^2}$ | $\Re \{s\} > 0$ |
| $e^{- s_0 t} \epsilon(t)$ | $\frac{1}{s + s_0}$ | $\Re \{s\} > \text{Re}\{-s_0\}$ |
| $\sin(\omega_0 t) \epsilon(t)$ | $\frac{\omega_0}{s^2 + \omega_0^2}$ | $\Re \{s\} > 0$ |
| $\cos(\omega_0 t) \epsilon(t)$ | $\frac{s}{s^2 + \omega_0^2}$ | $\Re \{s\} > 0$ |
| $t^n e^{-s_0 t} \epsilon(t)$ | $\frac{n!}{(s+s_0)^{n+1}}$ | $\Re \{s\} > \text{Re}\{-s_0\}$ |
| $e^{-s_0 t} \cos(\omega_0 t) \epsilon(t)$ | $\frac{s + s_0}{(s+s_0)^2 + \omega_0^2}$ | $\Re \{s\} > \Re \{-s_0\}$ |
| $e^{-s_0 t} \sin(\omega_0 t) \epsilon(t)$ | $\frac{\omega_0}{(s+s_0)^2 + \omega_0^2}$ | $\Re \{s\} > \Re \{-s_0\}$ |
| $t \cos(\omega_0 t) \epsilon(t)$ | $\frac{s^2 - \omega_0^2}{(s^2 + \omega_0^2)^2}$ | $\Re \{s\} > 0$ |
| $t \sin(\omega_0 t) \epsilon(t)$ | $\frac{2 \omega_0 s}{(s^2 + \omega_0^2)^2}$ | $\Re \{s\} > 0$ |

where $s_0 \in \mathbb{C}$, $\omega_0 \in \mathbb{R}$ and $n \in \mathbb{N}$. More one- and two-sided transforms may be found in the literature or [online](https://en.wikipedia.org/wiki/List_of_Laplace_transforms).

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


```
!pip install daft seaborn PyDrive
```

    Requirement already satisfied: daft in /usr/local/lib/python3.6/dist-packages (0.0.4)
    Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (0.7.1)
    Collecting PyDrive
      Downloading https://files.pythonhosted.org/packages/52/e0/0e64788e5dd58ce2d6934549676243dc69d982f198524be9b99e9c2a4fd5/PyDrive-1.3.1.tar.gz (987kB)
        100% |████████████████████████████████| 993kB 18.5MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from daft) (1.14.6)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from daft) (2.1.2)
    Requirement already satisfied: google-api-python-client>=1.2 in /usr/local/lib/python3.6/dist-packages (from PyDrive) (1.6.7)
    Requirement already satisfied: oauth2client>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from PyDrive) (4.1.3)
    Requirement already satisfied: PyYAML>=3.0 in /usr/local/lib/python3.6/dist-packages (from PyDrive) (3.13)
    Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from matplotlib->daft) (2018.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->daft) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->daft) (2.3.0)
    Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->daft) (1.11.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->daft) (2.5.3)
    Requirement already satisfied: httplib2<1dev,>=0.9.2 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->PyDrive) (0.11.3)
    Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->PyDrive) (3.0.0)
    Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->PyDrive) (4.0)
    Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->PyDrive) (0.2.2)
    Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->PyDrive) (0.4.4)
    Building wheels for collected packages: PyDrive
      Running setup.py bdist_wheel for PyDrive ... - \ done
      Stored in directory: /root/.cache/pip/wheels/fa/d2/9a/d3b6b506c2da98289e5d417215ce34b696db856643bad779f4
    Successfully built PyDrive
    Installing collected packages: PyDrive
    Successfully installed PyDrive-1.3.1

```
# Prepare the libraries for the exercise
import matplotlib
from matplotlib import pyplot as plt
import daft
import seaborn

import numpy as np
import tensorflow as tf
tf.enable_eager_execution()

import tensorflow_probability as tfp
tfd = tfp.distributions
np.random.seed(5218)
```

#### If you have trouble with mathematic mode in latex, following links are very good for reference!

[Basic](https://en.wikibooks.org/wiki/LaTeX/Mathematics)

[Advance](https://en.wikibooks.org/wiki/LaTeX/Mathematics)

# Question 1

Considering the directed graphical of Gaussian mixture model explained in the lectures. By making use of the d-separation criterion, show that posterior distribution of the latent variables factorizes with respect to the different data points. i.e.

\begin{align}
p(\mathbf{Z} \mid \mathbf{X}, \mathbf{\mu}, \sigma^2, \pi) =
\prod_{n=1}^N p(z_n \mid x_n, \mathbf{\mu}, \sigma^2, \pi)
\end{align}

*Figure 9.6*, page 433, "Pattern Recognition and Machine Learning", M.Bishop

Graphical representation of a Gaussian mixture model for a set of $N$ i.i.d. data points $\{x_n\}$, with corresponding latent points $\{z_n\}$, where $n = 1,...,N$.

# Question 2

Implement Gibbs sampler for Gaussian mixture models as defined in the lectures.

Test it using your previously generated synthetic data.

### The following code is used for generating synthetic data

The `gmm` method will generate 1-D data for a given value of `alpha` and `sigma0`

Your task is to implement Gibbs sampling to infer the value of `alpha` and `sigma0` from returned data `X`

NOTE: only `X` should be used, `Z` is returned only for plotting

```
def gmm(batch_size, n_clusters, alpha, sigma0):
  """ This is the solution for the process in 1.3 (only for 1-D data) """
  # parameters for Dirichlet distribution
  alpha = np.full(shape=(n_clusters,), fill_value=alpha, dtype='float32')

  # step 1: generate the assignment probability
  dirichlet = tfd.Dirichlet(concentration=alpha)
  theta = dirichlet.sample()

  # step 2: generate the centroid for each cluster
  normal_1 = tfd.Normal(loc=[0], scale=sigma0)
  # sampling `n_clusters` time, hence, mean for each cluster
  mu_k = normal_1.sample(n_clusters) # (n_clusters, n_dim)

  # ====== Now for the assignment, need 1 indicator for each
  # examples, hence we need `batch_size` amount of indicator
  # (step: 3(a))====== #
  categorical = tfd.OneHotCategorical(probs=theta)
  z = categorical.sample(batch_size) # (batch_size, n_clusters)
  z = tf.cast(z, tf.bool)

  # ====== sampling the data points (step: 3(b)) ====== #
  normal_2 = tfd.Normal(loc=mu_k, scale=1)
  # this make each draw sample will generate sample for
  # all 4 components
  normal_2 = tfd.Independent(normal_2, reinterpreted_batch_ndims=2)
  x_all_components = normal_2.sample(batch_size) # (batch_size, n_clusters, n_dim)
  # ====== selecting the right component for each sample (step: 3(b)) ====== #
  # (batch_size, n_clusters, n_dim) * (batch_size, n_clusters)
  # = (batch_size, n_dim)
  x = tf.boolean_mask(x_all_components, z)

  # ====== Return: X, Z, mu_k, theta ====== #
  return (x.numpy(),
          np.argmax(z.numpy(), axis=-1),
          mu_k.numpy(),
          theta.numpy())

```

```
X, Z, mu_k, theta = gmm(2500, n_clusters=3, alpha=1, sigma0=40)

plt.figure(figsize=(12, 6))
colors = seaborn.color_palette(palette='Set2', n_colors=len(np.unique(Z)))
plt.subplot(1, 2, 1)
# plotting the scatter points
plt.scatter(np.arange(len(X)), X, c=[colors[int(z)] for z in Z],
            s=4, alpha=0.6)
# plotting the mean
for i, mu in enumerate(mu_k):
  plt.axhline(mu, 0, len(X),
              label=r"$\mu=%.2f;\theta=%.2f$" % (mu_k[i], theta[i]),
              color=colors[i], linestyle='--', linewidth=2)
plt.grid(True)
plt.legend()
plt.subplot(1, 2, 2)
_ = plt.hist(X, bins=80)
```

### Gibbs sampling for GMM

From the above figure, $\mu$ and $\theta$ are the unknown quantities you need to infer using Gibbs sampling

The algorithm in outline:

* sample $\mu$ from $\mu \mid x,z,\theta$
* sample $\theta$ from $\theta \mid x,z,\mu$
* sample $z$ from $z \mid x,\theta,\mu$

# Question 3

Perform Gaussian mixture model inference on the fish data.

Fishes have been kept in different tanks (tankki) and in one tank same type feeding is used.

All fishes are supposed to be same species, but some might be wild and others grown in the tanks.

So can we see two different modes in the length distributions? And are there differences between tanks?

So fit 2-component GMM to fish length (pituus) in each tank (tankki) separately.

Use your favourite GMM estimator, preferably Gibbs sampler (you can get posterior of means).

What do you conclude?

Data is described in the following paper
(Harkonen, L., Hyvarinen, P., Mehtatalo, L. Vainikka, A. 2017. Growth, survival and social learning in the first hatchery generation of Eurasian perch (Perca fluviatilis). Aquaculture 466: 6471. http://dx.doi.org/10.1016/j.aquaculture.2016.09.027. )

### Read the csv dataset from Google Drive

Link to the data from Google Drive:

https://drive.google.com/open?id=1t-vN62A2oO3HkcZKq1cgYu8m5tuooU6-

```
# Here: all the package we need to connect to Google Drive using PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
```

```
# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
# 2. Load a file by ID and print its contents.
downloaded = drive.CreateFile({'id': "1t-vN62A2oO3HkcZKq1cgYu8m5tuooU6-"})
downloaded.GetContentFile('fish.csv')
```

    /usr/local/lib/python3.6/dist-packages/google/colab/auth.py:140: ResourceWarning: unclosed <ssl.SSLSocket fd=93, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.28.0.2', 35234), raddr=('74.125.142.84', 443)>
      if _check_adc():

### Use `pandas` package to load and do data analysis.

Important detail when you are preprocessing the data:

*   Only get sample with MITTAUSAIKA = "LOPETUS"
*  Remove all columns contain NaN (not-a-number) values

```
import pandas as pd

ds = pd.read_csv('fish.csv',
                 sep=";", decimal=',', encoding="Latin-1")
# remove all column that contain NaN values
ids = ds.apply(lambda x: np.all(x.notna()), axis=0)
ds = ds.iloc[:, ids.tolist()]
# we only take sample with "MITTAUSAIKA" = "LOPETUS"
selected_row = ds.MITTAUSAIKA == "LOPETUS"
ds = ds[selected_row]
print(ds.describe())

_ = ds.hist(figsize=(8, 8), bins=25)
_ = seaborn.pairplot(ds)
```

### Fitting GMM model

NOTE: the *curve blue line* is fitted using KDE (kernel density estimation), a non-parametric algorithm for estimating the density of input data.

KDE is different from GMM (a parametric algorithm), and should not be mistaken as GMM.

```
# ====== Getting the tank and fish length data ====== #
# Pituus: length
# Paino : weight
data = ds[['Allas', 'Pituus']]
# ====== grouping the data by the Tank ====== #
from sklearn.mixture import GaussianMixture
n_components = 2
# selecting random color for each component
colors = seaborn.color_palette(
    palette='Set2', n_colors=n_components)
# we need this to draw Gaussian distribution
import matplotlib.mlab as mlab

for pool_id in data.Allas.unique():
  # select all data from given tank
  pool_data = data.Pituus[data.Allas == pool_id]

  # Fitting Gaussian on Pool data
  gmm = GaussianMixture(n_components=int(n_components),
                        covariance_type='diag', n_init=8,
                        random_state=5218)
  # the input data must be at least 2D, so we
  # need to do some preprocessing
  pool_data = np.atleast_2d(pool_data.values).T
  gmm.fit(pool_data)

  # Plotting the histogram
  plt.figure(figsize=(8, 2))
  seaborn.distplot(pool_data, bins=18)

  # Visualizing the GMM
  mean = gmm.means_.ravel()
  precision = gmm.precisions_.ravel()
  xmin, xmax = plt.gca().get_xlim()
  X = np.linspace(start=xmin, stop=xmax,
                  num=1000)
  ax = plt.gca().twinx()
  for n in range(n_components):
    Y = mlab.normpdf(X, mean[n], np.sqrt(1 / precision[n]))
    _, = ax.plot(X, Y,
                 label='Component:%d' % n,
                 color=colors[n], linewidth=3, linestyle='--')

  # show extra info
  ax.set_xlim((np.min(pool_data) - 20,
               np.max(pool_data) + 20))
  plt.legend()
  plt.title("Pool #%s" % str(pool_id))
```


```python
import numpy as np
import keras
import pandas
from keras_tqdm import TQDMNotebookCallback
from sklearn import preprocessing

data = np.array(pandas.read_csv("~/trainingdata.csv", header=0))

print(data.shape)

```

    (72, 7)

```python
from sympy import *
init_printing(use_latex=True)

import matplotlib.pyplot as plt
%matplotlib inline

X = data[:,0:6]
X = preprocessing.scale(X)
print(X.shape)
display(X)

labels = data[:,6]
print(labels.shape)
display(labels)

Y = keras.utils.to_categorical(labels, len(np.unique(labels)))
```

    (72, 6)

    array([[-1.0233435 , -0.37716553, -0.34893403,  0.32330268, -0.05200758,
             0.37994185],
           [-0.7046986 , -0.45616456,  0.97833202, -0.01720953, -0.93794118,
            -0.68779499],
           [-0.66929361, -1.40415296,  1.04153517,  1.00432709, -1.61125071,
            -0.80643241],
           [-0.59848363, -1.32515393,  1.48395719,  0.89082302, -1.18600258,
            -1.6368944 ],
           [-0.74010359, -0.29816649, -0.53854347,  0.32330268, -0.09925737,
             0.6172167 ],
           [ 1.48155945,  0.17582771, -0.58594583,  1.91235964,  0.9717379 ,
             1.12142576],
           [ 1.56122067,  0.41282481, -0.60174661,  1.23133522,  1.18829945,
            -0.80643241],
           [-0.77550858,  0.33382578, -0.79135605, -0.24421767, -0.00475779,
            -0.56915756],
           [ 1.27798076,  0.80781998, -0.55961118, -0.4712258 ,  1.12923721,
             1.09176641],
           [-0.66929361, -0.77216069,  0.67811708, -0.4712258 , -1.56400092,
            -0.68779499],
           [ 1.45500571,  1.28181418,  0.28309742, -1.03874615,  1.71985961,
             1.56631611],
           [-0.66929361, -0.85115973,  1.02046745, -0.4712258 , -1.51675113,
            -0.92506984],
           [ 1.20717079,  1.51881128, -0.34893403, -0.58472987,  1.36548617,
             1.92222839],
           [ 1.95067555,  2.15080355, -0.8545592 , -1.15225021,  0.89298825,
             1.32904126],
           [ 1.39304698,  1.04481708, -0.60174661, -1.37925835,  1.01111273,
             0.97312898],
           [-0.49226867, -1.56215103,  1.73676977,  0.66381488, -0.80800425,
            -0.80643241],
           [-0.59848363, -1.2461549 ,  0.83612495,  1.23133522, -0.85525404,
            -0.80643241],
           [ 1.52581568,  0.49182384, -0.66494976,  2.0258637 ,  0.89298825,
             0.6172167 ],
           [-0.71650026, -0.61416263, -0.66494976,  0.13412923, -0.09925737,
            -0.21324528],
           [-0.66929361, -0.61416263,  1.67356662, -0.24421767, -1.37500175,
            -0.65813563],
           [ 1.38419573,  1.20281515, -0.60174661, -0.58472987,  1.12923721,
             1.09176641],
           [-0.88172354, -0.14016842, -1.23377807, -1.15225021,  0.042492  ,
            -1.10302598],
           [-0.59848363, -1.32515393,  1.10473831,  1.00432709, -0.94975363,
            -0.80643241],
           [-0.49226867, -1.2461549 ,  1.92637921,  0.32330268, -1.18600258,
            -1.51825697],
           [-0.6869961 , -0.45616456, -1.48659065, -0.10233758, -0.00475779,
            -0.21324528],
           [ 1.03899709,  1.61756007, -0.19092617, -1.15225021,  1.54267289,
             2.45609681],
           [-0.63388862, -0.69316166,  1.48395719, -0.35772173, -1.46950133,
            -0.80643241],
           [ 0.99474085,  0.72882095, -0.72815291,  1.79885557,  1.176487  ,
             1.41801933],
           [-0.63388862, -1.40415296,  1.3575509 ,  0.32330268, -1.42225154,
            -0.68779499],
           [-0.53947532, -0.06116939, -0.91776234, -0.95361809, -0.00475779,
             0.26130442],
           [-0.73125234, -0.06116939, -0.34893403,  0.70164957, -0.05200758,
             0.49857928],
           [-0.77550858, -0.77216069,  1.04153517, -0.35772173, -0.80800425,
            -0.68779499],
           [ 1.49041069,  1.04481708,  0.28309742, -0.81173801,  1.83798408,
             1.12142576],
           [-0.480467  ,  0.01782964, -1.04416863, -0.10233758, -0.05200758,
            -0.45052013],
           [-0.7046986 , -0.25866697, -0.91776234, -0.58472987, -0.19375696,
            -0.33188271],
           [-0.6869961 , -0.15991818, -0.53854347, -0.24421767, -0.00475779,
             0.02402957],
           [-0.59848363, -0.69316166,  1.2311446 , -0.24421767, -1.46950133,
            -1.04370727],
           [-0.77550858, -0.06116939, -1.10737178, -0.10233758, -0.09925737,
            -0.21324528],
           [ 1.13636081,  1.43981225,  0.17775885, -1.15225021,  1.48361065,
             0.67653542],
           [ 1.45500571,  1.20281515, -0.66494976, -1.60626649,  1.12923721,
             1.32904126],
           [-1.07055015, -0.14016842, -0.8545592 ,  0.32330268,  0.13699158,
             0.37994185],
           [-0.81091356, -0.15991818, -0.66494976,  0.20979861,  0.08974179,
             0.6172167 ],
           [-0.81091356,  0.01782964, -1.04416863, -0.81173801, -0.09925737,
            -0.45052013],
           [ 1.52581568,  1.28181418, -0.42793796,  2.36637591,  1.05048756,
             0.37994185],
           [-0.66929361, -1.483152  ,  0.83612495,  1.00432709, -1.75300008,
            -0.92506984],
           [-0.77550858, -0.37716553, -0.98096549, -0.10233758, -0.09925737,
             0.142667  ],
           [-0.71650026,  0.23507699, -0.72815291, -0.52797784, -0.05200758,
             0.49857928],
           [ 1.34879074,  1.20281515, -0.77028834, -0.69823394,  1.36548617,
             1.7146129 ],
           [-0.33294622, -0.37716553, -0.98096549, -0.24421767, -0.05200758,
             0.02402957],
           [-0.95253352, -0.15991818, -1.10737178,  0.32330268, -0.00475779,
            -0.56915756],
           [ 1.52581568,  2.15080355, -1.92901267, -0.92524208,  0.30236585,
             1.92222839],
           [ 1.45500571,  0.96581805, -0.34893403,  2.82039218,  1.24736169,
             0.85449156],
           [-0.59848363, -0.65366214,  1.04153517,  0.32330268, -0.94975363,
            -1.16234469],
           [ 1.34879074,  1.91380645, -0.34893403, -1.71977056,  1.176487  ,
             1.09176641],
           [ 0.95933587,  1.04481708,  0.36210135, -0.81173801,  1.77892185,
             0.8248322 ],
           [-0.56307864, -0.45616456,  0.59911315,  0.20979861, -0.71350467,
            -0.80643241],
           [-0.63388862, -1.40415296,  1.15214067,  0.89082302, -1.42225154,
            -0.9547292 ],
           [-0.88172354, -1.16715586,  1.98958235,  0.89082302, -0.66625488,
            -1.04370727],
           [-0.52767366, -1.08815683,  1.54716033, -0.01720953, -0.66625488,
            -0.80643241],
           [ 1.56122067,  0.96581805, -0.60174661,  2.36637591,  0.75123888,
             0.142667  ],
           [-0.66929361, -0.61416263,  1.3575509 , -0.24421767, -1.6585005 ,
            -0.80643241],
           [-0.480467  , -0.25866697, -0.98096549, -0.43339111, -0.05200758,
            -1.6368944 ],
           [-0.64273987, -0.37716553, -0.90196156, -0.81173801,  0.042492  ,
            -1.39961954],
           [-0.74010359, -1.483152  ,  1.67356662,  0.55031081, -1.04425321,
            -1.2018905 ],
           [ 1.34879074,  1.51881128, -0.66494976, -1.26575428,  1.03473763,
             0.6172167 ],
           [-0.59848363, -1.16715586,  1.3575509 ,  0.77731895, -0.99700342,
            -1.04370727],
           [-0.7046986 , -1.32515393,  1.79997291,  0.43680675, -0.71350467,
            -0.92506984],
           [-0.95253352, -0.37716553, -0.34893403,  0.66381488, -0.05200758,
             0.49857928],
           [-0.88172354,  0.17582771, -0.72815291, -0.62256456, -0.09925737,
            -1.16234469],
           [ 1.45500571,  1.59781032, -0.22252774, -2.06028276,  1.41273596,
             0.97312898],
           [-0.88172354,  0.17582771, -0.34893403,  0.32330268, -0.05200758,
             0.26130442],
           [ 1.56122067,  1.20281515, -0.72815291, -1.49276242,  1.07017497,
             1.92222839]])

    (72,)

    array([0., 1., 1., 1., 0., 2., 2., 0., 2., 1., 2., 1., 2., 2., 2., 1., 1.,
           2., 0., 1., 2., 0., 1., 1., 0., 2., 1., 2., 1., 0., 0., 1., 2., 0.,
           0., 0., 1., 0., 2., 2., 0., 0., 0., 2., 1., 0., 0., 2., 0., 0., 2.,
           2., 1., 2., 2., 1., 1., 1., 1., 2., 1., 0., 0., 1., 2., 1., 1., 0.,
           0., 2., 0., 2.])

```python
input_size = X.shape[1]
output_size = Y.shape[1]
display(X.shape[1])
```

```python
model = keras.models.Sequential()

model.add(keras.layers.Dense(100,input_dim=6,activation='relu', bias_initializer=keras.initializers.Constant(value=0.01)))
model.add(keras.layers.Dense(100,input_dim=6,activation='relu', bias_initializer=keras.initializers.Constant(value=0.01)))
model.add(keras.layers.Dense(100,input_dim=6,activation='relu', bias_initializer=keras.initializers.Constant(value=0.01)))

model.add(keras.layers.Dense(3,activation='softmax'))
#binary_crossentropy
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])

print(model.summary())
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_29 (Dense)             (None, 100)               700
    _________________________________________________________________
    dense_30 (Dense)             (None, 100)               10100
    _________________________________________________________________
    dense_31 (Dense)             (None, 100)               10100
    _________________________________________________________________
    dense_32 (Dense)             (None, 3)                 303
    =================================================================
    Total params: 21,203
    Trainable params: 21,203
    Non-trainable params: 0
    _________________________________________________________________
    None

```python
history = model.fit(X, Y,
    batch_size=56,
    epochs=100,
    verbose=0,
    callbacks=[TQDMNotebookCallback()],
    validation_split = 0.25)

```

    HBox(children=(IntProgress(value=0, description='Training', style=ProgressStyle(description_width='initial')),…

    HBox(children=(IntProgress(value=0, description='Epoch 0', max=54, style=ProgressStyle(description_width='init…

```python
plt.figure(1)

plt.subplot(211)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

plt.subplot(212)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.tight_layout()
plt.show()

score = model.evaluate(X, Y, verbose=1)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

```python

```


# Una mirada al concepto de Regresión lineal

###  Tabla de contenido

* [1. Introducción](#Introducción)
* [2. Antecedentes históricos](#Antecedentes)
* [3. Presentación del algoritmo](#Algoritmo)
* [4. Ejercicio de aplicación](#Ejercicio)
* [5. Conclusiones](#Conclusiones)
* [Referencias](#Referencias)

# Introducción

En el análisis y modelamiento de datos es importante el uso de diferentes herramientas matemáticas, esto con el fin de describir el comportamiento de los mismos. Con esto en mente, uno de los métodos más populares en medio de la estadística y de la ciencia de datos es la [*regresión lineal*](https://es.wikipedia.org/wiki/Regresi%C3%B3n_lineal). La idea central de este algoritmo es relacionar un conjunto de [variables respuestas o dependientes](https://en.wikipedia.org/wiki/Dependent_and_independent_variables) con un conjunto de [variables explicativas o independientes](https://en.wikipedia.org/wiki/Dependent_and_independent_variables), ambas de carácter cuantitativo, ya sea para hacer sociación, control de calidad, calibración, clasificación y predicción; por medio de una relación lineal entre las variables respuesta y los parámetros de interés. En caso que no sean cuantitativas se usan otros modelos como la regresión logística (Hocking,2003)

En lo que respecta al aprendizaje de máquina, la regresión lineal es de vital relevancia para clasificar y predicir. Para ello, se recurren a diferentes estrategias de optimización para estimar los parámetros como lo son [mínimos cuadrados](https://es.wikipedia.org/wiki/M%C3%ADnimos_cuadrados) o el método de [Gradiente Descendiente](https://www.iartificial.net/gradiente-descendiente-para-aprendizaje-automatico/); además de múltiples formas de validación (Bowles, 2015) (Müller, A. C., & Guido, S, 2016). Sin embargo, podemos tener datos que están relacionados de diferentes formas y debemos ser cuidadosos a la hora de plantear nuestros modelos; pues podemos implementar el modelo y tener malos ajustes.

Con lo anterior en mente, este cuaderno tiene como objetivo presentar el formalismo matemático de la regresión lineal junto con un ejemplo de aplicación, en el cual se encuentra la relación entre la potencia eléctrica de climatización y temperatura exterior de una planta. Este ejemplo y los datos usados se pueden encontrar en el blog de machine learning [Koldopina](https://koldopina.com/regresion-lineal-simple/).

# Antecedentes históricos

Los primeros desarrollos teóricas sobre regresión lineal se remontan a comienzos del siglo XIX cuando [Legendre](https://es.wikipedia.org/wiki/Adrien-Marie_Legendre), en 1805, y [Gauss](https://es.wikipedia.org/wiki/Carl_Friedrich_Gauss), en 1809,  implementaron este algoritmo para solucionar el movimientos de los planetas. En el planteamiento original se planteaba una relación lineal entre las variables explicativas y los parámetros de interés, es decir $y = \beta_1 x + \beta_0$, donde $y$ es la variable respuesta, $x$ es la variable explicativa y, $\beta_1$ y  $\beta_0$ los parámetros a determinar. A partir de esto, determinaron los parámetros usando mínimos cuadrados (Stigler, S. M. 1986).

*El estudio del movimiento planetario contribuyó al desarrollo de la regresión lineal y a la posterior implementación de los modelos lineales*

# Presentación del algoritmo

Si bien es cierto que hay cursos universitarios completos de regresión lineal, lo que vamos a abordar constituye las bases para modelos más elaborados. Si el lector se quiere familiarizar más con esto, puede ver los [usos de la regresión en el aprendizaje profundo](https://www.youtube.com/watch?v=E5RjzSK0fvY&ab_channel=edureka%21), [lecciones sobre el formalismo matemático mas detallado](https://www.youtube.com/watch?v=4b4MUYve_U8&ab_channel=stanfordonline) o [una explicación en vídeo](https://www.youtube.com/watch?v=SsFBnvkoZa4&ab_channel=EstimadosEstadisticos).

Para empezar tomemos una serie de variables explicativas $Y_i$ con $i=1,...,n$. Tales que

$$
Y_{i}=g_{i}(\theta)+\varepsilon_{i}=\alpha+b x_{2}+\varepsilon_{i}, \quad 1 \leq i \leq n
$$

con $\pmb{\theta}=(a, b)^{t}$, el vector de parámetros, y $x_{i}, i=1,2,3, \ldots, n,$ valores conocidos (variables explicativas)y $\varepsilon_{i}$ es una variable aleatoria que tiene una distribución normal con media cero y varianza $\sigma^2$. Además,

$$
g_{i}(\pmb{\theta})=a+b x_{i}, \quad i=1,2,3, \ldots, n
$$

Ahora, se quiere minimizar la distancia de los valores $y_i$ que predice nuestro modelo y los datos observados. Así

$$
Q=\sum_{i=1}^{n} \left[ y_{i}-g_{i}\left(\theta_{1}, \ldots, \theta_{x}\right)\right]^{2}=\sum_{i=1}^{n} \left[ y_{i}-a -b x_{i} \right]^{2}
$$

se deriva $Q$ con respecto a $\pmb{\theta}, \frac{d Q_f}{{d \theta}}$ y se iguala a cero $\frac{d Q}{d \theta}=0$

\begin{align}
    \left( \begin{matrix} \frac{\partial Q}{\partial a} \\ \frac{\partial Q}{\partial b} \end{matrix}\right) & = \left( \begin{matrix} (-1)\sum_{i=1}^n 2(y_i -a -bx_i) \\ \sum_{i=1}^n 2(y_i -a -bx_i)(-x_i)  \frac{\partial Q}{\partial b} \end{matrix}\right)
\end{align}

Así,
\begin{align}
    \left( \begin{matrix} (-1)\sum_{i=1}^n 2(y_i -a -bx_i) \\ \sum_{i=1}^n 2(y_i -a -bx_i)(-x_i)  \frac{\partial Q}{\partial b} \end{matrix}\right) & = \left( \begin{matrix} 0 \\ 0 \end{matrix}\right)
\end{align}

Despejando se obtiene

\begin{align}
a & = \frac{\bar{y} \sum_{i=1}^n x_i^2 -\bar{x} \sum_{i=1}^n x_i y_i }{\sum_{i=1}^n x_i^2  -n \bar{x}²} \\
b & = \frac{\sum_{i=1}^n x_i y_i - n \bar{x}\bar{y}}{\sum_{i=1}^n x_i^2  -n \bar{x}²} \\
\end{align}

De esta manera, obtenemos los parámetros de la regresión lineal simple.

Además, podemos calcular el coeficiente de correlación como
\begin{equation}
r_{x y}=\frac{\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)\left(y_{i}-\bar{y}\right)}{\sqrt{\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \sum_{i=1}^{n}\left(y_{i}-\bar{y}\right)^{2}}}
\end{equation}

# Ejercicio de aplicación

Como se mencionó anteriormente, este ejercicio se puede consultar en el blog [Koldopina](https://koldopina.com/regresion-lineal-simple/). La idea es entrenar un modelo de machine learning para predecir  una relación de la forma $y = \beta_1 x + \beta_0$ donde $y$ es la potencia eléctrica de climatización y $x$ es la temperatura exterior de una planta. Para empezar, vamos a importar las librerías necesarias para cargar los datos y podemos procesar matemáticamente

```python
#Importamos las librerias
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import random
import math
import csv
%matplotlib inline
```

Ahora, extraemos los datos del archivo [data.csv](https://github.com/daagonzalezgu/Curso-Big-Data-2021-I/blob/20d22c8d16a70e9235c2cf9ea509cfc04716b398/EnsayoJupyter/data.csv) y los organizamos para poder procesarlos, creando listas para cada columna.

```python
with open('data.csv', 'r') as f:
    reader = csv.reader(f, delimiter='|')
    OUTDOOR_TEMP = []
    ELECTRIC_POWER = []
    for index, row in enumerate(reader):
        if index == 0:
            header = row
        else:
            try:
                outNum = float(row[0].replace(",", "."))
            except:
                outNum = None
            OUTDOOR_TEMP.append(outNum)
            try:
                elecNum = float(row[1].replace(",", "."))
            except:
                elecNum = None
            ELECTRIC_POWER.append(elecNum)
```

Ahora, procedemos a visualizar los datos

```python
def info(header, data_list):
    """
    :param header: lista con los encabezados de las columnas
    :param data_list: lista con las listas de datos de las columnas: [lista1, lista2, etc...]
    :return: diccionario de diccionarios con número de registros, tipo de dato, número de na,
              media, std, min, max de cada columna.
    """

    from collections import defaultdict

    header = header
    columns = data_list
    values = defaultdict()
    for index, head in enumerate(header):
        aux = defaultdict()
        aux['len'] = len(columns[index])
        aux['clases'] = set([type(ele) for ele in columns[index]])
        aux['na'] = sum(1 for ele in columns[index] if ele == None)
        # media
        media = sum(ele for ele in columns[index] if ele != None) / len(columns[index])
        aux['media'] = media
        # std
        n = sum(1 for ele in columns[index] if ele != None)
        std = ((1 / (n - 1)) * sum((ele - media) ** 2 for ele in columns[index] if ele != None)) ** 0.5
        aux['std'] = std
        # minimo
        aux['min'] = min(ele for ele in columns[index] if ele != None)
        # maximo
        aux['max'] = max(ele for ele in columns[index] if ele != None)
        values[head] = aux
    return values
```

```python
print('_'*60 + 'COLUMNS')
print (header)

```

    ____________________________________________________________COLUMNS
    ['OUTDOOR_TEMP', 'ELECTRIC_POWER']

```python
print ('_'*60 + 'INFO')
print (info(header, [OUTDOOR_TEMP, ELECTRIC_POWER]))
```

    ____________________________________________________________INFO
    defaultdict(None, {'OUTDOOR_TEMP': defaultdict(None, {'len': 1496, 'clases': {<class 'float'>}, 'na': 0, 'media': 28.544429085561454, 'std': 6.364374947698145, 'min': 0.0, 'max': 38.310135}), 'ELECTRIC_POWER': defaultdict(None, {'len': 1496, 'clases': {<class 'float'>, <class 'NoneType'>}, 'na': 2, 'media': 285.83182486631, 'std': 41.51823016424719, 'min': 0.0, 'max': 391.71})})

Lo anterior nos dice que tenemos dos columnas, una llamada OUTDOOR_TEMP, que es la temperatura exterior y ELECTRIC_POWER, es la potencia eléctrica y tiene dos valores no definidos (na). Con ello, procedemos a limpiar los datos quitando esos valores (na)

```python
#Quitamos los nas
index_to_drop = [index for index, val in enumerate(ELECTRIC_POWER) if val is None]
ELECTRIC_POWER = [val for index, val in enumerate(ELECTRIC_POWER) if index not in index_to_drop]
OUTDOOR_TEMP = [val for index, val in enumerate(OUTDOOR_TEMP) if index not in index_to_drop]
```

Ahora, verificamos que esos valores no estén

```python
print (info(header, [OUTDOOR_TEMP, ELECTRIC_POWER]))
```

    defaultdict(None, {'OUTDOOR_TEMP': defaultdict(None, {'len': 1494, 'clases': {<class 'float'>}, 'na': 0, 'media': 28.54050408902272, 'std': 6.36757237847448, 'min': 0.0, 'max': 38.310135}), 'ELECTRIC_POWER': defaultdict(None, {'len': 1494, 'clases': {<class 'float'>}, 'na': 0, 'media': 286.21446452476556, 'std': 41.51646570695531, 'min': 0.0, 'max': 391.71})})

Seguido a ello, graficamos los datos para identificar valores atípicos que pueden afectar los valores que va a predecir nuestro modelo

```python
def visual(header, X, y):
    """
    :param header: Lista con los nombres de los encabezados
    :param X: Lista con los valores de la columna a colocar en el eje X
    :param y: Lista con los valores de la columna a colocar en el eje y
    :return: matplotlib figure plot
    """

    fs = 10  # fontsize
    fig, axs = plt.subplots(3, 2, figsize=(6, 6))
    plt.subplots_adjust(top=0.9, bottom=0.1, hspace=0.5, wspace=0.2, left=0.125, right=0.9)
    axs[0, 0].scatter(X, y, c='r', edgecolors=(0, 0, 0), alpha=0.2)
    axs[0, 0].set_title('Scatter %s vs %s' %(header[1], header[0]), fontsize=fs)
    axs[1, 0].hist(X, color='red')
    axs[1, 0].set_title('Hist %s' %header[0], fontsize=fs)
    axs[0, 1].hist2d(X, y)
    axs[0, 1].set_title('Hist 2D', fontsize=fs)
    axs[1, 1].hist(y, color='blue')
    axs[1, 1].set_title('Hist %s' %header[1], fontsize=fs)
    axs[2, 0].boxplot(X)
    axs[2, 0].set_title('Box %s' %header[0], fontsize=fs)
    axs[2, 1].boxplot(y)
    axs[2, 1].set_title('Box %s' %header[1], fontsize=fs)
    plt.show()
```

```python
#Ploteamos la gráfica
visual(header, OUTDOOR_TEMP, ELECTRIC_POWER)
```

```python
#Quitamos los outlier o valores atípicos
index_to_drop = [index for index, val in enumerate(OUTDOOR_TEMP) if val == 0]
ELECTRIC_POWER = [val for index, val in enumerate(ELECTRIC_POWER) if index not in index_to_drop]
OUTDOOR_TEMP = [val for index, val in enumerate(OUTDOOR_TEMP) if index not in index_to_drop]
```

```python
print (info(header, [OUTDOOR_TEMP, ELECTRIC_POWER]))
```

    defaultdict(None, {'OUTDOOR_TEMP': defaultdict(None, {'len': 1449, 'clases': {<class 'float'>}, 'na': 0, 'media': 29.42685514768802, 'std': 3.963016838173277, 'min': 18.765305, 'max': 38.310135}), 'ELECTRIC_POWER': defaultdict(None, {'len': 1449, 'clases': {<class 'float'>}, 'na': 0, 'media': 287.81427881297435, 'std': 35.88844333632387, 'min': 183.12, 'max': 391.71})})

```python
visual(header, OUTDOOR_TEMP, ELECTRIC_POWER)
```

Dividimos los datos para entrenar y para validar

```python
#creamos las listas para training y test aleatoriamente
all_data = [[x_val, y_val] for x_val, y_val in zip(OUTDOOR_TEMP, ELECTRIC_POWER)]
print (all_data[:5])
random.shuffle(all_data)
div = math.ceil(len(all_data)*0.3)
data_train = all_data[:div]
data_test = all_data[div:]

data_train_X = [ele[0] for ele in data_train]
data_train_y = [ele[1] for ele in data_train]
data_test_X = [ele[0] for ele in data_test]
data_test_y = [ele[1] for ele in data_test]
```

    [[31.321108, 324.54], [24.938467, 252.7], [33.316906, 331.1], [26.947517, 270.8], [34.769539, 288.66]]

Se crea el modelo y se procede a entrenarlo

```python
class Lin_reg():

    def __init__(self, X, Y):
        """
        :param X: lista con los valores de la variable de las abscisas
        :param y: lista con los valores de la variable de las ordenadas
        """
        self.X = X
        self.y = Y
        self.N = len(self.X)
        self.X_mean = sum(self.X) / len(self.X)
        self.y_mean = sum(self.y) / len(self.y)
        self.X_std = (1 / (self.N - 1) * sum((ele - self.X_mean) ** 2
                                             for ele in self.X)) ** 0.5
        self.y_std = (1 / (self.N - 1) * sum((ele - self.y_mean) ** 2
                                             for ele in self.y)) ** 0.5
        self.X_var = self.X_std ** 2
        self.y_var = self.y_std ** 2
        self.cov = sum([i * j for (i, j) in zip([ele - self.X_mean for ele in self.X],
                                                [ele - self.y_mean for ele in self.y])]) / (self.N)

        self.r = self.cov / (self.X_std * self.y_std)

    def Coeficientes(self):
        if len(self.X) != len(self.y):
            raise ValueError('unequal length')
        self.b = self.cov / self.X_var
        self.a = self.y_mean - (self.b * self.X_mean)
        return self.a, self.b

    def predict(self, X):
        yp = []
        for x in X:
            yp.append(self.a + self.b * x)
        return yp
```

```python
mylinreg=Lin_reg(data_train_X,data_train_y)
```

```python
a, b = mylinreg.Coeficientes()
print('La recta de regresión es: y = %f + %f * X'%(mylinreg.Coeficientes()))
print('El coeficiente de correlación es: r = %f' %mylinreg.r)
```

    La recta de regresión es: y = 54.837434 + 7.871854 * X
    El coeficiente de correlación es: r = 0.904796

Ahora se gráfican los datos de entrenamiento y los resultados de la regresión

```python
plt.scatter(data_train_X, data_train_y, c='r', edgecolors=(0, 0, 0), alpha=0.5)
plt.plot(data_train_X, [a + b * x for x in data_train_X], c="b")
plt.xlabel('Temperatura exterior (C)')
plt.ylabel('Potencia eléctrica kW')
plt.show()
```

Ahora, predecimos los valores y comparamos con los valores observados en los datos de validación

```python
predictions = mylinreg.predict(data_test_X)
```

```python
plt.scatter(data_test_y, predictions, c='r', edgecolors=(0, 0, 0), alpha=0.5)
plt.title('Valores predichos vs valores reales', fontsize=10)
plt.xlabel('Valores reales')
plt.ylabel('Valores predichos')
plt.show()
```

Con esto, calculamos el error promedio, el error cuadrático, el coeficiente de correlación y la desviación estándar

```python
#Metricas
#Mean Error - Desviación media
ME = sum(y_pred - y_test for y_pred, y_test in zip(predictions,data_test_y)) / len(predictions)
#Mean Absolute Error (error absoluto medio)
MAE = sum(abs(y_pred - y_test) for y_pred, y_test in zip(predictions,data_test_y)) / len(predictions)
#Mean Square Error (error cuadrático medio)
MSE = sum((y_pred - y_test)**2 for y_pred, y_test in zip(predictions, data_test_y)) / len(predictions)
#Root Mean Square Error - error de la raíz cuadrada de la media RMSE
RMSE = MSE ** 0.5
#Standard Deviation of Residuals . Desviación típica de los residuos
SDR = (1 / (len(data_test_y) - 1) * sum((y_test - y_pred) ** 2
        for y_pred, y_test in zip(predictions, data_test_y))) ** 0.5

print ('Mean Error: %f' %ME)
print ('Mean Absolute Error: %f' %MAE)
print ('Mean Square Error: %f' %MSE)
print ('Root Mean Square Error: %f' %RMSE)
print ('Standard Desviation of Residuals: %f' %SDR)
```

    Mean Error: -1.904745
    Mean Absolute Error: 12.997670
    Mean Square Error: 265.198086
    Root Mean Square Error: 16.284904
    Standard Desviation of Residuals: 16.292940

```python
data_test_mean = sum(ele
    for ele in data_test_y) / len(data_test_y)
predictions_mean = sum(ele
    for ele in predictions) / len(predictions)

data_test_std = (1 / (len(data_test_y) - 1) * sum((ele - data_test_mean) ** 2
    for ele in data_test_y)) ** 0.5
predictions_std = (1 / (len(predictions) - 1) * sum((ele - predictions_mean) ** 2
    for ele in predictions)) ** 0.5
cov = sum([i * j
    for (i, j) in zip([ele - data_test_mean
        for ele in data_test_y],
                      [ele - predictions_mean
        for ele in predictions])
]) / (len(predictions))

print('El coeficiente de correlación es: R2 = %f' % (cov**2 / (data_test_std ** 2 * predictions_std ** 2)))
```

    El coeficiente de correlación es: R2 = 0.799761

Por último, gráficamos los residuos para comprobar su distribución normal con media cero. Esto nos da una idea de que tan bien el modelo se reproduce los datos.

```python
#Distribución de los Residuos
sns.distplot((np.asarray(data_test_y) - np.asarray(predictions)), bins = 50)
plt.show()
```

# Conclusiones

A partir de lo mostrado, se tiene que la regresión lineal es uno de los métodos más importantes en el modelamiento y predicción de fenómenos, debido a su simplicidad matemática comparada con otros modelos. Además, que sirve de base para algoritmos más complejos tales como las redes neuranales o en modelos lineales generalizados. Por último, constituye un campo de estudio con muchas aplicaciones en la industria, demografía, ciencias exactas y demás; aunque tampoco constituye una receta mágica para todos nuestros problemas con datos.

# Referencias

* Bowles, M. (2015). Machine learning in Python: essential techniques for predictive analysis. John Wiley & Sons.
* Hocking, R. R. (2003), Methods and Applications of Linear Models, segunda edn, John Wiley and Sons, New Jersey.
* Müller, A. C., & Guido, S. (2016). Introduction to machine learning with Python: a guide for data scientists. " O'Reilly Media, Inc.".
* Ravishanker, N. & Dey, D. K. (2002), A First Course in Linear Model Theory, Chapman & Hall/CRC., New York.
* Stigler, S. M. (1986). The history of statistics: The measurement of uncertainty before 1900. Harvard University Press.