# The transport-length hillslope diffuser

# The basics:

This component uses an approach similar to Davy and Lague (2009)'s equation for fluvial erosion and transport, and applies it to hillslope diffusion.
Formulation and implementation were inspired by Carretier et al (2016), see this paper and references therein for justification.

## Theory

The elevation z of a point of the landscape (grid node) changes according to:

\begin{equation}
 \frac{\partial z}{\partial t} = -\epsilon + D + U  \tag{1}\label{eq:1}
\end{equation}

and we define:
\begin{equation}
 D = \frac{q_s}{L}  \tag{2}\label{eq:2}
\end{equation}

where $\epsilon$ is the local erosion rate [*L/T*], *D* the local deposition rate [*L/T*], *U* the uplift (or subsidence) rate [*L/T*], $q_s$ the incoming sediment flux per unit width [*L$^2$/T*] and *L* is the **transport length**.

We specify the erosion rate $\epsilon$ and the transport length *L*:

\begin{equation}
 \epsilon = \kappa S  \tag{3}\label{eq:3}
\end{equation}

\begin{equation}
 L = \frac{dx}{1-({S}/{S_c})^2}  \tag{4}\label{eq:4}
\end{equation}

where $\kappa$ [*L/T*] is an erodibility coefficient, $S$ is the local slope [*L/L*] and $S_c$ is the critical slope [*L/L*].

Thus, the elevation variation results from the difference between local rates of detachment and deposition.

The detachment rate is proportional to the local gradient. However, the deposition rate (*$q_s$/L*) depends on the local slope and the critical slope:
- when $S \ll S_c$, most of the sediment entering a node is deposited there, this is the pure diffusion case. In this case, the sediment flux $q_s$ does not include sediment eroded from above and is thus "local".
- when $S \approx S_c$, *L* becomes infinity and there is no redeposition on the node, the sediments are transferred further downstream.  This behaviour corresponds to mass wasting, grains can travel a long distance before being deposited. In that case, the flux $q_s$ is "non-local" as it incorporates sediments that have both been detached  locally and transited from upslope.
- for an intermediate $S$, there is a prgogressive transition between pure creep and "balistic" transport of the material. This is consistent with experiments (Roering et al., 2001; Gabet and Mendoza, 2012).

## Contrast with the non-linear diffusion model

Previous models typically use a "non-linear" diffusion model proposed by different authors (e.g. Andrews and Hanks, 1985; Hanks, 1999; Roering et al., 1999) and supported by $^{10}$Be-derived erosion rates (e.g. Binnie et al., 2007) or experiments (Roering et al., 2001). It is usually presented in the followin form:

$ $

\begin{equation}
 \frac{\partial z}{\partial t} = \frac{\partial q_s}{\partial x}  \tag{5}\label{eq:5}
\end{equation}

$ $
\begin{equation}
 q_s = \frac{\kappa' S}{1-({S}/{S_c})^2}   \tag{6}\label{eq:6}
\end{equation}

where $\kappa'$ [*L$^2$/T*] is a diffusion coefficient.

This description is thus based on the definition of a flux of transported sediment parallel to the slope:
- when the slope is small, this flux refers to diffusion processes such as aoil creep, rain splash or diffuse runoff
- when the slope gets closer to critical slope, the flux increases dramatically, simulating on average the cumulative effect of mass wasting events.

Despite these conceptual differences, Eq ($\ref{eq:3}$) and ($\ref{eq:4}$) predict similar topographic evolution to the 'non-linear' diffusion equations for $\kappa' = \kappa dx$, as shown in the following example.

# Example 1:

First, we import what we'll need:

```python
%matplotlib inline

import numpy as np
from matplotlib.pyplot import figure, show, plot, xlabel, ylabel, title
import pymt.models
```

    ➡ models: Avulsion, Plume, Sedflux3D, Subside, FrostNumber, Ku, ExponentialWeatherer, Flexure, FlowAccumulator, FlowDirectorD8, FlowDirectorDINF, FlowDirectorSteepest, FlowRouter, LinearDiffuser, OverlandFlow, SoilMoisture, StreamPowerEroder, TransportLengthHillslopeDiffuser, Vegetation, Hydrotrend, Child, Cem, Waves

Set the initial and run conditions:

```python
total_t = 2000000.  # total run time (yr)
dt = 1000.  # time step (yr)
nt = int(total_t // dt)  # number of time steps
uplift_rate = 0.0001  # uplift rate (m/yr)

kappa = 0.001  # erodibility (m/yr)
Sc = 0.6  # critical slope
```

Instantiate the components:
The hillslope diffusion component must be used together with a flow router/director that provides the steepest downstream slope for each node, with a D4 method (creates the field *topographic__steepest_slope* at nodes).

```python
fdir = pymt.models.FlowDirectorSteepest()
tl_diff = pymt.models.TransportLengthHillslopeDiffuser()
```

```python
config_file, config_dir = fdir.setup(
    grid_row_spacing=10.,
    grid_column_spacing=10.,
    grid_rows=100,
    grid_columns=100,
    clock_start=0.0,
    clock_stop=total_t,
    clock_step=dt,
)
fdir.initialize(config_file, config_dir)
```

```python
config_file, config_dir = tl_diff.setup(
    grid_row_spacing=10.,
    grid_column_spacing=10.,
    grid_rows=100,
    grid_columns=100,

    clock_start=0.0,
    clock_stop=total_t,
    clock_step=dt,

    erodibility=kappa,
    slope_crit=Sc,
)
tl_diff.initialize(config_file, config_dir)
```

Set the boundary conditions. The **FlowDirector** component uses a variable called *boundary_condition_flag* to set its boundary conditions. A value of 1, means the boundary is open and sediment is free to leave the grid. A value of 4 means the nodes are closed and so there is no flux through them. The **TransportLengthHillslopeDiffuser** uses these boundary conditions as input so we'll set them both here.

```python
status = fdir.get_value("boundary_condition_flag").reshape((100, 100))
status[:, (0, -1)] = 1 # E and W boundaries are open
status[(0, -1), :] = 4 # N and S boundaries are closed
```

```python
fdir.set_value("boundary_condition_flag", status)
tl_diff.set_value("boundary_condition_flag", status)
```

Start with an initial surface that's just random noise.

```python
z = np.random.rand(100 * 100)
```

```python
fdir.set_value("topographic__elevation", z)
```

Get the input values for **TransportLengthHillslopeDiffuser** from the flow director.

```python
tl_diff.set_value("topographic__elevation", z)
tl_diff.set_value("flow__receiver_node", fdir.get_value("flow__receiver_node"))
tl_diff.set_value("topographic__steepest_slope", fdir.get_value("topographic__steepest_slope"))
```

Run the components for 2 Myr and trace an East-West cross-section of the topography every 100 kyr:

```python
for t in range(nt - 1):
    fdir.update()

    tl_diff.set_value("topographic__elevation", z)
    tl_diff.set_value(
        "flow__receiver_node", fdir.get_value("flow__receiver_node")
    )
    tl_diff.set_value(
        "topographic__steepest_slope",
        fdir.get_value("topographic__steepest_slope"),
    )

    tl_diff.update()

    z = tl_diff.get_value("topographic__elevation").reshape((100, 100))
    z[1:-1, 1:-1] += uplift_rate * dt  # add the uplift

    fdir.set_value("topographic__elevation", z)

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)

        # plot east-west cross-section of topography:
        x_plot = range(0, 1000, 10)
        z_plot = z[1, :]
        figure('cross-section')
        plot(x_plot, z_plot)
```

And plot final topography:

```python
tl_diff.quick_plot("topographic__elevation")
```

# Example 2

In this example, we show that when the slope is steep ($S \geq S_c$), the transport-length hillsope diffusion simulates mass wasting, with long transport distances.

First, we create a grid: the western half of the grid is flat at 0m of elevation, the eastern half is a 45-degree slope.

```python
fdir = pymt.models.FlowDirectorSteepest()
tl_diff = pymt.models.TransportLengthHillslopeDiffuser()
```

```python
total_t = 1000000.  # total run time (yr)
dt = 1000.  # time step (yr)
nt = int(total_t // dt)  # number of time steps

kappa = 0.001  # erodibility (m / yr)
Sc = 0.6  # critical slope
```

```python
grid_params = {
    "grid_row_spacing": 10.,
    "grid_column_spacing": 10.,
    "grid_rows": 100,
    "grid_columns": 100,
}
clock_params = {
    "clock_start": 0.0,
    "clock_stop": total_t,
    "clock_step": dt,
}
```

```python
config_file, config_dir = fdir.setup(**grid_params, **clock_params)
fdir.initialize(config_file, config_dir)
```

```python
config_file, config_dir = tl_diff.setup(
    **grid_params,
    **clock_params,
    erodibility=kappa,
    slope_crit=Sc,
)
tl_diff.initialize(config_file, config_dir)
```

As before, set the boundary conditions for both components.

```python
status = fdir.get_value("boundary_condition_flag").reshape((100, 100))
status[:, (0, -1)] = 1 # E and W boundaries are open
status[(0, -1), :] = 4 # N and S boundaries are closed
```

```python
fdir.set_value("boundary_condition_flag", status)
tl_diff.set_value("boundary_condition_flag", status)
```

In this example, we'll use a different initial surface: a dipping plane.

```python
grid = fdir.var_grid("topographic__elevation")
n_vals = fdir.grid_size(grid)
x, y = fdir.grid[0].node_x, fdir.grid[0].node_y
```

```python
z = np.zeros(n_vals)
z[x > 500] = x[x < 490] / 10.0
```

To make sure we've set things up correctly.

```python
fdir.set_value("topographic__elevation", z)
fdir.quick_plot("topographic__elevation")
```

Now time step through the model, plotting things along the way.

```python
for t in range(1000):
    fdir.update()

    tl_diff.set_value("topographic__elevation", fdir.get_value("topographic__elevation"))
    tl_diff.set_value("flow__receiver_node", fdir.get_value("flow__receiver_node"))
    tl_diff.set_value("topographic__steepest_slope", fdir.get_value("topographic__steepest_slope"))

    tl_diff.update()

    fdir.set_value("topographic__elevation", tl_diff.get_value("topographic__elevation"))

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)

        z = tl_diff.get_value("topographic__elevation").reshape((100, 100))

        # plot east-west cross-section of topography:
        x_plot = range(0, 1000, 10)
        z_plot = z[1, :]
        figure('cross-section')
        plot(x_plot, z_plot)
```

```python
fdir.quick_plot("topographic__elevation")
```

The material is diffused from the top and along the slope and it accumulates at the bottom, where the topography flattens.

# Example 3

As a comparison, the following code uses linear diffusion on the same slope. Instead of using the **TransportLengthHillslopeDiffuser** component, we'll swap in the **LinearDiffuser** component. Everything else will be pretty much the same.

```python
fdir = pymt.models.FlowDirectorSteepest()
diff = pymt.models.LinearDiffuser()
```

Setup and initialize the models.

```python
config_file, config_dir = fdir.setup(**grid_params, **clock_params)
fdir.initialize(config_file, config_dir)
```

```python
config_file, config_dir = diff.setup(
    **grid_params,
    **clock_params,
    linear_diffusivity=0.1,
)
diff.initialize(config_file, config_dir)
```

Set boundary conditions.

```python
status = fdir.get_value("boundary_condition_flag").reshape((100, 100))
status[:, (0, -1)] = 1 # E and W boundaries are open
status[(0, -1), :] = 4 # N and S boundaries are closed
```

```python
fdir.set_value("boundary_condition_flag", status)
diff.set_value("boundary_condition_flag", status)
```

Set the initial topography.

```python
grid = fdir.var_grid("topographic__elevation")
n_vals = fdir.grid_node_count(grid)
x, y = fdir.grid[0].node_x, fdir.grid[0].node_y
```

```python
z = np.zeros(n_vals)
z[x > 500] = x[x < 490] / 10.0
```

```python
fdir.set_value("topographic__elevation", z)
fdir.quick_plot("topographic__elevation")
```

Run the model!

```python
for t in range(1000):
    fdir.update()

    diff.set_value("topographic__elevation", fdir.get_value("topographic__elevation"))

    diff.update()

    fdir.set_value("topographic__elevation", diff.get_value("topographic__elevation"))

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)

        z = diff.get_value("topographic__elevation").reshape((100, 100))

        # plot east-west cross-section of topography:
        x_plot = range(0, 1000, 10)
        z_plot = z[1, :]
        figure('cross-section')
        plot(x_plot, z_plot)
```

```python
fdir.quick_plot("topographic__elevation")
```

```python

```


#Probing convexity of some functions planted as exercises

For all problems we asume that $x_1, x_2 \in {\rm I\!R}$ and we  asume that $z = \alpha x_1 + (1 - \alpha)x_2$, is usable in all functions for probing the convexity and for last we asume that $\alpha \in [0,1]$

1. Show that $f(x) = a \cdot x + b$ is convex, $x \in {\rm I\!R}$

\begin{equation}
\begin{split}
& f(z) = f(\alpha x_1 + (1 - \alpha) x_2)\\
& f(z) = a (\alpha x_1 + (1 - \alpha) x_2) + b \\
& f(z) = a  \alpha x_1 + a(1-\alpha) x_2 + b \\
& f(z) = a  \alpha x_1 + a(1-\alpha) x_2 + b + \alpha b - \alpha b\\
& f(z) = a  \alpha x_1 + \alpha b  + a(1-\alpha) x_2 + b - \alpha b\\
& f(z) = \alpha (a x_1 + b) + a(1-\alpha)x_2 + b(1 - \alpha)\\
& f(z) = \alpha ( a x_1 + b) + (1-\alpha) (a x_2 + b)\\
& f(z) = \alpha f(x_1) + (1-\alpha) f(x_2)
\end{split}
\end{equation}

2. Show that $f(x) = ax^2  + b$ is convex, $x \in {\rm I\!R}$

\begin{equation}
\begin{split}
& f(z) = f(\alpha x_1 + (1 - \alpha) x_2)\\
& f(z) = a (\alpha x_1 + (1 - \alpha)x_2)^2 + b\\
& f(z) = a ( [\alpha x_1]^2 + 2 [\alpha x_1 + (1 - \alpha)x_2] + [(1 - \alpha)x_2]^2) + b\\
& f(z) = a(\alpha x_1)^2 + 2a(\alpha x_1 + (1 - \alpha)x_2) + a[(1 - \alpha)x_2]^2 + b\\
& f(z) = a(\alpha x_1)^2 + 2a\alpha x_1 + 2a(1 - \alpha)x_2 + a[(1 - \alpha)x_2]^2 + b\\
& f(z) = a(\alpha x_1)^2 + 2a\alpha x_1 + \alpha b + 2a(1 - \alpha)x_2 + a[(1 - \alpha)x_2]^2 + b - \alpha b\\
& f(z) = a(\alpha x_1)^2 + 2a\alpha x_1 + \alpha b + 2a(1 - \alpha)x_2 + a[(1 - \alpha)x_2]^2 + b  (1 - \alpha)\\
& f(z)= a(\alpha x_1)^2 + 2a\alpha x_1 + \alpha b + (1- \alpha)(2a + b) + a[(1 - \alpha)x_2]^2 \\
& f(z)= a(\alpha x_1)^2 + \alpha (2ax_1 + b) + (1- \alpha)(2a + b) + a[(1 - \alpha)x_2]^2
\end{split}
\end{equation}

3. Show that $g(x) = af(x) + b$ is convex, where $f(x)$ is a convex function, $x \in {\rm I\!R}$

\begin{equation}
\begin{split}
& f(z) = f(\alpha x_1 + (1-\alpha)x_2)\\
& g(x) = a f(x) + b\\
& g(x) = a f(z) + b\\
& g(x) = a f(\alpha x_1 + (1-\alpha)x_2) + b\\
& g(x) = a [\alpha f(x_1) + (1-\alpha) f(x_2)] + b\\
& g(x) = a \alpha f(x_1) + a (1 - \alpha) f(x_2) + b\\
& g(x) = a \alpha f(x_1) + a (1 - \alpha) f(x_2) + b + \alpha b - \alpha b\\
& g(x) = a \alpha f(x_1) + \alpha b + a (1 - \alpha) f(x_2) + b - \alpha b\\
& g(x) = \alpha (a f(x_1) + b) + a (1 - \alpha) f(x_2) + b (1 - \alpha)\\
& g(x) = \alpha (a f(x_1) + b) + (1 - \alpha) (a f(x_2) + b)\\
& g(x) = \alpha g(x_1) + (1 - \alpha) g(x_2)\\
\end{split}
\end{equation}

4. Show that, if $f(x)$ is convex, then $g(x) = f(ax+b)$ is convex as well

\begin{equation}
\begin{split}
& f(z) = f(\alpha x_1 + (1 - \alpha)x_2)\\
& g(x) = f(a x + b)\\
& g(x) = f(a (\alpha x_1 + (1 - \alpha)x_2) + b)\\
& g(x) = f(a\alpha x_1 + a(1 - \alpha)x_2 + b)\\
& g(x) = f(a\alpha x_1 + a(1 - \alpha)x_2 + b + \alpha b - \alpha b)\\
& g(x) = f(a\alpha x_1 + \alpha b  + a(1 - \alpha)x_2 + b - \alpha b)\\
& g(x) = f(\alpha [a x_1 + b]  + a [1 - \alpha] x_2 + b [1 - \alpha])\\
& g(x) = f(\alpha [a x_1 + b]  + [1 - \alpha] [a  x_2 + b])\\
& g(x) = \alpha f(a x_1 + b)  + (1 - \alpha) f(a  x_2 + b)\\
& g(x) = \alpha g(x_1)  + (1 - \alpha) g(x_2)\\
\end{split}
\end{equation}


# `DSML Workshop 07` - Introduction to ML

In this workshop we start with hands-on machine learning focusing mostly on the linear regression example covererd previously in the lecture.

Machine learning is a fast-growing field. Recent popularity of machine learning as a topic can largely be explained due to a combination of three different elements:
1. Gains in computing power (machine learning, as it grew out of computer science, has always been fundamentally concerned with computational algorithms)
1. Massive amounts of available data (the "raw materials" for machine learning methods)
1. Some notable algorithmic advances that have occurred over the last 30 years.

To start, though, we will cover some of the most basic algorithms to codify the underlying principles.

### Example: predicting peak electrical power

Let's consider the above problem a bit more formally, starting with a simple
example.  Suppose you want to predict what the peak electricity demand will be
during the day tomorrow for some area (we'll consider data from the area
surrounding Pittsburgh, PA).  This is actually a very important problem from a
logitics planning perspective: electricity generators, which for the most part
are based upon boiling water to move turbines, cannot turn on instantly, so in
order to guarantee that we have enough power to supply a given area, a system
operator typically needs to have some excess generation always waiting in the
wings.  The better we can forecast future demand, the smaller our excess margin
can be, leading to increased efficiency of the entire electrical grid.
The power consumption tomorrow depends on many factors: temperature, day of
week, season, holiday events, etc, not to mention some inherrent randomness
that we don't expect to even predict with perfect accuracy.  However, even for
someone working in the area, it would be very
difficult to come up with a model for electrical demand based soley upon "first
principles", thinking about the nature of electricity consumption or the devices
people may use, in an attempt to predict future consumption.

What _is_ easy, however, is simply to collect lots of data about past energy
consumption (the system operator serving the Pittsburgh region, PJM, maintains
a data set available [here](https://dataminer2.pjm.com/feed/hrl_load_metered/definition))
as well as the past factors that affect consumption, like the past weather for
the area. The files we are loading are the raw files we downloaded from this source. The final input data for our code is `Pittsburgh_load_data.csv`.

```python
import numpy as np
import matplotlib.pyplot as plt
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()
import seaborn as sns
import pandas as pd
from sklearn.linear_model import LinearRegression
import datetime
```

**Exercise** Import the data and have a first look.

If you have time left, also convert the 'Date' column to datetime.
Hint: the pandas function `to_datetime()` might be useful

```python
## Your Code goes here
df = None
```

Today we want to build a simple ML model to forecast peak demand.
Using the converted Date column, we can have a look at the interaction between the date and the peak demand, which can be found under the 'MAX' column.

```python
## Create a plot, plotting the date on the x-axis and the peak demand on the y-axis
```

```python
## Target solution 5 min
fig, ax = plt.subplots(figsize = (16,9))

# Plotting the data
ax.plot(df["Date"],df[["MAX"]])
ax.set_xlabel("Date")
ax.set_ylabel("Peak Demand (GW)")
plt.show()
```

**Exercise**: In the next steps we want to look at the summer months only. Create a `df_summer` which only contains the months of June, July and August. Hint: use the `dt.month` function to extract the month from the datetime object.

```python
## Create a dataframe df_summer which only contains the months of June, July and August
df_summer = None
```

```python
# let us define our dependent (y) and independent (x) variables for peak electricity load prediction

xp = df_summer['High_temp']

yp = df_summer['MAX']

#min_H_temp, max_H_temp = xp.min(), xp.max()
```

```python
fig, ax = plt.subplots(figsize = (8,6))
# Plotting the data
ax.scatter(xp, yp, marker='x',)
ax.set_xlabel("High Temperature (°C)")
ax.set_ylabel("Peak Demand (GW)")

plt.show()
#saving figures (You can comment-out this line inorder not to save figures)
#plt.savefig('summer data_peak demand.pdf')
```

As it turns out, electricity consumption in the summer in Pittsburgh is largely
driven by air conditioning, so with increasing high temperature comes increasing
electrical demand.  Thus, we may hypothesize that we can form a fairly good prediction of the peak demand using a linear model: that is, we hypothesize that
\begin{equation}
\mathrm{PeakDemand} \approx \theta_1 \cdot \mathrm{HighTemperature} + \theta_2
\end{equation}
where $\theta_1$ is the slope of the line and $\theta_2$ is the intercept term (together called the _parameters_ of the model).

```python
# define theta vector
theta = np.array([0, 2.0])

# set up plot
fig, ax = plt.subplots(figsize = (8,6))

# add objects to axis
ax.scatter(xp, yp, marker='x')
xlimp, ylimp =(plt.gca().get_xlim(), plt.gca().get_ylim())  #gets limity of x and y
ax.plot(xlimp, [theta[0]*xlimp[0]+theta[1], theta[0]*xlimp[1]+theta[1]], 'C1')
ax.set_xlim(xlimp)
ax.set_ylim(ylimp)
ax.set_xlabel("High Temperature (°C)")
ax.set_ylabel("Peak Demand (GW)")
plt.show()
#plt.savefig('summer data_peak demand_line.pdf')
```

Naturally, this model won't fit the data exactly (we can see from the chart that the figure doesn't lie precisely on an exact line), but if we can find slope and intercept terms that fit the data well, then for example, if we want to know what the peak demand will be tomorrow we can simply plug in the forecasted high temperature into the equation above get an estimate of the peak demand tomorrow (ignore the fact, for now that the high temperature tomorrow is also a prediction, we'll assume we just get this from a reliable source, and domains like weather forecasting are extremely well-studied in practice). This is of course equivalent to just "finding a point on the line".

### Finding good parameters

The question, of course, is how we find "good" values for $\theta_1$ and $\theta_2$ that fit this data well, i.e., so that the line fits the data as "closely" as possible.  The method we will describe for doing this (which is called _gradient descent_) is probably not the simplest algorithm for finding this fit.  In fact, as we will see, there is a very simple closed-form expression that will immediately give us the same solution for the framework we consider here.  But gradient descent is an _extremely_ powerful and general algorithm (and _is_ actually quite simple compared to some alternative approaches), and it is no exaggeration to say that gradient descent underlies virtually all modern machine learning.  So, with these caveats in place, let's dive in to understanding how we find "good" parameters $\theta_1$ and $\theta_2$ in some disciplined manner.

**Objective functions**  In order to find good values for the parameters, we need to formally define what "good" means in this setting.  This will actually be one of the key questions for machine learning algorithms in general, and difference notions of goodness lead to different algorithms.  Fortunately, there are some very well-studied definitions in this context, and so we have some "standard" options that we can try.  The notion that we will consider here captures the idea of the "squared error" between the prediction and the actual values.  That is, we consider all the days in the plot above, where $\mathrm{HighTemperature}^{(i)}$ denotes the high temperature and $\mathrm{PeakDemand}^{(i)}$ denotes the peak demand on day $i$.  Since _predicted_ peak demand for day $i$ is equal to
\begin{equation}
\theta_1 \cdot \mathrm{HighTemperature} + \theta_2
\end{equation}
we want to make this quantity as close as possible, averaged over all the days, to the true $\mathrm{PeakDemand}^{(i)}$.  We're going to measure this closeness in terms of the squared difference between the predicted and actual more.  More formally, we would like to minimize the quantity:
\begin{equation}
\frac{1}{\# \mathrm{days}} \sum_{i \in \mathrm{days}} \left ( \theta_1 \cdot \mathrm{HighTemperature}^{(i)} + \theta_2 - \mathrm{PeakDemand}^{(i)} \right )^2 \equiv E(\theta)
\end{equation}
which we abbreviate as $E(\theta)$ to emphasize the fact that we are going to be minimizing this error by tuning our $\theta$ variables.  This is known as the _objective function_ that we are trying to minimize.  A natural question that you may be asking is: why did we choose to measure closeness using this squared difference?  Why not use the average of absolute difference?  Or the maximum absolute difference?  These are good questions, and we'll defer answering them for now, except to say that we will definitely consider other possibilities later.  The squared error is simply a very common choice, mainly for reasons of mathematical convenience.

**Data scaling**  We'll shortly see what the gradient descent procedure looks like in our example above.  Before we apply the algorithm, though, we're going to make one small modification to our problem, and _normalize_ the data (both the inputs and the output) before we attempt to run the gradient descent algorithm.  We will see the reason for this more clearly shortly, but the brief reason is that if our slope and intercept terms are on very different "scales" (which would be the case here, because of the relatively large input values (units of degrees Celsius) compared to the output values (units of gigawatts)), then we would actually need to take very different step sizes in the two parameters $\theta_1$ and $\theta_2$.  This is possible to manually tune in our case, but when we start having many more parameters, it's not feasible.  We thus make our life much easier if we scale all the input and output data to be in the same rough range _before_ running gradient descent (Note: in class we talked about only scaling the input, which also happens to work fine here, but I think it may be easier conceptually to consider the case where we just normalize all the inputs and outputs in the same manner).

We can re-scale data in a number of ways, but a simple strategy is just to translate and scale the coordinates such that the values vary between zero and one in our dataset.  This can be easily achieved by the transformation
\begin{equation}
\tilde{x}^{(i)} = \frac{x^{(i)} - \min_i x^{(i)}}{\max_i x^{(i)} - \min_i x^{(i)}}
\end{equation}
and similarly for $\tilde{y}^{(i)}$.

**Exercise:** Re-scale input and output features using the formula shown above and plot the data. Call the normalized data vectors `x_nor` and `y_nor`.

```python
# YOUR CODE HERE
x_nor = None
y_nor = None
```

```python
# set up plot
fig, axes = plt.subplots(1,2,figsize = (16,9))

axes[0].scatter(xp, yp, marker='x')
axes[0].set_xlabel("Re-scaled Temperature")
axes[0].set_ylabel("Re-scaled Demand")

axes[1].scatter(x_nor, y_nor, marker = 'x')
axes[1].set_xlabel("Re-scaled Temperature")
axes[1].set_ylabel("Re-scaled Demand")

plt.show()
```

This look identical to the previous plot, of course, except that the units no longer correspond to traditional quantities like degrees Celsius or gigawatts, but just some linear transformation of these units.

### Visualizing gradient descent

Now let's look at the gradient descent algorithm, which we have derived mathematically in previous lectures.  This will initialize $\theta_1$ and $\theta_2$ to zero and repeatedly update them according to the partial derivative rules.  We will use the step size (also known as learning rate) $\alpha=1$, and print out the value of $\theta$ per each iteration.

\begin{equation}
\theta_j  := \theta_j  − \alpha \sum_{j=1}^m (\sum_{i=1}^n \theta_j x_j^{(i)} - y^{(i)})x_j^{(i)}
\end{equation}

Note: The above simplified equation omits the number of days (n) and the *2 term which results from partial differentiation and groups them into the alpha term for easier representation.

```python
theta = np.array([0., 0.])

alpha = 1.0

for t in range(20):
    print("Iteration {}: ".format(t), theta)

    # partial derivative theta1: 2*sum(((theta1*x + theta2)-y)*x)
    # term 2/N added here (usully included in alpha term), this is just a constant so could also be omitted
    theta[0] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor)*x_nor)

    # partial derivative theta2: 2*sum(((theta1*x + theta2)-y))
    # term 2/N added here (usully included in alpha term), this is just a constant so could also be omitted
    theta[1] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor) ) # this is the intercept with a slightly different partial derivative
```

Let's visualize what this looks like in a couple different ways.  First, let's look at what our line looks like during different iterations of gradient descent.  For this purpose, we'll wrap the above in a simple function that takes `iters` iterations of gradient descent (note that we can of course get all these plots within a single run of gradient descent, but we'll just use multiple calls to this function for illustration purposes).

```python
def gradient_descent(iters):
    theta = np.array([0., 0.])
    alpha = 1.0
    for t in range(iters):
        theta[0] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor)*x_nor)
        theta[1] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor) )
    return theta

def plot_fit(theta):

    Error = sum((theta[0]*x_nor + theta[1] - y_nor)**2) # simple least squares error, which underlies our OLS example

    # compute partial derivative (i.e. gradient) for theta 1 and 2
    def_theta1 = sum((theta[0] * x_nor + theta[1] - y_nor)*x_nor)
    def_theta2 = sum((theta[0] * x_nor + theta[1] - y_nor))

    # plot

    plt.figure(figsize = (8,6))
    plt.scatter(x_nor, y_nor, marker = 'x')
    plt.xlabel("Nomalized Temperature")
    plt.ylabel("Normalized Demand")
    xlim, ylim =(plt.gca().get_xlim(), plt.gca().get_ylim())

    plt.plot(xlim, [theta[0]*xlim[0]+theta[1], theta[0]*xlim[1]+theta[1]], 'C1')
    plt.xlim(xlim)
    plt.ylim(ylim)
    print('Theta = ', theta, 'Error = ',Error,'def_theta1 = ',def_theta1, 'def_theta2 = ', def_theta2 )
    #plt.savefig('gradient decent '+str(theta[0])+'.png', dpi = 300)
```

```python
plot_fit(gradient_descent(0))
```

```python
plot_fit(gradient_descent(1))
```

```python
plot_fit(gradient_descent(2))
```

```python
plot_fit(gradient_descent(3))
```

```python
plot_fit(gradient_descent(4))
```

```python
plot_fit(gradient_descent(5))
```

```python
plot_fit(gradient_descent(10))
```

```python
plot_fit(gradient_descent(100))
```

```python
plot_fit(gradient_descent(1000))
```

### Error versus iteration

We can also look at the average error versus iteration.

```python
def gradient_descent_err(iters):
    err = []
    theta = np.array([0., 0.])
    alpha = 1.0
    for t in range(iters):
        err.append(np.mean((theta[0] * x_nor + theta[1] - y_nor)**2))
        theta[0] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor)*x_nor)
        theta[1] -= alpha/len(xp) * 2 * sum((theta[0] * x_nor + theta[1] - y_nor) )
    return np.array(err)
plt.figure(figsize = (8,6))
plt.plot(np.arange(0,50), gradient_descent_err(50))
plt.xlabel("Iteration")
plt.ylabel("Average error")
plt.show()
```

### Getting the answer back in the original coordinates

Fortunately, we don't need to resort to solving the system in the original coordinates, we can simply solve on our normalized data and then find the corresponding equations for the original data.  Specifically, since our model gives the approximation
\begin{equation}
\begin{split}
\tilde{y} & \approx \tilde{x} \cdot \theta_1 + \theta_2\\
\Longrightarrow \;\; \frac{y-a}{b} & \approx \frac{x-c}{d} \cdot \theta_1 + \theta_2 \\
\Longrightarrow \;\; y-a & \approx ((x-c)\cdot \theta_1)/d)\cdot b + b \theta_2 \\
\Longrightarrow \;\; y & \approx x \cdot (b \theta_1/d) + b \theta_2 + a - c b \theta_1/d\\
\Longrightarrow \;\; y & \approx x \cdot \hat{\theta}_1 + \hat{\theta}_2\\
\end{split}
\end{equation}
where
\begin{equation}
a = \min_i y^{(i)}, \;\; b = \max_i y^{(i)} - \min_i y^{(i)}, \;\; c = \min_i x^{(i)}, \;\; d = \max_i x^{(i)} - \min_i x^{(i)},
\end{equation}
and where we define
\begin{equation}
\begin{split}
\hat{\theta}_1 & = b \theta_1/d \\
\hat{\theta}_2 & = b \theta_2 + a - c \cdot(b \theta_1/d).
\end{split}
\end{equation}
That might seem like a lot, but all it's saying is that there is an easy formula to convert between the solution we get for the normalized data and the unnormalized data.

### Getting the answer back in the original coordinates

Fortunately, we don't need to resort to solving the system in the original coordinates, we can simply solve on our normalized data and then find the corresponding equations for the original data.  Specifically, since our model gives the approximation
\begin{equation}
\begin{split}
\tilde{y} & \approx \tilde{x} \cdot \theta_1 + \theta_2\\
\Longrightarrow \;\; \frac{y-a}{b} & \approx \frac{x-c}{d} \cdot \theta_1 + \theta_2 \\
\Longrightarrow \;\; y-a & \approx ((x-c)\cdot \theta_1)/d)\cdot b + b \theta_2 \\
\Longrightarrow \;\; y & \approx x \cdot (b \theta_1/d) + b \theta_2 + a - c b \theta_1/d\\
\Longrightarrow \;\; y & \approx x \cdot \hat{\theta}_1 + \hat{\theta}_2\\
\end{split}
\end{equation}
where
\begin{equation}
a = \min_i y^{(i)}, \;\; b = \max_i y^{(i)} - \min_i y^{(i)}, \;\; c = \min_i x^{(i)}, \;\; d = \max_i x^{(i)} - \min_i x^{(i)},
\end{equation}
and where we define
\begin{equation}
\begin{split}
\hat{\theta}_1 & = b \theta_1/d \\
\hat{\theta}_2 & = b \theta_2 + a - c \cdot(b \theta_1/d).
\end{split}
\end{equation}
That might seem like a lot, but all it's saying is that there is an easy formula to convert between the solution we get for the normalized data and the unnormalized data.

```python
theta = gradient_descent(1000)
```

```python
theta
```

**Exercise**: Using the above expression, re-scale the theta parameters to the original scale by creating a new array entitled `theta_hat`.

```python
theta[0]
```

```python
# YOUR CODE HERE
```

```python
theta_hat  # not that
```

Finally we plot the derived linear regression line

```python
fig,ax = plt.subplots(figsize = (8,6))

ax.scatter(xp, yp, marker='x')

xlim, ylim =(plt.gca().get_xlim(), plt.gca().get_ylim())
ax.plot(xlim, [theta_hat[0]*xlim[0]+theta_hat[1], theta_hat[0]*xlim[1]+theta_hat[1]], 'C1')

ax.set_xlim(xlim)
ax.set_ylim(ylim)
ax.set_xlabel("High Temperature (°C)")
ax.set_ylabel("Peak Demand (GW)")
print(theta, theta_hat)
```

## `Libraries for machine learning in Python`

Finally, we conclude with some information about the types of libraries we will use to run machine learning algorithms in Python.  Although there are a number of machine learning packages available, by far the most popular Python library for general-purpose "classical" machine learning (this is in contrast to packages focused specficially on deep learning, such as [TensorFlow](http://www.tensorflow.org) and [Keras](https://keras.io/getting_started/)) is the [scikit-learn](http://scikit-learn.org/) library.  Scikit Learn is general purpose machine learning library with a number of common machine learning algorithms built in.

One important note, however, is that (despite some ongoing efforts to make it more scalable), scikit-learn is still best suited for small to medium-scale problems (say with ~10,000s of examples and ~1,000s of features).  For these size problems, most of the algorithms contained in the library will work reasonably fast, and the library has the advantage that one can train many different types of algorithms all with the same interface.  However, if you have data that is much bigger than this, then the algorithms start to get fairly slow compared to other more specialized libraries, and you are likely better off using an alternative library.

Another important caveat, and this is one that sadly often gets ignored, is that unlikely other software libraries, you _need_ to have some (even just basic) understanding of what the algorithms do in order to use scikit-learn effectively.  This is because virtually all algorithms will have some substantial number of hyperparameters, settings to the algorithm that can drastically effect performance (and really, affect _all_ the underlying aspects of the algorithm itself, the hypothesis, loss, and optimization problem).  Sadly, a surprisingly large number of the statements people make about data science techniques seem less about the actual algorithms and more about whatever default settings scikit-learn happens to have for each algorithm.  This is why you get people saying things like "support vector machines worked better than neural networks for this problem", which is a completely meaningless statement unless you know _what sort_ of support vector machine, and _what architecture_ neural network.  Maybe in 10 years we will be at a place where the ML algorithms truly are "self-contained", and practitioners don't need to know anything about the underlying algorithms to get good performance (certainly, some researchers and companies are attempting to develop tools that move in this direction).  But for the vast majority of tasks, we are still a lot ways away from this point, and you do absolutely need to understand the algorithms to get reasonable performance.

### Linear regression in scikit-learn

Let's look at how we specify a model, fit it to data, and make predictions in scikit-learn.  These three tasks form the common usage pattern for most interation with scikit-learn.  Let's first prepare our data.  Note that scikit-learn by default will fit a separate intercept term for linear regression models, so we don't include the all-ones entry in our features.

```python
X = df_summer["High_temp"].values.reshape((-1,1)) # if we pass a 1-feature array we need to re-shape it! This is not required for multi-dimenisonal arrays
y = df_summer["MAX"]
```

Now we can import and initialize our model. In general, scikit-learn has a different class for each different type of learning algorithm. In this case, we are importing the LinearRegression class. When we initialize the class, we pass various parameters to the constructor. In this case, we are specifying that we will fit an intercept term (i.e., we will not pass it as an explicit feature), and that we will not normalize the data. These are in fact the default parameters of the linear regression class, but we include them explicitly just for illustration.

Note that `scikit-learn` also has a pre-implemented scaling functionality such as the `StandardScaler` class ([see here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)). For convenience we will not scale the data here. Gradient descent is still able to find an optimal solution. For higher dimensional inputs, however, scaling is highly recommended. We will cover this in future workshops!

```python
# initialize model
linear_model = LinearRegression(fit_intercept=True, normalize=False)
```

```python
linear_model
```

Note that when we create this class we haven't actually passed any data to the system.  This is the standard interface for scikit-learn classes: the constructor just initializes the hyperparameters of the model, and when we actually want to fit it to data, we call the `model.fit()` function.

```python
# train model
linear_model.fit(X, y)
print(linear_model.coef_, linear_model.intercept_)
```

```python
# compare to own implementation (at 1000 iteration of gradient descent)
theta_hat
```

Finally, when we want to make a prediction on a new data point, we call the model.predict() function, passing in the feature values for the new points we want to predict. In the following example, we would be predicting what the peak demand would be given a 25 degree peak temperature.

```python
linear_model.intercept_
```

```python
## Predict the peak load at 40 deg c using our defined theta_hat
```

```python
# predict on new data
Xnew = np.array([[40]]) #predict peak load at 25 deg c

print(linear_model.predict(Xnew))
```

Note that we can pass multiple points to predict at once.

```python
Xnew = np.array([[25],[23]])
print(linear_model.predict(Xnew))
```

Using Scikit-Learn Library for fitting linear regression line:

```python
lr = LinearRegression()
lr.fit(xp.values.reshape(-1,1), yp)
model_pred = lr.predict(xp.values.reshape(-1,1))

fig,ax = plt.subplots(figsize = (8,6))

ax.scatter(xp, yp, marker="x")
ax.set_xlabel("High Temperature (°C)")
ax.set_ylabel("Peak Demand (GW)")

ax.plot(xp, model_pred, c='C2')
ax.legend(['Squared loss fit','Observed days'])
#plt.savefig('summer data_peak demand_line.pdf')
print(lr.coef_, lr.intercept_)
```

## `Regression Model Evaluation`

### Metrics

```python
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
```

```python
y_true = df_summer["MAX"]
y_pred = linear_model.predict(df_summer["High_temp"].values.reshape((-1,1)))
```

#### (Root) Mean Squared Error ((R)MSE)
The MSE calculates the mean of the squared residuals. As the fitting of the linear regression model is (typically) carried out by minimizing the squared error (see loss function in lecture slides), the MSE corresponds directly to the model optimization. The unit of the MSE is the target unit squared, i.e. in the above example using energy consumption in GW, the unit of the MSE is $(GW)^2$. As the residuals (the difference of prediction and observed values) are squared, large deviations are penalized stronger than small deviations. Therefore, the weight of outliers increases using MSE as a metric. This is useful in applications, where small prediction errors are not important, but large errors have to be avoided.

As the squared unit is hard to interpret, the RMSE can be used instead of the MSE. The RMSE is just the square root of the MSE, meaning that it is monotonic with respect to the MSE - a bigger MSE leads to a bigger RMSE and vice versa.

```python
print("Mean Squared Error:",mean_squared_error(y_pred, y_true),"(GW)^2")
print("Root Mean Squared Error:",mean_squared_error(y_pred, y_true)**0.5,"GW")
```

The RMSE can be easily interpreted as it can be set into relation with the c. 1.8 GW average demand mean. Generally, MSE and RMSE have always to be assessed given the model under evaluation. A MSE or RMSE with a certain value has no meaning unless the order of magnitude of the dataset is known (with an exception of the very unlikely case that the MSE is zero).

#### Mean Absolute Error (MAE)
Just as the RSME, the MAE is easy to interpret. It is just the mean absolute value of the error term. When there are no or little major outliers, RMSE and MAE often have the same order of magnitude, with the MAE always being smaller than the RMSE. While a linear regression is typically fitted using the least squares method, the MAE can still be a valuable metric. It is applicable when large errors are not disproportionately worse than smaller errors. For example, when prediciting monetary values, an error of 100 USD might always be twice as bad as an error of 50 USD and so on. If this is the case, the MAE can actually be the more suitable error metric. As the RSME (and the MAE for that matter), the MAE does only make sense when knowing the order of magnitude of the predicted values. On its own, it does not imply good or bad model fit.

```python
print("Mean Absolute Error:",mean_absolute_error(y_pred, y_true),"GW")
```

#### Coefficient of Determination ($R^2$)
The coefficient of determination (typically pronunced as "R squared"), is a metric of how well a model explains variance seen in the data. $R^2$ indicates the ratio of the explained and the overall variance (given some assumptions that we will not discuss here). Its value is always between 0 and 1. It can therefore be used as a means of comparison not only between regression methods, but also between completely different datasets - or even without knowing the data at all. For example, a value of 0.9 is always good, a value of 0.1 is always bad. Still, what threshold you would define as a good model fit depends on the domain of application. If you except a high degree of randomness in your data, it is harder to explain your variance using a predictor. If you expect your data to be highly deterministic, then it should be easily explainable using suitable features and prediction methods.

```python
print("Coefficient of determination:",r2_score(y_pred, y_true))
```

#### Other metrics
Scikit-learn does include other metrics that are not used as often as the ones explained above. To read up on those, just visit Scikit-learn's website. You can also define functions to calculate your own error metrics - depending on the application and with sound reasoning. E.g., it could make sense to weigh negative deviations higher than positive deviations, or the relative error instead of a form of absolute error could be decisive.

### References
[Model Evaluation](http://scikit-learn.org/stable/modules/model_evaluation.html)

__Note that the metrcis calculated above represent the training loss! They do not say anything about predictive performance!__

## End of Class Exercise

**Exercise 1**:

- Fit a linear regression by using `scikit-learn` library for average temperature and average demand data.
- Plot the data and fitted line. Annotate your graph and size it appropriately.
- Present relevant test metrics to quantify the __training__ loss

```python
# YOUR CODE HERE

# define target and feature vectors

# initialize lin model

# fit model

# predict training data

# define y true

```

```python
#Plot the data
```

```python
# Present test_metrics
```

**Exercise 2**:

(we will cover this in detail in next weeks tutorial)

- Create an `IsWeekday` feature and examine its effect on peak demand and high temperature data.
- Using `Scikit learn` build a linear model that incorporates `IsWeekday` as an additional feature.
- Plot the data and fitted line. Annotate your graph and size it appropriately.
- Provide appropriate test metrics

```python
# YOUR CODE HERE

# see WS08 for solution

```

---


Basic Guide for Data Analysis

 ## 1. Libraries

It's important to verify if you are importing all the libraries

```python
%matplotlib notebook
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.optimize as optimization
import scipy.odr.odrpack as odrpack
```

### Numpy

$*$ NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays. In this case array is a data structure that can be interpreted as $\mathbb{R}^{n}$ vectors coordinates.

$*$ We can also use mathematical tools with the best precision on python

$*$ The best library for Numerical Methods. Like integration, solution for linear and non linear ODE.

$*$ All the information can be found on the reference of the library https://numpy.org/doc/stable/reference/.
I encorage you to search your questions on StackOverflow

```python
errors = np.array([0.5,0.6,0.4,0.3,0.6])
incertezas = errors*2
errors
```

    array([0.5, 0.6, 0.4, 0.3, 0.6])

$-$ we can change one or more component of the array. ( Note that the values changed due to the operation above )

```python
errors[1:4] = errors[1:4]*np.cos(2)
errors
```

    array([ 0.5       , -0.2496881 , -0.16645873, -0.12484405,  0.6       ])

$-$ we can slice arrays on the components that we're interesed.

$-$ It's important to note is that python consider open intervals

###### When we declare a variable using an array slicing we dont have a copy of it, wich means that changing one part of the array we change the other variable. If you need to make a copy you can just use np.copy(array)

```python
slice_errors = errors[:3]
slice_errors[2] = 0

errors
```

    array([ 0.5       , -0.2496881 ,  0.        , -0.12484405,  0.6       ])

$-$ Now we gonna use NumPy to create values for x,y and $\sigma_y$

```python
x = np.linspace(0,100,10)

x
```

    array([  0.        ,  11.11111111,  22.22222222,  33.33333333,
            44.44444444,  55.55555556,  66.66666667,  77.77777778,
            88.88888889, 100.        ])

### Model

```python
def f(x, b, a):
    return a*x + b

# Creating the y values with the model
y = f(x,0.5,1)

# Random values between [0.005,0.006]
sigma_y = np.random.uniform(low=0.005, high=0.006, size=(10,))

sigma_y
```

    array([0.00502619, 0.00520858, 0.00510368, 0.005735  , 0.00598083,
           0.00593097, 0.00505421, 0.00596515, 0.00502132, 0.00565782])

## SciPy

$*$ SciPy is a free and open-source Python library used for scientific computing and technical computing. SciPy contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering.

$*$ The reference https://docs.scipy.org/doc/scipy/reference/index.html

### Scipy.optimization

$*$ Scipy.optimization is a sub-library of SciPy for optimization, wich means, in our case, linear and non linear regression.

###### $*$ The curve_fit method just consider $\sigma_y$ once it uses the Ordinary Least Squares ( You can propagate the uncertainty on x to y wich is the Total Least Square)
$*$ The reference https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html

```python
# xo is the parameters guess
xo = [0,0]

fit = optimization.curve_fit(f, x, y, p0=xo, sigma=sigma_y,absolute_sigma=True)

fit
```

    (array([0.5, 1. ]),
     array([[ 9.44539149e-06, -1.33549647e-07],
            [-1.33549647e-07,  2.74296898e-09]]))

$-$ The fit variable received 2 arrays. The first represent the parameters and the second the covariance matrix. In the next section we will see how to present this on a more elegant way

We can calculate the $\chiˆ2$ simple by using the formula

$$
\begin{align}
{\chi}^2=\sum_{k=1}^{n} \frac{(y_i - f(x_i))^2}{\sigma_y^2}
\end{align}
$$

```python
a = fit[0][1]
b = fit[0][0]

# To calculate the parameters errors with the covariance matrix
stdevs = np.sqrt(np.diag(fit[1]))
#stdevs is an array with 2 components stdevs[0] is the error of parameter (b) and stdevs[1] do parameter (a)

Chi2 = sum((y - f(x, b, a))**2/(sigma_y**2))
Chi2
```

    5.0686388782132475e-23

## Pandas

$*$ pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.

$*$ We're going to use Pandas to create a table to the parameters and to import the data.

$*$ The reference : https://pandas.pydata.org/docs/user_guide/index.html

```python
df = pd.DataFrame({'Parameter(a)' : [a,stdevs[1]], 'Parameter(b)' : [b,stdevs[0]]}, index=['Valor', 'Incerteza'])
df
```

      Parameter(a)
      Parameter(b)

      Valor
      1.000000
      0.500000

      Incerteza
      0.000052
      0.003073

### Matplotlib
$*$ Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy.

$*$ The reference https://matplotlib.org/tutorials/index.html#introductory

```python
h = plt.figure(1)
h = plt.plot(x,f(x,b,a), label="model", color='black')
h = plt.errorbar(x,y,sigma_y,fmt='.',color='red')
h = plt.grid()
h = plt.ylabel(r'$f(x)$')
h = plt.xlabel(r'$x$')
h = plt.legend()
# This line of code save the plot in the notebook file directory
plt.savefig('teste_1.png', format='png', dpi=150,bbox_inches = "tight")
plt.show(h)
```

    <IPython.core.display.Javascript object>

```python
g = plt.figure(2)
g.set_size_inches(6, 1)
g =plt.errorbar(x,y-f(x,b,a),sigma_y,fmt='.',color='red')
g = plt.hlines(0,x.min(),x.max(),color='black')
g = plt.grid()
g = plt.ylabel("f (x)")
g = plt.xlabel('x')
g = plt.savefig('teste_1_resid.png', format='png', dpi=150,bbox_inches = "tight")
g = plt.show(g)
```

    <IPython.core.display.Javascript object>

## Aditional information

$-$ Since all thoses libraries are getting more famous over time, the content is very vast. Hence, i encorage you to search information on StackOverflow.

$(*)$ Good and basic video of curve fitting in python:
https://www.youtube.com/watch?v=Jl-Ye38qkRc&t=810s

$(*)$ Nice article about curve fitting in python:
https://towardsdatascience.com/basic-curve-fitting-of-scientific-data-with-python-9592244a2509

# 2. Putting into pratice

#### Now, let's use uncertain in x and y

$*$ We'll use data in csv and txt

#### Importing txt with Pandas

```python
data = pd.read_csv('exemplo.txt',sep="\t")

# When the the decimal separator is "," we need to use
# data_1 = pd.read_csv('exemplo.txt', sep="\t", decimal=",")

data.head()
```

      x (cm)
      y (cm)
      sigma_x (cm)
      sigma_y (cm)

      0
      91.0
      -1.67
      0.182
      -0.0835

      1
      91.5
      -2.50
      0.183
      -0.1250

      2
      92.0
      -3.01
      0.184
      -0.1510

      3
      92.5
      -3.91
      0.185
      -0.1960

      4
      93.0
      -4.15
      0.186
      -0.2080

#### Important csv with Pandas

```python
data = pd.read_csv('exemplo.csv')
data.head()
```

      x (cm)
      y (cm)
      sigma_x (cm)
      sigma_y (cm)

      0
      91.0
      -1.67
      0.182
      -0.0835

      1
      91.5
      -2.50
      0.183
      -0.1250

      2
      92.0
      -3.01
      0.184
      -0.1510

      3
      92.5
      -3.91
      0.185
      -0.1960

      4
      93.0
      -4.15
      0.186
      -0.2080

```python
x = data['x (cm)']
y = data['y (cm)']

# x and y are a Series ( data structure similar to Data Frame )
plt.figure(0)
plt.scatter(x,y)
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

### Data Analysis

#### Model

```python
# Importante que notar que agora nossos parâmetros são representados por array
# Outro fator a se notar é a ordem na a f, ou seja, f(p,x) "p" vem primeiro

def f(p,x):
    return p[0]*x**2 + p[1]*x + p[2]
```

##### Using the values $x$, $y$, $\sigma_x$ e $\sigma_y$

```python
x = data['x (cm)']
y = data['y (cm)']
sigma_x = data['sigma_x (cm)']
sigma_y = data['sigma_y (cm)']

```

### Fitting

$*$ I'll use the Orthogonal Distance Regression (ODR) but you can use the Total Least Square (TLS). If you want know more about that you should check: https://towardsdatascience.com/total-least-squares-in-comparison-with-ols-and-odr-f050ffc1a86a#:~:text=Total%20least%20squares(aka%20TLS,often%20say%20a%20fitted%20value).

```python
# model object ( isso é um termo técnico para dizer que o modelo está guardado em um "objeto")
model_object = odrpack.Model(f)
# realdata object ( termo técnico para dizer que dizer que o modelo está guardado em um "objeto")
data_object = odrpack.RealData(x,y,sx=sigma_x, sy=sigma_y)
# Estabelece a relação entre data e modelo, e em geral necessita de das condições iniciais
# No caso do nosso modelo temos 3 parâmetros, ou seja beta0 = lista de 3 elementos
# Set up ODR with the model and data.
odr = odrpack.ODR(data_object, model_object, beta0=[1,1,1])
# Run the regression.
odr.set_job(fit_type=0)  # fit_type = 0 executa a regressão por ODR e fit_type = 2 por MMQ
# O tópico do fit_type é um complicado e vai além do proposito dessa introdução
out = odr.run()
# print the regression
out.pprint()
```

    Beta: [ 2.01583133e-01 -3.87100151e+01  1.85195389e+03]
    Beta Std Error: [7.60198651e-03 1.45258036e+00 6.93460557e+01]
    Beta Covariance: [[ 1.46180611e-04 -2.79299370e-02  1.33306903e+00]
     [-2.79299370e-02  5.33723001e+00 -2.54779487e+02]
     [ 1.33306903e+00 -2.54779487e+02  1.21640756e+04]]
    Residual Variance: 0.39533422646943844
    Inverse Condition #: 7.35277715433665e-08
    Reason(s) for Halting:
      Sum of squares convergence

### Plotting

```python
y_i = f([2.01584576e-01, -3.87102908e+01, 1.85196706e+03],x)

```

```python

ax2 = plt.figure(3)
ax2 = plt.errorbar(x, y, xerr=sigma_x, yerr=sigma_y, linestyle='None', marker='.',color='black')
#ax2 = plt.ylim(-1,2)
#ax2 = plt.xlim(0, 20000)
ax2 = plt.plot(x, y_i, label='model',color='red')
# axis names
ax2 = plt.xlabel('$x (cm)$')
ax2 = plt.ylabel('$y (cm)$')
ax2 = plt.grid()
#Saving the figure
ax2 = plt.savefig('exemplo.png',dpi=150,bbox_inches = "tight")

ax3 = plt.figure(4)
ax3 = plt.errorbar(x, y - y_i
                   ,xerr=sigma_x, yerr=sigma_y, linestyle='None', marker='.',color='black')

ax3 = plt.hlines(0,x.min(),x.max(),color='red')
ax3 = plt.grid()
# axis names
ax3 = plt.xlabel(r'$x (cm)$')
ax3 = plt.ylabel(r'$y(cm) - modelo$')
ax3 = plt.gcf()
#ax3 = plt.ylim(-1,1)
ax3.set_size_inches(6, 1)
#Saving the figure
ax3 = plt.savefig('exemplo_resid.png',dpi=150,bbox_inches = "tight")

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```python

```


# Sampled Softmax

For classification and prediction problems a typical criterion function is cross-entropy with softmax. If the number of output classes is high the computation of this criterion and the corresponding gradients could be quite costly. Sampled Softmax is a heuristic to speed up training in these cases. (see: [Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model](http://www.iro.umontreal.ca/~lisa/pointeurs/importance_samplingIEEEtnn.pdf), [Exploring the Limits of Language Modeling](https://arxiv.org/pdf/1602.02410v1.pdf), [What is Candidate Sampling](https://www.tensorflow.org/extras/candidate_sampling.pdf))

#### Select the notebook runtime environment devices / settings

Before we dive into the details we run some setup that is required for automated testing of this notebook.

```python
import os
import cntk as C

# Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        C.device.try_set_default_device(C.device.cpu())
    else:
        C.device.try_set_default_device(C.device.gpu(0))
```

## Basics

The softmax function is used in neural networks if we want to interpret the network output as a probability distribution over a set of classes $C$ with $|C|=N_C$.

Softmax maps an $N_C$-dimensional vector $z$, which has unrestricted values, to an $N_C$ dimensional vector $p$ with non-negative values that sum up to 1 so that they can be interpreted as probabilities. More precisely:

$$
\begin{align}
p_i &= softmax(z, i)\\
    &= \frac{exp(z_i)}{\sum_{k\in C} exp(z_k)}\\
\end{align}
$$

In what follows we assume that the input $z$ to the softmax is computed from some hidden vector $h$ of dimension $N_h$  in a specific way, namely:

$$ z = W h + b $$

where $W$ is a learnable weight matrix of dimension $(N_c, N_h)$ and $b$ is a learnable bias vector.
We restrict ourselves to this specific choice of $z$ because it helps in implementing an efficient sampled softmax.

In a typical use-case like for example a recurrent language model, the hidden vector $h$ would be the output of the recurrent layers and $C$ would be the set of words to predict.

As a training criterion, we use cross-entropy which is a function of the expected (true) class $t\in C$ and the probability predicted for it:

$$cross\_entropy := -log(p_t)$$

## Sampled Softmax from the outside

For the normal softmax the CNTK Python-api provides the function [cross_entropy_with_softmax](https://cntk.ai/pythondocs/cntk.ops.html?highlight=softmax#cntk.ops.cross_entropy_with_softmax). This takes as input the $N_C$-dimensional vector $z$. As mentioned for our sampled softmax implementation we assume that this z is computed by $ z = W h + b $. In sampled softmax this has to be part of the whole implementation of the criterion.

Below we show the code for `cross_entropy_with_sampled_softmax_and_embedding`. Let’s look at the signature first.

One fundamental difference to the corresponding function in the Python-api (`cross_entropy_with_softmax`) is that in the Python api function the input corresponds to $z$ and must have the same dimension as the target vector, while in cross_entropy_with_full_softmax the input corresponds to our hidden vector $h$ can have any dimension (hidden_dim).
Actually, hidden_dim will be typically much lower than the dimension of the target vector.

We also have some additional parameters `num_samples, sampling_weights, allow_duplicates` that control the random sampling.
Another difference to the api function is that we return a triple (z, cross_entropy_on_samples, error_on_samples).

We will come back to the details of the implementation below.

```python
from __future__ import print_function
from __future__ import division

# Creates a subgraph computing cross-entropy with sampled softmax.
def cross_entropy_with_sampled_softmax_and_embedding(
    hidden_vector,            # Node providing hidden input
    target_vector,            # Node providing the expected labels (as sparse vectors)
    num_classes,              # Number of classes
    hidden_dim,               # Dimension of the hidden vector
    num_samples,              # Number of samples to use for sampled softmax
    sampling_weights,         # Node providing weights to be used for the weighted sampling
    allow_duplicates = True,  # Boolean flag to control whether to use sampling with replacemement
                              # (allow_duplicates == True) or without replacement.
    ):
    # define the parameters learnable parameters
    b = C.Parameter(shape = (num_classes, 1), init = 0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())

    # Define the node that generates a set of random samples per minibatch
    # Sparse matrix (num_samples * num_classes)
    sample_selector = C.random_sample(sampling_weights, num_samples, allow_duplicates)

    # For each of the samples we also need the probablity that it in the sampled set.
    inclusion_probs = C.random_sample_inclusion_frequency(sampling_weights, num_samples, allow_duplicates) # dense row [1 * vocab_size]
    log_prior = C.log(inclusion_probs) # dense row [1 * num_classes]

    # Create a submatrix wS of 'weights
    W_sampled = C.times(sample_selector, W) # [num_samples * hidden_dim]
    z_sampled = C.times_transpose(W_sampled, hidden_vector) + C.times(sample_selector, b) - C.times_transpose (sample_selector, log_prior)# [num_samples]

    # Getting the weight vector for the true label. Dimension hidden_dim
    W_target = C.times(target_vector, W) # [1 * hidden_dim]
    z_target = C.times_transpose(W_target, hidden_vector) + C.times(target_vector, b) - C.times_transpose(target_vector, log_prior) # [1]

    z_reduced = C.reduce_log_sum_exp(z_sampled)

    # Compute the cross entropy that is used for training.
    # We don't check whether any of the classes in the random samples conincides with the true label, so it might
    # happen that the true class is counted
    # twice in the normalising demnominator of sampled softmax.
    cross_entropy_on_samples = C.log_add_exp(z_target, z_reduced) - z_target

    # For applying the model we also output a node providing the input for the full softmax
    z = C.times_transpose(W, hidden_vector) + b
    z = C.reshape(z, shape = (num_classes))

    zSMax = C.reduce_max(z_sampled)
    error_on_samples = C.less(z_target, zSMax)
    return (z, cross_entropy_on_samples, error_on_samples)

```

To give a better idea of what the inputs and outputs are and how this all differs from the normal softmax we give below a corresponding function using normal softmax:

```python
# Creates subgraph computing cross-entropy with (full) softmax.
def cross_entropy_with_softmax_and_embedding(
    hidden_vector,  # Node providing hidden input
    target_vector,  # Node providing the expected labels (as sparse vectors)
    num_classes,    # Number of classes
    hidden_dim      # Dimension of the hidden vector
    ):
    # Setup bias and weights
    b = C.Parameter(shape = (num_classes, 1), init = 0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())

    z = C.reshape( C.times_transpose(W, hidden_vector) + b, (1, num_classes))

    # Use cross_entropy_with_softmax
    cross_entropy = C.cross_entropy_with_softmax(z, target_vector)

    zMax = C.reduce_max(z)
    zT = C.times_transpose(z, target_vector)
    error_on_samples = C.less(zT, zMax)

    return (z, cross_entropy, error_on_samples)
```

As you can see the main differences to the api function `cross_entropy_with_softmax` are:
* We include the mapping $ z = W h + b $ into the function.
* We return a triple (z, cross_entropy, error_on_samples) instead of just returning the cross entropy.

## A toy example

To explain how to integrate sampled softmax let us look at a toy example. In this toy example we first transform one-hot input vectors via some random projection into a lower dimensional vector $h$. The modeling task is to reverse this mapping using (sampled) softmax. Well, as already said this is a toy example.

```python
import numpy as np
from math import log, exp, sqrt
from cntk.logging import ProgressPrinter
import timeit

# A class with all parameters
class Param:
    # Learning parameters
    learning_rate = 0.03
    minibatch_size = 100
    num_minbatches = 100
    test_set_size = 1000
    momentum_time_constant = 5 * minibatch_size
    reporting_interval = 10
    allow_duplicates = False

    # Parameters for sampled softmax
    use_sampled_softmax = True
    use_sparse = True
    softmax_sample_size = 10

    # Details of data and model
    num_classes = 50
    hidden_dim = 10

data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

# Creates random one-hot vectors of dimension 'num_classes'.
# Returns a tuple with a list of one-hot vectors, and list with the indices they encode.
def get_random_one_hot_data(num_vectors):
    indices = np.random.choice(
        range(Param.num_classes),
        size=num_vectors,
        p = data_sampling_distribution()).reshape((num_vectors, 1))
    list_of_vectors = C.Value.one_hot(indices, Param.num_classes)
    return (list_of_vectors, indices.flatten())

# Create a network that:
# * Transforms the input one hot-vectors with a constant random embedding
# * Applies a linear decoding with parameters we want to learn
def create_model(labels):
    # random projection matrix
    random_data = np.random.normal(scale = sqrt(1.0/Param.hidden_dim), size=(Param.num_classes, Param.hidden_dim)).astype(np.float32)
    random_matrix = C.constant(shape = (Param.num_classes, Param.hidden_dim), value = random_data)

    h = C.times(labels, random_matrix)

    # Connect the latent output to (sampled/full) softmax.
    if Param.use_sampled_softmax:
        sampling_weights = np.asarray(softmax_sampling_weights(), dtype=np.float32)
        sampling_weights.reshape((1, Param.num_classes))
        softmax_input, ce, errs = cross_entropy_with_sampled_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim,
            Param.softmax_sample_size,
            softmax_sampling_weights(),
            Param.allow_duplicates)
    else:
        softmax_input, ce, errs = cross_entropy_with_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim)

    return softmax_input, ce, errs

def train(do_print_progress):
    labels = C.input(shape = Param.num_classes, is_sparse = Param.use_sparse)
    z, cross_entropy, errs = create_model(labels)

    # Setup the trainer
    learning_rate_schedule = C.learning_rate_schedule(Param.learning_rate, C.UnitType.sample)
    momentum_schedule = C.momentum_as_time_constant_schedule(Param.momentum_time_constant)
    learner = C.momentum_sgd(z.parameters, learning_rate_schedule, momentum_schedule, True)
    progress_writers = None
    if do_print_progress:
        progress_writers = [ProgressPrinter(freq=Param.reporting_interval, tag='Training')]
    trainer = C.Trainer(z, (cross_entropy, errs), learner, progress_writers)

    minbatch = 0
    average_cross_entropy = compute_average_cross_entropy(z)
    minbatch_data = [0] # store minibatch values
    cross_entropy_data = [average_cross_entropy] # store cross_entropy values

    # Run training
    t_total= 0

    # Run training
    for minbatch in range(1,Param.num_minbatches):
        # Specify the mapping of input variables in the model to actual minibatch data to be trained with
        label_data, indices = get_random_one_hot_data(Param.minibatch_size)
        arguments = ({labels : label_data})

        # If do_print_progress is True, this will automatically print the progress using ProgressPrinter
        # The printed loss numbers are computed using the sampled softmax criterion
        t_start = timeit.default_timer()
        trainer.train_minibatch(arguments)
        t_end = timeit.default_timer()

        t_delta = t_end - t_start
        samples_per_second = Param.minibatch_size / t_delta

        # We ignore the time measurements of the first two minibatches
        if minbatch > 2:
            t_total += t_delta

        # For comparison also print result using the full criterion
        if minbatch % Param.reporting_interval == int(Param.reporting_interval/2):
            # memorize the progress data for plotting
            average_cross_entropy = compute_average_cross_entropy(z)
            minbatch_data.append(minbatch)
            cross_entropy_data.append(average_cross_entropy)

            if do_print_progress:
                print("\nMinbatch=%d Cross-entropy from full softmax = %.3f perplexity = %.3f samples/s = %.1f"
                    % (minbatch, average_cross_entropy, exp(average_cross_entropy), samples_per_second))

    # Number of samples we measured. First two minbatches were ignored
    samples_measured = Param.minibatch_size * (Param.num_minbatches - 2)
    overall_samples_per_second = samples_measured / t_total
    return (minbatch_data, cross_entropy_data, overall_samples_per_second)

def compute_average_cross_entropy(softmax_input):
    vectors, indices = get_random_one_hot_data(Param.test_set_size)
    total_cross_entropy = 0.0
    arguments = (vectors)
    z = softmax_input.eval(arguments).reshape(Param.test_set_size, Param.num_classes)

    for i in range(len(indices)):
        log_p = log_softmax(z[i], indices[i])
        total_cross_entropy -= log_p

    return total_cross_entropy / len(indices)

# Computes log(softmax(z,index)) for a one-dimensional numpy array z in an numerically stable way.
def log_softmax(z,    # numpy array
                index # index into the array
            ):
    max_z = np.max(z)
    return z[index] - max_z - log(np.sum(np.exp(z - max_z)))

np.random.seed(1)

print("start...")
train(do_print_progress = True)
print("done.")
```

    start...

    Minbatch=5 Cross-entropy from full softmax = 3.843 perplexity = 46.645 samples/s = 16981.6
     Minibatch[   1-  10]: loss = 2.321515 * 1000, metric = 0.00% * 1000;

    Minbatch=15 Cross-entropy from full softmax = 3.483 perplexity = 32.561 samples/s = 17406.4
     Minibatch[  11-  20]: loss = 2.042852 * 1000, metric = 0.00% * 1000;

    Minbatch=25 Cross-entropy from full softmax = 3.061 perplexity = 21.339 samples/s = 16680.9
     Minibatch[  21-  30]: loss = 1.701563 * 1000, metric = 0.00% * 1000;

    Minbatch=35 Cross-entropy from full softmax = 2.781 perplexity = 16.142 samples/s = 17080.2
     Minibatch[  31-  40]: loss = 1.435229 * 1000, metric = 0.00% * 1000;

    Minbatch=45 Cross-entropy from full softmax = 2.452 perplexity = 11.609 samples/s = 17973.4
     Minibatch[  41-  50]: loss = 1.268124 * 1000, metric = 0.00% * 1000;

    Minbatch=55 Cross-entropy from full softmax = 2.262 perplexity = 9.602 samples/s = 17152.6
     Minibatch[  51-  60]: loss = 1.052258 * 1000, metric = 0.00% * 1000;

    Minbatch=65 Cross-entropy from full softmax = 2.092 perplexity = 8.100 samples/s = 17673.8
     Minibatch[  61-  70]: loss = 0.979096 * 1000, metric = 0.00% * 1000;

    Minbatch=75 Cross-entropy from full softmax = 1.924 perplexity = 6.850 samples/s = 17338.6
     Minibatch[  71-  80]: loss = 0.908541 * 1000, metric = 0.00% * 1000;

    Minbatch=85 Cross-entropy from full softmax = 1.751 perplexity = 5.761 samples/s = 16892.4
     Minibatch[  81-  90]: loss = 0.825795 * 1000, metric = 0.00% * 1000;

    Minbatch=95 Cross-entropy from full softmax = 1.660 perplexity = 5.261 samples/s = 17125.3
    done.

In the above code we use two different methods to report training progress:
1. Using a function that computes the average cross entropy on full softmax.
2. Using the built-in ProgressPrinter

ProgressPrinter reports how the value of the training criterion changes over time.
In our case the training criterion is cross-entropy from **sampled** softmax.
The same is true for the error rate computed by progress printer, this is computed only for true-class vs sampled-classes and will therefore underestimate the true error rate.

Therefore while ProgressPrinter already gives us some idea how training goes on, if we want to compare the behavior for different sampling strategies (sample size, sampling weights, ...) we should not rely on numbers that are computed only using the sampled subset of classes.

## Importance sampling

Often the we don't have uniform distribution for the classes on the output side. The typical example is when we have words as output classes. A typical example are words where e.g. 'the' will be much more frequent than most others.

In such cases one often uses a non uniform distribution for drawing the samples in sampled softmax but instead increases the sampling weight for the frequent classes. This is also called importane sampling.
In our example the sampling distribution is controlled by the weight array `softmax_sampling_weights`.

As an example let's look at the case where the classes are distrubted according to zipf-distrubtion like:
$$
p[i] \propto \frac{1}{i+5},
$$
actually we use this distribution already in our example.

How does training behavior change if we switch uniform sampling to sampling with the zipfian distribution in sampled softmax?

```python
# We want to lot the data
import matplotlib.pyplot as plt
%matplotlib inline

# Define weights of zipfian distributuion
def zipf(index):
    return 1.0 / (index + 5)

# Use zipifian distribution for the classes
def zipf_sampling_weights():
    return np.asarray([ zipf(i) for i in range(Param.num_classes)], dtype=np.float32)

data_sampling_distribution = lambda: zipf_sampling_weights() / np.sum(zipf_sampling_weights())

print("start...")

# Train using uniform sampling (like before)
np.random.seed(1)
softmax_sampling_weights = lambda: np.repeat(1.0/Param.num_classes, Param.num_classes)
minibatch_data, cross_entropy_data, _ = train(do_print_progress = False)

# Train using importance sampling
np.random.seed(1)
softmax_sampling_weights = zipf_sampling_weights
minibatch_data2, cross_entropy_data2, _ = train(do_print_progress = False)

plt.plot(minibatch_data, cross_entropy_data, 'r--',minibatch_data, cross_entropy_data2, 'b--')
plt.xlabel('number of mini-batches')
plt.ylabel('cross entropy')
plt.show()

```

In the example above we compare uniform sampling (red) vs sampling with the same distribution the classes have (blue).
You will need to experiment to find the best settings for all the softmax parameters.

## What speedups to expect?

The speed difference between full softmax and sampled softmax in terms of training instances depends strongly on the concrete settings, namely
* Number of classes. Typically the speed-up will increase the more output classes you have.
* Number of samples used in sampled softmax
* Dimension of hiddlen layer input
* Minibatch size
* Hardware

Also you need to test how much you can reduce sample size without degradation of the result.

```python
print("start...")

# Reset parameters
class Param:
    # Learning parameters
    learning_rate = 0.03
    minibatch_size = 8
    num_minbatches = 100
    test_set_size = 1 # we are only interrested in speed
    momentum_time_constant = 5 * minibatch_size
    reporting_interval = 1000000 # Switch off reporting to speed up
    allow_duplicates = False

    # Parameters for sampled softmax
    use_sampled_softmax = True
    use_sparse = True
    softmax_sample_size = 10

    # Details of data and model
    num_classes = 50000
    hidden_dim = 10

data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)
softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

sample_sizes = [5, 10, 100, 1000]
speed_with_sampled_softmax = []

# Get the speed with sampled softmax for different sizes
for sample_size in sample_sizes:
    print("Measuring speed of sampled softmax for sample size %d ..." % (sample_size))
    Param.use_sampled_softmax = True
    Param.softmax_sample_size = sample_size
    _, _,  samples_per_second = train(do_print_progress = False)
    speed_with_sampled_softmax.append(samples_per_second)

# Get the speed with full softmax
Param.use_sampled_softmax = False
print("Measuring speed of full softmax ...")
_, _,  samples_per_second = train(do_print_progress = False)
speed_without_sampled_softmax = np.repeat(samples_per_second, len(sample_sizes))

# Plot the speed of sampled softmax (blue) as a function of sample sizes
# and compare it to the speed with full softmax (red).
plt.plot(sample_sizes, speed_without_sampled_softmax, 'r--',sample_sizes, speed_with_sampled_softmax, 'b--')
plt.xlabel('softmax sample size')
plt.ylabel('speed: instances / second')
plt.title("Speed 'sampled softmax' (blue) vs. 'full softmax' (red)")
plt.ylim(ymin=0)
plt.show()

```


```
import numpy as np
import matplotlib.pyplot as plt
```

## Multivariate normal distribution

The multivariate normal distribution is a **multidimensional generalisation** of the one-dimensional normal distribution. It represents the distribution of a multivariate random variable that is made up of **multiple random variables** that can be correlated with eachother.

Like the normal distribution, the multivariate normal is defined by sets of parameters: the **mean vector** $\mu$, which is the expected value of the distribution; and the **covariance matrix** $\Sigma$, which measures how dependend two random variables are and how they change together.

The multivariate normal with dimensionality $d$ has a joint probability density given by:

\begin{equation}
p(\mathbf{x} \mid \mathbf{\mu}, \Sigma) = \frac{1}{\sqrt{(2\pi)^d \lvert\Sigma\rvert}} \exp{ \left( -\frac{1}{2}(\mathbf{x} - \mathbf{\mu})^T \Sigma^{-1} (\mathbf{x} - \mathbf{\mu}) \right)}
\end{equation}

Where $\mathbf{x}$ is a random vector of size $d$, $\mathbf{\mu}$ is the mean vector, $\mathbf{\Sigma}$ is the (symmetric , positive definite) covariance matrix (of size $d \times d$, and $|\Sigma|$ its determinant. We denote this multivariate normal distribution as:

\begin{equation}
\mathcal{N}(\mathbf{\mu}, \Sigma)
\end{equation}

```
def multivariate_normal(X, Y, d, mean, covariance):
    # Pack X and Y into a single 3-dimensional array
    pos = np.dstack([X,Y])

    n = mu.shape[0]
    Sigma_det = np.linalg.det(Sigma)
    Sigma_inv = np.linalg.inv(Sigma)
    N = np.sqrt((2*np.pi)**n * Sigma_det)
    # This einsum call calculates (x-mu)T.Sigma-1.(x-mu) in a vectorized
    # way across all the input variables.
    fac = np.einsum('...k,kl,...l->...', pos-mu, Sigma_inv, pos-mu)

    return np.exp(-fac / 2) / N
```

Examples of two bivariate normal distributions are plotted below.

The figure on the left is a bivariate distribution with the covariance between $x_1$ and $x_2$ set to $0$  so that these 2 variables are independent:

\begin{equation}
\mathcal{N}\left(
\begin{bmatrix}
0 \\
0
\end{bmatrix},
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}\right)
\end{equation}

```
N = 60
X = np.linspace(-3, 3, N)
Y = np.linspace(-3, 4, N)
X, Y = np.meshgrid(X, Y)

# Mean vector and covariance matrix
mu = np.array([0., 0])
Sigma = np.array([[ 1. , 0], [0,  1]])

# The distribution on the variables X, Y packed into pos.
Z = multivariate_normal(X, Y, 2, mu, Sigma)
```

```
# Create a surface plot and projected filled contour plot under it.
fig = plt.figure(figsize=(20,7))

# Figure 1
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.set_zlim(-0.15,0.2)

# Figure 2
ax = fig.add_subplot(1, 2, 2, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')
cset = ax.contourf(X, Y, Z, zdir='z', offset=-0.15, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.set_zlim(-0.15,0.2)

plt.show()
```

```
# Create a surface plot and projected filled contour plot under it.
fig = plt.figure(figsize=(20,7))

# Figure 1
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.view_init(90, 0)

# Figure 2
ax = fig.add_subplot(1, 2, 2, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.view_init(0, 0)

plt.show()
```

The figure on the right is a bivariate distribution with the covariance between $x_1$ and $x_2$ set to be different than $0$ so that both variables are correlated. Increasing $x_1$ will increase the probability that $x_2$ will also increase:

\begin{equation}
\mathcal{N}\left(
\begin{bmatrix}
0 \\
1
\end{bmatrix},
\begin{bmatrix}
1 & 0.8 \\
0.8 & 1
\end{bmatrix}\right)
\end{equation}

```
N = 60
X = np.linspace(-4, 4, N)
Y = np.linspace(-4, 4, N)
X, Y = np.meshgrid(X, Y)

# Mean vector and covariance matrix
mu = np.array([0., 1.])
Sigma = np.array([[ 1. , 0.8], [0.8,  1]])

# The distribution on the variables X, Y packed into pos.
Z = multivariate_normal(X, Y, 2, mu, Sigma)

Sigma
```

    array([[1. , 0.8],
           [0.8, 1. ]])

```
# Create a surface plot and projected filled contour plot under it.
fig = plt.figure(figsize=(20,7))

# Figure 1
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.set_zlim(-0.15,0.2)

# Figure 2
ax = fig.add_subplot(1, 2, 2, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')
cset = ax.contourf(X, Y, Z, zdir='z', offset=-0.15, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.set_zlim(-0.15,0.2)

plt.show()
```

```
# Create a surface plot and projected filled contour plot under it.
fig = plt.figure(figsize=(20,7))

# Figure 1
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.view_init(90, 0)

# Figure 2
ax = fig.add_subplot(1, 2, 2, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')

# Adjust the limits, ticks and view angle
ax.view_init(0, 0)

plt.show()
```

```
from scipy.stats import multivriate_noarmal

pos2 = np.dstack([X,Y])

rv = multivariate_normal(mu, Sigma)
Z = rv.pdf(pos2)
plt.contourf(X, Y, Z, cmap='viridis')
```