## Optimal Power Flow
**Power Systems Optimization**

by Michael R. Davidson and Jesse D. Jenkins (last updated: October 4, 2020)

This notebook provides an introductory glimpse of the Optimal Power Flow (OPF) problem—which minimizes the short-run production costs of meeting electricity demand at a number of connected locations from a given set of generators subject to various technical and flow limit constraints. This will be our first treatment of a **network**, which is critical to all power systems.

We will first introduce a model of transmission flows that assumes we can control the flow along each path, in what is called a "transport model". This is a straightforward extension to Economic Dispatch (ED), where we have multiple supply and demand balance constraints, and a new set of flow constraints. This is also similar to other common optimization problems such as fleet routing of shipments.

We will then introduce a linear approximation to the optimal power flow problem known as "DC-OPF", where we begin to incorporate some of the physics involved in how electricity flows along transmission lines. With this, we recognize that given "injections" (i.e., generation) and withdrawals (i.e., demand) of power, flows along lines are not independently controllable. This can (very frequently) result in hitting flow constraints before we would if could control them as in the transport problem.

Our model does not explore the full functionality of DC-OPF, which can include inter-temporal constraints, additional generation constraints (e.g., on voltage), security constraints to ensure stability in the case of contingencies, and network losses.

Full "AC optimal power flow" models are also beyond the scope of this notebook, as the full set of physics associated with the interactions of AC flows introduces non-linearities that are much harder to solve.

We will start off with some simple systems, whose solutions can be worked out manually without resorting to any mathematical optimization model and software. But, eventually we will be solving larger system, thereby emphasizing the importance of such software and mathematical models.

## Introduction to OPF

Optimal Power Flow (OPF) is a power system optimal scheduling problem which captures the physics of electricity flows, adding a layer of complexity and more realism to the Economic Dispatch (ED) problem. OPF usually attempts to capture the entire network topology by representing the transmission line interconnections between different nodes (also known as buses) including various electrical parameters, such as resistance, series reactance, shunt admittance, etc. The full "AC" OPF turns out to be an extremely hard problem to solve (usually NP-hard). Hence, system operators and power marketers usually go about solving a linearized version of it, called the DC-OPF. The DC-OPF approximation works satisfactorily for bulk power transmission networks as long as such networks are not operated at the brink of instability or under heavily loaded conditions.

## "Transport" model
We will first examine the case where we allow for transmission but ignore the physics of electricity flows, and instead treat it like transporting an ordinary commodity.

$$
\begin{align}
\min \ & \sum_{g \in G} VarCost_g \times GEN_g & \\
\text{s.t.} & \\
 & \sum_{g \in G_i} GEN_g - Demand_i = \sum_{j \in J_i} FLOW_{ij} & \forall \quad i \in \mathcal{N}\\
 & FLOW_{ij} \leq MaxFlow_{ij} & \forall \quad i \in \mathcal{N}, \forall j \in J_i \\
 & FLOW_{ji} = - FLOW_{ij} & \forall \quad i, j \in \mathcal{N} \\
 & GEN_g \leq Pmax_g & \forall \quad g \in G \\
 & GEN_g \geq Pmin_g & \forall \quad g \in G
\end{align}
$$

We introduce a few new **sets** in the above:
- $\mathcal{N}$, the set of all nodes (or buses) in the network
- $J_i$, the subset of nodes that are connected to node $i$
- $G_i \subset G$, the subset of generators located at node $i$

The **decision variables** in the above problem are:

- $GEN_{g}$, the generation (in MW) produced by each generator, $g$
- $FLOW_{ij}$, the flow (in MW) along the line from $i$ to $j$

The **parameters** are:

- $Pmin_g$, the minimum operating bounds for the generator (based on engineering or natural resource constraints)
- $Pmax_g$, the maximum operating bounds for the generator (based on engineering or natural resource constraints)
- $Demand_i$, the demand (in MW) at bus $i$
- $MaxFlow_{ij}$, the maximum allowable flow along the line from $i$ to $j$
- $VarCost_g$, the variable cost of generator $g$

Notice how the problem above is equivalent to producing a single type of good at a set of factories and shipping them along capacity-limited corridors (roads, rail lines, etc.) to meet a set of demands in other locations.

### 1. Load packages

```julia
using JuMP, GLPK
using Plots; plotly();
using VegaLite  # to make some nice plots
using DataFrames, CSV, PrettyTables
ENV["COLUMNS"]=120; # Set so all columns of DataFrames and Matrices are displayed
```

    ┌ Info: For saving to png with the Plotly backend ORCA has to be installed.
    └ @ Plots /Users/michd/.julia/packages/Plots/ViMfq/src/backends.jl:373

### 2. Load and format data

We will load a modified 3-bus case stored in the [MATPOWER case format](https://matpower.org/docs/ref/matpower5.0/caseformat.html). It consists of:

- two generator buses with variable costs 50 and 100
- one load bus
- three lines connecting the buses

The location and numbering of the components:

```julia
datadir = joinpath("opf_data")
gen = CSV.read(joinpath(datadir,"gen.csv"), DataFrame);
gencost = CSV.read(joinpath(datadir,"gencost.csv"), DataFrame);
branch = CSV.read(joinpath(datadir,"branch.csv"), DataFrame);
bus = CSV.read(joinpath(datadir,"bus.csv"), DataFrame);

# Rename all columns to lowercase (by convention)
for f in [gen, gencost, branch, bus]
    rename!(f,lowercase.(names(f)))
end

# create generator ids
gen.id = 1:nrow(gen);
gencost.id = 1:nrow(gencost);

# create line ids
branch.id = 1:nrow(branch);
# add set of rows for reverse direction with same parameters
branch2 = copy(branch)
branch2.f = branch2.fbus
branch2.fbus = branch.tbus
branch2.tbus = branch2.f
branch2 = branch2[:,names(branch)]
append!(branch,branch2)

# Here are the buses:
bus
```

	bus_i	type	pd	qd	gs	bs	area	vm	va	basekv	zone	vmax	vmin

	Int64	Int64	Int64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Float64	Float64

3 rows × 13 columns

1	1	2	0	0.0	0	0	1	1	0	230	1	1.1	0.9

2	2	2	0	0.0	0	0	1	1	0	230	1	1.1	0.9

3	3	1	600	98.61	0	0	1	1	0	230	1	1.1	0.9

Columns pd and qd indicate the active and reactive power withdrawal at the bus. (We will ignore qd for this notebook.) We do not need any of other columns for our purposes.

```julia
# This is what the generator dataset looks like:
gen
```

	bus	pg	qg	qmax	qmin	vg	mbase	status	pmax	pmin	pc1	pc2	qc1min	qc1max	qc2min	qc2max

	Int64	Int64	Int64	Float64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64

2 rows × 22 columns (omitted printing of 6 columns)

1	1	40	0	30.0	-30.0	1	100	1	1000	0	0	0	0	0	0	0

2	2	170	0	127.5	-127.5	1	100	1	1000	0	0	0	0	0	0	0

```julia
# and generator cost dataset:
gencost
```

	model	startup	shutdown	n	x1	y1	id

	Int64	Int64	Int64	Int64	Int64	Int64	Int64

2 rows × 7 columns

1	2	0	0	2	50	0	1

2	2	0	0	2	100	0	2

In the above, model=2 indicates a polynomial variable cost formulation and the column n=2 indicates that there are two terms. Thus, we have a linear cost, ignoring any quadratic terms (and a zero constant term):

$$
VarCost_g = x1_g
$$

```julia
# Here are the transmission lines:
branch
```

	fbus	tbus	r	x	b	ratea	rateb	ratec	ratio	angle	status	angmin	angmax	id

	Int64	Int64	Float64	Float64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64

6 rows × 14 columns

1	1	3	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	1

2	1	2	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	2

3	2	3	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	3

4	3	1	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	1

5	2	1	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	2

6	3	2	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	3

For the first model, we are only using transmission line capacity (known as "ratings"), given in `ratea`, `rateb`, and `ratec`. These correspond to different ratings based on how long the line might be overloaded, with `ratec` known as an "emergency rating", which could exceed the long-term rating, `ratea`. We will use `ratea` for this model. The dataset also contains resistance and reactance.

### 3. Create solver function (transport)

```julia
#=
Function to solve transport flow problem
Inputs:
    gen -- dataframe with generator info
    branch -- dataframe with transmission lines info
    gencost -- dataframe with generator info
    bus -- dataframe with bus types and loads
Note: it is always a good idea to include a comment blog describing your
function's inputs clearly!
=#
function transport(gen, branch, gencost, bus)
    Transport = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets
      # Set of all generators
    G = gen.id
      # Set of all nodes
    N = bus.bus_i

      # sets J_i and G_i will be described using dataframe indexing below

    # Decision variables
    @variables(Transport, begin
        GEN[G]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
        FLOW[N,N]        # flow
        # Note: flow is not constrained to be positive
        # By convention, positive will indicate flow from the first to second node
        #  and a negative flow will indicate flow from the second to the first
        # This matrix is thus "anti-symmetric", which we will ensure with an appropriate
        #  constraint
    end)

    # Objective function
    @objective(Transport, Min,
        sum( gencost[g,:x1] * GEN[g]
                        for g in G)
    )

    # Supply demand balances
    @constraint(Transport, cBalance[i in N],
        sum(GEN[g] for g in gen[gen.bus .== i,:id])
                - bus[bus.bus_i .== i,:pd][1] ==
        sum(FLOW[i,j] for j in branch[branch.tbus .== i,:fbus]))

    # Max generation constraint
    @constraint(Transport, cMaxGen[g in G],
                    GEN[g] <= gen[g,:pmax])

    # Flow constraints
    for l in 1:nrow(branch)
        @constraint(Transport,
            FLOW[branch[l,:fbus][1],branch[l,:tbus][1]] <=
                        branch[l,:ratea])
    end

    @constraint(Transport, cFlowSymmetric[i in N, j in N],
                    FLOW[j,i] == -FLOW[i,j])

    # Solve statement (! indicates runs in place)
    optimize!(Transport)

    # Dataframe of optimal decision variables
    generation = DataFrame(
        id = gen.id,
        node = gen.bus,
        gen = value.(GEN).data
        )

    flows = value.(FLOW).data

    # Return the solution and objective as named tuple
    return (
        generation = generation,
        flows,
        cost = objective_value(Transport)
    )
end
```

    transport (generic function with 1 method)

### 4. Solve

```julia
solution = transport(gen, branch, gencost, bus)
solution.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	600.0

2	2	2	0.0

We generate all 600 MW from Gen A at Bus 1.

```julia
DataFrame(solution.flows)
```

	x1	x2	x3

	Float64	Float64	Float64

3 rows × 3 columns

1	0.0	100.0	500.0

2	-100.0	0.0	100.0

3	-500.0	-100.0	0.0

In turn, the following flows are created:

- $l_{13}$ = 500 MW
- $l_{12}$ = 100 MW
- $l_{23}$ = 100 MW

Hence, we are able to maximize the capacity of the line from 1 to 3 ($l_{13}$), and then route the remaining power through Bus 2.

## DC linear approximation

The above model is not physically correct as we cannot arbitrarily route power through lines. We will now introduce a linear approximation to the optimal power flow problem that incorporates this limitation and is tractable and reasonably accurate.

In the DC linear approximation, flows along the line from bus i to bus j are driven by voltage phase angle differences:

$$
FLOW_{ij} = B_{ij}(\theta_j-\theta_i)
$$

Modifying the above transport model to incorporate these new decision variables and constraints:

$$
\begin{align}
\min \ & \sum_{g \in G} VarCost_g \times GEN_g & \\
\text{s.t.} & \\
 & \sum_{g \in G_i} GEN_g - Demand_i = \sum_{j\in J(i)}B_{ij}(\theta_j-\theta_i) & \forall \quad i \in \mathcal{N}\\
 & B_{ij}(\theta_j-\theta_i) \leq MaxFlow_{ij} & \forall \quad i \in \mathcal{N}, \forall j \in J_i \\
 & GEN_g \leq Pmax_g & \forall \quad g \in G \\
 & GEN_g \geq Pmin_g & \forall \quad g \in G
\end{align}
$$

Note that we no longer require a constraint to enforce anti-symmetric flows.

We have the following **sets**:
- $\mathcal{N}$, the set of all nodes (or buses) in the network
- $J_i$, the subset of nodes that are connected to node $i$
- $G_i \subset G$, the subset of generators located at node $i$

The **decision variables** in the above problem are:

- $GEN_{g}$, the generation (in MW) produced by each generator, $g$
- $\theta_i$, the voltage phase angle at bus $i$ relative to the reference bus (THETA_i in the function below)

The **parameters** are:

- $Pmin_g$, the minimum operating bounds for the generator (based on engineering or natural resource constraints)
- $Pmax_g$, the maximum operating bounds for the generator (based on engineering or natural resource constraints)
- $Demand_i$, the demand (in MW) at bus $i$
- $MaxFlow_{ij}$, the maximum allowable flow along the line from $i$ to $j$
- $VarCost_g$, the variable cost of generator $g$
- $B_{ij}$, susceptance for line connecting buses $i$ and $j$

### 3. Create solver function (dcopf)

```julia
#=
Function to solve DC OPF problem
Inputs:
    gen -- dataframe with generator info
    branch -- dataframe with transmission lines info
    gencost -- dataframe with generator info
    bus -- dataframe with bus types and loads
Note: it is always a good idea to include a comment blog describing your
function's inputs clearly!
=#
function dcopf(gen, branch, gencost, bus)
    DCOPF = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets
      # Set of all generators
    G = gen.id
      # Set of all nodes
    N = bus.bus_i

    # Define conversion of impendance from p.u. to SI units
    # It is common for power systems data to be in "per units" (p.u.) units,
    #  with respect to some given base voltage level (which could vary by line).
    # In the current problem, the whole network is assumed to be at the same voltage,
    #  given in bus.basekv . Power is given in MW (i.e., 10^6 W).
    # To convert impedance, we use the formula: Z = V^2 / P
    pu_base = (bus.basekv[1]*1e3)^2 / 1e6

      # sets J_i and G_i will be described using dataframe indexing below

    # Decision variables
    @variables(DCOPF, begin
        GEN[G]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
        THETA[N]         # voltage phase angle of bus
    end)

    # Create slack bus with reference angle = 0
    # Note: by convention this is a generator bus. Hence, we will select bus 1
    fix(THETA[1],0)

    # Objective function
    @objective(DCOPF, Min,
        sum( gencost[g,:x1] * GEN[g]
                        for g in G)
    )

    # Supply demand balances
    #  Note: we multiply here by pu_convert
    @constraint(DCOPF, cBalance[i in N],
        sum(GEN[g] for g in gen[gen.bus .== i,:id])
                - bus[bus.bus_i .== i,:pd][1] ==
        sum(branch[(branch.fbus .== i) .& (branch.tbus .== j),:b][1] *
            pu_base *
            (THETA[j] - THETA[i])
            for j in branch[branch.fbus .== i,:tbus]))

    # Max generation constraint
    @constraint(DCOPF, cMaxGen[g in G],
                    GEN[g] <= gen[g,:pmax])

    # Flow constraints
    #  Note: we multiply here by pu_convert
    for l in 1:nrow(branch)
        @constraint(DCOPF,
            branch[l,:b] * pu_base *
            (THETA[branch[l,:tbus][1]] - THETA[branch[l,:fbus][1]]) <=
                        branch[l,:ratea])
    end

    # Solve statement (! indicates runs in place)
    optimize!(DCOPF)

    # Output variables
    generation = DataFrame(
        id = gen.id,
        node = gen.bus,
        gen = value.(GEN).data
        )

    angles = value.(THETA).data

    flows = DataFrame(
        fbus = branch.fbus,
        tbus = branch.tbus,
        flow = pu_base .* branch.b .* (angles[branch.tbus] .-
                        angles[branch.fbus]))

    # We output the marginal values of the demand constraints,
    # which will in fact be the prices to deliver power at a given bus.
    prices = DataFrame(
        node = bus.bus_i,
        value = dual.(cBalance).data)

    # Return the solution and objective as named tuple
    return (
        generation = generation,
        angles,
        flows,
        prices,
        cost = objective_value(DCOPF)
    )
end
```

    dcopf (generic function with 1 method)

### 4. Solve

```julia
solution = dcopf(gen, branch, gencost, bus)
solution.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	600.0

2	2	2	0.0

Hence, we generate all 600 MW from Gen A at Bus 1.

```julia
# These are the voltage phase angles of the buses relative to Bus 1.
solution.angles
```

    3-element Array{Float64,1}:
     0.0
     0.5309997663601028
     1.0619995327202056

```julia
solution.flows
```

	fbus	tbus	flow

	Int64	Int64	Float64

6 rows × 3 columns

1	1	3	400.0

2	1	2	200.0

3	2	3	200.0

4	3	1	-400.0

5	2	1	-200.0

6	3	2	-200.0

Thus, we notice that, in contrast to the transport model, we do not max out the capacity of $l_{13}$. The following flows are created:

- $l_{13}$ = 400 MW
- $l_{12}$ = 200 MW
- $l_{23}$ = 200 MW

### 5. Solve high demand case

Now, let's increase demand at Bus 3 to 800 MW. Intuition tells us we will no longer be able to generate all of our power from Gen A alone.

```julia
bus_high = copy(bus)
bus_high[3,:pd] = 800

sol_high = dcopf(gen, branch, gencost, bus_high)
sol_high.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	700.0

2	2	2	100.0

This situation is explained by flow patterns, where the capacity of $l_{13}$ is at its maximum, but in order to meet demand at Bus 3, more power needs to be injected in Bus 2.

```julia
sol_high.flows
```

	fbus	tbus	flow

	Int64	Int64	Float64

6 rows × 3 columns

1	1	3	500.0

2	1	2	200.0

3	2	3	300.0

4	3	1	-500.0

5	2	1	-200.0

6	3	2	-300.0

The following flows are created:

- $l_{13}$ = 500 MW
- $l_{12}$ = 200 MW
- $l_{23}$ = 300 MW

### 6. Compare prices

The marginal values of the demand constraints at a given bus represent the change in the objective that results from increasing demand at the bus by one unit. This is the natural definition of a "value" of power at that location, and is the basis for **locational marginal prices** (LMPs) found in electricity markets.

We examine first the regular case of demand = 600 MW, then the high demand case = 800 MW.

```julia
solution.prices
```

	node	value

	Int64	Float64

3 rows × 2 columns

1	1	50.0

2	2	50.0

3	3	50.0

All prices are the same in this case. The interpretation: if we were to add an incremental load at any of the buses, we could meet it from additional production from Gen A which has marginal cost of \$50 / MWh. We are not going to hit any transmission limits.

```julia
sol_high.prices
```

	node	value

	Int64	Float64

3 rows × 2 columns

1	1	50.0

2	2	100.0

3	3	150.0

Something interesting has happened!

First, note that the prices are different. Hence, we will not be able to meet incremental load from production by Gen A (except if we add load right at Gen A located at Bus 1). Similarly, load at Bus 2 can be met by increasing production from Gen B with marginal cost = \$100 / MWh.

However, why does Bus 3 have a marginal price of \$150 / MWh?!

The answer lies in what must happen to meet an incremental load at Bus 3 while respecting transmission constraints. We must increase from Gen B, but in doing so part of the power from Gen B will go through 2->1->3 in addition to 2->3. However, this will cause us to exceed the transmission constraint on line $l_{13}$, requiring us to throttle back power from Gen A.

The exact change for an incremental 1 MW load at Bus 3:
- Gen B $\uparrow$ 2 MW
- Gen A $\downarrow$ 1 MW

Hence:

$$
Price_3 = 2 \times VarCost_B - VarCost_A = \$150 \text{ / MWh}
$$

In a network with thousands of nodes, one can see quite quickly how prices may vary in unexpected ways; hence, the need for detailed mathematical models.

```julia

```


# SIRモデル

## 感染ダイナミクス (連続)

$$
\begin{align}
  \dfrac{\mathrm{d} S(t)}{\mathrm{d}t} &= - \beta S(t)I(t)
  \\
  \dfrac{\mathrm{d} I(t)}{\mathrm{d}t} &= (\beta S(t) - \gamma)I(t)
  \\
  \dfrac{\mathrm{d} R(t)}{\mathrm{d}t} &= \gamma I(t)
\end{align}
$$

## 感染ダイナミクス (離散)

$$
\begin{align}
   S(t+\Delta t) &= S(t) \mathrm{e}^{-\beta \Delta t I(t)}
  \\
  I(t+\Delta t) &= I(t)+ S(t) (1-\mathrm{e}^{-\beta \Delta t I(t)}) - \gamma \Delta t I(t)
  \\
  R(t+\Delta t) &= R(t) + \gamma \Delta t I(t)
\end{align}
$$

## 定式化およびアルゴリズム詳細

* [report.pdf](report.pdf)

```python
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
import SIR_Solver as SS

plt.style.use('default')
```

```python
prm = SS.Prameter(R0=2.0, gamma=1)
dt = 0.1
model = SS.Model(prm=prm, I=0.001, T=20, dt=dt)
model_Euler = SS.Model(prm=prm, I=0.001, T=20, dt=dt)
model_Heun = SS.Model(prm=prm, I=0.001, T=20, dt=dt)
model_Runge_Kutta = SS.Model(prm=prm, I=0.001, T=20, dt=dt)
```

```python
model.solve_Anly()
model_Euler.solve_Euler()
model_Heun.solve_Heun()
model_Runge_Kutta.solve_Runge_Kutta()
```

```python

plt.plot(model.I_list, color='black', label='Discrete model')
plt.plot(model_Euler.I_list, color='red', label='Euler')
plt.plot(model_Heun.I_list, color='blue', label='Heun')
plt.plot(model_Runge_Kutta.I_list, color='green', label='Runge-Kutta')

plt.xticks([i*50 for i in range(6)], [i*5 for i in range(6)])
plt.xlim([0,200])
plt.ylim([0.0,0.17])

plt.legend(fontsize=8)
```


## Definitions and Framing

* **Supervised Machine Learning:**
    Using known data to generate some useful predictions of on unseen data

    Label $y$: The target variable that we are trying to predict, for example 'spam' or 'not spam'.

    Features: Something about the data that is used to represent the data, that is later fed into a model. Complicated machine learning models can have multiple features $x_1, x_2, \ldots, x_n$. So this is a vector $\mathbf{x}$.

    Model: Maps unseen data to predictions of label $y$. It is defined by internal parameters that are learned using a training set of labeled data

    Labeled data can be represented as $(\mathbf{x}, y)$. In unlabeled data, $y$ is not known and is predicted by the model.

* **Models**

    A model defines the relationship between the features and the label. There are two key phases in the life of a model:

    Training: The phase where you the model is trained or learned. You show the model a number of examples of labeled data, and allow the model to learn the relationship between the features and the label. In other words, it is learning the values of the parameters in the model. These parameters in the model are often called hyperparameters.

    Inference: The phase where the model is used to generate labels $y'$ given features $\mathbf{x}$.

* **Regression vs. Classification**

    A regression predicts continuous values while a classification predicts discrete values.

## Descending into ML

Topics covered: Linear Regression, Training and Loss

* The $L_2$ loss is defined as

$$
\begin{align}
L_2 \textrm{ Loss } = \sum\limits_i (y_i - prediction_i(\mathbf{x}))^2
\end{align}
$$

where we sum over all data points in the training set $i$.

* The convention in machine learning is to represent the linear model as $y' = w_1 x_1 + b$ instead of the more traditional $y = mx + b$. We could easily generalize this regression from one feature to multiple features and the prediction would be given by $y' = w_1 x_1 + w_2 x_2 + w_3 x_3 + b$, where all the $w_i$'s are weights and all the $x_i$'s are features. The process of fitting the model is called training and the process of using the trained model to make a prediction is called inference.

* A commonly used loss function is the mean squared error (MSE), but this is neither the only loss function nor the best or most practical loss function for all cases.

## Reducing Loss

* One of the popular ways to find the minimum in the loss function is to use Gradient Descent. We calculate a gradient at each point and move in the direction of decreasing gradient. The step size that we use as we advance in the direction of decreasing gradient is called the learning rate. This has to be chosen carefully: in multi-dimensional problems, too large a learning rate can cause the code to become unstable.

* We can also get trapped in local minima if the space is *not-convex*. There is a whole sprawling field of non-convex optimization.

* There are two important flavors of gradient descent:

    **Stochastic Gradient Descent:**

    In theory, while using gradient descent, we need to calculate the gradient of the loss function considering all examples. However in practice, this is found to be excessive and computationally expensive. We therefore select only one example at random and calculate the gradient of the loss function considering only that one example. This is called stochastic gradient descent. Although this might require more steps to reach the optimum, overall there is usually less computation when dealing with very large data sets. Gradient calculations can be very expensive.

    Mathematically, we want to calculate

    $$
    \begin{align}
    w_{n+1} = w_n -\eta Q(w)
    \end{align}
    $$

    where $\eta$ is the learning rate and $Q(w) = \frac{1}{N}\sum_i Q_i(w)$ is the $L_2$ loss for the $i$-th example. What we do instead, in stochastic gradient descent is to first randomnly pick a particular example $k$ and calculate the loss function $Q_k(w)$ and then update $w$ using:

    $$
    \begin{align}
    w_{n+1} = w_n -\eta Q_k(w)
    \end{align}
    $$

    and perform this iteratively until the minimum criterion is reached.

    When there are multiple parameters to optimize, the gradient is of course a vector, and so is $\mathbf{w} = (w_1, w_2, \ldots)$ and we proceed in the direction of steepest gradient in steps $\eta$ large.

    **Mini-batch Gradient Descent**

    This is very similar to stochastic gradient descent except that instead of taking only one data point, we take batches of 10 or 100 of them.  Especially with datasets that contain duplicates, enormous datasets do not contain any further information than very large datasets. Mini-batch gradient descent exploits this fact and works very well in practice. Typical values of batch-size range between 10 and 1000. Stochastic gradient descent is an extreme example of mini-batch gradient descent with batch size 1.

* The algorithm to train the model in this case is iterative: we start with some intial guesses for the parameters, compute the loss function for those values, update the values of the parameters through some scheme (with the goal of moving in the direction of lower loss), calculate the loss for the updated values of the parameters, and proceed iteratively until we achieve convergence. Convergence is usually assumed when the loss becomes below some threshold value, or if the loss function starts changing extremely slowly.

## First steps with TensorFlow

* TensnsorFlow is a computational framework that allows you to build machine learning models. We can use both lower level APIs by defining models based on a series of mathematical operations or we could use predefined higher level APIs such as `tf.estimator`. These architectures can be linear regressors or neural networks.

* TensorFlow consists of the following two components:

    0. A graph protocol buffer
    0. A runtime that executes the distributed graph

* The **graph protocol buffer** or protobuf takes data structures written in a text file and then generates classes in Python (or other language) that allows you to load, save and interact with the data in a user friendly way. In this sense the protobuf and the runtime are akin to Python code and a Python interpreter.

* Because TensorFlow is built using APIs of various levels of abstraction, we have a choice of level. In general, I should chose the layer that offers the highest level of abstraction. Of course, the highest layers are also less flexible, so for some special modeling cases if I need more flexibility, I can just drop one run lower in the level of API.

### `tf.estimator` API

The `tf.estimator` API is one of the highest level APIs that has a lot of prepackaged tools of use. Below we show some code to exemplify its use. In the example below, we are going to estimate the median housing pricebased on just one input feature. The data is from the 1990 California housing census data. Available [here](https://developers.google.com/machine-learning/crash-course/california-housing-data-description). First we perform some imports:

### Step 0: Setup, imports, loading and inspecting the data

**Note:** `tensorflow` does not work with `python 3.7.x` yet. I had `python 3.7.2` loaded on my machine, so this was a good exercise in understanding `conda environments`. It is recommended that each project have ita own *environment*, which consists of a python ditribution (which can be versioned) and a series of packages (which can also be versioned). we first use

`conda env --name <envname>`

`-n` can be used instead of `--name`. The environment can be created with all the required packages in one line as follows (recommended):

`conda env -n tensorflowenv python=3.6 scipy numpy matplotlib scikit-learn`

Here `tensorflowenv` is the environment name that I choose to give. I then go into the new environment using

`conda activate tensorflowenv`

Now `jupyter` doesn't yet recognize this new kernel, so I need to manually go and install it (for each environment I create) using the following (see [here](https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook?rq=1) for more details:)

`python -m ipykernel install --name tensorflowenv --display-name "Python (tendorflowenv)"`

Now when I invoke `jupyter notebook` in the activate environment, I can use the dropdown menu to activate the kernel with the specific python version and packages that I desire. Also note that invoking `conda install pkgname` in the activated environment only installs the `pkgname` for that particular environment.

For more information regarding environments, see the [docs](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)

```python
import math
from matplotlib import cm
from matplotlib import gridspec
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import metrics
import tensorflow as tf

from tensorflow.python.data import Dataset
tf.logging.set_verbosity(tf.logging.ERROR)
pd.options.display.max_rows = 10
pd.options.display.float_format = '{:.1f}'.format
```

We next load the data

```python
california_housing_dataframe = pd.read_csv('https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv', sep=',')

california_housing_dataframe = california_housing_dataframe.reindex(
    np.random.permutation(california_housing_dataframe.index))
california_housing_dataframe['median_house_value'] /= 1000
```

```python
california_housing_dataframe.head()
```

      longitude
      latitude
      housing_median_age
      total_rooms
      total_bedrooms
      population
      households
      median_income
      median_house_value

      4689
      -118.1
      34.1
      28.0
      238.0
      58.0
      142.0
      31.0
      0.5
      500.0

      7444
      -118.3
      33.9
      13.0
      2887.0
      853.0
      2197.0
      800.0
      2.9
      207.9

      14389
      -122.1
      38.0
      42.0
      2225.0
      367.0
      864.0
      381.0
      4.1
      172.4

      9089
      -119.0
      36.1
      20.0
      1042.0
      183.0
      509.0
      175.0
      3.0
      73.0

      8388
      -118.5
      34.3
      33.0
      1549.0
      264.0
      881.0
      289.0
      5.1
      222.9

We are now going to use `total_rooms` as an input feature to predict our target (or label) `median_house_value`.  Note that the data is at the city block level, so the feature represents the total number of rooms in the block. We are going to use the `tf.estimator` API to implement a linear regressor to model the data. The API already implements a lot of the low-level nuts and bolts of the regression (or in general, other models) so we can focus on the training, evaluating, and visualizing aspects of the process. The specific class we will use is the `tf.estimator.LinearRegressor` class.

### Step 1: Defining the features

There are two main classes of features:

* **categorical:** a feature that can take on discrete values, like spam/not spam, sunny/rainy/cloudy, etc.

* **neumerical:** a feature that can take on continuous or a large number of discrete values, like price, temperature, etc. It seems that these are features that we can do arithmetic on.

In TensorFlow, we need to define what kind of feature we are working with. In our case, because we are working with `total_rooms` in this example, it is a numerical feature. This identification of the type of feature is done using `tf.feature_column` module, which has various functions built into it. See [here](https://www.tensorflow.org/guide/feature_columns) for documentation. Feature columns only contain a description of the data and not the data itself

```python
my_feature = california_housing_dataframe[['total_rooms']]
feature_columns = [tf.feature_column.numeric_column('total_rooms')]
```

### Step 2: Defining the target

We want to predict the meadian house value given the number of rooms, so this is our target

```python
targets = california_housing_dataframe['median_house_value']
```

### Step 3: Configure the LinearRegressor

We now configure a linear regression model using `LinearRegressor`. The optimization itself is carried out using a built-in implementation of mini-batch stochastic gradient descent. We also use the `tf.contrib.estimator.clip_gradients_by_norm` functions to impose a cut off on the step size so that the gradient descent doesn't become unstable.

```python
my_optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.0000001)
my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)
```

```python
linear_regressor = tf.estimator.LinearRegressor(feature_columns=feature_columns,
                                                optimizer=my_optimizer)
```

### Step 4: Define the input function

We now need to define an input function which tells tensorflow how to preprocess the data as well as how to batch, shuffle, and repeat it during model training. This function is a bit of a jump from what we have seen so far, so we will explore it step by step.

```python
def my_input_fn(features, targets, batch_size=1, shuffle=True, num_epochs=None):
    """Trains a linear regression model with one feature
    Inputs:
        features: a pandas dataframe of features
        targets: a pandas dataframe of targets
        batch_size: size of batches to be passed to the model
        shuffle: Boolean, whether to shuffle the training data or not
        num_epochs: number of epochs for which the data should be repeated. None means repeat indefinitely
    Returns:
        Tuple of (features, label) for the next data batch
    """
    # convert pandas data into a dict of numpy arrays
    features = {key:np.array(value) for (key, value) in dict(features).items()} # line 13

    # Construct a dataset and configure batching and repeating
    ds = Dataset.from_tensor_slices((features,targets)) # line 16
    ds = ds.batch(batch_size).repeat(num_epochs) # line 17

    # Shuffle the data if needed
    if shuffle:
        ds = ds.shuffle(buffer_size=10000) # line 21

    # return the next batch of data
    features, labels = ds.make_one_shot_iterator().get_next() # line 24
    return (features, labels)
```

Here is an explanation by line number of the code above.

* **line 13:**
    `features` is a pandas dictionary, and `dict(features)` converts it into a dictionary with the column name as key and all the values in the column as pandas series of values. Therefore, the items generates an iterator of type `(feature_name, values)`, which we use to generate our feature dictionary. `values` is of type `pandas.Series`.

* **lines 16:**

    This line makes use of the `tf.data` API, which lets you take in distributed data from various sources and transform it in different ways to generate datasets for machine learning models. There are two cheif abstractions that `tf.data` introduces, one of which is the `tf.data.Dataset` module (the other is the `tf.data.Iterator` module which is used in line 24 and discussed next). Using `tf.data.Dataset`, there are two distinct ways to create a dataset:

    1. Creating a source (for example `Dataset.from_tensor_slices()`) which constructs a dataset from one or more `tf.Tensor` objects.
    2. Applying a transformations (for example `Dataset.batch()`) which constructs a dataset from one or more `tf.data.Datasets` objects.

    A tensor object is in principle a matrix of n-dimensions, except that its value is not evaluated until called for using a `tf.run()` command. The objects can be passed into operations to yield other tensor objects, but again, actual evaluation of the numerical value of the tensor is held off until later. This enables the *flow* of a tensor through various operations without numerical evaluation. With this context, what line 16 is doing is the following:

    `Dataset.from_tensor_slices()` returns a dataset for each row of the input tensor. For example the code snippet below would return `[1, 2], [3, 4]` for the variable `ds`. In our case, we provide a tuple of the form `(features, targets)`, where `features` is a dictionary, and `targets` is a number, `from_tensor_slices` would essentially unpack this tuple, to provide multiple datasets, each of the form `({key: values}, target)`. In other words, for this dataset, if I were to print the Dataset object (with an appropriate `tf.Session.run()` call) I would see something like `({lattitide: -118, longitude: 30, ... , households: 600, ...}, <median_house_value>)`. It takes a row, converts it into a dictionary of the form `{column name: value}` an then constructs a iteratable and transformable `tf.Dataset` object which has a dictionary of features associated with the `target`. The code snippet in the next cell below should make things clear regarding the output format of the `tf.Dataset` call in this case.

```python
    t = tf.constant([[1, 2], [3, 4]])
    ds = tf.data.Dataset.from_tensor_slices(t)   # [1, 2], [3, 4]
```

```python
df_test = pd.DataFrame()
df_test['col1'] = [0, 1]
df_test['col2'] = [2, 3]
t = [100, 200]
x = {key: value for (key, value) in dict(df_test).items()}
print(x)
ds_test = Dataset.from_tensor_slices((x, t))
tensor_test = ds_test.make_one_shot_iterator().get_next()
with tf.Session() as session:
    print(session.run(tensor_test))
    print(session.run(tensor_test))
```

    {'col1': 0    0
    1    1
    Name: col1, dtype: int64, 'col2': 0    2
    1    3
    Name: col2, dtype: int64}
    ({'col1': 0, 'col2': 2}, 100)
    ({'col1': 1, 'col2': 3}, 200)

### Step 5: Train the model

We now call `train()` on `linear_regressor` to train the model.

```python
_ = linear_regressor.train(
    input_fn=lambda: my_input_fn(my_feature, targets),
    steps=100)
```

The use of the `lambda` function is somewhat unclear to me. It seems that it ensures that the input function is truly randomized? Or that datasets are correctly paired to models?

### Evaluate the model

We now just make predictions on the training data to see how well the model fit it during training.

```python
# Create an input function for predictions
prediction_input_fn = lambda: my_input_fn(my_feature, targets, num_epochs=1, shuffle=False)

# Call predict on the linear_regressor to make predictions
predictions = linear_regressor.predict(input_fn=prediction_input_fn)

#Format predictions as a numpy array so we can calculate error metrics
predictions = np.array([item['predictions'][0] for item in predictions])

#Calculate the mean squared error and the root mean squared error
mean_squared_error = metrics.mean_squared_error(predictions, targets)
root_mean_squared_error = math.sqrt(mean_squared_error)
print("Mean Squared Error (on training data): %0.3f" % mean_squared_error)
print("Root Mean Squared Error (on training data): %0.3f" % root_mean_squared_error)
```

    Mean Squared Error (on training data): 56367.025
    Root Mean Squared Error (on training data): 237.417

To know if this mse and rmse is any good, we compare the rmse to the difference between the min and max of our targets

```python
min_house_value = california_housing_dataframe["median_house_value"].min()
max_house_value = california_housing_dataframe["median_house_value"].max()
min_max_difference = max_house_value - min_house_value

print("Min. Median House Value: %0.3f" % min_house_value)
print("Max. Median House Value: %0.3f" % max_house_value)
print("Difference between Min. and Max.: %0.3f" % min_max_difference)
print("Root Mean Squared Error: %0.3f" % root_mean_squared_error)
```

    Min. Median House Value: 14.999
    Max. Median House Value: 500.001
    Difference between Min. and Max.: 485.002
    Root Mean Squared Error: 237.417

This seems really large - the rmse is nearly 50% of the max difference. Let us first look at some summary statistics on how well the model is doing.

```python
calibration_data = pd.DataFrame()
calibration_data['predictions'] = pd.Series(predictions)
calibration_data['targets'] = pd.Series(targets)
calibration_data.describe()
```

      predictions
      targets

      count
      17000.0
      17000.0

      mean
      0.1
      207.3

      std
      0.1
      116.0

      min
      0.0
      15.0

      25%
      0.1
      119.4

      50%
      0.1
      180.4

      75%
      0.2
      265.0

      max
      1.9
      500.0

```python
# Plotting the data

sample = california_housing_dataframe.sample(n=300)

# Get the min and max total_rooms values.
x_0 = sample["total_rooms"].min()
x_1 = sample["total_rooms"].max()

# Retrieve the final weight and bias generated during training.
weight = linear_regressor.get_variable_value('linear/linear_model/total_rooms/weights')[0]
bias = linear_regressor.get_variable_value('linear/linear_model/bias_weights')

# Get the predicted median_house_values for the min and max total_rooms values.
y_0 = weight * x_0 + bias
y_1 = weight * x_1 + bias

# Plot our regression line from (x_0, y_0) to (x_1, y_1).
plt.plot([x_0, x_1], [y_0, y_1], c='r') # Plotting the line using just 2 points

# Label the graph axes.
plt.ylabel("median_house_value")
plt.xlabel("total_rooms")

# Plot a scatter plot from our data sample.
plt.scatter(sample["total_rooms"], sample["median_house_value"])

# Display graph.
plt.show()
```

The rest of the exercise goes into a discussion on tweaking the hyperparameters to overcome this por fit. With a suitable choice of `learning_rate`, `steps` and `batch_size` we can get a reasonably good fit. The parameter `steps` is just the number of times the mini-batch gradient descent attempts to go towards the minimum, and `batch_size` is the number of data points the gradient descent optimizer is optimizing over per step, so `steps * batch_size` is just the total number of data points involved in the optimization (not necessary all unique because the data points are randomly sampled)

### Aside: Synthetics features and outliers

In this set of tasks, we are going to create a synthetic feature that is the ratio of two other features and we are going to use this new feature to train the linear regression model. We are also going to look at how we can remove outliers to improve the effectiveness of the model. We are going to use the same `my_input_fn` defined above to generate datasets. However let us define a new function `train_model` to train the model

```python
def train_model(learning_rate, steps, batch_size, input_feature):
    """
    Args:
        learning_rate: A `float`, the learning rate.
        steps: A non-zero `int`, the total number of training steps. A training step
               consists of a forward and backward pass using a single batch.
        batch_size: A non-zero `int`, the batch size.
        input_feature: A `string` specifying a column from `california_housing_dataframe`
                       to use as input feature.

    Returns:
        A Pandas `DataFrame` containing targets and the corresponding predictions done
        after training the model.
  """
    periods = 10 # line 15
    steps_per_period = steps / periods

    my_feature = input_feature
    my_feature_data = california_housing_dataframe[[my_feature]].astype('float32')
    my_label = 'median_house_value'
    targets = california_housing_dataframe[my_label].astype('float32')

    # Create input functions
    training_input_fn = lambda: my_input_fn(my_feature_data,
                                            targets,
                                            batch_size=batch_size,
                                            shuffle=True,
                                            num_epochs=None)
    predict_training_input_fn = lambda: my_input_fn(my_feature_data, # line 29
                                                   targets,
                                                   batch_size=1,
                                                   shuffle=False,
                                                   num_epochs=1)

    # Create feature columns
    feature_columns = [tf.feature_column.numeric_column(my_feature)] # line36

    # Create a LinearRegressor object
    my_optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5)
    linear_regressor = tf.estimator.LinearRegressor(feature_columns=feature_columns,
                                                   optimizer=my_optimizer)

    # Plotting commands to plot our model's fit after each period
    plt.figure(figsize=(15, 6))
    plt.subplot(1, 2, 1)
    plt.title("Learned Line by Period")
    plt.ylabel(my_label)
    plt.xlabel(my_feature)
    sample = california_housing_dataframe.sample(n=300)
    plt.scatter(sample[my_feature], sample[my_label])
    colors = [cm.coolwarm(x) for x in np.linspace(-1, 1, periods)]

    # We now train the model inside a loop so that we can periodically
    # assess loss metrics
    print("Training model...")
    print("RMSE (on training data):")
    root_mean_squared_errors = []
    for period in range(periods):
        linear_regressor.train(input_fn=training_input_fn,
                              steps=steps_per_period)
        # Compute predictions
        predictions = linear_regressor.predict(input_fn=predict_training_input_fn)
        predictions = np.array([item['predictions'][0] for item in predictions]) # line 64

        # Compute mean squared error
        root_mean_squared_error = math.sqrt(metrics.mean_squared_error(predictions, targets))

        # Add the loss metrics from this period to our list.
        root_mean_squared_errors.append(root_mean_squared_error)

        # Occasionally print the current loss.
        print("  period %02d : %0.2f" % (period, root_mean_squared_error))

        # Finally, track the weights and biases over time.

        # Apply some math to ensure that the data and line are plotted neatly.
        y_extents = np.array([0, sample[my_label].max()])

        weight = linear_regressor.get_variable_value('linear/linear_model/%s/weights' % input_feature)[0] # line 76
        bias = linear_regressor.get_variable_value('linear/linear_model/bias_weights') # line 77

        x_extents = (y_extents - bias) / weight
        x_extents = np.maximum(np.minimum(x_extents,
                                          sample[my_feature].max()),
                               sample[my_feature].min())
        y_extents = weight * x_extents + bias
        plt.plot(x_extents, y_extents, color=colors[period])
    print("Model training finished.")

    # Output a graph of loss metrics over periods.
    plt.subplot(1, 2, 2)
    plt.ylabel('RMSE')
    plt.xlabel('Periods')
    plt.title("Root Mean Squared Error vs. Periods")
    plt.tight_layout()
    plt.plot(root_mean_squared_errors)

    # Create a table with calibration data.
    calibration_data = pd.DataFrame()
    calibration_data["predictions"] = pd.Series(predictions)
    calibration_data["targets"] = pd.Series(targets)
    calibration_data.describe()

    print("Final RMSE (on training data): %0.2f" % root_mean_squared_error)

    return calibration_data
```

Again, we study all the lines that introduce something new.

* **line 29:**

    Here, we define a `predict_training_input_fn`. This is used when we call `linear_regressor.predict()` later, which needs an `input_fn` of its own. This is because we need a function that constructs the features that is fed into the `linear_regressor.predict()` in order to generate a prediction for the label. The `LinearRegressor` object is of course blind to the targets, and makes a prediction based on the outcome of the training phase. Therefore, the output of the `predict_training_input_fn` has to be a `tf.data.Dataset` object. However, this function is different from `training_input_function` in a few ways.

    1. `batch_size=1` because I create a prediction of a set of features one at a time
    2. `shuffle=False`because there is no need toa shuffle a batch of size 1.
    3. `num_epochs=1` is important. `tf.data.Dataset.repeat(count)` will throw an out of range error (`tf.errors.OutOfRangeError`) when the number of times the dataset is repeated exceeds `count`. In our case, `num_epochs` is used in place of count. `tf.LinearRegressor.predict()` continues to predict until an out of range error is thrown. So in the prediction case, we use this to ensure that the prediction is made 1 set of features at a time.

* **line 36:**

    Defining a feature column which is like a bridge, or intermediary between raw data and estimators. It transforms input data into formats that estimators can understand, giving a lot of richness to have input data speak with estimators. See the [docs](https://www.tensorflow.org/guide/feature_columns) for the different kind of feature columns, of which `numeric_column` is one.

* **line 64:**

    Some thoughts on `linear_regressor.predict()`. The output of this function returns a generator which iterators over dictionaries. Each dictionary is of the form, for example, `{'predictions': array([15.93631], dtype=float32)}`. For each row of the form `(features, label)`, there is an entry in the generator object.

* **line 66, 67:**

    `tf.estimator.get_variable_names()` gives me the names of the variables in the `tf.estimator` object. In this case, our estimator happens to be a `LinearRegressor` object. For example, calling `linear_regressor.get_variable_names()` outputs:

    ```
    ['global_step',
     'linear/linear_model/bias_weights',
     'linear/linear_model/total_rooms/weights']
    ```

    We can now get the values of these variables by calling the `get_variable_value(name)` method where `name` is one of the names returned by `get_variabale_names()`. For example,
    `linear_regressor.get_variable_value('linear/linear_model/{}/weights'.format('total_rooms'))` returns:

    `array([[4.999998e-05]], dtype=float32)` as the output.

#### Task 1: Creating my own synthetic feature

The idea here is that `total_rooms` and `population` are both reported on a per city block basis. But some city blocks might have a lot of people for the same number of rooms, so the number of rooms per person is in a sense a better indicator of house value. So perhaps that would make a better feature.

```python
california_housing_dataframe["rooms_per_person"] = (california_housing_dataframe['total_rooms'] /
                                                   california_housing_dataframe['population'])

calibration_data = train_model(
    learning_rate=0.05,
    steps=500,
    batch_size=5,
    input_feature="rooms_per_person"
)
```

#### Task 2: Identify outliers

When we plot the predictions against the actual `median_house_value`, ideally we should see a straight line with slope 1. Do we really see that, and does that help us identify outliers?

```python
plt.scatter(calibration_data['targets'], calibration_data['predictions'])
```

A histogram might help to see how the data is distributed

```python
california_housing_dataframe['rooms_per_person'].hist(bins=20)
```

It looks like there are some small set of data points for `rooms_per_person` that is very large and might be throwing off the fit, so we will clip those

```python
criterion = california_housing_dataframe['rooms_per_person'] < 4
california_housing_dataframe = california_housing_dataframe[criterion]
california_housing_dataframe.hist('rooms_per_person')
```

This looks much more reasonable. We now follow the fitting procedure defined above by calling `train_model`

```python
calibration_data = train_model(learning_rate=0.05,
    steps=500,
    batch_size=5,
    input_feature="rooms_per_person"
)
```

```python
# Replot the historgram and the parity plot to see if the plot looks any better now

fig, axs = plt.subplots(1, 2, figsize=(10,4))
axs[0].hist(california_housing_dataframe['rooms_per_person'])
axs[1].scatter(calibration_data['targets'], calibration_data['predictions'])
```

## Generalization

The fundamental tension in machine learning is generating a model that does not overfit our training set while at the same time making good predictions of our test data set. Over-fitting is usually caused by making a model more complex than necessary. Our goal in Machine Learning is to be able to closely determine a true (but hidden) probability distribution when we can see only a fixed number of drawn samples from that distribution.

There are three basic assumptions that guide generalization:

1. All the datapoints we see are i.i.d. i.e. they are specific values of the same random variable
2. This underlying distribution is stationary i.e. does not change with time. Clearly this cannot always be true.
3. We draw examples from partitions of the same distribution.

## Training and Test Sets

The two things we need to be careful about when splitting data into training sets and test sets is (1) We want to make sure that the training data is large enough to get meaningful results and (2) We are not making samples errors by making the training set unrepresentative of the overall data. For example, training set being shopping habits in summer and the test set begin shopping habits in winter.

## Validation Sets

One possible way to go about fitting a model to the data is to split the data into a training set and test set, find one set of hyperparameters, see how well it does in predicting the test set (by some appropriate loss metric), tweak the hyperparameters, check against the test data again, and repeat this process until we get a really good match to our test data. However, the problem with this approach is that eventually we might start overfitting to the peculiarities of the test set. Therefore, one way to beat this is to actually form three datasets: A large enough training set, and then two smaller sets - a validation set and a test set. We set the test set aside for the moment and do the hyperparameter tweaking only with the validation set. When we are satisfied with the fit we get with the validation set, only then do we check our model against the test set. If the loss between the validation and test sets are similar, then we know we have a pretty good model. If they loss is much larger in the test set, we have probably over fit to either the training set or we are overly capturing the peculiarities of the validation set.

Having a validation set is often a better strategy because there is fewer exposures of the model to the test set. Still, both validation sets and test sets wear out with time (the test set wears out faster than the validation set) so it is best to go out and get more data to add to these sets to see how well the model is doing.

We now carry out a programming exercise to learn and solidify these concepts. The objectives here are:

  *  To include more than one feature to improve the effectiveness of the model, and add some complexity to our linear regression model
  * Debug issues in our model input data
  * Use a test data set to check if a model is overfitting the validation data

### Setup

We first load all the necessary imports as above

```python
import math
from matplotlib import cm
from matplotlib import gridspec
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import metrics
import tensorflow as tf

from tensorflow.python.data import Dataset
tf.logging.set_verbosity(tf.logging.ERROR)
pd.options.display.max_rows = 10
pd.options.display.float_format = '{:.1f}'.format

california_housing_dataframe = pd.read_csv(
    "https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv", sep=",")
```

Next we define a function to do all the creation of the synthetic feature `'rooms_per_person'`

```python
def preprocess_features(california_housing_dataframe):
    """Prepares input features from California housing data set.

    Args:
      california_housing_dataframe: A Pandas DataFrame expected to contain data
      from the California housing data set.
    Returns:
      A DataFrame that contains the features to be used for the model, including
      synthetic features.
    """
    selected_features = california_housing_dataframe[
      ["latitude",
       "longitude",
       "housing_median_age",
       "total_rooms",
       "total_bedrooms",
       "population",
       "households",
       "median_income"]]
    processed_features = selected_features.copy()
    # Create a synthetic feature.
    processed_features["rooms_per_person"] = (
      california_housing_dataframe["total_rooms"] /
      california_housing_dataframe["population"])
    return processed_features
```

Similarly, we write a preprocessing function to divide `median_house_value` by 1000.

```python
def preprocess_targets(california_housing_dataframe):
    """Prepares target features (i.e., labels) from California housing data set.

    Args:
    california_housing_dataframe: A Pandas DataFrame expected to contain data
      from the California housing data set.
    Returns:
    A DataFrame that contains the target feature.
    """
    output_targets = pd.DataFrame()
    # Scale the target to be in units of thousands of dollars.
    output_targets["median_house_value"] = (
    california_housing_dataframe["median_house_value"] / 1000.0)
    return output_targets
```

Start from splitting up the data into train and validation models

```python

```


```python
import sympy as sp
from sympy import sin, cos, exp
```

```python
h, s, v, gam = sp.symbols('h s v gam')
lamH, lamS, lamV, lamGAM, w = sp.symbols('lamH lamS lamV lamGAM w')

rho0, H, beta, LD, R_m, g, rho0, g = sp.symbols('rho0, H, beta, LD, R_m, g, rho0, g')
u = sp.symbols('u')

# Define sub-expressions to make things a bit easier to read
v2 = v*v
rho = rho0 * sp.exp(-h/H)
D_m = rho * v2 / (2 * beta)
r = R_m + h

# Define equations of motion f(t, x, u)
f = sp.Matrix([v * sin(gam),       # dh/dt
     v * cos(gam),       # ds/dt
     -D_m - g*sin(gam),  # dV/dt
     (v2 * cos(gam)/r + D_m*LD*cos(u) - g*cos(gam))/v # dgam/dt. Hint: L/m = (D/m) * (L/D)
    ])

lamT = sp.Matrix([lamH, lamS, lamV, lamGAM])
X = sp.Matrix([h, s, v, gam])

# lamda_dot = - lambda_T . df/dx
dLamDt = [-lamT.dot(f.diff(x)) for x in X]

# Add lamU
dLamUDt = -(lamT.dot(f)).diff(u)
dLamDt.append(dLamUDt)

# Substitute some sub-expressions
dLamDt = [expr.subs({D_m: 'D_m', r: 'r', rho: 'rho'})
          for expr in dLamDt]

lam_vars = [str(lam) for lam in lamT]
lam_vars.append('lamU')
for var, expr in zip(lam_vars, dLamDt):
    print(str(var)+'dot','=',expr.simplify())
```

    lamHdot = D_m*LD*lamGAM*cos(u)/(H*v) - D_m*lamV/H + lamGAM*v*cos(gam)/r**2
    lamSdot = 0
    lamVdot = D_m*LD*lamGAM*cos(u)/v**2 - LD*lamGAM*rho*cos(u)/beta - g*lamGAM*cos(gam)/v**2 - lamGAM*cos(gam)/r - lamH*sin(gam) - lamS*cos(gam) + lamV*rho*v/beta
    lamGAMdot = -g*lamGAM*sin(gam)/v + g*lamV*cos(gam) + lamGAM*v*sin(gam)/r - lamH*v*cos(gam) + lamS*v*sin(gam)
    lamUdot = LD*lamGAM*rho*v*sin(u)/(2*beta)

```python

```


Solving a variant of problem 6.3, Gray, Costanzo, and Plesha (Gray, G.L., Costanzo, F., and Plesha, M.E., *Engineering Mechanics: Dynamics*, McGraw Hill, New York, 2010.), where kinetics are included ($I_1, I_2, I_3$) and a moment $M$ is applied to the rightmost gear.

The problem statement is ``A Moment $M$ is applied to the rightmost gear (gear 3). Find the acceleration of the gear as a resulting moment presuming inertias $I_1, I_2, I_3$.''

For my class: this is Wright State  University, ME 3210, Quiz 2, Spring 2016

This notebook uses Python 3.5 in iPython (now called Jupyter).

Image awaiting copyright approval.

```python
#
```

```python
# import symbolic capability to Python
from sympy import *
```

```python
# print things all pretty
from sympy.abc import *
init_printing()
```

```python
# Need to define variables as symbolic for sympy to use them.
r_A, r_B, r_C, r_D, F_AB, F_DC, M= symbols("r_A r_B r_C r_D F_AB F_DC M", real = True)
```

```python
I_1, I_2, I_3 = symbols("I_1 I_2 I_3", real = True)
```

```python
theta_1, theta_2, theta_3 = symbols("theta_1 theta_2 theta_3", real = True)
```

The Free Body Diagrams, and kinematic relationships.

Defining equations. In SymPy, and equation is an expression that is equal to zero. All terms must be brought to the same side in defining it.

```python
EA = F_DC *r_D  - I_3 * diff(theta_3(t),t,t)
EA
```

```python
EB = r_B* F_AB - r_C* F_DC - I_2 * diff(theta_2(t),t,t)
EB
```

```python
EC = r_A *(-F_AB) + M - I_1 * diff(theta_1(t),t,t)
EC
```

```python
E1 = diff(theta_3(t),t,t)* r_D - diff(theta_2(t),t,t) *r_C
E1
```

```python
E2 = r_B * diff(theta_2(t),t,t) - diff(theta_1(t),t,t) *r_A
E2
```

```python
solve(EA,diff(theta_3(t),t,t))
```

```python
solve(EB,F_AB)[0]
```

I'm going to do substitutions in stages, eliminating one variable at a time for clarity.

```python
step1 = EC.subs(F_AB, solve(EB,F_AB)[0])
step1
```

```python
step2 = step1.subs(F_DC,solve(EA,F_DC)[0])
step2
```

```python
step3 = step2.subs(diff(theta_1(t),t,t),solve(E2,diff(theta_1(t),t,t))[0])
step3
```

```python
step4 = step3.subs(diff(theta_2(t),t,t),solve(E1,diff(theta_2(t),t,t))[0])
step4
```

Finally, the acceleration of the right most gear as a function of the input moment

```python
solve(step4,diff(theta_3(t),t,t))
```

```python

```


# Variablen

Wenn Sie ein neues Jupyter Notebook erstellen, wählen Sie `Python 3.6` als Typ des Notebooks aus.

Innerhalb des Notebooks arbeiten Sie dann mit Python in der Version 3.6. Um zu verstehen, welche Bedeutung Variablen haben, müssen Sie also Variablen in Python 3.6 verstehen.

In Python ist eine Variable ein Kurzname für einen Speicherbereich, in den Daten abgelegt werden können. Auf diese Daten kann dann unter dem Namen der Variablen zugegriffen werden. Dies haben wir prinzipiell bereits in den ersten Übungen durchgeführt. Beispiele:

`d_i = 20 # Innendurchmesser eines Rohres`

Den Name einer Variablen muss eindeutig sein. Er darf nicht mit einer Ziffer beginnen und darf keine Operationszeichen wie `'+', '-', '*', ':', '^'` oder `'#'` enthalten. Soll einer Variablen ein Wert zugewiesen werden, so geschieht das mittels des Gleichheitszeichens, siehe oben.

Das `'#'`-Zeichen leitet einen Kommentar ein. Alles, was nach diesem Zeichen folgt, hat einen rein informativen Charakter aber wird von Python ignoriert.

Sobald eine Variable angelegt ist, kann diese in Berechnungen verwendet werden, z.B.

```python
import math
```

```python
d_i = 20e-3 # Innendurchmesser eines Rohres in m

A_i = math.pi*d_i**2/4 # Lichter Querschnitt in m**2
```

In einer Zelle dürfen mehrere Operationen stehen, siehe oben. Eine Zelle kann eine Ausgabe haben. Das ist dann immer das Ergebnis der Letzten Operation.

In der oben stehenden Zelle steht als letztes eine Wertzuweisung `A_i = ...`, die keine Ausgabe erzeugt.

Um interaktiv arbeiten zu können, sollte man nicht zu lange Berechnungen in einzelnen Zellen durchführen. Statt dessen sollte man nach wenigen sinnvollen Schritten Zwischenergebnisse anzeigen, um den Gang der Arbeit nachvollziehen zu können.

Stellt man einen Fehler fest, so können Werte geändert und die Zelle erneut ausgeführt werden.

Um das Ergebnis der oben durchgeführten Berechnung anzuzeigen, kann man die zuletzt erzeugte Variable aufrufen. Die Zelle sähe dann so aus:

```python
d_i = 20e-3 # Innendurchmesser eines Rohres in m

A_i = math.pi*d_i**2/4 # Lichter Querschnitt in m**2
A_i
```

    0.0003141592653589793

Variablen haben einen Typ, den man sich durch die Funktion `type(variable)` anzeigen kann, z.B.:

```python
type(A_i)
```

    float

# Aufgabe

Untersuchen Sie, welchen Typ die Variablen

`a=1`

`x=5.0`

`Name = 'Bernd'`

und

`Punkt = (3,4)`

haben.

```python
a = 1
type(a)
```

    int

```python
x=5.0
type(x)
```

    float

```python
name='Bernd'
type(name)
```

    str

```python
Punkt=(3,4)
type(Punkt)
```

    tuple

Untersuchen Sie, welche Auswirkung der Befehl

`2*name`

```python
2*name
```

    'BerndBernd'

mit dem oben definierten Namen hat. Welche Auswirkung hat analog

`2*Punkt`?

```python
2*Punkt
```

    (3, 4, 3, 4)

Stellen Sie eine Vermutung auf, welchen Typ das Produkt `a*x` und die Summe `a+x` mit den oben festgelegten Werten von `a` und `x` haben und überprüfen Sie diese.

```python
type(a+x)
```

    float

```python
type(a*x)
```

    float

Um die Anwendungen in der Mathematik nicht aus den Augen zu verlieren, bearbeiten Sie die folgende Aufgabe:

# Aufgabe

Berechnen Sie das Gewicht von 250m Kupferrohr CU15$\times$1 in kg. Entnehmen Sie dafür die Dichte $\varrho$ von Kupfer ihrem Tabellenbuch. Die Zusammenhänge sind durch die folgenden Formeln gegeben:

\begin{align}
  A &= \dfrac{\pi\,(d_a^2 - d_i^2)}{4}
  \\[2ex]
  V &= A\, l
  \\[2ex]
  m &= \varrho\, V
\end{align}

```python
import math
```

```python
# Ihre Lösung beginnt hier
d_a = 15e-2 # d_a in dm
d_i = 13e-2 # d_i in dm

l = 250e1 # l in dm
A = math.pi*(d_a**2 - d_i**2)/4 # Querschnitt in dm**2
V = A*l # Volumen in dm**3

rho = 8.96 # kg/dm**3
m = rho*V
m # Masse in kg
```

    98.52034561657587