## Optimal Power Flow
**Power Systems Optimization**

by Michael R. Davidson, Jesse D. Jenkins and Sambuddha Chakrabarti (last updated: October 7, 2020)

This notebook provides an introduction to the Optimal Power Flow (OPF) problem in electric power systems—which minimizes the short-run production costs of meeting electricity demand at a number of connected locations from a given set of generators subject to various technical and transmission network flow limit constraints. This will be our first treatment of a *network*, which is critical to all power systems.

We will first introduce a model of transmission flows that assumes we can control the flow along each path, in what is called a "**transport model**." This is a straightforward extension to [Economic Dispatch](04-Economic-Dispatch.ipynb) (ED), where we have multiple supply and demand balance constraints at each location or "node" in the network, and a new set of flow constraints between nodes. This is also similar to other common optimization problems such as fleet routing of shipments.

We will then introduce a linear approximation to the optimal power flow problem known as "**DC-OPF**", where we begin to incorporate some of the physics involved in how electricity flows along transmission lines. With this formulation, we recognize that given "injections" (i.e., generation) and "withdrawals" (i.e., demand) of power at each node in the network, flows along lines are not independently controllable. Instead, electrical power flows across transmission lines in relation to their physical properties, namely power flows across parallel circuits or paths in inverse proportion to the [electrical impedance](https://en.wikipedia.org/wiki/Electrical_impedance) of the lines. This can (very frequently) result in hitting flow constraints before we would if could control power flows across all lines as in the transport problem.

This notebook does not explore the full functionality of DC-OPF, which can include inter-temporal constraints, additional generation constraints (e.g., on voltage), security constraints to ensure stability in the case of contingencies, and network losses.

Full "AC optimal power flow" models are also beyond the scope of this notebook, as the full set of physics associated with the interactions of AC flows introduces non-convexities that make this problem much harder to solve. Due to the non-convex nature of the AC power flow problem, simplified formulations that linearize and approximate these complex constraints are frequently employed in power systems operations, including by electricity system operators, which use a linearized "security-constrained" optimal power flow (which ensures power flows would remain simulatenously feasible across a range of possible contingencies) to clear real-time electricity markets.

We will start off with some simple systems, whose solutions can be worked out manually without resorting to any mathematical optimization model and software. But, eventually we will be solving larger system, thereby emphasizing the importance of such software and mathematical models.

## Introduction to OPF

The Optimal Power Flow (OPF) problem is a power system optimal scheduling problem which captures the physics of electricity flows across electricity networks, adding a layer of complexity and more realism to the Economic Dispatch (ED) problem. OPF usually attempts to capture the entire network topology by representing the transmission line interconnections between different nodes (also known as buses, or locations where generators or demand inject or withdraw power into/from the network) including various electrical parameters, such as resistance, series reactance, shunt admittance, etc. The full alternating current or "AC" OPF is a non-convex problem and turns out to be an extremely hard problem to solve (usually NP-hard). Hence, system operators and power marketers usually go about solving a linearized version of it, called the "DC-OPF." The DC-OPF approximation works satisfactorily for bulk power transmission networks as long as such networks are not operated at the brink of instability or under heavily loaded conditions.

## "Transport" model
We will first examine the case where we allow for transmission but ignore the physics of electricity flows, and instead treat it like transporting an ordinary commodity.

$$
\begin{align}
\min \ & \sum_{g \in G} VarCost_g \times GEN_g & \\
\text{s.t.} & \\
 & \sum_{g \in G_i} GEN_g - Demand_i = \sum_{j \in J_i} FLOW_{ij} & \forall \quad i \in \mathcal{N}\\
 & FLOW_{ij} \leq MaxFlow_{ij} & \forall \quad i \in \mathcal{N}, \forall j \in J_i \\
 & FLOW_{ij} = - FLOW_{ji} & \forall \quad i, j \in \mathcal{N} \\
 & GEN_g \leq Pmax_g & \forall \quad g \in G \\
 & GEN_g \geq Pmin_g & \forall \quad g \in G
\end{align}
$$

We introduce a few new **sets** in the above:
- $\mathcal{N}$, the set of all nodes (or buses) in the network where generation, storage, or demand (load) are located
- $J_i \subset \mathcal{N}$, the subset of nodes that are connected to node $i$
- $G_i \subset G$, the subset of generators located at node $i$

The **decision variables** in the above problem are:

- $GEN_{g}$, the generation (in MW) produced by each generator, $g$
- $FLOW_{ij}$, the flow (in MW) along the line from $i$ to $j$

The **parameters** are:

- $Pmin_g$, the minimum operating bounds for the generator (based on engineering or natural resource constraints)
- $Pmax_g$, the maximum operating bounds for the generator (based on engineering or natural resource constraints)
- $Demand_i$, the demand (in MW) at bus $i$
- $MaxFlow_{ij}$, the maximum allowable flow along the line from $i$ to $j$
- $VarCost_g$, the variable cost of generator $g$

Notice how the problem above is equivalent to producing a single type of good at a set of factories and shipping them along capacity-limited corridors (roads, rail lines, etc.) to meet a set of demands in other locations.

### 1. Load packages

```julia
import Pkg; Pkg.add("VegaLite"); Pkg.add("PrettyTables")
using JuMP, GLPK
using Plots; plotly();
using VegaLite  # to make some nice plots
using DataFrames, CSV, PrettyTables
ENV["COLUMNS"]=120; # Set so all columns of DataFrames and Matrices are displayed
```

      Resolving package versions...
    No Changes to `~/.julia/environments/v1.5/Project.toml`
    No Changes to `~/.julia/environments/v1.5/Manifest.toml`
      Resolving package versions...
    No Changes to `~/.julia/environments/v1.5/Project.toml`
    No Changes to `~/.julia/environments/v1.5/Manifest.toml`

### 2. Load and format data

We will load a modified 3-bus case stored in the [MATPOWER case format](https://matpower.org/docs/ref/matpower5.0/caseformat.html). It consists of:

- two generator buses where 1000 MW generators are located, one with variable cost of 50/MWh and another with variable cost of 100/MWh
- one load bus where 600 MW of demand is located
- three lines connecting the buses, each with a maximum flow of 500 MW

The location and numbering of the components:

```julia
datadir = joinpath("OPF_data")
gen = CSV.read(joinpath(datadir,"gen.csv"), DataFrame);
gencost = CSV.read(joinpath(datadir,"gencost.csv"), DataFrame);
branch = CSV.read(joinpath(datadir,"branch.csv"), DataFrame);
bus = CSV.read(joinpath(datadir,"bus.csv"), DataFrame);

# Rename all columns to lowercase (by convention)
for f in [gen, gencost, branch, bus]
    rename!(f,lowercase.(names(f)))
end

# create generator ids
gen.id = 1:nrow(gen);
gencost.id = 1:nrow(gencost);

# create line ids
branch.id = 1:nrow(branch);
# add set of rows for reverse direction with same parameters
branch2 = copy(branch)
branch2.f = branch2.fbus
branch2.fbus = branch.tbus
branch2.tbus = branch2.f
branch2 = branch2[:,names(branch)]
append!(branch,branch2)

# Calculate the susceptance of each line, on the assumption that
# reactance is >> resistance, such that we can approximate
# resistance as = 0 and treat susceptance as the simple
# reciprocal of reactance (x).
# See https://en.wikipedia.org/wiki/Susceptance#Relationship_to_reactance
branch.sus = 1 ./ branch.x

# Here are the buses:
bus
```

	bus_i	type	pd	qd	gs	bs	area	vm	va	basekv	zone	vmax	vmin

	Int64	Int64	Int64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Float64	Float64

3 rows × 13 columns

1	1	2	0	0.0	0	0	1	1	0	230	1	1.1	0.9

2	2	2	0	0.0	0	0	1	1	0	230	1	1.1	0.9

3	3	1	600	98.61	0	0	1	1	0	230	1	1.1	0.9

Columns pd and qd indicate the [active and reactive power](https://en.wikipedia.org/wiki/AC_power#Active,_reactive,_and_apparent_power) withdrawal at the bus. (We will ignore qd for this notebook, since we are not considering full AC power flows.) We do not need any of other columns for our purposes.

```julia
# This is what the generator dataset looks like:
gen
```

	bus	pg	qg	qmax	qmin	vg	mbase	status	pmax	pmin	pc1	pc2	qc1min	qc1max	qc2min	qc2max

	Int64	Int64	Int64	Float64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64

2 rows × 22 columns (omitted printing of 6 columns)

1	1	40	0	30.0	-30.0	1	100	1	1000	0	0	0	0	0	0	0

2	2	170	0	127.5	-127.5	1	100	1	1000	0	0	0	0	0	0	0

```julia
# and generator cost dataset:
gencost
```

	model	startup	shutdown	n	x1	y1	id

	Int64	Int64	Int64	Int64	Int64	Int64	Int64

2 rows × 7 columns

1	2	0	0	2	50	0	1

2	2	0	0	2	100	0	2

In the above, model=2 indicates a polynomial variable cost formulation and the column n=2 indicates that there are two terms. Thus, we have a linear cost (in the x1 column) without any quadratic terms (and a zero constant term):

$$
VarCost_g = x1_g
$$

```julia
# Here are the transmission lines:
branch
```

	fbus	tbus	r	x	b	ratea	rateb	ratec	ratio	angle	status	angmin	angmax	id	sus

	Int64	Int64	Float64	Float64	Float64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Float64

6 rows × 15 columns

1	1	3	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	1	35.5872

2	1	2	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	2	35.5872

3	2	3	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	3	35.5872

4	3	1	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	1	35.5872

5	2	1	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	2	35.5872

6	3	2	0.00281	0.0281	0.00712	500	500	500	0	0	1	-360	360	3	35.5872

Note that while there are three lines, there are six entries here. There is an entry for each 'direction' of flow the lines can accomodate, hence six entries for three lines. The column `fbus` denotes the ID of the "from bus" or origin bus and the column `tbus` denotes the ID of the "to bus" or destination bus (e.g. `fbus`=1, `tbus`=3 is the flow in the direction from bus 1 to bus 3).

For this transport model formulation, we are only using transmission line capacity (known as "ratings"), given in `ratea`, `rateb`, and `ratec`. These correspond to different ratings based on how long the line might be overloaded, with `ratec` known as an "emergency rating", which could exceed the long-term rating, `ratea`. We will use `ratea` for this model. The dataset also contains resistance and reactance.

### 3. Create solver function (transport)

```julia
#=
Function to solve transport flow problem
Inputs:
    gen -- dataframe with generator info
    branch -- dataframe with transmission lines info
    gencost -- dataframe with generator info
    bus -- dataframe with bus types and loads
Note: it is always a good idea to include a comment blog describing your
function's inputs clearly!
=#
function transport(gen, branch, gencost, bus)
    Transport = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets
      # Set of all generators
    G = gen.id
      # Set of all nodes
    N = bus.bus_i
      # Note: sets J_i and G_i will be described using dataframe indexing below

    # Decision variables
    @variables(Transport, begin
        GEN[G]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
        FLOW[N,N]        # flow
        # Note: flow is not constrained to be positive
        # By convention, positive values will indicate flow from the first to second node
        # in the tuple, and a negative flow will indicate flow from the second to the first
        # This matrix is thus "anti-symmetric", which we will ensure with an appropriate
        # constraint.
    end)

    # Objective function: minimize sum of generation variable costs for all generators
    @objective(Transport, Min,
        sum( gencost[g,:x1] * GEN[g]
                        for g in G)
    )

    # Supply/demand balance constraints, accounting for power flows in/out of each node
    @constraint(Transport, cBalance[i in N],
        sum(GEN[g] for g in gen[gen.bus .== i,:id])
                - bus[bus.bus_i .== i,:pd][1] ==
        sum(FLOW[i,j] for j in branch[branch.tbus .== i,:fbus]))

    # Max generation constraints
    @constraint(Transport, cMaxGen[g in G],
                    GEN[g] <= gen[g,:pmax])

    # Flow constraints on each branch
    for l in 1:nrow(branch)
        @constraint(Transport,
            FLOW[branch[l,:fbus][1],branch[l,:tbus][1]] <=
                        branch[l,:ratea])
    end

    # Anti-symmetric flow constraints
    @constraint(Transport, cFlowSymmetric[i in N, j in N],
                    FLOW[i,j] == -FLOW[j,i])

    # Solve statement (! indicates runs in place)
    optimize!(Transport)

    # Dataframe of optimal decision variables
    generation = DataFrame(
        id = gen.id,
        node = gen.bus,
        gen = value.(GEN).data
        )

    flows = value.(FLOW).data

    # Return the solution and objective as named tuple
    return (
        generation = generation,
        flows,
        cost = objective_value(Transport),
        status = termination_status(Transport)
    )
end
```

    transport (generic function with 1 method)

### 4. Solve

```julia
solution = transport(gen, branch, gencost, bus)
solution.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	600.0

2	2	2	0.0

We generate all 600 MW from Gen A at Bus 1.

```julia
DataFrame(solution.flows)
```

	x1	x2	x3

	Float64	Float64	Float64

3 rows × 3 columns

1	0.0	100.0	500.0

2	-100.0	0.0	100.0

3	-500.0	-100.0	0.0

In turn, the following flows are created:

- $l_{13}$ = 500 MW
- $l_{12}$ = 100 MW
- $l_{23}$ = 100 MW

Hence, we are able to maximize the capacity of the line from 1 to 3 ($l_{13}$), and then route the remaining power through Bus 2.

## The "DC" Optimal Power Flow problem

The above model is not physically correct as we cannot arbitrarily route power through lines. We will now introduce a linear approximation to the optimal power flow problem that incorporates this limitation and is tractable and reasonably accurate. This is commonly called the "DC" optimal power flow problem, but in reality (as we'll see below), it is a linearization that still relates to the physics of AC power flows, it just simplifies (and ignores) certain non-convexities to produce a tractable linear programming problem that remains a valid approximation under certain circumstances.

In the "DC" or linear approximation of the AC optimal power flow problem, power flows along a line from bus $i$ to bus $j$ are driven by voltage [phase angle](https://en.wikipedia.org/wiki/Phasor) differences, denoted by $\theta_i$ and $\theta_j$:

$$
FLOW_{ij} = BaseMVA \times B_{ij} (\theta_i-\theta_j)
$$

Where $FLOW_{ij}$ is the flow across the line from node $i$ to node $j$ (in MW), $BaseMVA$ is the base power for the network (in MVA), $B_{ij}$ is the [susceptance](https://en.wikipedia.org/wiki/Susceptance) for the line connecting buses $i$ and $j$ (in per unit terms) and $(\theta_i-\theta_j)$ is the difference in voltage angles between buses (in radians).

Susceptance is the imaginary part of the [admittance](https://en.wikipedia.org/wiki/Admittance) of a line, where [admittance](https://en.wikipedia.org/wiki/Admittance) is a complex number that describes how easy it is for AC current to flow across a given conductor. Power flows in parallel circuits in an AC network in proportion to their admittance (or inverse proportion to [impedance](https://en.wikipedia.org/wiki/Electrical_impedance), which describes how hard it is a measure of the opposition that a circuit presents to a current when a voltage is applied).

Voltage [phase](https://www.allaboutcircuits.com/textbook/alternating-current/chpt-1/ac-phase/) angles describe the displacement of the AC voltage waveform at each node, relative to a reference or "slack" bus. A difference in voltage angles between buses $i$ and $j$ indicates that the peaks and troughs in the sinusoidal voltage waveform at bus $i$ are shifted in time relative to the voltage waveform at bus $j$, as in the image below.

 (*Image source: [allaboutcircuits.com](https://www.allaboutcircuits.com/textbook/alternating-current/chpt-1/ac-phase/)*)

In AC circuits, power flows from nodes with higher voltage angle to buses with lower voltage angle, just as power flows from higher voltage magnitude to lower voltage magnitude in DC circuits.

What causes the shift in voltage phase angle? An AC current flowing across a conductor encountering either [inductive reactance](https://en.wikipedia.org/wiki/Electrical_impedance#Inductive_reactance) (which relates to the magnetizing current, or the energy required to continually induce or establish magnetic fields around a conductor as AC current polarity flips each cycle) or [capacitive reactance](https://en.wikipedia.org/wiki/Electrical_impedance#Capacitive_reactance) (which relates to the charging current, or the energy required to sustain capacitive charges between two conductors separated by an insulator) will experience a shift in the voltage waveform, relative to the current wave form. Inductive reactance causes the voltage waveform to shift forward relative to the current, while capacitive reactive cases the voltage to shift backwards relative to the current. In overhead transmission lines, the primary source of voltage phase angle shifts is the inductance of the transmission lines themselves (although capacitance is relevant for underground lines).

(For (a bit) more on the physics of power flow on AC transmissio lines, you can review [this tutorial from the PJM Interconnection](https://learn.pjm.com/~/media/training/nerc-certifications/gen-exam-materials/bet/20160104-basics-of-elec-power-flow-on-ac.ashx))

The reason this approximation is known as the "DC OPF" is because we ignore [reactive power](https://en.wikipedia.org/wiki/AC_power#Reactive_power_ flows and focus only on flows of real power as in a DC network. But you can see why "DC" OPF is actually a misnomer: we're still dealing with AC voltages and susceptance terms here, we're just linearizing the problem through some simplifying assumptions. In particular, the three basic assumptions used to derive a linearized or "DC" OPF approximation from the underlying AC OPF problem are as follows:

1. The resistance for each branch is negligible relative to the reactance, and can therefore be approximated as ~0.
2. The voltage magnitude at each bus is constant and equal to the base voltage (e.g. equal to 1 p.u).
3. The voltage angle difference $(\theta_j-\theta_i)$ across any branch from bus $i$ to $j$ is sufficiently small such that $cos(\theta_i-\theta_j) \approx 1$ and $sin(\theta_i-\theta_j) \approx (\theta_i-\theta_j)$. Note that $\theta_i,\theta_j$ are measured in [radians](https://en.wikipedia.org/wiki/Radian).

Fortunately, under normal operating conditions, these conditions hold for electricity transmission networks (although note they are not generally acceptable simplifying assumptions for lower voltage distribution networks).

We can now modify the above transport model to incorporate these new power flow-related decision variables and constraints:

$$
\begin{align}
\min \ & \sum_{g \in G} VarCost_g \times GEN_g & \\
\text{s.t.} & \\
 & \sum_{g \in G_i} GEN_g - Demand_i = \sum_{j\in J(i)} FLOW_{i,j} & \forall \quad i \in \mathcal{N}\\
 & FLOW_{i,j} \leq MaxFlow_{ij} & \forall \quad i \in \mathcal{N}, \forall j \in J_i \\
 & FLOW_{i,j} = BaseMVA \times B_{ij}(\theta_i-\theta_j) & \forall \quad i \in \mathcal{N}, \forall j \in J_i \\
 & GEN_g \leq Pmax_g & \forall \quad g \in G \\
 & GEN_g \geq Pmin_g & \forall \quad g \in G \\
 & \theta_{slack} = 0
\end{align}
$$

Note that we no longer require a constraint to enforce anti-symmetric flows.

We have the following **sets**:
- $\mathcal{N}$, the set of all nodes (or buses) in the network
- $J_i \subset \mathcal{N}$, the subset of nodes that are connected to node $i$
- $G_i \subset G$, the subset of generators located at node $i$

The **decision variables** in the above problem are:

- $GEN_{g}$, the generation (in MW) produced by each generator, $g$
- $\theta_i$, the voltage phase angle at bus $i$ relative to the slack or reference bus ($\theta_{slack}$)
- $FLOW_{i,j}$, the flow from bus i to bus j (in MW)

Note that unlike the transport flow problem, in the OPF problem we *do not* directly choose the flows across lines, but rather choose the real power injections at generator buses via $GEN_{g}$ and the voltage angles $\theta_i$, which collectively determine the power flows across lines via the collection of constraints above. The $FLOW$ decision variable is thus an "auxialiary" variable, as it is precisely determined by the constraint $FLOW_{i,j} = BaseMVA \times B_{ij}(\theta_i-\theta_j)$ for all pairs ($i,j$) for which transmission lines exist.

(Note that we create $FLOW$ decisions for some pairs of nodes ($i,j$) that are not connected by lines; these variables are "free" variables, as they are not constrained and do not show up in/effect the objective function, so the solver will generally remove these variables in pre-solve step. We could try a different approach to sets for the $FLOW$ variable, e.g. by setting across lines instead of pairs of nodes, to avoid creating these unecessary free variables, but for convenience, we'll create and ignore them for now).

The **parameters** are:

- $Pmin_g$, the minimum operating bounds for the generator (based on engineering or natural resource constraints)
- $Pmax_g$, the maximum operating bounds for the generator (based on engineering or natural resource constraints)
- $Demand_i$, the demand (in MW) at bus $i$
- $MaxFlow_{ij}$, the maximum allowable flow along the line from $i$ to $j$
- $VarCost_g$, the variable cost of generator $g$
- $B_{ij}$, susceptance for line connecting buses $i$ and $j$
- $\theta_{slack}$, the "slack" bus or reference bus from which relative voltage angles at all other buses are calcluated. Thus $\theta_{slack}=0$.
- $BaseMVA$, the base power in MVA for the network (used to scale from standard units to per unit values or vice versa).

### 3. Create solver function (dcopf)

```julia
#=
Function to solve DC OPF problem
Inputs:
    gen -- dataframe with generator info
    branch -- dataframe with transmission lines info
    gencost -- dataframe with generator info
    bus -- dataframe with bus types and loads
Note: it is always a good idea to include a comment blog describing your
function's inputs clearly!
=#
function dcopf(gen, branch, gencost, bus)
    DCOPF = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets
      # Set of all generators
    G = gen.id
      # Set of all nodes
    N = bus.bus_i
      # sets J_i and G_i will be described using dataframe indexing below

    # Define per unit base units for the system
    # used to convert from per unit values to standard unit
    # values (e.g. p.u. power flows to MW/MVA)
    baseMVA = gen.mbase[1] # base MVA is 100 MVA for this system

    # Decision variables
    @variables(DCOPF, begin
        GEN[G]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
        THETA[N]         # voltage phase angle of bus
        FLOW[N,N]        # flows between all pairs of nodes
    end)

    # Create slack bus with reference angle = 0
    # Note: by convention this is a generator bus. Hence, we will select bus 1
    fix(THETA[1],0)

    # Objective function
    @objective(DCOPF, Min,
        sum( gencost[g,:x1] * GEN[g]
                        for g in G)
    )

    # Supply demand balances
    @constraint(DCOPF, cBalance[i in N],
        sum(GEN[g] for g in gen[gen.bus .== i,:id])
                - bus[bus.bus_i .== i,:pd][1] ==
        sum(FLOW[i,j] for j in branch[branch.fbus .== i,:tbus])
    )

    # Max generation constraint
    @constraint(DCOPF, cMaxGen[g in G],
                    GEN[g] <= gen[g,:pmax])

    # Flow constraints on each branch;
    # In DCOPF, line flow is a function of voltage angles
         # Create an array of references to the line constraints,
       # which we "fill" below in loop
    cLineFlows = JuMP.Containers.DenseAxisArray{Any}(undef, 1:nrow(branch))
    for l in 1:nrow(branch)
        cLineFlows[l] = @constraint(DCOPF,
            FLOW[branch[l,:fbus],branch[l,:tbus]] ==
            baseMVA*branch[l,:sus]*(THETA[branch[l,:fbus]] - THETA[branch[l,:tbus]])
        )
    end

    # Max line flow constraints
       # Create an array of references to the line constraints,
       # which we "fill" below in loop
    cLineLimits = JuMP.Containers.DenseAxisArray{Any}(undef, 1:nrow(branch))
    for l in 1:nrow(branch)
        cLineLimits[l] = @constraint(DCOPF,
           FLOW[branch[l,:fbus],branch[l,:tbus]]
           <= branch[l,:ratea]
        )
    end

    # Solve statement (! indicates runs in place)
    optimize!(DCOPF)

    # Output variables
    generation = DataFrame(
        id = gen.id,
        node = gen.bus,
        gen = value.(GEN).data
        )

    angles = value.(THETA).data

    flows = DataFrame(
        fbus = branch.fbus,
        tbus = branch.tbus,
        flow = baseMVA .* branch.sus .* (angles[branch.fbus] .- angles[branch.tbus])
    )

    # We output the marginal values of the demand constraints,
    # which will in fact be the prices to deliver power at a given bus.
    prices = DataFrame(
        node = bus.bus_i,
        value = dual.(cBalance).data)

    # Return the solution and objective as named tuple
    return (
        generation = generation,
        angles,
        flows,
        prices,
        cost = objective_value(DCOPF),
        status = termination_status(DCOPF)
    )
end
```

    dcopf (generic function with 1 method)

### 4. Solve

```julia
solution = dcopf(gen, branch, gencost, bus)
solution.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	600.0

2	2	2	0.0

Hence, we generate all 600 MW from Gen A at Bus 1.

```julia
# These are the voltage phase angles of the buses relative to Bus 1.
solution.angles
```

    3-element Array{Float64,1}:
      0.0
     -0.05619999999999999
     -0.11239999999999997

```julia
solution.flows
```

	fbus	tbus	flow

	Int64	Int64	Float64

6 rows × 3 columns

1	1	3	400.0

2	1	2	200.0

3	2	3	200.0

4	3	1	-400.0

5	2	1	-200.0

6	3	2	-200.0

Thus, we notice that, in contrast to the transport model, we do not max out the capacity of $l_{13}$. The following flows are created:

- $l_{1,3}$ = 400 MW
- $l_{1,2}$ = 200 MW
- $l_{2,3}$ = 200 MW

The reason we can't make maximum use of line $l_{1,3}$ is because power flows split across parallel circuits in inverse proportion to the impedance of the circuit paths. Here, all three branches have equal susceptance, and thus equal impedance (since we are assuming resistance is ~0). Thus, the path $l_{1,2} \rightarrow l_{2,3}$ has twice the impedance as the path $l_{1,3}$ and thus takes half as much power flow coming from Gen A at Bus 1.

### 5. Solve high demand case

Now, let's increase demand at Bus 3 to 800 MW. Despite spare capacity at Gen A, it turns out we will no longer be able to generate all of our power from Gen A alone.

```julia
bus_high = copy(bus)
bus_high[3,:pd] = 800

sol_high = dcopf(gen, branch, gencost, bus_high)
sol_high.generation
```

	id	node	gen

	Int64	Int64	Float64

2 rows × 3 columns

1	1	1	700.0

2	2	2	100.0

This situation is explained by flow patterns, where the capacity of $l_{13}$ is at its maximum, but in order to meet demand at Bus 3, more power needs to be injected in Bus 2, requiring the more costly generator at Bus 2 to dispatch, despite spare capacity at the generator at Bus 1.

```julia
sol_high.flows
```

	fbus	tbus	flow

	Int64	Int64	Float64

6 rows × 3 columns

1	1	3	500.0

2	1	2	200.0

3	2	3	300.0

4	3	1	-500.0

5	2	1	-200.0

6	3	2	-300.0

The following flows are created:

- $l_{1,3}$ = 500 MW
- $l_{1,2}$ = 200 MW
- $l_{2,3}$ = 300 MW

### 6. Compare prices

The marginal values of the demand constraints at a given bus represent the change in the objective that results from increasing demand at the bus by one unit. This is the natural definition of a "value" of power at that location, and is the basis for **[locational marginal prices](https://www.iso-ne.com/participate/support/faq/lmp)** (LMPs) found in electricity markets.

We examine first the regular case of demand = 600 MW, then the high demand case = 800 MW.

```julia
solution.prices
```

	node	value

	Int64	Float64

3 rows × 2 columns

1	1	50.0

2	2	50.0

3	3	50.0

All prices are the same in this case. The interpretation: if we were to add an incremental load at any of the buses, we could meet it from additional production from Gen A which has marginal cost of \$50 / MWh. We are not going to hit any transmission limits.

```julia
sol_high.prices
```

	node	value

	Int64	Float64

3 rows × 2 columns

1	1	50.0

2	2	100.0

3	3	150.0

Something interesting has happened!

First, note that the prices are different. Hence, we will not be able to meet incremental load from production by Gen A (except if we add load right at Gen A located at Bus 1). Similarly, load at Bus 2 can be met by increasing production from Gen B with marginal cost = \$100 / MWh.

However, why does Bus 3 have a marginal price of \$150 / MWh?! That's higher than the marginal cost of either of our two generators?

The answer lies in what must happen to meet an incremental load at Bus 3 while respecting transmission constraints. We must increase from Gen B, but in doing so, part of the power from Gen B will go through $l_{2,1} \rightarrow l_{1,3}$ in addition to $l_{2,3}$, since power flows split across parallel paths in proportion to admittance (or inverse proportion to impedance). However, without adjusting Gen A's output, an increase in production from Gen B will cause us to exceed the transmission constraint on line $l_{1,3}$, requiring us to throttle back power from Gen A to keep power flows feasible.

The exact change in generation for an incremental 1 MW load at Bus 3 is thus:
- Gen B $\uparrow$ 2 MW
- Gen A $\downarrow$ 1 MW

Hence:

$$
Price_3 = 2 \times VarCost_B - VarCost_A = \$150 \text{ / MWh}
$$

In a network with thousands of nodes and many parallel paths and loop flows, one can see quite quickly how prices may vary in unexpected ways; hence, the need for detailed mathematical models to compute locational marginal prices.

### 6. The IEEE 14 bus test system

We now explore a complicated system, the 14-bus IEEE test system, illustrated here:

The system consists of:
- 2 generators (located at nodes 1 and 2)
- 11 loads
- a meshed transmission network including transformers and multiple voltages

Our data files for the test system contain parameters for resistance and reactance of the transmission lines, which are related to complex impedance:

$$
Z = R + iX
$$

where $R$ = resistance is the real part, $X$ = reactance is the imaginary part. Recall from above that impedance is the inverse of admittance; hence, we have the following transformation for susceptance:

$$
B = \text{Im}\left(\frac{1}{R + iX}\right) = \frac{-X}{|R + iX|^2} = \frac{-X}{R^2 + X^2}
$$

But, since we neglect the resistance for the purpose of solving the DC-OPF, we can approximate the susceptance from above as:

$$
B = \frac{1}{X}
$$

The data are converted to have positive values of both $X$ and $B$, hence we remove the negative sign.

```julia
datadir = joinpath("ieee_test_cases")
gens = CSV.read(joinpath(datadir,"Gen14.csv"), DataFrame);
lines = CSV.read(joinpath(datadir,"Tran14_b.csv"), DataFrame);
loads = CSV.read(joinpath(datadir,"Load14.csv"), DataFrame);

# Rename all columns to lowercase (by convention)
for f in [gens, lines, loads]
    rename!(f,lowercase.(names(f)))
end

# create generator ids
gens.id = 1:nrow(gens);

# create line ids
lines.id = 1:nrow(lines);
# add set of rows for reverse direction with same parameters
lines2 = copy(lines)
lines2.f = lines2.fromnode
lines2.fromnode = lines.tonode
lines2.tonode = lines2.f
lines2 = lines2[:,names(lines)]
append!(lines,lines2)

# calculate simple susceptance, ignoring resistance as earlier
lines.b = 1 ./ lines.reactance

# keep only a single time period
loads = loads[:,["connnode","interval-1_load"]]
rename!(loads,"interval-1_load" => "demand");

lines
```

	fromnode	tonode	resistance	reactance	contingencymarked	capacity	id	b

	Int64	Int64	Float64	Float64	Int64	Int64	Int64	Float64

40 rows × 8 columns

1	1	2	0.01938	0.05917	1	10000	1	16.9005

2	1	5	0.05403	0.22304	1	10000	2	4.4835

3	2	3	0.04699	0.19797	1	10000	3	5.05127

4	2	4	0.05811	0.17632	1	10000	4	5.67151

5	2	5	0.05695	0.17388	1	10000	5	5.75109

6	3	4	0.06701	0.17103	1	10000	6	5.84693

7	4	5	0.01335	0.04211	1	10000	7	23.7473

8	4	7	0.0	0.20912	1	10000	8	4.78194

9	4	9	0.0	0.55618	1	10000	9	1.79798

10	5	6	0.0	0.25202	1	10000	10	3.96794

11	6	11	0.09498	0.1989	1	10000	11	5.02765

12	6	12	0.12291	0.25581	1	10000	12	3.90915

13	6	13	0.06615	0.13027	1	10000	13	7.67636

14	7	8	0.0	0.17615	1	10000	14	5.67698

15	7	9	0.0	0.11001	1	10000	15	9.09008

16	9	10	0.03181	0.0845	1	10000	16	11.8343

17	9	14	0.12711	0.27038	1	10000	17	3.6985

18	10	11	0.08205	0.19207	1	10000	18	5.20644

19	12	13	0.22092	0.19988	1	10000	19	5.003

20	13	14	0.17093	0.34802	1	10000	20	2.8734

21	2	1	0.01938	0.05917	1	10000	1	16.9005

22	5	1	0.05403	0.22304	1	10000	2	4.4835

23	3	2	0.04699	0.19797	1	10000	3	5.05127

24	4	2	0.05811	0.17632	1	10000	4	5.67151

25	5	2	0.05695	0.17388	1	10000	5	5.75109

26	4	3	0.06701	0.17103	1	10000	6	5.84693

27	5	4	0.01335	0.04211	1	10000	7	23.7473

28	7	4	0.0	0.20912	1	10000	8	4.78194

29	9	4	0.0	0.55618	1	10000	9	1.79798

30	6	5	0.0	0.25202	1	10000	10	3.96794

⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

```julia
gens
```

	connnode	c2	c1	c0	pgmax	pgmin	rgmax	rgmin	pgprev	id

	Int64	Float64	Int64	Int64	Float64	Int64	Int64	Int64	Float64	Int64

2 rows × 10 columns

1	1	0.0430293	20	0	332.4	0	100	-100	161.619	1

2	2	0.25	20	0	140.0	0	100	-100	97.4704	2

```julia
loads
```

	connnode	demand

	Int64	Float64

11 rows × 2 columns

1	2	-21.7

2	3	-94.2

3	4	-47.8

4	5	-7.6

5	6	-11.2

6	9	-29.5

7	10	-9.0

8	11	-3.5

9	12	-6.1

10	13	-13.5

11	14	-14.9

The structure of these data are different than the above case formats, hence we write a modified solver function:

```julia
#=
Function to solve DC OPF problem using IEEE test cases
Inputs:
    gen_info -- dataframe with generator info
    line_info -- dataframe with transmission lines info
    loads  -- dataframe with load info
=#
function dcopf_ieee(gens, lines, loads)
    DCOPF = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets based on data
      # Set of generator buses
    G = gens.connnode

      # Set of all nodes
    N = sort(union(unique(lines.fromnode),
            unique(lines.tonode)))

      # sets J_i and G_i will be described using dataframe indexing below

    # Define per unit base units for the system
    # used to convert from per unit values to standard unit
    # values (e.g. p.u. power flows to MW/MVA)
    baseMVA = 100 # base MVA is 100 MVA for this system

    # Decision variables
    @variables(DCOPF, begin
        GEN[N]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
        THETA[N]         # voltage phase angle of bus
        FLOW[N,N]        # flows between all pairs of nodes
    end)

    # Create slack bus with reference angle = 0; use bus 1 with generator
    fix(THETA[1],0)

    # Objective function
    @objective(DCOPF, Min,
        sum( gens[g,:c1] * GEN[g] for g in G)
    )

    # Supply demand balances
    @constraint(DCOPF, cBalance[i in N],
        sum(GEN[g] for g in gens[gens.connnode .== i,:connnode])
            + sum(load for load in loads[loads.connnode .== i,:demand])
        == sum(FLOW[i,j] for j in lines[lines.fromnode .== i,:tonode])
    )

    # Max generation constraint
    @constraint(DCOPF, cMaxGen[g in G],
                    GEN[g] <= gens[g,:pgmax])

    # Flow constraints on each branch;
    # In DCOPF, line flow is a function of voltage angles
       # Create an array of references to the line constraints,
       # which we "fill" below in loop
    cLineFlows = JuMP.Containers.DenseAxisArray{Any}(undef, 1:nrow(lines))
    for l in 1:nrow(lines)
        cLineFlows[l] = @constraint(DCOPF,
            FLOW[lines[l,:fromnode],lines[l,:tonode]] ==
            baseMVA * lines[l,:b] *
            (THETA[lines[l,:fromnode]] - THETA[lines[l,:tonode]])
        )
    end

    # Max line flow limits
       # Create an array of references to the line constraints,
       # which we "fill" below in loop
    cLineLimits = JuMP.Containers.DenseAxisArray{Any}(undef, 1:nrow(lines))
    for l in 1:nrow(lines)
        cLineLimits[l] = @constraint(DCOPF,
            FLOW[lines[l,:fromnode],lines[l,:tonode]] <=
            lines[l,:capacity]
        )
    end

    # Solve statement (! indicates runs in place)
    optimize!(DCOPF)

    # Output variables
    generation = DataFrame(
        node = gens.connnode,
        gen = value.(GEN).data[gens.connnode]
        )

    angles = value.(THETA).data

    flows = DataFrame(
        fbus = lines.fromnode,
        tbus = lines.tonode,
        flow = baseMVA * lines.b .* (angles[lines.fromnode] .-
                        angles[lines.tonode]))

    # We output the marginal values of the demand constraints,
    # which will in fact be the prices to deliver power at a given bus.
    prices = DataFrame(
        node = N,
        value = dual.(cBalance).data)

    # Return the solution and objective as named tuple
    return (
        generation = generation,
        angles,
        flows,
        prices,
        cost = objective_value(DCOPF),
        status = termination_status(DCOPF)
    )
end
```

    dcopf_ieee (generic function with 1 method)

```julia
solution = dcopf_ieee(gens, lines, loads);
```

```julia
solution.generation
```

	node	gen

	Int64	Float64

2 rows × 2 columns

1	1	119.0

2	2	140.0

```julia
solution.angles
```

    14-element Array{Float64,1}:
      0.0
     -0.03791491976786273
     -0.18200655970723797
     -0.1453797728538324
     -0.12249815256001242
     -0.22741136081397662
     -0.20659753163771283
     -0.20659753163771283
     -0.23880184324738518
     -0.2442681772629424
     -0.239406951515819
     -0.24660972634711795
     -0.2494179036348092
     -0.26611574994200304

```julia
solution.prices
```

	node	value

	Int64	Float64

14 rows × 2 columns

1	1	20.0

2	2	20.0

3	3	20.0

4	4	20.0

5	5	20.0

6	6	20.0

7	7	20.0

8	8	20.0

9	9	20.0

10	10	20.0

11	11	20.0

12	12	20.0

13	13	20.0

14	14	20.0

```julia

```


```python
%pylab inline
%config InlineBackend.figure_format = 'retina'
from ipywidgets import interact
```

    Populating the interactive namespace from numpy and matplotlib

# Question 1
# A
Consider the function $f(x)=x^3 -9x^2 +11x-11$, which has a root in the interval $(-10,10)$. Calculate by hand the first 3 iterations of Newton's method to estimate the root of $f(x)$, using the initial guess $x_0=0$. What happens?

---------------------------------------
## Solution
Starting with the initial guess $x_0=0$, the first three iterations using Newton's method are
\begin{align}
	x_1 &= 0 - \frac{-11}{11} = 1 \\
	x_2 &= 1 - \frac{1^3-9\cdot1^2+11\cdot1-11}{3\cdot1^2-18\cdot1+11} = 1-\frac{-8}{-4} = -1 \\
	x_3 &= -1 - \frac{(-1)^3-9(-1)^2+11(-1)-11}{3(-1)^2-18(-1)+11} = -1 - \frac{-32}{32} = 0 = x_0
\end{align}
Computing the first three iterates we find that Newton's method with $x_0=0$ results in a cycle with period 3.

# B
Write a python code that uses bisection to determine a better choice of $x_0$, then feed this new initial guess to a Newton's method solver. Iterate your Newton's method solver until you have found the root of $f(x)$ to within a tolerance of $\epsilon=10^{-6}$, and report the value of the root.

```python
# ====== bisection method followed by Newton's method ======
def f(x):
    return x**3 - 9*x**2 + 11*x - 11
def df(x):
    return 3*x**2 - 18*x + 11

# ------ endpoints, max iterations, and tolerance ------
a=-10 #left endpoint
b=10 #right endpoint
N=25 #max iterations for bisection & newton's
tol=(b-1)/2 #very broad tolerance for bisection. NOTE: this was chosen aftercomputing bisection by hand
            #until the subinterval no longer contained -1, 0, or 1. See text below

# ------ bisection (Burden & Faires Numerical Analysis 9th ed) ------
i=1
FA=f(a)
while i<=N:
    p = a + (b-a)/2
    FP=f(p)
    if FP==0 or (b-a)/2<tol: break #return p=x0 for newton's
    i+=1
    if FA*FP>0:
        a=p
        FA=FP
    else:
        b=p

print('Bisection complete. Using x0 =',p,'as initial guess for Newton\'s method')

# ------ newton's method adapted from Lab 2 ------
tol=10**(-6) #update tolerance
x=p #switch from p to x so notation is familiar

# ------ compute x_1 directly so we can do 1st comparison ------
xn = x - f(x)/df(x)
i=1 #initialize an iteration counter

while abs(xn-x)>tol and i<=N:
    x=xn
    xn = x - f(x)/df(x)
    i+=1
print('Convergence for Newton\'s method was achieved using',i,'iterations')
print('The root of f(x) is',x)
```

    Bisection complete. Using x0 = 7.5 as initial guess for Newton's method
    Convergence for Newton's method was achieved using 4 iterations
    The root of f(x) is 7.765951540351796

# Question 2
# A
Derive a third order method for solving $f(x) = 0$ in a way similar to the derivation of Newton’s method, using evaluations of $f(x_n)$, $f’(x_n)$, and $f’’(x_n)$. Show that in the course of derivation, a quadratic equation arises, and therefore two distinct schemes can be derived. **Hint: Expand $f(x)$ around $x_n$.**

---------------------------------------
## Solution
Expand $f(x)$ around $x_n$ to get
$$ f(x) = f(x_n) + (x - x_n)f'(x_n) + \frac{(x - x_n)^2}{2}f''(x_n) + O((x - x_n)^3). $$
Set $x = \hat{x}$ and use $f(\hat{x})=0$ to get
$$ 0 = f(x_n) + (\hat{x} - x_n)f'(x_n) + \frac{(\hat{x} - x_n)^2}{2}f''(x_n) + O((\hat{x} - x_n)^3). $$
Solving for $\hat{x}$, we have
$$\frac{1}{2}f''(x_n) \hat{x}^2 + (f'(x_n) - x_n f''(x_n))\hat{x} + f(x_n) - x_n f'(x_n) + \frac{x_n^2}{2}f''(x_n) = O((\hat{x} - x_n)^3).$$
Setting the right hand side to zero, we find two solutions for $\hat{x}$,
\begin{equation}
\hat{x} =  x_n - \frac{f'(x_n)}{f''(x_n)}\left[1 \pm \sqrt{1 - \frac{2 f''(x_n) f(x_n)}{f'(x_n)^2}}\right].
\end{equation}
**There is one valid solution if we require $x_n \to \hat{x}$ as $f(x_n)\to 0$.** To see this, set $f(x_n)=0$ in the above equation and solve for $y=\hat{x}-x_n$. Only one of the two solutions is zero (assuming that $f'(\hat{x}) \neq 0$). This might seem counter intuitive, but it is not once you look at it in simpler terms. There are two (possibly complex) roots of the polynomial $ax^2 + bx + c = 0$ (assuming $a \neq 0$), but only one of them will vanish as $c\to 0$ (assuming $b \neq 0$).

Hence,
\begin{equation}
\hat{x} =  x_n - \frac{f'(x_n)}{f''(x_n)}\left[1 - \sqrt{1 - \frac{2 f''(x_n) f(x_n)}{f'(x_n)^2}}\right]  + O((\hat{x} - x_n)^3).
\end{equation}
Using the above, we propose the fixed point iteration
$$ x_{n+1} = g(x_n) = x_n - \frac{f'(x_n)}{f''(x_n)}\left[1 - \sqrt{1 - \frac{2 f''(x_n) f(x_n)}{f'(x_n)^2}}\right]$$

------------------------------------------------------------------------------------------
**Comment:** The above is sufficient for full credit. However, one can futher simplify the method by expanding the square root term around $y = 0$, where $y = f(x_n)$. Assuming that $\hat{x}$ is a simple root, we have $f(x_n) = O(x_n - \hat{x})$, $f'(x_n) = O(1)$, and $f''(x_n) = O(1)$. To maintain the same accuracy, we need to expand the square root to explicitly include $O(y^2)$ terms. The $O(y^3)$ terms can be ignored and the same accuracy will be maintained. This leads to the iteration formula
$$ x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)} -  \frac{f''(x_n)f(x_n)^2}{f'(x_n)^3}.$$
One can show that the above iteration method also has cubic convergence.

# B
Show that the order of convergence (under appropriate conditions) is cubic.

---------------------------------------
## Solution
From the solution of part A we have
\begin{equation}
\hat{x} =  x_n - \frac{f'(x_n)}{f''(x_n)}\left[1 - \sqrt{1 - \frac{2 f''(x_n) f(x_n)}{f'(x_n)^2}}\right]  + O((\hat{x} - x_n)^3),
\end{equation}
and
$$ x_{n+1} = g(x_n) = x_n - \frac{f'(x_n)}{f''(x_n)}\left[1 - \sqrt{1 - \frac{2 f''(x_n) f(x_n)}{f'(x_n)^2}}\right].$$
Taking the difference of the above two equation yields
$$ \vert x_{n+1} - \hat{x} \vert = O((\hat{x} - x_n)^3). $$

# C
Implement the root-finding method in Python to compute the root of $f(x) = x^3 - 2$. Add a stopping criterion that requires $\vert f(x_n) \vert \leq 10^{-8}$. Save the value of $x_n$ at each iteration and create a plot showing the convergence rate.

```python
## parameters

max_steps = 50 ## max number of iterations to use
tol = 1e-8 ## 10^(-8)
x0 = 2.
xhat = 2.**(1/3.)

def f(x):
    return x**3 - 2.
def fp(x):
    return 3*x**2
def fpp(x):
    return 6*x

x = [x0]
for j in arange(max_steps):
    fj = f(x[j])
    fpj = fp(x[j])
    fppj = fpp(x[j])
    x.append(x[j] - fpj/fppj*(1 - sqrt(1 - 2*fppj*fj/fpj**2)))
    if absolute(fj) < tol:
        break ## this will stop the for loop

x = array(x) ## convert the list into an array

figure(1, [7, 4])
xplot = linspace(-1, 2, 200)
plot(xplot, 0*xplot, 'k') ## plot the line y=0
plot(xplot, f(xplot)) ## plot f(x)
plot(x, f(x), '*') ## plot the iterates of bisection
xlabel(r'$x$', fontsize=24) ## x axis label
ylabel(r'$f(x)$', fontsize=24); ## y axis label

## Convergence plot
figure(2, [7, 4])
err = absolute(x - xhat)
loglog(err[:-1], err[1:], '*') ## plot the iterates of bisection
err_plot = array([1e-3, 1e-2, 1e-1, 1.])
conv = err_plot**3  # quadratic. the theoretecal convergence curve
loglog(err_plot, conv, 'k')
xlim(1e-3, 1)
xlabel(r'$\vert x_n - \hat{x}\vert$', fontsize=24) ## x axis label
ylabel(r'$\vert x_{n+1} - \hat{x}\vert$', fontsize=24) ## y axis label
title('cubic convergence');
```

# D
Using your code and the function $f$ defined in part C, numerically estimate the number of iterations needed to reduce the initial error,  $\mathcal{E}_0 = \vert \hat{x} - x_0\vert$, by factor of $10^m$ for $m=1, \ldots 4$. Do this for each of the initial guesses $x_0 = 0.25, 1.25$.

```python
x_exact = 2.**(1./3.)
def number_of_iterations(x0, reduce_factor):
    initial_error = absolute(x0 - x_exact)
    x = [x0]
    for j in arange(max_steps):
        fj = f(x[j])
        fpj = fp(x[j])
        fppj = fpp(x[j])
        x.append(x[j] - fpj/fppj*(1 - sqrt(1 - 2*fppj*fj/fpj**2)))
        if absolute(x[-1] - x_exact) < initial_error/reduce_factor:
            break ## this will stop the for loop
    return j + 1

print('for x0 = 0.25')
print('number of iterations needed:')
print([number_of_iterations(0.25, 10.**m) for m in arange(1, 5)])

print('')

print('for x0 = 1.25')
print('number of iterations needed:')
print([number_of_iterations(1.25, 10.**m) for m in arange(1, 5)])

```

    for x0 = 0.25
    number of iterations needed:
    [2, 3, 3, 3]

    for x0 = 1.25
    number of iterations needed:
    [1, 1, 1, 1]

## JUST FOR FUN!!!

```python
x_exact = 2.**(1./3.)
def number_of_iterations_V2(x0, reduce_factor):
    initial_error = absolute(x0 - x_exact)
    x = [x0]
    for j in arange(max_steps):
        fj = f(x[j])
        fpj = fp(x[j])
        fppj = fpp(x[j])
        x.append(x[j] - fj/fpj - fppj*fj**2/fpj**3) ## See the comment in my answer to part A
        if absolute(x[-1] - x_exact) < initial_error/reduce_factor:
            break ## this will stop the for loop
    return j + 1

print('for x0 = -1')
print('number of iterations needed:')
print([number_of_iterations_V2(-1., 10.**m) for m in arange(1, 5)])

print('')

print('for x0 = 0.1')
print('number of iterations needed:')
print([number_of_iterations_V2(0.1, 10.**m) for m in arange(1, 5)])

print('')

print('for x0 = 1.25')
print('number of iterations needed:')
print([number_of_iterations_V2(1.25, 10.**m) for m in arange(1, 5)])

print('')

print('for x0 = 2.25')
print('number of iterations needed:')
print([number_of_iterations_V2(2.25, 10.**m) for m in arange(1, 5)])

print('')

print('for x0 = 3.25')
print('number of iterations needed:')
print([number_of_iterations_V2(3.25, 10.**m) for m in arange(1, 5)])
```

    for x0 = -1
    number of iterations needed:
    [2, 2, 3, 3]

    for x0 = 0.1
    number of iterations needed:
    [19, 20, 20, 21]

    for x0 = 1.25
    number of iterations needed:
    [1, 1, 2, 2]

    for x0 = 2.25
    number of iterations needed:
    [1, 2, 2, 3]

    for x0 = 3.25
    number of iterations needed:
    [2, 3, 3, 4]

```python

```


# Exercise 7) Learning and Planning

In this exercise, we will again investigate the inverted pendulum from the `gym` environment. We want to check, which benefits the implementation of planning offers.

Please note that the parameter $n$ has a different meaning in the context of planning (number of planning steps per actual step) than in the context of n-step learning.

```python
import numpy as np
import gym
gym.logger.set_level(40)
import random
import time
from tqdm.notebook import tqdm
import matplotlib.pyplot as plt
plt.style.use('seaborn-talk')
```

We will reuse the discretization routine from the previous exercise:

```python
d_T = 15
d_theta = 15
d_omega = 15

def discretize_state(states):
    limits = [1, 1, 8]
    nb_disc_intervals = [d_theta, d_theta, d_omega]

    # bring to value range [-1, 1]
    norm_states = [state / limit for state, limit in zip(states, limits)]
    interval_lengths = [2 / d for d in nb_disc_intervals]
    disc_state = [(norm_state + 1) // interval_length for norm_state, interval_length in zip(norm_states, interval_lengths)]
    disc_state = [(state - 1) if state == d else state for state, d in zip(disc_state, nb_disc_intervals)] # ensure that disc_state < d

    return np.array(disc_state)

def continualize_action(disc_action):
    limit = 2
    interval_length = 2 / (d_T-1)
    norm_action = disc_action * interval_length
    cont_action = (norm_action - 1) * limit
    return np.array(cont_action).flatten()
```

## 1) Dyna-Q

Write a Dyna-Q algorithm to solve the inverted pendulum. Check the quality of the result for different number of episodes, number of steps per episode and number of planning steps per interaction.

Make sure that the total number of learning steps stays the same for different n, such that comparisons are fair:

$\text{episodes} \cdot \text{steps} \cdot (1+n) = \text{const.}$

Interesting metrics for a comparison could be e.g. the execution time (the tqdm loading bar shows execution time of loops, alternatively you can use the time.time() command to get the momentary system time in seconds) and training stability.

## Solution 1)

The solution code is given below.

```python
def pendulumDynaQ(alpha, gamma, epsilon, n, nb_episodes, nb_steps):

    env = gym.make('Pendulum-v0')
    env = env.unwrapped

    action_values = np.zeros([d_theta, d_theta, d_omega, d_T])
    pi = np.zeros([d_theta, d_theta, d_omega])

    model = {} # dictionary

    cumulative_reward_history = [] # we can use this to figure out how well the learning worked

    for j in tqdm(range(nb_episodes), position=0, leave=True):

        ### BEGIN SOLUTION

        rewards = [] # this time, this list is only for monitoring

        state = env.reset() # initialize x_0
        state = tuple(discretize_state(state).astype(int))

        for k in range(nb_steps):

            # sample experiences for the model
            if np.random.uniform(0, 1) < epsilon:
                action = np.random.choice(d_T) # explorative action
            else:
                action = pi[state].astype(int) # exploitative action

            cont_action = continualize_action(action)
            next_state, reward, done, _ = env.step(cont_action)
            next_state = tuple(discretize_state(next_state).astype(int))

            # learn from the momentary experience
            action_values[state][action] += alpha * (reward
                                                          + gamma * np.max(action_values[next_state])
                                                          - action_values[state][action]
                                                         )
            pi[state] = np.argmax(action_values[state])
            model[state, action] = (next_state, reward) # update the model accoring to the latest experience
            state = next_state

            rewards.append(reward)

            # learn from the model, IMPORTANT: USE DIFFERENT VARIABLES HERE
            for i in range(n):
                # sample a random key from the dict: this state action combination has surely been taken in the past
                x, u = random.sample(model.keys(), 1)[0]
                x_1, r = model[x, u]
                action_values[x][u] += alpha * (r
                                                + gamma * np.max(action_values[x_1])
                                                - action_values[x][u]
                                               )
                pi[x] = np.argmax(action_values[x])

            if done:
                break

        cumulative_reward_history.append(np.sum(rewards))

    env.close()

    ### END SOLUTION

    return cumulative_reward_history, pi

```

Function to evaluate and render the measurement using the gym environment:

```python
def experiment(pi,nb_steps = 300):
# Runs the inverted pendulum experiment using policy pi for nb_steps steps

    env = gym.make('Pendulum-v0')
    env = env.unwrapped

    state = env.reset() # initialize x_0
    disc_state = tuple(discretize_state(state).astype(int)) # only tuples of integers can be used as index
    disc_action = pi[disc_state].astype(int)

    for k in range(nb_steps):

        cont_action = continualize_action(disc_action)
        env.render() # comment out for faster execution
        state, reward, done, _ = env.step(cont_action)
        disc_state = tuple(discretize_state(state).astype(int))

        if done:
            break

        disc_action = pi[disc_state].astype(int) # exploitative action

    env.close()
```

Let's use nb_episodes = 5000, nb_steps = 500, n = 0 as a first try. This is effectively Q learning.

\begin{align}
5000 \cdot 500 \cdot (0+1) &= 2.5 \cdot 10^6
\end{align}

The resulting policy is satisfactory.

```python
### train
print("Run without planning")
no_planning_history, no_planning_pi = pendulumDynaQ(alpha = 0.1, gamma = 0.9, epsilon = 0.1, n = 0, nb_episodes = 5000, nb_steps = 500)
```

    Run without planning

    HBox(children=(FloatProgress(value=0.0, max=5000.0), HTML(value='')))

```python
### run and render the experiment
experiment(no_planning_pi,nb_steps = 300)
```

Now let's try $n=9$ with the same nb_steps = 500:

\begin{align}
\text{nb_episodes} &= \frac{2.5 \cdot 10^6}{500 \cdot (9+1)} = 500
\end{align}

The resulting policy also looks good.

```python
### train
print("Run with planning")
with_planning_history, with_planning_pi = pendulumDynaQ(alpha = 0.1, gamma = 0.9, epsilon = 0.1, n = 9, nb_episodes = 500, nb_steps = 500)
```

    Run with planning

    HBox(children=(FloatProgress(value=0.0, max=500.0), HTML(value='')))

```python
### run and render the experiment
experiment(with_planning_pi,nb_steps = 300)
```

Now lets compare the cumulative rewards:

```python
plt.plot(no_planning_history)
plt.plot(with_planning_history)
plt.xlabel("episode")
plt.ylabel(r"$\sum R$")
plt.show()
```

The cumulative reward over the episodes both seems to have high variance.

So why should we prefer the planning method?
### Planning leads to the agent interacting less often with the "real" environment, such that in the end fewer interaction time is needed.

## 2) Simulation-based planning

Although it can be useful for small state spaces, building a system model by storing large amounts of state transitions like in task (1) is rarely feasible in engineering. As engineers, we are capable of a more efficient way of system modeling that we can utilize here: differential equations.

Using a state-space model allows to efficiently integrate existing pre-knowledge into the Dyna-Q algorithm we already used. To do so, write a class `pendulum_model` that implements a model of the pendulum. This class should work similar to `gym`: it should at least have a `step` and a `reset` method. In the step method, make use of forward Euler integration to simulate the system dynamics. In the reset method, allow to pass an optional initial state to the model, such that we can easily compare model and environment. If no initial state is passed to the `reset` function, a random initial state should be determined.

Integrate this model into a Dyna-Q algorithm.

Model of the pendulum in differential-equation form for change of the angular frequency $\omega$ and the angle $\theta$ depending on the torque $T_\mathrm{u}$:

\begin{align}
\dot{\omega} &= -\frac{3 g}{2 l} \text{sin}(\theta +\pi) + \frac{1}{J} T_\mathrm{u}
\\
\dot{\theta} &= \omega
\end{align}

Parameters (gravity constant $g$, mass $m$, length  $l$ and intertia $J$ of the pendulum):

\begin{align}
g&=10 \, \frac{\text{m}}{\text{s}^2}
&
m&=1 \, \text{kg}
&
l&=1 \, \text{m}
&
J&=\frac{1}{3} m l^2
\end{align}

Forward Euler integration:

\begin{align}
\dot{x}(k T_S) \approx \frac{x[k+1] - x[k]}{T_S}
\end{align}

with sampling time $T_S = 0.05 \, \text{s}$

Reward function:

\begin{align}
r_{k+1} = -(\theta^2[k] + 0.1 \, \text{s}^2 \cdot \omega^2[k] + 0.001 \frac{1}{(\text{N}\text{m})^2} \cdot T_\mathrm{u}^2[k])
\end{align}

Limitations of state and action space:
\begin{align}
\theta &\in [-\pi, \pi]
&
\omega &\in [-8  \, \frac{1}{\text{s}}, 8  \, \frac{1}{\text{s}}]
&
T_\mathrm{u} &\in [-2 \, \text{N}\text{m}, 2 \, \text{N}\text{m}]
\end{align}

And of course input and output space:
\begin{align}
\text{action}&=T_\mathrm{u}
&
\text{state}&=
\begin{bmatrix}
\text{cos}(\theta)\\
\text{sin}(\theta)\\
\omega
\end{bmatrix}
\end{align}

## Solution 2)

Model-based planning does not necessarily run faster than experience-based planning. However, experience-based planning fails to cover the whole state space especially in the earlier episodes when there are too few experiences. On the other hand, model-based planning can, of course, only be performed if a state-space model with accurate parametrization is available. In order to overcome parametric deviations between state-space model and environment, one could even use the measurements from the actual environment in order to identify the parameters of the model during runtime.

```python

class pendulum_model:
    def __init__(self, dt=0.05, m=1, g=10, l=1):

        ### BEGIN SOLUTION

        self.max_speed  = 8
        self.max_torque = 2

        self.dt = dt # sampling time in s
        self.g  =  g # gravity in m / s^2
        self.m  =  m # mass in kg
        self.l  =  l # length in m

        self.J = 1 / 3 * m * l ** 2 # pedulums moment of inertia in kg * m^2

        ### END SOLUTION

    def reset(self, state=None):

        ### BEGIN SOLUTION

        # if no state is give, set randomly
        if np.any(state == None):
            self.theta = np.random.uniform(-np.pi, +np.pi)
            self.omega = np.random.uniform(-self.max_speed, +self.max_speed)

        # else set initial state as given
        else:
            self.theta = np.arctan2(state[1], state[0])
            self.omega = state[2]

        state = np.array([np.cos(self.theta), np.sin(self.theta), self.omega])

        ### END SOLUTION

        return state

    def step(self, T_u):

        ### BEGIN SOLUTION

        T_u = np.clip(T_u, -self.max_torque, +self.max_torque)[0]

        reward = -(self.angle_normalize(self.theta)** 2 + 0.1 * self.omega ** 2 + 0.001 * (T_u ** 2))

        # differential-equations for state values
        self.omega = self.omega + self.dt * (-3 * self.g/(2 * self.l) * np.sin(self.theta + np.pi) + 1 / self.J * T_u)
        self.theta = self.theta + self.dt * self.omega

        self.omega = np.clip(self.omega, -self.max_speed, +self.max_speed)

        state = np.array([np.cos(self.theta), np.sin(self.theta), self.omega])

        ### END SOLUTION

        return state, reward

    def angle_normalize(self, theta):
        # usage of this helper function is optional
        return (((theta+np.pi) % (2*np.pi)) - np.pi)
```

The following cell is for debugging of the `pendulum_model` class.

```python
env = gym.make('Pendulum-v0')
env = env.unwrapped # removes a builtin time limit of k_T = 200, we want to determine the time limit ourselves

model = pendulum_model()

state = env.reset()
print(state)
m_state = model.reset(state) # model is set to state of env

for _ in range(10000):
    print(f"state: {state}")
    print(f"model: {m_state}")
    print()

    action = env.action_space.sample()

    state, reward, done, _ = env.step(action) # take action on env
    m_state, m_reward = model.step(action) # take the same action on model

    print(f"reward difference: {reward- m_reward}, env_reward:{reward}, model_reward:{m_reward}")

env.close()
```

Using $-\mathrm{sin}(\theta)$ instead of $\mathrm{sin}(\theta +\pi)$ makes no difference when assuming analytical precision, but due to numeric errors these formulations will still yield different results in numpy, mainly because $\pi$ is represented with finite (float) precision. In order to yield the same numbers as in `gym`, we will still make use of the (more cumbersome) $\mathrm{sin}(\theta +\pi)$.

Write a function for the Dyna-Q algorithm, which uses the model we defined above:

```python
def pendulumModelDynaQ(alpha, gamma, epsilon, n, nb_episodes, nb_steps):

    env = gym.make('Pendulum-v0')
    env = env.unwrapped
    model = pendulum_model()

    action_values = np.zeros([d_theta, d_theta, d_omega, d_T])
    pi = np.zeros([d_theta, d_theta, d_omega])

    cumulative_reward_history = [] # we can use this to figure out how well the learning worked

    for j in tqdm(range(nb_episodes), position=0, leave=True):

        ### BEGIN SOLUTION

        rewards = [] # this time, this list is only for monitoring

        state = env.reset() # initialize x_0
        state = tuple(discretize_state(state).astype(int))

        for k in range(nb_steps):

            # sample experiences for the model
            if np.random.uniform(0, 1) < epsilon:
                action = np.random.choice(d_T) # explorative action
            else:
                action = pi[state].astype(int) # exploitative action

            cont_action = continualize_action(action)
            next_state, reward, done, _ = env.step(cont_action)
            next_state = tuple(discretize_state(next_state).astype(int))

            # learn from the momentary experience
            action_values[state][action] += alpha * (reward
                                                          + gamma * np.max(action_values[next_state])
                                                          - action_values[state][action]
                                                         )
            pi[state] = np.argmax(action_values[state])
            # no model update is needed
            state = next_state

            rewards.append(reward)

            # learn from the model, IMPORTANT: USE DIFFERENT VARIABLES HERE
            for i in range(n):
                x = model.reset() # if no state is passed to the model, state is initialized randomly
                u_d = np.random.choice(d_T)

                x_d = tuple(discretize_state(x).astype(int))
                u = continualize_action(u_d)

                x_1, r = model.step(u)
                x_1_d = tuple(discretize_state(x_1).astype(int))
                action_values[x_d][u_d] += alpha * (r
                                                    + gamma * np.max(action_values[x_1_d])
                                                    - action_values[x_d][u_d]
                                                    )
                pi[x_d] = np.argmax(action_values[x_d])

            if done:
                break

        cumulative_reward_history.append(np.sum(rewards))

    env.close()

    ### END SOLUTION

    return cumulative_reward_history, pi
```

Use the following cell to compare the learing from experience from 1) to the learning using the defined model: (Beware, nb_steps = 10000 can take some time)

```python
### train both setups once
print("Run with planning from experience")
exp_planning_history, exp_planning_pi = pendulumDynaQ(alpha = 0.1, gamma = 0.9, epsilon = 0.1, n = 19, nb_episodes = 30, nb_steps = 10000)

print("Run with planning from model")
model_planning_history, model_planning_pi = pendulumModelDynaQ(alpha = 0.1, gamma = 0.9, epsilon = 0.1, n = 19, nb_episodes = 30, nb_steps = 10000)

plt.plot(exp_planning_history)
plt.plot(model_planning_history)
plt.xlabel("episode")
plt.ylabel(r"$\sum R$")
plt.show()

```

Use the following cell to execute the policy we got using the model:

```python
experiment(model_planning_pi,nb_steps = 300)
```

### Extra-task:
Change the model parameters (e.g. $g$, $m$, $l$) so that our model differs from the "real world" (we got from gym).
What do you observe

By changing the parameters our model differs from the "real world". Depending on the amount of difference, the learing curve looks worse than the one with the correct values.
The experiment result is also depending on the random starting position.
Depending on the parameter difference, the experiment can not be executed successfully any more.

Try to change the parameters on your own.

The Following learning curve results from a parameter change using $g =20 \, \frac{\text{m}}{\text{s}^2}, m = 5 \, \text{kg and } l = 2 \, \text{m}$:

```python
plt.plot(exp_planning_history, label="exp")
plt.plot(model_planning_history, label="model")
plt.xlabel("episode")
plt.ylabel(r"$\sum R$")
plt.legend(loc="upper left")
plt.show()

```