# Economic Dispatch

_**[Power Systems Optimization](https://github.com/east-winds/power-systems-optimization)**_

_by Michael R. Davidson and Jesse D. Jenkins (last updated: August 31, 2021)_

This notebook will introduce a key operational model—economic dispatch (ED)—which minimizes the short-run production costs of meeting electricity demand from a given set of generators subject to various technical constraints.

We build up the model in several stages. We begin with a single-time period and simple generator constraints just related to allowable output ranges. This allows us to compare the model with the intuition provided by the merit order supply curve (also known as the "dispatch stack").

We then add multiple time periods by considering the economic dispatch over an entire day. Due to the simple generator constraints, this simple model does not introduce any coupling across time.

More realism is added by considering complex engineering constraints that introduce time coupling—namely, ramp rates, which are an important limitation of many generators.

Finally, we examine the famous "duck curve" by adding more solar onto the system and explore the role that ramp rates play in the solution.

## Introduction to economic dispatch

Every day, system operators need to decide how to meet demand from a large variety of generators with different costs and engineering requirements. Prior to large-scale computational capabilities, this was accomplished by "priority lists" or similar heuristics to determine the ordering of plants. However, with modern optimization algorithms, operators can find the global minimum of production costs—and even incorporate coupling of engineering processes across time periods.

**Economic dispatch** (ED) is the problem of minimizing short-run costs of production in order to meet a given demand, considering relevant engineering constraints of the generators. It *does not* fully capture the physics of electricity flows, which will further constrain the feasible space and which we will come back to in later notebooks on optimal power flow, nor does it account for decisions and constraints related to turning on or "committing" large thermal generators, which we'll discuss in our next notebook on unit commitment. Typically, *it does* incorporate some type of network representation, which we have ignored here by considering only a single bus. More on the networks in the notebook on optimal power flow.

## Single-time period, simple generator constraints

We will first examine the case where we are optimizing dispatch for a single snapshot in time, with only very simple constraints on the generators.

$$
\begin{align}
\min \ & \sum_{g \in G} VarCost_g \times GEN_g & \\
\text{s.t.} & \\
 & \sum_{g} GEN_g = Demand & \\
 & GEN_g \leq Pmax_g & \forall \quad g \in G \\
 & GEN_g \geq Pmin_g & \forall \quad g \in G \\
\end{align}
$$

The **decision variable** in the above problem is:

- $GEN_{g}$, the generation (in MW) produced by each generator, $g$

The **parameters** are:

- $Pmin_g$, the minimum operating bounds for the generator (based on engineering or natural resource constraints)
- $Pmax_g$, the maximum operating bounds for the generator (based on engineering or natural resource constraints)
- $Demand$, the demand (in MW)

(For this simple problem, we will let $Pmin_g=0$ and revisit this later in the unit commitment notebook.)

In addition, we have:

$$
VarCost_g = VarOM_g + HeatRate_g \times FuelCost_g
$$

Note that, in contrast with the basic capacity expansion problem, we are not concerned with fixed costs. Why not?

The simplest response is that these costs have already been incurred and regardless of how much a generator produces, its fixed costs will not change. These costs are thus "[sunk costs](https://en.wikipedia.org/wiki/Sunk_cost)" and are constant in the objective function. Our optimal decision variables would not change by adjusting this constant up or down.

Therefore, we can safely ignore fixed costs for the purposes of finding optimal dispatch. (We will still have to consider them to calculate producer profits, however.)

[A famous electricity regulator](https://en.wikipedia.org/wiki/Ignacio_J._P%C3%A9rez_Arriaga), noting that it is frequently misunderstood and ignored even by experienced market participants, has called the application of the "[sunk cost fallacy](https://en.wikipedia.org/wiki/Sunk_cost#Fallacy_effect)" to electricity systems "[Grandma's Inheritance Theorem](http://link.springer.com/content/pdf/bbm%3A978-1-4471-5034-3%2F1.pdf)." In this hypothetical scenario, if you were to inherit a diamond ring or perhaps a financial contract-for-differences that pays out relative to cleared market prices (you have a very sophisticated grandma in this scenario!), you should not be tempted to change your production strategy.

Now, let's implement ED.

### 1. Load packages

We will try out another open source solver GLPK in this tutorial.

```julia
# New packages introduced in this tutorial
import Pkg; Pkg.add("PrettyTables"); Pkg.add("VegaLite"); Pkg.add("GLPK"); Pkg.add("FileIO")
```

        Updating registry at `~/.julia/registries/General`
        Updating git-repo `https://github.com/JuliaRegistries/General.git`
       Resolving package versions...
      No Changes to `~/.julia/environments/v1.6/Project.toml`
      No Changes to `~/.julia/environments/v1.6/Manifest.toml`
       Resolving package versions...
      No Changes to `~/.julia/environments/v1.6/Project.toml`
      No Changes to `~/.julia/environments/v1.6/Manifest.toml`
       Resolving package versions...
      No Changes to `~/.julia/environments/v1.6/Project.toml`
      No Changes to `~/.julia/environments/v1.6/Manifest.toml`
       Resolving package versions...
      No Changes to `~/.julia/environments/v1.6/Project.toml`
      No Changes to `~/.julia/environments/v1.6/Manifest.toml`

```julia
using JuMP, GLPK
using Plots; plotly();
using VegaLite  # to make some nice plots
using DataFrames, CSV, PrettyTables
using FileIO
ENV["COLUMNS"]=120; # Set so all columns of DataFrames and Matrices are displayed
```

    ┌ Info: For saving to png with the Plotly backend PlotlyBase has to be installed.
    └ @ Plots /Users/michd/.julia/packages/Plots/4UbNP/src/backends.jl:435

### 2. Load and format data

We will use data based on San Diego Gas and Electric (SDG&E, via the [PowerGenome](https://github.com/gschivley/PowerGenome) data platform) plus a few neighboring generators, consisting of:

- 25 generators (including some clustering of smaller generators, and excluding behind-the-meter solar)
- estimated hourly demand for 2020 (net load at the transmission substation level after subtracting 600MW of behind-the-meter solar from original demand)
- variable generation capacity factors
- estimated natural gas fuel costs

```julia
datadir = joinpath("ed_data")
# Note: joinpath is a good way to create path reference that is agnostic
# to what file system you are using (e.g. whether directories are denoted
# with a forward or backwards slash).
gen_info = CSV.read(joinpath(datadir,"Generators_data.csv"), DataFrame);
fuels = CSV.read(joinpath(datadir,"Fuels_data.csv"), DataFrame);
loads = CSV.read(joinpath(datadir,"Demand.csv"), DataFrame);
gen_variable = CSV.read(joinpath(datadir,"Generators_variability.csv"), DataFrame);

# Rename all columns to lowercase (by convention)
for f in [gen_info, fuels, loads, gen_variable]
    rename!(f,lowercase.(names(f)))
end
```

**Construct generator dataframe**

```julia
# Keep only the columns relevant to our ED model
# (We'll come back to other columns in our unit commitment notebooks)
select!(gen_info, 1:26, :stor)
gen_df = outerjoin(gen_info,  fuels, on = :fuel) # load in fuel costs and add to data frame
rename!(gen_df, :cost_per_mmbtu => :fuel_cost)   # rename column for fuel cost
gen_df[ismissing.(gen_df[:,:fuel_cost]), :fuel_cost] .= 0

# create "is_variable" column to indicate if this is a variable generation source (e.g. wind, solar)
# Note: Julia's strict vectorization syntax requires us to create a new variable using the '!' indexing operator
# and the '.=' broadcasting assignment. See: https://julia.guide/broadcasting
gen_df[!, :is_variable] .= false
gen_df[in(["onshore_wind_turbine","small_hydroelectric","solar_photovoltaic"]).(gen_df.resource),
    :is_variable] .= true;

# create full name of generator (including geographic location and cluster number)
#  for use with variable generation dataframe
gen_df.gen_full = lowercase.(gen_df.region .* "_" .* gen_df.resource .* "_" .* string.(gen_df.cluster) .* ".0");

# remove generators with no capacity (e.g. new build options that we'd use if this was capacity expansion problem)
gen_df = gen_df[gen_df.existing_cap_mw .> 0,:];

```

### Long and wide data format example

```julia
df = gen_df[1:5,1:5]
```

5 rows × 5 columns

	r_id	resource	region	existing_cap_mw	num_units

	Int64?	String?	String?	Float64?	Int64?

1	1	biomass	WEC_SDGE	21.4	20

2	2	hydroelectric_pumped_storage	WEC_SDGE	42.0	2

3	3	natural_gas_fired_combined_cycle	WEC_SDGE	607.1	1

4	4	natural_gas_fired_combined_cycle	WEC_SDGE	30.0	1

5	5	natural_gas_fired_combined_cycle	WEC_SDGE	49.4	1

```julia
df = copy(gen_df)
select!(df, 1:5)
df = df[1:5,:]
```

5 rows × 5 columns

	r_id	resource	region	existing_cap_mw	num_units

	Int64?	String?	String?	Float64?	Int64?

1	1	biomass	WEC_SDGE	21.4	20

2	2	hydroelectric_pumped_storage	WEC_SDGE	42.0	2

3	3	natural_gas_fired_combined_cycle	WEC_SDGE	607.1	1

4	4	natural_gas_fired_combined_cycle	WEC_SDGE	30.0	1

5	5	natural_gas_fired_combined_cycle	WEC_SDGE	49.4	1

```julia
stack(df,
    [:existing_cap_mw, :num_units],
    variable_name=:var,
    value_name=:val)
```

10 rows × 5 columns

	r_id	resource	region	var	val

	Int64?	String?	String?	String	Float64?

1	1	biomass	WEC_SDGE	existing_cap_mw	21.4

2	2	hydroelectric_pumped_storage	WEC_SDGE	existing_cap_mw	42.0

3	3	natural_gas_fired_combined_cycle	WEC_SDGE	existing_cap_mw	607.1

4	4	natural_gas_fired_combined_cycle	WEC_SDGE	existing_cap_mw	30.0

5	5	natural_gas_fired_combined_cycle	WEC_SDGE	existing_cap_mw	49.4

6	1	biomass	WEC_SDGE	num_units	20.0

7	2	hydroelectric_pumped_storage	WEC_SDGE	num_units	2.0

8	3	natural_gas_fired_combined_cycle	WEC_SDGE	num_units	1.0

9	4	natural_gas_fired_combined_cycle	WEC_SDGE	num_units	1.0

10	5	natural_gas_fired_combined_cycle	WEC_SDGE	num_units	1.0

**Modify load and variable generation dataframes**

1. Convert from GMT to GMT-8

The load and variable generation files are in GMT. We want to convert to local California time. Here, we will ignore daylight savings and simply subtract 8 hours. (A more proper solution would involve the use of the [TimeZones](https://github.com/JuliaTime/TimeZones.jl/) package.)

To accomplish this, we 'wrap' the last 8 hours of the data series (i.e., the last 8 hours of the year) around to the front of the series.

```julia
gen_variable.hour = mod.(gen_variable.hour .- 9, 8760) .+ 1
sort!(gen_variable, :hour)
loads.hour = mod.(loads.hour .- 9, 8760) .+ 1
sort!(loads, :hour);
```

2. Convert from "wide" to "long" format:

The file is in "wide" format, which has separate columns for values from different types of generators. This has more human readability, but often we want to have it in "long" format for computation.

"Long" format refers to a dataframe with a separate row entry for every value. The rest of the columns are "indexes" that uniquely identify the row. In Julia, we use the DataFrame package's [`stack`](https://juliadata.github.io/DataFrames.jl/stable/man/reshaping_and_pivoting/) function to accomplish this. (This is similar to the [`melt`](https://www.rdocumentation.org/packages/reshape2/versions/1.4.4/topics/melt) function for those familiar with DataFrames in R)

```julia
describe(gen_variable)
```

31 rows × 7 columns

	variable	mean	min	median	max	nmissing	eltype

	Symbol	Float64	Real	Float64	Real	Int64	DataType

1	hour	4380.5	1	4380.5	8760	0	Int64

2	wec_sdge_biomass_1.0	1.0	1	1.0	1	0	Int64

3	wec_sdge_hydroelectric_pumped_storage_1.0	1.0	1	1.0	1	0	Int64

4	wec_sdge_natural_gas_fired_combined_cycle_1.0	1.0	1	1.0	1	0	Int64

5	wec_sdge_natural_gas_fired_combined_cycle_2.0	1.0	1	1.0	1	0	Int64

6	wec_sdge_natural_gas_fired_combined_cycle_3.0	1.0	1	1.0	1	0	Int64

7	wec_sdge_natural_gas_fired_combined_cycle_4.0	1.0	1	1.0	1	0	Int64

8	wec_sdge_natural_gas_fired_combustion_turbine_1.0	1.0	1	1.0	1	0	Int64

9	wec_sdge_natural_gas_fired_combustion_turbine_2.0	1.0	1	1.0	1	0	Int64

10	wec_sdge_natural_gas_fired_combustion_turbine_3.0	1.0	1	1.0	1	0	Int64

11	wec_sdge_natural_gas_fired_combustion_turbine_4.0	1.0	1	1.0	1	0	Int64

12	wec_sdge_natural_gas_fired_combustion_turbine_5.0	1.0	1	1.0	1	0	Int64

13	wec_sdge_natural_gas_fired_combustion_turbine_6.0	1.0	1	1.0	1	0	Int64

14	wec_sdge_natural_gas_fired_combustion_turbine_7.0	1.0	1	1.0	1	0	Int64

15	wec_sdge_natural_gas_fired_combustion_turbine_8.0	1.0	1	1.0	1	0	Int64

16	wec_sdge_natural_gas_fired_combustion_turbine_9.0	1.0	1	1.0	1	0	Int64

17	wec_sdge_natural_gas_fired_combustion_turbine_10.0	1.0	1	1.0	1	0	Int64

18	wec_sdge_natural_gas_fired_combustion_turbine_11.0	1.0	1	1.0	1	0	Int64

19	wec_sdge_natural_gas_fired_combustion_turbine_12.0	1.0	1	1.0	1	0	Int64

20	wec_sdge_onshore_wind_turbine_1.0	0.253519	0.0	0.19465	0.9089	0	Float64

21	wec_sdge_small_hydroelectric_1.0	0.41567	0.2167	0.38475	0.6337	0	Float64

22	wec_sdge_solar_photovoltaic_1.0	0.247288	0.0	0.0	0.7959	0	Float64

23	wec_sdge_naturalgas_ccccsavgcf_0.0	1.0	1	1.0	1	0	Int64

24	wec_sdge_naturalgas_ccavgcf_0.0	1.0	1	1.0	1	0	Int64

25	wec_sdge_naturalgas_ctavgcf_0.0	1.0	1	1.0	1	0	Int64

26	wec_sdge_battery_0.0	1.0	1	1.0	1	0	Int64

27	wec_sdge_naturalgas_ccs100_0.0	1.0	1	1.0	1	0	Int64

28	wec_sdge_landbasedwind_ltrg1_1.0	0.341167	0.0	0.249565	0.93217	0	Float64

29	wec_sdge_landbasedwind_ltrg1_2.0	0.257347	0.0	0.194376	0.912162	0	Float64

30	wec_sdge_utilitypv_losangeles_1.0	0.346479	0.0	0.0	1.0	0	Float64

⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

```julia
gen_variable_long = stack(gen_variable,
                        Not(:hour),
                        variable_name=:gen_full,
                        value_name=:cf);
# Now we have a "long" dataframe;
# let's look at the first 6 entries of a wind resource for example
first(gen_variable_long[gen_variable_long.gen_full.=="wec_sdge_onshore_wind_turbine_1.0",:],6)
```

6 rows × 3 columns

	hour	gen_full	cf

	Int64	String	Float64

1	1	wec_sdge_onshore_wind_turbine_1.0	0.1694

2	2	wec_sdge_onshore_wind_turbine_1.0	0.1658

3	3	wec_sdge_onshore_wind_turbine_1.0	0.1502

4	4	wec_sdge_onshore_wind_turbine_1.0	0.1455

5	5	wec_sdge_onshore_wind_turbine_1.0	0.0932

6	6	wec_sdge_onshore_wind_turbine_1.0	0.0717

**Extract single time from the data**
(we will get to the full set later)

- demand
- variable generation

Note: variable generation profiles are often encoded in terms of hourly maximum capacity factor, expressed as a per unit value or percentage of installed capacity. Hourly capacity factor is then later multiplied by installed capacity to yield a maximum hourly generation for each resource.

```julia
hr = 2416  # pick 4pm on a spring day
loads_single = loads[loads[:,:hour] .== hr, Not(:hour)];
var_cf_single = gen_variable_long[
    gen_variable_long.hour .== hr,
    Not(:hour)]
```

30 rows × 2 columns

	gen_full	cf

	String	Float64

1	wec_sdge_biomass_1.0	1.0

2	wec_sdge_hydroelectric_pumped_storage_1.0	1.0

3	wec_sdge_natural_gas_fired_combined_cycle_1.0	1.0

4	wec_sdge_natural_gas_fired_combined_cycle_2.0	1.0

5	wec_sdge_natural_gas_fired_combined_cycle_3.0	1.0

6	wec_sdge_natural_gas_fired_combined_cycle_4.0	1.0

7	wec_sdge_natural_gas_fired_combustion_turbine_1.0	1.0

8	wec_sdge_natural_gas_fired_combustion_turbine_2.0	1.0

9	wec_sdge_natural_gas_fired_combustion_turbine_3.0	1.0

10	wec_sdge_natural_gas_fired_combustion_turbine_4.0	1.0

11	wec_sdge_natural_gas_fired_combustion_turbine_5.0	1.0

12	wec_sdge_natural_gas_fired_combustion_turbine_6.0	1.0

13	wec_sdge_natural_gas_fired_combustion_turbine_7.0	1.0

14	wec_sdge_natural_gas_fired_combustion_turbine_8.0	1.0

15	wec_sdge_natural_gas_fired_combustion_turbine_9.0	1.0

16	wec_sdge_natural_gas_fired_combustion_turbine_10.0	1.0

17	wec_sdge_natural_gas_fired_combustion_turbine_11.0	1.0

18	wec_sdge_natural_gas_fired_combustion_turbine_12.0	1.0

19	wec_sdge_onshore_wind_turbine_1.0	0.3006

20	wec_sdge_small_hydroelectric_1.0	0.4351

21	wec_sdge_solar_photovoltaic_1.0	0.5695

22	wec_sdge_naturalgas_ccccsavgcf_0.0	1.0

23	wec_sdge_naturalgas_ccavgcf_0.0	1.0

24	wec_sdge_naturalgas_ctavgcf_0.0	1.0

25	wec_sdge_battery_0.0	1.0

26	wec_sdge_naturalgas_ccs100_0.0	1.0

27	wec_sdge_landbasedwind_ltrg1_1.0	0.482802

28	wec_sdge_landbasedwind_ltrg1_2.0	0.302244

29	wec_sdge_utilitypv_losangeles_1.0	0.690915

30	wec_sdge_utilitypv_losangeles_2.0	0.743176

### 3. Create solver function

Here, we introduce a functionalized version of the optimization problem. We pass the problem parameters to the function, and it constructs and solves the model, returning the result. This will be useful when we wish to call the same basic model multiple times, e.g. for conducting sensitivities on parameters (among other things).

```julia
#=
Function to solve economic dispatch problem (single-time period, single-zone)
Inputs:
    gen_df -- dataframe with generator info
    loads  -- dataframe with load info
    gen_variable -- capacity factors of variable generators (in "long" format)
Note: it is always a good idea to include a comment blog describing your
function's inputs clearly!
=#
function economic_dispatch_single(gen_df, loads, gen_variable)
    ED = Model(GLPK.Optimizer) # You could use Clp as well, with Clp.Optimizer

    # Define sets based on data
      # A set of all variable generators
    G_var = gen_df[gen_df[!,:is_variable] .== 1,:r_id]
      # A set of all non-variable generators
    G_nonvar = gen_df[gen_df[!,:is_variable] .== 0,:r_id]
      # Set of all generators
    G = gen_df.r_id
    # Extract some parameters given the input data
      # Generator capacity factor time series for variable generators
    gen_var_cf = innerjoin(gen_variable,
                    gen_df[gen_df.is_variable .== 1 ,
                        [:r_id, :gen_full, :existing_cap_mw]],
                    on = :gen_full)

    # Decision variables
    @variables(ED, begin
        GEN[G]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
    end)

    # Objective function
    @objective(ED, Min,
        sum( (gen_df[i,:heat_rate_mmbtu_per_mwh] * gen_df[i,:fuel_cost] +
            gen_df[i,:var_om_cost_per_mwh]) * GEN[i]
                        for i in G_nonvar) +
        sum(gen_df[i,:var_om_cost_per_mwh] * GEN[i]
                        for i in G_var)
    )

    # Demand constraint
    @constraint(ED, cDemand,
        sum(GEN[i] for i in G) == loads[1,:demand])

    # Capacity constraint (non-variable generation)
    for i in G_nonvar
        @constraint(ED, GEN[i] <= gen_df[i,:existing_cap_mw])
    end

    # Variable generation capacity constraint
    for i in 1:nrow(gen_var_cf)
        @constraint(ED, GEN[gen_var_cf[i,:r_id] ] <=
                        gen_var_cf[i,:cf] *
                        gen_var_cf[i,:existing_cap_mw])
    end

    # Solve statement (! indicates runs in place)
    optimize!(ED)

    # Dataframe of optimal decision variables
    solution = DataFrame(
        r_id = gen_df.r_id,
        resource = gen_df.resource,
        gen = value.(GEN).data
        )

    # Return the solution and objective as named tuple
    return (
        solution = solution,
        cost = objective_value(ED),
    )
end
```

    economic_dispatch_single (generic function with 1 method)

### 4. Solve and print data

```julia
solution = economic_dispatch_single(gen_df, loads_single, var_cf_single);
solution.solution
```

25 rows × 3 columns

	r_id	resource	gen

	Int64?	String?	Float64

1	1	biomass	21.4

2	2	hydroelectric_pumped_storage	42.0

3	3	natural_gas_fired_combined_cycle	607.1

4	4	natural_gas_fired_combined_cycle	30.0

5	5	natural_gas_fired_combined_cycle	49.4

6	6	natural_gas_fired_combined_cycle	570.0

7	7	natural_gas_fired_combined_cycle	336.0

8	8	natural_gas_fired_combined_cycle	336.0

9	9	natural_gas_fired_combustion_turbine	0.0

10	10	natural_gas_fired_combustion_turbine	49.9

11	11	natural_gas_fired_combustion_turbine	0.0

12	12	natural_gas_fired_combustion_turbine	0.0

13	13	natural_gas_fired_combustion_turbine	86.8545

14	14	natural_gas_fired_combustion_turbine	0.0

15	15	natural_gas_fired_combustion_turbine	0.0

16	16	natural_gas_fired_combustion_turbine	0.0

17	17	natural_gas_fired_combustion_turbine	0.0

18	18	natural_gas_fired_combustion_turbine	0.0

19	19	natural_gas_fired_combustion_turbine	0.0

20	20	natural_gas_fired_combustion_turbine	0.0

21	21	natural_gas_fired_combustion_turbine	0.0

22	22	natural_gas_fired_combustion_turbine	0.0

23	23	onshore_wind_turbine	65.4106

24	24	small_hydroelectric	3.18493

25	25	solar_photovoltaic	284.75

We will now plot the **supply curve** (also known as the **dispatch stack**) for this single period to illustrate the concept of **merit order**. First, let's create a dataframe with the variable cost as a column, and then sort on it from lowest to highest.

```julia
supply_curve = leftjoin(gen_df,
                var_cf_single,
                on = :gen_full)
supply_curve[!, :varcost] .= 0.0
supply_curve[!, :cap] .= 0.0

# Store varcost of non-variable generators:
I = supply_curve[!,:is_variable] .== 0  # `I` contains indexes to the non-variable generators
supply_curve[I,:varcost] .=
    supply_curve[I,:heat_rate_mmbtu_per_mwh] .*
        supply_curve[I,:fuel_cost] .+
    supply_curve[I,:var_om_cost_per_mwh]

# Calculate available capacity for each generator
supply_curve[I,:cap] = supply_curve[I,:existing_cap_mw]

# Store varcost of variable generators (in this case, 0)
I = (supply_curve[:,:is_variable] .== 1) # `I` contains indexes to the variable generators
supply_curve[I,:varcost] = supply_curve[I,:var_om_cost_per_mwh]

# Calculate available capacity for each generator
# (adjusted for variable generation)
supply_curve[I,:cap] = supply_curve[I,:existing_cap_mw] .* supply_curve[I,:cf]

sort!(supply_curve, :varcost);
```

Now, let's plot this as a supply curve where:

- width = capacity of the generator (i.e., $Pmax$)
- height = variable cost of the generator ($VarCost$)

```julia
rectangle(w, h, x, y) = Shape(x .+ [0,w,w,0], y .+ [0,0,h,h])

p = plot(nrow(supply_curve))

marg_gen = 0
marg_price = 0
x = 0
for i in 1:nrow(supply_curve)
    plot!(p, rectangle(supply_curve[i,:cap],
            supply_curve[i,:varcost],x,0), opacity=.5,
            label = "")
    if (x < loads_single[1,:demand]) && (x + supply_curve[i,:cap] > loads_single[1,:demand])
        marg_gen = i
        marg_price = supply_curve[i,:varcost]
    end
    x = x + supply_curve[i,:cap]
end
vline!([loads_single[1,:demand]],linecolor="black",linewidth=5,
label = "demand")
hline!([marg_price],linecolor="blue",linewidth=5,
label = "SRMC")
title!("Supply and demand curves")
xlabel!("Capacity")
ylabel!("Marginal cost")
p
```

In the above, everything to the left of demand is dispatched at full capacity, except for the last generator (the "marginal generator") where demand stops. Everything to the right is not dispatched (=0). We stored the marginal generator in `marg_gen`:

```julia
supply_curve[marg_gen,:]
```

DataFrameRow (34 columns)

	r_id	resource	region	existing_cap_mw	num_units	cap_size	var_om_cost_per_mwh	var_om_cost_per_mwh_in

	Int64?	String?	String?	Float64?	Int64?	Float64?	Float64?	Int64?

13	13	natural_gas_fired_combustion_turbine	WEC_SDGE	96.6	3	32.2	10.8	0

Which is generating $GEN_{marg\_gen}=$

```julia
solution.solution[solution.solution.r_id .== supply_curve[marg_gen,:r_id],
    :gen][1]
```

    86.85450800000007

**Combine by generation type**

We are often interested in the total generation *by type*, not individual generators. We use `combine` and `groupby` methods to sum over resource:

```julia
sol_gen = combine(groupby(solution.solution, :resource),
            :gen => sum)
# We use the PrettyTable package here to round numbers when displaying
pretty_table(sol_gen, formatters = ft_printf("%5.3f"))
```

    ┌──────────────────────────────────────┬──────────┐
    │                             resource │  gen_sum │
    │                              String? │  Float64 │
    ├──────────────────────────────────────┼──────────┤
    │                              biomass │   21.400 │
    │         hydroelectric_pumped_storage │   42.000 │
    │     natural_gas_fired_combined_cycle │ 1928.500 │
    │ natural_gas_fired_combustion_turbine │  136.755 │
    │                 onshore_wind_turbine │   65.411 │
    │                  small_hydroelectric │    3.185 │
    │                   solar_photovoltaic │  284.750 │
    └──────────────────────────────────────┴──────────┘

Finally, let's add back the 600MW of behind-the-meter solar to get a full picture of all the electricity on the grid.

```julia
# we use the copy function to create a copy of the dataframe (as opposed to working with the original)
sol_gen_btm = copy(sol_gen)

# We'll use the same variability profile for BTM solar as the utility-scale solar here (in reality, they would probably differ)
btm = DataFrame(resource = ["solar_photovoltaic_btm"], gen_sum = var_cf_single[var_cf_single.gen_full .== "wec_sdge_solar_photovoltaic_1.0",:cf] * 600)
append!(sol_gen_btm, btm)
pretty_table(sol_gen_btm, formatters = ft_printf("%5.3f")) # We use the PrettyTable package to round numbers when displaying
```

    ┌──────────────────────────────────────┬──────────┐
    │                             resource │  gen_sum │
    │                              String? │  Float64 │
    ├──────────────────────────────────────┼──────────┤
    │                              biomass │   21.400 │
    │         hydroelectric_pumped_storage │   42.000 │
    │     natural_gas_fired_combined_cycle │ 1928.500 │
    │ natural_gas_fired_combustion_turbine │  136.755 │
    │                 onshore_wind_turbine │   65.411 │
    │                  small_hydroelectric │    3.185 │
    │                   solar_photovoltaic │  284.750 │
    │               solar_photovoltaic_btm │  341.700 │
    └──────────────────────────────────────┴──────────┘

Hence, the majority of energy is being met by combined cycle natural gas turbines (CCGT), second by solar PV, and third by behind-the-meter solar.

## Multiple-time period, simple generator constraints

We neglected the time dimension above and now we will add it back in. The key changes to the model arise from adding an additional index for time:

$$
\begin{align}
\min \ & \sum_{g \in G, t \in T} VarCost_g \times GEN_{g,t} & \\
\text{s.t.} & \\
 & \sum_{g} GEN_{g,t} = Demand_t & \forall \quad t \in T \\
 & GEN_{g,t} \leq Pmax_{g,t} & \forall \quad g \in G , t \in T \\
 & GEN_{g,t} \geq Pmin_{g,t} & \forall \quad g \in G , t \in T
\end{align}
$$

Note that for conventional generators, we often have $Pmax_{g,t} = Pmax_{g}$ and $Pmin_{g,t} = Pmin_{g}$ constant over time. (We will consider unit commitment later, which will alter this logic.)

For variable renewable generators, it is natural that $Pmax_{g,t}$ varies with time (i.e., based on solar irradiation or wind speeds). $Pmin_{g,t}$ for wind and solar resources is usually 0, but for hydropower resources, minimum streamflow constraints can produce a time-variant parameter.

### 3. Create solver function

(We reuse steps 1 and 2 above to load packages and data.)

```julia
#=
Function to solve economic dispatch problem (multi-time period, single-zone)
Inputs:
    gen_df -- dataframe with generator info
    loads  -- load by time
    gen_variable -- capacity factors of variable generators (in "long" format)
=#
function economic_dispatch_multi(gen_df, loads, gen_variable)
    ED = Model(GLPK.Optimizer)

    # Define sets based on data
    G_var = gen_df[gen_df[!,:is_variable] .== 1,:r_id]
    G_nonvar = gen_df[gen_df[!,:is_variable] .== 0,:r_id]
    G = gen_df.r_id
    T = loads.hour

    # Generator capacity factor time series for variable generators
    gen_var_cf = innerjoin(gen_variable,
                    gen_df[gen_df.is_variable .== 1 ,
                        [:r_id, :gen_full, :existing_cap_mw]],
                    on = :gen_full)

    # Decision variables
    @variables(ED, begin
        GEN[G, T]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
    end)

    # Objective function
    @objective(ED, Min,
        sum( (gen_df[gen_df.r_id .== i,:heat_rate_mmbtu_per_mwh][1] * gen_df[gen_df.r_id .== i,:fuel_cost][1] +
            gen_df[gen_df.r_id .== i,:var_om_cost_per_mwh][1]) * GEN[i,t]
                        for i in G_nonvar for t in T) +
        sum(gen_df[gen_df.r_id .== i,:var_om_cost_per_mwh][1] * GEN[i,t]
                        for i in G_var for t in T)
    )

    # Demand constraint
    @constraint(ED, cDemand[t in T],
        sum(GEN[i,t] for i in G) == loads[loads.hour .== t,:demand][1])

    # Capacity constraints (non-variable generation)
    @constraint(ED, Cap_nonvar[i in G_nonvar, t in T],
        GEN[i,t] <= gen_df[gen_df.r_id .== i,:existing_cap_mw][1])

    # Variable generation capacity constraints
    @constraint(ED, Cap_var[i in 1:nrow(gen_var_cf)],
            GEN[gen_var_cf[i,:r_id], gen_var_cf[i,:hour] ] <=
                        gen_var_cf[i,:cf] *
                        gen_var_cf[i,:existing_cap_mw])

    # Solve statement (! indicates runs in place)
    optimize!(ED)

    # Dataframe of optimal decision variables
    # This requires a few extra steps because we are indexing decision variables over two sets: generator and time
    solution = DataFrame(value.(GEN).data, :auto)  # :auto argument asks DataFrames to pick the variable names automatically
    ax1 = value.(GEN).axes[1]
    ax2 = value.(GEN).axes[2]
    cols = names(solution)
    insertcols!(solution, 1, :r_id => ax1)
    solution = stack(solution, Not(:r_id), variable_name=:hour)
    solution.hour = foldl(replace, [cols[i] => ax2[i] for i in 1:length(ax2)], init=solution.hour)
    rename!(solution, :value => :gen)
    solution.hour = convert.(Int64,solution.hour)

    # Return the solution and objective as named tuple
    return (
        solution = solution,
        cost = objective_value(ED),
        status = termination_status(ED)
    )

end
```

    economic_dispatch_multi (generic function with 1 method)

### 4. Solve a day's economic dispatch

We will subset our year-long dataset to a single spring day and solve.

```julia
n=100
T_period = (n*24+1):((n+1)*24)

loads_multi = loads[in.(loads.hour,Ref(T_period)),:]
gen_variable_multi = gen_variable_long[in.(gen_variable_long.hour,Ref(T_period)),:]

gen_df_sens = copy(gen_df)
gen_df_sens[gen_df_sens.resource .== "solar_photovoltaic",
    :existing_cap_mw] .= 3500

solution = economic_dispatch_multi(gen_df_sens, loads_multi, gen_variable_multi);
```

```julia
# **Add back in BTM solar and plot results**

sol_gen = innerjoin(solution.solution,
                    gen_df[!, [:r_id, :resource]],
                    on = :r_id)
sol_gen = combine(groupby(sol_gen, [:resource, :hour]),
            :gen => sum)
sol_gen_btm = copy(sol_gen)
sol_gen_btm[sol_gen_btm.resource .== "solar_photovoltaic", :resource] .= "_solar_photovoltaic"
sol_gen_btm[sol_gen_btm.resource .== "onshore_wind_turbine", :resource] .= "_onshore_wind_turbine"
sol_gen_btm[sol_gen_btm.resource .== "small_hydroelectric", :resource] .= "_small_hydroelectric"

btm = DataFrame(resource = repeat(["_solar_photovoltaic_btm"]; outer=length(T_period)),
    hour = T_period,
    gen_sum = gen_variable_multi[gen_variable_multi.gen_full .== "wec_sdge_solar_photovoltaic_1.0",:cf] * 600)
append!(sol_gen_btm, btm)

# Reorder resources
sort!(sol_gen_btm, [:hour, :resource])
CSV.write("results/sol_gen_btm.csv",sol_gen_btm)

sol_gen_sum = combine(groupby(sol_gen_btm, [:resource]),
            :gen_sum => sum)
```

8 rows × 2 columns

	resource	gen_sum_sum

	String?	Float64

1	_onshore_wind_turbine	2314.42

2	_small_hydroelectric	76.4384

3	_solar_photovoltaic	20546.0

4	_solar_photovoltaic_btm	3522.18

5	biomass	513.6

6	hydroelectric_pumped_storage	1008.0

7	natural_gas_fired_combined_cycle	29749.6

8	natural_gas_fired_combustion_turbine	1224.89

**Plot charts**

We use the [VegaLite](https://www.queryverse.org/VegaLite.jl/stable/) library to make some fancier plots.

We also introduce the pipe operator `|>`, a helpful syntax that "pipes" the data on the left into the first argument of the function on the right. (Some more examples are [here](https://syl1.gitbook.io/julia-language-a-concise-tutorial/useful-packages/pipe).)

```julia
sol_gen_btm |>
@vlplot(:line,
    x=:hour, y=:gen_sum,
    color={"resource:n", scale={scheme="category20"}})

```

```julia
g = sol_gen_btm |>
@vlplot(:area,
    x=:hour, y={:gen_sum, stack=:zero},
    color={"resource:n", scale={scheme="category20"}},
    width=600,height=600)
g |> FileIO.save("results/ED_multitime.png")
```

For reference, here is the load shape (after subtracting BTM solar) we are trying solve:

```julia
loads_multi |>
@vlplot(:line,
    x=:hour, y=:demand)
```

## Multiple-time period, complex generator constraints with time coupling

The above problem did not include more than one time index per constraint. Hence, we could actually solve this each time period separately without regard to what is happening before or after. (These are known as "separable" problems.)

We now introduce time coupling into the model, by considering ramp rates, which limit how much generators can change output from one period to the next (an engineering constraint).

The new model:

$$
\begin{align}
\min \ & \sum_{g \in G, t \in T} VarCost_g \times GEN_{g,t} & \\
\text{s.t.} & \\
 & \sum_{g} GEN_{g,t} = Demand_t & \forall \quad t \in T \\
 & GEN_{g,t} \leq Pmax_{g,t} & \forall \quad g \in G , t \in T \\
 & GEN_{g,t} \geq Pmin_{g,t} & \forall \quad g \in G , t \in T \\
 & GEN_{g,t+1} - GEN_{g,t} \leq RampUp_{g} & \forall \quad g \in G , t = 1..T-1 \\
  & GEN_{g,t} - GEN_{g,t+1} \leq RampDn_{g} & \forall \quad g \in G , t = 1..T-1
\end{align}
$$

### 3. Create solver function

(We reuse steps 1 and 2 above to load packages and data.)

First, we define a function to convert two-dimensional outputs from JuMP into easy-to-read dataframes.

```julia
#=
Function to convert JuMP outputs (technically, AxisArrays) with two-indexes to a dataframe
Inputs:
    var -- JuMP AxisArray (e.g., value.(GEN))
Reference: https://jump.dev/JuMP.jl/v0.19/containers/
=#
function value_to_df_2dim(var)
    solution = DataFrame(var.data, :auto)
    ax1 = var.axes[1]
    ax2 = var.axes[2]
    cols = names(solution)
    insertcols!(solution, 1, :r_id => ax1)
    solution = stack(solution, Not(:r_id), variable_name=:hour)
    solution.hour = foldl(replace, [cols[i] => ax2[i] for i in 1:length(ax2)], init=solution.hour)
    rename!(solution, :value => :gen)
    solution.hour = convert.(Int64,solution.hour)
    return solution
end
```

    value_to_df_2dim (generic function with 1 method)

```julia
#=
Function to solve economic dispatch problem (multi-time period, single-zone)
Inputs:
    gen_df -- dataframe with generator info
    loads  -- load by time
    gen_variable -- capacity factors of variable generators (in "long" format)
=#
function economic_dispatch_multi_time(gen_df, loads, gen_variable)
    ED = Model(GLPK.Optimizer)

    # Define sets based on data
    G_var = gen_df[gen_df[!,:is_variable] .== 1,:r_id]
    G_nonvar = gen_df[gen_df[!,:is_variable] .== 0,:r_id]
    G = gen_df.r_id
    T = loads.hour
    T_red = loads.hour[1:end-1]  # time periods used for ramp constraints

    # Generator capacity factor time series for variable generators
    gen_var_cf = innerjoin(gen_variable,
                    gen_df[gen_df.is_variable .== 1 ,
                        [:r_id, :gen_full, :existing_cap_mw]],
                    on = :gen_full)

    # Decision variables
    @variables(ED, begin
        GEN[G, T]  >= 0     # generation
        # Note: we assume Pmin = 0 for all resources for simplicty here
    end)

    # Objective function
    @objective(ED, Min,
        sum( (gen_df[gen_df.r_id .== i,:heat_rate_mmbtu_per_mwh][1] * gen_df[gen_df.r_id .== i,:fuel_cost][1] +
            gen_df[gen_df.r_id .== i,:var_om_cost_per_mwh][1]) * GEN[i,t]
                        for i in G_nonvar for t in T) +
        sum(gen_df[gen_df.r_id .== i,:var_om_cost_per_mwh][1] * GEN[i,t]
                        for i in G_var for t in T)
    )

    # Demand constraint
    @constraint(ED, cDemand[t in T],
        sum(GEN[i,t] for i in G) == loads[loads.hour .== t,:demand][1])

    # Capacity constraints (non-variable generation)
    @constraint(ED, Cap_nonvar[i in G_nonvar, t in T],
        GEN[i,t] <= gen_df[gen_df.r_id .== i,:existing_cap_mw][1])

    # Variable generation capacity constraints
    @constraint(ED, Cap_var[i in 1:nrow(gen_var_cf)],
            GEN[gen_var_cf[i,:r_id], gen_var_cf[i,:hour] ] <=
                        gen_var_cf[i,:cf] *
                        gen_var_cf[i,:existing_cap_mw])

    # Ramp up constraints
    @constraint(ED, RampUp[i in G, t in T_red],
        GEN[i,t+1] - GEN[i,t] <= gen_df[gen_df.r_id .== i,:existing_cap_mw][1] *
                                 gen_df[gen_df.r_id .== i,:ramp_up_percentage][1] )

    # Ramp down constraints
    @constraint(ED, RampDn[i in G, t in T_red],
        GEN[i,t] - GEN[i,t+1] <= gen_df[gen_df.r_id .== i,:existing_cap_mw][1] *
                                 gen_df[gen_df.r_id .== i,:ramp_dn_percentage][1] )

    # Solve statement (! indicates runs in place)
    optimize!(ED)

    # Dataframe of optimal decision variables
    solution = value_to_df_2dim(value.(GEN))

    # Dataframe of ramp up constraint marginal values
    # (aka "dual values" of the ramp constraint)
    # (to be used in solar sensitivity case)
    marginals = value_to_df_2dim(dual.(RampUp))

    # Return the solution, objective, and marginals as named tuple
    return (
        solution = solution,
        cost = objective_value(ED),
        marginals = marginals
    )

end
```

    economic_dispatch_multi_time (generic function with 1 method)

### 4. Solve a day's economic dispatch

```julia
solution = economic_dispatch_multi_time(gen_df, loads_multi, gen_variable_multi);

sol_gen = innerjoin(solution.solution,
                    gen_df[!, [:r_id, :resource]],
                    on = :r_id)
sol_gen = combine(groupby(sol_gen, [:resource, :hour]),
            :gen => sum)
sol_gen_btm = sol_gen
sol_gen_btm[sol_gen_btm.resource .== "solar_photovoltaic", :resource] .= "_solar_photovoltaic"
sol_gen_btm[sol_gen_btm.resource .== "onshore_wind_turbine", :resource] .= "_onshore_wind_turbine"
sol_gen_btm[sol_gen_btm.resource .== "small_hydroelectric", :resource] .= "_small_hydroelectric"

btm = DataFrame(resource = repeat(["_solar_photovoltaic_btm"]; outer=length(T_period)),
    hour = T_period,
    gen_sum = gen_variable_multi[gen_variable_multi.gen_full .== "wec_sdge_solar_photovoltaic_1.0",:cf] * 600)
append!(sol_gen_btm, btm)

# Reorder resources
sort!(sol_gen_btm, [:hour, :resource])

sol_gen_btm |>
@vlplot(:area,
    x=:hour, y={:gen_sum, stack=:zero},
    color={"resource:n", scale={scheme="category20"}},
    width=600,height=600) |>
    FileIO.save("results/ED_moderate.png")
```

### 5. Run sensitivity on solar capacity

Let's now examine what happens if we increase utility solar capacity from the current 500MW up to 3,500MW.

```julia
# Increase solar photovoltaic capacity
gen_df_sens = copy(gen_df)
gen_df_sens[gen_df_sens.resource .== "solar_photovoltaic",
    :existing_cap_mw] .= 3500

solution = economic_dispatch_multi_time(gen_df_sens, loads_multi, gen_variable_multi);

sol_gen = innerjoin(solution.solution,
                    gen_df[!, [:r_id, :resource]],
                    on = :r_id)

sol_gen = combine(groupby(sol_gen, [:resource, :hour]),
            :gen => sum)
sol_gen_btm = copy(sol_gen)
sol_gen_btm[sol_gen_btm.resource .== "solar_photovoltaic", :resource] .= "_solar_photovoltaic"
sol_gen_btm[sol_gen_btm.resource .== "onshore_wind_turbine", :resource] .= "_onshore_wind_turbine"
sol_gen_btm[sol_gen_btm.resource .== "small_hydroelectric", :resource] .= "_small_hydroelectric"

btm = DataFrame(resource = repeat(["_solar_photovoltaic_btm"]; outer=length(T_period)),
    hour = T_period,
    gen_sum = gen_variable_multi[gen_variable_multi.gen_full .== "wec_sdge_solar_photovoltaic_1.0",:cf] * 600)
append!(sol_gen_btm, btm)

# Reorder resources
sort!(sol_gen_btm, [:hour, :resource])

# Plot and save the resulting generation figure
p = sol_gen_btm |>
@vlplot(:area,
    x=:hour, y={:gen_sum, stack=:zero},
    color={"resource:n", scale={scheme="category20"}},
    width=600,height=600)
save("results/sol_gen_btm.png",p)
p
```

The large reduction in net load during the day and the loss of solar (orange) in the late afternoon timed with the rise in the evening peak is sometimes called the **"duck curve"** because of their passing resemblance to a duck's belly and head. Squint at it and then examine this (https://www.energy.gov/eere/articles/confronting-duck-curve-how-address-over-generation-solar-energy):

You may notice that the model still uses combustion turbines (light pink above) during hours when there is sufficient capacity of combined cycle plants, which are more efficient. This is because of ramp limits, which we will examine now.

All constraints (in a linear program) have a **"marginal value"** (also known as "shadow price" or "dual value") which is how much the objective would change if the right-hand-side (RHS) of the constraint were increased by one unit. Let's examine the marginals of the ramp up constraint for the combined cycle plants using the values from the new equation we added to `economic_dispatch_multi_time` (line 72).

```julia
marg_gen = innerjoin(solution.marginals,
                    gen_df[!, [:r_id, :resource]],
                    on = :r_id)
marg_gen[marg_gen.resource .== "natural_gas_fired_combined_cycle",:] |>
@vlplot(:line,
    x=:hour, y={:gen, title="Marginal"},
    color={"r_id:n", scale={scheme="category20"}})

```

Thus, during the ramp hours, the marginal value becomes negative—in this case, reaching ~ -9 for the combined cycle plant with `r_id=5`. This is the interpretation:

**If the RHS of the RampUp constraint for `r_id=5` combined cycle plant were increased by 1 unit, the problem objective would reduce by about 9.**

The precise value of the largest magnitude ramp marginal is:

```julia
marg_gen[argmin(marg_gen.gen),:]
```

DataFrameRow (4 columns)

	r_id	hour	gen	resource

	Int64?	Int64	Float64	String?

430	5	2418	-8.9672	natural_gas_fired_combined_cycle

**Why is this the case?** Due to the ramp constraint, we have to substitute some generation from `r_id=5` with a more expensive unit, in this case the more flexible `r_id=9` combustion turbine.

We can, in fact, recover this marginal value by re-examining the variable costs of these generators at the top of the notebook:

```julia
varcost_comp = supply_curve[in.(supply_curve.r_id,Ref([5,9])),[:r_id,:varcost]]
varcost_comp
```

2 rows × 2 columns

	r_id	varcost

	Int64?	Float64

1	5	29.3061

2	9	38.2733

If our guess is true, the difference between these should be our marginal value, and voila:

```julia
varcost_comp[2,2] - varcost_comp[1,2]
```

    8.967199999999998

Also notice that the marginals are zero for most of the day. Recalling from Notebook 1 our discussion on "binding" constraints, this is because the RampUp constraint is **not binding** during these hours. Alternatively, changing the RHS of the RampUp constraint does not affect the problem objective.


# Models, Data, Learning Problems

Welcome to the first lab of the lecture "Intelligent Data Analysis I". In this lab we start our first data analysis on a concrete problem. We are using Fisher's famous Iris data set. The goal is to classify flowers from the Iris family into one of three species, that look as follows:

Iris Setosa
Iris Versicolor
Iris Virginica

Our data set contains 50 flowers from each class, thus 150 in total. There are four features, the length and width of the petal (dt. Kronblatt) and sepal (dt. Kelchblatt) in centimetres.

Your goal is to go through the notebook, understand premade code and text as well as filling blanks and exercises left for you. You may also edit the notebook as you wish. A good way to learn is to add comments (lines starting with #) or modifying the code and see what changes.

The data set is distributed with sci-kit learn, the only thing we have to do is to important a function and call it.

```python
from sklearn.datasets import load_iris

data = load_iris()
X = data.data
y = data.target
print(type(X))
print(X.shape)

print("First three rows of data\n {}".format(X[:5]))
print("First three labels: {}".format(y[:5]))
```

    <class 'numpy.ndarray'>
    (150, 4)
    First three rows of data
     [[5.1 3.5 1.4 0.2]
     [4.9 3.  1.4 0.2]
     [4.7 3.2 1.3 0.2]
     [4.6 3.1 1.5 0.2]
     [5.  3.6 1.4 0.2]]
    First three labels: [0 0 0 0 0]

 Not only do we get the input matrix $X \in \mathbb{R}^{150 \times 4}$ and target $y \in \mathbb{R}^{150}$, but also meta information such as what the class labels $0, 1, 2$ stand for and what the features (i.e. columns of $X$) correspond to.

```python
print(data.target_names)
print(data.feature_names)
```

    ['setosa' 'versicolor' 'virginica']
    ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

As a first step we focus our analysis on the first two variables, the sepal length and sepal width. Since we obtain a representation of the data in two dimensions, we are able to plot it.

```python
X_2 = X[:, :2]
y_2 = y
```

```python
# Configures Jupyter to show graphics in the notebook
%matplotlib inline
import matplotlib.pyplot as plt # standard import

# We write a function so we can reuse it later.
def generate_scatter_plot(X, y):
    class_names = data.target_names
    class_colours = ['blue','yellow','green']

    fig = plt.figure(figsize=(12, 6)) # increase size of plot

    for i, class_colour in enumerate(class_colours):
        # plot the points only of this class label
        plt.scatter(X[y == i, 0], X[y == i, 1], c=class_colour, label=class_names[i])

    plt.xlabel(data.feature_names[0]) # label the axis
    plt.ylabel(data.feature_names[1])
    plt.legend(loc="best") # with legend

generate_scatter_plot(X_2, y)
```

We see that we could discriminate the iris setosa linearly from other two species. The linear function could even have a slope of about $1$. Let us substitute the first feature with the difference of the two features.

```python
import numpy as np
x_new = X[:, 0] - X[:, 1]
X_new = np.column_stack((x_new, X[:, 1]))
print (X_new.shape)
generate_scatter_plot(X_new, y)
plt.xlabel("sepal length - sepal width")
```

Remember that our main goal is to find a model,

$$ y_\theta: X \rightarrow Y $$

such that $y_\theta(x)$ models the knowledge we got from our training data plus the inductive bias. The plot gives the decision rule (or part of): "If sepal length - sepal width <= 2.2 => Classify iris setosa".

Exercise 1:

Implement the naive decision rule as given above. If the condition for iris setosa is not fulfilled, classify the result as 'iris versicolor'.

```python
def naive_decision_rule(x):
    # X matrix with 4 dimensional columns
    # returns for each row of X the expected class label (0 = setosa, 1 = versicolor, 2 = virginica)
    #print(x[3])
    if x[0]-x[1]  <= 2.2 : # sepal width
        return 0 # setosa
    else:
        return 1
```

The following function takes a decision rule (or model) and a matrix of data points to generate the predictions for this matrix.

```python
def predict(model, X):
    """Builds prediction on a matrix X given a model for each data point in a row.
    Returns a flat vector of predictions.
    """
    return np.apply_along_axis(model, axis=1, arr=X)

y_pred = predict(naive_decision_rule, X)
print(y_pred)
print(y)
```

    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]

The predictions of the first 50 numbers should be zero and one for all others. Now we have to judge the quality of our model, we do this by using the zero-loss function of the lecture.

Exercise 2:

Implement the zero loss function as defined in the lecture,
$$
\begin{align}
l(x, y; \theta) &= l_{0/1}(y_\theta(x_i), y_i) = \begin{cases} 0, & \mbox{ if } y_\theta(x_i) = y_i \\ 1, & \mbox{ otherwise } \end{cases} \\
l(X, y, theta) &= \sum_i{ l(x_i, y_i; \theta). }
\end{align}
$$
In lay-man terms one counts how often the label predicted differed from the observed label.

```python
def zero_one_loss(y_pred, y_true):
    # FILL IN
    loss=0
    for i in range(len(y_pred)) :
        if y_pred[i]==y_true[i] :
            loss=loss+0
        else :
            loss=loss+1

    return loss
```

```python
print('The 0-1-loss of the naive decision rule is ' + str(zero_one_loss(y_pred, y)) + ' (should be 50)')
```

    The 0-1-loss of the naive decision rule is 50 (should be 50)

Exercise 3:

Improve the decision rule to have a maximum number of missclassifications of $10$. As an informal constraint use "Occams Razor" as an inductive bias, i.e. as simplest as possible.

Discussion topic: Why could a complex model with zero missclassifications perform worse in reality (we got out and measure new flowers) than a simple model with more misclassifications?

```python
#['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
generate_scatter_plot(np.column_stack((X[:,0],X[:,3])), y)
plt.ylabel("sepal width")
plt.xlabel("petal width")
```

```python
# Place for your analysis.
def my_decision_rule(x):
    # FILL IN
     #print(x[3])
    if x[3] <= 0.75 : # sepal width
        return 0 # setosa
    elif x[3] >= 0.75 and x[3] <= 1.65 : # sepal width
        return 1 # versicolor
    elif x[3] > 1.65 and x[3] < 2.65  : # sepal width
        return 2 # virginica
    else:
        return -1
```

```python
# Evaluation script
y_pred = predict(my_decision_rule, X)
print (y_pred)
loss = zero_one_loss(y_pred, y)
print ("Your loss {}.".format(loss))

if loss <= 10:
    print("You have made it!")
else:
    print("Uhm, try again. Maybe you have flipped some class?")
```

    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1
     1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 1 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]
    Your loss 6.
    You have made it!

```python

```

2. Lab02_Model_Analysis/Models, Data, Learning Problems.ipynb


# 선형계획법 Linear Programming

```python
import matplotlib.pyplot as plt
import numpy as np
import numpy.linalg as nl
import scipy.optimize as so

```

ref :
* Wikipedia [link](https://en.wikipedia.org/wiki/Linear_programming)
* Stackoverflow [link](https://stackoverflow.com/questions/62571092/)
* Tips & Tricks on Linux, Matlab, vim, LaTex, etc [link](http://tipstrickshowtos.blogspot.com/2012/04/how-to-render-argmax-argmin-operator-in.html)

## Problem description
문제 설명

* Area of the farm 농장의 넓이 : $L = 10 (km^2)$
* Types of crops : wheat or rice
작물의 종류 : 밀 또는 쌀
* Available fertilizer 사용 가능한 비료의 양 : $F = 10 (kg)$
* Available pesticide 사용 가능한 살충제의 양 : $P = 5 (kg)$

|   | Wheat 밀 | rice 쌀 |
|:-----:|:-----:|:-----:|
| Needed Fertilizer per unit area $(kg/km^2)$
단위면적 당 필요 비료 양 $(kg/km^2)$ | $F_1$ | $F_2$ |
| Needed Pesticide per unit area $(kg/km^2)$
단위면적 당 필요 살충제 양 $(kg/km^2)$ | $P_1$ | $P_2$ |
| Selling price per unit area $(\$/km^2)$
단위면적 당 매출 $(\$/km^2)$ | $S_1$ | $S_2$ |
| Planting area $(km^2)$
재배 면적 $(km^2)$ | $x_1$ | $x_2$ |

* Under the constraints, what are the areas of wheat and rice maximizing the overall selling price?
제한조건 하에서 매출을 최대로 하는 밀과 쌀의 재배 면적?

$$
\underset{x_1, x_2}{\arg\max} \left(S_1 x_1 + S_2 x_2\right)
$$

subject to 제한조건

$$
\begin{align}
    x_1 + x_2 & \le L \\
    F_1 x_1 + F_2 x_2 & \le F \\
    P_1 x_1 + P_2 x_2 & \le P \\
    x_1, x_2 & \ge 0
\end{align}
$$

In matrix form 행렬 형태로는:

$$
\underset{x_1, x_2}{\arg\max} \begin{bmatrix} S_1 & S_2    \end{bmatrix}\begin{pmatrix} x_1 \\ x_2    \end{pmatrix}
$$

subject to 제한조건

$$
\begin{align}
    \begin{bmatrix}
        1 & 1 \\
        F_1 &  F_2 \\
        P_1 &  P_2 \\
    \end{bmatrix}
    \begin{pmatrix}
        x_1 \\
        x_2
    \end{pmatrix} & \le
    \begin{pmatrix}
        L \\
        F \\
        P
    \end{pmatrix} \\
    \begin{pmatrix}
        x_1 \\
        x_2
    \end{pmatrix}& \ge 0
\end{align}
$$

## Parameters Example
매개변수 예

```python
L = 10

```

```python
F = 10
F1 = 2
F2 = 3

```

```python
P = 5
P1 = 2
P2 = 1

```

```python
S1 = 20
S2 = 25

```

## Visualization 시각화

$$
\begin{align}
    x_1 + x_2 & \le L \\
    F_1 x_1 + F_2 x_2 & \le F \\
    P_1 x_1 + P_2 x_2 & \le P \\
    x_1, x_2 & \ge 0
\end{align}
$$

$$
\begin{align}
    x_2 & \le -x_1 + L \\
    x_2 & \le -\frac{F_1}{F_2} x_1 + \frac{F}{F_2} \\
    x_2 & \le -\frac{P_1}{P_2} x_1 + \frac{P}{P_2} \\
    x_1 & \ge 0 \\
    x_2 & \ge 0
\end{align}
$$

```python
x1 = np.linspace(0, 2.5, 101)
x2 = np.linspace(0, 5, 101)
X1, X2 = np.meshgrid(x1, x2)

C = S1 * X1 + S2 * X2

C[X2 > (-F1 * X1 + F) / F2] = np.nan
C[X2 > (-P1 * X1 + P) / P2] = np.nan

plt.pcolor(X1, X2, C, shading="auto")
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")

plt.title("$S_1 x_1 + S_2 x_2$")
plt.colorbar()

plt.grid(True)

```

## `scipy.optimize.linprog()`

```python
c_T = -np.array((S1, S2))

```

```python
A_ub = np.array(
    (
        (1, 1),
        (F1, F2),
        (P1, P2),
    )
)

```

```python
b_ub = np.array(
    ((L, F, P),)
).T

```

```python
bounds = (
    (0, None),
    (0, None),
)

```

```python
result = so.linprog(c_T, A_ub, b_ub, bounds=bounds)
result

```

```python

```


```python
from sympy import symbols, cos, sin, pi, simplify, pprint, tan, expand_trig, sqrt, trigsimp, atan2
from sympy.matrices import Matrix
```

```python
# rotation matrices in x, y, z axes

def rotx(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [1., 0., 0.],
    [0., cq,-sq],
    [0., sq, cq]
  ])

  return r

def roty(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [ cq, 0., sq],
    [ 0., 1., 0.],
    [-sq, 0., cq]
  ])

  return r

def rotz(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [cq,-sq, 0.],
    [sq, cq, 0.],
    [0., 0., 1.]
  ])

  return r
```

```python
def pose(theta, alpha, a, d):
  # returns the pose T of one joint frame i with respect to the previous joint frame (i - 1)
  # given the parameters:
  # theta: theta[i]
  # alpha: alpha[i-1]
  # a: a[i-1]
  # d: d[i]

  r11, r12 = cos(theta), -sin(theta)
  r23, r33 = -sin(alpha), cos(alpha)
  r21 = sin(theta) * cos(alpha)
  r22 = cos(theta) * cos(alpha)
  r31 = sin(theta) * sin(alpha)
  r32 = cos(theta) * sin(alpha)
  y = -d * sin(alpha)
  z = d * cos(alpha)

  T = Matrix([
    [r11, r12, 0.0, a],
    [r21, r22, r23, y],
    [r31, r32, r33, z],
    [0.0, 0.0, 0.0, 1]
  ])

  T = simplify(T)

  return T
```

```python
# get the pose (homogenous transforms) of each joint wrt to previous joint

q1, q2, q3, q4, q5, q6= symbols('q1:7')

d90 = pi / 2

T01 = pose(q1, 0, 0, 0.75)
T12 = pose(q2 - d90, -d90, 0.35, 0)
T23 = pose(q3, 0, 1.25, 0)
T34 = pose(q4, -d90, -0.054, 1.5)
T45 = pose(q5, d90, 0, 0)
T56 = pose(q6, -d90, 0, 0)
T6g = pose(0, 0, 0, 0.303)

T0g_a = simplify(T01 * T12 * T23 * T34 * T45 * T56 * T6g)
```

```python
# Total transform wrt gripper given
# yaw (alpha), pitch (beta), roll (beta)
# position px, py, pz

px, py, pz = symbols('px py pz', real = True)
alpha, beta, gamma = symbols('alpha beta gamma', real = True)

R = rotz(alpha) * roty(beta) * rotx(gamma) * (rotz(pi) * roty(-pi/2)).T

T0g_b = Matrix([
  [R[0, 0], R[0, 1], R[0, 2], px],
  [R[1, 0], R[1, 1], R[1, 2], py],
  [R[2, 0], R[2, 1], R[2, 2], pz],
  [0, 0, 0, 1]
])

T0g_b = simplify(trigsimp(T0g_b))
print(T0g_b)
```

    Matrix([
    [1.0*sin(alpha)*sin(gamma) + sin(beta)*cos(alpha)*cos(gamma),  1.0*sin(alpha)*cos(gamma) - 1.0*sin(beta)*sin(gamma)*cos(alpha), 1.0*cos(alpha)*cos(beta), px],
    [sin(alpha)*sin(beta)*cos(gamma) - 1.0*sin(gamma)*cos(alpha), -1.0*sin(alpha)*sin(beta)*sin(gamma) - 1.0*cos(alpha)*cos(gamma), 1.0*sin(alpha)*cos(beta), py],
    [                                   1.0*cos(beta)*cos(gamma),                                        -1.0*sin(gamma)*cos(beta),           -1.0*sin(beta), pz],
    [                                                          0,                                                                0,                        0,  1]])

```python
'''
px, py, pz = 0.49792, 1.3673, 2.4988
roll, pitch, yaw = 0.366, -0.078, 2.561

q1: 1.01249809363771
q2: -0.275800363737724
q3: -0.115686651053751
q4: 1.63446527240323
q5: 1.52050002599430
q6: -0.815781306199679
'''

Tb = T0g_b.evalf(subs = {
  gamma: 0.366, #roll
  beta: -0.078, #pitch
  alpha: 2.561, #yaw
  px: 0.49792,
  py: 1.3673,
  pz: 2.4988
})

print()
pprint(Tb)
print()
print(T0g_b)
```

    ⎡0.257143295038827   0.48887208255965   -0.833595473062543  0.49792⎤
    ⎢                                                                  ⎥
    ⎢0.259329420712765  0.796053601157403    0.54685182237706   1.3673 ⎥
    ⎢                                                                  ⎥
    ⎢0.93092726749696   -0.356795110642117  0.0779209320563015  2.4988 ⎥
    ⎢                                                                  ⎥
    ⎣        0                  0                   0             1.0  ⎦

    Matrix([
    [1.0*sin(alpha)*sin(gamma) + sin(beta)*cos(alpha)*cos(gamma),  1.0*sin(alpha)*cos(gamma) - 1.0*sin(beta)*sin(gamma)*cos(alpha), 1.0*cos(alpha)*cos(beta), px],
    [sin(alpha)*sin(beta)*cos(gamma) - 1.0*sin(gamma)*cos(alpha), -1.0*sin(alpha)*sin(beta)*sin(gamma) - 1.0*cos(alpha)*cos(gamma), 1.0*sin(alpha)*cos(beta), py],
    [                                   1.0*cos(beta)*cos(gamma),                                        -1.0*sin(gamma)*cos(beta),           -1.0*sin(beta), pz],
    [                                                          0,                                                                0,                        0,  1]])

```python
Ta = T0g_a.evalf(subs = {
  q1: 1.01249809363771,
  q2: -0.275800363737724,
  q3: -0.115686651053751,
  q4: 1.63446527240323,
  q5: 1.52050002599430,
  q6: -0.815781306199679
})

print()
pprint(Ta)
print()
print(T0g_a)
```

    ⎡0.257143295038831  0.488872082559654   -0.83359547306254   0.497920000000004⎤
    ⎢                                                                            ⎥
    ⎢0.259329420712762  0.796053601157401   0.546851822377065   1.36729999999999 ⎥
    ⎢                                                                            ⎥
    ⎢0.93092726749696   -0.356795110642117  0.0779209320563043  2.49880000000001 ⎥
    ⎢                                                                            ⎥
    ⎣        0                  0                   0                  1.0       ⎦

    Matrix([
    [((sin(q1)*sin(q4) + sin(q2 + q3)*cos(q1)*cos(q4))*cos(q5) + sin(q5)*cos(q1)*cos(q2 + q3))*cos(q6) - (-sin(q1)*cos(q4) + sin(q4)*sin(q2 + q3)*cos(q1))*sin(q6), -((sin(q1)*sin(q4) + sin(q2 + q3)*cos(q1)*cos(q4))*cos(q5) + sin(q5)*cos(q1)*cos(q2 + q3))*sin(q6) + (sin(q1)*cos(q4) - sin(q4)*sin(q2 + q3)*cos(q1))*cos(q6), -(sin(q1)*sin(q4) + sin(q2 + q3)*cos(q1)*cos(q4))*sin(q5) + cos(q1)*cos(q5)*cos(q2 + q3), -0.303*sin(q1)*sin(q4)*sin(q5) + 1.25*sin(q2)*cos(q1) - 0.303*sin(q5)*sin(q2 + q3)*cos(q1)*cos(q4) - 0.054*sin(q2 + q3)*cos(q1) + 0.303*cos(q1)*cos(q5)*cos(q2 + q3) + 1.5*cos(q1)*cos(q2 + q3) + 0.35*cos(q1)],
    [ ((sin(q1)*sin(q2 + q3)*cos(q4) - sin(q4)*cos(q1))*cos(q5) + sin(q1)*sin(q5)*cos(q2 + q3))*cos(q6) - (sin(q1)*sin(q4)*sin(q2 + q3) + cos(q1)*cos(q4))*sin(q6), -((sin(q1)*sin(q2 + q3)*cos(q4) - sin(q4)*cos(q1))*cos(q5) + sin(q1)*sin(q5)*cos(q2 + q3))*sin(q6) - (sin(q1)*sin(q4)*sin(q2 + q3) + cos(q1)*cos(q4))*cos(q6), -(sin(q1)*sin(q2 + q3)*cos(q4) - sin(q4)*cos(q1))*sin(q5) + sin(q1)*cos(q5)*cos(q2 + q3),  1.25*sin(q1)*sin(q2) - 0.303*sin(q1)*sin(q5)*sin(q2 + q3)*cos(q4) - 0.054*sin(q1)*sin(q2 + q3) + 0.303*sin(q1)*cos(q5)*cos(q2 + q3) + 1.5*sin(q1)*cos(q2 + q3) + 0.35*sin(q1) + 0.303*sin(q4)*sin(q5)*cos(q1)],
    [                                                                -(sin(q5)*sin(q2 + q3) - cos(q4)*cos(q5)*cos(q2 + q3))*cos(q6) - sin(q4)*sin(q6)*cos(q2 + q3),                                                                  (sin(q5)*sin(q2 + q3) - cos(q4)*cos(q5)*cos(q2 + q3))*sin(q6) - sin(q4)*cos(q6)*cos(q2 + q3),                                     -sin(q5)*cos(q4)*cos(q2 + q3) - sin(q2 + q3)*cos(q5),                                                                                -0.303*sin(q5)*cos(q4)*cos(q2 + q3) - 0.303*sin(q2 + q3)*cos(q5) - 1.5*sin(q2 + q3) + 1.25*cos(q2) - 0.054*cos(q2 + q3) + 0.75],
    [                                                                                0,                                                                                0,                                                                                0,                                                                                1]])


### Instructions

When running the notebook the first time, make sure to run all cells before making changes in the notebook. Hit Shift + Enter to run the selected cell or, in the top menu, click on: `Kernel` > `Restart Kernel and Run All Cells...` to rerun the whole notebook. If you make any changes in a cell, rerun that cell.

If you make any changes in a coding cell, rerun the notebook by `Run` > `Run Selected Cell and All Below`

```python
# Import dependencies
import sys
sys.path.append('python/')
import time
startTime = time.time()      # Calculate time for running this notebook
import numpy as np
import matplotlib.pyplot as plt
import load_galaxies as lg   # Load load_galaxies.py library
```

# Measured Data Plotting

Plotting radial velocity measurements is the first step of producing a rotation curve for a galaxy and an indication of how much mass the galaxy contains. Setting Newton's law of gravitation and the circular motion equation equal, the equation for circular velocity in terms of enclosed mass and radius can be derived:

\begin{equation}
v(r) = \sqrt{\frac{G M_{enc}(r)}{r}}
\end{equation}

>where:

        $G$ = gravitational constant

        $M_{enc}(r)$ = enclosed mass as a function of radius

        $r$ = radius or distance from the center of the galaxy

In the following activity, you will load radial velocity measurements of multiple galaxies from our Python library. The rotation curves are plotted on a single graph for comparison.

Knowing the radial velocity of stars at different radii allows you to estimate the mass enclosed for a given radius. By measuring the brightness (luminosity) of stars and the amount of gas, you can approximate the mass of "visible" matter. Compare it with the actual mass calculated from the radial velocities to get an idea of how much mass is "missing". The result is a ratio (mass-to-light ratio or M/L) that has been useful to describe the amount of dark matter in galaxies.

### Vocabulary

__Radial velocity__: the speed stars and gas are moving at different distances from the center of the galaxy

__Rotation curve__: a plot of a galaxy's radial velocity vs the radius

__NGC__: New General Catalogue of galaxies

__UGC__: Uppsala General Catalogue of galaxies

__kpc__: kiloparsec: 1 kpc = 3262 light years = 3.086e+19 meters = 1.917e+16 mile

### Load data of multiple galaxies

Load the radii, velocities, and errors in velocities of multiple galaxies from our Python library.

```python
# NGC 5533
r_NGC5533, v_NGC5533, v_err_NGC5533 = lg.NGC5533['m_radii'],lg.NGC5533['m_velocities'],lg.NGC5533['m_v_errors']

# NGC 891
r_NGC0891, v_NGC0891, v_err_NGC0891 = lg.NGC0891['m_radii'],lg.NGC0891['m_velocities'],lg.NGC0891['m_v_errors']

# NGC 7814
r_NGC7814, v_NGC7814, v_err_NGC7814 = lg.NGC7814['m_radii'],lg.NGC7814['m_velocities'],lg.NGC7814['m_v_errors']

# NGC 5005
r_NGC5005, v_NGC5005, v_err_NGC5005 = lg.NGC5005['m_radii'],lg.NGC5005['m_velocities'],lg.NGC5005['m_v_errors']

# NGC 3198
r_NGC3198, v_NGC3198, v_err_NGC3198 = lg.NGC3198['m_radii'],lg.NGC3198['m_velocities'],lg.NGC3198['m_v_errors']

# UGC 477
r_UGC477, v_UGC477, v_err_UGC477 = lg.UGC477['m_radii'],lg.UGC477['m_velocities'],lg.UGC477['m_v_errors']

# UGC 1281
r_UGC1281, v_UGC1281, v_err_UGC1281 = lg.UGC1281['m_radii'],lg.UGC1281['m_velocities'],lg.UGC1281['m_v_errors']

# UGC 1437
r_UGC1437, v_UGC1437, v_err_UGC1437 = lg.UGC1437['m_radii'],lg.UGC1437['m_velocities'],lg.UGC1437['m_v_errors']

# UGC 2953
r_UGC2953, v_UGC2953, v_err_UGC2953 = lg.UGC2953['m_radii'],lg.UGC2953['m_velocities'],lg.UGC2953['m_v_errors']

# UGC 4325
r_UGC4325, v_UGC4325, v_err_UGC4325 = lg.UGC4325['m_radii'],lg.UGC4325['m_velocities'],lg.UGC4325['m_v_errors']

# UGC 5253
r_UGC5253, v_UGC5253, v_err_UGC5253 = lg.UGC5253['m_radii'],lg.UGC5253['m_velocities'],lg.UGC5253['m_v_errors']

# UGC 6787
r_UGC6787, v_UGC6787, v_err_UGC6787 = lg.UGC6787['m_radii'],lg.UGC6787['m_velocities'],lg.UGC6787['m_v_errors']

# UGC 10075
r_UGC10075, v_UGC10075, v_err_UGC10075 = lg.UGC10075['m_radii'],lg.UGC10075['m_velocities'],lg.UGC10075['m_v_errors']
```

### Plot measured data with errorbars
Measured data points of 13 galaxies are plotted below.

Activity 1)

>In the coding cell below, change the limits of the x-axis to zoom in and out of the graph.

_Python help: change the limits of the x-axis by modifying the two numbers (left and right) of the line: `plt.xlim` then rerun the notebook or the cell._

Activity 2)

>Finding supermassive black holes: A high velocity at a radius close to zero (close to the center of the galaxy) indicates that there is a supermassive black hole present at the center of that galaxy, changing the velocities of the close-by stars. The reason the black hole does not have that much effect on the motion of stars at larger distances is because it acts as a point mass, which has negligible effect at large radii as the velocity drops off as $1 / \sqrt r$.

Can you find the galaxies with a possible central supermassive black hole and hide the curves of the rest of the galaxies?

_Python help: Turn off the display of all lines and go through them one by one. You can "turn off" the display of each galaxy by typing a `#` sign in front of the line `plt.errorbar`. This turns the line into a comment so that Python will ignore it._

_Insight: In the `04_Plotting_Rotation_Curves.ipynb` notebook, you will be able to calculate the rotation curve of the central supermassive black hole._

Activity 3)

>What do you notice about the size of the error bars at radii close and far from the center? What might be the reason for this?

```python
# Define radius for plotting
r = np.linspace(0,100,100)

# Plot
plt.figure(figsize=(14,8))                                        # size of the plot
plt.title('Measured radial velocity of multiple galaxies', fontsize=14)      # giving the plot a title
plt.xlabel('Radius (kpc)', fontsize=12)                           # labeling the x-axis
plt.ylabel('Velocity (km/s)', fontsize=12)                        # labeling the y-axis
plt.xlim(0,20)                                                    # limits of the x-axis (default from 0 to 20 kpc)
plt.ylim(0,420)                                                   # limits of the y-axis (default from 0 to 420 km/s)

# Plotting the measured data
plt.errorbar(r_NGC5533,v_NGC5533,yerr=v_err_NGC5533,    label='NGC 5533',  marker='o', markersize=6, linestyle='none', color='royalblue')
plt.errorbar(r_NGC0891,v_NGC0891,yerr=v_err_NGC0891,    label='NGC 891',   marker='o', markersize=6, linestyle='none', color='seagreen')
plt.errorbar(r_NGC7814,v_NGC7814,yerr=v_err_NGC7814,    label='NGC 7814',  marker='o', markersize=6, linestyle='none', color='m')
plt.errorbar(r_NGC5005,v_NGC5005,yerr=v_err_NGC5005,    label='NGC 5005',  marker='o', markersize=6, linestyle='none', color='red')
plt.errorbar(r_NGC3198,v_NGC3198,yerr=v_err_NGC3198,    label='NGC 3198',  marker='o', markersize=6, linestyle='none', color='gold')
plt.errorbar(r_UGC477,v_UGC477,yerr=v_err_UGC477,       label='UGC 477',   marker='o', markersize=6, linestyle='none', color='lightpink')
plt.errorbar(r_UGC1281,v_UGC1281,yerr=v_err_UGC1281,    label='UGC 1281',  marker='o', markersize=6, linestyle='none', color='aquamarine')
plt.errorbar(r_UGC1437,v_UGC1437,yerr=v_err_UGC1437,    label='UGC 1437',  marker='o', markersize=6, linestyle='none', color='peru')
plt.errorbar(r_UGC2953,v_UGC2953,yerr=v_err_UGC2953,    label='UGC 2953',  marker='o', markersize=6, linestyle='none', color='lightslategrey')
plt.errorbar(r_UGC4325,v_UGC4325,yerr=v_err_UGC4325,    label='UGC 4325',  marker='o', markersize=6, linestyle='none', color='darkorange')
plt.errorbar(r_UGC5253,v_UGC5253,yerr=v_err_UGC5253,    label='UGC 5253',  marker='o', markersize=6, linestyle='none', color='maroon')
plt.errorbar(r_UGC6787,v_UGC6787,yerr=v_err_UGC6787,    label='UGC 6787',  marker='o', markersize=6, linestyle='none', color='midnightblue')
plt.errorbar(r_UGC10075,v_UGC10075,yerr=v_err_UGC10075, label='UGC 10075', marker='o', markersize=6, linestyle='none', color='y')

plt.legend(bbox_to_anchor=(1,1), loc="upper left")
plt.show()
```

```python
#NBVAL_IGNORE_OUTPUT
#Because the timing won't be exactly the same each time.
# Time
executionTime = (time.time() - startTime)
ttt=executionTime/60
print(f'Execution time: {ttt:.2f} minutes')
```

    Execution time: 0.06 minutes

#### References

>De Naray, Rachel Kuzio, Stacy S. McGaugh, W. J. G. De Blok, and A. Bosma. __"High-resolution optical velocity fields of 11 low surface brightness galaxies."__ The Astrophysical Journal Supplement Series 165, no. 2 (2006): 461. https://doi.org/10.1086/505345.

>De Naray, Rachel Kuzio, Stacy S. McGaugh, and W. J. G. De Blok. __"Mass models for low surface brightness galaxies with high-resolution optical velocity fields."__ The Astrophysical Journal 676, no. 2 (2008): 920. https://doi.org/10.1086/527543.

>Epinat, B., P. Amram, M. Marcelin, C. Balkowski, O. Daigle, O. Hernandez, L. Chemin, C. Carignan, J.-L. Gach, and P. Balard. __“GHASP: An Hα Kinematic Survey of Spiral and Irregular GALAXIES – Vi. New HΑ Data Cubes for 108 Galaxies.”__ Monthly Notices of the Royal Astronomical Society 388, no. 2 (July 19, 2008): 500–550. https://doi.org/10.1111/j.1365-2966.2008.13422.x.

>Fraternali, F., R. Sancisi, and P. Kamphuis. __“A Tale of Two Galaxies: Light and Mass in NGC 891 and NGC 7814.”__ Astronomy & Astrophysics 531 (June 13, 2011). https://doi.org/10.1051/0004-6361/201116634.

>Karukes, E. V., P. Salucci, and Gianfranco Gentile. __"The dark matter distribution in the spiral NGC 3198 out to 0.22 $R_{vir}$."__ _Astronomy & Astrophysics_ 578 (2015): A13. https://doi.org/10.1051/0004-6361/201425339.

>Lelli, F., McGaugh, S. S., & Schombert, J. M. (2016). **SPARC: Mass models for 175 disk galaxies with Spitzer photometry and accurate rotation curves.** _The Astronomical Journal_, 152(6), 157. https://doi.org/10.3847/0004-6256/152/6/157

>Noordermeer, Edo. __"The rotation curves of flattened Sérsic bulges."__ _Monthly Notices of the Royal Astronomical Society_ 385, no. 3 (2007): 1359-1364. https://doi.org/10.1111/j.1365-2966.2008.12837.x

>Richards, Emily E., L. van Zee, K. L. Barnes, S. Staudaher, D. A. Dale, T. T. Braun, D. C. Wavle, et al. __“Baryonic Distributions in the Dark Matter Halo of NGC 5005.”__ Monthly Notices of the Royal Astronomical Society 449, no. 4 (June 1, 2015): 3981–96. https://doi.org/10.1093/mnras/stv568.
***


# Primal-Dual algorithm for Total Variation processing on graphs.
- **TIP**: *It is recommended to understand the code in the Tikhonov_reg notebook before understanding this notebook*.
- We shall be doing  pointcloud shape{x,y,z} processing using Total Variation on Graphs.
- To create the graph, see *Graph_construction* notebook.
- We shall be using Chambolle-Pock algorithm(2010).
- Consider the following functional to be minimized:
\begin{align*}
&f^{*} = \underset{f}{\arg\min} \  \tfrac{1}{2}\|f-f_0\|^2 +  \lambda \|\nabla_\omega f\|_{1,p} \\
&\text{where} \ \ \|\nabla_\omega f\|_{1,p} = \sum_{i \in V}\|\nabla_\omega f_i\|_{p}\\
&\text{and}\quad p\in\{1,2\}
\end{align*}
- For p = 1, this becomes anisotropic processing and for p = 2, this becomes isotropic processing.
- The above functional can be casted as the following saddle point problem :

$$ \min_{f}\max_{g}\langle Kf,g \rangle + B(f) - A^{*}(g)$$
- Here $g$ is the dual variable, $A \equiv \|.\|_{1,p}$, $B \equiv \tfrac{1}{2\lambda}\|. - f_0\|^{2}_{2}$ and $K\equiv \nabla_{\omega}$.
- For pointclouds $f \in \mathbb{R}^{|V|x3}$ and $g \in \mathbb{R}^{|V|x|V|x3}$. Here $|V|$ is the number of nodes in a graph *i.e.* the number of points in a pointcloud and 3 comes from 3 shape channels {x,y,z}.
- The following algorithm shall be implemented :
\begin{align*}
g^{(t+1)} &= \mathcal{P}_{\beta A^\ast }(g^{(t)}+\beta K \bar{f}^{(t)})\\
f^{(t+1)} &= \mathcal{P}_{\tau B}(f^{(t)}-\tau K^\ast g^{(t+1)})\\
\bar{f}^{(t+1)} &= f^{(t+1)}+ \theta(f^{(t+1)}-f^{(t)})
\end{align*}
- The proximal operator $\mathcal{P}_{\tau B}(f)$ is given as :
$$ \mathcal{P}_{\tau B}(f) = \tfrac{\lambda f + \tau f_0}{\lambda + \tau} $$
- The proximal operator $\mathcal{P}_{\beta A^{*}}(g)$ depends upon the value of $p$ in the norm. We shall find it, as we proceed in this notebook.
- The initialization shceme is $f^{(0)} = \bar{f}^{(0)} = f_0$ and $g^{(0)} = \nabla_\omega f_0$.
- Set $\tau, \beta >0$ such that $ \tau \beta \|{K}\|^2<1 $  and $\  \theta \in [0,1]$
- A common choice is to set $\tau = \beta = (4 \max_{i \in V}\sum_{j\in N(i)}\omega_{i,j})^{-1}$ and $\theta = 1$.

```python
import torch
from tqdm import tqdm
from torch_scatter import scatter
from utilities import *
dev = "cuda" if torch.cuda.is_available() else "cpu"
```

```python
# load the graph
graph = torch.load("./data/3d_signal.pt")
graph = graph.to(dev)
```

```python
#visualize the NOISY ponitcloud if required: #Open3d visualization works only locally!
displaySur(**dict(position=graph.x.cpu().numpy(), texture=graph.tex.cpu().numpy()))
```

# Implement a base Primal-Dual class
We shall be implementing the $\mathcal{P}_{\tau B}(f)$ and iterative scheme, but we will leave the $\mathcal{P}_{\beta A^{*}}(g)$ for now.

```python
class PD:
    """
    class to implement primal dual algo.
    """
    def __init__(self, graph, **kwargs):
        self.edge_index = graph.edge_index
        self.edge_attr = graph.edge_attr
        self.dim_size = kwargs["dim_size"]
        self.f0 = kwargs["noisy_sig"]
        self.lamb = kwargs["lamb"]
        self.tau = kwargs["tau"]
        self.beta = kwargs["beta"]
        self.theta = kwargs["theta"]
        self.dev = kwargs["dev"]

    def grad(self, Xb):
        return torch.sqrt(self.edge_attr) * (Xb[self.edge_index[0]] - Xb[self.edge_index[1]])

    def proj(self,T):
        pass

    def prox(self,U):
        return (self.lamb * U + self.tau * self.f0)/(self.lamb + self.tau)

    def grad_conj(self, P): # grad_conj = -divergence
        out1 = scatter(torch.sqrt(self.edge_attr) * P, self.edge_index[1], dim=0, dim_size=self.dim_size, reduce="add")
        out2= scatter(torch.sqrt(self.edge_attr) * P, self.edge_index[0], dim=0, dim_size=self.dim_size, reduce="add")
        return  (out2 - out1)

    def run(self,itr, X):
        Xb = X
        P = self.grad(X)
        for i in tqdm(range(itr)):
            P = self.proj(P + self.beta* self.grad(Xb))
            tmp_X = X.clone()
            X = self.prox(X - self.tau * self.grad_conj(P))
            Xb = X + self.theta*(X-tmp_X)
        return X
```

# Case-1, anisotropic p = 1
The proximal operator $\mathcal{P}_{\beta A^{*}}(g)$ in this case is given as:
$$ \mathcal{P}_{\beta A^{*}}(g)_{i,j,k} = \mathcal{Proj}_{\mathcal{B}_{\infty,\infty}}(g)_{i,j,k} = \frac{g_{i,j,k}}{\max\{1,|g_{i,j,k}|\}}$$
Recall that $g \in \mathbb{R}^{|V|x|V|x3}$

```python
class PD_TV1(PD):
    def proj(self,T):
        return (T/torch.clamp(torch.abs(T),1))
```

```python
lamb = 0.005
itr = 5000
theta = 1
# Calculates the \max_{i \in V}\sum_{j\in N(i)}\omega_{i,j}
k_prime = torch.max(scatter(graph.edge_attr, graph.edge_index[1],dim=0, dim_size=graph.x.shape[0], reduce="add")).item()
beta = 1/(2*k_prime)
tau = 1/(8*k_prime)
pd_tv1 = PD_TV1(graph, **dict(dim_size = graph.x.shape[0], noisy_sig=graph.x, tau=tau, lamb=lamb, beta = beta, theta=theta, dev=dev))
new_sig_tv1 = pd_tv1.run(itr,graph.x)
```

    100%|██████████| 5000/5000 [00:01<00:00, 2994.38it/s]

```python
displaySur(**dict(position=new_sig_tv1.cpu().numpy(), texture=graph.tex.cpu().detach())) # displays the processed pointcld.

```

# Case-2, isotropic p=2
The proximal operator $\mathcal{P}_{\beta A^{*}}(g)$ in this case is given as:
$$ \mathcal{P}_{\beta A^{*}}(g)_{i,j,k} = \mathcal{Proj}_{\mathcal{B}_{\infty,2}}(g)_{i,j,k} = \frac{g_{i,j,k}}{\max\{1,\|g_{i,.}\|_{2}\}}$$
Recall that $g \in \mathbb{R}^{|V|x|V|x3}$, here $\|g_{i,.}\|_{2}$ is the $L_2$ norm corresonding to $i^{th}$ row tensor of size $|V|x3$

```python
class PD_TV2(PD):
    def proj(self,T):
        aggr_out = scatter(torch.pow(T,2), self.edge_index[1], dim=0, dim_size=self.dim_size, reduce="add")
        a = torch.sqrt(aggr_out)
        b = torch.norm(a,dim=1)
        b = b.view(-1,1)
        return (T/torch.clamp(b[self.edge_index[1]],1))
```

```python
lamb = 0.05
itr = 5000
theta = 1
# Calculates the \max_{i \in V}\sum_{j\in N(i)}\omega_{i,j}
k_prime = torch.max(scatter(graph.edge_attr, graph.edge_index[1],dim=0, dim_size=graph.x.shape[0], reduce="add")).item()
beta = 1/(2*k_prime)
tau = 1/(8*k_prime)
pd_tv2 = PD_TV2(graph, **dict(dim_size = graph.x.shape[0], noisy_sig=graph.x, tau=tau, lamb=lamb, beta = beta, theta=theta, dev=dev))
new_sig_tv2 = pd_tv2.run(itr,graph.x)
```

    100%|██████████| 5000/5000 [00:02<00:00, 2249.51it/s]

```python
displaySur(**dict(position=new_sig_tv2.cpu().cpu().numpy(), texture=graph.tex.cpu().numpy())) # displays the processed pointcld.

```

# Use a non-squared Data-Fidelity term:
- The following minization is known to preserve the contrast/volume for color/shape processing:
\begin{equation}
 f^{*} = \underset{f}{\arg\min} \ \|f-f_0\|_{1} +  \lambda \|\nabla_\omega f\|_{1,p} \\
\end{equation}
- The inspiration comes from the 2010 paper of chambolle-pock

- However, we are interested in minimizing the non-squared $L2$ norm in Data-Fidelity term:

\begin{equation}
 f^{*} = \underset{f}{\arg\min} \ \|f-f_0\|_{2} +  \lambda \|\nabla_\omega f\|_{1,p = 1} \\
\end{equation}

- With a little bit of effort it can be shown that the proximal operator $\mathcal{P}_{\tau B}(f)$, where $B \equiv \tfrac{1}{\lambda}\|. - f_0\|_{2}$ is given as:
$$f_{0,i} + \left( 1 - \frac{\frac{\tau}{\lambda}}{\max({\frac{\tau}{\lambda}, \|f_i - f_{0,i}\|_{2}})}  \right)(f_i - f_{0,i})$$

```python
class PD_L2DF(PD):
    def prox(self,U):
        return self.f0 + (1 - ((self.tau/self.lamb)/ torch.where( torch.norm(U-self.f0, dim=1).view(-1,1) >= torch.tensor(self.tau/self.lamb).type(torch.float32).to(self.dev), torch.norm(U-self.f0, dim=1).view(-1,1) , torch.tensor(self.tau/self.lamb).type(torch.float32).to(self.dev)))) * (U-self.f0)
    def proj(self,T):
        return (T/torch.clamp(torch.abs(T),1))
```

```python
lamb = 0.15
itr = 5000
theta = 1
# Calculates the \max_{i \in V}\sum_{j\in N(i)}\omega_{i,j}
k_prime = torch.max(scatter(graph.edge_attr, graph.edge_index[1],dim=0, dim_size=graph.x.shape[0], reduce="add")).item()
beta = 1/(2*k_prime)
tau = 1/(8*k_prime)
pd_l2df = PD_L2DF(graph, **dict(dim_size = graph.x.shape[0], noisy_sig=graph.x, tau=tau, lamb=lamb, beta = beta, theta=theta, dev=dev))
new_sig_l2df = pd_l2df.run(itr,graph.x)
```

    100%|██████████| 5000/5000 [00:02<00:00, 1907.74it/s]

```python
displaySur(**dict(position=new_sig_l2df.cpu().numpy(), texture=graph.tex.cpu().numpy())) # displays the processed pointcld.

```

```python
# You should see something like this, in the following order:
#Original poincloud
#Case-1 (aniso TV)
#Case-2 (iso TV)
#Non-squared L2 DF term
from matplotlib import image as mpimg
from matplotlib import pyplot as plt
res = mpimg.imread("./data/out_shape.png")
f = plt.figure(figsize=(12,12))
plt.imshow(res, cmap="gray")
plt.show()
```

# Exercise

- Modify the `def prox()` method to solve for the non-squared L1 data-fidelity term in the following functional:
\begin{equation*}
 f^{*} = \underset{f}{\arg\min} \ \|f-f_0\|_{1} +  \lambda \|\nabla_\omega f\|_{1,p = 1} \\
\end{equation*}