# Farm Planning

## Objective and Prerequisites

This model is an example of a production planning problem. In such  problems — which are common across a wide range of industries — choices must be made regarding which products to produce and which resources to use to produce these products.

In this example, we will model and solve a multi-period production planning problem. In this case, the purpose of the application is to optimize the operation of a farm over five years. The farmer has a certain number of farmable acres and cows. The cows can be sold or used to produce milk, which can also be sold for profit. Food for the cows is grown on the farmable acres. The aim is to create an optimal operational plan for the next five years to maximize the total profit of the farm.
This includes determining optimal strategies for cultivation, for managing the cows, and for the overall operation of the farm

More information on this type of model can be found in example #8 of the fifth edition of Model Building in Mathematical Programming, by H. Paul Williams on pages 262-263 and 312-315.

This modeling example is at the advanced level, where we assume that you know Python and the Gurobi Python API and that you have advanced knowledge of building mathematical optimization models. Typically, the objective function and/or constraints of these examples are complex or require advanced features of the Gurobi Python API.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Problem Description

A farmer with a 200-acre farm wishes to create a five-year production plan.

Currently, he has a herd of 120 cows comprising 20 heifers (young female cows) and 100 adult dairy cows. Each heifer requires 2/3 of an acre to support it and each dairy cow requires one acre.

On average, a dairy cow produces 1.1 calves per year. Half of these calves will be bullocks that are sold shortly after birth for an average of $\$30$ each. The remaining calves, heifers, can either be sold for $\$40$ or raised until age two when they will become dairy cows. For the current year, all heifers identified for sale have already been sold.

The general practice is to sell all dairy cows at the age of 12 for an average of $\$120$ each. However, each year an average of $5\%$ of heifers and $2\%$ of dairy cows die. At present, the farmer’s herd of 120 cows is evenly distributed with 10 cows per age from newborn to 11 years old.

The milk from a dairy cow can be sold for $\$370$ annually. The farmer can currently house up to 130 cows, but this capacity limit can be increased for $\$200$ per additional cow.

Each dairy cow requires 0.6 tons of grain and 0.7 tons of sugar beet per year. Both of these can be grown on the farm. Each acre can yield 1.5 tons of sugar beet. However, only 80 acres are suitable for growing grain, and those acres have different levels of productivity as follows:

| Land Group | Area (Acres) | Grain Production (Tons/Acre) |
| --- | --- | --- |
| Group 1 | 20 | 1.10 |
| Group 2 | 30 | 0.90 |
| Group 3 | 20 | 0.80 |
| Group 4 | 10 | 0.65 |

Sugar beet can be bought for $\$70$ a ton and sold for $\$58$ a ton. Grain can be bought for $\$90$ a ton and sold for $\$75$ a ton.

The annual labor requirements for cows as well as grain and sugar beet production are as follows:

|  | Labor Required (Hr/Year) |
| --- | --- |
| Heifer | 10 |
| Milk-producing Cow | 42 |
| Acre Devoted to Grain | 4 |
| Acre Devoted to Sugar Beet| 14 |

Other annual costs are as follows:

|  | Cost (USD/Year) |
| --- | --- |
| Heifer | 50 |
| Milk-producing Cow | 100 |
| Acre Devoted to Grain | 15 |
| Acre Devoted to Sugar Beet| 10 |

Labor currently costs the farmer $\$4000$ per year and that cost provides 5,500 hours of labor. Additional labor can be paid for at the rate of $\$1.20$ per hour.

Any capital expenditure can be financed with a 10-year loan at $15\%$ interest annually. The interest and principal are paid back in 10 equal annual payments. It is prohibited for the annual cash flow to be negative.

The farmer does not want to reduce the total number of dairy cows at the end of the five-year period by more than $50\%$, nor does he want to increase their number by more than $75\%$.

What plan should the farmer follow over the next five years in order to maximize profit?

---
## Model Formulation

### Sets and Indices

$t \in \text{Years}=\{1,2,\dots,5\}$: Set of years.

$l \in \text{Lands}=\{1,2,3,4\}$: Set of land groups.

$k \in \text{Ages}=\{1,2,\dots,12\}$: Set of cow ages.

### Parameters

$\text{Installment} \in \mathbb{R}^+$: Annual payment for each $\$200$ loan. It can be computed as follows.

$$
\text{Installment}= \text{Loan}*r*\frac{(1+r)^n}{(1+r)^n-1},
$$

where $\text{Loan}$ is the loan amount of $\$200$, $r$ is the interest rate of $15\%$, and $n$ represents the ten-year period. Note that the book reports a value of $\$39.71$ instead of $\$39.85$.

$\text{Housing_cap} \in \mathbb{N}$: Number of cows that can be housed.

$\text{Land_cap} \in \mathbb{R}^+$: Land available (in Acres).

$\text{Labor_cap} \in \mathbb{R}^+$: Regular labor available (in hours) in a year.

$\text{GR_intake} \in \mathbb{R}^+$: Tons of grain consumed by a dairy cow in a year.

$\text{SB_intake} \in \mathbb{R}^+$: Tons of sugar beet consumed by a dairy cow in a year.

$\text{HF_land} \in \mathbb{R}^+$: Acres required for sustaining each heifer.

$\text{HF_labor} \in \mathbb{R}^+$: Hours of labor required in a year by each heifer.

$\text{Cow_labor} \in \mathbb{R}^+$: Hours of labor required in a year by each dairy cow.

$\text{GR_labor} \in \mathbb{R}^+$: Hours of labor required in a year by each acre of land devoted to grains.

$\text{SB_labor} \in \mathbb{R}^+$: Hours of labor required in a year by each acre of land devoted to sugar beet.

$\text{HF_decay} \in [0,1] \subset \mathbb{R}^+$: Average proportion of heifers that die in a year.

$\text{Cow_decay} \in [0,1] \subset \mathbb{R}^+$: Average proportion of dairy cows that die in a year.

$\text{Birthrate} \in \mathbb{R}^+$: Expected number of calves produced by a dairy cow in a year.

$\text{Min_final_cows} \in \mathbb{N}$: Minimum number of dairy cows at the end of the planning horizon.

$\text{Max_final_cows} \in \mathbb{N}$: Maximum number of dairy cows at the end of the planning horizon.

$\text{Initial_HF} \in \mathbb{R}^+$: Number of heifers of each age at the beginning of the planning horizon.

$\text{Initial_cows} \in \mathbb{R}^+$: Number of dairy cows of each age at the beginning of the planning horizon.

$\text{BL_price} \in \mathbb{R}^+$: Price for selling one bullock.

$\text{HF_price} \in \mathbb{R}^+$: Price for selling one heifer.

$\text{Cow_price} \in \mathbb{R}^+$: Price for selling one dairy cow.

$\text{Milk_price} \in \mathbb{R}^+$: Price for selling the milk produced by a dairy cow in a year.

$\text{GR_price} \in \mathbb{R}^+$: Price for selling a ton of grain.

$\text{SB_price} \in \mathbb{R}^+$: Price for selling a ton of sugar beet.

$\text{GR_cost} \in \mathbb{R}^+$: Cost for purchasing a ton of grain.

$\text{SB_cost} \in \mathbb{R}^+$: Cost for purchasing a ton of sugar beet.

$\text{Overtime_cost} \in \mathbb{R}^+$: Cost for getting an hour of overtime.

$\text{Regular_time_cost} \in \mathbb{R}^+$: Cost for having 5,500 hours of labor in regular time.

$\text{HF_cost} \in \mathbb{R}^+$: Yearly cost for supporting a heifer.

$\text{Cow_cost} \in \mathbb{R}^+$: Yearly cost for supporting a dairy cow.

$\text{GR_land_cost} \in \mathbb{R}^+$: Yearly cost for supporting an acre of land devoted to grain.

$\text{SB_land_cost} \in \mathbb{R}^+$: Yearly cost for supporting an acre of land devoted to sugar beet.

$\text{SB_yield} \in \mathbb{R}^+$: Sugar beet yield.

$\text{GR_yield}_l \in \mathbb{R}^+$: Grain yield at land group $l$.

$\text{GR_area}_l \in \mathbb{R}^+$: Number of acres suitable for growing grain in land group $l$.

### Decision Variables

$\text{Outlay}_t \in \mathbb{R}^+$: Amount of money (in units of $\$200$) spent on renting houses in year $t$.

$\text{Overtime}_t \in \mathbb{R}^+$: Number of extra labor hours needed in year $t$.

$\text{Newborn}_t \in \mathbb{R}^+$: Number of newborn heifers left in year $t$ to be raised.

$\text{HF_sell}_t \in \mathbb{R}^+$: Number of newborn heifers to sell in year $t$.

$\text{Profit}_t \in \mathbb{R}^+$: Profit attained in year $t$.

$\text{SB_buy}_t \in \mathbb{R}^+$: Number of tons of sugar beet to buy in year $t$.

$\text{SB_sell}_t \in \mathbb{R}^+$: Number of tons of sugar beet to sell in year $t$.

$\text{GR_buy}_t \in \mathbb{R}^+$: Number of tons of grain to buy in year $t$.

$\text{GR_sell}_t \in \mathbb{R}^+$: Number of tons of grain to sell in year $t$.

$\text{SB}_t \in \mathbb{R}^+$: Number of tons of sugar beet to grow in year $t$.

$\text{GR}_{t,l} \in \mathbb{R}^+$: Number of tons of grain to grow in year $t$ at land group $l$.

$\text{Cows}_{t,k} \in \mathbb{R}^+$: Number of cows of age $k$ available in year $t$.

### Objective Function

- **Profit**: Maximize the total profit (in USD) of the planning horizon. Notice that, to make capital expenditure as costly in latter years as in former ones, it is necessary to subtract pending loan payments.

\begin{equation}
\text{Maximize} \quad Z = \sum_{t \in \text{Years}}{\text{Profit}_t - installment*(t+4)*\text{Outlay}_t}
\end{equation}

### Constraints

- **Housing Capacity**: Livestock cannot exceed the installed capacity plus house rental in year $t$.

\begin{equation}
\text{Newborn}_t + \sum_{k \in \text{Ages} \setminus \{12\}}{\text{Cows}_{t,k}} \leq \text{Housing_cap} + \sum_{d \in \text{Years}: d \leq t}{\text{Outlay}_d} \quad \forall t \in \text{Years}
\end{equation}

- **Food Consumption**: There must be enough food to feed the livestock in year $t$.

- Grain.

\begin{equation}
\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{GR_intake}*\text{Cows}_{t,k}} \leq \text{GR_buy}_t - \text{GR_sell}_t + \sum_{l \in \text{Lands}}{\text{GR}_{t,l}} \quad \forall t \in \text{Years}
\end{equation}

- Sugar beet.

\begin{equation}
\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{SB_intake}*\text{Cows}_{t,k}} \leq \text{SB_buy}_t - \text{SB_sell}_t + \text{SB}_t \quad \forall t \in \text{Years}
\end{equation}

- **Grain Growing**: Grain produced on land $l$ cannot exceed the production capacity of year $t$.

\begin{equation}
\text{GR}_{t,l} \leq \text{GR_yield}_l*\text{GR_area}_l \quad \forall (t,l) \in \text{Years} \times \text{Lands}
\end{equation}

- **Land Capacity**: Use of space in year $t$ cannot exceed available land.

\begin{equation}
\frac{1}{\text{SB_yield}}*\text{SB}_t + \text{HF_land}*(\text{Newborn}_t + \text{Cow}_{t,1}) +
\end{equation}

\begin{equation}
\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cows}_{t,k}} +
\end{equation}

\begin{equation}
\sum_{l \in \text{Lands}}{\frac{\text{GR}_{t,l}}{\text{GR_yield}_l}} \leq \text{Land_cap} \quad \forall t \in \text{Years}
\end{equation}

- **Labor**: Labor required to run the farm in year $t$ cannot exceed contracted time plus overtime.

\begin{equation}
\text{HF_labor}*(\text{Newborn}_t + \text{Cow}_{t,1}) +
\end{equation}

\begin{equation}
\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cow_labor}*\text{Cow}_{t,k}} +
\end{equation}

\begin{equation}
\sum_{l \in \text{Lands}}{\frac{\text{GR_labor}*\text{GR}_{t,l}}{\text{GR_yield}_l}} +
\end{equation}

\begin{equation}
\frac{\text{SB_labor}*\text{SB}_t}{\text{SB_yield}} \leq \text{Labor_cap} + \text{Overtime}_t \qquad \forall t \in \text{Years}
\end{equation}

- **Continuity**: Livestock in year $t$ have to survive the previous year.

\begin{equation}
\text{Cows}_{t,1} = (1-\text{HF_decay})*\text{Newborn}_{t-1} \quad \forall t \in \text{Years} \setminus \{1\}
\end{equation}

\begin{equation}
\text{Cows}_{t,2} = (1-\text{HF_decay})*\text{Cows}_{t-1,1} \quad \forall t \in \text{Years} \setminus \{1\}
\end{equation}

\begin{equation}
\text{Cows}_{t,k+1} = (1-\text{Cow_decay})*\text{Cows}_{t-1,k} \quad \forall (t,k) \in \text{Years} \setminus \{1\} \times \text{Ages} \setminus \{1,12\}
\end{equation}

- **Heifers Birth**: Heifers born in year $t$ depend on the number of dairy cows.

\begin{equation}
\text{Newborn}_t + \text{HF_sell}_t = \sum_{k \in \text{Ages} \setminus \{1,12\}}{\frac{\text{Birthrate}}{2}*\text{Cows}_{t,k}} \quad \forall t \in \text{Years}
\end{equation}

- **Final Dairy Cows**: The number of dairy cows at the end of the planning horizon must be within tolerances.

\begin{equation}
\text{Min_final_cows} \leq \sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cows}_{5,k}} \leq \text{Max_final_cows}
\end{equation}

- **Initial Conditions**: Set the number of livestock available at the beginning of the planning horizon.

\begin{equation}
\text{Cows}_{1,1} = \text{Initial_HF}
\end{equation}

\begin{equation}
\text{Cows}_{1,2} = \text{Initial_HF}
\end{equation}

\begin{equation}
\text{Cows}_{1,k} = \text{Initial_cows} \quad \forall k \in \text{Ages} \setminus \{1,2\}
\end{equation}

- **Yearly Profit**: Profit in year $t$ is driven by operations from crops and livestock, after accounting for labor, land and financial costs.

\begin{equation}
\text{Profit}_t = \frac{\text{BL_price}*\text{Birthrate}}{2}*\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cows}_{t,k}}
+ \text{HF_price}*\text{HF_sell}_t
+ \text{Cow_price}*\text{Cows}_{t,12}
\end{equation}

\begin{equation}
+ \text{Milk_price}*\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cows}_{t,k}}
\end{equation}

\begin{equation}
+ \text{GR_price}*\text{GR_sell}_t
+ \text{SB_price}*\text{SB_sell}_t
\end{equation}

\begin{equation}
- \text{GR_cost}*\text{GR_buy}_t
- \text{SB_cost}*\text{SB_buy}_t
\end{equation}

\begin{equation}
- \text{Overtime_cost}*\text{Overtime}_t
- \text{Regular_time_cost}
\end{equation}

\begin{equation}
- \text{HF_cost}*(\text{Newborn}_t + \text{Cows}_{t,1})
- \text{Cow_cost}*\sum_{k \in \text{Ages} \setminus \{1,12\}}{\text{Cows}_{t,k}}
\end{equation}

\begin{equation}
- \text{GR_land_cost}*\sum_{l \in \text{Lands}}{\frac{\text{GR}_{t,l}}{\text{GR_yield}_l}}
- \text{SB_land_cost}*\frac{\text{SB}_t}{\text{SB_yield}} \\
- \text{Installment}*\sum_{d \in \text{Years}:d \leq t}{\text{Outlay}_d}
\quad \forall t \in \text{Years}
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
import numpy as np
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input Data
We define all the input data of the model.

```python
# Parameters

years = [1,2,3,4,5]
lands = [1,2,3,4]
ages = [1,2,3,4,5,6,7,8,9,10,11,12]
cow_ages = [2,3,4,5,6,7,8,9,10,11]

gr_area = {1: 20.0, 2: 30.0, 3: 20.0, 4: 10.0}
gr_yield = {1: 1.1,  2: 0.9, 3: 0.8, 4:0.65}
sb_yield = 1.5
housing_cap = 130
gr_intake = 0.6
sb_intake = 0.7
hf_land = 2/3.0
land_cap = 200
hf_labor = 10/100.0
cow_labor = 42/100.0
gr_labor = 4/100.0
sb_labor = 14/100.0
labor_cap = 5500/100.0
cow_decay = 0.02
hf_decay = 0.05
initial_hf = 9.5
initial_cows = 9.8
birthrate = 1.1
min_final_cows = 50
max_final_cows = 175
bl_price = 30
hf_price = 40
cow_price = 120
milk_price = 370
gr_price = 75
sb_price = 58
gr_cost = 90
sb_cost = 70
overtime_cost = 120
regular_time_cost = 4000
hf_cost = 50
cow_cost = 100
gr_land_cost = 15
sb_land_cost = 10
installment = 39.71
```

## Model Deployment

We create a model and the variables. For each, year we have continuous variables, which tell us how much sugar beet is grown (in tons), how much grain is bought (in tons), how much grain is sold (in tons), how much sugar beet is bought (in tons), how much sugar beet is sold (in tons), how much extra labor is recruited, how much capital outlay there is, how many heifers are sold at birth, how much profit there is, and how many cows are newborns (age 0).

For each year and each land group, there is a continuous variable, which tells us how much grain is grown on that land group. For each year and each cow age, there is a continuous variable that tells us how many cows exists in the current year of that age.

```python
model = gp.Model('Farming')
sb = model.addVars(years, vtype=GRB.CONTINUOUS, name="SB")
gr_buy = model.addVars(years, vtype=GRB.CONTINUOUS, name="GR_buy")
gr_sell = model.addVars(years, vtype=GRB.CONTINUOUS, name="GR_sell")
sb_buy = model.addVars(years, vtype=GRB.CONTINUOUS, name="SB_buy")
sb_sell = model.addVars(years, vtype=GRB.CONTINUOUS, name="SB_sell")
overtime = model.addVars(years, vtype=GRB.CONTINUOUS, name="Overtime")
outlay = model.addVars(years, vtype=GRB.CONTINUOUS, name="Outlay")
hf_sell = model.addVars(years, vtype=GRB.CONTINUOUS, name="HF_sell")
newborn = model.addVars(years, vtype=GRB.CONTINUOUS, name="Newborn")
profit = model.addVars(years, vtype=GRB.CONTINUOUS, name="Profit")
gr = model.addVars(years, lands, vtype=GRB.CONTINUOUS, name="GR")
cows = model.addVars(years, ages, vtype=GRB.CONTINUOUS, name="Cows")
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-

Next we insert the constraints:

There is only a housing capacity for 130 cows per year. It is possible that there are more than 130 cows, but housing these additional cows will require additional costs related to renting houses, which is covered with the `Outlay` variables.

```python
# 1. Housing capacity

HousingCap = model.addConstrs((newborn[year] +
                    cows[year,1] +
                    gp.quicksum(cows[year,age] for age in cow_ages) -
                    gp.quicksum(outlay[d] for d in years if d <= year)
                    <= housing_cap for year in years), name="Housing_cap")
```

After selling, buying, and producing grains and sugar beet, there needs to be enough of them in storage to feed all the cows.

```python
# 2.1 Food consumption (Grain)

GrainConsumption = model.addConstrs((gp.quicksum(gr_intake*cows[year, age] for age in cow_ages)
                  <= gr_buy[year] - gr_sell[year] + gr.sum(year, '*')
                  for year in years), name="Grain_consumption")

# 2.1 Food consumption (Sugar beet)
SugarbeetConsumption = model.addConstrs((gp.quicksum(sb_intake*cows[year, age] for age in cow_ages)
                  <= sb_buy[year] - sb_sell[year] + sb[year]
                  for year in years), name="Sugar_beet_consumption")
```

The grain produced on a land group cannot exceed the specified production capacity of that land group.

```python
# 3. Grain growing

GrainGrowing = model.addConstrs((gr[year, land] <= gr_yield[land]*gr_area[land]
                  for year in years for land in lands), name="Grain_growing")
```

Each cow needs a certain number of acres to support it. The amount depends on the age of the cow. There are 200 acres available at most.

```python
# 4. Land capacity

LandCap = model.addConstrs((sb[year]/sb_yield + hf_land*(newborn[year] + cows[year,1])
                  + gp.quicksum((1/gr_yield[land])*gr[year, land] for land in lands)
                  + gp.quicksum(cows[year, age] for age in cow_ages)
                  <= land_cap for year in years), name="Land_capacity")
```

Each cow and each acre requires a certain amount of worker time to maintain. The farm is currently able to provide a fixed number of worker hours in a year. Any additional work that needs to be done requires external workers, for which there will be an additional cost.

```python
# 5. Labor

Labor = model.addConstrs((hf_labor*(newborn[year] + cows[year,1])
                  + gp.quicksum(cow_labor*cows[year, age] for age in cow_ages)
                  + gp.quicksum(gr_labor/gr_yield[land]*gr[year,land] for land in lands)
                  + sb_labor/sb_yield*sb[year]
                  <= labor_cap + overtime[year] for year in years), name="Labor")
```

Each year a certain percentage of the cows die, depending on their age.

```python
# 6.1 Continuity

Continuity1 = model.addConstrs((cows[year,1] == (1-hf_decay)*newborn[year-1]
                  for year in years if year > min(years)),
                 name="Continuity_a")

# 6.2 Continuity

Continuity2 = model.addConstrs((cows[year,2] == (1-hf_decay)*cows[year-1,1]
                  for year in years if year > min(years)),
                 name="Continuity_b")

# 6.3 Continuity

Continuity3 = model.addConstrs((cows[year,age+1] == (1-cow_decay)*cows[year-1,age]
                  for year in years for age in cow_ages if year > min(years)),
                 name="Continuity_c")
```

Keep track of the number of cows; cows can come into/out of the model by being bought or sold or through birth.

```python
# 7. Heifers birth

HeifersBirth = model.addConstrs((newborn[year] + hf_sell[year]
                  == gp.quicksum(birthrate/2*cows[year,age] for age in cow_ages) for year in years)
                 , name="Heifers_birth")
```

At the end of the five years, the farmer wants at least 50 and at most 175 diary cows.

```python
# 8. Final dairy cows
FinalDairyCows = model.addRange(gp.quicksum(cows[max(years), age] for age in cow_ages), min_final_cows, max_final_cows, name="Final_dairy_cows" )
```

In the first year, there are 9.5 one-year-old cows and 9.5 two-year-old cows. In addition, there are 9.8 cows for each age from three to 12. Note that we are solving this as an LP model to make it easier to solve. This can lead to fractional values for variables, which are in reality integers.

```python
# 9.1-9.2 Initial conditions

InitialHeifers = model.addConstrs((initial_hf == cows[1, age] for age in ages if age < 3),
                 name="Initial_conditions")

# 9.3 Initial conditions

InitialCows = model.addConstrs((initial_cows == cows[1, age] for age in ages if age >= 3),
                 name="Initial_condition_cows")
```

The following constraints determine yearly profits. The costs for labor currently total $\$4,000$. The profit is influenced by the selling of bullocks and heifers, selling of 12-year-old-cows, selling of milk, selling of grain, selling of sugar beet, buying of grain, buying of sugar beet, labor costs, heifer costs, dairy cow costs, grain costs, sugar beet costs, and capital costs.

```python
# 10. Yearly profit

YearlyProfit = model.addConstrs((profit[year]
                  == bl_price*birthrate/2*gp.quicksum(cows[year, age] for age in cow_ages)
                  + hf_price*hf_sell[year] + cow_price*cows[year, 12]
                  + milk_price*gp.quicksum(cows[year, age] for age in cow_ages)
                  + gr_price*gr_sell[year] + sb_price*sb_sell[year]
                  - gr_cost*gr_buy[year] - sb_cost*sb_buy[year]
                  - overtime_cost*overtime[year] - regular_time_cost
                  - hf_cost*(newborn[year] + cows[year,1])
                  - cow_cost*gp.quicksum(cows[year, age] for age in cow_ages)
                  - gr_land_cost*gp.quicksum(gr[year, land]/gr_yield[land] for land in lands)
                  - sb_land_cost*sb[year]/sb_yield
                  - installment*gp.quicksum(outlay[d] for d in years if d <= year)
                  for year in years), name="Yearly_profit")
```

The total profit of the planning horizon consists of the calculated profits minus the loan payments that remain pending:

```python
# 0. Total profit

model.setObjective(gp.quicksum(profit[year] - installment*(year+4)*outlay[year] for year in years), GRB.MAXIMIZE)
```

Next, we start the optimization and Gurobi finds the optimal solution.

```python
model.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 116 rows, 131 columns and 734 nonzeros
    Model fingerprint: 0x1612a027
    Coefficient statistics:
      Matrix range     [4e-02, 3e+02]
      Objective range  [1e+00, 4e+02]
      Bounds range     [1e+02, 1e+02]
      RHS range        [7e+00, 4e+03]
    Presolve removed 84 rows and 67 columns
    Presolve time: 0.00s
    Presolved: 32 rows, 64 columns, 252 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    5.1200000e+32   5.000000e+30   5.120000e+02      0s
          24    1.2171917e+05   0.000000e+00   0.000000e+00      0s

    Solved in 24 iterations and 0.01 seconds
    Optimal objective  1.217191729e+05

---
## Analysis

The optimal plan results in a total profit of $\$121,719.17$ over the five-year period the model covers. The detailed plan for each year is as follows.

### Financial Plan

This plan determines the profit and outlay at each period of the planning horizon. For example, the profit in year 1 is $\$21,906.1$ and the outlay is zero dollars.

```python
rows = ["Profit", "Outlay"]
columns = years.copy()
finance_plan = pd.DataFrame(columns=columns, index=rows, data=0.0)

for year in years:
    if (abs(profit[year].x) > 1e-6):
        finance_plan.loc["Profit", year] = np.round(profit[year].x, 1)
    if (abs(outlay[year].x) > 1e-6):
        finance_plan.loc["Outlay", year] = np.round(outlay[year].x, 1)
finance_plan
```

      1
      2
      3
      4
      5

      Profit
      21906.1
      21888.7
      25816.1
      26825.8
      25282.6

      Outlay
      0.0
      0.0
      0.0
      0.0
      0.0

### Plan for Grains
This plan defines the number of tons of grain to grow for each type of land group (rows) at each year of the planning horizon (columns). It also defines the number of tons of grain to buy and sell at each year of the planning horizon. For example, we are going to grow 22 tons of grain at land group 1 during year 2 . In addition, we are going to buy 35.1 tons of grain during year 2.

```python
rows = lands.copy() + ["Buy", "Sell"]
columns = years.copy()
gr_plan = pd.DataFrame(columns=columns, index=rows, data=0.0)

for year, land in gr.keys():
    if (abs(gr[year, land].x) > 1e-6):
        gr_plan.loc[land, year] = np.round(gr[year, land].x, 1)
for year in years:
    if (abs(gr_buy[year].x) > 1e-6):
        gr_plan.loc["Buy", year] = np.round(gr_buy[year].x, 1)
    if (abs(gr_sell[year].x) > 1e-6):
        gr_plan.loc["Sell", year] = np.round(gr_sell[year].x, 1)
gr_plan
```

      1
      2
      3
      4
      5

      1
      22.0
      22.0
      22.0
      22.0
      22.0

      2
      0.0
      0.0
      2.8
      0.0
      0.0

      3
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      0.0
      0.0
      0.0
      0.0
      0.0

      Buy
      36.6
      35.1
      37.8
      40.1
      33.5

      Sell
      0.0
      0.0
      0.0
      0.0
      0.0

### Plan for Sugar Beet
This plan defines the number of tons of sugar beet to grow at each year of the planning horizon (columns). It also defines the number of tons of sugar beet to buy and sell at each year of the planning horizon. For example, we are going to grow 94 tons of sugar beet during year 2 . In addition, we are going to sell 27.4 tons of sugar beet during year 2.

```python
rows = ["Grow", "Buy", "Sell"]
columns = years.copy()
sb_plan = pd.DataFrame(columns=columns, index=rows, data=0.0)

for year in years:
    if (abs(sb[year].x) > 1e-6):
        sb_plan.loc["Grow", year] = np.round(sb[year].x, 1)
    if (abs(sb_buy[year].x) > 1e-6):
        sb_plan.loc["Buy", year] = np.round(sb_buy[year].x, 1)
    if (abs(sb_sell[year].x) > 1e-6):
        sb_plan.loc["Sell", year] = np.round(sb_sell[year].x, 1)
sb_plan
```

      1
      2
      3
      4
      5

      Grow
      91.1
      94.0
      97.7
      114.6
      131.3

      Buy
      0.0
      0.0
      0.0
      0.0
      0.0

      Sell
      22.8
      27.4
      24.6
      42.1
      66.6

### Plan for Heifers

This plan shows number of heifers to sell and raise at each period of the planning horizon. For example, we are going to sell 40.8 heifers and raise 11.6 heifers during year 2.

```python
rows = ["Sell", "Raise"]
columns = years.copy()
livestock_plan = pd.DataFrame(columns=columns, index=rows, data=0.0)

for year in years:
    if (abs(hf_sell[year].x) > 1e-6):
        livestock_plan.loc["Sell", year] = np.round(hf_sell[year].x, 1)
    if (abs(newborn[year].x) > 1e-6):
        livestock_plan.loc["Raise", year] = np.round(newborn[year].x, 1)
livestock_plan
```

      1
      2
      3
      4
      5

      Sell
      30.9
      40.8
      57.4
      57.0
      50.9

      Raise
      22.8
      11.6
      0.0
      0.0
      0.0

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

So far, we have considered parametric methods that reduce inference
or
prediction to parameter-fitting. However, for these to work, we had to
assume a
specific functional form for the unknown probability distribution of
the data.
Nonparametric methods eliminate the need to assume a specific
functional form by
generalizing to classes of functions.

## Kernel Density Estimation

We have
already made heavy use of this method with the histogram, which is a
special
case of kernel density estimation.  The histogram can be considered the
crudest
and most useful nonparametric method, that estimates the underlying
probability
distribution of the data.

To be formal and place the histogram on the same
footing as our earlier
estimations, suppose that $\mathscr{X}=[0,1]^d$ is the
$d$ dimensional unit
cube and that $h$ is the *bandwidth* or size of a *bin* or
sub-cube. Then,
there are $N\approx(1/h)^d$ such bins, each with volume $h^d$,
$\lbrace
B_1,B_2,\ldots,B_N \rbrace$. With all this in place, we can write the
histogram
has a probability density estimator of the form,

$$
\hat{p}_h(x) = \sum_{k=1}^N \frac{\hat{\theta}_k}{h} I(x\in B_k)
$$

 where

$$
\hat{\theta}_k=\frac{1}{n} \sum_{j=1}^n I(X_j\in B_k)
$$

 is the fraction of data points ($X_k$) in each bin, $B_k$. We want to
bound the
bias and variance of $\hat{p}_h(x)$. Keep in mind that we are trying
to estimate
a function of $x$, but the set of all possible probability
distribution
functions is extremely large and hard to manage. Thus, we need
to restrict our
attention to the following class of probability distribution of
so-called
Lipschitz functions,

$$
\mathscr{P}(L) = \lbrace p\colon \vert p(x)-p(y)\vert \le L \Vert x-y\Vert,
\forall \: x,y \rbrace
$$

 Roughly speaking, these are the density
functions whose slopes (i.e., growth
rates)  are bounded by $L$.
It turns out that the bias of the histogram
estimator is bounded in the
following way,

$$
\int\vert p(x)-\mathbb{E}(\hat{p}_h(x))\vert dx \le L h\sqrt{d}
$$

  Similarly, the variance is bounded by the following,

$$
\mathbb{V}(\hat{p}_h(x)) \le \frac{C}{n h^d}
$$

 for some constant $C$. Putting these two facts together means that the
risk is
bounded by,

$$
R(p,\hat{p}) = \int \mathbb{E}(p(x) -\hat{p}_h(x))^2 dx \le L^2 h^2 d +
\frac{C}{n h^d}
$$

 This upper bound is minimized by choosing

$$
h = \left(\frac{C}{L^2 n d}\right)^\frac{1}{d+2}
$$

 In particular, this means that,

$$
\sup_{p\in\mathscr{P}(L)} R(p,\hat{p}) \le C_0
\left(\frac{1}{n}\right)^{\frac{2}{d+2}}
$$

 where the constant $C_0$ is a function of $L$. There is a theorem
[[wasserman2004all]](#wasserman2004all) that shows this bound in tight, which
basically means
that the histogram is a really powerful probability density
estimator for
Lipschitz functions with risk that goes as
$\left(\frac{1}{n}\right)^{\frac{2}{d+2}}$.  Note that this class of functions
is not necessarily smooth because the Lipschitz condition admits
non-smooth
functions. While this is a reassuring result, we typically do
not know which
function class (Lipschitz or not) a particular probability
belongs to ahead of
time.  Nonetheless, the rate at which the risk changes with
both dimension $d$
and $n$ samples would be hard to understand without this
result.
[Figure](#fig:nonparametric_001) shows the probability distribution
function of
the $\beta(2,2)$ distribution compared to computed histograms for
different
values of $n$. The box plots on each of the points show how the
variation in
each bin of the histogram reduces with increasing $n$. The risk
function
$R(p,\hat{p})$ above is based upon integrating the squared difference
between
the histogram (as a piecewise function of $x$) and the probability
distribution
function.

**Programming Tip.**

The following snippet is the main element of
the code for [Figure](#fig:nonparametric_001).

```python
def generate_samples(n,ntrials=500):
    phat = np.zeros((nbins,ntrials))
    for k in range(ntrials):
        d = rv.rvs(n)
        phat[:,k],_=histogram(d,bins,density=True)
    return phat
```

The code uses the `histogram` function from Numpy.
To be consistent with the
risk function $R(p,\hat{p})$, we have to make sure
the `bins` keyword argument
is formatted correctly using a sequence of
bin-edges instead of just a single
integer. Also, the `density=True` keyword
argument normalizes the histogram
appropriately so that the comparison between
it and the probability distribution
function of the simulated beta distribution
is correctly scaled.

The box plots on
each of the points show how the variation in each bin of the histogram reduces
with increasing $n$.

## Kernel Smoothing

We can extend our methods
to other function classes using kernel functions.
A one-dimensional smoothing
kernel is a smooth function $K$ with
the following properties,

$$
\begin{align*}
\int K(x) dx &= 1 \\\
\int x K(x) dx &= 0 \\\
0< \int x^2 K(x)
dx &< \infty \\\
\end{align*}
$$

 For example, $K(x)=I(x)/2$ is the boxcar kernel, where $I(x)=1$
when $\vert
x\vert\le 1$ and zero otherwise. The kernel density estimator is
very similar to
the histogram, except now we put a kernel function on every
point as in the
following,

$$
\hat{p}(x)=\frac{1}{n}\sum_{i=1}^n \frac{1}{h^d} K\left(\frac{\Vert
x-X_i\Vert}{h}\right)
$$

 where $X\in \mathbb{R}^d$. [Figure](#fig:nonparametric_002) shows an
example of
a kernel density estimate using a Gaussian kernel function,
$K(x)=e^{-x^2/2}/\sqrt{2\pi}$. There are  five data points shown by the
vertical
lines in the upper panel. The dotted lines show the individual $K(x)$
function
at each of the data points. The lower panel shows the overall kernel
density
estimate, which is the scaled sum of the upper panel.

There is an important
technical result in [[wasserman2004all]](#wasserman2004all) that
states that
kernel density estimators are minimax in the sense we
discussed in the maximum
likelihood the section [ch:stats:sec:mle](#ch:stats:sec:mle). In
broad strokes,
this means that the analogous risk for the kernel
density estimator is
approximately bounded by the following factor,

$$
R(p,\hat{p}) \lesssim n^{-\frac{2 m}{2 m+d}}
$$

 for some constant $C$ where $m$ is a factor related to bounding
the derivatives
of the probability density function. For example, if the second
derivative of
the density function is bounded, then $m=2$. This means that
the convergence
rate for this estimator decreases with increasing dimension
$d$.

The upper panel shows the individual
kernel functions placed at each of the data points. The lower panel shows the
composite kernel density estimate which is the sum of the individual functions
in the upper panel.

### Cross-Validation

As a practical matter,
the tricky part of the kernel density estimator (which
includes the histogram as
a special case) is that we need to somehow compute
the bandwidth $h$ term using
data. There are several rule-of-thumb methods that
for some common kernels,
including Silverman's rule and Scott's rule for
Gaussian kernels. For example,
Scott's factor is to simply compute $h=n^{
-1/(d+4) }$ and Silverman's is $h=(n
(d+2)/4)^{ (-1/(d+4)) }$. Rules of
this kind are derived by assuming the
underlying probability density
function is of a certain family (e.g., Gaussian),
and then deriving the
best $h$ for a certain type of kernel density estimator,
usually equipped
with extra functional properties (say, continuous derivatives
of a
certain order). In practice, these rules seem to work pretty well,
especially for uni-modal probability density functions.  Avoiding these
kinds of
assumptions means computing the bandwidth from data directly and that is where
cross validation comes in.

Cross-validation is a method to estimate the
bandwidth from the data itself.
The idea is to write out the following
Integrated Squared Error (ISE),

$$
\begin{align*}
\textnormal{ISE}(\hat{p}_h,p)&=\int (p(x)-\hat{p}_h(x))^2
dx\\\
                             &= \int \hat{p}_h(x)^2 dx - 2\int p(x)
\hat{p}_h dx + \int p(x)^2 dx
\end{align*}
$$

 The problem with this expression is the middle term [^last_term],
[^last_term]: The last term is of no interest because we are
only interested in
relative changes in the ISE.

$$
\int p(x)\hat{p}_h dx
$$

 where $p(x)$ is what we are trying to estimate with $\hat{p}_h$. The
form of
the last expression looks like an expectation of $\hat{p}_h$ over the
density of
$p(x)$, $\mathbb{E}(\hat{p}_h)$. The approach is to
approximate this with the
mean,

$$
\mathbb{E}(\hat{p}_h) \approx \frac{1}{n}\sum_{i=1}^n \hat{p}_h(X_i)
$$

 The problem with this approach is that $\hat{p}_h$ is computed using
the same
data that the approximation utilizes.  The way to get around this is
to split
the data into two equally sized chunks $D_1$, $D_2$; and then compute
$\hat{p}_h$ for a sequence of different $h$ values over the $D_1$ set. Then,
when we apply the above approximation for the data ($Z_i$) in the $D_2$ set,

$$
\mathbb{E}(\hat{p}_h) \approx \frac{1}{\vert D_2\vert}\sum_{Z_i\in D_2}
\hat{p}_h(Z_i)
$$

  Plugging this approximation back into the integrated squared error
provides
the objective function,

$$
\texttt{ISE}\approx \int \hat{p}_h(x)^2 dx-\frac{2}{\vert
D_2\vert}\sum_{Z_i\in D_2} \hat{p}_h(Z_i)
$$

 Some code will make these steps concrete. We will need some tools from
Scikit-
learn.

```python
from sklearn.model_selection import train_test_split
from sklearn.neighbors.kde import KernelDensity
```

The `train_test_split` function makes it easy to split and
keep track of the
$D_1$ and $D_2$ sets we need for cross validation. Scikit-learn
already has a
powerful and flexible implementation of kernel density estimators.
To compute
the objective function, we need some
basic numerical integration tools from
Scipy. For this example, we
will generate samples from a $\beta(2,2)$
distribution, which is
implemented in the `stats` submodule in Scipy.

```python
import numpy as np
np.random.seed(123456)
```

```python
from scipy.integrate import quad
from scipy import stats
rv= stats.beta(2,2)
n=100                 # number of samples to generate
d = rv.rvs(n)[:,None] # generate samples as column-vector
```

**Programming Tip.**

The use of the `[:,None]` in the last line formats the
Numpy array returned by
the `rvs` function into a Numpy vector with a column
dimension of one. This is
required by the `KernelDensity` constructor because
the column dimension is
used for different features (in general) for Scikit-
learn. Thus, even though we
only have one feature, we still need to comply with
the structured input that
Scikit-learn relies upon. There are many ways to
inject the additional
dimension other than using `None`. For example, the more
cryptic, `np.c_`, or
the less cryptic `[:,np.newaxis]` can do the same, as can
the `np.reshape`
function.

 The next step is to split the data into two
halves and loop over
each of the $h_i$ bandwidths to create a separate kernel
density estimator
based on the $D_1$ data,

```python
train,test,_,_=train_test_split(d,d,test_size=0.5)
kdes=[KernelDensity(bandwidth=i).fit(train)
        for i in [.05,0.1,0.2,0.3]]
```

**Programming Tip.**

Note that the single underscore symbol in Python refers to
the last evaluated
result.  the above code unpacks the tuple returned by
`train_test_split` into
four elements.  Because we are only interested in the
first two, we assign the
last two to the underscore symbol. This is a stylistic
usage to make it clear
to the reader that the last two elements of the tuple are
unused.
Alternatively, we could assign the last two elements to a pair of dummy
variables that we do not use later, but then  the reader skimming the code may
think that those dummy variables are relevant.

 The last step is to loop over
the so-created kernel density estimators
and compute the objective function.

```python
for i in kdes:
    f = lambda x: np.exp(i.score_samples(x))
    f2 = lambda x: f([[x]])**2
    print('h=%3.2f\t %3.4f'%(i.bandwidth,quad(f2,0,1)[0]
          -2*np.mean(f(test))))
```

    h=0.05	 -1.1323
    h=0.10	 -1.1336
    h=0.20	 -1.1330
    h=0.30	 -1.0810

**Programming Tip.**

The lambda functions defined in the last block are
necessary because
Scikit-learn implements the return value of the kernel density
estimator as a
logarithm via the `score_samples` function. The numerical
quadrature function
`quad` from Scipy computes the $\int \hat{p}_h(x)^2 dx$ part
of the objective
function.

```python
%matplotlib inline
```

```python
from __future__ import division
from matplotlib.pylab import subplots
fig,ax=subplots()
xi = np.linspace(0,1,100)[:,None]
for i in kdes:
    f=lambda x: np.exp(i.score_samples(x))
    f2 = lambda x: f(x)**2
    _=ax.plot(xi,f(xi),label='$h$='+str(i.bandwidth))

_=ax.set_xlabel('$x$',fontsize=28)
_=ax.set_ylabel('$y$',fontsize=28)
_=ax.plot(xi,rv.pdf(xi),'k:',lw=3,label='true')
_=ax.legend(loc=0)
ax2 = ax.twinx()
_=ax2.hist(d,20,alpha=.3,color='gray')
_=ax2.axis(ymax=50)
_=ax2.set_ylabel('count',fontsize=28)
fig.tight_layout()
fig.savefig('fig-statistics/nonparametric_003.png')
```

Each line above is a
different kernel density estimator for the given bandwidth as an approximation
to the true density function. A plain histogram is imprinted on the bottom for
reference.

Scikit-learn has many more advanced tools to automate this kind
of
hyper-parameter (i.e., kernel density bandwidth) search. To utilize these
advanced tools, we need to format the current problem slightly differently by
defining the following wrapper class.

```python
class KernelDensityWrapper(KernelDensity):
    def predict(self,x):
        return np.exp(self.score_samples(x))
    def score(self,test):
        f = lambda x: self.predict(x)
        f2 = lambda x: f([[x]])**2
        return -(quad(f2,0,1)[0]-2*np.mean(f(test)))
```

This is tantamount to reorganizing the above previous code
into functions that
Scikit-learn requires. Next, we create the
dictionary of parameters we want to
search over (`params`) below
and then start the grid search with the `fit`
function,

```python
from sklearn.model_selection import GridSearchCV
params = {'bandwidth':np.linspace(0.01,0.5,10)}
clf = GridSearchCV(KernelDensityWrapper(), param_grid=params,cv=2)
clf.fit(d)
print (clf.best_params_)
```

    {'bandwidth': 0.17333333333333334}

The grid search iterates over all the elements in the `params`
dictionary and
reports the best bandwidth over that list of parameter values.
The `cv` keyword
argument above specifies that we want to split the data
into two equally-sized
sets for training and testing. We can
also examine the values of the objective
function for each point
on the grid as follow,

```python
clf.cv_results_['mean_test_score']
```

    array([0.60758058, 1.06324954, 1.11858734, 1.13187097, 1.12006532,
           1.09186225, 1.05391076, 1.01126161, 0.96717292, 0.92354959])

Keep in mind that the grid search examines multiple folds for cross
validation
to compute the above means and standard deviations. Note that there
is also a
`RandomizedSearchCV` in case you would rather specify a distribution
of
parameters instead of a list. This is particularly useful for searching very
large parameter spaces where an exhaustive grid search would be too
computationally expensive. Although kernel density estimators are easy to
understand and have many attractive analytical properties, they become
practically prohibitive for large, high-dimensional data sets.

## Nonparametric
Regression Estimators

Beyond estimating the underlying probability density, we
can use nonparametric
methods to compute estimators of the underlying function
that is generating the
data.  Nonparametric regression estimators of the
following form are known as
linear smoothers,

$$
\hat{y}(x) =  \sum_{i=1}^n \ell_i(x) y_i
$$

 To understand the performance of these smoothers,
we can define the risk as the
following,

$$
R(\hat{y},y) = \mathbb{E}\left( \frac{1}{n} \sum_{i=1}^n
(\hat{y}(x_i)-y(x_i))^2 \right)
$$

 and find the best $\hat{y}$ that minimizes this. The problem with
this metric
is that we do not know $y(x)$, which is why we are trying to
approximate it with
$\hat{y}(x)$. We could construct an estimation by using the
data at hand as in
the following,

$$
\hat{R}(\hat{y},y) =\frac{1}{n} \sum_{i=1}^n (\hat{y}(x_i)-Y_i)^2
$$

 where we have substituted the data $Y_i$ for the unknown function
value,
$y(x_i)$. The problem with this approach is that we are using the data
to
estimate the function and then using the same data to evaluate the risk of
doing
so. This kind of double-dipping leads to overly optimistic estimators.
One way
out of this conundrum is to use leave-one-out cross validation, wherein
the
$\hat{y}$ function is estimated using all but one of the data pairs,
$(X_i,Y_i)$. Then, this missing data element is used to estimate the above
risk.
Notationally, this is written as the following,

$$
\hat{R}(\hat{y},y) =\frac{1}{n} \sum_{i=1}^n (\hat{y}_{(-i)}(x_i)-Y_i)^2
$$

 where $\hat{y}_{(-i)}$ denotes computing the estimator without using
the
$i^{th}$ data pair. Unfortunately, for anything other than relatively small
data
sets, it quickly becomes computationally prohibitive to use leave-one-out
cross
validation in practice. We'll get back to this issue shortly, but let's
consider
a concrete example of such a nonparametric smoother.

## Nearest Neighbors
Regression

The simplest possible
nonparametric regression method is the $k$-nearest
neighbors regression. This is
easier to explain in words than to write out in
math. Given an input $x$, find
the closest one of the $k$ clusters that
contains it and then return the mean of
the data values in that cluster. As a
univariate example, let's consider the
following *chirp* waveform,

$$
y(x)=\cos\left(2\pi\left(f_o x + \frac{BW x^2}{2\tau}\right)\right)
$$

 This waveform is important in high-resolution radar applications.
The $f_o$ is
the start frequency and $BW/\tau$ is the frequency slope of the
signal. For our
example, the fact that it is nonuniform over its domain is
important. We can
easily create some data by sampling the
chirp as in the following,

```python
from numpy import cos, pi
xi = np.linspace(0,1,100)[:,None]
xin = np.linspace(0,1,12)[:,None]
f0 = 1 # init frequency
BW = 5
y = np.cos(2*pi*(f0*xin+(BW/2.0)*xin**2))
```

We can use this data to construct a simple nearest neighbor
estimator using
Scikit-learn,

```python
from sklearn.neighbors import KNeighborsRegressor
knr=KNeighborsRegressor(2)
knr.fit(xin,y)
```

    KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
                        metric_params=None, n_jobs=None, n_neighbors=2, p=2,
                        weights='uniform')

**Programming Tip.**

Scikit-learn has a fantastically consistent interface. The
`fit` function above
fits the model parameters to the data. The corresponding
`predict` function
returns the output of the model given an arbitrary input. We
will spend a lot
more time on Scikit-learn in the machine learning chapter. The
`[:,None]` part
at the end is just injecting a column dimension into the array
in order to
satisfy the dimensional requirements of Scikit-learn.

```python
from matplotlib.pylab import subplots
fig,ax=subplots()
yi = cos(2*pi*(f0*xi+(BW/2.0)*xi**2))
_=ax.plot(xi,yi,'k--',lw=2,label=r'$y(x)$')
_=ax.plot(xin,y,'ko',lw=2,ms=11,color='gray',alpha=.8,label='$y(x_i)$')
_=ax.fill_between(xi.flat,yi.flat,knr.predict(xi).flat,color='gray',alpha=.3)
_=ax.plot(xi,knr.predict(xi),'k-',lw=2,label='$\hat{y}(x)$')
_=ax.set_aspect(1/4.)
_=ax.axis(ymax=1.05,ymin=-1.05)
_=ax.set_xlabel(r'$x$',fontsize=24)
_=ax.legend(loc=0)
fig.set_tight_layout(True)
fig.savefig('fig-statistics/nonparametric_004.png')
```

The dotted line shows the chirp
signal and the solid line shows the nearest neighbor estimate. The gray circles
are the sample points that we used to fit the nearest neighbor estimator. The
shaded area shows the gaps between the estimator and the unsampled chirp.

[Figure](#fig:nonparametric_004) shows the sampled signal (gray
circles) against
the values generated by the nearest neighbor estimator (solid
line). The dotted
line is the full unsampled chirp signal, which increases in
frequency with $x$.
This is important for our example because it adds a
non-stationary aspect to
this problem in that the function gets progressively
wigglier with increasing
$x$.  The area between the estimated curve and the
signal is shaded in gray.
Because the nearest neighbor estimator uses only two
nearest neighbors, for each
new $x$, it finds the two adjacent $X_i$ that
bracket the $x$ in the training
data and then averages the corresponding $Y_i$
values to compute the estimated
value. That is, if you take every adjacent pair
of sequential gray circles in
the Figure, you find that the horizontal solid line
splits the pair on the
vertical axis. We can adjust the number of
nearest neighbors  by changing the
constructor,

```python
knr=KNeighborsRegressor(3)
knr.fit(xin,y)
```

    KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',
                        metric_params=None, n_jobs=None, n_neighbors=3, p=2,
                        weights='uniform')

```python
fig,ax=subplots()
_=ax.plot(xi,yi,'k--',lw=2,label=r'$y(x)$')
_=ax.plot(xin,y,'ko',lw=2,ms=11,color='gray',alpha=.8,label='$y(x_i)$')
_=ax.fill_between(xi.flat,yi.flat,knr.predict(xi).flat,color='gray',alpha=.3)
_=ax.plot(xi,knr.predict(xi),'k-',lw=2,label='$\hat{y}(x)$')
_=ax.set_aspect(1/4.)
_=ax.axis(ymax=1.05,ymin=-1.05)
_=ax.set_xlabel(r'$x$',fontsize=24)
_=ax.legend(loc=0)
fig.set_tight_layout(True)
fig.savefig('fig-statistics/nonparametric_005.png')
```

which produces the following corresponding [Figure](#fig:nonparametric_005).

This is the same as
[Figure](#fig:nonparametric_004) except that here there are three nearest
neighbors used to build the estimator.

For this
example, [Figure](#fig:nonparametric_005) shows that with
more nearest neighbors
the fit performs poorly, especially towards the end of
the signal, where there
is increasing variation, because the chirp is not
uniformly continuous.

Scikit-
learn provides many tools for cross validation. The following code
sets up the
tools for leave-one-out cross validation,

```python
from sklearn.model_selection import LeaveOneOut
loo=LeaveOneOut()
```

The `LeaveOneOut` object is an iterable that produces a set of
disjoint indices
of the data --- one for fitting the model (training set) and
one for evaluating
the model (testing set).  The next block loops over the
disjoint sets of
training and test indicies iterates provided by the `loo`
variable to evaluate
the estimated risk, which is accumulated in the `out`
list.

```python
out=[]
for train_index, test_index in loo.split(xin):
    _=knr.fit(xin[train_index],y[train_index])
    out.append((knr.predict(xi[test_index])-y[test_index])**2)

print( 'Leave-one-out Estimated Risk: ',np.mean(out),)
```

    Leave-one-out Estimated Risk:  1.0351713662681845

The last line in the code above reports leave-one-out's estimated
risk.
Linear smoothers of this type can be rewritten in using the following matrix,

$$
\mathscr{S} = \left[ \ell_i(x_j) \right]_{i,j}
$$

 so that

$$
\hat{\mathbf{y}} = \mathscr{S} \mathbf{y}
$$

   where $\mathbf{y}=\left[Y_1,Y_2,\ldots,Y_n\right]\in \mathbb{R}^n$
and $\hat{
\mathbf{y}
}=\left[\hat{y}(x_1),\hat{y}(x_2),\ldots,\hat{y}(x_n)\right]\in
\mathbb{R}^n$.
This leads to a quick way to approximate leave-one-out cross
validation as the
following,

$$
\hat{R}=\frac{1}{n}\sum_{i=1}^n\left(\frac{y_i-\hat{y}(x_i)}{1-\mathscr{S}_{i,i}}\right)^2
$$

 However, this does not reproduce the approach in the code above
because it
assumes that each $\hat{y}_{(-i)}(x_i)$ is consuming one fewer
nearest neighbor
than $\hat{y}(x)$.

We can get this $\mathscr{S}$ matrix from the `knr` object
as in the following,

```python
_= knr.fit(xin,y) # fit on all data
S=(knr.kneighbors_graph(xin)).todense()/float(knr.n_neighbors)
```

The `todense` part reformats the sparse matrix that is
returned into a regular
Numpy `matrix`. The following shows a subsection
of this $\mathcal{S}$ matrix,

```python
print(S[:5,:5])
```

    [[0.33333333 0.33333333 0.33333333 0.         0.        ]
     [0.33333333 0.33333333 0.33333333 0.         0.        ]
     [0.         0.33333333 0.33333333 0.33333333 0.        ]
     [0.         0.         0.33333333 0.33333333 0.33333333]
     [0.         0.         0.         0.33333333 0.33333333]]

The sub-blocks show the windows of the the `y` data that are being
processed by
the nearest neighbor estimator. For example,

```python
print(np.hstack([knr.predict(xin[:5]),(S*y)[:5]]))#columns match
```

    [[ 0.55781314  0.55781314]
     [ 0.55781314  0.55781314]
     [-0.09768138 -0.09768138]
     [-0.46686876 -0.46686876]
     [-0.10877633 -0.10877633]]

Or, more concisely checking all entries for approximate equality,

```python
np.allclose(knr.predict(xin),S*y)
```

    True

which shows that the results from the nearest neighbor
object and the matrix
multiply match.

**Programming Tip.**

Note that because we formatted the
returned $\mathscr{S}$ as a Numpy matrix, we
automatically get the matrix
multiplication instead of default element-wise
multiplication in the `S*y` term.
## Kernel Regression

For estimating the probability density, we started with
the histogram and moved
to the more general kernel density estimate. Likewise,
we can also extend
regression from nearest neighbors to kernel-based regression
using the
*Nadaraya-Watson* kernel regression estimator.  Given a bandwidth
$h>0$, the
kernel regression estimator is defined as the following,

$$
\hat{y}(x)=\frac{\sum_{i=1}^n K\left(\frac{x-x_i}{h}\right) Y_i}{\sum_{i=1}^n
K \left( \frac{x-x_i}{h} \right)}
$$

 Unfortunately, Scikit-learn does not implement this
regression estimator;
however, Jan Hendrik Metzen  makes a compatible
version available on
`github.com`.

```python
import sys
sys.path.append('../src-statistics')
xin = np.linspace(0,1,20)[:,None]
y = cos(2*pi*(f0*xin+(BW/2.0)*xin**2)).flatten()
```

    The history saving thread hit an unexpected error (OperationalError('disk I/O error')).History will not be written to the database.

```python
from kernel_regression import KernelRegression
```

This code makes it possible to internally optimize over the bandwidth
parameter
using leave-one-out cross validation by specifying a grid of
potential bandwidth
values (`gamma`), as in the following,

```python
kr = KernelRegression(gamma=np.linspace(6e3,7e3,500))
kr.fit(xin,y)
```

    KernelRegression(gamma=6000.0, kernel='rbf')

[Figure](#fig:nonparametric_006) shows the kernel estimator (heavy
black line)
using the Gaussian kernel compared to the nearest neighbor
estimator (solid
light black line). As before, the data points are shown as
circles.
[Figure](#fig:nonparametric_006) shows that the kernel estimator can
pick out
the sharp peaks that are missed by the nearest neighbor estimator.

The heavy black line is the Gaussian
kernel estimator. The light black line is the nearest neighbor estimator. The
data points are shown as gray circles. Note that unlike the nearest neighbor
estimator, the Gaussian kernel estimator is able to pick out the sharp peaks in
the training data.

Thus, the difference between nearest neighbor
and kernel estimation is that the
latter provides a smooth moving averaging of
points whereas the former provides
a discontinuous averaging. Note that kernel
estimates suffer near the
boundaries where there is mismatch between the edges
and the kernel
function. This problem gets worse in higher dimensions because
the data
naturally drift towards the boundaries (this is a consequence of the
*curse of
dimensionality*). Indeed, it is not possible to simultaneously
maintain local
accuracy (i.e., low bias) and a generous neighborhood (i.e., low
variance). One
way to address this problem is to create a local polynomial
regression using
the kernel function as a window to localize a region of
interest. For example,

$$
\hat{y}(x)=\sum_{i=1}^n K\left(\frac{x-x_i}{h}\right) (Y_i-\alpha - \beta
x_i)^2
$$

 and now we have to optimize over the two linear parameters $\alpha$
and
$\beta$. This method is known as *local linear regression*
[[loader2006local]](#loader2006local),
[[hastie2013elements]](#hastie2013elements). Naturally, this can be
extended to
higher-order polynomials. Note that these methods are not yet
implemented in
Scikit-learn.

```python
fig,ax=subplots()
#fig.set_size_inches((12,4))
_=ax.plot(xi,kr.predict(xi),'k-',label='kernel',lw=3)
_=ax.plot(xin,y,'o',lw=3,color='gray',ms=12)
_=ax.plot(xi,yi,'--',color='gray',label='chirp')
_=ax.plot(xi,knr.predict(xi),'k-',label='nearest')
_=ax.axis(ymax=1.1,ymin=-1.1)
_=ax.set_aspect(1/4.)
_=ax.axis(ymax=1.05,ymin=-1.05)
_=ax.set_xlabel(r'$x$',fontsize=24)
_=ax.set_ylabel(r'$y$',fontsize=24)
_=ax.legend(loc=0)
fig.savefig('fig-statistics/nonparametric_006.png')
```

## Curse of Dimensionality

The so-called curse of
dimensionality occurs as we move into higher and higher
dimensions. The term was
coined by Bellman in 1961 while he was studying
adaptive control processes.
Nowadays, the term is vaguely refers to anything
that becomes more complicated
as the number of dimensions increases
substantially. Nevertheless, the concept
is useful for recognizing
and characterizing the practical difficulties of high-
dimensional analysis and
estimation.

Consider the volume of an $d$-dimensional
sphere of radius $r$,

$$
V_s(d,r)=\frac{\pi ^{d/2} r^d}{\Gamma \left(\frac{d}{2}+1\right)}
$$

 Further, consider the sphere $V_s(d,1/2)$ enclosed by an $d$
dimensional unit
cube.  The volume of the cube is always equal to one, but
$\lim_{d\rightarrow\infty} V_s(d,1/2) = 0$. What does this mean? It means that
the volume of the cube is pushed away from its center, where the embedded
hypersphere lives.  Specifically, the distance from the center of the cube to
its vertices in $d$ dimensions is $\sqrt{d}/2$, whereas the distance from the
center of the inscribing sphere is $1/2$. This diagonal distance goes to
infinity as $d$ does. For a fixed $d$, the tiny spherical region at the center
of the cube has many long spines attached to it, like a hyper-dimensional sea
urchin or porcupine.

Another way to think about this is to consider the
$\epsilon>0$ thick peel of the
hypersphere,

$$
\mathcal{P}_{\epsilon} =V_s(d,r) - V_s(d,r-\epsilon)
$$

 Then, we consider the following limit,

$$
\begin{equation}
\lim_{d\rightarrow\infty}\mathcal{P}_{\epsilon}
=\lim_{d\rightarrow\infty} V_s(d,r)\left(1 -
\frac{V_s(d,r-\epsilon)}{V_s(d,r)}\right)
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation} \
=\lim_{d\rightarrow\infty} V_s(d,r)\left(1 -\lim_{d\rightarrow\infty}
\left(\frac{r-\epsilon}{r}\right)^d\right)
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation} \
=\lim_{d\rightarrow\infty} V_s(d,r)
\label{_auto3} \tag{3}
\end{equation}
$$

 So, in the limit, the volume of the $\epsilon$-thick peel
consumes the volume
of the hypersphere.

What are the consequences of this? For methods that rely on
nearest
neighbors, exploiting locality to lower bias becomes intractable. For
example, suppose we have an $d$ dimensional space and a point near the
origin we
want to localize around. To estimate behavior around this
point, we need to
average the unknown function about this point, but
in a high-dimensional space,
the chances of finding neighbors to
average are slim. Looked at from the
opposing point of view, suppose
we have a binary variable, as in the coin-
flipping problem. If we have
1000 trials, then, based on our earlier work, we
can be confident
about estimating the probability of heads.  Now, suppose we
have 10
binary variables.  Now we have $2^{ 10 }=1024$ vertices to estimate.
If
we had the same 1000 points, then at least 24 vertices would not
get any data.
To keep the same resolution, we would need 1000 samples
at each vertex for a
grand total of $1000\times 1024 \approx 10^6$
data points. So, for a ten fold
increase in the number of variables,
we now have about 1000 more data points to
collect to maintain the
same statistical resolution. This is the curse of
dimensionality.

Perhaps some code will clarify this. The following code
generates samples in
two dimensions that are plotted as points in
[Figure](#fig:curse_of_dimensionality_001) with the inscribed circle in two
dimensions. Note that for $d=2$ dimensions,  most of the points are contained
in
the circle.

```python
import numpy as np
v=np.random.rand(1000,2)-1/2.
```

```python
from matplotlib.patches import Circle
from matplotlib.pylab import subplots
fig,ax=subplots()
fig.set_size_inches((5,5))
_=ax.set_aspect(1)
_=ax.scatter(v[:,0],v[:,1],color='gray',alpha=.3)
_=ax.add_patch(Circle((0,0),0.5,alpha=.8,lw=3.,fill=False))
fig.savefig('fig-statistics/curse_of_dimensionality_001.pdf')
```

Two dimensional scatter of points randomly and independently uniformly
distributed in the unit square. Note that most of the points are contained in
the circle. Counter to intuition, this does not persist as the number of
dimensions increases.

The next code block describes the core computation in
[Figure](#fig:curse_of_dimensionality_002). For each of the dimensions, we
create a set of uniformly distributed random variates along each dimension
and
then compute how close each $d$ dimensional vector is to the origin.
Those that
measure one half are those contained in the hypersphere. The
histogram of each
measurment is shown in the corresponding panel in the
[Figure](#fig:curse_of_dimensionality_002). The dark vertical line shows the
threshold value. Values to the left
of this indicate the population that are
contained in the hypersphere. Thus,
[Figure](#fig:curse_of_dimensionality_002)
shows that as $d$ increases,
fewer points are contained in the inscribed
hypersphere. The following
code paraphrases  the content of
[Figure](#fig:curse_of_dimensionality_002).

```python
fig,ax=subplots()
for d in [2,3,5,10,20,50]:
    v=np.random.rand(5000,d)-1/2.
    ax.hist([np.linalg.norm(i) for i in v])
```

```python
siz = [ 2,3,5,10,20,50 ]
fig,axs=subplots(3,2,sharex=True)
fig.set_size_inches((10,6))
for ax,k in zip(axs.flatten(),siz):
    v=np.random.rand(5000,k)-1/2.
    _=ax.hist([np.linalg.norm(i) for i in v],color='gray',density=True);
    _=ax.vlines(0.5,0,ax.axis()[-1]*1.1,lw=3)
    _=ax.set_title('$d=%d$'%k,fontsize=20)
    _=ax.tick_params(labelsize='small',top=False,right=False)
    _=ax.spines['top'].set_visible(False)
    _=ax.spines['right'].set_visible(False)
    _=ax.spines['left'].set_visible(False)
    _=ax.yaxis.set_visible(False)
    _=ax.axis(ymax=3.5)

fig.set_tight_layout(True)
fig.savefig('fig-statistics/curse_of_dimensionality_002.pdf')
```

Each panel shows the histogram
of lengths of uniformly distributed $d$ dimensional random vectors. The
population to the left of the dark vertical line are those that are contained in
the inscribed hypersphere. This shows that fewer points are contained in the
hypersphere with increasing dimension.

##
Nonparametric Tests

Determining whether or not two sets of observations derive
from the same
underlying probability distribution is an important problem. The
most popular
way to do this is with a standard t-test, but that requires
assumptions about
normality that may be hard to justify, which leads to
nonparametric methods can
get at this questions without such assumptions.

Let
$V$ and $W$ be continuous random variables. The variable
$V$ is *stochastically
larger* than $W$ if,

$$
\mathbb{P}(V\ge x) \ge \mathbb{P}(W\ge x)
$$

 for all $x\in \mathbb{R}$ with strict inequality for at least one
$x$. The term
*stochastically smaller* means the obverse of this. For example,
the black line
density function shown in [Figure](#fig:nonparametric_tests_001) is
stochastically larger than the gray one.

```python
import numpy as np
from scipy import stats
from matplotlib.pylab import subplots

fig,ax=subplots()
xi = np.linspace(0,2,100)
_=ax.plot(xi,stats.norm(1,0.25).pdf(xi),lw=3,color='k')
_=ax.plot(xi,stats.beta(2,4).pdf(xi),lw=3,color='gray')
_=ax.spines['top'].set_visible(0)
_=ax.spines['right'].set_visible(0)
_=ax.tick_params(labelsize='medium',top=False,right=False)
ax.set_aspect(1/2.5)

fig.savefig('fig-statistics/nonparametric_tests_001.png')
```

The black line density function
is stochastically larger than the gray one.

### The Mann-Whitney-Wilcoxon Test

The Mann-Whitney-Wilcoxon Test approaches the
following alternative hypotheses

   * $H_0$ : $F(x) = G(x)$ for all $x$ versus
* $H_a$ : $F(x) \ge G(x)$, $F$ stochastically greater than $G$.

Suppose we have
two data sets $X$ and $Y$ and we want to know if they are drawn
from the same
underlying probability distribution or if one is stochastically
greater than the
other. There are $n_x$ elements in $X$ and $n_y$ elements in
$Y$. If we combine
these two data sets and rank them, then, under the null
hypothesis, any data
element should be as likely as any other to be assigned
any particular rank.
that is, the combined set $Z$,

$$
Z = \lbrace X_1,\ldots,X_{n_x}, Y_1,\ldots,Y_{n_y} \rbrace
$$

 contains $n=n_x+n_y$ elements. Thus, any assignment of $n_y$ ranks
from the
integers $\lbrace 1,\ldots,n \rbrace$  to $\lbrace Y_1,\ldots,Y_{n_y}
\rbrace$
should be equally likely (i.e., $\mathbb{P}={ \binom{n}{n_y} }^{-1}$).
Importantly, this property is independent of the $F$ distribution.

That is, we
can define the $U$ statistic as the following,

$$
U_X =\sum_{i=1}^{n_x}\sum_{j=1}^{n_y}\mathbb{I}(X_i\ge Y_j)
$$

 where $\mathbb{I}(\cdot)$ is the usual indicator function. For an
interpretation, this counts the number of times that elements of $Y$ outrank
elements of $X$.   For example, let us suppose that $X=\lbrace 1,3,4,5,6
\rbrace$ and $Y=\lbrace 2,7,8,10,11 \rbrace$.  We can get a this in one move
using Numpy broadcasting,

```python
import numpy as np
x = np.array([ 1,3,4,5,6 ])
y = np.array([2,7,8,10,11])
U_X = (y <= x[:,None]).sum()
U_Y = (x <= y[:,None]).sum()
print (U_X, U_Y)
```

    4 21

Note that

$$
U_X+U_Y =\sum_{i=1}^{n_x}\sum_{j=1}^{n_y} \mathbb{I}(Y_i\ge
X_j)+\mathbb{I}(X_i\ge Y_j)= n_x n_y
$$

 because $\mathbb{I}(Y_i\ge X_j)+\mathbb{I}(X_i\ge Y_j)=1$.  We
can verify this
in Python,

```python
print ((U_X+U_Y) == len(x)*len(y))
```

    True

Now that we can compute the $U_X$ statistic, we have to characterize it. Let us
consider $U_X$. If $H_0$ is true, then $X$ and $Y$ are identically distributed
random variables. Thus all $\binom{n_x+n_y}{n_x}$ allocations of the
$X$-variables in the ordered combined sample  are equally likely. Among these,
there are $\binom{n_x+n_y-1}{n_x}$ allocations have a $Y$ variable
as the
largest observation in the combined sample. For these, omitting this
largest
observation does not affect $U_X$ because it would not have been
counted anyway.
The other $\binom{n_x+n_y-1}{n_x-1}$ allocations have
an element of $X$ as the
largest observation. Omitting this observation
reduces $U_X$ by $n_y$.

With
all that, suppose $N_{n_x,n_y}(u)$ be the number of allocations of
$X$ and $Y$
elements that result in $U_X=u$. Under $H_0$ situation
of equally likely
outcomes, we have

$$
p_{n_x, n_y}(u) =
\mathbb{P}(U_X=u)=\frac{N_{n_x,n_y}(u)}{\binom{n_x+n_y}{n_x}}
$$

 From our previous discussion, we have the recursive relationship,

$$
N_{n_x,n_y}(u) = N_{n_x,n_y-1}(u) + N_{n_x-1,n_y}(u-n_y)
$$

 After dividing all of this by $\binom{n_x+n_y}{n_x}$ and using the
$p_{n_x,
n_y}(u)$ notation above, we obtain the following,

$$
p_{n_x, n_y}(u) = \frac{n_y}{n_x+n_y} p_{n_x,n_y-1}(u)+\frac{n_x}{n_x+n_y}
p_{n_x-1,n_y}(u-n_y)
$$

 where $0\le u\le n_x n_y$. To start this recursion, we need the
following
initial conditions,

$$
\begin{eqnarray*}
p_{0,n_y}(u_x=0) & = & 1 \\
p_{0,n_y}(u_x>0) & = & 0 \\
p_{n_x,0}(u_x=0) & = & 1 \\
p_{n_x,0}(u_x>0) & = & 0
\end{eqnarray*}
$$

 To see how this works in Python,

```python

def prob(n,m,u):
    if u<0: return 0
    if n==0 or m==0:
        return int(u==0)
    else:
        f = m/float(m+n)
        return (f*prob(n,m-1,u) +
                (1-f)*prob(n-1,m,u-m))
```

These are shown in [Figure](#fig:nonparametric_tests_002) and
approach a normal
distribution for large $n_x,n_y$, with the following
mean and variance,

$$
\begin{eqnarray}
\mathbb{E}(U) & = & \frac{n_x n_y}{2} \\
\mathbb{V}(U) & = &
\frac{n_x n_y (n_x+n_y+1)}{12}
\end{eqnarray}
\label{eq:ustatmv} \tag{4}
$$

 The variance becomes more complicated when there are ties.

```python
fig,axs=subplots(2,2)
fig.tight_layout()
ax=axs[0,0]
ax.tick_params(axis='both', which='major', labelsize=10)
_=ax.stem([prob(2,2,i) for i in range(2*2+1)],linefmt='k-',markerfmt='ko',basefmt='k-')
_=ax.set_title(r'$n_x=%d,n_y=%d$'%(2,2),fontsize=14)
ax=axs[0,1]
ax.tick_params(axis='both', which='major', labelsize=10)
_=ax.stem([prob(4,2,i) for i in range(4*2+1)],linefmt='k-',markerfmt='ko',basefmt='k-')
_=ax.set_title(r'$n_x=%d,n_y=%d$'%(4,2),fontsize=14)
ax=axs[1,0]
ax.tick_params(axis='both', which='major', labelsize=10)
_=ax.stem([prob(6,7,i) for i in range(6*7+1)],linefmt='k-',markerfmt='ko',basefmt='k-')
_=ax.set_title(r'$n_x=%d,n_y=%d$'%(6,7),fontsize=14)
ax=axs[1,1]
ax.tick_params(axis='both', which='major', labelsize=10)
_=ax.stem([prob(8,12,i) for i in range(8*12+1)],linefmt='k-',markerfmt='ko',basefmt='k-')
_=ax.set_title(r'$n_x=%d,n_y=%d$'%(8,12),fontsize=14)
fig.savefig('fig-statistics/nonparametric_tests_002.png')
```

The normal approximation to
the distribution improves with increasing $n_x, n_y$.

###
Example

We are trying to determine whether or not one network configuration is
faster
than another. We obtain the following round-trip times for each of the
networks.

```python

X=np.array([ 50.6,31.9,40.5,38.1,39.4,35.1,33.1,36.5,38.7,42.3 ])
Y=np.array([ 28.8,30.1,18.2,38.5,44.2,28.2,32.9,48.8,39.5,30.7 ])
```

Because there are too few elements to use the
`scipy.stats.mannwhitneyu`
function (which internally uses the normal
approximation to the U-statistic), we
can use our custom function above, but
first we need to compute the $U_X$
statistic using Numpy,

```python
U_X = (Y <= X[:,None]).sum()
```

For the p-value, we want to compute the probability that the observed
$U_X$
statistic at least as great as what was observed,

```python
print(sum(prob(10,10,i) for i in range(U_X,101)))
```

    0.08274697438784127

This is close to the usual five percent p-value threshold so it is
possible at
a slightly higher threshold to conclude that the two sets of
samples do *not*
originate from the same underlying distribution. Keep in mind
that the usual
five percent threshold is just a guideline. Ultimately, it is up
to the analyst
to make the call.

### Proving Mean and Variance for U-Statistic

To prove
Equation [4](#eq:ustatmv), we assume there are no ties.
One way to get at the
result $\mathbb{E}(U)= n_x n_y/2$,

$$
\mathbb{E}(U_Y) = \sum_j\sum_i\mathbb{P}(X_i \leq Y_j)
$$

 because $\mathbb{E}(\mathbb{I}(X_i\leq Y_j))=\mathbb{P}(X_i \leq
Y_j)$.
Further, because all the subscripted $X$ and $Y$ variables are drawn
independently from the same distribution, we have

$$
\mathbb{E}(U_Y) = n_x n_y \mathbb{P}(X \leq Y)
$$

 and also,

$$
\mathbb{P}(X \leq Y) + \mathbb{P}(X \ge Y) =1
$$

 because those are the two mutually exclusive conditions.  Because the
$X$
variables and $Y$ variables are drawn from the same distribution, we have
$\mathbb{P}(X \leq Y) = \mathbb{P}(X \ge Y)$, which means $ \mathbb{P}(X \leq
Y)=1/2$ and therefore $\mathbb{E}(U_Y)= n_x n_y /2$.  Another way to get the
same result, is to note that, as we showed earlier,  $U_X+U_Y = n_x n_y $.
Then,
taking the expectation of both sides noting that
$\mathbb{E}(U_X)=\mathbb{E}(U_Y)=\mathbb{E}(U)$, gives

$$
2 \mathbb{E}(U) = n_x n_y
$$

 which gives $\mathbb{E}(U)=n_x n_y /2$.

Getting the variance
is trickier. To
start, we compute the following,

$$
\mathbb{E}(U_X U_Y) = \sum_i \sum_j \sum_k \sum_l \mathbb{P}( X_i\ge Y_j
\land X_k \le Y_l )
$$

 Of these terms, we have $\mathbb{P}( Y_j \le X_i\le Y_j)=0$ because these
are
continuous random variables. Let's consider the terms of the following type,
$\mathbb{P}( Y_i \le X_k \le Y_l)$. To reduce the notational noise, let's
rewrite this as
$\mathbb{P}( Z \le X \le Y)$. Writing this out gives

$$
\mathbb{P}( Z \le X \le Y) = \int_{\mathbb{R}} \int_Z^\infty
(F(Y)-F(Z))f(y)f(z)dy dz
$$

 where $F$ is the cumulative density function and $f$ is the
probability density
function ($dF(x)/dx = f(x)$).  Let's break this up term by
term.  Using some
calculus for the term,

$$
\int_Z^\infty F(Y)f(y)dy = \int_{F(Z)}^1 F dF\ = \frac{1}{2}\left(1-F(Z)
\right)
$$

 Then, integrating out the $Z$ variable from this result, we obtain the
following,

$$
\int_{\mathbb{R}} \frac{1}{2}\left(1-\frac{F(Z)^2}{2}\right) f(z) dz =
\frac{1}{3}
$$

 Next, we  compute,

$$
\begin{eqnarray*}
\int_{\mathbb{R}} F(Z) \int_Z^\infty f(y) dy f(z) dz
&=&\int_{\mathbb{R}} (1-F(Z)) F(Z) f(z) dz \\
&=&\int_{\mathbb{R}} (1-F) F dF =\frac{1}{6}
\end{eqnarray*}
$$

 Finally, assembling the result, we have

$$
\mathbb{P}( Z \le X \le Y) = \frac{1}{3}- \frac{1}{6} = \frac{1}{6}
$$

 Also, terms like $\mathbb{P}(X_k\ge Y_i \land X_m \le Y_i) =
\mathbb{P}(X_m\le
Y_i \le X_k)=1/6$ by the same reasoning. That leaves the
terms like
$\mathbb{P}(X_k\ge Y_i\land X_m\le Y_l)=1/4$ because of mutual
independence  and
$\mathbb{P}(X_k\ge Y_i)=1/2$. Now that we have all the
terms, we have to
assemble the combinatorics to get the final answer.

There are $ n_y (n_y -1)
n_x  + n_x (n_x -1) n_y $ terms of type $\mathbb{P}(
Y_i \le X_k \le Y_l)$.
There are $ n_y (n_y -1) n_x (n_x -1)$ terms like
$\mathbb{P}(X_k\ge Y_i\land
X_m\le Y_l)$. Putting this all together,
this means that

$$
\mathbb{E}(U_X U_Y) = \frac{n_x n_y(n_x+n_y-2)}{6}+\frac{n_x
n_y(n_x-1)(n_y-1)}{4}
$$

 To assemble the $\mathbb{E}(U^2)$ result, we need to appeal to our earlier
result,

$$
U_X+U_Y = n_x n_y
$$

 Squaring both sides of this and taking the expectation gives,

$$
\mathbb{E}(U_X^2) + 2 \mathbb{E}(U_X U_Y)+\mathbb{E}(U_Y^2) = n_x^2 n_y^2
$$

 Because $\mathbb{E}(U_X^2)=\mathbb{E}(U_X^2)=\mathbb{E}(U)$, we
can simplify
this as the following,

$$
\begin{eqnarray*}
\mathbb{E}(U^2) &=& \frac{n_x^2 n_y^2 - 2 \mathbb{E}(U_X
U_Y)}{2}\\
\mathbb{E}(U^2) &=& \frac{n_x n_y (1+n_x +n_y +3 n_x n_y )}{12}
\end{eqnarray*}
$$

 Then, since $\mathbb{V}(U) = \mathbb{E}(U^2)- \mathbb{E}(U)^2$, we
finally have

$$
\mathbb{V}(U) = \frac{n_x n_y (1+ n_x +n_y )}{12}
$$


# Milk Collection Problem

## Objective and Prerequisites

In this example, you’ll discover how mathematical optimization can be leveraged to solve a capacitated vehicle routing problem: the Milk Collection Problem. With only one tanker truck with limited capacity, you will need to determine the best possible route for the tanker to take to collect milk every day from a set of farms. It’s a complicated problem to solve, but mathematical optimization will help show you the way!

This model is example 23 from the fifth edition of Model Building in Mathematical Programming by H. Paul Williams on pages 278-281 and 336-337.

This modeling example is at the advanced level, where we assume that you know Python and the Gurobi Python API and you have advanced knowledge of building mathematical optimization models. Typically, the objective function and/or constraints of these examples are complex or require advanced features of the Gurobi Python API

**Download the Repository** <br />
You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

---
## Problem Description

A small milk processing company is committed to collecting milk from 20 farms and taking it back to the depot for processing. The company has one tanker lorry with the capacity to carry 80 000 liters of milk. Eleven of the farms are small and need a collection only every other day. The other nine farms need a collection every day. The positions of the farms in relation to the depot (numbered 1) are given in the following table together with their collection requirements.

The goal is to find the optimal route for the tanker lorry on each day, bearing in mind that it has to:
1. Visit all the ‘every day’ farms.
2. Visit some of the ‘every other day’ farms.
3. Work within its capacity.

On alternate days, it must again visit the ‘every day’ farms and also visit the ‘every other day’ farms not visited on the
previous day.

---
## Model Formulation

### Sets and Indices
$i, j \in \text{Farms} = \{0,1,2, ..,20 \}$: Indices and set of farms. The depot index number is 0.

$\text{everyDay} = \{0,1,2, ..,9 \} \subset \text{Farms}$: Farms that need to visit every day.

$\text{otherDay} = \{10,11, 12, ..,20 \} \subset \text{Farms}$: Farms that need to visit every other day.

$k \in K = \{1,2 \} $: Day type for farms that are visited every other day.

$\text{Edges}= \{(i,j) \in Farms \times Farms \}$: Set of allowed Edges

$S_k \in S  $: Subtour in route of day $k$.

$G = (\text{Farms} , \text{Edges})$: A graph where the set $\text{Farms}$ defines the set of nodes and the set $\text{Edges}$ defines the set of edges.

### Parameters

$d_{i, j} \in \mathbb{R}^+$: Distance from farm $i$ to farm $j$, for all $(i, j) \in \text{Edges}$.

Notice that the distance from farm $i$ to farm $j$ is the same as the distance from farm $j$ to farm $i$, i.e. $d_{i, j} = d_{j, i}$.

$C \in \mathbb{R}^+$: The capacity of the tanker lorry.

$R_i \in \mathbb{R}^+$: Milk collection requirements of farm $i$.

### Decision Variables
$x_{i, j, k} \in \{0, 1\}$: This variable is equal to 1, if the tour on day $k$ connects directly farm $i$ with farm $j$. Otherwise, the decision variable is equal to zero.

$y_{i, k} \in \{0, 1\}$: This variable is equal to 1, if farm $i \in otherDay$ is visited on the tour of day $k \in K$.

### Objective Function
- **Shortest Route**. Minimize the total distance of both routes.

\begin{equation}
\text{Min} \quad Z = \sum_{k \in K} \sum_{(i,j) \in \text{Edges}} \frac{1}{2} d_{i,j} \cdot x_{i,j,k}
\tag{0}
\end{equation}

### Constraints
- **Symmetry Constraints**. For each edge $(i,j)$, ensure that the farm $i$ and $j$ are connected, if the former is visited immediately before or after visiting the latter.

\begin{equation}
x_{i, j, k} = x_{j, i, k} \quad \forall k \in dayType, \; (i, j) \in Edges
\tag{1}
\end{equation}

- **Entering and leaving an every day farm**. For each farm $i$, ensure that this farm is connected to two other farms.

\begin{equation}
\sum_{j: (i,j) \in \text{Edges}} x_{i,j,k} = 2 \quad \forall  i \in everyDay, \; k \in dayType
\tag{2}
\end{equation}

- **Entering and leaving an every other day farm**. For each farm $i$, ensure that this farm is connected to two other farms.

\begin{equation}
\sum_{j: (i,j) \in \text{Edges}} x_{i,j,k}  = 2 \cdot y_{i, k} \quad \forall  i \in otherDay, \; k \in dayType
\tag{3}
\end{equation}

- **Tanker capacity**. The tanker has limited capacity.
\begin{equation}
\sum_{i \in \text{otherDay}} R_{i} \cdot y_{i,k} \leq C -\sum_{i \in everyDay} R_{i} \quad \forall  k \in K
\tag{4}
\end{equation}

- **Farms visited**. Limit on visiting some farms only every other day.

\begin{equation}
y_{i,1} + y_{i,2}  = 1 \quad \forall  i \in \text{otherDay}
\tag{5}
\end{equation}

- **Subtour elimination**. These constraints ensure that for each route there is no cycle.

\begin{equation}
\sum_{(i,j) \in S_k}x_{i,j,k} \leq |S_k|-1 \quad \forall  k \in K, \;   S_k \in S
\tag{6}
\end{equation}

Where the subset $S$ is the set of farms visited by the tour, and this tour is defined by the values of the decision variables in the LHS of the inequality.

## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
%pip install gurobipy
```

```python
import gurobipy as gp
from gurobipy import GRB

import math
from itertools import combinations

# tested with Python 3.7.0 & Gurobi 9.1.0
```

### Input data
We define all the input data for the model.

```python
# Create a dictionary to capture the farm coordinates (10 miles) and collection requirements (1,000).

farms, coordinates, collect  = gp.multidict({
    0: [(0,0),0],
    1: [(-3,3),5],
    2: [(1,11),4],
    3: [(4,7),3],
    4: [(-5,9),6],
    5: [(-5,-2),7],
    6: [(-4,-7),3],
    7: [(6,0),4],
    8: [(3,-6),6],
    9: [(-1,-3),5],
    10: [(0,-6),4],
    11: [(6,4),7],
    12: [(2,5),3],
    13: [(-2,8),4],
    14: [(6,10),5],
    15: [(1,8),6],
    16: [(-3,1),8],
    17: [(-6,5),5],
    18: [(2,9),7],
    19: [(-6,-5),6],
    20: [(5,-4),6]
})

# List of farm  including the depot
farms = [*range(0,21)]

# List of farms that are visited everyday
everyDay = [*range(0,10)]

# List of farms that are visited each other day
otherDay = [*range(10,21)]

# List of day types
dayType = [1,2]

# Tanker lorry capacity (1,000)
tankerCap = 80

# Every day farms requirements
everyDayReq = 0
for i in everyDay:
    everyDayReq += collect[i]

```

### Data processing
Here, we calculate the distance for each pair of farms and store it in a Python dictionary.

```python
# Compute pairwise distance matrix
# numpy linalg norm = euclidean n=2

def distance(city1, city2):
    c1 = coordinates[city1]
    c2 = coordinates[city2]
    diff = (c1[0]-c2[0], c1[1]-c2[1])
    return math.sqrt(diff[0]*diff[0]+diff[1]*diff[1])

dist = {(c1, c2): distance(c1, c2) for c1, c2 in combinations(farms, 2)}
```

### Model Deployment
We now determine the capacitated vehicle routing model for the milk collection problem by defining decision variables, constraints, and objective function.

```python
# Create the model object m
m = gp.Model('MilkCollection.lp')

# Decision variables:

# Edge variables = 1, if farm 'i' is adjacent to farm 'j' on the tour of day type 'k'.
vars = m.addVars(dist, dayType, vtype=GRB.BINARY, name='x')

# Other day variables = 1, if farm 'i' is visited on the tour of day type 'k'.
other_var = m.addVars(otherDay, dayType, vtype=GRB.BINARY, name='y')

# Symmetry constraints: copy the object (not a constraint)
for i,j,k in vars.keys():
    vars[j, i, k] = vars[i, j, k]

# Every day constraints: two edges incident to an every day farm on tour of day type 'k'.
m.addConstrs((vars.sum(i,'*',k) == 2 for i in everyDay for k in dayType  ), name='everyDay')

# Other day constraints: two edges incident to an other day farm on tour of day type 'k'.
m.addConstrs((vars.sum(i,'*',k) == 2*other_var[i,k] for i in otherDay for k in dayType ), name='otherDay')

# Tanker capacity constraint.
m.addConstrs(( gp.quicksum(collect[i]*other_var[i,k] for i in otherDay ) <= tankerCap-everyDayReq for k in dayType ),
             name='tankerCap')

# Other day farms are visited on day type 1 or 2.
otherDayFarms = m.addConstrs((other_var.sum(i, '*') == 1 for i in otherDay), name='visited')

# Avoid symmetric alternative solutions
other_var[11,1].lb = 1

# Objective function: minimize total distance travel
m.setObjective(gp.quicksum(dist[i,j]*vars[i,j,k] for i,j in dist.keys() for k in dayType), GRB.MINIMIZE)
```

### Finding A Cycle
The following function determines a cycle not connected to the depot.

```python
# Find the edges from solution values, as a tuplelist for each dayType
def selected(vals):
    s = {k:gp.tuplelist() for k in dayType}
    for i, j, k in vals.keys():
        if vals[i,j,k] > 0.5:
            s[k].append((i,j))
    return s
# Alternately, using comprehension syntax:
#    return {k:gp.tuplelist((i, j) for i, j, k in vals.keys().select('*','*',k) if vals[i,j,k] > 0.5) for k in dayType}

# Given a tuplelist of edges, find the shortest subtour
def subtour(edges):
    nodes = set(i for e in edges for i in e)
    unvisited = list(nodes)
    cycle = list(nodes)
    while unvisited:
        thiscycle = []
        neighbors = unvisited
        while neighbors:
            current = neighbors[0]
            thiscycle.append(current)
            unvisited.remove(current)
            neighbors = [j for i, j in edges.select(current, '*')
                         if j in unvisited]
        if len(thiscycle) <= len(cycle): # even if it's the same, we reuse it so that we get the final tour in order
            cycle = thiscycle # New shortest subtour
    return cycle
```

### Callback Definition
Subtour constraints prevent multiple loops in a tour. Because there are an exponential number of these constraints, we don't want to add them all to the model. Instead, we use a callback function to find violated subtour constraints and add them to the model as lazy constraints.

```python
# Callback - use lazy constraints to eliminate sub-tours
def subtourelim(model, where):
    if where == GRB.Callback.MIPSOL:
        # get edges selected in the current solution
        vals = model.cbGetSolution(model._vars)
        edges = selected(vals)
        for k in dayType:
            tour = subtour(edges[k])
            if len(tour) < 0.5*len(edges[k]): # 0.5 due to symmetry: there are both edges i,j and j,i
                # add subtour elimination constr. for farms visited that day
                model.cbLazy(gp.quicksum(model._vars[i, j, k] for i, j in combinations(tour, 2))
                             <= len(tour)-1)
```

## Solve
We can now optimize the model with the lazy subtour constraints.

```python
# Optimize the model

m.reset()
m._vars = vars
m.Params.lazyConstraints = 1
m.optimize(subtourelim)
```

## Analysis

We print the optimal total distance traveled and the associated optimal routes for each day type.

```python
# Print optimal routes and distance traveled

print(f"The optimal distance traveled is: {10*round(m.objVal)} miles.")

vals = m.getAttr('X', vars)
edges = selected(vals)

for k in dayType:
    tour = subtour(edges[k])
    tour.append(0) # return to depot
    print ("Route for day type %i: %s" % (k, " -> ".join(map(str, tour))))
```

## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC