NYC-TAXI-EDA-FEATURE-ENGINEERING<br>
https://www.kaggle.com/frednavruzov/nyc-taxi-eda-feature-engineering

```python
import pandas as pd
import numpy as np
import sympy
import datetime as dt
import time
from math import *
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

from ipyleaflet import *
import folium
import json
import geopy.distance
from haversine import haversine
from tqdm import tqdm_notebook

sns.set()
%matplotlib inline
%config InlineBackend.figure_formats = {'png', 'retina'}

from matplotlib import font_manager, rc
plt.rcParams['axes.unicode_minus'] = False

import platform
if platform.system() == 'Darwin':
    rc('font', family='AppleGothic')
elif platform.system() == 'Windows':
    path = "c:/Windows/Fonts/malgun.ttf"
    font_name = font_manager.FontProperties(fname=path).get_name()
    rc('font', family=font_name)
```

```python
train = pd.read_csv("../dataset/train.csv") # 각자 데이터셋의 폴더
```

### data에 대한 기본적인 탐색

- traing data의 row 수와 column 수

```python
print("traing data의 row 수 : {}, column 수 : {}".format(train.shape[0], train.shape[1]))
```

```python
train.info()
```

```python
train.describe().round(2)
```

```python
# taxi['count'] = 1
```

# 전처리

- 아웃라이어 제거
1. 시간
    - pickup_datetime
2. 위치
3. 승객수

### 아웃라이어 제거
- 시간
- 승객수
- 위치

```python
# 운행 시간 0, 2시간 벗어나는 데이터 제거

# taxi = taxi[taxi["trip_duration"] >= 0]
# taxi = taxi[taxi["trip_duration"] <= 60*60*2]
```

```python
# 승객 수 0명 제거

# taxi = taxi[taxi["passenger_count"] != 0]
```

## pickup_datetime 살피기

```python
# pd.to_datetime을 해줘야 pickup_datetime coulmn의 data type이 datetime으로 됨
pickup_datetime_dt = pd.to_datetime(train["pickup_datetime"])
dropoff_datetime_dt = pd.to_datetime(train["dropoff_datetime"])
```

```python
train["pickup_datetime"] = pickup_datetime_dt
train["dropoff_datetime"] = dropoff_datetime_dt

train["pickup_date"] = train["pickup_datetime"].dt.date
train["dropoff_date"] = train["dropoff_datetime"].dt.date

train["pickup_month"] = train["pickup_datetime"].dt.month
train["dropoff_month"] = train["dropoff_datetime"].dt.month

train["pickup_weekday"] = train["pickup_datetime"].dt.weekday
train["dropoff_weekday"] = train["dropoff_datetime"].dt.weekday

train["pickup_hour"] = train["pickup_datetime"].dt.hour
train["dropoff_hour"] = train["dropoff_datetime"].dt.hour
```

```python
train.info()
```

```python
train.describe().round(2)
```

### EDA of pickup_datetime in train data
- year : 2016년
- month : 1~7월
- hour : 0~23시
- weekday : 월요일~일요일

```python
# # year
# print("데이터가 기록된 연도")
# print("가장 오래된 데이터의 기록 연도: {}년".format(taxi_df1["pickup_datetime"].dt.year.min()))
# print("가장 최신 데이터의 기록 연도: {}년".format(taxi_df1["pickup_datetime"].dt.year.max()))
# print('')

# # month
# print("데이터가 기록된 달")
# print("가장 오래된 데이터의 기록된 달: {}월".format(taxi_df1["pickup_month"].min()))
# print("가장 최신 데이터가 기록된 달: {}월".format(taxi_df1["pickup_month"].max()))
```

```python
train[train["trip_duration"] >= 60*60*24]
```

```python
train = train[train["trip_duration"] <= 60*60*24]
```

```python
plt.figure(figsize=(7, 5))
sns.distplot(train["trip_duration"], color="r")
plt.xlabel("Trip Duration")
plt.show()
```

```python
sns.set()
plt.figure(figsize=(7, 5))
sns.distplot(np.log(train['trip_duration']+1), color="r")
plt.xlabel("Log of Trip Duration")
plt.show()
```

```python
plt.figure(figsize=(7, 4))
sns.countplot(x="vendor_id", data=train, palette="husl")
plt.title("Vendor Distribution", fontsize=13)
plt.xlabel("Vendor")
plt.ylabel("Number of Trips")
plt.show()
```

```python
plt.figure(figsize=(7, 4))
sns.countplot(x="store_and_fwd_flag", data=train, palette="husl")
plt.title("Store & FWD Flag Distribution", fontsize=13)
plt.xlabel("Store & FWD Flag")
plt.ylabel("Number of Trips")
plt.show()
```

```python
plt.figure(figsize=(13, 4))

plt.subplot(121)
sns.countplot(x="pickup_month", data=train, palette="husl")
plt.title("Pickups Month Distribution", fontsize=13)
plt.xlabel("Pickup Months (January-June)")
plt.ylabel("Number of Trips")

plt.subplot(122)
sns.countplot(x="pickup_month", data=train, palette="husl", hue="vendor_id")
plt.title("Pickups Month Distribution", fontsize=13)
plt.xlabel("Pickup Months (January-June)")
plt.ylabel("Number of Trips")
plt.legend(loc=(1.04,0))

plt.show()
```

```python
plt.figure(figsize=(13, 4))

plt.subplot(121)
sns.countplot(x="pickup_weekday", data=train, palette="husl")
plt.title("Pickups Weekday Distribution", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Number of Trips")

plt.subplot(122)
sns.countplot(x="pickup_weekday", data=train, palette="husl", hue="vendor_id")
plt.title("Pickups Weekday Distribution", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Number of Trips")
plt.legend(loc=(1.04,0))

plt.show()
```

```python
plt.figure(figsize=(8, 4))
sns.countplot(x="pickup_hour", data=train, palette="husl")
plt.title("Pickups Hour Distribution", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Number of Trips")
plt.show()
```

```python
# Trip Duration의 경우, 상위 4개의 값이 워낙 큼. mean이 아닌 mode로 하면 차이는 더 많이 남)
# boxplot으로 보여주는 것도 좋을 것 같음 (단, log 처리하지 않으면 아웃라이어에 영향을 많이 받을 것)

data = train.loc[:, ["pickup_hour", "trip_duration"]].groupby("pickup_hour").mean()

plt.figure(figsize=(17, 4))

plt.subplot(121)
sns.barplot(x=data.index, y=data.trip_duration, data=data, palette="husl")
plt.title("Pickups Hours & Trip Duration", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Trip Duration")

plt.subplot(122)
sns.barplot(x=data.index, y=np.log(data.trip_duration+1), data=data, palette="husl")
plt.title("Pickups Hours & Log Trip Duration", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Log of Trip Duration")

plt.show()
```

```python
data = train.loc[:, ["pickup_weekday", "trip_duration"]].groupby("pickup_weekday").mean()

plt.figure(figsize=(17, 4))

plt.subplot(121)
sns.barplot(x=data.index, y=data.trip_duration, data=data, palette="husl")
plt.title("Pickups Weekday & Trip Duration", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Trip Duration")

plt.subplot(122)
sns.barplot(x=data.index, y=np.log(data.trip_duration+1), data=data, palette="husl")
plt.title("Pickups Weekday & Log Trip Duration", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Log of Trip Duration")

plt.show()
```

```python
plt.figure(figsize=(7, 5))
sns.boxplot(x=train["pickup_weekday"],
            y=train["trip_duration"].apply(np.log1p),
            data=train, palette="husl")
plt.title("Pickups Weekday & Log of Trip Duration", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Log of Trip Duration")

plt.show()
```

```python
data = train.loc[:, ["pickup_month", "trip_duration"]].groupby("pickup_month").mean()

plt.figure(figsize=(17, 4))

plt.subplot(121)
sns.barplot(x=data.index, y=data.trip_duration, data=data, palette="husl")
plt.title("Pickups Months & Trip Duration", fontsize=13)
plt.xlabel("Pickup Months (January-June)")
plt.ylabel("Trip Duration")

plt.subplot(122)
sns.barplot(x=data.index, y=np.log(data.trip_duration+1), data=data, palette="husl")
plt.title("Pickups Months & Log Trip Duration", fontsize=13)
plt.xlabel("Pickup Months (January-June)")
plt.ylabel("Log of Trip Duration")

plt.show()
```

```python
month = ["January", "February", "March", "April", "May", "June"]
weekday = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
working_day = [0, 1, 2, 3, 4] # Mon-Fri

wd = train.loc[:, ["pickup_weekday", "pickup_month", "trip_duration"]]
wd["working_day"] = wd["pickup_weekday"].isin(working_day)

plt.figure(figsize=(11, 5))
sns.violinplot(x=wd["pickup_month"],
               y=wd["trip_duration"].apply(np.log1p),
               hue="working_day",
               data=wd, palette="husl")
plt.title("Pickups Month & Log of Trip Duration", fontsize=13)
plt.xlabel("Pickup Month")
plt.ylabel("Log of Trip Duration")
plt.xticks(range(0, 6), month)
plt.show()
```

```python
plt.figure(figsize=(11, 7))
sns.countplot(x="pickup_weekday", data=train, hue="pickup_hour")
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Number of Trips")
plt.xticks(range(0,7), weekday)
plt.legend(loc=(1.04,0))
plt.show()
```

```python
plt.figure(figsize=(11,2))
sns.heatmap(data=pd.crosstab(train["pickup_weekday"],
                             train["pickup_hour"],
                             values=train["vendor_id"],
                             aggfunc="count",
                             normalize="index"), cmap="RdPu")

plt.title("Pickup Weekday vs. Hours", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Pickup Weekday")
plt.yticks(range(0,7), weekday, rotation="horizontal")
plt.show()
```

```python
plt.figure(figsize=(11,2))
sns.heatmap(data=pd.crosstab(train["pickup_month"],
                             train["pickup_hour"],
                             values=train["vendor_id"],
                             aggfunc="count",
                             normalize="index"), cmap="RdPu")

plt.title("Pickup Month vs. Hours", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Pickup Month")
plt.yticks(range(0,6), month, rotation="horizontal")
plt.show()
```

```python
plt.figure(figsize=(11,2))
sns.heatmap(data=pd.crosstab(train["pickup_month"],
                             train["pickup_weekday"],
                             values=train["vendor_id"],
                             aggfunc="count",
                             normalize="index"), cmap="RdPu")

plt.title("Pickup Month vs. Weekday", fontsize=13)
plt.xlabel("Pickup Weekday (Mon-Sun)")
plt.ylabel("Pickup Month")
plt.xticks(range(0,7), weekday, rotation=30)
plt.yticks(range(0,6), month, rotation="horizontal")
plt.show()
```

## 승객수

```python
train['passenger_count'].value_counts()
```

```python
print("가장 적은 탑승 인원: {}명".format(train["passenger_count"].min()))
print("가장 많은 탑승 인원: {}명".format(train["passenger_count"].max()))
```

```python
# 승객 수 0명 제거

# train = train[train["passenger_count"] != 0]
```

```python
plt.figure(figsize=(7, 5))
sns.countplot(x='passenger_count', data=train, palette="husl")
plt.title("Passenger Count Distribution", fontsize=13)
plt.xlabel("Passenger Count")
plt.ylabel("Number of Trips")
plt.show()
```

```python
# 아웃라이어 아직 제거하지 않아서 아웃라이어에 영향을 많이 받음

plt.figure(figsize=(17, 4))

plt.subplot(121)
sns.boxplot(x="passenger_count",
            y=train["trip_duration"],
            data=train,
            palette="husl")
plt.title("Passenger Count & Trip Duration", fontsize=13)
plt.xlabel("Passenger Count")
plt.ylabel("Trip Duration")

plt.subplot(122)
sns.boxplot(x="passenger_count",
            y=train["trip_duration"].apply(np.log1p),
            data=train,
            palette="husl")
plt.title("Passenger Count & Log Trip Duration", fontsize=13)
plt.xlabel("Passenger Count")
plt.ylabel("Log of Trip Duration")

plt.show()
```

```python
f, (ax1, ax2) = plt.subplots(ncols=2, sharey=True, figsize=(14, 4))

sns.boxplot(x="passenger_count",
            y=train["trip_duration"].apply(np.log1p),
            hue="vendor_id",
            data=train,
            palette="husl", ax=ax1)

ax1.set_xlabel(""); ax1.set_ylabel("Log of Trip Duration")

sns.boxplot(x="passenger_count",
            y=train["trip_duration"].apply(np.log1p),
            hue="store_and_fwd_flag",
            data=train,
            palette="husl", ax=ax2)
ax2.set_xlabel(""); ax2.set_ylabel("")

plt.suptitle("Passenger Count & Log of Trip Duration", y=1.05, fontsize=13)
plt.tight_layout()
f.text(0.5, -0.01, "Passenger Count", ha="center")

plt.show()
```

```python
plt.figure(figsize=(11,3))
sns.heatmap(data=pd.crosstab(train["passenger_count"],
                             train["pickup_hour"],
                             values=train["vendor_id"],
                             aggfunc="count",
                             normalize=False), cmap="RdPu")
plt.title("Passenger Count vs. Hours", fontsize=13)
plt.xlabel("Pickup Hours (0-23)")
plt.ylabel("Passenger Count")
plt.show()
```

```python
plt.figure(figsize=(11,3))
sns.heatmap(data=pd.crosstab(train["passenger_count"],
                             train["pickup_month"],
                             values=train["vendor_id"],
                             aggfunc="count",
                             normalize=False), cmap="RdPu")
plt.title("Passenger Count vs. Month", fontsize=13)
plt.xlabel("Pickup Month")
plt.ylabel("Passenger Count")
plt.xticks(range(0,6), month, rotation=30)
plt.show()
```

### EDA 결과
- 승객수 Max 9인, Min 0인 : 0인, 7인이상 삭제 검토 (총 65건)
- 승객수 지표 : 1인 승차객이 전체 약 146만 건 중 103만 건의 비중, 2인 승차객 21만건 / 5인 / 3인 순
- 승객수&승차시각 : 오전 8시 이후 1인 승차객 중심으로 전 시간대 고른 분포
- 승객수&승차요일 : 전반적으로 고른 분포 , 전반적으로 주말보다 수~금 승차건수가 높음
- 승객수&운행시간 : 1인 승객이 평균 1~2분정도 짧게 운행. 전반적으로 1000~1100초 (16분대), 탑승객 없는 택시 유일하게 30분 수준 (공항으로 이동?)

```python
import folium
import json
geo_path = '../dataset/geojson/state.geo.json'
geo_str = json.load(open(geo_path, encoding='utf-8'))
```

```python
# 뉴욕 시 경계를 벗어나는 위/경도 제거

# city_long_border = (-74.03, -73.75)
# city_lat_border = (40.63, 40.85)

train = train[train['pickup_longitude'] <= -73.75]
train = train[train['pickup_longitude'] >= -74.03]
train = train[train['pickup_latitude'] <= 40.85]
train = train[train['pickup_latitude'] >= 40.63]
train = train[train['dropoff_longitude'] <= -73.75]
train = train[train['dropoff_longitude'] >= -74.03]
train = train[train['dropoff_latitude'] <= 40.85]
train = train[train['dropoff_latitude'] >= 40.63]
```

```python
f, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(13,8))

train.plot(kind="scatter", x="pickup_longitude", y="pickup_latitude",
           color="yellow", grid=False, s=.02, alpha=.6, subplots=True, ax=ax1)

ax1.set_title("Pickups")
ax1.set_facecolor("black")

train.plot(kind="scatter", x="dropoff_longitude", y="dropoff_latitude",
           color="yellow", grid=False, s=.02, alpha=.6, subplots=True, ax=ax2)

ax2.set_title("Dropoffs")
ax2.set_facecolor("black")
```

```python
pickup_lat = tuple(train["pickup_latitude"])
pickup_lng = tuple(train["pickup_longitude"])
dropoff_lat = tuple(train["dropoff_latitude"])
dropoff_lng = tuple(train["dropoff_longitude"])
```

```python
pickup_loc = tuple(zip(pickup_lat, pickup_lng))
dropoff_loc = tuple(zip(dropoff_lat, dropoff_lng))
```

```python
print(len(pickup_loc))
print(len(dropoff_loc))
```

```python
# append가 아니라 전체 컬럼으로 바로
# vincenty distacne
import geopy.distance
from tqdm import tqdm_notebook

vincenty_distance = []
for i in tqdm_notebook(range(len(pickup_loc))):
    vincenty_distance.append(geopy.distance.vincenty(pickup_loc[i], dropoff_loc[i]).km)
```

```python
train.loc[:, "vincenty_distance"] = vincenty_distance
train.loc[:, ["id", "vincenty_distance"]].tail()
```

```python
train["log_duration"] = np.log1p(train["trip_duration"])
train["log_vincenty_distance"] = np.log1p(train["vincenty_distance"])
```

```python
plt.figure(figsize=(7, 5))
plt.scatter(train.log_vincenty_distance, train.log_duration, alpha=0.05)
plt.ylabel("log(Trip Duration)")
plt.xlabel("log(Vincenty Distance)")
plt.title("log(Vincenty Distance) vs log(Trip Duration)");
```

```python
sns.jointplot(x="log_vincenty_distance", y="log_duration", data=train);
```

```python
# taxi["pickup_datetime"] = pd.to_datetime(taxi.pickup_datetime)

# taxi.loc[:, "pickup_weekday"] = taxi["pickup_datetime"].dt.weekday
# taxi.loc[:, "pickup_hour_weekofyear"] = taxi["pickup_datetime"].dt.weekofyear
# taxi.loc[:, "pickup_hour"] = taxi["pickup_datetime"].dt.hour
# taxi.loc[:, "pickup_minute"] = taxi["pickup_datetime"].dt.minute
# taxi.loc[:, "pickup_dt"] = (taxi["pickup_datetime"] - taxi["pickup_datetime"].min()).dt.total_seconds()

```

```python
train.loc[:, "pickup_week_hour"] = train["pickup_weekday"] * 24 + train["pickup_hour"]
train.loc[:, "avg_speed_h"] = 1000 * train["vincenty_distance"] / train["trip_duration"]

fig, (ax1, ax2, ax3) = plt.subplots(ncols=3, sharey=True, figsize=(13, 6))

ax1.plot(train.groupby("pickup_hour").mean()["avg_speed_h"], 'bo-', lw=2, alpha=0.7)
ax2.plot(train.groupby("pickup_weekday").mean()["avg_speed_h"], 'go-', lw=2, alpha=0.7)
ax3.plot(train.groupby("pickup_week_hour").mean()["avg_speed_h"], 'ro-', lw=2, alpha=0.7)

ax1.set_xlabel("Hour")
ax2.set_xlabel("Weekday")
ax3.set_xlabel("Weekhour")

ax1.set_ylabel("Average Speed (km/h)")
fig.suptitle("Rush hour average traffic speed")
plt.show()
```

파라미터

- 이동 거리
- 시간대별

- 승/하차 위치 (맨하탄 내부 / 맨하탄 내외부(특히 공항))
- 스피드
- 승객 수

- 가장 큰 영향을 미치는 변수 확인 (아직 안 배움)
- 코드 깔끔히

### 속도와 duration간의 상관관계

```python
train_hour = train.loc[:, ["pickup_hour", "avg_speed_h", "trip_duration"]]
train_hour["trip_duration"] = train_hour["trip_duration"] / 60
train_hour = train_hour.groupby("pickup_hour").mean()
train_hour.tail()
```

```python
train_hour.plot(figsize=(7, 5))
plt.legend(loc='best')
plt.show()
```

```python
corr = train_hour.corr()
sns.heatmap(corr, cmap="RdPu");
```

```python
np.corrcoef(train_hour.avg_speed_h, train_hour.trip_duration)
```

```python
corr_matt = train[["pickup_datetime", "pickup_hour",
                   "pickup_month", "pickup_weekday",
                   "log_vincenty_distance", "log_duration"]]
```

```python
corr_matt.head()
```

```python
corr_matt = corr_matt.corr()

mask = np.array(corr_matt)
mask[np.tril_indices_from(mask)] = False
```

```python
corr_matt.head()
```

```python
fig, ax = plt.subplots()
fig.set_size_inches(14, 7)
sns.heatmap(corr_matt, mask=mask, vmax=.8, annot=True, square=True, cmap="RdPu")
plt.show()
```

```python
tmp = train[["trip_duration", "pickup_datetime"]]
tmp.head()
```


# Scenario E - Peakshape Variation (pVoigt model - results evaluation)

This file is used to evaluate the inference results.

In this scenario the peakshape in a spectrum with a fixed number of peaks is varied from Gaussian (n = 0.0) to Lorentzian (n = 1.0). All datasets contain 3 peaks and the noise level is kept constant at 1%.

The model used in the inference of the parameters is formulated as follows:

\begin{equation}
\large y = f(x) = \sum\limits_{m=1}^M \big[A_m \cdot f_{pseudo-Voigt}(x)\big] + \epsilon
\end{equation}

where:

\begin{equation}
\large f_{pseudo-Voigt}(x) = \eta \cdot \frac{\sigma_m^2}{(x-\mu_m)^2 + \sigma_m^2} + (1 - \eta) \cdot e^{-\frac{(x-\mu_m)^2}{2\cdot\sigma_m^2}}
\end{equation}

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pymc3 as pm
import arviz as az

#az.style.use('arviz-darkgrid')

print('Running on PyMC3 v{}'.format(pm.__version__))
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

    Running on PyMC3 v3.8

## Load results summary

```python
# load results from disk
#fname = './scenario_peakshape_pvoigt.csv'
fname = './scenario_peakshape_pvoigt_lrun.csv'
df = pd.read_csv(fname)
df.index += 1
df
```

      r_hat
      mcse
      ess
      bfmi
      r2
      waic
      epsilon
      epsilon_real
      eta
      eta_real
      cat

      1
      1.780909
      0.226636
      104.363636
      1.249257
      0.999856
      -4675.470979
      0.049972
      0.05
      0.005661
      0.0
      1

      2
      1.000000
      0.000000
      4383.363636
      1.029887
      0.999035
      -2331.742859
      0.109758
      0.05
      0.000332
      0.0
      1

      3
      1.070909
      0.049455
      684.363636
      1.014762
      0.999820
      -4207.406117
      0.059134
      0.05
      0.000384
      0.0
      1

      4
      1.005455
      0.003818
      1720.727273
      1.037991
      0.999818
      -4715.487126
      0.050034
      0.05
      0.008271
      0.0
      1

      5
      1.000000
      0.000727
      2124.454545
      1.044191
      0.999872
      -4726.544298
      0.049924
      0.05
      0.001852
      0.0
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      196
      1.000000
      0.000182
      2297.909091
      1.000964
      0.999846
      -4766.885947
      0.049235
      0.05
      0.998066
      1.0
      5

      197
      1.450909
      0.557818
      1605.181818
      1.008004
      0.999950
      -4836.525261
      0.048113
      0.05
      0.999041
      1.0
      5

      198
      1.000000
      0.000000
      4905.090909
      1.054635
      0.999424
      -2786.356407
      0.094399
      0.05
      0.977116
      1.0
      5

      199
      1.000000
      0.000000
      3922.636364
      0.997197
      0.999916
      -4743.426499
      0.049649
      0.05
      0.999503
      1.0
      5

      200
      1.000000
      0.000000
      3774.636364
      1.053122
      0.999882
      -4733.435338
      0.049827
      0.05
      0.999430
      1.0
      5

200 rows × 11 columns

```python
# number of succesfull convergences
suc = df.loc[(df['r_hat'] <= 1.1) & (df['r2'] >= 0.99)]
len(suc)
```

    189

```python
import seaborn as sns
sns.set(style="ticks", rc={'figure.figsize':(8,6)}, font_scale=1.35)

# color palette
pal = sns.cubehelix_palette(8, rot=-.5, dark=.3)

ax = sns.violinplot(x='eta_real', y='epsilon', data=suc, palette=pal, linewidth=2) \
                    .set_title("Peakshape eta (real) vs. Noise level (inferred)")

plt.savefig('violinplot_noise.png', dpi=150)
```

```python
#ax = sns.boxplot(x='eta_real', y='epsilon', data=suc, palette=pal, linewidth=2, fliersize=2) \
#                    .set_title("Peakshape eta (real) vs. Noise level (inferred)")
ax = sns.boxplot(x='eta_real', y='epsilon', data=suc, palette=sns.color_palette("Purples"),
                 linewidth=1, fliersize=2.5)

plt.savefig('boxplot_noise.png', dpi=150)
```

```python
ax = sns.violinplot(x='eta_real', y='eta', data=suc, palette=pal, linewidth=2) \
                    .set_title("Peakshape eta (real) vs. Peakshape eta (inferred)")
plt.savefig('violinplot_peakshape.png', dpi=150)
```

```python
#ax = sns.boxplot(x='eta_real', y='eta', data=suc, palette=pal, linewidth=2, fliersize=2) \
#                    .set_title("Peakshape eta (real) vs. Peakshape eta (inferred)")
#ax = sns.boxplot(x='eta_real', y='eta', data=suc, palette=pal, linewidth=2, fliersize=2)
ax = sns.boxplot(x='eta_real', y='eta', data=suc, palette=sns.color_palette("Purples"),
                 linewidth=1, fliersize=2.5)

plt.savefig('boxplot_peakshape.png', dpi=150)
```

```python
# table values
suc_000 = suc.loc[(suc['eta_real'] == 0.0)]
print("N = {0}".format(len(suc_000)))
suc_000['eta'].quantile([.25, .5, .75])
```

    N = 35

    0.25    0.000404
    0.50    0.000938
    0.75    0.001384
    Name: eta, dtype: float64

```python
suc_025 = suc.loc[(suc['eta_real'] == 0.25)]
print("N = {0}".format(len(suc_025)))
suc_025['eta'].quantile([.25, .5, .75])
```

    N = 39

    0.25    0.227337
    0.50    0.246641
    0.75    0.250805
    Name: eta, dtype: float64

```python
suc_050 = suc.loc[(suc['eta_real'] == 0.5)]
print("N = {0}".format(len(suc_050)))
suc_050['eta'].quantile([.25, .5, .75])
```

    N = 39

    0.25    0.489433
    0.50    0.499222
    0.75    0.500169
    Name: eta, dtype: float64

```python
suc_075 = suc.loc[(suc['eta_real'] == 0.75)]
print("N = {0}".format(len(suc_075)))
suc_075['eta'].quantile([.25, .5, .75])
```

    N = 38

    0.25    0.733126
    0.50    0.748641
    0.75    0.750410
    Name: eta, dtype: float64

```python
suc_100 = suc.loc[(suc['eta_real'] == 1.0)]
print("N = {0}".format(len(suc_100)))
suc_100['eta'].quantile([.25, .5, .75])
```

    N = 38

    0.25    0.990277
    0.50    0.997331
    0.75    0.998399
    Name: eta, dtype: float64

```python
ax = sns.scatterplot(x="eta", y="epsilon", data=suc, hue="eta_real", s=100,
                     palette="jet", legend="full")
plt.savefig('scatterplot_noise_peakshape.png', dpi=150)
```

##  Statistics per peak shape factor

```python
for i in [1,2,3,4,5]:
    cat = df.loc[(df['cat'] == i)]
    print('cat: {0} r-hat: {1:.2f}'.format(i, cat['r_hat'].mean()))
    print('cat: {0} r2   : {1:.4f}'.format(i, cat['r2'].mean()))
    print('cat: {0} waic : {1:.2f}'.format(i, cat['waic'].mean()))
    print('cat: {0} mcse : {1:.4f}'.format(i, cat['mcse'].mean()))
    print('cat: {0} ess  : {1:.1f}'.format(i, cat['ess'].mean()))
    print('cat: {0} bfmi : {1:.4f}'.format(i, cat['bfmi'].mean()))
    print('cat: {0} epsi : {1:.4f}'.format(i, cat['epsilon'].mean()))
    print('cat: {0} eta  : {1:.4f}'.format(i, cat['eta'].mean()))
    print('\n')
```

    cat: 1 r-hat: 1.08
    cat: 1 r2   : 0.9997
    cat: 1 waic : -4342.65
    cat: 1 mcse : 0.0823
    cat: 1 ess  : 3298.4
    cat: 1 bfmi : 1.0416
    cat: 1 epsi : 0.0585
    cat: 1 eta  : 0.0014

    cat: 2 r-hat: 1.01
    cat: 2 r2   : 0.9995
    cat: 2 waic : -3972.92
    cat: 2 mcse : 0.0267
    cat: 2 ess  : 3879.6
    cat: 2 bfmi : 1.0516
    cat: 2 epsi : 0.0762
    cat: 2 eta  : 0.2313

    cat: 3 r-hat: 1.01
    cat: 3 r2   : 0.9995
    cat: 3 waic : -4069.55
    cat: 3 mcse : 0.0199
    cat: 3 ess  : 3676.4
    cat: 3 bfmi : 1.0493
    cat: 3 epsi : 0.0713
    cat: 3 eta  : 0.4816

    cat: 4 r-hat: 1.00
    cat: 4 r2   : 0.9988
    cat: 4 waic : -3637.68
    cat: 4 mcse : 0.0039
    cat: 4 ess  : 3647.7
    cat: 4 bfmi : 1.0568
    cat: 4 epsi : 0.0971
    cat: 4 eta  : 0.7067

    cat: 5 r-hat: 1.02
    cat: 5 r2   : 0.9996
    cat: 5 waic : -4145.78
    cat: 5 mcse : 0.0217
    cat: 5 ess  : 3287.5
    cat: 5 bfmi : 1.0257
    cat: 5 epsi : 0.0649
    cat: 5 eta  : 0.9747

```python

```


## Exercise 1 (4 points): Idea of Maximal Margin (SVM) [Pen and Paper]

With the help of support vector machines, we can find a separation line which maximize the margin between two classes. In this exercise, we use a two-class example and approach to such a line in a geometric way. Figure 1 shows the two-dimensional data we are using throughout this exercise. Each point 𝑥 ∈ ℝ2 belongs either to the positive 𝜔1 (blue, circles) or negative class 𝜔−1 (orange, squares).

1. To find the separation line, the convex hull of a dataset is helpful. Start by drawing the convex hull for each class in Figure 1.

2. The line with the maximal margin is defined by the two points from the convex hulls of the two classes with minimal distance to each other. Find these two points on the convex hulls you have just drawn and label them with 𝑐1 and 𝑐-1 (for the class 𝜔1 and 𝜔-1, respectively). Note that every point on a convex hull is a possible candidate and they do not necessarily need to correspond with the data points.

3. Connect the points 𝑐1 and 𝑐-1 with a line.

4. The separation line goes through the centre of the line you have just drawn and is orthogonal to it, i.e. the two lines enclose a 90° angle. Draw the separation line 𝑠, the line 𝑙1 which passes through the support vectors of 𝜔1 and the line 𝑙-1 which passes through the support vectors of 𝜔-1.

5. Add two arbitrary points from each class, i.e. x1, 𝑥2 ∈ $\omega$𝜔1 and 𝑥3, 𝑥4 ∈ 𝜔-1, to the feature space so that the separation line 𝑠 found previously does not change.

6. Start fresh with Figure 2 and add a new data point 𝑥5 which belongs to a class of your choice to the feature space so that the new margin between the two classes equals 1. As before, sketch the three lines $\tilde{l}$1 , $\tilde{l}$-1 and $\tilde{s}$ based on the points $\tilde{c}$1 and $\tilde{l}$-1 on the convex hull.

7. In general, what is the difference between a line obtained via SVM and a line obtained via the perceptron learning algorithm?

## Exercise 2 (6 points): SVM [Pen and Paper]

From the previous exercise, we know which kind of separation line we can expect when using support vector machines and how to find it graphically. Determining the separation line analytically, however, is very hard; even for the simple problem of Exercise 1. The number of variables and conditions make this task impractical for a “Pen and Paper” exercise. But, to still get a basic idea of the SVM algorithm, we consider here an even simpler problem where we have only two points $x_i \in \mathbb{R}^{2}$ which belong to their own class each. The first point is from the set of positive samples $\omega_1 = \{x_i | T_i = 1\}$ and the second from the set of negative samples $\omega_{-1} = \{x_i | T_i = −1\}$. The two points are defined as
$$ x_1 =
\begin{pmatrix}
1\\
1\\
\end{pmatrix} \in \omega_1
\hspace{0.5cm}and\hspace{0.5cm}
\begin{pmatrix}
2\\
3\\
\end{pmatrix} \in \omega_{-1}
$$

Obviously, both points are also support vectors since they are the only representatives from their classes and hence must lie on the margin. Our goal is to find the parameters of a separation line with maximal margin, i.e. we search for 𝑤 and 𝑤0. In order to find our parameters, we need to optimize the Lagrange function
\begin{equation}
L(w, w_0, \alpha_1, \alpha_2) = L(\alpha_1, \alpha_2) = \sum_{i=1}^2 \alpha_i - \frac{1}{2}\sum_{i=1}^2\sum_{j=1}^2\alpha_i\alpha_jT_iT_jx_i^Tx_j
\end{equation}
subject to the constraint
$$\sum_{i=1}^2\alpha_iT_i=0$$

Technically, we also need the constraints $\alpha_i \geq 0$. But to keep things simple, we are assuming them satisfied here.
1. Set up the Lagrange function (Equation 1) as well as the constraint (Equation 2) of our problem set, i.e. plug in the values and simplify.

2. To optimize Equation 1 under the constraint of Equation 2, we need an additional Lagrange function with the Lagrange multiplier 𝜆
\begin{equation} \Lambda(\alpha_1, \alpha_2, \lambda) = L(\alpha_1, \alpha_2) + \lambda\left(\sum_{i=1}^2\alpha_iT_i\right) \end{equation}
Optimize this function and show that the optimal values are given as
$$\alpha_1^* = \frac{2}{5} \hspace{0.5cm} and \hspace{0.5cm} \alpha_2^* = \frac{2}{5} $$
It is sufficient to calculate only the extrema values. You do not need to show that they belong to a minimum/maximum (and not a saddle point) of $L(\alpha_1, \alpha_2)$. You can use Figure 3 to convince yourself that this is indeed the case here.

3. Based on the previous results, calculate the line parameters 𝑤 and 𝑤0.

4. Draw the resulting separation line into Figure 4.

5. Suppose you want to classify the new point 𝑥3 = (2, 2). Assign this point the correct label

    a) graphically based on the line you have just drawn into Figure 4.

    b) analytically by using the decision function 𝑓(𝑥) (script page 102).

6. Some general questions to conclude this exercise (no calculations required):

    a) How would Equation 1 change if we had more than three points?

    b) How can we infer from the coefficient 𝛼∗ 𝑖 whether the corresponding point 𝑥𝑖 contributes to the separation line (i.e. whether 𝑥𝑖 is a support vector)?


# Two modifications of  mean-variance portfolio theory

#### Daniel Csaba, Thomas J. Sargent and Balint Szoke

#### December 2016

### Remarks about estimating means and variances

The famous **Black-Litterman** (1992) portfolio choice model that we describe in this notebook is motivated by the finding that with high or moderate frequency data, means are more difficult to estimate than variances.

A model of **robust portfolio choice** that we'll describe also begins from the same
starting point.

To begin, we'll take for granted that means are more difficult to estimate that covariances and will focus on how  Black and Litterman, on the one hand, an robust control theorists, on the other, would recommend modifying the **mean-variance portfolio choice model** to take that into account.

At the end of this notebook, we shall  use some rates of convergence results and some simulations to verify how means are more difficult to estimate than variances.

Among the ideas in play in this notebook will be

   * mean-variance portfolio theory

   * Bayesian approaches to estimating linear regressions

   * A risk-sensitivity operator and its connection to robust control theory

```python
%matplotlib inline

import numpy as np
import scipy as sp
import scipy.stats as stat
import matplotlib.pyplot as plt
from ipywidgets import interact, FloatSlider

```

## Adjusting mean-variance portfolio choice theory for distrust of mean excess returns

This notebook describes two lines of thought that modify the classic mean-variance
portfolio choice model in ways designed to make its recommendations more plausible.

As we mentioned above, the two approaches build on a common hunch -- that because it is much easier statistically to estimate covariances of excess returns than it is to estimate their means, it makes sense to contemplated the consequences of adjusting investors'
subjective beliefs about mean returns in order to render more sensible decisions.

Both of the adjustments that we describe are designed to confront a  widely recognized  embarassment  to mean-variance portfolio theory, namely, that it usually implies taking
very extreme long-short portfolio positions.

##### Mean-variance portfolio choice

A risk free security earns one-period net return $r_f$.

An $n \times 1$ vector  of risky securities earns an $n \times 1$
vector $\vec r - r_f {\bf 1}$ of *excess returns*, where ${\bf 1}$ is
an $n \times 1$ vector of ones.

The excess return vector is multivariate normal with mean $\mu$ and covariance
matrix $\Sigma$, which we express either as

$$ \vec r - r_f {\bf 1} \sim {\mathcal N}(\mu, \Sigma) $$

or

$$ \vec r - r_f {\bf 1} = \mu + C \epsilon  $$

where $\epsilon \sim {\mathcal N}(0, I)$ is an $n \times 1 $ random vector.

Let $w $ be an $n\times 1$ vector of portfolio weights.

A portfolio consisting $w$ earns returns

$$ w' (\vec r - r_f {\bf 1}) \sim {\mathcal N}(w' \mu, w' \Sigma w ) $$

The **mean-variance portfolio choice problem** is to choose $w$ to maximize

$$ U(\mu,\Sigma;w) = w'\mu - \frac{\delta}{2} w' \Sigma w   ,  \quad (1) $$

where $\delta > 0 $ is a risk-aversion parameter.  The first-order condition for
maximizing (1) with respect to the vector $w$ is

$$ \mu = \delta \Sigma w ,  \quad (2) $$

which implies the following design of a risky portfolio:

$$ w = (\delta \Sigma)^{-1} \mu , \quad (3)  $$

##### Estimating $\mu$ and $\Sigma$

The key inputs into the portfolio choice model (3) are

   * estimates of the parameters $\mu, \Sigma$ of the random excess return vector $(\vec r - r_f {\bf 1})$

   * the risk-aversion parameter $\delta$

A standard way of estimating $\mu$ is maximum-likelihood or least squares; that
amounts to estimating $\mu$ by a sample mean of excess returns and estimating $\Sigma$
by a sample covariance matrix.

###### The Black-Litterman starting point

When estimates of $\mu$ and $\Sigma$ from historical sample means and covariances
have been combined with ``reasonable'' values of the risk-aversion parameter $\delta$
to compute an optimal portfolio from formula (3), a typical outcome has been
$w$'s with **extreme long and short positions**.  A common reaction to these outcomes
is that they are so unreasonable that a portfolio manager cannot recommend them to a
customer.

```python
#========================================
# Primitives of the laboratory
#========================================
np.random.seed(12)

N = 10                                           # number of assets
T = 200                                          # sample size

# random market portfolio (sum is normalized to 1)
w_m = np.random.rand(N)
w_m = w_m/(w_m.sum())

# True risk premia and variance of excess return
#          (constructed so that the Sharpe ratio is 1)
Mu = (np.random.randn(N) + 5)/100                   # mean excess return (risk premium)
S = np.random.randn(N, N)                           # random matrix for the covariance matrix
V = S @ S.T                                         # turn the random matrix into symmetric psd
Sigma = V * (w_m @ Mu)**2 / (w_m @ V @ w_m)         # make sure that the Sharpe ratio is one

# Risk aversion of market portfolio holder
delta = 1 / np.sqrt(w_m @ Sigma @ w_m)

# Generate a sample of excess returns
excess_return = stat.multivariate_normal(Mu, Sigma)
sample = excess_return.rvs(T)
```

```python
#==================================
# Mean-variance portfolio
#==================================

# Estimate mu and sigma
Mu_est = sample.mean(0).reshape(N, 1)
Sigma_est = np.cov(sample.T)

# Solve the constrained problem for the weights (with iota @ w = 1)
#iota = np.ones((N, 1))                                    # column vector of ones
#def lamb(mu, sigma, delta):
#    aux_vector = iota.T @ np.linalg.inv(delta * sigma)    # save memory
#    return ((1 - aux_vector @ mu) / (aux_vector @ iota))[0]    # lagrange multiplier on the const
#w = np.linalg.solve(d_m * sigma_hat, mu_hat) + lamb(mu_hat, sigma_hat, delta) * iota)

w = np.linalg.solve(delta * Sigma_est, Mu_est)

fig, ax = plt.subplots(figsize = (8, 5))
ax.set_title('Mean-variance portfolio weights recommendation and the market portfolio ', fontsize = 12)
ax.plot(np.arange(N) + 1, w, 'o', color = 'k', label = '$w$ (mean-variance)')
ax.plot(np.arange(N) + 1, w_m, 'o', color = 'r', label = '$w_m$ (market portfolio)')
ax.vlines(np.arange(N) + 1, 0, w, lw = 1)
ax.vlines(np.arange(N) + 1, 0, w_m, lw = 1)
ax.axhline(0, color = 'k')
ax.axhline(-1, color = 'k', linestyle = '--')
ax.axhline(1, color = 'k', linestyle = '--')
ax.set_xlim([0, N+1])
ax.set_xlabel('Assets', fontsize = 11)
ax.xaxis.set_ticks(np.arange(1, N + 1, 1))
plt.legend(numpoints = 1, loc = 'best', fontsize = 11)
plt.show()
```

Black and Litterman's  responded  to this situation in the following way:

  * They continue to accept (3) as a good model for choosing an optimal portfolio $w$

  * They want to continue to allow the customer to express his or her risk tolerance by
setting $\delta$

  * Leaving $\Sigma$ at its maximum-likelihood value, they  push  $\mu$ away from its maximum value in a way designed to make portfolio choices that are more plausible in terms of conforming to what most people actually do.

In particular, given $\Sigma$ and a reasonable value of $\delta$, Black and Litterman
reverse engineered a vector $\mu_{BL}$ of mean excess returns that makes the $w$ implied
by formula (3) equal the **actual** market portfolio $w_m$, so that

$$ w_m = (\delta \Sigma)^{-1} \mu_{BL} \quad (4) $$

##### Details

Let's define

$$ w_m' \mu \equiv ( r_m - r_f)  $$

as the (scalar) excess return on the market portfolio $w_m$.

Define

$$ \sigma^2 = w_m' \Sigma w_m  $$

as the variance of the excess return on the market portfolio $w_m$.

Define

$$ {\bf SR}_m = \frac{ r_m - r_f}{\sigma}  $$

as the **Sharpe-ratio** on the market portfolio $w_m$.

Let $\delta_m$ be the value of the risk aversion parameter that induces  an investor
to hold the market portfolio in light of the optimal portfolio choice rule (3).

Evidently, portfolio rule  (3) then implies that  $ r_m - r_f = \delta_m \sigma^2 $
or

$$ \delta_m = \frac{r_m - r_f}{\sigma^2}  $$

or

$$ \delta_m = \frac{\bf SR}{\sigma}, \quad (5) $$

Following the Black-Litterman philosophy, our first step will be to back a value
of $\delta_m$ from

  * an estimate of the Sharpe-ratio, and

  * our maximum likelihood estimate of $\sigma$ drawn from  our estimates or $w_m$ and $\Sigma$

The second key Black-Litterman step is then to use this value of $\delta$ together
with the maximum likelihood estimate of $\Sigma$ to deduce a $\mu_{\bf BL}$ that
verifies portfolio rule (3) at the market portfolio $w = w_m$:

$$ \mu_m = \delta_m \Sigma w_m   $$

The starting point of the Black-Litterman portfolio choice model is thus
a pair $(\delta_m, \mu_m) $ that tells the customer to hold the market portfolio.

```python
# =====================================================
# Derivation of Black-Litterman pair (\delta_m, \mu_m)
# =====================================================

# Observed mean excess market return
r_m = w_m @ Mu_est

# Estimated variance of market portfolio
sigma_m = w_m @ Sigma_est @ w_m

# Sharpe-ratio
SR_m = r_m/np.sqrt(sigma_m)

# Risk aversion of market portfolio holder
d_m = r_m/sigma_m

# Derive "view" which would induce market portfolio
mu_m = (d_m * Sigma_est @ w_m).reshape(N, 1)

fig, ax = plt.subplots(figsize = (8, 5))
ax.set_title(r'Difference between $\hat{\mu}$ (estimate) and $\mu_{BL}$ (market implied)', fontsize = 12)
ax.plot(np.arange(N) + 1, Mu_est, 'o', color = 'k', label = '$\hat{\mu}$')
ax.plot(np.arange(N) + 1, mu_m, 'o', color = 'r', label = '$\mu_{BL}$')
ax.vlines(np.arange(N) + 1, mu_m, Mu_est, lw = 1)
ax.axhline(0, color = 'k', linestyle = '--')
ax.set_xlim([0, N + 1])
ax.set_xlabel('Assets', fontsize = 11)
ax.xaxis.set_ticks(np.arange(1, N + 1, 1))
plt.legend(numpoints = 1, loc = 'best', fontsize = 11)
plt.show()
```

##### Adding ``views''

Black and Litterman start with  a baseline customer who asserts that he or she shares
the ``market's views'', which means that his or her believes that excess returns
are governed by

 $$ \vec r - r_f {\bf 1} \sim {\mathcal N}( \mu_{BL}, \Sigma)   , \quad (6) $$

Black and Litterman would advise that customer to hold the market portfolio of risky securities.

Black and Litterman then imagine a consumer who would like to express a view that
differs from the market's.  The consumer wants appropriately to mix his view with
the market's before using (3) to choose a portfolio.

Suppose that the customer's view is expressed by a hunch that rather than (6),
excess returns are governed  by

$$ \vec r - r_f {\bf 1} \sim {\mathcal N}( \hat \mu, \tau \Sigma)   , \quad (7) $$

where $\tau > 0$ is a scalar parameter that determines how the decision maker
wants to mix his view $\hat \mu$ with the market's view $\mu_{\bf BL}$.

Black and Litterman would then use a formula like the following one to mix the
views $\hat \mu$ and $\mu_{\bf BL}$:

$$\tilde \mu = (\Sigma^{-1} + (\tau \Sigma)^{-1})^{-1} (\Sigma^{-1} \mu_{BL}  + (\tau \Sigma)^{-1} \hat \mu)  , \quad (8) $$

Black and Litterman would then advice the customer to hold the portfolio associated
with these views implied by rule (3):

$$ \tilde w = (\delta \Sigma)^{-1} \tilde \mu , \quad (9)  $$

This portfolio $\tilde w$ will deviate from the portfolio $w_{BL}$ in amounts that
depend on the mixing parameter $\tau$.

If $\hat \mu$ is the maximum likelihood estimator and $\tau$ is chosen heavily to
weight this view, then the customer's portfolio will involve big short-long positions.

```python
def Black_Litterman(lamb, mu_1, mu_2, Sigma_1, Sigma_2):
    """
    This function calculates the Black-Litterman mixture
    mean excess return and covariance matrix
    """
    sigma1_inv = np.linalg.inv(Sigma_1)
    sigma2_inv = np.linalg.inv(Sigma_2)

    mu_tilde = np.linalg.solve(sigma1_inv + lamb * sigma2_inv,
                               sigma1_inv @ mu_1 + lamb * sigma2_inv @ mu_2)
    return mu_tilde

#===================================
# Example cont'
#     mean : MLE mean
#     cov  : scaled MLE cov matrix
#===================================

tau = 1
mu_tilde = Black_Litterman(1, mu_m, Mu_est, Sigma_est, tau * Sigma_est)

# The Black-Litterman recommendation for the portfolio weights
w_tilde = np.linalg.solve(delta * Sigma_est, mu_tilde)
```

```python
tau_slider = FloatSlider(min = 0.05, max = 10, step = 0.5, value = tau)

@interact(tau = tau_slider)
def BL_plot(tau):
    mu_tilde = Black_Litterman(1, mu_m, Mu_est, Sigma_est, tau * Sigma_est)
    w_tilde = np.linalg.solve(delta * Sigma_est, mu_tilde)

    fig, ax = plt.subplots(1, 2, figsize = (16, 6))
    ax[0].set_title(r'Relationship between $\hat{\mu}$, $\mu_{BL}$ and $\tilde{\mu}$', fontsize = 15)
    ax[0].plot(np.arange(N) + 1, Mu_est, 'o', color = 'k', label = r'$\hat{\mu}$ (subj view)')
    ax[0].plot(np.arange(N) + 1, mu_m, 'o', color = 'r', label = r'$\mu_{BL}$ (market)')
    ax[0].plot(np.arange(N) + 1, mu_tilde, 'o', color = 'y', label = r'$\tilde{\mu}$ (mixture)')
    ax[0].vlines(np.arange(N) + 1, mu_m, Mu_est, lw = 1)
    ax[0].axhline(0, color = 'k', linestyle = '--')
    ax[0].set_xlim([0, N + 1])
    ax[0].xaxis.set_ticks(np.arange(1, N + 1, 1))
    ax[0].set_xlabel('Assets', fontsize = 14)
    ax[0].legend(numpoints = 1, loc = 'best', fontsize = 13)

    ax[1].set_title('Black-Litterman portfolio weight recommendation', fontsize = 15)
    ax[1].plot(np.arange(N) + 1, w, 'o', color = 'k', label = r'$w$ (mean-variance)')
    ax[1].plot(np.arange(N) + 1, w_m, 'o', color = 'r', label = r'$w_{m}$ (market, BL)')
    ax[1].plot(np.arange(N) + 1, w_tilde, 'o', color = 'y', label = r'$\tilde{w}$ (mixture)')
    ax[1].vlines(np.arange(N) + 1, 0, w, lw = 1)
    ax[1].vlines(np.arange(N) + 1, 0, w_m, lw = 1)
    ax[1].axhline(0, color = 'k')
    ax[1].axhline(-1, color = 'k', linestyle = '--')
    ax[1].axhline(1, color = 'k', linestyle = '--')
    ax[1].set_xlim([0, N + 1])
    ax[1].set_xlabel('Assets', fontsize = 14)
    ax[1].xaxis.set_ticks(np.arange(1, N + 1, 1))
    ax[1].legend(numpoints = 1, loc = 'best', fontsize = 13)

    plt.show()
```

### Bayes interpretation of the Black-Litterman recommendation

Consider the following Bayesian interpretation of the Black-Litterman recommendation.

The prior belief over the mean excess returns is consistent with the market porfolio and is given by

$$ \mu \sim \mathcal{N}(\mu_{BL}, \Sigma).$$

Given a particular realization of the mean excess returns $\mu$ one observes the average excess returns $\hat \mu$ on the market according to the distribution

$$\hat \mu \mid \mu, \Sigma \sim \mathcal{N}(\mu, \tau\Sigma).$$

where $\tau$ is typically small capturing the idea that the variation in the mean is smaller than the variation of the individual random variable.

Given the realized excess returns one should then update the prior over the mean excess returns according to Bayes rule. The corresponding posterior over mean excess returns is normally distributed with mean
$$ (\Sigma^{-1} + (\tau \Sigma)^{-1})^{-1} (\Sigma^{-1}\mu_{BL}   + (\tau \Sigma)^{-1} \hat \mu)$$

The covariance matrix is

$$ (\Sigma^{-1} + (\tau \Sigma)^{-1})^{-1}.$$

Hence, the Black-Litterman recommendation is consistent with the Bayes update of the prior over the mean excess returns in light of the realized average excess returns on the market.

### Curve Decolletage

Consider two independent "competing" views on the excess market returns.

 $$ \vec r_e  \sim {\mathcal N}( \mu_{BL}, \Sigma) $$

and

 $$ \vec r_e \sim {\mathcal N}( \hat{\mu}, \tau\Sigma). $$

A special feature of the multivariate normal random variable $Z$ is that its density function depends only on the (Euclidiean) length of its realization $z$. Formally, let the $k$-dimensional random vector be $Z\sim \mathcal{N}(\mu, \Sigma)$, then $\bar{Z} \equiv \Sigma(Z-\mu)\sim \mathcal{N}(\mathbf{0}, I)$ and so the points where the density takes the same value can be described by the ellipse

$$\bar z \cdot \bar z =  (z - \mu)'\Sigma^{-1}(z - \mu) = \bar d \quad \quad (10)$$

where $\bar d\in\mathbb{R}_+$ denotes the (transformation) of a particular density value. The curves defined by equation (10) can be labelled as iso-likelihood ellipses.

> **Remark:** More generally there is a class of density functions that possesses this feature, i.e.

> $$\exists g: \mathbb{R}_+ \mapsto \mathbb{R}_+ \ \ \text{ and } \ \ c \geq 0, \ \ \text{s.t.  the density } \ \ f \ \ \text{of} \ \ Z  \ \ \text{ has the form } \quad f(z) = c g(z\cdot z)$$

> This property is called **spherical symmetry** (see p 81. in Leamer (1978)).

In our specific example, we can use the pair $(\bar d_1, \bar d_2)$ as being two "likelihood" values for which the corresponding isolikelihood ellipses in the excess return space are given by

\begin{align}
(\vec r_e - \mu_{BL})'\Sigma^{-1}(\vec r_e - \mu_{BL}) &= \bar d_1 \\
(\vec r_e - \hat \mu)'\left(\tau \Sigma\right)^{-1}(\vec r_e - \hat \mu) &= \bar d_2
\end{align}

Notice that for particular $\bar d_1$ and $\bar d_2$ values the two ellipses have a tangency point. These tangency points, indexed by the pairs $(\bar d_1, \bar d_2)$, characterize points $\vec r_e$ from which there exists no deviation where one can increase the likelihood of one view without decreasing the likelihood of the other view. The pairs $(\bar d_1, \bar d_2)$ for which there is such a point outlines a curve in the excess return space. This curve is reminiscent of the Pareto curve in an Edgeworth-box setting.

Leamer (1978) calls this curve *information contract curve* and describes it by the following program: maximize the likelihood of one view, say the Black-Litterman recommendation, while keeping the likelihood of the other view at least at a prespecified constant $\bar d_2$.

\begin{align*}
 \bar d_1(\bar d_2) &\equiv \max_{\vec r_e} \ \ (\vec r_e - \mu_{BL})'\Sigma^{-1}(\vec r_e - \mu_{BL}) \\
\text{subject to }  \quad  &(\vec r_e - \hat\mu)'(\tau\Sigma)^{-1}(\vec r_e - \hat \mu) \geq \bar d_2
\end{align*}

Denoting the multiplier on the constraint by $\lambda$, the first-order condition is

 $$ 2(\vec r_e - \mu_{BL} )'\Sigma^{-1} + \lambda 2(\vec r_e - \hat\mu)'(\tau\Sigma)^{-1} = \mathbf{0} $$

which defines the *information contract curve* between $\mu_{BL}$ and $\hat \mu$

 $$ \vec r_e = (\Sigma^{-1} + \lambda (\tau \Sigma)^{-1})^{-1} (\Sigma^{-1} \mu_{BL}  + \lambda (\tau \Sigma)^{-1}\hat \mu ) \quad \quad (11)$$

Note that if $\lambda = 1$, (11) is equivalent with (8) and it identifies one point on the information contract curve. Furthermore, because $\lambda$ is a function of the minimum likelihood $\bar d_2$ on the RHS of the constraint, by varying $\bar d_2$ (or $\lambda$), we can trace out the whole curve as the figure below illustrates.

```python
#========================================
# Draw a new sample for two assets
#========================================
np.random.seed(1987102)

N_new = 2                                           # number of assets
T_new = 200                                         # sample size

tau_new = .8

# random market portfolio (sum is normalized to 1)
w_m_new = np.random.rand(N_new)
w_m_new = w_m_new/(w_m_new.sum())

Mu_new = (np.random.randn(N_new) + 5)/100
S_new = np.random.randn(N_new, N_new)
V_new = S_new @ S_new.T
Sigma_new = V_new * (w_m_new @ Mu_new)**2 / (w_m_new @ V_new @ w_m_new)

excess_return_new = stat.multivariate_normal(Mu_new, Sigma_new)
sample_new = excess_return_new.rvs(T_new)

Mu_est_new = sample_new.mean(0).reshape(N_new, 1)
Sigma_est_new = np.cov(sample_new.T)

sigma_m_new = w_m_new @ Sigma_est_new @ w_m_new
d_m_new = (w_m_new @ Mu_est_new)/sigma_m_new
mu_m_new = (d_m_new * Sigma_est_new @ w_m_new).reshape(N_new, 1)
```

```python
N_r1, N_r2 = 100, 100
r1 = np.linspace(-0.04, .1, N_r1)
r2 = np.linspace(-0.02, .15, N_r2)

lamb_grid = np.linspace(.001, 20, 100)
curve = np.asarray([Black_Litterman(l, mu_m_new, Mu_est_new, Sigma_est_new,
                                    tau_new*Sigma_est_new).flatten() for l in lamb_grid])

lamb_slider = FloatSlider(min = .1, max = 7, step = .5, value = 1)

@interact(lamb = lamb_slider)
def decolletage(lamb):
    dist_r_BL = stat.multivariate_normal(mu_m_new.squeeze(), Sigma_est_new)
    dist_r_hat = stat.multivariate_normal(Mu_est_new.squeeze(), tau_new * Sigma_est_new)

    X, Y = np.meshgrid(r1, r2)
    Z_BL = np.zeros((N_r1, N_r2))
    Z_hat = np.zeros((N_r1, N_r2))

    for i in range(N_r1):
        for j in range(N_r2):
            Z_BL[i, j] = dist_r_BL.pdf(np.hstack([X[i, j], Y[i, j]]))
            Z_hat[i, j] = dist_r_hat.pdf(np.hstack([X[i, j], Y[i, j]]))

    mu_tilde_new = Black_Litterman(lamb, mu_m_new, Mu_est_new, Sigma_est_new,
                                   tau_new * Sigma_est_new).flatten()

    fig, ax = plt.subplots(figsize = (10, 6))
    ax.contourf(X, Y, Z_hat, cmap = 'viridis', alpha =.4)
    ax.contourf(X, Y, Z_BL, cmap = 'viridis', alpha =.4)
    ax.contour(X, Y, Z_BL, [dist_r_BL.pdf(mu_tilde_new)], cmap = 'viridis', alpha = .9)
    ax.contour(X, Y, Z_hat, [dist_r_hat.pdf(mu_tilde_new)], cmap = 'viridis', alpha = .9)
    ax.scatter(Mu_est_new[0], Mu_est_new[1])
    ax.scatter(mu_m_new[0], mu_m_new[1])

    ax.scatter(mu_tilde_new[0], mu_tilde_new[1], color = 'k', s = 20*3)

    ax.plot(curve[:, 0], curve[:, 1], color = 'k')
    ax.axhline(0, color = 'k', alpha = .8)
    ax.axvline(0, color = 'k', alpha = .8)
    ax.set_xlabel(r'Excess return on the first asset, $r_{e, 1}$', fontsize = 12)
    ax.set_ylabel(r'Excess return on the second asset, $r_{e, 2}$', fontsize = 12)
    ax.text(Mu_est_new[0] + 0.003, Mu_est_new[1], r'$\hat{\mu}$', fontsize = 20)
    ax.text(mu_m_new[0] + 0.003, mu_m_new[1] + 0.005, r'$\mu_{BL}$', fontsize = 20)

```

Note that the line that connects the two points $\hat \mu$ and $\mu_{BL}$ is linear, which comes from the fact that the covariance matrices of the two competing distributions (views) are proportional to each other.

To illustrate the fact that this is not necessarily the case, consider another example using the same parameter values, except that the "second view" constituting the constraint has covariance matrix $\tau I$ instead of $\tau \Sigma$. This leads to the following figure, on which the curve connecting $\hat \mu$ and $\mu_{BL}$ are bending.

```python
lamb_grid2 = np.linspace(.001, 20000, 1000)
curve2 = np.asarray([Black_Litterman(l, mu_m_new, Mu_est_new, Sigma_est_new,
                                    tau_new*np.eye(N_new)).flatten() for l in lamb_grid2])

lamb_slider2 = FloatSlider(min = 5, max = 1500, step = 100, value = 200)

@interact(lamb = lamb_slider2)
def decolletage(lamb):
    dist_r_BL = stat.multivariate_normal(mu_m_new.squeeze(), Sigma_est_new)
    dist_r_hat = stat.multivariate_normal(Mu_est_new.squeeze(), tau_new * np.eye(N_new))

    X, Y = np.meshgrid(r1, r2)
    Z_BL = np.zeros((N_r1, N_r2))
    Z_hat = np.zeros((N_r1, N_r2))

    for i in range(N_r1):
        for j in range(N_r2):
            Z_BL[i, j] = dist_r_BL.pdf(np.hstack([X[i, j], Y[i, j]]))
            Z_hat[i, j] = dist_r_hat.pdf(np.hstack([X[i, j], Y[i, j]]))

    mu_tilde_new = Black_Litterman(lamb, mu_m_new, Mu_est_new, Sigma_est_new,
                                   tau_new * np.eye(N_new)).flatten()

    fig, ax = plt.subplots(figsize = (10, 6))
    ax.contourf(X, Y, Z_hat, cmap = 'viridis', alpha = .4)
    ax.contourf(X, Y, Z_BL, cmap = 'viridis', alpha = .4)
    ax.contour(X, Y, Z_BL, [dist_r_BL.pdf(mu_tilde_new)], cmap='viridis', alpha =.9)
    ax.contour(X, Y, Z_hat, [dist_r_hat.pdf(mu_tilde_new)], cmap='viridis', alpha =.9)
    ax.scatter(Mu_est_new[0], Mu_est_new[1])
    ax.scatter(mu_m_new[0], mu_m_new[1])

    ax.scatter(mu_tilde_new[0], mu_tilde_new[1], color = 'k', s = 20*3)

    ax.plot(curve2[:, 0], curve2[:, 1], color = 'k')
    ax.axhline(0, color = 'k', alpha = .8)
    ax.axvline(0, color = 'k', alpha = .8)
    ax.set_xlabel(r'Excess return on the first asset, $r_{e, 1}$', fontsize = 12)
    ax.set_ylabel(r'Excess return on the second asset, $r_{e, 2}$', fontsize = 12)
    ax.text(Mu_est_new[0] + 0.003, Mu_est_new[1], r'$\hat{\mu}$', fontsize = 20)
    ax.text(mu_m_new[0] + 0.003, mu_m_new[1] + 0.005, r'$\mu_{BL}$', fontsize = 20)

```

### Black-Litterman recommendation as regularization

First, consider the OLS regression.

$$\min_{\beta} \Vert X\beta - y \Vert^2 $$

which yields the solution

$$ \hat{\beta}_{OLS} = (X'X)^{-1}X'y.$$

A common performance measure of estimators is the *mean squared error (MSE)*. An estimator is "good" if its MSE is realtively small. Suppose that $\beta_0$ is the "true" value of the coefficent, then the MSE of the OLS estimator is

$$\text{mse}(\hat \beta_{OLS}, \beta_0) := \mathbb E \Vert \hat \beta_{OLS} - \beta_0\Vert^2 = \underbrace{\mathbb E \Vert \hat \beta_{OLS} - \mathbb E  \beta_{OLS}\Vert^2}_{\text{variance}} + \underbrace{\Vert \mathbb E \hat\beta_{OLS} - \beta_0\Vert^2}_{\text{bias}}$$

From this decomposition one can see that in order for the MSE to be small, both the bias and the variance terms must be small. For example, consider the case when $X$ is a $T$-vector of ones (where $T$ is the sample size), so $\hat\beta_{OLS}$ is simply the sample average, while $\beta_0\in \mathbb{R}$ is defined by the true mean of $y$. In this example the MSE is

$$ \text{mse}(\hat \beta_{OLS}, \beta_0) = \underbrace{\frac{1}{T^2} \mathbb E \left(\sum_{t=1}^{T} (y_{t}- \beta_0)\right)^2 }_{\text{variance}} + \underbrace{0}_{\text{bias}}$$

However, because there is a trade-off between the estimator's bias and variance, there are cases when by permitting a small bias we can substantially reduce the variance so overall the MSE gets smaller. A typical scenario when this proves to be useful is when the number of coefficients to be estimated is large relative to the sample size.

In these cases one approach to handle the bias-variance trade-off is the so called *Tikhonov regularization*. A general form with regularization matrix $\Gamma$ can be written as

$$\min_{\beta} \Big\{ \Vert X\beta - y \Vert^2 + \Vert \Gamma (\beta - \tilde \beta) \Vert^2 \Big\}$$

which yields the solution

$$ \hat{\beta}_{Reg} = (X'X + \Gamma'\Gamma)^{-1}(X'y + \Gamma'\Gamma\tilde \beta).$$

Substituting the value of $\hat{\beta}_{OLS}$ yields

$$ \hat{\beta}_{Reg} = (X'X + \Gamma'\Gamma)^{-1}(X'X\hat{\beta}_{OLS} + \Gamma'\Gamma\tilde \beta).$$

Often, the regularization matrix takes the form $\Gamma = \lambda I$ with $\lambda>0$ and $\tilde \beta = \mathbf{0}$. Then the Tikhonov regularization is equivalent to what is called *ridge regression* in statistics.

To illustrate how this estimator addresses the bias-variance trade-off, we compute the MSE of the ridge estimator

$$ \text{mse}(\hat \beta_{\text{ridge}}, \beta_0) = \underbrace{\frac{1}{(T+\lambda)^2} \mathbb E \left(\sum_{t=1}^{T} (y_{t}- \beta_0)\right)^2 }_{\text{variance}} + \underbrace{\left(\frac{\lambda}{T+\lambda}\right)^2 \beta_0^2}_{\text{bias}}$$

The ridge regression shrinks the coefficients of the estimated vector towards zero relative to the OLS estimates thus reducing the variance term at the cost of introducing a "small" bias. However, there is nothing special about the zero vector. When $\tilde \beta \neq \mathbf{0}$ shrinkage occurs in the direction of $\tilde \beta$.

Now, we can give a regularization interpretation of the Black-Litterman portfolio recommendation. To this end, simplify first the equation (8) characterizing the Black-Litterman recommendation

\begin{align*}
\tilde \mu &= (\Sigma^{-1} + (\tau \Sigma)^{-1})^{-1} (\Sigma^{-1}\mu_{BL}  + (\tau \Sigma)^{-1}\hat \mu) \\
&= (1 + \tau^{-1})^{-1}\Sigma \Sigma^{-1} (\mu_{BL}  + \tau ^{-1}\hat \mu) \\
&= (1 + \tau^{-1})^{-1} ( \mu_{BL}  + \tau ^{-1}\hat \mu)
\end{align*}

In our case, $\hat \mu$ is the estimated mean excess returns of securities. This could be written as a vector autoregression where

* $y$ is the stacked vector of observed excess returns of size $(N T\times 1)$ -- $N$ securities and $T$ observations

* $X = \sqrt{T^{-1}}(I_{N} \otimes \iota_T)$ where $I_N$ is the identity matrix and $\iota_T$ is a column vector of ones.

Correspondingly, the OLS regression of $y$ on $X$ would yield the mean excess returns as coefficients. With $\Gamma = \sqrt{\tau T^{-1}}(I_{N} \otimes \iota_T)$ we can write the regularized version of the mean excess return estimation.

\begin{align*}
\hat{\beta}_{Reg} &= (X'X + \Gamma'\Gamma)^{-1}(X'X\hat{\beta}_{OLS} + \Gamma'\Gamma\tilde \beta) \\
&= (1 + \tau)^{-1}X'X (X'X)^{-1} (\hat \beta_{OLS}  + \tau \tilde \beta) \\
&= (1 + \tau)^{-1} (\hat \beta_{OLS}  + \tau \tilde \beta) \\
&= (1 + \tau^{-1})^{-1} ( \tau^{-1}\hat \beta_{OLS}  +  \tilde \beta)
\end{align*}

Given that $\hat \beta_{OLS} = \hat \mu$ and $\tilde \beta = \mu_{BL}$ in the Black-Litterman model we have the following interpretation of the model's recommendation.

The estimated (personal) view of the mean excess returns, $\hat{\mu}$ that would lead to extreme short-long positions are "shrunk" towards the conservative market view, $\mu_{BL}$.
that leads to the more  conservative market portfolio.

So the Black-Litterman procedure results in a recommendation that is a compromise between the conservative market portfolio and the  more extreme portfolio that is implied by estimated "personal" views.

### Digression on ${\sf T}$ operator

The Black-Litterman approach is partly inspired by the econometric insight that it
is easier to estimate covariances of excess returns than the means. That is what gave
Black and Litterman license to adjust investors' perception of mean excess returns while not tampering with the covariance matrix of excess returns.

The robust control theory is another approach that also hinges on adjusting mean excess
returns but not covariances.

Associated with a robust control problem is what Hansen and Sargent call
a ${\sf T}$ operator.

Let's define the ${\sf T}$ operator as it applies to the problem at hand.

Let $x$ be an $n \times 1$ Gaussian  random vector with mean vector $\mu$ and
covariance matrix $\Sigma = C C'$.  This means that $x$ can be represented
as

$$ x = \mu + C \epsilon  $$

where $\epsilon \sim {\mathcal N}(0,I)$.

Let $\phi(\epsilon)$ denote the associated standardized Gaussian density.

Let $m(\epsilon,\mu)$ be a **likelihood ratio**, meaning that it satisfies

  * $m(\epsilon, \mu) > 0 $

  * $ \int m(\epsilon,\mu) \phi(\epsilon) d \epsilon  =1 $

That is, $m(\epsilon, \mu)$ is a nonnegative random variable with mean 1.

Multiplying $\phi(\epsilon) $ by the likelihood ratio $m(\epsilon, \mu)$ produces
a distorted distribution for $\epsilon$, namely,

$$ \tilde \phi(\epsilon) = m(\epsilon,\mu) \phi(\epsilon)  $$

The next concept that we need is the **entropy** of the distorted distribution
$\tilde \phi$ with respect to $\phi$.

**Entropy**  is defined as

$$ {\rm ent} = \int \log m(\epsilon,\mu) m(\epsilon,\mu) \phi(\epsilon) d \epsilon $$

or

$$ {\rm ent} = \int \log m(\epsilon,\mu) \tilde \phi(\epsilon) d \epsilon $$

That is, relative entropy is the expected value of the likelihood ratio $m$ where the
expectation is taken with respect to the twisted density $\tilde \phi$.

Relative entropy is nonnegative.  It is a measure of the discrepancy between two probability
distributions.  As such, it plays an important role in governing the behavior of statistical
tests designed to discriminate one probability distribution from another.

We are ready to define the ${\sf T}$ operator.

Let $V(x)$ be a value function.

Define

$$ \eqalign{ {\sf T}\left(V(x)\right) & = \min_{m(\epsilon,\mu)} \int m(\epsilon,\mu)[V(\mu + C \epsilon) + \theta \log m(\epsilon,\mu) ] \phi(\epsilon) d \epsilon \cr
                         & = - \log \theta \int \exp \left( \frac{- V(\mu + C \epsilon)}{\theta} \right) \phi(\epsilon) d \epsilon } $$

This asserts that ${\sf T}$ is an indirect utility function for a minimization problem in
which an ``evil agent'' chooses a distorted probablity distribution $\tilde \phi$ to
lower  expected utility, subject to a penalty term that gets bigger the larger is relative
entropy.

Here the penalty parameter  $\theta \in [\underline \theta, +\infty] $ is a robustness parameter; when it is $+\infty$, there is no scope for the minimizing agent to distort the
distribution, so no robustness to alternative distributions is acquired.  As $\theta$ is lowered, more robustness is achieved.

**Note:**  The ${\sf T}$ operator is sometimes called a *risk-sensitivity* operator.

We shall apply ${\sf T}$ to the special case of a linear value function $w'
(\vec r - r_f {\bf 1}) $ where $\vec r - r_f {\bf 1} \sim \mathcal N(\mu , \Sigma)$
or $\vec r - r_f {\bf 1} = \mu + C \epsilon$ and  $\epsilon \sim {\mathcal N}(0,I)$.

The associated worst-case distribution of $\epsilon$ is Gaussian with mean $v =-\theta^{-1} C' w$ and covariance matrix $I$.  (When the value function is
affine, the worst-case distribution distorts the mean vector of $\epsilon$ but not the covariance matrix of $\epsilon$.)

For utility function argument  $w'
(\vec r - r_f {\bf 1}) $,
$$ {\sf T} (\vec r - r_f {\bf 1}) = w' \mu + \zeta - \frac{1}{2\theta} w' \Sigma w $$
and entropy is
$$ \frac{v'v}{2} = \frac{1}{2\theta^2}  w' C C' w. $$

#### A robust mean-variance portfolio model

According to criterion (1), the mean-variance portfolio choice problem chooses $w$ to maximize

$$ E [w ( \vec r - r_f {\bf 1})]] - {\rm var} [ w ( \vec r - r_f {\bf 1}) ]  \quad (10) $$

which equals

$$ w'\mu - \frac{\delta}{2} w' \Sigma w     $$

A robust decision maker can be modelled as replacing the mean return $ E [w ( \vec r - r_f {\bf 1})]$
with the risk-sensitive

$$ {\sf T} [w ( \vec r - r_f {\bf 1})] = w' \mu - \frac{1}{2 \theta} w' \Sigma w $$

that comes from replacing the mean $\mu$ of $ \vec r - r_f {\bf 1} $ with the worst-case mean

$$ \mu - \theta^{-1} \Sigma w   $$

Notice how the worst-case mean vector  depends on the portfolio $w$.

The operator ${\sf T}$ is the indirect utility function that emerges from solving
a  problem in which an agent who chooses probabilities does so in order to minimize
the expected utility of a maximizing agent (in our case, the maximizing agent chooses
portfolio weights $w$).

The robust version of the mean-variance portfolio choice problem is then
to choose a portfolio $w$ that maximizes

$$ {\sf T} [w ( \vec r - r_f {\bf 1})] - \frac{\delta}{2} w' \Sigma w , \quad (11) $$

or

$$ w' (\mu - \theta^{-1} \Sigma w ) - \frac{\delta}{2} w' \Sigma w, \quad (12) $$

The minimizer of (12) is

$$ w_{\rm rob} = \frac{1}{\delta + \gamma } \Sigma^{-1} \mu  , \quad (13) $$

where  $\gamma \equiv \theta^{-1}$ is sometimes called the risk-sensitivity parameter.

An increase in the risk-sensitivity paramter $\gamma$ shrinks the portfolio weights toward zero in the same way that an increase in risk aversion does.

----------------------------------------
# Appendix

We want to illustrate the "folk theorem" that with high or moderate frequency data, it is more difficult to esimate means than variances.

In order to operationalize this statement, we take two analog estimators:

 - sample average: $\bar X_N = \frac{1}{N}\sum_{i=1}^{N} X_i$
 - sample variance: $S_N = \frac{1}{N-1}\sum_{t=1}^{N} (X_i - \bar X_N)^2$

to estimate the unconditional mean and unconditional variance of the random variable $X$, respectively.

To measure the "difficulty of estimation", we use *mean squared error* (MSE), that is the average squared difference between the estimator and the true value. Assuming that the process $\{X_i\}$ is ergodic, both analog estimators are known to converge to their true values as the sample size $N$ goes to infinity. More precisely, for all $\varepsilon > 0$,

\begin{align}
\lim_{N\to \infty} \ \ P\left\{ \left |\bar X_N - \mathbb E X \right| > \varepsilon \right\} = 0 \quad \quad \text{and}\quad \quad  \lim_{N\to \infty} \ \ P \left\{ \left| S_N - \mathbb V X \right| > \varepsilon \right\} = 0
\end{align}

A necessary condition for these convergence results is that the associated MSEs vanish as $N$ goes to infintiy, or in other words,

$$\text{MSE}(\bar X_N, \mathbb E X) = o(1) \quad \quad  \text{and} \quad \quad \text{MSE}(S_N, \mathbb V X) = o(1)$$

Even if the MSEs converge to zero, the associated rates might be different. Looking at the limit of the *relative MSE* (as the sample size grows to infinity)

$$ \frac{\text{MSE}(S_N, \mathbb V X)}{\text{MSE}(\bar X_N, \mathbb E X)} = \frac{o(1)}{o(1)} \underset{N \to \infty}{\to} B $$

can inform us about the relative (asymptotic) rates.

We will show that in general, with dependent data, the limit $B$ depends on the sampling frequency. In particular, we find that the rate of convergence of the variance estimator is less sensitive to increased sampling frequency than the rate of convergence of the mean estimator. Hence, we can expect the relative asymptotic rate, $B$,  to get smaller with higher frequency data, illustrating that "it is more difficult to estimate means than variances". That is, we need significantly more data to obtain a given precision of the mean estimate than for our  variance estimate.

### A special case -- i.i.d. sample

We start our analysis with the benchmark case of iid data. Consider a sample of size $N$ generated by the following iid process,

$$X_i \sim \mathcal{N}(\mu, \sigma^2).$$

Taking  $\bar X_N$ to estimate the mean, the MSE is

$$ \text{MSE}(\bar X_N, \mu) = \frac{\sigma^2}{N} .$$

Taking $S_N$ to estimate the variance, the MSE is

$$ \text{MSE}(S_N, \sigma^2) = \frac{2\sigma^4}{N-1} .$$

Both estimators are unbiased and hence the MSEs reflect the corresponding variances of the estimators. Furthermore, both MSEs are $o(1)$ with a (multiplicative) factor of difference in their rates of convergence:

$$ \frac{\text{MSE}(S_N, \sigma^2)}{\text{MSE}(\bar X_N, \mu)} = \frac{N2\sigma^2}{N-1} \quad \underset{N \to \infty}{\to} \quad 2\sigma^2$$

We are interested in how this (asymptotic) relative rate of convergence changes as increasing sampling frequency puts dependence into the data.

### Dependence and sampling frequency

To investigate how sampling frequency affects relative rates of convergence, we assume that the data are generated by a mean-reverting continuous time process of the form

$$dX_t = -\kappa (X_t -\mu)dt + \sigma dW_t\quad\quad $$

where $\mu$ is the unconditional mean,  $\kappa > 0$ is a persistence parameter,
and $\{W_t\}$ is a standardized Brownian motion.

Observations arising from this system in particular discrete periods $\mathcal T(h) \equiv \{nh : n \in \mathbb Z \}$ with $h>0$ can be described by the following process

$$X_{t+1} = (1 - \exp(-\kappa h))\mu + \exp(-\kappa h)X_t + \epsilon_{t, h}$$

where
$$\epsilon_{t, h} \sim \mathcal{N}(0, \Sigma_h) \quad \text{with}\quad \Sigma_h = \frac{\sigma^2(1-\exp(-2\kappa h))}{2\kappa}$$

We call $h$ the *frequency* parameter, whereas $n$ represents the number of *lags* between observations. Hence, the effective distance between two observations $X_t$ and $X_{t+n}$ in the discrete time notation is equal to $h\cdot n$ in terms of the underlying continuous time process.

Straightforward calculations show that the autocorrelation function for the stochastic process $\{X_{t}\}_{t\in \mathcal T(h)}$ is

$$\Gamma_h(n) \equiv \text{corr}(X_{t + h n}, X_t) = \exp(-\kappa h n)$$

and the autocovariance function is

$$\gamma_h(n) \equiv \text{cov}(X_{t + h n}, X_t) = \frac{\exp(-\kappa h n)\sigma^2}{2\kappa} .$$

It follows that if $n=0$, the unconditional variance is given by $\gamma_h(0) = \frac{\sigma^2}{2\kappa}$ irrespective of the sampling frequency.

The following figure illustrates how the dependence between the observations is related to sampling frequency.
 - For any given $h$, the autocorrelation converges to zero as we increase the distance -- $n$ -- between the observations. This represents the "weak dependence" of the $X$ process.
 - Moreover, for a fixed lag length, $n$, the dependence vanishes as the sampling frequency goes to infinity. In fact, letting $h$ go to $\infty$ gives back the case of i.i.d. data.

```python
mu = .0
kappa = .1
sigma = .5
var_uncond = sigma**2 / (2 * kappa)

n_grid = np.linspace(0, 40, 100)
autocorr_h1 = np.exp(- kappa * n_grid * 1)
autocorr_h2 = np.exp(- kappa * n_grid * 2)
autocorr_h5 = np.exp(- kappa * n_grid * 5)
autocorr_h1000 = np.exp(- kappa * n_grid * 1e8)

fig, ax = plt.subplots(figsize = (8, 4))
ax.plot(n_grid, autocorr_h1, label = r'$h = 1$', color = 'darkblue', lw = 2)
ax.plot(n_grid, autocorr_h2, label = r'$h = 2$', color = 'darkred', lw = 2)
ax.plot(n_grid, autocorr_h5, label = r'$h = 5$', color = 'orange', lw = 2)
ax.plot(n_grid, autocorr_h1000, label = r'"$h = \infty$"', color = 'darkgreen', lw = 2)
ax.legend(loc = 'best', fontsize = 13)
ax.grid()
ax.set_title(r'Autocorrelation functions, $\Gamma_h(n)$', fontsize = 13)
ax.set_xlabel(r'Lags between observations, $n$', fontsize = 11)
plt.show()
```

### Frequency and the mean estimator

Consider again the AR(1) process generated by discrete sampling with frequency $h$. Assume that we have a sample of size $N$ and we would like to estimate the unconditional mean -- in our case the true mean is $\mu$.

Again, the sample average is an unbiased estimator of the unconditional mean.

$$ \mathbb{E}[\bar X_N] = \frac{1}{N}\sum_{i = 1}^N \mathbb{E}[X_i] = \mathbb{E}[X_0] = \mu $$

The variance of the sample mean is given by

\begin{align}
\mathbb{V}\left(\bar X_N\right) &= \mathbb{V}\left(\frac{1}{N}\sum_{i = 1}^N X_i\right) \\
&= \frac{1}{N^2} \left(\sum_{i = 1}^N \mathbb{V}(X_i) + 2 \sum_{i = 1}^{N-1} \sum_{s = i+1}^N \text{cov}(X_i, X_s) \right) \\
&= \frac{1}{N^2} \left( N \gamma(0) + 2 \sum_{i=1}^{N-1} i \cdot \gamma\left(h\cdot (N - i)\right) \right) \\
&= \frac{1}{N^2} \left( N \frac{\sigma^2}{2\kappa} + 2 \sum_{i=1}^{N-1} i \cdot \exp(-\kappa h (N - i)) \frac{\sigma^2}{2\kappa} \right)
\end{align}

It is explicit in the above equation that time dependence in the data inflates the variance of the mean estimator through the covariance terms. Moreover, as we can see, a higher sampling frequency---smaller $h$---makes all the covariance terms larger everything else being fixed. This implies a relatively slower rate of convergence of the sample average for high frequency data. Intuitively, the stronger dependence across observations  for high frequency data reduces the "information content" of each observation relative to the iid case.

We can upper bound the variance term in the following way.

\begin{align}
\mathbb{V}(\bar X_N) &= \frac{1}{N^2} \left( N \sigma^2 + 2 \sum_{i=1}^{N-1} i \cdot \exp(-\kappa h (N - i)) \sigma^2 \right) \\
&\leq \frac{\sigma^2}{2\kappa N} \left(1 + 2 \sum_{i=1}^{N-1} \cdot \exp(-\kappa h (i)) \right) \\
&= \underbrace{\frac{\sigma^2}{2\kappa N}}_{\text{i.i.d.  case}} \left(1 + 2 \frac{1 - \exp(-\kappa h)^{N-1}}{1 - \exp(-\kappa h)} \right)
\end{align}

Asymptotically the $\exp(-\kappa h)^{N-1}$ vanishes and the dependence in the data inflates the benchmark iid variance by a factor of $\left(1 + 2 \frac{1}{1 - \exp(-\kappa h)} \right)$. This long run factor is larger the higher is  the frequency (the smaller is $h$).

Therefore, we expect the asymptotic relative MSEs, $B$, to change with time dependent data. We just saw that the mean estimator's rate is roughly changing by a factor of $\left(1 + 2 \frac{1}{1 - \exp(-\kappa h)} \right)$. Unfortunately, the variance estimator's MSE is harder to derive. Nonetheless, we can approximate it by using (large sample) simulations, thus getting an idea about how the asymptotic relative MSEs changes in the sampling frequency $h$ relative to the iid case that we compute in closed form.

```python
def sample_generator(h, N, M):
    phi = (1 - np.exp(- kappa * h)) * mu
    rho = np.exp(- kappa * h)
    s = sigma**2 * (1 - np.exp(-2 * kappa * h)) / (2 * kappa)

    mean_uncond = mu
    std_uncond = np.sqrt(sigma**2 / (2 * kappa))

    eps_path = stat.norm(0, np.sqrt(s)).rvs((M, N))

    y_path = np.zeros((M, N + 1))
    y_path[:, 0] = stat.norm(mean_uncond, std_uncond).rvs(M)

    for i in range(N):
        y_path[:, i + 1] = phi + rho*y_path[:, i] + eps_path[:, i]

    return y_path

```

```python
# generate large sample for different frequencies
N_app, M_app = 1000, 30000                      # sample size, number of simulations
h_grid = np.linspace(.1, 80, 30)

var_est_store = []
mean_est_store = []
labels = []

for h in h_grid:
    labels.append(h)
    sample = sample_generator(h, N_app, M_app)
    mean_est_store.append(np.mean(sample, 1))
    var_est_store.append(np.var(sample, 1))

var_est_store = np.array(var_est_store)
mean_est_store = np.array(mean_est_store)

```

```python
# save mse of estimators
mse_mean = np.var(mean_est_store, 1) + (np.mean(mean_est_store, 1) - mu)**2
mse_var = np.var(var_est_store, 1) + (np.mean(var_est_store, 1) - var_uncond)**2

benchmark_rate = 2*var_uncond       # iid case

#relative MSE for large samples
rate_h = mse_var/mse_mean

```

```python
fig, ax = plt.subplots(figsize = (8, 5))
ax.plot(h_grid, rate_h, color = 'darkblue', lw = 2,
        label = r'large sample relative MSE, $B(h)$')
ax.axhline(benchmark_rate, color = 'k', linestyle = '--', label = r'iid benchmark')
ax.set_title('Relative MSE for large samples as a function of sampling frequency \n MSE($S_N$) relative to MSE($\\bar X_N$)',
             fontsize = 12)
ax.set_xlabel('Sampling frequency, $h$', fontsize = 11)
ax.set_ylim([1, 2.9])
ax.legend(loc = 'best', fontsize = 10)
plt.show()
```

The above figure illustrates the relationship between the asymptotic relative MSEs and the sampling frequency.

 * We can see that with low frequency data -- large values of $h$ -- the ratio of asymptotic rates approaches the iid case.

 * As $h$ gets smaller -- the higher the frequency -- the relative performance of the variance estimator is better in the sense that the ratio of asymptotic rates gets smaller. That is, as the time dependence gets more pronounced, the rate of convergence of the mean estimator's MSE deteriorates more than that of the variance estimator.

-----------------------------------------------------------

#### References

Black, F. and Litterman, R., 1992. "Global portfolio optimization". Financial analysts journal, 48(5), pp.28-43.

Dickey, J. 1975. "Bayesian alternatives to the F-test and least-squares estimate in the normal linear model", in: S.E. Fienberg and A. Zellner, eds., "Studies in Bayesian econometrics and statistics" (North-Holland, Amsterdam) 515-554.

Hansen, Lars Peter and Thomas J. Sargent. 2001. "Robust Control and Model Uncertainty." American Economic Review, 91(2): 60-66.

Leamer, E.E., 1978. **Specification searches: Ad hoc inference with nonexperimental data**, (Vol. 53). John Wiley & Sons Incorporated.

```python

```