#  CS109B Data Science 2: Advanced Topics in Data Science
## Lecture 3 - Coding Environment Setup and review of `statsmodels`
###  Notebook B

**Harvard University**

**Spring 2021**

**Instructors:** Mark Glickman, Pavlos Protopapas, and Chris Tanner

**Additional Instructor:** Eleni Kaxiras

*Content:* Eleni Kaxiras and Will Claybaugh

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
```

```python
import numpy as np
from scipy.interpolate import interp1d
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline
```

## Learning Goals

By the end of this lab, you should be able to:
* use `np.linalg.vander`
* use the weird R-style formulas in `statsmodels`
* practice least-squares regression in `statsmodels`

### Basis Functions

In our models we can use various types of functions as basis functions. Strictly speaking, in linear algebra where a basis for a subspace S of $\mathbb{R}^n$ is a set of vectors that spans S and is linearly independent. As a reminder, a set of vectors $\textbf{v}_1, \textbf{v}_2, ..., \textbf{v}_k$ are considered linearly independent if they cannot be written as a linear combination of each other, such that, if: $c_1\textbf{v}_1+c_2\textbf{v}_2+ ...+ c_k\textbf{v}_k = \textbf{0}$ then $c_1,c_2,...,c_k$ are all zero.

In data science where we have lots of imperfect data (with errors), as well as imperfect computers (with round-off errors), when we substitute their values into the matrices we almost always get column degeneracy, meaning, some of our columns become linear combinations of each other. Especially so if we use the monomial basis and go beyond ~5,6 degree of the polynomial.

Examples are:

- Monomials such as $x,x^2,x^4,x^5$
- Sigmoid/ReLU functions (neural networks)
- Fourier functions
- Wavelets
- Splines

The matrix produced when we substitute the values of our data into the basis functions is called the *design matrix*.

### Linear/Polynomial Regression

We will use the `diabetes` dataset.

Variables are:
- subject:   subject ID number
- age:       age diagnosed with diabetes
- acidity:   a measure of acidity called base deficit
Response:
- y:         natural log of serum C-peptide concentration

*Original source is Sockett et al. (1987) mentioned in Hastie and Tibshirani's book
"Generalized Additive Models".*

Reading the data in Pandas:

```python
diab = pd.read_csv("data/diabetes.csv")
diab.head()
```

      subject
      age
      acidity
      y

      0
      1
      5.2
      -8.1
      4.8

      1
      2
      8.8
      -16.1
      4.1

      2
      3
      10.5
      -0.9
      5.2

      3
      4
      10.6
      -7.8
      5.5

      4
      5
      10.4
      -29.0
      5.0

### Create the design matrix for a fictitious dataset
Let's keep just the `age` feature and create some columns of our own. Let's see how good this matrix is before we create the design matrix.

```python
diab_age = diab[['age']].copy()
# Age^2
diab_age['age2'] = diab_age.apply(lambda row: row['age']**2, axis=1)

# Create random numbers on normal distriution
diab_age['random'] = np.random.normal(0,1,len(diab_age))

# On column with the same values
diab_age['same'] = diab_age['age']
diab_age.head()
```

      age
      age2
      random
      same

      0
      5.2
      27.04
      1.239895
      5.2

      1
      8.8
      77.44
      -0.091829
      8.8

      2
      10.5
      110.25
      0.657542
      10.5

      3
      10.6
      112.36
      1.499066
      10.6

      4
      10.4
      108.16
      -1.155394
      10.4

```python
# Make a copy to a np matrix
A = diab_age.to_numpy(copy=True)
A[:5]
```

    array([[ 5.20000000e+00,  2.70400000e+01,  1.23989458e+00,
             5.20000000e+00],
           [ 8.80000000e+00,  7.74400000e+01, -9.18294950e-02,
             8.80000000e+00],
           [ 1.05000000e+01,  1.10250000e+02,  6.57542252e-01,
             1.05000000e+01],
           [ 1.06000000e+01,  1.12360000e+02,  1.49906618e+00,
             1.06000000e+01],
           [ 1.04000000e+01,  1.08160000e+02, -1.15539433e+00,
             1.04000000e+01]])

Let's check if the columns of A are linearly independent by using some linear algebra methods from `numpy.linalg` and `sympy`.

```python
from numpy.linalg import matrix_rank

# You can see the rank of your matrix -> How many columns are independent
matrix_rank(A)
```

    3

```python
# check out which rows are linearly independent
import sympy
_, inds = sympy.Matrix(A).T.rref()
inds
```

    (0, 1, 2)

```python
np.linalg.cond(A)
```

    4.553452628088643e+17

### Create the design matrix for `age` using a polynomial basis
Let's keep just the `age` feature again and create the design matrix using a polynomial of degree `n`. First we will use the basic `numpy` formula `vander()`.

```python

# We create a 3 degree age columns, takes into account the intercept term
vand = np.vander(diab_age.age, 2, increasing=True)
vand[:3], vand.shape
```

    (array([[ 1. ,  5.2],
            [ 1. ,  8.8],
            [ 1. , 10.5]]),
     (43, 2))

### The rank will be very high if you go higher than the 2 we specified
* We see the '2' indicating the linear independence
* But if you would subsitude in the dat for like 8 degrees, you have a lot of multicolineairty
* Means that you won't get a good solution -> Succeptible for random errors

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
## find the condition number of this matrix

np.linalg.cond(vand), matrix_rank(vand)
```

    (24.709088478926237, 2)

Exercise: Vandermonde matrix

Change the degree of the polynomial and comment on what happens to the condition and rank of the matrix.

```python
vand = np.vander(diab_age.age, 8, increasing=True)
vand[:3], vand.shape
```

    (array([[1.00000000e+00, 5.20000000e+00, 2.70400000e+01, 1.40608000e+02,
             7.31161600e+02, 3.80204032e+03, 1.97706097e+04, 1.02807170e+05],
            [1.00000000e+00, 8.80000000e+00, 7.74400000e+01, 6.81472000e+02,
             5.99695360e+03, 5.27731917e+04, 4.64404087e+05, 4.08675596e+06],
            [1.00000000e+00, 1.05000000e+01, 1.10250000e+02, 1.15762500e+03,
             1.21550625e+04, 1.27628156e+05, 1.34009564e+06, 1.40710042e+07]]),
     (43, 8))

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
## find the condition number of this matrix
np.linalg.cond(vand), matrix_rank(vand)
```

    (4207821293.899193, 8)

### Linear/Polynomial regression with statsmodels.

As you remember from 109a, we have two tools for Linear Regression:
- `statsmodels` [https://www.statsmodels.org/stable/regression.html](https://www.statsmodels.org/stable/regression.html), and
- `sklearn`[https://scikit-learn.org/stable/index.html](https://scikit-learn.org/stable/index.html)

Previously, in this notebook, we worked from a vector of target values and a design matrix we built ourself. In 109a we used e.g. `sklearn`'s PolynomialFeatures to build the matrix. Now we will look at `statsmodels` which allows users to fit statistical models using R-style **formulas**. They build the target value and design matrix for you.

**Note:** Categorical features (e.g. let's say we had a categorical feature called Region, are designated by `C(Region)`), polynomial features (e.g. age) are entered as `np.power(age, n)` where `n` is the degree of the polynomial **OR** `np.vander(age, n, increasing=True)`.

```
# Example: if our target variable is 'Lottery', while 'Region' is a categorical predictor and all the others are numerical:
df = dta.data[['Lottery', 'Literacy', 'Wealth', 'Region']]

formula='Lottery ~ Literacy + Wealth + C(Region) + Literacy * Wealth'
```

For more on these formulas see:

- https://www.statsmodels.org/stable/examples/notebooks/generated/formulas.html
- https://patsy.readthedocs.io/en/latest/overview.html

```python
import statsmodels.formula.api as smf

model1 = smf.ols('y ~ age', data=diab)
fit1_lm = model1.fit()
```

Let's build a dataframe to predict values on (sometimes this is just the test or validation set). Very useful for making pretty plots of the model predictions - predict for TONS of values, not just whatever's in the training set.

```python
x_pred = np.linspace(0.5,20,100)

predict_df = pd.DataFrame(data={"age":x_pred})
predict_df.head()
```

      age

      0
      0.500000

      1
      0.696970

      2
      0.893939

      3
      1.090909

      4
      1.287879

Use `get_prediction(<data>).summary_frame()` to get the model's prediction (and error bars!)

```python
prediction_output = fit1_lm.get_prediction(predict_df).summary_frame()
prediction_output.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      4.037574
      0.233301
      3.566412
      4.508736
      2.650278
      5.424870

      1
      4.053940
      0.228886
      3.591696
      4.516184
      2.669647
      5.438232

      2
      4.070305
      0.224489
      3.616940
      4.523670
      2.688952
      5.451658

      3
      4.086671
      0.220113
      3.642143
      4.531198
      2.708193
      5.465149

      4
      4.103036
      0.215759
      3.667302
      4.538770
      2.727369
      5.478703

Plot the data, the fitted model, the **confidence intervals**, and the *prediction intervals*. For more on how `statsmodels` calculates these intervals see: https://www.statsmodels.org/stable/_modules/statsmodels/regression/_prediction.html

```python
ax1 = diab.plot.scatter(x='age',y='y',c='brown',title="Diabetes data with least-squares linear fit")
ax1.set_xlabel("Age at Diagnosis")
ax1.set_ylabel("Log C-Peptide Concentration")

ax1.plot(predict_df.age, prediction_output['mean'],color="green")
ax1.plot(predict_df.age, prediction_output['mean_ci_lower'], color="blue",linestyle="dashed")
ax1.plot(predict_df.age, prediction_output['mean_ci_upper'], color="blue",linestyle="dashed")
ax1.plot(predict_df.age, prediction_output['obs_ci_lower'], color="green",linestyle="dashdot")
ax1.plot(predict_df.age, prediction_output['obs_ci_upper'], color="green",linestyle="dashdot");
```

Breakout Room Exercise

- Fit a 3rd degree polynomial model to predict `y` using only `age` and
- Plot the model and its confidence intervals.
- Change the degree of your polynomial and see what happens to the fitted curve.
- Does our model have an intercept? *Note*: we can discover the existence or not of an intercept in our model by running:
```
model_name.params
```

```python
# Fit a 3rd degree polynomial model to predict y using only age and
model2 = smf.ols('y ~ np.power(age, 3)', data=diab)
fit1_lm = model2.fit()

# Get predictions
x_pred = np.linspace(0.5,20,100)
predict_df2 = pd.DataFrame(data=np.power(diab['age'], 3))
predict_df2.head()

prediction_output2 = fit1_lm.get_prediction(predict_df2).summary_frame()
prediction_output2.head()

```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      599.964746
      296.906854
      0.349193
      1199.580300
      0.347546
      1199.581947

      1
      67794.781980
      33815.034002
      -496.064385
      136085.628345
      -496.064400
      136085.628360

      2
      332301.992819
      165756.554115
      -2450.158265
      667054.143902
      -2450.158268
      667054.143905

      3
      361894.147834
      180517.716594
      -2668.775552
      726457.071220
      -2668.775555
      726457.071223

      4
      304880.632820
      152078.227878
      -2247.578110
      612008.843750
      -2247.578113
      612008.843753

```python
# Plot the model and its confidence intervals.
ax2 = diab.plot.scatter(x='age',y='y',c='brown',title="Diabetes data with least-squares linear fit")
ax2.set_xlabel("Age at Diagnosis")
ax2.set_ylabel("Log C-Peptide Concentration")

ax2.plot(predict_df2.age, prediction_output2['mean'],color="green")
ax2.plot(predict_df2.age, prediction_output2['mean_ci_lower'], color="blue",linestyle="dashed")
ax2.plot(predict_df2.age, prediction_output2['mean_ci_upper'], color="blue",linestyle="dashed")
ax2.plot(predict_df2.age, prediction_output2['obs_ci_lower'], color="green",linestyle="dashdot")
ax2.plot(predict_df2.age, prediction_output2['obs_ci_upper'], color="green",linestyle="dashdot");

```

```python
# solution

```

Vandermonde matrix in formulas

It's easier to build higher order polynomials using `np.vandrer()`.

```python
# We need to add -1 because statsmodel will give one and
# the np.vander() has an intercept by itself (the one we will use)
# AKA we would end up with 2 intercepts if did not substract it
# Don't see the '-1' as a mathetimatical number, just that;
# you see to statsmodels that we don't want to use it's intercept
formula = "y ~ np.vander(age, 6, increasing=True) -1"
fit3_lm = smf.ols(formula=formula, data=diab).fit()
```

```python
# 3.14 Is the intercept, so the age^0
fit3_lm.params
```

    np.vander(age, 6, increasing=True)[0]    3.147753
    np.vander(age, 6, increasing=True)[1]    0.130248
    np.vander(age, 6, increasing=True)[2]    0.136034
    np.vander(age, 6, increasing=True)[3]   -0.028749
    np.vander(age, 6, increasing=True)[4]    0.002064
    np.vander(age, 6, increasing=True)[5]   -0.000050
    dtype: float64

```python
## To our point why the Vandermonde matrix is usually ill-conditioned,
# find the condition number of this matrix
np.linalg.cond(np.vander(predict_df.age, 6, increasing=True))
```

    11539278.684989277

```python
# solution
poly_predictions = fit3_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.243420
      0.734767
      1.754641
      4.732198
      1.308481
      5.178358

      1
      3.295358
      0.568205
      2.144066
      4.446650
      1.606277
      4.984440

      2
      3.353648
      0.438495
      2.465173
      4.242123
      1.831507
      4.875789

      3
      3.417256
      0.347322
      2.713515
      4.120997
      1.995013
      4.839499

      4
      3.485217
      0.295584
      2.886306
      4.084128
      2.111820
      4.858614

```python
# solution
x_pred = np.linspace(0.5,15,100)
predict_df = pd.DataFrame(data={"age":x_pred})

ax2 = diab.plot.scatter(x='age',y='y',c='Red',title="Diabetes data with least-squares cubic fit")
ax2.set_xlabel("Age at Diagnosis")
ax2.set_ylabel("Log C-Peptide Concentration")

ax2.plot(predict_df.age, poly_predictions['mean'],color="green")
ax2.plot(predict_df.age, poly_predictions['mean_ci_lower'], color="blue",linestyle="dashed", label='confidence interval')
ax2.plot(predict_df.age, poly_predictions['mean_ci_upper'], color="blue",linestyle="dashed");
ax2.legend();
```

Discussion

#### QR decomposition (**Beyond the scope of this class**)

As you know, to find the parameters of our model, we may try to solve the so-called *normal equations*, which, written in matrix form, are:

\begin{equation}
    (\boldsymbol{A^T}\cdot \boldsymbol{A}) \cdot \boldsymbol{b} = \boldsymbol{A} \cdot \boldsymbol{y}
\end{equation}

The direct solution is $\hat{\boldsymbol{b}}=(\boldsymbol{A}^T\cdot \boldsymbol{A})^{-1}\cdot \boldsymbol{A}^T \cdot \boldsymbol{y}$

Solving the least-squares problem directly via the normal equations is susceptible to roundoff error when the condition of the matrix $\boldsymbol{A}$ is large. An alternative technique involves QR decomposition (details in any good linear algebra book). `statsmodels` lets you use this technique via a parameter in the `.fit`:
```
   .fit(method='qr')
```

#### Let's try with QR now

```python
# Not part of the class but can perhaps give you a bit more feel of what we do
# If you don't have the right background, they can get you into a rabbit hole

```

```python
formula = "y ~ np.vander(age, 6, increasing=True) -1"
fit3_lm = smf.ols(formula=formula, data=diab).fit(method='qr')
```

```python
fit3_lm.params
```

    np.vander(age, 6, increasing=True)[0]    3.147753
    np.vander(age, 6, increasing=True)[1]    0.130248
    np.vander(age, 6, increasing=True)[2]    0.136034
    np.vander(age, 6, increasing=True)[3]   -0.028749
    np.vander(age, 6, increasing=True)[4]    0.002064
    np.vander(age, 6, increasing=True)[5]   -0.000050
    dtype: float64

```python
# solution
poly_predictions = fit3_lm.get_prediction(predict_df).summary_frame()
poly_predictions.head()
```

      mean
      mean_se
      mean_ci_lower
      mean_ci_upper
      obs_ci_lower
      obs_ci_upper

      0
      3.243420
      0.734767
      1.754641
      4.732198
      1.308481
      5.178358

      1
      3.281393
      0.607441
      2.050600
      4.512185
      1.537152
      5.025633

      2
      3.323028
      0.500327
      2.309268
      4.336788
      1.724518
      4.921538

      3
      3.367891
      0.413960
      2.529128
      4.206654
      1.874221
      4.861560

      4
      3.415567
      0.349163
      2.708094
      4.123039
      1.991473
      4.839660

```python
# solution
ax2 = diab.plot.scatter(x='age',y='y',c='Red',title="Diabetes data with least-squares cubic fit")
ax2.set_xlabel("Age at Diagnosis")
ax2.set_ylabel("Log C-Peptide Concentration")

ax2.plot(predict_df.age, poly_predictions['mean'],color="green")
ax2.plot(predict_df.age, poly_predictions['mean_ci_lower'], color="blue",linestyle="dashed", label='confidence interval')
ax2.plot(predict_df.age, poly_predictions['mean_ci_upper'], color="blue",linestyle="dashed");
ax2.legend();
```

```python

```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.io.wavfile
import subprocess
import librosa
import librosa.display
import IPython.display as ipd

from pathlib import Path, PurePath
from tqdm.notebook import tqdm

import os
import random
import pickle
import pandas as pd
from collections import defaultdict

import utility
import functions
```

## Task 1

#### Settings

```python
N_TRACKS = 1413
HOP_SIZE = 512
OFFSET = 1.0
DURATION = 30
```

do not run again below this line

```python
data_folder = Path("data/mp3s-32k/")
mp3_tracks = data_folder.glob("*/*/*.mp3")
tracks = data_folder.glob("*/*/*.wav")

tracks_list = list(enumerate(tracks))

# save tracks_list
open_file = open("tracks_list.pkl", "wb")
pickle.dump(tracks_list, open_file)
open_file.close()
```

```python
tracks_list[0]
```

    (0, WindowsPath('data/mp3s-32k/aerosmith/Aerosmith/01-Make_It.wav'))

```python
data_folder = Path("query/")
mp3_tracks = data_folder.glob("*.mp3")
query = data_folder.glob("*.wav")

query_list = list(enumerate(query))

# save query_list
open_file = open("query_list.pkl", "wb")
pickle.dump(query_list, open_file)
open_file.close()
```

#### Preprocessing

```python
# convert mp3 to wav
for track in tqdm(mp3_tracks, total=N_TRACKS):
    utility.convert_mp3_to_wav(str(track))
```

do not run again above this line

#### Audio signals

```python
open_file = open("tracks_list.pkl", "rb")
tracks_list = pickle.load(open_file)
open_file.close()
```

```python
# list of track title and author
tracks_title = []
for i in range(len(tracks_list)):
    aux = tracks_list[i][1].__str__().replace('_', ' ').replace('.wav', '').split("\\", 4)
    tracks_title.append([aux[-1], aux[2]])
tracks_titles = list(enumerate(tracks_title))
```

```python
tracks_titles[0]
```

    (0, ['01-Make It', 'aerosmith'])

```python
for idx, audio in tracks_list:
    if idx >= 2:
        break
    track, sr, onset_env, peaks = utility.load_audio_peaks(audio, OFFSET, DURATION, HOP_SIZE)
    utility.plot_spectrogram_and_peaks(track, sr, peaks, onset_env)
```

### 1 Minhashing

By using the function functions.zeros_ones_matrix we compute a zeros-ones matrix of  representative of the dataset.
The number of rows corresponds to the maximum possible peak index and each column is representative of a track.
For each track we assign ones-entries to the indexes of peaks (bins where peaks occur).

```python
m = round(sr*DURATION/HOP_SIZE) # maximum peak index
m
```

    1292

do not run again below this line

```python
# generate and store tracks matrix
tracks_matrix = functions.zeros_ones_matrix(m, 'tracks', tracks_list)
```

      0%|          | 0/1413 [00:00<?, ?it/s]

do not run again above this line

```python
open_file = open("tracks_matrix.pkl", "rb")
tracks_matrix = pickle.load(open_file)
open_file.close()

tracks_matrix.shape, tracks_matrix.nbytes
```

    ((1292, 1413), 1825596)

```python
# pandas visualization of tracks matrix
new_col = [item[1][0].split("-", 1)[1] for item in tracks_titles]
df = pd.DataFrame(tracks_matrix, columns=new_col)
df
```

      Make It
      Somebody
      Dream On
      One Way Street
      Mama Kin
      Write Me a Letter
      Movin Out
      Walking the Dog
      Draw the Line
      I Wanna Know Why
      ...
      Zooropa
      Babyface
      Numb
      Lemon
      Stay Faraway So Close
      Daddy s Gonna Pay For Your Crashed Car
      Some Days Are Better Than Others
      The First Time
      Dirty Day
      The Wanderer

      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

      4
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1287
      1
      0
      0
      0
      0
      0
      0
      0
      0
      1
      ...
      0
      0
      1
      0
      1
      0
      0
      0
      1
      0

      1288
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1289
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1

      1290
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1291
      0
      0
      0
      0
      0
      1
      0
      1
      0
      0
      ...
      1
      1
      0
      0
      0
      0
      0
      0
      0
      0

1292 rows × 1413 columns

We want to associate each track with a *signature* based on the previously defined representative matrix of the dataset.

In order to do this, first we generate $n_{hash}$ random hash functions from an *universal family of hash functions with a prime field*:

\begin{equation}
H = \{h_{c_0, c_1} | c_0 \in \{1,2,..., p - 1\},  c_1 \in \{1,2,..., p - 1\}\}
\end{equation}

\begin{equation}
h_{c_0, c_1}(x) = ((c_1x + c_0)\mod p)\mod n); x \in U
\end{equation}
- $U : \{0,1,..., m-1\}$
- number $n \le m$
- prime number $p \ge m$

The probability that distinct elements $x_1, x_2 \in U$ collide in $n$ is $P(h(x_1) = h(x_2)) \le \frac{1}{n}$

```python
# hash settings
p = 1789 # prime number
n = 1000
nh = 24 # number of random hash functions
```

do not run again below this line

```python
# generate random coefficients
c_0 = random.sample(range(0, p - 1), nh)
c_1 = random.sample(range(0, p - 1), nh)

# save coefficients
with open('coefficient.npy', 'wb') as f:
    np.save(f, c_0)
    np.save(f, c_1)
```

do not run again above this line

```python
with open('coefficient.npy', 'rb') as f:
    c_0 = np.load(f)
    c_1 = np.load(f)

c_0, c_1, [len(set(c_0)), len(set(c_1))]
```

    (array([1576,   17,  372,  516,  208,  297,  307, 1447, 1147,  684, 1078,
             706,  785,  523,  363,  774, 1589,  525, 1212,  539, 1584,  268,
              62,  318]),
     array([ 821, 1457, 1596, 1441, 1051,  889,  611,  204, 1724,  392,  361,
            1412,  560,  194,  394,  869, 1110,  605, 1693, 1389,  590, 1136,
             261, 1190]),
     [24, 24])

For each $k^{th}$ track (column of tracks matrix) we apply the $i^{th}$ *hash function* to each *row-index* with non-zero entry.

The $i^{th}$ component of the signature is equal to the *minimum* among these values.

- $k \in \{1, 2, 3,..., n_{tracks}\}$ where $n_{tracks}$ is the number of tracks in the dataset (variable N_TRACKS)
- $i \in \{1, 2, 3,..., n_{hash}\}$ where $n_{hash}$ is the number of random hash functions generated from the previously defined family (variable nh)

The function functions.minhash performs these operations.

The *signature matrix* - we compute through the function functions.generate_signature - has shape $(n_{hash}, n_{tracks})$.

do not run again below this line

```python
# generate and store signature matrix
signature_matrix = functions.generate_signature(tracks_list, tracks_matrix, 'tracks', nh, c_0, c_1, p, n)
```

      0%|          | 0/1413 [00:00<?, ?it/s]

do not run again above this line

```python
open_file = open("signature_matrix_tracks.pkl", "rb")
signature_matrix = pickle.load(open_file)
open_file.close()

signature_matrix.shape, signature_matrix.nbytes
```

    ((24, 1413), 33912)

```python
# pandas visualization of signature matrix
new_col = [item[1][0].split("-", 1)[1] for item in tracks_titles]
df = pd.DataFrame(signature_matrix, columns=new_col)
df
```

      Make It
      Somebody
      Dream On
      One Way Street
      Mama Kin
      Write Me a Letter
      Movin Out
      Walking the Dog
      Draw the Line
      I Wanna Know Why
      ...
      Zooropa
      Babyface
      Numb
      Lemon
      Stay Faraway So Close
      Daddy s Gonna Pay For Your Crashed Car
      Some Days Are Better Than Others
      The First Time
      Dirty Day
      The Wanderer

      0
      33
      4
      7
      20
      8
      5
      6
      4
      19
      1
      ...
      16
      8
      23
      16
      11
      5
      24
      1
      1
      3

      1
      20
      46
      7
      14
      5
      9
      7
      54
      8
      35
      ...
      45
      13
      24
      1
      14
      11
      1
      7
      3
      7

      2
      13
      19
      8
      2
      19
      3
      14
      2
      2
      48
      ...
      22
      6
      6
      9
      19
      38
      2
      10
      18
      7

      3
      10
      4
      15
      4
      19
      11
      5
      1
      0
      8
      ...
      13
      5
      9
      4
      18
      6
      11
      4
      0
      19

      4
      11
      1
      4
      12
      16
      45
      66
      27
      1
      11
      ...
      2
      15
      27
      2
      17
      10
      14
      27
      3
      6

      5
      21
      6
      14
      36
      6
      10
      21
      53
      18
      45
      ...
      20
      3
      7
      8
      8
      17
      17
      8
      30
      29

      6
      9
      1
      6
      3
      27
      2
      7
      9
      81
      9
      ...
      87
      14
      4
      0
      11
      22
      2
      17
      16
      2

      7
      1
      5
      0
      15
      18
      0
      22
      39
      25
      12
      ...
      34
      2
      2
      1
      12
      1
      28
      6
      12
      17

      8
      4
      14
      0
      0
      0
      35
      7
      68
      7
      9
      ...
      3
      3
      9
      6
      55
      57
      4
      4
      9
      3

      9
      7
      22
      10
      10
      11
      7
      10
      20
      34
      10
      ...
      59
      12
      12
      46
      41
      18
      10
      6
      4
      4

      10
      6
      58
      5
      0
      19
      19
      9
      5
      37
      48
      ...
      1
      6
      53
      26
      12
      20
      9
      0
      6
      2

      11
      7
      9
      11
      18
      0
      9
      6
      7
      11
      33
      ...
      12
      0
      11
      0
      5
      16
      13
      6
      4
      6

      12
      22
      13
      36
      5
      9
      3
      7
      54
      9
      5
      ...
      3
      6
      20
      1
      61
      18
      12
      2
      2
      9

      13
      19
      39
      10
      12
      13
      9
      19
      12
      6
      19
      ...
      17
      17
      4
      9
      6
      9
      6
      18
      3
      11

      14
      4
      17
      1
      1
      16
      7
      7
      6
      11
      1
      ...
      40
      8
      5
      9
      5
      17
      9
      8
      37
      7

      15
      2
      13
      16
      15
      16
      14
      4
      18
      14
      0
      ...
      20
      6
      16
      2
      14
      18
      0
      0
      6
      1

      16
      19
      10
      3
      2
      29
      19
      3
      1
      16
      0
      ...
      14
      0
      0
      30
      58
      32
      0
      12
      27
      2

      17
      4
      30
      19
      11
      24
      17
      10
      4
      4
      8
      ...
      44
      3
      10
      3
      6
      16
      3
      25
      39
      9

      18
      25
      14
      5
      1
      28
      11
      40
      19
      4
      7
      ...
      42
      9
      13
      4
      14
      23
      7
      10
      7
      2

      19
      1
      29
      14
      0
      0
      14
      35
      2
      3
      1
      ...
      4
      29
      0
      2
      9
      20
      10
      15
      1
      8

      20
      15
      10
      12
      14
      1
      24
      37
      6
      22
      5
      ...
      31
      11
      1
      1
      2
      15
      3
      22
      25
      6

      21
      58
      10
      8
      2
      13
      13
      13
      1
      25
      15
      ...
      3
      13
      1
      29
      18
      9
      10
      53
      18
      0

      22
      2
      9
      3
      3
      6
      3
      2
      1
      2
      7
      ...
      29
      21
      19
      2
      9
      15
      8
      19
      1
      3

      23
      0
      4
      2
      3
      5
      16
      12
      6
      5
      1
      ...
      9
      2
      4
      2
      19
      20
      3
      22
      0
      9

24 rows × 1413 columns

Now we're applying the same operations to the queries, stored in the query folder.

```python
open_file = open("query_list.pkl", "rb")
query_list = pickle.load(open_file)
open_file.close()
```

do not run again below this line

```python
# generate and store query matrix
query_matrix = functions.zeros_ones_matrix(m, 'query', query_list)
```

      0%|          | 0/10 [00:00<?, ?it/s]

do not run again below this line

```python
# generate and store signature query
signature_query = functions.generate_signature(query_list, query_matrix, "query", nh, c_0, c_1, p, n)
```

      0%|          | 0/10 [00:00<?, ?it/s]

```python
open_file = open("query_matrix.pkl", "rb")
query_matrix = pickle.load(open_file)
open_file.close()
query_matrix
```

    array([[0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           ...,
           [0, 0, 1, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 1, 0, 0]], dtype=int8)

```python
open_file = open("signature_matrix_query.pkl", "rb")
signature_query = pickle.load(open_file)
open_file.close()
signature_query
```

    array([[  7,  11,  61,  13,  27,  33,  16,   3,  41,  20],
           [  7,   8,   1,   3,  28,   1,   1,   7,  10,  30],
           [  8,  26,  29,  18,   3,   2,   3,   0,   6,   2],
           [ 15,   0,   5,   4,   0,  13,  25,   1,  14,  47],
           [  4,   3,   2,  26,  42,   0,   0,   2,  33,   0],
           [ 14,  10,  47,   7,  26,   4,   8,  13,  51,   0],
           [  6,  14,   7,   1,   9,  12,   9,   1,   6,  12],
           [  0,   6,  11,  13,   6,   0,  14,  11,   1,   3],
           [  0,   6,   9,   0,  26,  15,   3,  17,   9,   4],
           [ 10,   4, 101,   1,  19,   1,  56,   9,  21,   6],
           [  5,   8,  20,   6,   2,   0,  16,  33,  43,  10],
           [ 11,  11,  54,   9,   6,   7,  30,   9, -99,  27],
           [ 36,   5,  13,   2,   1,  31,   3,   1, -71,   1],
           [ 10,   6,  32,   3,  31,   3,   3,  33,  10,   2],
           [  1,   9,  82,   5,   6,  22,   6,  11,  16,  17],
           [ 16,   0,   1,   2,  33,   6,   6,   0,  28,  29],
           [  3,   5,  85,   5,  18,   8,   4,  35,  39,   0],
           [ 19,  10,   1,  18,  10,   3,   3,   6,   8,  34],
           [  5,  15,  84,   2,  16,   9,  65,  11,  37,   6],
           [ 14,   1,   9,  23,   0,   1,   3,   5,  79,  22],
           [ 12,  10,  26,   8,   2,   0,  36,   6,  37,  18],
           [  8,  10,   3,  11,   0,   1,  33,   5,  73,  29],
           [  3,   3,   3,   7,   8,  29,   6,   0,   2,   1],
           [  2,   1,   0,  12,  22,   3,   1,  27,   9,   0]], dtype=int8)

### 1 LSH matching

We look for possible matches with queries by performing Locality Sensitive Hashing.
Steps we made:
- split the signature matrix and the query matrix into equidimensional bands;
- generate hash function of random coefficients for each band;
- hash each split-signature belonging to a specific band to a bucket;
- a dictionary for each band maps bucket to track index;
- a dictionary (print of the following functions) maps query to a list of sets (one for each band) of candidates, track indexes that share the bucket with the query for that band;
- final candidates are those that share at least one bucket with the query;
- evaluate Jaccard similarity.

```python
bands = [4, 6, 8]
```

```python
# generate directories
for i in bands:
    path = f'{i}_bands'
    os.makedirs(path)
```

```python
open_file = open("4_bands/band_2.pkl", "rb")
band_2_4 = pickle.load(open_file)
open_file.close()
band_2_4
```

    defaultdict(None,
                {'coefficients': array([ 848,  918,  546,  499, 1380, 1527]),
                 667: [0, 867],
                 119: [1, 1310, 1328],
                 1246: [2, 770],
                 771: [3],
                 1424: [4, 1333],
                 753: [5],
                 762: [6, 651, 677],
                 1250: [7],
                 599: [8],
                 1126: [9],
                 352: [10],
                 1148: [11],
                 1202: [12],
                 1178: [13, 45, 328],
                 516: [14, 357, 503],
                 945: [15],
                 619: [16, 806, 1012, 1016],
                 20: [17],
                 499: [18],
                 454: [19],
                 1115: [20],
                 942: [21, 75, 1162],
                 70: [22],
                 1044: [23, 349, 466],
                 471: [24, 1195],
                 1415: [25],
                 510: [26, 1086],
                 577: [27],
                 216: [28, 229, 476, 1158],
                 1240: [29, 1144],
                 1506: [30],
                 489: [31, 1099],
                 616: [32],
                 1069: [33],
                 1410: [34, 483, 724],
                 115: [35, 928, 1117],
                 446: [36, 541, 647, 1054],
                 294: [37],
                 346: [38, 122],
                 65: [39],
                 1077: [40, 1187],
                 1384: [41, 51, 1204],
                 851: [42, 123],
                 1352: [43, 987, 1051],
                 535: [44],
                 1034: [46],
                 150: [47],
                 807: [48],
                 905: [49, 1228, 1314, 1323],
                 1073: [50, 249, 310],
                 402: [52],
                 173: [53, 334],
                 351: [54],
                 1026: [55, 1214, 1242],
                 1524: [56],
                 227: [57, 901],
                 1497: [58, 113],
                 328: [59],
                 769: [60, 374],
                 93: [61],
                 1253: [62],
                 694: [63],
                 277: [64],
                 907: [65, 828],
                 1258: [66, 746],
                 1324: [67, 708],
                 178: [68],
                 359: [69],
                 614: [70],
                 951: [71, 1256],
                 1366: [72],
                 623: [73],
                 859: [74],
                 444: [76, 705],
                 113: [77, 1113, 1318],
                 1477: [78],
                 562: [79],
                 970: [80],
                 114: [81],
                 848: [82, 279, 605],
                 628: [83, 1319],
                 794: [84],
                 1389: [85, 831],
                 465: [86, 736],
                 1015: [87],
                 1348: [88],
                 1085: [89, 798],
                 1201: [90],
                 135: [91],
                 773: [92, 1320],
                 852: [93, 103, 942],
                 1261: [94, 1047],
                 247: [95],
                 1454: [96],
                 866: [97],
                 1270: [98],
                 249: [99, 829],
                 272: [100, 556],
                 975: [101],
                 1008: [102, 760, 1065],
                 1039: [104],
                 821: [105, 759, 919],
                 504: [106],
                 108: [107],
                 143: [108, 934],
                 1206: [109, 545, 1057],
                 806: [110],
                 816: [111, 1345],
                 846: [112, 719, 927],
                 59: [114, 398, 530],
                 251: [115, 1249],
                 1516: [116],
                 803: [117, 1139],
                 1120: [118, 1171],
                 693: [119, 1363],
                 982: [120, 520],
                 412: [121, 922],
                 947: [124, 657, 1241],
                 1226: [125, 1151],
                 236: [126, 1254],
                 1418: [127],
                 421: [128, 313],
                 961: [129, 1354, 1411],
                 519: [130, 491, 1000],
                 79: [131, 340, 539, 564],
                 279: [132],
                 69: [133],
                 964: [134, 228],
                 1398: [135, 350],
                 725: [136],
                 1275: [137],
                 950: [138],
                 901: [139],
                 963: [140, 726],
                 539: [141, 1407],
                 1531: [142],
                 453: [143],
                 1141: [144, 181, 698],
                 1000: [145, 155],
                 995: [146, 1040],
                 1168: [147, 581],
                 1227: [148, 628, 756],
                 986: [149, 459],
                 681: [150],
                 1302: [151],
                 922: [152, 769],
                 663: [153],
                 1515: [154, 219, 420],
                 1255: [156],
                 938: [157],
                 1180: [158, 496],
                 1285: [159, 687],
                 1337: [160, 501],
                 122: [161, 1179],
                 676: [162],
                 743: [163, 849],
                 350: [164, 621],
                 68: [165],
                 902: [166, 312, 400],
                 563: [167, 191, 1287],
                 287: [168, 971],
                 1388: [169, 314, 1019],
                 943: [170, 369, 1200],
                 1520: [171, 1131],
                 1152: [172, 836, 841],
                 862: [173, 216, 1324],
                 755: [174, 811],
                 703: [175, 1399],
                 1387: [176, 998],
                 1327: [177],
                 1435: [178, 684, 1267],
                 1149: [179],
                 391: [180, 896, 984],
                 139: [182, 290, 325],
                 1122: [183],
                 929: [184],
                 690: [185],
                 641: [186],
                 481: [187],
                 1184: [188, 981],
                 355: [189],
                 443: [190],
                 1153: [192],
                 228: [193, 1084],
                 1089: [194, 747],
                 683: [195, 364],
                 1502: [196, 675],
                 592: [197, 390],
                 584: [198],
                 1362: [199],
                 121: [200],
                 625: [201],
                 1092: [202],
                 1082: [203],
                 1346: [204, 1167],
                 1171: [205, 1029],
                 439: [206, 1199],
                 1421: [207, 348],
                 373: [208, 341],
                 1426: [209, 618, 1217],
                 868: [210],
                 1303: [211, 321, 333, 1359],
                 1193: [212],
                 341: [213],
                 1547: [214, 408, 1347],
                 600: [215],
                 928: [217, 612],
                 494: [218],
                 50: [220, 432, 1074],
                 1542: [221, 271, 421, 951],
                 381: [222, 529],
                 1221: [223],
                 541: [224],
                 1196: [225, 1311],
                 595: [226, 338],
                 588: [227, 995, 1163],
                 553: [230, 780],
                 1167: [231, 345],
                 390: [232],
                 656: [233],
                 1229: [234, 415],
                 740: [235],
                 605: [236],
                 479: [237],
                 745: [238, 302],
                 741: [239],
                 49: [240, 1352, 1379],
                 1445: [241, 767],
                 1482: [242],
                 291: [243, 360],
                 260: [244],
                 780: [245],
                 917: [246],
                 1305: [247],
                 575: [248],
                 94: [250, 1042, 1276],
                 1490: [251],
                 571: [252],
                 1437: [253, 598, 805, 809],
                 1093: [254, 633],
                 433: [255, 1159],
                 708: [256, 482],
                 1074: [257],
                 1376: [258, 893],
                 1439: [259, 1270],
                 283: [260],
                 54: [261],
                 45: [262],
                 1062: [263],
                 1004: [264],
                 863: [265, 782],
                 1088: [266, 1388],
                 1263: [267, 667],
                 847: [268],
                 1347: [269, 1010, 1237],
                 1274: [270],
                 1002: [272],
                 1252: [273],
                 383: [274, 417, 883, 903],
                 369: [275, 1135],
                 789: [276, 366, 1356],
                 33: [277],
                 657: [278, 1174],
                 1407: [280, 960, 1129],
                 1427: [281],
                 1309: [282],
                 1300: [283, 874, 1201, 1218],
                 670: [284, 1147],
                 292: [285],
                 1219: [286, 343],
                 123: [287, 669, 1332],
                 1020: [288],
                 550: [289],
                 223: [291, 562, 835],
                 1494: [292],
                 612: [293, 674, 1351],
                 890: [294, 851, 1036],
                 817: [295, 452],
                 1367: [296],
                 850: [297],
                 837: [298, 729],
                 258: [299, 407],
                 42: [300],
                 1307: [301],
                 210: [303],
                 474: [304, 492],
                 73: [305],
                 1538: [306, 1245],
                 376: [307],
                 238: [308, 490, 788, 796, 1409],
                 654: [309, 524, 749],
                 1139: [311, 1112],
                 1228: [315],
                 170: [316, 936],
                 529: [317, 1004, 1198],
                 596: [318],
                 102: [319],
                 723: [320],
                 1054: [322],
                 1033: [323, 1039],
                 853: [324, 931, 1173],
                 1500: [326, 1302],
                 1399: [327, 1178],
                 215: [329, 1298],
                 199: [330],
                 451: [331],
                 172: [332, 779, 822, 1056, 1367],
                 136: [335],
                 1359: [336],
                 1199: [337, 1392],
                 60: [339, 904],
                 1245: [342, 955, 1189],
                 573: [344, 506],
                 1194: [346, 494],
                 728: [347, 685],
                 97: [351, 664, 1182],
                 647: [352, 1155],
                 459: [353, 680],
                 0: [354],
                 515: [355, 656],
                 488: [356, 1090],
                 132: [358],
                 336: [359],
                 1464: [361, 750],
                 1036: [362],
                 464: [363, 1202],
                 313: [365],
                 1486: [367],
                 1108: [368, 892, 963],
                 452: [370, 790],
                 1298: [371],
                 338: [372, 661],
                 790: [373],
                 342: [375],
                 262: [376, 900],
                 944: [377, 517],
                 96: [378, 686],
                 875: [379, 572, 672],
                 897: [380],
                 483: [381, 762, 1326],
                 511: [382, 1003],
                 1526: [383],
                 655: [384, 1006],
                 696: [385, 1077],
                 1189: [386],
                 507: [387, 765],
                 315: [388, 461, 1020],
                 314: [389, 1231],
                 1381: [391],
                 802: [392, 1094],
                 282: [393, 522],
                 1211: [394],
                 1339: [395],
                 820: [396, 488],
                 1447: [397],
                 542: [399],
                 306: [401],
                 51: [402, 546],
                 1222: [403],
                 991: [404, 710],
                 159: [405, 973, 989],
                 1248: [406, 966],
                 1511: [409, 1186, 1227],
                 880: [410],
                 1006: [411],
                 1231: [412, 518],
                 1047: [413, 683, 791, 1153],
                 648: [414, 714],
                 1469: [416],
                 831: [418, 655],
                 621: [419, 1028],
                 256: [422, 925],
                 1190: [423, 784, 1389],
                 893: [424, 1194],
                 203: [425],
                 826: [426, 881],
                 792: [427, 511],
                 343: [428, 681],
                 147: [429, 813],
                 557: [430],
                 566: [431],
                 912: [433],
                 1545: [434, 537, 1325],
                 273: [435],
                 458: [436],
                 38: [437],
                 1103: [438],
                 778: [439, 554],
                 1083: [440],
                 329: [441, 1412],
                 189: [442, 641, 718, 776],
                 788: [443],
                 895: [444],
                 1107: [445, 1072],
                 56: [446],
                 702: [447],
                 427: [448],
                 161: [449],
                 569: [450],
                 360: [451, 864],
                 731: [453, 600],
                 1281: [454, 860],
                 46: [455, 803],
                 232: [456, 1295],
                 72: [457],
                 804: [458],
                 1364: [460],
                 434: [462, 1184],
                 1289: [463],
                 337: [464, 678],
                 250: [465],
                 1135: [467],
                 1460: [468],
                 1480: [469],
                 716: [470, 1353, 1361],
                 551: [471],
                 40: [472],
                 82: [473],
                 1049: [474],
                 1493: [475, 489, 505],
                 1023: [477, 631, 1309],
                 196: [478],
                 205: [479],
                 1243: [480, 514],
                 759: [481],
                 1442: [484, 1165, 1321],
                 331: [485],
                 363: [486],
                 878: [487],
                 1191: [493, 1196],
                 873: [495],
                 1052: [497, 786],
                 1269: [498],
                 263: [499],
                 1318: [500],
                 955: [502, 527, 1123, 1252, 1291],
                 1117: [504, 730],
                 560: [507],
                 587: [508, 1177],
                 784: [509],
                 63: [510],
                 1251: [512],
                 107: [513, 620, 887],
                 1479: [515],
                 266: [516, 1092, 1341],
                 1440: [519],
                 508: [521],
                 1124: [523],
                 635: [525, 578, 932, 1067],
                 589: [526, 773],
                 230: [528],
                 316: [531],
                 1360: [532],
                 1488: [533],
                 1096: [534, 990],
                 422: [535],
                 927: [536],
                 1203: [538],
                 854: [540],
                 717: [542],
                 720: [543, 629, 1024],
                 709: [544],
                 1428: [547, 703],
                 393: [548, 552, 715],
                 267: [549],
                 1087: [550, 821],
                 281: [551, 964],
                 1007: [553],
                 819: [555, 1170],
                 1409: [557, 1007],
                 909: [558],
                 1448: [559],
                 768: [560, 1330],
                 356: [561],
                 818: [563],
                 152: [565, 1052],
                 576: [566],
                 1257: [567],
                 570: [568],
                 1507: [569],
                 1462: [570],
                 1086: [571],
                 1035: [573, 789, 1253],
                 913: [574],
                 815: [575],
                 540: [576],
                 1198: [577, 876],
                 1175: [579, 660],
                 764: [580, 1271],
                 581: [582],
                 456: [583, 1248],
                 758: [584, 871, 1181],
                 1392: [585],
                 1021: [586, 846],
                 999: [587, 1264],
                 799: [588, 1002, 1050],
                 1003: [589, 1119],
                 1504: [590, 840],
                 312: [591],
                 1207: [592],
                 564: [593],
                 1192: [594],
                 190: [595],
                 1224: [596],
                 1090: [597],
                 829: [599],
                 644: [601, 839],
                 311: [602],
                 1378: [603, 1243],
                 1102: [604, 1048],
                 41: [606],
                 736: [607],
                 1496: [608],
                 1254: [609],
                 1345: [610],
                 509: [611],
                 388: [613],
                 22: [614, 1278],
                 1476: [615],
                 839: [616],
                 467: [617, 1365],
                 420: [619, 745],
                 416: [622],
                 1284: [623],
                 1195: [624],
                 1094: [625],
                 1433: [626, 668, 801],
                 1503: [627],
                 691: [630],
                 742: [632],
                 1131: [634],
                 699: [635, 948],
                 924: [636],
                 801: [637, 949],
                 1292: [638],
                 496: [639],
                 1150: [640],
                 805: [642, 916],
                 1098: [643, 1305],
                 1498: [644, 1211],
                 814: [645],
                 468: [646, 734],
                 1336: [648],
                 1005: [649, 911],
                 958: [650, 1130],
                 664: [652],
                 973: [653, 751],
                 1404: [654],
                 442: [658, 983],
                 966: [659, 1308],
                 1066: [662],
                 765: [663, 1188],
                 358: [665],
                 899: [666],
                 61: [670],
                 1328: [671],
                 1223: [673],
                 177: [676, 1093],
                 787: [679],
                 613: [682],
                 748: [688],
                 1416: [689, 1290, 1357],
                 845: [690],
                 715: [691, 1251],
                 1321: [692],
                 811: [693, 833],
                 662: [694, 1107],
                 404: [695],
                 1402: [696],
                 330: [697, 1268],
                 1162: [699],
                 1373: [700],
                 1361: [701, 721],
                 78: [702, 758],
                 1259: [704],
                 426: [706, 1289],
                 253: [707, 1369],
                 1403: [709],
                 229: [711],
                 71: [712, 1114],
                 911: [713],
                 28: [716, 1203],
                 1159: [717],
                 1499: [720, 757, 845, 961, 972],
                 1230: [722],
                 658: [723],
                 697: [725, 834, 1377],
                 638: [727],
                 1489: [728],
                 1142: [731, 1374],
                 1396: [732, 823, 974, 1296],
                 140: [733],
                 653: [735],
                 876: [737],
                 1386: [738, 950],
                 144: [739],
                 555: [740, 992],
                 286: [741],
                 1355: [742, 868],
                 1457: [743, 913, 1292],
                 774: [744],
                 892: [748],
                 1317: [752, 1225],
                 490: [753, 854],
                 146: [754, 766, 891],
                 430: [755, 967],
                 361: [761],
                 727: [763, 1132],
                 675: [764, 1344],
                 1070: [768],
                 914: [771, 869],
                 126: [772, 1257],
                 1510: [774],
                 357: [775, 1138],
                 127: [777],
                 953: [778, 800],
                 486: [781],
                 793: [783],
                 687: [785, 810, 879, 1046, 1193],
                 730: [787, 1097],
                 661: [792],
                 894: [793],
                 1084: [794],
                 1099: [795, 946],
                 448: [797, 999],
                 1349: [799],
                 48: [802, 1070],
                 401: [804, 1150],
                 795: [807, 947],
                 1377: [808, 996],
                 1268: [812],
                 284: [814, 1018],
                 1058: [815],
                 1266: [816],
                 233: [817],
                 1147: [818],
                 157: [819],
                 891: [820],
                 993: [824],
                 257: [825],
                 1379: [826, 907],
                 321: [827, 1108, 1300],
                 240: [830],
                 1031: [832, 1382],
                 1136: [837],
                 1001: [838],
                 364: [842],
                 618: [843],
                 704: [844],
                 18: [847],
                 320: [848],
                 585: [850, 920],
                 1137: [852],
                 798: [853, 1216],
                 849: [855, 1206],
                 370: [856],
                 1472: [857, 980],
                 297: [858],
                 659: [859],
                 480: [861],
                 869: [862],
                 738: [863],
                 1038: [865],
                 1394: [866, 1011],
                 1401: [870, 1327],
                 1017: [872],
                 1425: [873, 1364],
                 1165: [875, 994],
                 1125: [877],
                 1325: [878],
                 1061: [880],
                 882: [882],
                 624: [884],
                 80: [885],
                 154: [886],
                 318: [888],
                 679: [889, 933],
                 984: [890],
                 327: [894, 1261, 1376],
                 643: [895],
                 968: [897],
                 941: [898, 986],
                 195: [899],
                 1484: [902],
                 1374: [905],
                 1176: [906],
                 400: [908],
                 349: [909, 958],
                 133: [910],
                 435: [912],
                 202: [914, 921],
                 990: [915],
                 1357: [917],
                 840: [918, 1095, 1141],
                 372: [923],
                 888: [924],
                 649: [926, 1380],
                 520: [929, 1350],
                 684: [930, 1366],
                 673: [935],
                 884: [937],
                 407: [938],
                 1065: [939],
                 182: [940, 1408],
                 201: [941],
                 194: [943],
                 168: [944],
                 1508: [945],
                 11: [952],
                 180: [953],
                 25: [954],
                 1423: [956],
                 64: [957, 1233],
                 718: [959],
                 32: [962],
                 1177: [965],
                 387: [968, 1068],
                 1273: [969],
                 179: [970, 1116],
                 354: [975, 1115],
                 921: [976],
                 700: [977],
                 1530: [978, 1005],
                 810: [979, 1385],
                 930: [982],
                 761: [985, 1078, 1395],
                 193: [988, 1281],
                 946: [991],
                 3: [993],
                 561: [997],
                 952: [1001, 1280],
                 1241: [1008],
                 1438: [1009],
                 1517: [1013],
                 1382: [1014],
                 1461: [1015],
                 1260: [1017],
                 1112: [1021, 1180],
                 1446: [1022],
                 1063: [1023],
                 1104: [1025, 1041, 1303],
                 76: [1026, 1076],
                 972: [1027],
                 476: [1030, 1338],
                 994: [1031],
                 395: [1032],
                 310: [1033],
                 1541: [1034],
                 1183: [1035],
                 590: [1037, 1104],
                 151: [1038],
                 1081: [1043],
                 1205: [1044],
                 735: [1045, 1405],
                 1412: [1049, 1088, 1161, 1208],
                 34: [1053, 1265],
                 254: [1055],
                 298: [1058],
                 1451: [1059],
                 870: [1060, 1137, 1244, 1358],
                 138: [1061],
                 241: [1062],
                 450: [1063],
                 836: [1064],
                 171: [1066],
                 609: [1069],
                 985: [1071, 1082],
                 724: [1073],
                 669: [1075],
                 498: [1079],
                 1157: [1080, 1145],
                 1326: [1081],
                 904: [1083],
                 13: [1085, 1223],
                 574: [1087],
                 531: [1089],
                 1164: [1091],
                 538: [1096],
                 533: [1098],
                 378: [1100],
                 268: [1101, 1337],
                 978: [1102],
                 293: [1103],
                 1105: [1105],
                 10: [1106],
                 640: [1109, 1213],
                 374: [1110],
                 579: [1111],
                 554: [1118],
                 1267: [1120],
                 1151: [1121, 1143],
                 1009: [1122],
                 834: [1124, 1235],
                 428: [1125],
                 1212: [1126],
                 1314: [1127],
                 1127: [1128],
                 484: [1133, 1397],
                 521: [1134],
                 1344: [1136],
                 1237: [1140],
                 524: [1142],
                 117: [1146],
                 219: [1148],
                 937: [1149, 1236],
                 559: [1152],
                 1220: [1154],
                 627: [1156, 1346],
                 27: [1157],
                 1279: [1160],
                 275: [1164],
                 334: [1166],
                 1247: [1168],
                 1534: [1169],
                 167: [1172, 1263],
                 384: [1175],
                 988: [1176],
                 636: [1183],
                 526: [1185],
                 1450: [1190],
                 1272: [1191],
                 603: [1192],
                 583: [1197],
                 1301: [1205],
                 622: [1207],
                 487: [1209],
                 1209: [1210],
                 242: [1212],
                 319: [1215],
                 105: [1219],
                 438: [1220],
                 340: [1221],
                 1027: [1222],
                 1443: [1224],
                 1536: [1226],
                 823: [1229],
                 207: [1230],
                 629: [1232],
                 389: [1234],
                 244: [1238],
                 1475: [1239],
                 1276: [1240],
                 221: [1246],
                 513: [1247],
                 406: [1250],
                 1391: [1255],
                 67: [1258],
                 1487: [1259],
                 224: [1260],
                 1051: [1262],
                 1072: [1266],
                 981: [1269],
                 908: [1272, 1317],
                 881: [1273],
                 432: [1274],
                 77: [1275],
                 979: [1277],
                 1294: [1279],
                 396: [1282],
                 155: [1283],
                 711: [1284],
                 523: [1285],
                 786: [1286],
                 21: [1288],
                 1121: [1293],
                 935: [1294],
                 309: [1297],
                 130: [1299],
                 602: [1301],
                 289: [1304],
                 572: [1306],
                 184: [1307],
                 325: [1312],
                 16: [1313],
                 1521: [1315],
                 642: [1316],
                 896: [1322],
                 1417: [1329],
                 271: [1331],
                 305: [1334],
                 611: [1335],
                 1186: [1336],
                 1310: [1339],
                 514: [1340],
                 1385: [1342],
                 1338: [1343],
                 726: [1348],
                 1495: [1349],
                 89: [1355],
                 677: [1360],
                 756: [1362],
                 567: [1368],
                 91: [1370],
                 424: [1371],
                 188: [1372],
                 1134: [1373],
                 751: [1375],
                 231: [1378],
                 1380: [1381],
                 379: [1383],
                 276: [1384],
                 462: [1386],
                 1154: [1387],
                 120: [1390],
                 1179: [1391],
                 440: [1393],
                 1465: [1394],
                 877: [1396],
                 1408: [1398],
                 1056: [1400],
                 874: [1401],
                 1330: [1402],
                 871: [1403],
                 183: [1404],
                 1068: [1406],
                 186: [1410]})

#### bands = 4

```python
candidates_4 = functions.lsh(bands[0], signature_query, signature_matrix)
```

    defaultdict(None, {0: [{2}, {2, 770}, {1210, 2, 394}, {2, 42}], 1: [{976}, {976}, {976}, {976, 310}], 2: [{1367}, {1056, 779, 332, 822, 1367}, {1367}, {1367}], 3: [{125}, {125, 1151}, {125, 510, 79}, {125, 1351}], 4: [{624, 1024, 1362, 1318}, {1024, 629, 543}, {1024}, {1024}], 5: [{837, 669, 1109}, {1332, 669, 287}, {914, 669}, {752, 650, 669}], 6: [{456}, {456, 1295}, {456, 1072}, {456, 1043}], 7: [{561, 1409}, {561}, {561, 572, 61, 390}, {561}], 8: [{396, 533}, {488, 396}, {396}, {793, 396}], 9: [{1166}, {1166}, {1166, 382, 895}, {1313, 1166}]})

```python
candidates_4
```

    [[2, 770, 42, 1210, 394],
     [976, 310],
     [1056, 822, 1367, 779, 332],
     [79, 1351, 125, 510, 1151],
     [624, 1024, 1362, 629, 1318, 543],
     [837, 650, 752, 914, 1332, 1109, 669, 287],
     [1072, 1043, 456, 1295],
     [561, 1409, 390, 572, 61],
     [533, 488, 793, 396],
     [1313, 1166, 382, 895]]

```python
# threshold = 0.2
functions.find_matches(signature_matrix, signature_query, cand, tracks_titles, 0.2)
```

    The 1st track you are looking for could be Dream On, sung by Aerosmith. Jaccard similarity is equal to 1.0.

    The 2nd track you are looking for could be Penny Lane, sung by Beatles. Jaccard similarity is equal to 0.2.

    The 2nd track you are looking for could be I Want To Break Free, sung by Queen. Jaccard similarity is equal to 1.0.

    The 2nd track you are looking for could be Shout, sung by Depeche Mode. Jaccard similarity is equal to 0.3.

    The 3th track you are looking for could be Everybody s Trying To Be My Baby, sung by Beatles. Jaccard similarity is equal to 0.2.

    The 3th track you are looking for could be October, sung by U2. Jaccard similarity is equal to 1.0.

    The 4th track you are looking for could be Ob-La-Di Ob-La-Da, sung by Beatles. Jaccard similarity is equal to 1.0.

    The 5th track you are looking for could be Karma Police, sung by Radiohead. Jaccard similarity is equal to 1.0.

    The 6th track you are looking for could be Heartbreaker, sung by Led Zeppelin. Jaccard similarity is equal to 1.0.

    The 6th track you are looking for could be Slither, sung by Metallica. Jaccard similarity is equal to 0.2.

    The 7th track you are looking for could be Go Your Own Way, sung by Fleetwood Mac. Jaccard similarity is equal to 1.0.

    The 7th track you are looking for could be Save Me, sung by Queen. Jaccard similarity is equal to 0.2.

    The 8th track you are looking for could be American Idiot, sung by Green Day. Jaccard similarity is equal to 1.0.

    The 9th track you are looking for could be Somebody, sung by Depeche Mode. Jaccard similarity is equal to 1.0.

    The 10th track you are looking for could be Black Friday, sung by Steely Dan. Jaccard similarity is equal to 1.0.

    [[[2, 1.0]],
     [[105, 0.2], [976, 1.0], [413, 0.3]],
     [[96, 0.2], [1367, 1.0]],
     [[125, 1.0]],
     [[1024, 1.0]],
     [[669, 1.0], [821, 0.2]],
     [[456, 1.0], [969, 0.2]],
     [[561, 1.0]],
     [[396, 1.0]],
     [[1166, 1.0]]]

```python
# threshold = 0.3
functions.find_matches(signature_matrix, signature_query, cand, tracks_titles, 0.3)
```

    The 1st track you are looking for could be Dream On, sung by Aerosmith. Jaccard similarity is equal to 1.0.

    The 2nd track you are looking for could be I Want To Break Free, sung by Queen. Jaccard similarity is equal to 1.0.

    The 3th track you are looking for could be October, sung by U2. Jaccard similarity is equal to 1.0.

    The 4th track you are looking for could be Ob-La-Di Ob-La-Da, sung by Beatles. Jaccard similarity is equal to 1.0.

    The 5th track you are looking for could be Karma Police, sung by Radiohead. Jaccard similarity is equal to 1.0.

    The 6th track you are looking for could be Heartbreaker, sung by Led Zeppelin. Jaccard similarity is equal to 1.0.

    The 7th track you are looking for could be Go Your Own Way, sung by Fleetwood Mac. Jaccard similarity is equal to 1.0.

    The 8th track you are looking for could be American Idiot, sung by Green Day. Jaccard similarity is equal to 1.0.

    The 9th track you are looking for could be Somebody, sung by Depeche Mode. Jaccard similarity is equal to 1.0.

    The 10th track you are looking for could be Black Friday, sung by Steely Dan. Jaccard similarity is equal to 1.0.

    [[[2, 1.0]],
     [[976, 1.0]],
     [[1367, 1.0]],
     [[125, 1.0]],
     [[1024, 1.0]],
     [[669, 1.0]],
     [[456, 1.0]],
     [[561, 1.0]],
     [[396, 1.0]],
     [[1166, 1.0]]]

#### bands = 6

```python
candidates_6 = functions.lsh(bands[1], signature_query, signature_matrix)
```

    defaultdict(None, {0: [{2}, {2, 659}, {1088, 2}, {2}, {2, 282, 1340}, {2}], 1: [{72, 771, 976}, {976}, {976, 1257, 527}, {664, 556, 976}, {976, 212}, {976, 1088, 182}], 2: [{1367}, {1367}, {203, 1367}, {112, 1367}, {726, 1367}, {1367}], 3: [{9, 125}, {125}, {125, 661}, {125}, {762, 125}, {125}], 4: [{1024, 137, 425}, {16, 1024}, {1024, 430, 1240}, {224, 33, 1024, 790, 1183}, {1024, 146}, {1024, 1359}], 5: [{669, 1118}, {669}, {1083, 669}, {841, 236, 669}, {669, 182, 375}, {669}], 6: [{456, 1248}, {456}, {456}, {456, 305}, {456}, {456, 641}], 7: [{561, 170}, {561, 691}, {561}, {561}, {561}, {561}], 8: [{396, 276}, {666, 523, 396, 45}, {396}, {396}, {416, 932, 396}, {396}], 9: [{1166}, {221, 1166, 847}, {429, 1166}, {1166}, {1085, 725, 1166}, {763, 1166}]})

```python
candidates_6
```

    [[1088, 2, 659, 282, 1340],
     [1088, 771, 72, 1257, 556, 527, 976, 212, 182, 664],
     [112, 726, 1367, 203],
     [661, 9, 762, 125],
     [1024, 224, 33, 137, 425, 430, 1359, 16, 146, 790, 1240, 1183],
     [182, 375, 841, 1083, 236, 669, 1118],
     [1248, 305, 641, 456],
     [561, 170, 691],
     [416, 276, 932, 666, 523, 396, 45],
     [1085, 725, 763, 429, 221, 1166, 847]]

#### bands = 8

```python
candidates_8 = functions.lsh(bands[2], signature_query, signature_matrix)
```

    defaultdict(None, {0: [{2}, {2}, {2, 1391, 853, 410}, {2, 199}, {2}, {2, 118}, {2}, {2}], 1: [{976, 1153}, {976}, {976, 105, 413}, {976, 357, 975}, {976, 482}, {976}, {976}, {976}], 2: [{674, 1367}, {547, 1367}, {640, 109, 1367}, {1367}, {541, 1367, 671}, {1367}, {1367}, {96, 751, 1271, 253, 1367}], 3: [{125}, {125}, {508, 125, 1078}, {41, 125}, {125}, {125}, {955, 988, 125}, {125}], 4: [{1024, 1001}, {1024}, {1024, 1256, 427, 1005, 1328, 786, 764}, {1024, 1208}, {1024}, {1024, 1073}, {1024}, {1024}], 5: [{669}, {572, 669, 534}, {669}, {669}, {724, 669}, {821, 669, 926}, {628, 669}, {669}], 6: [{456, 914}, {456, 140, 492, 1261, 633, 470, 409, 1083}, {456, 969}, {456}, {456, 292}, {456, 457}, {456}, {456}], 7: [{561}, {472, 561}, {561}, {561}, {385, 666, 561}, {561, 477}, {561}, {561, 811, 1053}], 8: [{396}, {396}, {396, 492, 422}, {396}, {33, 396}, {396}, {396}, {396}], 9: [{1166}, {1166}, {149, 1166}, {435, 1166, 1019}, {209, 1166}, {1166}, {1166}, {1166, 685, 1158}]})

```python
candidates_8
```

    [[2, 853, 118, 199, 410, 1391],
     [976, 1153, 482, 357, 105, 413, 975],
     [640, 96, 674, 547, 109, 751, 1367, 253, 1271, 541, 671],
     [1078, 988, 41, 955, 508, 125],
     [1024, 1256, 1001, 427, 1005, 1328, 1073, 786, 1208, 764],
     [724, 821, 534, 628, 572, 669, 926],
     [292, 456, 969, 409, 457, 140, 492, 1261, 914, 470, 633, 1083],
     [561, 385, 1053, 472, 666, 811, 477],
     [33, 422, 396, 492],
     [209, 435, 149, 1158, 1019, 685, 1166]]

#### Masking matching

We check the results by performing for each signature query a boolean comparison with all the signatures (not only with the candidates).

```python
threshold_0 = 0.5
matches_0 = functions.find_similarities(signature_matrix, signature_query, tracks_titles, threshold_0)
```

    The 1st track you are looking for could be Dream On, sung by Aerosmith. Jaccard similarity is equal to 1.0.

    The 2nd track you are looking for could be I Want To Break Free, sung by Queen. Jaccard similarity is equal to 1.0.

    The 3th track you are looking for could be October, sung by U2. Jaccard similarity is equal to 1.0.

    The 4th track you are looking for could be Ob-La-Di Ob-La-Da, sung by Beatles. Jaccard similarity is equal to 1.0.

    The 5th track you are looking for could be Karma Police, sung by Radiohead. Jaccard similarity is equal to 1.0.

    The 6th track you are looking for could be Heartbreaker, sung by Led Zeppelin. Jaccard similarity is equal to 1.0.

    The 7th track you are looking for could be Go Your Own Way, sung by Fleetwood Mac. Jaccard similarity is equal to 1.0.

    The 8th track you are looking for could be American Idiot, sung by Green Day. Jaccard similarity is equal to 1.0.

    The 9th track you are looking for could be Somebody, sung by Depeche Mode. Jaccard similarity is equal to 1.0.

    The 10th track you are looking for could be Black Friday, sung by Steely Dan. Jaccard similarity is equal to 1.0.

```python
threshold_1 = 0.25
matches_1 = functions.find_similarities(signature_matrix, signature_query, tracks_titles, threshold_1)
```

    The 1st track you are looking for could be Dream On, sung by Aerosmith. Jaccard similarity is equal to 1.0.

    The 1st track you are looking for could be Kyoto Song, sung by Cure. Jaccard similarity is equal to 0.4.

    The 1st track you are looking for could be Cry Freedom, sung by Dave Matthews Band. Jaccard similarity is equal to 0.3.

    The 2nd track you are looking for could be Shout, sung by Depeche Mode. Jaccard similarity is equal to 0.3.

    The 2nd track you are looking for could be I Want To Break Free, sung by Queen. Jaccard similarity is equal to 1.0.

    The 3th track you are looking for could be October, sung by U2. Jaccard similarity is equal to 1.0.

    The 4th track you are looking for could be Ob-La-Di Ob-La-Da, sung by Beatles. Jaccard similarity is equal to 1.0.

    The 4th track you are looking for could be Sail Away Sweet Sister, sung by Queen. Jaccard similarity is equal to 0.3.

    The 5th track you are looking for could be Karma Police, sung by Radiohead. Jaccard similarity is equal to 1.0.

    The 6th track you are looking for could be Cross-Tie Walker, sung by Creedence Clearwater Revival. Jaccard similarity is equal to 0.3.

    The 6th track you are looking for could be Heartbreaker, sung by Led Zeppelin. Jaccard similarity is equal to 1.0.

    The 7th track you are looking for could be Go Your Own Way, sung by Fleetwood Mac. Jaccard similarity is equal to 1.0.

    The 7th track you are looking for could be Fun It, sung by Queen. Jaccard similarity is equal to 0.3.

    The 8th track you are looking for could be American Idiot, sung by Green Day. Jaccard similarity is equal to 1.0.

    The 9th track you are looking for could be Somebody, sung by Depeche Mode. Jaccard similarity is equal to 1.0.

    The 10th track you are looking for could be Hero Of The Day, sung by Metallica. Jaccard similarity is equal to 0.3.

    The 10th track you are looking for could be Black Friday, sung by Steely Dan. Jaccard similarity is equal to 1.0.

```python
matches_1
```

    [[[2, 1.0], [224, 0.4], [302, 0.3]],
     [[413, 0.3], [976, 1.0]],
     [[1367, 1.0]],
     [[125, 1.0], [967, 0.3]],
     [[1024, 1.0]],
     [[171, 0.3], [669, 1.0]],
     [[456, 1.0], [928, 0.3]],
     [[561, 1.0]],
     [[396, 1.0]],
     [[787, 0.3], [1166, 1.0]]]


# 一、从最优化引出拉格朗日数乘法

## 1、一个最简单的无约束优化问题

在学习与工程之中，我们时常会遇到一些优化的问题，也就是要对某个目标函数求取极值，最简单的形式如下。

$$\min_{x \in \mathbb{R}^{n}} f(x)$$

这个式子表达的含义是以x为自变量，求取f(x)的最小值。如果想要求取最大值，可以把f加一个负号，这样把$\max_{x}f(x)$转换为等价的$\min_{x}-f(x)$就行了。另外需要注意的是x并非一个标量数值，而是n维实数空间上的一个向量。

这样的例子有很多，最常见的就比如我们在机器学习中经常碰到的最小化代价函数。对于这样的问题，我们求解的方法很简单，只要能够求出一个点$x^{\prime}$，使得 $\nabla f\left(x^{\prime}\right)=0$即可。

当然我们也知道，只有当f是凸函数的时候，这样求出来的才是全局最优点，否则不过是个局部最优点罢了。不过这就不在我们的讨论范围内了。现在我们姑且假定讨论的f都是凸函数。

## 2、带有等式约束的优化问题

我们在上一节的基础上进一步，如果f是有约束的呢？

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad h_{i}(x)=0, i \in[1, l]$$

在这个问题中，我们对在定义域上的x有了$l$个等式约束$h_i$。这样一来我们就不能随意的计算导数等于0的点作为最优点了。那要怎么做呢？

这时就需要用到拉格朗日乘数法了，具体而言，我们首先来构建这么一个拉格朗日函数:

$$L(x, \lambda)=f(x)+\sum_{i=1}^{l} \lambda_{i} h_{i}(x)$$
$$x \in \mathbb{R}^{n}, \lambda \in \mathbb{R}^{l}$$

这里λ是新引入的参数，被称为拉格朗日乘子，在这里它是一个$l$维的向量。

接着我们需要求解方程:

$$x \in \mathbb{R}^{n}, \lambda \in \mathbb{R}^{l}$$

求解出来的结果$x^\prime$必然就是最大值或最小值对应的点，只要稍加检验就可以得出结果了。

那么为什么会这样呢？这里我们稍加解释，如下图所示，f(x,y)是我们的目标函数f(x)，而g(x,y)则是我们的约束条件h(x)。

显然我们的目标就是求出在f和g的交点上的最小值。稍加思索，我们不难发现，这样的点一定是两个g和f的某个等高线相切的一个点。原因很简单：假定我们的目标$x^*$不是切点，那么它必然可以沿着负梯度的方向朝着更低的方向前进，那这就意味着它必然不是最小值对应的点。

所以我们就有结论:

若$x^*$是被等式h约束的最优化问题$\min f$的解，则f和h在$x^*$处相切。

既然f和h在$x^*$上相切，那就意味着二者在这一点的梯度的方向相同，即:

$$\nabla f=\lambda \nabla h, \quad \lambda \neq 0$$

稍加整理，我们就可以得出结论，只要x满足:

$$\left\{\begin{array}{l}\frac{d f}{d x}+\lambda \frac{d h}{d x}=0 \\ h(x)=0\end{array}\right.$$

那么x就是我们希望的最优点。稍加整理就是上面的拉格朗日函数的形式了。

## 3、再加上不等式约束

然而实际生活中我们可能碰到的问题比上面的还要复杂，不仅仅是有等式约束，还可能会有不等式约束。

现在假定$f(x),c_i(x),h_i(x)$都是定义在$\mathbb{R}^n$上的连续可微函数，则考虑约束最优化问题:

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad h_{j}(x)=0, j \in[1, l]$$

这就是我们实际会遇到的优化问题的基本形式。如何求解这个优化问题也就是这篇文章主要要探讨的。

关于具体如何求解，我们后面再说，先把一些需要用到的数学概念理清。

## 4、预备的数学知识

**数学符号说明**

- min/max符号

$$\min _{x} f(x)$$

表示求取当f(x)最小时，x的取值

$$\max _{x \in D} f(x, \lambda)$$

表示固定变量$\lambda$在满足x在D集合里，f(x)取最大值时，x的取值

$$v=\max _{\lambda} \min _{x} f(x, \lambda)$$

这个式子要从右往左看，首先把$\min _{x} f(x, \lambda)$这一项理解成关于λ的函数g(λ)，然后再对g求最大值，即:

$$g(\lambda)=\min _{x} f(x, \lambda)$$

$$v=\max _{\lambda}g(\lambda)$$

- inf和sup符号

inf是infimum(下确界)的简称，而sup是supremum(上确界)的简称。它们和min还有max很相似，但是细节部分略有不同。具体来讲，一个例子足以:

有函数$f(x)=\frac{sin(x)}{x}$:

显然f在0处没有定义。那么:

$\max{f}$不存在

$\sup{f}$存在，且为1

- 其他符号

$dom f$: 函数f的定义域

$⋂S_i$：对若干集合$S_i$求交集

**凸函数和凹函数**

没错，其实是上凹下凸，这是国际惯例，我们国内的定义(也就是中学教的)是反过来的，我们在此后一律使用国际标准。

另外关于凸函数和凹函数有性质如下:

1. 若函数$f:\mathbb{R}^n \to \mathbb{R}$是凸的，则:

$$\forall x, y \in \operatorname{dom} f, \forall \theta \in[0,1]$$

$$\theta f(x)+(1-\theta) f(y) \leq f(\theta x+(1-\theta) y)$$

表示凸函数图像上的任意两点做连线，连线上的函数值（红色直线）都会大于被截断这段弧线（绿色弧线）上的函数值。$\theta$就是来控制线上点的范围的。

2. 若函数$f:\mathbb{R}^n \to \mathbb{R}$是凹的，则:

$$\forall x, y \in \operatorname{dom} f, \forall \theta \in[0,1]$$

$$\theta f(x)+(1-\theta) f(y) \geq f(\theta x+(1-\theta) y)$$

**仿射函数**

仿射函数，即最高次数为1的多项式函数。常数项为零的仿射函数称为线性函数。

假设有$\vec{x} \in \mathbb{R}^{n}, \vec{b} \in \mathbb{R}^{m}$，矩阵$A_{m×n}$那么：

$$\vec{x} \rightarrow A \vec{x}+\vec{b}$$

被称为$\mathbb{R}^n \rightarrow \mathbb{R}^m$的仿射变换，这一过程被称为仿射函数。

$$f(x)=A x+b, x \in \mathbb{R}^{n}$$

比如最简单的:

$$a_{1} x_{1}+a_{2} x_{2}+\cdots+a_{n} x_{n}+b$$

就是一个仿射函数。

仿射函数有一个非常重要的性质，那就是它既凹又凸，准确来讲在凹凸函数的那个不等式中，仿射函数是可以取等号的【可以先理解为仿射函数几何图像就是一条直线，那直线肯定是取等号的】:

$$f(\theta x+(1-\theta) y)=\theta f(x)+(1-\theta) f(y)$$

**凸优化**

优化问题的基本形式:

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad h_{j}(x)=0, j \in[1, l]$$

而所谓的凸优化，是满足一些特定条件的优化问题，它要求:

1. $f(x)$是凸函数
2. $c_i(x)$是凸函数
3. $h_j(x)$是仿射函数

也就是:

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad a_j(x)=b_j, j \in[1, l]$$

凸优化有一个重要的性质：任意位置的局部最优解同时也是全局最优解。

# 三、从广义拉格朗日函数到拉格朗日对偶函数

原问题 (Primal Problem):

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad h_{j}(x)=0, j \in[1, l]$$

首先我们做两个约定:

- 我们不假定原函数f的凹凸性，也就是f可以是非凸非凹函数
- 记问题的定义域$D=(\operatorname{dom} f) \cap\left(\bigcap_{i=1}^{k} c_{i}\right) \cap\left(\bigcap_{i=1}^{l} h_{i}\right)$
- D!=∅
- 我们约定最终求出来的最优结果用$p^*$

## 1、什么是对偶

对于原问题我们通常用拉格朗日对偶的方式来求解。啥是对偶？对偶说白了就是实质相同但从不同角度提出不同提法的一对问题。有时候原问题 (Primal Problem) 不太好解，但是对偶问题 (Dual Problem) 却很好解，我们就可以通过求解对偶问题来迂回地解答原问题。

在这里我们用的是拉格朗日对偶的方法来解决，既然我们说用对偶的方法的原因在于原问题不好解，那么这里也是如此吗？

当然了。

首先我们来看看原问题有什么难的地方:

1. 约束条件太多

   很显然约束越多，问题就越难解决，原问题中总共有$k+l$个约束，相当麻烦

2. 原问题凹凸性不明确

   之前我们说过，“不假定原函数f的凹凸性”，这就意味着我们无法将凸优化的方法应用在原问题中

那么它的拉格朗日对偶问题有什么优点呢？

1. 只有一个约束

2. 拉格朗日对偶问题一定是凹的

这里先做一个感性的认识，细节方面接下来慢慢说。

## 2、广义 Lagrange 函数

早前我们说过，对于等式约束问题可以用构建拉格朗日函数的方法来求解。这里也一样，我们可以为原问题构建一个广义拉格朗日函数$\mathcal{L}$:

$$\mathcal{L}: \mathbb{R}^{n} \times \mathbb{R}^{\mathbb{k}} \times \mathbb{R}^{l} \rightarrow R$$
$$\mathcal{L}(x, \lambda, \mu)=f(x)+\sum_{i=1}^{k} \lambda_{i} c_{i}(x)+\sum_{j=1}^{l} \mu_{j} h_{j}(x)$$
$$\vec{x} \in \mathbb{R}^{n}, \vec{\lambda} \in \mathbb{R}^{\mathbb{k}}, \vec{\mu} \in \mathbb{R}^{l}$$

$\vec{\lambda}$和$\vec{\mu}$被称为拉格朗日乘子向量。

有了这个广义拉格朗日函数，现在我们分别可以定义另外对偶函数g。

## 3、Lagrange 对偶函数g

我们可以根据$\mathcal{L}$定义一个拉格朗日对偶函数(Lagrange Dual Function)$g(\lambda, \mu)$:

$$
\begin{align}
g(\lambda, \mu)&=\inf _{x \in D} \mathcal{L}(x, \lambda, \mu)\\
&=\inf _{x \in D}\left(f(x)+\sum_{i=1}^{k} \lambda_{i} c_{i}(x)+\sum_{j=1}^{l} \mu_{j} h_{j}(x)\right)\\
\lambda \geq 0 &
\end{align}
$$

这个对偶函数g有一个非常重要的性质：它一定是一个凹函数，我们可以证明一下。

首先我们明确一个凹函数具有性质:

$$f(\theta x+(1-\theta) y) \geqslant \theta f(x)+(1-\theta) f(y)$$

要证明g是一个凹函数就是要证明它满足这个不等式。

**第一步**

我们首先记:

$$g(\lambda, \mu)=\inf _{x \in D}\left\{\mathcal{L}\left(x_{1}, \lambda, \mu\right), \mathcal{L}\left(x_{2}, \lambda, \mu\right), \cdots, \mathcal{L}\left(x_{n}, \lambda, \mu\right)\right\}, n \rightarrow+\infty$$

这个好理解，就是把连续的函数离散成无限个点的集合，这样就可以把g看作一个逐点求下界的无穷集合。

**第二步**

简记:

$$\gamma=(\lambda, \mu), g(\gamma)=g(\lambda, \mu)$$

这个主要是为了一会推导的时候方便写。

**第三步**

证明过程如下:
$$
\begin{align}
g\left(\theta \gamma_{1}+(1-\theta) \gamma_{2}\right)&=\inf \left\{\mathcal{L}\left(x_{1}, \theta \gamma_{1}+(1-\theta) \gamma_{2}\right), \mathcal{L}\left(x_{2}, \theta \gamma_{1}+(1-\theta) \gamma_{2}\right), \cdots, \mathcal{L}\left(x_{n}, \theta \gamma_{1}+(1-\theta) \gamma_{2}\right)\right\}\\
&\geqslant \inf \left\{\theta \mathcal{L}\left(x_{1}, \gamma_{1}\right)+(1-\theta) \mathcal{L}\left(x_{1}, \gamma_{2}\right), \cdots, \theta \mathcal{L}\left(x_{n}, \gamma_{1}\right)+(1-\theta) \mathcal{L}\left(x_{n}, \gamma_{2}\right)\right\}\\
&\geqslant \theta \inf \left\{\mathcal{L}\left(x_{1}, \gamma_{1}\right), \mathcal{L}\left(x_{2}, \gamma_{1}\right), \cdots, \mathcal{L}\left(x_{n}, \gamma_{1}\right)\right\}+(1-\theta) \inf \left\{\mathcal{L}\left(x_{1}, \gamma_{2}\right), \mathcal{L}\left(x_{2}, \gamma_{2}\right), \cdots, \mathcal{L}\left(x_{n}, \gamma_{2}\right)\right\}\\
&=\theta g\left(\gamma_{1}\right)+(1-\theta) g\left(\gamma_{2}\right)
\end{align}
$$

解释过程：

第一行到第二行是因为$L(x_i, \gamma)$中的x的值已固定，所以$f_{i}(x), i=0,1,2, \cdots, m$ 和 $h_{i}(x), i=0,1,2, \cdots, p$

都应该看做常数，所以此时的$L(x_i, \gamma)$是$\gamma$的仿射函数，而仿射函数是既凸且凹的，对第一行右边中的每一个拉格朗日函数都运用其凹性，就可以得到第二行。

而从第二行到第三行运用的是一个简单的数学原理：

设有两个实数集合a和b:

$$a=\left\{a_{1}, a_{2}, \cdots, a_{n}\right\}$$

$$b=\left\{b_{1}, b_{2}, \cdots, b_{n}\right\}$$

则对于所有的i、j有：

$$\min \left\{a_{i}+b_{j}\right\} \geq \min \{a\}+\min \{b\}, \quad i, j \in N^{+}$$

最后通过图像来解释:

上图中，每条直线表示的是一个$L(x_i, \gamma)$假想有一条平行于上图中y轴方向的直线，这条直线沿着x轴方向平移，这条直线与上图中所有的$L(x_i, \gamma)$相交，这些交点的最小值（y轴方向的值，因为y轴方向对应于$L(x_i, \gamma)$的值，x轴方向对应于每一个$x_i$）就是$g(\gamma)$，也就是（第一步公式）要表达的意思。

由于这条直线每到一处，就对应于一个$x_i$从而逐点逐点地获得$g(\gamma)$，所以就称对偶函数是一族关于$\gamma$的仿射函数的逐点下确界。

**总结: 不管原函数f的凹凸性，它的对偶函数g一定是凹函数。**

## 4、对偶函数与原函数的关系

我们之所以需要这么一个对偶函数是因为g有一个非常重要的作用，什么意思呢？其实这是 Dual Function 一个很重要的性质:

$$\forall \lambda \geqslant 0 \Rightarrow g(\lambda, \mu) \leqslant p^{*}$$

换而言之，只要λ不小于0，g的值永远不会超过$p^*$

要证明这个定理并不难，我们首先假设$\hat x \in D$是原问题的一个可行解:

$$
\begin{align}
&\because c_{i}(\hat{x}) \leqslant 0, h_{i}(\hat{x})=0\\
&\therefore \lambda_{i} c_{i} \leqslant 0, \mu_{i} h_{i}=0\\
&\therefore \mathcal{L}(\hat{x}, \lambda, \mu)=f(x)+\sum_{i=1}^{k} \lambda_{i} c_{i}(\hat{x})+\sum_{j=1}^{l} \mu_{j}h_{j}(\hat{x}) \leqslant f(x)\\
&Meanwhile\ g(\lambda, \mu)=\inf \mathcal{L}, \quad p^{*}=\min f\\
&\therefore g(\lambda, \mu) \leqslant \mathcal{L} \leqslant p^{*}
\end{align}
$$

这给了我们一个重要的启示，无论如何，$p^*$都不会小于$\max{g(\lambda, \mu)}$。

# 四、从原问题到拉格朗日对偶问题

在此之前，我们讲了原问题，讲了广义拉格朗日函数和拉格朗日对偶函数，又讲了这两个之间的关系。那么这有什么用呢？

用处当然很大啦！现在我们不妨这么一想，既然Lagrange对偶函数给出了最优解$p^*$的下界，那么从拉格朗日对偶函数中可以得到的最佳下界是什么呢?

我们来捋一捋:

1. 首先我们明确一件事，我们的目的是找到最优解$p^*$

2. 有时候$p^*$其实并不一定能解出来，这种情况下，我们希望可以给出一个尽可能地逼近$p^*$的值

3. 既然我们已经知道了g可以给出下界，那么哪个值能够尽可能逼近呢？

4. 答案是:$\max g(\lambda, \mu)\ (s.t\ λ⩾0)$

明白了这一点，我们终于可以介绍拉格朗日对偶问题了。

首先再次回顾一下原问题。

$$Primal Problem$$

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad h_{j}(x)=0, j \in[1, l]$$

然后隆重介绍对偶问题的形式。

$$Lagrange Dual Problem$$

$$\max _{\lambda, \mu} g(\lambda, \mu)=\max _{\lambda, \mu} \inf _{x \in D} \mathcal{L}(x, \lambda, \mu)$$
$$s.t \quad \lambda_{i} \geqslant 0, i=1,2, \ldots, k$$

参考上面那幅图，其实这个所谓的对偶问题在干什么已经很清楚了。我们把原问题的结果设为$p^*$，而对偶问题的结果记为$d^*$。

现在回顾一下之前说的求解对偶问题的好处，我们再来看看为什么需要拉格朗日对偶。

首先对于原问题:

1. 原问题的约束太多了，又是等式又是不等式
2. 原问题不一定是一个凸优化问题，所以即便找到了貌似是$p^*$的点，也很可能不过是个局部最优点。

那 Lagrange 对偶问题就好一些吗？还真是，我们来看看:

1. 约束少了，这是很明显的，少了$l$个，而且剩下的k个约束也比原来的简单一些
2. 最最重要的，对偶问题一定是一个凸优化问题，所以很多凸优化的手段全可以用上了

总结一下就是:

我们希望求$p^*$，但是$p^*$实在不好求，所以退而求其次去求$d^*$，这个就好求多了。

并且由于$d^* \leqslant p^*$，我们在求出了$d^*$后即便不能得到$p^*$，也可以得到$p^*$的下界；而在一些非常理想的情况下，$d^* = p^*$。

$d^* = p^*$如果真能这样那简直就是天堂了，显然一般情况下这两个是不同的，但是在满足某些特殊条件时，这个等式就可以成立了。具体这些条件是什么，我们在接下来的几节里再来谈谈。

# 五、弱对偶与强对偶

所谓强弱对偶，指的是 Lagrange 对偶问题的一种性质。

## 1、弱对偶

弱对偶就是$𝑑^∗<𝑝^∗$

## 2、强对偶

强对偶就是$𝑑^∗=𝑝^∗$

之所以称之为“强”对偶，正是因为这种性质相对于“弱”对偶而言对我们的意义更加重大。很显然，当我们待求解的问题是强对偶的话，那就意味着我们通过对偶问题求的的解就不再仅仅是原问题解的一个下界了，而是$𝑝^∗$本身。用人话来讲就是：

**如果满足强对偶，那只要我们求出了$d^*$，我们就知道了$𝑝^∗$的值**

显然，强对偶并不是什么时候都成立的，必须要满足一定的条件才会有强对偶这种东西。

下一节，我们着重讲解这些条件是什么 (充分条件、必要条件)。

# 六、关于强对偶和最优的一些条件

首先要明确一件事情：使满足强对偶的条件有很多种，我们这里只介绍最基本的。

## 1、Convex + Slater

如果:

- 原问题是凸优化

- 满足 Slater 条件

则一定满足强对偶

注: 这是强对偶的一个充分条件

下面具体解释一下含义。

如果原问题是凸问题，必须满足：

1. $f(x)$是凸函数
2. $c_i(x)$是凸函数
3. $h_j(x)$是仿射函数

满足Slater条件：

$$\exists x \in D$$

$$c_{i}(x)<0, i \in[1, k]$$

$$Ax=b$$

其中$Ax=b$是把$a_j x=b_j$合起来写的写法，我们的主要关注点在$c_i$上。我们把满足该条件的点称为是严格可行的，因为不等式约束要求严格成立。

之所以称上述的 Slater 条件是严格的，是因为 Slater 还有一个弱化的版本。

$$c_{i}(x)是仿射函数,i=[1, p] \& p \leqslant k$$

$$则弱化的Slater条件为:$$

$$\exists x \in D$ $$

$$c_{i}(x) \leqslant 0, i \in[1, p]$$

$$c_{i}(x)<0, i \in[p+1, k]$$

$$Ax=b$$

任意满足:

1. 凸优化
2. 强或弱 Slater 条件

的优化问题同样满足强对偶性 (充分条件)。

## 2、KKT 条件

>对于无约束最优化问题，其搜索空间是无界的，只要确定了搜索方向和步长因子，便可以在一轮或几轮迭代之后找到最优解或近似最有解。这里举个不太恰当的例子，无约束最优化如同在浩瀚的宇宙中寻找体积最大的星球，你按照一定的策略去找，不用担心越界。
>\
>而约束的优化问题就不同了，在寻找最优解的过程中始终要在某一个约束范围空间内进行。还是以上述例子说明，约束最优化如同在浩瀚的宇宙中寻找体积最大的星球，你按照一定的策略去找，但是不能离开银河系。KKT条件正是这类最优化问题的最优解必须要满足的条件，言外之意便是只要是最优解，必然满足这个条件，反之则不然；但对于凸问题，确是充分必要条件，这一点很重要，很多问题是凸的，这就为问题求解带来很大的便利性。

有必要明确一些很重要的点。

首先是 KKT 条件实际上应该分为两种情况讨论:

- Case 1: 原问题为非凸问题情况下的 KKT

- Case 2: 原问题为凸问题情况下的 KKT

为什么要分两点来讨论？因为在两种情况下，KKT 条件作为一个“条件”的充分性和必要性是不同的，自然用法也不同。

再有一个问题是，无论哪种情况，KKT 都是着眼于“最优解”这个议题进行讨论的。

下面分两种情况讨论。

**非凸问题下的 KKT**

当原问题并非凸优化(或者不清楚、不关心是不是凸优化)时，KKT 条件是一种用来描述强对偶情况下最优解性质的条件

换而言之，若强对偶性质成立，那么满足最优解的点一定满足 KKT 条件；KKT 条件是强对偶一个必要条件，但无法作为充分条件来使用

我们首先给出 KKT 条件的具体数学描述，然后逐行解释。

假设:

1. 原问题中的函数均可微
2. 强对偶成立

$x_*$和$(\lambda^*, \mu^*)$分别为原问题和对偶问题的某对最优解

则:

$$\begin{array}
{lr}c_{i}\left(x^{*}\right) \leqslant 0 & i=1, \ldots, k \\
h_{i}\left(x^{*}\right)=0 & i=1, \ldots, l \\
\lambda_{i}^{*} \geqslant 0 & i=1, \ldots, k \\
\lambda_{i}^{*} c_{i}\left(x^{*}\right)=0 & i=1, \ldots, k \\
\nabla f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} \nabla c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*} \nabla h_{i}\left(x^{*}\right)=0
\end{array}$$

这就是 KKT 条件了，第一行和第二行是原问题的约束，必须遵守；第三行则是弱对偶的必要条件。最后一行，我们知道，既然$x^*$是最优点，那么理所当然的，原问题$\mathcal{L}$关于$x^*$的导数必须等于0。

现在难点主要在第四行，其实第四行的不等式是一个被称为“互补松弛条件”的东西。我们做一个简单的证明。

首先我们知道

$$
\begin{align}
f\left(x^{*}\right)&=g\left(\lambda^{*}, \mu^{*}\right) \\
&=\inf _{x} \mathcal{L}\left(x^{*}, \lambda^{*}, \mu^{*}\right)
\end{align}
$$

所以有:

$$
\begin{align}
f\left(x^{*}\right)& \leq \mathcal{L}\left(x^{*}, \lambda^{*}, \mu^{*}\right) \\
&=f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*} h_{i}\left(x^{*}\right)
\end{align}
$$

同时我们知道，$\lambda \geq 0$ 且 $c_i(x) \leq 0$，而$h_i=0$，所以有：

$$f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*} \leq f(x^*)$$

联立上面几个式子，有:

$$
\begin{align}
f\left(x^{*}\right)&=g\left(\lambda^{*}, \mu^{*}\right) \\
&=\inf_x f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*}\\
&=f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*}\\
&=f\left(x^{*}\right)
\end{align}
$$

注意到第二行和第三行想等，而$\lambda_i c_i \leq 0$，所以为了让等式成立，必然有:

$$\lambda_i c_i(x) = 0$$

这被称为互补松弛条件。

把这几个条件一股脑叠加在一起，就构成了 KKT 条件。对于目标函数和约束函数可微的优化问题，如果强对偶成立，则任意一对原问题的最优解和对偶问题的最优解满足 KKT 条件。

**凸问题下的 KKT**

当原问题为凸优化时，KKT 条件在非凸的基础上有多了找到最优点的功能
在这种情况下，那么满足 KKT 条件的点一定是原问题和对偶问题的最优解；KKT 条件成了强对偶和最优解的充要条件

也就是说:

1. 若原问题是Convex的
2. $\exists x^*, \lambda^*, \mu^*$满足:

$$\begin{array}
{lr}c_{i}\left(x^{*}\right) \leqslant 0 & i=1, \ldots, k \\
h_{i}\left(x^{*}\right)=0 & i=1, \ldots, l \\
\lambda_{i}^{*} \geqslant 0 & i=1, \ldots, k \\
\lambda_{i}^{*} c_{i}\left(x^{*}\right)=0 & i=1, \ldots, k \\
\nabla f\left(x^{*}\right)+\sum_{i=1}^{k} \lambda_{i}^{*} \nabla c_{i}\left(x^{*}\right)+\sum_{i=1}^{l} \mu_{i}^{*} \nabla h_{i}\left(x^{*}\right)=0
\end{array}$$

那么$x^*$和$(\lambda^*, \mu^*)$分别是原问题和对偶问题的最优解，且最优对偶间隙为0，强对偶性满足。

**几种条件之间的关系的总结**

1. 对于任意问题

强对偶 + 最优解 ⇒ KKT条件

2. 对于 Convex + 可微 的问题

KKT 条件 ⇒ 强对偶 + 最优解

3. Convex + Slater ⇒ 强对偶

# 七、总结

**原问题：**带约束问题不好求解

$$\min_{x \in \mathbb{R}^{n}} f(x)$$
$$s.t \quad c_{i}(x)\leq 0, i \in[1, k]$$
$$\quad \quad h_{j}(x)=0, j \in[1, l]$$

**拉格朗日函数：**将带约束问题变成无约束问题，但是仍然不好解

$$\mathcal{L}: \mathbb{R}^{n} \times \mathbb{R}^{\mathbb{k}} \times \mathbb{R}^{l} \rightarrow R$$
$$\mathcal{L}(x, \lambda, \mu)=f(x)+\sum_{i=1}^{k} \lambda_{i} c_{i}(x)+\sum_{j=1}^{l} \mu_{j} h_{j}(x)$$
$$\vec{x} \in \mathbb{R}^{n}, \vec{\lambda} \in \mathbb{R}^{\mathbb{k}}, \vec{\mu} \in \mathbb{R}^{l}$$

简化为：

$$\min_x\max_{\lambda, \mu}\mathcal{L}(x, \lambda, \mu)$$

**拉格朗日对偶函数：**拉格朗日函数不好解，我们看看它的对偶问题（对偶函数一定是凹函数），对偶问题就是能够得出和原问题相同解的问题

$$
\begin{align}
g(\lambda, \mu)&=\inf _{x \in D} \mathcal{L}(x, \lambda, \mu)\\
&=\inf _{x \in D}\left(f(x)+\sum_{i=1}^{k} \lambda_{i} c_{i}(x)+\sum_{j=1}^{l} \mu_{j} h_{j}(x)\right)\\
\end{align}
$$

$$\max _{\lambda, \mu} g(\lambda, \mu)=\max _{\lambda, \mu} \inf _{x \in D} \mathcal{L}(x, \lambda, \mu)$$
$$s.t \quad \lambda_{i} \geqslant 0, i=1,2, \ldots, k$$

$$p^*是原问题的解$$

简化为：

$$\max_{\lambda, \mu}\min_x\mathcal{L}(x, \lambda, \mu)$$

**KKT约束：**因为对偶问题可以通过不断迭代找到接近于原问题的解，但是不一定等于，所以我们加上kkt条件约束就能保证等于

kkt约束将上图中的$p^*$和$d^*$的距离消除，保证$p^* = d^*$

```python

```


```python
# !pip install jupyter_dashboards
# !jupyter dashboards quick-setup --sys-prefix
# !pip install ipywidgets
# !jupyter nbextension enable --py widgetsnbextension
```

## Smoothing
### Linear Interpolation

Linear lnterpolation
Use both unigram and bigram counts, but we believe them to different extents

\begin{align}
{\lambda_1} & = \ \dfrac{C(w_i-1)}{C(w_i-1)+\lambda} \\
\end{align}

```python
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual,IntSlider
import ipywidgets as widgets
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
def f(lambd,c):
    context = np.arange(c)
    lamb1 = context/(context + lambd)
    plt.plot(context,lamb1)
    plt.ylim(0,1)

```

```python
interact(f, lambd=IntSlider(min=1, max=300),
            c=IntSlider(min=1, max=10000,step=1))
```

    interactive(children=(IntSlider(value=1, description='lambd', max=300, min=1), IntSlider(value=1, description=…

    <function __main__.f(lambd, c)>

## Conclusion:
- Fixed Lambda
   - for higher context count and value of lamda does not matter, we have high value of lambda1,we have high confidence on bigram count
   - for lower context count, value of lamda1 or trust on bigram count is decreases

Higher lambda penalizes ngrams with lower context count