# **Modelos de Classificação**

#### Este laboratório irá cobrir os passos para tratar a base de dados de taxa de cliques (click-through rate - CTR) e criar um modelo de classificação para tentar determinar se um usuário irá ou não clicar em um banner.

#### Para isso utilizaremos a base de dados [Criteo Labs](http://labs.criteo.com/) que foi utilizado em uma competição do [Kaggle](https://www.kaggle.com/c/criteo-display-ad-challenge).

#### ** Nesse notebook: **
+  #### *Parte 1:* Utilização do one-hot-encoding (OHE) para transformar atributos categóricos em numéricos
+  #### *Parte 2:* Construindo um dicionário OHE
+  #### *Parte 3:* Geração de atributos OHE na base de dados CTR
 + #### *Visualização 1:* Frequência de atributos
+  #### *Parte 4:* Predição de CTR e avaliação da perda logarítimica (logloss)
 + #### *Visualização 2:* Curva ROC
+  #### *Parte 5:* Reduzindo a dimensão dos atributos através de hashing (feature hashing)

#### Referências de métodos: [Spark's Python API](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD)e [NumPy Reference](http://docs.scipy.org/doc/numpy/reference/index.html)

### ** Part 1: Utilização do one-hot-encoding (OHE) para transformar atributos categóricos em numéricos **

#### ** (1a) One-hot-encoding **

#### Para um melhor entendimento do processo da codificação OHE vamos trabalhar com uma base de dados pequena e sem rótulos. Cada objeto dessa base pode conter três atributos, o primeiro indicando o animal, o segundo a cor e o terceiro qual animal que ele come.

#### No esquema OHE, queremos representar cada tupla `(IDatributo, categoria)` através de um atributo binário. Nós podemos fazer isso no Python criando um dicionário que mapeia cada possível tupla em um inteiro que corresponde a sua posição no vetor de atributos binário.

#### Para iniciar crie um dicionário correspondente aos atributos categóricos da base construída logo abaixo. Faça isso manualmente.

```python
# Data for manual OHE
# Note: the first data point does not include any value for the optional third feature
sampleOne = [(0, 'mouse'), (1, 'black')]
sampleTwo = [(0, 'cat'), (1, 'tabby'), (2, 'mouse')]
sampleThree =  [(0, 'bear'), (1, 'black'), (2, 'salmon')]
sampleDataRDD = sc.parallelize([sampleOne, sampleTwo, sampleThree])
```

```python
# EXERCICIO
sampleOHEDictManual = {}
sampleOHEDictManual[(0,'mouse')] = 0
sampleOHEDictManual[(0,'cat')] = 1
sampleOHEDictManual[(0,'bear')] = 2
sampleOHEDictManual[(1,'black')] = 3
sampleOHEDictManual[(1,'tabby')] = 4
sampleOHEDictManual[(2,'mouse')] = 5
sampleOHEDictManual[(2,'salmon')] = 6
```

```python
# TEST One-hot-encoding (1a)
assert (0, 'mouse') in sampleOHEDictManual, "(0, 'mouse') not in sampleOHEDictManual"
assert (0, 'cat') in sampleOHEDictManual, "(0, 'cat') not in sampleOHEDictManual"
assert (0, 'bear') in sampleOHEDictManual, "(0, 'bear') not in sampleOHEDictManual"
assert (1, 'black') in sampleOHEDictManual, "(1, 'black') not in sampleOHEDictManual"
assert (1, 'tabby') in sampleOHEDictManual, "(1, 'tabby') not in sampleOHEDictManual"
assert (2, 'mouse') in sampleOHEDictManual, "(2, 'mouse') not in sampleOHEDictManual"
assert (2, 'salmon') in sampleOHEDictManual, "(2, 'salmon') not in sampleOHEDictManual"
```

#### ** (1b) Vetores Esparsos **

#### Pontos de dados categóricos geralmente apresentam um pequeno conjunto de OHE não-nulos relativo ao total de possíveis atributos. Tirando proveito dessa propriedade podemos representar nossos dados como vetores esparsos, economizando espaço de armazenamento e cálculos computacionais.

#### No próximo exercício transforme os vetores com nome precedidos de `Dense` para vetores esparsos. Utilize a classe [SparseVector](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.SparseVector) para representá-los e verifique que ambas as representações retornam o mesmo resultado nos cálculos dos produtos interno.

#### Use `SparseVector(tamanho, *args)` para criar um novo vetor esparso onde `tamanho` é o tamanho do vetor e `args` pode ser um dicionário, uma lista de tuplas (índice, valor) ou duas arrays separadas de índices e valores ordenados por índice.

```python
import numpy as np
from pyspark.mllib.linalg import SparseVector
```

```python
# EXERCICIO
aDense = np.array([0., 3., 0., 4.])
aSparse = SparseVector(4, [(1,3),(3,4)])

bDense = np.array([0., 0., 0., 1.])
bSparse = SparseVector(4, [(3,1)])

w = np.array([0.4, 3.1, -1.4, -.5])
print (aDense.dot(w))
print (aSparse.dot(w))
print (bDense.dot(w))
print (bSparse.dot(w))
```

    7.3
    7.3
    -0.5
    -0.5

```python
# TEST Sparse Vectors (1b)
assert isinstance(aSparse, SparseVector), 'aSparse needs to be an instance of SparseVector'
assert isinstance(bSparse, SparseVector), 'aSparse needs to be an instance of SparseVector'
assert aDense.dot(w) == aSparse.dot(w), 'dot product of aDense and w should equal dot product of aSparse and w'
assert bDense.dot(w) == bSparse.dot(w), 'dot product of bDense and w should equal dot product of bSparse and w'
```

#### **(1c) Atributos OHE como vetores esparsos **

#### Agora vamos representar nossos atributos OHE como vetores esparsos. Utilizando o dicionário `sampleOHEDictManual`, crie um vetor esparso para cada amostra de nossa base de dados. Todo atributo que ocorre em uma amostra deve ter valor 1.0. Por exemplo, um vetor para um ponto com os atributos 2 e 4 devem ser `[0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0]`.

```python
# Reminder of the sample features
# sampleOne = [(0, 'mouse'), (1, 'black')]
# sampleTwo = [(0, 'cat'), (1, 'tabby'), (2, 'mouse')]
# sampleThree =  [(0, 'bear'), (1, 'black'), (2, 'salmon')]
```

```python
# EXERCICIO
sampleOneOHEFeatManual = SparseVector(7, [(sampleOHEDictManual[x],1.0) for x in sampleOne])
sampleTwoOHEFeatManual = SparseVector(7, [(sampleOHEDictManual[x],1.0) for x in sampleTwo])
sampleThreeOHEFeatManual = SparseVector(7, [(sampleOHEDictManual[x],1.0) for x in sampleThree])
```

```python
# TEST OHE Features as sparse vectors (1c)
assert isinstance(sampleOneOHEFeatManual, SparseVector), 'sampleOneOHEFeatManual needs to be a SparseVector'
assert isinstance(sampleTwoOHEFeatManual, SparseVector), 'sampleTwoOHEFeatManual needs to be a SparseVector'
assert isinstance(sampleThreeOHEFeatManual, SparseVector), 'sampleThreeOHEFeatManual needs to be a SparseVector'
```

#### **(1d) Função de codificação OHE **

#### Vamos criar uma função que gera um vetor esparso codificado por um dicionário de OHE. Ele deve fazer o procedimento similar ao exercício anterior.

```python
# EXERCICIO
def oneHotEncoding(rawFeats, OHEDict, numOHEFeats):
    """Produce a one-hot-encoding from a list of features and an OHE dictionary.

    Note:
        You should ensure that the indices used to create a SparseVector are sorted.

    Args:
        rawFeats (list of (int, str)): The features corresponding to a single observation.  Each
            feature consists of a tuple of featureID and the feature's value. (e.g. sampleOne)
        OHEDict (dict): A mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The total number of unique OHE features (combinations of featureID and
            value).

    Returns:
        SparseVector: A SparseVector of length numOHEFeats with indicies equal to the unique
            identifiers for the (featureID, value) combinations that occur in the observation and
            with values equal to 1.0.
    """
    return SparseVector(numOHEFeats, [(OHEDict[x], 1.0) for x in rawFeats])

# Calculate the number of features in sampleOHEDictManual
numSampleOHEFeats = len(sampleOHEDictManual)

# Run oneHotEnoding on sampleOne
sampleOneOHEFeat = oneHotEncoding(sampleOne, sampleOHEDictManual, numSampleOHEFeats)

print (sampleOneOHEFeat)
```

    (7,[0,3],[1.0,1.0])

```python
# TEST Define an OHE Function (1d)
assert sampleOneOHEFeat == sampleOneOHEFeatManual, 'sampleOneOHEFeat should equal sampleOneOHEFeatManual'
```

#### **(1e) Aplicar OHE em uma base de dados **

#### Finalmente, use a função da parte (1d) para criar atributos OHE para todos os 3 objetos da base de dados artificial.

```python
# EXERCICIO
sampleOHEData = sampleDataRDD.map(lambda x: oneHotEncoding(x, sampleOHEDictManual, numSampleOHEFeats))
print (sampleOHEData.collect())
```

    [SparseVector(7, {0: 1.0, 3: 1.0}), SparseVector(7, {1: 1.0, 4: 1.0, 5: 1.0}), SparseVector(7, {2: 1.0, 3: 1.0, 6: 1.0})]

```python
# TEST Apply OHE to a dataset (1e)
sampleOHEDataValues = sampleOHEData.collect()
assert len(sampleOHEDataValues) == 3, 'sampleOHEData should have three elements'
```

### ** Part 2: Construindo um dicionário OHE **

#### **(2a) Tupla RDD de `(IDatributo, categoria)` **

#### Crie um RDD de pares distintos de `(IDatributo,  categoria)`. Em nossa base de dados você deve gerar `(0, 'bear')`, `(0, 'cat')`, `(0, 'mouse')`, `(1, 'black')`, `(1, 'tabby')`, `(2, 'mouse')`, `(2, 'salmon')`. Repare que `'black'` aparece duas vezes em nossa base de dados mas contribui apenas para um item do RDD: `(1, 'black')`, por outro lado `'mouse'` aparece duas vezes e contribui para dois itens: `(0, 'mouse')` and `(2, 'mouse')`.

#### Dica: use [flatMap](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.flatMap) e [distinct](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.distinct).

```python
# EXERCICIO
sampleDistinctFeats = (sampleDataRDD
                       .flatMap(lambda x: x)
                       .distinct()
                      )

```

```python
# TEST Pair RDD of (featureID, category) (2a)
assert sorted(sampleDistinctFeats.collect()) == [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),(1, 'tabby'), (2, 'mouse'), (2, 'salmon')], 'incorrect value for sampleDistinctFeats'
```

#### ** (2b) Dicionário OHE de atributos únicos **

#### Agora, vamos criar um RDD de tuplas para cada `(IDatributo, categoria)` em `sampleDistinctFeats`. A chave da tupla é a própria tupla original, e o valor será um inteiro variando de 0 até número de tuplas - 1.

#### Em seguida, converta essa `RDD` em um dicionário, utilizando o comando `collectAsMap`.

#### Use o comando  [zipWithIndex](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.zipWithIndex) seguido de [collectAsMap](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.collectAsMap).

```python
# EXERCICIO
sampleOHEDict = (sampleDistinctFeats
                           .zipWithIndex()
                           .collectAsMap())
print (sampleOHEDict)
```

    {(0, 'mouse'): 0, (1, 'black'): 1, (0, 'cat'): 2, (1, 'tabby'): 3, (2, 'mouse'): 4, (0, 'bear'): 5, (2, 'salmon'): 6}

```python
# TEST OHE Dictionary from distinct features (2b)
assert sorted(sampleOHEDict.keys()) == [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),(1, 'tabby'), (2, 'mouse'), (2, 'salmon')], 'sampleOHEDict has unexpected keys'
assert sorted(sampleOHEDict.values()) == list(range(7)), 'sampleOHEDict has unexpected values'
```

#### **(2c) Criação automática do dicionário OHE **

#### Agora use os códigos dos exercícios anteriores para criar uma função que retorna um dicionário OHE a partir dos atributos categóricos de uma base de dados.

```python
# EXERCICIO
def createOneHotDict(inputData):
    """Creates a one-hot-encoder dictionary based on the input data.

    Args:
        inputData (RDD of lists of (int, str)): An RDD of observations where each observation is
            made up of a list of (featureID, value) tuples.

    Returns:
        dict: A dictionary where the keys are (featureID, value) tuples and map to values that are
            unique integers.
    """
    return (inputData
             .flatMap(lambda x: x)
             .distinct()
             .zipWithIndex()
             .collectAsMap()
             )

sampleOHEDictAuto = createOneHotDict(sampleDataRDD)
print (sampleOHEDictAuto)
```

    {(0, 'mouse'): 0, (1, 'black'): 1, (0, 'cat'): 2, (1, 'tabby'): 3, (2, 'mouse'): 4, (0, 'bear'): 5, (2, 'salmon'): 6}

```python
# TEST Automated creation of an OHE dictionary (2c)
assert sorted(sampleOHEDictAuto.keys()) == [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'), (1, 'tabby'), (2, 'mouse'), (2, 'salmon')], 'sampleOHEDictAuto has unexpected keys'
assert sorted(sampleOHEDictAuto.values()) == list(range(7)), 'sampleOHEDictAuto has unexpected values'
```

### **Part 3: Parse CTR data and generate OHE features**

#### Antes de começar essa parte, vamos carregar a base de dados e verificar o formato dela.

#### Repare que o primeiro campo é o rótulo de cada objeto, sendo 0 se o usuário não clicou no banner e 1 caso tenha clicado. O restante dos atributos ou são numéricos ou são strings representando categorias anônimas. Vamos tratar todos os atributos como categóricos.

```python
import os.path
fileName = os.path.join('Data', 'dac_sample.txt')

if os.path.isfile(fileName):
    rawData = (sc
               .textFile(fileName, 2)
               .map(lambda x: x.replace('\t', ',')))  # work with either ',' or '\t' separated data
    print (rawData.take(1))
```

    ['0,1,1,5,0,1382,4,15,2,181,1,2,,2,68fd1e64,80e26c9b,fb936136,7b4723c4,25c83c98,7e0ccccf,de7995b8,1f89b562,a73ee510,a8cd5504,b2cb9c98,37c9c164,2824a5f6,1adce6ef,8ba8b39a,891b62e7,e5ba7672,f54016b9,21ddcdc9,b1252a9d,07b5194c,,3a171ecb,c5c50484,e8b83407,9727dd16']

#### **(3a) Carregando e dividindo os dados **

#### Da mesma forma que no notebook anterior, vamos dividir os dados entre treinamento, validação e teste. Use o método [randomSplit](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.randomSplit) com os pesos (weights) e semente aleatória (seed) especificados para criar os conjuntos, então faça o [cache](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.cache) de cada RDD, pois utilizaremos cada uma delas com frequência durante esse exercício.

```python
# EXERCICIO
weights = [.8, .1, .1]
seed = 42
# Use randomSplit with weights and seed
rawTrainData, rawValidationData, rawTestData = rawData.randomSplit(weights, seed)
# Cache the data
rawTrainData.cache()
rawValidationData.cache()
rawTestData.cache()

nTrain = rawTrainData.count()
nVal = rawValidationData.count()
nTest = rawTestData.count()
print (nTrain, nVal, nTest, nTrain + nVal + nTest)
print (rawData.take(1))
```

    80053 9941 10006 100000
    ['0,1,1,5,0,1382,4,15,2,181,1,2,,2,68fd1e64,80e26c9b,fb936136,7b4723c4,25c83c98,7e0ccccf,de7995b8,1f89b562,a73ee510,a8cd5504,b2cb9c98,37c9c164,2824a5f6,1adce6ef,8ba8b39a,891b62e7,e5ba7672,f54016b9,21ddcdc9,b1252a9d,07b5194c,,3a171ecb,c5c50484,e8b83407,9727dd16']

```python
# TEST Loading and splitting the data (3a)
assert all([rawTrainData.is_cached, rawValidationData.is_cached, rawTestData.is_cached]), 'you must cache the split data'
assert nTrain == 80053, 'incorrect value for nTrain'
assert nVal == 9941, 'incorrect value for nVal'
assert nTest == 10006, 'incorrect value for nTest'
```

#### ** (3b) Extração de atributos **

#### Como próximo passo, crie uma função para ser aplicada em cada objeto do RDD para gerar uma RDD de tuplas (IDatributo, categoria). Ignore o primeiro campo, que é o rótulo e gere uma lista de tuplas para os atributos seguintes. Utilize o comando [enumerate](https://docs.python.org/2/library/functions.html#enumerate) para criar essas tuplas.

```python
# EXERCICIO
def parsePoint(point):
    """Converts a comma separated string into a list of (featureID, value) tuples.

    Note:
        featureIDs should start at 0 and increase to the number of features - 1.

    Args:
        point (str): A comma separated string where the first value is the label and the rest
            are features.

    Returns:
        list: A list of (featureID, value) tuples.
    """
    return list(enumerate(point.split(',')[1:]))

parsedTrainFeat = rawTrainData.map(parsePoint)

numCategories = (parsedTrainFeat
                 .flatMap(lambda x: x)
                 .distinct()
                 .map(lambda x: (x[0], 1))
                 .reduceByKey(lambda x, y: x + y)
                 .sortByKey()
                 .collect()
                )

print (numCategories[2][1])
```

    864

```python
# TEST Extract features (3b)
assert numCategories[2][1] == 864, 'incorrect implementation of parsePoint'
assert numCategories[32][1] == 4, 'incorrect implementation of parsePoint'
```

#### **(3c) Crie o dicionário de OHE dessa base de dados **

#### Note que a função parsePoint retorna um objeto em forma de lista `(IDatributo, categoria)`, que é o mesmo formato utilizado pela função `createOneHotDict`. Utilize o RDD `parsedTrainFeat` para criar um dicionário OHE.

```python
# EXERCICIO
ctrOHEDict = createOneHotDict(parsedTrainFeat)
numCtrOHEFeats = len(ctrOHEDict.keys())
print (numCtrOHEFeats)
print (ctrOHEDict[(0, '')])
```

    234358
    50

```python
# TEST Create an OHE dictionary from the dataset (3c)
assert numCtrOHEFeats == 234358, 'incorrect number of features in ctrOHEDict'
assert (0, '') in ctrOHEDict, 'incorrect features in ctrOHEDict'
```

#### ** (3d) Aplicando OHE à base de dados **

#### Agora vamos usar o dicionário OHE para criar um RDD de objetos [LabeledPoint](http://spark.apache.org/docs/1.3.1/api/python/pyspark.mllib.html#pyspark.mllib.regression.LabeledPoint) usando atributos OHE. Complete a função `parseOHEPoint`. Dica: essa função é uma extensão da função `parsePoint` criada anteriormente e que usa a função `oneHotEncoding`.

```python
from pyspark.mllib.regression import LabeledPoint
```

```python
# EXERCICIO
def parseOHEPoint(point, OHEDict, numOHEFeats):
    """Obtain the label and feature vector for this raw observation.

    Note:
        You must use the function `oneHotEncoding` in this implementation or later portions
        of this lab may not function as expected.

    Args:
        point (str): A comma separated string where the first value is the label and the rest
            are features.
        OHEDict (dict of (int, str) to int): Mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The number of unique features in the training dataset.

    Returns:
        LabeledPoint: Contains the label for the observation and the one-hot-encoding of the
            raw features based on the provided OHE dictionary.
    """
    splitted = point.split(',')
    label, features = float(splitted[0]), list(enumerate(splitted[1:]))
    return LabeledPoint(label, oneHotEncoding(features, OHEDict, numOHEFeats) )

OHETrainData = rawTrainData.map(lambda point: parseOHEPoint(point, ctrOHEDict, numCtrOHEFeats))
OHETrainData.cache()
print (OHETrainData.take(1))

# Check that oneHotEncoding function was used in parseOHEPoint
backupOneHot = oneHotEncoding
oneHotEncoding = None
withOneHot = False
try: parseOHEPoint(rawTrainData.take(1)[0], ctrOHEDict, numCtrOHEFeats)
except TypeError: withOneHot = True
oneHotEncoding = backupOneHot
```

    [LabeledPoint(0.0, (234358,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,117215,117216,117217,117218,117219,117220,117221,117222,117223,117224,117225,117226,117227,117228,117229,117230,117231,117232,117233,117234,117235],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]))]

```python
# TEST Apply OHE to the dataset (3d)
numNZ = sum(parsedTrainFeat.map(lambda x: len(x)).take(5))
numNZAlt = sum(OHETrainData.map(lambda lp: len(lp.features.indices)).take(5))
assert numNZ == numNZAlt, 'incorrect implementation of parseOHEPoint'
assert withOneHot, 'oneHotEncoding not present in parseOHEPoint'
```

#### **Visualização 1: Frequência dos Atributos **

#### Vamos agora visualizar o número de vezes que cada um dos 233.286 atributos OHE aparecem na base de treino.  Para isso primeiro contabilizamos quantas vezes cada atributo aparece na base, então alocamos cada atributo em um balde de histograma. Os baldes tem tamanhos de potência de 2, então o primeiro balde conta os atributos que aparecem exatamente uma vez ( $ \scriptsize 2^0 $ ), o segundo atributos que aparecem duas vezes ( $ \scriptsize 2^1 $ ), o terceiro os atributos que aparecem de 3 a 4 vezes ( $ \scriptsize 2^2 $ ), o quinto balde é para atributos que ocorrem de cinco a oito vezes ( $ \scriptsize 2^3 $ ) e assim por diante. O gráfico de dispersão abaixo mostra o logarítmo do tamanho dos baldes versus o logarítmo da frequência de atributos que caíram nesse balde.

```python
def bucketFeatByCount(featCount):
    """Bucket the counts by powers of two."""
    for i in range(11):
        size = 2 ** i
        if featCount <= size:
            return size
    return -1

featCounts = (OHETrainData
              .flatMap(lambda lp: lp.features.indices)
              .map(lambda x: (x, 1))
              .reduceByKey(lambda x, y: x + y))
featCountsBuckets = (featCounts
                     .map(lambda x: (bucketFeatByCount(x[1]), 1))
                     .filter(lambda kv: kv[0] != -1)
                     .reduceByKey(lambda x, y: x + y)
                     .collect())
print (featCountsBuckets)
```

    [(8, 11480), (2, 23578), (16, 7750), (128, 1504), (64, 2663), (256, 733), (32, 4792), (512, 416), (1024, 251), (4, 16870), (1, 164030)]

```python
import matplotlib.pyplot as plt
% matplotlib inline

x, y = zip(*featCountsBuckets)
x, y = np.log(x), np.log(y)

def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999',
                gridWidth=1.0):
    """Template for generating the plot layout."""
    plt.close()
    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')
    ax.axes.tick_params(labelcolor='#999999', labelsize='10')
    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:
        axis.set_ticks_position('none')
        axis.set_ticks(ticks)
        axis.label.set_color('#999999')
        if hideLabels: axis.set_ticklabels([])
    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')
    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])
    return fig, ax

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 10, 1), np.arange(4, 14, 2))
ax.set_xlabel(r'$\log_e(bucketSize)$'), ax.set_ylabel(r'$\log_e(countInBucket)$')
plt.scatter(x, y, s=14**2, c='#d6ebf2', edgecolors='#8cbfd0', alpha=0.75)
pass
```

#### **(3e) Atributos não observados **

#### Naturalmente precisaremos aplicar esse mesmo procedimento para as outras bases (validação e teste), porém nessas bases podem existir atributos não observados na base de treino.

#### Precisamos adaptar a função `oneHotEncoding` para ignorar os atributos que não existem no dicionário.

```python
# EXERCICIO
def oneHotEncoding(rawFeats, OHEDict, numOHEFeats):
    """Produce a one-hot-encoding from a list of features and an OHE dictionary.

    Note:
        If a (featureID, value) tuple doesn't have a corresponding key in OHEDict it should be
        ignored.

    Args:
        rawFeats (list of (int, str)): The features corresponding to a single observation.  Each
            feature consists of a tuple of featureID and the feature's value. (e.g. sampleOne)
        OHEDict (dict): A mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The total number of unique OHE features (combinations of featureID and
            value).

    Returns:
        SparseVector: A SparseVector of length numOHEFeats with indicies equal to the unique
            identifiers for the (featureID, value) combinations that occur in the observation and
            with values equal to 1.0.
    """
    return SparseVector(numOHEFeats, [(OHEDict[x], 1.0) for x in rawFeats if x in OHEDict])

OHEValidationData = rawValidationData.map(lambda point: parseOHEPoint(point, ctrOHEDict, numCtrOHEFeats))
OHEValidationData.cache()
print (OHEValidationData.take(1))
```

    [LabeledPoint(0.0, (234358,[2,8,15,21,45,49,50,52,61,96,135,160,164,671,1355,5140,5141,5142,117223,117226,117238,117242,117258,117261,117262,117267,117269,117284,117287,118378,119236,120375,122236,122237,122238,122239,122240,122241,170030],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]))]

```python
# TEST Handling unseen features (3e)
numNZVal = (OHEValidationData
            .map(lambda lp: len(lp.features.indices))
            .sum())
assert numNZVal == 367585, 'incorrect number of features'
```

### ** Part 4: Predição do CTR e avaliação da perda-log (logloss) **

#### ** (4a) Regressão Logística **

#### Um classificador que podemos utilizar nessa base de dados é a regressão logística, que nos dá a probabilidade de um evento de clique em banner ocorrer. Vamos utilizar a função  [LogisticRegressionWithSGD](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.classification.LogisticRegressionWithSGD) para treinar um modelo usando  `OHETrainData` com a configuração de parâmetros dada.  `LogisticRegressionWithSGD` retorna um [LogisticRegressionModel](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LogisticRegressionModel).

#### Em seguida, imprima  `LogisticRegressionModel.weights` e `LogisticRegressionModel.intercept` para verificar o modelo gerado.

```python
from pyspark.mllib.classification import LogisticRegressionWithSGD

# fixed hyperparameters
numIters = 50
stepSize = 10.
regParam = 1e-6
regType = 'l2'
includeIntercept = True
```

```python
# EXERCICIO
model0 = LogisticRegressionWithSGD.train(OHETrainData, numIters, stepSize, regParam=regParam, regType=regType, intercept=includeIntercept)
sortedWeights = sorted(model0.weights)
print (sortedWeights[:5], model0.intercept)
```

    /home/olivetti/Downloads/spark-2.2.1-bin-hadoop2.7/python/pyspark/mllib/classification.py:313: UserWarning: Deprecated in 2.0.0. Use ml.classification.LogisticRegression or LogisticRegressionWithLBFGS.
      "Deprecated in 2.0.0. Use ml.classification.LogisticRegression or "

    [-0.46297159426279577, -0.39040230182817892, -0.3871281985827924, -0.35815003494268316, -0.34963241495474701] 0.5616041364601837

```python
# TEST Logistic regression (4a)
assert np.allclose(model0.intercept,  0.5616041364601837), 'incorrect value for model0.intercept'
assert np.allclose(sortedWeights[0:5], [-0.46297159426279577, -0.39040230182817892, -0.3871281985827924, -0.35815003494268316, -0.34963241495474701]), 'incorrect value for model0.weights'
```

#### ** (4b) Log loss **

#### Uma forma de avaliar um classificador binário é através do log-loss, definido como: $$  \begin{align} \scriptsize \ell_{log}(p, y) = \begin{cases} -\log (p) & \text{if } y = 1 \\\ -\log(1-p) & \text{if } y = 0 \end{cases} \end{align} $$ onde $ \scriptsize p$ é uma probabilidade entre 0 e 1 e  $ \scriptsize y$ é o rótulo binário (0 ou 1). Log loss é um critério de avaliação muito utilizado quando deseja-se predizer eventos raros. Escreva uma função para calcular o log-loss, e avalie algumas entradas de amostra.

```python
# EXERCICIO
import numpy as np

def computeLogLoss(p, y):
    """Calculates the value of log loss for a given probabilty and label.

    Note:
        log(0) is undefined, so when p is 0 we need to add a small value (epsilon) to it
        and when p is 1 we need to subtract a small value (epsilon = 1e-11) from it.

    Args:
        p (float): A probabilty between 0 and 1.
        y (int): A label.  Takes on the values 0 and 1.

    Returns:
        float: The log loss value.
    """
    if p == 0.0:
        p += 1e-11
    elif p == 1.0:
        p -= 1e-11
    return -(y*np.log(p) + (1. - y)*np.log(1-p))

print (computeLogLoss(.5, 1))
print( computeLogLoss(.5, 0))
print (computeLogLoss(.99, 1))
print (computeLogLoss(.99, 0))
print (computeLogLoss(.01, 1))
print (computeLogLoss(.01, 0))
print (computeLogLoss(0, 1))
print (computeLogLoss(1, 1))
print (computeLogLoss(1, 0))
```

    0.69314718056
    0.69314718056
    0.0100503358535
    4.60517018599
    4.60517018599
    0.0100503358535
    25.3284360229
    1.00000008275e-11
    25.3284359402

```python
# TEST Log loss (4b)
assert np.allclose([computeLogLoss(.5, 1), computeLogLoss(.01, 0), computeLogLoss(.01, 1)], [0.69314718056, 0.0100503358535, 4.60517018599]), 'computeLogLoss is not correct'
assert np.allclose([computeLogLoss(0, 1), computeLogLoss(1, 1), computeLogLoss(1, 0)], [25.3284360229, 1.00000008275e-11, 25.3284360229]), 'computeLogLoss needs to bound p away from 0 and 1 by epsilon'
```

#### ** (4c)  Baseline log loss **

#### Agora, vamos utilizar a função da Parte (4b) para calcular um baseline da métrica de log-loss na nossa base de treino. Uma forma de calcular um baseline é predizer sempre a média dos rótulos observados. Primeiro calcule a média dos rótulos da base e, em seguida, calcule o log-loss médio para a base de treino.

```python
# EXERCICIO
# Note that our dataset has a very high click-through rate by design
# In practice click-through rate can be one to two orders of magnitude lower
classOneFracTrain = OHETrainData.map(lambda lp: lp.label).mean()
print (classOneFracTrain)

logLossTrBase = OHETrainData.map(lambda lp: computeLogLoss(classOneFracTrain, lp.label)).mean()
print( 'Baseline Train Logloss = {0:.3f}\n'.format(logLossTrBase))
```

    0.2271245299988764
    Baseline Train Logloss = 0.536

```python
# TEST Baseline log loss (4c)
assert np.allclose(classOneFracTrain, 0.2271245299988764), 'incorrect value for classOneFracTrain'
assert np.allclose(logLossTrBase, 0.535778466496), 'incorrect value for logLossTrBase'
```

#### ** (4d) Probabilidade da Predição **

#### O modelo gerado na Parte (4a) possui um método chamado `predict`, porém esse método retorna apenas 0's e 1's. Para calcular a probabilidade de um evento, vamos criar uma função `getP` que recebe como parâmetro o ponto x, o conjunto de pesos `w` e o `intercept`.

#### Calcule o modelo de regressão linear nesse ponto x e aplique a  [função sigmoidal](http://en.wikipedia.org/wiki/Sigmoid_function) $ \scriptsize \sigma(t) = (1+ e^{-t})^{-1} $ para retornar a probabilidade da predição do objeto x.

```python
# EXERCICIO
from math import exp #  exp(-t) = e^-t

def getP(x, w, intercept):
    """Calculate the probability for an observation given a set of weights and intercept.

    Note:
        We'll bound our raw prediction between 20 and -20 for numerical purposes.

    Args:
        x (SparseVector): A vector with values of 1.0 for features that exist in this
            observation and 0.0 otherwise.
        w (DenseVector): A vector of weights (betas) for the model.
        intercept (float): The model's intercept.

    Returns:
        float: A probability between 0 and 1.
    """
    # calculate rawPrediction = w.x + intercept
    rawPrediction = x.dot(w) + intercept

    # Bound the raw prediction value
    rawPrediction = min(rawPrediction, 20)
    rawPrediction = max(rawPrediction, -20)

    # calculate (1+e^-rawPrediction)^-1
    return 1./(1. + np.exp(-rawPrediction))

trainingPredictions = OHETrainData.map(lambda lp: getP(lp.features, model0.weights, model0.intercept))

print (trainingPredictions.take(5))
```

    [0.28030535919449739, 0.094841892570971031, 0.28288128455914763, 0.16384346875926237, 0.56959784508095868]

```python
# TEST Predicted probability (4d)
assert np.allclose(trainingPredictions.sum(), 18198.8525175), 'incorrect value for trainingPredictions'
```

#### ** (4e) Avalie o modelo **

#### Finalmente, crie uma função `evaluateResults` que calcula o log-loss médio do modelo em uma base de dados. Em seguida, execute essa função na nossa base de treino.

```python
# EXERCICIO
def evaluateResults(model, data):
    """Calculates the log loss for the data given the model.

    Args:
        model (LogisticRegressionModel): A trained logistic regression model.
        data (RDD of LabeledPoint): Labels and features for each observation.

    Returns:
        float: Log loss for the data.
    """
    return (data
            .map(lambda lp: (lp.label, getP(lp.features, model0.weights, model0.intercept)))
            .map(lambda lp: computeLogLoss(lp[1], lp[0]))
            .mean()
            )

logLossTrLR0 = evaluateResults(model0, OHETrainData)
print ('OHE Features Train Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'.format(logLossTrBase, logLossTrLR0))
```

    OHE Features Train Logloss:
    	Baseline = 0.536
    	LogReg = 0.457

```python
# TEST Evaluate the model (4e)
assert np.allclose(logLossTrLR0, 0.45704573867), 'incorrect value for logLossTrLR0'
```

    0.45704573867

#### ** (4f) log-loss da validação **

#### Agora aplique o modelo na nossa base de validação e calcule o log-loss médio, compare com o nosso baseline.

```python
# EXERCICIO
logLossValBase = OHEValidationData.map(lambda lp: computeLogLoss(classOneFracTrain, lp.label)).mean()

logLossValLR0 = evaluateResults(model0, OHEValidationData)
print ('OHE Features Validation Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'.format(logLossValBase, logLossValLR0))
```

    OHE Features Validation Logloss:
    	Baseline = 0.527
    	LogReg = 0.458

```python
# TEST Validation log loss (4f)
assert np.allclose(logLossValBase, 0.526558409461), 'incorrect value for logLossValBase'
assert np.allclose(logLossValLR0, 0.458434994198), 'incorrect value for logLossValLR0'
```

#### **Visualização 2: Curva ROC  **

#### A curva ROC nos mostra o custo-benefício entre a taxa de falso positivo e a taxa de verdadeiro positivo, conforme diminuimos o limiar de predição. Um modelo aleatório é representado por uma linha pontilhada. Idealmente nosso modelo deve formar uma curva acima dessa linha.

```python
labelsAndScores = OHEValidationData.map(lambda lp:
                                            (lp.label, getP(lp.features, model0.weights, model0.intercept)))
labelsAndWeights = labelsAndScores.collect()
labelsAndWeights.sort(key=lambda kv: kv[1], reverse=True)
labelsByWeight = np.array([k for (k, v) in labelsAndWeights])

length = labelsByWeight.size
truePositives = labelsByWeight.cumsum()
numPositive = truePositives[-1]
falsePositives = np.arange(1.0, length + 1, 1.) - truePositives

truePositiveRate = truePositives / numPositive
falsePositiveRate = falsePositives / (length - numPositive)

# Generate layout and plot data
fig, ax = preparePlot(np.arange(0., 1.1, 0.1), np.arange(0., 1.1, 0.1))
ax.set_xlim(-.05, 1.05), ax.set_ylim(-.05, 1.05)
ax.set_ylabel('True Positive Rate (Sensitivity)')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
plt.plot(falsePositiveRate, truePositiveRate, color='#8cbfd0', linestyle='-', linewidth=3.)
plt.plot((0., 1.), (0., 1.), linestyle='--', color='#d6ebf2', linewidth=2.)  # Baseline model
pass
```

```python

```


```python
import prtools as pr
import numpy as np
from matplotlib import pyplot as plt
from sklearn.model_selection import train_test_split

from jupyterthemes import jtplot
jtplot.style(theme="oceans16")
```

$$\sum_{i=1}^N (x_{i}^Tw-y_i)^2 + \lambda ||w|| = ||Xw-Y||^2 + \lambda ||w||^2$$

# Exercise 3.1

(a)
\begin{equation}
X =
    \begin{bmatrix}
    x_{11} & x_{12} & x_{13} \\
    x_{21} & x_{22} & x_{23} \\
    x_{31} & x_{32} & x_{33}
    \end{bmatrix}
\end{equation}

\begin{equation}
X^TX =
    \begin{bmatrix}
    x_{11}^2 & x_{12}^2 & x_{13}^2 \\
    x_{21}^2 & x_{22}^2 & x_{23}^2 \\
    x_{31}^2 & x_{32}^2 & x_{33}^2
    \end{bmatrix}^T
\end{equation}

\
A is positive semidefinite if, for any vector x with real components, the dot product of Ax and x is nonnegative,
\
<Ax, xi> ≥ 0

(b)

 半正定矩阵+微扰则 $\to$ 正定矩阵 $\to$ 可逆 (该结论由SVD分解推理出)

 $X^TX v = \lambda v$

 ($X^TX + \lambda I) v = \lambda v + \lambda Iv = (\lambda + 1)v $

 若所有特征值均不小于零，则称为半正定。若所有特征值均大于零，则称为正定。

(c)

 $\frac{d}{dw}||Xw-Y||^2 + \lambda ||w||^2 = 0$

 $w = (X^TX + \lambda I)^{-1} X^TY$

(e)

 $||w||^2 < \tau, where \ \tau \propto \frac{1}{\lambda}$

 w = 0

# Exercise 3.2

加上所有参数（不包括$w_0$）的绝对值之和，即l1范数，此时叫做Lasso回归；

加上所有参数（不包括$w_0$）的平方和，即l2范数的平方，此时叫做ridge回归.

# Exercise 3.3

```python
def genstdNoise(x):
    return np.random.randn(x.shape[0], x.shape[1])
```

(a)

```python
x_train = np.random.uniform(low=0.0, high=1.0, size=(10,1))
y_trian = x_train + genstdNoise(x_train)
x_test = np.random.uniform(low=0.0, high=1.0, size=(10000,1))
y_test = x_test + genstdNoise(x_test)

trainData = pr.gendatr(x_train, targets=y_trian)
testData = pr.gendatr(x_test, targets=y_test)

print("trainData shape: ", trainData.shape)
print("testData shape: ", testData.shape)
```

    trainData shape:  (10, 1)
    testData shape:  (10000, 1)

```python
pr.scatterr(testData)
lamb = 0.001
w = pr.ridger(trainData, lamb)
y_hat = w(x_test)
pr.plotr(w, color="r")
print("SSE: ", np.sum((+y_hat - +y_test) ** 2))
print("w:", +w)
```

(b)

 拟合过程中通常都倾向于让权值尽可能小，最后构造一个所有参数都比较小的模型。因为一般认为参数值小的模型比较简单，能适应不同的数据集，也在一定程度上避免了过拟合现象。可以设想一下对于一个线性回归方程，若参数很大，那么只要数据偏移一点点，就会对结果造成很大的影响；但如果参数足够小，数据偏移得多一点也不会对结果造成什么影响，专业一点的说法是抗扰动能力强。

 https://www.cnblogs.com/zingp/p/10375691.html#_label0

```python
lamb = np.arange(0.01, 10, 0.05)
SSE = np.zeros(len(lamb))
for i in range(len(lamb)):
    w = pr.ridger(trainData, lamb[i])
    y_hat = w(x_test)
    SSE[i] = (np.sum((+y_hat - y_test) ** 2))

plt.plot(lamb, SSE)
plt.xlabel("lambda")
plt.ylabel("SSE")
plt.xticks(np.arange(0,2,0.5))
plt.show()
```

# Exercise 3.4

I have shown in the previous question, but it will be better to use CV to do that learning curve.

# Exercise 3.5

if $\lambda \to 0$, the term $\lambda ||w||^2$ is negligible, which means when minimizing the loss/cost, minimizing the $||Xw-Y||^2$ is more significant to reduce the loss/cost. The solution tends to base on the solution of minimizing the $||Xw-Y||^2$.

# Exercise 3.6

Not done

# Exercise 3.7

(a)

No, none of the entries ever become zero really. The probability that this happens is 0. In the limit, for  larer and larger, w should of course shrink to 0 however.

```python
x = np.random.randn(20, 2)
y = x[:,0] + 0.2 * np.random.randn(20)
data = pr.gendatr(x, y)
```

```python
w = pr.ridger(data, 0.001)
pr.scatterr(data)
pr.plotr(w, gridsize=5)
print(+w)
```

(b)

In this setting there will be a finite  for which at least on of the entries (most often the second of course!) becomes zero. For an even larger , also the other entry will become 0

```python
w = pr.lassor(data, 1)
pr.scatterr(data)
pr.plotr(w, gridsize=5)
print(+w)
```

# Exercise 3.8

(a)

 hypothesis class: $\omega \in \Re^d$

 loss: $lim_{w \in \Re^d} ||w^TX - Y||^2$

 Regularization: $\lambda ||w||$ or $\lambda ||w||^2$

(b)
the answer is the same as (a), but it doesn't have regularization term

# Exercise 3.9

(a)
SSE Loss for a single point= $(wx_i - y_i)^2$

(a)
LLE Loss for a signle point = $-log \ h(x,y|\theta)$

(c)
AUC

(d)
K-NN

# Exercise 3.10

not done

# Exercise 3.11

(a)
$$p(x,y) = p(y|x)p(x) = \frac{1}{2 \sigma \tau \pi} e^{-( \frac{(y-(xw+w_0))^2}{2 \sigma^2} + \frac{(x-v)^2}{2 \tau^2})}$$
$$p(x,y)p(w) = \frac{1}{\sigma \tau s (2\pi)^{3/2}} e^{-( \frac{(y-(xw+w_0))^2}{2 \sigma^2} + \frac{(x-v)^2}{2 \tau^2}) + \frac{w^2}{2 s^2} )}$$
$$max_w log \ p(x,y)p(w) \simeq min_w ( \frac{(y-(xw+w_0))^2}{2 \sigma^2} + \frac{(x-v)^2}{2 \tau^2}) + \frac{w^2}{2 s^2} )$$
$$\to_{only \ consider \ w \ terms} min_w (\frac{(y-(xw+w_0))^2}{2 \sigma^2} + \frac{w^2}{2 s^2})$$

(b)

 in order to minize (3.1), w should be small, but it cannot set too samll so as to underfit.

(c)
shown in 3.3

# Exercise 3.12

not done

# Exercise 3.13

```python
t = pr.gendath(n=(500, 500))
```

```python
for i in range(30):
    a = pr.gendath(n=(20, 20))

    # e = t * pr.ldc(a) * pr.testc()
    LDC = pr.ldc(a)
    error = pr.testc(LDC.eval(t))
    print("error rate:", error)
```

    error rate: 0.199
    error rate: 0.18
    error rate: 0.225
    error rate: 0.169
    error rate: 0.192
    error rate: 0.184
    error rate: 0.181
    error rate: 0.171
    error rate: 0.174
    error rate: 0.158
    error rate: 0.204
    error rate: 0.135
    error rate: 0.189
    error rate: 0.198
    error rate: 0.211
    error rate: 0.218
    error rate: 0.157
    error rate: 0.185
    error rate: 0.249
    error rate: 0.187
    error rate: 0.176
    error rate: 0.179
    error rate: 0.238
    error rate: 0.203
    error rate: 0.19
    error rate: 0.174
    error rate: 0.242
    error rate: 0.162
    error rate: 0.194
    error rate: 0.233

(a)

 the training set changes every time $\to$ the fitting curve changes every time $\to$ the error rate changes every time

```python
a = pr.gendath(n=(20,20))
w = pr.ldc(a)
```

```python
for i in range(30):
    t = pr.gendath(n=(500, 500))
    error = pr.testc(w.eval(t))
    print("error rate:", error)
```

    error rate: 0.239
    error rate: 0.223
    error rate: 0.249
    error rate: 0.252
    error rate: 0.245
    error rate: 0.258
    error rate: 0.256
    error rate: 0.268
    error rate: 0.262
    error rate: 0.24
    error rate: 0.265
    error rate: 0.228
    error rate: 0.235
    error rate: 0.245
    error rate: 0.25
    error rate: 0.243
    error rate: 0.229
    error rate: 0.238
    error rate: 0.235
    error rate: 0.241
    error rate: 0.239
    error rate: 0.244
    error rate: 0.226
    error rate: 0.228
    error rate: 0.253
    error rate: 0.23
    error rate: 0.234
    error rate: 0.254
    error rate: 0.217
    error rate: 0.261

(b)

 the testing set changes every time $\to$ the error rate changes every time

 I believe the variance of the latter results is smaller than the fomer one

# Exercise 3.14
https://www.jianshu.com/p/0414b6ccc88c

```python
data = pr.gendath(n=(1000,1000))
feature = +data
label = pr.genlab(n=(1000,1000), lab=[-1, 1])
noiseFeature = np.hstack((feature, np.random.rand(2000,60)))
noiseData = pr.prdataset(noiseFeature, label)
```

```python
u = [pr.nmc(), pr.ldc(), pr.qdc()]
e_nmc = pr.cleval(noiseData, u[0], trainsize=[65, 128, 256, 512], nrreps=20)
e_ldc = pr.cleval(noiseData, u[1], trainsize=[65, 128, 256, 512], nrreps=20)
e_qdc = pr.cleval(noiseData, u[2], trainsize=[65, 128, 256, 512], nrreps=20)
plt.legend()
```

(a)

 the error rate of the testing curve is decreasing, and the one of the training curve is increasing, when the quantity of the training obejcts increases.

(b)

 w tends to be stable when the training set is large enough

 I prefer QDA, which has the lowest error rate

  Ans: Simpler classifiers work generally better when sample sizes are small. None of the classifiers is best.

(c)

 it will be level-off

# Exercise 3.15

```python
data = pr.gendath(n=(500,500))
feature = +data
label = pr.genlab(n=(500,500), lab=[-1, 1])
noiseFeature = np.hstack((+data, np.random.rand(1000,60)))
noiseData = pr.prdataset(noiseFeature, label)
```

```python
e_nmc = pr.cleval(noiseData, pr.knnc((1)), trainsize=[25, 50, 100, 200], nrreps=10)
```

The 1-NN classifier has a zero apparent error

# Exercise 3.16

```python
data = pr.gendatd(n=(1000,1000))
feature = +data
label = pr.genlab(n=(1000,1000), lab=[-1, 1])
noiseFeature = np.hstack((feature, np.random.rand(2000,60)))
noiseData = pr.prdataset(noiseFeature, label)
```

```python
u = [pr.nmc(), pr.ldc(), pr.qdc()]
e_nmc = pr.cleval(noiseData, u[0], trainsize=[65, 128, 256, 512], nrreps=20)
e_ldc = pr.cleval(noiseData, u[1], trainsize=[65, 128, 256, 512], nrreps=20)
e_qdc = pr.cleval(noiseData, u[2], trainsize=[65, 128, 256, 512], nrreps=20)
plt.legend()
```

(a)

  The curves will perfomance in a different kind of way because of the difference in data distributions and the way the classifiers fit these distributions.

# Exercise 3.17

Not done

# Exercise 3.18

Not done

# Exercise 3.19

 Ask for TAs

 This dataset consists of features of handwritten numerals (0-9) extracted from a collection of Dutch utility maps. 200 patterns per class (for a total of 2,000 patterns) have been digitized in binary images.

 In each file the 2000 patterns are stored in ASCI on 2000 lines. The first 200 patterns are of class 0, followed by sets of 200 patterns for each of the classes 1-9. Corresponding patterns in different feature sets (files) correspond to the same original character.

```python
kar = np.loadtxt("mfeat-kar.txt", dtype=np.float64)
label = pr.genlab(n=[200]*10, lab = [0,1,2,3,4,5,6,7,8,9])
data = pr.prdataset(kar, targets=label)
print(data)
```

    2000 by 64 prdataset with 10 classes: [200 200 200 200 200 200 200 200 200 200]

```python
classifier = pr.qdc()
e_qdc = pr.clevalf(data, classifier, trainsize=0.4, nrreps=10)
plt.legend()
plt.show()
```

# Exercise 3.20

(a)

```python
data = pr.gendatb(n=(20,20), s=2.0)
knn1 = pr.knnc((1))
knn3 = pr.knnc((3))

for n in range(2,11):
    e_knn1 = pr.prcrossval(data, knn1, k=n, nrrep=3)
    e_knn3 = pr.prcrossval(data, knn3, k=n, nrrep=3)

    e1 = np.sum(e_knn1, axis=1)/e_knn1.shape[1]
    e3 = np.sum(e_knn3, axis=1)/e_knn3.shape[1]

    print("N = " + str(n), "\t", "mean", "\t", "variance")
    print("1-nn \t", np.round(np.mean(e1),3), "\t", np.round(np.var(e1),3))
    print("3-nn \t", np.round(np.mean(e3),3), "\t", np.round(np.var(e3),3))
    print()
```

    N = 2 	 mean 	 variance
    1-nn 	 0.108 	 0.002
    3-nn 	 0.133 	 0.003

    N = 3 	 mean 	 variance
    1-nn 	 0.083 	 0.001
    3-nn 	 0.14 	 0.001

    N = 4 	 mean 	 variance
    1-nn 	 0.142 	 0.004
    3-nn 	 0.142 	 0.0

    N = 5 	 mean 	 variance
    1-nn 	 0.092 	 0.003
    3-nn 	 0.15 	 0.006

    N = 6 	 mean 	 variance
    1-nn 	 0.076 	 0.002
    3-nn 	 0.137 	 0.002

    N = 7 	 mean 	 variance
    1-nn 	 0.087 	 0.003
    3-nn 	 0.115 	 0.004

    N = 8 	 mean 	 variance
    1-nn 	 0.101 	 0.002
    3-nn 	 0.118 	 0.004

    N = 9 	 mean 	 variance
    1-nn 	 0.09 	 0.006
    3-nn 	 0.133 	 0.014

    N = 10 	 mean 	 variance
    1-nn 	 0.075 	 0.003
    3-nn 	 0.108 	 0.007

(b) + (c)

 Typically you get a less biased estimate of the error when you increase n, but the variance
typically increases a bit with larger n

(d)
Typically you get a less biased estimate of the error when you increase n, but the variance typically increases a bit with larger n

# Exercise 3.21

Not done

# Exercise 3.22 + 3.23

```python
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
# confmat from prtools cannot be used on python, that's suck

# I will do 3.23 instead of 3.22, they are redundant
```

(22.b) lab is the true target, whereas lab2 is the predicted target

(23.a)

```python
kar = np.loadtxt("mfeat-kar.txt", dtype=np.float64)
zer = np.loadtxt("mfeat-zer.txt", dtype=np.float64)
label = pr.genlab(n=[200]*10, lab = [0,1,2,3,4,5,6,7,8,9])
data_kar = kar
data_zer = zer

print("kar set: ", data_kar.shape)
print("zer set ", data_zer.shape)
```

    kar set:  (2000, 64)
    zer set  (2000, 47)

```python
x_train, x_test, y_train, y_test = train_test_split(data_kar, label, test_size=0.3, random_state=40)
print("total test sample:", len(x_test))
```

    total test sample: 600

```python
qdc = QuadraticDiscriminantAnalysis().fit(x_train, y_train)
y_hat = qdc.predict(x_test)
print(confusion_matrix(y_test, y_hat, labels=[0,1,2,3,4,5,6,7,8,9]))
error = np.sum(y_hat != y_test)/len(y_hat)
print("error rate: ", error)
```

    [[59  0  0  0  0  0  0  0  3  0]
     [ 0 55  1  0  0  1  0  0  2  1]
     [ 0  0 55  0  0  0  0  0  0  1]
     [ 0  0  0 60  0  3  0  0  0  0]
     [ 0  0  0  0 51  0  0  0  0  0]
     [ 0  0  0  2  0 52  0  0  0  0]
     [ 0  0  0  0  0  0 71  0  1  0]
     [ 0  1  2  2  0  0  0 52  0  4]
     [ 0  0  0  1  0  0  0  0 66  0]
     [ 0  0  0  0  0  0  0  0  0 54]]
    error rate:  0.041666666666666664

```python
x_train, x_test, y_train, y_test = train_test_split(data_zer, label, test_size=0.3, random_state=40)
print("total test sample:", len(x_test))
```

    total test sample: 600

```python
qdc = QuadraticDiscriminantAnalysis().fit(x_train, y_train)
y_hat = qdc.predict(x_test)
print(confusion_matrix(y_test, y_hat, labels=[0,1,2,3,4,5,6,7,8,9]))
error = np.sum(y_hat != y_test)/len(y_hat)
print("error rate: ", error)
```

    [[58  0  0  0  0  1  0  0  3  0]
     [ 0 53  0  1  2  1  3  0  0  0]
     [ 0  0 51  2  0  2  1  0  0  0]
     [ 0  0  2 55  0  4  1  0  0  1]
     [ 0  2  0  0 49  0  0  0  0  0]
     [ 0  0  0  2  1 49  2  0  0  0]
     [ 0  0  0  0  0  0 21  0  1 50]
     [ 0  0  0  2  1  0  0 57  0  1]
     [ 2  1  0  1  1  4  1  0 55  2]
     [ 0  0  0  0  0  2 37  0  0 15]]
    error rate:  0.22833333333333333

(c)

 I don't know what does the question mean. Why should I judge it from a confusion matrix, and why not observe it from the feature learning curve?

# Exercise 3.24

(a)

The answer is b=2c. If it is a cost matrix, I can't agree more, but it is a confustion matrix, so I prefer c=2b.

# Exercise 3.25

Not done


## Physics 152L Radioactivity Lab ##

### 1. Read in the data

Import the usual libraries, and then read in the data from the saved CSV file into a Pandas dataframe.

We can use df.info() to get a basic summary of the data.

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

df = pd.read_csv("./Demo files/Sawtooth test.csv")

df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 11 entries, 0 to 10
    Data columns (total 2 columns):
     #   Column        Non-Null Count  Dtype
    ---  ------        --------------  -----
     0   Time (s)      11 non-null     int64
     1   Position (m)  11 non-null     int64
    dtypes: int64(2)
    memory usage: 304.0 bytes

### 2. Splitting Up The Data ###

Inspection of the dataframe shows us that there are five data sets from the different
parts of the experiment that are combined into our single CSV file, and hence into our
initial dataframe.

We can use df.filter() to parse this main dataframe and create separate dataframes for
each part of the experiment.

```python
background = df.filter(['Time (s) Background',
                        'Geiger Counts (counts/sample) Background',
                        'RootN (cts/5s) Background',
                        'Background Subtracted (cts/5s) Background'], axis=1).dropna()
background2 = df.filter(['Time (s) Background2',
                        'Geiger Counts (counts/sample) Background2',
                        'RootN (cts/5s) Background2',
                        'Background Subtracted (cts/5s) Background2'], axis=1).dropna()
barium = df.filter(['Time (s) Barium',
                        'Geiger Counts (counts/sample) Barium',
                        'RootN (cts/5s) Barium',
                        'Background Subtracted (cts/5s) Barium'], axis=1).dropna()
radium = df.filter(['Time (s) Radium',
                        'Geiger Counts (counts/sample) Radium',
                        'RootN (cts/5s) Radium',
                        'Background Subtracted (cts/5s) Radium'], axis=1).dropna()
radsamples = df.filter(['Time (s) RadSamples',
                        'Geiger Counts (counts/sample) RadSamples',
                        'RootN (cts/5s) RadSamples',
                        'Background Subtracted (cts/5s) RadSamples'], axis=1).dropna()
```

### 3.  Part I:  Background Measurments ###

We begin by plotting the data from part I.

In order to calculate the average background rate, with uncertainty, we do the following:
* Get the total integrated counts
* Get the total time
* Calculate the average rate
* Calculate the uncertainty, using the fact that $\Delta N = \sqrt{N}$.

There are two background measurements in the CSV file, and so we will do the same thing
for each of these two data sets.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(background['Time (s) Background'],
         background['Geiger Counts (counts/sample) Background'],'r.', label = 'Raw Data')
plt.plot(background['Time (s) Background'],
         background['RootN (cts/5s) Background'],'b--', label = 'Cumulative Error')
plt.plot(background['Time (s) Background'],
         background['Background Subtracted (cts/5s) Background'],'g-',
         label = 'Background Subtracted')
plt.legend()

entries = len(background)
count_sum = background['Geiger Counts (counts/sample) Background'].sum()
end_time = background['Time (s) Background'][entries-1]
rate = count_sum/end_time*60
erate = np.sqrt(count_sum)/end_time*60.0

print ('Counts = ',count_sum,' Time = ',end_time,' Rate = ',rate,' +/- ',erate)
```

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(background2['Time (s) Background2'],
         background2['Geiger Counts (counts/sample) Background2'],'r.',label = 'Raw Data' )
plt.plot(background2['Time (s) Background2'],
         background2['RootN (cts/5s) Background2'],'b--', label = 'Cumulative Error' )
plt.plot(background2['Time (s) Background2'],
         background2['Background Subtracted (cts/5s) Background2'],
         'g-', label = 'Background Subtracted')
plt.legend()

entries = len(background2)
count_sum = background2['Geiger Counts (counts/sample) Background2'].sum()
end_time = background2['Time (s) Background2'][entries-1]
rate = count_sum/end_time*60
erate = np.sqrt(count_sum)/end_time*60.0

print ('Counts = ',count_sum,' Time = ',end_time,' Rate = ',rate,' +/- ',erate)
```

### 3.  Barium Measurements

Plot the barium data.  The expected distribution is:

\begin{equation}
R(t) = R_0 e^{-Bt} \nonumber
\end{equation}

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         barium['Background Subtracted (cts/5s) Barium'],'g.',
         label='Background Subtracted')
```

So, the distribution looks roughly exponential, which is good.  But, in order to extract the parameters of this distribution, i.e. $R_0$ and $B$, we take a slightly different approach.

Taking the natural logarithm of both sides, we have that:

\begin{equation}
ln(R) = ln(R_0 e^{-Bt}) = ln(R_0) - Bt \nonumber
\end{equation}

Thus, if we plot $ln(R)$ vs. $t$, then we expect a $linear$ graph, with a positive y-intercept (=$ln(R_0)$) and negative slope (=$-B$).

We can use the linear regression fitting model from sklearn to fit this data, and extract
the numerical values of the slope and intercept.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         np.log(barium['Background Subtracted (cts/5s) Barium']),'g.',
         label='Background Subtracted')

from sklearn import linear_model

reg = linear_model.LinearRegression()

x_data = barium['Time (s) Barium'].values.reshape(-1, 1)
y_data = np.log(barium['Background Subtracted (cts/5s) Barium'].values.reshape(-1, 1))

reg.fit(x_data,y_data)

print ('Fit: y_intercept = ',reg.intercept_)
print ('Fit: slope = ',reg.coef_)

y_fit = x_data*reg.coef_ + reg.intercept_
plt.plot(x_data,y_fit,'r-',label='Linear Fit')
plt.legend()
```

Note that the linear fit is NOT a particularly good one, because by the end of the data set, it
appears that the radioactive material has mostly decayed away, and we are not seeing the
same exponential fall-off.

So, what to do?  Well, we can limit the region of the fit to just the first 450 second or so, where the decay does appear to follow the expected log-linear dependence.

```python
barium2 = barium[barium['Time (s) Barium'] < 450]

fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         np.log(barium['Background Subtracted (cts/5s) Barium']),'g.',
         label='Background Subtracted')

from sklearn import linear_model

reg = linear_model.LinearRegression()

x_data = barium2['Time (s) Barium'].values.reshape(-1, 1)
y_data = np.log(barium2['Background Subtracted (cts/5s) Barium'].values.reshape(-1, 1))

reg.fit(x_data,y_data)

print ('Fit: y_intercept = ',reg.intercept_)
print ('Fit: slope = ',reg.coef_)

print ('Half-Life = ',float(-np.log(2)/reg.coef_),' seconds.')

y_fit = x_data*reg.coef_ + reg.intercept_
plt.plot(x_data,y_fit,'r-',label='Linear Fit')
plt.legend()
```

###  4. Measurements with Different Absorbers ###

Let's begin by looking at the raw data.

We that there are five distinct "sections" in the data set that correspond to five
different experimental conditions:

* Gamma + Beta Source with no absorber
* Gamma + Beta Source with absorber to block beta particles
* Gamma + Beta Source with Cardboard absorber
* Gamma + Beta Source with Aluminum absorber
* Gamma + Beta Source with Lead absorber

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radium['Time (s) Radium'],radium['Background Subtracted (cts/5s) Radium'],'g-')
```

For each of the five measurements, we will set a time range (t1min and t1max).  Then, we filter the data set to include only measurements during these time windows.  Once we have the filtered data set, we can calculate an average rate over that time window.  In each case, we plot a horizontal line at the appropriate rate for that time window on top of the raw data.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radium['Time (s) Radium'],radium['Background Subtracted (cts/5s) Radium'],'g-')

t1min = 0.0
t1max = 90.0
gammabeta = radium[radium['Time (s) Radium']>t1min]
gammabeta = gammabeta[gammabeta['Time (s) Radium']<t1max]

gammabeta_rate = gammabeta['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Gamma+Beta Rate = ',gammabeta_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([gammabeta_rate,gammabeta_rate])
plt.plot(x,y,'b-')

t1min = 100.0
t1max = 170.0
gamma = radium[radium['Time (s) Radium']>t1min]
gamma = gamma[gamma['Time (s) Radium']<t1max]

gamma_rate = gamma['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Gamma Rate = ',gamma_rate,' cts/5 sec')
print ('Beta Rate = ',gammabeta_rate-gamma_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([gamma_rate,gamma_rate])
plt.plot(x,y,'b-')

print ("")

t1min = 180.0
t1max = 240.0
cardboard = radium[radium['Time (s) Radium']>t1min]
cardboard = cardboard[cardboard['Time (s) Radium']<t1max]

cardboard_rate = cardboard['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Cardboard Rate = ',cardboard_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([cardboard_rate,cardboard_rate])
plt.plot(x,y,'b-')

t1min = 260.0
t1max = 305.0
aluminum = radium[radium['Time (s) Radium']>t1min]
aluminum = aluminum[aluminum['Time (s) Radium']<t1max]

aluminum_rate = aluminum['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Aluminum Rate = ',aluminum_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([aluminum_rate,aluminum_rate])
plt.plot(x,y,'b-')

t1min = 340.0
t1max = 410.0
lead = radium[radium['Time (s) Radium']>t1min]
lead = lead[lead['Time (s) Radium']<t1max]

lead_rate = lead['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Lead Rate = ',lead_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([lead_rate,lead_rate])
plt.plot(x,y,'b-')
```

###  4. Measurements with Different Sources (With and Without Lead Absorber)

In this part of the experiment, there are ten separate measurements, corresponding to five different sources, with data taken both with and without a lead absorber.

We begin by plotting the raw data, and we can clearly see the ten different situations.

As in the last section, we will calculate the average rate for each of these ten time windows, and plot this average rate on top of the data.  Of course, we need to define the ten time windows (as four arrays - two for no absorber and two for absorber).

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radsamples['Time (s) RadSamples'],
         radsamples['Background Subtracted (cts/5s) RadSamples'],'g-')
```

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radsamples['Time (s) RadSamples'],
         radsamples['Background Subtracted (cts/5s) RadSamples'],'g-')

tsmin = np.array([0.0,250.0,500.0,720.0,990.0])
tsmax = np.array([105.0,340.0,580.0,830.0,1070.0])
tlmin = np.array([120.0,375.0,650.0,850.0,1100.0])
tlmax = np.array([220.0,460.0,720.0,950.0,1190.0])

for i in range(5):
    t1min = tsmin[i]
    t1max = tsmax[i]
    rate = radsamples[radsamples['Time (s) RadSamples']>t1min]
    rate = rate[rate['Time (s) RadSamples']<t1max]

    no_lead_rate = rate['Background Subtracted (cts/5s) RadSamples'].mean()*1.0
    print ('NoLeadRate[',i+1,'] = ',no_lead_rate,' cts/5 sec')

    x = np.array([t1min,t1max])
    y = np.array([no_lead_rate,no_lead_rate])
    plt.plot(x,y,'b-')

    t1min = tlmin[i]
    t1max = tlmax[i]
    rate = radsamples[radsamples['Time (s) RadSamples']>t1min]
    rate = rate[rate['Time (s) RadSamples']<t1max]

    lead_rate = rate['Background Subtracted (cts/5s) RadSamples'].mean()*1.0
    print ('LeadRate[',i+1,'] = ',lead_rate,' cts/5 sec')

    x = np.array([t1min,t1max])
    y = np.array([lead_rate,lead_rate])
    plt.plot(x,y,'b-')
    print()
```

```python

```

```python

```