Project 1: Arduino as a Laboratory Interface
========================

The purpose of this project is twofold:

1. to familiarize yourself with some of the tools we'll be using in this course: the Arduino compiler, simulator, and microcontroller, electronic breadboards, oscilloscopes, "C" programming, "python" programming and jupyter notebooks, AND

2. to build a simple circuit that enables us to study the behvior of an LED.

As you progress through the project please be sure to ask questions and get help if you are confused or unclear on how to proceed. Since most students taking this course have used python in intro physics, and have at least one course in programming computers in "c" or "c++". As a result the treatment of those topics will be fairly brief. If you have not use python and/or c/c++ before please alert your instructor and they will provide you with extra materials and instruction as necessary.

Here are some references you may find helpful:

1. [Dive into python](https://diveintopython3.problemsolving.io) (intro to the python language) [http://python.org](http://python.org)
2. Arduino Simulator [http://tinkercad.com](http://tinkercad.com) (this simulator also simulates components and breadboards!)
3. Arduino programming [http://arduino.cc](http://arduino.cc)
4. Intro to C programming: [C programming text](http://www-personal.acfr.usyd.edu.au/tbailey/ctext/ctext.pdf)

We're going to jump right into using python and "C". You should consult the references here to dig deeper, but we'll go over what you need for now. For this project we'll be completing a series of tasks:

Tasks:
-------

1. Set up the circuit [shown below](#Project-1-Schematic-Diagram). Choose values for R1 and R2 that make sense. Ask if you need help!

2. [Program your Arduino](#Arduino-Programming) to collect data as you adjust the driving current through the diode.

2. Run the program to collect voltage vs. time data from the Arduino

3. Analyze (you may read the data into excel, but please use jupyter to do any fitting).

4. Compare results with expectations. See the example [linear](#Linear-Analysis) and [non-linear](#Non-Linear-Analysis) analysis python programs given below.

5. Carry out the [Statistical Exercises](Normal%20Distribution%20and%20Estimation%20of%20Parameters.ipynb) for project one.

6. Submit a project report describing your results.

Criteria for success:

See the [rubric](../report_rubric.pdf) and the [sample](../sample/Sample%20Report.ipynb).

Controlling a Voltage with Arduino
--------------------------------

The next few cells we'll be reviewing the charging and discharging of capacitors. We need this because we'll be using an RC circuit to control the driving voltage across the diode circuit that forms the core of this project.

Charging and Discharging Capacitors
---------------------------------

Let's start with RC Circuits. Suppose we have a charging circuit like that shown in Fig 1 where the voltage source (V1 in Fig. 1) can be turned on and off periodicallly. This produces a current that alternately charges and discharges the capacitor. Suppose the voltage supplied by V1 is 10V when it's "on" and 0V when it's "off". If we assume the capacitor is initially uncharged when V1 is first turned on, there will be a potential drop of 10V across the resistor R1 initially. This will produce an initial current of 10V/10k$\Omega$ = 1mA. As the capacitor charges this current will diminish, approaching zero in a few RC time constants (in this case, as you can see from Fig. 1, RC=50ms). If the source is set up to toggle between "on" and "off" every 500ms, there will be ample time for the capacitor to become very nearly fully charged before the voltage toggles to "off". When the voltage does toggle "off" the current will flow in the opposite direction slowly draining the charge on the capacitor. As you learned in intro physics the voltage on the capacitor as a function of time during the charging and discharging cycles is exponential (see e.g., Chabay & Sherwood, vII, 4th ed, pp785-786).

\begin{equation}
V(t) = V_0\left(1-e^{-t/RC}\right); {\rm (charging)}
\end{equation}

\begin{equation}
V(t) = V_0 e^{-t/RC}; {\rm (discharging)}
\end{equation}

Let's use some python to visualize this result.

```python
# invoke the matplotlib magic to get access to plot inline
#

import matplotlib.pyplot as pl # plotting library
import numpy as np             # numerical library

#
# Set up some constants
#
V0=10.0 # Volts
R1=10e3 # Ohms
C1=5e-6 # Farads

"""
for the first 0.25 sec we'll be charging.
"""
t1 = np.linspace(0, 0.25, 100)     # time array, 100 values evenly spaced beteen 0 and 0.25 seconds
Vc = V0*(1.0-np.exp(-t1/(R1*C1)))  # array of voltages on the capacitor at different times
pl.xlabel('time (s)')              # make the graph purty, give it x and y labls and a title
pl.ylabel('capacitor voltage (V)')
pl.title("Voltage on capacitor during charging cycle")
pl.grid()                          # add a grid so it's easier to read
pl.plot(t1, Vc, 'b-')              # plot the data, t1 on the horizontal, Vc on the vertical, using a blue line
```

You can see how the capacitor voltage is very close to 10V at the end of the 250ms charging cycle.

```python
Vf = Vc[-1]   # take the last voltage in the Vc array and check it. Note that a '-1' index gets the last value
print(Vf)
```

    9.932620530009146

So, the voltage on the capacitor is about 68mV shy of 10V. Or, if you'd like (using a cash analog) we could've gotten up to a \$10 reward, but we still have a mere 7 cents to go.

Close!

What about the discharging?

```python
Vc2 = Vf*np.exp(-t1/(R1*C1))       # compute the voltage during the discharge cycle.
pl.xlabel('time (s)')              # make the graph purty
pl.ylabel('capacitor voltage (V)')
pl.title("Voltage on capacitor during discharge cycle")
pl.grid()
pl.plot(t1, Vc2, 'b-') # plot the data, t1 on the horizontal, Vc2 on the vertical using a blue line

Vf2 = Vc2[-1] # take the last voltage
print("The final voltage after 250ms of discharge is:", Vf2)
```

Again (using a cash analogy), here we started with \$9.93 and we ended up with less than 7 cents in the end.

Practically nothing. ;-)

You should familiarize yourself with these equations. Here are some questions to help you check your understanding:

1. What is the voltage on the capacitor at 0.1s after the voltage source turns "on"? (ans 8.65V)

2. What is the voltage on the capacitor at 0.15s after the voltage source turns "off"? (ans 0.50V)

3. How long after the voltage source turns "on" does the voltage on the capacitor reach 5V? (ans 0.035s)

4. How long after the voltage source turns "off" does the voltage on the capacitor reach 2V? (ans 0.080s)

We can also use LTSpice to simulate this circuit.

How is this relevant?
------------------------

The output pins on a basic Arduino are digital. This means they can either be set "high" (typically 5V, or sometimes 3.3V) or "low" (usually 0V), but nothing in-between. We wish to adjust the current flowing through a device (in this case, an LED) and make simultaneous measurements of the voltage drop across the diode over a range of voltages and currents.

# So what are we doing?

This week we'll be using the Arduino <http://arduino.cc> microcontroller to study the behavior of an LED. The idea is to drive the LED with different currents and to measure the voltage drop across the LED. There is a standard model ([https://en.wikipedia.org/wiki/Shockley_diode_equation](https://en.wikipedia.org/wiki/Shockley_diode_equation)) for this relationship:

$$ I = I_o (e^{\frac{q V_d}{\eta k_b T}}-1) $$

We'll want to compare the data we collect with the predictions of this model.

What follows is an introduction to Arduino programming. If you've already coded a lot in "C" or "C++" this should all be pretty easy. If you haven't, it's time to learn! The good news is that our coding will be pretty elementary. All we really need are simple loops, variables, arrays and functions.

Arduino Programming
------------------

If you open the Arduino IDE and choose File -> Examples -> Basic -> Blink you'll find this code in your IDE window:

    /*
      Blink
      Turns on an LED on for one second, then off for one second, repeatedly.

      Most Arduinos have an on-board LED you can control. On the Uno and
      Leonardo, it is attached to digital pin 13. If you're unsure what
      pin the on-board LED is connected to on your Arduino model, check
      the documentation at http://arduino.cc

      This example code is in the public domain.

      modified 8 May 2014
      by Scott Fitzgerald
     */

    // the setup function runs once when you press reset or power the board
    void setup() {
      // initialize digital pin 13 as an output.
      pinMode(13, OUTPUT);
    }

    // the loop function runs over and over again forever
    void loop() {
      digitalWrite(13, HIGH);   // turn the LED on (HIGH is the voltage level)
      delay(1000);              // wait for a second
      digitalWrite(13, LOW);    // turn the LED off by making the voltage LOW
      delay(1000);              // wait for a second
    }

You can actually run this on your Arduino right away! Make sure you go to Tools -> Board and select the correct Arduino type. Then go to "Tools -> Port" and select the serial port to which your Arduino is assigned. (If you diconnect, then check "Tools -> Port", then reconnect and check "Tools -> Port" again you shoud be able to see which is the "new" port in the list the corresponds to your Arduino.)

Let's go through this code and describe what's happening.

If you've already taken "c/c++" programming you'll be familiar with most of these ideas right away. If you've not taken a programming course you may need some extra support/documentation. Please see me if this is the case and I'll be sure to provide some additional resources.

First the program is processed by a 'compiler' that reads the text and interprets it as a set of steps to be translated into machine instructions for the microcontroller. Lines in the text that start with a "#" are meant to be processed *before* the translation process begins, so these are called "compiler directives" rather than instructions.

Next, anything between '/*' and '*/' is a comment. Anything after '//' on a line, is a comment.

There are two critical functions in an Ardiuno program 'setup' and 'loop'.

Setup
-----

The 'setup' function is executed any time the Arduino is reset. This happens:

1. when you upload a new program
2. when you first connect with a serial port
3. when you push the reset button, or
4. when ground the reset pin.

Let's look at the setup function for this example. Note that it only does one thing: declare that pin 13 on the Arduino is going to be used for output.  It turns out pin 13 on most Arduinos is connected to an LED, so when the pin goes high, you can see light. Other things that typically happen in setup are initialization of variables, arrays, setting various ports' intial values and generally getting things ready to go.

    void setup() {
      // initialize digital pin 13 as an output.
      pinMode(13, OUTPUT);
    }

Loop
----

The 'loop' function is called immediately after 'setup'. After loop completes it is called again, and again, and again, forever (or until power is removed/lost, or the board is reset). The loop function is where most of the work of the program happens.

    void loop() {
      digitalWrite(13, HIGH);   // turn the LED on (HIGH is the voltage level)
      delay(1000);              // wait for a second
      digitalWrite(13, LOW);    // turn the LED off by making the voltage LOW
      delay(1000);              // wait for a second
    }

This program's 'loop' function first sets pin 13 high, waits for 1 sec, then sets pin 13 low, waits another second, then exits. This lights the LED, then turns it off, over and over, each second, forever. Notice that this is a *lot* like the voltage source in our example circuit! In fact, that similarity is exactly what we'll use to develop this project.

Basic Concepts in "c"
===========

In case you're not very familiar with the "c" programming language, these explanations may help get you up to speed.

Macros
------

In the example program there is a line:

    digitalWrite(13, HIGH);   // turn the LED on (HIGH is the voltage level)

it's a bit hard to tell what's going on. The reader may not remember/realize that pin 13 is connected to the built-in LED on the Arduino. Wouldn't it be nice if we could communicate that to them in some simple way? That's one of the most useful applications of a commonly used compiler directive called a "macro". You can create your own macros (I like to do this at the very beginning of my programs, so they are easy to find). Here's how it works:

    #define LED 13            // Pin 13 is the LED

The first character on the line is '#', which tells the compiler that it's a special directive, not a step in the program. The 'define' means we're defining a macro, in this case a constant. When the compiler sees this macro it then knows that anywhere in the program that has 'LED' it's to be replaced by the value '13'. Then our digitalWrite function call looks like this:

    digitalWrite(LED, HIGH);   // turn the LED on (HIGH is the voltage level)

Isn't that clearer? Of course it is. ;-) Below is a list of macros that we can use to define constants needed by the program to run project 1.

    #define OUTPIN 3          // apply output voltage to pin 3
    #define IN_ANALOG 0       // which analog pin are we reading?
    #define NUMVALUES 40      // how much data to collect?
    #define STARTUP_DELAY 500 // how long to hold voltage high initially
    #define DECAY_DELAY   10  // how long to measure
    #define LOOP_DELAY 500    // how many millis to delay each time

Because these definitions are constant, they can't be changed during execution. It is a convention to use all upper case characters in the names of constants so the reader can tell that these things are not variable in the context in which they appear.

Variables
---------

Most programs need some values that can change. These are *variables*. A variable needs to be decared with a type and within a scope. The *type* of a variable determines the amount of memory required for the variable's value and the scope determines the times during which the variable is needed during program execution. Basically if the variable is declared outside of any function, it's scope is *global* and it is always available. If it is declared within a function it's scope is *local* (only within the function). It's memory is 'allocated' within the function, and 'released' outside the function. Let's look at an example:

    int x;

This declares `x` to be an integer (16 bits on the Arduino) and since it's not inside a function, it's scope is global. Here's another:

    int foo(int y) {
        int x=y;

        if (y>9) {
            x = 3*y;
        }

        return 100 - x;
    }

Here `x` is a local variable that is only defined within the function `foo`. You can see that it is used as a part of computing the result of the function. If you tried to access the value of `x` outside this function the compiler would complain that `x` is undefined.

Arrays
------

An array is a collection of memory of a particular type. An array is declared using square brakets: `[]` like so:

    int myArray[16];

this would allocated an array called `myArray` with space for 16 integers. You can access individual elements within the array by using that particular element's address or `index`. The index values start at zero and go up to one less than the number of elements, in this case, 15. For example to set the `zeroth` element of `myArray` to 9 you'd use:

    myArray[0] = 9;

An array element can be use just like a normal variable. We will find an array useful so that we can aqcuire and save data quickly.

Handling Conditions
------------------

Sometimes we want one thing to happen `if` a variable has one value, and something `else` to happen if it has a different value. This is most easily accomplished using an `if` statement. Here is a simple example. Suppose we have a heater and a temperature sensor. We decide that we want the heater to run if the temperature sensor measures any voltage *below* 0.488V, and we want to turn it off if it measures *above* 0.488V. Suppose further that the temperature sensor is connected to analog input pin 0, and the heater is connected to digital output pin 3. This code would do the trick:

    #define IN_ANALOG 0
    #define OUT_DIGITAL 3

    int val;  // val will get the integer representation of the input voltage

    val = analogRead(IN_ANALOG);  // read current value on IN_ANALOG pin, val is between 0 and 1023.

    if (val < 100) {
        Serial.println("Voltage is below 0.488 Volts");
        digitalWrite(OUT_DIGITAL, HIGH);  // turn the heater on
    } else {
        Serial.println("Volgate is above 0.488 Volts");
        digitalWrite(OUT_DIGITAL, LOW);   // turn the heater off
    }

The `AnalogRead` function is used to measure analog voltages on the analog input pins of the Arduino Microcontroller. Voltages bewteen 0.0V and 5.0V are mapped onto integer values between 0 and 1023. The `DigitalWrite` function sets an output pin `high` (which is 5.0V) or `low` (which is 0.0V) as shown. This example sets pin 3 high (5V) if the voltage on analog input 0 is below 0.488 Volts and sets pin 3 low (0V) if the voltage on analog input 0 is above 0.488 V.

For Loops
---------

There are several different ways to create looping behaviors in "c", but the most common is the `for` loop. Here is an example `for` loop that sets all the elements of `myArray` to zero.

    #define N 16

    int i;
    int myArray[N];

    for (i=0; i<N; i++) {
        myArray[i] = 0;
    }

The `for` loop begins with `for` and contains a set of three expressions (clauses) separated by semi-colons. The three expressions:

1. initialize the loop control variable(s), `(i=0)`
2. check to see if the loop should continue and `(i<N)`
3. update the loop control variables `(i++)`

respectively. Let's look at those parts one at a time. First `i` is initialized to zero. This way we know where things are starting. Since the goal of this loop is to set all the elements of `myArray` to zero this means we'll be starting with the `zeroth` element. Before the loop exectutes each iteration it evaluates the middle clause to see if it should continue execution. In this case it checks to see if `i<N`. Since `N` is `#define`ed to be 16, this is checking to make sure `i<16`. The first time through the loop `i=0` so this is clearly satisfied. Remeber that the last element of the array has an index `i=15` so checking that `i<16` will allow no higher index than this. Good! Each time the loop completes the `update` clause `(i++)` is exectuted. `i++` is shorthand in "c" for `i=i+1`. This increments the loop control variable `i` by one at the end of each iteration. Note the net effect is that the loop is exectuted once for each value of `i` from `i=0` to `i=15`.

Functions
---------

A function is a body of code that encapsulates some operation. Sometimes functions produce a result that is desired, sometimes they produce side effects that are needed. Here is an example function that produces a result:

    int foo(x) {
        return 3*x;
    }

The function `foo` accepts an integer argument and returns a result that is three times the value of the argument passed. Yes, it's a silly function, but you get the idea. Here is a function that has a side effect:

    void updateHeater(int threshold) {
        int val;  // val will get the integer representation of the input voltage

        val = analogRead(IN_ANALOG);  // read current value on IN_ANALOG pin, val is between 0 and 1023.

        if (val < threshold) {
            digitalWrite(OUT_DIGITAL, HIGH);  // turn the heater on if val < threshold
        } else {
            digitalWrite(OUT_DIGITAL, LOW);   // otherwise turn the heater off
        }
    }

So you can call `updateHeater(100)` and it will turn the heater on or off based on a threshold of 100. Note that as a general rule of coding *style* it's preferable to create functions that

1. return a result, but have no side effects OR
2. have side effects, but return no result

Communication
------------

You can send and receive data over the serial port using the [Serial object](https://www.arduino.cc/en/Reference/Serial) on the Arduino. We'll learn more of the details as the course continues. For this week we only need two functions: `print` [(documentation)](https://www.arduino.cc/en/Serial/Print) and: `println` [(documentation)](https://www.arduino.cc/en/Serial/Println). The primary difference between these is that the `println` version sends a carriage return/linefeed so that the next thing printed will be on a new line. You can print(ln) any single value, integer, float or a string of characters. For floating point values you can specify the number of digits past the decimal point as an optional second argument. So, the following program:

    int x=3;
    float y=2.345;

    void setup() {
      Serial.begin(9600);
      Serial.print("The answer is:");
      Serial.print(x);
      Serial.print(",");
      Serial.println(y, 2);
    }

would print out:

    The answer is:3,2.35

Note that the floating point value is rounded to 2 digits as requested.

## Wait, What am I actually supposed to do?

Your tasks:

1. Set up the circuit shown below. Choose values for R1 and R2 that make sense. Ask if you need help!

2. Program your Arduino to collect data as you adjust the driving current through the diode.

2. Run the program to collect voltage vs. time data from the Arduino

3. Analyze (you may read the data from excel, but please use jupyter to do any fitting).

4. Compare results with expectations. See the example linear and non-linear analysis python programs given below.

5. Carry out the [Statistical Exercises](Normal%20Distribution%20and%20Estimation%20of%20Parameters.ipynb) for project one.

6. Submit a project report (see sample) describing your results.

## Project 1 Schematic Diagram

# Linear Analysis

```python
#
# Example "straight line" fit. This example is from the "RC" lab you may have done in PHYS-163.
# You'll be using diode data, but the concept is the same so long as you can manipulate the data
# so that it's susceptible to a straight line snalysis.
#
#
# This cell loads the data taken earlier and analyzes it by fitting the ln(Vc) vs. t to a straight line.
#
# You are welcome to use this approach if you like
#

import pandas as pd  # load up the pandas (pyData) package
data = pd.read_csv('data.csv')  # read in the data taken previously

(m, b), pcov = np.polyfit(data.time.values, np.log(data.voltage.values), 1, cov=True)  # do the fit

dm = np.sqrt(pcov[0,0]) # uncertainty in the slope
db = np.sqrt(pcov[1,1]) # uncertainty in the intercept

ystar=data.time.values*m + b

pl.title("Decay of charge on a capacitor")
pl.ylabel("ln(V)")
pl.xlabel("time (ms)")
pl.plot(data.time.values, np.log(data.voltage.values), 'b.')
pl.plot(data.time.values, ystar,'r-')
pl.grid()
print("m = %2.3f +/- %2.3f" % (m, dm))
print("b = %2.3f +/- %2.3f" % (b, db))

```

# Don't forget the statistics assignment

You can find it [here](Normal%20Distribution%20and%20Estimation%20of%20Parameters.ipynb).

```python

```

proj1/Arduino as a Laboratory Interface.ipynb


# Encoding of categorical variables

## Sources

* [Encoding categorical variables](https://kiwidamien.github.io/encoding-categorical-variables.html)
* [Benchmarking Categorical Encoders](https://towardsdatascience.com/benchmarking-categorical-encoders-9c322bd77ee8)
* [Smarter Ways to Encode Categorical Data for Machine Learning](https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159)
* [Encoding Categorical Variables in Practice](https://medium.com/epfl-extension-school/encoding-categorical-variables-in-practice-a536907f2013)
* [Types of Categorical Data Encoding Schemes](https://medium.com/analytics-vidhya/types-of-categorical-data-encoding-schemes-a5bbeb4ba02b)
* [Guide to Encoding Categorical Values in Python ](https://pbpython.com/categorical-encoding.html)
* [All about Categorical Variable Encoding](https://towardsdatascience.com/all-about-categorical-variable-encoding-305f3361fd02)
* [Encoding categorical variables: one-hot and beyond](http://www.win-vector.com/blog/2017/04/encoding-categorical-variables-one-hot-and-beyond/)

# Background

Many machine learning algorithms are not able to use non-numeric data. While many features we might use, such as a person's age, or height, are numeric there are many that are not. Usually these features are represented by strings, and we need some way of transforming them to numbers before using scikit-learn's algorithms. The different ways of doing this are called encodings.

**Note**:This notebook does not address the translation of text into numbers/vectors. This is the topic of __word embeddings__ where the aim is to generate vectors such that words with similar meaning are close to one another and that the arithmetic of vectors corresponds to ''arithmetic'' of meaning, e.g. embedding('queen') ~ embedding('king') - embedding('man') + embedding('woman').

## Terminology

**Levels**: A levels of a non-numeric feature are the number of distinct values. The examples listed above are all examples of levels. The number of levels can vary wildly: the number of races for a patient is typically four (asian, black, hispanic, and white), the number of states for the US is 51 (if we include DC separately), while the number of professions is in the thousands.

**Ordinal**: If the levels are ordered, then we call the feature ordinal. For example, if a class grade such as "B+" or "A" is a non-numeric feature, but the letters are not just different, they are ordered (an "A" is better than a "B+", which is better than a "C-" etc).

**Nominal**: If the levels are just different without an ordering, we call the feature categorical. For example, professions or car brands are categorical. If we use an encoding that maps levels to numbers, we introduce an ordering on the categories, which may not be desirable. Most of this article will be about encoding categorical variables.

## Considerations when choosing encodings

**Do you have many levels?** If so, using an encoding that has a level-per-feature is difficult for tree-based models. Trees separate on features that "split" the data into different classes effectively. If there are many levels, it is likely only a tiny fraction of the data belong to one level, so it will be hard for trees to "find" that feature to split on. Typically this isn't a problem for linear models.

**Are there many examples of each level?** If there are only 5 doctors in your dataset, you probably are not going to know the doctor category very well (nor will it generalize). Some encoders deal with this gracefully, while others won't. You might consider making an explicit "other" category for levels, or grouping categories together. This is a problem for all models.

**Could you have new categories at test time?** Some categorical variables can be completely specified at training time (e.g. the levels for race or blood type would be known even with zero training examples). Other categories, such as profession, are so broad that we probably learn the levels from the training data. Some encoders deal with levels that are only in the test set better than others.

**Are the categories related?** Many encoding schemes treat two different levels as "equally different" from one another. If looking at color of a car, a typical encoding has no idea that "brick red" and "red" are more related than "red" and "yellow". One way of solving this problem is to cluster the categories into higher levels, and then encode that category as well.

**Is it reversible?** Does it store a lookup table? Given the encoding of a feature, can you recover the original value? If so, we call the encoding reversible. Generally, reversible features also require a lot of storage if there are a lot of levels (to figure out how to go backward). I would generally see reversible as a negative if it requries storing a lookup table as well.

## Example data

```python
import requests
import category_encoders as ce
import numpy as np
import pandas as pd

def encode_var(var, encoder, y=None):
    if y is None:
        encoder.fit(var)
    else:
        encoder.fit(var, y)
    new_var = encoder.transform(var)
    if isinstance(new_var, pd.DataFrame):
        new_var.insert(0, 'original', var)
        return new_var
    else:
        return pd.DataFrame({'original': var, 'encoder': new_var})

def print_res(res, rows_per_level=2):
    out = pd.DataFrame(columns=res.columns)
    for lvl in res.original.unique():
        out = out.append(res[res.original==lvl].head(rows_per_level))
    return out
```

```python
download_data = False
```

```python
if download_data:
    url = "http://mlr.cs.umass.edu/ml/machine-learning-databases/autos/imports-85.data"
    r = requests.get(url)
    with open('imports-85.data', 'wb') as f:
        f.write(r.content)
```

```python
# Define the headers since the data does not have any
headers = ["symboling", "normalized_losses", "make", "fuel_type", "aspiration",
           "num_doors", "body_style", "drive_wheels", "engine_location",
           "wheel_base", "length", "width", "height", "curb_weight",
           "engine_type", "num_cylinders", "engine_size", "fuel_system",
           "bore", "stroke", "compression_ratio", "horsepower", "peak_rpm",
           "city_mpg", "highway_mpg", "price"]
# Read in the CSV file and convert "?" to NaN
df = pd.read_csv("imports-85.data",
                  header=None, names=headers, na_values="?" )
```

```python
df.head()
```

      symboling
      normalized_losses
      make
      fuel_type
      aspiration
      num_doors
      body_style
      drive_wheels
      engine_location
      wheel_base
      ...
      engine_size
      fuel_system
      bore
      stroke
      compression_ratio
      horsepower
      peak_rpm
      city_mpg
      highway_mpg
      price

      0
      3
      NaN
      alfa-romero
      gas
      std
      two
      convertible
      rwd
      front
      88.6
      ...
      130
      mpfi
      3.47
      2.68
      9.0
      111.0
      5000.0
      21
      27
      13495.0

      1
      3
      NaN
      alfa-romero
      gas
      std
      two
      convertible
      rwd
      front
      88.6
      ...
      130
      mpfi
      3.47
      2.68
      9.0
      111.0
      5000.0
      21
      27
      16500.0

      2
      1
      NaN
      alfa-romero
      gas
      std
      two
      hatchback
      rwd
      front
      94.5
      ...
      152
      mpfi
      2.68
      3.47
      9.0
      154.0
      5000.0
      19
      26
      16500.0

      3
      2
      164.0
      audi
      gas
      std
      four
      sedan
      fwd
      front
      99.8
      ...
      109
      mpfi
      3.19
      3.40
      10.0
      102.0
      5500.0
      24
      30
      13950.0

      4
      2
      164.0
      audi
      gas
      std
      four
      sedan
      4wd
      front
      99.4
      ...
      136
      mpfi
      3.19
      3.40
      8.0
      115.0
      5500.0
      18
      22
      17450.0

5 rows × 26 columns

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 205 entries, 0 to 204
    Data columns (total 26 columns):
    symboling            205 non-null int64
    normalized_losses    164 non-null float64
    make                 205 non-null object
    fuel_type            205 non-null object
    aspiration           205 non-null object
    num_doors            203 non-null object
    body_style           205 non-null object
    drive_wheels         205 non-null object
    engine_location      205 non-null object
    wheel_base           205 non-null float64
    length               205 non-null float64
    width                205 non-null float64
    height               205 non-null float64
    curb_weight          205 non-null int64
    engine_type          205 non-null object
    num_cylinders        205 non-null object
    engine_size          205 non-null int64
    fuel_system          205 non-null object
    bore                 201 non-null float64
    stroke               201 non-null float64
    compression_ratio    205 non-null float64
    horsepower           203 non-null float64
    peak_rpm             203 non-null float64
    city_mpg             205 non-null int64
    highway_mpg          205 non-null int64
    price                201 non-null float64
    dtypes: float64(11), int64(5), object(10)
    memory usage: 41.8+ KB

# Notation

$$ \begin{align}
N: & \text{Number of observations}\\
N_i: & \text{Number of observations with level i}\\
n: & \text{Number of levels}\\
i: & \text{The i-th level}\\
k: & \text{The k-th variable of the encoding}\\
x_k(i): & \text{Value for the k-th variable of the encoding when the original variable has label $i$.}\\
\bar{y}: & \text{The average value of the target}\\
\bar{y}_i: & \text{The average value of the target for level i}
\end{align}$$

```python

```

# Unsupervised Encoding Methods

## Label Encoding

**Description**

Label encoding replaces the *n* labels with values from *0* to *n-1*, in lexicographical order.

**Construction**

$$x(i) = i$$

**When to use**

Never (only for ordinal variables, for which there is the `OrdinalEncoder` class.

```python
df["num_cylinders"].value_counts().sort_index()
```

    eight       5
    five       11
    four      159
    six        24
    three       1
    twelve      1
    two         4
    Name: num_cylinders, dtype: int64

```python
from sklearn.preprocessing import LabelEncoder
```

```python
encoder = LabelEncoder()
res = encode_var(df["num_cylinders"], encoder)
print_res(res)
```

      original
      encoder

      0
      four
      2

      1
      four
      2

      2
      six
      3

      12
      six
      3

      4
      five
      1

      5
      five
      1

      71
      eight
      0

      72
      eight
      0

      55
      two
      6

      56
      two
      6

      18
      three
      4

      49
      twelve
      5

## Ordinal Encoding

**Description**

Same as label encoding, but values are ordered in the order of labels.

**When to use**

For ordinal variables, where the difference between all labels represents the same distance.

```python
df["num_cylinders"].value_counts(sort=False)
```

    three       1
    five       11
    two         4
    six        24
    four      159
    twelve      1
    eight       5
    Name: num_cylinders, dtype: int64

```python
df["num_cylinders"] = df["num_cylinders"].astype('category').cat.reorder_categories(ordered=True, new_categories=['two', 'three', 'four', 'five', 'six', 'eight', 'twelve'])
```

```python
df["num_cylinders"].value_counts(sort=False)
```

    two         4
    three       1
    four      159
    five       11
    six        24
    eight       5
    twelve      1
    Name: num_cylinders, dtype: int64

```python
encoder = ce.ordinal.OrdinalEncoder()
res = encode_var(df[["num_cylinders"]], encoder)
print_res(res)
```

      original
      num_cylinders

      0
      four
      3

      1
      four
      3

      2
      six
      5

      12
      six
      5

      4
      five
      4

      5
      five
      4

      71
      eight
      6

      72
      eight
      6

      55
      two
      1

      56
      two
      1

      18
      three
      2

      49
      twelve
      7

## Dummy/One Hot Encoding

**Description**

Creates a binary indicator for each level.

**Construction**

$$x_k(i) =
\begin{cases}
1 & \text{if } k=i \\
0 & \text{otherwise}
\end{cases}$$

**When to use**

When the main interest is in differences in average values for each level and there are sufficient observations for each level.

```python
df['drive_wheels'].value_counts()
```

    fwd    120
    rwd     76
    4wd      9
    Name: drive_wheels, dtype: int64

```python
encoder = ce.one_hot.OneHotEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      drive_wheels_1
      drive_wheels_2
      drive_wheels_3

      3
      fwd
      0
      1
      0

      5
      fwd
      0
      1
      0

      0
      rwd
      1
      0
      0

      1
      rwd
      1
      0
      0

      4
      4wd
      0
      0
      1

      9
      4wd
      0
      0
      1

## Binary Encoding

**Description**

Similar to One-Hot Encoding, but values of categories are stored as binary bitstrings. Each binary digit create one encoding column, i.e. if there are $n$ levels then there are $\log_2n$ features.

**Construction**

Each level is associated with its order, which is then translated into bitstrings. The encoding variables are the different valeus in the bitstring.

**When to use**

For categorical variables with many levels.

```python
encoder = ce.binary.BinaryEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      drive_wheels_0
      drive_wheels_1
      drive_wheels_2

      3
      fwd
      0
      1
      0

      5
      fwd
      0
      1
      0

      0
      rwd
      0
      0
      1

      1
      rwd
      0
      0
      1

      4
      4wd
      0
      1
      1

      9
      4wd
      0
      1
      1

## BaseN Encoding

**Description**

Base-N encoder encodes the categories into arrays of their base-N representation. A base of $1$ is equivalent to one-hot encoding (not really base-1, but useful), a base of $2$ is equivalent to binary encoding. A base of $n$ is equivalent to vanilla ordinal encoding.

**Construction**

Each level is associated with its order, which is then translated into bitstrings. The encoding variables are the different valeus in the bitstring.

**When to use**

For categorical variables with many levels.

```python
encoder = ce.basen.BaseNEncoder(base=2)
res = encode_var(df[["make"]], encoder)
print_res(res)
```

      original
      make_0
      make_1
      make_2
      make_3
      make_4
      make_5

      150
      toyota
      0
      1
      0
      1
      0
      0

      151
      toyota
      0
      1
      0
      1
      0
      0

      89
      nissan
      0
      0
      1
      1
      0
      1

      90
      nissan
      0
      0
      1
      1
      0
      1

      50
      mazda
      0
      0
      1
      0
      0
      1

      51
      mazda
      0
      0
      1
      0
      0
      1

      30
      honda
      0
      0
      0
      1
      1
      0

      31
      honda
      0
      0
      0
      1
      1
      0

      76
      mitsubishi
      0
      0
      1
      1
      0
      0

      77
      mitsubishi
      0
      0
      1
      1
      0
      0

      182
      volkswagen
      0
      1
      0
      1
      0
      1

      183
      volkswagen
      0
      1
      0
      1
      0
      1

      138
      subaru
      0
      1
      0
      0
      1
      1

      139
      subaru
      0
      1
      0
      0
      1
      1

      194
      volvo
      0
      1
      0
      1
      1
      0

      195
      volvo
      0
      1
      0
      1
      1
      0

      107
      peugot
      0
      0
      1
      1
      1
      0

      108
      peugot
      0
      0
      1
      1
      1
      0

      21
      dodge
      0
      0
      0
      1
      0
      1

      22
      dodge
      0
      0
      0
      1
      0
      1

      10
      bmw
      0
      0
      0
      0
      1
      1

      11
      bmw
      0
      0
      0
      0
      1
      1

      67
      mercedes-benz
      0
      0
      1
      0
      1
      0

      68
      mercedes-benz
      0
      0
      1
      0
      1
      0

      3
      audi
      0
      0
      0
      0
      1
      0

      4
      audi
      0
      0
      0
      0
      1
      0

      118
      plymouth
      0
      0
      1
      1
      1
      1

      119
      plymouth
      0
      0
      1
      1
      1
      1

      132
      saab
      0
      1
      0
      0
      1
      0

      133
      saab
      0
      1
      0
      0
      1
      0

      125
      porsche
      0
      1
      0
      0
      0
      0

      126
      porsche
      0
      1
      0
      0
      0
      0

      43
      isuzu
      0
      0
      0
      1
      1
      1

      44
      isuzu
      0
      0
      0
      1
      1
      1

      0
      alfa-romero
      0
      0
      0
      0
      0
      1

      1
      alfa-romero
      0
      0
      0
      0
      0
      1

      18
      chevrolet
      0
      0
      0
      1
      0
      0

      19
      chevrolet
      0
      0
      0
      1
      0
      0

      47
      jaguar
      0
      0
      1
      0
      0
      0

      48
      jaguar
      0
      0
      1
      0
      0
      0

      130
      renault
      0
      1
      0
      0
      0
      1

      131
      renault
      0
      1
      0
      0
      0
      1

      75
      mercury
      0
      0
      1
      0
      1
      1

## Simple Encoder

**Description**

Compares each level to the reference level, with the intercept as the grand mean.

**Construction**

$$x_k(i) = \begin{cases}
\frac{n-1}{n} & \text{if } k = i \\
-\frac{1}{n} & \text{otherwise}
\end{cases}$$

**When to use**

Same as dummy encoding, but when the interest is in deviations from the grand mean rather than deviations from the reference level.

```python
### Not available in Python
from simple_coding import SimpleEncoder
```

```python
encoder = SimpleEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      intercept
      drive_wheels_0
      drive_wheels_1

      3
      fwd
      1
      0.666667
      -0.333333

      5
      fwd
      1
      0.666667
      -0.333333

      0
      rwd
      1
      -0.333333
      -0.333333

      1
      rwd
      1
      -0.333333
      -0.333333

      4
      4wd
      1
      -0.333333
      0.666667

      9
      4wd
      1
      -0.333333
      0.666667

## Sum encoding / Deviation Encoding / Effect Encoding

**Description**

Sum encoding compares each group effect to the grand mean, i.e. the mean of group means (which is not the overall mean). The encoding representation is the difference of indicator variables minus the indicator for one baseline group. Sum encoding is similar to One-Hot encoding, but the interpretation of effects is different (effect relative to the grand mean vs. effect relative to a baseline group).

**Construction**

$$x_k(i) = \begin{cases}
-1 & \text{if } i = 1\\
1 & \text{if } i+1 = k\\
0 & \text{otherwise}
\end{cases}$$

**When to use**

When the main interest is in differences in average values for each level compared to the grand mean and there are sufficient observations for each level.

```python
encoder = ce.sum_coding.SumEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      intercept
      drive_wheels_0
      drive_wheels_1

      3
      fwd
      1
      0.0
      1.0

      5
      fwd
      1
      0.0
      1.0

      0
      rwd
      1
      1.0
      0.0

      1
      rwd
      1
      1.0
      0.0

      4
      4wd
      1
      -1.0
      -1.0

      9
      4wd
      1
      -1.0
      -1.0

## (Orthogonal) Polynomial Encoding

**Description**
Orthogonal polynomial coding is a form of trend analysis in that it is looking for the linear, quadratic and cubic trends in the categorical variable. Orthogonal polynomials are equations such that each is associated with a power of the variable.

**Construction**

Each encoding variable are the coefficients of the orthogonal polynomials of order $n-1$. See this [Post](https://stats.stackexchange.com/questions/105115/polynomial-contrasts-for-regression) for an explanation of the construction.

**When to use**
This type of coding system should be used only with an ordinal variable in which the levels are equally spaced.

```python
encoder = ce.polynomial.PolynomialEncoder()
res = encode_var(df[["num_cylinders"]], encoder)
print_res(res)
```

      original
      intercept
      num_cylinders_0
      num_cylinders_1
      num_cylinders_2
      num_cylinders_3
      num_cylinders_4
      num_cylinders_5

      0
      four
      1
      -1.889822e-01
      -3.273268e-01
      4.082483e-01
      0.080582
      -5.455447e-01
      0.493464

      1
      four
      1
      -1.889822e-01
      -3.273268e-01
      4.082483e-01
      0.080582
      -5.455447e-01
      0.493464

      2
      six
      1
      1.889822e-01
      -3.273268e-01
      -4.082483e-01
      0.080582
      5.455447e-01
      0.493464

      12
      six
      1
      1.889822e-01
      -3.273268e-01
      -4.082483e-01
      0.080582
      5.455447e-01
      0.493464

      4
      five
      1
      1.617449e-17
      -4.364358e-01
      -1.109626e-16
      0.483494
      -6.714569e-16
      -0.657952

      5
      five
      1
      1.617449e-17
      -4.364358e-01
      -1.109626e-16
      0.483494
      -6.714569e-16
      -0.657952

      71
      eight
      1
      3.779645e-01
      1.195122e-17
      -4.082483e-01
      -0.564076
      -4.364358e-01
      -0.197386

      72
      eight
      1
      3.779645e-01
      1.195122e-17
      -4.082483e-01
      -0.564076
      -4.364358e-01
      -0.197386

      55
      two
      1
      -5.669467e-01
      5.455447e-01
      -4.082483e-01
      0.241747
      -1.091089e-01
      0.032898

      56
      two
      1
      -5.669467e-01
      5.455447e-01
      -4.082483e-01
      0.241747
      -1.091089e-01
      0.032898

      49
      twelve
      1
      5.669467e-01
      5.455447e-01
      4.082483e-01
      0.241747
      1.091089e-01
      0.032898

      18
      three
      1
      -3.779645e-01
      9.521795e-17
      4.082483e-01
      -0.564076
      4.364358e-01
      -0.197386

## Helmert Encoding

**Description**

Helmert coding compares each level of a categorical variable to the mean of the subsequent levels. The first contrast compares the mean of the dependent variable for the second level with the mean of the dependent variable for the first level. The second contrast compares the mean for the third level with the mean for the first two levels, etc.

**Construction**

$$x_k(i) = \begin{cases}
-1 & \text{if } i < k\\
k & \text{if } i = k\\
0 & \text{otherwise}
\end{cases}$$

The representation for level k has -1 for each level before k and then k as value for the current level, starting at *k=2* up to *k=n*.

**When to use**

When levels of a categorical variable are ordered from lowest to highest or from smallest to largest.

```python
encoder = ce.helmert.HelmertEncoder()
res = encode_var(df[["body_style"]], encoder)
print_res(res)
```

      original
      intercept
      body_style_0
      body_style_1
      body_style_2
      body_style_3

      3
      sedan
      1
      0.0
      2.0
      -1.0
      -1.0

      4
      sedan
      1
      0.0
      2.0
      -1.0
      -1.0

      2
      hatchback
      1
      1.0
      -1.0
      -1.0
      -1.0

      9
      hatchback
      1
      1.0
      -1.0
      -1.0
      -1.0

      7
      wagon
      1
      0.0
      0.0
      3.0
      -1.0

      28
      wagon
      1
      0.0
      0.0
      3.0
      -1.0

      69
      hardtop
      1
      0.0
      0.0
      0.0
      4.0

      74
      hardtop
      1
      0.0
      0.0
      0.0
      4.0

      0
      convertible
      1
      -1.0
      -1.0
      -1.0
      -1.0

      1
      convertible
      1
      -1.0
      -1.0
      -1.0
      -1.0

```python
df["body_style"].unique()
```

    array(['convertible', 'hatchback', 'sedan', 'wagon', 'hardtop'],
          dtype=object)

## Backward Difference Encoding

**Description**

Similar to Helmert encoding, but differences are taken with respect to the prior adjacent level (not all prior levels).

**Construction**

$$x_k(i) = \begin{cases}
-\frac{n-k}{n} & \text{if } i \leq k\\
\frac{n}{k} & \text{otherwise}
\end{cases}$$

**When to use**

When levels of a categorical variable are ordered from lowest to highest or from smallest to largest, but the interest is in step-wise differences.

```python
encoder = ce.backward_difference.BackwardDifferenceEncoder()
res = encode_var(df[["body_style"]], encoder)
print_res(res)
```

      original
      intercept
      body_style_0
      body_style_1
      body_style_2
      body_style_3

      0
      convertible
      1
      -0.8
      -0.6
      -0.4
      -0.2

      1
      convertible
      1
      -0.8
      -0.6
      -0.4
      -0.2

      2
      hatchback
      1
      0.2
      -0.6
      -0.4
      -0.2

      9
      hatchback
      1
      0.2
      -0.6
      -0.4
      -0.2

      3
      sedan
      1
      0.2
      0.4
      -0.4
      -0.2

      4
      sedan
      1
      0.2
      0.4
      -0.4
      -0.2

      7
      wagon
      1
      0.2
      0.4
      0.6
      -0.2

      28
      wagon
      1
      0.2
      0.4
      0.6
      -0.2

      69
      hardtop
      1
      0.2
      0.4
      0.6
      0.8

      74
      hardtop
      1
      0.2
      0.4
      0.6
      0.8

## Frequency / Count Encoding

**Description**

Frequency encoding replaces the levels of a categorical variable with their absolute or relative frequency.

**Construction**

$$x(i) = \frac{N_i}{N}$$

**When to use**

When common or uncommon levels have similar influences.

```python
encoder = ce.count.CountEncoder(normalize=True)
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      0.585366

      5
      fwd
      0.585366

      0
      rwd
      0.370732

      1
      rwd
      0.370732

      4
      4wd
      0.043902

      9
      4wd
      0.043902

## Hashing

**Description**

Hashing converts categorical variables to a higher dimensional space of integers, where the distance between two vectors of categorical variables in approximately maintained the transformed numerical dimensional space.

**Construction**

Any hash funtion from the `hashlib` package.

**When to use**

This method is advantageous when the cardinality of categorical is very high.

```python
encoder = ce.hashing.HashingEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)
```

      original
      col_0
      col_1
      col_2
      col_3
      col_4
      col_5
      col_6
      col_7

      0
      rwd
      0
      0
      0
      0
      0
      1
      0
      0

      1
      rwd
      0
      0
      0
      0
      0
      1
      0
      0

      3
      fwd
      0
      0
      0
      0
      1
      0
      0
      0

      5
      fwd
      0
      0
      0
      0
      1
      0
      0
      0

      4
      4wd
      0
      0
      0
      0
      0
      0
      1
      0

      9
      4wd
      0
      0
      0
      0
      0
      0
      1
      0

# Supervised Encoding Methods / Bayesian Encoders

## Target / Mean / Impact / Likelihood Encoding

**Description**

Target encoding replaces each category with the average value of all ovservations for that category, mixed with a prior.

For the case of *categorical target*: features are replaced with a blend of posterior probability of the target given particular categorical value and the prior probability of the target over all the training data.

For the case of *continuous target*: features are replaced with a blend of the expected value of the target given particular categorical value and the expected value of the target over all the training data.

**Construction**

$$x(i) = \bar{y} (1-s) + s \bar{y}_i$$
where $s$ is a smoothing parameter calculated as:
$$s=\frac{1}{1+\exp{\left(-\frac{n-mdl}{a}\right)}}$$
with 'mdl' standing for 'min data in leaf' and $a$ is a regularization parameter. The mdl defines a threshold where prior, i.e. the mean of the target, and target mean for a given category value have the same weight.

**When to use**

This encoding method brings out the relation between similar categories, but the connections are bounded within the categories and target itself. The advantages of the mean target encoding are that it does not affect the volume of the data and helps in faster learning. Target encoding is powerful in prediction tasks, but runs the risk of target leakage. The leakage can be controlled via regularization, data augmentation by adding noise to the encoding representation, or through double validation.

```python
encoder = ce.target_encoder.TargetEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      9244.779661

      5
      fwd
      9244.779661

      0
      rwd
      19757.613333

      1
      rwd
      19757.613333

      4
      4wd
      10243.702296

      9
      4wd
      10243.702296

## James-Stein Encoding

**Description**

Target encoding based on the James-Stein mean estimator, rather than the sample mean. The idea is to improve the estimation of the category's mean target by shrinking them towards the central average.

**Construction**

$$x(i) = \bar{y} B + (1-B) \bar{y}_i$$
where $B$ is a shrinkage parameter. A common value is
$$B=\frac{Var\left[y_k\right]}{Var\left[y_k\right]+Var\left[y\right]}$$
but the value could also be set via cross-valdiation. The intuition behind the equation is that if the mean estiamte of a category is uncertain (high variance), then stronger shrinkage should be applied.

**When to use**

Same as for target encoder.

```python
encoder = ce.james_stein.JamesSteinEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      9244.779661

      5
      fwd
      9244.779661

      0
      rwd
      19757.613333

      1
      rwd
      19757.613333

      4
      4wd
      10241.000000

      9
      4wd
      10241.000000

## M-estimator Encoding

**Description**

A simplified version of the target encoder as it has only one hyerparameter.

**Construction**

$$x(i) = \frac{N_i + m \times \bar{y}}{\bar{y}_i + m}$$
where $m$ is a smoothing parameter.

**When to use**

Same as for target encoder.

```python
encoder = ce.m_estimate.MEstimateEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      9278.076717

      5
      fwd
      9278.076717

      0
      rwd
      19671.422755

      1
      rwd
      19671.422755

      4
      4wd
      10570.569928

      9
      4wd
      10570.569928

## Leave One Out Encoding

**Description**

The encoding is calculated for each observation $j$ by calculating the average value of the target of all observations with the same target value as observation $j$ except $j$.

**Construction**

$$ x_k^{(j)} = \frac{a}{b}\frac{\sum_{i\neq j}(y_i(x_i==k)-y_j}{\sum_{i\neq j}x_i==k}$$

**When to use**

Same as for target encoder.

```python
encoder = ce.leave_one_out.LeaveOneOutEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      9244.779661

      5
      fwd
      9244.779661

      0
      rwd
      19757.613333

      1
      rwd
      19757.613333

      4
      4wd
      10241.000000

      9
      4wd
      10241.000000

## Catboost Encoding

**Decription**

Catboost is an improvement over the leave-one-out encoder. It is intended to overcome the target leackage problems.

**Construction**
Same as LOO encoder, but the sums only range up to the current observations:
$$ x_k(i) = \frac{a}{b}\frac{\sum_{j\neq i, j\leq k}(y_j(x_j==k)-y_i}{\sum_{j\neq i, j\leq k}x_j==k}$$

**When to use**

```python
encoder = ce.cat_boost.CatBoostEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      9278.076717

      5
      fwd
      9278.076717

      0
      rwd
      19671.422755

      1
      rwd
      19671.422755

      4
      4wd
      10570.569928

      9
      4wd
      10570.569928

## Weight of Evidence Encoder

**Description**

The encoding is a measure of the *strength* that seperates effect from no effect. The encoding can only be used for binary targets.

**Construction**

The encoding is calculated from a modified odds ratio, which is intended to prevent target leakage:
$$ \begin{align}
numerator =& \ \frac{N_i+a}{N_i\bar{y}_i + 2a} \\
denominator =& \ \frac{N-N_i+a}{N\bar{y}-N_i\bar{y}_i  2a} \\
x(i) =& \ \log\left(\frac{numerator}{denominator}\right)
\end{align}
$$

**When to use**

For binary targets

```python
encoder = ce.woe.WOEEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['fuel_type']=='gas')
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      0.275848

      5
      fwd
      0.275848

      0
      rwd
      -0.435318

      1
      rwd
      -0.435318

      4
      4wd
      0.162519

      9
      4wd
      0.162519

## Probability Ratio Encoding

**Description**

Probability Ratio Encoding is similar to Weight Of Evidence encoding, but the encoding is based on the ratio of probabilities of the positive to the negative class.

**Construction**

$$x(i) = \frac{y^+/N}{y_k^-/N}=\frac{y_k^+}{y_k^-}$$
This is the same as WoE encoding with no regularization, i.e. $a=0$.

**When to use**

Same as weight of evidence encoding.

```python
encoder = ce.woe.WOEEncoder(regularization=0.)
res = encode_var(df[["drive_wheels"]], encoder, df['fuel_type']=='gas')
print_res(res)
```

      original
      drive_wheels

      3
      fwd
      0.287682

      5
      fwd
      0.287682

      0
      rwd
      -0.448132

      1
      rwd
      -0.448132

      4
      4wd
      inf

      9
      4wd
      inf

# Other encodings

* DRACuLa
* Reverse Helmert
* Forward Differences
* Thermometer Encoder

# Cheat Sheet

# Exercises

**Exercise 1**: Calculate the regressions of 'num_cylinders' on price for each possible encoding. What is the correct interpretation for the coefficients?

**Exercise 2**: Try to find the best encoding for each variable to maximize the generalization performance of a linear regression model to predict the price of a car.