# Exploring Data with Python

A significant part of a a data scientist's role is to explore, analyze, and visualize data. There's a wide range of tools and programming languages that they can use to do this; and of of the most popular approaches is to use Jupyter notebooks (like this one) and Python.

Python is a flexible programming language that is used in a wide range of scenarios; from web applications to device programming. It's extremely popular in the data science and machine learning community because of the many packages it supports for data analysis and visualization.

In this notebook, we'll explore some of these packages, and apply basic techniques to analyze data. This is not intended to be a comprehensive Python programming exercise; or even a deep dive into data analysis. Rather, it's intended as a crash course in some of the common ways in which data scientists can use Python to work with data.

> **Note**: If you've never used the Jupyter Notebooks environment before, there are a few things you should be aware of:
>
> - Notebooks are made up of *cells*. Some cells (like this one) contain *markdown* text, while others (like the one beneath this one) contain code.
> - You can run each code cell by using the ▷ icon at the top left of the cell. Alternatively, you can run all of the cells in the notebook in a single command by using the ▷▷ icon on the toolbar at the top of the notebook (*this toolbar contains additional buttons - you can hold the mouse over each one to see a tooltip describing what it does*).
> - The output from each code cell will be displayed immediately below the cell.
> - Even though the code cells can be run individually, some variables used in the code are global to the notebook. That means that you should run all of the code cells **in order**. There may be dependencies between code cells, so if you skip a cell, subsequent cells might not run correctly.
> - The notebook is connected to a Python *kernel* on a Jupyter server (you can see which one at the top right of the page). If you stop the kernel or disconnect from the server (for example, by closing and reopening the notebook, or ending and resuming your session), the output from cells that have been run will still be displayed; but any variables or functions defined in those cells will have been lost - you must rerun the cells before running any subsequent cells that depend on them.

## Exploring data arrays with NumPy

Lets start by looking at some simple data.

Suppose a college takes a sample of student grades for a data science class.

Run the code in the cell below by clicking its green ▷ button (at the top left of the cell) to see the data.

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from matplotlib.axes import Axes
from matplotlib.figure import Figure
from scipy import stats
from sklearn.preprocessing import MinMaxScaler
%matplotlib inline
```

```python
data = [
    50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48,
    52, 63, 64,
]
print(data)
```

    [50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64]

The data has been loaded into a Python **list** structure, which is a good data type for general data manipulation, but not optimized for numeric analysis. For that, we're going to use the **NumPy** package, which includes specific data types and functions for working with Numbers in Python.

Run the cell below to load the data into a NumPy **array**.

```python
grades = np.array(data)
print(grades)
```

    [50 50 47 97 49  3 53 42 26 74 82 62 37 15 70 27 36 35 48 52 63 64]

Just in case you're wondering about the differences between a **list** and a NumPy **array**, let's compare how these data types behave when we use them in an expression that multiplies them by 2.

```python
print (type(data),'x 2:', data * 2)
print('---')
print (type(grades),'x 2:', grades * 2)
```

    <class 'list'> x 2: [50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64, 50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64]
    ---
    <class 'numpy.ndarray'> x 2: [100 100  94 194  98   6 106  84  52 148 164 124  74  30 140  54  72  70
      96 104 126 128]

Note that multiplying a list by 2 creates a new list of twice the length with the original sequence of list elements repeated. Multiplying a NumPy array on the other hand performs an element-wise calculation in which the array behaves like a *vector*, so we end up with an array of the same size in which each element has been multiplied by 2.

The key takeaway from this is that NumPy arrays are specifically designed to support mathematical operations on numeric data - which makes them more useful for data analysis than a generic list.

You might have spotted that the class type for the numpy array above is a **numpy.ndarray**. The **nd** indicates that this is a structure that can consists of multiple *dimensions* (it can have *n* dimensions). Our specific instance has a single dimension of student grades.

Run the cell below to view the **shape** of the array.

```python
grades.shape
```

    (22,)

The shape confirms that this array has only one dimension, which contains 22 elements (there are 22 grades in the original list). You can access the individual elements in the array by their zer0-based ordinal position. Let's get the first element (the one in position 0).

```python
grades[0]
```

    50

Alright, now you know your way around a NumPy array, it's time to perform some analysis of the grades data.

You can apply aggregations across the elements in the array, so let's find the simple average grade (in other words, the *mean* grade value).

```python
grades.mean()
```

    49.18181818181818

So the mean grade is just around 50 - more or less in the middle of the possible range from 0 to 100.

Let's add a second set of data for the same students, this time recording the typical number of hours per week they devoted to studying.

```python
study_hours = [
    10.0, 11.5, 9.0, 16.0, 9.25, 1.0, 11.5, 9.0, 8.5, 14.5, 15.5,
    13.75, 9.0, 8.0, 15.5, 8.0, 9.0, 6.0, 10.0, 12.0, 12.5, 12.0,
]

student_data = np.array([study_hours, grades])

student_data
```

    array([[10.  , 11.5 ,  9.  , 16.  ,  9.25,  1.  , 11.5 ,  9.  ,  8.5 ,
            14.5 , 15.5 , 13.75,  9.  ,  8.  , 15.5 ,  8.  ,  9.  ,  6.  ,
            10.  , 12.  , 12.5 , 12.  ],
           [50.  , 50.  , 47.  , 97.  , 49.  ,  3.  , 53.  , 42.  , 26.  ,
            74.  , 82.  , 62.  , 37.  , 15.  , 70.  , 27.  , 36.  , 35.  ,
            48.  , 52.  , 63.  , 64.  ]])

Now the data consists of a 2-dimensional array - an array of arrays. Let's look at its shape.

```python
student_data.shape
```

    (2, 22)

The **student_data** array contains two elements, each of which is an array containing 22 elements.

To navigate this structure, you need to specify the position of each element in the hierarchy. So to find the first value in the first array (which contains the study hours data), you can use the following code.

```python
student_data[0][0]
```

    10.0

Now you have a multidimensional array containing both the student's study time and grade information, which you can use to compare data. For example, how does the mean study time compare to the mean grade?

```python
avg_study = student_data[0].mean()
avg_grade = student_data[1].mean()

print(f'Average study hours: {avg_study:.2f}')
print(f'Average grade: {avg_grade:.2f}')
```

    Average study hours: 10.52
    Average grade: 49.18

## Exploring tabular data with Pandas

While NumPy provides a lot of the functionality you need to work with numbers, and specifically arrays of numeric values; when you start to deal with two-dimensional tables of data, the **Pandas** package offers a more convenient structure to work with - the **DataFrame**.

Run the following cell to import the Pandas library and create a DataFrame with three columns. The first column is a list of student names, and the second and third columns are the NumPy arrays containing the study time and grade data.

```python
df_students = pd.DataFrame({
    'Name': [
        'Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic',
        'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana',
        'Kian', 'Jenny', 'Jakeem', 'Helena', 'Ismat', 'Anila', 'Skye',
        'Daniel', 'Aisha',
    ],
    'StudyHours': student_data[0],
    'Grade': student_data[1]
})

df_students
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.00
      50.0

      1
      Joann
      11.50
      50.0

      2
      Pedro
      9.00
      47.0

      3
      Rosie
      16.00
      97.0

      4
      Ethan
      9.25
      49.0

      5
      Vicky
      1.00
      3.0

      6
      Frederic
      11.50
      53.0

      7
      Jimmie
      9.00
      42.0

      8
      Rhonda
      8.50
      26.0

      9
      Giovanni
      14.50
      74.0

      10
      Francesca
      15.50
      82.0

      11
      Rajab
      13.75
      62.0

      12
      Naiyana
      9.00
      37.0

      13
      Kian
      8.00
      15.0

      14
      Jenny
      15.50
      70.0

      15
      Jakeem
      8.00
      27.0

      16
      Helena
      9.00
      36.0

      17
      Ismat
      6.00
      35.0

      18
      Anila
      10.00
      48.0

      19
      Skye
      12.00
      52.0

      20
      Daniel
      12.50
      63.0

      21
      Aisha
      12.00
      64.0

Note that in addition to the columns you specified, the DataFrame includes an *index* to unique identify each row. We could have specified the index explicitly, and assigned any kind of appropriate value (for example, an email address); but because we didn't specify an index, one has been created with a unique integer value for each row.

### Finding and filtering data in a DataFrame

You can use the DataFrame's **loc** method to retrieve data for a specific index value, like this.

```python
df_students.loc[5]
```

    Name          Vicky
    StudyHours        1
    Grade             3
    Name: 5, dtype: object

You can also get the data at a range of index values, like this:

```python
df_students.loc[0:5]
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.00
      50.0

      1
      Joann
      11.50
      50.0

      2
      Pedro
      9.00
      47.0

      3
      Rosie
      16.00
      97.0

      4
      Ethan
      9.25
      49.0

      5
      Vicky
      1.00
      3.0

In addition to being able to use the **loc** method to find rows based on the index, you can use the **iloc** method to find rows based on their ordinal position in the DataFrame (regardless of the index):

```python
df_students.iloc[0:5]
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.00
      50.0

      1
      Joann
      11.50
      50.0

      2
      Pedro
      9.00
      47.0

      3
      Rosie
      16.00
      97.0

      4
      Ethan
      9.25
      49.0

Look carefully at the `iloc[0:5]` results, and compare them to the `loc[0:5]` results you obtained previously. Can you spot the difference?

The **loc** method returned rows with index *label* in the list of values from *0* to *5* - which includes *0*, *1*, *2*, *3*, *4*, and *5* (six rows). However, the **iloc** method returns the rows in the *positions* included in the range 0 to 5, and since integer ranges don't include the upper-bound value, this includes positions *0*, *1*, *2*, *3*, and *4* (five rows).

**iloc** identities data values in a DataFrame by *position*, which extends beyond rows to columns. So for example, you can use it to find the values for the columns in positions 1 and 2 in row 0, like this:

```python
df_students.iloc[0,[1,2]]
```

    StudyHours    10
    Grade         50
    Name: 0, dtype: object

Let's return to the **loc** method, and see how it works with columns. Remember that **loc** is used to locate data items based on index values rather than positions. In the absence of an explicit index column, the rows in our dataframe are indexed as integer values, but the columns are identified by name:

```python
df_students.loc[0,'Grade']
```

    50.0

Here's another useful trick. You can use the **loc** method to find indexed rows based on a filtering expression that references named columns other than the index, like this:

```python
df_students.loc[df_students['Name'] == 'Aisha']
```

      Name
      StudyHours
      Grade

      21
      Aisha
      12.0
      64.0

Actually, you don't need to explicitly use the **loc** method to do this - you can simply apply a DataFrame filtering expression, like this:

```python
df_students[df_students['Name'] == 'Aisha']
```

      Name
      StudyHours
      Grade

      21
      Aisha
      12.0
      64.0

And for good measure, you can achieve the same results by using the DataFrame's **query** method, like this:

```python
df_students.query('Name == "Aisha"')
```

      Name
      StudyHours
      Grade

      21
      Aisha
      12.0
      64.0

The three previous examples underline an occasionally confusing truth about working with Pandas. Often, there are multiple ways to achieve the same results. Another example of this is the way you refer to a DataFrame column name. You can specify the column name as a named index value (as in the `df_students['Name']` examples we've seen so far), or you can use the column as a property of the DataFrame, like this:

```python
df_students[df_students.Name == 'Aisha']
```

      Name
      StudyHours
      Grade

      21
      Aisha
      12.0
      64.0

### Handling missing values

One of the most common issues data scientists need to deal with is incomplete or missing data.

For example, suppose we add rows for two more students to our dataset.

```python
new_rows = [
    {'Name':'Bill', 'StudyHours':8},
    {'Name':'Ted',},
]
df_students = df_students.append(new_rows, ignore_index=True)
df_students
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.00
      50.0

      1
      Joann
      11.50
      50.0

      2
      Pedro
      9.00
      47.0

      3
      Rosie
      16.00
      97.0

      4
      Ethan
      9.25
      49.0

      5
      Vicky
      1.00
      3.0

      6
      Frederic
      11.50
      53.0

      7
      Jimmie
      9.00
      42.0

      8
      Rhonda
      8.50
      26.0

      9
      Giovanni
      14.50
      74.0

      10
      Francesca
      15.50
      82.0

      11
      Rajab
      13.75
      62.0

      12
      Naiyana
      9.00
      37.0

      13
      Kian
      8.00
      15.0

      14
      Jenny
      15.50
      70.0

      15
      Jakeem
      8.00
      27.0

      16
      Helena
      9.00
      36.0

      17
      Ismat
      6.00
      35.0

      18
      Anila
      10.00
      48.0

      19
      Skye
      12.00
      52.0

      20
      Daniel
      12.50
      63.0

      21
      Aisha
      12.00
      64.0

      22
      Bill
      8.00
      NaN

      23
      Ted
      NaN
      NaN

Note that the new student records don't include values for all columns. When the DataFrame is retrieved, the missing values show up as **NaN** (*not a number*). These values are used to indicate missing numeric data.

So how would we know that the DataFrame contains missing values? You can use the **isnull** method to identify which individual values are null, like this:

```python
df_students.isnull()
```

      Name
      StudyHours
      Grade

      0
      False
      False
      False

      1
      False
      False
      False

      2
      False
      False
      False

      3
      False
      False
      False

      4
      False
      False
      False

      5
      False
      False
      False

      6
      False
      False
      False

      7
      False
      False
      False

      8
      False
      False
      False

      9
      False
      False
      False

      10
      False
      False
      False

      11
      False
      False
      False

      12
      False
      False
      False

      13
      False
      False
      False

      14
      False
      False
      False

      15
      False
      False
      False

      16
      False
      False
      False

      17
      False
      False
      False

      18
      False
      False
      False

      19
      False
      False
      False

      20
      False
      False
      False

      21
      False
      False
      False

      22
      False
      False
      True

      23
      False
      True
      True

Of course, with a larger DataFrame, it would be inefficient to review all of the rows and columns individually; so we can get the sum of missing values for each column, like this:

```python
df_students.isnull().sum()
```

    Name          0
    StudyHours    1
    Grade         2
    dtype: int64

So now we know that there's one missing **StudyHours** value, and two missing **Grade** values.

To see them in context, we can filter the dataframe to include only rows where any of the columns (axis 1 of the DataFrame) are null.

```python
df_students[df_students.isnull().any(axis=1)]
```

      Name
      StudyHours
      Grade

      22
      Bill
      8.0
      NaN

      23
      Ted
      NaN
      NaN

So now that we've found the null values, what can we do about them?

One common approach is to *impute* replacement values. For example, if the number of study hours is missing, we could just assume that the student studied for an average amount of time and replace the missing value with the mean study hours. To do this, we can use the **fillna** method, like this:

```python
df_students['StudyHours'] = (
    df_students['StudyHours'].fillna(df_students['StudyHours'].mean())
)
df_students
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.000000
      50.0

      1
      Joann
      11.500000
      50.0

      2
      Pedro
      9.000000
      47.0

      3
      Rosie
      16.000000
      97.0

      4
      Ethan
      9.250000
      49.0

      5
      Vicky
      1.000000
      3.0

      6
      Frederic
      11.500000
      53.0

      7
      Jimmie
      9.000000
      42.0

      8
      Rhonda
      8.500000
      26.0

      9
      Giovanni
      14.500000
      74.0

      10
      Francesca
      15.500000
      82.0

      11
      Rajab
      13.750000
      62.0

      12
      Naiyana
      9.000000
      37.0

      13
      Kian
      8.000000
      15.0

      14
      Jenny
      15.500000
      70.0

      15
      Jakeem
      8.000000
      27.0

      16
      Helena
      9.000000
      36.0

      17
      Ismat
      6.000000
      35.0

      18
      Anila
      10.000000
      48.0

      19
      Skye
      12.000000
      52.0

      20
      Daniel
      12.500000
      63.0

      21
      Aisha
      12.000000
      64.0

      22
      Bill
      8.000000
      NaN

      23
      Ted
      10.413043
      NaN

Alternatively, it might be important to ensure that you only use data you know to be absolutely correct; so you can drop rows or columns that contains null values by using the **dropna** method. In this case, we'll remove rows (axis 0 of the DataFrame) where any of the columns contain null values.

```python
df_students = df_students.dropna(axis=0, how='any')
df_students
```

      Name
      StudyHours
      Grade

      0
      Dan
      10.00
      50.0

      1
      Joann
      11.50
      50.0

      2
      Pedro
      9.00
      47.0

      3
      Rosie
      16.00
      97.0

      4
      Ethan
      9.25
      49.0

      5
      Vicky
      1.00
      3.0

      6
      Frederic
      11.50
      53.0

      7
      Jimmie
      9.00
      42.0

      8
      Rhonda
      8.50
      26.0

      9
      Giovanni
      14.50
      74.0

      10
      Francesca
      15.50
      82.0

      11
      Rajab
      13.75
      62.0

      12
      Naiyana
      9.00
      37.0

      13
      Kian
      8.00
      15.0

      14
      Jenny
      15.50
      70.0

      15
      Jakeem
      8.00
      27.0

      16
      Helena
      9.00
      36.0

      17
      Ismat
      6.00
      35.0

      18
      Anila
      10.00
      48.0

      19
      Skye
      12.00
      52.0

      20
      Daniel
      12.50
      63.0

      21
      Aisha
      12.00
      64.0

### Explore data in the DataFrame

Now that we've cleaned up the missing values, we're ready to explore the data in the DataFrame. Let's start by comparing the mean study hours and grades.

```python
mean_study = df_students['StudyHours'].mean()

mean_grade = df_students['Grade'].mean()

print(f'Average weekly study hours: {mean_study:.2f}')
print(f'Average grade: {mean_grade:.2f}')
```

    Average weekly study hours: 10.52
    Average grade: 49.18

OK, let's filter the DataFrame to find only the students who studied for more than the average amount of time.

```python
df_students[df_students['StudyHours'] > mean_study]
```

      Name
      StudyHours
      Grade

      1
      Joann
      11.50
      50.0

      3
      Rosie
      16.00
      97.0

      6
      Frederic
      11.50
      53.0

      9
      Giovanni
      14.50
      74.0

      10
      Francesca
      15.50
      82.0

      11
      Rajab
      13.75
      62.0

      14
      Jenny
      15.50
      70.0

      19
      Skye
      12.00
      52.0

      20
      Daniel
      12.50
      63.0

      21
      Aisha
      12.00
      64.0

Note that the filtered result is itself a DataFrame, so you can work with its columns just like any other DataFrame.

For example, let's find the average grade for students who undertook more than the average amount of study time.

```python
df_students[df_students['StudyHours'] > mean_study]['Grade'].mean()
```

    66.7

Let's assume that the passing grade for the course is 60.

We can use that information to add a new column to the DataFrame, indicating whether or not each student passed.

```python
df_students['Pass'] = df_students['Grade'] >= 60

df_students
```

    /Users/susumuasaga/ml-basics/conda-env/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      """Entry point for launching an IPython kernel.

      Name
      StudyHours
      Grade
      Pass

      0
      Dan
      10.00
      50.0
      False

      1
      Joann
      11.50
      50.0
      False

      2
      Pedro
      9.00
      47.0
      False

      3
      Rosie
      16.00
      97.0
      True

      4
      Ethan
      9.25
      49.0
      False

      5
      Vicky
      1.00
      3.0
      False

      6
      Frederic
      11.50
      53.0
      False

      7
      Jimmie
      9.00
      42.0
      False

      8
      Rhonda
      8.50
      26.0
      False

      9
      Giovanni
      14.50
      74.0
      True

      10
      Francesca
      15.50
      82.0
      True

      11
      Rajab
      13.75
      62.0
      True

      12
      Naiyana
      9.00
      37.0
      False

      13
      Kian
      8.00
      15.0
      False

      14
      Jenny
      15.50
      70.0
      True

      15
      Jakeem
      8.00
      27.0
      False

      16
      Helena
      9.00
      36.0
      False

      17
      Ismat
      6.00
      35.0
      False

      18
      Anila
      10.00
      48.0
      False

      19
      Skye
      12.00
      52.0
      False

      20
      Daniel
      12.50
      63.0
      True

      21
      Aisha
      12.00
      64.0
      True

DataFrames are amazingly versatile, and make it easy to manipulate data. Most DataFrame operations return a new copy of the DataFrame; so if you want to modify a DataFrame but keep the existing variable, you need to assign the result of the operation to the existing variable. For example, the following code sorts the student data into descending order of Grade, and assigns the resulting sorted DataFrame to the original **df_students** variable.

```python
df_students = df_students.sort_values('Grade', ascending=False)

df_students
```

      Name
      StudyHours
      Grade
      Pass

      3
      Rosie
      16.00
      97.0
      True

      10
      Francesca
      15.50
      82.0
      True

      9
      Giovanni
      14.50
      74.0
      True

      14
      Jenny
      15.50
      70.0
      True

      21
      Aisha
      12.00
      64.0
      True

      20
      Daniel
      12.50
      63.0
      True

      11
      Rajab
      13.75
      62.0
      True

      6
      Frederic
      11.50
      53.0
      False

      19
      Skye
      12.00
      52.0
      False

      1
      Joann
      11.50
      50.0
      False

      0
      Dan
      10.00
      50.0
      False

      4
      Ethan
      9.25
      49.0
      False

      18
      Anila
      10.00
      48.0
      False

      2
      Pedro
      9.00
      47.0
      False

      7
      Jimmie
      9.00
      42.0
      False

      12
      Naiyana
      9.00
      37.0
      False

      16
      Helena
      9.00
      36.0
      False

      17
      Ismat
      6.00
      35.0
      False

      15
      Jakeem
      8.00
      27.0
      False

      8
      Rhonda
      8.50
      26.0
      False

      13
      Kian
      8.00
      15.0
      False

      5
      Vicky
      1.00
      3.0
      False

## Visualizing data with Matplotlib

DataFrames provide a great way to explore and analyze tabular data, but sometimes a picture is worth a thousand rows and columns. The **Matplotlib** library provides the foundation for plotting data visualizations that can greatly enhance your ability the analyze the data.

Let's start with a simple bar chart that shows the grade of each student.

```python
plt.bar(df_students['Name'], df_students['Grade'])

plt.show()
```

Well, that worked; but the chart could use some improvements to make it clearer what we're looking at.

Note that you used the **pyplot** class from Matplotlib to plot the chart. This class provides a whole bunch of ways to improve the visual elements of the plot. For example, the following code:

- Specifies the color of the bar chart.
- Adds a title to the chart (so we know what it represents)
- Adds labels to the X and Y (so we know which axis shows which data)
- Adds a grid (to make it easier to determine the values for the bars)
- Rotates the X markers (so we can read them)

```python
plt.bar(df_students['Name'], df_students['Grade'], color='orange')

plt.title('Student Grades')
plt.xlabel('Student')
plt.ylabel('Grade')
plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)
plt.xticks(rotation=90)

plt.show()
```

A plot is technically contained with a **Figure**. In the previous examples, the figure was created implicitly for you; but you can create it explicitly. For example, the following code creates a figure with a specific size.

```python
plt.figure(figsize=(8,3))

plt.bar(df_students['Name'], df_students['Grade'], color='orange')

plt.title('Student Grades')
plt.xlabel('Student')
plt.ylabel('Grade')
plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)
plt.xticks(rotation=90)

plt.show()
```

A figure can contain multiple subplots, each on its own *axis*.

For example, the following code creates a figure with two subplots - one is a bar chart showing student grades, and the other is a pie chart comparing the number of passing grades to non-passing grades.

```python
fig: Figure
ax1: Axes
ax2: Axes
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10,4))

ax1.bar(df_students['Name'], df_students['Grade'], color='orange')
ax1.set_title('Grades')
ax1.set_xticklabels(df_students.Name, rotation=90)

pass_counts = df_students['Pass'].value_counts()
ax2.pie(pass_counts, labels=pass_counts)
ax2.set_title('Passing Grades')
ax2.legend(pass_counts.keys().tolist())

fig.suptitle('Student Data')

fig.show()
```

Until now, you've used methods of the Matplotlib.pyplot object to plot charts. However, Matplotlib is so foundational to graphics in Python that many packages, including Pandas, provide methods that abstract the underlying Matplotlib functions and simplify plotting. For example, the DataFrame provides its own methods for plotting data, as shown in the following example to plot a bar chart of study hours.

```python
df_students.plot.bar('Name', 'StudyHours', color='teal', figsize=(6,4))
```

## Getting started with statistical analysis

Now that you know how to use Python to manipulate and visualize data, you can start analyzing it.

A lot of data science is rooted in *statistics*, so we'll explore some basic statistical techniques.

> **Note**: This is not intended to teach you statistics - that's much too big a topic for this notebook. It will however introduce you to some statistical concepts and techniques that data scientists use as they explore data in preparation for machine learning modeling.

### Descriptive statistics and data distribution

When examining a *variable* (for example a sample of student grades), data scientists are particularly interested in its *distribution* (in other words, how are all the different grade values spread across the sample). The starting point for this exploration is often to visualize the data as a histogram, and see how frequently each value for the variable occurs.

```python
var_data = df_students['Grade']

fig = plt.figure(figsize=(10,4))

plt.hist(var_data)

plt.title('Data Distribution')
plt.xlabel('Value')
plt.ylabel('Frequency')

fig.show()
```

The histogram for grades is a symmetric shape, where the most frequently occurring grades tend to be in the middle of the range (around 50), with fewer grades at the extreme ends of the scale.

#### Measures of central tendency

To understand the distribution better, we can examine so-called *measures of central tendency*; which is a fancy way of describing statistics that represent the "middle" of the data. The goal of this is to try to find a "typical" value. Common ways to define the middle of the data include:

- The *mean*: A simple average based on adding together all of the values in the sample set, and then dividing the total by the number of samples.
- The *median*: The value in the middle of the range of all of the sample values.
- The *mode*: The most commonly occurring value in the sample set\*.

Let's calculate these values, along with the minimum and maximum values for comparison, and show them on the histogram.

> \*Of course, in some sample sets , there may be a tie for the most common value - in which case the dataset is described as *bimodal* or even *multimodal*.

```python
var = df_students['Grade']

min_val = var.min()
max_val = var.max()
mean_val = var.mean()
med_val = var.median()
mod_val = var.mode()[0]

print(f'Minimum:{min_val:.2f}')
print(f'Mean:{mean_val:.2f}')
print(f'Median:{med_val:.2f}')
print(f'Mode:{mod_val:.2f}')
print(f'Maximum:{max_val:.2f}')

fig = plt.figure(figsize=(10,4))

plt.hist(var)

plt.axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)
plt.axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)
plt.axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)
plt.axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)
plt.axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)

plt.title('Data Distribution')
plt.xlabel('Value')
plt.ylabel('Frequency')

fig.show()
```

For the grade data, the mean, median, and mode all seem to be more or less in the middle of the minimum and maximum, at around 50.

Another way to visualize the distribution of a variable is to use a *box* plot (sometimes called a *box-and-whiskers* plot). Let's create one for the grade data.

```python
var = df_students['Grade']

fig = plt.figure(figsize=(10,4))

plt.boxplot(var)

plt.title('Data Distribution')

fig.show()
```

The box plot shows the distribution of the grade values in a different format to the histogram. The *box* part of the blot shows where the inner two *quartiles* of the data reside - so in this case, half of the grades are between approximately 36 and 63. The *whiskers* extending from the box show the outer two quartiles; so the other half of the grades in this case are between 0 and 36 or 63 and 100. The line in the box indicates the *median* value.

It's often useful to combine histograms and box plots, with the box plot's orientation changed to align it with the histogram (in some ways, it can be helpful to think of the histogram as a "front elevation" view of the distribution, and the box plot as a "plan" view of the distribution from above.)

```python
def show_distribution(var_data):
    """
    Show histogram and box plot combination of `var_data`.
    """

    min_val = var_data.min()
    max_val = var_data.max()
    mean_val = var_data.mean()
    med_val = var_data.median()
    mod_val = var_data.mode()[0]

    print(f'Minimum:{min_val:.2f}')
    print(f'Mean:{mean_val:.2f}')
    print(f'Median:{med_val:.2f}')
    print(f'Mode:{mod_val:.2f}')
    print(f'Maximum:{max_val:.2f}')

    fig: Figure
    ax1: Axes
    ax2: Axes
    fig, (ax1, ax2) = plt.subplots(2, 1, figsize = (10,4))

    ax1.hist(var_data)
    ax1.set_ylabel('Frequency')

    ax1.axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)
    ax1.axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)
    ax1.axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)
    ax1.axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)
    ax1.axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)

    ax2.boxplot(var_data, vert=False)
    ax2.set_xlabel('Value')

    fig.suptitle('Data Distribution')

    fig.show()
```

```python
col = df_students['Grade']
show_distribution(col)
```

All of the measurements of central tendency are right in the middle of the data distribution, which is symmetric with values becoming progressively lower in both directions from the middle.

To explore this distribution in more detail, you need to understand that statistics is fundamentally about taking *samples* of data and using probability functions to extrapolate information about the full *population* of data. For example, the student data consists of 22 samples, and for each sample there is a grade value. You can think of each sample grade as a variable that's been randomly selected from the set of all grades awarded for this course. With enough of these random variables, you can calculate something called a *probability density function*, which estimates the distribution of grades for the full population.

The Pandas DataFrame class provides a helpful plot function to show this density.

```python
def show_density(var_data):
    """
    Show kernel density estimation plot of `var_data`, using Gaussian kernels.
    """

    plt.figure(figsize=(10,4))

    var_data.plot.density()

    plt.title('Data Density')

    plt.axvline(
        var_data.mean(), color='cyan', linestyle='dashed', linewidth = 2
    )
    plt.axvline(
        var_data.median(), color='red', linestyle='dashed', linewidth=2
    )
    plt.axvline(
        var_data.mode()[0], color='yellow', linestyle='dashed', linewidth = 2
    )

    plt.show()
```

```python
col = df_students['Grade']
show_density(col)
```

As expected from the histogram of the sample, the density shows the characteristic 'bell curve" of what statisticians call a *normal* distribution with the mean and mode at the center and symmetric tails.

Now let's take a look at the distribution of the study hours data.

```python
col = df_students['StudyHours']
show_distribution(col)
```

The distribution of the study time data is significantly different from that of the grades.

Note that the whiskers of the box plot only extend to around 6.0, indicating that the vast majority of the first quarter of the data is above this value. The minimum is marked with an **o**, indicating that it is statistically an *outlier* - a value that lies significantly outside the range of the rest of the distribution.

Outliers can occur for many reasons. Maybe a student meant to record "10" hours of study time, but entered "1" and missed the "0". Or maybe the student was abnormally lazy when it comes to studying! Either way, it's a statistical anomaly that doesn't represent a typical student. Let's see what the distribution looks like without it.

```python
col = df_students[df_students['StudyHours'] > 1]['StudyHours']
show_distribution(col)
```

With the outliers removed, the box plot shows all data within the four quartiles. Note that the distribution is not symmetric like it is for the grade data though - there are some students with very high study times of around 16 hours, but the bulk of the data is between 7 and 13 hours; The few extremely high values pull the mean towards the higher end of the scale.

Let's look at the density for this distribution.

```python
show_density(col)
```

This kind of distribution is called *right skewed*. The mass of the data is on the left side of the distribution, creating a long tail to the right because of the values at the extreme high end; which pull the mean to the right.

#### Measures of variance

So now we have a good idea where the middle of the grade and study hours data distributions are. However, there's another aspect of the distributions we should examine: how much variability is there in the data?

Typical statistics that measure variability in the data include:

- **Range**: The difference between the maximum and minimum. There's no built-in function for this, but it's easy to calculate using the **min** and **max** functions.
- **Variance**: The average of the squared difference from the mean. You can use the built-in **var** function to find this.
- **Standard Deviation**: The square root of the variance. You can use the built-in **std** function to find this.

```python
for col_name in ['Grade','StudyHours']:
    col = df_students[col_name]
    rng = col.max() - col.min()
    var = col.var()
    std = col.std()
    print(f'{col_name}:')
    print(f' - Range: {rng:.2f}')
    print(f' - Variance: {var:.2f}')
    print(f' - Std.Dev: {std:.2f}')
```

    Grade:
     - Range: 94.00
     - Variance: 472.54
     - Std.Dev: 21.74
    StudyHours:
     - Range: 15.00
     - Variance: 12.16
     - Std.Dev: 3.49

Of these statistics, the standard deviation is generally the most useful. It provides a measure of variance in the data on the same scale as the data itself (so grade points for the Grade distribution and hours for the StudyHours distribution). The higher the standard deviation, the more variance there is when comparing values in the distribution to the distribution mean - in other words, the data is more spread out.

When working with a *normal* distribution, the standard deviation works with the particular characteristics of a normal distribution to provide even greater insight. Run the cell below to see the relationship between standard deviations and the data in the normal distribution.

```python
col = df_students['Grade']

density = stats.gaussian_kde(col)

col.plot.density()

s = col.std()
m = col.mean()

x1 = [m - s, m + s]
y1 = density(x1)
plt.plot(x1, y1, color='magenta')
plt.annotate('1 std (68.26%)', (x1[1],y1[1]))

x2 = [m - s*2, m + s*2]
y2 = density(x2)
plt.plot(x2,y2, color='green')
plt.annotate('2 std (95.45%)', (x2[1],y2[1]))

x3 = [m - s*3, m + s*3]
y3 = density(x3)
plt.plot(x3,y3, color='orange')
plt.annotate('3 std (99.73%)', (x3[1],y3[1]))

plt.axvline(col.mean(), color='cyan', linestyle='dashed', linewidth=1)

plt.axis('off')

plt.show()
```

The horizontal lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

So, since we know that the mean grade is 49.8, the standard deviation is 21.47, and distribution of grades is approximately normal; we can calculate that 68.26% of students should achieve a grade between 28.33 and 71.27.

The descriptive statistics we've used to understand the distribution of the student data variables are the basis of statistical analysis; and because they're such an important part of exploring your data, there's a built-in **Describe** method of the DataFrame object that returns the main descriptive statistics for all numeric columns.

```python
df_students.describe()
```

      StudyHours
      Grade

      count
      22.000000
      22.000000

      mean
      10.522727
      49.181818

      std
      3.487144
      21.737912

      min
      1.000000
      3.000000

      25%
      9.000000
      36.250000

      50%
      10.000000
      49.500000

      75%
      12.375000
      62.750000

      max
      16.000000
      97.000000

## Comparing data

Now that you know something about the statistical distribution of the data in your dataset, you're ready to examine your data to identify any apparent relationships between variables.

First of all, let's get rid of any rows that contain outliers so that we have a sample that is representative of a typical class of students. We identified that the StudyHours column contains some outliers with extremely low values, so we'll remove those rows.

```python
df_sample = df_students[df_students['StudyHours'] > 1]
df_sample
```

      Name
      StudyHours
      Grade
      Pass

      3
      Rosie
      16.00
      97.0
      True

      10
      Francesca
      15.50
      82.0
      True

      9
      Giovanni
      14.50
      74.0
      True

      14
      Jenny
      15.50
      70.0
      True

      21
      Aisha
      12.00
      64.0
      True

      20
      Daniel
      12.50
      63.0
      True

      11
      Rajab
      13.75
      62.0
      True

      6
      Frederic
      11.50
      53.0
      False

      19
      Skye
      12.00
      52.0
      False

      1
      Joann
      11.50
      50.0
      False

      0
      Dan
      10.00
      50.0
      False

      4
      Ethan
      9.25
      49.0
      False

      18
      Anila
      10.00
      48.0
      False

      2
      Pedro
      9.00
      47.0
      False

      7
      Jimmie
      9.00
      42.0
      False

      12
      Naiyana
      9.00
      37.0
      False

      16
      Helena
      9.00
      36.0
      False

      17
      Ismat
      6.00
      35.0
      False

      15
      Jakeem
      8.00
      27.0
      False

      8
      Rhonda
      8.50
      26.0
      False

      13
      Kian
      8.00
      15.0
      False

### Comparing numeric and categorical variables

The data includes two *numeric* variables (**StudyHours** and **Grade**) and two *categorical* variables (**Name** and **Pass**). Let's start by comparing the numeric **StudyHours** column to the categorical **Pass** column to see if there's an apparent relationship between the number of hours studied and a passing grade.

To make this comparison, let's create box plots showing the distribution of StudyHours for each possible Pass value (true and false).

```python
df_sample.boxplot('StudyHours', 'Pass', figsize=(8,5))
```

Comparing the StudyHours distributions, it's immediately apparent (if not particularly surprising) that students who passed the course tended to study for more hours than students who didn't. So if you wanted to predict whether or not a student is likely to pass the course, the amount of time they spend studying may be a good predictive feature.

### Comparing numeric variables

Now let's compare two numeric variables. We'll start by creating a bar chart that shows both grade and study hours.

```python
df_sample.plot.bar('Name', ['Grade','StudyHours'], figsize=(8,5))
```

The chart shows bars for both grade and study hours for each student; but it's not easy to compare because the values are on different scales. Grades are measured in grade points, and range from 3 to 97; while study time is measured in hours and ranges from 1 to 16.

A common technique when dealing with numeric data in different scales is to *normalize* the data so that the values retain their proportional distribution, but are measured on the same scale. To accomplish this, we'll use a technique called *MinMax* scaling that distributes the values proportionally on a scale of 0 to 1. You could write the code to apply this transformation; but the **Scikit-Learn** library provides a scaler to do it for you.

```python
scaler = MinMaxScaler()

df_normalized = df_sample[['Name', 'Grade', 'StudyHours']].copy()

df_normalized[['Grade','StudyHours']] = scaler.fit_transform(
    df_normalized[['Grade','StudyHours']]
)

df_normalized.plot.bar('Name', ['Grade','StudyHours'], figsize=(8,5))
```

With the data normalized, it's easier to see an apparent relationship between grade and study time. It's not an exact match, but it definitely seems like students with higher grades tend to have studied more.

So there seems to be a correlation between study time and grade; and in fact, there's a statistical *correlation* measurement we can use to quantify the relationship between these columns.

```python
df_normalized['Grade'].corr(df_normalized['StudyHours'])
```

    0.9117666413789675

The correlation statistic is a value between -1 and 1 that indicates the strength of a relationship. Values above 0 indicate a *positive* correlation (high values of one variable tend to coincide with high values of the other), while values below 0 indicate a *negative* correlation (high values of one variable tend to coincide with low values of the other). In this case, the correlation value is close to 1; showing a strongly positive correlation between study time and grade.

> **Note**: Data scientists often quote the maxim "*correlation* is not *causation*". In other words, as tempting as it might be, you shouldn't interpret the statistical correlation as explaining *why* one of the values is high. In the case of the student data, the statistics demonstrates that students with high grades tend to also have high amounts of study time; but this is not the same as proving that they achieved high grades *because* they studied a lot. The statistic could equally be used as evidence to support the nonsensical conclusion that the students studied a lot *because* their grades were going to be high.

Another way to visualise the apparent correlation between two numeric columns is to use a *scatter* plot.

```python
df_sample.plot.scatter('StudyHours', 'Grade', title='Study Time vs Grade')
```

Again, it looks like there's a discernible pattern in which the students who studied the most hours are also the students who got the highest grades.

We can see this more clearly by adding a *regression* line (or a *line of best fit*) to the plot that shows the general trend in the data. To do this, we'll use a statistical technique called *least squares regression*.

> **Warning - Math Ahead!**
>
> Cast your mind back to when you were learning how to solve linear equations in school, and recall that the *slope-intercept* form of a linear equation looks like this:
>
> \begin{equation}y = mx + b\end{equation}
>
> In this equation, $y$ and $x$ are the coordinate variables, $m$ is the slope of the line, and $b$ is the y-intercept (where the line goes through the Y axis).
>
> In the case of our scatter plot for our student data, we already have our values for $x$ (StudyHours) and $y$ (Grade), so we just need to calculate the intercept and slope of the straight line that lies closest to those points. Then we can form a linear equation that calculates a new $y$ value on that line for each of our $x$ (StudyHours) values - to avoid confusion, we'll call this new $y$ value $f(x)$ (because it's the output from a linear equation function based on $x$). The difference between the original $y$ (Grade) value and the $f(x)$ value is the *error* between our regression line and the actual Grade achieved by the student. Our goal is to calculate the slope and intercept for a line with the lowest overall error.
>
> Specifically, we define the overall error by taking the error for each point, squaring it, and adding all the squared errors together. The line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.

Fortunately, you don't need to code the regression calculation yourself - the **SciPy** package includes a **stats** class that provides a **linregress** method to do the hard work for you. This returns (among other things) the coefficients you need for the slope equation - slope ($m$) and intercept ($b$) based on a given pair of variable samples you want to compare.

```python
df_regression = df_sample[['Grade', 'StudyHours']].copy()

m, b, r, p, se = stats.linregress(
    df_regression['StudyHours'], df_regression['Grade']
)
print(f'slope: {m:.4f}')
print(f'y-intercept: {b:.4f}')
print('so...')
print(f' f(x) = {m:.4f}x + {b:.4f}')

df_regression['fx'] = m * df_regression['StudyHours'] + b

df_regression['error'] = df_regression['Grade'] - df_regression['fx']

df_regression.plot.scatter('StudyHours', 'Grade')

plt.plot(df_regression['StudyHours'], df_regression['fx'], color='gray')

plt.show()
```

Note that this time, the code plotted two distinct things - the scatter plot of the sample study hours and grades is plotted as before, and then a line of best fit based on the least squares regression coefficients is plotted.

The slope and intercept coefficients calculated for the regression line are shown above the plot.

The line is based on the **$f(x)$** values calculated for each **StudyHours** value. Run the following cell to see a table that includes the following values:

- The **StudyHours** for each student.
- The **Grade** achieved by each student.
- The **$f(x)$** value calculated using the regression line coefficients.
- The *error* between the calculated **$f(x)$** value and the actual **Grade** value.

Some of the errors, particularly at the extreme ends, and quite large (up to over 17.5 grade points); but in general, the line is pretty close to the actual grades.

```python
df_regression[['StudyHours', 'Grade', 'fx', 'error']]
```

      StudyHours
      Grade
      fx
      error

      3
      16.00
      97.0
      83.098400
      13.901600

      10
      15.50
      82.0
      79.941687
      2.058313

      9
      14.50
      74.0
      73.628262
      0.371738

      14
      15.50
      70.0
      79.941687
      -9.941687

      21
      12.00
      64.0
      57.844698
      6.155302

      20
      12.50
      63.0
      61.001410
      1.998590

      11
      13.75
      62.0
      68.893193
      -6.893193

      6
      11.50
      53.0
      54.687985
      -1.687985

      19
      12.00
      52.0
      57.844698
      -5.844698

      1
      11.50
      50.0
      54.687985
      -4.687985

      0
      10.00
      50.0
      45.217846
      4.782154

      4
      9.25
      49.0
      40.482777
      8.517223

      18
      10.00
      48.0
      45.217846
      2.782154

      2
      9.00
      47.0
      38.904421
      8.095579

      7
      9.00
      42.0
      38.904421
      3.095579

      12
      9.00
      37.0
      38.904421
      -1.904421

      16
      9.00
      36.0
      38.904421
      -2.904421

      17
      6.00
      35.0
      19.964144
      15.035856

      15
      8.00
      27.0
      32.590995
      -5.590995

      8
      8.50
      26.0
      35.747708
      -9.747708

      13
      8.00
      15.0
      32.590995
      -17.590995

### Using the regression coefficients for prediction

Now that you have the regression coefficients for the study time and grade relationship, you can use them in a function to estimate the expected grade for a given amount of study.

```python
def f(x):
    """
    f(x) = 6.31 * x - 17.9
    """
    m = 6.31
    b = -17.9
    return m*x + b

study_time = 14

prediction = f(study_time)

expected_grade = max(0,min(100,prediction))

print (
    f'Studying for {study_time} hours per week '
    f'may result in a grade of {expected_grade:.0f}'
)
```

    Studying for 14 hours per week may result in a grade of 70

So by applying statistics to sample data, you've determined a relationship between study time and grade; and encapsulated that relationship in a general function that can be used to predict a grade for a given amount of study time.

This technique is in fact the basic premise of machine learning. You can take a set of sample data that includes one or more *features* (in this case, the number of hours studied) and a known *label* value (in this case, the grade achieved) and use the sample data to derive a function that calculates predicted label values for any given set of features.

## Further Reading

To learn more about the Python packages you explored in this notebook, see the following documentation:

- [NumPy](https://numpy.org/doc/stable/)
- [Pandas](https://pandas.pydata.org/pandas-docs/stable/)
- [Matplotlib](https://matplotlib.org/contents.html)


# Introduction #

In this exercise, you'll work on building some intuition around feature extraction. First, we'll walk through the example we did in the tutorial again, but this time, with a kernel you choose yourself. We've mostly been working with images in this course, but what's behind all of the operations we're learning about is mathematics. So, we'll also take a look at how these feature maps can be represented instead as arrays of numbers and what effect convolution with a kernel will have on them.

Run the cell below to get started!

```python
# Setup feedback system
from learntools.core import binder
binder.bind(globals())
from learntools.computer_vision.ex2 import *
```

# Apply Transformations #

The next few exercises walk through feature extraction just like the example in the tutorial. Run the following cell to load an image we'll use for the next few exercises.

```python
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
plt.rc('figure', autolayout=True)
plt.rc('axes', labelweight='bold', labelsize='large',
       titleweight='bold', titlesize=18, titlepad=10)
plt.rc('image', cmap='magma')

image_path = '../input/computer-vision-resources/car_illus.jpg'
image = tf.io.read_file(image_path)
image = tf.io.decode_jpeg(image, channels=1)
image = tf.image.resize(image, size=[400, 400])

plt.figure(figsize=(6, 6))
plt.imshow(tf.squeeze(image), cmap='gray')
plt.axis('off')
plt.show();
```

You can run this cell to see some standard kernels used in image processing.

```python
import learntools.computer_vision.visiontools as visiontools
from learntools.computer_vision.visiontools import edge, bottom_sobel, emboss, sharpen

kernels = [edge, bottom_sobel, emboss, sharpen]
names = ["Edge Detect", "Bottom Sobel", "Emboss", "Sharpen"]

plt.figure(figsize=(12, 12))
for i, (kernel, name) in enumerate(zip(kernels, names)):
    plt.subplot(1, 4, i+1)
    visiontools.show_kernel(kernel)
    plt.title(name)
plt.tight_layout()
```

# 1) Define Kernel #

Use the next code cell to define a kernel. You have your choice of what kind of kernel to apply. One thing to keep in mind is that the *sum* of the numbers in the kernel determines how bright the final image is. Generally, you should try to keep the sum of the numbers between 0 and 1 (though that's not required for a correct answer).

In general, a kernel can have any number of rows and columns. For this exercise, let's use a $3 \times 3$ kernel, which often gives the best results. Define a kernel with `tf.constant`.

```python
# YOUR CODE HERE: Define a kernel with 3 rows and 3 columns.
kernel = tf.constant([
    #____,
])
# Uncomment to view kernel
# visiontools.show_kernel(kernel)

# Check your answer
q_1.check()
```

```python
#%%RM_IF(PROD)%%
kernel = np.array([
    [-2, -1, 0],
    [-1, 1, 1],
    [0, 1, 2],
])
q_1.assert_check_failed()
```

```python
tf.constant#%%RM_IF(PROD)%%
kernel = tf.constant([
'abc'
])
q_1.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
kernel = tf.constant([0, 1, 2])
q_1.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
kernel = tf.constant([
    [-2, -1, 0],
    [-1, 1, 1],
    [0, 1, 2],
])
visiontools.show_kernel(kernel)
q_1.assert_check_passed()
```

```python
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
q_1.hint()
#_COMMENT_IF(PROD)_
q_1.solution()
```

Now we'll do the first step of feature extraction, the filtering step. First run this cell to do some reformatting for TensorFlow.

```python
# Reformat for batch compatibility.
image = tf.image.convert_image_dtype(image, dtype=tf.float32)
image = tf.expand_dims(image, axis=0)
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 2) Apply Convolution #

Now we'll apply the kernel to the image by a convolution. The *layer* in Keras that does this is `layers.Conv2D`. What is the *backend function* in TensorFlow that performs the same operation?

```python
# YOUR CODE HERE: Give the TensorFlow convolution function (without arguments)
conv_fn = ____

# Check your answer
q_2.check()
```

```python
#%%RM_IF(PROD)%%
conv_fn = 'abc'
q_2.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
conv_fn = tf.nn.conv2d(
    input=image,
    filters=kernel,
    strides=1, # or (1, 1)
    padding='SAME',
)
q_2.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
conv_fn = tf.nn.conv2d
q_2.assert_check_passed()
```

```python
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
q_2.hint()
#_COMMENT_IF(PROD)_
q_2.solution()
```

Once you've got the correct answer, run this next cell to execute the convolution and see the result!

```python
image_filter = conv_fn(
    input=image,
    filters=kernel,
    strides=1, # or (1, 1)
    padding='SAME',
)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_filter)
)
plt.axis('off')
plt.show();
```

Can you see how the kernel you chose relates to the feature map it produced?

# 3) Apply ReLU #

Now detect the feature with the ReLU function. In Keras, you'll usually use this as the activation function in a `Conv2D` layer. What is the *backend function* in TensorFlow that does the same thing?

```python
# YOUR CODE HERE: Give the TensorFlow ReLU function (without arguments)
relu_fn = ____

# Check your answer
q_3.check()
```

```python
#%%RM_IF(PROD)%%
relu_fn = 'abc'
q_3.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
relu_fn = tf.nn.relu(image_filter)
q_3.assert_check_failed()
```

```python
#%%RM_IF(PROD)%%
relu_fn = tf.nn.relu
q_3.assert_check_passed()
```

```python
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
q_3.hint()
#_COMMENT_IF(PROD)_
q_3.solution()
```

Once you've got the solution, run this cell to detect the feature with ReLU and see the result!

The image you see below is the feature map produced by the kernel you chose. If you like, experiment with some of the other suggested kernels above, or, try to invent one that will extract a certain kind of feature.

```python
image_detect = relu_fn(image_filter)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_detect)
)
plt.axis('off')
plt.show();
```

In the tutorial, our discussion of kernels and feature maps was mainly visual. We saw the effect of `Conv2D` and `ReLU` by observing how they transformed some example images.

But the operations in a convolutional network (like in all neural networks) are usually defined through mathematical functions, through a computation on numbers. In the next exercise, we'll take a moment to explore this point of view.

Let's start by defining a simple array to act as an image, and another array to act as the kernel. Run the following cell to see these arrays.

```python
# Sympy is a python library for symbolic mathematics. It has a nice
# pretty printer for matrices, which is all we'll use it for.
import sympy
sympy.init_printing()
from IPython.display import display

image = np.array([
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 1, 1],
    [0, 1, 0, 0, 0, 0],
])

kernel = np.array([
    [1, -1],
    [1, -1],
])

display(sympy.Matrix(image))
display(sympy.Matrix(kernel))
# Reformat for Tensorflow
image = tf.cast(image, dtype=tf.float32)
image = tf.reshape(image, [1, *image.shape, 1])
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 4) Observe Convolution on a Numerical Matrix #

What do you see? The image is simply a long vertical line on the left and a short horizontal line on the lower right. What about the kernel? What effect do you think it will have on this image? After you've thought about it, run the next cell for the answer.

```python
# View the solution (Run this code cell to receive credit!)
q_4.check()
```

Now let's try it out. Run the next cell to apply convolution and ReLU to the image and display the result.

```python
image_filter = tf.nn.conv2d(
    input=image,
    filters=kernel,
    strides=1,
    padding='VALID',
)
image_detect = tf.nn.relu(image_filter)

# The first matrix is the image after convolution, and the second is
# the image after ReLU.
display(sympy.Matrix(tf.squeeze(image_filter).numpy()))
display(sympy.Matrix(tf.squeeze(image_detect).numpy()))
```

Is the result what you expected?

# Conclusion #

In this lesson, you learned about the first two operations a convolutional classifier uses for feature extraction: **filtering** an image with a **convolution** and **detecting** the feature with the **rectified linear unit**.

# Keep Going #

Move on to [**Lesson 3**](#$NEXT_NOTEBOOK_URL$) to learn the final operation: **condensing** the feature map with **maximum pooling**!