```python
%%HTML
<link rel="stylesheet" type="text/css" href="//fonts.googleapis.com/css?family=Quicksand:300" />
<link rel="stylesheet" type="text/css" href="custom.css">
```

<link rel="stylesheet" type="text/css" href="//fonts.googleapis.com/css?family=Quicksand:300" />
<link rel="stylesheet" type="text/css" href="custom.css">

# From zero to Data Science with Python

---

# Jupyter Notebook

## Reasons to use:

+ easier to edit code
+ plays well with Pandas data structures
+ output (graphs) stored under code
+ shareable

## Notebook shortcuts

All necessary commands have corresponding button at the top of the screen. As you become more familiar with using Jupyter Notebook, you'll become more efficient by using hotkeys instead. A list of some useful ones:

- `ctrl  + enter` : run cell
- `shift + enter` : run cell and create/select the one after
- `esc`           : enter command mode
- `ctrl  + s`     : save notebook

While in command mode, the shortcuts work a bit differently:

- `↑`/`↓` : browser through different cells
- `y`     : changes cell to a code cell
- `m`     : changes cell to a `markdown` cell
- `a`     : create new cell above
- `b`     : create new cell below
- `l`     : toggle line numbers
- `dd`    : delete cell

You can always access the docs/help by added the `?` in front of any function in python. You can escape this helper view by pressing `q`.

```python
? len
```

---

# Introduction to Python

## The basic types: `int`, `str`, `float`, `bool`, `None`

```python
1 + 1
```

    2

```python
'1' + '1'
```

    '11'

```python
type(1)
```

    int

```python
type('1')
```

    str

```python
1 * 3
```

    3

```python
'1' * 3
```

    '111'

```python
"THIS IS A SENTENCE IN CAPTIAL LETTERS!".lower()
```

    'this is a sentence in captial letters!'

```python
"this is a sentence in lowercase letters!".upper()
```

    'THIS IS A SENTENCE IN LOWERCASE LETTERS!'

```python
"2018/02/01".replace("/", "-")
```

    '2018-02-01'

```python
str(1)
```

    '1'

```python
type( str(1) )
```

    str

```python
print('The number five looks like : ' + str(5))
```

    The number five looks like : 5

```python
type(1.0)
```

    float

```python
float(1)
```

    1.0

```python
type(True), type(False)
```

    (bool, bool)

```python
type(None)
```

    NoneType

## Lists

```python
a = [1, 'q', False]
```

```python
type(a)
```

    list

```python
a[1]
```

    'q'

```python
a[3]
```

```python
b = []
b.append('w')
b.append(1211.35)
b
```

    ['w', 1211.35]

```python
len(b)
```

    2

```python
len('this sentence has 31 characters')
```

    31

```python
c0 = [1, 2, 3]
c1 = ['x', 'y', 'z']
c = c0 + c1
```

```python
c
```

    [1, 2, 3, 'x', 'y', 'z']

```python
'y' in c
```

    True

```python
'2018-02-01'.split('-')
```

    ['2018', '02', '01']

```python
for i in [0, 1, 2, 3, 4]:
    print(i)
```

    0
    1
    2
    3
    4

```python
for i, j in enumerate(['a', 'b', 'c', 'd', 'e']):
    print('Index: ', i, ', Character: ', j)
```

    Index:  0 , Character:  a
    Index:  1 , Character:  b
    Index:  2 , Character:  c
    Index:  3 , Character:  d
    Index:  4 , Character:  e

```python
list(enumerate(['a', 'b', 'c', 'd', 'e']))
```

    [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e')]

```python
for i in range(5):
    print(i)
```

    0
    1
    2
    3
    4

```python
for i in range(5, 10):
    print(i)
```

    5
    6
    7
    8
    9

```python
for i in range(0, 10, 2):
    print(i)
```

    0
    2
    4
    6
    8

```python
for i in range(10, 5):
    print(i)
```

Full list of built-in Python functions: [https://docs.python.org/3/library/functions.html](https://docs.python.org/3/library/functions.html)

## Functions

```python
def double(x):
    """
    This functions doubles a number.

    example:
    double(2) => 4
    """
    y = x * 2
    return y
```

```python
double(3)
```

    6

```python
double('3')
```

    '33'

```python
? double
```

```python
double.__doc__
```

    '\n    This functions doubles a number.\n    \n    example:\n    double(2) => 4\n    '

```python
def is_even(x):
    if type(x) != int:
        return "Enter only integers!"  # this type of returns are harmful though, you should raise or return None
    elif x % 2 == 0:
        return True
    else:
        return False
```

```python
is_even(42)
```

    True

```python
is_even(1.5)
```

    'Enter only integers!'

```python
def factorial(n):
    """
    factorial(n) = n * (n-1) * (n-2) * ... * 3 * 2 * 1
    """
    answer = 1
    current_factor = 1
    while current_factor <= n:
        answer *= current_factor
        current_factor += 1
    return answer
```

```python
factorial(4)
```

    24

## Exercise

The Fibonacci numbers is the sequence beginning 1, 1, 2, 3, 5, 8, 13, ... and defined by the recursive equation:

`F(n) = F(n-1) + F(n-2), F(0) = F(1) = 1`

Thus if we want to calculate `F(3)`, we proceed as follows:

`F(3) = F(2) + F(1) = F(1) + F(0) + 1 = 1 + 1 + 1 = 3`

Defining a Python function in this way, though, is a bit tricky. Instead of performing the calculation from the top-down, let's try from the bottom-up: for a given `n`, add `F(0)`, `F(1)`, `F(2)`, ..., `F(n-1)`, and `F(n)`. Write a function `fibonacci()` in this way, using the logic of `while`, `if`, and `else`.

```python
assert fibonacci(6) == 8
```

## List comprehensions

```python
[ k for k in [0, 1, 2, 3, 4] ]
```

    [0, 1, 2, 3, 4]

```python
[ i * 2 for i in [0, 1, 2, 3, 4] ]
```

    [0, 2, 4, 6, 8]

```python
[ is_even(i) for i in [0, 1, 2, 3, 4]]
```

    [True, False, True, False, True]

```python
[ i * 2 for i in range(5) ]
```

    [0, 2, 4, 6, 8]

```python
[(i, double(j)) for i, j in enumerate(['a', 'b', 'c', 'd', 'e'])]
```

    [(0, 'aa'), (1, 'bb'), (2, 'cc'), (3, 'dd'), (4, 'ee')]

```python
[ double(i) for i in [1, 2, 3, 4, 5] if is_even(i) ]
```

    [4, 8]

```python
for i in map(double, [1, 2, 3, 4, 5]):
    print(i)
```

    2
    4
    6
    8
    10

## Indexing and slicing

```python
arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
```

```python
arr[0]
```

    1

```python
arr[1]
```

    2

```python
arr[:2]
```

    [1, 2]

```python
arr[1:3]
```

    [2, 3]

```python
arr[1::2]
```

    [2, 4, 6, 8, 10]

```python
arr[:-1]
```

    [1, 2, 3, 4, 5, 6, 7, 8, 9]

```python
arr[-2:]
```

    [9, 10]

```python
arr[3:-1]
```

    [4, 5, 6, 7, 8, 9]

```python
arr[-3:-1]
```

    [8, 9]

```python
arr[::-1]
```

    [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]

```python
arr[::2]
```

    [1, 3, 5, 7, 9]

```python
arr[:-1] == arr[:-1:]
```

    True

```python
arr[0:5]
```

    [1, 2, 3, 4, 5]

## Dicts

```python
d = {
    'a': 1,
    'b': 2,
    'c': 3
}
print('Type: ', type(d), ', d: ', d)
```

    Type:  <class 'dict'> , d:  {'a': 1, 'b': 2, 'c': 3}

```python
d['c']
```

    3

```python
d['c'] = 4
print( d )
```

    {'a': 1, 'b': 2, 'c': 4}

```python
d.keys()
```

    dict_keys(['a', 'b', 'c'])

```python
'b' in d.keys()
```

    True

```python
d.values()
```

    dict_values([1, 2, 4])

```python
d.items()
```

    dict_items([('a', 1), ('b', 2), ('c', 4)])

```python
for k, v in d.items():
    print('Key: ', k, ', Value: ', v)
```

    Key:  a , Value:  1
    Key:  b , Value:  2
    Key:  c , Value:  4

```python
for thing in d.items():
    print(thing[0], thing[1])
```

    a 1
    b 2
    c 4

```python
e = ('a', 1, True)
print('Type: ', type(e), ', e: ', e)
```

    Type:  <class 'tuple'> , e:  ('a', 1, True)

```python
e[0]
```

    'a'

```python
e[0] = 'z'
```

```python
{ thing[0]: thing[1] for thing in [('a', 1), ('b', 2), ('c', 3)] }
```

    {'a': 1, 'b': 2, 'c': 3}

```python
{ k: double(v) for k, v in [('a', 1), ('b', 2), ('c', 3)] }
```

    {'a': 2, 'b': 4, 'c': 6}

```python
d['f']
```

This problem can arise in the following way. When building a dict, you need to check first if the key you want to add is already present (to avoid the above `KeyError`). If not, you need to add it before adding your desired value:

```python
d = {}
for k, v in [('a', 2), ('b', 3), ('a', 4), ('c', 0), ('b', 5), ('a', 8)]:
    if k not in d.keys():
        d[k] = []
    d[k].append(v)
d
```

    {'a': [2, 4, 8], 'b': [3, 5], 'c': [0]}

This seems un-Pythonic to me. Enter the `defaultdict`:

```python
from collections import defaultdict
```

```python
d = defaultdict(list)
for k, v in [('a', 2), ('b', 3), ('a', 4), ('c', 0), ('b', 5), ('a', 8)]:
    d[k].append(v)
d
```

    defaultdict(list, {'a': [2, 4, 8], 'b': [3, 5], 'c': [0]})

You may also notice that not everything can be a key in a dictionary. Lists cannot be keys, but tuples can!

```python
d = { (1, 1) : 'London', (2, 2): 'Amsterdam' }
d
```

    {(1, 1): 'London', (2, 2): 'Amsterdam'}

```python
d[(2,2)]
```

    'Amsterdam'

The reason that lists cannot be used is because they are mutable (which is a bad characteristic for a key).

## Itertools

`itertools` is part of the python standard library, just like the `collections` module. It has a few very useful functions in it. We'll briefly discuss them here.

```python
import itertools as it
```

```python
it.product([1,2,3], [4,5,6])
```

    <itertools.product at 0x10bcacca8>

The reason this doesn't produce the expected output is because it returns a **generator** and not a **list**. What a generator is is outside the scope of this lecture; a decent explanation can be found [here](https://realpython.com/blog/python/introduction-to-python-generators/).

```python
list( it.product([1, 2, 3], [4, 5, 6]) )
```

    [(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]

```python
list( it.permutations([1, 2, 3, 4, 5], 2) )
```

    [(1, 2),
     (1, 3),
     (1, 4),
     (1, 5),
     (2, 1),
     (2, 3),
     (2, 4),
     (2, 5),
     (3, 1),
     (3, 2),
     (3, 4),
     (3, 5),
     (4, 1),
     (4, 2),
     (4, 3),
     (4, 5),
     (5, 1),
     (5, 2),
     (5, 3),
     (5, 4)]

```python
list( it.combinations([1, 2, 3, 4, 5], 2) )
```

    [(1, 2),
     (1, 3),
     (1, 4),
     (1, 5),
     (2, 3),
     (2, 4),
     (2, 5),
     (3, 4),
     (3, 5),
     (4, 5)]

### Key lessons

- list comprehensions are shorter and often simpler
- it's OK to use `for`-loops if you need them, but sometimes its better not to
- map/apply type functions seperate concerns, `collections`/`itertools` help
- there are usually multiple ways to do something
- pick the obvious way of doing something but think about it beforehand

## Files

```python
with open('data/chickweight.csv', 'r') as ff:
    for _ in range(5):
        print(ff.readline())
```

    rownum,weight,Time,Chick,Diet

    1,42,0,1,1

    2,51,2,1,1

    3,59,4,1,1

    4,64,6,1,1

`chickweight.csv` is a CSV (**Comma-Separated Values**) file. It stores tabular information in human-readable form, using (usually but not necessarily) commas to separate values. A Python module exists for helping read from and write to CSV files but using it typically not necessary.

```python
with open('data/chickweight.csv', 'r') as ff:
    contents = ff.read()
```

```python
type(contents)
```

    str

```python
contents[:100]
```

    'rownum,weight,Time,Chick,Diet\n1,42,0,1,1\n2,51,2,1,1\n3,59,4,1,1\n4,64,6,1,1\n5,76,8,1,1\n6,93,10,1,1\n7,1'

```python
print('a', '\n', 'b')
```

    a
     b

```python
print('a', '\\n', 'b')
```

    a \n b

```python
lines = contents.split('\n')
```

```python
type(lines)
```

    list

```python
lines[0]
```

    'rownum,weight,Time,Chick,Diet'

```python
lines[1]
```

    '1,42,0,1,1'

```python
len(lines)
```

    579

```python
header = lines[0]
last_ten_lines = lines[-10:]
small_file_list = [header] + last_ten_lines
small_file_str = '\n'.join(small_file_list)
```

```python
small_file_str
```

    'rownum,weight,Time,Chick,Diet\n569,67,4,50,4\n570,84,6,50,4\n571,105,8,50,4\n572,122,10,50,4\n573,155,12,50,4\n574,175,14,50,4\n575,205,16,50,4\n576,234,18,50,4\n577,264,20,50,4\n578,264,21,50,4'

```python
with open('data/small_chickweight.csv', 'w') as ff:
    ff.write(small_file_str)
```

## Exercise

Use the techniques above to create a new file from `data/chickweight.csv` for every distinct `Chick` containing only the `weight` and `Time` columns.

## JSON

**JavaScript Object Notation** is another popular human-readable file format. A JSON document consists of either an array (think: Python list) or associative array (think: Python dict) where the keys are strings. The values in these arrays can be numbers, Booleans, `None`s, strings, or other (associative) arrays. Let's examine a sample JSON document (generated via the [JSON Generator](https://www.json-generator.com/):

```python
with open('data/generated.json', 'r') as ff:
    doc_string = ff.read()
print(doc_string)
```

    [
      {
        "_id": "5a70cfbc45459ab8b01b0772",
        "index": 0,
        "guid": "fdd155b6-5d61-4ae0-a912-d11533c94e9c",
        "isActive": true,
        "balance": "$3,635.14",
        "picture": "http://placehold.it/32x32",
        "age": 40,
        "eyeColor": "green",
        "name": "Corrine Pugh",
        "gender": "female",
        "company": "BESTO",
        "email": "corrinepugh@besto.com",
        "phone": "+1 (856) 523-2643",
        "address": "726 Lincoln Terrace, Cowiche, Alabama, 9320",
        "about": "Ullamco ex dolor aliqua dolor duis esse ipsum laboris adipisicing. Commodo nisi veniam sit esse ex. Occaecat anim labore voluptate elit ut ut elit veniam laboris esse qui velit. Aute velit sint adipisicing ullamco consectetur id. Aliquip quis irure eu ad Lorem eiusmod anim in magna aute adipisicing fugiat eiusmod excepteur.\r\n",
        "registered": "2016-03-08T10:47:14 -01:00",
        "latitude": 64.881052,
        "longitude": 152.110462,
        "tags": [
          "duis",
          "excepteur",
          "tempor",
          "aliqua",
          "proident",
          "amet",
          "amet"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Hannah Pacheco"
          },
          {
            "id": 1,
            "name": "Loraine Burton"
          },
          {
            "id": 2,
            "name": "Darlene Burris"
          }
        ],
        "greeting": "Hello, Corrine Pugh! You have 3 unread messages.",
        "favoriteFruit": "strawberry"
      },
      {
        "_id": "5a70cfbca8acc89ddf73fe58",
        "index": 1,
        "guid": "789e1e8e-b093-4182-a482-0d206a9b26ec",
        "isActive": true,
        "balance": "$1,078.48",
        "picture": "http://placehold.it/32x32",
        "age": 34,
        "eyeColor": "blue",
        "name": "Stanton Kelley",
        "gender": "male",
        "company": "TOURMANIA",
        "email": "stantonkelley@tourmania.com",
        "phone": "+1 (974) 491-2133",
        "address": "258 Rockwell Place, Osage, Puerto Rico, 549",
        "about": "Mollit ex Lorem in nisi fugiat mollit irure eiusmod excepteur. Exercitation anim Lorem dolor nulla magna. Ea in est irure irure Lorem mollit incididunt.\r\n",
        "registered": "2017-10-23T03:42:05 -02:00",
        "latitude": 10.181705,
        "longitude": 22.472016,
        "tags": [
          "adipisicing",
          "commodo",
          "enim",
          "aliqua",
          "do",
          "quis",
          "cupidatat"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Luna Campos"
          },
          {
            "id": 1,
            "name": "Elaine Gentry"
          },
          {
            "id": 2,
            "name": "Elvira Baird"
          }
        ],
        "greeting": "Hello, Stanton Kelley! You have 4 unread messages.",
        "favoriteFruit": "strawberry"
      },
      {
        "_id": "5a70cfbca6467b1911ba63f5",
        "index": 2,
        "guid": "6ebc5c58-e9d9-467a-845f-b13229c5a505",
        "isActive": true,
        "balance": "$3,602.10",
        "picture": "http://placehold.it/32x32",
        "age": 37,
        "eyeColor": "brown",
        "name": "Dollie Nelson",
        "gender": "female",
        "company": "ENERSAVE",
        "email": "dollienelson@enersave.com",
        "phone": "+1 (883) 489-3713",
        "address": "491 Dennett Place, Downsville, Washington, 5649",
        "about": "Laborum ea dolor cupidatat proident cillum est minim labore. Incididunt eu nisi eu aliquip consectetur do laborum. Dolor ad sit sunt excepteur tempor eu excepteur. Esse officia officia sit Lorem ut fugiat duis fugiat ut do. Magna quis exercitation ullamco fugiat cupidatat sit fugiat et reprehenderit cupidatat. Minim et dolor ex irure veniam enim non quis esse.\r\n",
        "registered": "2014-06-24T08:53:19 -02:00",
        "latitude": -49.748363,
        "longitude": -37.100133,
        "tags": [
          "mollit",
          "nulla",
          "occaecat",
          "aliqua",
          "ad",
          "minim",
          "officia"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Crystal Sloan"
          },
          {
            "id": 1,
            "name": "Preston Rosario"
          },
          {
            "id": 2,
            "name": "Bowers Mcfarland"
          }
        ],
        "greeting": "Hello, Dollie Nelson! You have 7 unread messages.",
        "favoriteFruit": "apple"
      },
      {
        "_id": "5a70cfbc5f072e9d911e49df",
        "index": 3,
        "guid": "48f596dd-c440-4940-8f53-9dc17a4d8b8f",
        "isActive": false,
        "balance": "$3,052.70",
        "picture": "http://placehold.it/32x32",
        "age": 33,
        "eyeColor": "brown",
        "name": "Deena Schmidt",
        "gender": "female",
        "company": "VIOCULAR",
        "email": "deenaschmidt@viocular.com",
        "phone": "+1 (880) 417-3391",
        "address": "818 Cooper Street, Forbestown, Maryland, 8259",
        "about": "Quis adipisicing aute elit dolor pariatur dolor. Dolor amet velit occaecat ex aliquip minim. Incididunt sit magna dolor elit consequat. Eu culpa consequat do irure incididunt aute.\r\n",
        "registered": "2014-09-15T05:32:02 -02:00",
        "latitude": -10.133346,
        "longitude": 89.592497,
        "tags": [
          "ullamco",
          "incididunt",
          "deserunt",
          "ipsum",
          "veniam",
          "voluptate",
          "exercitation"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Rice Keith"
          },
          {
            "id": 1,
            "name": "Noel Morgan"
          },
          {
            "id": 2,
            "name": "Elva Sandoval"
          }
        ],
        "greeting": "Hello, Deena Schmidt! You have 5 unread messages.",
        "favoriteFruit": "apple"
      },
      {
        "_id": "5a70cfbc4d421b6fdbf78baf",
        "index": 4,
        "guid": "a46324b5-ff81-4b2c-8136-dc5adb830adc",
        "isActive": false,
        "balance": "$2,986.41",
        "picture": "http://placehold.it/32x32",
        "age": 32,
        "eyeColor": "green",
        "name": "Gay Griffith",
        "gender": "male",
        "company": "TECHADE",
        "email": "gaygriffith@techade.com",
        "phone": "+1 (964) 583-3859",
        "address": "322 Lawton Street, Coleville, Ohio, 3802",
        "about": "Exercitation sunt ullamco labore velit aute culpa et voluptate. Consequat et ut cupidatat non ullamco irure duis nostrud. Elit Lorem anim minim consequat consequat in deserunt sint quis pariatur commodo. Eiusmod commodo nostrud in eiusmod ut sit. Cupidatat et ullamco ad tempor veniam consequat dolor aliqua est.\r\n",
        "registered": "2015-10-10T03:17:41 -02:00",
        "latitude": 44.352609,
        "longitude": 101.846246,
        "tags": [
          "duis",
          "quis",
          "proident",
          "qui",
          "quis",
          "velit",
          "labore"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Harding Thompson"
          },
          {
            "id": 1,
            "name": "Eugenia Kemp"
          },
          {
            "id": 2,
            "name": "Payne Fischer"
          }
        ],
        "greeting": "Hello, Gay Griffith! You have 2 unread messages.",
        "favoriteFruit": "strawberry"
      },
      {
        "_id": "5a70cfbc082993456638e3b9",
        "index": 5,
        "guid": "874298b3-7319-4f5d-a7be-e97e2edd79ab",
        "isActive": false,
        "balance": "$1,738.60",
        "picture": "http://placehold.it/32x32",
        "age": 35,
        "eyeColor": "brown",
        "name": "Mclaughlin James",
        "gender": "male",
        "company": "CAPSCREEN",
        "email": "mclaughlinjames@capscreen.com",
        "phone": "+1 (953) 462-3163",
        "address": "331 Fane Court, Chical, Nevada, 8750",
        "about": "Dolore anim incididunt sit ea reprehenderit veniam ullamco. Incididunt laborum in reprehenderit cillum qui Lorem. Id eiusmod anim consequat laborum. Do commodo mollit officia aliquip non officia id cillum occaecat nulla excepteur cupidatat. Pariatur culpa laborum velit ut esse nisi nostrud quis do dolor commodo cillum eiusmod.\r\n",
        "registered": "2014-06-23T02:54:36 -02:00",
        "latitude": 51.342523,
        "longitude": -167.974659,
        "tags": [
          "dolore",
          "ad",
          "labore",
          "labore",
          "cillum",
          "ea",
          "eu"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Mattie Buck"
          },
          {
            "id": 1,
            "name": "Emma Strong"
          },
          {
            "id": 2,
            "name": "Janis Farmer"
          }
        ],
        "greeting": "Hello, Mclaughlin James! You have 3 unread messages.",
        "favoriteFruit": "strawberry"
      },
      {
        "_id": "5a70cfbc432201017cdc344b",
        "index": 6,
        "guid": "8db3a87b-d25d-4452-b746-876172ded0b2",
        "isActive": false,
        "balance": "$3,907.58",
        "picture": "http://placehold.it/32x32",
        "age": 33,
        "eyeColor": "blue",
        "name": "Benton Humphrey",
        "gender": "male",
        "company": "DAIDO",
        "email": "bentonhumphrey@daido.com",
        "phone": "+1 (804) 497-2066",
        "address": "897 Loring Avenue, Allendale, Montana, 9518",
        "about": "Id commodo amet enim velit cillum minim anim. Aliquip laborum tempor nostrud dolor ad ullamco anim. Veniam minim ad dolor quis aliqua cillum adipisicing mollit pariatur id nostrud ut. Ullamco officia duis sint aliqua. Esse veniam ad laboris ut irure.\r\n",
        "registered": "2016-03-30T01:18:36 -02:00",
        "latitude": 76.716756,
        "longitude": -73.392068,
        "tags": [
          "velit",
          "proident",
          "sit",
          "ut",
          "eiusmod",
          "nulla",
          "aute"
        ],
        "friends": [
          {
            "id": 0,
            "name": "Henrietta Berger"
          },
          {
            "id": 1,
            "name": "Hazel Weber"
          },
          {
            "id": 2,
            "name": "Aileen Wade"
          }
        ],
        "greeting": "Hello, Benton Humphrey! You have 1 unread messages.",
        "favoriteFruit": "apple"
      }
    ]

```python
type(doc_string)
```

    str

```python
len(doc_string)
```

    9212

Notice that, although what's printed out **looks** like Python lists, dicts, ints, etc. it's actually one big string. Manipulating the string is possible but there is a better way: convert the string into Python objects.

```python
import json

doc_json = json.loads(doc_string)
```

```python
type(doc_json)
```

    list

```python
len(doc_json)
```

    7

```python
print(doc_json[0])
```

    {'_id': '5a70cfbc45459ab8b01b0772', 'index': 0, 'guid': 'fdd155b6-5d61-4ae0-a912-d11533c94e9c', 'isActive': True, 'balance': '$3,635.14', 'picture': 'http://placehold.it/32x32', 'age': 40, 'eyeColor': 'green', 'name': 'Corrine Pugh', 'gender': 'female', 'company': 'BESTO', 'email': 'corrinepugh@besto.com', 'phone': '+1 (856) 523-2643', 'address': '726 Lincoln Terrace, Cowiche, Alabama, 9320', 'about': 'Ullamco ex dolor aliqua dolor duis esse ipsum laboris adipisicing. Commodo nisi veniam sit esse ex. Occaecat anim labore voluptate elit ut ut elit veniam laboris esse qui velit. Aute velit sint adipisicing ullamco consectetur id. Aliquip quis irure eu ad Lorem eiusmod anim in magna aute adipisicing fugiat eiusmod excepteur.\r\n', 'registered': '2016-03-08T10:47:14 -01:00', 'latitude': 64.881052, 'longitude': 152.110462, 'tags': ['duis', 'excepteur', 'tempor', 'aliqua', 'proident', 'amet', 'amet'], 'friends': [{'id': 0, 'name': 'Hannah Pacheco'}, {'id': 1, 'name': 'Loraine Burton'}, {'id': 2, 'name': 'Darlene Burris'}], 'greeting': 'Hello, Corrine Pugh! You have 3 unread messages.', 'favoriteFruit': 'strawberry'}

```python
type(doc_json[0])
```

    dict

```python
from pprint import pprint

pprint(doc_json[0])
```

    {'_id': '5a70cfbc45459ab8b01b0772',
     'about': 'Ullamco ex dolor aliqua dolor duis esse ipsum laboris adipisicing. '
              'Commodo nisi veniam sit esse ex. Occaecat anim labore voluptate '
              'elit ut ut elit veniam laboris esse qui velit. Aute velit sint '
              'adipisicing ullamco consectetur id. Aliquip quis irure eu ad Lorem '
              'eiusmod anim in magna aute adipisicing fugiat eiusmod '
              'excepteur.\r\n',
     'address': '726 Lincoln Terrace, Cowiche, Alabama, 9320',
     'age': 40,
     'balance': '$3,635.14',
     'company': 'BESTO',
     'email': 'corrinepugh@besto.com',
     'eyeColor': 'green',
     'favoriteFruit': 'strawberry',
     'friends': [{'id': 0, 'name': 'Hannah Pacheco'},
                 {'id': 1, 'name': 'Loraine Burton'},
                 {'id': 2, 'name': 'Darlene Burris'}],
     'gender': 'female',
     'greeting': 'Hello, Corrine Pugh! You have 3 unread messages.',
     'guid': 'fdd155b6-5d61-4ae0-a912-d11533c94e9c',
     'index': 0,
     'isActive': True,
     'latitude': 64.881052,
     'longitude': 152.110462,
     'name': 'Corrine Pugh',
     'phone': '+1 (856) 523-2643',
     'picture': 'http://placehold.it/32x32',
     'registered': '2016-03-08T10:47:14 -01:00',
     'tags': ['duis', 'excepteur', 'tempor', 'aliqua', 'proident', 'amet', 'amet']}

```python
doc_json[0]['friends']
```

    [{'id': 0, 'name': 'Hannah Pacheco'},
     {'id': 1, 'name': 'Loraine Burton'},
     {'id': 2, 'name': 'Darlene Burris'}]

We also also go the other way: save Python dicts, arrays, etc. as JSON docs.

```python
address_dict = {
    'number': 202,
    'street': 'Wibautstraat',
    'city': 'Amsterdam'
}
address_string = json.dumps(address_dict)
```

```python
print(address_string)
```

    {"number": 202, "street": "Wibautstraat", "city": "Amsterdam"}

```python
type(address_string)
```

    str

```python
with open('data/address.json', 'w') as ff:
    ff.write(address_string)
```

Note that JSON can't fully reproduce the structure of all Python objects. In particular, Python tuples are converted into JSON arrays and Python defaultdicts are converted into JSON associative arrays. Were such JSON to be then converted back into Python via the `json` module, we would end up with lists and (normal) dicts.

One additional bit of trickiness you'll see with files whose name ends in `.json` is that they can sometimes consist of multiple JSON documents, each separated by a newline.

## Exercise

Using `data/generated.json`, create a new file `data/generated-short.json`. This file should have one line per entry in the original `data/generated.json`, where each line is a JSON document consisting of the following name/value pairs from the original entry: `'name'`, `'age'`, `'balance'`, and `'isActive'` and the lines in the file are separated by newline characters (`'\n'`).

## Loading modules

Not all Python functions are included in the list of built-in functions or the standard library - some have to be downloaded and installed from the internet. Before moving on, make sure that the following modules can be loaded without difficulty:

```python
import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
import seaborn as sns; sns.set()
```

Should this give a problem, you can use pip from your ipython notebook and install any python package you might want.

```python
import pip
from pip._internal import  main as pip_main

def install(package):
    pip_main(['install', package])

install('requests')
```

## Lists, NumPy and performance

A downside of python is that because it is a dynamically-typed language (you don't have to specify the type of your variables while writing code, the interpreter will figure this out on the fly) we lose some performance. The `numpy` module is meant to counter this by giving us an API with python objects that call highly-performant C code so that we can still write python code for performant numerical problems.

Let's have a look. We can use the `%%timeit` notebook magic.

```python
%%timeit
sum( range(100000) )
```

    2.23 ms ± 157 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
sum( [i for i in range(100000)] )
```

    6.63 ms ± 717 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
np.sum( np.arange(100000) )
```

    94.1 µs ± 5.16 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
from time import time

start = time()
np.sum( np.arange(0,100000) )
print( time() - start )
```

    0.0006170272827148438

```python
%%timeit
np.sum( [i for i in range(100000)] )
```

    11.7 ms ± 258 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
sum( np.arange(100000) )
```

    7.81 ms ± 154 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
s = 0
for i in range(100000):
    s += 1
```

    4.7 ms ± 112 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Keep all your functions restricted to numpy if you want performant code. The moment you mix python number lists with numpy arrays the code seems to drastically lose performance. Using the standard python `sum` function on a numpy array is the worst case!

Similar conclusions can be made when dealing with vectorized operations.

```python
%%timeit
[ 2 * i for i in range(100000) ]
```

    7.54 ms ± 350 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
np.arange(100000) * 2
```

    79.8 µs ± 2.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
%%timeit
[ math.sqrt(i) for i in range(100000) ]
```

    14 ms ± 664 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
np.sqrt( np.arange(100000) )
```

    256 µs ± 10.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The reason behind the performance cost has to do with type-checking/casting that needs to be done at runtime instead of compile time. Numpy will run precompiled C-code, any combination with normal python code will therefore add overhead.

## NumPy

Numpy arrays are very different from python lists. You can easily cast between the two, but you should notice they will behave differently around operators.

```python
py_arr = [1, False, 1.]
np_arr = np.array(py_arr)

print(py_arr, type(py_arr))
print(np_arr, np_arr.dtype)
```

    [1, False, 1.0] <class 'list'>
    [ 1.  0.  1.] float64

```python
py_arr * 2
```

    [1, False, 1.0, 1, False, 1.0]

```python
np_arr * 2
```

    array([ 2.,  0.,  2.])

```python
print(np_arr.dtype)
print(np_arr.shape)
```

    float64
    (3,)

The datatype of the array automatically depends on the array content.

```python
np.array([1, 25]).dtype
```

    dtype('int64')

```python
np.array([1.0, 2.5]).dtype
```

    dtype('float64')

```python
np.array(['Phase 1: collect underpants','Phase 2: ?','Phase 3: profit']).dtype
```

    dtype('<U27')

```python
np.array([1, 2.5]).dtype
```

    dtype('float64')

There are [many](http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html) datatypes out there. For now, we will only focus on numerical `float64` arrays and matrices.

You can declare arrays very easily in numpy. Here are some common initialization functions.

```python
np.zeros(3)
```

    array([ 0.,  0.,  0.])

```python
np.ones(3)
```

    array([ 1.,  1.,  1.])

```python
np.zeros((3, 3))
```

    array([[ 0.,  0.,  0.],
           [ 0.,  0.,  0.],
           [ 0.,  0.,  0.]])

```python
np.ones((3, 3))
```

    array([[ 1.,  1.,  1.],
           [ 1.,  1.,  1.],
           [ 1.,  1.,  1.]])

```python
np.eye(3)
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

```python
np.arange(1, 10, 0.1)
```

    array([ 1. ,  1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,
            2.1,  2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,
            3.2,  3.3,  3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,
            4.3,  4.4,  4.5,  4.6,  4.7,  4.8,  4.9,  5. ,  5.1,  5.2,  5.3,
            5.4,  5.5,  5.6,  5.7,  5.8,  5.9,  6. ,  6.1,  6.2,  6.3,  6.4,
            6.5,  6.6,  6.7,  6.8,  6.9,  7. ,  7.1,  7.2,  7.3,  7.4,  7.5,
            7.6,  7.7,  7.8,  7.9,  8. ,  8.1,  8.2,  8.3,  8.4,  8.5,  8.6,
            8.7,  8.8,  8.9,  9. ,  9.1,  9.2,  9.3,  9.4,  9.5,  9.6,  9.7,
            9.8,  9.9])

```python
np.linspace(start=1, stop=10, num=20)
```

    array([  1.        ,   1.47368421,   1.94736842,   2.42105263,
             2.89473684,   3.36842105,   3.84210526,   4.31578947,
             4.78947368,   5.26315789,   5.73684211,   6.21052632,
             6.68421053,   7.15789474,   7.63157895,   8.10526316,
             8.57894737,   9.05263158,   9.52631579,  10.        ])

```python
np.logspace(start=1, stop=7, num=7, base=10)
```

    array([  1.00000000e+01,   1.00000000e+02,   1.00000000e+03,
             1.00000000e+04,   1.00000000e+05,   1.00000000e+06,
             1.00000000e+07])

```python
np.arange(15).reshape([3, 5])
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14]])

As stated before, we can do vectorized operations on these arrays as if we are doing true mathematical operators on them. If you remember your linear algebra from college, the following might feel more familiar.

```python
np_nums = np.array([1.0, 3.0, 5.0])
np_nums + 1
```

    array([ 2.,  4.,  6.])

```python
np.arange(15).reshape([3, 5]) + 1
```

    array([[ 1,  2,  3,  4,  5],
           [ 6,  7,  8,  9, 10],
           [11, 12, 13, 14, 15]])

```python
np_nums - np_nums
```

    array([ 0.,  0.,  0.])

```python
np.ones((4, 3))
```

    array([[ 1.,  1.,  1.],
           [ 1.,  1.,  1.],
           [ 1.,  1.,  1.],
           [ 1.,  1.,  1.]])

```python
np_nums * np.ones((4, 3))
```

    array([[ 1.,  3.,  5.],
           [ 1.,  3.,  5.],
           [ 1.,  3.,  5.],
           [ 1.,  3.,  5.]])

but...

```python
np_nums * np.ones((3, 4))
```

```python
np_nums.shape, np.ones((3, 4)).shape
```

    ((3,), (3, 4))

```python
np_nums.reshape((3, 1))
```

    array([[ 1.],
           [ 3.],
           [ 5.]])

```python
np_nums.reshape((3, 1)).shape
```

    (3, 1)

```python
np_nums.reshape((3, 1)) * np.ones((3, 4))
```

    array([[ 1.,  1.,  1.,  1.],
           [ 3.,  3.,  3.,  3.],
           [ 5.,  5.,  5.,  5.]])

NumPy arrays have common summary statistics methods associated with them:

```python
print( np.arange(1, 10) )
```

    [1 2 3 4 5 6 7 8 9]

```python
print( np.arange(1, 10).sum() )
```

    45

```python
print( np.arange(1, 10).cumsum() )
```

    [ 1  3  6 10 15 21 28 36 45]

```python
print( np.arange(1, 10).mean() )
```

    5.0

```python
print( np.arange(1, 10).std() )
```

    2.58198889747

```python
print( np.arange(1, 10).max() )
```

    9

```python
print( np.arange(1, 10).min() )
```

    1

NumPy also allows for indexing and slicing, but has some extra flexibilities.

```python
arr = np.array([9, 4, 3, 5, 2, 1, 5, 3])
arr[1]
```

    4

```python
arr[1:5]
```

    array([4, 3, 5, 2])

```python
arr[1:5] = 3
arr
```

    array([9, 3, 3, 3, 3, 1, 5, 3])

```python
arr == 3
```

    array([False,  True,  True,  True,  True, False, False,  True], dtype=bool)

(This is called a bitmask.)

```python
arr[arr == 3]
```

    array([3, 3, 3, 3, 3])

```python
arr[arr != 3]
```

    array([9, 1, 5])

Array slices can be multidimensional too!

```python
orig = np.random.normal(loc=0, scale=1, size=[4,4])
orig
```

    array([[ 0.58655646, -0.11468599, -0.76400848, -0.49096007],
           [ 0.33159074, -0.5886225 , -0.03742059,  1.18871613],
           [-1.14929503, -0.4189929 ,  1.23538325, -0.25124764],
           [-0.06174092, -1.64996835, -0.01030903, -0.50062249]])

```python
deriv = orig.copy()
```

```python
deriv < 0
```

    array([[False,  True,  True,  True],
           [False,  True,  True, False],
           [ True,  True, False,  True],
           [ True,  True,  True,  True]], dtype=bool)

```python
deriv[deriv < 0] = 0
```

```python
orig
```

    array([[ 0.58655646, -0.11468599, -0.76400848, -0.49096007],
           [ 0.33159074, -0.5886225 , -0.03742059,  1.18871613],
           [-1.14929503, -0.4189929 ,  1.23538325, -0.25124764],
           [-0.06174092, -1.64996835, -0.01030903, -0.50062249]])

```python
deriv
```

    array([[ 0.58655646,  0.        ,  0.        ,  0.        ],
           [ 0.33159074,  0.        ,  0.        ,  1.18871613],
           [ 0.        ,  0.        ,  1.23538325,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ]])

We are now using the `random` submodule from NumPy. The random processes generated by this submodule depend on a seed number to determine their output, which often times is based on something quasi-unique, such as the system time. To ensure reproducible results, set the seed value.

```python
np.random.seed(seed=0)
```

```python
np.random.normal(loc=0, scale=1, size=[4,4])
```

    array([[ 1.76405235,  0.40015721,  0.97873798,  2.2408932 ],
           [ 1.86755799, -0.97727788,  0.95008842, -0.15135721],
           [-0.10321885,  0.4105985 ,  0.14404357,  1.45427351],
           [ 0.76103773,  0.12167502,  0.44386323,  0.33367433]])

```python
np.random.normal(loc=0, scale=1, size=[4,4])
```

    array([[ 1.49407907, -0.20515826,  0.3130677 , -0.85409574],
           [-2.55298982,  0.6536186 ,  0.8644362 , -0.74216502],
           [ 2.26975462, -1.45436567,  0.04575852, -0.18718385],
           [ 1.53277921,  1.46935877,  0.15494743,  0.37816252]])

```python
np.random.seed(seed=0)
np.random.normal(loc=0, scale=1, size=[4,4])
```

    array([[ 1.76405235,  0.40015721,  0.97873798,  2.2408932 ],
           [ 1.86755799, -0.97727788,  0.95008842, -0.15135721],
           [-0.10321885,  0.4105985 ,  0.14404357,  1.45427351],
           [ 0.76103773,  0.12167502,  0.44386323,  0.33367433]])

The normal distribution is of course not the only distribution from which you can sample in NumPy; for instance:

+ `np.random.beta()`
+ `np.random.poisson()`
+ `np.random.uniform()`
+ `np.random.zipf()`

For a full list see the NumPy [documentation](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.random.html#distributions).

## Matplotlib

```python
x = np.linspace(start=0, stop=2 * np.pi, num=256, endpoint=True)

y_sin = np.sin(x)
y_cos = np.cos(x)
plt.plot(x, y_sin)
plt.plot(x, y_cos)
plt.show()
```

```python
plt.plot(x, y_cos)
plt.plot(x, y_sin)

plt.ylim(-1.1, 1.1)
plt.xlim(0, 2 * np.pi)

plt.xticks([0, np.pi/2, np.pi, 3*np.pi/2, 2*np.pi],
           [r'$0$', r'$+\pi/2$', r'$+\pi$', r'$3\pi/2$', r'$2\pi$' ])
plt.yticks([-1, 0, +1], [r'$-1$', r'$0$', r'$+1$'])
plt.show()
```

```python
plt.plot(x, y_cos, linewidth=3, linestyle='--')  # <-- thicker line and linestyle
plt.plot(x, y_sin, color='red')  # <-- different color

plt.ylim(-1.1, 1.1)
plt.xlim(0, 2*np.pi)

plt.xticks([0, np.pi/2, np.pi, 3*np.pi/2, 2*np.pi],
           [r'$0$', r'$+\pi/2$', r'$+\pi$', r'$3\pi/2$', r'$2\pi$' ])
plt.yticks([-1, 0, +1], [r'$-1$', r'$0$', r'$+1$'])
# plt.savefig("sincos.png", dpi=600, transparent=True)
plt.show()
```

```python
plt.plot(x, y_cos, linewidth=3, linestyle='--', label=r'$\cos x$')
plt.plot(x, y_sin, color='red', label=r'$\sin x$')
plt.ylim(-1.1, 1.2)
plt.xlim(0, 2*np.pi)
plt.xticks([0, np.pi/2, np.pi, 3*np.pi/2, 2*np.pi],
       [r'$0$', r'$+\pi/2$', r'$+\pi$', r'$3\pi/2$', r'$2\pi$' ])
plt.yticks([-1, 0, +1],[r'$-1$', r'$0$', r'$+1$'])
t = 2 * np.pi / 3
plt.scatter([t,],[np.sin(t),], 50, color ='red')
plt.annotate(r'$\sin(\frac{2\pi}{3})=\frac{\sqrt{3}}{2}$',
             xy=(t, np.sin(t)),
             xycoords='data',
             xytext=(+40, 10),
             textcoords='offset points',
             fontsize=16,
             arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2", edgecolor='black'))
plt.title("All the bells and whistles!")
plt.show()
```

The `plt` module contains everything you need to create and adjust plots!

For instance:
* `plt.plot`
* `plt.scatter`
* `plt.hist`
* `plt.xlabel`
* `plt.colorbar`
* `plt.savefig`

Typically, 90% of all plots are done with `plt.plot`, `plt.scatter` or `plt.hist`. The documentation allows for much more, but I hardly find the need to do much more.

## Exercise

Play around with matplotlib and create a few plots with functions from NumPy (the random distributions make nice plots!). If you don't know how to create the function, try looking for it in the NumPy documentation.

## Exercise

Here's a nice puzzle:

> There's a wind blowing while a plane is flying a return flight. The wind direction is parallel to the flight direction. Does a round-trip by plane take more time, less time, or the same time?

Let's go and use full-on math for this one. Denote $v$ the speed of the airplane and let $w$ be the effect on the speed caused by the wind. Then the time to get from **A** to **B** (with the wind) and the time to get from **B** to **A** (against the wind) is defined via:

\begin{align}
t_{AB}& = \frac{d}{v+w}\\
t_{BA}& = \frac{d}{v-w}
\end{align}

Notice then that the total time needed make a round trip becomes:

\begin{align}
T  = t_{AB} + t_{BA}& = \frac{d}{v+w} + \frac{d}{v-w}\\
& = d \frac{(v-w)+(v+w)}{(v+w)(v-w)}\\
& = 2d \frac{v}{v^2-w^2}\\
\end{align}

Let's go and plot this as a function $T(w)$ of the wind speed:

```python
w = np.linspace(start=-99, stop=99, num=2000)
v = 100
d = 1200
t = 2 * d * v / (v**2 - w**2) # ** is the exponential operator
```

```python
plt.plot(w, t)
plt.xlim(-105, 105)
plt.title("Effect of wind on total travel time")
```

When the magnitude of $v$ approaches that of $w$ the traveling time goes to infinity, which makes sense.

The main point though, is that our single line of code that is calculating all this is very elegant:

    t = 2 * d * v / (v**2 - w**2)

If you are used to working with maths the vectorized operations aren't just very performant, they are also very elegant in writing.

## Simulation

NumPy has great support for random number generation. Here're a few results from randomly-generated sequences:

### Uniform distribution

```python
rns = np.random.uniform(low=0, high=1, size=100000)
plt.hist(rns, color="steelblue", bins=10)
plt.show()
```

### Normal distribution

```python
rns = np.random.normal(loc=0, scale=1, size=10000)
plt.hist(rns, color="steelblue", bins=100)
plt.show()
```

### Random integer distribution

```python
rns = np.random.randint(low=1, high=4, size=10000)
plt.hist(rns, color="steelblue", bins=5)
plt.show()
```

```python
rns = np.random.randint(low=1, high=4, size=10000)
plt.hist(rns, color="steelblue", bins=7, range=(0.25, 3.75))
plt.show()
```

### Weighted integer sample

```python
rns = np.random.choice(a=[4, 5, 6], size=10000, replace=True, p=[0.1, 0.1, 0.8])
plt.hist(rns, color="steelblue", bins=7, range=(3.25, 6.75))
plt.show()
```

### Binomial distribution

```python
rns = np.random.binomial(n=1, p=0.33, size=10000)
plt.hist(rns, color="steelblue", bins=5, range=[-0.75, 1.75])
plt.show()
```

```python
rns = np.random.binomial(10, 0.33, 10000)
plt.hist(rns, color="steelblue", bins=10)
plt.show()
```

### Poisson distribution

```python
rns = np.random.poisson(2, 10000)
plt.hist(rns, color="steelblue", bins=10)
plt.show()
```

### NumPy + matplotlib Exercises

Here is a small list of some assignments. You may want to consult the numpy documentation to see if there are helper functions that can make your life a lot easier.

1. Create 1000 random uniform numbers and find out what the largest number is.
2. Estimate the probability that a random standard normal value lies between 0.1 and 0.5.
3. Calculate the mean of 1000 random values. Is this what you expected? Also repeat for 10, 100 and 10000. Preferably make a nice function.
4. Use simulation to estimate the probability that two people in this room have the same birthday. Where does the probability cross the 50% mark?

#### Solution to (1)

```python
rns = np.random.uniform(low=0, high=1, size=1000)
rns.max(), rns.argmax()
```

    (0.99923088350172717, 309)

#### Solution to (2)

```python
rns = np.random.uniform(low=0, high=1, size=100000)
rns.size
```

    100000

```python
rns > 0.1
```

    array([ True,  True,  True, ...,  True,  True,  True], dtype=bool)

```python
(rns > 0.1).sum()
```

    89878

```python
(rns < 0.5).sum()
```

    50131

```python
((rns > 0.1) & (rns < 0.5)).sum()
```

    40009

```python
def calc_prob(num, rand_alg = np.random.normal, lower=0.1, upper=0.5):
    rand_vals = rand_alg(size=num)
    lower_predicate = (rand_vals > lower)
    upper_predicate = (rand_vals < upper)
    return np.mean(lower_predicate & upper_predicate)
```

```python
calc_prob(10000, rand_alg=np.random.uniform)
```

    0.40260000000000001

```python
calc_prob(10000, rand_alg=np.random.normal)
```

    0.14729999999999999

#### Solution to (3)

```python
rns = np.random.uniform(low=0, high=1, size=100000)
rns.mean()
```

    0.49897521419284974

```python
rns = np.random.poisson(3, size=100000)
rns.mean()
```

    3.0139900000000002

```python
rns = np.random.normal(loc=0, scale=1, size=100000)
rns.mean()
```

    0.0024206829066252896

```python
def prob_mean(dist, n):
    return dist(size=n).mean()
```

```python
prob_mean(np.random.poisson, 10000)
```

    0.99450000000000005

#### Solution to (4)

```python
def simulate_overlap(num_people=14):
    birthdays = np.random.randint(1, 366, size=num_people)
    return len(np.unique(birthdays)) != num_people

def est_prob(num_people, n_sim=1000):
    return np.mean( [simulate_overlap(num_people) for _ in range(n_sim)] )
```

```python
x = np.arange(100)
y = [est_prob(num_people, n_sim=10000) for num_people in x]
```

```python
plt.plot(x, y);
```

For larger values of `n_sims` we expect to be closer to the true value and to have a smoother curve. We can see that simulation can deliver an adequate answer if the problem is hard to derive, mathematically.

## pandas

NumPy is very performant and useful but it doesn't resemble a sheet of data with rows and columns as much as we'd like. It's more like an array or a matrix. This is fine for the more science-y applications but it is less ideal for the applications where we are analyzing a data sheet that you've been handed which can contain categorical and temporal data.

Back in the day before code, this meant that you needed to use your mouse and click away in Excel.

Thankfully, we have pandas and Python nowadays. The syntax of pandas is mostly similar to NumPy, so let's get familiar with it.

```python
random_numbers = np.random.randn(10)
range_series = pd.Series(data=random_numbers)
range_series
```

    0   -0.576753
    1    1.589395
    2    1.250364
    3   -1.138342
    4   -1.184932
    5   -0.552428
    6    0.361902
    7   -1.795201
    8    0.979794
    9    0.171570
    dtype: float64

Notice that a Series is just like our NumPy array but that we get an Index attached:

```python
range_series.index
```

    RangeIndex(start=0, stop=10, step=1)

In this case the Index is numeric, but it could even be a string or a date:

```python
dates = pd.date_range(start='01/02/2018', periods=7)
dates
```

    DatetimeIndex(['2018-01-02', '2018-01-03', '2018-01-04', '2018-01-05',
                   '2018-01-06', '2018-01-07', '2018-01-08'],
                  dtype='datetime64[ns]', freq='D')

```python
from datetime import date

start_date = date(year=2018, month=2, day=1)
type(start_date)
```

    datetime.date

```python
dates = pd.date_range(start=start_date, periods=7)
dates
```

    DatetimeIndex(['2018-02-01', '2018-02-02', '2018-02-03', '2018-02-04',
                   '2018-02-05', '2018-02-06', '2018-02-07'],
                  dtype='datetime64[ns]', freq='D')

```python
random_numbers = np.random.randn(7)
date_series = pd.Series(data=random_numbers, index=dates)
date_series
```

    2018-02-01    1.327079
    2018-02-02    0.168353
    2018-02-03    0.509586
    2018-02-04   -0.769438
    2018-02-05   -1.468650
    2018-02-06   -1.558707
    2018-02-07    0.323132
    Freq: D, dtype: float64

The default frequency daily, but we can change that:

```python
weekly_dates = pd.date_range(start=start_date, periods=52, freq='W')
weekly_dates[:10]
```

    DatetimeIndex(['2018-02-04', '2018-02-11', '2018-02-18', '2018-02-25',
                   '2018-03-04', '2018-03-11', '2018-03-18', '2018-03-25',
                   '2018-04-01', '2018-04-08'],
                  dtype='datetime64[ns]', freq='W-SUN')

```python
weekly_dates = pd.date_range(start=start_date, periods=52, freq='7D')
weekly_dates[:10]
```

    DatetimeIndex(['2018-02-01', '2018-02-08', '2018-02-15', '2018-02-22',
                   '2018-03-01', '2018-03-08', '2018-03-15', '2018-03-22',
                   '2018-03-29', '2018-04-05'],
                  dtype='datetime64[ns]', freq='7D')

```python
from datetime import timedelta

offset = timedelta(days=3)
weekly_dates + offset
```

    DatetimeIndex(['2018-02-04', '2018-02-11', '2018-02-18', '2018-02-25',
                   '2018-03-04', '2018-03-11', '2018-03-18', '2018-03-25',
                   '2018-04-01', '2018-04-08', '2018-04-15', '2018-04-22',
                   '2018-04-29', '2018-05-06', '2018-05-13', '2018-05-20',
                   '2018-05-27', '2018-06-03', '2018-06-10', '2018-06-17',
                   '2018-06-24', '2018-07-01', '2018-07-08', '2018-07-15',
                   '2018-07-22', '2018-07-29', '2018-08-05', '2018-08-12',
                   '2018-08-19', '2018-08-26', '2018-09-02', '2018-09-09',
                   '2018-09-16', '2018-09-23', '2018-09-30', '2018-10-07',
                   '2018-10-14', '2018-10-21', '2018-10-28', '2018-11-04',
                   '2018-11-11', '2018-11-18', '2018-11-25', '2018-12-02',
                   '2018-12-09', '2018-12-16', '2018-12-23', '2018-12-30',
                   '2019-01-06', '2019-01-13', '2019-01-20', '2019-01-27'],
                  dtype='datetime64[ns]', freq='W-SUN')

```python
from datetime import datetime

start_datetime = datetime(year=2018, month=2, day=1, hour=9, minute=0, second=0)
type(start_datetime)
```

    datetime.datetime

```python
hourly_datetimes = pd.date_range(start=start_datetime, periods=72, freq='H')
hourly_datetimes[:10]
```

    DatetimeIndex(['2018-02-01 09:00:00', '2018-02-01 10:00:00',
                   '2018-02-01 11:00:00', '2018-02-01 12:00:00',
                   '2018-02-01 13:00:00', '2018-02-01 14:00:00',
                   '2018-02-01 15:00:00', '2018-02-01 16:00:00',
                   '2018-02-01 17:00:00', '2018-02-01 18:00:00'],
                  dtype='datetime64[ns]', freq='H')

There are many useful frequency options, see [here](https://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases).

As long as it is sortable and unique for each row, it can be used as an Index.

Pandas Series can be treated in very much the same way as a NumPy array:

```python
range_series[4]
```

    -1.184932370947223

```python
range_series[4:7]
```

    4   -1.184932
    5   -0.552428
    6    0.361902
    dtype: float64

```python
date_series['2018-02-04']
```

    -0.76943832647710497

```python
date_series['2018-02-04':'2018-02-07']
```

    2018-02-04   -0.769438
    2018-02-05   -1.468650
    2018-02-06   -1.558707
    2018-02-07    0.323132
    Freq: D, dtype: float64

Note that for DatetimeIndices both endpoints are included in a slice.

```python
normal_series = pd.Series(data=np.random.normal(loc=0, scale=1, size=10000))
normal_series.mean(), normal_series.std()
```

    (-0.0036489664047400384, 0.99369952796654792)

Just like NumPy arrays, each Series must contain values of only one type:

```python
normal_series.dtype
```

    dtype('float64')

 If we wanted a data structure consisting of multiple Series of the same length but of different data types, one thing we could try is a Python dict:

```python
a = np.random.randn(10)
b = np.random.randint(low=0, high=10, size=10)
c = [ 'foo' if x > 0.5 else 'bar' for x in np.random.rand(10) ]

d = {'a': a, 'b': b, 'c': c}
d
```

    {'a': array([-0.8122872 ,  1.96108308,  0.78628605, -0.09426957,  0.5544281 ,
            -0.08118371,  0.23068922, -1.06895393, -0.27336916, -1.06171749]),
     'b': array([5, 4, 9, 5, 7, 3, 3, 3, 3, 3]),
     'c': ['bar', 'foo', 'bar', 'foo', 'bar', 'bar', 'bar', 'bar', 'bar', 'foo']}

Pandas can convert these into a table-like structure called a DataFrame. A pandas DataFrame can be seen as a collection of Series where each Series is a column in the DataFrame. The Index of the Series represents the row number.

This DataFrame object will be the main object you will talk to when using pandas.

```python
df = pd.DataFrame(data=d)
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
      <th>c</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_dates = pd.DataFrame(
    data=d,
    index=pd.date_range(date(2018, 2, 1), periods=10)
)

df_dates
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
      <th>c</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2018-02-01</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-02</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2018-02-03</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-04</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2018-02-05</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-06</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-07</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-08</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-09</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>2018-02-10</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_dates.loc['2018-02-02']
```

    a    1.96108
    b          4
    c        foo
    Name: 2018-02-02 00:00:00, dtype: object

## DataFrame properties

```python
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
      <th>c</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.head(8)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
      <th>c</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.head().T
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>a</th>
      <td>-0.812287</td>
      <td>1.96108</td>
      <td>0.786286</td>
      <td>-0.0942696</td>
      <td>0.554428</td>
    </tr>
    <tr>
      <th>b</th>
      <td>5</td>
      <td>4</td>
      <td>9</td>
      <td>5</td>
      <td>7</td>
    </tr>
    <tr>
      <th>c</th>
      <td>bar</td>
      <td>foo</td>
      <td>bar</td>
      <td>foo</td>
      <td>bar</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.shape
```

    (10, 3)

```python
df.index
```

    RangeIndex(start=0, stop=10, step=1)

```python
df.dtypes
```

    a    float64
    b      int64
    c     object
    dtype: object

```python
df.describe()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
      <th>b</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>10.000000</td>
      <td>10.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>0.014071</td>
      <td>4.500000</td>
    </tr>
    <tr>
      <th>std</th>
      <td>0.933491</td>
      <td>2.068279</td>
    </tr>
    <tr>
      <th>min</th>
      <td>-1.068954</td>
      <td>3.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>-0.677558</td>
      <td>3.000000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>-0.087727</td>
      <td>3.500000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>0.473493</td>
      <td>5.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>1.961083</td>
      <td>9.000000</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.columns
```

    Index(['a', 'b', 'c'], dtype='object')

```python
df.columns = ['normal', 'uniform', 'foo_bar']
```

```python
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>uniform</th>
      <th>foo_bar</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
    </tr>
  </tbody>
</table>
</div>

## Columnar operations

```python
df['normal']
```

    0   -0.812287
    1    1.961083
    2    0.786286
    3   -0.094270
    4    0.554428
    5   -0.081184
    6    0.230689
    7   -1.068954
    8   -0.273369
    9   -1.061717
    Name: normal, dtype: float64

```python
type(df['normal'])
```

    pandas.core.series.Series

```python
df['normal'].sum()
```

    0.14070539625244338

```python
df['normal'].mean(), df['normal'].std()
```

    (0.014070539625244339, 0.93349082617896373)

```python
df['normal'].hist(bins=5)
```

```python
df[['normal','uniform']]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>uniform</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
</div>

### Adding a column

```python
df['shifted_normal'] = 1 + df['normal']
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>uniform</th>
      <th>foo_bar</th>
      <th>shifted_normal</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
      <td>0.187713</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
      <td>2.961083</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
      <td>1.786286</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
      <td>0.905730</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
      <td>1.554428</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
      <td>0.918816</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
      <td>1.230689</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
      <td>-0.068954</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
      <td>0.726631</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
      <td>-0.061717</td>
    </tr>
  </tbody>
</table>
</div>

```python
df['weighted_normal'] = df['normal'] * df['uniform']
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>uniform</th>
      <th>foo_bar</th>
      <th>shifted_normal</th>
      <th>weighted_normal</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
      <td>0.187713</td>
      <td>-4.061436</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
      <td>2.961083</td>
      <td>7.844332</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
      <td>1.786286</td>
      <td>7.076574</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
      <td>0.905730</td>
      <td>-0.471348</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
      <td>1.554428</td>
      <td>3.880997</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
      <td>0.918816</td>
      <td>-0.243551</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
      <td>1.230689</td>
      <td>0.692068</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
      <td>-0.068954</td>
      <td>-3.206862</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
      <td>0.726631</td>
      <td>-0.820107</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
      <td>-0.061717</td>
      <td>-3.185152</td>
    </tr>
  </tbody>
</table>
</div>

```python
def is_positive(x):
    return x > 0
```

```python
df['is_positive'] = df['normal'].apply(is_positive)
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>uniform</th>
      <th>foo_bar</th>
      <th>shifted_normal</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>5</td>
      <td>bar</td>
      <td>0.187713</td>
      <td>-4.061436</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>4</td>
      <td>foo</td>
      <td>2.961083</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>9</td>
      <td>bar</td>
      <td>1.786286</td>
      <td>7.076574</td>
      <td>True</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>5</td>
      <td>foo</td>
      <td>0.905730</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>7</td>
      <td>bar</td>
      <td>1.554428</td>
      <td>3.880997</td>
      <td>True</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>3</td>
      <td>bar</td>
      <td>0.918816</td>
      <td>-0.243551</td>
      <td>False</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>3</td>
      <td>bar</td>
      <td>1.230689</td>
      <td>0.692068</td>
      <td>True</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>3</td>
      <td>bar</td>
      <td>-0.068954</td>
      <td>-3.206862</td>
      <td>False</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>3</td>
      <td>bar</td>
      <td>0.726631</td>
      <td>-0.820107</td>
      <td>False</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>3</td>
      <td>foo</td>
      <td>-0.061717</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

### Drop a column

```python
df = df.drop(['shifted_normal', 'uniform'], axis=1)
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>bar</td>
      <td>-4.061436</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>bar</td>
      <td>7.076574</td>
      <td>True</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>foo</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>bar</td>
      <td>3.880997</td>
      <td>True</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>bar</td>
      <td>-0.243551</td>
      <td>False</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>bar</td>
      <td>0.692068</td>
      <td>True</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>bar</td>
      <td>-3.206862</td>
      <td>False</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>bar</td>
      <td>-0.820107</td>
      <td>False</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>foo</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

### Select only rows meeting some criterion:

```python
df['normal'] < 0
```

    0     True
    1    False
    2    False
    3     True
    4    False
    5     True
    6    False
    7     True
    8     True
    9     True
    Name: normal, dtype: bool

```python
df[ df['normal'] < 0 ]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>bar</td>
      <td>-4.061436</td>
      <td>False</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>foo</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>bar</td>
      <td>-0.243551</td>
      <td>False</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>bar</td>
      <td>-3.206862</td>
      <td>False</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>bar</td>
      <td>-0.820107</td>
      <td>False</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>foo</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

```python
df[ (df['normal'] > 0) & (df['foo_bar'] == 'foo') ]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
  </tbody>
</table>
</div>

```python
df[ (df['normal'] > 0) | (df['foo_bar'] == 'foo') ]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>bar</td>
      <td>7.076574</td>
      <td>True</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>foo</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>bar</td>
      <td>3.880997</td>
      <td>True</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>bar</td>
      <td>0.692068</td>
      <td>True</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>foo</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

```python
df['normal'].max()
```

    1.9610830828815413

```python
df['normal'] == df['normal'].max()
```

    0    False
    1     True
    2    False
    3    False
    4    False
    5    False
    6    False
    7    False
    8    False
    9    False
    Name: normal, dtype: bool

```python
df[ df['normal'] == df['normal'].max() ]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
  </tbody>
</table>
</div>

### Saving a DataFrame

```python
df.to_csv('data/random_df.csv', sep=',', header=True, index=False)
```

## Getting data into a DataFrame

### CSV

```python
df_random = pd.read_csv('data/random_df.csv')
df_random
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>bar</td>
      <td>-4.061436</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>bar</td>
      <td>7.076574</td>
      <td>True</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>foo</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>bar</td>
      <td>3.880997</td>
      <td>True</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>bar</td>
      <td>-0.243551</td>
      <td>False</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>bar</td>
      <td>0.692068</td>
      <td>True</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>bar</td>
      <td>-3.206862</td>
      <td>False</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>bar</td>
      <td>-0.820107</td>
      <td>False</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>foo</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_random.dtypes
```

    normal             float64
    foo_bar             object
    weighted_normal    float64
    is_positive           bool
    dtype: object

```python
df_random = pd.read_csv('data/random_df.csv', header=0, sep=',')
df_random
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>normal</th>
      <th>foo_bar</th>
      <th>weighted_normal</th>
      <th>is_positive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>-0.812287</td>
      <td>bar</td>
      <td>-4.061436</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.961083</td>
      <td>foo</td>
      <td>7.844332</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.786286</td>
      <td>bar</td>
      <td>7.076574</td>
      <td>True</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.094270</td>
      <td>foo</td>
      <td>-0.471348</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.554428</td>
      <td>bar</td>
      <td>3.880997</td>
      <td>True</td>
    </tr>
    <tr>
      <th>5</th>
      <td>-0.081184</td>
      <td>bar</td>
      <td>-0.243551</td>
      <td>False</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.230689</td>
      <td>bar</td>
      <td>0.692068</td>
      <td>True</td>
    </tr>
    <tr>
      <th>7</th>
      <td>-1.068954</td>
      <td>bar</td>
      <td>-3.206862</td>
      <td>False</td>
    </tr>
    <tr>
      <th>8</th>
      <td>-0.273369</td>
      <td>bar</td>
      <td>-0.820107</td>
      <td>False</td>
    </tr>
    <tr>
      <th>9</th>
      <td>-1.061717</td>
      <td>foo</td>
      <td>-3.185152</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

`read_csv()` has tons of options to help navigate the morass of CSV formatting: [check them out](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html).

### JSON

```python
df_generated = pd.read_json('data/generated.json')
```

```python
df_generated.head().T
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>_id</th>
      <td>5a70cfbc45459ab8b01b0772</td>
      <td>5a70cfbca8acc89ddf73fe58</td>
      <td>5a70cfbca6467b1911ba63f5</td>
      <td>5a70cfbc5f072e9d911e49df</td>
      <td>5a70cfbc4d421b6fdbf78baf</td>
    </tr>
    <tr>
      <th>about</th>
      <td>Ullamco ex dolor aliqua dolor duis esse ipsum ...</td>
      <td>Mollit ex Lorem in nisi fugiat mollit irure ei...</td>
      <td>Laborum ea dolor cupidatat proident cillum est...</td>
      <td>Quis adipisicing aute elit dolor pariatur dolo...</td>
      <td>Exercitation sunt ullamco labore velit aute cu...</td>
    </tr>
    <tr>
      <th>address</th>
      <td>726 Lincoln Terrace, Cowiche, Alabama, 9320</td>
      <td>258 Rockwell Place, Osage, Puerto Rico, 549</td>
      <td>491 Dennett Place, Downsville, Washington, 5649</td>
      <td>818 Cooper Street, Forbestown, Maryland, 8259</td>
      <td>322 Lawton Street, Coleville, Ohio, 3802</td>
    </tr>
    <tr>
      <th>age</th>
      <td>40</td>
      <td>34</td>
      <td>37</td>
      <td>33</td>
      <td>32</td>
    </tr>
    <tr>
      <th>balance</th>
      <td>$3,635.14</td>
      <td>$1,078.48</td>
      <td>$3,602.10</td>
      <td>$3,052.70</td>
      <td>$2,986.41</td>
    </tr>
    <tr>
      <th>company</th>
      <td>BESTO</td>
      <td>TOURMANIA</td>
      <td>ENERSAVE</td>
      <td>VIOCULAR</td>
      <td>TECHADE</td>
    </tr>
    <tr>
      <th>email</th>
      <td>corrinepugh@besto.com</td>
      <td>stantonkelley@tourmania.com</td>
      <td>dollienelson@enersave.com</td>
      <td>deenaschmidt@viocular.com</td>
      <td>gaygriffith@techade.com</td>
    </tr>
    <tr>
      <th>eyeColor</th>
      <td>green</td>
      <td>blue</td>
      <td>brown</td>
      <td>brown</td>
      <td>green</td>
    </tr>
    <tr>
      <th>favoriteFruit</th>
      <td>strawberry</td>
      <td>strawberry</td>
      <td>apple</td>
      <td>apple</td>
      <td>strawberry</td>
    </tr>
    <tr>
      <th>friends</th>
      <td>[{'id': 0, 'name': 'Hannah Pacheco'}, {'id': 1...</td>
      <td>[{'id': 0, 'name': 'Luna Campos'}, {'id': 1, '...</td>
      <td>[{'id': 0, 'name': 'Crystal Sloan'}, {'id': 1,...</td>
      <td>[{'id': 0, 'name': 'Rice Keith'}, {'id': 1, 'n...</td>
      <td>[{'id': 0, 'name': 'Harding Thompson'}, {'id':...</td>
    </tr>
    <tr>
      <th>gender</th>
      <td>female</td>
      <td>male</td>
      <td>female</td>
      <td>female</td>
      <td>male</td>
    </tr>
    <tr>
      <th>greeting</th>
      <td>Hello, Corrine Pugh! You have 3 unread messages.</td>
      <td>Hello, Stanton Kelley! You have 4 unread messa...</td>
      <td>Hello, Dollie Nelson! You have 7 unread messages.</td>
      <td>Hello, Deena Schmidt! You have 5 unread messages.</td>
      <td>Hello, Gay Griffith! You have 2 unread messages.</td>
    </tr>
    <tr>
      <th>guid</th>
      <td>fdd155b6-5d61-4ae0-a912-d11533c94e9c</td>
      <td>789e1e8e-b093-4182-a482-0d206a9b26ec</td>
      <td>6ebc5c58-e9d9-467a-845f-b13229c5a505</td>
      <td>48f596dd-c440-4940-8f53-9dc17a4d8b8f</td>
      <td>a46324b5-ff81-4b2c-8136-dc5adb830adc</td>
    </tr>
    <tr>
      <th>index</th>
      <td>0</td>
      <td>1</td>
      <td>2</td>
      <td>3</td>
      <td>4</td>
    </tr>
    <tr>
      <th>isActive</th>
      <td>True</td>
      <td>True</td>
      <td>True</td>
      <td>False</td>
      <td>False</td>
    </tr>
    <tr>
      <th>latitude</th>
      <td>64.8811</td>
      <td>10.1817</td>
      <td>-49.7484</td>
      <td>-10.1333</td>
      <td>44.3526</td>
    </tr>
    <tr>
      <th>longitude</th>
      <td>152.11</td>
      <td>22.472</td>
      <td>-37.1001</td>
      <td>89.5925</td>
      <td>101.846</td>
    </tr>
    <tr>
      <th>name</th>
      <td>Corrine Pugh</td>
      <td>Stanton Kelley</td>
      <td>Dollie Nelson</td>
      <td>Deena Schmidt</td>
      <td>Gay Griffith</td>
    </tr>
    <tr>
      <th>phone</th>
      <td>+1 (856) 523-2643</td>
      <td>+1 (974) 491-2133</td>
      <td>+1 (883) 489-3713</td>
      <td>+1 (880) 417-3391</td>
      <td>+1 (964) 583-3859</td>
    </tr>
    <tr>
      <th>picture</th>
      <td>http://placehold.it/32x32</td>
      <td>http://placehold.it/32x32</td>
      <td>http://placehold.it/32x32</td>
      <td>http://placehold.it/32x32</td>
      <td>http://placehold.it/32x32</td>
    </tr>
    <tr>
      <th>registered</th>
      <td>2016-03-08T10:47:14 -01:00</td>
      <td>2017-10-23T03:42:05 -02:00</td>
      <td>2014-06-24T08:53:19 -02:00</td>
      <td>2014-09-15T05:32:02 -02:00</td>
      <td>2015-10-10T03:17:41 -02:00</td>
    </tr>
    <tr>
      <th>tags</th>
      <td>[duis, excepteur, tempor, aliqua, proident, am...</td>
      <td>[adipisicing, commodo, enim, aliqua, do, quis,...</td>
      <td>[mollit, nulla, occaecat, aliqua, ad, minim, o...</td>
      <td>[ullamco, incididunt, deserunt, ipsum, veniam,...</td>
      <td>[duis, quis, proident, qui, quis, velit, labore]</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_generated.dtypes
```

    _id               object
    about             object
    address           object
    age                int64
    balance           object
    company           object
    email             object
    eyeColor          object
    favoriteFruit     object
    friends           object
    gender            object
    greeting          object
    guid              object
    index              int64
    isActive            bool
    latitude         float64
    longitude        float64
    name              object
    phone             object
    picture           object
    registered        object
    tags              object
    dtype: object

```python
df_generated['registered']
```

    0    2016-03-08T10:47:14 -01:00
    1    2017-10-23T03:42:05 -02:00
    2    2014-06-24T08:53:19 -02:00
    3    2014-09-15T05:32:02 -02:00
    4    2015-10-10T03:17:41 -02:00
    5    2014-06-23T02:54:36 -02:00
    6    2016-03-30T01:18:36 -02:00
    Name: registered, dtype: object

```python
df_generated = pd.read_json('data/generated.json', convert_dates=['registered',])
```

```python
df_generated.dtypes
```

    _id                      object
    about                    object
    address                  object
    age                       int64
    balance                  object
    company                  object
    email                    object
    eyeColor                 object
    favoriteFruit            object
    friends                  object
    gender                   object
    greeting                 object
    guid                     object
    index                     int64
    isActive                   bool
    latitude                float64
    longitude               float64
    name                     object
    phone                    object
    picture                  object
    registered       datetime64[ns]
    tags                     object
    dtype: object

```python
df_generated['registered']
```

    0   2016-03-08 11:47:14
    1   2017-10-23 05:42:05
    2   2014-06-24 10:53:19
    3   2014-09-15 07:32:02
    4   2015-10-10 05:17:41
    5   2014-06-23 04:54:36
    6   2016-03-30 03:18:36
    Name: registered, dtype: datetime64[ns]

```python
df_generated['balance']
```

    0    $3,635.14
    1    $1,078.48
    2    $3,602.10
    3    $3,052.70
    4    $2,986.41
    5    $1,738.60
    6    $3,907.58
    Name: balance, dtype: object

```python
x = '$3,635.14'
float(x[1:].replace(',', ''))
```

    3635.14

```python
def convert_balance_to_float(balance):
    return float(balance[1:].replace(',', ''))
```

```python
df_generated['float_balance'] = df_generated['balance'].apply(convert_balance_to_float)
```

```python
df_generated['float_balance']
```

    0    3635.14
    1    1078.48
    2    3602.10
    3    3052.70
    4    2986.41
    5    1738.60
    6    3907.58
    Name: float_balance, dtype: float64

## Exercise:

Add four new columns to `df_generated`: `street` (includes hous number and street name), `city`, `state`, and `postcode` based on the `address` column. Do so using the `apply` method. Make sure that the `dtype` for the `postcode` is `int`.

Advanced: split `street` into `house_number` and `street_name`, again ensuring that `house_number` is of `dtype` `int`.

## Split-Apply-Combine

A moment of abstraction.

Very often we will want to perform group operations on data. In a financial example we might want to perform an operation for every week or for every month of data. In another example we might want to apply methods per geolocation/type of person/per website, etc.

Whenever we are doing such an operation, we might look at it as a ```split-apply-combine``` operation, shown visually below:

pandas has great support for these kinds of operations. Based on the key of a DataFrame we will split the data and then apply functions to the grouped data.

## Enter ChickWeight

For this next bit we will return to the ChickWeight dataset. This dataset contains information about different diets for chickens. The goal of the dataset is to find out which diet will get the chickens as fat as possible.

```python
df_chickweight = pd.read_csv('data/chickweight.csv')
# data: http://koaning.s3-website-us-west-2.amazonaws.com/data/pydata/chickweight.csv
df_chickweight.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>rownum</th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>42</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>51</td>
      <td>2</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>59</td>
      <td>4</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>64</td>
      <td>6</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>76</td>
      <td>8</td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_chickweight = pd.read_csv('data/chickweight.csv', index_col=0)
df_chickweight
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
    </tr>
    <tr>
      <th>rownum</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>42</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>51</td>
      <td>2</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>59</td>
      <td>4</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>64</td>
      <td>6</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>76</td>
      <td>8</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>6</th>
      <td>93</td>
      <td>10</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>106</td>
      <td>12</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>125</td>
      <td>14</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>9</th>
      <td>149</td>
      <td>16</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>10</th>
      <td>171</td>
      <td>18</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>11</th>
      <td>199</td>
      <td>20</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>12</th>
      <td>205</td>
      <td>21</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>13</th>
      <td>40</td>
      <td>0</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>14</th>
      <td>49</td>
      <td>2</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>15</th>
      <td>58</td>
      <td>4</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>16</th>
      <td>72</td>
      <td>6</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>17</th>
      <td>84</td>
      <td>8</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>18</th>
      <td>103</td>
      <td>10</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>19</th>
      <td>122</td>
      <td>12</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>20</th>
      <td>138</td>
      <td>14</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>21</th>
      <td>162</td>
      <td>16</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>22</th>
      <td>187</td>
      <td>18</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>23</th>
      <td>209</td>
      <td>20</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>24</th>
      <td>215</td>
      <td>21</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <th>25</th>
      <td>43</td>
      <td>0</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>26</th>
      <td>39</td>
      <td>2</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>27</th>
      <td>55</td>
      <td>4</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>28</th>
      <td>67</td>
      <td>6</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>29</th>
      <td>84</td>
      <td>8</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>30</th>
      <td>99</td>
      <td>10</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>549</th>
      <td>154</td>
      <td>12</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>550</th>
      <td>170</td>
      <td>14</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>551</th>
      <td>222</td>
      <td>16</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>552</th>
      <td>261</td>
      <td>18</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>553</th>
      <td>303</td>
      <td>20</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>554</th>
      <td>322</td>
      <td>21</td>
      <td>48</td>
      <td>4</td>
    </tr>
    <tr>
      <th>555</th>
      <td>40</td>
      <td>0</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>556</th>
      <td>53</td>
      <td>2</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>557</th>
      <td>64</td>
      <td>4</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>558</th>
      <td>85</td>
      <td>6</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>559</th>
      <td>108</td>
      <td>8</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>560</th>
      <td>128</td>
      <td>10</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>561</th>
      <td>152</td>
      <td>12</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>562</th>
      <td>166</td>
      <td>14</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>563</th>
      <td>184</td>
      <td>16</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>564</th>
      <td>203</td>
      <td>18</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>565</th>
      <td>233</td>
      <td>20</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>566</th>
      <td>237</td>
      <td>21</td>
      <td>49</td>
      <td>4</td>
    </tr>
    <tr>
      <th>567</th>
      <td>41</td>
      <td>0</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>568</th>
      <td>54</td>
      <td>2</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>569</th>
      <td>67</td>
      <td>4</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>570</th>
      <td>84</td>
      <td>6</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>571</th>
      <td>105</td>
      <td>8</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>572</th>
      <td>122</td>
      <td>10</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>573</th>
      <td>155</td>
      <td>12</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>574</th>
      <td>175</td>
      <td>14</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>575</th>
      <td>205</td>
      <td>16</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>576</th>
      <td>234</td>
      <td>18</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>577</th>
      <td>264</td>
      <td>20</td>
      <td>50</td>
      <td>4</td>
    </tr>
    <tr>
      <th>578</th>
      <td>264</td>
      <td>21</td>
      <td>50</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
<p>578 rows × 4 columns</p>
</div>

The nice thing about this object is that we can easily summarise it.

```python
df_chickweight.describe()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>578.000000</td>
      <td>578.000000</td>
      <td>578.000000</td>
      <td>578.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>121.818339</td>
      <td>10.717993</td>
      <td>25.750865</td>
      <td>2.235294</td>
    </tr>
    <tr>
      <th>std</th>
      <td>71.071960</td>
      <td>6.758400</td>
      <td>14.568795</td>
      <td>1.162678</td>
    </tr>
    <tr>
      <th>min</th>
      <td>35.000000</td>
      <td>0.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>63.000000</td>
      <td>4.000000</td>
      <td>13.000000</td>
      <td>1.000000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>103.000000</td>
      <td>10.000000</td>
      <td>26.000000</td>
      <td>2.000000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>163.750000</td>
      <td>16.000000</td>
      <td>38.000000</td>
      <td>3.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>373.000000</td>
      <td>21.000000</td>
      <td>50.000000</td>
      <td>4.000000</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_chickweight['Chick'].unique()
```

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
           18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
           35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50])

```python
df_chickweight['Diet'].unique()
```

    array([1, 2, 3, 4])

```python
df_chickweight[ df_chickweight['Chick'] == 1 ]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
    </tr>
    <tr>
      <th>rownum</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>42</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>51</td>
      <td>2</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>59</td>
      <td>4</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>64</td>
      <td>6</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>76</td>
      <td>8</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>6</th>
      <td>93</td>
      <td>10</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>106</td>
      <td>12</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>125</td>
      <td>14</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>9</th>
      <td>149</td>
      <td>16</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>10</th>
      <td>171</td>
      <td>18</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>11</th>
      <td>199</td>
      <td>20</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>12</th>
      <td>205</td>
      <td>21</td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>

We could also do aggregation by using the `groupby` method. It is similar to `GROUP BY` in SQL.

```python
grouped_chickweight = df_chickweight.groupby(["Diet"])
```

```python
for thing in grouped_chickweight:
    print(thing)
```

    (1,         weight  Time  Chick  Diet
    rownum
    1           42     0      1     1
    2           51     2      1     1
    3           59     4      1     1
    4           64     6      1     1
    5           76     8      1     1
    6           93    10      1     1
    7          106    12      1     1
    8          125    14      1     1
    9          149    16      1     1
    10         171    18      1     1
    11         199    20      1     1
    12         205    21      1     1
    13          40     0      2     1
    14          49     2      2     1
    15          58     4      2     1
    16          72     6      2     1
    17          84     8      2     1
    18         103    10      2     1
    19         122    12      2     1
    20         138    14      2     1
    21         162    16      2     1
    22         187    18      2     1
    23         209    20      2     1
    24         215    21      2     1
    25          43     0      3     1
    26          39     2      3     1
    27          55     4      3     1
    28          67     6      3     1
    29          84     8      3     1
    30          99    10      3     1
    ...        ...   ...    ...   ...
    191        113    16     17     1
    192        123    18     17     1
    193        133    20     17     1
    194        142    21     17     1
    195         39     0     18     1
    196         35     2     18     1
    197         43     0     19     1
    198         48     2     19     1
    199         55     4     19     1
    200         62     6     19     1
    201         65     8     19     1
    202         71    10     19     1
    203         82    12     19     1
    204         88    14     19     1
    205        106    16     19     1
    206        120    18     19     1
    207        144    20     19     1
    208        157    21     19     1
    209         41     0     20     1
    210         47     2     20     1
    211         54     4     20     1
    212         58     6     20     1
    213         65     8     20     1
    214         73    10     20     1
    215         77    12     20     1
    216         89    14     20     1
    217         98    16     20     1
    218        107    18     20     1
    219        115    20     20     1
    220        117    21     20     1

    [220 rows x 4 columns])
    (2,         weight  Time  Chick  Diet
    rownum
    221         40     0     21     2
    222         50     2     21     2
    223         62     4     21     2
    224         86     6     21     2
    225        125     8     21     2
    226        163    10     21     2
    227        217    12     21     2
    228        240    14     21     2
    229        275    16     21     2
    230        307    18     21     2
    231        318    20     21     2
    232        331    21     21     2
    233         41     0     22     2
    234         55     2     22     2
    235         64     4     22     2
    236         77     6     22     2
    237         90     8     22     2
    238         95    10     22     2
    239        108    12     22     2
    240        111    14     22     2
    241        131    16     22     2
    242        148    18     22     2
    243        164    20     22     2
    244        167    21     22     2
    245         43     0     23     2
    246         52     2     23     2
    247         61     4     23     2
    248         73     6     23     2
    249         90     8     23     2
    250        103    10     23     2
    ...        ...   ...    ...   ...
    311        145    12     28     2
    312        156    14     28     2
    313        184    16     28     2
    314        207    18     28     2
    315        212    20     28     2
    316        233    21     28     2
    317         39     0     29     2
    318         48     2     29     2
    319         59     4     29     2
    320         74     6     29     2
    321         87     8     29     2
    322        106    10     29     2
    323        134    12     29     2
    324        150    14     29     2
    325        187    16     29     2
    326        230    18     29     2
    327        279    20     29     2
    328        309    21     29     2
    329         42     0     30     2
    330         48     2     30     2
    331         59     4     30     2
    332         72     6     30     2
    333         85     8     30     2
    334         98    10     30     2
    335        115    12     30     2
    336        122    14     30     2
    337        143    16     30     2
    338        151    18     30     2
    339        157    20     30     2
    340        150    21     30     2

    [120 rows x 4 columns])
    (3,         weight  Time  Chick  Diet
    rownum
    341         42     0     31     3
    342         53     2     31     3
    343         62     4     31     3
    344         73     6     31     3
    345         85     8     31     3
    346        102    10     31     3
    347        123    12     31     3
    348        138    14     31     3
    349        170    16     31     3
    350        204    18     31     3
    351        235    20     31     3
    352        256    21     31     3
    353         41     0     32     3
    354         49     2     32     3
    355         65     4     32     3
    356         82     6     32     3
    357        107     8     32     3
    358        129    10     32     3
    359        159    12     32     3
    360        179    14     32     3
    361        221    16     32     3
    362        263    18     32     3
    363        291    20     32     3
    364        305    21     32     3
    365         39     0     33     3
    366         50     2     33     3
    367         63     4     33     3
    368         77     6     33     3
    369         96     8     33     3
    370        111    10     33     3
    ...        ...   ...    ...   ...
    431        128    12     38     3
    432        154    14     38     3
    433        192    16     38     3
    434        232    18     38     3
    435        280    20     38     3
    436        290    21     38     3
    437         42     0     39     3
    438         50     2     39     3
    439         61     4     39     3
    440         78     6     39     3
    441         89     8     39     3
    442        109    10     39     3
    443        130    12     39     3
    444        146    14     39     3
    445        170    16     39     3
    446        214    18     39     3
    447        250    20     39     3
    448        272    21     39     3
    449         41     0     40     3
    450         55     2     40     3
    451         66     4     40     3
    452         79     6     40     3
    453        101     8     40     3
    454        120    10     40     3
    455        154    12     40     3
    456        182    14     40     3
    457        215    16     40     3
    458        262    18     40     3
    459        295    20     40     3
    460        321    21     40     3

    [120 rows x 4 columns])
    (4,         weight  Time  Chick  Diet
    rownum
    461         42     0     41     4
    462         51     2     41     4
    463         66     4     41     4
    464         85     6     41     4
    465        103     8     41     4
    466        124    10     41     4
    467        155    12     41     4
    468        153    14     41     4
    469        175    16     41     4
    470        184    18     41     4
    471        199    20     41     4
    472        204    21     41     4
    473         42     0     42     4
    474         49     2     42     4
    475         63     4     42     4
    476         84     6     42     4
    477        103     8     42     4
    478        126    10     42     4
    479        160    12     42     4
    480        174    14     42     4
    481        204    16     42     4
    482        234    18     42     4
    483        269    20     42     4
    484        281    21     42     4
    485         42     0     43     4
    486         55     2     43     4
    487         69     4     43     4
    488         96     6     43     4
    489        131     8     43     4
    490        157    10     43     4
    ...        ...   ...    ...   ...
    549        154    12     48     4
    550        170    14     48     4
    551        222    16     48     4
    552        261    18     48     4
    553        303    20     48     4
    554        322    21     48     4
    555         40     0     49     4
    556         53     2     49     4
    557         64     4     49     4
    558         85     6     49     4
    559        108     8     49     4
    560        128    10     49     4
    561        152    12     49     4
    562        166    14     49     4
    563        184    16     49     4
    564        203    18     49     4
    565        233    20     49     4
    566        237    21     49     4
    567         41     0     50     4
    568         54     2     50     4
    569         67     4     50     4
    570         84     6     50     4
    571        105     8     50     4
    572        122    10     50     4
    573        155    12     50     4
    574        175    14     50     4
    575        205    16     50     4
    576        234    18     50     4
    577        264    20     50     4
    578        264    21     50     4

    [118 rows x 4 columns])

```python
for diet, df_diet in grouped_chickweight:
    print(diet)
    print(df_diet.head())
    print('-' * 35)
```

    1
            weight  Time  Chick  Diet
    rownum
    1           42     0      1     1
    2           51     2      1     1
    3           59     4      1     1
    4           64     6      1     1
    5           76     8      1     1
    -----------------------------------
    2
            weight  Time  Chick  Diet
    rownum
    221         40     0     21     2
    222         50     2     21     2
    223         62     4     21     2
    224         86     6     21     2
    225        125     8     21     2
    -----------------------------------
    3
            weight  Time  Chick  Diet
    rownum
    341         42     0     31     3
    342         53     2     31     3
    343         62     4     31     3
    344         73     6     31     3
    345         85     8     31     3
    -----------------------------------
    4
            weight  Time  Chick  Diet
    rownum
    461         42     0     41     4
    462         51     2     41     4
    463         66     4     41     4
    464         85     6     41     4
    465        103     8     41     4
    -----------------------------------

```python
grouped_chickweight.describe()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead tr th {
        text-align: left;
    }

    .dataframe thead tr:last-of-type th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr>
      <th></th>
      <th colspan="8" halign="left">Chick</th>
      <th colspan="5" halign="left">Time</th>
      <th colspan="8" halign="left">weight</th>
    </tr>
    <tr>
      <th></th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
      <th>min</th>
      <th>25%</th>
      <th>50%</th>
      <th>75%</th>
      <th>max</th>
      <th>count</th>
      <th>mean</th>
      <th>...</th>
      <th>75%</th>
      <th>max</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
      <th>min</th>
      <th>25%</th>
      <th>50%</th>
      <th>75%</th>
      <th>max</th>
    </tr>
    <tr>
      <th>Diet</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>220.0</td>
      <td>9.963636</td>
      <td>5.700160</td>
      <td>1.0</td>
      <td>5.0</td>
      <td>10.0</td>
      <td>14.0</td>
      <td>20.0</td>
      <td>220.0</td>
      <td>10.481818</td>
      <td>...</td>
      <td>16.0</td>
      <td>21.0</td>
      <td>220.0</td>
      <td>102.645455</td>
      <td>56.656553</td>
      <td>35.0</td>
      <td>57.75</td>
      <td>88.0</td>
      <td>136.50</td>
      <td>305.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>120.0</td>
      <td>25.500000</td>
      <td>2.884324</td>
      <td>21.0</td>
      <td>23.0</td>
      <td>25.5</td>
      <td>28.0</td>
      <td>30.0</td>
      <td>120.0</td>
      <td>10.916667</td>
      <td>...</td>
      <td>16.5</td>
      <td>21.0</td>
      <td>120.0</td>
      <td>122.616667</td>
      <td>71.607495</td>
      <td>39.0</td>
      <td>65.50</td>
      <td>104.5</td>
      <td>163.00</td>
      <td>331.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>120.0</td>
      <td>35.500000</td>
      <td>2.884324</td>
      <td>31.0</td>
      <td>33.0</td>
      <td>35.5</td>
      <td>38.0</td>
      <td>40.0</td>
      <td>120.0</td>
      <td>10.916667</td>
      <td>...</td>
      <td>16.5</td>
      <td>21.0</td>
      <td>120.0</td>
      <td>142.950000</td>
      <td>86.541761</td>
      <td>39.0</td>
      <td>67.50</td>
      <td>125.5</td>
      <td>198.75</td>
      <td>373.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>118.0</td>
      <td>45.525424</td>
      <td>2.902141</td>
      <td>41.0</td>
      <td>43.0</td>
      <td>46.0</td>
      <td>48.0</td>
      <td>50.0</td>
      <td>118.0</td>
      <td>10.754237</td>
      <td>...</td>
      <td>16.0</td>
      <td>21.0</td>
      <td>118.0</td>
      <td>135.262712</td>
      <td>68.828714</td>
      <td>39.0</td>
      <td>71.25</td>
      <td>129.5</td>
      <td>184.75</td>
      <td>322.0</td>
    </tr>
  </tbody>
</table>
<p>4 rows × 24 columns</p>
</div>

### Applying your own

We can use built-in functions on our grouped objects, but we can also just apply our own functions.

Note that these functions need to be able to be applied to a DataFrame object, as long as the function does this, it will work.

```python
def show_size(frame):
    return "Dude we have " + str(frame.shape[0]) + " rows here!"

grouped_chickweight.apply(show_size)
```

    Diet
    1    Dude we have 220 rows here!
    2    Dude we have 120 rows here!
    3    Dude we have 120 rows here!
    4    Dude we have 118 rows here!
    dtype: object

Realize that this means that **any** function can be used here. You will want to think about performance when dealing with large datasets. This functionality is one of the things that make the pandas API very powerful.

### Advanced Groups
We can also create groups based on two columns in the table.

```python
df_chickweight.groupby(['Diet', 'Time']).apply(show_size)
```

    Diet  Time
    1     0       Dude we have 20 rows here!
          2       Dude we have 20 rows here!
          4       Dude we have 19 rows here!
          6       Dude we have 19 rows here!
          8       Dude we have 19 rows here!
          10      Dude we have 19 rows here!
          12      Dude we have 19 rows here!
          14      Dude we have 18 rows here!
          16      Dude we have 17 rows here!
          18      Dude we have 17 rows here!
          20      Dude we have 17 rows here!
          21      Dude we have 16 rows here!
    2     0       Dude we have 10 rows here!
          2       Dude we have 10 rows here!
          4       Dude we have 10 rows here!
          6       Dude we have 10 rows here!
          8       Dude we have 10 rows here!
          10      Dude we have 10 rows here!
          12      Dude we have 10 rows here!
          14      Dude we have 10 rows here!
          16      Dude we have 10 rows here!
          18      Dude we have 10 rows here!
          20      Dude we have 10 rows here!
          21      Dude we have 10 rows here!
    3     0       Dude we have 10 rows here!
          2       Dude we have 10 rows here!
          4       Dude we have 10 rows here!
          6       Dude we have 10 rows here!
          8       Dude we have 10 rows here!
          10      Dude we have 10 rows here!
          12      Dude we have 10 rows here!
          14      Dude we have 10 rows here!
          16      Dude we have 10 rows here!
          18      Dude we have 10 rows here!
          20      Dude we have 10 rows here!
          21      Dude we have 10 rows here!
    4     0       Dude we have 10 rows here!
          2       Dude we have 10 rows here!
          4       Dude we have 10 rows here!
          6       Dude we have 10 rows here!
          8       Dude we have 10 rows here!
          10      Dude we have 10 rows here!
          12      Dude we have 10 rows here!
          14      Dude we have 10 rows here!
          16      Dude we have 10 rows here!
          18      Dude we have 10 rows here!
          20       Dude we have 9 rows here!
          21       Dude we have 9 rows here!
    dtype: object

Looks like some chickens 'disappeared' prematurely.

## pandas & visualisation

```python
df_chickweight.groupby(['Time','Diet'])['weight']
```

    <pandas.core.groupby.SeriesGroupBy object at 0x114f44dd8>

```python
agg = df_chickweight.groupby(['Time','Diet'])['weight'].apply(np.mean)
```

```python
agg.head(10)
```

    Time  Diet
    0     1       41.400000
          2       40.700000
          3       40.800000
          4       41.000000
    2     1       47.250000
          2       49.400000
          3       50.400000
          4       51.800000
    4     1       56.473684
          2       59.800000
    Name: weight, dtype: float64

```python
agg.reset_index().head(10)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Time</th>
      <th>Diet</th>
      <th>weight</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>1</td>
      <td>41.400000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0</td>
      <td>2</td>
      <td>40.700000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0</td>
      <td>3</td>
      <td>40.800000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0</td>
      <td>4</td>
      <td>41.000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>2</td>
      <td>1</td>
      <td>47.250000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>2</td>
      <td>2</td>
      <td>49.400000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2</td>
      <td>3</td>
      <td>50.400000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2</td>
      <td>4</td>
      <td>51.800000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>4</td>
      <td>1</td>
      <td>56.473684</td>
    </tr>
    <tr>
      <th>9</th>
      <td>4</td>
      <td>2</td>
      <td>59.800000</td>
    </tr>
  </tbody>
</table>
</div>

pandas has a wrapper for matplotlib that we can use. So if we quickly want to create a plot of our DataFrame we can simply do it with this one-liner:

```python
agg.unstack()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>Diet</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
    <tr>
      <th>Time</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>41.400000</td>
      <td>40.7</td>
      <td>40.8</td>
      <td>41.000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>47.250000</td>
      <td>49.4</td>
      <td>50.4</td>
      <td>51.800000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>56.473684</td>
      <td>59.8</td>
      <td>62.2</td>
      <td>64.500000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>66.789474</td>
      <td>75.4</td>
      <td>77.9</td>
      <td>83.900000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>79.684211</td>
      <td>91.7</td>
      <td>98.4</td>
      <td>105.600000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>93.052632</td>
      <td>108.5</td>
      <td>117.1</td>
      <td>126.000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>108.526316</td>
      <td>131.3</td>
      <td>144.4</td>
      <td>151.400000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>123.388889</td>
      <td>141.9</td>
      <td>164.5</td>
      <td>161.800000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>144.647059</td>
      <td>164.7</td>
      <td>197.4</td>
      <td>182.000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>158.941176</td>
      <td>187.7</td>
      <td>233.1</td>
      <td>202.900000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>170.411765</td>
      <td>205.6</td>
      <td>258.9</td>
      <td>233.888889</td>
    </tr>
    <tr>
      <th>21</th>
      <td>177.750000</td>
      <td>214.7</td>
      <td>270.3</td>
      <td>238.555556</td>
    </tr>
  </tbody>
</table>
</div>

```python
agg.unstack().plot();
```

### Exercise

1. Add a title to the plot we've just made. Try googling to figure out how to do this.
2. Figure out which chickens died prematurely.
3. Figure out which chicken became the fattest.
4. Figure out during what timestep the chickens tend to gain the most weight. Hint; google what the `shift` function does.

```python
agg.unstack().plot(title="Diets!")
```

```python
df_chickweight[df_chickweight['Time'] == df_chickweight['Time'].max()]['Chick'] # all the chicks not in this list
```

    rownum
    12      1
    24      2
    36      3
    48      4
    60      5
    72      6
    84      7
    107     9
    119    10
    131    11
    143    12
    155    13
    167    14
    194    17
    208    19
    220    20
    232    21
    244    22
    256    23
    268    24
    280    25
    292    26
    304    27
    316    28
    328    29
    340    30
    352    31
    364    32
    376    33
    388    34
    400    35
    412    36
    424    37
    436    38
    448    39
    460    40
    472    41
    484    42
    496    43
    518    45
    530    46
    542    47
    554    48
    566    49
    578    50
    Name: Chick, dtype: int64

```python
df_chickweight.sort_values('weight', ascending=False).head(1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
    </tr>
    <tr>
      <th>rownum</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>400</th>
      <td>373</td>
      <td>21</td>
      <td>35</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_chickweight['weight_shifted'] = df_chickweight.groupby('Chick')['weight'].shift()
```

```python
df_chickweight
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>weight</th>
      <th>Time</th>
      <th>Chick</th>
      <th>Diet</th>
      <th>weight_shifted</th>
    </tr>
    <tr>
      <th>rownum</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>42</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>51</td>
      <td>2</td>
      <td>1</td>
      <td>1</td>
      <td>42.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>59</td>
      <td>4</td>
      <td>1</td>
      <td>1</td>
      <td>51.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>64</td>
      <td>6</td>
      <td>1</td>
      <td>1</td>
      <td>59.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>76</td>
      <td>8</td>
      <td>1</td>
      <td>1</td>
      <td>64.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>93</td>
      <td>10</td>
      <td>1</td>
      <td>1</td>
      <td>76.0</td>
    </tr>
    <tr>
      <th>7</th>
      <td>106</td>
      <td>12</td>
      <td>1</td>
      <td>1</td>
      <td>93.0</td>
    </tr>
    <tr>
      <th>8</th>
      <td>125</td>
      <td>14</td>
      <td>1</td>
      <td>1</td>
      <td>106.0</td>
    </tr>
    <tr>
      <th>9</th>
      <td>149</td>
      <td>16</td>
      <td>1</td>
      <td>1</td>
      <td>125.0</td>
    </tr>
    <tr>
      <th>10</th>
      <td>171</td>
      <td>18</td>
      <td>1</td>
      <td>1</td>
      <td>149.0</td>
    </tr>
    <tr>
      <th>11</th>
      <td>199</td>
      <td>20</td>
      <td>1</td>
      <td>1</td>
      <td>171.0</td>
    </tr>
    <tr>
      <th>12</th>
      <td>205</td>
      <td>21</td>
      <td>1</td>
      <td>1</td>
      <td>199.0</td>
    </tr>
    <tr>
      <th>13</th>
      <td>40</td>
      <td>0</td>
      <td>2</td>
      <td>1</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>14</th>
      <td>49</td>
      <td>2</td>
      <td>2</td>
      <td>1</td>
      <td>40.0</td>
    </tr>
    <tr>
      <th>15</th>
      <td>58</td>
      <td>4</td>
      <td>2</td>
      <td>1</td>
      <td>49.0</td>
    </tr>
    <tr>
      <th>16</th>
      <td>72</td>
      <td>6</td>
      <td>2</td>
      <td>1</td>
      <td>58.0</td>
    </tr>
    <tr>
      <th>17</th>
      <td>84</td>
      <td>8</td>
      <td>2</td>
      <td>1</td>
      <td>72.0</td>
    </tr>
    <tr>
      <th>18</th>
      <td>103</td>
      <td>10</td>
      <td>2</td>
      <td>1</td>
      <td>84.0</td>
    </tr>
    <tr>
      <th>19</th>
      <td>122</td>
      <td>12</td>
      <td>2</td>
      <td>1</td>
      <td>103.0</td>
    </tr>
    <tr>
      <th>20</th>
      <td>138</td>
      <td>14</td>
      <td>2</td>
      <td>1</td>
      <td>122.0</td>
    </tr>
    <tr>
      <th>21</th>
      <td>162</td>
      <td>16</td>
      <td>2</td>
      <td>1</td>
      <td>138.0</td>
    </tr>
    <tr>
      <th>22</th>
      <td>187</td>
      <td>18</td>
      <td>2</td>
      <td>1</td>
      <td>162.0</td>
    </tr>
    <tr>
      <th>23</th>
      <td>209</td>
      <td>20</td>
      <td>2</td>
      <td>1</td>
      <td>187.0</td>
    </tr>
    <tr>
      <th>24</th>
      <td>215</td>
      <td>21</td>
      <td>2</td>
      <td>1</td>
      <td>209.0</td>
    </tr>
    <tr>
      <th>25</th>
      <td>43</td>
      <td>0</td>
      <td>3</td>
      <td>1</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>26</th>
      <td>39</td>
      <td>2</td>
      <td>3</td>
      <td>1</td>
      <td>43.0</td>
    </tr>
    <tr>
      <th>27</th>
      <td>55</td>
      <td>4</td>
      <td>3</td>
      <td>1</td>
      <td>39.0</td>
    </tr>
    <tr>
      <th>28</th>
      <td>67</td>
      <td>6</td>
      <td>3</td>
      <td>1</td>
      <td>55.0</td>
    </tr>
    <tr>
      <th>29</th>
      <td>84</td>
      <td>8</td>
      <td>3</td>
      <td>1</td>
      <td>67.0</td>
    </tr>
    <tr>
      <th>30</th>
      <td>99</td>
      <td>10</td>
      <td>3</td>
      <td>1</td>
      <td>84.0</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>549</th>
      <td>154</td>
      <td>12</td>
      <td>48</td>
      <td>4</td>
      <td>125.0</td>
    </tr>
    <tr>
      <th>550</th>
      <td>170</td>
      <td>14</td>
      <td>48</td>
      <td>4</td>
      <td>154.0</td>
    </tr>
    <tr>
      <th>551</th>
      <td>222</td>
      <td>16</td>
      <td>48</td>
      <td>4</td>
      <td>170.0</td>
    </tr>
    <tr>
      <th>552</th>
      <td>261</td>
      <td>18</td>
      <td>48</td>
      <td>4</td>
      <td>222.0</td>
    </tr>
    <tr>
      <th>553</th>
      <td>303</td>
      <td>20</td>
      <td>48</td>
      <td>4</td>
      <td>261.0</td>
    </tr>
    <tr>
      <th>554</th>
      <td>322</td>
      <td>21</td>
      <td>48</td>
      <td>4</td>
      <td>303.0</td>
    </tr>
    <tr>
      <th>555</th>
      <td>40</td>
      <td>0</td>
      <td>49</td>
      <td>4</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>556</th>
      <td>53</td>
      <td>2</td>
      <td>49</td>
      <td>4</td>
      <td>40.0</td>
    </tr>
    <tr>
      <th>557</th>
      <td>64</td>
      <td>4</td>
      <td>49</td>
      <td>4</td>
      <td>53.0</td>
    </tr>
    <tr>
      <th>558</th>
      <td>85</td>
      <td>6</td>
      <td>49</td>
      <td>4</td>
      <td>64.0</td>
    </tr>
    <tr>
      <th>559</th>
      <td>108</td>
      <td>8</td>
      <td>49</td>
      <td>4</td>
      <td>85.0</td>
    </tr>
    <tr>
      <th>560</th>
      <td>128</td>
      <td>10</td>
      <td>49</td>
      <td>4</td>
      <td>108.0</td>
    </tr>
    <tr>
      <th>561</th>
      <td>152</td>
      <td>12</td>
      <td>49</td>
      <td>4</td>
      <td>128.0</td>
    </tr>
    <tr>
      <th>562</th>
      <td>166</td>
      <td>14</td>
      <td>49</td>
      <td>4</td>
      <td>152.0</td>
    </tr>
    <tr>
      <th>563</th>
      <td>184</td>
      <td>16</td>
      <td>49</td>
      <td>4</td>
      <td>166.0</td>
    </tr>
    <tr>
      <th>564</th>
      <td>203</td>
      <td>18</td>
      <td>49</td>
      <td>4</td>
      <td>184.0</td>
    </tr>
    <tr>
      <th>565</th>
      <td>233</td>
      <td>20</td>
      <td>49</td>
      <td>4</td>
      <td>203.0</td>
    </tr>
    <tr>
      <th>566</th>
      <td>237</td>
      <td>21</td>
      <td>49</td>
      <td>4</td>
      <td>233.0</td>
    </tr>
    <tr>
      <th>567</th>
      <td>41</td>
      <td>0</td>
      <td>50</td>
      <td>4</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>568</th>
      <td>54</td>
      <td>2</td>
      <td>50</td>
      <td>4</td>
      <td>41.0</td>
    </tr>
    <tr>
      <th>569</th>
      <td>67</td>
      <td>4</td>
      <td>50</td>
      <td>4</td>
      <td>54.0</td>
    </tr>
    <tr>
      <th>570</th>
      <td>84</td>
      <td>6</td>
      <td>50</td>
      <td>4</td>
      <td>67.0</td>
    </tr>
    <tr>
      <th>571</th>
      <td>105</td>
      <td>8</td>
      <td>50</td>
      <td>4</td>
      <td>84.0</td>
    </tr>
    <tr>
      <th>572</th>
      <td>122</td>
      <td>10</td>
      <td>50</td>
      <td>4</td>
      <td>105.0</td>
    </tr>
    <tr>
      <th>573</th>
      <td>155</td>
      <td>12</td>
      <td>50</td>
      <td>4</td>
      <td>122.0</td>
    </tr>
    <tr>
      <th>574</th>
      <td>175</td>
      <td>14</td>
      <td>50</td>
      <td>4</td>
      <td>155.0</td>
    </tr>
    <tr>
      <th>575</th>
      <td>205</td>
      <td>16</td>
      <td>50</td>
      <td>4</td>
      <td>175.0</td>
    </tr>
    <tr>
      <th>576</th>
      <td>234</td>
      <td>18</td>
      <td>50</td>
      <td>4</td>
      <td>205.0</td>
    </tr>
    <tr>
      <th>577</th>
      <td>264</td>
      <td>20</td>
      <td>50</td>
      <td>4</td>
      <td>234.0</td>
    </tr>
    <tr>
      <th>578</th>
      <td>264</td>
      <td>21</td>
      <td>50</td>
      <td>4</td>
      <td>264.0</td>
    </tr>
  </tbody>
</table>
<p>578 rows × 5 columns</p>
</div>

```python
df_chickweight['weight_diff'] = df_chickweight['weight'] - df_chickweight['weight_shifted']
df_chickweight.groupby('Time')['weight_diff'].mean().reset_index().plot('Time', 'weight_diff')
```

## Machine Learning: Just a Little

Let's do a small regression. This is a mere exercise and a very modest intro to Scikit-learn. There's much more out there.

```python
ml_df = df_chickweight[~np.isnan(df_chickweight['weight_shifted'])]
```

```python
ml_df.plot(x='weight_shifted', y='weight', kind='scatter')
```

```python
y = np.array(ml_df['weight'], dtype='float64')
x = np.array(ml_df['weight_shifted']).T
```

```python
from sklearn import linear_model
reg = linear_model.LinearRegression(fit_intercept=True)
```

```python
reg.fit(x, y)
```

```python
reg.fit(x.reshape(-1, 1), y)
```

```python
reg.coef_, reg.intercept_
```

    (array([ 1.06152295]), 8.5528159557474623)

```python
preds = reg.predict(x.reshape(-1, 1))
```

```python
plt.plot(x, y, "o")
plt.plot(x, preds, color='r')
```

We've just trained a machine learning model which can predict the weight of a chick given its weight the time period before.

## Working with time series

```python
dates = pd.date_range('2016-01-01', '2017-01-01')
dates[:10]
```

    DatetimeIndex(['2016-01-01', '2016-01-02', '2016-01-03', '2016-01-04',
                   '2016-01-05', '2016-01-06', '2016-01-07', '2016-01-08',
                   '2016-01-09', '2016-01-10'],
                  dtype='datetime64[ns]', freq='D')

```python
dates[-1]
```

    Timestamp('2017-01-01 00:00:00', freq='D')

```python
values1 = np.cumsum(np.random.normal(0, 1, dates.shape))
values2 = np.cumsum(np.random.normal(0, 1, dates.shape))
```

```python
df_time = pd.DataFrame({'values1': values1, 'values2': values2}, index=dates)
```

```python
df_time.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values1</th>
      <th>values2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2016-01-01</th>
      <td>-1.202854</td>
      <td>-0.945468</td>
    </tr>
    <tr>
      <th>2016-01-02</th>
      <td>-1.629328</td>
      <td>-1.424312</td>
    </tr>
    <tr>
      <th>2016-01-03</th>
      <td>-0.297628</td>
      <td>-0.431331</td>
    </tr>
    <tr>
      <th>2016-01-04</th>
      <td>-0.205496</td>
      <td>0.277939</td>
    </tr>
    <tr>
      <th>2016-01-05</th>
      <td>-0.182044</td>
      <td>1.042470</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_time.plot()
```

```python
df_smooth = df_time.rolling(window=10, axis=0).mean()
```

```python
df_smooth.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values1</th>
      <th>values2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2016-01-01</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-02</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-03</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-04</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-05</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_smooth.head(20)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values1</th>
      <th>values2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2016-01-01</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-02</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-03</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-04</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-05</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-06</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-07</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-08</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-09</th>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-10</th>
      <td>0.336643</td>
      <td>0.137472</td>
    </tr>
    <tr>
      <th>2016-01-11</th>
      <td>0.920831</td>
      <td>0.267195</td>
    </tr>
    <tr>
      <th>2016-01-12</th>
      <td>1.459406</td>
      <td>0.602617</td>
    </tr>
    <tr>
      <th>2016-01-13</th>
      <td>1.895791</td>
      <td>0.765302</td>
    </tr>
    <tr>
      <th>2016-01-14</th>
      <td>2.318210</td>
      <td>0.968805</td>
    </tr>
    <tr>
      <th>2016-01-15</th>
      <td>2.568835</td>
      <td>1.026869</td>
    </tr>
    <tr>
      <th>2016-01-16</th>
      <td>3.166831</td>
      <td>1.137327</td>
    </tr>
    <tr>
      <th>2016-01-17</th>
      <td>3.622090</td>
      <td>1.328860</td>
    </tr>
    <tr>
      <th>2016-01-18</th>
      <td>4.015601</td>
      <td>1.383000</td>
    </tr>
    <tr>
      <th>2016-01-19</th>
      <td>4.377611</td>
      <td>1.310752</td>
    </tr>
    <tr>
      <th>2016-01-20</th>
      <td>4.618585</td>
      <td>1.140468</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_smooth.plot()
```

```python
df_smooth.columns = [ 'smooth_' + column_name for column_name in df_smooth.columns ]
```

```python
df_smooth.columns
```

    Index(['smooth_values1', 'smooth_values2'], dtype='object')

```python
pd.concat([df_time, df_smooth], axis=1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values1</th>
      <th>values2</th>
      <th>smooth_values1</th>
      <th>smooth_values2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2016-01-01</th>
      <td>-1.202854</td>
      <td>-0.945468</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-02</th>
      <td>-1.629328</td>
      <td>-1.424312</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-03</th>
      <td>-0.297628</td>
      <td>-0.431331</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-04</th>
      <td>-0.205496</td>
      <td>0.277939</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-05</th>
      <td>-0.182044</td>
      <td>1.042470</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-06</th>
      <td>-1.196204</td>
      <td>0.349791</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-07</th>
      <td>0.294781</td>
      <td>0.720578</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-08</th>
      <td>1.559086</td>
      <td>0.316354</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-09</th>
      <td>2.080212</td>
      <td>0.767629</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2016-01-10</th>
      <td>4.145903</td>
      <td>0.701068</td>
      <td>0.336643</td>
      <td>0.137472</td>
    </tr>
    <tr>
      <th>2016-01-11</th>
      <td>4.639025</td>
      <td>0.351763</td>
      <td>0.920831</td>
      <td>0.267195</td>
    </tr>
    <tr>
      <th>2016-01-12</th>
      <td>3.756425</td>
      <td>1.929913</td>
      <td>1.459406</td>
      <td>0.602617</td>
    </tr>
    <tr>
      <th>2016-01-13</th>
      <td>4.066221</td>
      <td>1.195514</td>
      <td>1.895791</td>
      <td>0.765302</td>
    </tr>
    <tr>
      <th>2016-01-14</th>
      <td>4.018697</td>
      <td>2.312969</td>
      <td>2.318210</td>
      <td>0.968805</td>
    </tr>
    <tr>
      <th>2016-01-15</th>
      <td>2.324206</td>
      <td>1.623106</td>
      <td>2.568835</td>
      <td>1.026869</td>
    </tr>
    <tr>
      <th>2016-01-16</th>
      <td>4.783754</td>
      <td>1.454374</td>
      <td>3.166831</td>
      <td>1.137327</td>
    </tr>
    <tr>
      <th>2016-01-17</th>
      <td>4.847367</td>
      <td>2.635907</td>
      <td>3.622090</td>
      <td>1.328860</td>
    </tr>
    <tr>
      <th>2016-01-18</th>
      <td>5.494204</td>
      <td>0.857755</td>
      <td>4.015601</td>
      <td>1.383000</td>
    </tr>
    <tr>
      <th>2016-01-19</th>
      <td>5.700309</td>
      <td>0.045152</td>
      <td>4.377611</td>
      <td>1.310752</td>
    </tr>
    <tr>
      <th>2016-01-20</th>
      <td>6.555639</td>
      <td>-1.001769</td>
      <td>4.618585</td>
      <td>1.140468</td>
    </tr>
    <tr>
      <th>2016-01-21</th>
      <td>5.701295</td>
      <td>-1.407599</td>
      <td>4.724812</td>
      <td>0.964532</td>
    </tr>
    <tr>
      <th>2016-01-22</th>
      <td>4.119324</td>
      <td>-2.219264</td>
      <td>4.761102</td>
      <td>0.549615</td>
    </tr>
    <tr>
      <th>2016-01-23</th>
      <td>2.689068</td>
      <td>-1.579575</td>
      <td>4.623386</td>
      <td>0.272106</td>
    </tr>
    <tr>
      <th>2016-01-24</th>
      <td>2.059804</td>
      <td>-0.633275</td>
      <td>4.427497</td>
      <td>-0.022519</td>
    </tr>
    <tr>
      <th>2016-01-25</th>
      <td>2.556925</td>
      <td>-0.994812</td>
      <td>4.450769</td>
      <td>-0.284311</td>
    </tr>
    <tr>
      <th>2016-01-26</th>
      <td>3.273056</td>
      <td>-1.821385</td>
      <td>4.299699</td>
      <td>-0.611886</td>
    </tr>
    <tr>
      <th>2016-01-27</th>
      <td>3.502346</td>
      <td>0.057817</td>
      <td>4.165197</td>
      <td>-0.869696</td>
    </tr>
    <tr>
      <th>2016-01-28</th>
      <td>3.247997</td>
      <td>0.229985</td>
      <td>3.940576</td>
      <td>-0.932473</td>
    </tr>
    <tr>
      <th>2016-01-29</th>
      <td>3.874419</td>
      <td>-1.522412</td>
      <td>3.757987</td>
      <td>-1.089229</td>
    </tr>
    <tr>
      <th>2016-01-30</th>
      <td>3.098916</td>
      <td>-2.143491</td>
      <td>3.412315</td>
      <td>-1.203401</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>2016-12-03</th>
      <td>-13.101481</td>
      <td>-29.910958</td>
      <td>-12.609278</td>
      <td>-30.217000</td>
    </tr>
    <tr>
      <th>2016-12-04</th>
      <td>-14.105354</td>
      <td>-28.328899</td>
      <td>-12.757486</td>
      <td>-30.175793</td>
    </tr>
    <tr>
      <th>2016-12-05</th>
      <td>-14.667113</td>
      <td>-29.712922</td>
      <td>-12.930993</td>
      <td>-30.125140</td>
    </tr>
    <tr>
      <th>2016-12-06</th>
      <td>-15.478283</td>
      <td>-28.665579</td>
      <td>-13.116878</td>
      <td>-29.983080</td>
    </tr>
    <tr>
      <th>2016-12-07</th>
      <td>-13.583451</td>
      <td>-26.649210</td>
      <td>-13.223630</td>
      <td>-29.616711</td>
    </tr>
    <tr>
      <th>2016-12-08</th>
      <td>-12.569898</td>
      <td>-25.415938</td>
      <td>-13.408744</td>
      <td>-29.065817</td>
    </tr>
    <tr>
      <th>2016-12-09</th>
      <td>-12.386322</td>
      <td>-25.029441</td>
      <td>-13.338398</td>
      <td>-28.451891</td>
    </tr>
    <tr>
      <th>2016-12-10</th>
      <td>-13.165306</td>
      <td>-23.568852</td>
      <td>-13.415730</td>
      <td>-27.714242</td>
    </tr>
    <tr>
      <th>2016-12-11</th>
      <td>-12.783364</td>
      <td>-23.041410</td>
      <td>-13.472492</td>
      <td>-27.021311</td>
    </tr>
    <tr>
      <th>2016-12-12</th>
      <td>-10.968730</td>
      <td>-23.199240</td>
      <td>-13.280930</td>
      <td>-26.352245</td>
    </tr>
    <tr>
      <th>2016-12-13</th>
      <td>-12.498982</td>
      <td>-23.238819</td>
      <td>-13.220680</td>
      <td>-25.685031</td>
    </tr>
    <tr>
      <th>2016-12-14</th>
      <td>-12.209784</td>
      <td>-22.721555</td>
      <td>-13.031123</td>
      <td>-25.124296</td>
    </tr>
    <tr>
      <th>2016-12-15</th>
      <td>-13.160083</td>
      <td>-23.212741</td>
      <td>-12.880420</td>
      <td>-24.474278</td>
    </tr>
    <tr>
      <th>2016-12-16</th>
      <td>-12.848348</td>
      <td>-23.130548</td>
      <td>-12.617427</td>
      <td>-23.920775</td>
    </tr>
    <tr>
      <th>2016-12-17</th>
      <td>-13.833194</td>
      <td>-23.575057</td>
      <td>-12.642401</td>
      <td>-23.613360</td>
    </tr>
    <tr>
      <th>2016-12-18</th>
      <td>-13.080879</td>
      <td>-23.551875</td>
      <td>-12.693499</td>
      <td>-23.426954</td>
    </tr>
    <tr>
      <th>2016-12-19</th>
      <td>-12.946079</td>
      <td>-21.670961</td>
      <td>-12.749475</td>
      <td>-23.091106</td>
    </tr>
    <tr>
      <th>2016-12-20</th>
      <td>-12.998124</td>
      <td>-21.917588</td>
      <td>-12.732757</td>
      <td>-22.925979</td>
    </tr>
    <tr>
      <th>2016-12-21</th>
      <td>-14.264855</td>
      <td>-23.841269</td>
      <td>-12.880906</td>
      <td>-23.005965</td>
    </tr>
    <tr>
      <th>2016-12-22</th>
      <td>-13.060445</td>
      <td>-22.926829</td>
      <td>-13.090077</td>
      <td>-22.978724</td>
    </tr>
    <tr>
      <th>2016-12-23</th>
      <td>-12.381289</td>
      <td>-23.373590</td>
      <td>-13.078308</td>
      <td>-22.992201</td>
    </tr>
    <tr>
      <th>2016-12-24</th>
      <td>-12.188702</td>
      <td>-22.254727</td>
      <td>-13.076200</td>
      <td>-22.945518</td>
    </tr>
    <tr>
      <th>2016-12-25</th>
      <td>-12.188768</td>
      <td>-21.664237</td>
      <td>-12.979068</td>
      <td>-22.790668</td>
    </tr>
    <tr>
      <th>2016-12-26</th>
      <td>-12.211951</td>
      <td>-21.103311</td>
      <td>-12.915429</td>
      <td>-22.587944</td>
    </tr>
    <tr>
      <th>2016-12-27</th>
      <td>-11.009907</td>
      <td>-23.177501</td>
      <td>-12.633100</td>
      <td>-22.548189</td>
    </tr>
    <tr>
      <th>2016-12-28</th>
      <td>-11.043070</td>
      <td>-23.544025</td>
      <td>-12.429319</td>
      <td>-22.547404</td>
    </tr>
    <tr>
      <th>2016-12-29</th>
      <td>-10.727957</td>
      <td>-24.378335</td>
      <td>-12.207507</td>
      <td>-22.818141</td>
    </tr>
    <tr>
      <th>2016-12-30</th>
      <td>-10.074889</td>
      <td>-23.220822</td>
      <td>-11.915183</td>
      <td>-22.948465</td>
    </tr>
    <tr>
      <th>2016-12-31</th>
      <td>-10.549604</td>
      <td>-24.428269</td>
      <td>-11.543658</td>
      <td>-23.007165</td>
    </tr>
    <tr>
      <th>2017-01-01</th>
      <td>-11.455182</td>
      <td>-24.967937</td>
      <td>-11.383132</td>
      <td>-23.211275</td>
    </tr>
  </tbody>
</table>
<p>367 rows × 4 columns</p>
</div>

```python
pd.concat([df_time, df_smooth], axis=1).plot(figsize = (10, 4), title="Smoothed. Albeit delayed.")
```

## Exercise

There are many other rolling functions in python. Try some. How can you find them from the notebook?

```python
df_time.plot(figsize=(10, 3));
```

```python
df_smooth = df_time['values1'].rolling(10).cov(df_time['values2'])
df_smooth.plot(figsize=(10, 3));
```

## Aggregating Time

You can also use time not just to map values, but also to group values. Let's calculate some values and group them by date.

There are two ways of doing that;

* create a new column and aggregate that
* use the index to do this for you

We'll demonstrate the latter, simply because it is safer and a whole lot less work.

```python
df_time.resample('W').mean()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values1</th>
      <th>values2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2016-01-03</th>
      <td>-1.043270</td>
      <td>-0.933704</td>
    </tr>
    <tr>
      <th>2016-01-10</th>
      <td>0.928034</td>
      <td>0.596547</td>
    </tr>
    <tr>
      <th>2016-01-17</th>
      <td>4.062242</td>
      <td>1.643364</td>
    </tr>
    <tr>
      <th>2016-01-24</th>
      <td>4.617092</td>
      <td>-0.848368</td>
    </tr>
    <tr>
      <th>2016-01-31</th>
      <td>3.133673</td>
      <td>-0.845215</td>
    </tr>
    <tr>
      <th>2016-02-07</th>
      <td>5.068653</td>
      <td>0.900954</td>
    </tr>
    <tr>
      <th>2016-02-14</th>
      <td>3.577649</td>
      <td>-0.924682</td>
    </tr>
    <tr>
      <th>2016-02-21</th>
      <td>0.665007</td>
      <td>3.123118</td>
    </tr>
    <tr>
      <th>2016-02-28</th>
      <td>1.400017</td>
      <td>5.022458</td>
    </tr>
    <tr>
      <th>2016-03-06</th>
      <td>2.982486</td>
      <td>2.876473</td>
    </tr>
    <tr>
      <th>2016-03-13</th>
      <td>-0.342328</td>
      <td>3.498061</td>
    </tr>
    <tr>
      <th>2016-03-20</th>
      <td>-2.954335</td>
      <td>0.695576</td>
    </tr>
    <tr>
      <th>2016-03-27</th>
      <td>0.394468</td>
      <td>-0.328559</td>
    </tr>
    <tr>
      <th>2016-04-03</th>
      <td>-2.111267</td>
      <td>-2.311715</td>
    </tr>
    <tr>
      <th>2016-04-10</th>
      <td>-0.753374</td>
      <td>-2.784537</td>
    </tr>
    <tr>
      <th>2016-04-17</th>
      <td>-4.421193</td>
      <td>-3.046315</td>
    </tr>
    <tr>
      <th>2016-04-24</th>
      <td>-2.466026</td>
      <td>-5.156145</td>
    </tr>
    <tr>
      <th>2016-05-01</th>
      <td>-3.426302</td>
      <td>-7.355652</td>
    </tr>
    <tr>
      <th>2016-05-08</th>
      <td>-3.551178</td>
      <td>-8.871298</td>
    </tr>
    <tr>
      <th>2016-05-15</th>
      <td>-4.837539</td>
      <td>-9.711201</td>
    </tr>
    <tr>
      <th>2016-05-22</th>
      <td>-9.898535</td>
      <td>-9.635329</td>
    </tr>
    <tr>
      <th>2016-05-29</th>
      <td>-14.546133</td>
      <td>-11.040674</td>
    </tr>
    <tr>
      <th>2016-06-05</th>
      <td>-16.523255</td>
      <td>-11.134737</td>
    </tr>
    <tr>
      <th>2016-06-12</th>
      <td>-15.268189</td>
      <td>-13.612706</td>
    </tr>
    <tr>
      <th>2016-06-19</th>
      <td>-15.805042</td>
      <td>-12.621504</td>
    </tr>
    <tr>
      <th>2016-06-26</th>
      <td>-12.925192</td>
      <td>-13.366764</td>
    </tr>
    <tr>
      <th>2016-07-03</th>
      <td>-13.809686</td>
      <td>-13.973168</td>
    </tr>
    <tr>
      <th>2016-07-10</th>
      <td>-13.756602</td>
      <td>-12.267922</td>
    </tr>
    <tr>
      <th>2016-07-17</th>
      <td>-14.335205</td>
      <td>-13.554078</td>
    </tr>
    <tr>
      <th>2016-07-24</th>
      <td>-10.522536</td>
      <td>-14.373727</td>
    </tr>
    <tr>
      <th>2016-07-31</th>
      <td>-9.873278</td>
      <td>-11.343291</td>
    </tr>
    <tr>
      <th>2016-08-07</th>
      <td>-10.468867</td>
      <td>-9.767552</td>
    </tr>
    <tr>
      <th>2016-08-14</th>
      <td>-11.050006</td>
      <td>-12.875145</td>
    </tr>
    <tr>
      <th>2016-08-21</th>
      <td>-11.765581</td>
      <td>-14.991923</td>
    </tr>
    <tr>
      <th>2016-08-28</th>
      <td>-13.252107</td>
      <td>-16.566560</td>
    </tr>
    <tr>
      <th>2016-09-04</th>
      <td>-10.560958</td>
      <td>-18.226928</td>
    </tr>
    <tr>
      <th>2016-09-11</th>
      <td>-10.480465</td>
      <td>-20.386428</td>
    </tr>
    <tr>
      <th>2016-09-18</th>
      <td>-10.770104</td>
      <td>-18.711203</td>
    </tr>
    <tr>
      <th>2016-09-25</th>
      <td>-8.468545</td>
      <td>-20.057780</td>
    </tr>
    <tr>
      <th>2016-10-02</th>
      <td>-7.610634</td>
      <td>-23.186193</td>
    </tr>
    <tr>
      <th>2016-10-09</th>
      <td>-12.633554</td>
      <td>-25.828228</td>
    </tr>
    <tr>
      <th>2016-10-16</th>
      <td>-14.099397</td>
      <td>-25.140328</td>
    </tr>
    <tr>
      <th>2016-10-23</th>
      <td>-17.655951</td>
      <td>-25.385477</td>
    </tr>
    <tr>
      <th>2016-10-30</th>
      <td>-18.218125</td>
      <td>-24.913854</td>
    </tr>
    <tr>
      <th>2016-11-06</th>
      <td>-18.214347</td>
      <td>-27.519348</td>
    </tr>
    <tr>
      <th>2016-11-13</th>
      <td>-12.819316</td>
      <td>-29.257077</td>
    </tr>
    <tr>
      <th>2016-11-20</th>
      <td>-12.865225</td>
      <td>-28.290879</td>
    </tr>
    <tr>
      <th>2016-11-27</th>
      <td>-11.801224</td>
      <td>-29.704187</td>
    </tr>
    <tr>
      <th>2016-12-04</th>
      <td>-12.643922</td>
      <td>-30.162771</td>
    </tr>
    <tr>
      <th>2016-12-11</th>
      <td>-13.519105</td>
      <td>-26.011907</td>
    </tr>
    <tr>
      <th>2016-12-18</th>
      <td>-12.657143</td>
      <td>-23.232833</td>
    </tr>
    <tr>
      <th>2016-12-25</th>
      <td>-12.861180</td>
      <td>-22.521314</td>
    </tr>
    <tr>
      <th>2017-01-01</th>
      <td>-11.010366</td>
      <td>-23.545743</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_time.resample('W').mean().plot();
```

Note that the `pd.date_range` function can also be used in other ways

```python
times = pd.date_range('2017-01-01', periods=24 * 3, freq='H')
values = np.cumsum(np.random.normal(0, 1, times.shape))
df_hours = pd.DataFrame({'values': values}, index = times)
df_hours.plot()
```

Again, we can do resampling.

```python
df_hours.resample('D').sum()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2017-01-01</th>
      <td>-11.118210</td>
    </tr>
    <tr>
      <th>2017-01-02</th>
      <td>-13.714294</td>
    </tr>
    <tr>
      <th>2017-01-03</th>
      <td>15.315960</td>
    </tr>
  </tbody>
</table>
</div>

You may have noticed that this `.resample()` method is similar to `groupby`.

```python
def some_func(frame):
    return "we have {} rows".format(frame.shape[0])

df_hours.resample('D').apply(some_func)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>values</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2017-01-01</th>
      <td>we have 24 rows</td>
    </tr>
    <tr>
      <th>2017-01-02</th>
      <td>we have 24 rows</td>
    </tr>
    <tr>
      <th>2017-01-03</th>
      <td>we have 24 rows</td>
    </tr>
  </tbody>
</table>
</div>

There are many more things we can do with times. Let's look at the online documentation to learn more about these features.