```python
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
```

# Atlanta Police Department

The Atlanta Police Department provides Part 1 crime data at http://www.atlantapd.org/i-want-to/crime-data-downloads

A recent copy of the data file is stored in the cluster. Please, do not copy this data file into your home directory!

```python
### Load libraries
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
help(plt.legend)
```

    Help on function legend in module matplotlib.pyplot:

    legend(*args, **kwargs)
        Places a legend on the axes.

        To make a legend for lines which already exist on the axes
        (via plot for instance), simply call this function with an iterable
        of strings, one for each legend item. For example::

            ax.plot([1, 2, 3])
            ax.legend(['A simple line'])

        However, in order to keep the "label" and the legend element
        instance together, it is preferable to specify the label either at
        artist creation, or by calling the
        :meth:`~matplotlib.artist.Artist.set_label` method on the artist::

            line, = ax.plot([1, 2, 3], label='Inline label')
            # Overwrite the label by calling the method.
            line.set_label('Label via method')
            ax.legend()

        Specific lines can be excluded from the automatic legend element
        selection by defining a label starting with an underscore.
        This is default for all artists, so calling :meth:`legend` without
        any arguments and without setting the labels manually will result in
        no legend being drawn.

        For full control of which artists have a legend entry, it is possible
        to pass an iterable of legend artists followed by an iterable of
        legend labels respectively::

           legend((line1, line2, line3), ('label1', 'label2', 'label3'))

        Parameters
        ----------
        loc : int or string or pair of floats, default: 'upper right'
            The location of the legend. Possible codes are:

                ===============   =============
                Location String   Location Code
                ===============   =============
                'best'            0
                'upper right'     1
                'upper left'      2
                'lower left'      3
                'lower right'     4
                'right'           5
                'center left'     6
                'center right'    7
                'lower center'    8
                'upper center'    9
                'center'          10
                ===============   =============

            Alternatively can be a 2-tuple giving ``x, y`` of the lower-left
            corner of the legend in axes coordinates (in which case
            ``bbox_to_anchor`` will be ignored).

        bbox_to_anchor : :class:`matplotlib.transforms.BboxBase` instance                          or tuple of floats
            Specify any arbitrary location for the legend in `bbox_transform`
            coordinates (default Axes coordinates).

            For example, to put the legend's upper right hand corner in the
            center of the axes the following keywords can be used::

               loc='upper right', bbox_to_anchor=(0.5, 0.5)

        ncol : integer
            The number of columns that the legend has. Default is 1.

        prop : None or :class:`matplotlib.font_manager.FontProperties` or dict
            The font properties of the legend. If None (default), the current
            :data:`matplotlib.rcParams` will be used.

        fontsize : int or float or {'xx-small', 'x-small', 'small', 'medium',                   'large', 'x-large', 'xx-large'}
            Controls the font size of the legend. If the value is numeric the
            size will be the absolute font size in points. String values are
            relative to the current default font size. This argument is only
            used if `prop` is not specified.

        numpoints : None or int
            The number of marker points in the legend when creating a legend
            entry for a line/:class:`matplotlib.lines.Line2D`.
            Default is ``None`` which will take the value from the
            ``legend.numpoints`` :data:`rcParam<matplotlib.rcParams>`.

        scatterpoints : None or int
            The number of marker points in the legend when creating a legend
            entry for a scatter plot/
            :class:`matplotlib.collections.PathCollection`.
            Default is ``None`` which will take the value from the
            ``legend.scatterpoints`` :data:`rcParam<matplotlib.rcParams>`.

        scatteryoffsets : iterable of floats
            The vertical offset (relative to the font size) for the markers
            created for a scatter plot legend entry. 0.0 is at the base the
            legend text, and 1.0 is at the top. To draw all markers at the
            same height, set to ``[0.5]``. Default ``[0.375, 0.5, 0.3125]``.

        markerscale : None or int or float
            The relative size of legend markers compared with the originally
            drawn ones. Default is ``None`` which will take the value from
            the ``legend.markerscale`` :data:`rcParam <matplotlib.rcParams>`.

        *markerfirst*: [ *True* | *False* ]
            if *True*, legend marker is placed to the left of the legend label
            if *False*, legend marker is placed to the right of the legend
            label

        frameon : None or bool
            Control whether a frame should be drawn around the legend.
            Default is ``None`` which will take the value from the
            ``legend.frameon`` :data:`rcParam<matplotlib.rcParams>`.

        fancybox : None or bool
            Control whether round edges should be enabled around
            the :class:`~matplotlib.patches.FancyBboxPatch` which
            makes up the legend's background.
            Default is ``None`` which will take the value from the
            ``legend.fancybox`` :data:`rcParam<matplotlib.rcParams>`.

        shadow : None or bool
            Control whether to draw a shadow behind the legend.
            Default is ``None`` which will take the value from the
            ``legend.shadow`` :data:`rcParam<matplotlib.rcParams>`.

        framealpha : None or float
            Control the alpha transparency of the legend's frame.
            Default is ``None`` which will take the value from the
            ``legend.framealpha`` :data:`rcParam<matplotlib.rcParams>`.

        mode : {"expand", None}
            If `mode` is set to ``"expand"`` the legend will be horizontally
            expanded to fill the axes area (or `bbox_to_anchor` if defines
            the legend's size).

        bbox_transform : None or :class:`matplotlib.transforms.Transform`
            The transform for the bounding box (`bbox_to_anchor`). For a value
            of ``None`` (default) the Axes'
            :data:`~matplotlib.axes.Axes.transAxes` transform will be used.

        title : str or None
            The legend's title. Default is no title (``None``).

        borderpad : float or None
            The fractional whitespace inside the legend border.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.borderpad`` :data:`rcParam<matplotlib.rcParams>`.

        labelspacing : float or None
            The vertical space between the legend entries.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.labelspacing`` :data:`rcParam<matplotlib.rcParams>`.

        handlelength : float or None
            The length of the legend handles.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.handlelength`` :data:`rcParam<matplotlib.rcParams>`.

        handletextpad : float or None
            The pad between the legend handle and text.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.handletextpad`` :data:`rcParam<matplotlib.rcParams>`.

        borderaxespad : float or None
            The pad between the axes and legend border.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.borderaxespad`` :data:`rcParam<matplotlib.rcParams>`.

        columnspacing : float or None
            The spacing between columns.
            Measured in font-size units.
            Default is ``None`` which will take the value from the
            ``legend.columnspacing`` :data:`rcParam<matplotlib.rcParams>`.

        handler_map : dict or None
            The custom dictionary mapping instances or types to a legend
            handler. This `handler_map` updates the default handler map
            found at :func:`matplotlib.legend.Legend.get_legend_handler_map`.

        Notes
        -----

        Not all kinds of artist are supported by the legend command.
        See :ref:`plotting-guide-legend` for details.

        Examples
        --------

        .. plot:: mpl_examples/api/legend_demo.py

Load data (don't change this if you're running the notebook on the cluster)

We have two files
- `/home/data/APD/COBRA083016_2015.xlsx` for 2015
- `/home/data/APD/COBRA083016.xlsx` from 2009 to current date

```python
%%time
df = pd.read_excel('/home/data/APD/COBRA083016_2015.xlsx', sheetname='Query')
```

    CPU times: user 12.6 s, sys: 21.5 ms, total: 12.7 s
    Wall time: 39.6 s

```python
df.shape
```

    (30011, 23)

```python
for c in df.columns:
    print(c)
```

    MI_PRINX
    offense_id
    rpt_date
    occur_date
    occur_time
    poss_date
    poss_time
    beat
    apt_office_prefix
    apt_office_num
    location
    MinOfucr
    MinOfibr_code
    dispo_code
    MaxOfnum_victims
    Shift
    Avg Day
    loc_type
    UC2 Literal
    neighborhood
    npu
    x
    y

```python
df.head()
```

      MI_PRINX
      offense_id
      rpt_date
      occur_date
      occur_time
      poss_date
      poss_time
      beat
      apt_office_prefix
      apt_office_num
      ...
      loc_type
      UC2 Literal
      neighborhood
      npu
      x
      y
      Zone
      occur_ts
      occur_month
      occur_woy

      206914
      1371687
      150562000
      05/14/2013
      05/14/2013
      09:00:00
      05/14/2013
      11:30:00
      205
      NaN
      NaN
      ...
      18.0
      LARCENY-FROM VEHICLE
      Woodfield
      C
      -84.40912
      33.82308
      2
      2013-05-14 09:00:00
      5.0
      20.0

      207443
      4346442
      150010052
      01/01/2015
      12/31/2014
      22:00:00
      01/01/2015
      00:07:00
      512
      NaN
      NaN
      ...
      NaN
      LARCENY-FROM VEHICLE
      Downtown
      M
      -84.39361
      33.75246
      5
      2014-12-31 22:00:00
      12.0
      1.0

      207444
      4346443
      150010079
      01/01/2015
      01/01/2015
      00:03:00
      01/01/2015
      00:03:00
      606
      NaN
      3377
      ...
      26.0
      ROBBERY-PEDESTRIAN
      Grant Park
      W
      -84.35917
      33.73991
      6
      2015-01-01 00:03:00
      1.0
      1.0

      207445
      4346444
      150010151
      01/01/2015
      12/31/2014
      23:45:00
      01/01/2015
      00:21:00
      208
      NaN
      NaN
      ...
      18.0
      LARCENY-NON VEHICLE
      Buckhead Forest
      B
      -84.37462
      33.84564
      2
      2014-12-31 23:45:00
      12.0
      1.0

      207446
      4346445
      150010214
      01/01/2015
      01/01/2015
      00:30:00
      01/01/2015
      01:05:00
      407
      1000
      1009
      ...
      26.0
      AGG ASSAULT
      Fairburn Mays
      H
      -84.50968
      33.74349
      4
      2015-01-01 00:30:00
      1.0
      1.0

5 rows × 27 columns

```python
df.describe()
```

      MI_PRINX
      offense_id
      beat
      MinOfucr
      MaxOfnum_victims
      loc_type
      x
      y

      count
      3.001100e+04
      3.001100e+04
      30011.000000
      30011.000000
      30011.000000
      26903.000000
      30011.000000
      30011.000000

      mean
      4.361347e+06
      1.518675e+08
      359.417813
      594.219886
      1.194695
      21.109356
      -84.408346
      33.756058

      std
      1.931052e+04
      1.029128e+06
      169.563281
      114.321851
      0.799062
      16.579831
      0.046894
      0.045981

      min
      1.371687e+06
      1.500101e+08
      101.000000
      210.000000
      0.000000
      1.000000
      -84.546070
      33.637450

      25%
      4.353944e+06
      1.510128e+08
      208.000000
      512.000000
      1.000000
      13.000000
      -84.432445
      33.729060

      50%
      4.361446e+06
      1.518913e+08
      401.000000
      640.000000
      1.000000
      18.000000
      -84.398210
      33.756000

      75%
      4.368948e+06
      1.527329e+08
      505.000000
      670.000000
      1.000000
      21.000000
      -84.374420
      33.781470

      max
      4.376451e+06
      1.536580e+08
      709.000000
      730.000000
      44.000000
      99.000000
      -84.290480
      33.883250

```python
df.offense_id.min(), df.offense_id.max()
```

    (150010052, 153658045)

```python
df.columns
```

    Index([u'MI_PRINX', u'offense_id', u'rpt_date', u'occur_date', u'occur_time',
           u'poss_date', u'poss_time', u'beat', u'apt_office_prefix',
           u'apt_office_num', u'location', u'MinOfucr', u'MinOfibr_code',
           u'dispo_code', u'MaxOfnum_victims', u'Shift', u'Avg Day', u'loc_type',
           u'UC2 Literal', u'neighborhood', u'npu', u'x', u'y'],
          dtype='object')

```python
crime_summary = df.groupby(['UC2 Literal', 'neighborhood']).offense_id.count()
```

```python
crime_summary.index
```

    MultiIndex(levels=[[u'AGG ASSAULT', u'AUTO THEFT', u'BURGLARY-NONRES', u'BURGLARY-RESIDENCE', u'LARCENY-FROM VEHICLE', u'LARCENY-NON VEHICLE', u'RAPE', u'ROBBERY-COMMERCIAL', u'ROBBERY-PEDESTRIAN', u'ROBBERY-RESIDENCE'], [u'Adair Park', u'Adams Park', u'Adamsville', u'Almond Park', u'Amal Heights', u'Ansley Park', u'Arden/Habersham', u'Ardmore', u'Argonne Forest', u'Arlington Estates', u'Ashley Courts', u'Ashview Heights', u'Atkins Park', u'Atlanta Industrial Park', u'Atlanta University Center', u'Atlantic Station', u'Audobon Forest', u'Audobon Forest West', u'Baker Hills', u'Bakers Ferry', u'Bankhead', u'Bankhead Courts', u'Bankhead/Bolton', u'Beecher Hills', u'Ben Hill', u'Ben Hill Acres', u'Ben Hill Forest', u'Ben Hill Pines', u'Ben Hill Terrace', u'Benteen Park', u'Berkeley Park', u'Betmar LaVilla', u'Blair Villa/Poole Creek', u'Blandtown', u'Bolton', u'Bolton Hills', u'Boulder Park', u'Boulevard Heights', u'Brandon', u'Brentwood', u'Briar Glen', u'Brookhaven', u'Brookview Heights', u'Brookwood', u'Brookwood Hills', u'Browns Mill Park', u'Buckhead Forest', u'Buckhead Heights', u'Buckhead Village', u'Bush Mountain', u'Butner/Tell', u'Cabbagetown', u'Campbellton Road', u'Candler Park', u'Capitol Gateway', u'Capitol View', u'Capitol View Manor', u'Carey Park', u'Carroll Heights', u'Carver Hills', u'Cascade Avenue/Road', u'Cascade Green', u'Cascade Heights', u'Castleberry Hill', u'Castlewood', u'Center Hill', u'Chalet Woods', u'Channing Valley', u'Chastain Park', u'Chosewood Park', u'Collier Heights', u'Collier Hills', u'Collier Hills North', u'Colonial Homes', u'Cross Creek', u'Custer/McDonough/Guice', u'Deerwood', u'Dixie Hills', u'Downtown', u'Druid Hills', u'East Ardley Road', u'East Atlanta', u'East Chastain Park', u'East Lake', u'Edgewood', u'Elmco Estates', u'English Avenue', u'English Park', u'Fairburn', u'Fairburn Heights', u'Fairburn Mays', u'Fairburn Road/Wisteria Lane', u'Fairburn Tell', u'Fairway Acres', u'Fernleaf', u'Florida Heights', u'Fort McPherson', u'Fort Valley', u'Garden Hills', u'Georgia Tech', u'Glenrose Heights', u'Grant Park', u'Green Acres Valley', u'Green Forest Acres', u'Greenbriar', u'Greenbriar Village', u'Grove Park', u'Hammond Park', u'Hanover West', u'Harland Terrace', u'Harris Chiles', u'Harvel Homes Community', u'Heritage Valley', u'High Point', u'Hills Park', u'Home Park', u'Horseshoe Community', u'Hunter Hills', u'Huntington', u'Inman Park', u'Ivan Hill', u'Joyland', u'Just Us', u'Kings Forest', u'Kingswood', u'Kirkwood', u'Knight Park/Howell Station', u'Lake Claire', u'Lake Estates', u'Lakewood', u'Lakewood Heights', u'Laurens Valley', u'Leila Valley', u'Lenox', u'Lincoln Homes', u'Lindbergh/Morosgo', u'Lindridge/Martin Manor', u'Loring Heights', u'Magnum Manor', u'Margaret Mitchell', u'Marietta Street Artery', u'Mays', u'Meadowbrook Forest', u'Mechanicsville', u'Mellwood', u'Memorial Park', u'Midtown', u'Midwest Cascade', u'Monroe Heights', u'Morningside/Lenox Park', u'Mozley Park', u'Mt. Gilead Woods', u'Mt. Paran Parkway', u'Mt. Paran/Northside', u'Niskey Cove', u'Niskey Lake', u'North Buckhead', u'Norwood Manor', u'Oakcliff', u'Oakland', u'Oakland City', u'Old Fairburn Village', u'Old Fourth Ward', u'Old Gordon', u'Orchard Knob', u'Ormewood Park', u'Paces', u'Peachtree Battle Alliance', u'Peachtree Heights East', u'Peachtree Heights West', u'Peachtree Hills', u'Peachtree Park', u'Penelope Neighbors', u'Peoplestown', u'Perkerson', u'Peyton Forest', u'Piedmont Heights', u'Pine Hills', u'Pittsburgh', u'Pleasant Hill', u'Polar Rock', u'Pomona Park', u'Poncey-Highland', u'Princeton Lakes', u'Randall Mill', u'Rebel Valley Forest', u'Regency Trace', u'Reynoldstown', u'Ridgecrest Forest', u'Ridgedale Park', u'Ridgewood Heights', u'Riverside', u'Rockdale', u'Rosedale Heights', u'Rue Royal', u'Sandlewood Estates', u'Scotts Crossing', u'Sherwood Forest', u'South Atlanta', u'South River Gardens', u'South Tuxedo Park', u'Southwest', u'Springlake', u'Summerhill', u'Swallow Circle/Baywood', u'Sweet Auburn', u'Sylvan Hills', u'Tampa Park', u'The Villages at Carver', u'The Villages at Castleberry H', u'The Villages at East Lake', u'Thomasville Heights', u'Tuxedo Park', u'Underwood Hills', u'Venetian Hills', u'Vine City', u'Virginia Highland', u'Washington Park', u'Wesley Battle', u'West End', u'West Highlands', u'West Lake', u'West Manor', u'West Paces Ferry/Northside', u'Westhaven', u'Westminster/Milmar', u'Westover Plantation', u'Westview', u'Westwood Terrace', u'Whitewater Creek', u'Whittier Mill Village', u'Wildwood (NPU-C)', u'Wildwood (NPU-H)', u'Wildwood Forest', u'Wilson Mill Meadows', u'Wisteria Gardens', u'Woodfield', u'Woodland Hills', u'Wyngate']],
               labels=[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9], [0, 1, 2, 3, 8, 9, 10, 11, 14, 15, 18, 20, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 41, 42, 44, 45, 46, 47, 48, 49, 52, 54, 55, 56, 57, 58, 59, 60, 62, 63, 65, 67, 69, 70, 72, 73, 74, 75, 76, 77, 78, 81, 83, 84, 85, 86, 87, 88, 89, 90, 91, 95, 96, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 109, 110, 112, 113, 114, 115, 117, 119, 121, 122, 123, 125, 129, 130, 132, 133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 146, 148, 149, 150, 156, 157, 158, 160, 161, 162, 163, 164, 165, 166, 167, 169, 171, 173, 174, 176, 177, 178, 180, 181, 182, 183, 185, 187, 188, 191, 193, 194, 195, 196, 198, 199, 200, 201, 203, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 219, 220, 221, 222, 223, 224, 227, 228, 230, 232, 234, 235, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 22, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 93, 95, 96, 97, 98, 99, 100, 101, 102, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 132, 133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 146, 147, 148, 149, 150, 151, 153, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 180, 181, 182, 183, 184, 185, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 219, 220, 221, 222, 223, 224, 225, 227, 228, 229, 230, 231, 232, 233, 234, 235, 237, 0, 1, 2, 3, 5, 7, 10, 11, 12, 13, 14, 15, 20, 22, 23, 28, 30, 31, 32, 33, 34, 42, 44, 45, 46, 47, 48, 51, 52, 53, 55, 57, 58, 59, 60, 63, 65, 67, 69, 70, 73, 75, 77, 78, 81, 82, 83, 84, 85, 86, 87, 89, 90, 93, 95, 97, 98, 100, 101, 102, 104, 106, 107, 109, 114, 115, 117, 119, 120, 121, 123, 124, 125, 126, 129, 130, 132, 133, 135, 136, 137, 140, 143, 146, 148, 149, 150, 156, 157, 160, 162, 165, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 180, 181, 182, 183, 187, 191, 193, 198, 199, 200, 201, 203, 205, 206, 208, 209, 210, 213, 214, 215, 216, 218, 219, 221, 222, 223, 227, 228, 231, 237, 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 94, 95, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 146, 147, 148, 149, 150, 151, 153, 154, 155, 156, 157, 159, 160, 161, 162, 163, 164, 165, 166, 167, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 92, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 109, 110, 111, 112, 113, 114, 115, 116, 117, 119, 120, 121, 122, 123, 124, 125, 126, 127, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 145, 146, 147, 148, 149, 150, 151, 153, 154, 156, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 38, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 71, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 95, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 109, 110, 111, 112, 113, 114, 115, 117, 119, 120, 121, 122, 123, 125, 126, 127, 129, 130, 131, 132, 133, 134, 135, 136, 137, 139, 140, 141, 142, 143, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 156, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 180, 181, 182, 183, 184, 185, 187, 188, 189, 190, 191, 192, 193, 196, 198, 199, 200, 201, 202, 203, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 227, 228, 230, 231, 232, 233, 234, 235, 236, 237, 238, 0, 3, 7, 11, 14, 18, 19, 20, 22, 30, 42, 45, 46, 48, 52, 54, 57, 63, 65, 70, 77, 78, 83, 84, 86, 90, 91, 95, 100, 101, 104, 106, 107, 115, 125, 130, 133, 134, 135, 136, 141, 143, 146, 150, 156, 160, 162, 164, 170, 171, 173, 174, 176, 177, 180, 185, 191, 192, 193, 196, 199, 200, 201, 203, 205, 206, 208, 210, 213, 215, 216, 221, 232, 0, 1, 2, 11, 15, 20, 25, 30, 33, 34, 42, 48, 52, 54, 60, 63, 65, 70, 75, 78, 81, 82, 83, 84, 85, 86, 95, 98, 100, 101, 104, 106, 107, 109, 115, 123, 125, 127, 130, 133, 135, 136, 143, 146, 156, 162, 165, 169, 171, 173, 174, 176, 177, 182, 183, 191, 201, 203, 206, 211, 213, 214, 215, 217, 219, 221, 227, 231, 237, 0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 12, 14, 15, 19, 20, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 55, 56, 57, 58, 60, 62, 63, 65, 68, 69, 70, 74, 75, 76, 77, 78, 80, 81, 83, 84, 85, 86, 87, 88, 89, 90, 93, 95, 97, 98, 100, 101, 104, 106, 107, 109, 110, 112, 115, 117, 119, 120, 121, 123, 125, 126, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 143, 146, 147, 148, 149, 150, 156, 157, 160, 162, 163, 164, 165, 166, 169, 170, 171, 172, 173, 174, 176, 177, 178, 180, 182, 183, 185, 187, 192, 193, 195, 196, 198, 199, 200, 201, 202, 203, 204, 205, 206, 208, 209, 210, 211, 213, 214, 215, 216, 217, 219, 220, 221, 222, 223, 224, 227, 228, 230, 231, 232, 234, 237, 0, 2, 3, 10, 11, 14, 15, 20, 24, 27, 34, 35, 38, 41, 45, 51, 52, 55, 57, 63, 65, 69, 70, 75, 77, 78, 81, 82, 83, 84, 86, 90, 95, 97, 100, 101, 107, 109, 113, 117, 123, 129, 134, 135, 136, 141, 143, 146, 149, 150, 158, 160, 162, 163, 165, 169, 172, 173, 174, 176, 177, 178, 179, 180, 183, 184, 187, 191, 198, 199, 201, 203, 206, 208, 209, 211, 212, 213, 214, 215, 219, 220, 223, 227, 228, 234]],
               names=[u'UC2 Literal', u'neighborhood'])

```python
crime_summary.reset_index().head(20)
```

      UC2 Literal
      neighborhood
      offense_id

      0
      AGG ASSAULT
      Adair Park
      23

      1
      AGG ASSAULT
      Adams Park
      13

      2
      AGG ASSAULT
      Adamsville
      22

      3
      AGG ASSAULT
      Almond Park
      8

      4
      AGG ASSAULT
      Argonne Forest
      1

      5
      AGG ASSAULT
      Arlington Estates
      6

      6
      AGG ASSAULT
      Ashley Courts
      2

      7
      AGG ASSAULT
      Ashview Heights
      15

      8
      AGG ASSAULT
      Atlanta University Center
      21

      9
      AGG ASSAULT
      Atlantic Station
      7

      10
      AGG ASSAULT
      Baker Hills
      1

      11
      AGG ASSAULT
      Bankhead
      28

      12
      AGG ASSAULT
      Bankhead/Bolton
      4

      13
      AGG ASSAULT
      Beecher Hills
      1

      14
      AGG ASSAULT
      Ben Hill
      6

      15
      AGG ASSAULT
      Ben Hill Forest
      1

      16
      AGG ASSAULT
      Ben Hill Pines
      2

      17
      AGG ASSAULT
      Ben Hill Terrace
      4

      18
      AGG ASSAULT
      Benteen Park
      4

      19
      AGG ASSAULT
      Berkeley Park
      6

```python

```

```python

```

```python

```

```python
df["Zone"] = df.beat // 100
```

```python
df
```

      MI_PRINX
      offense_id
      rpt_date
      occur_date
      occur_time
      poss_date
      poss_time
      beat
      apt_office_prefix
      apt_office_num
      ...
      MaxOfnum_victims
      Shift
      Avg Day
      loc_type
      UC2 Literal
      neighborhood
      npu
      x
      y
      Zone

      206914
      1371687
      150562000
      05/14/2013
      05/14/2013
      09:00:00
      05/14/2013
      11:30:00
      205
      NaN
      NaN
      ...
      1
      Day
      Tue
      18.0
      LARCENY-FROM VEHICLE
      Woodfield
      C
      -84.40912
      33.82308
      2

      207443
      4346442
      150010052
      01/01/2015
      12/31/2014
      22:00:00
      01/01/2015
      00:07:00
      512
      NaN
      NaN
      ...
      1
      Morn
      Wed
      NaN
      LARCENY-FROM VEHICLE
      Downtown
      M
      -84.39361
      33.75246
      5

      207444
      4346443
      150010079
      01/01/2015
      01/01/2015
      00:03:00
      01/01/2015
      00:03:00
      606
      NaN
      3377
      ...
      1
      Morn
      Thu
      26.0
      ROBBERY-PEDESTRIAN
      Grant Park
      W
      -84.35917
      33.73991
      6

      207445
      4346444
      150010151
      01/01/2015
      12/31/2014
      23:45:00
      01/01/2015
      00:21:00
      208
      NaN
      NaN
      ...
      1
      Morn
      Thu
      18.0
      LARCENY-NON VEHICLE
      Buckhead Forest
      B
      -84.37462
      33.84564
      2

      207446
      4346445
      150010214
      01/01/2015
      01/01/2015
      00:30:00
      01/01/2015
      01:05:00
      407
      1000
      1009
      ...
      2
      Morn
      Thu
      26.0
      AGG ASSAULT
      Fairburn Mays
      H
      -84.50968
      33.74349
      4

      207447
      4346446
      150010359
      01/01/2015
      12/31/2014
      21:00:00
      01/01/2015
      00:15:00
      512
      NaN
      NaN
      ...
      1
      Eve
      Wed
      18.0
      AUTO THEFT
      Downtown
      M
      -84.39303
      33.75094
      5

      207448
      4346447
      150010370
      01/01/2015
      12/31/2014
      23:15:00
      01/01/2015
      01:47:00
      505
      NaN
      NaN
      ...
      1
      Morn
      Thu
      13.0
      LARCENY-FROM VEHICLE
      Midtown
      E
      -84.38882
      33.77511
      5

      207449
      4346448
      150010413
      01/01/2015
      12/31/2014
      23:00:00
      01/01/2015
      02:00:00
      303
      NaN
      NaN
      ...
      3
      Morn
      Thu
      13.0
      LARCENY-FROM VEHICLE
      Mechanicsville
      V
      -84.39903
      33.73631
      3

      207450
      4346449
      150010417
      01/01/2015
      01/01/2015
      02:00:00
      01/01/2015
      02:03:00
      304
      NaN
      NaN
      ...
      1
      Morn
      Thu
      14.0
      LARCENY-NON VEHICLE
      Summerhill
      V
      -84.38686
      33.74229
      3

      207451
      4346450
      150010419
      01/01/2015
      01/01/2015
      01:50:00
      01/01/2015
      02:00:00
      403
      NaN
      A
      ...
      1
      Morn
      Thu
      20.0
      AGG ASSAULT
      Oakland City
      S
      -84.42831
      33.71759
      4

      207452
      4346451
      150010442
      01/01/2015
      01/01/2015
      00:30:00
      01/01/2015
      01:45:00
      302
      NaN
      4347
      ...
      1
      Morn
      Thu
      26.0
      AUTO THEFT
      Pittsburgh
      V
      -84.40177
      33.73396
      3

      207453
      4346452
      150010468
      01/01/2015
      01/01/2015
      02:15:00
      01/01/2015
      02:30:00
      509
      ROOM
      702
      ...
      1
      Morn
      Thu
      NaN
      AGG ASSAULT
      Downtown
      M
      -84.38406
      33.76139
      5

      207454
      4346453
      150010565
      01/01/2015
      12/31/2014
      22:00:00
      01/01/2015
      03:09:00
      508
      NaN
      NaN
      ...
      2
      Morn
      Thu
      18.0
      LARCENY-FROM VEHICLE
      Downtown
      M
      -84.38840
      33.76444
      5

      207455
      4346454
      150010601
      01/01/2015
      01/01/2015
      03:00:00
      01/01/2015
      07:00:00
      509
      NaN
      NaN
      ...
      2
      Morn
      Thu
      13.0
      AGG ASSAULT
      Downtown
      M
      -84.38554
      33.76929
      5

      207456
      4346455
      150010604
      01/01/2015
      12/31/2014
      19:45:00
      01/01/2015
      03:20:00
      104
      NaN
      8201
      ...
      1
      Morn
      Wed
      26.0
      BURGLARY-RESIDENCE
      Ashview Heights
      T
      -84.42057
      33.74647
      1

      207457
      4346456
      150010608
      01/01/2015
      01/01/2015
      02:48:00
      01/01/2015
      03:26:00
      305
      NaN
      NaN
      ...
      1
      Morn
      Thu
      NaN
      ROBBERY-PEDESTRIAN
      South Atlanta
      Y
      -84.38374
      33.71730
      3

      207458
      4346457
      150010661
      01/01/2015
      01/01/2015
      02:00:00
      01/01/2015
      02:40:00
      207
      NaN
      NaN
      ...
      1
      Morn
      Thu
      21.0
      ROBBERY-PEDESTRIAN
      Underwood Hills
      D
      -84.41623
      33.80076
      2

      207459
      4346458
      150010872
      01/01/2015
      01/01/2015
      05:40:00
      01/01/2015
      06:00:00
      611
      NaN
      NaN
      ...
      2
      Morn
      Thu
      26.0
      BURGLARY-RESIDENCE
      The Villages at East Lake
      O
      -84.31650
      33.74577
      6

      207460
      4346459
      150010893
      01/01/2015
      01/01/2015
      04:30:00
      01/01/2015
      05:00:00
      304
      NaN
      NaN
      ...
      1
      Morn
      Thu
      13.0
      AGG ASSAULT
      Summerhill
      V
      -84.38909
      33.74245
      3

      207461
      4346460
      150010977
      01/01/2015
      01/01/2015
      06:30:00
      01/01/2015
      06:45:00
      602
      NaN
      NaN
      ...
      1
      Morn
      Thu
      13.0
      LARCENY-NON VEHICLE
      Old Fourth Ward
      M
      -84.37191
      33.76728
      6

      207462
      4346461
      150010989
      01/01/2015
      12/31/2014
      21:00:00
      01/01/2015
      08:22:00
      108
      NaN
      NaN
      ...
      2
      Morn
      Thu
      20.0
      LARCENY-FROM VEHICLE
      Dixie Hills
      J
      -84.45890
      33.75336
      1

      207463
      4346462
      150011005
      01/01/2015
      12/31/2014
      15:00:00
      01/01/2015
      06:00:00
      303
      NaN
      NaN
      ...
      1
      Eve
      Wed
      NaN
      LARCENY-FROM VEHICLE
      Mechanicsville
      V
      -84.39806
      33.73833
      3

      207464
      4346463
      150011023
      01/01/2015
      01/01/2015
      08:18:00
      01/01/2015
      08:48:00
      409
      NaN
      NaN
      ...
      1
      Day
      Thu
      20.0
      AGG ASSAULT
      Southwest
      R
      -84.49750
      33.69867
      4

      207465
      4346464
      150011124
      01/01/2015
      01/01/2015
      02:00:00
      01/01/2015
      02:30:00
      112
      NaN
      NaN
      ...
      1
      Morn
      Thu
      3.0
      ROBBERY-PEDESTRIAN
      Collier Heights
      I
      -84.49110
      33.78520
      1

      207466
      4346465
      150011131
      01/01/2015
      12/31/2014
      11:00:00
      01/01/2015
      05:00:00
      508
      NaN
      637
      ...
      4
      Unk
      Wed
      14.0
      LARCENY-NON VEHICLE
      Downtown
      M
      -84.39003
      33.75896
      5

      207467
      4346466
      150011181
      01/01/2015
      12/31/2014
      20:30:00
      01/01/2015
      10:45:00
      103
      NaN
      NaN
      ...
      1
      Morn
      Thu
      NaN
      AUTO THEFT
      English Avenue
      L
      -84.40498
      33.76527
      1

      207468
      4346467
      150011209
      01/01/2015
      01/01/2015
      10:15:00
      01/01/2015
      10:15:00
      603
      NaN
      NaN
      ...
      0
      Day
      Thu
      18.0
      LARCENY-NON VEHICLE
      Old Fourth Ward
      M
      -84.37140
      33.77097
      6

      207469
      4346468
      150011234
      01/01/2015
      12/31/2014
      22:30:00
      01/01/2015
      03:00:00
      302
      NaN
      1423
      ...
      1
      Morn
      Thu
      18.0
      AUTO THEFT
      Pittsburgh
      V
      -84.40177
      33.73396
      3

      207470
      4346469
      150011237
      01/01/2015
      12/31/2014
      22:30:00
      01/01/2015
      11:30:00
      410
      NaN
      C17
      ...
      1
      Morn
      Thu
      26.0
      LARCENY-NON VEHICLE
      Fort Valley
      R
      -84.45149
      33.70055
      4

      207471
      4346470
      150011253
      01/01/2015
      12/31/2014
      23:47:00
      01/01/2015
      11:38:00
      501
      NaN
      NaN
      ...
      1
      Morn
      Thu
      26.0
      LARCENY-NON VEHICLE
      Atlantic Station
      E
      -84.39541
      33.79231
      5

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      237423
      4376422
      153651689
      12/31/2015
      11/01/2015
      12:00:00
      12/31/2015
      15:45:00
      402
      NaN
      NaN
      ...
      1
      Unk
      Unk
      20.0
      BURGLARY-RESIDENCE
      Oakland City
      S
      -84.42550
      33.72696
      4

      237424
      4376423
      153651697
      12/31/2015
      12/18/2015
      12:00:00
      12/20/2015
      12:00:00
      612
      NaN
      NaN
      ...
      1
      Unk
      Unk
      18.0
      LARCENY-NON VEHICLE
      East Atlanta
      W
      -84.33524
      33.73099
      6

      237425
      4376424
      153651707
      12/31/2015
      12/31/2015
      15:45:00
      12/31/2015
      16:15:00
      509
      NaN
      NaN
      ...
      1
      Eve
      Thu
      9.0
      LARCENY-NON VEHICLE
      Downtown
      M
      -84.38702
      33.76044
      5

      237426
      4376425
      153651745
      12/31/2015
      12/23/2015
      22:00:00
      12/31/2015
      14:00:00
      310
      NaN
      34C
      ...
      1
      Unk
      Unk
      26.0
      BURGLARY-RESIDENCE
      Browns Mill Park
      Z
      -84.39417
      33.68294
      3

      237427
      4376426
      153651751
      12/31/2015
      12/31/2015
      15:30:00
      12/31/2015
      16:00:00
      306
      NaN
      NaN
      ...
      4
      Eve
      Thu
      18.0
      LARCENY-FROM VEHICLE
      Sylvan Hills
      X
      -84.41113
      33.69731
      3

      237428
      4376427
      153651774
      12/31/2015
      12/31/2015
      06:00:00
      12/31/2015
      13:30:00
      202
      NaN
      NaN
      ...
      1
      Day
      Thu
      20.0
      LARCENY-FROM VEHICLE
      Randall Mill
      A
      -84.42172
      33.85375
      2

      237429
      4376428
      153651776
      12/31/2015
      12/31/2015
      01:00:00
      12/31/2015
      16:45:00
      401
      NaN
      NaN
      ...
      1
      Day
      Thu
      23.0
      AUTO THEFT
      West End
      T
      -84.41435
      33.74042
      4

      237430
      4376429
      153651836
      12/31/2015
      12/31/2015
      14:00:00
      12/31/2015
      16:30:00
      213
      NaN
      NaN
      ...
      1
      Eve
      Thu
      NaN
      LARCENY-NON VEHICLE
      Piedmont Heights
      F
      -84.37101
      33.79789
      2

      237431
      4376430
      153651969
      12/31/2015
      12/31/2015
      17:45:00
      12/31/2015
      18:15:00
      210
      NaN
      NaN
      ...
      1
      Eve
      Thu
      8.0
      LARCENY-NON VEHICLE
      Lenox
      B
      -84.36212
      33.84676
      2

      237432
      4376431
      153651972
      12/31/2015
      12/31/2015
      18:18:00
      12/31/2015
      18:18:00
      611
      NaN
      NaN
      ...
      1
      Eve
      Thu
      20.0
      BURGLARY-RESIDENCE
      East Lake
      O
      -84.30354
      33.74624
      6

      237433
      4376432
      153652007
      12/31/2015
      12/31/2015
      18:20:00
      12/31/2015
      18:38:00
      403
      NaN
      NaN
      ...
      1
      Eve
      Thu
      31.0
      AGG ASSAULT
      Oakland City
      S
      -84.42892
      33.71896
      4

      237434
      4376433
      153652008
      12/31/2015
      12/31/2015
      18:30:00
      12/31/2015
      18:38:00
      101
      NaN
      NaN
      ...
      1
      Eve
      Thu
      NaN
      ROBBERY-PEDESTRIAN
      Atlanta University Center
      T
      -84.40436
      33.75323
      1

      237435
      4376434
      153652014
      12/31/2015
      12/31/2015
      16:45:00
      12/31/2015
      17:00:00
      503
      NaN
      NaN
      ...
      1
      Eve
      Thu
      13.0
      AUTO THEFT
      Midtown
      E
      -84.38387
      33.78305
      5

      237436
      4376435
      153652081
      12/31/2015
      12/31/2015
      19:00:00
      12/31/2015
      19:20:00
      303
      NaN
      NaN
      ...
      1
      Eve
      Thu
      26.0
      AGG ASSAULT
      Mechanicsville
      V
      -84.39741
      33.74116
      3

      237437
      4376436
      153652105
      12/31/2015
      12/31/2015
      19:05:00
      12/31/2015
      19:20:00
      303
      NaN
      6
      ...
      1
      Eve
      Thu
      26.0
      AGG ASSAULT
      Mechanicsville
      V
      -84.40200
      33.74187
      3

      237438
      4376437
      153652157
      12/31/2015
      12/31/2015
      19:10:00
      12/31/2015
      19:45:00
      207
      NaN
      NaN
      ...
      2
      Eve
      Thu
      18.0
      LARCENY-FROM VEHICLE
      Berkeley Park
      D
      -84.41575
      33.79835
      2

      237439
      4376438
      153652170
      12/31/2015
      12/31/2015
      18:06:00
      12/31/2015
      18:12:00
      208
      NaN
      NaN
      ...
      1
      Eve
      Thu
      8.0
      LARCENY-NON VEHICLE
      North Buckhead
      B
      -84.36235
      33.85166
      2

      237440
      4376439
      153652249
      12/31/2015
      12/31/2015
      19:45:00
      12/31/2015
      20:00:00
      412
      NaN
      NaN
      ...
      1
      Eve
      Thu
      13.0
      ROBBERY-PEDESTRIAN
      Heritage Valley
      P
      -84.50263
      33.70432
      4

      237441
      4376440
      153652313
      12/31/2015
      12/31/2015
      11:30:00
      12/31/2015
      20:00:00
      113
      NaN
      B313
      ...
      1
      Eve
      Thu
      26.0
      BURGLARY-RESIDENCE
      Carey Park
      G
      -84.47292
      33.78810
      1

      237442
      4376441
      153652478
      12/31/2015
      12/31/2015
      19:00:00
      12/31/2015
      20:30:00
      204
      NaN
      NaN
      ...
      2
      Eve
      Thu
      18.0
      LARCENY-FROM VEHICLE
      Underwood Hills
      D
      -84.42161
      33.80224
      2

      237443
      4376442
      153652499
      12/31/2015
      12/31/2015
      09:00:00
      12/31/2015
      09:00:00
      208
      NaN
      NaN
      ...
      1
      Day
      Thu
      18.0
      LARCENY-FROM VEHICLE
      Buckhead Forest
      B
      -84.37291
      33.84286
      2

      237444
      4376443
      153652534
      12/31/2015
      12/31/2015
      21:26:00
      12/31/2015
      22:28:00
      313
      NaN
      NaN
      ...
      1
      Eve
      Thu
      20.0
      AUTO THEFT
      Blair Villa/Poole Creek
      Z
      -84.37888
      33.66185
      3

      237445
      4376444
      153652703
      12/31/2015
      12/31/2015
      22:00:00
      12/31/2015
      22:30:00
      107
      NaN
      NaN
      ...
      1
      Eve
      Thu
      20.0
      BURGLARY-RESIDENCE
      Grove Park
      J
      -84.44629
      33.77318
      1

      237446
      4376445
      153652724
      12/31/2015
      12/31/2015
      22:00:00
      12/31/2015
      22:30:00
      508
      NaN
      NaN
      ...
      1
      Eve
      Thu
      NaN
      AGG ASSAULT
      Downtown
      M
      -84.39152
      33.75700
      5

      237447
      4376446
      153652840
      12/31/2015
      12/31/2015
      22:10:00
      12/31/2015
      22:30:00
      612
      NaN
      NaN
      ...
      1
      Eve
      Thu
      18.0
      AUTO THEFT
      East Atlanta
      W
      -84.33588
      33.73126
      6

      237448
      4376447
      153652866
      12/31/2015
      12/31/2015
      22:30:00
      12/31/2015
      23:00:00
      610
      NaN
      NaN
      ...
      1
      Eve
      Thu
      23.0
      LARCENY-NON VEHICLE
      Kirkwood
      O
      -84.32665
      33.74774
      6

      237449
      4376448
      153652876
      12/31/2015
      12/31/2015
      17:00:00
      12/31/2015
      23:00:00
      303
      NaN
      1408
      ...
      1
      Eve
      Thu
      26.0
      BURGLARY-RESIDENCE
      Mechanicsville
      V
      -84.40649
      33.74005
      3

      237450
      4376449
      153652888
      01/01/2016
      12/31/2015
      23:05:00
      01/01/2016
      00:50:00
      313
      NaN
      22
      ...
      1
      Morn
      Thu
      13.0
      AGG ASSAULT
      South River Gardens
      Z
      -84.36187
      33.67102
      3

      237451
      4376450
      153653047
      12/31/2015
      12/31/2015
      21:30:00
      12/31/2015
      23:47:00
      603
      NaN
      NaN
      ...
      2
      Eve
      Thu
      13.0
      LARCENY-FROM VEHICLE
      Old Fourth Ward
      M
      -84.36450
      33.77073
      6

      237452
      4376451
      153658045
      12/31/2015
      12/31/2015
      13:02:00
      12/31/2015
      13:15:00
      704
      NaN
      NaN
      ...
      1
      Day
      Thu
      NaN
      LARCENY-NON VEHICLE
      NaN
      NaN
      -84.44824
      33.63751
      7

30011 rows × 24 columns

## Exploring Dates

```python
df[['offense_id', 'occur_date', 'occur_time', 'rpt_date']][1:10]
```

      offense_id
      occur_date
      occur_time
      rpt_date

      207443
      150010052
      12/31/2014
      22:00:00
      01/01/2015

      207444
      150010079
      01/01/2015
      00:03:00
      01/01/2015

      207445
      150010151
      12/31/2014
      23:45:00
      01/01/2015

      207446
      150010214
      01/01/2015
      00:30:00
      01/01/2015

      207447
      150010359
      12/31/2014
      21:00:00
      01/01/2015

      207448
      150010370
      12/31/2014
      23:15:00
      01/01/2015

      207449
      150010413
      12/31/2014
      23:00:00
      01/01/2015

      207450
      150010417
      01/01/2015
      02:00:00
      01/01/2015

      207451
      150010419
      01/01/2015
      01:50:00
      01/01/2015

Convert into date-time type

```python
df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)
```

```python
#df[['offense_id', 'occur_date', 'occur_time', 'occur_ts', 'rpt_date']][1:10]
```

```python

```

```python
df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)
```

```python
df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear
```

```python
df.describe()
```

      MI_PRINX
      offense_id
      beat
      MinOfucr
      MaxOfnum_victims
      loc_type
      x
      y
      Zone
      occur_month
      occur_woy

      count
      3.001100e+04
      3.001100e+04
      30011.000000
      30011.000000
      30011.000000
      26903.000000
      30011.000000
      30011.000000
      30011.000000
      29997.000000
      29997.000000

      mean
      4.361347e+06
      1.518675e+08
      359.417813
      594.219886
      1.194695
      21.109356
      -84.408346
      33.756058
      3.527307
      6.654399
      27.453312

      std
      1.931052e+04
      1.029128e+06
      169.563281
      114.321851
      0.799062
      16.579831
      0.046894
      0.045981
      1.700308
      3.365509
      14.700785

      min
      1.371687e+06
      1.500101e+08
      101.000000
      210.000000
      0.000000
      1.000000
      -84.546070
      33.637450
      1.000000
      1.000000
      1.000000

      25%
      4.353944e+06
      1.510128e+08
      208.000000
      512.000000
      1.000000
      13.000000
      -84.432445
      33.729060
      2.000000
      4.000000
      15.000000

      50%
      4.361446e+06
      1.518913e+08
      401.000000
      640.000000
      1.000000
      18.000000
      -84.398210
      33.756000
      4.000000
      7.000000
      28.000000

      75%
      4.368948e+06
      1.527329e+08
      505.000000
      670.000000
      1.000000
      21.000000
      -84.374420
      33.781470
      5.000000
      10.000000
      40.000000

      max
      4.376451e+06
      1.536580e+08
      709.000000
      730.000000
      44.000000
      99.000000
      -84.290480
      33.883250
      7.000000
      12.000000
      53.000000

```python
df.shape
```

    (30011, 27)

```python
df.columns
```

    Index([         u'MI_PRINX',        u'offense_id',          u'rpt_date',
                  u'occur_date',        u'occur_time',         u'poss_date',
                   u'poss_time',              u'beat', u'apt_office_prefix',
              u'apt_office_num',          u'location',          u'MinOfucr',
               u'MinOfibr_code',        u'dispo_code',  u'MaxOfnum_victims',
                       u'Shift',           u'Avg Day',          u'loc_type',
                 u'UC2 Literal',      u'neighborhood',               u'npu',
                           u'x',                 u'y',              u'Zone',
                    u'occur_ts',       u'occur_month',         u'occur_woy'],
          dtype='object')

```python
df.iloc[9]
```

    MI_PRINX                            4346450
    offense_id                        150010419
    rpt_date                         01/01/2015
    occur_date                       01/01/2015
    occur_time                         01:50:00
    poss_date                        01/01/2015
    poss_time                          02:00:00
    beat                                    403
    apt_office_prefix                       NaN
    apt_office_num                            A
    location             1231 CAMPBELLTON PL SW
    MinOfucr                                430
    MinOfibr_code                          1315
    dispo_code                               10
    MaxOfnum_victims                          1
    Shift                                  Morn
    Avg Day                                 Thu
    loc_type                                 20
    UC2 Literal                     AGG ASSAULT
    neighborhood                   Oakland City
    npu                                       S
    x                                  -84.4283
    y                                   33.7176
    occur_ts                2015-01-01 01:50:00
    Name: 207451, dtype: object

```python
#resdf.index
```

```python
resdf.loc[['AUTO_THEFT', 6]]
```

```python
resdf = df.groupby(['UC2 Literal', 'occur_date']).offense_id.count()
resdf
```

    UC2 Literal        occur_date
    AGG ASSAULT        01/01/2009    1
                       01/01/2015    6
                       01/02/2015    6
                       01/03/2015    7
                       01/04/2015    4
                       01/05/2015    3
                       01/06/2015    4
                       01/07/2015    6
                       01/08/2015    5
                       01/09/2015    7
                       01/10/2015    3
                       01/11/2015    9
                       01/12/2015    3
                       01/13/2015    1
                       01/14/2015    7
                       01/15/2015    2
                       01/16/2015    3
                       01/17/2015    3
                       01/18/2015    7
                       01/19/2015    9
                       01/20/2015    5
                       01/21/2015    2
                       01/22/2015    6
                       01/23/2015    3
                       01/24/2015    3
                       01/25/2015    3
                       01/26/2015    8
                       01/27/2015    6
                       01/28/2015    5
                       01/29/2015    9
                                    ..
    ROBBERY-RESIDENCE  10/02/2015    1
                       10/03/2015    1
                       10/04/2015    1
                       10/08/2015    1
                       10/10/2015    1
                       10/18/2015    1
                       10/20/2015    1
                       10/21/2015    1
                       10/24/2015    1
                       11/01/2015    1
                       11/02/2015    2
                       11/09/2015    2
                       11/11/2015    1
                       11/13/2015    1
                       11/20/2015    1
                       11/23/2014    1
                       11/27/2015    1
                       11/28/2015    1
                       12/02/2015    1
                       12/04/2015    1
                       12/06/2015    1
                       12/10/2015    1
                       12/11/2015    2
                       12/19/2015    1
                       12/21/2015    1
                       12/22/2015    1
                       12/27/2015    1
                       12/29/2015    1
                       12/30/2015    1
                       12/31/2015    1
    Name: offense_id, dtype: int64

```python
resdf['BURGLARY-RESIDENCE'].as_matrix()
```

    array([13,  1,  6, 12,  7,  9, 12,  6,  1, 11, 10,  1, 10,  7, 17, 16, 10,
           18, 15, 18, 10,  7, 16,  1,  8, 10,  9, 11, 11, 16,  8, 15, 16, 10,
            9, 11, 14,  4, 13,  9,  8, 13,  8,  3,  6, 10, 10, 16,  5,  2,  6,
            3,  3,  7, 10,  4,  8, 13,  9,  5,  7, 13, 13, 12,  9,  6, 12, 12,
           13,  5,  8,  9,  7,  9,  9, 14,  6,  8, 15,  9, 12,  7, 13,  8,  5,
           10, 16, 15,  9,  6, 13,  3, 14,  9, 17, 15, 17, 12,  4, 18,  9, 14,
            9, 15,  5, 11, 11, 13, 11,  7,  8,  6,  6, 16, 11,  9,  8,  6, 10,
            3,  7, 13,  8, 12, 12,  8,  9, 11, 17,  7, 17, 15,  8,  4,  9, 23,
            1, 20, 12, 10,  1,  7,  6, 21, 11, 12,  1, 12, 14, 13,  6,  5, 18,
           18, 15, 13, 13,  6, 11, 14, 13,  8, 15, 10,  6, 11,  6,  4,  7,  7,
           12,  5,  8, 10, 14, 15, 12, 12,  9, 11, 17, 11, 17, 22,  9,  6, 10,
           11, 10, 19, 14, 11,  2, 13, 10,  4, 11, 16,  9,  6, 12, 16,  9, 14,
           13,  8,  6, 11,  5, 13,  6,  7,  9,  8, 19,  9, 10,  7, 10,  9,  3,
            9,  9, 12,  4, 12,  6,  5, 13,  7,  1,  9, 10,  1,  9, 11,  4, 10,
           10,  8, 14,  7,  8,  5,  9, 14, 14,  8, 17,  8, 13, 13, 13, 13,  8,
            9,  7,  6,  8, 21, 10, 11, 17,  6, 10,  6, 12, 10,  6,  8, 10,  8,
           13, 13,  7,  8, 11,  7,  5, 12, 15,  8, 17, 12, 10,  1,  9, 10, 14,
           10, 10, 12,  7,  8, 12,  9, 14,  5,  8,  7,  6,  9, 10,  7, 20, 13,
            1, 19,  7, 12, 16,  1, 11,  9, 13, 11,  8, 16,  8, 11,  7,  8, 11,
            4, 12,  1, 18,  9, 15, 12,  3,  2,  4, 15, 17, 12,  1,  8,  2, 16,
           14, 11, 13, 11, 15,  9, 15,  6, 11,  1, 14, 19, 18, 12, 18,  1, 17,
            8,  1, 14,  1, 14, 13,  2, 18, 19,  1,  8, 13, 21,  1, 10,  1, 20,
           14,  1, 21,  8,  2, 20, 13,  1,  9,  1,  8,  2, 10, 10,  1,  8,  6,
           12,  1,  6,  2, 15,  2,  9])

```python
resdf['BURGLARY-RESIDENCE'].iloc(0)
```

    <pandas.core.indexing._iLocIndexer at 0x7f73610>

```python
resdf['BURGLARY-RESIDENCE']
```

    occur_date
    01/01/2015    13
    01/02/2014     1
    01/02/2015     6
    01/03/2015    12
    01/04/2015     7
    01/05/2015     9
    01/06/2015    12
    01/07/2015     6
    01/08/2014     1
    01/08/2015    11
    01/09/2015    10
    01/10/2014     1
    01/10/2015    10
    01/11/2015     7
    01/12/2015    17
    01/13/2015    16
    01/14/2015    10
    01/15/2015    18
    01/16/2015    15
    01/17/2015    18
    01/18/2015    10
    01/19/2015     7
    01/20/2015    16
    01/21/2014     1
    01/21/2015     8
    01/22/2015    10
    01/23/2015     9
    01/24/2015    11
    01/25/2015    11
    01/26/2015    16
                  ..
    12/13/2015    13
    12/14/2015    21
    12/15/2014     1
    12/15/2015    10
    12/16/2014     1
    12/16/2015    20
    12/17/2015    14
    12/18/2014     1
    12/18/2015    21
    12/19/2015     8
    12/20/2014     2
    12/20/2015    20
    12/21/2015    13
    12/22/2014     1
    12/22/2015     9
    12/23/2014     1
    12/23/2015     8
    12/24/2014     2
    12/24/2015    10
    12/25/2015    10
    12/26/2014     1
    12/26/2015     8
    12/27/2015     6
    12/28/2015    12
    12/29/2014     1
    12/29/2015     6
    12/30/2014     2
    12/30/2015    15
    12/31/2014     2
    12/31/2015     9
    Name: offense_id, dtype: int64

```python
%matplotlib inline
fig = plt.figure(figsize=(10,6)) # 10inx10in
#plt.plot(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'])
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'].values, marker='x')
# plt.scatter(resdf['BURGLARY-NONRES'].index, resdf['BURGLARY-NONRES'], marker='o')

# plt.ylim(0, 500)
# plt.title('BURGLARY-RESIDENCE')
# plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
# fig.savefig('BurglaryResidence_over_month.svg')
# x = 1
```

```python
def getTheMonth(x):
    return x.month

df['occur_month'] = df['occur_ts'].map(getTheMonth)
```

```python
df = pd.read_excel('/home/data/APD/COBRA083016_2015.xlsx', sheetname='Query')
df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)
df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear
```

```python
%matplotlib inline
resdf = df.groupby(['UC2 Literal', 'occur_month']).offense_id.count()
fig = plt.figure(figsize=(10,6))
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'], marker='x')
plt.ylim(0, 500)
plt.title('BURGLARY-RESIDENCE')
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.savefig('quiz3-burglary-residence.png')

''
```

```python
plt.savefig('quiz3-burglary-residence.png')

```

# Part 1 - Observations from the data

```python
pd.unique(df) # how to get duplicate records?
```

```python
pd.unique(df['UC2 Literal'])
```

    array([u'LARCENY-FROM VEHICLE', u'ROBBERY-PEDESTRIAN',
           u'LARCENY-NON VEHICLE', u'AGG ASSAULT', u'AUTO THEFT',
           u'BURGLARY-RESIDENCE', u'ROBBERY-RESIDENCE', u'BURGLARY-NONRES',
           u'ROBBERY-COMMERCIAL', u'RAPE'], dtype=object)

```python
len(pd.unique(df.MI_PRINX))
```

    30011

# Part 2 - Seasonal Model

```python
## load complete dataset
dff = pd.read_excel('/home/data/APD/COBRA083016.xlsx', sheetname='Query')
```

```python
dff.shape
```

```python
for evt in ['occur', 'poss']:
    dff['%s_ts'%evt] = pd.to_datetime(dff['%s_date'%evt]+' '+dff['%s_time'%evt])
dff['rpt_ts'] = pd.to_datetime(dff.rpt_date)
```

```python
', '.join(dff.columns)
```

```python
dff['occur_year'] = dff.occur_ts.dt.year
dff['occur_month'] = dff.occur_ts.dt.month
dff['occur_dayweek'] = dff.occur_ts.dt.dayofweek
```

### Crime per year
Let's look at the

```python
crime_year = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year']).offense_id.count()
```

```python
%matplotlib inline
fig = plt.figure(figsize=(40,30))
crime_types = crime_year.index.levels[0]
years = crime_year.index.levels[1]
for c in range(len(crime_types)):
    y_max = max(crime_year.loc[crime_types[c]])

    plt.subplot(4,3,c+1)
    plt.hlines(crime_year.loc[crime_types[c]].iloc[-1]*100/y_max, years[0], years[-1], linestyles="dashed", color="r")
    plt.bar(crime_year.loc[crime_types[c]].index, crime_year.loc[crime_types[c]]*100/y_max, label=crime_types[c], alpha=0.5)
    ##plt.legend()
    plt.ylim(0, 100)
    plt.xticks(years+0.4, [str(int(y)) for y in years], rotation=0, fontsize=24)
    plt.yticks([0,20,40,60,80,100], ['0%','20%','40%','60%','80%','100%'], fontsize=24)
    plt.title(crime_types[c], fontsize=30)
    None
```

Let's look at residential burglary.

```python
c = 3
crime_types[c]
```

```python
crime_year_month = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year', 'occur_month']).offense_id.count()
```

```python
c = 3 ## 'BURGLARY-RESIDENCE'
resburglaries = crime_year_month.loc[crime_types[c]]
fig = plt.figure(figsize=(20,10))
for y in years:
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s"%crime_types[c], fontsize=20)
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None
```

Normalized over the annual average

```python
c = 3 ## 'BURGLARY-RESIDENCE'
fig = plt.figure(figsize=(20,10))
for y in years:
    avg = resburglaries.loc[y].mean()
    plt.hlines(avg, 1, 13, linestyle='dashed')
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s (with annuale averages)"%crime_types[c], fontsize=20)
plt.xticks(list(range(1,13)), ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None
```

```python
c = 3 ## 'BURGLARY-RESIDENCE'
fig = plt.figure(figsize=(20,10))
for y in years:
    avg = resburglaries.loc[y].mean()
    std = resburglaries.loc[y].std()
    ##plt.hlines(avg, 1, 13, linestyle='dashed')
    plt.plot(resburglaries.loc[y].index, (resburglaries.loc[y]-avg)/std, label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s (normalized)"%crime_types[c], fontsize=20)
plt.xticks(list(range(1,13)), ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
plt.ylabel("Standard deviations $\sigma_y$")
None
```

```python

```

```python
seasonal_adjust = resburglaries.reset_index().groupby(by=['occur_month']).offense_id.agg('mean')
```

```python

```

```python

```

```python

```

```python

```

```python

```

### Fitting the regression line
Suppose there are $n$ data points {{math|{(''xi'', ''yi''), ''i'' {{=}} 1, ..., ''n''}.}} The function that describes x and y is:

$$y_i = \alpha + \beta x_i + \varepsilon_i.$$

The goal is to find the equation of the straight line

$$y = \alpha + \beta x,$$

which would provide a "best" fit for the data points. Here the "best" will be understood as in the [[Ordinary least squares|least-squares]] approach: a line that minimizes the sum of squared residuals of the linear regression model. In other words, {{mvar|α}} (the {{mvar|y}}-intercept) and {{mvar|β}} (the slope) solve the following minimization problem:

$$\text{Find }\min_{\alpha,\,\beta} Q(\alpha, \beta), \qquad \text{for } Q(\alpha, \beta) = \sum_{i=1}^n\varepsilon_i^{\,2} = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2\ $$

By using either [[calculus]], the geometry of [[inner product space]]s, or simply expanding to get a quadratic expression in {{mvar|α}} and {{mvar|β}}, it can be shown that the values of {{mvar|α}} and {{mvar|β}} that minimize the objective function {{mvar|Q}}<ref>Kenney, J. F. and Keeping, E. S. (1962) "Linear Regression and Correlation." Ch. 15 in ''Mathematics of Statistics'', Pt. 1, 3rd ed. Princeton, NJ: Van Nostrand, pp. 252–285</ref> are

: <math>\begin{align}
  \hat\beta &= \frac{ \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 } \\[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i - x_i \bar{y} - \bar{x} y_i + \bar{x} \bar{y})} { \sum_{i=1}^n (x_i^2 - 2 x_i \bar{x} + \bar{x}^2) } \\[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i) - \bar{y} \sum_{i=1}^{n}  x_i  - \bar{x} \sum_{i=1}^{n} y_i + n  \bar{x} \bar{y}} { \sum_{i=1}^n (x_i^2) - 2 \bar{x}  \sum_{i=1}^n  x_i  + n \bar{x}^2 } \\[6pt]
            &= \frac{ \frac{1}{n} \sum_{i=1}^{n} x_i y_i - \bar{x} \bar{y} }{ \frac{1}{n}\sum_{i=1}^n {x_i^2} -   \overline{x}^2 } \\[6pt]
            &= \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } = \frac{ \operatorname{Cov}[x, y] }{ \operatorname{Var}[x] } \\
            &= r_{xy} \frac{s_y}{s_x}, \\[6pt]
 \hat\alpha & = \bar{y} - \hat\beta\,\bar{x},
\end{align}</math>

where {{math|''rxy''}} is the [[Correlation#Pearson's product-moment coefficient|sample correlation coefficient]] between {{mvar|x}} and {{mvar|y}}; and {{math|''sx''}} and {{math|''sy''}} are the [[sample standard deviation]] of {{mvar|x}} and {{mvar|y}}. A horizontal bar over a quantity indicates the average value of that quantity. For example:

:<math>\overline{xy} = \frac{1}{n} \sum_{i=1}^n x_i y_i.</math>

Substituting the above expressions for <math>\hat{\alpha}</math> and <math>\hat{\beta}</math> into

: <math>f = \hat{\alpha} + \hat{\beta} x,</math>

yields

: <math>\frac{ f - \bar{y}}{s_y} = r_{xy} \frac{ x - \bar{x}}{s_x}  </math>

This shows that {{math|''rxy''}} is the slope of the regression line of the [[Standard score|standardized]] data points (and that this line passes through the origin).

It is sometimes useful to calculate {{math|''rxy''}} from the data independently using this equation:

:<math>r_{xy} = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \sqrt{ \left(\overline{x^2} - \bar{x}^2\right)\left(\overline{y^2} - \bar{y}^2\right)} } </math>

The [[coefficient of determination]] (R squared) is equal to <math>r_{xy}^2</math> when the model is linear with a single independent variable. See [[Correlation#Pearson's product-moment coefficient|sample correlation coefficient]] for additional details.

===Linear regression without the intercept term===
Sometimes it is appropriate to force the regression line to pass through the origin, because {{mvar|x}} and {{mvar|y}} are assumed to be proportional. For the model without the intercept term, {{math|''y'' {{=}} ''βx''}}, the OLS estimator for {{mvar|β}} simplifies to

: <math>\hat{\beta} = \frac{ \sum_{i=1}^n x_i y_i }{ \sum_{i=1}^n x_i^2 } = \frac{\overline{x y}}{\overline{x^2}} </math>

Substituting {{math|(''x'' − ''h'', ''y'' − ''k'')}} in place of {{math|(''x'', ''y'')}} gives the regression through {{math|(''h'', ''k'')}}:

: <math>\begin{align}
  \hat\beta &= \frac{\overline{(x - h) (y - k)}}{\overline{(x - h)^2}} \\[6pt]
            &= \frac{\overline{x y} + k \bar{x} - h \bar{y} - h k }{\overline{x^2} - 2 h \bar{x} + h^2} \\[6pt]
            &= \frac{\overline{x y} - \bar{x} \bar{y} + (\bar{x} - h)(\bar{y} - k)}{\overline{x^2} - \bar{x}^2 + (\bar{x} - h)^2} \\[6pt]
            &= \frac{\operatorname{Cov}[x,y] + (\bar{x} - h)(\bar{y}-k)}{\operatorname{Var}[x] + (\bar{x} - h)^2}
\end{align}</math>

The last form above demonstrates how moving the line away from the center of mass of the data points affects the slope.

```python
### in case we want to save a DataFrame
#writer = pd.ExcelWriter('myresults.xlsx')
#df.to_excel(writer,'Results')
#writer.save()
```

```python
#resdf
```

```python

```

```python

```

```python

```

```python

```


## Uncomment & Run the below cell if you're using Colab

```python
# !git clone https://github.com/mayahq/g-index-benchmark
# %cd /content/g-index-benchmark/notebooks/
```

    fatal: destination path 'g-index-benchmark' already exists and is not an empty directory.
    [Errno 2] No such file or directory: '/content/g-index-benchmark/notebooks/'
    /home/atlas/dagbert-code/g-index-benchmark/notebooks

## Imports

```python
import glob
import json
import os
import sys
from collections import defaultdict
from os import PathLike
from pprint import pprint
from statistics import mean
from typing import Union

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from tqdm import tqdm

module_path = os.path.abspath(os.path.join(".."))
if module_path not in sys.path:
    sys.path.append(module_path)

import ipywidgets as widgets
from ipywidgets import Dropdown, FloatLogSlider, FloatSlider, IntSlider, interact

from g_index import Experiment
from node_utils import node_divergence
from utils import *

# Experiment Settings
EXP_ROOT = "../experiments/"
EXP_FILES = glob.glob(EXP_ROOT + "/*.json")
DD_CACHE = cache_dd()
DOMAIN_LENGTHS = get_domain_lengths(AVAILABLE_DOMAINS)

##Plot Settings
ALPHA, S = 0.90, 240
FIGSIZE = (16,9)
LABEL_SIZE = 24
TICK_SIZE = 20
TITLE_SIZE = 24
LEGEND_SIZE = 20

colors = ["#375E97", "#FB6542", "#FFBB00", "#3F681C"]
sim_colors = ["#C03221", "#A049AB", "#3C28B8", "#545E75", "#3F826D"]
models = ["gpt2-774M", "gpt2-345M", "gpt2-1.5B", "gptneo-2.7B"]
model_colormap = {model: color for color, model in zip(colors, models)}
model_name_correction = {
    "gpt2-medium": "gpt2-345M",
    "gpt2-large": "gpt2-774M",
    "gpt2-xl": "gpt2-1.5B",
    "EleutherAI gpt-neo-2.7B": "gptneo-2.7B",
}
```

## 1. Domains

### What are domains

A set of tasks \$C$ can be clustered into `Task Domains`.

Tasks within the same domain are closer to each other than tasks in different domains.

### List of Available Domains

```python
for i, domain in enumerate(AVAILABLE_DOMAINS):
    print(i, " ", domain)
```

    0   template-slider
    1   template-form
    2   url-skill
    3   cron-schedule
    4   gmail-send
    5   telegram-3-reply
    6   telegram-2-reply
    7   youtube-pause
    8   youtube-resume
    9   http
    10   cron-reminder
    11   google-search
    12   twitter
    13   youtube-play
    14   facebook
    15   googlesearch-to-csv

### What is Domain Distance ( $\Omega$ )

Domain Distance of a Task $T'$ from a set of tasks $C$ is the
similarity between optimal DAG program for $T'$ and it's nearest neighbour in $C$.

\begin{equation}
	\label{domain-distcalc}
	\begin{aligned}
		 \Omega(T', C) & = \min\limits_{T \in C} \Delta(G_{P'_{opt}}, G_{P_T})                                          &   & \\
		              & \text{where }  G_{P'_{opt}} \text{denotes the DAG of the optimal solution $P'_{opt}$ of $T'$} &   & \\
		              & \text{and }    G_{P'_T} \text{ denotes the DAG for a program $P_T$ that solves
		 $T \in C$}    &                                                                                &
	\end{aligned}
\end{equation}

Domain distance can be calculated for a set of tasks $T'$ by averaging over all tasks.

### Calculating domain distance between two domains

Domain distance can be calculated between any two domains in `AVAILABLE_DOMAINS`

```python
domain_1 = Dropdown(options=AVAILABLE_DOMAINS, description="Domain 1")
domain_2 = Dropdown(options=AVAILABLE_DOMAINS, description="Domain 2")

@interact(domain_1=domain_1, domain_2=domain_2)
def get_dd_i(domain_1, domain_2):
    print(calculate_dd(domain_1, domain_2))
```

    interactive(children=(Dropdown(description='Domain 1', options=('template-slider', 'template-form', 'url-skill…

### Calculating Domain Distance between two JSON files

Treating each file as it's own domain with a single sample Domain Distance can be calculated between two domain files.

```python
domain_1 = widgets.Text(
    value="../domains/youtube-pause/youtube-pause-6e696d.json",
    placeholder="Enter Relative File Path",
    description="Domain 1 :",
    disabled=False,
)
domain_2 = widgets.Text(
    value="../domains/facebook/facebook-b4ee9a.json",
    placeholder="Enter Relative File Path ",
    description="Domain 2 :",
    disabled=False,
)

@interact(domain_1=domain_1, domain_2=domain_2)
def get_dd_files_i(domain_1, domain_2):
    dd = calculate_dd(domain_1, domain_2)
    if dd is not None:
        print(f"DD: {dd}")
```

    interactive(children=(Text(value='../domains/youtube-pause/youtube-pause-6e696d.json', description='Domain 1 :…

### Plotting the Domain Distance Matrix

Domain Distance between various domain can easily be visualised with the following heatmap.

```python
sns.set(rc={"figure.figsize": (16, 12)})
generate_dd_matrix()
```

## 2. Experiments

### Components of an experiment

The Major Components of an experiment are:
1. Intelligent System (`IS`)
2. Curricula Domains ($C$)
3. Task Domains ($\{T'\}$)
4. Experience (`E`)
5. Generalization Difficulty (GD)
6. Performance Details
7. AveragePerformance ($\theta$)

Note: Uncheck `truncate_domains` to view full domain data

```python
experiments = Dropdown(options=EXP_FILES, description="Experiments")
@interact(exp_file=experiments,truncate_domains=True)
def print_exp_components_i(exp_file, truncate_domains=False):
    exp = Experiment(exp_file)
    exp.get_exp_components(truncate_domains=truncate_domains, print_members=True)
```

    interactive(children=(Dropdown(description='Experiments', options=('../experiments/GPT2_1_5B_2.json', '../expe…

## 3. g-index

### What is `g-index`?

`g-index` is used to measure skill-acquisition efficiency of any intelligence system on a set of real-world tasks.

### Calculating `g-index` from an experiment files

Steps:
1. Select an Experiment file in the Experiment Class
2. Run `calculate_g_index` function

```python
experiments = Dropdown(options=EXP_FILES, description="Experiments")
@interact(exp_file=experiments)
def get_g_index_i(exp_file):
    exp = Experiment(exp_file)
    print(f"g-index: {exp.calculate_g_index()}")
```

    interactive(children=(Dropdown(description='Experiments', options=('../experiments/GPT2_1_5B_2.json', '../expe…

### Calculating `g-index` from Values

```python
style = {"description_width": "100px"}
n_tasks_domain = IntSlider(value=5,min=1,max=100,step=1,description="Samples per Task domain",style=style)
n_curricula_domain = IntSlider(value=5,min=1,max=100,step=1,description="Samples per Curricula domain",style=style)
sim_dd = FloatSlider(value=0.0,min=0.0,max=1.00,step=0.01,description="Generalization Difficulty",style=style)
sim_P = FloatLogSlider(value=-3, min=-10, max=10, step=1, description="Prior", style=style)
sim_E = FloatLogSlider(value=1, min=1, max=12, step=1, description="Experience", style=style)
sim_PTheta = FloatSlider(value=0.1,min=0.00,max=1.00,step=0.01,description="Performance θ",style=style)

@interact(n_tasks_domain= n_tasks_domain,n_curricula_domain = n_curricula_domain,sim_dd = sim_dd,sim_P= sim_P,sim_E = sim_E,sim_PTheta = sim_PTheta)
def get_g_index_sim_i(n_tasks_domain,n_curricula_domain,sim_dd,sim_P,sim_E,sim_PTheta):
    exp = Experiment()
    g_index =  exp.simulate_g_index(n_tasks_domain= n_tasks_domain,n_curricula_domain = n_curricula_domain,sim_dd = sim_dd,sim_P= sim_P,sim_E = sim_E,sim_PTheta = sim_PTheta)
    print(f"g-index: {g_index}")
```

    interactive(children=(IntSlider(value=5, description='Samples per Task domain', min=1, style=SliderStyle(descr…

## 4. Reproducing results mentioned in the paper

## Disclaimer

The experiments data & the domain samples provided are a small fraction of the data used during the research.
Due to this, your results might vary from the reported results.

Interested people can go to the section 5 to get more info.

### Section-4.1 Simulation Plots

#### g-index vs Training Samples ( for varying $\theta$ )

<big> Setting up the following Simulation variables</big>

1. Number of samples per curricula domain (Varying)
2. Average performance (Varying)
3. Number of samples per task domain (Fixed)
4. Iterations per domain
5. Raw experience (compute)
6. Priors

```python
ns_curricula_domain = [ 20 * i for i in range(1, 33)]# No of Samples per domain from 20 to 640
sim_PthetaValues = np.around( np.arange(0.2, 1.2, 0.2), 2).tolist()  # Average Performance Values from 0.2 to 1.0
num_domains = len(AVAILABLE_DOMAINS)
n_tasks_domain = 5
iters_per_domain = 20
sim_E = 1e2
sim_P = 1e-3
sim_dd = None
dd_cache = cache_dd() # To speed up the simulations by using precalculated domain distances across domains.
```

```python
results_gt = defaultdict(dict)
for sim_Ptheta in tqdm(sim_PthetaValues):
    dummy_exp = Experiment(dd_cache=dd_cache) # Setup a dummy experiment to simulate values
    ns_gi = {"samples_per_domain": [], "g_index": []}
    for ns_per_domain in ns_curricula_domain:
        for _ in range(iters_per_domain):
            # Randomly distribute samples across domains
            CurriculaDistribution = dict(zip(AVAILABLE_DOMAINS, resplit(ns_per_domain, num_domains)))
            # Get the value of g-index from simulations
            g_index = dummy_exp.simulate_g_index(n_tasks_domain=n_tasks_domain,sim_P=sim_P,sim_E=sim_E,sim_PTheta=sim_Ptheta,
                                                 CurriculaDistribution=CurriculaDistribution,sim_dd=sim_dd)
            ns_gi["samples_per_domain"].append(ns_per_domain)
            ns_gi["g_index"].append(g_index)
    results_gt[sim_Ptheta] = ns_gi
```

    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:56<00:00, 11.25s/it]

```python
fig, axes = plt.subplots(figsize=FIGSIZE)
xlabel = "Training Samples (per domain)"
ylabel = "g-index"
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.xlim(20, 640)
plt.ylim(0,13000)
set_axes() # Fix axes spines & ticks, see utils.py for more details

for idx, sim_Ptheta in enumerate(results_gt.keys()):
    vals = results_gt[sim_Ptheta]
    x = vals["samples_per_domain"]
    y = vals["g_index"]
    # Curve Smoothing with 3 degree polynomial
    fx = np.poly1d(np.polyfit(np.array(x), np.array(y), 3))
    z = fx(x)
    # Plot the confidence interval
    sns.lineplot(x=x, y=y, ax=axes, legend=False,linewidth=0, color=sim_colors[idx])
    # Plot against the smoothed value
    sns.lineplot(x=x,y=z,ax=axes,legend=False,linewidth=2,label=f"$\\theta$ = {sim_Ptheta}",color=sim_colors[idx])
plt.legend(loc="upper right", labelcolor="black",fontsize=LEGEND_SIZE)
plt.tight_layout()
plt.title(f"{ylabel} vs {xlabel}", loc="center", fontsize=TITLE_SIZE)
plt.show()
```

#### g-index vs Compute ( for varying $\theta$ )

<big> Setting up the following Simulation variables</big>

1. Compute steps (varying)
2. Average performance (varying)
3. Number of samples per curricula domain (fixed)
4. Number of samples  per task domain (fixed)
5. Iterations per domain (fixed)
6. Priors (fixed)

```python
sim_compute_steps = [50 * i for i in range(1, 31)] # Compute from 50-1500
sim_PthetaValues = np.around(np.arange(0.2, 1.2, 0.2), 2).tolist()  # Average Performance Values from 0.2 to 1.0
n_samples_per_domain = 40
n_tasks_domain = 5
num_domains = len(AVAILABLE_DOMAINS)
sim_P = 1e-2
iters_per_compute_step = 5
dd_cache = cache_dd()
sim_dd = None
```

```python
results_gcp = defaultdict(dict)

for sim_Ptheta in tqdm(sim_PthetaValues):
    dummy_exp = Experiment(dd_cache=dd_cache) # Setup a dummy experiment to simulate values
    gi = {"compute": [], "g_index": []}
    for sim_E in sim_compute_steps:
        for itp in range(iters_per_compute_step):

            CurriculaDistribution = dict(zip(AVAILABLE_DOMAINS, resplit(n_samples_per_domain, num_domains)))
            g_index = dummy_exp.simulate_g_index(
                n_tasks_domain=n_tasks_domain,
                sim_P=sim_P,
                sim_E=sim_E,
                sim_PTheta=sim_Ptheta,
                CurriculaDistribution=CurriculaDistribution,
                sim_dd=sim_dd)
            gi["compute"].append(sim_E)
            gi["g_index"].append(g_index)

    results_gcp[sim_Ptheta] = gi
```

    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:51<00:00, 10.21s/it]

```python
fig, axes = plt.subplots(figsize=FIGSIZE)
xlabel = "Compute"
ylabel = "g-index"
plt.xlim(50, 1500)
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
set_axes() # Fix axes spines & ticks, see utils.py for more details

for idx, sim_Ptheta in enumerate(results_gcp.keys()):
    vals = results_gcp.get(sim_Ptheta)
    x = vals["compute"]
    y = vals["g_index"]
    # Plot the confidence interval
    sns.lineplot(x=x, y=y, ax=axes, legend=False, linewidth=0, color=sim_colors[idx])
    #Plot the actual non-smoothed line
    sns.lineplot(x=x,y=y,ax=axes,legend=False,linewidth=2,label=f"$\\theta$ = {sim_Ptheta}",color=sim_colors[idx])

plt.legend(loc="upper right", labelcolor="black",fontsize=LEGEND_SIZE)
plt.tight_layout()
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=LEGEND_SIZE)
plt.show()
```

#### G-index vs $\theta$ ( for varying $\Omega$ ) HeatMap

<big> Setting up the following Simulation variables</big>

1. Average performance (varying)
2. Domain distance between domains (varying)
3. Number of samples per curricula domain (fixed)
4. Number of samples  per task domain (fixed)
5. Raw Experience ( compute, fixed )
6. Priors (fixed)
7. Iterations per domain (fixed)

```python
sim_PthetaValues = np.around(np.linspace(0.00, 1, 50), 2) # Average Performance Values from 0.2 to 1.0
n_curricula_domain = 40
n_tasks_domain = 5
num_domains = len(AVAILABLE_DOMAINS)
sim_E = 1e2
sim_P = 1e-2
n_iters_per_temp = 10
dd_cache = cache_dd()
sim_dd = None
```

```python
results_gt = defaultdict(dict)

for sim_PTheta in tqdm(sim_PthetaValues):
    gi_dd = {"theta": [], "g_index": [], "domain_distance": []}
    dummy_exp = Experiment() # Setup a dummy experiment to simulate values
    for _ in range(n_iters_per_temp):
        #Randomly sample domain_distance values for every two domains from a normal distribution
        random_dd = {tA: {tB: val for tB, val in zip(AVAILABLE_DOMAINS,np.clip(np.random.normal(0.5, 0.4, 16), 0, 1).tolist())} for tA in AVAILABLE_DOMAINS}
        g_index = dummy_exp.simulate_g_index(
            n_tasks_domain=n_tasks_domain,
            n_curricula_domain=n_curricula_domain,
            sim_P=sim_P,
            sim_E=sim_E,
            sim_PTheta=sim_PTheta,
            sim_dd=random_dd,
        )
        gi_dd["theta"].append(sim_PTheta)
        gi_dd["g_index"].append(g_index)
        # get random row and just get it's domain_dist average
        gi_dd["domain_distance"].append(random_row_mean(random_dd))
    results_gt[sim_PTheta] = gi_dd
```

    100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:25<00:00,  2.00it/s]

```python
ylabel = "g-index"
xlabel = "$\\theta$"
fig, axes = plt.subplots(figsize=FIGSIZE)
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.xlim(0, 1.01)
plt.ylim(0, 4000)
set_axes() # Fix axes spines & ticks, see utils.py for more details

xtotal, ytotal = [], []
for sim_PTheta in results_gt.keys():
    vals = results_gt.get(sim_PTheta)
    x = vals["theta"]
    y = vals["g_index"]
    h = vals["domain_distance"]
    xtotal.extend(x)
    ytotal.extend(y)
    #Plot the points x & y
    sns.scatterplot(x=x, y=y, hue=h, ax=axes, legend=False, s=S, alpha=ALPHA, palette="rocket")
#Gather all points x & y  & apply polynomial smoothing
fx = np.poly1d(np.polyfit(np.array(xtotal), np.array(ytotal), 3))
z = fx(xtotal)
#Plot the smoothed line
sns.lineplot(x=xtotal, y=z, ax=axes, legend=False, linewidth=1, color="black")
# Add the colorbar for heatmap
sm = plt.cm.ScalarMappable(cmap="rocket")
axes.figure.colorbar(sm).set_label(label="Domain Distance",size=LEGEND_SIZE)

plt.tight_layout()
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=TITLE_SIZE)
plt.show()
```

### Section-4.2 Experiment Plots

#### Average Performance ( $\theta$ )  vs Training Samples

```python
results = {"IS": [], "avg_perf": [], "total_samples": [], "g_index": []}

for exp_file in EXP_FILES:
    exp = Experiment(exp_file)
    exp_c = exp.get_exp_components() # Get components of an experiment
    avg_perf = exp_c.AveragePerformance # Average Performance of an experiment
    IS = exp_c.IS # Get the IS name of an experiment
    CurriculaDomains = exp_c.CurriculaDomains # Get Curricula domains for an experiment
    total_samples = sum([domain["num_samples"] for domain in CurriculaDomains]) # Total curricula samples for an experiment
    g_index = exp.calculate_g_index() # Get the g-index for an experiment

    results["IS"].append(IS)
    results["avg_perf"].append(avg_perf)
    results["total_samples"].append(total_samples)
    results["g_index"].append(g_index)
```

```python
gi = results["g_index"]
g_i_dots_scales = [scale_dots(min(gi), c) for c in gi] #Calculate the scatterplot dots sizes based on g-index values
#Create a data list of (Total Samples,Avg Performance, IS name, gindex_dot_sizes)
data_list = [[x, y, model_name, marker_size] for (x, y, model_name, marker_size) in zip(results["total_samples"], results["avg_perf"], results["IS"], g_i_dots_scales)]
```

```python
fig, axes = plt.subplots()
ylabel = "$\\theta$"
xlabel = "Training Samples"
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.ylim(0, 1)
set_axes(disable_ticks=False) # Fix axes spines & ticks, see utils.py for more details

color_name = "black"
axes.spines["top"].set_color(color_name)
axes.spines["bottom"].set_color(color_name)
axes.spines["left"].set_color(color_name)
axes.spines["right"].set_color(color_name)

#plot the points step by step
for entry in data_list:
    color = model_colormap[model_name_correction[entry[2]]] # Get color value for a specified model
    sns.scatterplot(x=[entry[0]], y=[entry[1]], color=color, s=entry[3], alpha=ALPHA) # plot every point on the axes

plt.tight_layout()
# Dummy plot to generate legend
markers = [plt.Line2D([0, 0], [0, 0], color=color, marker="o", linestyle="") for color in model_colormap.values()]
plt.legend(markers, model_colormap.keys(),fontsize=LEGEND_SIZE)
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=TITLE_SIZE)
plt.show()
```

#### Average Domain Performance ( $\theta$ ) vs Program Size

```python
domains_considered = AVAILABLE_DOMAINS
results_pst = {"IS": [], "domain_name": [], "program_size": [], "theta": []}

for exp_file in EXP_FILES:
    exp = Experiment(exp_file) # Setup a dummy experiment to simulate values
    for domain in domains_considered:
        exp_c = exp.get_exp_components() # Get components of an experiment
        IS = exp_c.IS # Get the IS name of an experiment
        perf_details = exp_c.PerformanceDetails # Per domain performance of an experiment
        perf_details = next(item for item in perf_details if item["name"] == domain)
        program_size = DOMAIN_LENGTHS[domain]

        exp_c = exp.get_exp_components() # Get components of an experiment
        results_pst["IS"].append(IS)
        results_pst["domain_name"].append(domain)
        results_pst["program_size"].append(program_size)
        results_pst["theta"].append(perf_details["performance"])
df = pd.DataFrame(results_pst).groupby(by=["domain_name", "IS"]).mean()
```

```python
fig, axes = plt.subplots()
xlabel = "Program size"
ylabel = "$\\theta$"
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.ylim(0, 1)
set_axes(disable_ticks=False) # Fix axes spines & ticks, see utils.py for more details

for temp in domains_considered:
    tdf = df.loc[temp]
    tdf = tdf.reindex(list(model_name_correction.keys()))
    x = tdf["program_size"]
    y = tdf["theta"]
    for model in model_name_correction.keys():
        x = tdf.loc[model]["program_size"]
        y = tdf.loc[model]["theta"]
        color = model_colormap[model_name_correction[model]]
        sns.scatterplot(x=[x], y=[y], color=color, s=S, alpha=ALPHA)

plt.tight_layout()
# Dummy Plot to Generate Legend
markers = [
    plt.Line2D([0, 0], [0, 0], color=color, marker="o", linestyle="")
    for color in model_colormap.values()
]
plt.legend(markers, model_colormap.keys(),fontsize=LEGEND_SIZE)
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=TITLE_SIZE)
plt.show()
```

#### Skill level vs Program Size

```python
results_def = {"IS": [], "domain_name": [], "program_size": [], "skill_level": []}
for exp_file in EXP_FILES:
    exp = Experiment(exp_file)
    for domain in domains_considered:
        exp_c = exp.get_exp_components() # Get components of an experiment
        IS = exp_c.IS # Get the IS name of an experiment
        perf_details = exp_c.PerformanceDetails # Per domain performance of an experiment

        perf_details = next(item for item in perf_details if item["name"] == domain)
        program_size = DOMAIN_LENGTHS[domain]
        results_def["IS"].append(IS)
        results_def["domain_name"].append(domain)
        results_def["program_size"].append(program_size)
        results_def["skill_level"].append(perf_details["perfects"])
df = pd.DataFrame(results_def).groupby(by=["domain_name", "IS"]).mean()
```

```python
fig, axes = plt.subplots()
xlabel = "Program size"
ylabel = "Skill level"
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.ylim(0, 1)
set_axes(disable_ticks=False) # Fix axes spines & ticks, see utils.py for more details

#plotting the points domain-wise
for domain in domains_considered:
    tdf = df.loc[domain] #temporary_df for the domain
    tdf = tdf.reindex(model_name_correction.keys()) #reindex by model names
    x = tdf["program_size"]
    y = tdf["skill_level"]
    for model in model_name_correction.keys(): #plot individual points per domain
        x = tdf.loc[model]["program_size"]
        y = tdf.loc[model]["skill_level"]
        color = model_colormap[model_name_correction[model]] # get the color for a model.
        sns.scatterplot(x=[x], y=[y], color=color, s=S, alpha=ALPHA)

plt.tight_layout()
# Dummy Plot to Generate Legend
markers = [plt.Line2D([0, 0], [0, 0], color=color, marker="o", linestyle="") for color in model_colormap.values()]
plt.legend(markers, model_colormap.keys(),fontsize=LEGEND_SIZE)
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=TITLE_SIZE)
plt.show()
```

#### Compute Vs Average Performance ( $\theta$ )

```python
results_ct = {"IS": [], "avg_perf": [], "compute": [], "g_index": []}
for exp_file in EXP_FILES:
    exp = Experiment(exp_file)
    exp_c = exp.get_exp_components(return_raw_experience=True) # Get components of an experiment with raw compute
    IS = exp_c.IS # Get the IS name of an experiment
    avg_perf = exp_c.AveragePerformance # Average Performance of an experiment
    compute = exp_c.E # Get compute for an experiment
    g_index = exp.calculate_g_index() #Calculate g-index for the experiment
    results_ct["IS"].append(model_name_correction[IS])
    results_ct["avg_perf"].append(avg_perf)
    results_ct["compute"].append(compute)
    results_ct["g_index"].append(g_index)
```

```python
g_i_dots_scales = [scale_dots(min(gi), c) for c in results_ct["g_index"]]#Calculate the scatterplot dots sizes based on g-index values
#Create a data list of (Total Samples,Avg Performance, IS name, gindex_dot_sizes)
data_list = [[x,y,marker_size,model_name] for (x,y,marker_size,model_name) in zip(results_ct["compute"], results_ct["avg_perf"], g_i_dots_scales, results_ct["IS"])]
```

```python
fig, axes = plt.subplots()
ylabel = "Performance"
xlabel = "Compute"
plt.xlabel(xlabel, fontsize=LABEL_SIZE)
plt.ylabel(ylabel, fontsize=LABEL_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
axes.tick_params(labelsize=TICK_SIZE)
plt.ylim(0, 1.01)
set_axes(disable_ticks=False) # Fix axes spines & ticks, see utils.py for more details

#plot the points step by step
for entry in data_list:
    sns.scatterplot(x=[entry[0]],y=[entry[1]],color=model_colormap[entry[3]],s=entry[2],alpha=ALPHA) # plot every point on the axes

markers = [plt.Line2D([0, 0], [0, 0], color=color, marker="o", linestyle="") for color in model_colormap.values()]
plt.legend(markers, model_colormap.keys(),fontsize=LEGEND_SIZE)
plt.tight_layout()
plt.title(f"{ylabel} vs {xlabel}", loc="center",fontsize=TITLE_SIZE)
plt.show()
```

## 5. Request the data

Please send us a mail at the [humans@mayahq.com](mailto:humans@mayahq.com) with the following details to get the data.

Email Subject: `g-index data request`

```markdown
Name:
I/We are an : [Individual/Organisation]
Our use case : [Brief Description]
```

## 6. Cite us!

If this repository, the paper or any of its content is useful for your research, please cite:

```
@misc{venkatasubramanian2021measure,
      title={Towards A Measure Of General Machine Intelligence},
      author={Gautham Venkatasubramanian and Sibesh Kar and Abhimanyu Singh and Shubham Mishra and Dushyant Yadav and Shreyansh Chandak},
      year={2021},
      eprint={2109.12075},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}
```