# Corona Virus Data

The Corona Virus, COVID-19, is one of the most significant events of recent years, affecting the entire world. Social distancing measures and shelter in place orders have been enacted to slow the spread of the pandemic. These interventions come with the cost of slowed social and economic activity.

The goal of this notebook is to explore corona virus data from a list of sources including those aggregated by [Johns Hopkins University Center for Systems Science and Engineering](https://github.com/CSSEGISandData/COVID-19) and the [New York Times](https://github.com/nytimes/covid-19-data). This work is intended to make use of *data science* skills, including programming, data visualization and modeling to remind readers of the bigger picture.

The structure of this notebook is as follows:

1. [Convenience Functions](#functions)
2. [Data Preprocessing](#data)
3. [Exploratory Data Analysis](#eda)
4. [SIR Model Background](#SIR)
5. [Model Implementation and Fits](#fits)

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re
import plotly.express as px
from fuzzywuzzy import fuzz, process
from collections import defaultdict
```

# Convenience Functions

```python
# Functions to load data
def load_raw_global():
    """Loads confirmed, deaths, recovered data frames in a dict."""
    path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/' \
            'csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_'
    world = {i: pd.read_csv(path+i.lower()+'_global.csv').drop(columns=['Lat', 'Long'])
             for i in ['Confirmed', 'Deaths', 'Recovered']}
    return world

def load_local():
    """Loads state, county data in tidy format, packaged in a dict."""
    path = 'https://raw.githubusercontent.com/nytimes/covid-19-data/master/'
    data ={'state': pd.read_csv(path + 'us-states.csv'),
            'county': pd.read_csv(path + 'us-counties.csv')}
    data['state'].rename(columns={'date': 'Date', 'state':'State',
                                  'cases':'Confirmed','deaths':'Deaths'}, inplace=True)
    data['county'].rename(columns={'date': 'Date', 'county':'County',
                                   'state':'State','cases':'Confirmed',
                                   'deaths':'Deaths'}, inplace=True)
    return data

def search_country(data, countries):
    """Narrow search results for confirmed, deaths"""
    data = data[data['Country/Region'].isin(countries)]
    return confirmed, deaths

def time_series(data):
    """Transpose data with time series index, and swaps index and columns"""
    idx = pd.to_datetime(data.columns, errors='coerce')
    ts = (data
          .T
          .loc[data.columns.drop(labels=['Province/State', 'Country/Region', 'Lat', 'Long'],
                                 errors='ignore')
              ]
          .set_index(idx)
         )
    return ts

def load_merged_daily_global(datadict):
    """Create dataframes of daily confirmed, deaths, recovered in tidy (long) format"""
    d_ = list(datadict.values())
    merged = d_[0]
    for d in d_:
        merged = pd.merge(merged, d)
    for key in ['Confirmed', 'Deaths', 'Recovered']:
        merged['Daily ' + key] = (merged
                                  .groupby('Country/Region')
                                  [key]
                                  .diff()
                                  .fillna(merged[key])
                                 )
    return merged

def load_tidy_global(data):
    """Load confirmed, deaths, recovered in tidy (long) format"""
    datadict = {}
    for key, name in zip(data, ['Confirmed', 'Deaths', 'Recovered']):
        country_aggregated = data[key].groupby('Country/Region').agg(sum).reset_index()
        datadict[key] = pd.melt(country_aggregated, 'Country/Region', var_name='Date', value_name=name)
    return datadict

def load_merged_daily_local(data):
    """Create dict of dataframes of state or county level daily cases, deaths in tidy (long) format in a dictionary."""
    data['Date'] = pd.to_datetime(data.Date)
    iscounty = 'County' in data.columns
    group = 'State'
    if iscounty:
        group = ['County', 'State']
    for i in ['Confirmed', 'Deaths']:
        data['Daily ' + i]= data.groupby(group)[i].diff().fillna(data[i])
    return data

def load_daily_county(county):
    """Create dict of dataframes of county-level daily cases, deaths
        in tidy (long) format in a dictionary"""
    daily_county = {}
    county['Date'] = pd.to_datetime(county['Date'])
    multi_idx = county.set_index(['County', 'State']).index
    county['County_State'] = multi_idx
    for i in ['Cases', 'Deaths']:
        df = (county
                     .pivot(index='Date', columns='County_State', values=i)
                     .fillna(0)
                     .diff()
                     .reset_index()
                    )
        daily_county[i] = pd.melt(df, id_vars='Date', var_name=['County_State'],
                                  value_name='Daily '+i)
        daily_county[i]['County'] = daily_county[i]['County_State'].apply(lambda x: x[0])
        daily_county[i]['State'] = daily_county[i]['County_State'].apply(lambda x: x[1])
    return daily_county

# Functions to filter data
def top_n_countries(world_confirmed, feature='Confirmed', n=10):
    """Return list of top n countries with highest feature (Confirmed, Deaths, Recovered, Daily ...)"""
    top_n = (world_confirmed
              .groupby('Country/Region')
              .agg('max')
              .sort_values(feature, ascending=False)
              .head(n)
              .index
              .values)
    return list(top_n)

def top_n_states(states, feature='Confirmed', n=10):
    top_n = (states
              .groupby('State')
              .agg('max')
              .sort_values(feature, ascending=False)
              .head(n)
              .index
              .values)
    return list(top_n)

def top_n_counties(counties, feature='Confirmed', n=10):
    top_n = (counties
                .groupby('County_State')
                .agg['max']
                .sort_values('Confirmed', ascending=False)
                .head(n)
                .index
                .values)
    return list(top_n)

def filter_countries(data, countries):
    return data[data['Country/Region'].isin(countries)]

def filter_states(data, states):
    return data[data['States'].isin(states)]

def filter_counties(data, counties):
    return data[data['County_States'].isin(counties)]

# Text processing
def convert2num(text):
    """
    Converts unit free text into numbers

    Parameters:
    ----------------
    text: string
        description

    Returns:
    ----------------
    num: float
         Data converted to numerical form

    """
    if text is np.nan:
        return 0
    if type(text) in [np.float64, float, int, np.int64, np.float32, np.float16,
                      np.int32, np.int16]:
        return text

    factor = 1
    dd = defaultdict(lambda: 1, {'m': 1e6, 'b': 1e9, 'tr': 1e12})
    unit = re.search(r'([a-z]*)(illion)', text)
    if unit:
        factor = dd[unit.group(1)]
        text = text.replace(unit.group(),'').rstrip()

    found =  re.search(r'-?\d+[\.]?\d*', text.replace('$', '').replace(',', ''))

    if found:
        try:
            num= float(found.group())*factor
        except:
            num = text
            print('Had trouble converting %s' % text)
            print('Obtained: ', found.group())
            return num
    else:
        num = text
    return num

def remove_stop_words(stop_words, text):
    """
    Description

    Parameters:
    ----------------
    stop_words: list
        List of strings to remove
    text: string
        String to process

    Returns:
    ----------------
    filtered: string
         String with stop words removed
    """
    text = text.lower()
    for word in stop_words:
        text = re.sub(word.lower(), '', text)

    return text

def search_fields(search_term, search_space, scorer='partial', cutoff=80, max_results=15):
    """
    Searches fields in countries dataset by partial string matching

    Parameters:
    ----------------
    search_term: string
        A string that approximates the matches
    search_space: list-like object
        Words to search from
    scorer: {'simple', 'partial', 'token sort', 'tokens set ratio'}
        Strings to specify scorers fuzz.ratio, fuzz.partial_ratio, fuzz.token_set_ratio
        respectively
    cutoff: int
        Minimum score to count as a match
    max_results: int
        Limit on how many results to show

    Returns:
    ----------------
    results: list
        List of words that meet cutoff score

    """
    scorers = {'simple':fuzz.ratio,
           'partial': fuzz.partial_ratio,
           'token sort': fuzz.token_sort_ratio,
           'token set': fuzz.token_set_ratio}

    if scorer not in scorers.keys():
        print(scorer, ' not found among valid scorers. Using default instead.')
        scorer = 'partial'

    results = process.extract(search_term, search_space,
                       scorer=scorers[scorer], limit=max_results)

    filtered_results = [(field, score) for field, score in results if score >=cutoff]

    return filtered_results

def find_aliases(match_space, search_space, stop_words=[],
                 scorers=['partial', 'token sort'], cutoff=70, max_results=5):
    """
    Creates a dictionary between entries of one list and the analogs of another
    through consecutive string matches

    Parameters:
    ----------------
    match_space: list-like
        Index to translate to best matches
    search_space: list-like
        Analogous list to search from
    stop_words: list
        Words to exclude from the match_space
    scorers: list of strings
        List containing first scorer and second scorer for string matching
    cutoff: int
        Threshold for words to consider for matching
    max_results: int
        Expected number of results with top scores

    Returns:
    ----------------
    best_matches: dict
        Dictionary with match_space words as keys and best matches from search_space
        as values
    unmatched: list
        List of words from the match_space with no matches

    """
    match_set = set(match_space)
    search_set = set(search_space)
    intersection = match_set & search_set
    diff = match_set - intersection
    search_space = search_set - intersection
    unmatched = []
    best_matches = {word: word for word in intersection}
    for word in diff:

        # First filter keeping only max scores
        match1 = search_fields(remove_stop_words(stop_words, word), search_space,
                              scorer=scorers[0], max_results=max_results, cutoff=cutoff)
        if len(match1)>0:
            best_score = match1[0][1]
            first_round_best = [tup[0] for tup in match1 if tup[1]==best_score]

            # Second filter
            match2 = search_fields(remove_stop_words(stop_words, word), first_round_best,
                                             scorer=scorers[1], max_results=1,
                                             cutoff=cutoff)
            if len(match2) > 0:
                best_matches[word] = match2[0][0]
            else:
                best_matches[word] = match1[0][0]
        else:
            unmatched.append(word)

    return best_matches, unmatched

def translate(text, d):
        """Translate text to alias if present in dicitonary"""
        try:
            return d[text]
        except:
            return text

def load_population_data():
    """Load and prepare country populations with same keys as covid19 data."""
    # Load population data
    populations_counties = pd.read_csv('https://www2.census.gov/programs-surveys/popest/datasets/'\
                           '2010-2019/counties/totals/co-est2019-alldata.csv',encoding='ISO-8859-1')
    populations_states = populations_counties[populations_counties.STNAME==populations_counties.CTYNAME]
    populations_countries = pd.read_csv('countries.csv')

    # Load COVID-19 data
    state, county = load_local().values()
    world = load_raw_global()

    # Sets of keys from each data source
    covid19_keys = {'county': set(county.groupby(['County','State']).max().index.values),
                        'state': set(state.State.unique()),
                        'country': set(world['Confirmed'].groupby('Country/Region').max().index.values),
                       }

    pop_keys = {'county': set(populations_counties[~populations_counties
                                                     .duplicated(['STNAME', 'CTYNAME'])
                                                ]
                                    .groupby(['CTYNAME', 'STNAME'])
                                    .max()
                                    .index
                            ),
                    'state': set(populations_states.groupby('STNAME').max().index),
                    'country': set(populations_countries.country.unique())
    }

    # Preparing initial nation, state and county dataframes
    df_countries = populations_countries.applymap(convert2num)[['country', 'Population']]
    df_countries.rename(columns={'country': 'Country/Region'}, inplace=True)
    df_states = (populations_states[~populations_states[['STNAME', 'CTYNAME']]
                                .duplicated()][['STNAME', 'POPESTIMATE2019']]
            .rename(columns={'STNAME': 'State', 'POPESTIMATE2019': 'Population'})
            )

    territories = ['American Samoa', 'Guam', 'Northern Mariana Islands', 'Puerto Rico',
                 'Virgin Islands']
    df_territories = (df_countries[df_countries['Country/Region'].isin(territories)]
                  .rename(columns={'Country/Region': 'State'})
                 )
    df_states_territories = pd.concat([df_states, df_territories], ignore_index=True)

    #  Country Key Dictionary, including cruise ships
    country_search_space = df_countries['Country/Region']
    country_match_space = covid19_keys['country']
    covid19_pop_countries, unmatched_countries = find_aliases(country_match_space, country_search_space)
    pop_covid19_countries = {val: key for key, val in covid19_pop_countries.items()}
    # manual inputs
    pop_covid19_countries['United States'] = 'US'
    pop_covid19_countries['Congo, Republic of the'] = 'Congo (Brazzaville)'
    pop_covid19_countries['Congo, Democratic Republic of the'] = 'Congo (Kinshasa)'
    df_cruise_ships = pd.DataFrame([['The Diamond Princess',3711], ['MS Zaandam', 1243]],
                                      columns=['Country/Region', 'Population'])
    df_global = df_countries.append(df_cruise_ships, ignore_index=True)
    # State Key Dictionary
    state_search_space = df_states_territories.State
    state_match_space = covid19_keys['state']
    covid19_pop_states, unmatched_states = find_aliases(state_match_space, state_search_space)
    pop_covid19_states = {val: key for key, val in covid19_pop_states.items()}

    # County Key Dictionary, including NYC entry
    county_search_space = county.County.unique()
    county_match_space = populations_counties.CTYNAME.unique()
    covid19_pop_counties, unmatched_counties = find_aliases(county_search_space, county_match_space)
    boroughs = ['Richmond County', 'Queens County', 'Kings County', 'Bronx County', 'New York County']
    inNYstate = populations_counties.STNAME=='New York'
    same_counties = populations_counties.CTYNAME.isin(boroughs)
    inNYC = same_counties & inNYstate
    NYC_pop = populations_counties[inNYC].loc[:, ('CTYNAME', 'POPESTIMATE2019')]
    NYC_2019est = populations_counties[inNYC].loc[:,'POPESTIMATE2019'].sum()
    df_NYC = pd.DataFrame({'CTYNAME': 'New York City', 'STNAME': 'New York',
                       'POPESTIMATE2019': NYC_2019est}, index=[0])
    populations_counties.append(df_NYC, ignore_index=True)
    covid19_pop_counties['New York City'] = 'New York City'
    pop_covid19_counties = {val: key for key, val in covid19_pop_counties.items()}

    pop_covid19_US = {**pop_covid19_counties, **pop_covid19_states}
    df_counties = (populations_counties
                              .append(df_NYC, ignore_index=True)
                              .loc[:, ('CTYNAME', 'STNAME','POPESTIMATE2019')]
                              .rename(columns={'CTYNAME':'County', 'STNAME': 'State',
                                               'POPESTIMATE2019':'Population'})

                             )

    df_counties.loc[:,('County','State')] = (df_counties[['County','State']]
                                        .applymap(lambda x: translate(x, pop_covid19_US))
                                       )

    return df_global, df_states_territories, df_counties

def create_per_capita_features(data, population):
    """Merges data set with population data and creates per capita numerical features"""

    num_types = ['int8','int16', 'int32', 'int64', 'float16', 'float32', 'float64']
    df = pd.merge(data, population)
    numerical_fields = [i for i in data.columns
                        if data[i].dtype in num_types]
    for i in numerical_fields:
        df[i+' per capita'] = df[i]/df['Population']
    return df

```

#  Population Data Preprocessing

## Translating to Common Keys

The COVID-19 data contains raw number of confirmed cases, deaths and recovered. To compare these numbers among different countries, states, and counties, we may wish to factor in populations, which requires population estimates for each entity. To eventually join these datasets, which will require *joins* based on common *keys*, specifically the geographic names.

This requires that the population data and COVID-19 data to have consistent analogous keys:

1. COVID-19 Data:
    - countries
    - states
    - counties
2. Population Data:
    - countries
    - states
    - counties

Our approach will be:
1. loading population data
2. loading COVID-19 data
3. create sets of unique location entities of each data set
3. compare number of location entities in each data set
4. use string matching library `fuzzywuzzy` to create dictionaries between location keys
5. check a few examples manually

```python
# Load population data
populations_counties = pd.read_csv('https://www2.census.gov/programs-surveys/popest/datasets/'\
                                   '2010-2019/counties/totals/co-est2019-alldata.csv',encoding='ISO-8859-1')
populations_states = populations_counties[populations_counties.STNAME==populations_counties.CTYNAME]
populations_countries = pd.read_csv('countries.csv')

# Load COVID-19 data
state, county = load_local().values()
world = load_raw_global()
```

```python
# Sets of keys from each data source
covid19_keys = {'county': set(county.groupby(['County','State']).max().index.values),
                    'state': set(state.State.unique()),
                    'country': set(world['Confirmed'].groupby('Country/Region').max().index.values),
                   }

pop_keys = {'county': set(populations_counties[~populations_counties
                                                 .duplicated(['STNAME', 'CTYNAME'])
                                            ]
                                .groupby(['CTYNAME', 'STNAME'])
                                .max()
                                .index
                        ),
                'state': set(populations_states.groupby('STNAME').max().index),
                'country': set(populations_countries.country.unique())
}
```

```python
# Comparing set cardinalities
d = {'Countries (Population)': [populations_countries.country.nunique(),
                                len(pop_keys['country'])],
     'Countries (COVID-19)': [world['Confirmed']['Country/Region'].nunique(),
                              len(covid19_keys['country'])],
     'States (Population)': [populations_states.STNAME.nunique(),
                             len(pop_keys['state'])],
     'States (COVID-19)': [state.State.nunique(),
                           len(covid19_keys['state'])],
     'Counties (Population)': [populations_counties.CTYNAME.nunique(),
                               len(pop_keys['country'])],
     'Counties (COVID-19)': [county.County.nunique(),
                             len(covid19_keys['county'])]
     }
df = pd.DataFrame(d, index=['Unique Names','Unique Entities'])
df
```

      Countries (Population)
      Countries (COVID-19)
      States (Population)
      States (COVID-19)
      Counties (Population)
      Counties (COVID-19)

      Unique Names
      250
      188
      51
      55
      1927
      1726

      Unique Entities
      250
      188
      51
      55
      250
      2947

```python
# Set differences and intersections
covid19_keys['state'] - pop_keys['state']
```

    {'Guam', 'Northern Mariana Islands', 'Puerto Rico', 'Virgin Islands'}

```python
covid19_keys['state'] & pop_keys['country']
```

    {'Georgia',
     'Guam',
     'Northern Mariana Islands',
     'Puerto Rico',
     'Virgin Islands'}

```python
pop_keys['country'] & pop_keys['state']
```

    {'Georgia'}

```python
covid19_keys['country'] - pop_keys['country']
```

    {'Bahamas',
     'Congo (Brazzaville)',
     'Congo (Kinshasa)',
     'Diamond Princess',
     'Gambia',
     'Holy See',
     'MS Zaandam',
     'Taiwan*',
     'US',
     'West Bank and Gaza'}

### Note:

- The population data has more entities than covid-19 data in the case of countries and counties
- States include:
    - 50 US States
    - District of Columbia
    - territories: American Samoa, Guam, Northern Mariana Islands, Puerto Rico, Virgin Islands

So...
- population data is likely available for all entities with covid19 data
- population data of states and countries should be merged for convenient joins, were it not for Georgia
- state population data should be augmented with the territory data from country population dataset:
    - American Samoa
    - Guam
    - Northern Mariana Islands
    - Puerto Rico
    - Virgin Islands

### Augmenting States Data with Territories from Country Data

```python
df_countries = populations_countries.applymap(convert2num)[['country', 'Population']]
df_countries.rename(columns={'country': 'Country/Region'}, inplace=True)
df_countries.head()
```

      Country/Region
      Population

      0
      American Samoa
      49437.0

      1
      Afghanistan
      36643815.0

      2
      Albania
      3074579.0

      3
      Angola
      32522339.0

      4
      Algeria
      42972878.0

```python
territories = ['American Samoa', 'Guam', 'Northern Mariana Islands', 'Puerto Rico',
                 'Virgin Islands']
df_territories = (df_countries[df_countries['Country/Region'].isin(territories)]
                  .rename(columns={'Country/Region': 'State'})
                 )
df_territories
```

      State
      Population

      0
      American Samoa
      49437.0

      91
      Guam
      168485.0

      166
      Northern Mariana Islands
      51433.0

      181
      Puerto Rico
      3189068.0

      238
      Virgin Islands
      106235.0

```python
df_states = (populations_states[~populations_states[['STNAME', 'CTYNAME']]
                                .duplicated()][['STNAME', 'POPESTIMATE2019']]
            .rename(columns={'STNAME': 'State', 'POPESTIMATE2019': 'Population'})
            )
df_states.head()
```

      State
      Population

      0
      Alabama
      4903185

      68
      Alaska
      731545

      98
      Arizona
      7278717

      114
      Arkansas
      3017804

      190
      California
      39512223

```python
df_states_territories = pd.concat([df_states, df_territories], ignore_index=True)
df_states_territories
```

      State
      Population

      0
      Alabama
      4903185.0

      1
      Alaska
      731545.0

      2
      Arizona
      7278717.0

      3
      Arkansas
      3017804.0

      4
      California
      39512223.0

      5
      Colorado
      5758736.0

      6
      Connecticut
      3565287.0

      7
      Delaware
      973764.0

      8
      District of Columbia
      705749.0

      9
      Florida
      21477737.0

      10
      Georgia
      10617423.0

      11
      Hawaii
      1415872.0

      12
      Idaho
      1787065.0

      13
      Illinois
      12671821.0

      14
      Indiana
      6732219.0

      15
      Iowa
      3155070.0

      16
      Kansas
      2913314.0

      17
      Kentucky
      4467673.0

      18
      Louisiana
      4648794.0

      19
      Maine
      1344212.0

      20
      Maryland
      6045680.0

      21
      Massachusetts
      6892503.0

      22
      Michigan
      9986857.0

      23
      Minnesota
      5639632.0

      24
      Mississippi
      2976149.0

      25
      Missouri
      6137428.0

      26
      Montana
      1068778.0

      27
      Nebraska
      1934408.0

      28
      Nevada
      3080156.0

      29
      New Hampshire
      1359711.0

      30
      New Jersey
      8882190.0

      31
      New Mexico
      2096829.0

      32
      New York
      19453561.0

      33
      North Carolina
      10488084.0

      34
      North Dakota
      762062.0

      35
      Ohio
      11689100.0

      36
      Oklahoma
      3956971.0

      37
      Oregon
      4217737.0

      38
      Pennsylvania
      12801989.0

      39
      Rhode Island
      1059361.0

      40
      South Carolina
      5148714.0

      41
      South Dakota
      884659.0

      42
      Tennessee
      6829174.0

      43
      Texas
      28995881.0

      44
      Utah
      3205958.0

      45
      Vermont
      623989.0

      46
      Virginia
      8535519.0

      47
      Washington
      7614893.0

      48
      West Virginia
      1792147.0

      49
      Wisconsin
      5822434.0

      50
      Wyoming
      578759.0

      51
      American Samoa
      49437.0

      52
      Guam
      168485.0

      53
      Northern Mariana Islands
      51433.0

      54
      Puerto Rico
      3189068.0

      55
      Virgin Islands
      106235.0

### Note:

- Concatenation of state and country population data sets looks successful

- Time to move toward checking for consistency between keys

### Creating Dictionaries Between State Keys

```python
from pprint import pprint
state_search_space = df_states_territories.State
state_match_space = covid19_keys['state']
covid19_pop_states, unmatched_states = find_aliases(state_match_space, state_search_space)
pop_covid19_states = {val: key for key, val in covid19_pop_states.items()}
```

```python
unmatched_states
```

    []

```python
covid19_pop_states
```

    {'Mississippi': 'Mississippi',
     'Northern Mariana Islands': 'Northern Mariana Islands',
     'Hawaii': 'Hawaii',
     'Nevada': 'Nevada',
     'South Carolina': 'South Carolina',
     'Guam': 'Guam',
     'Colorado': 'Colorado',
     'California': 'California',
     'North Carolina': 'North Carolina',
     'Michigan': 'Michigan',
     'Oregon': 'Oregon',
     'New Hampshire': 'New Hampshire',
     'Kentucky': 'Kentucky',
     'Alaska': 'Alaska',
     'West Virginia': 'West Virginia',
     'Missouri': 'Missouri',
     'Georgia': 'Georgia',
     'Massachusetts': 'Massachusetts',
     'Texas': 'Texas',
     'Utah': 'Utah',
     'South Dakota': 'South Dakota',
     'Oklahoma': 'Oklahoma',
     'Delaware': 'Delaware',
     'Iowa': 'Iowa',
     'Illinois': 'Illinois',
     'North Dakota': 'North Dakota',
     'Wyoming': 'Wyoming',
     'Indiana': 'Indiana',
     'Louisiana': 'Louisiana',
     'Wisconsin': 'Wisconsin',
     'Maine': 'Maine',
     'Florida': 'Florida',
     'Puerto Rico': 'Puerto Rico',
     'Minnesota': 'Minnesota',
     'New Jersey': 'New Jersey',
     'Connecticut': 'Connecticut',
     'Virgin Islands': 'Virgin Islands',
     'Alabama': 'Alabama',
     'Vermont': 'Vermont',
     'Arkansas': 'Arkansas',
     'New Mexico': 'New Mexico',
     'District of Columbia': 'District of Columbia',
     'Kansas': 'Kansas',
     'Montana': 'Montana',
     'Nebraska': 'Nebraska',
     'Pennsylvania': 'Pennsylvania',
     'New York': 'New York',
     'Rhode Island': 'Rhode Island',
     'Maryland': 'Maryland',
     'Washington': 'Washington',
     'Tennessee': 'Tennessee',
     'Ohio': 'Ohio',
     'Virginia': 'Virginia',
     'Arizona': 'Arizona',
     'Idaho': 'Idaho'}

### Note
- no unmatched keys

So state populations data is ready to be joined with covid19 state data.

### Creating Dictionaries Between Country Keys

```python
country_search_space = df_countries['Country/Region']
country_match_space = covid19_keys['country']
covid19_pop_countries, unmatched_countries = find_aliases(country_match_space, country_search_space)
pop_covid19_countries = {val: key for key, val in covid19_pop_countries.items()}
unmatched_countries
```

    ['US',
     'Diamond Princess',
     'MS Zaandam',
     'Congo (Brazzaville)',
     'Congo (Kinshasa)']

### Note

- The US is an abbreviation for United States
- Congo (Kinshasa) : [Democratic Republic of Congo](https://en.wikipedia.org/wiki/Democratic_Republic_of_the_Congo)
- Congo (Brazzaville) : [Republic of Congo](https://en.wikipedia.org/wiki/Republic_of_the_Congo)
- [The Diamond Princess](https://en.wikipedia.org/wiki/Diamond_Princess_(ship)) had a total population of 3711
- [MS Zaandaam](https://en.wikipedia.org/wiki/MS_Zaandam#cite_note-GraudianBlog-3) had 1243 passengers and 586 crew, for a total population of 1829

**Manual Modifications to be made include:**
- adding US: United States entries to the key dictionary
- appending cruise ship population data

```python
df_countries[df_countries['Country/Region'].str.contains('Congo')]
```

      Country/Region
      Population

      51
      Congo, Republic of the
      5293070.0

      52
      Congo, Democratic Republic of the
      101780263.0

```python
pop_covid19_countries['United States'] = 'US'
pop_covid19_countries['Congo, Republic of the'] = 'Congo (Brazzaville)'
pop_covid19_countries['Congo, Democratic Republic of the'] = 'Congo (Kinshasa)'
```

```python
df_cruise_ships = pd.DataFrame([['The Diamond Princess',3711], ['MS Zaandam', 1243]],
                                      columns=['Country/Region', 'Population'])
df_global = df_countries.append(df_cruise_ships, ignore_index=True)
```

```python
df_global
```

      Country/Region
      Population

      0
      American Samoa
      49437.0

      1
      Afghanistan
      36643815.0

      2
      Albania
      3074579.0

      3
      Angola
      32522339.0

      4
      Algeria
      42972878.0

      ...
      ...
      ...

      247
      United States
      332639102.0

      248
      Yemen
      29884405.0

      249
      Zimbabwe
      14546314.0

      250
      The Diamond Princess
      3711.0

      251
      MS Zaandam
      1243.0

252 rows × 2 columns

### Converting to a Common Key for Countries

```python
def translate(text, d):
    """Translate text to alias if present in dicitonary"""
    try:
        return d[text]
    except:
        return text

df_global.loc[:,'Country/Region'] = df_global[['Country/Region']].applymap(lambda x: translate(x,pop_covid19_countries))
df_global
```

      Country/Region
      Population

      0
      American Samoa
      49437.0

      1
      Afghanistan
      36643815.0

      2
      Albania
      3074579.0

      3
      Angola
      32522339.0

      4
      Algeria
      42972878.0

      ...
      ...
      ...

      247
      US
      332639102.0

      248
      Yemen
      29884405.0

      249
      Zimbabwe
      14546314.0

      250
      The Diamond Princess
      3711.0

      251
      MS Zaandam
      1243.0

252 rows × 2 columns

### Checking for Consistency Between County Keys

All states were included in the covid19 data, so that leaves the county names to check. Since we're dealing with tuples, we'll have to invidually extract and translate the county portions.

```python
populations_counties.CTYNAME.unique()
```

    array(['Alabama', 'Autauga County', 'Baldwin County', ..., 'Uinta County',
           'Washakie County', 'Weston County'], dtype=object)

```python
covid19_pop_counties, unmatched_counties = find_aliases(county.County.unique(), populations_counties.CTYNAME.unique())
```

```python
unmatched_counties
```

    []

```python
covid19_pop_counties['New York City']
```

    'New York'

### Notes:

- All county names had matches
- New York City was connected to New York county, which only includes Manhattan. This entity in the COVID-19 data comprises of all boroughs, which includes:
    - Staten Island (Richmond County)
    - Queens (Queens County)
    - Brooklyn (Kings County)
    - Bronx (Bronx County)
    - Manhattan (New York County)

This implies we need to manually add each of the populations for the total NYC estimate

```python
boroughs = ['Richmond County', 'Queens County', 'Kings County', 'Bronx County', 'New York County']
inNYstate = populations_counties.STNAME=='New York'
same_counties = populations_counties.CTYNAME.isin(boroughs)
inNYC = same_counties & inNYstate
NYC_pop = populations_counties[inNYC].loc[:, ('CTYNAME', 'POPESTIMATE2019')]
NYC_pop
```

      CTYNAME
      POPESTIMATE2019

      1863
      Bronx County
      1418207

      1884
      Kings County
      2559903

      1891
      New York County
      1628706

      1901
      Queens County
      2253858

      1903
      Richmond County
      476143

```python
NYC_2019est = populations_counties[inNYC].loc[:,'POPESTIMATE2019'].sum()
```

```python
df_NYC = pd.DataFrame({'CTYNAME': 'New York City', 'STNAME': 'New York',
                       'POPESTIMATE2019': NYC_2019est}, index=[0])
populations_counties.append(df_NYC, ignore_index=True)
```

      SUMLEV
      REGION
      DIVISION
      STATE
      COUNTY
      STNAME
      CTYNAME
      CENSUS2010POP
      ESTIMATESBASE2010
      POPESTIMATE2010
      ...
      RDOMESTICMIG2019
      RNETMIG2011
      RNETMIG2012
      RNETMIG2013
      RNETMIG2014
      RNETMIG2015
      RNETMIG2016
      RNETMIG2017
      RNETMIG2018
      RNETMIG2019

      0
      40.0
      3.0
      6.0
      1.0
      0.0
      Alabama
      Alabama
      4779736.0
      4780125.0
      4785437.0
      ...
      1.917501
      0.578434
      1.186314
      1.522549
      0.563489
      0.626357
      0.745172
      1.090366
      1.773786
      2.483744

      1
      50.0
      3.0
      6.0
      1.0
      1.0
      Alabama
      Autauga County
      54571.0
      54597.0
      54773.0
      ...
      4.847310
      6.018182
      -6.226119
      -3.902226
      1.970443
      -1.712875
      4.777171
      0.849656
      0.540916
      4.560062

      2
      50.0
      3.0
      6.0
      1.0
      3.0
      Alabama
      Baldwin County
      182265.0
      182265.0
      183112.0
      ...
      24.017829
      16.641870
      17.488579
      22.751474
      20.184334
      17.725964
      21.279291
      22.398256
      24.727215
      24.380567

      3
      50.0
      3.0
      6.0
      1.0
      5.0
      Alabama
      Barbour County
      27457.0
      27455.0
      27327.0
      ...
      -5.690302
      0.292676
      -6.897817
      -8.132185
      -5.140431
      -15.724575
      -18.238016
      -24.998528
      -8.754922
      -5.165664

      4
      50.0
      3.0
      6.0
      1.0
      7.0
      Alabama
      Bibb County
      22915.0
      22915.0
      22870.0
      ...
      1.385134
      -4.998356
      -3.787545
      -5.797999
      1.331144
      1.329817
      -0.708717
      -3.234669
      -6.857092
      1.831952

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      3189
      50.0
      4.0
      8.0
      56.0
      39.0
      Wyoming
      Teton County
      21294.0
      21298.0
      21296.0
      ...
      -1.583464
      -2.575509
      1.905293
      23.259519
      12.508872
      4.670450
      1.253214
      -0.171608
      -12.432212
      1.797445

      3190
      50.0
      4.0
      8.0
      56.0
      41.0
      Wyoming
      Uinta County
      21118.0
      21121.0
      21089.0
      ...
      -9.181105
      -18.196975
      -4.487730
      -10.870861
      -15.033634
      -10.484550
      -11.485101
      -18.923455
      -13.554993
      -9.181105

      3191
      50.0
      4.0
      8.0
      56.0
      43.0
      Wyoming
      Washakie County
      8533.0
      8528.0
      8530.0
      ...
      -7.652085
      -12.839390
      -3.084589
      -1.307811
      -18.938032
      0.000000
      -15.204038
      -15.950541
      -17.246806
      -8.289759

      3192
      50.0
      4.0
      8.0
      56.0
      45.0
      Wyoming
      Weston County
      7208.0
      7208.0
      7198.0
      ...
      0.000000
      -9.065551
      -10.549265
      6.191515
      0.420345
      9.201171
      0.693097
      -36.227798
      -6.910452
      2.599090

      3193
      NaN
      NaN
      NaN
      NaN
      NaN
      New York
      New York City
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

3194 rows × 164 columns

```python
covid19_pop_counties['New York City'] = 'New York City'
pop_covid19_counties = {val: key for key, val in covid19_pop_counties.items()}

pop_covid19_US = {**pop_covid19_counties, **pop_covid19_states}
```

```python
df_counties = (populations_counties
                              .append(df_NYC, ignore_index=True)
                              .loc[:, ('CTYNAME', 'STNAME','POPESTIMATE2019')]
                              .rename(columns={'CTYNAME':'County', 'STNAME': 'State',
                                               'POPESTIMATE2019':'Population'})

                             )
df_counties.loc[:,('County','State')] = (df_counties[['County','State']]
                                        .applymap(lambda x: translate(x, pop_covid19_US))
                                       )

```

```python
df_counties
```

      County
      State
      Population

      0
      Alabama
      Alabama
      4903185

      1
      Autauga
      Alabama
      55869

      2
      Baldwin
      Alabama
      223234

      3
      Barbour
      Alabama
      24686

      4
      Bibb
      Alabama
      22394

      ...
      ...
      ...
      ...

      3189
      Teton
      Wyoming
      23464

      3190
      Uinta
      Wyoming
      20226

      3191
      Washakie
      Wyoming
      7805

      3192
      Weston County
      Wyoming
      6927

      3193
      New York City
      New York
      8336817

3194 rows × 3 columns

## Summary Code: Population Data Preprocessing

All the steps of the above are summarized into:

```python
df_countries, df_states, df_counties = load_population_data()
```

## Creating Per Capita Features

We can get per capita cases, deaths and recovered by:
- Merging COVID-19 and Population Data Sets
- Creating a Per Capita Feature for each data set

### Persistence

```python
populations = [df_countries, df_states, df_counties]
files = ['pop_countries.csv', 'pop_states.csv', 'pop_counties.csv']
[df.to_csv(file) for df, file in zip(populations, files)]

```

    [None, None, None]

```python
df_countries, df_states, df_counties = [pd.read_csv(file, index_col=0) for file in files]
```

```python
# Load tidy global data dictionaries
data = load_raw_global()
tidy_global = load_tidy_global(data)
merged_global = load_merged_daily_global(tidy_global)

# Load local data, and daily data dictionaries
state, county = load_local().values()
merged_state = load_merged_daily_local(state)
merged_county = load_merged_daily_local(county)

# Create per capita features on merged data sets
datasets = [merged_global, merged_state.drop(columns='fips'), merged_county.drop(columns='fips')]
populations = [df_countries, df_states, df_counties]
world_data, state_data, county_data = [create_per_capita_features(data, pop) for data, pop in zip(datasets, populations)]
```

```python
world_data['Location'] = world_data['Country/Region']
state_data['Location'] = state_data['State']
county_data['Location'] = county_data['County'] + ', ' + county_data['State']
```

```python
set(state_data.Location) & set(county_data.Location)
```

    set()

```python
county_data.Locat
```

```python
world_data.head()
```

      Country/Region
      Date
      Confirmed
      Deaths
      Recovered
      Daily Confirmed
      Daily Deaths
      Daily Recovered
      Population
      Confirmed per capita
      Deaths per capita
      Recovered per capita
      Daily Confirmed per capita
      Daily Deaths per capita
      Daily Recovered per capita

      0
      Afghanistan
      1/22/20
      0
      0
      0
      0.0
      0.0
      0.0
      36643815.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      1
      Afghanistan
      1/23/20
      0
      0
      0
      0.0
      0.0
      0.0
      36643815.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      Afghanistan
      1/24/20
      0
      0
      0
      0.0
      0.0
      0.0
      36643815.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      Afghanistan
      1/25/20
      0
      0
      0
      0.0
      0.0
      0.0
      36643815.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      Afghanistan
      1/26/20
      0
      0
      0
      0.0
      0.0
      0.0
      36643815.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

```python
state_data.head()
```

      Date
      State
      Confirmed
      Deaths
      Daily Confirmed
      Daily Deaths
      Population
      Confirmed per capita
      Deaths per capita
      Daily Confirmed per capita
      Daily Deaths per capita

      0
      2020-01-21
      Washington
      1
      0
      1.0
      0.0
      7614893.0
      1.313216e-07
      0.0
      1.313216e-07
      0.0

      1
      2020-01-22
      Washington
      1
      0
      0.0
      0.0
      7614893.0
      1.313216e-07
      0.0
      0.000000e+00
      0.0

      2
      2020-01-23
      Washington
      1
      0
      0.0
      0.0
      7614893.0
      1.313216e-07
      0.0
      0.000000e+00
      0.0

      3
      2020-01-24
      Washington
      1
      0
      0.0
      0.0
      7614893.0
      1.313216e-07
      0.0
      0.000000e+00
      0.0

      4
      2020-01-25
      Washington
      1
      0
      0.0
      0.0
      7614893.0
      1.313216e-07
      0.0
      0.000000e+00
      0.0

```python
county_data.head()
```

      Date
      County
      State
      Confirmed
      Deaths
      Daily Confirmed
      Daily Deaths
      Population
      Confirmed per capita
      Deaths per capita
      Daily Confirmed per capita
      Daily Deaths per capita

      0
      2020-01-21
      Snohomish
      Washington
      1
      0
      1.0
      0.0
      822083
      0.000001
      0.0
      0.000001
      0.0

      1
      2020-01-22
      Snohomish
      Washington
      1
      0
      0.0
      0.0
      822083
      0.000001
      0.0
      0.000000
      0.0

      2
      2020-01-23
      Snohomish
      Washington
      1
      0
      0.0
      0.0
      822083
      0.000001
      0.0
      0.000000
      0.0

      3
      2020-01-24
      Snohomish
      Washington
      1
      0
      0.0
      0.0
      822083
      0.000001
      0.0
      0.000000
      0.0

      4
      2020-01-25
      Snohomish
      Washington
      1
      0
      0.0
      0.0
      822083
      0.000001
      0.0
      0.000000
      0.0

```python
county_data['County'] +', ' + county_data['State']
```

    0         Snohomish, Washington
    1         Snohomish, Washington
    2         Snohomish, Washington
    3         Snohomish, Washington
    4         Snohomish, Washington
                      ...
    124446           Dent, Missouri
    124447         Thayer, Nebraska
    124448            Archer, Texas
    124449            Fisher, Texas
    124450           Haskell, Texas
    Length: 124451, dtype: object

```python
import time

def timed(func):
    def wrapper(*args, **kwargs):
        start = time.time()
        val = func(*args,**kwargs)
        end = time.time()
        print('Time: ', end-start)
        return val
    return wrapper

numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']

for i in df_countries.columns:
    print(df_countries[i].dtype in numerics)

```

## Exploratory Data Analysis

```python
# Change this list to see other countries
countries = top_n_countries(confirmed,10)
countries
```

```python
fig1 = px.line(filter_countries(confirmed,countries), x='Date', y='Confirmed',
              color='Country/Region', hover_name='Country/Region', title='Cumulative Confirmed')
fig1.show()
```

```python
fig2 = px.line(filter_countries(daily_confirmed,countries), x='Date', y='New Confirmed',
              color='Country/Region', hover_name='Country/Region', title='Daily Confirmed Cases')
fig2.show()
```

## Deaths

```python
fig3 = px.line(filter_countries(deaths,countries), x='Date', y='Deaths',
              color='Country/Region', hover_name='Country/Region', title='Cumulative Deaths')
fig3.show()
```

```python
fig4 = px.line(filter_countries(daily_deaths, countries), x='Date', y='New Deaths',
              color='Country/Region', hover_name='Country/Region', title='Daily Reported Deaths')
fig4.show()
```

```python
fig5 = px.line(filter_countries(recovered, countries), x='Date', y='Recovered',
              color='Country/Region', hover_name='Country/Region', title='Cumulative Recovered')
fig5.show()
```

```python
fig6 = px.line(filter_countries(daily_recovered, countries), x='Date', y='New Recovered',
              color='Country/Region', hover_name='Country/Region', title='Daily Recovered')
fig6.show()
```

## SIR Model Background

Under this model the population is compartmentalized into time dependent groups denoted as susceptible $S(t)$, infected $I(t)$, and removed $R(t)$ and  we have the following system of differential equations to describe their dynamics:

$$\begin{align}
\dfrac{dS}{dt} &= -\beta SI \tag{1}\\
\dfrac{dI}{dt}&= \beta SI - \nu I \tag{2}\\
\dfrac{dR}{dt}&= \nu I \tag{3}
\end{align}$$

### Understanding the Model Parameters

We can break the factor $\beta$ further down by considering:
- $\kappa$ : the contacts per unit time an infected individual has with the rest of the population
- $\tau$ : the probability that contact leads to transmission
- $N$ : the total population

Which leads to: $$ \beta = \dfrac{\kappa \tau}{N}$$

We then can look at the term:

$$\beta SI = \dfrac{\kappa \tau SI}{N} = \Big(\kappa \tau I \Big)\Big(\dfrac{S}{N}\Big)$$

and interpret $\kappa \tau I$ as the contacts per unit time by the infected population to the rest of the population that would lead to transmission. We multiply this by the fraction or probability of encountering a susceptible individual, $\dfrac{S}{N}$, to arrive at new infections per unit time.

The parameter $\nu$ can be interpreted as the probability of an infected host being removed per time, either through natural recovery, cure, or death. The inverse $\dfrac{1}{\nu}$ can be interpreted as the time per removal, or death of an invidual. In other words, the lifetime of the virus per host.

### Threshold for Epidemics

If we look at the behavior of the infected population growth rate in (2) in more detail:

$$\begin{align}
\dfrac{dI}{dt} &= \beta SI - \nu I\\
               &= (\beta S - \nu)I\\
               &= \Bigg[\dfrac{\kappa \tau S}{N} - \nu\Bigg]I\\
               &= \nu\Bigg[\dfrac{\kappa \tau}{\nu}\dfrac{S}{N} - 1\Bigg]I\\
 \end{align}$$

 With this rearrangment, we see how the term $\dfrac{\kappa \tau}{\nu}\dfrac{S}{N} - 1$ determines the sign for the rate of change for the infected population.

 We define a threshold quantity, **effective reproductive rate** $R_e$:

 $$ R_e = \dfrac{\kappa \tau}{\nu}\dfrac{S}{N}$$

 This effective reproductive number factors is appropriately named because it depends on the lifetime of the virus $\dfrac{1}{\nu}$ coupled with the effective contacts per host $\kappa \tau$, scaled by the composition of the population, $\dfrac{S}{N}$.

 and compare this quantity with 1 to see the sign for growth rate:

 $$\dfrac{dI}{dt} = \nu[R_e - 1]I$$

 ### Public Health Interventions

 - Staying home while sick, social distancing, quarantines, lowers the contacts $\kappa$, and lowers the effective $S$ and $I$ into the removed $R$ category.

 - Washing hands, sneezing into elbows and other habits reduce the fraction of transmissions per contact, $\tau$.

 - Cures and vaccinations helps reduce the lifetime of the virus, or increase $\nu$.

 - Testing allows faster reactions like quarantining, social distancing, which all lower $\kappa$.

 - Becoming informed and acting appropriately on this information overall drastically lowers $R_e$.

## Model Implementation and Fits

### Model Functions

```python
from scipy.integrate import solve_ivp
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error
from hyperopt import hp, fmin, tpe

def SIR(t, y, N, kappa, tau, nu):
    """
    Expresses SIR model in initial value ODE format, including time,
    state variables, and parameters required to compute the derivatives

    Parameters
    ---------------
    t : array
        Independent variable
    y : array-like
        Initial state of system, [S, I, R]
    parameters : array-like
        Parameters of model [N, kappa, tau, nu]

    Returns
    ----------------
    f : array-like
        Array of derivatives for the system to be numerically integrated
    """
    S, I, R = y

    dSdt = - kappa*tau/N*S*I
    dIdt = (kappa*tau/N*S-nu)*I
    dRdt = nu*I
    f = [dSdt, dIdt, dRdt]
    return f

def create_SIR_curves(parameters, data=data):
    N, kappa, tau, nu = parameters
    S, I, R = N-1, 1, 0
    sys0 = (S, I, R)
    total_points = len(data)
    start = 0
    stop = total_points
    t = np.linspace(start, stop, total_points)
    sol = solve_ivp(fun=SIR, t_span=(start,stop), y0=sys0, args=parameters, t_eval=t)
    df = pd.DataFrame(sol.y.T, columns=['S', 'I', 'R'])
    df['Time'] = t
    df['Cases'] = df['I']+df['R']
    df[data.name] = data.values
    return df

def plot_SIR(parameters, df):
    N, kappa, tau, nu = parameters
    Re = kappa*tau/(nu)
    print(Re)

    df.plot(x='Time', y=df.columns.drop('Time'),figsize=(15,10))
    plt.title('SIR')

def model_loss(parameters):
    # initial state variables
    df = create_SIR_curves(parameters)
    loss = mean_squared_error(df['Cases'].values, df[df.columns[-1]].values)
    return loss

```

###

```python
# parameters
sns.set(context='talk')
N = 10000
kappa = 3.5
tau = .05
nu = .01
parameters = (N, kappa, tau, nu)
country = 'Italy'
data = filter_countries(confirmed,[country])

# define configuration space
space = [hp.loguniform('N', 3, 20),
         hp.uniform('kappa', 0, 10),
         hp.uniform('tau', 0, 1),
         hp.loguniform('nu', -2, 2),
         ]
```

```python
data
```

```python
best = fmin(model_loss, space, algo=tpe.suggest, max_evals=2000)
```

```python
parameters = (int(best['N']),best['kappa'],best['tau'], best['nu'])
parameters
```

```python
df = create_SIR_curves(parameters)
plot_SIR(parameters,df)
```

```python
model_loss(parameters)
```

```python
mean_squared_error(df['Cases'].values, df[df.columns[-1]].values)
```

```python
data.values
```

```python
df.columns.drop('Time')
```

```python
df = create_SIR_curves(parameters, data)
df
```

```python
((df.Cases-df['Korea, South'])**2).mean()
```

```python
((df.Cases-df['Korea, South'])**2).mean()
```

```python
from hyperopt import hp, fmin, tpe

# define loss function

def loss()
```

```python
len(ts_confirmed)
```

```python
ts_confirmed['Korea, South'].values
```

```python
dots = sns.load_dataset("dots")
dots
```

## Population Simulations

Here's a simulation of infected population using the approach from [MIT 6.002x](https://courses.edx.org/courses/course-v1:MITx+6.00.2x_5+1T2016/courseware/44b64e16aa524037be90cd2aa3552ef6/eb30b49da504469c81d87d037126ce40/?child=first)

```python
a = [('aa',1), ('a',3), ('b', 2)]

a.sort(key=lambda x: x[1])
a
```

```python
import plotly.graph_objects as go

import pandas as pd

# Load dataset
df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv")

df.columns = [col.replace("AAPL.", "") for col in df.columns]

# Initialize figure
fig = go.Figure()

# Add Traces

fig.add_trace(
    go.Scatter(x=list(df.index),
               y=list(df.High),
               name="High",
               line=dict(color="#33CFA5")))

fig.add_trace(
    go.Scatter(x=list(df.index),
               y=[df.High.mean()] * len(df.index),
               name="High Average",
               visible=False,
               line=dict(color="#33CFA5", dash="dash")))

fig.add_trace(
    go.Scatter(x=list(df.index),
               y=list(df.Low),
               name="Low",
               line=dict(color="#F06A6A")))

fig.add_trace(
    go.Scatter(x=list(df.index),
               y=[df.Low.mean()] * len(df.index),
               name="Low Average",
               visible=False,
               line=dict(color="#F06A6A", dash="dash")))

# Add Annotations and Buttons
high_annotations = [dict(x="2016-03-01",
                         y=df.High.mean(),
                         xref="x", yref="y",
                         text="High Average:<br> %.2f" % df.High.mean(),
                         ax=0, ay=-40),
                    dict(x=df.High.idxmax(),
                         y=df.High.max(),
                         xref="x", yref="y",
                         text="High Max:<br> %.2f" % df.High.max(),
                         ax=0, ay=-40)]
low_annotations = [dict(x="2015-05-01",
                        y=df.Low.mean(),
                        xref="x", yref="y",
                        text="Low Average:<br> %.2f" % df.Low.mean(),
                        ax=-40, ay=40),
                   dict(x=df.High.idxmin(),
                        y=df.Low.min(),
                        xref="x", yref="y",
                        text="Low Min:<br> %.2f" % df.Low.min(),
                        ax=0, ay=40)]

fig.update_layout(
    updatemenus=[
        dict(
            type="buttons",
            direction="right",
            active=0,
            x=0.57,
            y=1.2,
            buttons=list([
                dict(label="None",
                     method="update",
                     args=[{"visible": [True, False, True, False]},
                           {"title": "Yahoo",
                            "annotations": []}]),
                dict(label="High",
                     method="update",
                     args=[{"visible": [True, True, False, False]},
                           {"title": "Yahoo High",
                            "annotations": high_annotations}]),
                dict(label="Low",
                     method="update",
                     args=[{"visible": [False, False, True, True]},
                           {"title": "Yahoo Low",
                            "annotations": low_annotations}]),
                dict(label="Both",
                     method="update",
                     args=[{"visible": [True, True, True, True]},
                           {"title": "Yahoo",
                            "annotations": high_annotations + low_annotations}]),
            ]),
        )
    ])

# Set title
fig.update_layout(
    title_text="Yahoo",
    xaxis_domain=[0.05, 1.0]
)

fig.show()
```

```python
a = {'a':1,'b':2,'c':3}
b = [2,4,6]

for i,j in zip(a,b):
    print(i,j)
```

```python
a
```

```python
a.values()

```

```python
import numpy as np
np.log(2/3)*(-450e-6)
```

```python
def func(a):
    a = 1
    return a
```

```python
a =2

b = func(a)
print(a,b)
```

```python
a,b,c = [1,2,3]
```

```python
a
```

```python
b
```

```python
c
```

```python
a = ['a', 'b', 'c']
b = [1,2,3]

list(zip(a,b))
```

**Note:** You are not required to complete this notebook and it will not be submitted with your project, but it is designed to quickly introduce the relevant parts of the Pomegranate library that you will need to complete the part of speech tagger.

Dropdown Example

The data-toggle="dropdown" attribute is used to open the dropdown menu.

    <button class="btn btn-primary dropdown-toggle" id="menu1" type="button" data-toggle="dropdown">Dropdown Example
    </button>

HTML

CSS

JavaScript

About Us

```python
from collections import defaultdict

d = defaultdict(lambda: defaultdict(int))
d['a']['b']
```

```python
d = {'a':1, 'b':2, 'c':3}
max(d.values())
```

```python
a = [1,2,3,4]
for i in a:

```

```python
sum(d.values())
```

```python
from itertools import product
for i in product(d.values(), repeat=2):
    print(i)
```

```python
d.keys()
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

Preface: Installing Devito (do not include in manuscipt)

This tutorial and the coming second part are based on Devito version 3.1.0. It requires the installation of the full software with examples, not only the code generation API. To install:

  git clone -b v3.1.0 https://github.com/opesci/devito
  cd devito
  conda env create -f environment.yml
  source activate devito
  pip install -e .

That final dot is important, don't miss it out!

Useful links

Devito documentation

Devito source code and examples

Tutorial notebooks with latest Devito/master

### Geophysics tutorial

# Full-waveform inversion 2: adjoint modeling

Mathias Louboutin1\*, Philipp Witte1, Michael Lange2, Navjot Kukreja2, Fabio Luporini2, Gerard Gorman2, and Felix J. Herrmann1,3

1 Seismic Laboratory for Imaging and Modeling (SLIM), The University of British Columbia

2 Imperial College London, London, UK

3 now at Georgia Institute of Technology, USA

Corresponding author: mloubout@eoas.ubc.ca

## Introduction

This tutorial is the second part of a three part tutorial series on full-waveform inversion (FWI), in which we provide a step by step walk through of setting up forward and adjoint wave equation solvers and an optimization framework for inversion. In Part 1 (Louboutin et al., 2017), we showed how to use [Devito](http://www.opesci.org/devito-public) to set up and solve acoustic wave equations with (impulsive) seismic sources and sample wavefields at the receiver locations to forward model shot records. In the second part of this tutorial series, we will discuss how to set up and solve adjoint wave equations with Devito and from that, how we can calculate gradients and function values of the FWI objective function.

The gradient of FWI is most commonly computed via the adjoint state method, by cross-correlating forward and adjoint wavefields and summing the contributions over all time steps (Plessix, 2006). Calculating the gradient for one source location consists of three steps:

* Solve the forward wave equation to create a shot record. The time varying wavefield must be stored for use in step 3; techniques such as subsampling can be used to reduce the storage requirements.

* Compute the data residual (or misfit) between the predicted and observed data.

* Solve the corresponding discrete adjoint model using the data residual as the source. Within the adjoint (reverse) time loop, cross correlate the second time derivative of the adjoint wavefield with the forward wavefield. These cross correlations are summed to form the gradient.

We start with the definition and derivation of the adjoint wave equation and its Devito stencil and then show how to compute the gradient of the conventional least squares FWI misfit function. As usual, this tutorial is accompanied by all the code you need to reproduce the figures. Go to [github.com/seg/tutorials-2018](https://github.com/seg/tutorials-2018) and follow the links.

## A simple experiment

To demonstrate the gradient computation in the simplest possible way, we perform a small seismic transmission experiment with a circular imaging phantom, i.e. a constant velocity model with a circular high velocity inclusion in its centre, as shown in Figure 1. For a transmission experiment, we place 21 seismic sources on the left-hand side of the model and 101 receivers on the right-hand side.

We will use the forward propagator from part 1 to independently model the 21 "observed" shot records using the true model. As the initial model for our gradient calculation, we use a constant velocity model with the same velocity as the true model, but without the circular velocity perturbation. We will then model the 21 predicted shot records for the initial model, calculate the data residual and gradient for each shot, and sum them to obtain the full gradient.

> _++ Figure 1 is generated later in the manuscript ++_
>
> **Figure 1:** (a) The velocity model, with sources and receivers arranged vertically. (b) The initial estimate. (c) The difference between the model and the initial estimate.

## The adjoint wave equation

Adjoint wave equations are a main component in seismic inversion algorithms and are required for computing gradients of both linear and non-linear objective functions. To ensure stability of the adjoint modeling scheme and the expected convergence of inversion algorithms, it is very important that the adjoint wave equation is in fact the adjoint (transpose) of the forward wave equation. The derivation of the adjoint wave equation in the acoustic case is simple, as it is self-adjoint if we ignore the absorbing boundaries for the moment. However, in the general case, discrete wave equations do not have this property (such as the coupled anisotropic TTI wave equation (Zhang et al., 2011)) and require correct derivations of their adjoints. We concentrate here, as in part 1, on the acoustic case and follow an optimize-discretize approach, which means we write out the adjoint wave equation for the continuous case first and then discretize it, using finite difference operators of the same order as for the forward equation. With the variables defined as in part 1 and the data residual $\delta d(x,y,t; x_r, y_r)$, located at $x_r, y_r$ (receiver locations) as the adjoint source, the continuous adjoint wave equation is given by:

$$
 m(x,y) \frac{\mathrm{d}^2 v(t,x,y)}{\mathrm{d}t^2}\ -\ \Delta v(t,x,y)\ -\ \eta(x,y) \frac{\mathrm{d} v(t,x,y)}{\mathrm{d}t}\ \ =\ \ \delta d(t,x,y;x_r, y_r)
$$

The adjoint acoustic wave equation is equivalent to the forward equation with the exception of the damping term $\eta(x,y) \mathrm{d}v(t,x,y)/\mathrm{d}t$, which contains a first time derivative and therefore has a change of sign in its adjoint. (A second derivative matrix is the same as its transpose, whereas a first derivative matrix is equal to its negative transpose and vice versa.)

Following the pattern of part 1, we first define the discrete adjoint wavefield $\mathbf{v}$ as a Devito `TimeFunction` object. For reasons we'll explain later, we do not need to save the adjoint wavefield:

```python
# NOT FOR MANUSCRIPT
from examples.seismic import Model, demo_model, plot_velocity, plot_perturbation

shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # Need origin to define relative source and receiver locations

model = demo_model('circle-isotropic',
                   vp=3.0,
                   vp_background=2.5,
                   origin=origin,
                   shape=shape,
                   spacing=spacing,
                   nbpml=40)

# For the manuscript, we'll re-form the model using the Vp field from
# this newly created model.
vp = model.vp

model0 = demo_model('circle-isotropic',
                    vp=2.5,
                    vp_background=2.5,
                    origin=origin,
                    shape=shape,
                    spacing=spacing,
                    nbpml=40)

t0 = 0.     # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing

nt = int(1 + (tn-t0) / dt)  # Discrete time axis length
time = np.linspace(t0, tn, nt)  # Discrete modelling time
```

```python
# NOT FOR MANUSCRIPT
from devito import TimeFunction
```

```python
v = TimeFunction(name="v", grid=model.grid,
                 time_order=2, space_order=4,
                 save=False)
```

Now symbolically set up the PDE:

```python
pde = model.m * v.dt2 - v.laplace - model.damp * v.dt
```

As before, we then define a stencil:

```python
# NOT FOR MANUSCRIPT
from devito import Eq
from sympy import solve
```

```python
stencil_v = Eq(v.backward, solve(pde, v.backward)[0])
```

Just as for the forward wave equation, `stencil_v` defines the update for the adjoint wavefield of a single time step. The only difference is that, while the forward modeling propagator goes forward in time, the adjoint propagator goes backwards in time, since the initial time conditions for the forward propagator turn into final time conditions for the adjoint propagator. As for the forward stencil, we can write out the corresponding discrete expression for the update of the adjoint wavefield:

$$
\mathbf{v}[\text{time}-\text{dt}] = 2\mathbf{v}[\text{time}] - \mathbf{v}[\text{time}+\text{dt}] + \frac{\text{dt}^2}{\mathbf{m}}\Delta \mathbf{v}[\text{time}], \quad \text{time} = n_{t-1} \cdots 1
$$

with $\text{dt}$ being the time stepping interval. Once again, this expression does not contain any (adjoint) source terms so far, which will be defined as a separate `SparseFunction` object. Since the source term for the adjoint wave equation is the difference between an observed and modeled shot record, we first define an (empty) shot record `residual` with ``101`` receivers and coordinates defined in `rec_coords`. We then set the data field `rec.data` of our shot record to be the data residual between the observed data `d_obs` and the predicted data `d_pred`. The symbolic residual source expression `res_term` for our adjoint wave equation is then obtained by *injecting* the data residual into the modeling scheme (`residual.inject`). Since we solve the time-stepping loop backwards in time, the `res_term` is used to update the previous adjoint wavefield `v.backward`, rather than the next wavefield. As in the forward modeling example, the source is scaled by  $\mathrm{dt}^2/\mathbf{m}$. In Python, we have:

```python
# NOT FOR MANUSCRIPT
from examples.seismic import Receiver

nshots = 21  # Number of shots to create gradient from
nreceivers = 101  # Number of receiver locations per shot

# Recs are distributed across model, at depth of 20 m.
z_extent, _ = model.domain_size
z_locations = np.linspace(0, z_extent, num=nreceivers)
rec_coords = np.array([(980, z) for z in z_locations])
```

```python
# NOT FOR MANUSCRIPT
from examples.seismic import PointSource
```

```python
residual = PointSource(name='residual', ntime=nt,
                       grid=model.grid, coordinates=rec_coords)

res_term = residual.inject(field=v.backward,
                           expr=residual * dt**2 / model.m,
                           offset=model.nbpml)
```

```python
# NOT FOR MANUSCRIPT
rec = Receiver(name='rec', npoint=nreceivers, ntime=nt,
               grid=model.grid, coordinates=rec_coords)
```

```python
# NOT FOR MANUSCRIPT
from examples.seismic import RickerSource

# At first, we want only a single shot.
# Src is 5% across model, at depth of 500 m.
z_locations = np.linspace(0, z_extent, num=nshots)
src_coords = np.array([(z_extent/50, z) for z in z_locations])
```

```python
# NOT FOR MANUSCRIPT
f0 = 0.010  # kHz, peak frequency.
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   time=time, coordinates=src_coords[nshots//2])
```

```python
# NOT FOR MANUSCRIPT
plt.plot(src.time, src.data)
plt.xlabel("Time (ms)")
plt.ylabel("Amplitude")
plt.show()
```

```python
# NOT FOR MANUSCRIPT
# Generates Figure 1
from matplotlib.gridspec import GridSpec

# Set up figure, grid, and parameters.
fig = plt.figure(figsize=(8, 12))

gs = GridSpec(3, 2)
ax0 = fig.add_subplot(gs[:2, :2])
ax1 = fig.add_subplot(gs[2, 0])
ax2 = fig.add_subplot(gs[2, 1])

extent = [model.origin[0], model.origin[0] + 1e-3 * model.shape[0] * model.spacing[0],
          model.origin[1] + 1e-3*model.shape[1] * model.spacing[1], model.origin[1]]
model_param = dict(vmin=2.5, vmax=3.0, cmap="GnBu", aspect=1, extent=extent)
diff_param  = dict(vmin=-1, vmax=0, cmap="GnBu", aspect=1, extent=extent)

# Part (a)
im = ax0.imshow(np.transpose(model.vp), **model_param)
ax0.scatter(*rec_coords.T/1000, lw=0, c='green', s=8)
ax0.scatter(*src_coords.T/1000, lw=0, c='red',s=24)
ax0.set_ylabel('Depth (km)',  fontsize=14)
ax0.text(0.5, 0.08, "model.vp", ha="center", color='k', size=18)
ax0.text(0.5, 0.5, "3000 m/s", ha="center", va='center', color='w', size=14)
ax0.text(0.8, 0.5, "2500 m/s", ha="center", va='center', color='navy', size=14)
ax0.text(0.04, 0.04, "sources", ha="left", color='r', size=12)
ax0.text(0.96, 0.04, "receivers", ha="right", color='green', size=12)
ax0.set_ylim(1, 0)
plt.setp(ax0.get_xticklabels(), fontsize=12)
plt.setp(ax0.get_yticklabels(), fontsize=12)

# Part (b)
im = ax1.imshow(np.transpose(model0.vp), **model_param)
ax1.set_xlabel('X position (km)',  fontsize=14)
ax1.set_yticklabels([0, '', 0.5, '', 1])
ax1.text(0.5, 0.08, "model0.vp", ha="center", color='k', size=14)
ax1.set_ylabel('Depth (km)',  fontsize=14)
plt.setp(ax1.get_xticklabels(), fontsize=12)
plt.setp(ax1.get_yticklabels(), fontsize=12)

# Part (c)
im = ax2.imshow(np.transpose(model0.vp - model.vp), **diff_param)
ax2.set_xlabel('X position (km)',  fontsize=14)
ax2.set_yticklabels([])
ax2.text(0.5, 0.08, "model0.vp – model.vp", ha="center", color='w', size=14)
ax2.text(0.5, 0.5, "–500 m/s", ha="center", va='center', color='w', size=12)
ax2.text(0.85, 0.5, "0 m/s", ha="center", va='center', color='w', size=12)
plt.setp(ax2.get_xticklabels(), fontsize=12)
plt.setp(ax2.get_yticklabels(), fontsize=12)

plt.savefig("../Figures/Figure_1.pdf")
plt.savefig("../Figures/Figure_1.png", dpi=400)
plt.show()
```

In this demonstration, there is no real data. Instead we will generate the 'observed' data via forward modeling with the true model `model`. The synthetic data is generated from the initial model `model0`. The resulting data, and their difference, are shown in Figure 2.

```python
# NOT FOR MANUSCRIPT
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, src, rec, space_order=4)

# Compute 'real' data with forward operator.
obs , _, _ = solver.forward(src=src, m=model.m)
```

    CustomCompiler: compiled /tmp/devito-gx10ovxj/7a282d6885f4ad3a7b16fd67992b9457900f0291.c [0.62 s]
    =========================================================================================
    Section section_1<714,1> with OI=0.73 computed in 0.000 s [0.52 GFlops/s]
    Section section_2<714,101> with OI=1.37 computed in 0.001 s [2.20 GFlops/s]
    Section main<714,179,179> with OI=1.95 computed in 0.090 s [8.15 GFlops/s, 0.25 GPts/s]
    =========================================================================================

```python
# NOT FOR MANUSCRIPT
# Compute initial data with forward operator.
pred, u0, _ = solver.forward(src=src, m=model0.m, save=True)
```

    CustomCompiler: compiled /tmp/devito-gx10ovxj/460b72ec62af10f5fed261f3a591dc9679cffa88.c [0.25 s]
    =========================================================================================
    Section section_1<714,1> with OI=0.73 computed in 0.000 s [0.26 GFlops/s]
    Section section_2<714,101> with OI=1.37 computed in 0.001 s [1.86 GFlops/s]
    Section main<714,179,179> with OI=1.95 computed in 0.113 s [6.49 GFlops/s, 0.20 GPts/s]
    =========================================================================================

```python
# NOT FOR MANUSCRIPT
fig = plt.figure(figsize=(15, 5))

extent = [model.origin[0],  # Horizontal min
          model.origin[0] + 1e-3 * model.shape[0] * model.spacing[0],
          tn/1000,  # Vertical min (bottom)
          t0/1000]  # Vertical max (top)

ma = np.percentile(obs.data, 99.5)
params = dict(vmin=-ma, vmax=ma, cmap="Greys", aspect=1, extent=extent)
text_params = dict(ha="center", color='w', size=16)

ax0 = fig.add_subplot(131)
im = plt.imshow(obs.data, **params)
ax0.set_ylabel('Time (s)',  fontsize=16)
ax0.text(0.5, 0.08, "obs", **text_params)
plt.xticks(fontsize=10)
plt.yticks(fontsize=10)

ax1 = fig.add_subplot(132)
im = plt.imshow(pred.data, **params)
ax1.set_xlabel('Z position (km)',  fontsize=16)
ax1.set_yticklabels([])
ax1.text(0.5, 0.08, "pred", **text_params)
plt.xticks(fontsize=10)
plt.yticks(fontsize=10)

ax2 = fig.add_subplot(133)
im = plt.imshow(pred.data - obs.data, **params)
ax2.set_yticklabels([])
ax2.text(0.5, 0.08, "pred – obs", **text_params)
plt.xticks(fontsize=10)
plt.yticks(fontsize=10)

plt.savefig("../Figures/Figure2.pdf")
plt.savefig("../Figures/Figure2.png", dpi=400)
plt.show()
```

> **Figure 2.** Shot records for a shot at a Z position of 0.5 km. (a) The observed data, using the known model with the high velocity disc, contains a perturbation not present in (b) the 'predicted' data, using the initial estimate of the model, which contais no disc. (c) The residual.

Finally, we create the full propagator by adding the residual source expression to our previously defined stencil and set the flag `time_axis=Backward`, to specify that the propagator runs in backwards in time:

```python
# NOT FOR MANUSCRIPT
from devito import Operator, Backward
```

```python
op_adj = Operator([stencil_v] + res_term, time_axis=Backward)
```

In contrast to forward modeling, we do not record any measurements at the surface since we are only interested in the adjoint wavefield itself. The full script for setting up the adjoint wave equation, including an animation of the adjoint wavefield is available in **`adjoint_modeling.ipynb`**.

## Computing the FWI gradient

The goal of FWI is to estimate a discrete parametrization of the subsurface by minimizing the misfit between the observed shot records of a seismic survey and numerically modeled shot records. The predicted shot records are obtained by solving an individual wave equation per shot location and depend on the parametrization $\mathbf{m}$ of our wave propagator. The most common function for measuring the data misfit between the observed and modeled data is the $\ell_2$ norm, which leads to the following objective function (Lions (1971), Tarantola (1984)):

$$
\mathop{\hbox{minimize}}_{\mathbf{m}} \hspace{.2cm} f(\mathbf{m})= \sum_{i=1}^{n_\mathrm{s}} \frac{1}{2} \left\lVert \mathbf{d}^\mathrm{pred}_i (\mathbf{m}, \mathbf{q}_i) - \mathbf{d}_i^\mathrm{obs} \right\rVert_2^2,
$$

where the index $i$ runs over the total number of shots $n_\mathrm{s}$ and the model parameters are the squared slowness. Optimization problems of this form are called nonlinear least-squares problems, since the predicted data modeled with the forward modeling propagator (`op_fwd()` in part 1) depends nonlinearly on the unknown parameters $\mathbf{m}$. The full derivation of the FWI gradient using the adjoint state method is outside the scope of this tutorial, but conceptually we obtain the gradient by applying the chain rule and taking the partial derivative of the inverse wave equation $\mathbf{A}(\mathbf{m})^{-1}$ with respect to $\mathbf{m}$, which yields the following expression (Plessix, 2006, Virieux and Operto, 2009):

$$
 \nabla f (\mathbf{m})= -  \sum_{i=1}^{n_\mathrm{s}}  \sum_{\text{time}=1}^{n_t} \mathbf{u}[\text{time}]\odot \ddot{\mathbf{v}}[\text{time}].
$$

The inner sum $\text{time}=1,...,n_t$ runs over the number of computational time steps $n_t$ and $\ddot{\mathbf{v}}$ denotes the second temporal derivative of the adjoint wavefield $\mathbf{v}$. Computing the gradient of Equation 3, therefore corresponds to performing the point-wise multiplication (denoted  by the symbol $\odot$) of the forward wavefields with the second time derivative of the adjoint wavefield and summing over all time steps.

To avoid the need to store the adjoint wavefield, the FWI gradient is calculated in the reverse time-loop while solving the adjoint wave equation. To compute the gradient $\mathbf{g}$ for the current time step $\mathbf{v}[\text{time}]$:

$$
 \mathbf{g} = \mathbf{g} - \frac{\mathbf{v}[\text{time-dt}] - 2\mathbf{v}[\text{time}] + \mathbf{v}[\text{time+dt}]}{\mathrm{dt}^2} \odot \mathbf{u}[\text{time}], \quad \text{time}=1 \cdots n_{t-1}
$$

The second time derivative of the adjoint wavefield is computed with a second order finite-difference stencil and uses the three adjoint wavefields that are kept in memory during the adjoint time loop (Equation 2).

In Devito we define the gradient as a `Function` since the gradient is computed as the sum over all time steps and therefore has no time dependence:

```python
# NOT FOR MANUSCRIPT
from devito import TimeFunction, Function
```

```python
# NOT FOR MANUSCRIPT
# This is the same u as in Part 1.
u = TimeFunction(name="u", grid=model.grid,
                 time_order=2, space_order=4,
                 save=True, time_dim=nt)
```

```python
grad = Function(name="grad", grid=model.grid)
```

The update for the gradient as defined in Equations 4 and 5 is then:

```python
grad_update = Eq(grad, grad - u * v.dt2)
```

Now we must add the gradient update expression to the adjoint propagator `op_grad`. This yields a single symbolic expression with update instructions for both the adjoint wavefield and the gradient:

```python
op_grad = Operator([stencil_v] + res_term + [grad_update],
                   time_axis=Backward)
```

Solving the adjoint wave equation by running the following now computes the FWI gradient for a single source. Its value is stored in `grad.data`.

```python
op_grad(u=u0, v=v, m=model0.m,
        residual=pred.data-obs.data,
        time=nt, dt=dt)
```

    CustomCompiler: compiled /tmp/devito-gx10ovxj/ec66845144ec0c1fb8e1bb5f532cda21c551712e.c [0.28 s]
    =========================================================================================
    Section section_1<714,101> with OI=0.80 computed in 0.003 s [1.18 GFlops/s]
    Section main<714,179,179> with OI=2.83 computed in 0.303 s [6.11 GFlops/s, 0.08 GPts/s]
    =========================================================================================

```python
# NOT FOR MANUSCIPT
plt.figure(figsize=(8,8))
plt.imshow(np.transpose(grad.data[40:-40,40:-40]), extent=extent, cmap='RdBu', vmin=-1e3, vmax=1e3)
plt.colorbar(shrink=0.75)
plt.show()
```

```python
# NOT FOR MANUSCRIPT
tmp = grad.copy()
```

Now we can iterate over all the shot locations, running the same sequence of commands each time.

```python
# NOT FOR MANUSCRIPT
from devito import configuration
configuration['log_level'] = 'WARNING'

# Create the symbols.
u0 = TimeFunction(name='u0', grid=model.grid, time_order=2, space_order=4, save=True, time_dim=nt)
u = TimeFunction(name='u', grid=model.grid, time_order=2, space_order=4, save=True, time_dim=nt)
v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4, save=False)

# Define the wave equation, but with a negated damping term
eqn = model.m * v.dt2 - v.laplace - model.damp * v.dt

# Use SymPy to rearrange the equation into a stencil expression.
stencil = Eq(v.backward, solve(eqn, v.backward)[0])

# Define the residual injection.
residual = PointSource(name='residual', ntime=nt, coordinates=rec_coords, grid=model.grid)
res_term = residual.inject(field=v.backward, expr=residual * dt**2 / model.m, offset=model.nbpml)

# Correlate u and v for the current time step and add it to the gradient.
grad = Function(name='grad', grid=model.grid, dtype=model.m.dtype)
grad_update = Eq(grad, grad - u * v.dt2)

# Compose the operator.
op_grad2 = Operator([stencil] + res_term + [grad_update], time_axis=Backward)

# Iterate over the shots.
for i in range(nshots):
    print("Source {} of {}".format(i, nshots))
    # Opdate source location.
    src.coordinates.data[0, :] = src_coords[i]

    # Generate data from true model and current model estimate.
    obs, _, _ = solver.forward(src=src, m=model.m)
    pred, _, _ = solver.forward(src=src, m=model0.m, u=u0, save=True)

    # Compute the gradient from the residual.
    v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4)
    residual.data[:] = pred.data - obs.data
    op_grad2(u=u0, v=v, m=model0.m, residual=residual, grad=grad, dt=dt)

configuration['log_level'] = 'INFO'
```

    Source 0 of 21
    Source 1 of 21
    Source 2 of 21
    Source 3 of 21
    Source 4 of 21
    Source 5 of 21
    Source 6 of 21
    Source 7 of 21
    Source 8 of 21
    Source 9 of 21
    Source 10 of 21
    Source 11 of 21
    Source 12 of 21
    Source 13 of 21
    Source 14 of 21
    Source 15 of 21
    Source 16 of 21
    Source 17 of 21
    Source 18 of 21
    Source 19 of 21
    Source 20 of 21

```python
# NOT FOR MANUSCRIPT
plt.figure(figsize=(8,8))
plt.imshow(np.transpose(grad.data)[40:-40, 40:-40], extent=extent, vmin=-1e4, vmax=1e4, cmap='RdBu')
plt.colorbar(shrink=0.75)
plt.show()
```

```python
# NOT FOR MANUSCRIPT
# Generates Figure 3
from plot_utils import add_subplot_axes

fig = plt.figure(figsize=(12,6))

ax0 = fig.add_subplot(121)
ma = 1e3
im = ax0.imshow(np.transpose(tmp.data[40:-40,40:-40]), extent=extent, cmap='RdBu', vmin=-ma, vmax=ma)
cax = add_subplot_axes(ax0, [0.3, 0.075, 0.4, 0.02])
fig.colorbar(im, cax=cax, orientation='horizontal')
cax.text(0, 1, "{:.0f} ".format(-ma), ha='right', va='top', size=10)
cax.text(1, 1, " {:.0f}".format(ma), ha='left', va='top', size=10)
cax.set_axis_off()
ax0.text(0.5, 0.05, "Shot @ 500 m", ha='center', va='center', size=16)
ax0.set_xlabel('X position (m)',  fontsize=16)
ax0.set_ylabel('Z position (m)',  fontsize=16)
plt.setp(ax0.get_xticklabels(), fontsize=12)
plt.setp(ax0.get_yticklabels(), fontsize=12)

ax1 = fig.add_subplot(122)
ma = 1e4
im = ax1.imshow(np.transpose(grad.data[40:-40,40:-40]), extent=extent, cmap='RdBu', vmin=-ma, vmax=ma)
cax = add_subplot_axes(ax1, [0.3, 0.075, 0.4, 0.02])
fig.colorbar(im, cax=cax, orientation='horizontal')
cax.text(0, 1, "{:.0f} ".format(-ma), ha='right', va='top', size=10)
cax.text(1, 1, " {:.0f}".format(ma), ha='left', va='top', size=10)
cax.set_axis_off()
ax1.text(0.5, 0.05, "All {} shots".format(nshots), ha='center', va='center', size=16)
ax1.set_xlabel('X position (m)',  fontsize=16)
ax1.set_yticklabels([])
plt.setp(ax1.get_xticklabels(), fontsize=12)
plt.setp(ax1.get_yticklabels(), fontsize=12)

plt.savefig("../Figures/Figure3.pdf")
plt.savefig("../Figures/Figure3.png", dpi=400)
plt.show()
```

> **Figure 3.** Gradient plots for (a) a single shot at 0.5 km and (b) the sum of all shots.

This gradient can then be used for a simple gradient descent optimization loop, as illustrated at the end of the notebook `adjoint_modeling.ipynb`. After each update, a new gradient is computed for the new velocity model until sufficient decrease of the objective or chosen number of iteration is reached. A detailed treatment of optimization and more advanced algorithms will be described in the third and final part of this tutorial series.

Verification (do not include in manuscipt)

The next step of the adjoint modeling and gradient part is verification with unit testing, i.e. we ensure that the adjoints and gradients are implemented correctly. Incorrect adjoints can lead to unpredictable behaviour during and inversion and in the worst case cause slower convergence or convergence to wrong solutions.

Since our forward-adjoint wave equation solvers correspond to forward-adjoint pairs, we need to ensure that the adjoint defined dot test holds within machine precision (see **`tests/test_adjointA.py`** for the dot test). Furthermore, we verify the correct implementation of the FWI gradient by ensuring that using the gradient leads to first order convergence. The gradient test can be found in **`tests/test_gradient.py`**.

## Conclusions

We need the gradient of the FWI objective function in order to find the optimal solution. It is computed by solving adjoint wave equations and summing the point-wise product of forward and adjoint wavefields over all time steps. Using Devito, the adjoint wave equation is set up in a similar fashion as the forward wave equation, with the main difference being the (adjoint) source, which is the residual between the observed and predicted shot records.

With the ability to model shot records and compute gradients of the FWI objective function, we are ready to demonstrate how to set up more gradient-based algorithms for FWI in Part 3 next month.

## Acknowledgments

This research was carried out as part of the SINBAD II project with the support of the member organizations of the SINBAD Consortium. This work was financially supported in part by EPSRC grant EP/L000407/1 and the Imperial College London Intel Parallel Computing Centre.

## References

[1] Michael Lange, Navjot Kukreja, Mathias Louboutin, Fabio Luporini, Felippe Vieira Zacarias, Vincenzo Pandolfo, Paulius Velesko, Paulius Kazakas, and Gerard Gorman. Devito: Towards a generic finite difference DSL using symbolic python. In 6th Workshop on Python for High-Performance and Scientific Computing, pages 67–75, 11 2016. doi: 10.1109/PyHPC.2016.9.

[2] J. L. Lions. Optimal control of systems governed by partial differential equations. Springer-Verlag Berlin Heidelberg, 1st edition, 1971. ISBN 978-3-642-65026-0.

[3] Mathias Louboutin, Philipp A. Witte, Michael Lange, Navjot Kukreja, Fabio Luporini, Gerard Gorman, and Felix J. Herrmann. Full-waveform inversion - part 1: forward modeling. Submitted to The Leading Edge for the tutorial section on October 30, 2017., 2017.

[4] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. Peer J Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peerj-cs.103. URL https: //doi.org/10.7717/peerj-cs.103.

[5] R.-E. Plessix. A review of the adjoint-state method for computing the gradient of a functional with geophysical applications. Geophysical Journal International, 167(2):495, 2006. doi: 10.1111/j.1365-246X.2006.02978.x. URL +http://dx.doi.org/10.1111/j.1365-246X.2006.02978.x

[6] Albert Tarantola. Inversion of seismic reflection data in the acoustic approximation. GEOPHYSICS, 49(8): 1259–1266, 1984. doi: 10.1190/1.1441754. URL https://doi.org/10.1190/1.1441754

[7] J. Virieux and S. Operto. An overview of full-waveform inversion in exploration geophysics. GEOPHYSICS, 74 (5):WCC1–WCC26, 2009. doi: 10.1190/1.3238367. URL http://library.seg.org/doi/abs/10.1190/1.3238367

[8] Yu Zhang, Houzhu Zhang, and Guanquan Zhang. A stable tti reverse time migration and its implementation. GEOPHYSICS, 76(3):WA3–WA11, 2011. doi: 10.1190/1.3554411. URL https://doi.org/10.1190/1.3554411.

© 2017 The authors — licensed CC-BY-SA