# Homework 2

In this homework, we are going to play with Twitter data.

The data is represented as rows of of [JSON](https://en.wikipedia.org/wiki/JSON#Example) strings.
It consists of [tweets](https://dev.twitter.com/overview/api/tweets), [messages](https://dev.twitter.com/streaming/overview/messages-types), and a small amount of broken data (cannot be parsed as JSON).

For this homework, we will only focus on tweets and ignore all other messages.

## Tweets

A tweet consists of many data fields. [Here is an example](https://gist.github.com/arapat/03d02c9b327e6ff3f6c3c5c602eeaf8b). You can learn all about them in the Twitter API doc. We are going to briefly introduce only the data fields that will be used in this homework.

* `created_at`: Posted time of this tweet (time zone is included)
* `id_str`: Tweet ID - we recommend using `id_str` over using `id` as Tweet IDs, becauase `id` is an integer and may bring some overflow problems.
* `text`: Tweet content
* `user`: A JSON object for information about the author of the tweet
    * `id_str`: User ID
    * `name`: User name (may contain spaces)
    * `screen_name`: User screen name (no spaces)
* `retweeted_status`: A JSON object for information about the retweeted tweet (i.e. this tweet is not original but retweeteed some other tweet)
    * All data fields of a tweet except `retweeted_status`
* `entities`: A JSON object for all entities in this tweet
    * `hashtags`: An array for all the hashtags that are mentioned in this tweet
    * `urls`: An array for all the URLs that are mentioned in this tweet

## Data source

All tweets are collected using the [Twitter Streaming API](https://dev.twitter.com/streaming/overview).

## Users partition

Besides the original tweets, we will provide you with a Pickle file, which contains a partition over 452,743 Twitter users. It contains a Python dictionary `{user_id: partition_id}`. The users are partitioned into 7 groups.

```python
# You can load pickle_content from a file on the local file system
# while testing on your laptop
# To test on your laptop, set `ON_EMR=False`
# To test on AWS for final submission, set `ON_EMR=True`

ON_EMR = False
```

## Grading

We ask you use the `OutputLogger` object `my_output` to store the
results of your program.
We have provided function calls to `my_output.append()` method for
storing the results in all necessary places.
Please make sure NOT to remove these lines

In the last cell of this file, we write the content of `my_output`
to a pickle file which the grader will read in and use for grading.

```python
import os
import pickle

class OutputLogger:
    def __init__(self):
        self.ans = {}

    def append(self, key, value):
        self.ans[key] = value

    def write_to_disk(self):
        if ON_EMR:
            filepath = os.path.expanduser("answer.pickle")
            print("FilePath = {}".format(filepath))
            with open(filepath, 'wb') as f:
                pickle.dump(self.ans, f)
            proc = subprocess.Popen(["/usr/local/hadoop/hadoop-2.7.4/bin/hadoop", "fs", "-copyFromLocal", filepath, "/user/spark/answer.pickle"])
            proc.wait()
            os.remove(filepath)
        else:
            filepath = os.path.expanduser("~/answer.pickle")
            with open(filepath, 'wb') as f:
                pickle.dump(self.ans, f)

my_output = OutputLogger()
```

# Part 0: Load data to a RDD

The tweets data is stored on AWS S3. We have in total a little over 1 TB of tweets. We provide 10 MB of tweets for your local development. For the testing and grading on the homework server, we will use different data.

## Testing on the homework server
On EdX, we provide three different input sizes to test your program: 10 MB, 1 GB, and 10 GB. For any run, we will only be using one of these four datasets.

For submission and for local testing, make sure to read the path of the file you want to operate with from `./hw2-files.txt`. Otherwise your program will receive no points.

## Local test

For local testing, please create your own `hw2-files.txt` file, which contains a single file path on the local disk, e.g.
`file://<absolute_path_to_current_directory>/hw2-files-10mb.txt`. For final submission, we will create this file on our server for testing with the appropriate file path. If your implementation is correct, you should not worry about which file system (i.e. local file system or HDFS) Spark will read data from.

Now let's see how many lines there are in the input files.

1. Make RDD from the data in the file given by the file path present in `hw2-files.txt`.
2. Mark the RDD to be cached (so in next operation data will be loaded in memory)
3. call the `count` method to print number of lines in all these files

<b>It should print<b>
```
Number of elements: 2150
```

```python
"""
This is a useful cell for debugging.
Use timer_start() and timer_stop() at different parts of your code
for checking the amount of time a segment takes.
"""
from time import time

timer = []
prev_ts = None

def timer_start():
    global prev_ts
    prev_ts = time()

def timer_stop(title):
    timer.append((title, time() - prev_ts))
```

```python
timer_start()

from pyspark import SparkContext

sc = SparkContext()

timer_stop("set up sc")
```

```python
timer_start()

with open('./hw2-files.txt') as f:
    file_path = [w.strip() for w in f.readlines() if w.strip()]

count = 0
for path in file_path:
    print("Reading file: " + path)
    baseRdd = sc.textFile(path).cache()
    count = count + baseRdd.count()

my_output.append("num-tweets", count)
print(baseRdd.take(1))
print('Number of elements:', count)
timer_stop("read data")
```

    Reading file: ./hw2-files-10mb.txt
    ['{"created_at":"Tue Feb 23 17:42:31 +0000 2016","id":702186789189132288,"id_str":"702186789189132288","text":"I\'m voting 4 #BernieSanders bc he doesn\'t ride a CAPITALIST PIG adorned w\\/ #GoldmanSachs $. SYSTEM RIGGED CLASS WAR https:\\/\\/t.co\\/P7pFm2MT9e","source":"\\u003ca href=\\"http:\\/\\/twitter.com\\" rel=\\"nofollow\\"\\u003eTwitter Web Client\\u003c\\/a\\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":470520068,"id_str":"470520068","name":"Marni Halasa","screen_name":"MarniHalasa1","location":"NYC","url":"http:\\/\\/www.revolutionissexy.com","description":"Performance Artist\\/Activist: unofficial OWS Freedom Fairy, Prof Figure Skater, Parade Personality, Media Liaison, Journalist, Lawyer. I protest w\\/skates & wings","protected":false,"verified":false,"followers_count":1156,"friends_count":1484,"listed_count":102,"favourites_count":4109,"statuses_count":10443,"created_at":"Sat Jan 21 21:15:53 +0000 2012","utc_offset":null,"time_zone":null,"geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"000000","profile_background_image_url":"http:\\/\\/abs.twimg.com\\/images\\/themes\\/theme1\\/bg.png","profile_background_image_url_https":"https:\\/\\/abs.twimg.com\\/images\\/themes\\/theme1\\/bg.png","profile_background_tile":false,"profile_link_color":"3B94D9","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"000000","profile_text_color":"000000","profile_use_background_image":false,"profile_image_url":"http:\\/\\/pbs.twimg.com\\/profile_images\\/461558466218184704\\/Al04oPWW_normal.jpeg","profile_image_url_https":"https:\\/\\/pbs.twimg.com\\/profile_images\\/461558466218184704\\/Al04oPWW_normal.jpeg","profile_banner_url":"https:\\/\\/pbs.twimg.com\\/profile_banners\\/470520068\\/1416425356","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":{"id":"01a9a39529b27f36","url":"https:\\/\\/api.twitter.com\\/1.1\\/geo\\/id\\/01a9a39529b27f36.json","place_type":"city","name":"Manhattan","full_name":"Manhattan, NY","country_code":"US","country":"United States","bounding_box":{"type":"Polygon","coordinates":[[[-74.026675,40.683935],[-74.026675,40.877483],[-73.910408,40.877483],[-73.910408,40.683935]]]},"attributes":{}},"contributors":null,"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"BernieSanders","indices":[13,27]},{"text":"GoldmanSachs","indices":[75,88]}],"urls":[],"user_mentions":[],"symbols":[],"media":[{"id":702186787679002624,"id_str":"702186787679002624","indices":[116,139],"media_url":"http:\\/\\/pbs.twimg.com\\/media\\/Cb6rKb2UYAAR1Kc.jpg","media_url_https":"https:\\/\\/pbs.twimg.com\\/media\\/Cb6rKb2UYAAR1Kc.jpg","url":"https:\\/\\/t.co\\/P7pFm2MT9e","display_url":"pic.twitter.com\\/P7pFm2MT9e","expanded_url":"http:\\/\\/twitter.com\\/MarniHalasa1\\/status\\/702186789189132288\\/photo\\/1","type":"photo","sizes":{"thumb":{"w":150,"h":150,"resize":"crop"},"large":{"w":2048,"h":1536,"resize":"fit"},"medium":{"w":1200,"h":900,"resize":"fit"},"small":{"w":680,"h":510,"resize":"fit"}}}]},"extended_entities":{"media":[{"id":702186787679002624,"id_str":"702186787679002624","indices":[116,139],"media_url":"http:\\/\\/pbs.twimg.com\\/media\\/Cb6rKb2UYAAR1Kc.jpg","media_url_https":"https:\\/\\/pbs.twimg.com\\/media\\/Cb6rKb2UYAAR1Kc.jpg","url":"https:\\/\\/t.co\\/P7pFm2MT9e","display_url":"pic.twitter.com\\/P7pFm2MT9e","expanded_url":"http:\\/\\/twitter.com\\/MarniHalasa1\\/status\\/702186789189132288\\/photo\\/1","type":"photo","sizes":{"thumb":{"w":150,"h":150,"resize":"crop"},"large":{"w":2048,"h":1536,"resize":"fit"},"medium":{"w":1200,"h":900,"resize":"fit"},"small":{"w":680,"h":510,"resize":"fit"}}}]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1456249351199"}']
    Number of elements: 2150

# Part 1: Parse JSON strings to JSON objects

Python has built-in support for JSON.

```python
import json

json_example = '''
{
    "id": 1,
    "name": "A green door",
    "price": 12.50,
    "tags": ["home", "green"]
}
'''

json_obj = json.loads(json_example)
json_obj
```

    {'id': 1, 'name': 'A green door', 'price': 12.5, 'tags': ['home', 'green']}

## Broken tweets and irrelevant messages

The data of this assignment may contain broken tweets (invalid JSON strings). So make sure that your code is robust for such cases.

You can filter out such broken tweet by checking if:
* the line is not in json format

In addition, some lines in the input file might not be tweets, but messages that the Twitter server sent to the developer (such as [limit notices](https://dev.twitter.com/streaming/overview/messages-types#limit_notices)). Your program should also ignore these messages.

These messages would not contain the `created_at` field and can be filtered out accordingly.
* Check if json object of the broken tweet has a `created_at` field

*Hint:* [Catch the ValueError](http://stackoverflow.com/questions/11294535/verify-if-a-string-is-json-in-python)

**********************************************************************************

**Tasks**

(1) Parse raw JSON tweets to obtain valid JSON objects.

(2) From all valid tweets, construct a pair RDD of `(user_id, text)`, where `user_id` is the `id_str` data field of the `user` dictionary (read [Tweets](#Tweets) section above), `text` is the `text` data field.

```python
import json

def safe_parse(raw_json):
    try:
        json_object = json.loads(raw_json)
    except ValueError as e:
        return None
    return json_object

```

```python
"""
# Remember to construct an RDD of (user_id, text) here.
"""

def createTweetEntry(s):
    json = safe_parse(s)
    if (json is None or not "user" in json or not "id_str" in json["user"] or not "text" in json):
        return None
    else:
        return (json["user"]["id_str"], json["text"])

tweetRdd = baseRdd.map(createTweetEntry).filter(lambda x: x is not None)

print(tweetRdd.take(5))

```

    [('470520068', "I'm voting 4 #BernieSanders bc he doesn't ride a CAPITALIST PIG adorned w/ #GoldmanSachs $. SYSTEM RIGGED CLASS WAR https://t.co/P7pFm2MT9e"), ('2176120173', "RT @TrumpNewMedia: .@realDonaldTrump #America get out & #VoteTrump if you don't #VoteTrump NOTHING will change it's that simple!\n#Trump htt…"), ('145087572', 'RT @Libertea2012: RT TODAY: #Colorado’s leading progressive voices to endorse @BernieSanders! #Denver 11AM - 1PM in MST CO State Capitol…'), ('23047147', '[VID] Liberal Tears Pour After Bernie Supporter Had To Deal With Trump Fans https://t.co/GFio6jeCBy via @MadWorldNews'), ('526506000', 'RT @justinamash: .@tedcruz is the only remaining candidate I trust to take on what he correctly calls the Washington Cartel. https://t.co/N…')]

## Number of unique users

Count the number of different users in all valid tweets

(hint: [the `distinct()` method](https://spark.apache.org/docs/latest/programming-guide.html#transformations) is an easy way to do this, but try to see if there is a faster way to do this).

*******************************************************************************

**It should print**
```
The number of unique users is: 1748
```

```python
timer_start()

users_count = tweetRdd.map(lambda x: (x[0], 1)).\
                reduceByKey(lambda x,y: x*y).\
                map(lambda x: x[1]).\
                reduce(lambda x,y: x+y)

my_output.append("num-unique-users", users_count)
print('The number of unique users is:', users_count)
timer_stop("Count unique users")
```

    The number of unique users is: 1748

# Part 2: Number of posts from each user partition

Load the Pickle file `/twitter/users-partition.pickle`, you will get a dictionary which represents a partition over 452,743 Twitter users, `{user_id: partition_id}`. The users are partitioned into 7 groups. For example, if the dictionary is loaded into a variable named `partition`, the partition ID of the user `59458445` is `partition["59458445"]`. These users are partitioned into 7 groups. The partition ID is an integer between 0-6.

Note that the user partition we provide doesn't cover all users appear in the input data.

## Load the pickle file

For local testing, you can load the pickle file from the local file system, namely

```
proc = subprocess.Popen(["cat", "./users-partition.pickle"],
                        stdout=subprocess.PIPE)
pickle_content = proc.communicate()[0]
```
However, for submission, please keep following code block unchanged, since on the server the pickle file is located on the HDFS.

```python
import subprocess
import pickle

if ON_EMR:
    proc = subprocess.Popen(["/usr/local/hadoop/hadoop-2.7.4/bin/hadoop", "fs", "-cat", "/user/spark/twitter/users-partition.pickle"],
                            stdout=subprocess.PIPE)
    pickle_content = proc.communicate()[0]

else:
    #!wget 'http://mas-dse-open.s3.amazonaws.com/Twitter/users-partition.pickle' -O './users-partition.pickle'
    proc = subprocess.Popen(["cat", "./users-partition.pickle"],
                        stdout=subprocess.PIPE)
    pickle_content = proc.communicate()[0]

partition = pickle.loads(pickle_content)
len(partition)
```

    452743

## Tweets per user partition

1. Count the number of posts from each user partition

2. Count the number of posts from group 0, 1, ..., 6, plus the number of posts from users who are not in any partition. Assign users who are not in any partition to the group 7.

3. Put the results of this step into a pair RDD `(group_id, count)` that is sorted by key.

Print the post count using the `print_post_count` function we provided.

**It should print**

```
Group 0 posted 87 tweets
Group 1 posted 242 tweets
Group 2 posted 41 tweets
Group 3 posted 349 tweets
Group 4 posted 101 tweets
Group 5 posted 358 tweets
Group 6 posted 434 tweets
Group 7 posted 521 tweets
```

```python
def print_post_count(counts):
    for group_id, count in counts:
        print('Group %d posted %d tweets' % (group_id, count))
```

```python
timer_start()

def getTweetUserPartition(tweetId):
    if (tweetId in partition):
        return partition[tweetId]
    return 7

counts_per_partition = tweetRdd.\
        map(lambda x: (getTweetUserPartition(x[0]), 1)).\
        reduceByKey(lambda x,y: x+y).sortByKey().collect()

# Following code adds your solution to `my_output`
assert(type(counts_per_partition) is list and
       len(counts_per_partition) == 8 and
       len(counts_per_partition[0]) == 2)
my_output.append("counts_per_part", counts_per_partition)
print_post_count(counts_per_partition)
timer_stop("Count tweets per user partition")
```

    Group 0 posted 87 tweets
    Group 1 posted 242 tweets
    Group 2 posted 41 tweets
    Group 3 posted 349 tweets
    Group 4 posted 101 tweets
    Group 5 posted 358 tweets
    Group 6 posted 434 tweets
    Group 7 posted 521 tweets

# Part 3:  Tokens that are relatively popular in each user partition

In this step, we are going to find tokens that are relatively popular in each user partition.

We define the number of mentions of a token $t$ in a specific user partition $k$ as the number of users from the user partition $k$ that ever mentioned the token $t$ in their tweets. Note that even if some users might mention a token $t$ multiple times or in multiple tweets, a user will contribute at most 1 to the counter of the token $t$.

Please make sure that the number of mentions of a token is equal to the number of users who mentioned this token but NOT the number of tweets that mentioned this token.

Let $N_t^k$ be the number of mentions of the token $t$ in the user partition $k$. Let $N_t^{all} = \sum_{i=0}^7 N_t^{i}$ be the number of total mentions of the token $t$.

We define the relative popularity of a token $t$ in a user partition $k$ as the log ratio between $N_t^k$ and $N_t^{all}$, i.e.

\begin{equation}
p_t^k = \log \frac{N_t^k}{N_t^{all}}.
\end{equation}

You can compute the relative popularity by calling the function `get_rel_popularity`.

We load a tweet tokenizer for you in the following cells. This Tokenizer object is called `tok`. Don't forget to execute the two cells below.

You can expand the following cell if needed to see the minutae of the Tokenizer.

```python
#!/usr/bin/env python

"""
This code implements a basic, Twitter-aware tokenizer.

A tokenizer is a function that splits a string of text into words. In
Python terms, we map string and unicode objects into lists of unicode
objects.

There is not a single right way to do tokenizing. The best method
depends on the application.  This tokenizer is designed to be flexible
and this easy to adapt to new domains and tasks.  The basic logic is
this:

1. The tuple regex_strings defines a list of regular expression
   strings.

2. The regex_strings strings are put, in order, into a compiled
   regular expression object called word_re.

3. The tokenization is done by word_re.findall(s), where s is the
   user-supplied string, inside the tokenize() method of the class
   Tokenizer.

4. When instantiating Tokenizer objects, there is a single option:
   preserve_case.  By default, it is set to True. If it is set to
   False, then the tokenizer will downcase everything except for
   emoticons.

The __main__ method illustrates by tokenizing a few examples.

I've also included a Tokenizer method tokenize_random_tweet(). If the
twitter library is installed (http://code.google.com/p/python-twitter/)
and Twitter is cooperating, then it should tokenize a random
English-language tweet.

Julaiti Alafate:
  I modified the regex strings to extract URLs in tweets.
"""

__author__ = "Christopher Potts"
__copyright__ = "Copyright 2011, Christopher Potts"
__credits__ = []
__license__ = "Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-nc-sa/3.0/"
__version__ = "1.0"
__maintainer__ = "Christopher Potts"
__email__ = "See the author's website"

######################################################################

import re
from html import entities

######################################################################
# The following strings are components in the regular expression
# that is used for tokenizing. It's important that phone_number
# appears first in the final regex (since it can contain whitespace).
# It also could matter that tags comes after emoticons, due to the
# possibility of having text like
#
#     <:| and some text >:)
#
# Most imporatantly, the final element should always be last, since it
# does a last ditch whitespace-based tokenization of whatever is left.

# This particular element is used in a couple ways, so we define it
# with a name:
emoticon_string = r"""
    (?:
      [<>]?
      [:;=8]                     # eyes
      [\-o\*\']?                 # optional nose
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      |
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      [\-o\*\']?                 # optional nose
      [:;=8]                     # eyes
      [<>]?
    )"""

# The components of the tokenizer:
regex_strings = (
    # Phone numbers:
    r"""
    (?:
      (?:            # (international)
        \+?[01]
        [\-\s.]*
      )?
      (?:            # (area code)
        [\(]?
        \d{3}
        [\-\s.\)]*
      )?
      \d{3}          # exchange
      [\-\s.]*
      \d{4}          # base
    )"""
    ,
    # URLs:
    r"""http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"""
    ,
    # Emoticons:
    emoticon_string
    ,
    # HTML tags:
     r"""<[^>]+>"""
    ,
    # Twitter username:
    r"""(?:@[\w_]+)"""
    ,
    # Twitter hashtags:
    r"""(?:\#+[\w_]+[\w\'_\-]*[\w_]+)"""
    ,
    # Remaining word types:
    r"""
    (?:[a-z][a-z'\-_]+[a-z])       # Words with apostrophes or dashes.
    |
    (?:[+\-]?\d+[,/.:-]\d+[+\-]?)  # Numbers, including fractions, decimals.
    |
    (?:[\w_]+)                     # Words without apostrophes or dashes.
    |
    (?:\.(?:\s*\.){1,})            # Ellipsis dots.
    |
    (?:\S)                         # Everything else that isn't whitespace.
    """
    )

######################################################################
# This is the core tokenizing regex:

word_re = re.compile(r"""(%s)""" % "|".join(regex_strings), re.VERBOSE | re.I | re.UNICODE)

# The emoticon string gets its own regex so that we can preserve case for them as needed:
emoticon_re = re.compile(regex_strings[1], re.VERBOSE | re.I | re.UNICODE)

# These are for regularizing HTML entities to Unicode:
html_entity_digit_re = re.compile(r"&#\d+;")
html_entity_alpha_re = re.compile(r"&\w+;")
amp = "&"

######################################################################

class Tokenizer:
    def __init__(self, preserve_case=False):
        self.preserve_case = preserve_case

    def tokenize(self, s):
        """
        Argument: s -- any string or unicode object
        Value: a tokenize list of strings; conatenating this list returns the original string if preserve_case=False
        """
        # Try to ensure unicode:
        try:
            s = str(s)
        except UnicodeDecodeError:
            s = s.encode('string_escape')
            s = str(s)
        # Fix HTML character entitites:
        s = self.__html2unicode(s)
        # Tokenize:
        words = word_re.findall(s)
        # Possible alter the case, but avoid changing emoticons like :D into :d:
        if not self.preserve_case:
            words = map((lambda x : x if emoticon_re.search(x) else x.lower()), words)
        return words

    def tokenize_random_tweet(self):
        """
        If the twitter library is installed and a twitter connection
        can be established, then tokenize a random tweet.
        """
        try:
            import twitter
        except ImportError:
            print("Apologies. The random tweet functionality requires the Python twitter library: http://code.google.com/p/python-twitter/")
        from random import shuffle
        api = twitter.Api()
        tweets = api.GetPublicTimeline()
        if tweets:
            for tweet in tweets:
                if tweet.user.lang == 'en':
                    return self.tokenize(tweet.text)
        else:
            raise Exception("Apologies. I couldn't get Twitter to give me a public English-language tweet. Perhaps try again")

    def __html2unicode(self, s):
        """
        Internal metod that seeks to replace all the HTML entities in
        s with their corresponding unicode characters.
        """
        # First the digits:
        ents = set(html_entity_digit_re.findall(s))
        if len(ents) > 0:
            for ent in ents:
                entnum = ent[2:-1]
                try:
                    entnum = int(entnum)
                    s = s.replace(ent, unichr(entnum))
                except:
                    pass
        # Now the alpha versions:
        ents = set(html_entity_alpha_re.findall(s))
        ents = filter((lambda x : x != amp), ents)
        for ent in ents:
            entname = ent[1:-1]
            try:
                s = s.replace(ent, unichr(entities.name2codepoint[entname]))
            except:
                pass
            s = s.replace(amp, " and ")
        return s

```

```python
from math import log

tok = Tokenizer(preserve_case=False)

def get_rel_popularity(c_k, c_all):
    return log(1.0 * c_k / c_all) / log(2)

def print_tokens(tokens, gid = None):
    group_name = "overall"
    if gid is not None:
        group_name = "group %d" % gid
    print('=' * 5 + ' ' + group_name + ' ' + '=' * 5)
    for t, n in tokens:
        print("%s\t%.4f" % (t, n))
    print
```

## Tokenize tweets

1. Tokenize the tweets using the `tokenize` function that is a method of the `Tokenizer` class that we have instantiated as `tok`.

1. Count the number of mentions for each tokens regardless of specific user group.

1. Call `print_count` function to show how many different tokens we have.

**It should print**
```
Number of tokens: 7677
```

```python
# YOUR CODE HERE
#for x in tok.tokenize("x[1] Hello world"):
#    print(x)

# Create list of tokens per user.
tokenizedRdd = tweetRdd.map(lambda x: (x[0], list(tok.tokenize(x[1])))).cache()

# Use flatMap to get the list returned by each lambda function applied to a value and merge them at the end.
# Then we removed duplicated elements.
num_of_tokens = tokenizedRdd.flatMap(lambda x: (x[1])).distinct().count()

my_output.append("num-tokens", num_of_tokens)
print("Number of tokens:", num_of_tokens)
```

    Number of tokens: 7677

## Token popularity

Tokens that are mentioned by too few users are usually not very interesting. So we want to only keep tokens that are mentioned by at least 100 users. Filter out tokens that don't meet this requirement.

Call `print_count` function to show how many different tokens we have after the filtering.

Call `print_tokens` function to show top 20 most frequent tokens.

**It should print**
```
Number of tokens: 46
===== overall =====
:	1046.0000
rt	920.0000
.	767.0000
the	587.0000
trump	560.0000
…	520.0000
to	501.0000
,	497.0000
in	385.0000
a	383.0000
is	382.0000
of	300.0000
!	285.0000
for	275.0000
and	263.0000
on	218.0000
i	216.0000
he	191.0000
that	190.0000
"	181.0000
```

```python
timer_start()

def tokenListExplode(l):
    ret = []
    for token in l:
        ret.append((token, 1))
    return ret

freqTokenThreshold = 100

# YOUR CODE HERE
# Create list of unique tokens per user.
# We use the set data structure to remove duplicates and convert back to list at the end.
# Then we merge all tweets by the same user and remove duplicates again.
uniqueTokenizedRdd = tokenizedRdd.map(lambda x: (x[0], list(set(x[1])))).\
    reduceByKey(lambda x,y: x + y).map(lambda x: (x[0], list(set(x[1]))))

# Sum all token count, filter only tokens that appeared more than a threshold and
# reverse pairs to sort by count amount.
# Note that we only reverse the pair order to sort by count value and then reverse back.
freqTokens = uniqueTokenizedRdd.flatMap(lambda x: tokenListExplode(x[1])).\
    reduceByKey(lambda x,y: x + y).\
    filter(lambda x: x[1] >= freqTokenThreshold).\
    map(lambda x: (x[1], x[0])).sortByKey(ascending = False).map(lambda x: (x[1], x[0])).cache()

top20 = freqTokens.take(20)
num_freq_tokens = freqTokens.count()

my_output.append("num-freq-tokens", num_freq_tokens)
my_output.append("top-20-tokens", top20)
print("Number of tokens:", num_freq_tokens)
print_tokens(top20)
timer_stop("Count overall most popular tokens")
```

    Number of tokens: 46
    ===== overall =====
    :	1046.0000
    rt	920.0000
    .	767.0000
    the	587.0000
    trump	560.0000
    …	520.0000
    to	501.0000
    ,	497.0000
    in	385.0000
    a	383.0000
    is	382.0000
    of	300.0000
    !	285.0000
    for	275.0000
    and	263.0000
    on	218.0000
    i	216.0000
    he	191.0000
    that	190.0000
    "	181.0000

## Relative Popularity

For all tokens that are mentioned by at least 100 users, compute their relative popularity in each user group. Then print the top 10 tokens with highest relative popularity in each user group. In case two tokens have same relative popularity, break the tie by printing the alphabetically smaller one.

**Hint:** Let the relative popularity of a token $t$ be $p$. The order of the items will be satisfied by sorting them using (-p, t) as the key.

**It should print**
```
===== group 0 =====
with	-3.6088
cruz	-3.6554
his	-3.6582
amp	-3.8651
on	-3.9608
to	-4.0145
&	-4.0875
https	-4.1699
i	-4.1699
what	-4.1699
===== group 1 =====
sanders	-2.2854
gop	-2.4060
hillary	-2.4330
’	-2.4463
bernie	-2.4835
"	-2.6925
are	-2.7249
this	-2.7633
for	-2.8179
about	-2.8346
===== group 2 =====
with	-4.3458
donald	-4.5146
...	-4.7004
gop	-4.7279
i	-4.9475
on	-4.9608
he	-4.9925
…	-5.1155
https	-5.1699
what	-5.1699
===== group 3 =====
bernie	-1.5945
sanders	-1.6609
hillary	-2.2188
and	-2.5154
"	-2.5930
in	-2.6114
will	-2.6160
https	-2.6674
...	-2.7004
you	-2.7004
===== group 4 =====
what	-3.4330
have	-3.4725
bernie	-3.5380
this	-3.5518
it	-3.6881
?	-3.6912
for	-3.7110
about	-3.7415
hillary	-3.7549
that	-3.7625
===== group 5 =====
what	-1.8007
not	-1.8745
https	-2.0000
his	-2.0144
cruz	-2.0704
it	-2.1031
on	-2.1243
&	-2.1399
amp	-2.1489
;	-2.1592
===== group 6 =====
will	-1.3847
have	-1.4725
!	-1.5850
cruz	-1.6919
trump	-1.7199
https	-1.7549
-	-1.7673
;	-1.7807
be	-1.7952
amp	-1.8144
===== group 7 =====
donald	-1.0740
trump	-1.6535
bernie	-1.7790
sanders	-1.7829
’	-1.8613
of	-1.9069
?	-1.9186
with	-1.9307
the	-1.9588
be	-1.9758
```

```python
# Auxiliar method for the next section.
def getTokenKey(pair):
    return (-1 * pair[1], pair[0])

def mergeGroupTokens(tokens1, tokens2):
    maxLen = 10
    mergedTokens = tokens1 + tokens2
    mergedTokens = sorted(mergedTokens, key = getTokenKey)
    if len(mergedTokens) <= maxLen:
        return mergedTokens

    delta = maxLen - len(mergedTokens)
    return mergedTokens[:maxLen]

pair1 = ("aaa", -0.444)
pair2 = ("aaa", -0.414)
pair3 = ("aaa", -0.314)
print(compTokens(pair1, pair2))

print(mergeGroupTokens([pair1, pair2], [pair3]))
```

    -0.030000000000000027
    [('aaa', -0.314), ('aaa', -0.414), ('aaa', -0.444)]

```python
timer_start()

def removeUnfreqTokens(l, freqTokens):
    ret = []
    for x in l:
        if x in freqTokens:
            ret.append(x)
    return ret

def separateByTokenAndGroup(pair):
    groupId = pair[0]
    tokens = pair[1]
    ret = []
    for t in tokens:
        ret.append(((groupId, t), 1))
    return ret

# YOUR CODE HERE
freqTokensList = freqTokens.map(lambda x: x[0]).collect()

# We start creating a RDD of Tweet user group x List of Unique tokens.
uniqueFreqTokenByGroupRdd = uniqueTokenizedRdd.map(lambda x: (getTweetUserPartition(x[0]), x[1]))

# Then we filter to only display the most frequent tokens for each entry.
uniqueFreqTokenByGroupRdd = uniqueFreqTokenizedRdd.map(lambda x: (x[0], removeUnfreqTokens(x[1], freqTokensList)))

# Preparing tokens to be counted per group, using each entry as ((group, token), 1).
# We use flat map to transform a group list entry into multiple entries indexed by group and token.
# Each value shouldn't become a long list since we didn't reduced all tweets yet.
tokenCountByGroup = uniqueFreqTokenByGroupRdd.flatMap(separateByTokenAndGroup)

# Reducing all by key to count the number of times which each token appear.
tokenCountByGroup = tokenCountByGroup.reduceByKey(lambda x, y: x + y)

# Now we need to transform the count value into the popularity value.
freqTokensMap = freqTokens.collectAsMap()
tokenFreqByGroup = tokenCountByGroup.map(lambda x: (x[0], get_rel_popularity(x[1], freqTokensMap[x[0][1]])))

# Sorting by popularity decreasing order since it is based on log(x), x < 1.
# We sort by group and untie by popularity and Lexicographical order.
# Note that we reverse popularity just to be able to untie by lexicographical order here.
tokenFreqByGroup = tokenFreqByGroup.sortBy(lambda x: (x[0][0], -1 * x[1], x[0][1]), ascending = True)

# To filter only N biggest elements per group, we convert each value to a list and merge the lists
# with a special function
tokenFreqByGroup = tokenFreqByGroup.map(lambda x: (x[0][0], [(x[0][1], x[1])])).\
                    reduceByKey(mergeGroupTokens)

popular_10_in_each_group = tokenFreqByGroup.collectAsMap()

# Following code add your solution to `my_output`
my_output.append("popular_10_in_each_group", popular_10_in_each_group)
for k in range(8):
    print_tokens(popular_10_in_each_group[k], k)
timer_stop("Print popular tokens in each group")
```

    ===== group 0 =====
    with	-3.6088
    cruz	-3.6554
    his	-3.6582
    amp	-3.8651
    on	-3.9608
    to	-4.0145
    &	-4.0875
    https	-4.1699
    i	-4.1699
    what	-4.1699
    ===== group 1 =====
    sanders	-2.2854
    gop	-2.4060
    hillary	-2.4330
    ’	-2.4463
    bernie	-2.4835
    "	-2.6925
    are	-2.7249
    this	-2.7633
    for	-2.8179
    about	-2.8346
    ===== group 2 =====
    with	-4.3458
    donald	-4.5146
    ...	-4.7004
    gop	-4.7279
    i	-4.9475
    on	-4.9608
    he	-4.9925
    …	-5.1155
    https	-5.1699
    what	-5.1699
    ===== group 3 =====
    bernie	-1.5945
    sanders	-1.6609
    hillary	-2.2188
    and	-2.5154
    "	-2.5930
    in	-2.6114
    will	-2.6160
    https	-2.6674
    ...	-2.7004
    you	-2.7004
    ===== group 4 =====
    what	-3.4330
    have	-3.4725
    bernie	-3.5380
    this	-3.5518
    it	-3.6881
    ?	-3.6912
    for	-3.7110
    about	-3.7415
    hillary	-3.7549
    that	-3.7625
    ===== group 5 =====
    what	-1.8007
    not	-1.8745
    https	-2.0000
    his	-2.0144
    cruz	-2.0704
    it	-2.1031
    on	-2.1243
    &	-2.1399
    amp	-2.1489
    ;	-2.1592
    ===== group 6 =====
    will	-1.3847
    have	-1.4725
    !	-1.5850
    cruz	-1.6919
    trump	-1.7199
    https	-1.7549
    -	-1.7673
    ;	-1.7807
    be	-1.7952
    amp	-1.8144
    ===== group 7 =====
    donald	-1.0740
    trump	-1.6535
    bernie	-1.7790
    sanders	-1.7829
    ’	-1.8613
    of	-1.9069
    ?	-1.9186
    with	-1.9307
    the	-1.9588
    be	-1.9758

## Important: Write your solutions to disk

Following cell write your solutions to disk which would be read in by the grader for grading.

```python
my_output.write_to_disk()
```

## Optional Ungraded Exercise
The users partition is generated by a machine learning algorithm that tries to group the users by their political preferences. Three of the user groups are showing supports to Bernie Sanders, Ted Cruz, and Donald Trump.

If your program looks okay on the local test data, you can try it on the larger input by submitting your program to the homework server. Observe the output of your program to larger input files, can you guess the partition IDs of the three groups mentioned above based on your output?

```python
# Change the values of the following three items to your guesses
users_support = [
    (-1, "Bernie Sanders"),
    (-1, "Ted Cruz"),
    (-1, "Donald Trump")
]

for gid, candidate in users_support:
    print("Users from group %d are most likely to support %s." % (gid, candidate))
```

Uncomment the lines in the cell below to print out the time taken between `time_start()` and `time_end()` at various stages of your pipeline.

```python
# total_time = 0.0
# for item in timer:
#     print("{}\t{}".format(item[0], item[1]))
#     total_time += item[1]
# print("Total time: {}".format(total_time))
```

```python

```


```
#@title Copyright 2022 The Cirq Developers
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Best practices

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

```
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    import cirq
    print("installed cirq.")
```

```
import cirq_google as cg
import sympy
```

This section lists some best practices for creating a circuit that performs well
on Google hardware devices. This is an area of active research, so users are
encouraged to try multiple approaches to improve results.

This guide is split into three parts:
*  Getting your circuit to run
*  Making it run faster
*  Lowering error

## Getting a circuit to run on hardware

In order to run on hardware, the circuit must only use qubits and gates that the
device supports.  Using inactive qubits, non-adjacent qubits, or non-native
gates will immediately cause a circuit to fail.

Validating a circuit with a device, such as
`cg.Sycamore.validate_circuit(circuit)` will test a lot of these
conditions.  Calling the `validate_circuit` function will work with any
device, including those retrieved directly from the API using the
[engine object](./specification.md#serializable-devices), which can help
identify any qubits used in the circuit that have been disabled on the actual
device.

### Using built-in tranformers as a first pass

Using built-in transformers will allow you to compile to the correct gate set. As they are
automated solutions, they will not always perform as well as a hand-crafted solution, but
they provide a good starting point for creating a circuit that is likely to run successfully
on hardware. Best practice is to inspect the circuit after optimization to make sure
that it has compiled without unintended consequences.

```
# Create your circuit here
my_circuit = cirq.Circuit()

# Convert the circuit to run on a Google target gateset.
# The google specific `cirq.CompilationTargetGateset` specifies the target gateset
# and a sequence of appropriate optimization routines that should be executed to compile
# a circuit to run on this target.
sycamore_circuit = cirq.optimize_for_target_gateset(my_circuit, gateset=cg.SycamoreTargetGateset())
```

### Using CircuitOperation to reduce circuit size

Particularly large batches (or sweeps) of circuits may encounter errors when
sent to Quantum Engine due to an upper limit on request size. If the circuits
in question have a repetitive structure, `cirq.CircuitOperation`s can be used
to reduce the request size and avoid this limit.

```
# Repeatedly apply Hadamard and measurement to 10 qubits.
my_circuit = cirq.Circuit()
qubits = cirq.GridQubit.rect(2, 5)
for i in range(100):
    my_circuit.append(cirq.H.on_each(*qubits))
    for qb in qubits:
        my_circuit.append(cirq.measure(qb, key=cirq.MeasurementKey.parse_serialized(f'{i}:m{qb}')))

print(my_circuit)

# The same circuit, but defined using CircuitOperations.
# This is ~1000x smaller when serialized!
q = cirq.NamedQubit("q")
sub_circuit = cirq.FrozenCircuit(cirq.H(q), cirq.measure(q, key='m'))
circuit_op = cirq.CircuitOperation(sub_circuit).repeat(100)
short_circuit = cirq.Circuit(
    circuit_op.with_qubits(q).with_measurement_key_mapping({'m': f'm{q}'}) for q in qubits
)
print(short_circuit)
```

When compiling circuits with `CircuitOperation`s, providing a context
with `deep=True` will preserve the `CircuitOperation`s while
optimizing their contents. This is useful for producing a concise,
device-compatible circuit.

```
syc_circuit = cirq.optimize_for_target_gateset(
    short_circuit,
    gateset=cg.SycamoreTargetGateset(),
    context=cirq.TransformerContext(deep=True)
)
print(syc_circuit)
```

## Running circuits faster

The following sections give tips and tricks that allow you to improve your
repetition rate (how many repetitions per second the device will run).

This will allow you to make the most out of limited time on the
device by getting results faster. The shorter experiment time may
also reduce error due to drift of qubits away from calibration.

There are costs to sending circuits over the network, to compiling each
circuit into waveforms, to initializing the device,
and to sending results back over the network.
These tips will aid you in removing some of this overhead by combining your
circuits into sweeps or batches.

### Use sweeps when possible

Round trip network time to and from the engine typically adds latency on the order of a second
to the overall computation time.  Reducing the number of trips and allowing the engine to
properly batch circuits can improve the throughput of your calculations.  One way to do this
is to use parameter sweeps to send multiple variations of a circuit at once.

One example is to turn single-qubit gates on or off by using parameter sweeps.
For instance, the following code illustrates how to combine measuring in the
Z basis or the X basis in one circuit.

```
q = cirq.GridQubit(1, 1)
sampler = cirq.Simulator()

# STRATEGY #1: Have a separate circuit and sample call for each basis.
circuit_z = cirq.Circuit(
    cirq.measure(q, key='out'))
circuit_x = cirq.Circuit(
    cirq.H(q),
    cirq.measure(q, key='out'))
samples_z = sampler.sample(circuit_z, repetitions=5)
samples_x = sampler.sample(circuit_x, repetitions=5)

print("Measurement in Z Basis:", samples_z, sep="\n")
print("Measurement in X Basis:", samples_x, sep="\n")
```

```
# STRATEGY #2: Have a parameterized circuit.
circuit_sweep = cirq.Circuit(
    cirq.H(q)**sympy.Symbol('t'),
    cirq.measure(q, key='out'))

samples_sweep = sampler.sample(circuit_sweep,
                               repetitions=5,
                               params=[{'t': 0}, {'t': 1}])
print(samples_sweep)
```

One word of caution is there is a limit to the total number of repetitions.  Take some care
that your parameter sweeps, especially products of sweeps, do not become so excessively large
that they exceed this limit.

### Use batches if sweeps are not possible

The engine has a method called `run_batch()` that can be used to send multiple
circuits in a single request.  This can be used to increase the efficiency
of your program so that more repetitions are completed per second.

The circuits that are grouped into the same batch must
measure the same qubits and have the same number of repetitions for each
circuit.  Otherwise, the circuits will not be batched together
on the device, and there will be no gain in efficiency.

### Flatten sympy formulas into symbols

Symbols are extremely useful for constructing parameterized circuits (see above).  However,
only some sympy formulas can be serialized for network transport to the engine.
Currently, sums and products of symbols, including linear combinations, are supported.
See `cirq_google.arg_func_langs` for details.

The sympy library is also infamous for being slow, so avoid using complicated formulas if you
care about performance.  Avoid using parameter resolvers that have formulas in them.

One way to eliminate formulas in your gates is to flatten your expressions.
The following example shows how to take a gate with a formula and flatten it
to a single symbol with the formula pre-computed for each value of the sweep:

```
# Suppose we have a gate with a complicated formula.  (e.g. "2^t - 1")
# This formula cannot be serialized
# It could potentially encounter sympy slowness.
gate_with_formula = cirq.XPowGate(exponent=2 ** sympy.Symbol('t') - 1)
sweep = cirq.Linspace('t', start=0, stop=1, length=5)

# Instead of sweeping the formula, we will pre-compute the values of the formula
# at every point and store it a new symbol called '<2**t - 1>'
sweep_for_gate, flat_sweep = cirq.flatten_with_sweep(gate_with_formula, sweep)

print(repr(sweep_for_gate))

print(list(flat_sweep.param_tuples()))
```

## Improving circuit fidelity

The following tips and tricks show how to modify your circuit to
reduce error rates by following good circuit design principles that
minimize the length of circuits.

Quantum Engine will execute a circuit as faithfully as possible.
This means that moment structure will be preserved. That is, all gates in a
moment are guaranteed to be executed before those in any later moment and
after gates in previous moments.  Many of these tips focus on having a
good moment structure that avoids problematic missteps that can cause
unwanted noise and error.

Note: See the [Circuit optimization, gate alignment, and spin echoes tutorial](../tutorials/google/spin_echoes.ipynb) for an example of the best practices discussed in this section.

### Short gate depth

In the current NISQ (noisy intermediate scale quantum) era, gates and devices still
have significant error. Both gate errors and T1 decay rate can cause long circuits
to have noise that overwhelms any signal in the circuit.

The recommended gate depths vary significantly with the structure of the circuit itself
and will likely increase as the devices improve. Total circuit fidelity can be roughly
estimated by multiplying the fidelity for all gates in the circuit. For example,
using a error rate of 0.5% per gate, a circuit of depth 20 and width 20 could be estimated
at 0.995^(20 * 20) = 0.135. Using separate error rates per gates (i.e. based on calibration
metrics) or a more complicated noise model can result in more accurate error estimation.

### Terminal Measurements

Make sure that measurements are kept in the same moment as the final moment in
the circuit.  Make sure that any circuit optimizers do not alter this by
incorrectly pushing measurements forward. This behavior can be avoided by
measuring all qubits with a single gate or by adding
the measurement gate after all optimizers have run.

Currently, only terminal measurements are supported by the hardware.  If you
absolutely need intermediate measurements for your application, reach out to
your Google sponsor to see if they can help devise a proper circuit using
intermediate measurements.

### Keep qubits busy

Qubits that remain idle for long periods tend to dephase and decohere. Inserting a
[Spin Echo](https://en.wikipedia.org/wiki/Spin_echo) into your circuit onto
qubits that have long idle periods, such as a pair
of involutions, such as two successive Pauli Y gates, will generally increase
performance of the circuit.

Be aware that this should be done after calling
`cirq.optimize_for_target_gateset`, since this function will 'optimize'
these operations out of the circuit. You can also tag the spin echo operations with a no-compile tag, and include these tags in `context.tags_to_ignore`, so that the transformer ignores all tagged operations marked with any of `context.tags_to_ignore`.

### Delay initialization of qubits

The |0⟩ state is more robust than the |1⟩ state. As a result, one should
not initialize a qubit to |1⟩ at the beginning of the circuit until shortly
before other gates are applied to it.

### Align single-qubit and two-qubit layers

Devices are generally calibrated to circuits that alternate single-qubit gates with
two-qubit gates in each layer. Staying close to this paradigm will often improve
performance of circuits.  This will also reduce the circuit's total duration,
since the duration of a moment is its longest gate.  Making sure that each layer
contains similar gates of the same duration can be challenging, but it will
likely have a measurable impact on the fidelity of your circuit.

Devices generally operate in the Z basis, so that rotations around the Z axis will become
book-keeping measures rather than physical operations on the device. These
virtual Z operations have zero duration and have no cost, if they add no moments
to your circuit.  In order to guarantee that they do not add moments, you can
make sure that virtual Z are aggregated into their own layer.  Alternatively,
you can use the `cirq.eject_z` optimizer to propagate these Z gates forward through
commuting operators.

See the function `cirq.stratified_circuit` for an automated way to organize gates
into moments with similar gates.

### Qubit picking

On current NISQ devices, qubits cannot be considered identical.  Different
qubits can have vastly different performance and can vary greatly from day
to day.  It is important for experiments to have a dynamic method to
pick well-performing qubits that maximize the fidelity of the experiment.
There are several techniques that can assist with this.

*   Analyze calibration metrics:  performance of readout, single-qubit, and
two-qubit gates are measured as a side effect of running the device's
calibration procedure.  These metrics can be used as a baseline to evaluate
circuit performance or identify outliers to avoid.  This data can be inspected
programmatically by retrieving metrics from the [API](calibration.md) or
[visually by applying a cirq.Heatmap](../tutorials/google/visualizing_calibration_metrics.ipynb)
to that data or by using the built-in
heatmaps in the Cloud console page for the processor.  Note that, since this
data is only taken during calibration (e.g. at most daily), drifts and other
concerns may affect the values significantly, so these metrics should only be
used as a first approximation.  There is no substitute for actually running characterizations
on the device.
*   Loschmidt echo:  Running a small circuit on a string of qubits and then
applying the circuit's inverse can be used as a quick but effective way to
judge qubit quality.  See
[this tutorial](../tutorials/google/echoes.ipynb) for instructions.
*   XEB:  Cross-entropy benchmarking is another way to gauge qubit performance
on a set of random circuits.  See tutorials on
[parallel XEB](../qcvv/parallel_xeb.ipynb)
or [isolated XEB](../qcvv/isolated_xeb.ipynb) for instructions.

### Refitting gates

Virtual Z gates (or even single qubit gates) can be added to adjust for errors
in two qubit gates.  Two qubit gates can have errors due to drift, coherent
error, unintended cross-talk, or other sources.  Refitting these gates and
adjusting the circuit for the observed unitary of the two qubit gate
compared to the ideal unitary can substantially improve results.
However, this approach can use a substantial amount of resources.

This technique involves two distinct steps.  The first is *characterization*,
which is to identify the true behavior of the two-qubit gate.  This typically
involves running many varied circuits involving the two qubit gate in a method
(either periodic or random) to identify the parameters of the gate's behavior.

Entangling gates used in Google's architecture fall into a general category of FSim gates,
standing for *Fermionic simulation*.  The generalized version of this gate can
be parameterized into 5 angles, or degrees of freedom.  Characterization will
attempt to identify the values of these five angles.

The second step is calibrating (or refitting) the gate.  Out of the five angles
that comprise the generalized FSim gate, three can be corrected for by adding
Z rotations before or after the gate.  Since these gates are propagated forward
automatically, they add no duration or error to the circuit and can essentially
be added "for free".  See the [devices page](devices.md#virtual_z_gates) for more
information on Virtual Z gates.  Note that it is important to keep the single-qubit and
two-qubit gates aligned (see above) while performing this procedure so that
the circuit stays the same duration.

For more on calibration and detailed instructions on how to perform these procedures, see the following tutorials:

* [Calibration API](../tutorials/google/calibration_api.ipynb)
* [Floquet calibration example](../tutorials/google/floquet_calibration_example.ipynb)
* [XEB calibration example](../tutorials/google/xeb_calibration_example.ipynb)


[](https://colab.research.google.com/github/pySTEPS/pysteps/blob/master/examples/my_first_nowcast.ipynb)

# My first precipitation nowcast

In this example, we will use pysteps to compute and plot an extrapolation nowcast using the NSSL's Multi-Radar/Multi-Sensor System
([MRMS](https://www.nssl.noaa.gov/projects/mrms/)) rain rate product.

The MRMS precipitation product is available every 2 minutes, over the contiguous US.
Each precipitation composite has 3500 x 7000 grid points, separated 1 km from each other.

## Set-up Colab environment

**Important**: In colab, execute this section one cell at a time. Trying to excecute all the cells at once may results in cells being skipped and some dependencies not being installed.

First, let's set up our working environment. Note that these steps are only needed to work with google colab.

To install pysteps locally, you can follow [these instructions](https://pysteps.readthedocs.io/en/latest/user_guide/install_pysteps.html).

First, let's install the latest Pysteps version from the Python Package Index (PyPI) using pip. This will also install the minimal dependencies needed to run pysteps.

#### Install optional dependencies

Now, let's install the optional dependendies that will allow us to plot and read the example data.
- pygrib: to read the MRMS data grib format
- pyproj: needed by pygrib

**NOTE:** Do not import pysteps in this notebook until the following optional dependencies are loaded. Otherwise, pysteps will assume that they are not installed and some of its functionalities won't work.

```python
# These libraries are needed for the pygrib library in Colab.
# Note that is needed if you install pygrib using pip.
# If you use conda, the libraries will be installed automatically.
! apt-get install libeccodes-dev libproj-dev

# Install the python packages
! pip install pyproj
! pip install pygrib
```

```python
# Uninstall existing shapely
# We will re-install shapely in the next step by ignoring the binary
# wheels to make it compatible with other modules that depend on
# GEOS, such as Cartopy (used here).
!pip uninstall --yes shapely
```

```python
# To install cartopy in Colab using pip, we need to install the library
# dependencies first.

!apt-get install -qq libgdal-dev libgeos-dev
!pip install shapely --no-binary shapely
!pip install cartopy
```

#### Install pysteps

Now that all dependencies are installed, we can install pysteps.

```python
# ! pip install pysteps
! pip install git+https://github.com/pySTEPS/pysteps
```

## Getting the example data

Now that we have the environment ready, let's install the example data and configure the pysteps's default parameters by following [this tutorial](https://pysteps.readthedocs.io/en/latest/user_guide/example_data.html).

First, we will use the [pysteps.datasets.download_pysteps_data()](https://pysteps.readthedocs.io/en/latest/generated/pysteps.datasets.download_pysteps_data.html) function to download the data.

```python
# Import the helper functions
from pysteps.datasets import download_pysteps_data, create_default_pystepsrc

# Download the pysteps data in the "pysteps_data"
download_pysteps_data("pysteps_data")
```

Next, we need to create a default configuration file that points to the downloaded data.
By default, pysteps will place the configuration file in `$HOME/.pysteps` (unix and Mac OS X) or `$USERPROFILE/pysteps` (windows).
To quickly create a configuration file, we will use the [pysteps.datasets.create_default_pystepsrc()](https://pysteps.readthedocs.io/en/latest/generated/pysteps.datasets.create_default_pystepsrc.html#pysteps.datasets.create_default_pystepsrc) helper function.

```python
# If the configuration file is placed in one of the default locations
# (https://pysteps.readthedocs.io/en/latest/user_guide/set_pystepsrc.html#configuration-file-lookup)
# it will be loaded automatically when pysteps is imported.
config_file_path = create_default_pystepsrc("pysteps_data")
```

Since pysteps was already initialized in this notebook, we need to load the new configuration file and update the default configuration.

```python
# Import pysteps and load the new configuration file
import pysteps
_ = pysteps.load_config_file(config_file_path, verbose=True)
```

Let's see what the default parameters look like (these are stored in the
[pystepsrc file](https://pysteps.readthedocs.io/en/latest/user_guide/set_pystepsrc.html)). We will be using them to load the MRMS data set.

```python
# The default parameters are stored in pysteps.rcparams.
from pprint import pprint
pprint(pysteps.rcparams.data_sources['mrms'])
```

This should have printed the following lines:

- `fn_ext`: 'grib2' -- The file extension
- `fn_pattern`: 'PrecipRate_00.00_%Y%m%d-%H%M%S' -- The file naming convention of the MRMS data.
- `importer`: 'mrms_grib' -- The name of the importer for the MRMS data.
- `importer_kwargs`: {} -- Extra options provided to the importer. None in this example.
- `path_fmt`: '%Y/%m/%d' -- The folder structure in which the files are stored. Here, year/month/day/filename.
- `root_path`: '/content/pysteps_data/mrms' -- The root path of the MRMS-data.
- `timestep`: 2 -- The temporal interval of the (radar) rainfall data

Note that the default `timestep` parameter is 2 minutes, which corresponds to the time interval at which the MRMS product is available.

## Load the MRMS example data

Now that we have installed the example data, let's import the example MRMS dataset using the [load_dataset()](https://pysteps.readthedocs.io/en/latest/generated/pysteps.datasets.load_dataset.html) helper function from the `pysteps.datasets` module.

We import 1 hour and 10 minutes of data, which corresponds to a sequence of 35 frames of 2-D precipitation composites.
Note that importing the data takes approximately 30 seconds.

```python
from pysteps.datasets import load_dataset

# We'll import the time module to measure the time the importer needed
import time

start_time = time.time()

# Import the data
precipitation, metadata, timestep = load_dataset('mrms',frames=35)  # precipitation in mm/h

end_time = time.time()

print("Precipitation data imported")
print("Importing the data took ", (end_time - start_time), " seconds")
```

Let's have a look at the values returned by the `load_dataset()` function.

- `precipitation`: A numpy array with (time, latitude, longitude) dimensions.
- `metadata`: A dictionary with additional information (pixel sizes, map projections, etc.).
- `timestep`: Time separation between each sample (in minutes)

```python
# Let's inspect the shape of the imported data array
precipitation.shape
```

Note that the shape of the precipitation is 4 times smaller than the raw MRMS data (3500 x 7000).
The `load_dataset()` function uses the default parameters from `importers` to read the data. By default, the MRMS importer upscales the data 4x. That is, from ~1km resolution to ~4km. It also uses single precision to reduce the memory requirements.
Thanks to the upscaling, the memory footprint of this example dataset is ~200Mb instead of the 3.1Gb of the raw (3500 x 7000) data.

```python
timestep # In minutes
```

```python
pprint(metadata)
```

# Time to make a nowcast

So far, we have 1 hour and 10 minutes of precipitation images, separated 2 minutes from each other.
But, how do we use that data to run a precipitation forecast?

A simple way is by extrapolating the precipitation field, assuming it will continue to move as observed in the recent past, and without changes in intensity. This is commonly known as *Lagrangian persistence*.

The first step to run our nowcast based on Lagrangian persistence, is the estimation of the motion field from a sequence of past precipitation observations.
We use the Lucas-Kanade (LK) optical flow method implemented in pysteps.
This method follows a local tracking approach that relies on the OpenCV package.
Local features are tracked in a sequence of two or more radar images.
The scheme includes a final interpolation step to produce a smooth field of motion vectors.
Other optical flow methods are also available in pysteps.
Check the full list [here](https://pysteps.readthedocs.io/en/latest/pysteps_reference/motion.html).

Now let's use the first 5 precipitation images (10 min) to estimate the motion field of the radar pattern and the remaining 30 images (1h) to evaluate the quality of our forecast.

```python
# precipitation[0:5] -> Used to find motion (past data). Let's call it training precip.
train_precip = precipitation[0:5]

# precipitation[5:] -> Used to evaluate forecasts (future data, not available in "real" forecast situation)
# Let's call it observed precipitation because we will use it to compare our forecast with the actual observations.
observed_precip = precipitation[3:]
```

Let's see what this 'training' precipitation event looks like using the [pysteps.visualization.plot_precip_field](https://pysteps.readthedocs.io/en/latest/generated/pysteps.visualization.precipfields.plot_precip_field.html) function.

```python
from matplotlib import pyplot as plt
from pysteps.visualization import plot_precip_field

# Set a figure size that looks nice ;)
plt.figure(figsize=(9, 5), dpi=100)

# Plot the last rainfall field in the "training" data.
# train_precip[-1] -> Last available composite for nowcasting.
plot_precip_field(train_precip[-1], geodata=metadata, axis="off", map="cartopy")
plt.show()  # (This line is actually not needed if you are using jupyter notebooks)
```

Did you note the **shaded grey** regions? Those are the regions were no valid observations where available to estimate the precipitation (e.g., due to ground clutter, no radar coverage, or radar beam blockage).
Those regions need to be handled with care when we run our nowcast.

### Data exploration

Before we produce a forecast, let's explore the precipitation data. In particular, let's see how the distribution of the rain rate values looks.

```python
import numpy as np

# Let's define some plotting default parameters for the next plots
# Note: This is not strictly needed.
plt.rc('figure', figsize=(4,4))
plt.rc('figure', dpi=100)
plt.rc('font', size=14) # controls default text sizes
plt.rc('axes', titlesize=14) # fontsize of the axes title
plt.rc('axes', labelsize=14) # fontsize of the x and y labels
plt.rc('xtick', labelsize=14) # fontsize of the tick labels
plt.rc('ytick', labelsize=14) # fontsize of the tick labels

# Let's use the last available composite for nowcasting from the "training" data (train_precip[-1])
# Also, we will discard any invalid value.
valid_precip_values = train_precip[-1][~np.isnan(train_precip[-1])]

# Plot the histogram
bins= np.concatenate( ([-0.01,0.01], np.linspace(1,40,39)))
plt.hist(valid_precip_values,bins=bins,log=True, edgecolor='black')
plt.autoscale(tight=True, axis='x')
plt.xlabel("Rainfall intensity [mm/h]")
plt.ylabel("Counts")
plt.title('Precipitation rain rate histogram in mm/h units')
plt.show()
```

The histogram shows that rain rate values have a non-Gaussian and asymmetric distribution that is bounded at zero. Also, the probability of occurrence decays extremely fast with increasing rain rate values (note the logarithmic y-axis).

For better performance of the motion estimation algorithms, we can convert the rain rate values (in mm/h) to a more log-normal distribution  of rain rates by applying the following logarithmic transformation:

\begin{equation}
R\rightarrow
\begin{cases}
    10\log_{10}R, & \text{if } R\geq 0.1\text{mm h$^{-1}$} \\
    -15,          & \text{otherwise}
\end{cases}
\end{equation}

The transformed precipitation corresponds to logarithmic rain rates in units of dBR. The value of −15 dBR is equivalent to assigning a rain rate of approximately 0.03 mm h$^{−1}$ to the zeros.

```python
from pysteps.utils import transformation

# Log-transform the data to dBR.
# The threshold of 0.1 mm/h sets the fill value to -15 dBR.
train_precip_dbr, metadata_dbr = transformation.dB_transform(train_precip, metadata,
                                                             threshold=0.1,
                                                             zerovalue=-15.0)
```

Let's inspect the resulting **transformed precipitation** distribution.

```python
# Only use the valid data!
valid_precip_dbr = train_precip_dbr[-1][~np.isnan(train_precip_dbr[-1])]

plt.figure(figsize=(4, 4), dpi=100)

# Plot the histogram
counts, bins, _ = plt.hist(valid_precip_dbr, bins=40, log=True, edgecolor="black")
plt.autoscale(tight=True, axis="x")
plt.xlabel("Rainfall intensity [dB]")
plt.ylabel("Counts")
plt.title("Precipitation rain rate histogram in dB units")

# Let's add a lognormal distribution that fits that data to the plot.
import scipy

bin_center = (bins[1:] + bins[:-1]) * 0.5
bin_width = np.diff(bins)

# We will only use one composite to fit the function to speed up things.
# First, remove the no precip areas."
precip_to_fit = valid_precip_dbr[valid_precip_dbr > -15]

fit_params = scipy.stats.lognorm.fit(precip_to_fit)

fitted_pdf = scipy.stats.lognorm.pdf(bin_center, *fit_params)

# Multiply pdf by the bin width and the total number of grid points: pdf -> total counts per bin.
fitted_pdf = fitted_pdf * bin_width * precip_to_fit.size

# Plot the log-normal fit
plt.plot(bin_center, fitted_pdf, label="Fitted log-normal")
plt.legend()
plt.show()
```

That looks more like a log-normal distribution. Note the large peak at -15dB. That peak corresponds to "zero" (below threshold) precipitation. The jump with no data in between -15 and -10 dB is caused by the precision of the data, which we had set to 1 decimal. Hence, the lowest precipitation intensities (above zero) are 0.1 mm/h (= -10 dB).

## Compute the nowcast

These are the minimal steps to compute a short-term forecast using Lagrangian extrapolation of the precipitation patterns:

 1. Estimate the precipitation motion field.
 1. Use the motion field to advect the most recent radar rainfall field and produce an extrapolation forecast.

### Estimate the motion field

Now we can estimate the motion field. Here we use a local feature-tracking approach (Lucas-Kanade).
However, check the other methods available in the [pysteps.motion](https://pysteps.readthedocs.io/en/latest/pysteps_reference/motion.html) module.

```python
# Estimate the motion field with Lucas-Kanade
from pysteps import motion
from pysteps.visualization import plot_precip_field, quiver

# Import the Lucas-Kanade optical flow algorithm
oflow_method = motion.get_method("LK")

# Estimate the motion field from the training data (in dBR)
motion_field = oflow_method(train_precip_dbr)

## Plot the motion field.
# Use a figure size that looks nice ;)
plt.figure(figsize=(9, 5), dpi=100)
plt.title("Estimated motion field with the Lukas-Kanade algorithm")

# Plot the last rainfall field in the "training" data.
# Remember to use the mm/h precipitation data since plot_precip_field assumes
# mm/h by default. You can change this behavior using the "units" keyword.
plot_precip_field(train_precip[-1], geodata=metadata, axis="off")

# Plot the motion field vectors
quiver(motion_field, geodata=metadata, step=40)

plt.show()
```

### Extrapolate the observations

We have all ingredients to make an extrapolation nowcast now.
The final step is to advect the most recent radar rainfall field along the estimated motion field, producing an extrapolation forecast.

```python
from pysteps import nowcasts

start = time.time()

# Extrapolate the last radar observation
extrapolate = nowcasts.get_method("extrapolation")

# You can use the precipitation observations directly in mm/h for this step.
last_observation = train_precip[-1]

last_observation[~np.isfinite(last_observation)] = metadata["zerovalue"]

# We set the number of leadtimes (the length of the forecast horizon) to the
# length of the observed/verification preipitation data. In this way, we'll get
# a forecast that covers these time intervals.
n_leadtimes = observed_precip.shape[0]

# Advect the most recent radar rainfall field and make the nowcast.
precip_forecast = extrapolate(train_precip[-1], motion_field, n_leadtimes)

# This shows the shape of the resulting array with [time intervals, rows, cols]
print("The shape of the resulting array is: ", precip_forecast.shape)

end = time.time()
print("Advecting the radar rainfall fields took ", (end - start), " seconds")
```

Let's inspect the last forecast time (hence this is the forecast rainfall an hour ahead).

```python
# Plot precipitation at the end of the forecast period.
plt.figure(figsize=(9, 5), dpi=100)
plot_precip_field(precip_forecast[-1], geodata=metadata, axis="off", map='cartopy')
plt.show()
```

## Evaluate the forecast quality

Many verification methods are already present in pysteps (see a complete list [here](https://pysteps.readthedocs.io/en/latest/pysteps_reference/verification.html)). We just have to import them.

Here, we will evaluate our forecast using the Fractions Skill Score (FSS).
This metric provides an intuitive assessment of the dependency of forecast skill on spatial scale and intensity. This makes the FSS an ideal skill score for high-resolution precipitation forecasts.

More precisely, the FSS is a neighborhood spatial verification method that directly compares the fractional coverage of events in windows surrounding the observations and forecasts.
The FSS varies from 0 (total mismatch) to 1 (perfect forecast).
For most situations, an FSS value of > 0.5 serves as a good indicator of a useful forecast ([Roberts and Lean, 2008](https://journals.ametsoc.org/doi/full/10.1175/2007MWR2123.1) and [Skok and Roberts, 2016](https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/qj.2849)).

```python
from pysteps import verification

fss = verification.get_method("FSS")

# Compute fractions skill score (FSS) for all lead times for different scales using a 1 mm/h detection threshold.
scales = [
    2,
    4,
    8,
    16,
    32,
    64,
]  # In grid points.

scales_in_km = np.array(scales)*4

# Set the threshold
thr = 1.0  # in mm/h

score = []

# Calculate the FSS for every lead time and all predefined scales.
for i in range(n_leadtimes):
    score_ = []
    for scale in scales:
        score_.append(
            fss(precip_forecast[i, :, :], observed_precip[i, :, :], thr, scale)
        )
    score.append(score_)

# Now plot it
plt.figure()
x = np.arange(1, n_leadtimes+1) * timestep
plt.plot(x, score, lw=2.0)
plt.xlabel("Lead time [min]")
plt.ylabel("FSS ( > 1.0 mm/h ) ")
plt.title("Fractions Skill Score")
plt.legend(
    scales_in_km,
    title="Scale [km]",
    loc="center left",
    bbox_to_anchor=(1.01, 0.5),
    bbox_transform=plt.gca().transAxes,
)
plt.autoscale(axis="x", tight=True)
plt.show()
```

As you can see, the FSS decreases with increasing lead time.
This is expected, as the forecasting quality slowly decreases when we forecast further ahead.
Upscaling the forecast, however, clearly leads to higher skill (up to longer ahead) compared to the forecast on the highest resolutions.

## Concluding remarks
Congratulations, you have successfully made your first nowcast using the pysteps library!
This was a simple extrapolation-based nowcast and a lot more advanced options are possible too, see [the pysteps examples gallery](https://pysteps.readthedocs.io/en/latest/auto_examples/index.html) for some nice examples.