# Homework 2

In this homework, we are going to play with Twitter data.

The data is represented as rows of of [JSON](https://en.wikipedia.org/wiki/JSON#Example) strings.
It consists of [tweets](https://dev.twitter.com/overview/api/tweets), [messages](https://dev.twitter.com/streaming/overview/messages-types), and a small amount of broken data (cannot be parsed as JSON).

For this homework, we will only focus on tweets and ignore all other messages.

## Tweets

A tweet consists of many data fields. [Here is an example](https://gist.github.com/arapat/03d02c9b327e6ff3f6c3c5c602eeaf8b). You can learn all about them in the Twitter API doc. We are going to briefly introduce only the data fields that will be used in this homework.

* `created_at`: Posted time of this tweet (time zone is included)
* `id_str`: Tweet ID - we recommend using `id_str` over using `id` as Tweet IDs, becauase `id` is an integer and may bring some overflow problems.
* `text`: Tweet content
* `user`: A JSON object for information about the author of the tweet
    * `id_str`: User ID
    * `name`: User name (may contain spaces)
    * `screen_name`: User screen name (no spaces)
* `retweeted_status`: A JSON object for information about the retweeted tweet (i.e. this tweet is not original but retweeteed some other tweet)
    * All data fields of a tweet except `retweeted_status`
* `entities`: A JSON object for all entities in this tweet
    * `hashtags`: An array for all the hashtags that are mentioned in this tweet
    * `urls`: An array for all the URLs that are mentioned in this tweet

## Data source

All tweets are collected using the [Twitter Streaming API](https://dev.twitter.com/streaming/overview).

## Users partition

Besides the original tweets, we will provide you with a Pickle file, which contains a partition over 452,743 Twitter users. It contains a Python dictionary `{user_id: partition_id}`. The users are partitioned into 7 groups.

```python
# You can load pickle_content from a file on the local file system
# while testing on your laptop
# To test on your laptop, set `ON_EMR=False`
# To test on AWS for final submission, set `ON_EMR=True`

ON_EMR = False
```

## Grading

We ask you use the `OutputLogger` object `my_output` to store the
results of your program.
We have provided function calls to `my_output.append()` method for
storing the results in all necessary places.
Please make sure NOT to remove these lines

In the last cell of this file, we write the content of `my_output`
to a pickle file which the grader will read in and use for grading.

```python
import os
import pickle

class OutputLogger:
    def __init__(self):
        self.ans = {}

    def append(self, key, value):
        self.ans[key] = value

    def write_to_disk(self):
        if ON_EMR:
            filepath = os.path.expanduser("answer.pickle")
            print("FilePath = {}".format(filepath))
            with open(filepath, 'wb') as f:
                pickle.dump(self.ans, f)
            proc = subprocess.Popen(["/usr/local/hadoop/hadoop-2.7.4/bin/hadoop", "fs", "-copyFromLocal", filepath, "/user/spark/answer.pickle"])
            proc.wait()
            os.remove(filepath)
        else:
            filepath = os.path.expanduser("~/answer.pickle")
            with open(filepath, 'wb') as f:
                pickle.dump(self.ans, f)

my_output = OutputLogger()
```

# Part 0: Load data to a RDD

The tweets data is stored on AWS S3. We have in total a little over 1 TB of tweets. We provide 10 MB of tweets for your local development. For the testing and grading on the homework server, we will use different data.

## Testing on the homework server
On EdX, we provide three different input sizes to test your program: 10 MB, 1 GB, and 10 GB. For any run, we will only be using one of these four datasets.

For submission and for local testing, make sure to read the path of the file you want to operate with from `./hw2-files.txt`. Otherwise your program will receive no points.

## Local test

For local testing, please create your own `hw2-files.txt` file, which contains a single file path on the local disk, e.g.
`file://<absolute_path_to_current_directory>/hw2-files-10mb.txt`. For final submission, we will create this file on our server for testing with the appropriate file path. If your implementation is correct, you should not worry about which file system (i.e. local file system or HDFS) Spark will read data from.

Now let's see how many lines there are in the input files.

1. Make RDD from the data in the file given by the file path present in `hw2-files.txt`.
2. Mark the RDD to be cached (so in next operation data will be loaded in memory)
3. call the `count` method to print number of lines in all these files

<b>It should print<b>
```
Number of elements: 2150
```

```python
"""
This is a useful cell for debugging.
Use timer_start() and timer_stop() at different parts of your code
for checking the amount of time a segment takes.
"""
from time import time

timer = []
prev_ts = None

def timer_start():
    global prev_ts
    prev_ts = time()

def timer_stop(title):
    timer.append((title, time() - prev_ts))
```

```python
timer_start()

from pyspark import SparkContext

sc = SparkContext()

timer_stop("set up sc")
```

```python
timer_start()

with open('./hw2-files.txt') as f:
    file_path = [w.strip() for w in f.readlines() if w.strip()]

# Your code here
raise NotImplementedError()

my_output.append("num-tweets", count)
print('Number of elements:', count)
timer_stop("read data")
```

# Part 1: Parse JSON strings to JSON objects

Python has built-in support for JSON.

```python
import json

json_example = '''
{
    "id": 1,
    "name": "A green door",
    "price": 12.50,
    "tags": ["home", "green"]
}
'''

json_obj = json.loads(json_example)
json_obj
```

## Broken tweets and irrelevant messages

The data of this assignment may contain broken tweets (invalid JSON strings). So make sure that your code is robust for such cases.

You can filter out such broken tweet by checking if:
* the line is not in json format

In addition, some lines in the input file might not be tweets, but messages that the Twitter server sent to the developer (such as [limit notices](https://dev.twitter.com/streaming/overview/messages-types#limit_notices)). Your program should also ignore these messages.

These messages would not contain the `created_at` field and can be filtered out accordingly.
* Check if json object of the broken tweet has a `created_at` field

*Hint:* [Catch the ValueError](http://stackoverflow.com/questions/11294535/verify-if-a-string-is-json-in-python)

**********************************************************************************

**Tasks**

(1) Parse raw JSON tweets to obtain valid JSON objects.

(2) From all valid tweets, construct a pair RDD of `(user_id, text)`, where `user_id` is the `id_str` data field of the `user` dictionary (read [Tweets](#Tweets) section above), `text` is the `text` data field.

```python
import json

def safe_parse(raw_json):
    """
    Input is a String
    Output is a JSON object if the tweet is valid and None if not valid
    """
    # YOUR CODE HERE
    raise NotImplementedError()

```

```python
"""
# Remember to construct an RDD of (user_id, text) here.
"""

# YOUR CODE HERE
raise NotImplementedError()

```

## Number of unique users

Count the number of different users in all valid tweets

(hint: [the `distinct()` method](https://spark.apache.org/docs/latest/programming-guide.html#transformations) is an easy way to do this, but try to see if there is a faster way to do this).

*******************************************************************************

**It should print**
```
The number of unique users is: 1748
```

```python
timer_start()

# YOUR CODE HERE
raise NotImplementedError()

my_output.append("num-unique-users", users_count)
print('The number of unique users is:', users_count)
timer_stop("Count unique users")
```

# Part 2: Number of posts from each user partition

Load the Pickle file `/twitter/users-partition.pickle`, you will get a dictionary which represents a partition over 452,743 Twitter users, `{user_id: partition_id}`. The users are partitioned into 7 groups. For example, if the dictionary is loaded into a variable named `partition`, the partition ID of the user `59458445` is `partition["59458445"]`. These users are partitioned into 7 groups. The partition ID is an integer between 0-6.

Note that the user partition we provide doesn't cover all users appear in the input data.

## Load the pickle file

For local testing, you can load the pickle file from the local file system, namely

```
proc = subprocess.Popen(["cat", "./users-partition.pickle"],
                        stdout=subprocess.PIPE)
pickle_content = proc.communicate()[0]
```
However, for submission, please keep following code block unchanged, since on the server the pickle file is located on the HDFS.

```python
import subprocess
import pickle

if ON_EMR:
    proc = subprocess.Popen(["/usr/local/hadoop/hadoop-2.7.4/bin/hadoop", "fs", "-cat", "/user/spark/twitter/users-partition.pickle"],
                            stdout=subprocess.PIPE)
    pickle_content = proc.communicate()[0]

else:
    #!wget 'http://mas-dse-open.s3.amazonaws.com/Twitter/users-partition.pickle' -O './users-partition.pickle'
    proc = subprocess.Popen(["cat", "./users-partition.pickle"],
                        stdout=subprocess.PIPE)
    pickle_content = proc.communicate()[0]

partition = pickle.loads(pickle_content)
len(partition)
```

## Tweets per user partition

1. Count the number of posts from each user partition

2. Count the number of posts from group 0, 1, ..., 6, plus the number of posts from users who are not in any partition. Assign users who are not in any partition to the group 7.

3. Put the results of this step into a pair RDD `(group_id, count)` that is sorted by key.

Print the post count using the `print_post_count` function we provided.

**It should print**

```
Group 0 posted 87 tweets
Group 1 posted 242 tweets
Group 2 posted 41 tweets
Group 3 posted 349 tweets
Group 4 posted 101 tweets
Group 5 posted 358 tweets
Group 6 posted 434 tweets
Group 7 posted 521 tweets
```

```python
def print_post_count(counts):
    for group_id, count in counts:
        print('Group %d posted %d tweets' % (group_id, count))
```

```python
timer_start()

# YOUR CODE HERE
raise NotImplementedError()

# Following code adds your solution to `my_output`
assert(type(counts_per_partition) is list and
       len(counts_per_partition) == 8 and
       len(counts_per_partition[0]) == 2)
my_output.append("counts_per_part", counts_per_partition)
print_post_count(counts_per_partition)
timer_stop("Count tweets per user partition")
```

# Part 3:  Tokens that are relatively popular in each user partition

In this step, we are going to find tokens that are relatively popular in each user partition.

We define the number of mentions of a token $t$ in a specific user partition $k$ as the number of users from the user partition $k$ that ever mentioned the token $t$ in their tweets. Note that even if some users might mention a token $t$ multiple times or in multiple tweets, a user will contribute at most 1 to the counter of the token $t$.

Please make sure that the number of mentions of a token is equal to the number of users who mentioned this token but NOT the number of tweets that mentioned this token.

Let $N_t^k$ be the number of mentions of the token $t$ in the user partition $k$. Let $N_t^{all} = \sum_{i=0}^7 N_t^{i}$ be the number of total mentions of the token $t$.

We define the relative popularity of a token $t$ in a user partition $k$ as the log ratio between $N_t^k$ and $N_t^{all}$, i.e.

\begin{equation}
p_t^k = \log \frac{N_t^k}{N_t^{all}}.
\end{equation}

You can compute the relative popularity by calling the function `get_rel_popularity`.

We load a tweet tokenizer for you in the following cells. This Tokenizer object is called `tok`. Don't forget to execute the two cells below.

You can expand the following cell if needed to see the minutae of the Tokenizer.

```python
#!/usr/bin/env python

"""
This code implements a basic, Twitter-aware tokenizer.

A tokenizer is a function that splits a string of text into words. In
Python terms, we map string and unicode objects into lists of unicode
objects.

There is not a single right way to do tokenizing. The best method
depends on the application.  This tokenizer is designed to be flexible
and this easy to adapt to new domains and tasks.  The basic logic is
this:

1. The tuple regex_strings defines a list of regular expression
   strings.

2. The regex_strings strings are put, in order, into a compiled
   regular expression object called word_re.

3. The tokenization is done by word_re.findall(s), where s is the
   user-supplied string, inside the tokenize() method of the class
   Tokenizer.

4. When instantiating Tokenizer objects, there is a single option:
   preserve_case.  By default, it is set to True. If it is set to
   False, then the tokenizer will downcase everything except for
   emoticons.

The __main__ method illustrates by tokenizing a few examples.

I've also included a Tokenizer method tokenize_random_tweet(). If the
twitter library is installed (http://code.google.com/p/python-twitter/)
and Twitter is cooperating, then it should tokenize a random
English-language tweet.

Julaiti Alafate:
  I modified the regex strings to extract URLs in tweets.
"""

__author__ = "Christopher Potts"
__copyright__ = "Copyright 2011, Christopher Potts"
__credits__ = []
__license__ = "Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-nc-sa/3.0/"
__version__ = "1.0"
__maintainer__ = "Christopher Potts"
__email__ = "See the author's website"

######################################################################

import re
from html import entities

######################################################################
# The following strings are components in the regular expression
# that is used for tokenizing. It's important that phone_number
# appears first in the final regex (since it can contain whitespace).
# It also could matter that tags comes after emoticons, due to the
# possibility of having text like
#
#     <:| and some text >:)
#
# Most imporatantly, the final element should always be last, since it
# does a last ditch whitespace-based tokenization of whatever is left.

# This particular element is used in a couple ways, so we define it
# with a name:
emoticon_string = r"""
    (?:
      [<>]?
      [:;=8]                     # eyes
      [\-o\*\']?                 # optional nose
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      |
      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
      [\-o\*\']?                 # optional nose
      [:;=8]                     # eyes
      [<>]?
    )"""

# The components of the tokenizer:
regex_strings = (
    # Phone numbers:
    r"""
    (?:
      (?:            # (international)
        \+?[01]
        [\-\s.]*
      )?
      (?:            # (area code)
        [\(]?
        \d{3}
        [\-\s.\)]*
      )?
      \d{3}          # exchange
      [\-\s.]*
      \d{4}          # base
    )"""
    ,
    # URLs:
    r"""http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"""
    ,
    # Emoticons:
    emoticon_string
    ,
    # HTML tags:
     r"""<[^>]+>"""
    ,
    # Twitter username:
    r"""(?:@[\w_]+)"""
    ,
    # Twitter hashtags:
    r"""(?:\#+[\w_]+[\w\'_\-]*[\w_]+)"""
    ,
    # Remaining word types:
    r"""
    (?:[a-z][a-z'\-_]+[a-z])       # Words with apostrophes or dashes.
    |
    (?:[+\-]?\d+[,/.:-]\d+[+\-]?)  # Numbers, including fractions, decimals.
    |
    (?:[\w_]+)                     # Words without apostrophes or dashes.
    |
    (?:\.(?:\s*\.){1,})            # Ellipsis dots.
    |
    (?:\S)                         # Everything else that isn't whitespace.
    """
    )

######################################################################
# This is the core tokenizing regex:

word_re = re.compile(r"""(%s)""" % "|".join(regex_strings), re.VERBOSE | re.I | re.UNICODE)

# The emoticon string gets its own regex so that we can preserve case for them as needed:
emoticon_re = re.compile(regex_strings[1], re.VERBOSE | re.I | re.UNICODE)

# These are for regularizing HTML entities to Unicode:
html_entity_digit_re = re.compile(r"&#\d+;")
html_entity_alpha_re = re.compile(r"&\w+;")
amp = "&"

######################################################################

class Tokenizer:
    def __init__(self, preserve_case=False):
        self.preserve_case = preserve_case

    def tokenize(self, s):
        """
        Argument: s -- any string or unicode object
        Value: a tokenize list of strings; conatenating this list returns the original string if preserve_case=False
        """
        # Try to ensure unicode:
        try:
            s = str(s)
        except UnicodeDecodeError:
            s = s.encode('string_escape')
            s = str(s)
        # Fix HTML character entitites:
        s = self.__html2unicode(s)
        # Tokenize:
        words = word_re.findall(s)
        # Possible alter the case, but avoid changing emoticons like :D into :d:
        if not self.preserve_case:
            words = map((lambda x : x if emoticon_re.search(x) else x.lower()), words)
        return words

    def tokenize_random_tweet(self):
        """
        If the twitter library is installed and a twitter connection
        can be established, then tokenize a random tweet.
        """
        try:
            import twitter
        except ImportError:
            print("Apologies. The random tweet functionality requires the Python twitter library: http://code.google.com/p/python-twitter/")
        from random import shuffle
        api = twitter.Api()
        tweets = api.GetPublicTimeline()
        if tweets:
            for tweet in tweets:
                if tweet.user.lang == 'en':
                    return self.tokenize(tweet.text)
        else:
            raise Exception("Apologies. I couldn't get Twitter to give me a public English-language tweet. Perhaps try again")

    def __html2unicode(self, s):
        """
        Internal metod that seeks to replace all the HTML entities in
        s with their corresponding unicode characters.
        """
        # First the digits:
        ents = set(html_entity_digit_re.findall(s))
        if len(ents) > 0:
            for ent in ents:
                entnum = ent[2:-1]
                try:
                    entnum = int(entnum)
                    s = s.replace(ent, unichr(entnum))
                except:
                    pass
        # Now the alpha versions:
        ents = set(html_entity_alpha_re.findall(s))
        ents = filter((lambda x : x != amp), ents)
        for ent in ents:
            entname = ent[1:-1]
            try:
                s = s.replace(ent, unichr(entities.name2codepoint[entname]))
            except:
                pass
            s = s.replace(amp, " and ")
        return s

```

```python
from math import log

tok = Tokenizer(preserve_case=False)

def get_rel_popularity(c_k, c_all):
    return log(1.0 * c_k / c_all) / log(2)

def print_tokens(tokens, gid = None):
    group_name = "overall"
    if gid is not None:
        group_name = "group %d" % gid
    print('=' * 5 + ' ' + group_name + ' ' + '=' * 5)
    for t, n in tokens:
        print("%s\t%.4f" % (t, n))
    print
```

## Tokenize tweets

1. Tokenize the tweets using the `tokenize` function that is a method of the `Tokenizer` class that we have instantiated as `tok`.

1. Count the number of mentions for each tokens regardless of specific user group.

1. Call `print_count` function to show how many different tokens we have.

**It should print**
```
Number of tokens: 7677
```

```python
# YOUR CODE HERE
raise NotImplementedError()

my_output.append("num-tokens", num_of_tokens)
print("Number of tokens:", num_of_tokens)
```

## Token popularity

Tokens that are mentioned by too few users are usually not very interesting. So we want to only keep tokens that are mentioned by at least 100 users. Filter out tokens that don't meet this requirement.

Call `print_count` function to show how many different tokens we have after the filtering.

Call `print_tokens` function to show top 20 most frequent tokens.

**It should print**
```
Number of tokens: 46
===== overall =====
:	1046.0000
rt	920.0000
.	767.0000
the	587.0000
trump	560.0000
…	520.0000
to	501.0000
,	497.0000
in	385.0000
a	383.0000
is	382.0000
of	300.0000
!	285.0000
for	275.0000
and	263.0000
on	218.0000
i	216.0000
he	191.0000
that	190.0000
"	181.0000
```

```python
timer_start()

# YOUR CODE HERE
raise NotImplementedError()

my_output.append("num-freq-tokens", num_freq_tokens)
my_output.append("top-20-tokens", top20)
print("Number of tokens:", num_freq_tokens)
print_tokens(top20)
timer_stop("Count overall most popular tokens")
```

## Relative Popularity

For all tokens that are mentioned by at least 100 users, compute their relative popularity in each user group. Then print the top 10 tokens with highest relative popularity in each user group. In case two tokens have same relative popularity, break the tie by printing the alphabetically smaller one.

**Hint:** Let the relative popularity of a token $t$ be $p$. The order of the items will be satisfied by sorting them using (-p, t) as the key.

**It should print**
```
===== group 0 =====
with	-3.6088
cruz	-3.6554
his	-3.6582
amp	-3.8651
on	-3.9608
to	-4.0145
&	-4.0875
https	-4.1699
i	-4.1699
what	-4.1699
===== group 1 =====
sanders	-2.2854
gop	-2.4060
hillary	-2.4330
’	-2.4463
bernie	-2.4835
"	-2.6925
are	-2.7249
this	-2.7633
for	-2.8179
about	-2.8346
===== group 2 =====
with	-4.3458
donald	-4.5146
...	-4.7004
gop	-4.7279
i	-4.9475
on	-4.9608
he	-4.9925
…	-5.1155
https	-5.1699
what	-5.1699
===== group 3 =====
bernie	-1.5945
sanders	-1.6609
hillary	-2.2188
and	-2.5154
"	-2.5930
in	-2.6114
will	-2.6160
https	-2.6674
...	-2.7004
you	-2.7004
===== group 4 =====
what	-3.4330
have	-3.4725
bernie	-3.5380
this	-3.5518
it	-3.6881
?	-3.6912
for	-3.7110
about	-3.7415
hillary	-3.7549
that	-3.7625
===== group 5 =====
what	-1.8007
not	-1.8745
https	-2.0000
his	-2.0144
cruz	-2.0704
it	-2.1031
on	-2.1243
&	-2.1399
amp	-2.1489
;	-2.1592
===== group 6 =====
will	-1.3847
have	-1.4725
!	-1.5850
cruz	-1.6919
trump	-1.7199
https	-1.7549
-	-1.7673
;	-1.7807
be	-1.7952
amp	-1.8144
===== group 7 =====
donald	-1.0740
trump	-1.6535
bernie	-1.7790
sanders	-1.7829
’	-1.8613
of	-1.9069
?	-1.9186
with	-1.9307
the	-1.9588
be	-1.9758
```

```python
timer_start()

# YOUR CODE HERE
raise NotImplementedError()

# Following code add your solution to `my_output`
my_output.append("popular_10_in_each_group", popular_10_in_each_group)
for k in range(8):
    print_tokens(popular_10_in_each_group[k], k)
timer_stop("Print popular tokens in each group")
```

## Important: Write your solutions to disk

Following cell write your solutions to disk which would be read in by the grader for grading.

```python
my_output.write_to_disk()
```

## Optional Ungraded Exercise
The users partition is generated by a machine learning algorithm that tries to group the users by their political preferences. Three of the user groups are showing supports to Bernie Sanders, Ted Cruz, and Donald Trump.

If your program looks okay on the local test data, you can try it on the larger input by submitting your program to the homework server. Observe the output of your program to larger input files, can you guess the partition IDs of the three groups mentioned above based on your output?

```python
# Change the values of the following three items to your guesses
users_support = [
    (-1, "Bernie Sanders"),
    (-1, "Ted Cruz"),
    (-1, "Donald Trump")
]

for gid, candidate in users_support:
    print("Users from group %d are most likely to support %s." % (gid, candidate))
```

Uncomment the lines in the cell below to print out the time taken between `time_start()` and `time_end()` at various stages of your pipeline.

```python
# total_time = 0.0
# for item in timer:
#     print("{}\t{}".format(item[0], item[1]))
#     total_time += item[1]
# print("Total time: {}".format(total_time))
```

```python

```

Big Data Analytics Using Spark/Programming Assignment 2/HW-2.ipynb


##### Copyright 2021 The Cirq Developers

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Quantum circuits on Rigetti devices

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

```
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")
    import cirq
```

    installing cirq...
         |████████████████████████████████| 56 kB 668 kB/s
         |████████████████████████████████| 593 kB 9.5 MB/s
         |████████████████████████████████| 541 kB 16.4 MB/s
         |████████████████████████████████| 48 kB 4.6 MB/s
         |████████████████████████████████| 1.8 MB 47.1 MB/s
         |████████████████████████████████| 53 kB 1.9 MB/s
         |████████████████████████████████| 220 kB 55.7 MB/s
         |████████████████████████████████| 52 kB 1.1 MB/s
         |████████████████████████████████| 97 kB 5.9 MB/s
         |████████████████████████████████| 49 kB 5.6 MB/s
         |████████████████████████████████| 10.1 MB 42.9 MB/s
         |████████████████████████████████| 65 kB 3.2 MB/s
         |████████████████████████████████| 145 kB 70.0 MB/s
         |████████████████████████████████| 38.1 MB 1.3 MB/s
         |████████████████████████████████| 44 kB 139 kB/s
         |████████████████████████████████| 229 kB 66.2 MB/s
         |████████████████████████████████| 243 kB 55.2 MB/s
         |████████████████████████████████| 1.6 MB 31.7 MB/s
         |████████████████████████████████| 109 kB 58.9 MB/s
         |████████████████████████████████| 546 kB 40.0 MB/s
      Building wheel for lark (setup.py) ... done
      Building wheel for retrying (setup.py) ... done
      Building wheel for rpcq (setup.py) ... done
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    markdown 3.3.7 requires importlib-metadata>=4.4; python_version < "3.10", but you have importlib-metadata 3.10.1 which is incompatible.
    google-colab 1.0.0 requires six~=1.15.0, but you have six 1.16.0 which is incompatible.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    installed cirq.

Running this notebook requires the pyQuil QVM and Compiler. If you are running on Google Colab or a Linux Debian machine, you can run the below cell to install them if necessary. If you are on a non-Linux Debian machine, see [these instructions](https://pyquil-docs.rigetti.com/en/stable/start.html#downloading-the-qvm-and-compiler){:.external} for installation.

```
! [ -z "$(which qvm)" ] &&\
  apt update &&\
  apt install jq &&\
  export LATEST_FOREST_SDK_VERSION=$(curl -s https://downloads.rigetti.com/qcs-sdk/versions | jq -r '.versions[0].sdk') &&\
  curl -f https://downloads.rigetti.com/qcs-sdk/forest-sdk-$LATEST_FOREST_SDK_VERSION-linux-deb.tar.bz2 -o $PWD/forest-sdk-$LATEST_FOREST_SDK_VERSION-linux-deb.tar.bz2 &&\
  tar -xf forest-sdk-$LATEST_FOREST_SDK_VERSION-linux-deb.tar.bz2 &&\
  ./forest-sdk-$LATEST_FOREST_SDK_VERSION-linux-deb/forest-sdk-$LATEST_FOREST_SDK_VERSION-linux-deb.run &&\
  quilc --version &&\
  qvm --version
```

    Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
    Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
    Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
    Hit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
    Get:5 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
    Get:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
    Ign:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
    Hit:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release
    Get:9 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]
    Hit:10 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease
    Get:11 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]
    Hit:12 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
    Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,512 kB]
    Get:14 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,799 kB]
    Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,231 kB]
    Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,286 kB]
    Get:18 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,992 kB]
    Get:19 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [1,021 kB]
    Get:20 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 Packages [45.3 kB]
    Fetched 13.2 MB in 5s (2,876 kB/s)
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    45 packages can be upgraded. Run 'apt list --upgradable' to see them.
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following package was automatically installed and is no longer required:
      libnvidia-common-460
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      libjq1 libonig4
    The following NEW packages will be installed:
      jq libjq1 libonig4
    0 upgraded, 3 newly installed, 0 to remove and 45 not upgraded.
    Need to get 276 kB of archives.
    After this operation, 930 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB]
    Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB]
    Fetched 276 kB in 0s (1,212 kB/s)
    Selecting previously unselected package libonig4:amd64.
    (Reading database ... 155632 files and directories currently installed.)
    Preparing to unpack .../libonig4_6.7.0-1_amd64.deb ...
    Unpacking libonig4:amd64 (6.7.0-1) ...
    Selecting previously unselected package libjq1:amd64.
    Preparing to unpack .../libjq1_1.5+dfsg-2_amd64.deb ...
    Unpacking libjq1:amd64 (1.5+dfsg-2) ...
    Selecting previously unselected package jq.
    Preparing to unpack .../jq_1.5+dfsg-2_amd64.deb ...
    Unpacking jq (1.5+dfsg-2) ...
    Setting up libonig4:amd64 (6.7.0-1) ...
    Setting up libjq1:amd64 (1.5+dfsg-2) ...
    Setting up jq (1.5+dfsg-2) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1.3) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 54.1M  100 54.1M    0     0  25.5M      0  0:00:02  0:00:02 --:--:-- 25.5M
    Verifying archive integrity...     0%     7%    14%    22%    29%    37%    44%    51%    59%    66%    74%    81%    89%    96%  100%   MD5 checksums are OK. All good.
    Uncompressing forest-sdk-deb     0%     7%    14%    22%    29%    37%    44%    51%    59%    66%    74%    81%    89%    96%  100%
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    libblas-dev is already the newest version (3.7.1-4ubuntu1).
    libblas-dev set to manually installed.
    liblapack-dev is already the newest version (3.7.1-4ubuntu1).
    The following package was automatically installed and is no longer required:
      libnvidia-common-460
    Use 'apt autoremove' to remove it.
    The following NEW packages will be installed:
      libffi-dev
    0 upgraded, 1 newly installed, 0 to remove and 45 not upgraded.
    Need to get 156 kB of archives.
    After this operation, 362 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libffi-dev amd64 3.2.1-8 [156 kB]
    Fetched 156 kB in 0s (728 kB/s)
    Selecting previously unselected package libffi-dev:amd64.
    (Reading database ... 155649 files and directories currently installed.)
    Preparing to unpack .../libffi-dev_3.2.1-8_amd64.deb ...
    Unpacking libffi-dev:amd64 (3.2.1-8) ...
    Setting up libffi-dev:amd64 (3.2.1-8) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Selecting previously unselected package forest-sdk.
    (Reading database ... 155684 files and directories currently installed.)
    Preparing to unpack forest-sdk_2.23.0.deb ...
    Unpacking forest-sdk (2.23.0) ...
    Setting up forest-sdk (2.23.0) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    1.23.0 [e6c0939]
    1.17.1 [cf3f91f]

Next, run the pyQuil QVM and Compiler if they are not already running on their default ports.

```
import subprocess

subprocess.Popen(["qvm", "--quiet", "-S"])
subprocess.Popen(["quilc", "--quiet", "-R"])
```

    <subprocess.Popen at 0x7fd6121add10>

# Running a Bell state circuit

To demonstrate the basic functionality of the Cirq Rigetti integration, we begin constructing a basic Bell state circuit.

```
bell_circuit = cirq.Circuit()
qubits = cirq.LineQubit.range(2)
bell_circuit.append(cirq.H(qubits[0]))
bell_circuit.append(cirq.CNOT(qubits[0], qubits[1]))
bell_circuit.append(cirq.measure(qubits[0], qubits[1], key='m'))
print(bell_circuit)
```

    0: ───H───@───M('m')───
              │   │
    1: ───────X───M────────

Next, we'll import `RigettiQCSService` and list available quantum processors.

```
from cirq_rigetti import RigettiQCSService

quantum_processors = RigettiQCSService.list_quantum_processors().quantum_processors
processors_list = [quantum_processor.id for quantum_processor in quantum_processors]
print(processors_list)
```

    ['Aspen-11', 'Aspen-M-1']

For now, we'll instantiate the `RigettiQCSService` as a pyQuil [Quantum Virtual Machine](https://docs.rigetti.com/qcs/references/qvm){:.external} based on the topology of one of the available Rigetti quantum processors. At the time of this writing, `Aspen-11` is available.

Note, in addition to the quantum processors listed above, you can also instantiate the `RigettiQCSService` by naming an arbitrary virtual device the pyQuil QVM supports. See the documentation for pyquil [get_qc](https://pyquil-docs.rigetti.com/en/stable/apidocs/pyquil.api.html#pyquil.api.get_qc){:.external} for more information.

```
from cirq_rigetti import circuit_transformers, circuit_sweep_executors, get_rigetti_qcs_service

SERVICE_NAME = processors_list[0]
print(SERVICE_NAME)
service = get_rigetti_qcs_service(SERVICE_NAME, as_qvm=True, noisy=False)
result = service.run(bell_circuit, repetitions=1000)

print(result.histogram(key='m'))
```

    Aspen-11
    Counter({3: 529, 0: 471})

We'll use the built-in `cirq.plot_state_histogram` to visually verify the results of our Bell state.

```
cirq.plot_state_histogram(result.histogram(key='m'))
```

As expected, we see states 0 (ie '00') and 3 (ie '11') as the dominant results.

You may initialize both the `RigettiQCSService` and `RigettiQCSSampler` with execution functions from the `cirq_rigetti.circuit_sweep_executor` module and transformation functions from `cirq_rigetti.circuit_transformations`.

You may invoke these functions with arguments for controlling your circuit execution at a more fine grained level. For instance, you may want add [Pragma statements](https://pyquil-docs.rigetti.com/en/stable/compiler.html#region-specific-compiler-features-through-pragma){:.external} to set the [initial rewiring](https://pyquil-docs.rigetti.com/en/stable/compiler.html#initial-rewiring){:.external} strategy, invoke [active qubit reset](https://arxiv.org/abs/2001.04449){:.external} prior to execution, or explicitly address physical qubits on the quantum computer.

```
from pyquil.quilbase import Reset, Pragma

def hook(program, measurement_id_map):
    program._instructions.insert(0, Reset())
    program._instructions.insert(1, Pragma('INITIAL_REWIRING', freeform_string='GREEDY'))
    print(program)
    return program, measurement_id_map

# assign qubits explicitly to hardware or virtual machine qubits.
qubit_id_map = {
   qubits[0]: 4,
   qubits[1]: 5,
}
executor = circuit_sweep_executors.with_quilc_compilation_and_cirq_parameter_resolution
transformer = circuit_transformers.build(qubit_id_map=qubit_id_map, qubits=qubits, post_transformation_hooks=[hook])
service = get_rigetti_qcs_service(SERVICE_NAME, as_qvm=True, executor=executor, transformer=transformer)
result = service.run(bell_circuit, repetitions=1000)

cirq.plot_state_histogram(result.histogram(key='m'))
```

# Running a parameterized circuit

Of course, you may be running a parameterized circuit and would like to leverage the [Quil compilers](https://pyquil-docs.rigetti.com/en/stable/compiler.html){:.external} support for parametric compilation. This affords a speedup in execution times as the Cirq Rigetti integration will only compile the circuit once for a single parameter sweep.

We start by initializing the `RigettiQCSSampler` and specifying a circuit sweep executor that supports parametric compilation. Note, that this class accepts the same `executor` and `transformer` types as `RigettiQCSService`.

```
from cirq_rigetti import get_rigetti_qcs_sampler

executor = circuit_sweep_executors.with_quilc_parametric_compilation
sampler = get_rigetti_qcs_sampler(SERVICE_NAME, as_qvm=True, executor=executor)
```

Next, we will initialize a parameterized circuit in Cirq along with a set of parameter values.

```
import sympy

qubit = cirq.LineQubit.range(1)[0]

circuit = cirq.Circuit(
    cirq.H(qubit)**sympy.Symbol('t'),
    cirq.measure(qubit, key='m'))
params = cirq.Linspace('t', start=0, stop=4, length=5)

print(circuit)
```

    0: ───H^t───M('m')───

In much the same way that we invoke, `RigettiQCSSerivce.run`, we invoke `RigettiQCSSampler.run_sweep` with our parameters here.

```
import pandas as pd

results = sampler.run_sweep(circuit, params, repetitions=10)
data = {f't={result.params["t"]}': [measurement[0] for measurement in result.measurements['m']] for result in results}

pd.DataFrame(data)
```

      t=0.0
      t=1.0
      t=2.0
      t=3.0
      t=4.0

      0
      0
      0
      0
      1
      0

      1
      0
      1
      0
      0
      0

      2
      0
      0
      0
      1
      0

      3
      0
      0
      0
      0
      0

      4
      0
      0
      0
      0
      0

      5
      0
      0
      0
      1
      0

      6
      0
      1
      0
      0
      0

      7
      0
      1
      0
      1
      0

      8
      0
      0
      0
      1
      0

      9
      0
      0
      0
      0
      0

      <button class="colab-df-convert" onclick="convertToInteractive('df-8b7f6894-aff7-4811-9f82-84dcbd27acc2')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>


# DO NOT USE
I made this back when I thought I would be using squares to determine covid heat, not a random forest algorithm

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

```python
df = pd.read_csv('latimes-place-totals.csv')
```

```python
df.head()
```

      date
      county
      fips
      place
      confirmed_cases
      note
      x
      y

      0
      2020-10-25
      Alameda
      1.0
      Alameda
      456
      NaN
      -122.274444
      37.756111

      1
      2020-10-25
      Alameda
      1.0
      Albany
      63
      NaN
      -122.297778
      37.886944

      2
      2020-10-25
      Alameda
      1.0
      Berkeley
      749
      NaN
      -122.272778
      37.871667

      3
      2020-10-25
      Alameda
      1.0
      Dublin
      320
      NaN
      -121.935833
      37.702222

      4
      2020-10-25
      Alameda
      1.0
      Emeryville
      102
      NaN
      -122.285278
      37.831389

```python
df['place'].nunique()
```

    1245

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 167270 entries, 0 to 167269
    Data columns (total 8 columns):
     #   Column           Non-Null Count   Dtype
    ---  ------           --------------   -----
     0   date             167270 non-null  object
     1   county           167270 non-null  object
     2   fips             165673 non-null  float64
     3   place            167270 non-null  object
     4   confirmed_cases  167270 non-null  int64
     5   note             5176 non-null    object
     6   x                164796 non-null  float64
     7   y                164796 non-null  float64
    dtypes: float64(3), int64(1), object(4)
    memory usage: 10.2+ MB

```python
df.place.unique()
```

    array(['Alameda', 'Albany', 'Berkeley', ..., 'Pico', 'San Ysidro',
           'Santa Clarita and Stevenson Ranch'], dtype=object)

```python
#df_test = df.groupby(by = 'place')
df_test = pd.DataFrame({'count' : df.groupby( ['place', 'date'] ).size()}).reset_index()
```

```python
df_test.head()
```

      place
      date
      count

      0
      90755: Long Beach
      2020-08-17
      1

      1
      90802: Long Beach
      2020-04-29
      1

      2
      90802: Long Beach
      2020-04-30
      1

      3
      90802: Long Beach
      2020-05-01
      1

      4
      90802: Long Beach
      2020-05-02
      1

```python
df_most_recent = df.loc[df['date'] == '2020-10-25']
df_before_most_recent =
```

```python
df_recent = df.groupby(by = 'date')
df_recent = df.loc[df['date'] == '2020-10-25']
df.loc[df['a'] > 10, ['a','c']]
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 683 entries, 0 to 682
    Data columns (total 8 columns):
     #   Column           Non-Null Count  Dtype
    ---  ------           --------------  -----
     0   date             683 non-null    object
     1   county           683 non-null    object
     2   fips             672 non-null    float64
     3   place            683 non-null    object
     4   confirmed_cases  683 non-null    int64
     5   note             1 non-null      object
     6   x                677 non-null    float64
     7   y                677 non-null    float64
    dtypes: float64(3), int64(1), object(4)
    memory usage: 48.0+ KB

```python
df_x_y = df[['x', 'y']][:40]
plt.figure(figsize=(6, 6));
plt.scatter(df_x_y['x'], df_x_y['y'])
```

Dataframe 1 columns: Covid info
- Index: (Lat1, Lon1, Lat2, Lon2)
- Latitude 1 column
- Longitude 1 column
- Latitude 2 column
- Longitude 2 column
- number of cases in that area
    - if there's a dot, count that
    - if there isn't a dot, leave blank (NaN)
- Covid 'heat'
    - if the number of cases column is NOT blank, rate on scale from 1-10
    - if the number of cases column is blank, take average of the 8 around it
        - TARGET BLANK BOX: (Lat1, Lon1, Lat1 - .1, Lon1 + .1)
        - top left (Lat1 + .1, Lon1 - .1, Lat1, Lon1)
        - middle left (Lat1, Lon1 - .1, Lat1 - .1, Lon1)
        - bottom left (Lat1 - .1, Lon1 - .1, Lat1 - .2, Lon1)
        - bottom middle
        - bottom right
        - middle right
        - top right
        - top middle
*
PLOT AND SEE WHAT HAPPENS
*

Dataframe 2 columns: GS info
- index: GS number (coordinates?)
- coordinates (CANNOT BE INDEX)
- Gas station (GS) names
- GS latitude
- GS Longitude
- heat for this GS based on region
    - From DF1, for DF1 index = DF2 coordinates, get DF1 heat column value
*
PLOT: scatter plot of all gas stations (x = longitude, y = latitude, hue = heat), compare with above
*

Dataframe 3: Error analysis
- LOCATION INPUT: current coordinates, desired coordinates
- RUSH INPUT: Are you in a rush?
-
- Only look at DF2 columns that are within this region
- Find y = mx+b Line for these inputs

Columns:
- Index: GS number (from DF2)
- Heat (from DF2)
- Distance from point to Line (total Least squares)
- Error metric:
    - Weight the heat and distance columns:
        - Do some sort of data science-y thing that determines the optimal weights
        - Questions for weight optimization:
            - How concerned are you about covid?
                - Not concerned
                - Mildly concerned
                - Very concerned
            - Are you in a rush?
                - Yes!! I am rushing asap!!!
                - Yes
                - No
        - Default to "very concerned" unless they are in a big rush (puts a higher weight on the heat of the gas station)
        - If in a big rush, put higher weight (slightly) on the distance RMSE
    - Overall goal: find the gas station which is the minimum of this error metric

# DO KNN FOR GAS STATION CLASSIFICATION!!!

```python
# Python program to find the distance between
# a given point and a given line in 2 D.

# from https://www.geeksforgeeks.org/perpendicular-distance-between-a-point-and-a-line-in-2-d/

import math

# Function to find distance given ax + by + c = 0
# if we have the equation y =
def shortest_distance(x, y, a, b, c):
    d = abs((a * x + b * y + c)) / (math.sqrt(a * a + b * b))
    return d

def get_a_b_c(x1, y1, x2, y2):
    a = -(y2 - y1)
    b = (x2 - x1)
    c = (a * x1) + (b * y1)
    return a, b, c

a = get_a_b_c(2, -1, 3, -3)[0]
b = get_a_b_c(2, -1, 3, -3)[1]
c = get_a_b_c(2, -1, 3, -3)[2]

x = 6
y = 8
shortest_distance(x, y, a, b, c)
```

    10.285912696499032

```python
## Visualize what's above

x1 = 2
y1 = -1
x2 = 3
y2 = -3

def line_eq(x1, y1, x2, y2):
    slope = (y1 - y2) / (x1 - x2)
    b = y1 - (slope * x1)
    return (slope, b)

x_graph = np.linspace(x1, x2)
y_graph = (line_eq(x1, y1, x2, y2)[0] * x_graph) + line_eq(x1, y1, x2, y2)[1]

plt.plot(x_graph, y_graph, '-r')
plt.title('Example')
```

```python
from sympy.solvers import solve, linsolve
from sympy import symbols
```

```python
# for red line above:
x1 = 2
y1 = -1
x2 = 3
y2 = -3
slope = line_eq(x1, y1, x2, y2)[0]
b = line_eq(x1, y1, x2, y2)[1]

#given a point with coordinates (x_a, y_a), find equation of perpendicular line
x_a = 2.2
y_a = 2.5
# the slope will be the negative inverse of the line it's perpendicular to
slope_a = -1/slope
# find b by plugging in the coordinates of our point
b_a = y_a - (slope_a * x_a)

print('slope: ', slope)
print('b: ', b)
print('slope_a: ', slope_a)
print('b_a: ', b_a)

```

    slope:  -2.0
    b:  3.0
    slope_a:  0.5
    b_a:  1.4

```python
# Brute force to find point of intersection (poi) between these lines
x_poi = (b_a - b) / (slope - slope_a)
y_poi = (slope * x_poi) + b
print((x_poi, y_poi))
```

    (0.64, 1.72)

```python
print((x1, x2))
print((x_poi, x_a))
```

    (2, 3)
    (0.64, 2.2)

```python
x_z = np.linspace(0, 2)
y_z = (slope * x_z) + b

x_z2 = np.linspace(x_poi, x_a)
y_z2 = (slope_a * x_z2) + b_a

fig = plt.figure(figsize = (10,10))
ax1 = fig.add_subplot(111)

ax1.scatter(x_z, y_z, s=10, c='b', marker="s", label='first')
ax1.scatter(x_z2, y_z2, s=10, c='r', marker="o", label='second')
plt.legend(loc='upper left');
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.show()
```

```python
x_graph = np.linspace(x_a, x_poi)
y_graph = (slope_a * x_graph) + b_a

plt.plot(x_graph, y_graph, '-r')
plt.title('Example')
```

```python
x_graph_2 = np.linspace(x1, x2)
y_graph_2 = (line_eq(x_a, y_a, x_b, y_b)[0] * x_graph_2) + line_eq(x_a, y_a, x_b, y_b)[1]
plt.plot(x_graph, y_graph, '-r')
plt.title('Example')
```

```python
x_graph = np.linspace(x1, x2)
y_graph = (line_eq(x1, y1, x2, y2)[0] * x_graph) + line_eq(x1, y1, x2, y2)[1]

x_graph_2 = np.linspace(x_a, x_poi)
y_graph_2 = (slope_a * x_graph) + b_a

#x_graph_2 = np.linspace(x1, x2)
#y_graph_2 = (line_eq(x_a, y_a, x_b, y_b)[0] * x_graph_2) + line_eq(x_a, y_a, x_b, y_b)[1]

fig = plt.figure()
ax1 = fig.add_subplot(111)

ax1.scatter(x_graph, y_graph, s=10, c='b', marker="s", label='first')
ax1.scatter(x_graph_2, y_graph_2, s=10, c='r', marker="o", label='second')
plt.legend(loc='upper left');
plt.show()
```

```python
line_eq(x1, y1, x2, y2)[0]
```

    -2.0

```python
df.head()
```

      date
      county
      fips
      place
      confirmed_cases
      note
      x
      y

      0
      2020-10-25
      Alameda
      1.0
      Alameda
      456
      NaN
      -122.274444
      37.756111

      1
      2020-10-25
      Alameda
      1.0
      Albany
      63
      NaN
      -122.297778
      37.886944

      2
      2020-10-25
      Alameda
      1.0
      Berkeley
      749
      NaN
      -122.272778
      37.871667

      3
      2020-10-25
      Alameda
      1.0
      Dublin
      320
      NaN
      -121.935833
      37.702222

      4
      2020-10-25
      Alameda
      1.0
      Emeryville
      102
      NaN
      -122.285278
      37.831389

```python

```

```python

```

```python

```

```python

```

```python

```

```python
#find the point at which these two equations intersect: use sympy

x, y, z = symbols('x, y, z')
#linsolve([slope*x + y + z - 1, x + y + 2*z - 3 ], (x, y, z))
coords = linsolve([y - (slope*x + b), y - (slope_a*x + b_a)], (x, y))

x_b = float(coords.args[0][0])
y_b = float(coords.args[0][1])

print(x_b, y_b)
```


# Jupyter Book
> Documentación usando jupyter notebook.

- toc: true
- badges: false
- comments: true
- categories: [jupyter]
- image: images/jb.png

# Introducción

[Jupyter Book](https://jupyterbook.org/intro.html) es un proyecto de código abierto para crear libros y documentos mediante Jupyter Notebooks y/o Markdown.

Algunas características importantes del uso de Jupyter Book:

* contenido con calidad de publicación que incluya figuras, símbolos matemáticos, citas y referencias cruzadas!
* escribir contenido como Jupyter Notebooks, markdown o reStructuredText
* Agregue interactividad a su libro, por ejemplo, alternar la visibilidad de las celdas, conectarse con un servicio en línea como **Binder** e incluir resultados interactivos (por ejemplo, figuras y widgets).
* generar una variedad de resultados, incluidos sitios web (HTML, CSS, JS), markdown y PDF.
* una interfaz de línea de comandos para crear libros rápidamente, por ejemplo, `jupyter-book build mybook`

En esta sesión, se muestra un ejemplo de cómo crear un Jupyter Book desde cero y algunas de las características clave que ofrece Jupyter Book.

> **Nota**: Puede encontrar los códigos de este ejemplo en el siguiente [repositorio](https://gitlab.com/FAAM/jb_example). Por otro lado, puede revisar el siguente [link](https://faam.gitlab.io/jb_example/jb_example/index.html) para ver la compilación con [GitLab CI/CD](https://docs.gitlab.com/ee/ci/).

# Primeros pasos

## Instalación

Para instalar **Jupyter Book**, necesitará usar la línea de comando. Si ha instalado Anaconda, puede usar:

```
conda install -c conda-forge jupyter-book
```

De lo contrario, puede instalar con pip:

```
pip install jupyter-book
```

## Crear una estructura de libro

Jupyter Book viene con una herramienta que le permite crear y construir libros rápidamente. Para crear el esqueleto del libro, escriba lo siguiente en la línea de comando:

```
jupyter-book create jupiter
```

> **Nota**: Aquí llamamos al libro `jupiter`, pero puedes elegir llamar a tu libro como quieras.

Ahora tendrás un nuevo directorio llamado `jupiter` (o como quieras llamar a tu libro), con el siguiente contenido:

```
jupiter
  ├── _config.yml
  ├── _toc.yml
  ├── content.md
  ├── intro.md
  ├── markdown.md
  ├── notebooks.ipynb
  ├── references.bib
  └── requirements.txt
```

en donde:
* `_config.yml`: archivo que contiene las configuraciones del proyecto.
* `_toc.yml`: archivo que ordena los capítulos del libro.
* `content.md`: archivo genérico `.md`.
* `intro.md`: archivo genérico `.md`.
* `markdown.md`: archivo genérico `.md`.
* `notebooks.ipynb`: archivo genérico `.ipynb`.
* `references.bib`: archivo para añadir las referencias.
* `requirements.txt`: archivo que contiene las dependencias  python) del proyecto.

## Re-estructuración del directorio

Jupyter Book admite varios tipos de archivos:

* Markdown (.md)
* notebooks (.ipynb)
* etc.

Como Markdown y Jupyter Notebooks probablemente serán los tipos de archivo más comunes que usará, se mostrará un ejemplo de ello.

Lo primero será eliminar los archivos de inicio en el directorio:

* content.md
* intro.md
* markdown.md
* notebooks.ipynb

Así que ejecutamos por línea de comando:
```
rm content.md intro.md markdown.md notebooks.ipynb
```

Por otro lado, nuestro proyecto estaré conformado por tres archivos:

* index.md
* Introduction.md
* great_red_spot.ipynb

Luego, debemos indicar cómo serán mostrados estos documentos en el archivo `_toc.yml`. La estructura será la siguiente:

```
format: jb-book
root: index
chapters:
- file: Introduction
- file: great_red_spot
```

En este caso, `root: index` corresponde al primer archivo que se visualiza en el jupyter-book. Dentro del archivo `index.md` escribiremos:

# Home

jupyter book example

## Contenidos

```{tableofcontents}
```

## Agregar un archivo Markdown

Se comienza por agregar un archivo de markdown. Con algún editor a elección (por ejemplo, jupyter notebook o jupyterlab) se crea un nuevo archivo markdown llamado `Introduction.md`.

Se usa este archivo como demostración de algunos de los principales tipos de contenido que puede agregar en Jupyter-Book.

### Texto

Se agrega un texto de Markdown simple a nuestro archivo. Si no está familiarizado con la sintaxis de markdown, consulte [Markdown Cheat Sheet](https://www.markdownguide.org/cheat-sheet/). Puede copiar y pegar el siguiente contenido directamente en su archivo `Introduction.md`.

```
# Jupiter Book

This book contains information about the planet **Jupiter** - the fifth planet from the sun and the largest planet in the solar system!
```

### Figuras

Puedes incluir figuras en tu Jupyter Book usando la siguiente sintaxis:

```{figure} my_image.png
---
height: 150px
name: my-image
---
Here is my image's caption!
```

Si bien la imagen puede estar contenida y referenciada desde el directorio raíz, también se puede incluir imágenes a través de URL. Incluyamos una imagen del planeta Júpiter en nuestro archivo `Introduction.md` usando lo siguiente:

```{figure} https://solarsystem.nasa.gov/system/resources/detail_files/2486_stsci-h-p1936a_1800.jpg
---
height: 300px
name: jupiter-figure
---
The beautiful planet Jupiter!
```

La razón por la que le damos un "nombre" a nuestra imagen es para que podamos hacer referencia a ella fácilmente con la sintaxis:

```
{numref}`jupiter-figure`
```

Se agregará una oración que incluya esta referencia. El archivo completo ahora debería verse así:

# Jupiter Book

This book contains information about the planet Jupiter - the fifth planet from the sun and the largest planet in the solar system! {numref}`jupiter-figure` below shows an image of Jupiter captured by the Hubble Space Telescope on June 27, 2019.

```{figure} https://solarsystem.nasa.gov/system/resources/detail_files/2486_stsci-h-p1936a_1800.jpg
---
height: 300px
name: jupiter-figure
---
The beautiful planet Jupiter! Source: [NASA](https://solarsystem.nasa.gov/resources/2486/hubbles-new-portrait-of-jupiter/?category=planets_jupiter).
```

En este punto, probablemente se debería crear nuetro libro para asegurarnos de que tenga el aspecto esperado. Para hacer eso, primero necesitamos modificar nuestro archivo `_toc.yml`. Este archivo contiene la tabla de contenido de nuestro libro. Abra ese archivo ahora y elimine todo lo que hay allí. Luego, simplemente agregue lo siguiente:

```
- file: introduction
```

Ahora podemos construir nuestro libro desde la línea de comandos asegurándonos de que estamos en el directorio raíz de nuestro libro y luego usando:

```
jupyter-book build .
```

Una vez finalizada la compilación, tendrá un nuevo subdirectorio llamado`_build/html/` en la raíz de su libro, navegue hasta esa ubicación y abra `_build/html/index.html`. Debería verse algo como esto:

### Equaciones matemáticas

Jupyter Book usa [MathJax](https://www.mathjax.org/) para componer matemáticas, lo que le permite agregar matemáticas de estilo LaTeX a su libro. Puede agregar matemáticas en línea, bloques matemáticos y ecuaciones numeradas a su libro Jupyter. Sigamos adelante y creemos un nuevo encabezado en nuestro archivo Introduction.md que incluye algunas matemáticas.

Las matemáticas en línea se pueden definir usando `$` de la siguiente manera:

Jupiter has a mass of:  $m_{j} \approx 1.9 \times 10^{27} kg$

Jupiter has a mass of:  $m_{j} \approx 1.9 \times 10^{27} kg$

Los bloques matemáticos se pueden definir usando la notación `$$`:

$$m_{j} \approx 1.9 \times 10^{27} kg$$

$$m_{j} \approx 1.9 \times 10^{27} kg$$

> **Nota**: Si lo prefiere, los bloques matemáticos también se pueden definir con `\begin{equation}` en lugar de `$$`.

Las ecuaciones numeradas se pueden definir así (este es el estilo que te recomiendo que uses con más frecuencia):

```{math}
:label: my_label
m_{j} \approx 1.9 \times 10^{27} kg
```

Agreguemos más contenido a nuestro libro. Copie y agregue el siguiente texto a su archivo `Introduction.md`:

## The Mass of Jupiter

We can estimate the mass of Jupiter from the period and size of an object orbiting it. For example, we can use Jupiter's moon Callisto to estimate it's mass.

Callisto's period: $p_{c}=16.7 days$

Callisto's orbit radius: $r_{c}=1,900,000 km$

Now, using [Kepler's Law](https://solarsystem.nasa.gov/resources/310/orbits-and-keplers-laws/) we can work out the mass of Jupiter.

```{math}
:label: eq1
m_{j} \approx \frac{r_{c}}{p_{c}} \times 7.9 \times 10^{10}
```

```{math}
:label: eq2
m_{j} \approx 1.9 \times 10^{27} kg
```

A continuación, puede reconstruir su libro (`jupyter-book build .`) y abrir `_build/html/index.html` para asegurarse de que todo se esté procesando como se esperaba.

### Controlando el diseño de la página
Hay varias formas diferentes de controlar el diseño de las páginas de su Jupyter Book. El cambio de diseño que utilizo con más frecuencia es agregar contenido a un margen en la página. Puede agregar un margen usando la siguiente directiva:

```{margin} An optional title
Some margin content.
```

Agreguemos algo de contenido marginal al libro:

```{margin} Did you know?
Jupiter is 11.0x larger than Earth!
```

### Advertencias

Hay todo tipo de [advertencias](https://jupyterbook.org/content/content-blocks.html) diferentes que puede usar en Jupyter Book que se enumeran aquí en la documentación de Jupyter Book. Las advertencias se crean con la sintaxis:

```{note}
I am a useful note!
```

No dude en agregar la siguiente advertencia a `Introduction.md`:

```{hint}
NASA provides a lot more information about the physical characteristics of Jupiter [here](https://solarsystem.nasa.gov/planets/jupiter/by-the-numbers/).
```

### Citas y bibliografía
El último contenido corresponde a referencias y una bibliografía. Puede agregar citas de cualquier trabajo almacenado en el archivo Bibtex `Reference.bib` que se encuentra en el directorio raíz de su libro.

Para incluir una cita en su libro, agregue una entrada bibtex a `references.bib`, por ejemplo:

```
@article{mayor1995jupiter,
    title={A Jupiter-mass companion to a solar-type star},
    author={Mayor, Michel and Queloz, Didier},
    journal={Nature},
    volume={378},
    number={6555},
    pages={355--359},
    year={1995},
    publisher={Nature Publishing Group}
}

@article{guillot1999interiors,
    title={Interiors of giant planets inside and outside the solar system},
    author={Guillot, Tristan},
    journal={Science},
    volume={286},
    number={5437},
    pages={72--77},
    year={1999},
    publisher={American Association for the Advancement of Science}
}
```

> **Nota:** Consulte la [documentación de BibTex](http://www.bibtex.org/Using/) para obtener información sobre el estilo de referencia de BibTex. Google Scholar facilita la exportación de un formato de cita bibtex.

A continuación, puede hacer referencia al trabajo en su libro utilizando la siguiente directiva:

```
{cite}`mayor1995jupiter`
```

O para múltiples citas:

```
{cite}`mayor1995jupiter,guillot1999interiors`
```

Luego puede crear una bibliografía a partir de `reference.bib` usando:

```{bibliography} references.bib
```

Por ejemplo, intente agregar esto a su archivo `Introduction.md`:

There might even be more planets out there with a similar mass to Jupiter {cite}`mayor1995jupiter,guillot1999interiors`!

## Bibliography

```{bibliography} references.bib
```

Su archivo final `Introduction.md` debería verse así:

# Jupiter Book

This book contains information about the planet Jupiter - the fifth planet from the sun and the largest planet in the solar system! {numref}`jupiter-figure` below shows an image of Jupiter captured by the Hubble Space Telescope on June 27, 2019.

```{figure} https://solarsystem.nasa.gov/system/resources/detail_files/2486_stsci-h-p1936a_1800.jpg
---
height: 300px
name: jupiter-figure
---
The beautiful planet Jupiter! Source: [NASA](https://solarsystem.nasa.gov/resources/2486/hubbles-new-portrait-of-jupiter/?category=planets_jupiter).
```

## The Mass of Jupiter

We can estimate the mass of Jupiter from the period and size of an object orbiting it. For example, we can use Jupiter's moon Callisto to estimate it's mass.

Callisto's period: $p_{c}=16.7 days$

Callisto's orbit radius: $r_{c}=1,900,000 km$

Now, using [Kepler's Law](https://solarsystem.nasa.gov/resources/310/orbits-and-keplers-laws/) we can work out the mass of Jupiter.

```{math}
:label: eq1
m_{j} \approx \frac{r_{c}}{p_{c}} \times 7.9 \times 10^{10}
```

```{math}
:label: eq2
m_{j} \approx 1.9 \times 10^{27} kg
```

```{margin} Did you know?
Jupiter is 11.0x larger than Earth!
```

```{hint}
NASA provides a lot more information about the physical characteristics of Jupiter [here](https://solarsystem.nasa.gov/planets/jupiter/by-the-numbers/).
```

There might even be more planets out there with a similar mass to Jupiter {cite}`mayor1995jupiter,guillot1999interiors`!

## Bibliography

```{bibliography} references.bib
```

Y debería renderizarse así:

## Agregar un archivo de contenido de Jupyter Notebook

Todos los flujos de trabajo de formato y estilo que vimos en markdown también se aplican a un Jupyter Notebook; simplemente agréguelos a una celda de markdown y listo.

Comencemos con lo siguiente:

* Cree un nuevo notebook llamado `great_red_spot.ipynb`;
* Agregue este archivo a su `_toc.yml`;
* Agregue una celda de markdown con el siguiente contenido:

# The Great Red Spot

Jupiter’s iconic Great Red Spot (GRS) is actually an enormous storm that is bigger than Earth that has raged for hundreds of years! {numref}`great-red-spot` below shows an image of Jupiter captured by the Hubble Space Telescope on June 27, 2019.

```{figure} https://solarsystem.nasa.gov/system/resources/detail_files/626_PIA21775.jpg
---
height: 300px
name: great-red-spot
---
Jupiter's Great Red Spot! Source: [NASA](https://solarsystem.nasa.gov/resources/626/jupiters-great-red-spot-in-true-color/?category=planets_jupiter).
```

Jupiter's GRS has been observed to be shrinking for about the last century and a half! [Here](https://github.com/UBC-DSCI/jupyterdays/tree/master/jupyterdays/sessions/beuzen/data) is some data of the length of the GRS spanning the last ~150 years which we can use to investigate this phenomenon.

¡Ahora intente construir su libro (`jupyter-book build .`) para asegurarse de que todo se vea bien! Usando la barra de contenido del lado izquierdo, navegue a la nueva página “The Great Red Spot”, que debería verse así:

¡Ok genial! Ahora importemos los datos a los que hicimos referencia para que podamos crear algunos gráficos.

Cree una nueva celda de código debajo de la celda de rebaja actual y agregue el siguiente código para leer en nuestro conjunto de datos de GRS como un marco de datos de Pandas.

```python
import pandas as pd
pd.options.plotting.backend = "plotly"

url = "https://raw.githubusercontent.com/UBC-DSCI/jupyterdays/master/jupyterdays/sessions/beuzen/data/GRS_data.csv"
df = pd.read_csv(url)
df['Year'] = df['Year'].astype(int)
df.head()
```

> **Nota**: Estamos imprimiendo la salida en la pantalla con el uso de `df.head()` y esto se mostrará en nuestro Jupyter Book renderizado.

Si reconstruye su libro (`jupyter-book build .`) en este punto, verá algo como lo siguiente:

Ahora, podemos usar estos datos para crear algunos gráficos.

Las tramas en su Jupyter Book pueden ser estáticas (por ejemplo, `matplotlib`, `seaborn`) o interactivas (por ejemplo, `altair`, `plotly`, `bokeh`).
Para este tutorial, crearemos algunos gráficos de ejemplo usando `Plotly` (a través del backend de Pandas).

Primero creemos un diagrama de dispersión simple de nuestros datos. Cree una nueva celda de código en su cuaderno y agregue el siguiente código:

```python
import plotly.io as pio
pio.renderers.default = "notebook"
fig = df.plot.scatter(x="Year", y="GRS Length", color="Recorder",
                      range_x=[1870, 2030], range_y=[10, 40],
                      width=650, height=400)
fig.update_layout(title={'text': "Great Red Spot Size", 'x':0.5, 'y':0.92})
fig.update_traces(marker=dict(size=7))
```

Ya que estamos en eso, creemos también una trama animada. Cree otra celda de código nueva y agregue el siguiente código:
```python
fig = df.plot.scatter(x="Year", y="GRS Length",
                      animation_frame="Year",
                      range_x=[1870, 2030], range_y=[10, 40],
                      width=600, height=520)
fig.update_layout(title={'text': "Great Red Spot Size Animation", 'x':0.5, 'y':0.94})
fig.layout.updatemenus[0].buttons[0].args[1]["frame"]["duration"] = 200
fig.update_traces(marker=dict(size=10))
```

> **Nota:** Plotly tiene [diferentes renderizadores](https://plotly.com/python/renderers/) disponibles para generar gráficos. Es posible que deba experimentar con renderizadores para obtener el resultado que desea en su Jupyter Book. He descubierto que `pio.renderers.default = "notebook"` funciona con la versión actual de Jupyter Book.

¡Ahora, reconstruyamos nuestro libro y echemos un vistazo!

Es posible que desee ocultar parte del código en su libro, ¡no hay problema! Eso también se hace fácilmente con Jupyter Book.

El que nos interesa aquí es ocultar la entrada de código. Podemos hacerlo fácilmente agregando la etiqueta `hide-input` a la celda que deseamos ocultar. Hay varias formas de agregar etiquetas a la celda en Jupyter Notebooks. En Jupyter Lab, haga clic en el icono de engranaje en la barra lateral izquierda y luego agregue la etiqueta deseada como se muestra a continuación:

Continúe y agregue las etiquetas `hide-input` a ambas celdas de trazado en su archivo `great_red_spot.ipynb`. Cuando reconstruyas el libro, verás que la entrada del código está oculta (pero se puede alternar con el ícono `+`):

> **Nota**: También puede almacenar el contenido de la libreta como valores, gráficos o marcos de datos en variables que se pueden utilizar en toda su libreta mediante la herramienta `glue`.

# Referencias

* [Jupyter-Book - Documentation](https://jupyterbook.org/intro.html)
* [Tutorial - Jupyter Book](https://ubc-dsci.github.io/jupyterdays/sessions/beuzen/jupyter_book_tutorial.html)


## Install TensorTrade

```python
!python3 -m pip install -e .. -U
```

    Obtaining file:///Users/adam/Desktop/Capfolio/tensortrade
    Requirement already satisfied, skipping upgrade: numpy==1.16.4 in /usr/local/lib/python3.7/site-packages (from tensortrade==0.2.0b0) (1.16.4)
    Requirement already satisfied, skipping upgrade: pandas==0.25.0 in /usr/local/lib/python3.7/site-packages (from tensortrade==0.2.0b0) (0.25.0)
    Requirement already satisfied, skipping upgrade: gym==0.14.0 in /usr/local/lib/python3.7/site-packages (from tensortrade==0.2.0b0) (0.14.0)
    Requirement already satisfied, skipping upgrade: pyyaml==5.1.2 in /usr/local/lib/python3.7/site-packages (from tensortrade==0.2.0b0) (5.1.2)
    Requirement already satisfied, skipping upgrade: sympy==1.4 in /usr/local/lib/python3.7/site-packages (from tensortrade==0.2.0b0) (1.4)
    Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.7/site-packages (from pandas==0.25.0->tensortrade==0.2.0b0) (2019.2)
    Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /usr/local/Cellar/jupyterlab/1.0.0_5/libexec/vendor/lib/python3.7/site-packages (from pandas==0.25.0->tensortrade==0.2.0b0) (2.7.3)
    Requirement already satisfied, skipping upgrade: pyglet<=1.3.2,>=1.2.0 in /usr/local/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.2.0b0) (1.3.2)
    Requirement already satisfied, skipping upgrade: cloudpickle~=1.2.0 in /usr/local/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.2.0b0) (1.2.1)
    Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.2.0b0) (1.4.1)
    Requirement already satisfied, skipping upgrade: six in /usr/local/Cellar/jupyterlab/1.0.0_5/libexec/vendor/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.2.0b0) (1.11.0)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in /usr/local/lib/python3.7/site-packages (from sympy==1.4->tensortrade==0.2.0b0) (1.1.0)
    Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.7/site-packages (from pyglet<=1.3.2,>=1.2.0->gym==0.14.0->tensortrade==0.2.0b0) (0.17.1)
    Installing collected packages: tensortrade
      Attempting uninstall: tensortrade
        Found existing installation: TensorTrade 0.2.0b0
        Uninstalling TensorTrade-0.2.0b0:
          Successfully uninstalled TensorTrade-0.2.0b0
      Running setup.py develop for tensortrade
    Successfully installed tensortrade

## Setup Data Fetching

```python
import ssl
import pandas as pd

ssl._create_default_https_context = ssl._create_unverified_context # Only used if pandas gives a SSLError

def fetch(exchange_name, symbol, timeframe):
    url = "https://www.cryptodatadownload.com/cdd/"
    filename = "{}_{}USD_{}.csv".format(exchange_name, symbol, timeframe)
    volume_column = "Volume {}".format(symbol)
    new_volume_column = "Volume_{}".format(symbol)

    df = pd.read_csv(url + filename, skiprows=1)
    df = df[::-1]
    df = df.drop(["Symbol"], axis=1)
    df = df.rename({"Volume USD": "volume", volume_column: new_volume_column}, axis=1)
    df = df.set_index("Date")
    df.columns = [symbol + ":" + name.lower() for name in df.columns]

    return df
```

## Setup Trading Environment

```python
from tensortrade.exchanges import Exchange
from tensortrade.exchanges.services.execution.simulated import execute_order
from tensortrade.data import Stream, DataFeed, Module
from tensortrade.instruments import USD, BTC, ETH, LTC
from tensortrade.wallets import Wallet, Portfolio
from tensortrade.environments import TradingEnvironment

coinbase_data = pd.concat([
    fetch("Coinbase", "BTC", "1h"),
    fetch("Coinbase", "ETH", "1h")
], axis=1)

coinbase = Exchange("coinbase", service=execute_order)(
    Stream("USD-BTC", list(coinbase_data['BTC:close'])),
    Stream("USD-ETH", list(coinbase_data['ETH:close']))
)

coinbase_btc = coinbase_data.loc[:, [name.startswith("BTC") for name in coinbase_data.columns]]

with Module("coinbase") as coinbase_ns:
    nodes = [Stream(name, list(coinbase_data[name])) for name in coinbase_data.columns]

feed = DataFeed([coinbase_ns])

portfolio = Portfolio(USD, [
    Wallet(coinbase, 10000 * USD),
    Wallet(coinbase, 10 * BTC),
    Wallet(coinbase, 5 * ETH),
])

env = TradingEnvironment(
    feed=feed,
    portfolio=portfolio,
    action_scheme='managed-risk',
    reward_scheme='risk-adjusted',
    window_size=20
)
```

## Example Data Feed Observation

```python
env.feed.next()
```

    {'coinbase:/USD:/total': 10000,
     'coinbase:/BTC:high': 2513.38,
     'coinbase:/BTC:/free': 10,
     'coinbase:/BTC:low': 2495.12,
     'coinbase:/USD:/free': 10000,
     'coinbase:/BTC:/locked': 0,
     'coinbase:/BTC:close': 2509.17,
     'coinbase:/BTC:volume_btc': 114.6,
     'coinbase:/BTC:/total': 10,
     'coinbase:/ETH:close': 275.01,
     'coinbase:/USD-BTC': 2509.17,
     'coinbase:/BTC:/worth': 25091.7,
     'coinbase:/BTC:volume': 287000.32,
     'coinbase:/ETH:open': 279.98,
     'coinbase:/USD-ETH': 275.01,
     'coinbase:/BTC:open': 2505.56,
     'coinbase:/ETH:high': 279.99,
     'coinbase:/ETH:low': 272.1,
     'coinbase:/ETH:/locked': 0,
     'coinbase:/ETH:/free': 5,
     'coinbase:/ETH:volume_eth': 2455.28,
     'coinbase:/ETH:/total': 5,
     'coinbase:/USD:/locked': 0,
     'coinbase:/ETH:volume': 679358.87,
     'coinbase:/ETH:/worth': 1375.05,
     'net_worth': 36466.75}

## Setup and Train DQN Agent

```python
from tensortrade.agents import DQNAgent

agent = DQNAgent(env)

agent.train(n_steps=200, save_path="examples/agents/")
```

    ====      EPISODE ID: eb726f14-9b6a-46be-a069-c79b007ff909      ====

## Plot Performance

```python
%matplotlib inline

portfolio.performance.plot()
```

```python
portfolio.performance.net_worth.plot()
```

## Setup and Train A2C Agent

```python
from tensortrade.agents import A2CAgent

agent = A2CAgent(env)

agent.train(n_steps=200, save_path="examples/agents/")
```

```python
portfolio.performance.plot()
```

```python
portfolio.performance.net_worth.plot()
```

## Setup and Train Parallel DQN Agent

```python
from tensortrade.agents import ParallelDQNAgent

def create_env():
    feed = DataFeed([coinbase_ns])

    portfolio = Portfolio(USD, [
        Wallet(coinbase, 10000 * USD),
        Wallet(coinbase, 10 * BTC),
        Wallet(coinbase, 5 * ETH),
    ])

    env = TradingEnvironment(
        feed=feed,
        portfolio=portfolio,
        action_scheme='managed-risk',
        reward_scheme='risk-adjusted',
        window_size=20
    )

    return env

agent = ParallelDQNAgent(create_env)

agent.train(n_envs=4, n_steps=200, save_path="examples/agents/")
```

    ====      EPISODE ID (0/2147483647): 0a15a273-c1e9-4f8e-9a50-6ea95795b484      ====
    ====      EPISODE ID (0/2147483647): d311dbdc-33d7-4704-b394-9468481e8c68      ====
    ====      EPISODE ID (0/2147483647): 9bca0f01-925f-4e73-86eb-c91ba8f9d20c      ====
    ====      EPISODE ID (0/2147483647): 3d12e8dd-674e-45c9-9009-59ab235342af      ====

```python

```