# Introduction to web scraping

This workshop is a one-hour beginner's introduction to web scraping.

This notebook deliberately has more content that we can reasonably cover in one hour. **The most important material is in bold**, and we'll focus on that material in person. To get the most out of this workshop, I'd suggest spending some time working through it in full after the workshop.

We'll cover the following topics:

[Motivation](#motivation)

_Why would you want to scrape data from the web?_

[How the Web works](#web)

_A high-level appreciation of how the Web works will help us to scrape data effectively._

[Making a request](#request)

_How can we ask other computers on the Internet to send us data using Python?_

[Parsing HTML](#parsing)

_Web pages are just files in a special format. Extracting information out of these files involves parsing HTML._

[Terms of Service](#terms)

_Don't go scraping willy-nilly!_

[Further resources](#resources)

_So you want to learn more about web scraping._

## Motivation

It's 2019. The web is everywhere.

* If you want to buy a house, real estate agents have [websites](https://www.wendytlouie.com/) where they list the houses they're currently selling.
* If you want to know whether to where a rain jacket or shorts, you check the weather on a [website](https://weather.com/weather/tenday/l/Berkeley+CA+USCA0087:1:US).
* If you want to know what's happening in the world, you read the news [online](https://www.sfchronicle.com/).
* If you've forgotten which city is the capital of Australia, you check [Wikipedia](https://en.wikipedia.org/wiki/Australia).

**The point is this: there is an enormous amount of information (also known as data) on the web.**

If we (in our capacities as, for example, data scientists, social scientists, digital humanists, businesses, public servants or members of the public) can get our hands on this information, **we can answer all sorts of interesting questions or solve important problems**.

* Maybe you're studying gender bias in student evaluations of professors. One option would be to scrape ratings from [Rate My Professors](https://www.ratemyprofessors.com/) (provided you follow their [terms of service](https://www.ratemyprofessors.com/TermsOfUse_us.jsp#use))
* Perhaps you want to build an app that shows users articles relating to their specified interests. You could scrape stories from various news websites and then use NLP methods to decide which articles to show which users.
* [Geoff Boeing](https://geoffboeing.com/) and [Paul Waddell](https://ced.berkeley.edu/ced/faculty-staff/paul-waddell) recently published [a great study](https://arxiv.org/pdf/1605.05397.pdf) of the US housing market by scraping millions of Craiglist rental listings. Among other insights, their study shows which metropolitan areas in the US are more or less affordable to renters.

## How the Web works

Here's our high-level description of the web.

**The internet is a bunch of computers connected together.** Some computers are laptops, some are desktops, some are smart phones, some are servers owned by companies. Each computer has its own address on the internet. Using these addresses, **one computer can ask another computer for some information (data). We say that the first computer sends a _request_ to the second computer, asking for some particular information. The second computer sends back a _response_**. The response could include the information requested, or it could be an error message. Perhaps the second computer doesn't have that information any more, or the first computer isn't allowed to access that information.

We said that there is an enormous amount of information available on the web. When people put information on the web, they generally have two different audiences in mind, two different types of consumers of their information: humans and computers. If they want their information to be used primarily by humans, they'll make a website. This will let them lay out the information in a visually appealing way, choose colours, add pictures, and make the information interactive. If they want their information to be used by computers, they'll make a web API. A web API provides other computers structured access to their data. We won't cover APIs in this workshop, but you should know that i) APIs are very common and ii) if there is an API for a website/data source, you should use that over web scraping. Many data sources that you might be interested in (e.g. social media sites) have APIs.

**Websites are just a bunch of files on one of those computers. They are just plain text files, so you can view them if you want. When you type in the address of a website in your browser, your computer sends a request to the computer located at that address. The request says "hey buddy, please send me the file(s) for this website". If everything goes well, the other computer will send back the file(s) in the response**. Everytime you navigate to a new website or page in your browser, this process repeats.

**There are three main languages that that website files are written with: HyperText Markup Language (HTML), Cascading Style Sheets (CSS) and JavaScript (JS)**. They normally have `.html`, `.css` and `.js` file extensions. Each language (and thus each type of file) serves a different purpose. **HTML files are the ones we care about the most, because they are the ones that contain the text you see on a web page**. CSS files contain the instructions on how to make the content in a HTML visually appealing (all the colours, font sizes, border widths, etc.). JavaScript files have the instructions on how to make the information on a website interactive (things like changing colour when you click something, entering data in a form). In this workshop, we're going to focus on HTML.

**It's not too much of a simplification to say:**

\begin{equation}
\textrm{Web scraping} = \textrm{Making a request for a HTML file} + \textrm{Parsing the HTML response}
\end{equation}

## Making a request

**The first step in web scraping is to get the HTML of the website we want to scrape. The [requests](http://docs.python-requests.org/en/master/) library is the easiest way to do this in Python.**

```python
import requests

url = 'https://en.wikipedia.org/wiki/Canberra'

response = requests.get(url)
```

Great, it looks like everything worked! Let's see our beautiful HTML:

```python
response
```

Huh, that's weird. Doesn't look like HTML to me.

What the `requests.get` function returned (and the thing in our `response` variable) was a Response object. It itself isn't the HTML that we wanted, but rather a collection of metadata about the request/response interaction between your computer and the Wikipedia server.

For example, it knows whether the response was successful or not (`response.ok`), how long the whole interaction took (`response.elapsed`), what time the request took place (`response.headers['Date']`) and a whole bunch of other metadata.

```python
response.ok
```

```python
response.headers['Date']
```

**Of course, what we really care about is the HTML content. We can get that from the `Response` object with `response.text`. What we get back is a string of HTML, exactly the contents of the HTML file at the URL that we requested.**

```python
html = response.text
print(html[:1000])
```

### Challenge

Get the HTML for [the Wikipedia page about HTML](https://en.wikipedia.org/wiki/HTML).
Print out the first 1000 characters and compare it to the HTML you see when you view the source HTML in your broswer.

```python
# your solution here
```

```python
# solution
url = 'https://en.wikipedia.org/wiki/HTML'
response = requests.get(url)
html = response.text
```

### Challenge

Write a function called `get_html` that takes a URL as an argument and returns the HTML contents as a string. Test your function on the page for [Sir Tim Berners-Lee](https://en.wikipedia.org/wiki/Tim_Berners-Lee).

```python
# your solution here
```

```python
# solution
def get_html(url):
    response = requests.get(url)
    return response.text

url = 'https://en.wikipedia.org/wiki/Tim_Berners-Lee'
html = get_html(url)
```

### Challenge

What happens if the request doesn't go so smoothly? Add a defensive measure to your function to check that the response recieved was successful.

```python
# your solution here
```

```python
# solution
def get_html(url):
    response = requests.get(url)
    assert response.ok, "Whoops, this request didn't go as planned!"
    return response.text

url = 'https://en.wikipedia.org/wiki/Tim_Berners-Lee'
html = get_html(url)
```

## Parsing HTML

**The second step in web scraping is parsing HTML. This is where things can get a little tricky.**

**Imagine you're in the field of education, in fact your specialty is studying higher education institutions. You're wondering how different disciplines change over time. Is it true that disciplines are incorporating more computational techniques as the years go on? Is that true for all disciplines or only some? Can we spot emerging themes across a whole university?**

**To answer these questions, we're going to need data. We're going to collect a dataset of all courses registered at UC Berkeley, not just those being taught this semester but all courses currently approved to be taught. These are listed on [this page](http://guide.berkeley.edu/courses/), called the Academic Guide. Well, actually they're not directly listed on that page. That page lists the departments/programs/units that teach currently approved courses. If we click on each department (for the sake of brevity, I'm just going to call them all "departments"), we can see the list of all courses they're approved to teach. For example, [here's](http://guide.berkeley.edu/courses/aerospc/) the page for Aerospace Studies. We'll call these pages departmental pages.**

### Challenge

View the source HTML of [the page listing all departments](http://guide.berkeley.edu/courses/), and see if you can find the part of the HTML where the departments are listed. There's a lot of other stuff in the file that we don't care too much about. You could try `Crtl-F`ing for the name of a department you can see on the webpage.

**Solution**

You should see something like this:

```
<div id="atozindex">
<h2 class="letternav-head" id='A'><a name='A'>A</a></h2>
<ul>
<li><a href="/courses/aerospc/">Aerospace Studies (AEROSPC)</a></li>
<li><a href="/courses/africam/">African American Studies (AFRICAM)</a></li>
<li><a href="/courses/a,resec/">Agricultural and Resource Economics (A,RESEC)</a></li>
<li><a href="/courses/amerstd/">American Studies (AMERSTD)</a></li>
<li><a href="/courses/ahma/">Ancient History and Mediterranean Archaeology (AHMA)</a></li>
<li><a href="/courses/anthro/">Anthropology (ANTHRO)</a></li>
<li><a href="/courses/ast/">Applied Science and Technology (AST)</a></li>
<li><a href="/courses/arabic/">Arabic (ARABIC)</a></li>
<li><a href="/courses/arch/">Architecture (ARCH)</a></li>
```

**This is HTML. HTML uses "tags", code that surrounds the raw text which indicates the structure of the content. The tags are enclosed in `<` and `>` symbols. The `<li>` says "this is a new thing in a list and `</li>` says "that's the end of that new thing in the list". Similarly, the `<a ...>` and the `</a>` say, "everything between us is a hyperlink". In this HTML file, each department is listed in a list with `<li>...</li>` and is also linked to its own page using `<a>...</a>`. In our browser, if we click on the name of the department, it takes us to that department's own page. The way the browser knows where to go is because the `<a>...</a>` tag tells it what page to go to. You'll see inside the `<a>` bit, there's a `href=...`. That tells us the (relative) location of the page it's linked to.**

### Challenge

Look at HTML source of [the page for the Aerospace Studies department](http://guide.berkeley.edu/courses/aerospc/), and try to find the part of the file where the information on each course is. Again, try searching for it using `Crtl-F`.

**Solution**

```
<div class="courseblock">

<button class="btn_toggleCoursebody" aria-expanded="false" aria-controls="cb_aerospc1a" data-toggle="#cb_aerospc1a">

<a name="spanaerospc1aspanspanfoundationsoftheu.s.airforcespanspan1unitspan"></a>
<h3 class="courseblocktitle">
<span class="code">AEROSPC 1A</span>
<span class="title">Foundations of the U.S. Air Force</span>
<span class="hours">1 Unit</span>
</h3>
```

The content that we care about is enclosed within HTML tags. It looks like the course code is enclosed in a `span` tag, which has a `class` attribute with the value `"code"`. What we'll have to do is extract out the information we care about by specifying what tag it's enclosed in.

But first, we're going to need to get the HTML of the first page.

### Challenge

Get the HTML content of `http://guide.berkeley.edu/courses/` and store it in a variable called `academic_guide_html`. You can use the `get_html` function you wrote before.

Print the first 500 characters to see what we got back.

```python
# your solution here
```

```python
# solution
academic_guide_url = 'http://guide.berkeley.edu/courses/'
academic_guide_html = get_html(academic_guide_url)
print(academic_guide_html[:500])
```

Great, we've got the HTML contents of the Academic Guide site we want to scrape. Now we can parse it. ["Parsing"](https://en.wikipedia.org/wiki/Parsing) means to turn a string of data into a structured representation. When we're parsing HTML, we're taking the Python string and turning it into a tree. The Python package `BeautifulSoup` does all our HTML parsing for us. We give it our HTML as a string and it returns a parsed HTML tree. Here, we're also telling BeautifulSoup to use the `lxml` parser behind the scenes.

```python
from bs4 import BeautifulSoup

academic_guide_soup = BeautifulSoup(academic_guide_html, 'lxml')
```

We said before that all the departments were listed on the Academic Guide page with links to their departmental page, where the actual courses are listed. So we can find all the departments by looking in our parsed HTML for all the links. Remember that the links are represented in the HTML with the `<a>...</a>` tag, so we ask our `academic_guide_soup` to find us all the tags called `a`. What we get back is a list of all the `a` elements in the HTML page.

```python
links = academic_guide_soup.find_all('a')
# print a random link element
links[48]
```

So now we have a list of `a` elements, each one represents a link on the Academic Guide page. But there are other links on this page in addition to the ones we care about, for example, a link back to the UC Berkeley home page. How can we filter out all the links we don't care about?

### Challenge

Look through the list `links`, or the HTML source, and figure out how we can identify just the links that we care about, namely the links to departmental pages.

```python
# your solution here
```

```python
# solution
import re

def is_departmental_page(link):
    """
    Return true if `link` points to a departmental page.

    By examining the source HTML by eye, I noticed that
    the links we care about (i.e. the departmental pages)
    all point to a relative path that starts with "/courses/".
    This function uses that idea to determine if the link is
    a departmental page.
    """
    # some links don't have a href attribute, only a name attribute
    # we don't care about them
    try:
        href = link.attrs['href']
    except KeyError:
        return False
    pattern = r'/courses/(.*)/'
    match = re.search(pattern, href)
    return bool(match)

print(links[0])
print(is_departmental_page(links[0]))
print()
print(links[48])
print(is_departmental_page(links[48]))
```

Let's use our new `is_departmental_page` function to filter out the links we don't care about. How many departments do we have?

```python
departmental_page_links = [link for link in links if is_departmental_page(link)]
len(departmental_page_links)
```

Each link in our `departmental_page_links` list contains a HTML element representing a link. Each element contains not only the relative location of the link but also the text that is linked (i.e. the words on the page that are underlined and you can click on to go to the linked page). In BeautifulSoup, we can get that text by asking for it with `element.text`, like this:

```python
departmental_page_links[0].text
```

### Challenge

From the `departmental_page_links`, we can extract out the name and the code for each department. Try doing this.

```python
# your solution here
```

```python
# solution
import re

def extract_department_name_and_code(departmental_link):
    """
    Return the (name, code) for a department.

    The easiest way to do this is to use regular expressions.
    We're not going to cover regular expressions in this workshop,
    but here's how to do it anyway.
    """
    text = departmental_link.text
    pattern = r'([^(]+) \((.*)\)'
    match = re.search(pattern, text)
    if match:
        return match.group(1), match.group(2)

extract_department_name_and_code(links[48])
```

From each link in our `departmental_page_links` list, we can get the relative link that it points to like this:

```python
departmental_page_links[0].attrs['href']
```

### Challenge

Write a function that extracts out the relative link of a link element.

*Hint: This has a similar solution to our `is_departmental_page` function from before.*

```python
# your solution here
```

```python
# solution
def extract_relative_link(departmental_link):
    """
    We noted above that all the departmental links point to "/courses/something/",
    where the "something" looks a lot like their code. This function
    extracts out that "something", so we can add it to the base URL of
    the Academic Guide page and get full paths to each departmental page.
    """
    href = departmental_link.attrs['href']
    pattern = r'/courses/(.*)/'
    match = re.search(pattern, href)
    if match:
        return match.group(1)

extract_relative_link(departmental_page_links[0])
```

Alright! Now we've identified all the departmental links on the Academic Guide page, we've found their name and code, and we know the relative link they point to. Next, we can use this relative link to construct the full URL they point to, which we'll then use to scrape the HTML for each departmental page.

Let's write a function that takes a departmental link and returns the absolute URL of its departmental page.

```python
def construct_absolute_url(departmental_link):
    relative_link = extract_relative_link(departmental_link)
    return academic_guide_url + relative_link

construct_absolute_url(departmental_page_links[37])
```

To summarize so far, we've gone from the URL of the Academic Guide website, found all the departments that offer approved courses, identified their name and code and the link to their departmental page which lists all the courses they teach.

Now we want to find the get the HTML for each departmental page and scrape it for all the courses they offer. Let's focus on one page for now, the Aerospace Studies page. Once we select the link, we use our functions from above to: i) get the name (I guess we already know it's Aerospace, but whatever) and code, get the full URL, get the HTML for that URL and then parse the HTML.

```python
aerospace_link = departmental_page_links[0]
aerospace_name, aerospace_code = extract_department_name_and_code(aerospace_link)
aerospace_url = construct_absolute_url(aerospace_link)
aerospace_html = get_html(aerospace_url)
aerospace_soup = BeautifulSoup(aerospace_html, 'lxml')
print(aerospace_html[:500])
```

Right at the start of this section on parsing HTML, we saw the HTML for a departmental page. Here it is again.

```
<div class="courseblock">

<button class="btn_toggleCoursebody" aria-expanded="false" aria-controls="cb_aerospc1a" data-toggle="#cb_aerospc1a">

<a name="spanaerospc1aspanspanfoundationsoftheu.s.airforcespanspan1unitspan"></a>
<h3 class="courseblocktitle">
<span class="code">AEROSPC 1A</span>
<span class="title">Foundations of the U.S. Air Force</span>
<span class="hours">1 Unit</span>
</h3>
```

It looks like each course is listed in a `div` element that has a `class` attribute with value `"courseblock"`. We can use this information to identify all the courses on a page and then extract out the information from them. You've seen how to do this before, here it is again:

```python
aerospace_courseblocks = aerospace_soup.find_all(class_='courseblock')
len(aerospace_courseblocks)
```

Looks like the Aerospace department has seven current courses they're approved to teach (at the time of writing). Looking at the page in our browser, that looks right to me! So now we have a list called `aerospace_courseblocks` that holds seven elements that each refer to one course taught by the Aerospace department. Now we can extract out any information we care about. We just have to look at the page in our browser, decide what information we care about, then look at the HTML source to see where that information is kept in the HTML structure. Finally, we write a function for each piece of information we want to extract out of a course.

### Challenge
Write functions to take a courseblock and extract:
- The course code (e.g. AEROSPC 1A)
- The coure name
- The number of units
- The textual description of the course

```python
# your solution here
```

```python
# solution
def extract_course_code(courseblock):
    span = courseblock.find(class_='code')
    return span.text

def extract_course_title(courseblock):
    span = courseblock.find(class_='title')
    return span.text

def extract_course_units(courseblock):
    span = courseblock.find(class_='hours')
    return span.text

def extract_course_description(courseblock):
    span = courseblock.find(class_='coursebody')
    return span.text

def extract_one_course(courseblock):
    course = {}
    course['course_code'] = extract_course_code(courseblock)
    course['course_title'] = extract_course_title(courseblock)
    course['course_units'] = extract_course_units(courseblock)
    course['course_description'] = extract_course_description(courseblock)
    return course

first_aerospace_course = extract_one_course(aerospace_courseblocks[0])
for value in first_aerospace_course.values():
    print(value)
    print()
```

Let's write a function to scrape these four pieces of information from every course from every department and save it as a csv file.

```python
def scrape_one_department(department_link):
    department_name, department_code = extract_department_name_and_code(department_link)
    department_url = construct_absolute_url(department_link)
    department_html = get_html(department_url)
    department_soup = BeautifulSoup(department_html, 'lxml')
    department_courseblocks = department_soup.find_all(class_='courseblock')
    result = []
    for courseblock in department_courseblocks:
        course = extract_one_course(courseblock)
        course['department_name'] = department_name
        course['department_code'] = department_code
        result.append(course)
    return result

aerospace_courses = scrape_one_department(aerospace_link)
for value in aerospace_courses[0].values():
    print(value)
    print()
```

```python
import time

def scrape_all_departments(be_nice=True):
    academic_guide_url = 'http://guide.berkeley.edu/courses/'
    academic_guide_html = get_html(academic_guide_url)
    academic_guide_soup = BeautifulSoup(academic_guide_html, 'lxml')
    links = academic_guide_soup.find_all('a')
    departmental_page_links = [link for link in links if is_departmental_page(link)]

    result = []
    for departmental_page_link in departmental_page_links:
        department_result = scrape_one_department(departmental_page_link)
        result.extend(department_result)
        if be_nice:
            time.sleep(1)
    return result
```

```python
import pandas as pd
result = scrape_all_departments(be_nice=False)
df = pd.DataFrame(result)
print(str(len(df)) + ' courses scraped')
df.head()
```

9360 courses scraped! (At the time of writing). Wow, that was a lot easier than doing it by hand!

## Terms of Service

As you've seen, web scraping involves making requests from other computers for their data. It costs people money to maintain the computers that we request data from: it needs electricity, it requires staff, sometimes you need to upgrade the computer, etc. But we didn't pay anyone for using their resources.

Because we're making these requests programmatically, we could make many, many requests per second. For example, we could put a request in a never-ending loop which would constantly request data from a server. But computers can't handle too much traffic, so eventually this might crash someone else's computer. Moreover, if we make too many requests when we're web scraping, that might restrict the number of people who can view the web page in their browser. This isn't very nice.

Websites often have Terms of Service, documents that you agree to whenever you visit a site. Some of these terms prohibit web scraping, because it puts too much strain on their servers, or they just don't want their data accessed programmatically. Whatever the reason, we need to respect a websites Terms of Service. **Before you scrape a site, you should always check its terms of service to make sure it's allowed.**

Often, there are better ways of accessing the same data. For the Wikipedia sites we scraped, there's actually an [API](https://www.mediawiki.org/wiki/REST_API) that we could have used. In fact, Wikipedia would prefer that we access their data that way. There's even a [Python package](https://pypi.org/project/wikipedia/) that wraps around this API to make it even easier to use. Furthermore, Wikipedia actually makes all of its content available for [direct download](https://dumps.wikimedia.org/). **The point of the story is: before web scraping, see if you can get the same data elsewhere.** This will often be easier for you and preferred by the people who own the data.

Moreover, if you're affiliated with an institution, you may be breaching existing contracts by engaging in scraping. UC Berkeley's Library [recommends](http://guides.lib.berkeley.edu/text-mining) following this workflow:

## Further resources

### Resources for learning

* Work through this notebook in full.
    * _We glossed over a lot of details. As your next step, I'd suggest spending as much time as you need to understand every line of text and code in this notebook._

* [Web-scraping with Python](http://shop.oreilly.com/product/0636920078067.do)

    * _A great textbook for learning more about web scraping using Python._

* [Fantastic Data and Where To Find Them: An introduction to APIs, RSS, and Scraping](https://www.youtube.com/watch?v=A42voDYkFZw)

    * This is a recorded video of a workshop on collecting data via the web at PyCon, a Python conference._

* [D-Lab workshops](http://dlab.berkeley.edu/training)

    * _We teach workshops on web scraping and NLP-related tools throughout the semester. Check this page for the latest scheduled workshops._

* [D-Lab consulting](http://dlab.berkeley.edu/consulting)

    * _We also offer free consulting for members of UC Berkeley's community. Reach out to us if you ever need a hand with a web scraping or NLP project!_

### A few libraries to be aware of

* [requests-HTML](http://html.python-requests.org/)

    * _Did you see how easy it was to request data using the `requests` library? Well the author of that library, Kenneth Reitz, has another library for parsing HTML. I'm not that familiar with it, but it looks promising if it's by Reitz!_

* [furl](https://github.com/gruns/furl)

    * _Lets you extract out different parts of a URL._

* [cssutils](http://cthedot.de/cssutils/)

    * _We didn't talk about CSS at all, but you can also scrape data depending on its visual characteristics. This is a great library for parsing CSS files, but you can get some of the same functionality with BeautifulSoup._

* [scrapy](https://scrapy.org/)

    * _Need to do some serious web scraping? You'll wanna check out Scrapy._

* [newspaper](https://newspaper.readthedocs.io/en/latest/)

    * _If you know you're focussed on newspaper articles, this is a great little library for parsing common formats._


Aside: [http://illinois.edu/calendar/detail/990/33249682](http://illinois.edu/calendar/detail/990/33249682)

# Non-Proliferation and Safeguards

This unit covers discussions of nuclear non-proliferation and safeguards in the context of nuclear power and the nuclear fuel cycle.

```python
from IPython.display import IFrame
IFrame("https://embed.polleverywhere.com/free_text_polls/LaKxXWF9y4JKkue?short_poll=true", width="700", height="400")
```

```python
IFrame("https://embed.polleverywhere.com/multiple_choice_polls/YiHjKwlUd1Cit6s?short_poll=true", width="700", height="400")
```

```python
IFrame("https://embed.polleverywhere.com/free_text_polls/lBcbliuSuQGKea6?short_poll=true", width="700", height="400")
```

Source: [Huck Magazine, Photography by Campaign for Nuclear Disarmament](http://www.huckmagazine.com/perspectives/activism-2/anti-nuclear-protest-art-will-stop-tracks/)

## Learning Objectives

At the end of this lesson, you will be equipped to:

- Recognize the position and history of movements against nuclear weapons.
- List key events in the history of nuclear weapons.
- Identify nuclear weapons states.
- Characterize the agreements between nuclear weapons states.
- Analyze the relationship between nuclear weapons and nuclear power.
- Name the national and international agencies responsible for enforcing non-proliferation treaties.
- Recognize methods used to enforce nuclear safeguards.
- Apply knowledge of the NPT to current and recent events.
- Differentiate between fact and fiction.
- Discuss the failures of our nuclear weapons program (Bikini Atoll,

## Art to Consider

### Hiroshima survivor Hiroharu Kono

### Yusaku Kamekura

# How Did We Get Here?

## The Making of the Atomic Bomb

The best way to learn about this topic is to read Richard Rhodes excellent book, The Making of The Atomic Bomb.

However, let's summarize:

### 1930
German nationalists acheive shocking election victory.

### 1938
Theory of Nuclear Fission (Discovered on December 17,  1938 and explained theoretically in January 1939).

### 1939
 Szilard and Wigner draft the Einstein-Szilard letter.

### 1940 - 1946
Manhattan Project

### July, 16 1945

- Trinity Nuclear Test
- "The Gadget"/"Fat Man"
  - 6.19-kilogram (13.6 lb) Pu
  - Implosion type plutonium weapon
  - energy equivalent of 22 kilotons of TNT (92 TJ)

### 8:15 a.m., August 6, 1945

- Hiroshima
- population 340,000 - 350,000
- Enola Gay, B-29
- "Little Boy"
  - contained 64 kg (141 lb) of enriched uranium.
  - less than 1 kg completely detonated.
  - energy equivalent of 15 kilotons of TNT (63 TJ).
  - gun type.
- 70,000 people dead instantly.
- 90,000–166,000 after the first year. About 70% of the city's buildings were destroyed, and another 7% severely damaged.

### 11:02 a.m., August 9, 1945

- Nagasaki.
- population 263,000
- Bockscar, B-29
- "Fat Man"
  - contained 14 pounds (6.4 kg) Pu
  - energy equivalent of 21 kilotons of TNT (88 TJ; 1 g mass equivalent)
  - implosion type
- 35,000 people were killed instantly
- 39,000 - 80,000 by the end of 1945
- Fireball temp 3,900 °C (7,050 °F)
- Winds over 1,000 km/h (620 mph)

### 6:45 a.m., March 1, 1954

- Castle Bravo.
- dry fuel hydrogen bomb
- energy released: 15 megatons of TNT
- vastly exceeded expected  4 to 8 megatons (because of an unexpectedly high involvement of lithium-7 in the fusion reaction)
- shifting winds ---> greatest contamination every perpetrated by the U.S.
- Japanese fishing boat (fishermen died, contaminated fish were sold)

### 11:32 a.m. October 30, 1961

- Tsar Bomba / Ivan / Vanya
- Soviet RDS-220, three-stage hydrogen bomb
  - 2nd/3rd stages had 238U "tamper" to reduce fast fissions & fallout
  - 97% of the total yield resulted from thermonuclear fusion
  - energy released: 50 megaton TNT (210 PJ)

### October 16–28, 1962
Cuban Missile Crisis

### 1963
Partial test ban treaty.

## 1970 : NPT

Opened for signature in 1968, the treaty entered into force in 1970

The goals of the NPT were to:

- Prevent the spread of nuclear weapons
- Promote cooperation in peaceful uses of nuclear technology
- Acheive nuclear disarmament

Accordingly:

- Countries with weapons (UK, US, USSR, France, China) agree not to share.
- Countries without nuclear weapons promised not to develop them in return for nuclear power technology.
- Those that already have weapons agreed to disarm. (Article 6: Countries must "negotiate in good faith.")

NPT is reviewed annually.

### Membership

By File:NPT Participation.svg: Allstar86, L.tak, Danlaycock
derivative work: Danlaycock - This file was derived from:
 NPT Participation.svg, CC BY-SA 3.0, Link

#### Renegged
Only North Korea has backed out of their agreement.

#### Never signed

Indian, Pakistan, and Israel never joined. All three now have weapons.

### 1996

Comprehensive Test Ban Treaty

## Megatons to Megawatts (1993 - 2003)

- **20** year program
- **500 tons** of HEU (**90%** enriched)
- Soviet era warheads -> downblended to $<5\%$ enriched
- To be mplemented on commercial terms without government funds.
  - United States named Enrichment Corporation (USEC) as the commercial agent
  - the Russian Federation designated Techsnabexport (TENEX) as its commercial agent.
- The first shipment of LEU took place in May 1995.

**Practice: Can You prove this is an accurate statement?**

Think back to enrichment. This is another feed factor problem.

\begin{align}
x_p &= 0.05\\
x_w &= 0.002\\
x_f &= 0.9\\
\end{align}

Thus, we can find the feed factor (kg feed per kg enriched):

\begin{align}
FF &= \frac{F}{P}\\
   &= \frac{x_p - x_w}{x_f - x_w}\\
   &= \frac{0.05 - 0.002}{0.9 - 0.002}\\
   & \\
   &= 0.0535
\end{align}

We also know the amount of feed, F, was 500 tons.

\begin{align}
FF = \frac{F}{P}\\
FF = 0.0535\\
F = 5.0\times 10^5\\
\end{align}

```python
F = 500000 # kg
x_p = 0.05  # sold to US
x_w = 0.002 # depleted U
x_f = 0.9   # feed from  HEU

def ff(x_p, x_w, x_f):
    num = x_p - x_w
    denom = x_f - x_w
    return num/denom

print(ff(x_p, x_w, x_f),
      " kg of the HEU are needed to produce 1kg fuel.")
print("Therefore, ",1/ff(x_p, x_w, x_f),
      " kg of the new will be produced per kg of HEU")
print("Therefore, ",F/ff(x_p, x_w, x_f),
      " kg of the new low enriched uranium was produced")

```

    0.053452115812917596  kg of the HEU are needed to produce 1kg fuel.
    Therefore,  18.708333333333332  kg of the new will be produced per kg of HEU
    Therefore,  9354166.666666666  kg of the new low enriched uranium was produced

Now, based on our fuel utilization discussions, how many MWh is that?

\begin{align}
M = \frac{Q}{BU}\\
Q = \frac{P_0 \cdot CF \cdot T}{\eta_{th}}\\
\end{align}

We want to solve for $P_0 CF T$, which will be in units of MWd (and we can convert to MWh.)
\begin{align}
\Rightarrow &M = \frac{P_0 \cdot CF \cdot T}{\eta_{th}BU}\\
\Rightarrow &\frac{M}{CF} = \frac{P_0 \cdot CF \cdot T}{\eta_{th}BU}\\
\Rightarrow &P_0 \cdot CF \cdot T [MWh] = M\eta_{th}BU[MWd]\times 24 \left[\frac{h}{d}\right]  \\
\end{align}

So, let's assume some typical values appropriate for 5% enriched fuel (this will give high-ish burnups).

\begin{align}
\eta_{th} = 0.33\mbox{ [MWe/MWth]}\\
BU = 50000\mbox{ [MWd/ton] } = 50 \mbox{ [MWd/kg] }\\
\end{align}

```python
M = F/ff(x_p, x_w, x_f)
#cf = 0.90
eta = 0.37
h_per_d = 24
bu = 50

PCFT = M*eta*bu*h_per_d
print("And finally, the program produced ",'%e' % PCFT, " MWh.")
```

    And finally, the program produced  4.153250e+09  MWh.

```python

from IPython.core.display import HTML
HTML("""<video controls src="{0}" loop=1 autoplay width=100%/>""".format('https://img.buzzfeed.com/buzzfeed-static/static/2014-07/3/11/enhanced/webdr04/anigif_enhanced-buzz-21275-1404402138-31.gif?output-format=mp4'))

```

<video controls src="https://img.buzzfeed.com/buzzfeed-static/static/2014-07/3/11/enhanced/webdr04/anigif_enhanced-buzz-21275-1404402138-31.gif?output-format=mp4" loop=1 autoplay width=100%/>

# Friday: IAEA, Safeguards, MUF, SQ, Etc.

## IAEA Safeguards

150 million euro per year.

Director General...

### Material Accountancy

## Significant Quantities
Direct use nuclear material

|======|========|======|
| Pu    | 	8 kg  | Pu |
| 233U	 | 8 kg  | 233U |
| HEU (235U < 20%) | 	25 kg 235U |  |
| Indirect use nuclear material | | |
| U(235U < 20%)b |	75 kg 235U | (or 10 t natural U | |
| Th	| 20 t |  Th |

### Example: Significant Quantity of Enriched U

Determine the

### Locks and Surveillance

## Material Unaccounted For

$$MUF = (PB + X - Y) - PE$$

PB = Beginning Physical inventory
X = Increases to inventory
Y = Removal from inventory
PE = Ending Physical inventory

- Understand the concepts of detection, delay, and response technologies.
- Formulate different response strategies (including deterrence, denial, containment, pursuit, and recapture) for different facilities and considering on-site and/or o ff-site response.
- Defense-in-depth
- PRA
- Knowledge of the history o f nuclear weapons development
- Know the countries who have developed nuclear weapons and the political reasons for their development
- Understand the skills, capabilities, and materials needed by a proliferator to produce a nuclear weapon.
- Knowledge o f current the NPT and the Additional Protocols
- Be able to calculate HEU and Pu production quantities for various nuclear facilities.
- Identify proliferation-risks/safeguards-weaknesses in a nuclear fuel cycle or in a collection of facilities.
- Understand the methods for safeguarding nuclear material at declared facilities and describe the technology used to measure bulk nuclear materials.
- Describe methods for identifying covert nuclear activities and how they can be applied to treaty verification.
- Know the types “Special Nuclear Material”
- Understand the types of radiation that is emitted by SNM
- Understand the challenges o f detecting SNM
- Be able to calculate uranium enrichment from spectra
- Know what detection systems should be used and under what circumstances
- Understand the various forms of SNM for each step o f an open and closed fuel cycle
- Calculate and identify possible points o f MUF
- De fine and calculate separative work units for uranium enrichment
- Be able to describe the methods used for accounting and control of SNM
- Discuss the elements o f proliferation resistance
- Know the quantities of SNM created in SNF for various burn-ups and reactor types
- Statistics applied in safeguards
- Describe the fundamentals of how nuclear m aterial safeguards are implemented to secure nuclear material from diversion by the country.


## Output Handling

In Colaboratory the output of each cell is hosted in a separate iframe sandbox with limited access to the global notebook environment. APIs are available to enable some Python to Javascript and Javascript to Python communications, these APIs are generally only available for outputs which have been executed in the current session.

### Viewing Notebooks

Colaboratory attempts to provide a rich viewing experience where all resources needed to view outputs are persisted in the notebook itself and can be used for later viewing when not connected to a kernel.

In cases where outputs require execution to work properly this should be communicated to the user.

## display.Javascript to execute JavaScript from Python

The Javascript is persisted in the outputs of the notebook document and will be executed when the document is reloaded.

```
import IPython
js_code = '''
document.querySelector("#output-area").appendChild(document.createTextNode("hello world!"));
'''
display(IPython.display.Javascript(js_code))
```

    <IPython.core.display.Javascript object>

## Evaluate a Javascript expression from Python with eval_js

Evaluates the Javascript expression within the context of the outputframe of the current cell.

This executes the Javascript as a blocking call from Python and returns the result of the expression.

If the Javascript expression results in a [Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) then the call will block until the promise has settled and use the resulting value.

**Note:** This differs from `display.Javascript` in that the Javascript expression is not persisted in the notebook document and will not be executed when the document is reloaded.

```
import IPython
from google.colab import output

display(IPython.display.Javascript('''
  window.someValue = new Promise(resolve => {
    setTimeout(() => {
      resolve("hello world!");
    }, 100);
  });
'''))

value = output.eval_js('someValue');
value
```

    <IPython.core.display.Javascript object>

    'hello world!'

## Javascript to Python communication

Python code can register callbacks which can be invoked by Javascript in the outputframe.

**Note:** This is only available to 'trusted' outputs- outputs which have been executed within the current session.

```
import IPython
from google.colab import output

def Concat(a, b):
  # Use display.JSON to transfer a structured result.
  return IPython.display.JSON({'result': ' '.join((a, b))})

output.register_callback('notebook.Concat', Concat)
```

```javascript
%%javascript
(async function() {
  const result = await google.colab.kernel.invokeFunction(
    'notebook.Concat', // The callback name.
    ['hello', 'world!'], // The arguments.
    {}); // kwargs
  const text = result.data['application/json'];
  document.querySelector("#output-area").appendChild(document.createTextNode(text.result));
})();
```

    <IPython.core.display.Javascript object>

### Persisting Updates

Since `invokeFunction` is only available for outputs which have executed within the current session
then it may be desirable to update the notebook so the changes take effect on notebook reload. An example
of doing this.

```
import IPython
from google.colab import output

display(IPython.display.HTML('''
    The items:
    <br><ol id="items"></ol>
    <button id='button'>Click to add</button>

    '''))

def add_list_item():
  # Use redirect_to_element to direct the elements which are being written.
  with output.redirect_to_element('#items'):
    # Use display to add items which will be persisted on notebook reload.
    display(IPython.display.HTML('<li> Another item</li>'))

output.register_callback('notebook.AddListItem', add_list_item)
```

The items:

<button id='button'>Click to add</button>

### Using register_callback to make a button invoke a function

```
import IPython
import uuid
from google.colab import output

class InvokeButton(object):
  def __init__(self, title, callback):
    self._title = title
    self._callback = callback

  def _repr_html_(self):
    callback_id = 'button-' + str(uuid.uuid4())
    output.register_callback(callback_id, self._callback)

    template = """<button id="{callback_id}">{title}</button>
        """
    html = template.format(title=self._title, callback_id=callback_id)
    return html

def do_something():
  print('here')

InvokeButton('click me', do_something)
```

<button id="button-508b2891-2393-4644-a7e8-68e99af5f509">click me</button>

## Tagged Outputs

Output tagging is useful for removing or replacing chunks of outputs from Python.

```
import time
import sys
from google.colab import output

print('Starting.')

with output.use_tags('some_outputs'):
  sys.stdout.write('working....\n')
  sys.stdout.flush();
  time.sleep(2)

  sys.stdout.write('still working...\n')
  sys.stdout.flush();
  time.sleep(2)

# Now clear the previous outputs.
output.clear(output_tags='some_outputs')
print('All done!')

```

    Starting.
    All done!

## Pausing output processing

If your Javascript code is doing an asynchronous operation (such as loading other resources), use `colab.output.pauseOutputUntil` to pause execution of subsequent output messages until the promise has completed.

This has an additional benefit that when the notebook is later loaded then output will wait for all messages and pauses to complete before resizing the cell, reducing the amount of jank when loading notebooks.

```
import IPython

display(IPython.display.Javascript('''
  const promise = new Promise((resolve, reject) => {
    const script = document.createElement('script');
    script.src = 'data:,window.value = "hello world!"';
    script.onload = resolve;
    script.onerror = reject;
    document.head.appendChild(script);
  });
  // Pause subsequent outputs until the script has been loaded.
  google.colab.output.pauseOutputUntil(promise);
'''))

display(IPython.display.Javascript('''
    // Can immediately reference scripts loaded earlier since
    // output processing was blocked on them.
    document.body.appendChild(document.createTextNode(window.value));
'''))

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

## Serving resources

Outputs in the browser can request resources from the kernel by requesting `https://localhost:{port}`. The protocol will automatically be translated from `https` to `http` and the `localhost` will be the kernel executing the code.

By default the responses to any kernel requests will be cached in the notebook JSON to make them available to viewers not connected to the kernel. To skip the caching of resources, set the HTTP header `x-colab-notebook-cache-control: no-cache` to any responses. See the 'Custom server' example below.

Outputs can only request resources from the kernel if they have been executed in the current session, when viewed the outputs will only have access to the cached responses.

### Jupyter nbextensions
Jupyter's nbextensions is a mechanism to bundle resources into Python libraries, see the [documentation](http://testnb.readthedocs.io/en/latest/examples/Notebook/Distributing%20Jupyter%20Extensions%20as%20Python%20Packages.html#Defining-the-server-extension-and-nbextension) for details.

An example of referencing these resources from outputs:

```
%%html
<link rel="stylesheet" href="/nbextensions/google.colab/tabbar.css">
<div class='goog-tab'>
  Some content
</div>
```

<link rel="stylesheet" href="/nbextensions/google.colab/tabbar.css">

  Some content

### Custom server

A webserver can be started on the kernel to serve up arbitrary resources.

The client needs to reference the resource with `https://localhost:{port}` but this will automatically be translated to `http://localhost:{port}`.

By default responses will be cached in the notebook for offline access.

```
import portpicker
import threading
import socket
import IPython

from six.moves import socketserver
from six.moves import SimpleHTTPServer

class V6Server(socketserver.TCPServer):
  address_family = socket.AF_INET6

class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler):
  def do_GET(self):
    self.send_response(200)
    # If the response should not be cached in the notebook for
    # offline access:
    # self.send_header('x-colab-notebook-cache-control', 'no-cache')
    self.end_headers()
    self.wfile.write(b'''
      document.querySelector('#output-area').appendChild(document.createTextNode('Script result!'));
    ''')

port = portpicker.pick_unused_port()

def server_entry():
    httpd = V6Server(('::', port), Handler)
    # Handle a single request then exit the thread.
    httpd.serve_forever()

thread = threading.Thread(target=server_entry)
thread.start()

# Display some HTML referencing the resource.
display(IPython.display.HTML(''.format(port=port)))
```

## Cross-output communication

The output of each cell is hosted in a separate sandboxed iframe without direct access to the contents of other cells, but they can create [BroadcastChannel](https://developer.mozilla.org/en-US/docs/Web/API/BroadcastChannel/BroadcastChannel)s to communicate between cells.

```javascript
%%javascript
const listenerChannel = new BroadcastChannel('channel');
listenerChannel.onmessage = (msg) => {
  const div = document.createElement('div');
  div.textContent = msg.data;
  document.body.appendChild(div);
};
```

    <IPython.core.display.Javascript object>

This second cell will be in a separate sandboxed iframe.

```javascript
%%javascript
const senderChannel = new BroadcastChannel('channel');
senderChannel.postMessage('Hello world!');
```

    <IPython.core.display.Javascript object>

## Rendering SymPy in Colab

Rendering [SymPy](http://www.sympy.org) equations in Colab requires the MathJax javascript library to be available within each cell output. The following defines an execution hook that loads it automatically each time you execute a cell.

```
from IPython.display import Math, HTML

def load_mathjax_in_cell_output():
  display(HTML(""))
get_ipython().events.register('pre_run_cell', load_mathjax_in_cell_output)
```

```
# try it out:
import sympy
sympy.init_printing()
x = sympy.symbols('x')
sympy.Integral(sympy.sqrt(1 / x), x)
```

## Camera Capture
Using a webcam to capture images for processing on the runtime.

```
from IPython.display import display, Javascript
from google.colab.output import eval_js
from base64 import b64decode

def take_photo(filename='photo.jpg', quality=0.8):
  js = Javascript('''
    async function takePhoto(quality) {
      const div = document.createElement('div');
      const capture = document.createElement('button');
      capture.textContent = 'Capture';
      div.appendChild(capture);

      const video = document.createElement('video');
      video.style.display = 'block';
      const stream = await navigator.mediaDevices.getUserMedia({video: true});

      document.body.appendChild(div);
      div.appendChild(video);
      video.srcObject = stream;
      await video.play();

      // Resize the output to fit the video element.
      google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);

      // Wait for Capture to be clicked.
      await new Promise((resolve) => capture.onclick = resolve);

      const canvas = document.createElement('canvas');
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      canvas.getContext('2d').drawImage(video, 0, 0);
      stream.getVideoTracks()[0].stop();
      div.remove();
      return canvas.toDataURL('image/jpeg', quality);
    }
    ''')
  display(js)
  data = eval_js('takePhoto({})'.format(quality))
  binary = b64decode(data.split(',')[1])
  with open(filename, 'wb') as f:
    f.write(binary)
  return filename
```

```
from IPython.display import Image
try:
  filename = take_photo()
  print('Saved to {}'.format(filename))

  # Show the image which was just taken.
  display(Image(filename))
except Exception as err:
  # Errors will be thrown if the user does not have a webcam or if they do not
  # grant the page permission to access it.
  print(str(err))
```

    <IPython.core.display.Javascript object>

    Saved to photo.jpg

## Showing CV2 Images

The ``cv2.imshow()`` and ``cv.imshow()`` functions from the [opencv-python](https://github.com/skvark/opencv-python) package are incompatible with Jupyter notebook; see https://github.com/jupyter/notebook/issues/3935.

As a replacement, you can use the following function:

```
from google.colab.patches import cv2_imshow
```

For example, here we download and display a PNG image of the Colab logo:

```
!curl -o logo.png https://colab.research.google.com/img/colab_favicon_256px.png
import cv2
img = cv2.imread('logo.png', cv2.IMREAD_UNCHANGED)
cv2_imshow(img)
```

```

```