# The Seats-Votes Curve in a Single Election

*Levi John Wolf

University of Bristol

levi.john.wolf@gmail.com

(Supported by NSF #1657689)*

In the last notebook, we talked about the concept underlying *historical* seats-votes relationships. At its core, though, estimating seats-votes relationships in this manner is

1. **wasteful**: in each election, we observe a ton of district-level information. However, we disregard nearly all of this information about district-level *vote* results in favor of exclusive examination of the *district* winners. While winning a district or not *is* information about district-level vote (it's a censored variable with a threshold of $.5$), this is not the same as the original information in the vote shares.
2. **dubious**: over many elections, the seats-votes model does *not* account for structural changes to the electorate, the party system, or the electoral process. This means that each election is considered comparable to every other election, even when they're separated by *large* expanses of time or structural changes to the electoral system.

These kinds of objections are noted by Browning & King (1987) but are most forcefully discussed by Gelman & King (1994). Incorporating older arguments (e.g. Mackerras (1962), among others), this line of argument seeks to define seats votes curves *in a single election* using a basic counterfactual argument: when shifts in party popular vote occur, these votes tend to be distributed uniformly at random among districts.

In this notebook, we'll talk about how this kind of seats-votes curve is built within a single election, how the *uniform partisan swing* assumption works, and how it might not quite capture the true reality of how votes change from election to election. But, first, though, we'll have to read the data again.

```python
import pandas
import pysal
import numpy
import seaborn
import seatsvotes
import statsmodels.api as sm
import matplotlib.pyplot as plt
%matplotlib inline
```

    /home/lw17329/anaconda/envs/ana/lib/python3.6/site-packages/pysal/__init__.py:65: VisibleDeprecationWarning: PySAL's API will be changed on 2018-12-31. The last release made with this API is version 1.14.4. A preview of the next API version is provided in the `pysal` 2.0 prelease candidate. The API changes and a guide on how to change imports is provided at https://pysal.org/about
      ), VisibleDeprecationWarning)

```python
house = seatsvotes.data.congress(geo=True)
house.crs = {"init":"epsg:4269"}
house = house.to_crs(epsg=5070)
```

```python
house.head()
```

      index
      congress
      year
      delsouth
      state_fips
      contest_ui
      inc
      king_dist
      lewis_dist
      state_name
      turnout
      vote_share
      geom_uid
      geometry

      0
      870
      107
      2000
      0
      06
      107006026
      1
      NaN
      26
      california
      96500.0
      1.000000
      006103107026
      POLYGON ((-2035775.761381865 1494606.630854639...

      1
      871
      107
      2000
      0
      06
      107006029
      1
      NaN
      29
      california
      226079.0
      0.797487
      006103107029
      POLYGON ((-2026676.55813503 1472460.195548936,...

      2
      872
      107
      2000
      0
      06
      107006030
      1
      NaN
      30
      california
      95011.0
      0.875930
      006103107030
      POLYGON ((-2011853.30096701 1467542.286440545,...

      3
      873
      107
      2000
      1
      12
      107012013
      -1
      NaN
      13
      florida
      275486.0
      0.361427
      012105107013
      (POLYGON ((1339081.34287222 619572.4289258342,...

      4
      874
      107
      2000
      1
      12
      107012014
      -1
      NaN
      14
      florida
      242614.0
      0.000000
      012105107014
      (POLYGON ((1388687.535007545 543393.538895309,...

# Seats & Votes in a single election

At its core, analyzing a seats-votes relationship in a single election is somewhat suspect. This is because the relationship between votes and seats is a *strict hinge-point* relationship in most two-party elections. Once the reference party gets beyond 50% in the two-party vote (i.e. once Dems get more votes than Republicans), the district is "won," and represented by a $1$. In the other direction, the district is lost. This makes for a pretty boring scatterplot:

```python
plt.scatter(house.vote_share, house.vote_share > .5, marker='.', color='r')
plt.vlines(.5, 0,1, linestyle=':', color='k', )
```

Indeed, there's no real mechanism to be learned here; the basic relationship is simply one of a censored variable; when Dems win a majority of the two-party vote, they win the seat; when Dems win a minority, Republicans win.

### Uniform Partisan Swing

So, how do we model the seats-votes curve in a single election? We can't do it *directly* based on the split of districts that are won (or lost). Instead, a classic method to construct the seats-votes curve relies on the assumption of *uniform partisan swing*, that changes in a party's popular vote are reasonably modelled by assigning those changes to *districts* uniformly. So, if Dems change in their popular vote by 10%, then we simply subtract 10% of each districts' vote shares.

Unfortunately, this returns us to the same issue noted before about the difference between *average district voteshare* and *party popular vote*. This distinction leads to some rather finnicky issues with assuming uniform partisan swing. For example, a change of 10% in a district with *many* voters will result in a larger bump to the popular vote than a 10% change to a small district. Thus, it becomes more common to examine a party's average voteshare across districts, rather than the party's popular vote. Further, it again becomes common to assume that, when vote shares are expected to go below $0$ or above $100%$, the vote shares stop changing for that district. Complexities remain about *uncontested* districts, and we'll treat those later.

Thinking about a *seats-votes* curve estimated using uniform partisan swing, we can build one in a pretty direct fashion. We'll focus first on building the seats-votes curve under uniform swing around a point we *did observe:* a single election. Let's pick out an election in particular, the 2006 election:

```python
house06 = house.query('year == 2006')
```

Since this is only a single election, we only get one single observation of the average Democrat vote share, and one observation of how many districts the Dems won in that election. Below, we'll plot this as a single point, marked as a red `x`:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot(house06.vote_share.mean(), (house06.vote_share > .5).mean(), marker='x', color='r')
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

Now, uniform partisan swing basically works by asking:

> if a party were to win $\delta$% more in each district, how many districts that party win?

By asking that question for a few different values of $\delta$, we can build up a sequence of hypothetical district average vote shares which show the underlying seat-vote relationship latent in our data, *assuming (of course)* that the uniform swing hypothesis holds as a reasonable model of how votes in districts change as the average district vote changes.

So, let's see what happens when Democrats win 1% more in each district. Below, we'll add `.01` to each observation, and record the number of seats Dems would have won if each district increases in Democrat vote share by exactly one percentage point:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.plot((house06.vote_share + .01).mean(),
        ((house06.vote_share + .01) > .5).mean(), marker='x', color='k')
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

Wow! there's a very small increase in the share of seats Dems win when every district's voteshare increases by 1%. This suggests that one or more districts $i$ have a vote share $v_i$ such that $v_i < .5$ but $v_i + .01 > .5$. Thus, when all districts increase in their Democrat vote share by a single percentage point, the districts flip from being won by Republicans to being won by Democrats.

Given that we can do this with an increase towards *Democrats*, we can also *decrease* all Democrat vote shares by a single percentage point and model a uniform partisan swing towards Republicans:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.plot((house06.vote_share + .01).mean(),
        ((house06.vote_share + .01) > .5).mean(), marker='x', color='grey')
ax.plot((house06.vote_share - .01).mean(),
        ((house06.vote_share - .01) > .5).mean(), marker='x', color='k')
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

More districts flip! This suggests that some district is won by Democrats with a margin smaller than 1%.

Again, contingent on the uniform partisan swing assumption being realistic, we can do this over and over again for different values of $\delta$ and obtain the "seats-votes curve" for that election:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot((house06.vote_share.mean() + numpy.linspace(-.25, .25, num=20)),
        [((house06.vote_share + delta) > .5).mean() for delta
         in numpy.linspace(-.25,.25,num=20)], marker='x', color='grey')
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

The curve above is the "empirical" seats votes curve, like that implemented by Simon Jackman in the `pscl` library in R. This "empirical" seats votes curve requires the uniform partisan swing assumption to construct; each point on the curve is given by some swing $\delta$ away from the observed average party vote share $\bar{v}$ and the fraction of districts at swing $\delta$ that are won by Democrats $(v_i + \delta > .5)$. Letting $\mathcal{I}$ stand for the indicator function which is 1 when the statement inside of it is true and zero otherwise, this means every point on the empirical seats-votes curve is a coordinate:

$$ \left(\bar{v}, \sum_i^N \mathcal{I}\left( (v_i + \delta) > .5\right)n^{-1}\right)$$

And the functional form of the "empirical" seats-votes curve is:

$$f(\delta) = \sum_i^N \mathcal{I}\left((v_i + \delta) > .5\right)n^{-1} $$

Looking at this definition, a few things become apparent. First, this method practically ignores districts that go way above $1$ or below $0$; while these are impossible (no district can have more Republican votes than there are voters in general), the seats-votes curve only considers districts relative to the threshold of victory, $.5$. This means that some vote shares from $v_i + \delta$ may actually be invalid vote shares. Second, while many doubt whether or not this model is a *realistic* model of how the electorate behaves, it's still suggested as a good first approximation for how the electoral system actually works, especially in the small region around the observed voteshare (e.g. Jackman 2014). There is absolutely *no* uncertainty in this model of the seats-votes curve, however. Further, assuming that the average swing affects all districts equally can be patently unrealistic when thinking about elections as social or geographical processes. Finally, we can see that the empirical seats-votes curve is actually *much* bumpier than the plot above suggests: there is a finite number ($N$) of observed $v_i$, but $\delta$ changes continuously. This means that the $f(\delta)$ function changes in a stepwise fashion: as $\delta$ increases, some $v_i + \delta$ switches from being below $.5$ to being above $.5$; when this happens, the function increases by a single step $1/n$. In general, if districts are allowed to have *idential* $v_i$, the changes may only occur in multiples of $1/n$, but these remain *integral* changes.

Indeed, this step change property belies a much more important fact: this "model" of the seats-votes curve built from this restrictive "uniform partisan swing" assumption is actually a much more basic expression of the empirical structure of the data. Nagle (2018) provides a very good discussion of this; the seats-votes curve (when constructed this way) contains the same information as the empirical cumulative distribution function for $v_i$.

This means that no theory of partisan swing is necessary to construct this curve at all. We'll discuss this now.

# Seats-Votes Curves as an Empirical CDF

Going back to our first observed point from the 2006 election, we'll zoom in really tight on the observed result.

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot(house06.vote_share.mean(), (house06.vote_share > .5).mean(), marker='x', color='r')
ax.axis((.55,.58, .52,.55))
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

Now, we've discussed that *really small* changes in $\delta$ should reveal the integrality property. So, instead of changing $\delta$ by $.01$, we'll change $\delta$ **really finely** (like, $.00004$) near our observed value. This reveals the stepwise structure:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot((house06.vote_share.mean() + numpy.linspace(-.01, .01, num=500)),
        [((house06.vote_share + delta) > .5).mean()
         for delta in numpy.linspace(-.01,.01,num=500)],
        marker='x', color='grey')
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.axis((.55,.58, .52,.55))
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

Now we can see the stepwise structure. Again, this happens because seats are either *won* or they're *lost* by a party; no fractional seats are possible, so the seats-votes curve always assigns a discrete number of "wins" to Democrats. If there are 435 seats, the "estimated" seats-votes curve can only change $1/435$ at any step. Sometimes, the seats-votes curve can change more than $1/435$ at a time, like if two districts tie exactly in their vote shares. But, it can never change less than $1/435$ due to integrality. If we visualize this as light small lines in the plot we can see this effect very clearly:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
ax.plot((house06.vote_share.mean() + numpy.linspace(-.01, .01, num=500)),
        [((house06.vote_share + delta) > .5).mean() for delta in numpy.linspace(-.01,.01,num=500)], marker='x', color='grey')
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.hlines((numpy.arange(1,435)/435), 0,1, color='k', linewidth=.15)
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
ax.axis((.55,.58, .52,.55))
plt.show()
```

Each of the horizontal lines is a change in exactly one seat. Note that two seats tie for the voteshares right around $.565$; we can see this by noting that the two district vote share percentages are so close to one another, they fall through the grid of values together. But, unless something ties *exactly*, the seats-votes curve only ever changes by a single seat at each percent vote value, and if the grid of $\delta$ became even finer, the curve would only increase exactly $1/435$ at a time.

To complete the last bit of relationship between the seats-votes curve and a cumulative density function, we need to obtain an expression that relates the two kinds of curves.

So, can see that when a party popular vote share is exactly $\bar{v}$, then the seats-votes curve will be the count where $v_i>.5$. If we're *instead* at some swung vote share $\bar{v} + \delta$, the height of the seats-votes curve will be the percent of seats won at that *swung* vote share ($v_i + \delta > .5$). Moving to the *empirical cumulative density function* (ECDF), the height of the ECDF at an arbitrary value $x$ is the percent of all observations less than or equal to that value ($v_i < x$). Arbitrarily, let $x = \bar{v} - \delta$, we can see that the criteria for the ECDF is *identical to* the within-election seats-votes curve, shifted by a factor, $.5 - \bar{v}$:

$$
\begin{align}
v_i &< x \\
v_i &< \bar{v} - \delta \\
v_i &< \bar{v} - \delta + (.5 - .5) \\
v_i + \delta + .5- \bar{v} &< .5 \\
v_i + \delta + (.5- \bar{v}) &< .5
\end{align}
$$

This holds for any $x$ in the domain of the cumulative density function, since $\delta$ is a free parameter and $\bar{v}$ is observed.

The $.5 - \bar{v}$ factor isn't totally arbitrary: it's the *aggregate margin*, or the difference between the between the party's average voteshare in districts and $.5$, which is the threshold the party would need to control the legislature. Second, note that we've defined $x = \bar{v} - \delta$. This "flips" the direction of the seats-votes curve, since swings of size $\delta$ in the seats-votes curve are swings of size $-\delta$ in the ECDF. For a more rigorous discussion, again see my dissertation (Wolf, 2017) or Nagle (2018).
In practice, this means that the ECDF *for Republicans* and the seats-votes curve *for Democrats* are simply shifted & scaled versions of the same curve. Since we can construct the ECDF without reference to a theory of partisan swing, this suggests the "empirical" seats-votes curve is sufficiently more general than it might appear at first glance.

As far as our ability to compute seats-votes curves is concerned, this means we can use all of the standard methods for computing histograms to "estimate" empirical seats-votes curves under the uniform partisan swing assumption. Below, I'll show the empirical CDF and swing-built seats-votes curve from above, zoomed in:

```python
f = plt.figure()
ax = plt.gca()
ax.axis((0,1,0,1))
shift = .5 - house06.vote_share.mean()
ax.plot((house06.vote_share.mean() + numpy.linspace(-.10, .10, num=1000)),
        [((house06.vote_share + delta) > .5).mean()
         for delta in numpy.linspace(-.10,.10,num=1000)],
        marker='x', color='grey', label='Uniform Swing')
ax.plot(house06.vote_share.mean(),
        (house06.vote_share > .5).mean(), marker='x', color='r')
ax.hist(1-house06.vote_share - shift, cumulative=True, histtype='step',
        density=True, bins=9000, color='red', zorder=100,
        label='Republican Shifted ECDF')
ax.hist(1-house06.vote_share, cumulative=True, histtype='step',
        density=True, bins=9000, color='purple', zorder=100,
        label='Republican ECDF')
ax.hlines((numpy.arange(1,436)/435), 0,1, color='k', linewidth=.15)
ax.arrow(.475,.501,-shift, 0, color='b',
          length_includes_head=True, zorder=100, label='Shift')
ax.hlines(1000,10001,1000, color='b', label='Shift') # because the shift label is disappearing
ax.legend(loc='center right', bbox_to_anchor=(2,.5), fontsize=20, frameon=False)
ax.axis((.46, .59, .48,.58))
ax.set_title("Seats-Votes & ECDF Equivalence")
ax.set_xlabel("Average District Pct. Dem Vote")
ax.set_ylabel("Pct. of Congress Won by Dems")
plt.show()
```

We can do this for the nation for each year to get a collection of seats-votes curves that apply to each year, or use *all* seats across the elections to build one collected seats-votes curve. In practice, when we have multiple seats-votes curves, it's usually best to visualize each separately, and consider the *consensus* curve to to be the median among the many different replications. We'll see this strategy in Gelman & King (1994), as well as McGann et al. (2015) and Wolf (2018).

Below, though, I'll plot seats-votes curves using the shifted Republican ECDF for each election since 1994.

```python
for year,chunk in house.groupby('year'):
    shift = .5 - chunk.vote_share.mean()
    plt.hist(numpy.clip(1-chunk.vote_share.dropna().values - shift, 0,1),
             cumulative=True, density=True, histtype='step',
             color='k', bins=100, alpha=.2, zorder=1000,
             label='Single Year' if year==2006 else None)
plt.scatter(house.groupby('year').vote_share.mean(),
            house.groupby('year').vote_share.apply(lambda x: (x > .5).mean()),
            label='Observed District\nAverages', marker='x', color='b')
plt.hist(numpy.clip((1-house.dropna(subset=['vote_share']).vote_share.values)
                    - (.5 - house.vote_share.mean()), 0,1),
         cumulative=True, density=True, histtype='step',
         color='orangered', bins=100, linewidth=3, label='Pooled')
plt.legend(loc='upper left', fontsize=14)
plt.xlabel("Average District %Dem")
plt.ylabel("Percent of Legislature Won")
plt.show()
```

# Is the seats-votes curve realistic?

The question of whether or not a seats-votes curve is *realistic* is quite distinct from *whether or not we can estimate it*. Again, following Nagle (2018), we can clearly "estimate" seats-votes curves for any number of replications from a model of elections. We can do this quickly, easily, and in a pretty straight-forward fashion. I'll talk a lot later about how there may be many different ways of constructing *stochastic* estimates of seats-votes curves, those that take into account the various random and co-varying factors that are involved in electoral processes. But, at its core, the question of whether or not we can estimate seats-votes curves easily is solved using Nagle (2018)'s relization about empirical CDFs and seats-votes curves.

Their *realism* depends, though, on their interpretation. Interpreting seats-votes curves as empirical CDFs is not problematic at all. However, interpreting them *as a model* of how congressional control might change as district average vote shares change... that is problematic. In general, this interpretation hinges on how realistic the assumption of uniform partisan swing is.

While we may take the uniform partisan swing curves as a good first-approximation for how the electoral system will work, there is a common perception that extremes of the seats-votes curves, the points that are far away from an observed $(\bar{v},\bar{s})$ point, are not "realible" representations of reality. In my own dissertation research, I found that policymakers, public officials, and stakeholders in Washington and and Arizona did believe that the whole range of $\bar{v}$ to $1-\bar{v}$ were possible, but that it was probably more likely that the structure of partisanship in the state would not change radically or flip in the control of the congressional delegation. This suggests that, for practitioners I interviewed, seats-votes curves may indeed have a narrow band of validity around the observed election results, and this band may encompass values where a party both wins or loses. This doesn't mean that the *estimation* of the curve is unrealible, but it *does* mean the model becomes unrealistic at large swings.

However, another component of the assumption of uniform partisan swing that I examined in my dissertation considered whether the assumption of *uniformity* makes sense *geographically*. That is, do nearby districts tend to swing the same direction (or, more-so than they may otherwise if swing were uniformly random)? It may be the case that small but *highly correlated* swings will behave differently from small uncorrelated swings. If *this* is true, it suggests that assuming that *all districts accrue the same changes* might be unrealistic, and this unrealism may affect the validity of the seats-votes curve as a conceptual model.

To examine this more in depth, we can see the votes over the 2002-2010 elections to Congress.

```python
aughties = house.query('year > 2000 & year < 2012')
```

Now, admitting that there are a few inter-censal redistrictings during this period, I'll assume that the redistricting map remains *mainly* unchanged, and that it is reasonable to compute swing between two districts in subsequent elections so long as they have the same district number.

```python
aughties['district_id'] = aughties.state_fips.astype(str).str.rjust(2,'0') \
                        + aughties.lewis_dist.astype(str).str.rjust(2, '0')
```

    /home/lw17329/anaconda/envs/ana/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

Now, I'm going to pivot the dataframe so that each row is a congressional district and each column is a year in which a vote share is recorded.

```python
pivot_table = aughties.pivot(index='district_id',
                               columns='year',
                               values='vote_share')
pivot_table.head()
```

      year
      2002
      2004
      2006
      2008
      2010

      district_id

      0101
      0.384417
      0.368377
      0.318440
      0.000000
      0.000000

      0102
      0.300415
      0.284929
      0.304612
      0.503119
      0.489062

      0103
      0.489307
      0.387705
      0.392786
      0.459005
      0.405193

      0104
      0.000000
      0.251688
      0.297387
      0.251417
      0.000000

      0105
      0.747845
      0.730523
      1.000000
      0.518012
      0.420643

Then, we'll compute the `diff` along columns and drop the first (since it has nothing to diff against):

```python
swings = pivot_table.diff(axis=1).drop(2002, axis=1)
swings.head()
```

      year
      2004
      2006
      2008
      2010

      district_id

      0101
      -0.016039
      -0.049937
      -0.318440
      0.000000

      0102
      -0.015486
      0.019683
      0.198507
      -0.014057

      0103
      -0.101602
      0.005081
      0.066219
      -0.053811

      0104
      0.251688
      0.045700
      -0.045970
      -0.251417

      0105
      -0.017322
      0.269477
      -0.481988
      -0.097368

We can see the swing distributions in each year:

```python
swings.hist(bins=40)
```

Now, swing is *quite* noisy when considering both contested and uncontested elections together. We see that, in general, there appear to be three modes in these distributions. These tend to be where elections change between contested and uncontested across two elections. If we focus in on the cases where elections are contested, we'll have a bit of a cleaner time when we move to the map. Thus, let's focus on the middle 80% first, and return later to the problem of uncontested elections.

```python
midswing = aughties.query('vote_share < .9 & vote_share > .1')\
                    .pivot(index='district_id',
                           columns='year',
                           values='vote_share')\
                    .diff(axis=1).drop(2002,axis=1)
midswing.hist(bins=40)
plt.tight_layout()
plt.show()
```

Given this drop of uncontested elections, contests where *either* year is uncontested result in null swing. But, looking generally at a map of the swings (here, we'll focus on 2004), we can see that it does indeed appear to cluster in space:

```python
spswing = aughties.merge(midswing[2004].to_frame('swing'),
                         left_on='district_id',
                         right_index=True,how='left')
spswing = spswing.query('state_name not in ("alaska", "hawaii")')
```

```python
ax = spswing[~spswing.swing.isnull()].plot('swing', cmap='RdYlBu')
spswing[spswing.swing.isnull()].plot(color='lightgrey', edgecolor='k',
                                     linewidth=.1, zorder=-1,ax=ax)
```

Given that we've focused only on contested elections in both cycles, it appears that the changes in vote share between years *does* tend to be spatially dependent. To verify this, we can start by using Moran's $I$, which is kind of like a spatial correlation coefficient. This will tell us how correlated a given congressional district's swing is with other nearby districts. Over all districts, this will tell us whether the *swing* at any site tends to be related to swing in nearby sites. If districts that are near each other tend to experience the same swing, then it is not really the case that swing exists uniformly at random across contested elections. Instead, there are *correlated swings*: areas that are near each other tend to swing together.

```python
Is = []
for y in numpy.arange(2004,2012,2):
    spswing = aughties.merge(midswing[y].to_frame('swing'),
                         left_on='district_id',
                         right_index=True,how='left')
    spswing = spswing.query('state_name not in ("alaska", "hawaii")')
    spswing = spswing.dropna(subset=['swing'])
    W = pysal.weights.Queen.from_dataframe(spswing)
    Is.append(pysal.Moran(spswing.swing.values, W))
```

We can visualize this in on of two ways. First, we can simply look at the estimate of Moran's $I$ and its standard error in each year:

```python
plt.errorbar(numpy.arange(2004,2012,2),
             [mi.I for mi in Is],
             [2*(mi.VI_rand**.5) for mi in Is],
             fmt='none', color='k')
plt.scatter(numpy.arange(2004,2012,2),
             [mi.I for mi in Is], marker='x', color='r', zorder=100)
plt.xticks(numpy.arange(2004,2012,2))
seaborn.despine()
plt.xlabel("Moran's I for queen congressional district swings")
```

Or, we can look at the scatterplot relating the swing in a contested election to the swing in neighboring districts in each year:

```python
f,ax = plt.subplots(2,2,figsize=(8,8))
for i,year in enumerate(numpy.arange(2004,2012,2)):
    ax_ = ax.flatten()[i]
    seaborn.regplot(Is[i].z,
                    pysal.weights.lag_spatial(Is[i].w, Is[i].z),
                    ax=ax_, scatter_kws=dict(color='k', marker='.', s=4),
                    line_kws=dict(color='r'))
    ax_.hlines(0,*plt.xlim(), linestyle='--', color='grey')
    ax_.vlines(0,*plt.ylim(), linestyle='--', color='grey')
    ax_.plot((-.1,.1), (-.1,.1), color='skyblue', linestyle='--')
    ax_.set_title("Moran Plot for {}".format(year))
    ax_.set_xlabel("Swing")
f.tight_layout()
plt.show()
```

Thus, we see clearly that swing is *not* uniformly random over space; areas that are near each other tend to break towards (or away from) candidates in a simlar fashion.

notebooks/01 - Seat-Vote, Rank-Vote, & Single Elections.ipynb


```python
import numpy as np
from sys import exit
from functools import reduce
import sympy
from sympy.matrices import Matrix, eye, zeros, ones, diag, GramSchmidt
```

```python
def legendre_symbol(a, n):
        assert(n > a > 0 and n%2 == 1)
        t = 1
        while a != 0:
            while a % 2 == 0:
                a /= 2
                r = n % 8
                if r == 3 or r == 5:
                    t = -t
            a, n = n, a
            if a % 4 == n % 4 == 3:
                t = -t
            a %= n
        if n == 1:
            return t
        else:
            return 0

def chinese_remainder(n, a):
    sum = 0
    prod = reduce(lambda a, b: a*b, n)
    for n_i, a_i in zip(n, a):
        p = prod // n_i
        sum += a_i * mul_inv(p, n_i) * p
    return sum % prod

def mul_inv(a, b):
    b0 = b
    x0, x1 = 0, 1
    if b == 1: return 1
    while a > 1:
        q = a // b
        a, b = b, a%b
        x0, x1 = x1 - q * x0, x0
    if x1 < 0: x1 += b0
    return x1

def stringToAscii(s):
    return [ord(c) for c in s]

def stringToBinary(s):
    return [bin(ord(c)) for c in s]
```

```python
msg = "Máté"
key = 580
print(msg)
asciiList = stringToAscii(msg)
binaryList = stringToBinary(msg)
print(binaryList)
k = ""
for j in binaryList:
    k+=j[2:]
print(list(k))
squareFreeCores = []
for j in range(key,key+len(k)):
    factors = sympy.primefactors(j)
    squareFreeFactorization = []
    for l in factors:
        helper = j
        count = 0
        while helper%l==0:
            helper/=l
            count+=1
        if count%2==1:
            squareFreeFactorization.append(l)
    squareFreeCores+=squareFreeFactorization
primes = list(set(squareFreeCores))
primes.sort()
print(primes)
a=len(primes)
b=len(k)
print(a,b)
A = np.zeros((b,a))
for x in range(key,key+b):
    for y in primes:
        z = x
        cnt = 0
        while z % y == 0:
            z/=y
            cnt+=1
        if cnt%2 == 1:
            A[x-key,primes.index(y)]=int(1)
print(A)
print("Solution exists",np.linalg.det(A)) ## We need a non-zero determinant!
msgBin = []
for x in k:
    msgBin.append(int(x))
print(msgBin)
m = np.array(msgBin)
print(m)
A_inv = np.linalg.inv(A) ## for some reason, this seems to be super slow
print(A_inv)
solution = A_inv.dot(m)
print(solution)
print(np.linalg.solve(A,m))
## Once the solution vector for the residues of the primes is established, then we just need to apply
## the Chinese Remainder Theorem for getting the final prime modulus (see paper)

```

    Máté
    ['0b1001101', '0b11100001', '0b1110100', '0b11101001']
    ['1', '0', '0', '1', '1', '0', '1', '1', '1', '1', '0', '0', '0', '0', '1', '1', '1', '1', '0', '1', '0', '0', '1', '1', '1', '0', '1', '0', '0', '1']
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 43, 53, 59, 67, 73, 83, 97, 101, 149, 151, 197, 199, 293, 587, 593, 599, 601, 607]
    30 30
    [[0. 0. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      1. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 1. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [1. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 1. 0. 0. 0.]
     [1. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 1. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.
      0. 0. 0. 0. 0. 0.]
     [1. 0. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 1. 0. 0.]
     [1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 1. 0.]
     [1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 1.]
     [1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]
     [0. 1. 0. 1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
      0. 0. 0. 0. 0. 0.]]
    Solution exists -1.0
    [1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1]
    [1 0 0 1 1 0 1 1 1 1 0 0 0 0 1 1 1 1 0 1 0 0 1 1 1 0 1 0 0 1]
    [[ 0.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [-0. -0. -0. -0. -0. -0. -0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -0. -0.
      -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]
     [-1.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  1.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.
       0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  0.  0.]
     [ 1.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  1.  0.  0.
       0.  0.  0.  0.  0.  0.  0. -2.  0.  0.  0. -1.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  1.  0.]
     [ 0.  0.  0.  0.  0. -1.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       1.  0. -1.  0.  0.  0.  0.  1.  0.  0.  0.  0.]
     [ 1. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0.
      -0. -0. -0. -0. -0. -0. -0. -1. -0. -0. -0. -0.]
     [-0. -0. -0. -0. -0. -0. -0. -0. -1.  1. -0. -0. -0. -0. -0. -0. -0. -0.
      -0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -1. -0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 1.  0.  0.  0.  0.  0.  0.  0.  2.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0. -1.  0.  1.  0.  0. -1.  0.  0.  0. -1.]
     [-0. -0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -1. -0. -0. -0.
      -0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -0. -0.]
     [-0. -0. -0. -0. -0. -0. -0. -0.  1. -0.  1. -0. -0. -0. -0. -0. -0. -0.
      -0. -0. -1. -0. -0. -0. -0. -1. -0. -0. -0. -0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  1.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 1.  1. -0. -0. -0. -0. -0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -0. -0.
      -0. -0. -0. -0. -0. -0. -0. -1. -0. -0. -0. -1.]
     [-0. -0.  1. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0. -0.
      -0. -0. -1. -0. -0. -0. -0. -0. -0. -0. -0. -0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0. -1.  0.  0.  0.  0.  0.  1.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  1.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  1.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  1.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0. -1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
       0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]]
    [-1.  1.  0. -1.  1.  0.  2.  1.  1.  1.  0.  0.  3.  0.  1.  1.  2.  1.
      0.  1.  1.  1. -1.  0.  2.  1.  0.  1.  0.  0.]
    [-1.  1.  0. -1.  1.  0.  2.  1.  1.  1. -0.  0.  3. -0.  1.  1.  2.  1.
     -0.  1.  1.  1. -1.  0.  2.  1.  0.  1.  0.  0.]

```python
for i in range(len(solution)):
    solution[i]=int(solution[i])%2
print(solution)
moduli = [1]
for i in range(1,len(primes)):
    for j in range(1,primes[i]):
        if solution[i]==1 and legendre_symbol(j,primes[i])==-1:
            moduli.append(j)
            break
        if solution[i]==0 and legendre_symbol(j,primes[i])==1:
            moduli.append(j)
            break
print(primes)
print(moduli)
print(len(primes),len(moduli))
modulus=chinese_remainder(primes,moduli)
print(modulus)
## let's suppose that p = 1 mod 4
```

    [1. 1. 0. 1. 1. 0. 0. 1. 1. 1. 0. 0. 1. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 0.
     0. 1. 0. 1. 0. 0.]
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 43, 53, 59, 67, 73, 83, 97, 101, 149, 151, 197, 199, 293, 587, 593, 599, 601, 607]
    [1, 2, 1, 3, 2, 1, 1, 2, 5, 2, 1, 1, 2, 1, 2, 2, 1, 2, 1, 2, 2, 3, 2, 1, 1, 2, 1, 7, 1, 1]
    30 30
    7419833483365953441636195213120205307487092525147381

```python
realBits = []
for r in range(key,key+len(msgBin)):
    realBits.append(legendre_symbol(r,modulus)%2)
print("Bug is fixed:",realBits==msgBin)

```

    Bug is fixed: False

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  The presentation below largely follows part II in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Solution to ODE Initial Value Problems - Part 2

# Overview -- so far

So far we have discussed 3 basic techniques for integration of ODE IVP's

* **Single-Step Multi-Stage** schemes (explicit and implicit)
* **Taylor's Series** Methods
* **Linear Multi-Step** Schemes (explicit and implicit)

as well as
* **truncation error** of each method (and it's relation to step-error)
* **adaptive stepping**

# Overview -- to do

This notebook will take the analysis of these methods to discuss the other important issues of
* **Convergence** (Global Error) -- does the discrete solution converge to the continuous analytic solution?
* **Stability** -- Are there limits to how large a time-step we can take?
* **Stiff Equations**:  consequences of stability

# Convergence

We can think of an ODE method as providing a sequence of approximations $U_N$ where here $N$ is the number of time steps needed to reach the final time of interest $t_f$, in effect we are increasing the resolution of the method.  We then say that a method is convergent if this sequence converges to the true solution at the same time
$$
    \lim_{N\rightarrow \infty} U_N = u(t_f).
$$

We can also define this in a more familiar way in terms of $\Delta t$ such that
$$
    N \Delta t = t_f ~~~~ \Rightarrow ~~~~ N = \frac{t_f}{\Delta t}
$$
so that our definition of convergence becomes
$$
    \lim_{\Delta t \rightarrow 0} U_{\Delta t} = u(t_f).
$$

In general for a method to be convergent it must be
 - **consistent** which as before meant that the local truncation error $T = \mathcal{O}(\Delta t^p)$ where $p > 0$,
 - **zero-stable** which implies that the sum total of the errors as $\Delta t \rightarrow 0$ is bounded and has the same order as $T$ (the truncation error) which we know goes to zero as $\Delta t \rightarrow 0$.

put another way,  a method is convergent if the **global error**

$$
    E = | U_N(t_f) - u(t_f)| \rightarrow 0\quad\mathrm{as}\quad N\rightarrow\infty
$$

### Example:  Forward Euler on a Linear Problem

Consider the simple linear problem
$$
    \frac{\text{d}u}{\text{d}t} = \lambda u \quad \text{with} \quad u(0) = u_0
$$
which we know has the solution $u(t) = u_0 e^{\lambda t}$.  Applying Euler's method to this problem leads to
$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t\lambda U_n \\
            &= (1 + \Delta t \lambda) U_n
\end{aligned}$$

We also know the local truncation error is defined by
$$\begin{aligned}
    T_n &= \frac{1}{\Delta t} \left[ U_{n+1} - u_{n+1} \right ] \\
        &= \frac{1}{\Delta t} \left[ (1 + \lambda \Delta t) u_n - u_{n+1} \right ]
\end{aligned}$$
which can be rearranged to find

$$
    u_{n+1} = (1 + \Delta t \lambda) u_n - \Delta t T_n.
$$

Note that all values here are in terms of the exact solution where as the application of Euler's method is in terms of the approximate solution.

Defining the global error as
$$
    E_{n} = u_n - U_n
$$

we can subtract the last two expressions to find

$$\begin{aligned}
    u_{n+1} - U_{n+1} &= \underbrace{(1 + \Delta t \lambda) u_n - \Delta t T_n}_{u_{n+1}} - \underbrace{(1 + \Delta t \lambda) U_n}_{U_{n+1}} \\
    E_{n+1} &= (1 + \Delta t \lambda) (u_n - U_n) - \Delta t T_n \\
            &= (1 + \Delta t \lambda) E_n - \Delta t T_n,
\end{aligned}$$

a recursive definition for the global error that connects it to the local truncation error.

Working backwards from the global error $E_n$ at maximum time step $n = t_f/\Delta t$,  the first few iterates of this look like

$$\begin{align}
    E_n &= (1 + \lambda \Delta t) E_{n - 1} - \Delta t T_{n-1} \\
    &= (1 + \lambda \Delta t) \left [(1 + \lambda \Delta t) E_{n - 2} - \Delta t T_{n-2} \right ] - \Delta t T_{n-1} \\
    &= (1 + \lambda \Delta t) \left [(1 + \lambda \Delta t) \left \{(1 + \lambda \Delta t) E_{n - 3} - \Delta t T_{n-3} \right \} - \Delta t T_{n-2} \right ] - \Delta t T_{n-1}
\end{align}$$

Which at this point looks like
$$
    E_n = (1 + \lambda \Delta t)^3E_{n-3} - \Delta t\sum_{i=1}^3 (1+\Delta t\lambda)^{i-1}T_{n-i}
$$

So taking it back $n$ steps ($3\rightarrow n$) gives
Which at this point looks like
$$
    E_n = (1 + \lambda \Delta t)^nE_0 - \Delta t\sum_{i=1}^n (1+\Delta t\lambda)^{i-1}T_{n-i}
$$

Expanding this expression out backwards in time to $n=0$ leads to
$$
    E_n = (1 + \Delta t \lambda)^n E_0 - \Delta t \sum^n_{i=1} (1 + \Delta t \lambda)^{i-1} T_{n - i}.
$$

Noting that $(1 + \Delta t \lambda)$ are the first two terms in the Taylor series for $e^{\lambda\Delta t}$ we will bound this term by

$$
    e^{\Delta t \lambda} \geq |1 + \Delta t \lambda|
$$

which then implies the term in the summation can be bounded by

$$
    |1 + \Delta t \lambda|^{i - 1} \leq e^{(n-1) \Delta t |\lambda|} \leq e^{n \Delta t |\lambda|} \leq e^{|\lambda| t_f}
$$

Using this expression in the expression for the global error we find
$$\begin{aligned}
    E_n &= (1 + \Delta t \lambda)^n E_0 - \Delta t \sum^n_{i=1} (1 + \Delta t \lambda)^{i-1} T_{n-i} \\
    |E_n| &\leq e^{|\lambda| \Delta t n} |E_0| + \Delta t \sum^n_{i=1} e^{|\lambda| t_f} |T_{n-i}| \\
          &\leq e^{|\lambda| t_f} \left(|E_0| + \Delta t \sum^n_{i=1} |T_{n - i}|\right) \\
          &\leq e^{|\lambda| t_f} \left(|E_0| + n \Delta t \max_{1 \leq i \leq n} |T_{n - i}|\right)
\end{aligned}$$

In other words the global error is bounded by the original global error and the maximum one-step error made multiplied by the number of time steps taken.  If $N = \frac{t_f}{\Delta t}$ as before and taking into account the local truncation error we can simplify this expression further to

$$
    |E_n| \leq e^{|\lambda| t_f} \left[|E_0| + t_f \left(\frac{1}{2} \Delta t |u''| + \mathcal{O}(\Delta t^2)\right ) \right]
$$

If we assume that we have used the correct initial condition $u_0$ then $E_0 \rightarrow 0$ as $\Delta t \rightarrow 0$ and we see that the global error $E_n$ is bounded by $\mathcal{O}(\Delta t)$, the same as the local truncation error:
$$
    |E_n| \leq e^{|\lambda| t_f} t_f \left(\frac{1}{2} \Delta t |u''| + \mathcal{O}(\Delta t^2)\right ) = \mathcal{O}(\Delta t).
$$

More generally,  it can be shown that for all single-step multi-stage schemes (of which Euler is the simplest), the Global error has the same order as the Truncation error, and therefore all of these schemes converge.

## Absolute Stability
Although zero-stability guarantees stability it is much more difficult to work with in general as the limit $\Delta t \rightarrow 0$ can be difficult to compute (and in general isn't ideal as it requires an infinite amount of computation).  Instead we often consider a finite $\Delta t$ and examine if the method is stable for this particular choice of $\Delta t$.  This has the practical upside that it will also tell us what particular $\Delta t$ will ensure that our method is indeed stable.

### Example
Consider the problem
$$
    u'(t) = \lambda (u - \cos t) - \sin t ~~~~ \text{with} ~~~~ u(0) = 1
$$

whose exact solution is
$$
    u(t) = \cos t.
$$

We can compute an estimate for what $\Delta t$ we need to use by examining the truncation error for Euler's method

$$\begin{aligned}
    T &= \frac{1}{2} \Delta t u''(t) + \mathcal{O}(\Delta t^2) \\
      &= -\frac{1}{2} \Delta t \cos t + \mathcal{O}(\Delta t^2)
\end{aligned}$$

and therefore
$$
    |E_n| \leq \Delta t \max_{0 \leq t \leq t_f} |\cos t| = \Delta t.
$$

If we want a solution where $|E_n| < 10^{-3}$ then $\Delta t \approx 10^{-3}$.  Turning to the application of Euler's method lets apply this to the case where $\lambda = -10$ and $\lambda = -2100$.

```python
# Implement vectorized Forward Euler
def euler(f, t_span, u0, N):
    """ simple implementation of constant step-size forward euler method for vector valued f
        This doc string should have so much more in it
    """
    t = numpy.linspace(t_span[0], t_span[1],N)
    if numpy.isscalar(u0):
        u = numpy.empty(t.shape)
    else:
        u = numpy.empty((len(t),len(u0)))
    u[0] = u0
    delta_t = t[1] - t[0]
    for (n, t_n) in enumerate(t[:-1]):
        K1 = delta_t * f(t_n, u[n])
        u[n + 1] = u[n] + K1
    return t, u.transpose()
```

```python
# Compare accuracy between Euler for different values of lambda
f = lambda t, u, lam: lam * (u - numpy.cos(t)) - numpy.sin(t)
u_exact = lambda t: numpy.cos(t)
u_0 = u_exact(0.)

t_span = [0., 2.]
num_steps = [2**n for n in range(4, 9)]
```

```python
#num_steps = [2**n for n in range(15,20)]
delta_t = numpy.empty(len(num_steps))
error_10 = numpy.empty(len(num_steps))
error_2100 = numpy.empty(len(num_steps))

t_f = t_span[1]
u_f = u_exact(t_f)

for (i, N) in enumerate(num_steps):
    # Compute Euler solution
    ff = lambda t, u: f(t, u, -10.)
    t, U_10 = euler(ff, t_span, u_0, N)
    delta_t[i] = t[1] - t[0]
    error_10[i] = numpy.abs(U_10[-1] - u_f) / numpy.abs(u_f)
    ff = lambda t, u: f(t, u, -2100.)
    t, U_2100 = euler(ff, t_span, u_0, N)
    error_2100[i] = numpy.abs(U_2100[-1] - u_f) / numpy.abs(u_f)

print('Error: lambda =   10: {}'.format(error_10))
print('Error: lambda = 2100: {}'.format(error_2100))

# Plot error vs. delta_t
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)

axes.loglog(delta_t, error_10, 'bo', label='Forward Euler')

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_10[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")

axes.legend(loc=4)
axes.set_title("Error - $\lambda = 10$")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

axes = fig.add_subplot(1, 2, 2)

axes.loglog(delta_t, error_2100, 'bo', label='Forward Euler')
axes.loglog(delta_t, order_C(delta_t[1], error_2100[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")

axes.set_title("Error - $\lambda = 2100$")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

plt.show()
```

So what went wrong with $\lambda = -2100$, the global error should go as

$$
    E_n = \mathcal{O}(\Delta t)?
$$

If $\Delta t \approx 10^{-3}$ then for the case $\lambda = -10$ the previous **global error** is multiplied by $1+\lambda\Delta t$

$$1 + 10^{-3} \cdot -10 = 0.99$$

which means the contribution from $E^n$ will slowly decrease as we take more time steps.  For the other case we have

$$1 + 10^{-3} \cdot -2100 = -1.1$$

which means that for this $\Delta t$ the error made in previous time steps will grow!  For this not to happen we would have to have $\Delta t < 1 / 2100$ which would lead to convergence again.

### A simpler example
Consider our simplest test problem
$$
    u'(t) = \lambda u, \quad u(0) = u_0
$$

whose exact solution is
$$
    u(t) = u_0 e^{\lambda t}.
$$
which for real $\lambda < 0$ should decay to zero as $t\rightarrow\infty$

### Question?

What is the **largest** step-size $\Delta t$ for an Euler method, such that the discrete solution $U_N\rightarrow 0$ as $N\rightarrow\infty$?

Euler's method on this simplest problem can be written

$$
\begin{align}
    U_{n+1} &= U_n + \Delta t f(U_n) = U_n + \Delta t \lambda U_n\\
            &= (1 + \lambda\Delta t)U_n\\
\end{align}
$$

i.e. at each step the last solution is multiplied by the factor $(1 + \lambda\Delta t)$

After $N$ steps

$$
    U_N = (1 + \lambda\Delta t)^N U_0
$$

Assuming $\lambda < 0$ what is the largest step size $\Delta t$ such that $U_N\rightarrow 0$ as $N\rightarrow\infty$?

Hopefully it's clear that for this scheme to be stable requires

$$
    |1 + \lambda\Delta t| < 1
$$

The key parameter is actually the combination of $\lambda$ and the time step $z=\lambda\Delta t$, so stability requires that

$$
    | 1 + z | < 1
$$
or  𝑧∈[?,?]

```python
plt.figure(figsize=(8,6))
z = numpy.linspace(-3,2,200)
plt.plot(z,numpy.abs(1 + z))
plt.plot(z,numpy.ones(z.shape),'k--')
plt.plot([0.,0],[0.,3.])
plt.xlabel('$z$',fontsize=16)
plt.ylabel('$|1 + z|$',fontsize=16)
plt.grid()
plt.show()

```

Given $z\in[-2,0]$,  the  time step is simply

$$
    \Delta t = \frac{z}{\lambda}
$$
so the maximum time step we can take for $\lambda<0$ is

$$
    \Delta t = -\frac{2}{\lambda} > 0
$$

```python
# Compare accuracy between Euler for different values of time steps and $\lambda = -1$
lam = -1.
f = lambda t, u :  lam * u
u_exact = lambda t: numpy.exp(lam * t)
u_0 = u_exact(0.)

t_span = [0., 10.]

N = 21
N = 11
N = 6
N = 5
t, u = euler(f, t_span, u_0, N)

```

```python
tt = numpy.linspace(0.,t_span[1],100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(tt, u_exact(tt),label='$u_{{exact}}$')
axes.plot(t, u, 'ro-', label='$u_{{euler}}$')
axes.set_xlabel('t')
axes.set_title('$\Delta t = {}$'.format(t[1]-t[0]), fontsize=18)
axes.grid()
axes.legend(loc='best',fontsize=14)
plt.show()

```

### Absolute Stability of the Forward Euler Method for complex $z$

The region $|1 + z| < 1$  is the **region of absolute stability**.  For more general problems $\lambda$ may be complex (stemming from eigenvalues of real problems),  so we generally consider the region of stability on the complex plane.

$$
    | 1 + z | < 1
$$
for $z\in\mathbb{C}$

```python
# Plot the region of absolute stability for Forward Euler
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
axes.set_xlim((-3, 3.0))
axes.set_ylim((-3,3))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler")
plt.show()
```

### Why complex eigenvalues?  And why should we care?

#### A simple example:  The simple harmonic oscillator

The linear harmonic oscillator can be written as a 2nd-order ODE for position $x$ of a mass on a spring
$$
    \frac{d^2 x}{dt^2} + x = 0, \quad x(0)=x_0,\,  \frac{dx}{dt} = 0.
$$
Which has solutions
$$
    x(t) = x_0\cos(t)
$$

Alternatively, we can write this as a 2 dimensional system of first order equations by  defining the velocity $v= dx/dt$ and
$$
\begin{align}
    \frac{dx}{dt} &= v \\
    \frac{dv}{dt} &= -x \\
\end{align}
$$
or more cleanly as
$$
    \frac{d\mathbf{u}}{dt} = A\mathbf{u},\quad\mathbf{u}(0)=\mathbf{u_0}
$$
with
$$
\mathbf{u}=\begin{bmatrix} x\\ v\\\end{bmatrix}\quad A= \begin{bmatrix} 0 & 1 \\ -1 & 0 \\ \end{bmatrix}\quad\mathbf{u}_0=\begin{bmatrix} x_0\\ 0\\\end{bmatrix}
$$

All diagonalizable linear dynamical systems have solutions of the form

$$
    \mathbf{u}(t) = \sum_{i=1}^n c_i e^{\lambda_i t}\mathbf{s}_i
$$

where $\lambda_i$, $\mathbf{s}_i$ are corresponding eigenvalues and eigenvectors of $A$ and each eigenvector satisfies
$$
    \frac{d\mathbf{s}_i}{dt} = \lambda_i\mathbf{s}_i
$$
which is exactly our model problem.

In the case where
$$
A= \begin{bmatrix} 0 & 1 \\ -1 & 0 \\ \end{bmatrix}
$$

The eigenvalues are _ _ _ _ _ _ _ _ ?

$\pm i$

Even with complex Eigenvalues and Eigenvectors, the solutions are real and can be shown to be perfect circles on the phase plane

$$
\begin{align}
    x(t) &= \cos(t)\\
    y(t) &=-\sin(t)\\
\end{align}
$$

However,  the plotting the eigenvalues on the stability diagram show's that Euler's method will be completely unstable for this problem for all time steps $\Delta t$

```python
# Plot the region of absolute stability for Forward Euler
x = numpy.linspace(-2, 2, 11)
v = numpy.linspace(-2, 2, 11)
X,V = numpy.meshgrid(x,v)
t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1,2,1)
axes.quiver(X, V, V, -X)
axes.plot(1.5*numpy.cos(t), -1.5*numpy.sin(t))
axes.plot([-2, 2],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-2, 2],'k--')
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('v',fontsize=16)

axes.set_title('Phase Plane: simple Harmonic oscillator',fontsize=16)
axes.set_aspect('equal')

axes = fig.add_subplot(1, 2, 2)
axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
axes.plot([0., 0.0], [-1., 1.],'rx',markersize=10)
axes.set_xlim((-3, 3.0))
axes.set_ylim((-3,3))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler",fontsize=16)
plt.show()
```

$\pm i$ is outside Forward Euler stability regime. No $\Delta t$ can bring it within stability region.

```python
f = lambda t, u : numpy.array([ u[1], -u[0]])
t_span = [0., 4*numpy.pi]
u_0 = numpy.array([1., 0.])
t, u = euler(f,t_span,u_0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.quiver(X, V, V, -X)
axes.plot(1.*numpy.cos(t), -numpy.sin(t), label = 'true')
axes.plot(u[0],u[1],'r', label = 'Euler method')
axes.plot([-2, 2],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-2, 2],'k--')
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('v',fontsize=16)

axes.set_title("Euler's method: simple Harmonic oscillator", fontsize=16)
axes.set_aspect('equal')
axes.legend()
```

### Absolute Stability of Single-Step Multi-Stage schemes -- The R method

This approach can be applied to **all** Single-Step Multi-Stage schemes including all the Runge-Kutta and Taylor Series methods.  The recipe is straightforward

*  Apply the stepping scheme for one step of the model problem $u' = \lambda u$, $u(0)=u_0$ assuming complex $\lambda$
*  This will result in a discrete approximation

$$
    U(\Delta t) = R(z) U_0
$$

where $z=\lambda\Delta t\in\mathbb{C}$, which will be a discrete approximation to the true solution

$$
    U(\Delta t) = e^{z} U_0
$$
* Absolute stability will require that $|R(z)| < 1$

### Absolute Stability of the backward Euler Method

Now try this on backward Euler.

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

which for our model problem is

$$
    U_{n+1} = U_n + \Delta t\lambda U_{n+1}
$$

or rearranging and solving for $U_{n+1}$ gives

$$
    U_{n+1} = \frac{1}{1-\Delta t\lambda} U_n
$$

so
$$
    R(z) = \frac{1}{1-z}
$$
and absolute stability requires

$$
    |R(z)| \leq 1 \leftrightarrow |1 - z| \geq 1$$

so in fact the stability region encompasses the entire complex plane except for a circle centered at $(1, 0)$ of radius 1 implying that the backward Euler method is in fact stable for any choice of $\Delta t$.

The backward version of Euler's method is defined as

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

If we again consider the test problem from before we find that

$$\begin{aligned}
    0 &= U_{n+1} (1 - \Delta t \lambda) - U_n \\
    &= \xi (1 - z) - 1
\end{aligned}$$

which has the root $\xi = \frac{1}{1 - z}$.  We then have

$$\left|\frac{1}{1-z}\right| \leq 1 \leftrightarrow |1 - z| \geq 1$$

so in fact the stability region encompasses the entire complex plane except for a circle centered at $(1, 0)$ of radius 1 implying that the backward Euler method is in fact stable for any choice of $\Delta t$.

```python
# Plot the region of absolute stability for Backward Euler
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.set_facecolor((255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.fill(numpy.cos(t) + 1.0, numpy.sin(t), 'w')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Backward Euler")
plt.show()
```

For oscillator, Backward Euler is **stable**, but **not convergent** (solution decays, spins inward)

## General Stability Regions for Linear Methods

We can also map out  stability regions for Linear Multi-step methods (although the analysis is slightly more complicated).

All of the Linear Multi-step methods from the previous lecture scan be written
$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j f(t,U_{n+j})
$$

which for our model problem, $f(t,u) = \lambda u$ becomes
$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j \lambda U_{n+j}
$$

or  rearranging and again defining $z=\Delta t\lambda$ becomes

$$
    \sum^r_{j=0} (\alpha_j - \beta_j z) U_{n+j} = 0.
$$
which takes the form of a "Linear Difference Equation"

### Characteristic Polynomials and Linear Difference Equations

To help us analyze stability regions we need to take a small aside and consider linear difference equations and their solutions.  Say we wanted to solve

$$\sum^r_{j=0} \alpha_j U_{n+j} = 0$$

given initial conditions $U_0, U_1, \ldots, U_{r-1}$.  This expression has a solution in the general form $U_n = \xi^n$.

Plugging this into the equation we have

$$\sum^r_{j=0} \alpha_j \xi^{n+j} = 0$$

which simplifies to

$$\sum^r_{j=0} \alpha_j \xi^j = 0 $$

by dividing by $\xi^n$.  If $\xi$ then is a root of the polynomial

$$\rho(\xi) \equiv \sum^r_{j=0} \alpha_j \xi^j$$

then $\xi$ solves the original difference equation.

### Example:  the Fibonacci Numbers

A classic example of  linear difference equation is the Fibonnacci sequence, $0,1,1,2,3,5,\ldots$.

Which are solutions to the two-term difference equation

$$
    U_{n+2} = U_{n+1} + U_n, \quad U_0=0,\, U_1 = 1
$$

or $\alpha = \begin{bmatrix} -1 & -1 & 1 \\\end{bmatrix}$

Substituting $U_n = \xi^n$ into
$$
U_{n+2} - U_{n+1} - U_n =0
$$
yields
$$
    \xi^n\left(\xi^2 - \xi -1\right) = 0
$$
which has solutions $\xi =  \frac{1\pm\sqrt{5}}{2}$.  Where the positive root is our favorite number $\phi=1.618033$ which is the golden ratio.  The negative root is $1-\phi$. Note, these roots are related to the eigenvalues of a related matrix problem

Thus all Fibonacci numbers can be written as a linear combination of the two roots

$$
    U_n = A\phi^n + B(1-\phi)^n
$$

where $A$ and $B$ are determined by the first two terms in the sequence [0,1].

Note, in this case $\phi > 1$ (while $|(1-\phi)|<1$) thus $U_n\rightarrow\infty$ as $n\rightarrow\infty$.

$$
    U_n = A\phi^n + B(1-\phi)^n
$$

$A\phi^n$ gets larger, while $B(1-\phi)^n \rightarrow 0$.

### Stability for LMM's
Anyway,  Back to the Linear difference equations arising from applying any Linear multistep method to the model problem
$$
    \sum^r_{j=0} (\alpha_j - \beta_j z) U_{n+j} = 0.
$$

Letting
$$
    \rho(\xi) = \sum^r_{j=0} \alpha_j \xi^j, \quad \sigma(\xi) = \sum^r_{j=0} \beta_j \xi^j
$$
and substituting $U_n = \xi^n$ into the difference equations reduces to
$$
    \pi(\xi, z) = \rho(\xi) - z \sigma(\xi) = 0
$$
where $\pi(\xi, z)$ is the **stability polynomial** of the method, whose roots are solutions of the difference method.

### Absolute stability of the linear multi-step methods

For $U_n$ to stay bounded as $n\rightarrow\infty$,  it requires that all roots $\xi_i$ of $\pi(\xi, z)$ satisfy
$$
    |\xi_i| \leq 1
$$
If this is the case, then the multi-step method is **zero-stable**.  Roughly you can think of these roots as multiplying the past truncation errors so that if they are less than or equal to 1 they do not grow in magnitude.  We then define the region of **absolute stability** as the values for $z$ for which this is true.  This approach reduces to the $R$ method for  one-step methods as a special case.

### Example:  Forward Euler's Method

Examining forward Euler's method we have

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \lambda U_n \\
    &= U_{n+1} - U_n (1 + \Delta t \lambda)\\
\end{aligned}
$$

setting $U_n = \xi^n$ implies
$$
\begin{aligned}
   0 &= \xi^n\left(\xi -  (1 + z)\right)\\
\end{aligned}$$

or $\pi(\xi, z) = \xi -  (1 + z)$ whose root is $\xi = 1 + z$ and we have re-derived the stability region we had found before.

### Example:  Backwards Euler's Method

Similarly for backwards Euler

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \lambda U_{n+1} \\
    &= U_{n+1}(1 - \Delta t\lambda) - U_n \\
    &= \xi(1 - z)  - 1 \\
    &=\pi(\xi, z)
\end{aligned}$$

whose root is $\xi = 1/(1 - z)$

### Example: Adams-Bashforth 2

As an example consider the Adams-Bashforth 2-stage LMM
$$
    U_{n+2} = U_{n+1} + \frac{\Delta t}{2} (-f(U_n) + 3 f(U_{n+1}))
$$

Let $f(U) = \lambda U$,  $z = \Delta t \lambda$
$$
    U_{n+2} - U_{n+1} - \frac{1}{2} (-z U_n + 3 z U_{n+1}) = 0 \\
$$

substituting $U_n=\xi^n$, and factoring out $\xi^n$ yields
$$
\begin{aligned}
    \pi(\xi, z) &= \xi^{2} - \xi - \frac{1}{2} (-z  + 3 z \xi) \\
    &= 2 \xi^2 - (2 + 3z) \xi + z \\
\end{aligned}$$

## Plotting Stability Regions

Given a stability polynomial $\pi(\xi,z)$, absolute stability requires that

$$
    \max{|\xi_i|} \leq 1
$$
where $\xi_i$ are the (generally complex) roots of $\pi(\xi, z)$ and therefore a function of $z$.  In the case of single step schemes, the problem reduces to finding the region of the complex plane where $|R(z)| < 1$.

Either way,  the simplest way to visualize the regions of the complex plane where the scheme is stable, is to calculate either $|R(z)|$ or $\max|\xi_i(z)|$ and plot the 1-contour and determine all regions that are $<1$.

### Some useful codes

```python
def stability_plot(X, Y, C, axes, title=None, continuous=True):
    """
    Utility function to make stability diagram given complex stability scalar C

    parameters:
    -----------

    X, Y:  numpy.meshgrids for complex plane
    C: numpy array
        Field to plot,  either |R(z)| for a single step scheme, or max(|xi_i(z)|) for a LMM scheme
    axes: matplotlib axes object
        subplot or plot to draw in.
    title: string
        subplot title if not None
    continuous: boolean
        if True, plot a continous coloring of C
        if False, plot Heaviside(C)
    """
    if  continuous:
        Ch = C
    else:
        Ch = numpy.heaviside(C-1,0.)
    pcolor_plot = axes.pcolor(X, Y, Ch, vmin=0, vmax=1, cmap=plt.get_cmap('Greens_r'))
    axes.contour(X, Y, C, 'k', levels=[1.0])
    fig = plt.gcf()
    fig.colorbar(pcolor_plot)
    axes.plot(x, numpy.zeros(x.shape),'k--')
    axes.plot(numpy.zeros(y.shape), y,'k--')

    axes.set_xlabel('Re', fontsize=16)
    axes.set_ylabel('Im', fontsize=16)
    if title is not None:
        axes.set_title(title, fontsize=16)

    axes.set_aspect('equal')

def plot_stability_ssms(R, x, y, axes=None, title=None):
    """
    plot stability regions for single-step multi-stage ODE schemes given the function R(z)
    such that U_{n+1} = R(z)U_n   and z is complex

    parameters:
    -----------

    R: calleable
        function of a complex variable z such that if |R|<=1, the scheme is absolutely stable
    x: numpy array
        array of values for the real axis
    y: numpy array
        values to plot for the imaginary axis
    axes: matplotlib axes object
        subplot or plot to draw in.  If axes=None create a new figure
    title: string
        subplot title if
    """

    X,Y = numpy.meshgrid(x,y)
    Z = X + 1j * Y
    if axes is None:
        fig = plt.figure(figsize=(8,6))
        axes = fig.add_subplot(1,1,1)

    abs_R = numpy.abs(R(Z))
    stability_plot(X, Y, numpy.abs(R(Z)), axes, title)

def plot_stability_lmm(pi_coeff, x, y, axes=None, title=None):
    """
    plot stability regions for linear multi-step  ODE schemes given the coefficients of the stability polynomial
    pi(xi, z)

    parameters:
    -----------

    pi_coeff: calleable (function of z)
        function that returns array of stability polynomial pi(z)
    x: numpy array
        array of values for the real axis
    y: numpy array
        values to plot for the imaginary axis
    axes: matplotlib axes object
        subplot or plot to draw in.  If axes=None create a new figure
    title: string
        subplot title if not None
    """

    X,Y = numpy.meshgrid(x,y)
    Z = X + 1j * Y
    if axes is None:
        fig = plt.figure(figsize=(8,6))
        axes = fig.add_subplot(1,1,1)

    norm_max = numpy.empty(Z.shape)
    for i,row in enumerate(Z):
        for j, z in enumerate(row):
            norm_max[i,j] = max(numpy.abs(numpy.roots(pi_coeff(z))))

    stability_plot(X, Y, norm_max, axes, title)

```

```python
x = numpy.linspace(-3,3,100)
y = numpy.linspace(-3,3,100)

R_euler = lambda z: 1 + z
R_beuler = lambda z: 1./(1. - z)
R_trap = lambda z: (1. + 0.5*z)/(1. - 0.5*z)
R_RK2 = lambda z: 1 + z + z**2/2
R_Taylor4 = lambda z: 1 + z + z**2/2 + z**3/6. + z**4/24.
```

```python
fig = plt.figure(figsize=(12,12))
axes = fig.add_subplot(2,2,1)
plot_stability_ssms(R_euler, x, y, axes=axes, title='Stability of Eulers method')

axes = fig.add_subplot(2,2,2)
plot_stability_ssms(R_beuler, x, y, axes=axes, title='Stability of Backwards Eulers')

axes = fig.add_subplot(2,2,3)
plot_stability_ssms(R_trap, x, y, axes=axes, title='Stability of Trapezoidal method')

axes = fig.add_subplot(2,2,4)
plot_stability_ssms(R_Taylor4, x, y, axes=axes, title='Stability of Taylor4 method')

```

**Taylor4**: can do oscillators (slightly extends into real)

```python
x = numpy.linspace(-3,3,100)
y = numpy.linspace(-3,3,100)

pi_euler = lambda z: numpy.array([1., -(1 + z)])
pi_AB2 = lambda z: numpy.array([2, -(2. + 3*z), z])
```

`pi_euler` and `pi_AB2` are functions that return numpy arrays of coefficients of stability polynomials.

```python
fig = plt.figure(figsize=(12,6))
axes = fig.add_subplot(1,2,1)
plot_stability_lmm(pi_euler, x, y, axes=axes, title='Stability of Eulers method')
axes = fig.add_subplot(1,2,2)
plot_stability_lmm(pi_AB2, x, y, axes=axes, title='Stability of Adams-Bashforth2 method')
```

## Application to Stiff ODEs

Consider again the ODE we examined before

$$u'(t) = \lambda (u - \cos t) - \sin t$$

except this time with general initial condition $u(t_0) = u_0$.  What happens to solutions that are slightly different from $u_0 = 1$ or $t_0 = 0$?  The general solution of the ODE is

$$u(t) = e^{\lambda (t - t_0)} (u_0 - \cos t_0) + \cos t$$.

```python
# Plot "hairy" solutions to the ODE
u = lambda t_0, u0, lam, t: numpy.exp(lam * (t - t_0)) * (u0 - numpy.cos(t_0)) + numpy.cos(t)

fig = plt.figure(figsize=(12, 10))

for i,lam in enumerate([-1, -10, 1, 10]):
    axes = fig.add_subplot(2, 2, i+1)
    for u0 in numpy.linspace(-1, 1, 10):
        for t_0 in numpy.linspace(0.0, 9.0, 10):
            t = numpy.linspace(t_0,10.0,100)
            axes.plot(t, u(t_0, u0, lam, t),'b')
    t = numpy.linspace(0.0,10.0,100)
    axes.plot(t, numpy.cos(t), 'r', linewidth=5)
    axes.set_ylim((-2, 2))

    axes.set_title("Perturbed Solutions $\lambda = %s$" % lam)
    axes.set_xlabel('$t$')
    axes.set_ylabel('$u(t)$')
axes.set_ylim((-2, 2))
plt.show()

```

Plot note: Each blue line is a different initial condition $u_0$

```python
# Plot "inverse hairy" solutions to the ODE
u = lambda t_0, u0, lam, t: numpy.exp(lam * (t - t_0)) * (u0 - numpy.cos(t_0)) + numpy.cos(t)

num_steps = 10
error = numpy.ones(num_steps) * 1.0
t_hat = numpy.linspace(0.0, 10.0, num_steps + 1)
t_whole = numpy.linspace(0.0, 10.0, 1000)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
eta = 1.0
lam = 0.1

for n in range(1,num_steps):
    t = numpy.linspace(t_hat[n-1], t_hat[n], 100)
    U = u(t_hat[n-1], eta, lam, t)
    axes.plot(t, U, 'b')
    axes.plot(t_whole, u(t_hat[n-1], eta, lam, t_whole),'b--')
    axes.plot([t[-1], t[-1]], (U[-1], U[-1] + -1.0**n * error[n]), 'r')
    eta = U[-1] + -1.0**n * error[n]

t = numpy.linspace(0.0, 10.0, 100)
axes.plot(t, numpy.cos(t), 'g')

axes.set_title("Perturbed Solutions $\lambda = %s$" % lam)
axes.set_xlabel('$t$')
axes.set_ylabel('$u(t)$')
axes.set_ylim((-10,10))
plt.show()

```

### Example:  Chemical systems

Consider the transition of a chemical $A$ to a chemical $C$ through the process

$$A \overset{K_1}{\rightarrow} B \overset{K_2}{\rightarrow} C.$$

If we let

$$\vec{u} = \begin{bmatrix} [A] \\ [B] \\ [C] \end{bmatrix}$$

then we can model this simple chemical reaction with the system of ODEs

$$\frac{\text{d} \vec{u}}{\text{d} t} =
\begin{bmatrix}
    -K_1 & 0 & 0 \\
    K_1 & -K_2 & 0 \\
    0 & K_2 & 0
\end{bmatrix} \vec{u}$$

The solution of this system is of the form

$$u_j(t) = c_{j1} e^{-K_1 t} + c_{j2}e^{-K_2 t} + c_{j3}$$

```python
# Solve the chemical systems example using a Forward Euler scheme
# Problem parameters
K_1 = 3
K_2 = 1
N = 40
K_1 = 30.0
K_2 = 1.0

A = numpy.array([[-K_1, 0, 0], [K_1, -K_2, 0], [0, K_2, 0]])
f = lambda u: numpy.dot(A, u)
```

```python
t_span = [0, 2.]
u_0 = [2.5, 5.0, 2.0]
f = lambda t, u : A.dot(u)
t, U = euler(f, t_span, u_0, N)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t, U.transpose())
axes.legend(["[A]", "[B]", "[C]"])
axes.set_title("Chemical System")
axes.set_xlabel("$t$")
axes.set_title("$[A], [B], [C]$")
axes.set_ylim((0.0, 10.))
axes.set_xlim((0.0, 2.0))
axes.grid()

plt.show()
```

### What is stiffness?

In general a **stiff** ODE is one where $u'(t) \ll f'(t, u)$.  For linear dynamical systems like the chemical decay problem,  the **stiffness ratio**

$$\frac{\max_p |\lambda_p|}{\min_p |\lambda_p|}$$

can be used to characterize the stiffness of the system.  In our last example this ratio was $K_1 / K_2$ if $K_1 > K_2$.  As we increased this ratio we observed that the numerical method became unstable only a reduction in $\Delta t$ lead to stable solution again.  For explicit time step methods this is problematic as the reduction of the time step for only one of the species leads to very expensive evaluations.  For example, forward Euler has the stability criteria

$$|1 + \Delta t \lambda| < 1$$

where $\lambda$ will have to be the maximum eigenvalue of the system.

```python
# set the eigenvalues and time step
K_1 = 30.
K_2 = 1.0
delta_t = 0.05
eigenvalues = [-K_1, -K_2]

```

```python
# Plot the region of absolute stability for Forward Euler
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
for lam in eigenvalues:
    print(lam * delta_t)
    axes.plot(lam * delta_t, 0.0, 'ko')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler")
plt.show()
```

```python
# solve using backwards Euler
A = numpy.array([[-K_1, 0, 0], [K_1, -K_2, 0], [0, K_2, 0]])

N = 100
t = numpy.linspace(0.0, 2.0, N)
delta_t = t[1] - t[0]

U = numpy.empty((t.shape[0], 3))
U[0, :] = [2.5, 5.0, 2.0]

B = numpy.eye(3) - delta_t*A
for n in range(t.shape[0] - 1):
    # backwards Euler (originally wrong)
    #U[n+1, :] = U[n, :] / (1.0 - delta_t * f(U[n, :]))
    U[n+1, :] = numpy.linalg.solve(B, U[n])
```

```python

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t, U)
axes.legend(["[A]", "[B]", "[C]"])
axes.set_title("Chemical System")
axes.set_xlabel("$t$")
axes.set_title("$[A], [B], [C]$")
axes.set_ylim((0.0, 10.))
axes.set_xlim((0.0, 2.0))
axes.grid()

plt.show()
```

### A-Stability
What if we could expand the absolute stability region to encompass more of the left-half plane or even better, all of it.  A method that has this property is called **A-stable**.  We have already seen one example of this with backward Euler which as a stability region of

$$|1 - z| \geq 1$$

which covers the full left-half plane.  It turns out that for linear multi-step methods a theorem by Dahlquist proves that there are no LMMs that satisfies the A-stability criterion beyond second order (trapezoidal rule).  There are higher-order Runge-Kutta methods do however.

Perhaps this is too restrictive though.  Often large eigenvalues for systems (for instance coming from a PDE discretization for the heat equation) lie completely on the real line.  If the stability region can encompass as much of the real line as possible while leaving out the rest of the left-half plane we can possibly get a more efficient method.  There are a number of methods that can be constructed that have this property but are higher-order.

```python
# Plot the region of absolute stability for Backward Euler
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

K_1 = 3.0
K_2 = 1.0
delta_t = 1.0
eigenvalues = [-K_1, -K_2]

axes.set_facecolor((255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.fill(numpy.cos(t) + 1.0, numpy.sin(t), 'w')
for lam in eigenvalues:
    print(lam * delta_t)
    axes.plot(lam * delta_t, 0.0, 'ko')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Backward Euler")
plt.show()
```

# L-Stability

It turns out not all A-stable methods are alike.  Consider the backward Euler method and the trapezoidal method defined by

$$\frac{U_{n+1} - U_n}{\Delta t} = \frac{1}{2}(f(U_n) + f(U_{n+1})$$

whose stability polynomial is

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \frac{1}{2} (\lambda U_n + \lambda U_{n+1}) \\
      &= U_{n+1}\left(1 - \frac{1}{2} \Delta t \lambda \right ) - U_n \left(1 + \frac{1}{2}\Delta t \lambda \right) \\
      &= \left(\xi - \frac{1 + \frac{1}{2}z}{1 - \frac{1}{2} z}\right) \left(1 - \frac{1}{2} z \right )\\
\end{aligned}$$
which shows that it is A-stable.  Lets apply both these methods to a problem we have seen before and see what happens.

```python
# Compare accuracy between Euler and Trapezoidal for the stiff ODE
f = lambda t, lam, u: lam * (u - numpy.cos(t)) - numpy.sin(t)
u_exact = lambda t_0, eta, lam, t: numpy.exp(lam * (t - t_0)) * (eta - numpy.cos(t_0)) + numpy.cos(t)

t_0 = 0.0
t_f = 1.0
eta = 2.0
lam = -1e6
```

```python
num_steps = [10, 20, 40, 50]
# num_steps = numpy.arange(100, 1000, 100)

delta_t = numpy.empty(len(num_steps))
error_euler = numpy.empty(len(num_steps))
error_trap = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]
    u = u_exact(t_0, eta, lam, t_f)

    # Compute Euler solution
    U_euler = numpy.empty(t.shape)
    U_euler[0] = eta
    for (n, t_n) in enumerate(t[1:]):
        U_euler[n+1] = (U_euler[n] - lam * delta_t[i] * numpy.cos(t_n) - delta_t[i] * numpy.sin(t_n)) / (1.0 - lam * delta_t[i])
    error_euler[i] = numpy.abs(U_euler[-1] - u) / numpy.abs(u)

    # Compute using trapezoidal
    U_trap = numpy.empty(t.shape)
    U_trap[0] = eta
    for (n, t_n) in enumerate(t[1:]):
        U_trap[n+1] = (U_trap[n] + delta_t[i] * 0.5 * f(t_n, lam, U_trap[n]) - 0.5 * lam * delta_t[i] * numpy.cos(t_n) - 0.5 * delta_t[i] * numpy.sin(t_n)) / (1.0 - 0.5 * lam * delta_t[i])
    error_trap[i] = numpy.abs(U_trap[-1] - u)
```

```python

# Plot error vs. delta_t
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)
axes.plot(t, U_euler, 'ro-')
axes.plot(t, u_exact(t_0, eta, lam, t),'k')

axes = fig.add_subplot(1, 2, 2)
axes.loglog(delta_t, error_euler, 'bo')
order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_euler[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")
axes.loglog(delta_t, order_C(delta_t[1], error_euler[1], 2.0) * delta_t**2.0, 'b--', label="2nd Order")

axes.legend(loc=4)
axes.set_title("Comparison of Errors for Backwards Euler")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

# Plots for trapezoid
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)
axes.plot(t, U_trap, 'ro-')
axes.plot(t, u_exact(t_0, eta, lam, t),'k')

axes = fig.add_subplot(1, 2, 2)
axes.loglog(delta_t, error_trap, 'bo', label='Trapezoidal')
order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_trap[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")
axes.loglog(delta_t, order_C(delta_t[1], error_trap[1], 2.0) * delta_t**2.0, 'b--', label="2nd Order")

axes.legend(loc=4)
axes.set_title("Comparison of Errors for Trapezoidal Rule")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

plt.show()
```

It turns out that if we look at a one-step method and define the following ratio

$$U_{n+1} = R(z) U_n$$

we can define another form of stability, called **L-stable**, where we require that the method is A-stable and that

$$\lim_{z \rightarrow \infty} |R(z)| = 0.$$

Turns out that backwards Euler is L-stable while trapezoidal rule is not.

```python
R_beuler = lambda z: 1./(1. - z)
R_trap = lambda z: (1. + 0.5*z)/(1. - 0.5*z)
```

```python
fig = plt.figure(figsize=(12,6))
axes = fig.add_subplot(1,2,1)
plot_stability_ssms(R_beuler, x, y, axes=axes, title='Stability of Backwards Eulers')

axes = fig.add_subplot(1,2,2)
plot_stability_ssms(R_trap, x, y, axes=axes, title='Stability of Trapezoidal')
```

## Backward Differencing Formulas

A class of LMM methods that are useful for stiff ODE problems are the backward difference formula (BDF) methods which have the form

$$\alpha_0 U_n + \alpha_1 U_{n+1} + \cdots + \alpha_r U_{n+r} = \Delta \beta_r f(U_{n+r})$$

These methods can be derived directly from backwards finite differences from the point $U_{n+r}$ and the rest of the points back in time.  One can then derive r-step methods that are rth-order accurate this way.  Some of the methods are

$$\begin{aligned}
    r = 1:& & U_{n+1} - U_n = \Delta t f(U_{n+1}) \\
    r = 2:& &3 U_{n+2} - 4 U_{n+1} + U_n = 2 \Delta t f(U_{n+2}) \\
    r = 3:& &11U_{n+3} - 18U_{n+2} + 9U_{n+1} - 2 U_n = 6 \Delta t f(U_{n+3}) \\
    r = 4:& &25 U_{n+4} - 48 U_{n+3} +36 U_{n+2} -16 U_{n+1} +3 U_n = 12 \Delta t f(U_{n+4})
\end{aligned}$$

$r = 1$: Backward Euler

$r = 2$: Second order first finite difference formula

$r = 3$:

Higher order BDF schemes: interpolation of higher order polynomials

```python
pi_BDF1 = lambda z: numpy.array([ 1. - z, -1.])
pi_BDF2 = lambda z: numpy.array([ 3. - 2*z, -4. , 1.])
pi_BDF3 = lambda z: numpy.array([ 11 - 6.*z, -18., 9., -2.])
pi_BDF4 = lambda z: numpy.array([ 25. - 12*z, -48., 36, -16., 3])
```

```python
x = numpy.linspace(-10,10,50)
y = numpy.linspace(-10,10,50)
fig = plt.figure(figsize=(12,12))
axes = fig.add_subplot(2,2,1)
plot_stability_lmm(pi_BDF1, x, y, axes=axes, title='Stability of BDF1 method')

axes = fig.add_subplot(2,2,2)
plot_stability_lmm(pi_BDF2, x, y, axes=axes, title='Stability of BDF2 method')

axes = fig.add_subplot(2,2,3)
plot_stability_lmm(pi_BDF3, x, y, axes=axes, title='Stability of BDF3 method')

axes = fig.add_subplot(2,2,4)
plot_stability_lmm(pi_BDF4, x, y, axes=axes, title='Stability of BDF4 method')

```

BDF1 and 2 are A-stable

Higher order $\rightarrow$ less stable (slight instability region in negative reals)

```python

```