# Assignment 3:  Histograms, Data Analysis and Fitting

The file "edmonton.pickle" contains the historical temperature data from Environment Canada, for weather stations that include "Edmonton" in their name( downloaded from climate.weather.gc.ca).

The data are organized by "station"; a station object contains "name", "latitude", "longitude", "firstYear", "climateID", "stationID", "dates", "minT", "maxT","doy", and "year" and consist of the readings from a single station.

The important objects are "dates", which is an array of datetime objects, "minT", which is a corresponding array of minimum temperature for each date, "maxT" which is the array of maximum temperatures for those dates.  "doy" and "year" refer to "day of the year" (from 0-365) and the year (from 1880-2019).

The snippet of code below will allow you to read in the data.  You will need to download "edmonton.pickle", and "station.py" into your working directory.

```python
import station
import pickle
import datetime as dt

with open('edmonton.pickle','rb') as f:
    s=pickle.load(f)

for station in s:
    print(station.name)
    print(station.maxT)
```

    EDMONTON
    [18.9 12.8 16.7 ... 25.6 26.7 21.1]
    EDMONTON CITY CENTRE AWOS
    [-12.5  -6.1   4.4 ...  16.   14.   18.3]
    EDMONTON CALDER
    [ 22.8  26.1  18.9  19.4  20.   17.2  16.7  19.4  21.7  24.4  22.8  17.8
      16.1  21.1  16.1  18.3  16.7  19.4  21.1  17.8  26.1  21.7  26.1  27.2
      20.6  16.1  16.7  13.3  20.6  22.8  25.   28.3  30.6  30.6  30.   31.1
      25.   25.   27.2  29.4  31.1  32.2  20.6  25.   25.   22.2  21.7  16.1
      22.8  21.7  21.1  21.1  25.   29.4  25.   26.1  30.   18.9  20.6  20.
      22.2  21.1  24.4  26.1  20.   22.8  26.1  23.3  24.4  21.7  20.   18.9
      21.7  16.1  23.3  16.7  18.9  12.8  13.3  16.1  22.2  23.3  15.6  15.
      14.4  16.1  17.8  25.   22.8  17.8  17.2  14.4  17.2  16.7  15.   17.2
      16.7  19.4  22.2  22.2  17.8  17.2  16.1  23.3  23.9  23.3  22.2  16.1
      10.6  14.4  16.1  22.8  24.4  27.8  24.4  22.8  13.9  15.6  18.9  16.1
      15.   18.9  24.4  25.6  26.1  16.1  14.4  10.    3.3   7.2  10.6   8.9
       8.3  12.8  14.4  15.   10.   12.8  18.3  14.4  13.9  10.6   5.6   6.7
       3.9   3.9   5.   -0.6  -1.7  -0.6  -2.2   0.    7.8   5.6   8.9  17.2
      15.6   8.9   5.6   5.6   5.6   2.8  -1.1  -0.6   9.4  15.6  11.7  16.1
       7.2   3.9   0.   -0.6   0.6  -9.4  -9.4 -10.6   0.    0.   -7.2  -6.1
     -12.8  -7.8 -13.9 -14.4 -10.6 -12.8 -20.  -12.2  -3.3   3.3   6.1   0.6
     -11.1 -19.4 -13.9 -22.2  -6.1 -13.3 -14.4  -1.1   7.2   6.1   3.9  -1.7
      -2.8   1.1   4.4   0.    5.6   5.6   1.7   2.2  -3.9 -10.6 -13.3 -14.4
     -12.8 -13.9 -25.  -25.  -26.1 -20.6 -15.   -6.1 -10.6 -12.2 -11.1 -12.2
       0.    2.2   7.8   1.7   5.6   3.9   4.4   3.9  -1.7 -10.  -11.1  -0.6
       8.9   5.    6.1   7.2   8.3   5.    7.8 -10.6 -10.   -3.3   5.6   5.
       7.8  -3.9   0.    2.2  -1.7  -4.4  -5.   -4.4  -7.2  -4.4 -10.6  -9.4
      -2.8   3.9  10.    8.9   1.7  -0.6  -7.2 -16.7 -17.8 -15.6 -21.1 -22.2
     -11.7  -6.7   5.    3.9   3.9   5.   -0.6  -6.7  -4.4   1.7  -2.2   0.6
       5.    6.7  11.7  12.8   1.7   1.1   2.2   8.3   6.1   6.7   7.2   6.7
       7.8   7.2   6.1  13.3  12.2   5.6   8.3  12.2  15.6  18.3  21.1  20.6
      20.   19.4  21.1  20.6  20.   17.8  13.9   5.    4.4   8.9  12.2  12.2
      15.6   8.9  10.   15.6  18.9  14.4  15.   17.8  18.3  18.9  24.4  24.4
      19.4  24.4  18.3  15.   20.   22.2  21.1  21.1  28.3  15.   20.6  13.3
      16.7  19.4  23.3  18.3  12.8  15.   17.8  17.2  24.4  25.6  17.2  18.3
      21.1  30.   15.   14.4  20.6  17.2  22.8  17.2  18.9  12.8  17.8  18.9
      20.   25.   25.   25.   24.4  17.2  21.1  22.2  20.   18.3  23.3  20.
      14.4  11.1  12.8  15.   15.   18.3  22.2  25.6  26.1  25.6  21.7  20.
      22.2  26.1  26.7  27.2  24.4  19.4  23.9  25.6  23.9  21.7  19.4  24.4
      30.   24.4  22.8  23.3  28.3  24.4  27.2  27.8  24.4  26.1  25.6  16.1
      19.4  20.   20.6  21.1  25.   28.9  23.3  23.3  23.9  26.7  26.1  22.2
      22.2  23.9  26.1  26.1  26.1  29.4  28.3  25.   15.   23.9  22.2  21.1
      20.6  22.8  22.8  25.6  20.   16.7  18.3  25.   23.3  23.3  23.9  24.4
      15.6  19.4  23.3  18.3  16.1  12.2  16.7  23.9  18.3  11.1  10.   20.6
      22.2  23.3  23.9  22.8  18.3  21.7  24.4  23.9  23.3  23.3  24.4  23.3
      20.6  23.9  24.4  23.3  16.1  14.4  11.1   6.1  15.   15.   13.3  23.3
      21.1  20.   16.7  17.8  13.3  12.8   6.1   8.3   7.2   7.8  13.9  10.
      10.    7.8   7.2   2.8   0.6   0.   13.9  16.1  12.8   9.4   7.2  12.8
       8.3  11.1  12.2  10.6   4.4   6.7   8.3  10.   10.    1.7   1.7   4.4
       8.9   5.6   6.1   7.2   6.1   1.7  -2.2   1.7   7.2   3.3  -2.8  -9.4
      -0.6  -0.6   5.6   5.6  10.    5.    0.6  -5.  -13.9 -14.4 -13.3 -16.1
       0.    6.1 -11.1   1.1   7.2   3.3   6.7   7.2   3.3  -6.7  -8.9   2.8
       0.6  -3.9   1.7  -1.1  -3.9  -2.2  -6.1  -7.8 -11.7 -11.7 -10.6 -12.2
     -20.  -12.8 -12.8   4.4   6.1  -0.6 -11.1 -17.8 -16.7 -12.8 -14.4 -17.8
     -19.4 -15.   -7.2   2.2   3.9   1.1   3.3   5.    2.8   3.3   1.7   2.2
       3.3   0.  -15.  -13.   -4.4   1.1   2.8   4.4   5.6   2.8   6.1   3.3
       8.3   8.3   6.7   3.9   3.9   3.9   5.    4.4   7.2   4.4   7.8   5.6
       8.3   8.9  10.    8.9   5.    2.2   2.8   3.9   3.3   6.7   5.    3.3
       6.7   5.6  11.1   8.3  11.7  10.    7.2   3.9   6.1  10.6   9.4  -3.3
      -2.8   3.3   1.7  -0.6  -1.1   2.2   2.2   2.2  -0.6  -3.3   6.7  10.
      10.    9.4   8.3   7.2   7.2]
    EDMONTON INT'L A
    [22.8  8.3 12.2 ...  8.4 13.1 16.6]
    EDMONTON INTERNATIONAL CS
    [-14.7 -10.6  -6.1 ...   9.1  12.8  12. ]
    EDMONTON CITY CENTRE A
    [ 7.2  8.3  6.1 ... -0.6 -4.7 -6.3]
    EDMONTON BLATCHFORD
    [ 2.1  0.4 -8.1 ...  9.2 12.6 12.1]
    EDMONTON NAMAO A
    [-7.2 -8.3 -8.3 ... -5.7 -4.7 -7.1]
    EDMONTON NAMAO AWOS A
    [24.5 24.  28.  ...  8.6 11.8 11.5]
    EDMONTON INTL A
    [10.5  4.8  0.5 ...  8.7 12.5 12.2]
    EDMONTON SOUTH CAMPUS
    [ 28.2  27.9  25.1  17.   19.4  23.3  24.6  25.2  28.2  29.8  27.2  27.6
      21.2  24.8  28.7  17.8  13.6  13.9  12.4  19.5  23.3  20.9  18.5  23.4
      28.   27.3  26.4  14.   14.5  17.3  20.8  20.9  21.   22.6  24.5  27.3
      28.   28.3  29.6  26.   25.5  28.3  29.1  19.5  21.7  22.6  23.2  21.7
      19.2  17.4  19.9  22.6  28.1  29.9  21.5  22.6  26.8  26.1  26.1  26.6
      22.9  19.7  25.8  30.6  31.7  27.2  25.4  25.8  20.7  21.   19.2  19.2
      22.7  23.7  26.   28.   30.6  27.6  27.3  29.5  24.   20.6  21.1  25.1
      28.7  25.5  28.2  34.6  31.8  22.1  11.8  20.2  27.1  22.4  27.5  27.2
      20.8  23.2  22.9  26.4  25.6  22.7  11.5  20.4  15.7  14.5  22.2  18.4
      18.4  20.8  17.5  18.5  15.1  15.9  21.1  21.3  26.5  17.8  12.5  10.3
      14.8   9.    0.8   3.6   2.7   3.1   5.   11.4  14.    9.8   2.5   1.3
       8.5  11.2  17.3  13.2   8.2   8.1   3.1   4.7   4.    3.1   7.1   9.
      10.6  11.5   7.3  -1.7  -1.8   0.6   6.5   7.7   1.5  11.5  14.3  17.6
      24.9  17.   13.8  17.4  12.1  14.3  13.2  12.8  16.1  11.9  10.6   8.1
       8.9   8.6   8.4   3.2  -0.1   5.7   2.1  -6.1 -10.   -6.1  -9.1   3.6
       2.9  -2.9   4.5   9.    8.9   4.5  -5.6  -2.3   4.3   6.4   8.5   6.3
       5.7   1.4  -3.5   3.3  -1.1  -1.4   6.1   3.5   2.7  -0.9  -4.7  -4.5
      -1.5  -6.6  -6.6  -7.6  -3.6  -1.4   2.4  -2.4   4.1   5.5   6.1   5.8
      -5.1   0.2   3.7   5.8   2.    1.4  -3.5  -4.6  -6.7  -5.7  -9.   -8.5
      -3.   -2.5  -2.9  -6.1   3.3   8.6   6.3  -1.3  -3.8  -3.1  -6.4 -10.9
      -8.5  -9.6  -3.5   0.4  -3.4   2.6  -0.3 -10.2 -15.6 -14.5 -12.5   2.7
       2.1   0.   -0.9   1.7   4.4   9.7   9.9  -3.1  -9.    3.1   3.8 -13.5
     -20.3 -26.7 -22.8 -23.  -20.  -18.1 -19.3 -21.8 -21.6 -20.2 -17.3 -15.8
     -12.7 -18.3 -14.9 -16.7 -13.9  -2.3  -3.  -10.4  -1.7 -11.1 -14.7 -14.7
      -7.3  -1.5  -5.9 -16.1 -18.5 -12.6  -5.5 -10.1 -11.5  -7.4  -6.   -2.7
       2.    7.1   4.7   4.8   5.    5.7   8.   10.6  12.5  15.5  15.3  11.9
      13.9   9.6   3.8   2.5   4.3   4.3   8.9   9.2   9.3   6.2   7.5   5.4
      10.2   3.1   8.8   6.1  13.4  15.2  15.9  12.1  10.3   9.8  14.    9.
      10.1  13.2  14.8  17.3  13.2  11.9  15.9  22.5  17.5  10.7  11.6  11.6
       5.6   5.4   8.    7.   10.6   5.6   6.4   9.9  13.4  12.4  18.   21.
      20.1  24.3  23.5  19.5  16.8   9.8  15.1  17.7  16.5  17.4  18.6  19.1
      22.2  20.7  15.6  20.2  22.1  24.7  28.   30.8  19.7  22.2  27.9  21.8
      18.8  20.2  18.5  12.2   8.7  14.   15.7  18.9  22.3  27.3  24.4  21.
      22.2  24.   25.3  23.6  15.3  15.1  17.6  20.3  21.2  16.   18.7  20.8
      22.2  18.7  22.2  19.1  18.1  17.6  18.9  19.   21.   20.3  20.4  16.7
      18.5  20.9  25.8  26.1  24.4  24.1  23.5  24.4  15.5  20.1  23.3  24.2
      27.5  29.4  23.   22.2  26.7  25.3  21.6  23.5  21.4  23.6  24.7  28.1
      20.6  24.3  26.5  18.1  22.3  26.4  17.2  16.3  15.8  17.7  21.3  24.9
      25.4  15.8  17.2  15.8  18.7  24.9  28.3  20.8  20.7  23.   21.   17.4
      21.7  19.5  19.2  13.8  15.8  16.7  20.8  25.   25.2  22.5  21.7  22.9
      15.4  13.5  11.3  14.7  21.6  20.8  20.5  21.6  18.6  18.2  17.9  19.5
      20.6  22.7  24.7  16.2  16.4  14.3  10.5   5.2   5.3   4.3   3.8   8.5
      12.2  12.3]
    EDMONTON VILLENEUVE A
    [-15.  -14.  -10.5 ...  21.5  13.8  14.8]
    EDMONTON VILLENEUVE A
    [11.2 15.5 14.6 ...  9.3 12.2 11.1]
    EDMONTON STONY PLAIN
    [ -7.8 -13.9 -18.9 ...   4.    9.5  13. ]
    EDMONTON WOODBEND
    [ -2.2  -5.6  -9.4 ...   6.5  -1.  -10. ]
    FORT EDMONTON
    [-13.3 -21.7 -27.2 ...  18.5  20.   17.5]
    EDMONTON STONY PLAIN CS
    [ 1.8  9.2  7.  ...  9.2 12.8 10.2]
    EDMONTON TIEBEKE ESTATES
    [ 25.5  23.   27.5  27.   22.   26.   26.5  23.   20.   20.5  24.5  25.5
      13.   10.5  14.   17.5  18.5  18.   22.5  16.   14.   16.   21.   15.5
      14.   17.5  18.5  22.   24.5  23.   17.   13.   19.5  25.5  24.5  21.5
      23.5  15.5  11.5   9.   10.5  11.5   8.    4.    2.5   3.   11.   13.
      11.5  12.5  17.5  14.   11.5  11.   10.   10.   14.   10.    6.    9.
      15.   10.   19.   18.   11.5  21.   19.5  13.   10.    9.   11.    6.
       8.   13.    2.5   2.5   6.5   6.5   2.5   0.    2.    7.5   3.    4.
      10.    6.    5.    2.5   2.5   2.5   3.   -0.5  -2.5  -1.5   1.   -0.5
       0.5   1.5  -1.5   2.   -3.   -4.   -5.    2.    5.    7.    6.5   5.5
       5.    3.5  -2.5  -3.   -9.5  -5.5  -3.5  -6.   -3.   -6.   -4.5  -5.
       0.5   7.5   5.    4.    5.5  10.5   7.    5.5   7.5   8.5  11.    8.5
       7.5   8.   10.   12.    9.5   5.5  14.    8.    5.5   1.5   8.    9.
       1.    5.5   5.    3.5  -6.5  -5.5  -2.    2.5   8.5  15.5  18.   19.5
      19.   24.5  15.5  12.   14.   16.   17.   17.   16.   19.5  17.   15.5
      19.   15.5  17.5   8.5  12.   14.   16.    4.    9.   13.5  13.   14.5
      16.5  11.5  16.   18.5  21.   16.   20.5  19.   13.   16.5  16.   13.5
      19.   10.   14.5  15.   18.   18.   21.   22.5  26.   25.5  21.5  21.5
      12.   11.   10.5   7.5  13.   19.   16.5  14.   13.5  21.   22.   15.5
      20.   20.   19.   18.5  18.   16.   19.5  23.5  25.5  25.5  22.   21.
      21.   21.5  17.   17.5  20.5  22.5  24.   20.5  20.   21.5  25.5  28.5
      24.   17.   17.5  13.5  18.5  22.   25.5  25.   27.5  25.5  21.   25.
      24.   25.5  27.5  23.5  25.5  26.5  23.   20.   22.   24.   26.5  25.5
      21.   22.   22.   20.5  17.   19.   17.5  19.5  23.5  17.   22.5  24.5
      21.   19.5  22.   25.5  26.   24.   26.   15.5  14.   16.5   7.5  10.5
      11.5  10.5   7.5   9.   13.5  19.5  17.   22.   15.5  15.5  10.5  15.5
      17.5  17.5  22.5  24.   25.   23.   17.5  11.5   7.    2.5   5.5  18.
      20.5  20.   16.   23.5  21.5  19.   13.   11.5   4.5   3.5  -1.    3.5
       6.5  13.5  22.   16.   13.5   8.5  15.5  12.    5.5  12.   15.5  13.
      14.5  10.5   9.    9.   14.   18.   15.    9.   12.   11.5   3.5   4.5
       8.    9.5   6.5   8.5   9.5   8.5  -5.5  -6.   -5.   -5.5  -8.   -9.5
      -3.    3.    2.5   2.5  -2.   -2.5   0.    3.5  -0.5   1.5   5.    7.5
       8.5   7.    3.5   6.    1.    2.   -1.5  -4.   -4.    1.5   0.5  -4.5
       2.    7.    1.  -11.  -22.  -19.5 -17.5 -20.5 -19.5 -20.  -23.5 -18.
     -14.5  -0.5  -2.5  -2.5  -3.5  -5.5   2.    2.    2.5   8.5   3.    6.
       3.5   5.    6.    7.    2.5   1.5  -5.   -9.   -6.   -4.    4.5   2.5
      -1.    0.    2.    2.5   4.    0.5   2.    0.5   3.5   4.    5.    1.
       0.5   3.    3.    5.    6.5   2.5  -4.5  -8.  -11.  -13.5 -16.5 -10.
     -11.   -9.5  -0.5  -1.5 -13.5 -11.5  -3.5  -6.   -8.   -4.   -4.   -3.5
      -9.5 -17.5 -16.   -5.5   6.5  10.5   6.5   1.5   7.    5.5   3.    9.
      11.    9.5   3.    4.5   8.    9.5   5.    6.    7.    7.5   9.5  11.
       4.   -6.5  -6.5 -12.  -12.   -6.    1.    4.5   9.5  11.5   6.    5.
       8.    8.    6.5   6.    6.5  11.5  12.5  12.    7.5   8.    5.5   5.5
       9.    9.5   6.5   4.5   9.   18.5  17.5  12.5   2.    8.   12.5  17.
      22.   17.5  21.5  26.5  25.   17.   15.5  12.5  14.5  19.5  18.5  14.
      13.   18.5  19.   14.5  15.5  18.5  24.   22.   17.5  19.   17.5  16.5
      15.5  13.5  14.5  15.5  24.   28.   25.5  22.   19.5  24.   26.5   9.5
      10.5  22.   22.   15.5  12.5  13.5  20.   20.   15.   22.   23.   16.
      16.   18.   19.5  17.5  17.5  16.   15.5  18.   19.5  26.   27.   22.
      21.5  17.5  13.5  13.   15.5  20.5  15.   20.   20.   23.   24.5  29.
      21.   24.   25.5  24.5  27.5  31.5  19.5  24.   20.5  24.   24.   19.
      21.5  19.5  21.5  22.5  21.5  20.   21.   17.   16.   20.   23.5  19.5
      13.5  20.5  24.   24.   25.5  29.   23.   24.5  24.5  21.   18.5  21.
      24.5  24.   27.5  28.   25.   27.5  27.   30.   27.   19.   21.   22.5
      22.5  24.5  20.   22.   24.5  28.5  21.   22.   24.5  23.5  23.5  18.5
      22.5  17.5  17.   14.5  13.   13.   14.   13.   13.   16.5  19.   24.5
      25.5  18.5  20.5  16.5  10.5  15.5  13.5  15.5  25.   28.5  29.   25.
      17.   14.5  19.5  17.   10.5  15.   12.    6.   15.   19.   15.5  16.
      12.5  12.5  10.5  10.5   7.    6.5   7.   15.5   8.5   8.5  10.5   9.
       7.    0.5  -2.5]

Look at the data.

1.  Make plots that show the max and min temperatures as a function of date for a single station
2.  Histogram the max and min temperatures for a single station at a few dates throughout the year. Make sure the histograms have a reasonable number and range for the bins.
3.  For each pair of stations, find the periods of time during which they both measured temperatures.  If there is an overlapping period, find the mean and standard deviation of the differences between the max and min temperatures measured at the two stations for those periods.  Fill out a table with rows:  Name of Station 1, Name of Station 2, number of days that both measured temperatures, Average Difference in max, standard deviation of difference of Max, Average difference in min, standard deviation of difference in min
4.  For a few of the pairs which have significant differences, make a 2d color histogram of Ta-Tb versus Ta, where Ta and Tb refer to the measurements at the two stations.

```python

```

Combine all the data from the different stations into an average max and min for each each day of the year. To do this, you will to pick one station as the standard, and correct each of the others by its average difference.

Keep the data as an np.array.
Plot averages for the year.

```python

```

## No-atmosphere model of heating and cooling

The atmosphere plays a critical role in weather and climate, but modelling it is difficult.  Anyone who has watched the weather forecast sees weather systems move into and out of regions and winds/fronts/ high and low pressure regions- all atmospheric phenomena.  However, we are going to see whether we can fit the data with a "no-atmosphere" model.  If there were no atmosphere, the the only source of heat would be solar radiation, and the only cooling would be from black body radiation.  We incorporate this as an ordinary differential equation.

\begin{equation} \frac {dT}{dt}=\alpha F(t) - \beta T^4 \end{equation}

Here $T$ is the (absolute!)temperature since $T^4$ is the Stefan-Boltzman law, $t$ the time, $F$ the solar flux (in Watts/m$^2$) and $\alpha$ and $\beta$ are the parameters of the model that we will allow to fit.  If we were to calculate from first principles, $\alpha$ would include the reflectivity of the surface and the clouds , as well as the heat capacity per square meter of the layer of the earth/atmosphere that heat up and cool down.  Similarly, $\beta$ includes the Stefan-Boltzman constant, the emissivity of the earth, and the heat capacity.  So this model is quite simple.

However, it is still interesting to see how well such a simple average model can work.  Since everything in the model is independent of time (except for the orbit) our model does not allow any differences year to year or any difference between locations at the same latitude.

To start the problem, I have modified the earth-sun solution from Problem Set 2 to include the rotation of the earth, which is important to calculate the solar flux at Edmonton.    This involves adding the vector from the center of the earth to Edmonton, $\vec{x_{Ed}}$ and solving the differential equation
\begin{equation} \frac{d\vec{x_{Ed}}}{dt}=\vec{\omega} \times \vec{x_{Ed}}\end{equation} where $\vec{\omega}$ in the constant rotation vector of the earth.

We start by finding the "north pole vector", which we know is aligned with the earth-sun vector on June 21 and December 21st.  If we look at the Horizon web page for Dec 22, we see that X~0, Y=1 at the solstice, and that the axial tilt (obliquity) is 23.4392911 degrees.  Thus $$\hat{n}=(0,\sin(23.4392911),\cos(23.4392911))\approx(0,0.398,0.918)$$ and $$\vec{\omega}=\frac{2\pi\hat{n}}{24\times 3600 \times 365.2425/366.2425}$$
(Notice here the factor 365.2425/366.2425- which converts from normal days to "sidereal days"; which take into account the fact that the earths revolution around the sun means noon-noon is a little longer than one rotation).

We pick $\vec{x_{Ed}}=\cos(53.55)(1,0,0)+\sin(53.55)\hat{n}$ at the solstice. (Only the latitude matters- in principle we need to set longitude as well, but in our model the longitude doesn't really matter).

We run the code below to generate the inputs to the flux calcuation- the sun-earth vector and the Edmonton vector.  We pick t to cover the year, with say 48 bins per day.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
import datetime as dt
import time

GM=132712440041.93938e9 #m^3/s^2
GMEarth=398600.435436e9 #m^3/s^2
MEarth=5.97219e24 #kg
MSun=GM/GMEarth*MEarth  #  this keeps the mass ratio right, with most precise GM values!
AU=149597870700.0 #m (exactly)
D=24*3600.0 #s

#  Factors from Horizon web site at 2458839.916666667 = A.D. 2019-Dec-22 10:00:00.0000 TDB
#  This is picked to be at the hour closest to the solstice- notice that the x position of earth is very small
x0=np.array([4.858434307587728E-04 *AU, 9.836920738092132E-01 *AU, -4.745398623388847E-05 *AU])
v0=np.array([-1.749013293502635E-02 *AU/D,-5.128449611745279E-05 *AU/D,  4.120640971206839E-07 *AU/D])
tilt=23.4392911/180*np.pi
n=np.array([0, np.sin(tilt),np.cos(tilt)])  # vector of earth's axis
omega=2*np.pi/(D*365.2425/366.2425)*n  #rotation axis
latitude=53.55/180*np.pi  # latitude of the "Edmonton" weather station
radius=6.37e6  #Earth's radius=6370 km from Horizons
x_ed0=radius*(np.array([np.cos(latitude),0,0])+np.sin(latitude)*n)  #Edmonton location at solstice (at least for some year!)
factor=(MSun+MEarth)/MSun  # to convert xe to earth-sun distance
xsun=np.array([0,0,0])

solarConstant=1367.6 #W/m**2 from Horizons.  At 1 AU

cm=MEarth*x0/(MEarth+MSun)
x=x0-cm #  x is distance wrt to the cm

vcm=MEarth*v0/(MEarth+MSun)  #velocity of CM
v=v0-vcm

def dvdt(xvArgument,t):
    xv=xvArgument.reshape(3,3)
    xearth_sun=factor*xv[0] #position wrt sun
    distance=np.sqrt(np.dot(xearth_sun,xearth_sun))
    v_ed=np.cross(omega,xv[2])  # velocity of Edmonton is omega x x_ed
    return np.array([xv[1],-GM/distance**3*xearth_sun,v_ed]).reshape(9)

spy=365.2425*24*3600
t=np.linspace(0,380*24*3600,380*48)  # 48 bins per day, starting December 22.  Go 380 days to cover whole next year
y0=np.array([x,v,x_ed0]).reshape(9)

run=True
if run:
    cpuT0=time.process_time()
    ephemeris = odeint(dvdt, y0, t,rtol=1e-12)
    print("CPU Time=",time.process_time()-cpuT0)
    np.save('problem3',ephemeris)
else:
    ephemeris=np.load('problem3.npy')

```

    CPU Time= 31.738442715999998

Write a function to calculate the Flux at any instant of time, from the output of this calculation.  To do this you will need to interpolate ephemeris (since it is only returned at discrete times).

Ephemeris contains three vectors- the position of the earth wrt to CM, the velocity of the earth wrt CM, and the position of Edmonton with respect to the center of the earth.

Once we have the three vectors, the flux is
\begin{equation} F(t)=\begin{cases}\phi_0 \frac{\vec{x_{Ed}} \cdot (\vec {x_s}-\vec{x_e})}{|\vec{x_{Ed}}| |\vec {x_s}-\vec{x_e}|} & \text{if } \vec{x_{Ed}} \cdot (\vec {x_s}-\vec{x_e})>0\\0 & \text{if } \vec{x_{Ed}} \cdot (\vec {x_s}-\vec{x_e})<0
\end{cases} \end{equation}

Plot your function versus date (for one year)

```python

```

Now write a model that integrates the ordinary differential equation:

\begin{equation} \frac {dT}{dt}=\alpha F(t) - \beta T^4 \end{equation}

using F(t) from the calculation of ephemeris.

You will need to experiment to find values of $\alpha$ and $\beta$ that are semi-reasonable.

Find the maximum and minimum temperature for each day from this model, and calculate the difference between this model and the data.

Fit the model to the data.  Plot the residuals.  Comment on the reasonableness of the fit.

```python

```

(Optional) If you would like to experiment some more, we can add some elements to the model and see how the fit improves.  One can add
feedback("clouds?") by making $\alpha$ and $\beta$ functions (first or second order polynomials) of T. You could also change $T^4$ to $(T-\Delta)^4$- basically saying the cooling infrared radiation doesn't come from the ground, but from higher in the atmosphere where the temperatures are colder.

```python

```

Assignments/Blank/Assignment 3 Data and Fitting.ipynb


# Introduction to Digital agro

## Crop simulation models

____

### Mikhail Gasanov

E-mail: Mikhail.Gasanov[a]skoltech.ru

tg:@misha_grol

## Clone utils and files from GitHub

```python
# !git clone https://github.com/mishagrol/Seminar_Sobol.git
# !cp -r ./Seminar_Sobol/* .
```

# How to start with PCSE/WOFOST model

_____

### Documentation: [PCSE](https://pcse.readthedocs.io/)

```python
%matplotlib inline
import sys, os
import pcse
import pandas as pd
import matplotlib
import yaml
matplotlib.style.use("ggplot")
import matplotlib.pyplot as plt
print("This notebook was built with:")
print("python version: %s " % sys.version)
print("PCSE version: %s" %  pcse.__version__)
```

    This notebook was built with:
    python version: 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:19:23)
    [GCC Clang 10.0.1 ]
    PCSE version: 5.4.2

```python
import warnings
warnings.filterwarnings("ignore")
```

```python
wofostPP = pcse.start_wofost(mode="wlp")
```

You have just successfully initialized a PCSE/WOFOST object in the Python interpreter, which is in its initial state and waiting to do some simulation. We can now advance the model state for example with 1 day:

```python
wofostPP.run()
```

Advancing the crop simulation with only 1 day, is often not so useful so the number of days to simulate can be specified as well:

```python
wofostPP.run(days=10)
```

## Getting information about state and rate variables
Retrieving information about the calculated model states or rates can be done with the `get_variable()` method on a PCSE object. For example, to retrieve the leaf area index value in the current model state you can do:

### Leaf Area Index

```python
# Leaf Area Index at this date
print(wofostPP.day)
print('LAI', wofostPP.get_variable('LAI'))
```

    2000-01-12
    LAI 0.2870809817505803

The `get_variable()` method can retrieve any state or rate variable that is defined somewhere in the model.

Finally, we can finish the crop season by letting it run until the model terminates because the crop reaches maturity or the harvest date:

```python
wofostPP.run_till_terminate()
```

## Retrieving and displaying WOFOST output
We can retrieve the results of the simulation at each time step using  `get_output()`. In python terms this returns a list of dictionaries, one dictionary for each time step of the the simulation results. Each dictionary contains the key:value pairs of the state or rate variables that were stored at that time step.

```python
output = wofostPP.get_output()
```

The most convenient way to handle the output from WOFOST is to used the `pandas` module to convert it into a dataframe. Pandas DataFrames can be converted to a variety of formats including excel, CSV or database tables.

```python
df_crop = pd.DataFrame(output).set_index('day')
```

```python
summary_output = wofostPP.get_summary_output()
msg = "Reached maturity at {DOM} with total biomass {TAGP:.1f} kg/ha, " \
      "a yield of {TWSO:.1f} kg/ha with a maximum LAI of {LAIMAX:.2f}."
for crop_cycle in summary_output:
    print(msg.format(**crop_cycle))
```

    Reached maturity at 2000-05-31 with total biomass 15261.8 kg/ha, a yield of 7179.8 kg/ha with a maximum LAI of 6.13.

```python
fig, (axis1, axis2) = plt.subplots(nrows=1, ncols=2, figsize=(16,8))
df_crop.LAI.plot(ax=axis1, label="LAI", color='k')
df_crop.TAGP.plot(ax=axis2, label="Total biomass")
df_crop.TWSO.plot(ax=axis2, label="Yield")
axis1.set_title("Leaf Area Index")
axis2.set_title("Crop biomass")
fig.autofmt_xdate()
r = fig.legend()
```

# Running PCSE/WOFOST with custom input data

This Jupyter notebook will show you how to read inputs from files for running PCSE/WOFOST.

thanks to **Allard de Wit**

**Prerequisites for running this notebook**

Several packages need to be installed for running PCSE/WOFOST:

  1. `PCSE` and its dependencies. See the [PCSE user guide](http://pcse.readthedocs.io/en/stable/installing.html) for more information;
  2. The `pandas` module for processing and storing WOFOST output;
  3. The `matplotlib` module for generating charts

## Reading model parameters
### Crop parameters

```python
data_dir = 'data/'
```

```python
from pcse.fileinput import CABOFileReader
cropfile = os.path.join(data_dir, 'crop', 'SUG0601.crop')
cropdata = CABOFileReader(cropfile)
```

```python
#potato
from pcse.fileinput import CABOFileReader
cropfile_potato = os.path.join(data_dir, 'crop', 'POT701.CAB')
cropdata_potato = CABOFileReader(cropfile_potato)
```

```python
# Number of parameters for our crop
len(cropdata_potato)
```

    63

### Soil parameters
The soildata dictionary provides the parameter name/value pairs related to the soil type and soil physical properties. The number of parameters is variable depending on the soil water balance type that is used for the simulation. For this example, we will use the water balance for freely draining soils and use the soil file for medium fine sand: `ec3.soil`. This file is also taken from the soil files in the [WOFOST Control Centre](http://www.wageningenur.nl/wofost).

```python
soilfile = os.path.join(data_dir, 'soil', 'ec3.soil')
soildata = CABOFileReader(soilfile)
print(soildata)
```

    ** $Id: ec3.new 1.2 1997/09/18 17:33:54 LEM release $
    **
    ** SOIL DATA FILE for use with WOFOST Version 5.0, June 1990
    **
    ** EC3-medium fine
    ------------------------------------
    SMW: 0.104 <class 'float'>
    SMFCF: 0.3 <class 'float'>
    SM0: 0.41 <class 'float'>
    CRAIRC: 0.06 <class 'float'>
    K0: 25.586 <class 'float'>
    SOPE: 1.47 <class 'float'>
    KSUB: 1.47 <class 'float'>
    SPADS: 0.1 <class 'float'>
    SPODS: 0.03 <class 'float'>
    SPASS: 0.2 <class 'float'>
    SPOSS: 0.05 <class 'float'>
    DEFLIM: -0.3 <class 'float'>
    RDMSOL: 120 <class 'int'>
    SOLNAM: EC3-medium fine <class 'str'>
    SMTAB: [-1.0, 0.41, 1.0, 0.398, 1.3, 0.389, 1.491, 0.38, 2.0, 0.34, 2.4, 0.287, 2.7, 0.241, 3.4, 0.148, 4.204, 0.104, 6.0, 0.09] <class 'list'>
    CONTAB: [0.0, 1.408, 1.0, 0.167, 1.3, -0.215, 1.491, -0.638, 1.7, -0.854, 2.0, -1.155, 2.4, -1.796, 2.7, -2.26, 3.0, -2.745, 3.4, -3.357, 3.7, -3.824, 4.0, -4.276, 4.204, -4.678] <class 'list'>

### Site parameters

The site parameters provide ancillary parameters that are not related to the crop or the soil. Examples are the initial conditions of the water balance such as the initial soil moisture content (WAV) and the initial and maximum surface storage (SSI, SSMAX). Also the atmospheric $CO_{2}$
concentration is a typical site parameter. For the moment, we can define these parameters directly on the Python commandline as a simple python dictionary. However, it is more convenient to use the `WOFOST71SiteDataProvider` that documents the site parameters and provides sensible defaults:

```python
from pcse.util import WOFOST71SiteDataProvider
sitedata = WOFOST71SiteDataProvider(WAV=100, CO2=360)
print(sitedata)
```

    {'IFUNRN': 0, 'NOTINF': 0, 'SSI': 0.0, 'SSMAX': 0.0, 'WAV': 100.0, 'SMLIM': 0.4, 'CO2': 360.0}

### Packaging all parameters
Finally, we need to pack the different sets of parameters into one variable using the `ParameterProvider`. This is needed because PCSE expects one variable that contains all parameter values. Using this approach has the additional advantage that parameter value can be easily overridden in case of running multiple simulations with slightly different parameter values:

```python
from pcse.base import ParameterProvider
parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
```

## Agromanagement
The agromanagement inputs provide the start date of the agricultural campaign, the start_date/start_type of the crop simulation, the end_date/end_type of the crop simulation and the maximum duration of the crop simulation. The latter is included to avoid unrealistically long simulations for example as a results of a too high temperature sum requirement.

The agromanagement inputs are defined with a special syntax called [YAML](http://yaml.org/) which allows to easily create more complex structures which is needed for defining the agromanagement. The agromanagement file for sugar beet in Wageningen `sugarbeet_calendar.agro` can be read with the `YAMLAgroManagementReader`:

```python
from pcse.fileinput import YAMLAgroManagementReader
#crop rotation for Moscow region
agromanagement_file = os.path.join(data_dir, 'agro', 'sugarbeet_calendar_Moscow_short.agro')
#agromanagement_file = os.path.join(data_dir, 'agro', 'sugarbeet_calendar.agro')
agromanagement = YAMLAgroManagementReader(agromanagement_file)
print(agromanagement)
```

    !!python/object/new:pcse.fileinput.yaml_agro_loader.YAMLAgroManagementReader
    listitems:
    - 2019-06-01:
        CropCalendar:
          crop_end_date: 2019-10-15
          crop_end_type: harvest
          crop_name: sugar-beet
          crop_start_date: 2019-06-02
          crop_start_type: emergence
          max_duration: 300
          variety_name: sugar-beet-601
        StateEvents: null
        TimedEvents:
        - comment: All fertilizer amounts in kg/ha
          event_signal: apply_npk
          events_table:
          - 2019-06-22:
              K_amount: 122
              N_amount: 128
              P_amount: 25
          name: Timed N/P/K application table

We can create a crop rotation in the model

```python
K_kg = 60
P_kg = 60
N_kg = 120
year=2017

yaml_agro = f"""
- {year}-05-01:
    CropCalendar:
        crop_name: 'sugar-beet'
        variety_name: 'sugar-beet-601'
        crop_start_date: {year}-05-20
        crop_start_type: sowing
        crop_end_date:
        crop_end_type: maturity
        max_duration: 250
    TimedEvents:
    -   event_signal: apply_npk
        name:  Timed N/P/K application table
        comment: All fertilizer amounts in kg/ha
        events_table:
        - {year}-06-22: {{N_amount : {N_kg}, P_amount: {P_kg}, K_amount: {K_kg}}}
    StateEvents: null
"""
agromanagement = yaml.safe_load(yaml_agro)
print(yaml_agro)
#crop_end_date: {year_date_1}-11-15
```

    - 2017-05-01:
        CropCalendar:
            crop_name: 'sugar-beet'
            variety_name: 'sugar-beet-601'
            crop_start_date: 2017-05-20
            crop_start_type: sowing
            crop_end_date:
            crop_end_type: maturity
            max_duration: 250
        TimedEvents:
        -   event_signal: apply_npk
            name:  Timed N/P/K application table
            comment: All fertilizer amounts in kg/ha
            events_table:
            - 2017-06-22: {N_amount : 120, P_amount: 60, K_amount: 60}
        StateEvents: null

## Daily weather observations
Daily weather variables are needed for running the simulation. There are several data providers in PCSE for reading weather data, see the section on [weather data providers](http://pcse.readthedocs.io/en/stable/reference_guide.html#weather-data-providers) to get an overview.

For this example we will use weather data from an excel file which provides daily weather data for Wageningen for the period 2004 to 2008. We will read the data from the file using the ExcelWeatherDataProvider:

### NASA Weather Data Provider from NASA [DataBase](https://power.larc.nasa.gov/)

```python
#NASA Weather system

#Sometimes it does not work (server), upload excel file
from pcse.db import NASAPowerWeatherDataProvider
```

```python

weather = NASAPowerWeatherDataProvider(51, 5, force_update=True)

```

```python
print(weather)
```

    Weather data provided by: NASAPowerWeatherDataProvider
    --------Description---------
    NASA/POWER SRB/FLASHFlux/MERRA2/GEOS 5.12.4 (FP-IT) 0.5 x 0.5 Degree Daily Averaged Data
    ----Site characteristics----
    Elevation:   38.5
    Latitude:  51.000
    Longitude:  5.000
    Data available for 1983-07-01 - 2021-02-07
    Number of missing days: 6

### Problems with missing days (~1-5 %)

```python
def weather_loader(latitude, longitude):

    path = './data/meteo/'
    #API request to NASA database
    weather = NASAPowerWeatherDataProvider(latitude, longitude, force_update=True)

    # Print done if downloaded
    print('____DONE_____','latitude',latitude, 'longitude',longitude,'____')

    # export pcse.weather format to pandas df
    df_weather = pd.DataFrame(weather.export())

    #print('initial number of days:', len(df_weather))

    #create full range of dates
    r = pd.date_range(start=df_weather.DAY.min(), end=df_weather.DAY.max())

    #extend range of dates
    full_range_weather = df_weather.set_index('DAY').reindex(r).rename_axis('DAY').reset_index()
    missing_days = (full_range_weather.isna()).sum().sum()

    print('num_of_missing_days', missing_days)

    #fill weather with fill forward method in pandas
    filled_weather = full_range_weather.fillna(method='ffill', axis=0)

    #save as csv file

    filled_weather=filled_weather[['DAY', 'IRRAD', 'TMIN', 'TMAX', 'VAP', 'WIND', 'RAIN']]
    filled_weather['SNOWDEPTH'] = 'NaN'
    filled_weather[['IRRAD']] = filled_weather[['IRRAD']]/1000.
    filled_weather[['VAP']] = filled_weather[['VAP']]/10.
    filled_weather.DAY=filled_weather.DAY.dt.strftime('%Y%m%d')

    text = open(path+"pattern.csv", "r")
    text = ''.join([i for i in text]).replace("1111", str(weather.longitude))
    text = ''.join([i for i in text]).replace("2222", str(weather.latitude))
    text = ''.join([i for i in text]).replace("3333", str(weather.elevation))
    text = ''.join([i for i in text]).replace("4444", str(weather.angstA))
    text = ''.join([i for i in text]).replace("5555", str(weather.angstB))
    x = open(path+f'NASA_weather_latitude_{latitude}_longitude_{longitude}.csv',"w")
    x.writelines(text)
    x.close()

    path_to_save_csv_file = path+f'NASA_weather_latitude_{latitude}_longitude_{longitude}.csv'
    filled_weather.to_csv(path_to_save_csv_file, mode='a', header=False, index=False)

    #add info to weather database and save it to csv

    #LOAD WEATHER as csv file
    weather = pcse.fileinput.CSVWeatherDataProvider(path_to_save_csv_file)
    return weather
```

```python
weather = weather_loader(55,55)
```

    ____DONE_____ latitude 55 longitude 55 ____
    num_of_missing_days 195

```python
df_weather = pd.DataFrame(weather.export())
```

```python
fig, (ax1,ax2) = plt.subplots(nrows=1, ncols=2, figsize=(16,4))
df_weather.set_index('DAY')['ET0'][-365:].plot(ax=ax1, label='ET0')
df_weather.set_index('DAY')['TMAX'][-365:].plot(ax=ax2, label='T MAX')
ax1.set_title('ET 0')
ax2.set_title('T°C Max')
```

## Importing, initializing and running a PCSE model

Internally, PCSE uses a simulation engine to run a crop simulation. This engine takes a configuration file that specifies the components for the crop, the soil and the agromanagement that need to be used for the simulation. So any PCSE model can be started by importing the engine and initializing it with a given configuration file and the corresponding parameters, weather data and agromanagement.

However, as many users of PCSE only need a particular configuration (for example the WOFOST model for potential production), preconfigured Engines are provided in `pcse.models`. For the sugarbeet example we will import the WOFOST model for water-limited simulation under freely draining soil conditions:

```python
from pcse.models import Wofost71_WLP_FD
wofsim = Wofost71_WLP_FD(parameters, weather, agromanagement)
wofsim.run_till_terminate()
df_results = pd.DataFrame(wofsim.get_output())
df_results = df_results.set_index("day")
df_results.tail()
```

      DVS
      LAI
      TAGP
      TWSO
      TWLV
      TWST
      TWRT
      TRA
      RD
      SM
      WWLOW

      day

      2018-01-21
      1.638268
      4.361832
      13001.94206
      8070.729387
      2295.014434
      2636.198239
      2072.527759
      0.014248
      120.0
      0.154055
      18.486627

      2018-01-22
      1.638268
      4.361832
      13001.94206
      8070.729387
      2295.014434
      2636.198239
      2072.527759
      0.023385
      120.0
      0.153960
      18.475179

      2018-01-23
      1.638268
      4.361832
      13001.94206
      8070.729387
      2295.014434
      2636.198239
      2072.527759
      0.023202
      120.0
      0.154166
      18.499908

      2018-01-24
      1.638268
      4.361832
      13001.94206
      8070.729387
      2295.014434
      2636.198239
      2072.527759
      0.028208
      120.0
      0.154058
      18.486978

      2018-01-25
      1.638268
      4.361832
      13001.94206
      8070.729387
      2295.014434
      2636.198239
      2072.527759
      0.026663
      120.0
      0.153811
      18.457348

We can then run the simulation and retrieve the time series of daily simulation output using the get_output() method on the WOFOST object. Finally, we convert the simulation results to a pandas dataframe:

```python
summary_output = wofsim.get_summary_output()
```

```python
wofsim.get_summary_output()
```

    [{'DVS': 1.6382678571428573,
      'LAIMAX': 4.361831916594967,
      'TAGP': 13001.942059594281,
      'TWSO': 8070.729386700372,
      'TWLV': 2295.0144341030245,
      'TWST': 2636.198238790885,
      'TWRT': 2072.5277589248244,
      'CTRAT': 17.5463787829338,
      'RD': 120.0,
      'DOS': datetime.date(2017, 5, 20),
      'DOE': datetime.date(2017, 6, 1),
      'DOA': datetime.date(2017, 7, 23),
      'DOM': None,
      'DOH': None,
      'DOV': None}]

```python
msg = "Reached maturity at {DOM} with total biomass {TAGP} kg/ha "\
"and a yield of {TWSO} kg/ha."
print(msg.format(**summary_output[0]))
```

    Reached maturity at None with total biomass 13001.942059594281 kg/ha and a yield of 8070.729386700372 kg/ha.

# Sensitivity analysis of models
___

```python
# !pip install SALib
```

    Collecting SALib
      Downloading https://files.pythonhosted.org/packages/f7/33/cee4d64f7c40f33c08cf5ef5c9b1fb5e51f194b5deceefb5567112800b70/SALib-1.3.11.tar.gz (856kB)
         |████████████████████████████████| 860kB 3.3MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.18.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from SALib) (3.2.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from SALib) (1.0.5)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (1.2.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->SALib) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->SALib) (1.15.0)
    Building wheels for collected packages: SALib
      Building wheel for SALib (setup.py) ... done
      Created wheel for SALib: filename=SALib-1.3.11-py2.py3-none-any.whl size=729665 sha256=93e720543725a20b1f31d6b4dcd5681574a59bb5c49e364a65b935ee9a92a0b0
      Stored in directory: /root/.cache/pip/wheels/62/ed/f9/a0b98754ffb2191b98324b96cbbeb1bd5d9598b39ab996b429
    Successfully built SALib
    Installing collected packages: SALib
    Successfully installed SALib-1.3.11

## Sobol’ Sequences versus Random numbers and regular grid

```python
from SALib.sample import saltelli
from SALib.analyze import sobol
from SALib.test_functions import Ishigami
import numpy as np
```

__Docs [SALib](https://salib.readthedocs.io/en/latest/#)__

```python

```

In this example, we will perform a Sobol’ sensitivity analysis of the _Ishigami_ function, shown below. The _Ishigami_ function is commonly used to test uncertainty and sensitivity analysis methods because it exhibits strong nonlinearity and nonmonotonicity.

$f(x)=\sin \left(x_{1}\right)+ \text{a}\, \operatorname{sin}^{2}\left(x_{2}\right)+ \text{b}\,  x_{3}^{4} \sin \left(x_{1}\right)$

```python
problem = {
  'num_vars': 3,
  'names': ['x1', 'x2', 'x3'],
  'bounds': [[-np.pi, np.pi]]*3
}
```

```python
# Generate samples
param_values = saltelli.sample(problem, 10, calc_second_order=True)
param_values.shape
```

    (80, 3)

Here, `param_values` is a NumPy matrix. If we run `param_values.shape`, we see that the matrix is **8000 by 3**. The Saltelli sampler generated 8000 samples. The Saltelli sampler generates $N∗(2D+2)$ samples, where in this example $N$ is 1000 (the argument we supplied) and $D$ is 3 (the number of model inputs). The keyword argument `calc_second_order=False` will exclude second-order indices, resulting in a smaller sample matrix with $N∗(D+2)$ rows instead.

```python
# Run model (example)
Y = Ishigami.evaluate(param_values)

# Perform analysis
Si = sobol.analyze(problem, Y, print_to_console=True)
# Returns a dictionary with keys 'S1', 'S1_conf', 'ST', and 'ST_conf'
# (first and total-order indices with bootstrap confidence intervals)
T_Si, first_Si, (idx, second_Si) = sobol.Si_to_pandas_dict(Si)
```

    Parameter S1 S1_conf ST ST_conf
    x1 -0.203434 0.660158 1.345837 2.206907
    x2 0.838328 0.650453 0.716922 0.567090
    x3 -0.592087 0.390549 0.459872 0.420236

    Parameter_1 Parameter_2 S2 S2_conf
    x1 x2 0.208665 1.159449
    x1 x3 1.165212 0.917190
    x2 x3 0.268558 0.759311

Consider the model output as
\begin{eqnarray*}
Y=f(X)=f\left(X_{1}, \ldots, X_{p}\right),
\end{eqnarray*}
where $f$ in our case part of agro-model simulator, $X$ are $p$ varied input parameters and $Y$ is the predicted output. Following the techniques by Sobol we represent the multi-variate random function $f$ using Hoeffding decomposition:
\begin{equation}
f(X_1,\dots,X_p) = f_0 + \sum_i^p f_i + \sum_i^p\sum_{j>i}^p f_{ij} + \dots + f_{1\dots p},
\end{equation}
where $f_0$ is a constant term, $f_i = f_i(X_i)$ denotes main effects, $f_{ij} = f_{ij}(X_i, X_j)$ and others describe higher-order interactions. These terms can be written as
\begin{equation*}
\begin{split}
f_0 &= E(Y),\\
f_i &= E_{X_{\sim i}}(Y | X_i) - E(Y),\\
f_{ij} &= E_{X_{\sim ij}}(Y | X_i, X_j) - f_i - f_j - f_0,\\
\dots
\end{split}
\end{equation*}
where $E$ is mathematical expectation and $X_{\sim i}$ denotes all parameters except $i^\text{th}$. Under the assumption that the input parameters are independent, total variance $V(Y)$ of the crop yield can be decomposed as follows:
\begin{equation*}
V(Y) = \sum_i^p V_i + \sum_i^p\sum_{j>i}^p V_{ij} + \dots + V_{12\dots p},
\end{equation*}
where partial variances are
\begin{equation*}
\begin{split}
V_i &= V[f_i(X_i)] = V_{X_i}\left[E_{X_{\sim i}}(Y | X_i)\right],\\
V_{ij} &= V[f_{ij}(X_i,X_j)] = V_{X_iX_j}\left[E_{X_{\sim ij}}(Y | X_i, X_j)\right] - V_i - V_j,\\
\dots
\end{split}
\end{equation*}

## Sobol index (first order, second order, total index)

This way, sensitivity indices (SI) can be introduced as
\begin{equation}
\Large
S_i = \frac{V_i}{V(Y)},~S_{ij} = \frac{V_{ij}}{V(Y)},~\dots
\end{equation}
In order to incorporate all of the interactions for a particular parameter, one can compute the total effect index:
\begin{equation}
S_{T_i} = \frac{E_{X_{\sim i}}\left[V_{X_i}(Y|X_{\sim i})\right]}{V(Y)} = 1 - \frac{V_{X_{\sim i}}\left[E_{X_i}(Y | X_{\sim i})\right]}{V(Y)}
\end{equation}

From this assumption we can conclude:
\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

More -
* [Wiki](https://en.wikipedia.org/wiki/Sobol_sequence)
* [Habr](https://habr.com/ru/post/440892/)
* Feature selection [Skoltech ML 2020](https://github.com/adasegroup/ML2020_lectures/blob/master/lecture9/Lecture_9_Model_Feature_Selection_Sensitivity.pdf)

# Sensitivity analysis of WOFOST model

## Install modules

```python
  from SALib.sample import saltelli
  from SALib.analyze import sobol
  from SALib.test_functions import Ishigami
  import numpy as np
  import pandas as pd
```

## Parameters

```python
NPK = {
    'num_vars':3,
    'names':['N_kg', 'P_kg', 'K_kg'],
    'bounds':[[30., 60.],
              [60., 90.],
              [100., 130.]]
}
```

```python
Soil_parameters = {
  'num_vars':5,
    'names':['SMV', 'SMFCF', 'SM0', 'CRAIRC', 'K0'],
    'bounds':[[0.7, 1.3],
              [0.1, 0.5],
              [0.2, 0.6],
             [0.04, 0.08],
             [22.5, 27.5]]}
```

## Generate input parameters

```python
param_values = saltelli.sample(Soil_parameters, 10)
```

$n = N \times (D \times 2 +2)$

```python
param_values.shape
```

    (120, 5)

## Loop for yield prediction

```python
from pcse.fileinput import YAMLAgroManagementReader
agromanagement_file = os.path.join(data_dir, 'agro', './sugarbeet_calendar.agro')
agromanagement = YAMLAgroManagementReader(agromanagement_file)
#print(agromanagement)
Soil_parameters = {
  'num_vars':5,
    'names':['SMV', 'SMFCF', 'SM0', 'CRAIRC', 'K0'],
    'bounds':[[0.7, 1.3],
              [0.1, 0.5],
              [0.2, 0.6],
             [0.04, 0.08],
             [22.5, 27.5]]}
param_values = saltelli.sample(Soil_parameters, N=10, calc_second_order=True)
```

Soil parameters in [PCSE model](https://pcse.readthedocs.io/en/stable/code.html?highlight=K0#pcse.soil.WaterbalanceFD)

```python
def sensitivity_soil(soil_parameters):
    SMV, SMFCF, SM0, CRAIRC, K0 = soil_parameters
    soildata['SMV'] = SMV
    soildata['SMFCF'] = SMFCF
    soildata['SM0'] = SM0
    soildata['CRAIRC'] = CRAIRC
    soildata['K0'] = K0
    parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
    wofsim = Wofost71_WLP_FD(parameters, wdp, agromanagement)
    wofsim.run_till_terminate()
    #df_results = pd.DataFrame(wofsim.get_output())
    #df_results = df_results.set_index("day")
    #df_results.tail()
    summary_output = wofsim.get_summary_output()
    yield_list.append(summary_output[0]['TWSO'])
```

```python
%%time
yield_list = []
param_values = saltelli.sample(Soil_parameters, 10, calc_second_order=True)
for step in range(len(param_values)):
    sensitivity_soil(param_values[step])
    print(param_values[step])
```

    [ 0.83183594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 1.24433594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.11835938  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.55976563  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06003906 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06707031 22.84667969]
    [ 0.83183594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.13867188  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.40742187  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06707031 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 23.90136719]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 0.94433594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.31835938  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.35976562  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04003906 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 25.34667969]
    [ 1.13183594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.33867188  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.20742188  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04707031 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 26.40136719]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 0.79433594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.21835938  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.25976562  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.05003906 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 26.59667969]
    [ 1.28183594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.23867188  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.50742187  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.07707031 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 25.15136719]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 1.09433594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.41835937  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.45976563  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.07003906 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 24.09667969]
    [ 0.98183594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.43867188  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.30742188  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.05707031 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 22.65136719]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 1.16933594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.26835938  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.20976563  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05503906 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 25.97167969]
    [ 0.90683594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.28867188  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.25742188  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05207031 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 23.27636719]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 0.86933594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.46835938  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.40976563  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07503906 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 23.47167969]
    [ 1.20683594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.48867187  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.45742187  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07207031 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 25.77636719]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 0.71933594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.16835938  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.50976562  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.06503906 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 24.72167969]
    [ 1.05683594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.18867188  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.35742188  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.04207031 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 27.02636719]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 1.01933594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.36835938  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.30976562  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.04503906 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 27.22167969]
    [ 0.75683594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.38867188  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.55742187  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.06207031 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 24.52636719]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.85058594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.48085937  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.52226562  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.04128906 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 24.56542969]
    [ 0.73808594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.17617188  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.21992188  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.05832031 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 25.62011719]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.15058594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.28085938  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.32226562  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.06128906 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 27.06542969]
    [ 1.03808594  0.28085938  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.37617188  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.41992188  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.07832031 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 23.12011719]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 27.06542969]
    CPU times: user 1min 21s, sys: 229 ms, total: 1min 21s
    Wall time: 1min 21s

```python
np_yield = np.array(yield_list)
Si = sobol.analyze(Soil_parameters, np_yield, print_to_console=False)
```

```python
Si_dict = dict(Si)
Si_df = pd.DataFrame()
Si_df = Si_df.append(pd.Series(Si_dict['S1']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['S1_conf']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST_conf']), ignore_index=True)
Si_df = Si_df.T
Si_df.columns = ['Si', 'ST', 'Si_conf', 'ST_conf']
Si_df.rename(index={0:'SMV',1:'SMFCF', 2:'SM0', 3:'CRAIRC', 4:'K0'}, inplace=True)
Si_df
```

      Si
      ST
      Si_conf
      ST_conf

      SMV
      0.000000
      0.000000
      0.000000
      0.000000

      SMFCF
      -0.077417
      0.284878
      0.839456
      0.638290

      SM0
      0.267504
      0.381437
      0.693409
      0.688329

      CRAIRC
      0.000000
      0.000000
      0.000000
      0.000000

      K0
      0.000000
      0.000000
      0.000000
      0.000000

Is it ok?

\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

### For 5 years

```python
def sensitivity_weather(year):
    K_kg = 60
    P_kg = 60
    N_kg = 120
    year_date=year
    print(year)
    print(year_date)
    yaml_agro = f"""
    - {year_date}-06-01:
        CropCalendar:
            crop_name: 'sugar-beet'
            variety_name: 'sugar-beet-601'
            crop_start_date: {year_date}-06-02
            crop_start_type: emergence
            crop_end_date: {year_date}-10-15
            crop_end_type: harvest
            max_duration: 300
        TimedEvents:
        -   event_signal: apply_npk
            name:  Timed N/P/K application table
            comment: All fertilizer amounts in kg/ha
            events_table:
            - {year_date}-06-22: {{N_amount : {N_kg}, P_amount: {P_kg}, K_amount: {K_kg}}}
        StateEvents: null
        """
    agromanagement = yaml.safe_load(yaml_agro)
    parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
    wofsim = Wofost71_WLP_FD(parameters, moscow_weather, agromanagement)
    wofsim.run_till_terminate()
    summary_output = wofsim.get_summary_output()
    yield_list_weather.append(summary_output[0]['TWSO'])
```

## Visualizing simulation results

Finally, we can generate some figures of WOFOST variables such as the development (DVS), total biomass (TAGP), leaf area index (LAI) and root-zone soil moisture (SM) using the MatPlotLib plotting package:

```python
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12,10))
for var, ax in zip(["DVS", "TWSO", "LAI", "SM"], axes.flatten()):
    ax.plot_date(df_results.index, df_results[var], 'b-')
    ax.set_title(var)
fig.autofmt_xdate()
```

# Visualization for sensitivity analysis

Plots by [Water programming group](https://waterprogramming.wordpress.com/2019/08/27/a-python-implementation-of-grouped-radial-convergence-plots-to-visualize-sobol-sensitivity-analysis-results/)

How to repeat: [Repo of SampleVIS](https://github.com/charlesrouge/SampleVis)

```python
import numpy as np
from SALib.analyze import sobol
from SALib.sample import saltelli
from fishery import fish_game
import matplotlib.pyplot as plt
import itertools
import math
```

### Why number of samples is important?

```python
# Set up dictionary with system parameters
problem = {
  'num_vars': 6,
  'names': ['a', 'b', 'c','h',
            'K','m'],
  'bounds': [[ 0.002, 2],
             [0.005, 1],
             [0.2, 1],
             [0.001, 1],
             [100, 5000],
             [0.1, 1.5]]
}

# Array with n's to use
nsamples = np.arange(50, 500, 50)

# Arrays to store the index estimates
S1_estimates = np.zeros([problem['num_vars'],len(nsamples)])
ST_estimates = np.zeros([problem['num_vars'],len(nsamples)])

# Loop through all n values, create sample, evaluate model and estimate S1 & ST
for i in range(len(nsamples)):
    print('n= '+ str(nsamples[i]))
    # Generate samples
    sampleset = saltelli.sample(problem, nsamples[i],calc_second_order=False)
    # Run model for all samples
    output = [fish_game(*sampleset[j,:]) for j in range(len(sampleset))]
    # Perform analysis
    results = sobol.analyze(problem, np.asarray(output), calc_second_order=False,print_to_console=False)
    # Store estimates
    ST_estimates[:,i]=results['ST']
    S1_estimates[:,i]=results['S1']

np.save('ST_estimates.npy', ST_estimates)
np.save('S1_estimates.npy', S1_estimates)

S1_estimates = np.load('S1_estimates.npy')
ST_estimates = np.load('ST_estimates.npy')

# Generate figure showing evolution of indices
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_estimates[j,:], linewidth=5)
ax1.set_title('Evolution of S1 index estimates', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_estimates[j,:], linewidth=5)
ax2.set_title('Evolution of ST index estimates', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=11)
plt.show()
#plt.savefig('indexevolution.png')

# Calculate parameter rankings
S1_ranks = np.zeros_like(S1_estimates)
ST_ranks = np.zeros_like(ST_estimates)
for i in range(len(nsamples)):
    orderS1 = np.argsort(S1_estimates[:,i])
    orderST = np.argsort(ST_estimates[:,i])
    S1_ranks[:,i] = orderS1.argsort()
    ST_ranks[:,i] = orderST.argsort()

# Generate figure showing evolution of ranks
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_ranks[j,:], linewidth=3)
ax1.set_title('Parameter ranking based on S1', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_ranks[j,:], linewidth=3)
ax2.set_title('Parameter ranking based on ST', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=14)
#plt.show()
#plt.savefig('rankingevolution.png')
```

## Radial plot for SA

```python
import numpy as np
import itertools
import matplotlib.pyplot as plt
import seaborn as sns
import math
from numpy import genfromtxt
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
sns.set_style('whitegrid', {'axes_linewidth': 0, 'axes.edgecolor': 'white'})
```

## Plot function

```python
def is_significant(value, confidence_interval, threshold="conf"):
    if threshold == "conf":
        return value - abs(confidence_interval) > 0
    else:
        return value - abs(float(threshold)) > 0

def grouped_radial(SAresults, parameters, radSc=2.0, scaling=1, widthSc=0.5, STthick=1, varNameMult=1.3, colors=None, groups=None, gpNameMult=1.5, threshold="conf"):
    # Derived from https://github.com/calvinwhealton/SensitivityAnalysisPlots
    fig, ax = plt.subplots(1, 1)
    color_map = {}

    # initialize parameters and colors
    if groups is None:

        if colors is None:
            colors = ["k"]

        for i, parameter in enumerate(parameters):
            color_map[parameter] = colors[i % len(colors)]
    else:
        if colors is None:
            colors = sns.color_palette("deep", max(3, len(groups)))

        for i, key in enumerate(groups.keys()):
            #parameters.extend(groups[key])

            for parameter in groups[key]:
                color_map[parameter] = colors[i % len(colors)]

    n = len(parameters)
    angles = radSc*math.pi*np.arange(0, n)/n
    x = radSc*np.cos(angles)
    y = radSc*np.sin(angles)

    # plot second-order indices
    for i, j in itertools.combinations(range(n), 2):
        #key1 = parameters[i]
        #key2 = parameters[j]

        if is_significant(SAresults["S2"][i][j], SAresults["S2_conf"][i][j], threshold):
            angle = math.atan((y[j]-y[i])/(x[j]-x[i]))

            if y[j]-y[i] < 0:
                angle += math.pi

            line_hw = scaling*(max(0, SAresults["S2"][i][j])**widthSc)/2

            coords = np.empty((4, 2))
            coords[0, 0] = x[i] - line_hw*math.sin(angle)
            coords[1, 0] = x[i] + line_hw*math.sin(angle)
            coords[2, 0] = x[j] + line_hw*math.sin(angle)
            coords[3, 0] = x[j] - line_hw*math.sin(angle)
            coords[0, 1] = y[i] + line_hw*math.cos(angle)
            coords[1, 1] = y[i] - line_hw*math.cos(angle)
            coords[2, 1] = y[j] - line_hw*math.cos(angle)
            coords[3, 1] = y[j] + line_hw*math.cos(angle)

            ax.add_artist(plt.Polygon(coords, color="0.75"))

    # plot total order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["ST"][i], SAresults["ST_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, color='w'))
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, lw=STthick, color='0.4', fill=False))

    # plot first-order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["S1"][i], SAresults["S1_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["S1"][i]**widthSc)/2, color='0.4'))

    # add labels
    for i, key in enumerate(parameters):
        ax.text(varNameMult*x[i], varNameMult*y[i], key, ha='center', va='center',
                rotation=angles[i]*360/(2*math.pi) - 90,
                color=color_map[key])

    if groups is not None:
        for i, group in enumerate(groups.keys()):
            print(group)
            group_angle = np.mean([angles[j] for j in range(n) if parameters[j] in groups[group]])

            ax.text(gpNameMult*radSc*math.cos(group_angle), gpNameMult*radSc*math.sin(group_angle), group, ha='center', va='center',
                rotation=group_angle*360/(2*math.pi) - 90,
                color=colors[i % len(colors)])

    ax.set_facecolor('white')
    ax.set_xticks([])
    ax.set_yticks([])
#     ax.
    plt.axis('equal')
    plt.axis([-2*radSc, 2*radSc, -2*radSc, 2*radSc])
    #plt.show()

    return fig
```

## Range of soil parameters

```python
problem = {
    'num_vars':6,
    'names':['SOC', 'Sand', 'Clay', 'pH', 'CN', 'BD'],
    'bounds':[[2.58, 6.20],
              [0.01, 0.30],
              [0.01, 0.30],
              [4.6, 6.9],
              [10.9, 12.4],
              [900, 1350]]
}
```

```python
#names for csv files
list_of_csv=['soybean-000-2015.csv', 'sugar-beet-2011.csv', 'sugar-beet-2017.csv',
'spring-barley-2012.csv', 'sugar-beet-2014.csv']
list_of_names=['soybean-000-2015', 'sugar-beat-2011', 'sugar-beat-2017',
'spring-barley-2012', 'sugar-beat-2014']
list_of_totals=['total_SI_'+x for x in list_of_names]
list_of_first=['fisrt_SI_'+x for x in list_of_names]
list_of_second=['second_SI_'+x for x in list_of_names]
list_of_SI=['SI_'+x for x in list_of_names]
```

```python
for j, i in enumerate(list_of_csv):
        all_data_csv = genfromtxt('./'+str(i), delimiter=',')
        output = all_data_csv[:,2]
        print(i)
        list_of_SI[j] = sobol.analyze(problem, output, calc_second_order=True, conf_level=0.95, print_to_console=False)
```

    soybean-000-2015.csv
    sugar-beet-2011.csv
    sugar-beet-2017.csv
    spring-barley-2012.csv
    sugar-beet-2014.csv

```python
groups={"Soil physics" : ["Sand", "Clay", "BD"],
        "Soil chemistry" : ["pH", "SOC", "CN"]}

fig = grouped_radial(list_of_SI[4], ['BD', 'Sand', 'Clay', 'pH', 'CN', 'SOC'], groups=groups, threshold=0.001)
red_patch = mpatches.Patch(color='red', label='The red data')
plt.title(list_of_names[4], loc='left')

plt.show()
```

## Homework

__[Tasks](https://skoltech-my.sharepoint.com/:w:/g/personal/mikhail_gasanov_skoltech_ru/EeTPQxbrzVdPqnSENKYyoTUBay1RDYgMMW3GO3qFT2ge5g?e=4hk45V)__

Usefull

__SA and UQ__

1) [Rhodium project](https://github.com/Project-Platypus/Rhodium.git)

2) [SALib](https://github.com/SALib/SALib)

__Model__

3) [PCSE](https://pcse.readthedocs.io/en/stable/index.html)

4) How to install PCSE at local machine
    `conda env create -f` [py3_pcse.yml](https://github.com/mishagrol/Seminar_Sobol/blob/master/py3_pcse.yml)

Аny questions  -

Telegram - `@misha_grol`

Part 1 – Crop Yield Prediction (PCSE, MONICA)

You can use the seminars’ colab:

“https://colab.research.google.com/drive/1j4AHD8KkTRThPuNsQzQFWYSJtptQ6bUA”

1) Assess the yield of one of the crops for the Moscow region over several years (potatoes, sugar beets for 2-3 years)

    Crop  - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/crop)

    Soil - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/soil)

    Weather  - NASAdataprovider in PCSE (https://pcse.readthedocs.io/en/stable/code.html?highlight=NASA#pcse.db.NASAPowerWeatherDataProvider)

    Agromanagement - (https://github.com/mishagrol/Seminar_Sobol/blob/master/data/agro/sugarbeet_calendar_Moscow_short.agro)

Part 2 – Sensitivity Analysis (SALib)

1) Perform sensitivity analysis of one of the model blocks (crop, soil, agromanagement *) with SALib. You can choose one of the methods that you consider necessary (Sobol, FAST, ...).

    Generate samples – In report provide the size of the resulting matrix and the sample size (N)

    Conduct parameter sensitivity analysis  - In report provide S1 and ST indices.

2) Generate plots (Hist, Radial convergence plot, etc.)

*3) Estimate the required number of simulations to obtain reliable values of the sensitivity indices. Try to estimate the sample size at the confidence interval of the sensitivity indices.

* Please note that working with discrete data can cause certain difficulties.

#Agro Hack https://agro-code.ru/

### bonus

__Morris method__

Generate a sample using the Method of Morris

Three variants of Morris' sampling for elementary effects is supported:

- Vanilla Morris
- Optimised trajectories when ``optimal_trajectories=True`` (using
    Campolongo's enhancements from 2007 and optionally Ruano's enhancement
    from 2012; ``local_optimization=True``)
- Groups with optimised trajectories when ``optimal_trajectories=True`` and
    the problem definition specifies groups (note that ``local_optimization``
    must be ``False``)

At present, optimised trajectories is implemented using either a brute-force
approach, which can be very slow, especially if you require more than four
trajectories, or a local method based which is much faster. Both methods now
implement working with groups of factors.

Note that the number of factors makes little difference,
but the ratio between number of optimal trajectories and the sample size
results in an exponentially increasing number of scores that must be
computed to find the optimal combination of trajectories.  We suggest going
no higher than 4 from a pool of 100 samples with the brute force approach.
With local_optimization = True (which is default),
it is possible to go higher than the previously suggested 4 from 100.

```python
import sys

from SALib.analyze import morris
from SALib.sample.morris import sample
from SALib.test_functions import Sobol_G
from SALib.util import read_param_file
from SALib.plotting.morris import horizontal_bar_plot, covariance_plot, \
    sample_histograms
import matplotlib.pyplot as plt

#sys.path.append('../..')

# Read the parameter range file and generate samples
#problem = read_param_file('/Users/mikhailgasanov/Documents/GIT/SALib/src/SALib/test_functions/params/Sobol_G.txt')
# or define manually without a parameter file:
problem = {
 'num_vars': 8,
 'names': ['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7', 'x8'],
 'groups': None,
 'bounds': [[0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0]]
}
# Files with a 4th column for "group name" will be detected automatically, e.g.
# param_file = '../../src/SALib/test_functions/params/Ishigami_groups.txt'

# Generate samples
param_values = sample(problem, N=1000, num_levels=4,
                      optimal_trajectories=None)

# To use optimized trajectories (brute force method),
# give an integer value for optimal_trajectories

# Run the "model" -- this will happen offline for external models
Y = Sobol_G.evaluate(param_values)

# Perform the sensitivity analysis using the model output
# Specify which column of the output file to analyze (zero-indexed)
Si = morris.analyze(problem, param_values, Y, conf_level=0.95,
                    print_to_console=True,
                    num_levels=4, num_resamples=100)
# Returns a dictionary with keys 'mu', 'mu_star', 'sigma', and 'mu_star_conf'
# e.g. Si['mu_star'] contains the mu* value for each parameter, in the
# same order as the parameter file

fig, (ax1, ax2) = plt.subplots(1, 2)
horizontal_bar_plot(ax1, Si, {}, sortby='mu_star', unit=r"tCO$_2$/year")
covariance_plot(ax2, Si, {}, unit=r"tCO$_2$/year")

fig2 = plt.figure()
sample_histograms(fig2, param_values, problem, {'color': 'y'})
plt.show()
```

```python

```


```python
import pp
from FermatPrincipleThreaded import Fermat
import astropy.coordinates as ac
import astropy.units as au
#import astropy.time as at
from IRI import *
import numpy as np
from Geometry import *
import sympy
import time

import tempfile

def ppRayProp(file,inRays,N,pathlength,frequency):
    #enu = ENUFrame.ENU()
    fermat =  FermatPrincipleCartesian.Fermat(neFunc = None,type = 's',frequency = frequency)
    fermat.loadFunc(file)
    rays = {}
    for ray in inRays:
        datumIdx = ray.id
        #origin = astropy.coordinates.SkyCoord(*(ray.origin*astropy.units.km),frame=enu).transform_to('itrs').cartesian.xyz.to(astropy.units.km).value
        #direction = astropy.coordinates.SkyCoord(*ray.dir,frame=enu).transform_to('itrs').cartesian.xyz.value
        origin = ray.origin
        direction = ray.dir
        time = ray.time
        #print(ray)
        x,y,z,s = fermat.integrateRay(origin,direction,pathlength,time=time,N=N)
        rays[datumIdx] = {'x':x,'y':y,'z':z,'s':s}
    return rays

def test():
    sol = SolitonModel(5)
    neFunc = sol.generateSolitonsModel()
    theta = np.linspace(-np.pi/8.,np.pi/8.,25)
    inRays = []
    origin = ac.ITRS(sol.enu.location).cartesian.xyz.to(au.km).value
    count = 0
    for t in theta:
        for p in theta:
            direction = ac.SkyCoord(np.sin(t),
                                    np.sin(p),
                                    1.,frame=sol.enu).transform_to('itrs').cartesian.xyz.value
            ray = Ray(origin,direction,id = count,time =0)
            inRays.append(ray)
            count += 1

    ncpus = 1
    t1 = time.time()
    N = len(inRays)/ncpus
    # Creates jobserver with ncpus workers
    jobs = []
    job_server = pp.Server(ncpus, ppservers=())
    for i in range(ncpus):
        file = 'neFunc-thread{0}.npz'.format(i)
        np.savez(file,neFunc=neFunc)
        #file = sol.saveNeFunc(neFunc)
        job = job_server.submit(ppRayProp,
                   args=(file,inRays[i:(i+1)*N],100,2000,120e6),
                   depfuncs=(),
                   modules=('FermatPrincipleCartesian',),
                   globals={})
        jobs.append(job)
    for job in jobs:
        result = job()
    print(time.time() - t1)
    job_server.print_stats()
if __name__ == '__main__':
    test()
```


# sbPoynETNRPy: An Einstein Toolkit Thorn for Computing the 4-Velocity Time-Component $u^0$, the Magnetic Field Measured by a Comoving Observer $b^{\mu}$, and the Poynting Vector $S^i$

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

**Notebook Status:**  Validated

**Validation Notes:** This module has been validated against the hand-written smallbPoynET in WVUThorns_diagnostics (a trsuted code), which itself is based on expressions in IllinoisGRMHD... which was validated against the original GRMHD code of the Illinois NR group.

## Introduction:
In the [previous tutorial notebook](Tutorial-u0_smallb_Poynting-Cartesian.ipynb), we constructed within SymPy full expressions for the 4-velocity time-component $u^0$, the magnetic field (measured by a comoving observer) $b^{\mu}$, and the Poynting vector $S^i$.

Here we will work through the steps necessary to construct an Einstein Toolkit diagnostic thorn (module) that uses ADMBase and HydroBase variables as input into the NRPy+-generated SymPy expressions for $b^{\mu}$, $b^2$, and the Poynting Vector $S^i$, outputting to gridfunctions `smallb4U[]`, `smallb2etk` (the "etk" suffix must be appended because base gridfunction names ending in numbers are not allowed in NRPy+), and `SPoyn[]`, respectively.

# Table of Contents
$$\label{toc}$$

This tutorial is organized as follows

1. [Step 1](#initializenrpy): Call on NRPy+ to convert the SymPy expressions for $b^{\mu}$, $b^2$, and the Poynting Vector $S^i$ to C code kernels
1. [Step 2](#etk): Build up the needed Einstein Toolkit infrastructure to implement the NRPy+-generated C code kernels
    1. [Step 2.a](#etkc): Write the C code functions called by the Einstein Toolkit scheduler that incorporate the above ".h" files
    1. [Step 2.b](#cclfiles): CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure
    1. [Step 2.c](#etksys): Inform the Einstein Toolkit build system of the C code
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Call on NRPy+ to convert the SymPy expressions for $b^{\mu}$, $b^2$, and the Poynting Vector $S^i$ to C code kernels \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step 1a: import all needed modules from NRPy+:
import indexedexp as ixp                   # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
from outputC import outputC                # NRPy+: Basic C code output functionality
import grid as gri                         # NRPy+: Functions having to do with numerical grids
import NRPy_param_funcs as par             # NRPy+: parameter interface
import sympy as sp                         # SymPy: The Python computer algebra package upon which NRPy+ depends
```

```python
# Step 1b: Initialize  parameters (stub; there are none for this module)
thismodule = __name__
```

We will to disable verbose output in the NRPy+ outputC function. This is an important step in this case, because our final expressions are very large. Verbose output, when enabled, will print (in comments) the input SymPy expressions to the top of the file *without* CSE, resulting here in an *enormous* output file.

We will also declare the additional gridfunctions we need for this thorn:

**Inputs from ADMBase:**
* the physical metric $\gamma_{ij}$
* the spacetime gauge quantities $\alpha$ and $\beta^i$

**Inputs from HydroBase:**
* the Valencia 3-velocity $v^i_{(n)}$
* the densitized magnetic field of a normal observer $\tilde{B}^i$

**Output gridfunctions:**
* the magnetic field as observed in a frame comoving with the plasma $b^\mu$ (`smallb4U[]}`)
* twice the magnetic pressure $2 P_{\rm mag} = b_\mu b^\mu = b^2$ (`smallb2etk`)
* the Poynting vector $S^i$ (`SPoyn[]`)

```python
# Step 1c: Set spatial dimension (must be 3 for BSSN)
DIM = 3
par.set_parval_from_str("grid::DIM",DIM)

# Step 1d: declare the additional gridfunctions (i.e., functions whose values are declared
#          at every grid point, either inside or outside of our SymPy expressions) needed
#         for this thorn

# INPUT GRIDFUNCTIONS:
gammaDD = ixp.register_gridfunctions_for_single_rank2("AUX","gammaDD", "sym01") # The AUX or EVOL designation is *not*
                                                                                # used in diagnostic modules.
betaU = ixp.register_gridfunctions_for_single_rank1("AUX","betaU") # The AUX or EVOL designation is *not*
                                                                   # used in diagnostic modules.
alpha = gri.register_gridfunctions("AUX","alpha") # The AUX or EVOL designation is *not*
                                                  # used in diagnostic modules.
ValenciavU = ixp.register_gridfunctions_for_single_rank1("AUX","ValenciavU") # The AUX or EVOL designation is *not*
                                                                             # used in diagnostic modules.
BU = ixp.register_gridfunctions_for_single_rank1("AUX","BU") # The AUX or EVOL designation is *not*
                                                             # used in diagnostic modules.

# OUTPUT GRIDFUNCTIONS:
smallb4U = ixp.register_gridfunctions_for_single_rank1("AUX","smallb4U",DIM=4) # The AUX or EVOL designation is *not*
                                                                               # used in diagnostic modules.
smallb2etk = gri.register_gridfunctions("AUX","smallb2etk") # The AUX or EVOL designation is *not*
                                                            # used in diagnostic modules.
PoynSU = ixp.register_gridfunctions_for_single_rank1("AUX","PoynSU") # The AUX or EVOL designation is *not*
                                                                     # used in diagnostic modules.

# Step 1f: Call the NRPy+ module to set up the SymPy expressions for the output, as well as the C code for computing u^0
import u0_smallb_Poynting__Cartesian.u0_smallb_Poynting__Cartesian as u0etc
u0etc.compute_u0_smallb_Poynting__Cartesian(gammaDD,betaU,alpha,ValenciavU,BU)

# Step 1g: Set the gridfunction memory access type to "ETK":
par.set_parval_from_str("GridFuncMemAccess","ETK")

# Step 1h: Make output directories:
!mkdir sbPoynETNRPy     2>/dev/null # 2>/dev/null: Don't throw an error or warning if the directory already exists.
!mkdir sbPoynETNRPy/src 2>/dev/null # 2>/dev/null: Don't throw an error or warning if the directory already exists.

# Step 1i: Output routine for computing u0:
with open("sbPoynETNRPy/src/u0.h", "w") as file:
    file.write(str(u0etc.computeu0_Cfunction))
    print("Wrote to file \""+file.name+"\"")

# Step 1j: Use NRPy+'s outputC to convert the SymPy expressions for smallb4U, smallb2etk, and PoynSU to C code:
#outputC([u0etc.smallb4U[0],u0etc.smallb4U[1],u0etc.smallb4U[2],u0etc.smallb4U[3],u0etc.smallb2etk,
outputC([u0etc.smallb4U[0],u0etc.smallb4U[1],u0etc.smallb4U[2],u0etc.smallb4U[3],u0etc.smallb2etk,
         u0etc.PoynSU[0],u0etc.PoynSU[1],u0etc.PoynSU[2]],
         [gri.gfaccess("","smallb4U0"),gri.gfaccess("","smallb4U1"),gri.gfaccess("","smallb4U2"),gri.gfaccess("","smallb4U3"),
          gri.gfaccess("","smallb2etk"),
          gri.gfaccess("","PoynSU0"),gri.gfaccess("","PoynSU1"),gri.gfaccess("","PoynSU2")],
       filename="sbPoynETNRPy/src/smallb4U_smallb2etk_PoynSU.h",
       params="outCverbose=False") # <- Force outCverbose=False for this
                                   #    module to avoid gigantic C file filled with the
                                   #    non-CSE expressions for the Weyl scalars.
```

    Wrote to file "sbPoynETNRPy/src/u0.h"
    Wrote to file "sbPoynETNRPy/src/smallb4U_smallb2etk_PoynSU.h"

# Step 2: Build up the needed Einstein Toolkit infrastructure to implement the NRPy+-generated C code kernels \[Back to [top](#toc)\]
$$\label{etk}$$

## Step 2.a: Write the C code functions called by the Einstein Toolkit scheduler that incorporate the above ".h" files \[Back to [top](#toc)\]
$$\label{etkc}$$

```python
%%writefile sbPoynETNRPy/src/sbPoynETNRPy.c

#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

void sbPoynETNRPy_lowlevel(const cGH* restrict const cctkGH,const int *cctk_lsh,
                           const CCTK_REAL *gammaDD00GF,const CCTK_REAL *gammaDD01GF,const CCTK_REAL *gammaDD02GF,
                           const CCTK_REAL *gammaDD11GF,const CCTK_REAL *gammaDD12GF,const CCTK_REAL *gammaDD22GF,
                           const CCTK_REAL *alphaGF,
                           const CCTK_REAL *betaU0GF,const CCTK_REAL *betaU1GF,const CCTK_REAL *betaU2GF,
                           const CCTK_REAL *vel,const CCTK_REAL *Bvec,

                           CCTK_REAL *smallb4U0GF,CCTK_REAL *smallb4U1GF,CCTK_REAL *smallb4U2GF,CCTK_REAL *smallb4U3GF,
                           CCTK_REAL *smallb2etkGF,
                           CCTK_REAL *PoynSU0GF,CCTK_REAL *PoynSU1GF,CCTK_REAL *PoynSU2GF) {

    DECLARE_CCTK_PARAMETERS;

#pragma omp parallel for
    for(int i2=0;i2<cctk_lsh[2];i2++) for(int i1=0;i1<cctk_lsh[1];i1++) for(int i0=0;i0<cctk_lsh[0];i0++) {
        const CCTK_REAL gammaDD00 = gammaDD00GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL gammaDD01 = gammaDD01GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL gammaDD02 = gammaDD02GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL gammaDD11 = gammaDD11GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL gammaDD12 = gammaDD12GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL gammaDD22 = gammaDD22GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];

        const CCTK_REAL alpha = alphaGF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];

        const CCTK_REAL betaU0 = betaU0GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL betaU1 = betaU1GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];
        const CCTK_REAL betaU2 = betaU2GF[CCTK_GFINDEX3D(cctkGH, i0,i1,i2)];

        // Valencia 3-velocity may be adjusted due to the velocity ceiling.
        CCTK_REAL ValenciavU0 = vel[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 0)];
        CCTK_REAL ValenciavU1 = vel[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 1)];
        CCTK_REAL ValenciavU2 = vel[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 2)];

        const CCTK_REAL BU0 = Bvec[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 0)];
        const CCTK_REAL BU1 = Bvec[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 1)];
        const CCTK_REAL BU2 = Bvec[CCTK_GFINDEX4D(cctkGH, i0,i1,i2, 2)];

        CCTK_REAL u0;
#include "u0.h"
#include "smallb4U_smallb2etk_PoynSU.h"
    }
}

extern void sbPoynETNRPy(CCTK_ARGUMENTS) {

    DECLARE_CCTK_PARAMETERS;
    DECLARE_CCTK_ARGUMENTS;

    if(sbPoynETNRPy_calc_every<=0 || cctk_iteration%sbPoynETNRPy_calc_every!=0) { return; }

    /* Calculate smallb4U[], smallb2etk, and PoynSU[]: */
    sbPoynETNRPy_lowlevel(cctkGH,cctk_lsh,
                          gxx,gxy,gxz,gyy,gyz,gzz,
                          alp,
                          betax,betay,betaz,
                          vel,Bvec,

                          smallb4U0,smallb4U1,smallb4U2,smallb4U3,
                          smallb4_sq,
                          PoynSU0,PoynSU1,PoynSU2);
}
```

    Overwriting sbPoynETNRPy/src/sbPoynETNRPy.c

## Step 2.b: CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure \[Back to [top](#toc)\]
$$\label{cclfiles}$$

Writing a module ("thorn") within the Einstein Toolkit requires that three "ccl" files be constructed, all in the root directory of the thorn:

1. `interface.ccl`: defines the gridfunction groups needed, and provides keywords denoting what this thorn provides and what it should inherit from other thorns.
1. `param.ccl`: specifies free parameters within the thorn.
1. `schedule.ccl`: allocates storage for gridfunctions, defines how the thorn's functions should be scheduled in a broader simulation, and specifies the regions of memory written to or read from gridfunctions.

Let's start with `interface.ccl`. The [official Einstein Toolkit (Cactus) documentation](http://einsteintoolkit.org/usersguide/UsersGuide.html) defines what must/should be included in an `interface.ccl` file [**here**](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-178000D2.2).

```python
%%writefile sbPoynETNRPy/interface.ccl

# With "implements", we give our thorn its unique name.
implements: sbPoynETNRPy

# By "inheriting" other thorns, we tell the Toolkit that we
#   will rely on variables/function that exist within those
#   functions.
inherits:   ADMBase Boundary Grid HydroBase MethodofLines

# Tell the Toolkit that we want the various Weyl scalars
#    and invariants to be visible to other thorns by using
#    the keyword "public". Note that declaring these
#    gridfunctions *does not* allocate memory for them;
#    that is done by the schedule.ccl file.
public:
CCTK_REAL smallb4U_group type=GF timelevels=3
{
  smallb4U0,smallb4U1,smallb4U2,smallb4U3
} "smallb4U 4-vector"

public:
CCTK_REAL smallb4_sq_group type=GF timelevels=3
{
  smallb4_sq
} "smallb^{mu} squared == twice the magnetic pressure"

public:
CCTK_REAL PoynSU_group type=GF timelevels=3
{
  PoynSU0,PoynSU1,PoynSU2
} "Poynting 3-vector"
```

    Overwriting sbPoynETNRPy/interface.ccl

We will now write the file `param.ccl`. This file allows the listed parameters to be set at runtime. We also give allowed ranges and default values for each parameter. More information on this file's syntax can be found in the [official Einstein Toolkit documentation](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-183000D2.3).

The first parameter specifies how many time levels need to be stored. Generally when using the ETK's adaptive-mesh refinement (AMR) driver [Carpet](https://carpetcode.org/), three timelevels are needed so that the diagnostic quantities can be properly interpolated and defined across refinement boundaries.

The second parameter determines how often we will calculate $b^\mu$, $b^2$, and $S^i$.

The third parameter sets the maximum allowed Lorentz factor when computing $u^0$ (i.e., $\Gamma_{\rm max}$, as defined in the [previous tutorial notebook](Tutorial-u0_smallb_Poynting-Cartesian.ipynb)).

```python
%%writefile sbPoynETNRPy/param.ccl

shares: HydroBase
USES CCTK_INT timelevels

restricted:
CCTK_INT timelevels "Number of active timelevels" STEERABLE=RECOVER
{
  0:3 :: ""
} 3

restricted:
CCTK_INT sbPoynETNRPy_calc_every "Compute these quantities every sbPoynETNRPy_calc_every iterations." STEERABLE=ALWAYS
{
  *:* :: ""
} 1

restricted:
CCTK_REAL GAMMA_SPEED_LIMIT "Maximum Lorentz factor."
{
 1:* :: "Positive > 1, though you'll likely have troubles in GRMHD far above 10, or far above 2000 in GRFFE."
} 10.0

```

    Overwriting sbPoynETNRPy/param.ccl

Finally, we will write the file `schedule.ccl`; its official documentation is found [here](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-186000D2.4).

This file registers the function we wish to call, `sbPoynETNRPy`, with the Einstein Toolkit scheduler.

```python
%%writefile sbPoynETNRPy/schedule.ccl

STORAGE: smallb4U_group[timelevels]
STORAGE: smallb4_sq_group[timelevels]
STORAGE: PoynSU_group[timelevels]

schedule group sbPoynETNRPy_group in MoL_PseudoEvolution after ADMBase_SetADMVars
{
} "Schedule sbPoynETNRPy group"

schedule sbPoynETNRPy in sbPoynETNRPy_group
{
    LANG: C
    READS: admbase::gxx(Everywhere)
    READS: admbase::gxy(Everywhere)
    READS: admbase::gxz(Everywhere)
    READS: admbase::gyy(Everywhere)
    READS: admbase::gyz(Everywhere)
    READS: admbase::gzz(Everywhere)

    READS: admbase::alpha(Everywhere)

    READS: admbase::betax(Everywhere)
    READS: admbase::betay(Everywhere)
    READS: admbase::betaz(Everywhere)

    READS: HydroBase::vel(Everywhere)
    READS: HydroBase::Bvec(Everywhere)

    WRITES: sbPoynETNRPy::smallb4U0(Everywhere)
    WRITES: sbPoynETNRPy::smallb4U1(Everywhere)
    WRITES: sbPoynETNRPy::smallb4U2(Everywhere)
    WRITES: sbPoynETNRPy::smallb4U3(Everywhere)

    WRITES: sbPoynETNRPy::smallb4_sq(Everywhere)

    WRITES: sbPoynETNRPy::PoynSU0(Everywhere)
    WRITES: sbPoynETNRPy::PoynSU1(Everywhere)
    WRITES: sbPoynETNRPy::PoynSU2(Everywhere)
} "Call sbPoynETNRPy main function, to compute $b^mu$, $b^2$, and $S^i$"
```

    Overwriting sbPoynETNRPy/schedule.ccl

## Step 2.c: Inform the Einstein Toolkit build system of the C code \[Back to [top](#toc)\]
$$\label{etksys}$$

The `make.code.defn` lists the source files that need to be compiled. Naturally, this thorn has only the one C file $-$ written above $-$ to compile:

```python
%%writefile sbPoynETNRPy/src/make.code.defn

SRCS = sbPoynETNRPy.c
```

    Overwriting sbPoynETNRPy/src/make.code.defn

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-ETK_thorn-u0_smallb_Poynting.pdf](Tutorial-ETK_thorn-u0_smallb_Poynting.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-ETK_thorn-u0_smallb_Poynting")
```

    Created Tutorial-ETK_thorn-u0_smallb_Poynting.tex, and compiled LaTeX file
        to PDF file Tutorial-ETK_thorn-u0_smallb_Poynting.pdf