## Contents of this notebook

- 0: Load packages, scripts etc.
- 1: Construct trees. Uses the technology data stored in an .xlsx-file to construct dictionaries and other objects that fully characterize the production tree. These are then combined to an actual tree object using the nestingtree.nestingtree class.
- 2: Adding parameters to the database. Only some of the share-parameters etc. can be deduced directly from technology data. The rest of these as well as starting values for endogenous and exogenous variables must be collected in a database.
- 3: The model. Uses the tree object as well as the prepared database to construct a model using the *gmspython* class and its childclass *abate*.
The script ends by exporting the model as a pickle to be loaded in the calibration script.

### Loads

```python
clean_up=True # removes gams-related files in work-folder if true
%run StdPackages.ipynb
os.chdir(directory['py'])
import abatement, techdata_to_tree, sys, ShockFunction
os.chdir(directory["curr"])
data_folder = os.getcwd()+'\\Data'
gams_folder = data_folder + "\\..\\gamsmodels\\Main"
#Functions
def flatten_list(list_):
    return [item for sublist in list_ for item in sublist]
```

# **1: Construct trees**

#### Load technology data.
This runs the script in located in the file *techdata_to_tree.py*:

```python
# inputfile = "techdata_dors_2.xlsx"
inputfile = "techdata_dors_3_overlap.xlsx"
output = techdata_to_tree.load_techcats(pd.read_excel(data_folder + "/" + inputfile, sheet_name=["inputdisp", "endofpipe", "inputprices"]))
```

#### The output of the code is a dictionary with three keys, referring to the two modules + a list of inputprices stored in the xlsx-file as well:

```python
output.keys()
```

    dict_keys(['ID', 'inputprices', 'EOP'])

The two different modules correspond to the two types of technology catalogs that the model can handle.\
*Input-displacing* (ID) and *End of pipe* (EOP).

```python
modules = ["ID", "EOP"]
```

ID contains a dictionary related to input-displacing technologies, EOP does so for end of pipe (we use this naming convention throughout).\
They contain the following keys:

```python
for module in modules:
    print("Keys of " + module + ": ", output[module].keys(), "\n")
```

    Keys of ID:  dict_keys(['techs_inputs', 'techs', 'components', 'upper_categories', 'mu', 'Q2P', 'unit_costs', 'current_coverages', 'current_coverages_split', 'coverage_potentials', 'basetechs', 'basetech_inputs'])

    Keys of EOP:  dict_keys(['techs_inputs', 'techs', 'components', 'upper_categories', 'mu', 'Q2P', 'unit_costs', 'current_coverages', 'current_coverages_split', 'coverage_potentials'])

As an example, "techs" shows, for each technology, which technology good it produces, e.g. for input-displacing:

```python
output["ID"]["techs"]
```

    {'ID_t2': ['U_ID_t2_1', 'U_ID_t2_2'],
     'ID_t3': ['U_ID_t3_1'],
     'ID_t4': ['U_ID_t4_1', 'U_ID_t4_2']}

And "upper_categories" shows the mapping between energy services (E) and their respective components (C) for ID.\
All of the trees in the `output` object are dictionaries where the keys are nodes in the tree and the values are lists of connected branches.

```python
output["ID"]["upper_categories"]
```

    {'EH': ['C_EH_1', 'C0_EH'], 'EL': ['C_EL_1', 'C_EL_2', 'C_EL_3', 'C0_EL']}

... and the mapping between emission types M ($CO_2$, $SO_2$ etc.) and the components C.\
Note importantly that this mapping does not constitute an actual part of the tree, because components are the outputs of the EOP sector.

```python
output["EOP"]["upper_categories"]
```

    {'CO2': ['C_CO2_1', 'C_CO2_2']}

### Initialize nesting tree, call it "Abatement"
The construction of the production tree starts with an initialization of the *nesting_tree* class:

```python
nts = {"ID":nesting_tree.nesting_tree(name="ID"), "EOP":nesting_tree.nesting_tree(name="EOP")}
```

The "trees" attribute starts off empty, reflecting that no (sub)trees have been added yet.

```python
nts["ID"].trees
```

    {}

#### The following cells add each of the subtrees to the tree-object, "nt"

##### First, input-displacing subtrees

Upper part, connecting energy services to components (this is an input-tree, so we do not supply additional keyword arguments).\
Again, we use the naming convention that a prefix reflects whether the tree is related to input-displacement (ID) or end of pipe (EOP).
The prefix is followed by an underscore and then letters reflecting which elements in the tree are connected.

The first tree connects energy services (E) to components (C), related to input-displacing technologies (ID)

```python
nts["ID"].add_tree(output["ID"]["upper_categories"], tree_name = 'ID_EC', **{"type_f":"CES_norm"})
```

Before proceeding to the next tree, lets print some information from this tree.\
First, we see that the tree has been added to the list of trees in the "aggregate" tree, nt:

```python
nts["ID"].trees
```

    {'ID_EC': <nesting_trees.nt at 0x23349e3e088>}

Second, information about the tree is automatically added when the tree is added, e.g. the "type_f" attribute states that the functional form in this subtree is constant elasticity of substitution (CES)

```python
nts["ID"].trees["ID_EC"].__dict__
```

    {'name': 'ID_EC',
     'tree': {'EH': ['C_EH_1', 'C0_EH'],
      'EL': ['C_EL_1', 'C_EL_2', 'C_EL_3', 'C0_EL']},
     'type_io': 'input',
     'version': 'std',
     'n': 'n',
     'nn': 'nn',
     'nnn': 'nnn',
     'map_': 'map_ID_EC',
     'kno': 'kno_ID_EC',
     'bra': 'bra_ID_EC',
     'inp': 'inp_ID_EC',
     'out': 'out_ID_EC',
     'temp_namespace': None,
     'type_f': 'CES_norm',
     'database': <DataBase.GPM_database at 0x23349e3e388>}

However, even though the namespace includes e.g. "map_":"map_ID_EC", the database of the tree still does not contain this mapping. This requires running the run() method.\
Instead of doing that for each tree individually, we instead add all trees and call the run_all() methods which also calls the run() on each tree.

Next, we add the middle part, connecting components C to their technology goods U

```python
nts["ID"].add_tree(output["ID"]["components"], tree_name = "ID_CU", **{"type_f":"MNL"})
```

The two remaining parts are the bottom ones. \
First, the one that connects technologies to their outputs (technology goods). That these are outputs is specified explicitly by using the `type_io` keyword.\
Second, the one that connects technologies to their inputs (non-capital inputs X, and capital K)

```python
nts["ID"].add_tree(output["ID"]["techs"], tree_name="ID_TU", **{'type_io': 'output', 'type_f': 'CET_norm'})
nts["ID"].add_tree(output["ID"]["techs_inputs"], tree_name="ID_TX")
```

Baseline components and technology goods, and their respective sets of inputs are also part of the aggregate tree. We distinguish between those with their own inputs (these baseline technologies are the ones that come from knowing which energy mix a (set of) technologies replaces. The remaining baseline technology goods (and all baseline components by construction) are outputs of the "IO technology". This IO technology in turn draws on all inputs from the economy. This is the one we calibrate to make sure we replicate IO data.

```python
nts["ID"].add_tree(output["ID"]["basetechs"], tree_name="ID_BU", **{"type_io":"output", "type_f":"linear_out"})
nts["ID"].add_tree(output["ID"]["basetech_inputs"], tree_name="ID_BX")
```

```python
# if output["ID"]["baseline_U_inputs"]:
#     nts["ID"].add_tree(output["ID"]["baseline_U_inputs"], tree_name="ID_UbaseX")
# else:
#     print("No replacing baseline technology goods")
```

#### Next, we add the end of pipe subtrees.
First, the one connecting components (which are the final product in the end-of-pipe tree) to the technology goods from which they are created:

```python
nts["EOP"].add_tree(output["EOP"]["components"], tree_name = "EOP_CU", **{"type_f":"MNL"})
```

Second, the bottom part. This consists of technologies and their outputs and inputs respectively, similarly to with input-displacing technologies:

```python
nts["EOP"].add_tree(output["EOP"]["techs"], tree_name="EOP_TU", **{'type_io': 'output', 'type_f': 'CET_norm'})
nts["EOP"].add_tree(output["EOP"]["techs_inputs"], tree_name="EOP_TX")
```

#### Trees related to final goods: One taking energy services and remaining inputs to produce a composite good Y, and one splitting Y into sX

```python
Y_inputs = list(output["ID"]["upper_categories"].keys())
Y_outputs = ["Y_out_" + c for c in list(output["inputprices"].keys()[:-1])]
```

```python
Q2P_addition = []
for inp in list(output["inputprices"].index[:-1]) + ["K"]:
    Y_inputs += ["Y_in_" + inp]
    Q2P_addition.append(("Y_in_" + inp, inp))
```

```python
output["ID"]["Q2P"] = output["ID"]["Q2P"].append(pd.MultiIndex.from_tuples(Q2P_addition, names=["n", "nn"]))
```

```python
#Share parameters are simply 1/N on both the input and output side
for inp in Y_inputs:
    output["ID"]["mu"][(inp, "Y")] = 1/len(Y_inputs)
for out in Y_outputs:
    output["ID"]["mu"][(out, "Y")] = 1/len(Y_outputs)
```

```python
nts["ID"].add_tree({"Y":Y_inputs}, tree_name="ID_Y_in")
nts["ID"].add_tree({"Y":Y_outputs}, tree_name="ID_Y_out", **{'type_io': 'output', 'type_f': 'CET'})
```

##### Now, all trees have been added:

```python
print(nts["ID"].trees, "\n", nts["EOP"].trees)
```

    {'ID_EC': <nesting_trees.nt object at 0x0000023349E3E088>, 'ID_CU': <nesting_trees.nt object at 0x0000023349E42608>, 'ID_TU': <nesting_trees.nt object at 0x0000023349E3C7C8>, 'ID_TX': <nesting_trees.nt object at 0x0000023349E3CC08>, 'ID_BU': <nesting_trees.nt object at 0x0000023349E2FB88>, 'ID_BX': <nesting_trees.nt object at 0x0000023349E2FF48>, 'ID_Y_in': <nesting_trees.nt object at 0x0000023349D28648>, 'ID_Y_out': <nesting_trees.nt object at 0x0000023349D28708>}
     {'EOP_CU': <nesting_trees.nt object at 0x0000023349D3C848>, 'EOP_TU': <nesting_trees.nt object at 0x0000023349D36448>, 'EOP_TX': <nesting_trees.nt object at 0x0000023349D36488>}

The next step is to use the method `run_all`. This runs through a number of steps, where it sets up sets/subsets/mappings identifying which elements e.g. are inputs, intermediate goods, and outputs in the aggregate tree, i.e. the combination of the subtrees we just added. *Tutorial_nesting_tree* includes a brief review of these.\
There are still some of the objects in `output` that we have not used. We will return to these later as they become relevant.

```python
nts["ID"].__dict__
```

    {'name': 'ID',
     'version': 'std',
     'trees': {'ID_EC': <nesting_trees.nt at 0x23349e3e088>,
      'ID_CU': <nesting_trees.nt at 0x23349e42608>,
      'ID_TU': <nesting_trees.nt at 0x23349e3c7c8>,
      'ID_TX': <nesting_trees.nt at 0x23349e3cc08>,
      'ID_BU': <nesting_trees.nt at 0x23349e2fb88>,
      'ID_BX': <nesting_trees.nt at 0x23349e2ff48>,
      'ID_Y_in': <nesting_trees.nt at 0x23349d28648>,
      'ID_Y_out': <nesting_trees.nt at 0x23349d28708>},
     'database': <DataBase.GPM_database at 0x23349de66c8>}

```python
standard_sets = ('inp','out','int','wT','map_all','kno_out','kno_inp')
for module in modules:
    namespace = {k: module + '_' + k for k in standard_sets}
    nts[module].run_all(**namespace)
    #Also replaces keys with module-specific names, i.e. changes 'inp' to 'ID_inp' in the attributes/keys.
    for std_set in standard_sets:
        if hasattr(nts[module], std_set):
            setattr(nts[module], module + "_" + std_set, getattr(nts[module], std_set))
            delattr(nts[module], std_set)
```

This constructs the aggregate tree. \
For a few highlights, let's check:
1. The outputs of the aggregate tree (aggregate simply refers to the combination of all the individual (sub)trees\
This includes energy services (from ID) and components (from EOP)

```python
list(nts["ID"].database.series["ID_out"].vals)
```

    ['Y_out_inp3', 'Y_out_electricity', 'Y_out_oil']

2. The outputs of a particular tree, say EOP_TU. Note that in this particular case, the reported list is empty. This is because an 'output' refers to whether it is an output of the aggregate tree and not just this particular one.

```python
list(nts["EOP"].trees["EOP_TU"].database.series["out_EOP_TU"].vals)
```

    ['EOP_t2', 'EOP_t1']

3. Likewise, we can check the inputs of EOP_TU\
This reports that the Us are inputs, which they really are not. This is not a problem, because the tree is constructed in the correct way. But not that U's are not inputs. Not of the aggregate tree (in which Us are intermediates), but not in the TU-tree either, since here, they are outputs (they are produced by technologies).

```python
nts["EOP"].trees["EOP_TU"].database.series["inp_EOP_TU"].vals
```

    Index(['U_EOP_t1_1', 'U_EOP_t2_1'], dtype='object', name='n')

Lastly, lets print the objects that the `nt` class instance contains:

```python
nts["ID"].__dict__
```

    {'name': 'ID',
     'version': 'std',
     'trees': {'ID_EC': <nesting_trees.nt at 0x23349e3e088>,
      'ID_CU': <nesting_trees.nt at 0x23349e42608>,
      'ID_TU': <nesting_trees.nt at 0x23349e3c7c8>,
      'ID_TX': <nesting_trees.nt at 0x23349e3cc08>,
      'ID_BU': <nesting_trees.nt at 0x23349e2fb88>,
      'ID_BX': <nesting_trees.nt at 0x23349e2ff48>,
      'ID_Y_in': <nesting_trees.nt at 0x23349d28648>,
      'ID_Y_out': <nesting_trees.nt at 0x23349d28708>},
     'database': <DataBase.GPM_database at 0x23349de66c8>,
     'n': 'n',
     'nn': 'nn',
     'nnn': 'nnn',
     'fg': 'fg',
     'prune_trees': {'OnlyQ', 'bra', 'inp', 'kno', 'out'},
     'ID_inp': 'ID_inp',
     'ID_out': 'ID_out',
     'ID_int': 'ID_int',
     'ID_wT': 'ID_wT',
     'ID_map_all': 'ID_map_all',
     'ID_kno_out': 'ID_kno_out',
     'ID_kno_inp': 'ID_kno_inp'}

## 1.5: Construct some sets needed before calculating starting values

```python
def multiindex_series(idx_level_names, idx_name=None, series_name=None):
    if idx_name is None and series_name is not None:
        idx_name = series_name
    elif idx_name is not None and series_name is None:
        series_name = idx_name
    elif idx_name is None and series_name is None:
        raise Exception("Supply either index name or series name")
    idx = pd.MultiIndex(levels=[[]]*len(idx_level_names), codes=[[]]*len(idx_level_names), names=idx_level_names)
    ser = pd.Series(index=idx, dtype=float)
    ser.rename(series_name, inplace=True)
    #ser.index.name = idx_name
    return ser
```

```python
def find_key_from_value(d, value):
    assert isinstance(d, dict)
    out = []
    for (k, v) in d.items():
        if value in v:
            out.append(k)
    if len(out) == 1:
        return out[0]
    else:
        raise Exception("Value exists in multiple keys")
```

```python
def find_true_input(inp, Q2P):
    true = list(Q2P[Q2P.get_level_values(0).isin([inp])].get_level_values(1))[0]
    return true
```

```python
def find_str_in_name(name, energy_services):
    found_e = [e for e in energy_services if e in name]
    if len(found_e) != 1:
        raise Exception("found zero or more than 1 E")
    else:
        return found_e[0]
```

```python
def find_X_in_name(name, inputs):
    found = [X for X in inputs if X in name]
    if len(found) != 1:
        raise Exception("found zero or more than 1 E")
    else:
        return found[0]
```

```python
#sumX. The sum of energy use for baseline technologies (IO_tech + replacing baselines)

# sumXinEaggs = pd.Series([], index=)

sumXinE2baselineinputs = multiindex_series(idx_level_names=["n", "nn"], series_name="map_sumXinE2baselineinputs")
sumXinE2E = multiindex_series(idx_level_names=["n", "nn"], series_name="map_sumXinE2E")
sumXinE2X = multiindex_series(idx_level_names=["n", "nn"], series_name="map_sumXinE2X")

for basetech in output["ID"]["basetech_inputs"]:
    e = find_str_in_name(basetech, list(output["ID"]["upper_categories"].keys()))
    for x in output["ID"]["basetech_inputs"][basetech]:
        inp = find_true_input(x, output["ID"]["Q2P"])
        sumXinE2baselineinputs[("sum_" + e + "_" + inp, x)] = np.nan
```

```python
#All the sumXinE aggregates can now be fetched from the mapping just created:
sumXinEaggs = sumXinE2baselineinputs.index.get_level_values(0).unique()
```

```python
#mapping from sumXinE to E, simply extracted from its name
energy_services = list(output["ID"]["upper_categories"].keys())
for agg in sumXinEaggs:
    sumXinE2E[(agg, find_str_in_name(agg, energy_services))] = np.nan
```

```python
#mapping from sumXinE to Xs below explicit technologies

for sumx in sumXinEaggs:
    pure_x_of_sumx = find_X_in_name(sumx, list(output["inputprices"].index))
    for t in output["ID"]["techs_inputs"]:
        for t_x in output["ID"]["techs_inputs"][t]:
            pure_x_of_t_x = find_true_input(t_x, output["ID"]["Q2P"])
            if pure_x_of_sumx == pure_x_of_t_x:
                sumXinE2X[(sumx, t_x)] = np.nan

```

```python
#The sum of inputs outside of energy services. For ID, this is simply what goes into final goods production. For EOP, inputs used in the entire EOP are added
sumXrest2X = {"ID":multiindex_series(idx_level_names=["n", "nn"], series_name="sumXrest"), "EOP":multiindex_series(idx_level_names=["n", "nn"], series_name="sumXrest")}
```

```python
for Y_inp in Y_inputs:
    if Y_inp not in output["ID"]["upper_categories"]:
        #if not an E, add to sumX2X
        for module in modules:
            sumXrest2X[module][("sumXrest_" + find_true_input(Y_inp, output["ID"]["Q2P"]), Y_inp)] = np.nan
```

```python
#Create the set of aggregates for sumXrest:
sumXrestaggs = sumXrest2X["ID"].index.get_level_values(0).unique()
```

```python
for sumx in sumXrestaggs:
    pure_x_of_sumx = find_X_in_name(sumx, list(output["inputprices"].index))
    for t in output["EOP"]["techs_inputs"]:
        for t_x in output["EOP"]["techs_inputs"][t]:
            pure_x_of_t_x = find_true_input(t_x, output["EOP"]["Q2P"])
            if pure_x_of_sumx == pure_x_of_t_x:
                sumXrest2X["EOP"][(sumx, t_x)] = np.nan
```

```python

```

```python
#change name of the sum of an input to "sum_x"
# sumX = {"ID":sumX["ID"].reset_index(), "EOP":sumX["EOP"].reset_index()}
# sumX2X = {}
# for module in modules:
#     sumX[module]["n"] = "sum_" + sumX[module]["n"]
#     sumX2X[module] = sumX[module].set_index(["n", "nn"])["sumX"].index
```

```python
phi = multiindex_series(idx_level_names=["n", "nn"], series_name="phi")
M0 = pd.Series([], name="M0", dtype="float64")
M = pd.Series([], name="M", dtype="float64")
#Here sets emission intensity equal to 0.1 by construction for now
```

```python
#add each input to phi as well (needed for the Phat equation)
default_emsint = 0
emission_intensities = {
    "oil":{
        "CO2":0.5
    },
    "electricity":{
        "CO2":0.00
    },
    "inp3":{
        "CO2":0.0
    }

}
for m in output["EOP"]["upper_categories"].keys():
    M0[m] = 5 #Emissions default simply chosen to be 5
    M[m] = 5
    for x in sumXrest2X["EOP"].append(sumXinE2X).index.get_level_values(1):
        true_input = find_true_input(x, output["ID"]["Q2P"].append(output["EOP"]["Q2P"]))
        if (true_input in emission_intensities and m in emission_intensities[true_input]):
            phi[(m, x)] = emission_intensities[true_input][m]
        else:
            phi[(m, x)] = default_emsint
    #baseline technologies' inputs
    for basetech in output["ID"]["basetech_inputs"]:
        for x in output["ID"]["basetech_inputs"][basetech]:
            true_input = find_true_input(x, output["ID"]["Q2P"])
            if (true_input in emission_intensities and m in emission_intensities[true_input]):
                phi[(m, x)] = emission_intensities[true_input][m]
            else:
                phi[(m, x)] = default_emsint
```

```python
#Emission prices,  and theta which measures the potentials of components.
pM = M.copy()
pM[:] = 5
pM.name = "pM"
pM.index.name = "n"

pMhat = M.copy()
pMhat[:] = 5
pMhat.name = "pMhat"
pMhat.index.name = "n"
```

# **2: Adding parameters to the database (starting values and share parameters)**

The next step is to construct a database that contains all the share-parameters that we can deduce from technology data, as well as appropriate starting values for endogenous variables. These starting values will be set to the value that they would have if the entire tree was Leontief.
The end goal is to make sure the database contained by `nt` includes all these parameters and starting values.
To do so, we construct an empty database, add the share-parameters and starting values to this, and then finally merge that database with the one already in `nt`.

The GPM database of the nesting_tree instance (`nt`) does not contain the $\mu$ parameters calculated from technology data yet.\
The $\mu$-parameters are stored under the key of the same name in `output`.\
Let's check out the database of `nt` now, to see that it does not include any symbol called $\mu$ yet:

```python
print("The database nt.database is of type " + str(type(nts["ID"].database)))
print("Does the database include a symbol called mu? ")
if "mu" in nts["ID"].database.series:
    print("YES!")
else:
    print("NO!")
```

    The database nt.database is of type <class 'DataBase.GPM_database'>
    Does the database include a symbol called mu?
    NO!

First we create an empty database:

```python
dbs = {"ID":DataBase.GPM_database(), "EOP":DataBase.GPM_database()}
```

We store the share-parameters from technology data in an object called `mu`. This does not contain all share-parameters of the tree (e.g. it does not include the share parameters of the inputs under baseline technology goods, or the share parameters for technology goods under components).

```python
mu = {"ID":output["ID"]["mu"].copy(), "EOP":output["EOP"]["mu"].copy()}
# for module in modules:
#     mu[module].name = module + "_" + mu[module].name
```

Check out the mu parameters (prints just the five first entries here).\
Note: The mu-object is a series with a multiindex, where the first level is called `n` and the second level `nn`. The first level contains branches and the second level contains nodes.

```python
mu["ID"].head()
```

    n                  nn
    ID_t2_electricity  ID_t2     1.00
    ID_t2_K            ID_t2    24.00
    C_EL_1             EL        0.08
    U_ID_t2_1          ID_t2     0.80
    U_ID_t2_2          ID_t2     0.20
    Name: mu, dtype: float64

#### Calculate starting values (corresponding to the solution if the entire tree was Leontief).
For demands/quantities, we use the fact that Leontief demand is given by
$$q_j = \mu_j \left(\frac{p_i}{p_j}\right)^\sigma q_i = \mu_j q_i$$ for CES demand where $i$ refers to a node and $j$ refers to the branch. The CET and the MNL form is identical under Leontief.\
For prices, we generally use that when we know the price of each branch under a node as well as the quantities of these branches, the zero profit condition can be rearranged to give us the price of the node:
$$ q_i p_i = \sum_j q_j p_j \quad \Leftrightarrow \quad  p_i = \frac{\sum_j q_j p_j}{q_i}$$

Output quantities are held in the `qS` object. Output quantities are held fixed in the partial equilibrum (the output prices are the variables that adjust), and we simply set them to `output_quantity` in this example.

```python
output_quantity_Yout = 100 * len(Y_inputs) / len(Y_outputs) #Makes qD[E] = 100 by construction
output_quantity_M = 100
qS = {"ID":pd.Series([], name="qS", dtype="float64"), "EOP":pd.Series([], name="qS", dtype="float64")}
```

`qS` is now an empty series to which we add the value of 100 for each output of the tree:

```python
qS["EOP"]
```

    Series([], Name: qS, dtype: float64)

```python
for nt in nts:
    for out in nts[nt].database.series[nt+"_out"]:
        if nt == "ID":
            quantity = output_quantity_Yout
        elif nt == "EOP":
            quantity = output_quantity_M
        qS[nt][out] = quantity
```

All other quantities are kept in the object `qD`, which we initialize:

```python
qD = {"ID":pd.Series([], name="qD", dtype="float64"), "EOP":pd.Series([], name="qD", dtype="float64")}
```

First, we calculate the composite Y and the quantities of the inputs for Y (E + dX rest)

```python
qD["ID"]["Y"] = sum(qS["ID"])
```

```python
for Yinp in Y_inputs:
    qD["ID"][Yinp] = mu["ID"].loc[(Yinp, "Y")] * qD["ID"]["Y"]
```

Second, we calculate components' starting values for input-displacing (we did it for EOP with qS, because components are outputs there). \
We use the demand equation stated earlier (multiplication of share-parameter and relevant node/component). This calculates quantities for all components, including baseline components

```python
for E in output["ID"]["upper_categories"]:
    for C in output["ID"]["upper_categories"][E]:
        qD["ID"][C] = mu["ID"].loc[(C, E)] * qD["ID"][E]
```

Next, we use that we have estimates of U to set these (current coverages split according to overlap), and then calculate the $\mu$s residually

```python
output["ID"]["current_coverages_split"]
```

    n          nn
    U_ID_t2_1  EL    0.008
    U_ID_t2_2  EL    0.002
    U_ID_t4_1  EL    0.002
    U_ID_t3_1  EH    0.020
    U_ID_t4_2  EL    0.008
    Name: current_coverages_split, dtype: float64

```python
for index, curr_coverage in output["ID"]["current_coverages_split"].iteritems():
    qD["ID"][index[0]] = curr_coverage * qD["ID"][index[1]]
```

The $\mu$s of technology goods (in their component nest) residually. This includes the share-parameters for the baseline technology goods $\bar U$. Since we still have not set the starting values for the quantities of these baseline technology goods, we set these using the share-parameters that are calculated here as well.

```python
output["ID"]["components"]
```

    {'C_EL_1': ['U_ID_t2_1', 'U0_ID_C_EL_1'], 'C0_EL': ['U0_ID_C0_EL']}

```python
mu["ID"]
```

    n                  nn
    ID_t2_electricity  ID_t2     1.000000
    ID_t2_K            ID_t2    24.000000
    C_EL_1             EL        0.100000
    U_ID_t2_1          ID_t2     1.000000
    C0_EL              EL        0.900000
    EL                 Y         0.200000
    Y_in_electricity   Y         0.200000
    Y_in_oil           Y         0.200000
    Y_in_inp3          Y         0.200000
    Y_in_K             Y         0.200000
    Y_out_electricity  Y         0.333333
    Y_out_oil          Y         0.333333
    Y_out_inp3         Y         0.333333
    Name: mu, dtype: float64

```python
for C in output["ID"]["components"]:
    #The ':-1' leaves out the baseline technology good, which we handle separately afterwards
    nonbase_U = output["ID"]["components"][C][:-1]
    base_mu = 1
    for U in nonbase_U:
        mu["ID"][(U, C)] =  qD["ID"][U]/qD["ID"][C]
        base_mu -= mu["ID"][(U, C)]
    #Quantities of baseline U
    mu["ID"][(output["ID"]["components"][C][-1], C)] = base_mu
    qD["ID"][output["ID"]["components"][C][-1]] = base_mu * qD["ID"][C]
    if base_mu <= 0:
        print(C)
        print(qD["ID"][C])
        print(qD["ID"][U])
        print(nonbase_U)
        print(base_mu)
        raise Exception("base_mu is not positive")

```

For end of pipe, we simply set the share-parameters of the technology goods under components as $1/N$ where $N$ is the number of branches/technology goods under a given component. Having set these, we can calculate the corresponding quantities using the CES Leontief demand equation.

```python
for C in output["EOP"]["components"]:
    for U in output["EOP"]["components"][C]:
        mu["EOP"].loc[(U, C)] = 1/len(output["EOP"]["components"][C])
        qD["EOP"][U] = mu["EOP"].loc[(U, C)] * qS["EOP"][C]
```

Then, we calculate the starting values of technologies $\tau$ as the sum of their relevant $U$, since the CET function should be scale-preserving.\
The drawback here, for EOP only, is that the quantities of $U$ do not necessarily adhere to the relative sizes of their share-parameters (which are set based on the technology catalog).

```python
for module in modules:
    for tech in output[module]["techs"]:
        qD[module][tech] = qD[module][output[module]["techs"][tech]].sum()
```

Lastly, we set the starting values of the inputs $X$ that go into technologies. These again use the demand equation. The share-parameters of these were given directly from technology data.

```python
for module in modules:
    techs = output[module]["techs_inputs"]
    for tech in techs:
        for inp in techs[tech]:
            qD[module][inp] = mu[module].loc[(inp, tech)] * qD[module][tech]
```

Set the quantity of the IO-technology as the sum of the quantities of the goods ($U_0$ and $C_0$) that it provides. Afterwards, calculate the relevant share parameters using the fraction of IO-tech to the goods.

```python
#Quantity of the baseline technologies
for basetech in output["ID"]["basetechs"]:
    qD["ID"][basetech] = 0
    for baseU in output["ID"]["basetechs"][basetech]:
        qD["ID"][basetech] += qD["ID"][baseU]
    #share parameters for each the non-replacing baseline technologies:
    for baseU in output["ID"]["basetechs"][basetech]:
        mu["ID"][(baseU, basetech)] = qD["ID"][baseU] / qD["ID"][basetech]
```

##### Prices:

The next part contains the setting of starting values for prices.
The prices of inputs (the very bottom of the tree) are fully exogenous, whereas all other prices are endogenous. The prices of outputs (those whose quantities are contained in `qS` are kept in the object `PbT` (refers to "prices before taxes") whereas the rest of the prices are kept in `PwT`.

Objects for storing prices of goods:

```python
PwThat = {"ID":pd.Series([], name="PwThat", dtype="float64"), "EOP":pd.Series([], name="PwThat", dtype="float64")}
PwT = {"ID":pd.Series([], name="PwT", dtype="float64"), "EOP":pd.Series([], name="PwT", dtype="float64")}
PbT = {"ID":pd.Series([], name="PbT", dtype="float64"), "EOP":pd.Series([], name="PbT", dtype="float64")}
```

Prices of inputs are also stated in the catalog:

```python
for module in modules:
    for t, inputs in output[module]["techs_inputs"].items():
        for inp in inputs:
            PwT[module][inp] = output["inputprices"][inp.split("_")[-1]]
            p = output["inputprices"][inp.split("_")[-1]]
            #Add emission prices to Phat
            taxes = 0
            for m in pM.index:
                taxes += phi[(m, inp)] * pM[m]
            PwThat[module][inp] = p + taxes
```

```python
#Prices on the dX in final goods production
for Y_inp in Y_inputs:
    if Y_inp not in output["ID"]["upper_categories"].keys():
        p = output["inputprices"][find_true_input(Y_inp, output["ID"]["Q2P"])]
        PwT["ID"][Y_inp] = p
        taxes = 0
        for m in pM.index:
            taxes += phi[(m, Y_inp)] * pM[m]
        PwThat["ID"][Y_inp] = p + taxes
```

Add the prices of technologies: $p^\tau$. These correspond to the weighted average of the prices of its inputs.

```python
for module in modules:
    for t, inputs in output[module]["techs_inputs"].items():
        PwThat[module][t] = pd.concat([PwThat[module][inputs], qD[module][inputs]], axis=1).product(axis=1).sum() / qD[module][t]
```

These prices are equal to unit costs by construction, which we check here for good measure. The differences between prices and unit costs are:

```python
round(output["ID"]["unit_costs"].set_index("tech")["unit_cost"] - PwThat["ID"][output["ID"]["unit_costs"]["tech"]], 2)
```

    tech
    ID_t2    0.0
    dtype: float64

For $p^U$, we simply assume $p^U = p^\tau$, which, since the share-parameters in the CET nest splitting $\tau$ into its produced technology goods sum to one, is consistent with zero profits (mroe generally, scale preservation ensures zero profits with this assumption)

```python
for module in modules:
    for t in output[module]["techs"]:
        for U in output[module]["techs"][t]:
            PwThat[module][U] = PwThat[module][t]
```

Prices on components for end of pipe should be allocated to PbT because that includes the prices of outputs:

```python
for C, U_list in output["EOP"]["components"].items():
    PbT["EOP"][C] = pd.concat([PwThat["EOP"][U_list], qD["EOP"][U_list]], axis=1).product(axis=1).sum() / qS["EOP"][C]
```

Insert baseline input quantities and baseline technology good prices (only relevant for ID since baselines do not exist in EOP).

This also (for now, should be set with input-output macro data later) sets the share-parameters of the inputs for the baseline technology goods. It assumes that the baseline uses all energy inputs in equal proportions and that these share-parameters sum to 1. Using these share-parameters, the corresponding quantities are added to `qD`.

```python
for basetech in output["ID"]["basetech_inputs"]:
    inputs = output["ID"]["basetech_inputs"][basetech]
    for inp in inputs:
        mu["ID"][(inp, basetech)] = 1/len(inputs)
        PwT["ID"][inp] = output["inputprices"][inp.split("_")[-1]]
        p = output["inputprices"][inp.split("_")[-1]]
        taxes = 0
        for m in pM.index:
            taxes += phi[(m, inp)] * pM[m]
        PwThat["ID"][inp] = p + taxes
        qD["ID"][inp] = mu["ID"][(inp, basetech)] * qD["ID"][basetech]
    #Price of the IO technology
    PwThat["ID"][basetech] = pd.concat([PwThat["ID"][inputs], qD["ID"][inputs]], axis=1).product(axis=1).sum() / qD["ID"][basetech]
```

```python
#Price of each output from the baseline technologies are simply set to the price that they themselves have
for basetech in output["ID"]["basetechs"]:
    for U0 in output["ID"]["basetechs"][basetech]:
        PwThat["ID"][U0] = PwThat["ID"][basetech]
```

```python
# Quantities and prices of inputs to replacement-baseline-technologies and the cost of these as well:
# for base_U, inputs in output["ID"]["baseline_U_inputs"].items():
#     for inp in inputs:
#         qD["ID"][inp] = mu["ID"][(inp, base_U)] * qD["ID"][base_U]
#         PwT["ID"][inp] = output["inputprices"][inp.split("_")[-1]] #Actual prices
#         p = output["inputprices"][inp.split("_")[-1]]
#         taxes = 0
#         for m in pM.index:
#             taxes += phi[(m, inp)] * pM[m]
#         PwThat["ID"][inp] = p + taxes
#     PwThat["ID"][base_U] =  pd.concat([PwThat["ID"][inputs], qD["ID"][inputs]], axis=1).product(axis=1).sum() / qD["ID"][base_U]
```

Prices of components:

```python
for C, U_list in output["ID"]["components"].items():
    PwThat["ID"][C] = pd.concat([PwThat["ID"][U_list], qD["ID"][U_list]], axis=1).product(axis=1).sum() / qD["ID"][C]
```

Prices of energy services ("upper categories" in ID).

```python
for E, C_list in output["ID"]["upper_categories"].items():
    PwThat["ID"][E] = pd.concat([PwThat["ID"][C_list], qD["ID"][C_list]], axis=1).product(axis=1).sum() / qD["ID"][E]
```

Price of composite of Y and outputs of Y

```python
PwThat["ID"]["Y"] = pd.concat([PwThat["ID"][Y_inputs], qD["ID"][Y_inputs]], axis=1).product(axis=1).sum() / qD["ID"]["Y"]
```

```python
#Price of final goods are equal to price of Y
for Y_out in Y_outputs:
    PbT["ID"][Y_out] = PwThat["ID"]["Y"]
```

We now add the calculated values to the (still) empty database `db` instantiated earlier.

#### Now we merge the first databases into the nesting tree objects.

```python
for module in modules:
    qS[module].index.name = "n"
    qD[module].index.name = "n"
    PwThat[module].index.name = "n"
    PwT[module].index.name = "n"
    PbT[module].index.name = "n"
    dbs[module]["qS"] = qS[module]
    dbs[module]["qD"] = qD[module]
    dbs[module]["PwThat"] = PwThat[module]
    dbs[module]["PbT"] = PbT[module]
    dbs[module]["mu"] = mu[module]
```

Merge `db` with the database attached to the nesting tree object `nt`:

```python
for module in modules:
    DataBase.GPM_database.merge_dbs(nts[module].database, dbs[module], "first")
```

As a confirmation that this went succesfully, we check that the databse in `nt` contains a symbol called 'mu':

```python
nts["ID"].database.series["mu"].vals.head()
```

    n                  nn
    ID_t2_electricity  ID_t2     1.0
    ID_t2_K            ID_t2    24.0
    C_EL_1             EL        0.1
    U_ID_t2_1          ID_t2     1.0
    C0_EL              EL        0.9
    Name: mu, dtype: float64

Success! We now proceed to use the class *abate*, a childclass of *gmspython*.

## Add sets used for calibration to the database

```python
#mappings for the current applications for ID
map_ID_TC = multiindex_series(idx_level_names=["n", "nn"], series_name="map_ID_TC")
for tech in output["ID"]["techs"]:
    for U in output["ID"]["techs"][tech]:
        C = find_key_from_value(output["ID"]["components"], U)
        map_ID_TC[(tech, C)] = np.nan
map_ID_TC = map_ID_TC.index

```

```python
map_ID_BUC = multiindex_series(idx_level_names=["n", "nn"], series_name="map_ID_BUC")
for btech in output["ID"]["basetechs"]:
    for U0 in output["ID"]["basetechs"][btech]:
        C = find_key_from_value(output["ID"]["components"], U0)
        map_ID_BUC[(U0, C)] = np.nan
map_ID_BUC = map_ID_BUC.index

```

```python

```

```python

```

```python
#ID_nonBU and mubar: all technology goods except those from baseline technologies.

mubar = multiindex_series(idx_level_names=["n", "nn"], series_name="mubar")
for C in output["ID"]["components"]:
    for U in output["ID"]["components"][C]:
        if U not in flatten_list(output["ID"]["basetechs"].values()):
            mubar[(U, C)] = 1
map_ID_nonBUC = mubar.index
```

Sets, variables and mappings related to finding $\gamma_\tau$

```python
currapp_ID_modified = pd.Series([], name="currapp_ID_modified", dtype="float64")
gamma_tau = multiindex_series(idx_level_names=["n", "nn"], series_name="gamma_tau")
map_currapp_ID2T = multiindex_series(idx_level_names=["n", "nn"], series_name="map_currapp_ID2T")
map_currapp_ID2E = multiindex_series(idx_level_names=["n", "nn"], series_name="map_currapp_ID2E")
for tech, E in output["ID"]["current_coverages"].index:
    gamma_tau[(tech, E)] = 0.5
    map_currapp_ID2T[(tech + "_" + E, tech)] = np.nan
    map_currapp_ID2E[(tech + "_" + E, E)] = np.nan
    currapp_ID_modified[tech + "_" + E] = output["ID"]["current_coverages"][(tech, E)] + 0.001
currapp_ID_modified.index.name = "n"
currapp_ID_subset = currapp_ID_modified.index
map_currapp_ID2T = map_currapp_ID2T.index
map_currapp_ID2E = map_currapp_ID2E.index
currapp_ID = currapp_ID_modified.copy()
currapp_ID.name = "currapp_ID"
map_T2E = gamma_tau.index
```

```python

```

```python
#Current applications for EOP too
currapp_EOP = pd.Series([], name="currapp_EOP", dtype="float64")
# map_currapp_EOP2T = multiindex_series(idx_level_names=["n", "nn"], series_name="map_currapp_EOP2T")
# map_currapp_EOP2M = multiindex_series(idx_level_names=["n", "nn"], series_name="map_currapp_EOP2M")
for tech, m in output["EOP"]["current_coverages"].index:
#     map_currapp_EOP2T[(tech + "_" + m, tech)] = np.nan
#     map_currapp_EOP2M[(tech + "_" + m, m)] = np.nan
    currapp_EOP[tech + "_" + m] = output["EOP"]["current_coverages"][(tech, m)]
currapp_EOP.index.name = "n"
currapp_EOP_subset = currapp_EOP.index
# map_currapp_EOP2T = map_currapp_EOP2T.index
# map_currapp_EOP2M = map_currapp_EOP2M.index

```

#### g_exo_vals consists of $\bar \sigma$ and $\bar \mu$. These are targets in the minimzation object

$\bar \sigma$ parameters used for the minimization object. These are the sigmas in the MNL nests (C->U)

```python
# sigmabar = {"ID":pd.Series([], name="sigmabar", dtype="float64"), "EOP":pd.Series([], name="sigmabar", dtype="float64")}
# for c in output["ID"]["components"].keys():
#     sigmabar["ID"][c] = 1
```

$\bar \mu$ parameters used for minimization object. These are the share parameters in the CET split from technologies to technology goods.

```python
# mubar = {"ID":mu["ID"].loc[pd.IndexSlice[mu["ID"].index.get_level_values(0).isin(flatten_list(list(output["ID"]["techs"].values()))), \
#                      mu["ID"].index.get_level_values(1).isin(list(output["ID"]["techs"].keys()))]]}
# mubar["ID"].name = "mubar"
```

#### g_tech_endo consists of parameters that are endogenized in the calibration procedure

```python
mu_EC = mu["ID"].loc[pd.IndexSlice[pd.Series(mu["ID"].index.get_level_values(0).str.startswith("C")), \
                     pd.Series(mu["ID"].index.get_level_values(1).str.startswith("E"))]].index

mu_EC = nts["ID"].trees["ID_EC"].database["map_ID_EC"].vals
# sigma_CU = sigmabar["ID"].index
# sigma_CU.name = "n"

# mu_tautoU = mu["ID"].loc[pd.IndexSlice[mu["ID"].index.get_level_values(0).str.startswith("U"), mu["ID"].index.get_level_values(1).isin(output["ID"]["techs"].keys())]].index
```

```python

```

```python

```

```python
#all mus from C->U except for baseline technology goods since these simply have their share parameter set to 1.
#USES mubar:
mu_CU = mu["ID"].loc[mubar.index].index

#self.g("map_ID_CU").rctree_pd(self.g("bra_no_ID_BU"))

# mu_CU = mu["ID"].loc[mu["ID"].index.get_level_values(0).isin(flatten_list(output["ID"]["components"].values())), \
#                      mu["ID"].index.get_level_values(1).isin(output["ID"]["components"].keys())]

# mu_CU = mu_CU.loc[~mu_CU.index.get_level_values(0).isin(flatten_list(output["ID"]["basetechs"].values()))]

# mu_CU = mu_CU.index
# mu_CU.name = "n"
```

```python
mu_CU
```

    MultiIndex([('U_ID_t2_1', 'C_EL_1')],
               names=['n', 'nn'])

```python
#Inputs to baseline technologies, to hit sumXinE:
mu_BX = mu["ID"].loc[nts["ID"].trees["ID_BX"].database.series["map_ID_BX"].vals].index # <--- LOL
#mu_BX = gm.get("map_ID_BX")

```

```python

```

```python
mu_Y_inp = mu["ID"].loc[nts["ID"].trees["ID_Y_in"].database.series["map_ID_Y_in"].vals].index
```

```python
#mu_Y_inp = gm.get("map_ID_Y_in")
```

```python
mu_Y_out = mu["ID"].loc[nts["ID"].trees["ID_Y_out"].database.series["map_ID_Y_out"].vals].index
# mu_Y_out = gm.get("map_ID_Y_out")

```

```python
# mu_BU = mu["ID"].loc[pd.IndexSlice[mu["ID"].index.get_level_values(0).str.startswith("IO"), mu["ID"].index.get_level_values(0).str.startswith("IO")]].index
```

```python
t = pd.IndexSlice[mu["ID"].index.get_level_values(1).isin(output["ID"]["basetechs"].keys()), \
                  mu["ID"].index.get_level_values(0).isin(flatten_list(output["ID"]["basetechs"].values()))]
mu_gamma_C = mu["ID"].loc[t].index
# mu_gamma_C = gm.get("map_ID_BU")
```

Share-parameters of Leontief technologies (to be kept exogenous throughout)

```python
# t = pd.IndexSlice[pd.Series(mu["ID"].index.get_level_values(1).isin(output["ID"]["techs"].keys())), pd.Series(mu["ID"].index.get_level_values(1).isin(output["ID"]["techs"].keys()))]

# t2 = pd.IndexSlice[pd.Series(mu["ID"].index.get_level_values(1).isin(output["ID"]["baseline_U_inputs"].keys())), \
#                    pd.Series(mu["ID"].index.get_level_values(1).isin(output["ID"]["baseline_U_inputs"].keys()))]

# mu_leontief_techs = mu["ID"].loc[t].index
#.append(mu["ID"].loc[t].index)
```

```python
tech_endoincalib_mu = {"ID":mu_EC.append(mu_gamma_C).append(mu_CU).append(mu_BX).append(mu_Y_inp).append(mu_Y_out), "EOP":None}
# tech_endoincalib_sigma = {"ID":sigma_CU, "EOP":None}
```

```python
tech_endoincalib_mu
```

    {'ID': MultiIndex([(                 'C_EL_1',          'EL'),
                 (                  'C0_EL',          'EL'),
                 (           'U0_ID_C_EL_1', 'basetech_EL'),
                 (            'U0_ID_C0_EL', 'basetech_EL'),
                 (              'U_ID_t2_1',      'C_EL_1'),
                 ('basetech_EL_electricity', 'basetech_EL'),
                 (        'basetech_EL_oil', 'basetech_EL'),
                 (       'basetech_EL_inp3', 'basetech_EL'),
                 (          'basetech_EL_K', 'basetech_EL'),
                 (                     'EL',           'Y'),
                 (       'Y_in_electricity',           'Y'),
                 (               'Y_in_oil',           'Y'),
                 (              'Y_in_inp3',           'Y'),
                 (                 'Y_in_K',           'Y'),
                 (      'Y_out_electricity',           'Y'),
                 (              'Y_out_oil',           'Y'),
                 (             'Y_out_inp3',           'Y')],
                names=['n', 'nn']),
     'EOP': None}

g_endovars_exoincalib includes the endogenous variables that we exogenize in calibration because we have data on them (C due to potentials, sum of U due to current applications, sum of dX for IO tech and replacing tech due to IO data)

```python
#Mapping from technology goods to
# U2E = multiindex_series(idx_level_names=["n", "nn"], series_name="map_E2E")
# for E in output["ID"]["upper_categories"]:
#     for C in output["ID"]["upper_categories"][E]:
#         for U in output["ID"]["components"][C]:
#             U2E[(U, E)] = np.nan
```

The sumU object is constructed because we need to calibrate to those. sumU hits current application.

```python
#sumU stuff
sumU = {"ID":multiindex_series(idx_level_names=["n", "nn"], series_name="sumU"), "EOP":multiindex_series(idx_level_names=["n", "nn"], series_name="sumU")}

for module in modules:
    for t in output[module]["techs"]:
        for U in output[module]["techs"][t]:
            C = find_key_from_value(output[module]["components"], U)
            upper = find_key_from_value(output[module]["upper_categories"], C)
            sumU[module][("sumU_" + t + "_" + upper, U)] = qD[module][U]
```

```python
sumU_map = {"ID":sumU["ID"].index, "EOP":sumU["EOP"].index}
```

```python
qsumU = sumU.copy()
for module in modules:
    qsumU[module].index = qsumU[module].index.droplevel(1)
    qsumU[module] = qsumU[module].groupby("n").sum()
```

```python
sumUaggs = {}
# sumXaggs = {}
for module in modules:
    sumUaggs[module] = sumU_map[module].get_level_values(0).unique()
#     sumXaggs[module] = sumX2X[module].get_level_values(0).unique()
```

```python

```

```python
#mapping from current applications to their sumU and EL
map_currapp2sumUE = multiindex_series(idx_level_names=["n", "nn", "nnn"], series_name="map_currapp2sumUE")
for currapp in currapp_ID_subset:
    map_currapp2sumUE[(currapp, "sumU_" + currapp, find_str_in_name(currapp, energy_services))] = np.nan
map_currapp2sumUE = map_currapp2sumUE.index
```

```python
#mapping from current applications to their sumU and M0
map_currapp2sumUM = multiindex_series(idx_level_names=["n", "nn", "nnn"], series_name="map_currapp2sumUM")
for currapp in currapp_EOP_subset:
    map_currapp2sumUM[(currapp, "sumU_" + currapp, find_str_in_name(currapp, list(M.index)))] = np.nan
map_currapp2sumUM = map_currapp2sumUM.index
```

```python
# endovars_exoincalib_sumU = sumU_map.copy()
# endovars_exoincalib_sumX = sumX
# endovars_exoincalib_C = {"ID":mu_EC.get_level_values(0), "EOP":None}
# endovars_exoincalib_E = {"ID":mu_EC.get_level_values(1).unique(), "EOP":None}
# endovars_exoincalib_E["ID"].name = "n"
```

```python
#calib_values_currentapplications = pd.concat([qD[qD.index.str.startswith("U") & ~qD.index.str.contains("EOP")], qD[qD.index.str.startswith("C")]])
```

```python
# calib_values_potentials = mu.loc[pd.IndexSlice[pd.Series(mu.index.get_level_values(0).str.startswith("C")), \
#                                  pd.Series(mu.index.get_level_values(1).str.startswith("E"))]]
calib_values_potentials = {}
for module in modules:
    calib_values_potentials[module] = output[module]["coverage_potentials"]
```

Initialize dict: tech_dbs with databases for ID and EOP

```python
tech_dbs = {"ID":DataBase.GPM_database(), "EOP":DataBase.GPM_database()}
alwaysexo_mu = {"ID":mu["ID"].drop(tech_endoincalib_mu["ID"]).index, "EOP":mu["EOP"].index}
```

```python
# alwaysexo = gm.get("map_ID_TX").append(gm.get("map_ID_TU")).append(gm.g("map_ID_CU").rctree_pd(gm.g("bra_ID_BU")))
```

```python
# ll = (
#     l.append(pd.Series([2])).append(pd.Series([2]))
#     .append(pd.Series([3]))
# )
```

```python
alwaysexo_mu["ID"]
```

    MultiIndex([('ID_t2_electricity',  'ID_t2'),
                (          'ID_t2_K',  'ID_t2'),
                (        'U_ID_t2_1',  'ID_t2'),
                (     'U0_ID_C_EL_1', 'C_EL_1'),
                (      'U0_ID_C0_EL',  'C0_EL')],
               names=['n', 'nn'])

```python

```

```python
for module in modules:
    tech_dbs[module]["PwT"] = PwT[module] #New basic prices
    #db["endovars_exoincalib_sumU"] = endovars_exoincalib_sumU
    #db["endovars_exoincalib_sumX"] = endovars_exoincalib_sumX
    tech_dbs[module][module + "_" +"params_alwaysexo_mu"] = alwaysexo_mu[module]
    tech_dbs[module]["calib_values_currentapplications"] = qsumU[module]
    tech_dbs[module]["calib_values_potentials"] = calib_values_potentials[module]
    #if module == "ID":
#         tech_dbs[module][module + "_" + "endovars_exoincalib_E"] = endovars_exoincalib_E[module]
#         tech_dbs[module][module + "_" + "endovars_exoincalib_C"] = endovars_exoincalib_C[module]
#         tech_dbs[module][module + "_" +"tech_endoincalib_mu"] = tech_endoincalib_mu[module]
#         tech_dbs[module][module + "_" +"tech_endoincalib_sigma"] = tech_endoincalib_sigma[module]
```

```python
#Minimization stuff
for module in modules:
    tech_dbs[module]["minobj"] = 1
    if module == "ID":
        tech_dbs[module]["mubar"] = mubar
        tech_dbs[module]["map_ID_TC"] = map_ID_TC
        tech_dbs[module]["map_ID_BUC"] = map_ID_BUC
        tech_dbs[module]["map_ID_nonBUC"] = map_ID_nonBUC
        tech_dbs[module]["gamma_tau"] = gamma_tau
        tech_dbs[module]["map_T2E"] = map_T2E
        tech_dbs[module]["currapp_ID_modified"] = currapp_ID_modified
        tech_dbs[module]["currapp_ID_subset"] = currapp_ID_subset
        tech_dbs[module]["map_currapp_ID2T"] = map_currapp_ID2T
        tech_dbs[module]["map_currapp_ID2E"] = map_currapp_ID2E
        tech_dbs[module]["map_currapp2sumUE"] = map_currapp2sumUE

#         tech_dbs[module]["minobj_sigma"] = sigmabar[module]
#         tech_dbs[module]["minobj_mu_subset"] = mubar[module].index
#         tech_dbs[module]["minobj_sigma_subset"] = sigmabar[module].index
#         tech_dbs[module]["weight_sigma"] = 1
        tech_dbs[module]["weight_mu"] = 10
    if module == "EOP":
        tech_dbs[module]["currapp_EOP"] = currapp_EOP
        tech_dbs[module]["currapp_EOP_subset"] = currapp_EOP_subset
        tech_dbs[module]["map_currapp2sumUM"] = map_currapp2sumUM
        tech_dbs[module]["weight_sigmaG"] = 10
        tech_dbs[module]["weight_muG"] = 1

```

```python
for module in modules:
    tech_dbs[module][module + "_" +"sumUaggs"] = sumU_map[module].get_level_values(0).unique()
    tech_dbs[module][module + "_" +"sumU2U"] = sumU_map[module]
    tech_dbs[module]["qsumU"] = qsumU[module]
#     dbs[module]["qsumU"].name = "qsumU"
```

```python
#aggregates for IO data
tech_dbs["ID"]["sumXinEaggs"] = sumXinEaggs
tech_dbs["ID"]["sumXrestaggs"] = sumXrestaggs
for module in modules:
    tech_dbs[module]["map_sumXrest2X_" + module] = sumXrest2X[module].index

```

```python
#Mappings related to new IO structure and calibration
tech_dbs["ID"]["map_sumXinE2X"] = sumXinE2X.index
tech_dbs["ID"]["map_sumXinE2E"] = sumXinE2E.index
tech_dbs["ID"]["map_sumXinE2baselineinputs"] = sumXinE2baselineinputs.index
# tech_dbs["ID"]["map_U2E"] = U2E.index
```

```python
# qsumX = {"ID":pd.Series([], name="qsumX", dtype="float64"), "EOP":pd.Series([], name="qsumX", dtype="float64")}

qsumX = pd.Series([], name="qsumX", dtype="float64")
for sumx in sumXinEaggs:
    total = 0
    for x in sumXinE2X.index[sumXinE2X.index.get_level_values(0).isin([sumx])].get_level_values(1):
        total += qD["ID"][x]
    qsumX[sumx] = total

for sumx in sumXrestaggs:
    total = 0
    for x in sumXrest2X["EOP"].index[sumXrest2X["EOP"].index.get_level_values(0).isin([sumx])].get_level_values(1):
        if x in qD["EOP"]:
            total += qD["EOP"][x]
        elif x in qD["ID"]:
            total += qD["ID"][x]
        else:
            raise Exception("WHAT")
    qsumX[sumx] = total

qsumX.index.name = "n"
```

```python
# sumX2X[module].loc[pd.IndexSlice[pd.Series(sumX2X[module].get_level_values(0).isin([sumx]))]]
# pd.IndexSlice[pd.Series(mu["ID"].index.get_level_values(1).isin(sumx)), pd.Series(mu["ID"].index.get_level_values(1).isin(output["ID"]["techs"].keys()))]
```

```python
# for module in modules:
tech_dbs["ID"]["qsumX"] = qsumX
```

```python

```

Emission accounts stuff:

```python
#Add the sumX-variable to the emission intensity variable (phi)

for m in output["EOP"]["upper_categories"].keys():
    for sumx in qsumX.index:
        true_input = find_X_in_name(sumx, list(output["inputprices"].index))
        if true_input in emission_intensities.keys():
            if m in emission_intensities[true_input]:
                phi[(m, sumx)] = emission_intensities[true_input][m]
                continue
        phi[(m, sumx)] = default_emsint
#Update index names of M0 and M
M0.index.name = "n"
M.index.name = "n"
```

```python
#Store in database
tech_dbs["ID"]["phi"] = phi
tech_dbs["ID"]["map_M2X"] = phi.index
tech_dbs["ID"]["M0"] = M0
tech_dbs["EOP"]["M"] = M
tech_dbs["ID"]["M_subset"] = M0.index
```

```python
#mapping that shows which components in EOP contribute to the abatement of which emission types
map_M2C = multiindex_series(idx_level_names=["n", "nn"], idx_name="map_M2C")
for m in output["EOP"]["upper_categories"]:
    for c in output["EOP"]["upper_categories"][m]:
        map_M2C[(m, c)] = np.nan
map_M2C = map_M2C.index
```

```python
tech_dbs["EOP"]["map_M2C"] = map_M2C
```

```python
#technical parameters in G-functions (muG and sigmaG)
muG = pd.Series([], name="muG", dtype="float64")
sigmaG = pd.Series([], name="sigmaG", dtype="float64")
for c in map_M2C.get_level_values(1):
    muG[c] = 0
    sigmaG[c] = 0.3 #Neutral value here I guess could be 1

muG.index.name = "n"
sigmaG.index.name = "n"
# EOP_C_subset = muG.index
```

```python
# EOP_C_subset.name = "n"
# tech_dbs["EOP"]["EOP_C_subset"] = EOP_C_subset
```

```python
#Potentials
theta = calib_values_potentials["EOP"].copy()
theta.index = theta.index.droplevel(1)
theta.name = "theta"
```

```python
#Calibration procedure unfixes all muG and sigmaG.
muGbar = muG.copy()
sigmaGbar = sigmaG.copy()
```

```python
#add to database
module = "EOP"
tech_dbs[module]["pM"] = pM
tech_dbs[module]["muG"] = muG
tech_dbs[module]["sigmaG"] = sigmaG
tech_dbs[module]["minobj_muG"] = muGbar
tech_dbs[module]["minobj_sigmaG"] = sigmaGbar
tech_dbs[module]["theta"] = theta

module = "ID"
tech_dbs[module]["pMhat"] = pMhat

```

Finally, we adjust the prices according to the emission content of them

```python
dbs["ID"].get("mu").tail(35)
```

    n                        nn
    ID_t2_electricity        ID_t2           1.000000
    ID_t2_K                  ID_t2          24.000000
    C_EL_1                   EL              0.100000
    U_ID_t2_1                ID_t2           1.000000
    C0_EL                    EL              0.900000
    EL                       Y               0.200000
    Y_in_electricity         Y               0.200000
    Y_in_oil                 Y               0.200000
    Y_in_inp3                Y               0.200000
    Y_in_K                   Y               0.200000
    Y_out_electricity        Y               0.333333
    Y_out_oil                Y               0.333333
    Y_out_inp3               Y               0.333333
    U_ID_t2_1                C_EL_1          0.100000
    U0_ID_C_EL_1             C_EL_1          0.900000
    U0_ID_C0_EL              C0_EL           1.000000
    U0_ID_C_EL_1             basetech_EL     0.090909
    U0_ID_C0_EL              basetech_EL     0.909091
    basetech_EL_electricity  basetech_EL     0.250000
    basetech_EL_oil          basetech_EL     0.250000
    basetech_EL_inp3         basetech_EL     0.250000
    basetech_EL_K            basetech_EL     0.250000
    Name: mu, dtype: float64

# **3: The model**

We now build a gams model using a *gmspython* childclass (*abate*). We build the partial equilibrium model from the following principles:
* The firm takes input prices as given. These are prices with taxes ('PwThat') that are defined over the global set *inp*. In this case this includes e.g. the price on electricity and capital.
* The firm further takes the quantity of supply as given (it has to be either this or output prices for the model to be square). We denote the quantity of supply 'qS', and define it over the subset of goods that are outputs from the sector *out*.
* The firm's demand for intermediate goods, and their prices are endogenous to the firm: In our case this involves $\tau$, $U$ and $C$ in ID and only $\tau$ and $U$ in EOP. These intermediate goods are not traded on any market, but is simply a construct we use to build the nests. In this way the quantity of $U$ is both supplied, and demanded by the firm/sector itself. As we do not need both a supply and demand variable (they are the same in this case), we let the quantity/prices for intermediate goods go under the variables 'qD'/'PwThat', defined over the subset *int*.
* The demand for inputs is also endogenous to the firm. We denote this 'qD' defined over the subset of goods 'inp'.
* As briefly mentioned above, we either have to let the output prices / quantities be endogeonous (and the other exogenous) for the module to be square. In this case we let the price on outputs be endogenous. We denote the price 'PbT' for price before taxes. This is defined over the subset *out*. Distinguishing between PbT and PwThat allows for the flexibility of including a tax module / including a mark-up on profits at a later point.

We will get back to the way we initialize the *gmspython* module next; here we initialize the class to be able to print the variables needed for this specific model.
The model, denoted `m`, is an instance of the class `abate` which is itself a childclass of the more general `gmspython` class.

```python
#m = abatement.abate(nt=nts["ID"], tech_db=tech_dbs["ID"], work_folder=directory["work"], use_EOP={"tech_db":tech_dbs["EOP"], "nt":nts["EOP"]}, **{'data_folder': gams_folder, "name":"Abatement"})
gm = abatement.abate(nt=nts['ID'],tech=output,work_folder=directory['work'],**{'data_folder':directory['gams'],'name': 'atest'})
```

```python
gm.add_subsets()
```

```python
gm.get("ID_params_endoincalib_mu")
```

    MultiIndex([(                 'C_EL_1',          'EL'),
                (                  'C0_EL',          'EL'),
                (           'U0_ID_C_EL_1',      'C_EL_1'),
                (            'U0_ID_C0_EL',       'C0_EL'),
                ('basetech_EL_electricity', 'basetech_EL'),
                (        'basetech_EL_oil', 'basetech_EL'),
                (       'basetech_EL_inp3', 'basetech_EL'),
                (          'basetech_EL_K', 'basetech_EL'),
                (                     'EL',           'Y'),
                (       'Y_in_electricity',           'Y'),
                (               'Y_in_oil',           'Y'),
                (              'Y_in_inp3',           'Y'),
                (                 'Y_in_K',           'Y'),
                (      'Y_out_electricity',           'Y'),
                (              'Y_out_oil',           'Y'),
                (             'Y_out_inp3',           'Y'),
                (           'U0_ID_C_EL_1', 'basetech_EL'),
                (            'U0_ID_C0_EL', 'basetech_EL')],
               names=['n', 'nn'])

```python
m = "ID"
```

```python
gm.model.database[gm.n(m+'_nonB_u2c')] = gm.get(m+'_map_all')[(gm.get(m+'_map_all').get_level_values(gm.n('n')).isin(gm.get(m+'_u'))) & (gm.get(m+'_map_all').get_level_values(gm.n('nn')).isin(gm.get(m+'_c')))]
```

```python
gm.get(m+"_nonB_u2c")
```

    MultiIndex([('U_ID_t2_1', 'C_EL_1')],
               names=['n', 'nn'])

```python
m.model.database.symbols
```

```python
m.get("ID_nonB_u2c")
```

```python
m.model.database.update_all_sets(clean_up=False) #Makes n include the sum variables
```

```python
m.model.functions = {"std_pdf":"$FUNCTION std_pdf({x}): ((1/(sqrt(2*Pi)))*exp(-(Sqr({x}))/2)) $ENDFUNCTION"}
```

Recall that running a model requires constructing three methods/properties for `m`:

1. `m.initialize_variables()`
2. `m.endo_groups` and `m.exo_groups`
3. `m.add_blocks()`

We briefly review each in turn.

`m.initialize_variables()` sets default initial values for the variables and parameters that are not already present in the attached database (which we constructed earlier). For example, we did not specify the values for the mark-up parameters anywhere, so the method `initialize_variables()` *must* specify the value that these parameters should have. We can see that in this case they will simply be set to 1 (no markup):

The default values for markups are zero, i.e. no markups:

We run the method using the keyword argument `check_variables:True`. This makes the program check whether the database contains all the necessary values of a specific variable/parameter (say, $\mu$), and if not, merges the default value (in this case equal to 1) onto the series of share-parameters already contained in the database.

```python
m.g("map_currapp2sumUM").write()
```

    'map_currapp2sumUM[n,nn,nnn]'

Since all of our starting values were set 'as if' the entire tree was Leontief, it is worth noting that the default values for substitutions of elasticity/transformation are 1, e.g. for $\sigma$:

```python
m.default_var_series("sigma").head()
```

    n
    C_EL_1         0.0001
    C0_EL          0.0001
    basetech_EL    0.0001
    Y              0.0001
    ID_t2          0.0001
    Name: sigma, dtype: float64

The second requirement(s) are the properties `endo_groups` and `exo_groups`. These collect the variables in groups and specify which of these subgroups are part of the two upper groups of endogenous and exogenous variables/parameters respectively. The choice of whether a variable should be endogenous or exogenous changes e.g. depending on whether the model is about to be solved for calibration purposes or not.
In the basic case (not calibration mode), the endogenous variables are split into multiple groups:

```python
m.endo_groups.keys()
```

    dict_keys(['Abatement_g_ID_prices_alwaysendo', 'Abatement_g_emissions_alwaysendo', 'Abatement_g_ID_quants_alwaysendo', 'Abatement_g_EOP_prices_alwaysendo', 'Abatement_g_prices_endogenouswithEOP', 'Abatement_g_EOP_quants_alwaysendo', 'Abatement_g_emissions_endoinEOP', 'Abatement_g_ID_quants_exoincalib', 'Abatement_g_EOP_quants_exoincalib', 'Abatement_g_ID_minobj_exoincalib_endoinbaseline'])

... which themselves include the actual variables. Here we print the values of prices with taxes in the endovars group (the first 5 only). This uses the method *.var_endo*.

```python
m.initialize_variables()
```

```python
m.add_groups()
```

```python
m.add_blocks()
```

```python
m.var_endo("currapp_ID")
```

    n
    ID_t2_EL    0.05
    Name: currapp_ID, dtype: float64

And next we print the exogenous prices with taxes by doing the same thing, but for exogenous groups instead. This is found in the var_exo method:

Finally, we need to specify the *add_blocks* method. This is the method that writes blocks of equations to the model. In our case, we write a different block of equations for each tree.
For example, a set of equations are:

```python
print(m.eqtext(list(m.ns_local)[0]))
```

    E_zp_out_ID_EC[n]$(ID_out_ID_EC[n])..	PbT[n]*qS[n] =E= sum(nn$(map_ID_EC[nn,n]), qD[nn]*PwThat[nn]);
    	E_zp_nout_ID_EC[n]$(kno_no_ID_EC[n])..	PwThat[n]*qD[n] =E= sum(nn$(map_ID_EC[nn,n]), qD[nn]*PwThat[nn]);
    	E_q_out_ID_EC[n]$(bra_o_ID_EC[n])..	qD[n] =E= sum(nn$(map_ID_EC[n,nn]), mu[n,nn] * (PbT[nn]/PwThat[n])**(sigma[nn]) * qS[nn] / sum(nnn$(map_ID_EC[nnn,nn]), mu[nnn,nn] * (PbT[nn]/PwThat[nnn])**(sigma[nn])));
    	E_q_nout_ID_EC[n]$(bra_no_ID_EC[n])..	qD[n] =E= sum(nn$(map_ID_EC[n,nn]), mu[n,nn] * (PwThat[nn]/PwThat[n])**(sigma[nn]) * qD[nn] / sum(nnn$(map_ID_EC[nnn,nn]), mu[nnn,nn] * (PwThat[nn]/PwThat[nnn])**(sigma[nn])));

#### Model summary:

To sum up, the model we have in mind here has the following main settings (where a \$ denotes a condition):
* Endogenous variables: $PwThat\$(int)$, $PbT\$(out)$, $qD\$(wT)$. (Recall that the subset wT is the union of intermediate goods and inputs)
* Exogenous variables: $PwThat\$(inp)$, $qS\$(out)$.
* Equations: For the CES nest we have the following two equations:
    $$\begin{align}
        q_j =& \mu_j\left(\dfrac{p_j}{p_i}\right)^{-\sigma}q_i, \tag{CES-1}\\
        p_iq_i =& \sum_j q_jp_j \tag{CES-2}
    \end{align}$$
    (CES-1) is the CES demand function that has to hold for all *branches* ($q_j$) where $q_i$ is the relevant knot in the nesting tree. (CES-2) is a zero-profit condition that has to hold for production functions with constant returns to scale (alternatively, we can use a price index). This has to hold for every *knot* where $j$ sums over the relevant branches. A similar thing has to hold for the CET tree.

### **Running the model**
Running the model, including the invoking the methods and properties described in the previous section, we simply run the following:

```python
m.write_and_run()
```

To check whether the model was successfully solved, we check the modelstat (16.0 means solved correctly, 5.0 means not)

```python
m.model_instances["baseline"].__dict__
```

    {'execute_name': 'CollectAndRun.gms',
     'name': 'gmodel',
     'settings': <DB2Gams_l2.gams_settings at 0x22b938a3788>,
     'export_settings': {'dropattrs': ['settings', 'opt', 'job'],
      'pklattrs': {'settings': 'gams_settings'},
      'opt': 'conopt4.opt'},
     'opt': <gams.options.GamsOptions at 0x22b93b18248>,
     'opt_file': 'conopt4.opt',
     'import_settings': {},
     'job': <gams.execution.GamsJob at 0x22b93b34bc8>,
     'out_db': <DataBase.GPM_database at 0x22b93b7cb08>,
     'modelstat': 16.0,
     'solvestat': 1.0}

Run in the ID state:

```python
m.setstate("ID")
```

```python
m.reset_settings()
```

```python
m.state
```

    'ID'

```python
m.initialize_variables(**{"check_variables":True})
```

```python
m.add_blocks()
```

```python
m.add_groups()
```

```python
m.reset_settings()
```

```python
m.write_and_run()
```

```python
m.model_instances["baseline"].modelstat
```

    16.0

```python
m.setstate("EOP")
```

```python
m.reset_settings()
```

```python
m.write_and_run()
```

```python
m.model_instances["baseline"].modelstat
```

    16.0

The model does not run when all sigmas are not set to be Leontief, i.e. when they use their default value of 1.

We wish to solve the model while instead of the default value of substitution and transformation elasticities of 0.1, we set them equal to zero (in practice not exactly, but very close).
Since the starting values have been set under the assumption of Leontief nests, this will help the solver find an initial solution.
Since EOP is not consistent with Leontief, these elasticities specifically must not be zero. To make sure we will always find a solution, we choose a high number, e.g. 2:

```python
# m.model.settings.databases["Abatement_0"]["sigma"].vals.loc[:] = 0.0001
# m.model.settings.databases["Abatement_0"]["eta"].vals.loc[:] = -0.0001
#m.get("sigma")[:] = 0.0001
# m.get("sigma")[m.get("EOP_C_subset")] = 2
#m.get("eta")[:] = -0.0001
# m.get("eta")[m.get("eta").index.str.startswith("EOP")] = -2

#m.model.settings.databases["Abatement"]["sigma"].vals[condition] = 0.2
#m.write_and_run(options_run={'output':sys.stdout})
m.write_and_run()
if m.model_instances["baseline"].modelstat == 16.0:
    print("\nSuccess! The modelstat was 16.0")
```

    Success! The modelstat was 16.0

Save model by pickling it, using the export method:

```python
m.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\Abatement\\Data\\..\\gamsmodels\\Main\\gmspython_Abatement'

The model is calibrated in the next jupyter file.


### ■예제61. 자료형 확인하기 (type)
#### 파이썬의 자료형은 하나의 클래스. 파이썬은 숫자나 문자, 문자열, 리스트, 튜플, 사전, 함수 등을 각각 하나의 클래스로 취급
#### 코드를 작성하다가 변수 이름만 보고 이 자료가 어떤 자료형인지 확인해야 하는 경우 type()을 활용하면 쉽게 확인 가능

```python
# 예제
numdata=57
print(type(numdata))
strdata='파이썬'
print(type(strdata))
listdata=[1,2,3]
print(type(listdata))
```

    <class 'int'>
    <class 'str'>
    <class 'list'>

### ※ 문제135. 딕셔너리 자료형을 만들고 위와 같이 type을 확인하시오~

```python
dictdata={1:'a',2:'b',3:'c'}
print(type(dictdata))
```

    <class 'dict'>

### ■ 예제62. 나눗셈에서 나머지만 구하기(%)
#### 나누기 연산에서 나머지만 구하는 연산을 모듈(module)로 연산이라 하며 기호는 mod로 표시

```python
# 예시
print(12%3)
print(12%5)
```

    0
    2

### ※ 문제136. 아래와 같이 두개의 숫자를 각각 물어보게하고 아래의 메세지가 출력되게 하시오
    첫번째 숫자를 입력하세요 ~ 1113
    두번째 숫자를 입력하세요 ~ 23
    1113을 23으로 나누면 9가 나머지로 남습니다.

```python
n1=int(input(''))
n2=int(input(''))
print(f'{n1}을 {n2}로 나누면 {n1%n2}가 나머지로 남습니다.')
```

    1113
    23
    1113을 23로 나누면 9가 나머지로 남습니다.

### ■ 예제63. 몫과 나머지 구하기 (divmod)

```python
# 예제
n1=1113
n2=23
q ,r =divmod(n1,n2)
print(q, r)
```

    48 9

#### ※ divmod 함수가 리턴하는 값이 2개여서 변수를 2개 사용함

### ■ 예제64. Pandas를 이용한 데이터 검색
      SQL  -----------> Pandas
    Pandas ----------->   SQL

#### 예제
```python
import pandas as pd
emp[['ename','sal']][emp['ename']=='SCOTT']
         열                 행
```

```python
import pandas as pd
emp=pd.read_csv('K:\\Itwill\\2. Python\\자료\\emp3.csv')
print(emp[['ename','sal']][emp['sal']>=3000])

```

        ename   sal
    0    KING  5000
    9    FORD  3000
    11  SCOTT  3000

### ※ 문제137. 부서번호가 10번의 부서위치를 출력하시오

```python
dept=pd.read_csv('K:/Itwill/2. Python/자료/dept3.csv')
print(dept[['loc']][dept['deptno']==10])
```

            loc
    0  NEW YORK

### ■ 예제65. Pandas를 이용한 join
```sql
select e.ename, d.loc
    from emp e, dept d
    where e.deptno=d.deptno;
    ```
```python
import pandas as pd

emp=pd.read_csv('K:/Itwill/2. Python/자료/emp3.csv')
dept=pd.read_csv('K:/Itwill/2. Python/자료/dept3.csv')
result=pd.merge(emp, dept, on="deptno") # on절에 join조건 작성
print(result[['ename','loc']])
```

```python
import pandas as pd

emp=pd.read_csv('K:/Itwill/2. Python/자료/emp3.csv')
dept=pd.read_csv('K:/Itwill/2. Python/자료/dept3.csv')
result=pd.merge(emp, dept, on="deptno")
print(result[['ename','loc']])
```

         ename       loc
    0     KING  NEW YORK
    1    CLARK  NEW YORK
    2   MILLER  NEW YORK
    3    BLAKE   CHICAGO
    4   MARTIN   CHICAGO
    5    ALLEN   CHICAGO
    6   TURNER   CHICAGO
    7    JAMES   CHICAGO
    8     WARD   CHICAGO
    9    JONES    DALLAS
    10    FORD    DALLAS
    11   SMITH    DALLAS
    12   SCOTT    DALLAS
    13   ADAMS    DALLAS

### ※ 문제138. 아래의 SQL을 Pandas로 구현하시오
```sql
select e.ename, d.loc
    from emp e, dept d
    where e.deptno=d.deptno
        and d.loc='DALLAS'
        ```

```python
result=pd.merge(emp, dept, on='deptno')
print(result[['ename','loc']][result['loc']=='dallas'.upper()])
```

        ename     loc
    9   JONES  DALLAS
    10   FORD  DALLAS
    11  SMITH  DALLAS
    12  SCOTT  DALLAS
    13  ADAMS  DALLAS

### ※ 문제139. 아래의 SQL을 Pandas로 구현하시오
```sql
select e.ename, d.loc
    from emp e, dept d
    where e.deptno = d.deptno(+)
    ```

```python
result=pd.merge(emp, dept, on='deptno', how='left')
print(result[['ename','loc']])
```

         ename       loc
    0     KING  NEW YORK
    1    BLAKE   CHICAGO
    2    CLARK  NEW YORK
    3    JONES    DALLAS
    4   MARTIN   CHICAGO
    5    ALLEN   CHICAGO
    6   TURNER   CHICAGO
    7    JAMES   CHICAGO
    8     WARD   CHICAGO
    9     FORD    DALLAS
    10   SMITH    DALLAS
    11   SCOTT    DALLAS
    12   ADAMS    DALLAS
    13  MILLER  NEW YORK

### ※ 문제140. 아래의 SQL을 Pandas로 구현하시오
```sql
select e.ename, d.loc
    from emp e full outer join dept d
    on (e.deptno = d.deptno);
    ```

```python
result=pd.merge(emp, dept, on='deptno',how='outer')
print(result[['ename','loc']])
```

         ename       loc
    0     KING  NEW YORK
    1    CLARK  NEW YORK
    2   MILLER  NEW YORK
    3    BLAKE   CHICAGO
    4   MARTIN   CHICAGO
    5    ALLEN   CHICAGO
    6   TURNER   CHICAGO
    7    JAMES   CHICAGO
    8     WARD   CHICAGO
    9    JONES    DALLAS
    10    FORD    DALLAS
    11   SMITH    DALLAS
    12   SCOTT    DALLAS
    13   ADAMS    DALLAS
    14     NaN    BOSTON

### ■ 예제66. Pandas를 이용한 서브쿼리
```sql
select ename, sal
    from emp
    where sal = (select max(sal)
                     from emp);
                     ```
 ```python
import pandas as pd

emp=pd.read_csv('K:/Itwill/2. Python/자료/emp3.csv')
maxsal=emp['sal'].max()
print(maxsal)
```

```python
import pandas as pd

emp=pd.read_csv('K:/Itwill/2. Python/자료/emp3.csv')
maxsal=emp['sal'].max()
result=emp[['ename','sal']][emp['sal']==maxsal]
print(result)
```

      ename   sal
    0  KING  5000

### ※ 문제141. 아래의 SQL을 Pandas로 구현하시오
```sql
select ename, sal
    from emp
    where sal > (select sal
                     from emp
                     where ename='JONES')
                     ```

```python
js=emp['sal'][emp['ename']=='JONES'].values[0]
result=emp[['ename','sal']][emp['sal']>js]
result
```

      ename
      sal

      0
      KING
      5000

      9
      FORD
      3000

      11
      SCOTT
      3000

### ■ 예제67. Pandas와 Oracle 그룹함수 비교
```sql
select max(sal)
    from emp
    ```
```python
import pandas as pd
emp=pd.read_csv('k:/Itwill/2. Python/자료/emp3.csv')
emp['sal'].max()
          .min()
          .sum()
          .var()
          .std()
        ```

### ※ 문제142. 아래의 SQL을 Pandas로 구현하시오
```sql
select max(sal)
    from emp
    where deptno=20;
    ```

```python
emp[['sal']][emp['deptno']==20].max()
```

    sal    3000
    dtype: int64

### ※ 문제143. 아래의 SQL을 Pandas로 구현하시오
```sql
select min(sal)
    from emp
    where job='SALESMAN';
    ```

```python
emp['sal'][emp['job']=='SALESMAN'].min()
```

    1250

```
     오라클       vs      판다스
    table              data frame
    column             Series
    number             int
 char,varchar2         str
    ```

### ※ 문제144. 아래의 SQL을 Pandas로 구현하시오
```sql
select job, max(sal)
    from emp
    group by job;
    ```

```python
result=emp.groupby('job')['sal'].max().reset_index() # Series -> DF로 변경
result
```

      job
      sal

      0
      ANALYST
      3000

      1
      CLERK
      1300

      2
      MANAGER
      2975

      3
      PRESIDENT
      5000

      4
      SALESMAN
      1600

#### ※ reset_index는 Series 로 출력하는게 아니라 DataFrame(table)로 출력하는 키워드

### ※ 문제145. 아래의 SQL을 Pandas로 구현하시오
```sql
select deptno, sum(sal)
    from emp
    group by deptno;
    ```

```python
import pandas as pd

emp=pd.read_csv('k:/Itwill/2. Python/자료/emp3.csv')
emp.groupby('deptno')['sal'].sum().reset_index()
```

      deptno
      sal

      0
      10
      8750

      1
      20
      10875

      2
      30
      9400

### ※ 문제146. 아래의 SQL을 Pandas로 구현하시오
```sql
select deptno, sum(sal)
    from emp
    where deptno!=20
    group by deptno;
    ```

```python
result=emp.groupby('deptno')['sal'].sum().reset_index()
result[['deptno','sal']][result['deptno']!=20]
```

      deptno
      sal

      0
      10
      8750

      2
      30
      9400

#### ※ result가 DataFrame이므로 열과 행을 기술할 수 있다.

### ※ 문제147. 아래의 SQL을 Pandas로 구현하시오
```sql
select deptno, sum(sal)
    from emp
    group by deptno
    order by sum(sal) desc;
    ```

```python
result=emp.groupby('deptno')['sal'].sum()
result2=result.sort_values(ascending=False).reset_index()
result2
```

      deptno
      sal

      0
      20
      10875

      1
      30
      9400

      2
      10
      8750

#### ※ sort_values는 Series에서 사용할 수 있다.
```python
    Series.sort_values()
    ```

### ※ 문제148. 아래의 SQL을 판다스로 구현하시오
```sql
select ename, sal, job
    from emp
    where job = 'SALESMAN'
    order by sal desc;
    ```

```python
result=emp[['ename','sal','job']][emp['job']=='SALESMAN']
result2=result.sort_values('sal',ascending=False).reset_index() # sort_values('정렬할 column', 정렬방식) ∵ DataFrame
result2
```

      index
      ename
      sal
      job

      0
      5
      ALLEN
      1600
      SALESMAN

      1
      6
      TURNER
      1500
      SALESMAN

      2
      4
      MARTIN
      1250
      SALESMAN

      3
      8
      WARD
      1250
      SALESMAN

#### ※ 설명
```python
Series.sort_values(asceding= )
DataFrame.sort_values('column',ascending= )
```

### ※ 문제149.(점심시간 문제) 아래의 SQL을 Pandas로 구현하시오
```sql
select job, sum(sal)
    from emp
    where job='SALESMAN'
    group by job
    order by sum(sal) desc;
    ```

```python
result=emp.groupby('job')['sal'].sum()
result2=result.sort_values(ascending=False).reset_index()
result2[['job','sal']][result2['job']=='SALESMAN']
```

      job
      sal

      2
      SALESMAN
      5600

```python

```

```python

```

### ※(점심시간 문제)(파이썬 알고리즘 문제 28번) 확률밀도함수 수학 공식을 이용하여 확률을 구하시오
#### 확률변수 $X$가 정규분포 $N(5,2^2)$ 를 따를 때, 확률 $P(3 \leq X \leq 7)$ 의 값을  무엇인가?

```python
from sympy import exp, sqrt, pi, Integral, Symbol, S
from scipy.stats import norm
import numpy as np
import matplotlib.pyplot as plt

x= np.arange(140,160,0.001)
plt.plot(x, norm.pdf(x,150,5))
plt.show()
```

```python
x = Symbol('x')
f = exp(-(x-5)**2/(2*2**2))/(2*sqrt(2*pi))
result = Integral(f,(x,3,7)).doit().evalf()
result1= Integral(f,(x,S.NegativeInfinity,S.Infinity)).doit().evalf() # 전체확률 확인하는 방법
print(result)
print(result1)
```

    0.682689492137086
    1.00000000000000

```python

```


# ECCO Ocean Mixed Layer Energy Budget

The following sample code verifies that the ocean mixed layer energy budget is balanced in an example region,
i.e., the ocean heat storage tendency is equal to the sum of surface heat fluxes
plus ocean heat convergence (advection, diffusion), less surface radiation out the bottom.

```python
from dask_jobqueue import PBSCluster
cluster = PBSCluster()

from dask.distributed import Client
client = Client(cluster)
```

```python
print(client)
```

    <Client: scheduler='tcp://10.148.13.116:39011' processes=0 cores=0>

```python
cluster.scale(9)
```

```python
cluster
```

    VBox(children=(HTML(value='
PBSCluster
'), HBox(children=(HTML(value='\n
\n