```python
clean_up=True # removes gams-related files in work-folder if true
%run StdPackages.ipynb
os.chdir(py['main'])
import global_settings,ReadData,ShockFunction,Production,Household,GE,Invest,Trade,Government,diagnostics
from DataBase_wheels import small_updates
os.chdir(curr)
data_folder = os.getcwd()+'\\Data\\IO'
gams_folder = os.getcwd()+'\\gamsmodels\\GE'
```

    The file_gams_py_gdb0.gdx is still active and was not deleted.
    The file_gams_py_gdb3.gdx is still active and was not deleted.

# Set up a dynamic, general equilibrium model

*The current general equilibrium model is a small open economy that features exogenous long run interest-, inflation-, and growth rates. These settings are defined in the global settings:*

```python
name = 'GE'
gs_v = 'gs_v1'
tindex = range(1,4)
gs_vals = {'t':tindex}
gs = global_settings.gs_v1(kwargs_vals=gs_vals)
```

## **1: Setup**

The fundamental setup:
* The model consists of three fundamental sets that most equations/variables are defined over: (1) $n$ the set of goods in the economy, (2) $s$ the set of sectors, $t$ the time index.
* We distinguish between prices of various sorts $(PbT,Peq,PwT)$, quantities $(qS,qD)$, and values $(vD,vS)$. Beyond this, a number of other variables are additionally used in various modules.
* Market clearing ensures that the sum of supply equals the sum of supply, for a subset of goods $n$. As goods can be demanded/supplied by more than one sector, the market clearing condition is:
$$\begin{align}
    \sum_{s\in d\_qS[s,n]} qS[t,s,n] = \sum_{s\in d\_qD[s,n]} qD[t,s,n],
\end{align}$$
where $d\_qS$ and $d\_qD$ are dummies identifying which sectors are active for the relevant good.

## **2: Data**

The relevant data needed to run each module may vary. However, in general, the general equilibrium module should be adjusted to be consistent with input-output data. In a single year, the IO baseline data should at least cover:
* The equilibrium price for all traded goods.
* The inputs/outputs in values for each domestic sector, for each type of goods. The system must be balanced, such that the sum of demand equals the sum of supply for each $s$.

The following reads in the IO data and defines a number of default subsets:

```python
dsheets = {'Production_v': data_folder+'\\IO_v.xlsx', 'Production_p': data_folder+'\\IO_p.xlsx'}
GE_data = ReadData.read_data.main(dsheets,name='GE_data',components=['domstic','trade','HH','tax','invest'])
```

To the IO data, we add inventory data on durables:

```python
DataBase.GPM_database.merge_dbs(GE_data,excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+'dur.xlsx',{k: 'vars' for k in ('prod','HH','G')}),'second')
```

Furthermore, we read in data on tax rates:

```python
DataBase.GPM_database.merge_dbs(GE_data,excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+'Tax.xlsx',{'tbaseyear': 'vars'}),'second')
```

Note that - while not necessary - the tax rates should be of a size such that the total tax revenue corresponds to the income from the IO data; the government module can adjust some tax rates to ensure this, however, this will be in a somwhat random fashion. To check for this assertion, the following IO function computes the tax income on sectorial level, when tax rates consists of three componenets: Input taxes $tauD$, output taxes $tauSflat$ and lump sum taxes $tauLump$:

```python
pd.DataFrame({'Model': Government.taxRevenue(GE_data), 'IO': GE_data['vD'].rctree_pd(GE_data['n_tax']).droplevel(-1)})
```

      Model
      IO

      s

      F
      0.0909422
      -0.018329

      HH
      1.80099
      1.61747

      I_B
      0.250148
      0.263686

      I_M
      0.207596
      0.202365

      a
      0.442536
      0.058697

      b
      1.60557
      0.827386

      itory
      0.00155662
      0.00228146

Some of these discrepancies can be adjusted using the lump-sum tax rate; however, not all sectors are taxed lump-sum. The following adjusts the lump sum taxes, but leaves the two sectors 'inventory' and 'foreign' sectors unbalanced:

```python
GE_data['tauLump'] = Government.balanceIO_lumpsum(GE_data)
pd.DataFrame({'Model': Government.taxRevenue(GE_data), 'IO': GE_data['vD'].rctree_pd(GE_data['n_tax']).droplevel(-1)})
```

      Model
      IO

      s

      F
      0.0909422
      -0.018329

      HH
      1.61747
      1.61747

      I_B
      0.263686
      0.263686

      I_M
      0.202365
      0.202365

      a
      0.058697
      0.058697

      b
      0.827386
      0.827386

      itory
      0.00155662
      0.00228146

To adjust the final two sectors' tax revenues, we can move the taxes on demand upward/downward by the same amount for all goods. we make this adjustment on the demand taxes ($tauD$), as the sectors that are not taxed lump-sum (itory, F) are not supplying anything in the model:

```python
rates = Government.balanceIO_advalorem(GE_data)
GE_data['tauD'] = rates[~(rates==0)]
```

And adjust the prices with taxes, to be defined as the equilibrium price, plus taxes:

```python
GE_data['PwT'].vals = GE_data.get('PwT').add(GE_data.get('tauD'),fill_value=0)
```

Export the database:

```python
GE_data.export(repo=gams_folder)
```

Beyond this, we require a lot of data. This is handled along the way in the specific modules.

## **3: Model components**

In the following we go through the various modules, including (1) which data we require for the model to run, (2) how to initialize the model, (3) run and calibrate the model separately, and (4) store the calibrated version.

## **3.1: The *Production.py* module**

*The production module is handled in more detail in the notebook 'Ex1\_production.ipynb'. Here we run briefly through the model.*

The production module is build primarily on nesting trees. The trees specify how the sectors combine inputs to produce its respective outputs. A number of different type of nesting functions are available (see *gams\_production.py*). In the following, the two production sectors in our toy model simply use the relevant inputs in a nested CES manner.

### *Required Data:*

The necessary data includes:
1. IO data on the sectors (already included).
2. The nesting structure of inputs including technical parameters detailing the production structure (CES parameters).
3. An inventory of the durables owned by each sector.

In the current case, the model includes two durables for each production sector: Machines ($iM$) and buildings ($iB$). This information is read from the IO data, but can also be added manually along the way. The production module includes quite a lot of flexibility (e.g. in nesting structure and functional forms). The following illustrates an example where a lot of default values are applied:

*Define settings for the module:*

```python
name_module = 'p'
trees = {'a': {'file': 'S1.xlsx', 'sheets': ['lower_nests', 'upper_nest']},
         'b': {'file': 'S2.xlsx', 'sheets': ['lower_nests', 'upper_nest']}}
temp_namespace = {'a': 'a_in', 'b':'b_in'} # used when reading in nesting trees, to distinguish input and output elements with identical names.
kwargs_st = {'sector': True, 'ss': GE_data.get('s_prod')} # settings for initializing the module
```

*Read in data to nesting tree:*

```python
nts = {}
for s,t in trees.items():
    nts[s] = nesting_tree.nesting_tree(name=name_module) # initialize tree
    for tree in t['sheets']:
        nts[s].add_tree(data_folder+'\\'+t['file'],tree_name=tree,**{'sheet':tree}) # add nesting structure
        DataBase.GPM_database.merge_dbs(nts[s].trees[tree].database, excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+t['file'],{tree:'vars'}),'second') # add data
        if tree.startswith('lower'):
            nts[s].trees[tree].temp_namespace = temp_namespace
    nts[s].run_all(s0=s) # add default attributes from nesting structure
nesting_tree.merge_nts(list(nts.values())[0], list(nts.values())[1:]) # merge trees into one.
nt = list(nts.values())[0]
```

### *Static model:*

Start by setting up a static model from the nesting tree:

```python
gm_static = Production.pr_static(nt=nt,work_folder = work_folder,**{'data_folder':work_folder,'name':'p_static'})
```

Add data from the general equilibrium data, but restrict it to the data on production sectors (not necessary, but neat):

```python
GE_prod = small_updates.subset_db(GE_data.copy(),GE_data.get('s_prod'))
```

Calibrate the model (See *Ex1_production* for more on these settings):

*1: Calibrate to inputs that are exogenous in the baseline settings:*

```python
gm_static.write_and_run(name='v1',add_checkpoint='v1')
db_temp = gm_static.slice_exo(GE_prod,copy=True)
gm_static.model_instances['v1'].solve_sneakily(db_star=db_temp, from_cp = True, cp_init = gm_static.checkpoints['v1'], kwargs_shock={'n_steps':10})
```

    {'Modelstat': 16.0, 'Solvestat': 1.0}

*2: read current solution back to main database:*

```python
db = gm_static.model_instances['v1'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_static.model.database.symbols]; # delete symbols that are created only in order to calibrate model.
gm_static.model.database.merge_dbs(gm_static.model.database,db,'second')
```

*3: calibrate to the other moments in the data:*

```python
gm_static.reset_settings()
gm_static.setstate('DC')
db_temp = gm_static.slice_exo(GE_prod,copy=True)
gm_static.calibrate_sneaky(db_temp,**{'n_steps':100})
```

    {'Modelstat': 16.0, 'Solvestat': 1.0}

*inspect solution compared to IO data:*

```python
db = gm_static.model_instances['baseline'].out_db
```

```python
var= 'qD'
diagnostics.compare_data.std_diagnostics_var(db,GE_data,var)
```

### *Dynamic model:*

Next, we set up the dynamic model, building on the static solution, as well as the IO data. Specifically, we: (1) Initialize using nesting structure, (2) add specification of durables from IO data, (3) Initialize parameter values and initial guess of variables using static model, (4) run.

```python
gm_p = Production.pr_dynamic(nt=nt,work_folder=work_folder,kwargs_st = {'sector':True, 'ss': GE_data.get('s_prod')}, gs_v=gs,**{'data_folder':gams_folder,'name':name_module}) # initialize model.
gm_p.add_dur(GE_data.get('dur'),dur2inv = GE_data.get('dur2inv')) # add specification of durables
gm_p.ivfs(db,merge=False) # initialzie levels from static model
gm_p.initialize_variables(**{'check_variables': True}) # give all variables without a initial level some (semi) random number.
gm_p.model.database[gm_p.n('mu')].vals = db.get(gm_p.n('mu')) # update values of mu-parameters to static solution
```

Finally, update the prices $PwT$ on investment goods that are not automatically loaded from the static version (where the investment behavior was not included):

```python
gm_p.model.database[gm_p.n('PwT')] = DataBase_wheels.repeat_variable_windex(GE_data['PwT'].rctree_pd(GE_data['inv']),gs.get('txE')).combine_first(gm_p.get('PwT'))
```

To calibrate to IO data, we start by adding the baseline year to the IO data:

```python
GE_prod_t = DataBase.GPM_database()
for var in GE_prod.variables_flat:
    GE_prod_t[var] = DataBase_wheels.repeat_variable_windex(GE_prod.get(var),gm_p.get('t0'))
```

We further adjust the capital depreciation rates to ensure the model is in a steady state. The depreciation rates are then fitted to data at a later point in the proces (see Example1_cont.ipynb):

```python
gm_p.ss_rDepr(GE_data)
```

The calibration method we apply form a grid of values between the database in the model, and the target database (GE\_prod\_t in this case),  and asks GAMS to solve the model on this grid. However, this only works on exogenous variables. To make sure that this works we do the following:
1. Set the *state* of the model to calibration.
2. Subset the target database *GE\_prod\_t* to only include exogenous variables (all *gmspython* models can access the exogenous/endogenous variables by calling the *self.var_exo(symbol)* method).
3. Run the calibration function.

```python
gm_p.setstate('DC')
```

*Subset variables to exogenous variables:*

```python
GE_prod_t = gm_p.slice_exo(GE_prod_t,copy=False)
```

*Calibrate sneakily (target the database GE\_prod\_t, if files w. same names exists overwrite them, sneak up on the solution in n\_steps):*

```python
gm_p.calibrate_sneaky(GE_prod_t,overwrite=True,**{'n_steps': 10,'diff':True})
```

    {'Modelstat': 16.0, 'Solvestat': 1.0}

*Store as pickle to run from at a later point:*

```python
db = gm_p.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_p.model.database.symbols]; # delete symbols that are created only in order to calibrate model.
gm_p.model.database.merge_dbs(gm_p.model.database,db,'second')
gm_p.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_p'

```python
compare_vars,year = 'qD',2
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_prod[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_prod[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(8,6));
```

## **3.2: The *Household.py* module**

*The household module is handled in more detail in the notebook 'Ex1\_household.ipynb'.*

Similarly to the production module, the household module is also initialized using nesting trees. The trees specify how the consumer derives utility from aggregating goods. The nesting trees used for the household are different, however, in a number of ways. For instance, whereas the 'most upper' nest of the production module always symbolizes outputs (supply), and the 'most lower' nest inputs (demand), there is no such restriction on the household nesting tree.

The current module uses a CRRA intertemporal utility setup, that is only tied to the upper-most aggregate level of consumption in the nesting tree (this can easily be altered). Furthermore, we fix the labor supply (but can similarly easily be endogenized / included in the nesting tree).

### *Required Data:*

The necessary data includes:
1. IO data on the sectors (already included).
2. Nesting structure of the utility function, including preference parameters (impatience, CES, CRRA).
3. An inventory of the assets/endowments of the household sector(s), as well as relevant tax rates (see tax module for more).

*Define settings for the module (simpler setup than production, as we only use one tree, i.e. no looping)*

```python
name_module = 'HH'
tree_name = 'HH_agg'
file, sheet = 'HH.xlsx', 'nesting'
```

*Define partial equilibrium data, and general equilibrium data that are subsetted for the household sector:*

```python
PE_data = ReadData.PE_from_GE(GE_data,GE_data.get('s_HH')[0])
GE_HH = small_updates.subset_db(GE_data.copy(),GE_data.get('s_HH'))
```

*read in nesting tree (similar to production, but a simpler setup):*

```python
nt = nesting_tree.nesting_tree_hh(name=name_module,**{'version':'v1'}) # the version specifies calibration type.
nt.add_tree(data_folder+'\\'+file,tree_name=tree_name,**{'sheet':sheet}) # initialize tree
data = excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+file,{sheet:'vars'})
DataBase.GPM_database.merge_dbs(nt.trees[tree_name].database, data ,'second') # add data
nt.run_all(PE_data,s0=GE_data.get('s_HH')[0],postfix='_'+name_module,int_temp = data['crra'].index) # add attributes
```

### *Static model:*

As in the production module, we start by setting up a static version of the model and calibrate this to the IO data. The steps are generally the same here as in the production module: (1) Initialize static model from nesting tree, (2) Add IO data to the model database, (3) solve and calibrate model.

*set up data and model:*

```python
gm_static = Household.hh_static(nt=nt,work_folder=work_folder,**{'data_folder':work_folder,'name':'hh_static'})
DataBase.GPM_database.merge_dbs(gm_static.model.database,GE_HH,'second')
```

*slice IO data to only use exongeous data:*

```python
gm_static.setstate('DC') # set state to 'dynamic calibration (DC), don't initialize settings
GE_HH_exo = gm_static.slice_exo(GE_HH,copy=True)
```

*calibrate (sneakily):*

```python
gm_static.calibrate_sneaky(GE_HH_exo,kwargs_init={'check_variables':True},**{'n_steps':10})
db_static = gm_static.model_instances['baseline'].out_db
```

### *Dynamic model:*

The dynamic model is then based on the static one (as was the case w. the production module). Specifically we (1) initialize from the nesting tree, (2) add the savings subset, (3) initialize levels from the static model solution:

```python
gm_hh = Household.hh_dynamic(nt=nt,work_folder=work_folder, gs_v = gs,**{'data_folder':gams_folder,'name':name_module})
gm_hh.add_svngs() # define subset of savings; call this element 'svngs' as default.
gm_hh.ivfs(db_static,merge=False) # initialize levels from static model
gm_hh.initialize_variables(**{'check_variables':True})
gm_hh.model.database[gm_hh.n('mu')].vals = db_static.get(gm_hh.n('mu')) # update calibrated parameters
```

Next, we add the baseline year to the IO data, slice the IO data to the values that are exogenous in the model and calibrate:

```python
GE_HH_t = DataBase.GPM_database()
for var in GE_HH.variables_flat:
    GE_HH_t[var] = DataBase_wheels.repeat_variable_windex(GE_HH.get(var),gm_hh.get('t0'))
gm_hh.setstate('DC') # set state to 'dynamic calibration (DC)'
GE_HH_t = gm_hh.slice_exo(GE_HH_t,copy=False)
gm_hh.calibrate_sneaky(GE_HH_t,overwrite=True,**{'n_steps': 10,'diff':True})
```

    {'Modelstat': 16.0, 'Solvestat': 1.0}

*Store as pickle to run from at a later point:*

```python
db = gm_hh.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_hh.model.database.symbols]; # delete symbols that are created only in order to calibrate model.
gm_hh.model.database.merge_dbs(gm_hh.model.database,db,'second')
gm_hh.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_HH'

```python
compare_vars,year = 'qD',2
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_HH[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_HH[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(4,3));
```

## **3.3: The *investment.py* module**

*The investment module is handled in more detail in the notebook 'Ex1\_invest.ipynb'.*

The investment module essentially works as a production module (when static). The dynamic module is slightly different.

### *Required Data:*

*Define settings for the module:*

```python
name_module = 'inv'
trees = {'I_B': {'file': 'inv_B.xlsx', 'sheets': ['nest']},
         'I_M': {'file': 'inv_M.xlsx', 'sheets': ['nest']}}
kwargs_st = {'sector': True, 'ss': GE_data.get('s_inv')} # settings for initializing the module
namespace = {k: name+'_'+k for k in ('inp','out','int','wT','map_all','kno_out','kno_inp','n_out','endo_PbT','exo_mu','PwT_dom')} # as the production module uses the same nesting tree, we adjust the standard names here
```

*Define tree:*

```python
nts = {}
for s,t in trees.items():
    nts[s] = nesting_tree.nesting_tree(name=name_module) # initialize tree
    for tree in t['sheets']:
        nts[s].add_tree(data_folder+'\\'+t['file'],tree_name=tree,**{'sheet':tree}) # add nesting structure
        DataBase.GPM_database.merge_dbs(nts[s].trees[tree].database, excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+t['file'],{tree:'vars'}),'second') # add data
    nts[s].run_all(s0=s,**namespace) # add default attributes from nesting structure
nesting_tree.merge_nts(list(nts.values())[0], list(nts.values())[1:]) # merge trees into one.
nt = list(nts.values())[0]
```

### *Static model:*

*similar to production module: initialize from nt, add IO data, calibrate.*

```python
gm_static = Invest.pr_static(nt=nt,work_folder = work_folder,kwargs_ns=namespace,**{'data_folder':work_folder,'name':'I_static'})
GE_inv = small_updates.subset_db(GE_data.copy(),GE_data.get('s_inv'))
DataBase.GPM_database.merge_dbs(gm_static.model.database,GE_inv,'second')
gm_static.setstate('DC')
db_temp = gm_static.slice_exo(GE_inv,copy=True)
gm_static.calibrate_sneaky(db_temp,kwargs_init={'check_variables':True},**{'n_steps':100, 'gridtype': 'pol','phi':0.9})
db_static = gm_static.model_instances['baseline'].out_db
```

### *Dynamic model:*

Set up model and build on static setup/solution:

```python
gm_inv = Invest.inv_dynamic(nt=nt,work_folder=work_folder,gs_v=gs,kwargs_st=kwargs_st,kwargs_ns=namespace,**{'data_folder':gams_folder,'name': name_module})
gm_inv.ivfs(db_static,merge=False)
gm_inv.initialize_variables(**{'check_variables':True})
gm_inv.model.database[gm_inv.n('mu')].vals = db_static.get(gm_inv.n('mu'))
gm_inv.model.database[gm_inv.n('markup')].vals = db_static.get(gm_inv.n('markup'))
```

The investment module does not need a dynamic calibration; thus we simply solve the model in baseline mode and store it:

```python
gm_inv.setstate('DC')
gm_inv.setstate('B')
gm_inv.write_and_run(overwrite=True) # the overwrite=True option overwrites existing file with same names.
```

```python
db = gm_inv.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_inv.model.database.symbols];
gm_inv.model.database.merge_dbs(gm_inv.model.database,db,'second')
gm_inv.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_inv'

```python
compare_vars,year = 'Peq',2
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_inv[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_inv[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(6,4.5));
```

### *Inventory investments:*

The IO data can also include 'inventory investments'. In this example, we simply include a "demand" for inventory that tends to zero over time (in an AR(1) manner). The inclusion of inventory investments are thus primarily to match IO data; behavior is ad hoc.

```python
GE_itory = small_updates.subset_db(GE_data.copy(),GE_data.get('s_itory'))
for var in GE_itory.variables_flat:
    GE_itory[var] = DataBase_wheels.repeat_variable_windex(GE_itory.get(var),gs.get('t0'))
```

```python
name_module = 'itory'
gm_itory = Invest.itoryD(work_folder=work_folder,databases=[GE_itory],gs_v=gs,**{'data_folder':gams_folder,'name':name_module})
gm_itory.write_and_run(kwargs_init={'check_variables':True},overwrite=True)
```

Export as pickle:

```python
db = gm_itory.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_itory.model.database.symbols];
gm_itory.model.database.merge_dbs(gm_itory.model.database,db,'second')
gm_itory.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_itory'

```python
compare_vars,year = 'qD',1
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_data[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_data[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(6,4.5));
```

## **3.4: The *Trade.py* module**

*The trade module is handled in more detail in the notebook 'Ex1\_trade.ipynb'.*

The trade model can be specified in various ways. In this simple module, the trade modules simply specifies the foreign demand for domestically produced goods in an Armington-like manner. As there are more than one foreign/domestic goods, the prices entering the demand function are the domestic price relative to the foreign price of a similar good; this similarity is declared in a mapping denoted *dom2for[n,nn]* coupling domestic goods to a foreign counterpart.

*Define settings for the module:*

```python
name_module = 'trade'
data = {'file': 'trade.xlsx', 'sheets': {'vars': 'vars', 'dom2for': 'maps'}} # read in the sheet 'vars' as variables, and 'dom2for' as mappings.}
kwargs_st = {'sector': True, 'ss': GE_data.get('s_for')} # settings for initializing the module
```

*load data:*

```python
db = excel2py.xl2PM.pm_from_workbook(data_folder+'\\'+data['file'],data['sheets'])
```

*initialize model:*

```python
gm_trade = Trade.trade_dynamic(work_folder=work_folder,kwargs_st=kwargs_st,gs_v=gs,**{'data_folder':gams_folder,'name':name_module})
```

*subset GE data to relevant sectors:*

```python
GE_trade = small_updates.subset_db(GE_data.copy(),GE_data.get('s_for'))
```

*initialize relevant subsets from GE data, and add data loaded from excel sheets:*

```python
gm_trade.add_sets_from_GE(GE_trade)
DataBase.GPM_database.merge_dbs(gm_trade.model.database,db,'second')
```

*initialize variables from GE data:*

```python
gm_trade.ivfs(GE_trade,merge=False) # initialize levels from static model
```

*set to calibration mode and solve:*

```python
GE_trade_t = DataBase.GPM_database()
for var in GE_trade.variables_flat:
    GE_trade_t[var] = DataBase_wheels.repeat_variable_windex(GE_trade.get(var),gm_trade.get('t0'))
gm_trade.setstate('DC')
GE_trade_t = gm_trade.slice_exo(GE_trade_t,copy=False)
gm_trade.calibrate_sneaky(GE_trade_t,overwrite=True,**{'n_steps':2,'diff':True})
```

    {'Modelstat': 15.0, 'Solvestat': 1.0}

*Export as pickle:*

```python
db = gm_trade.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_trade.model.database.symbols];
gm_trade.model.database.merge_dbs(gm_trade.model.database,db,'second')
gm_trade.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_trade'

```python
compare_vars,year = 'qD',2
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_data[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_data[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(6,4.5));
```

## **3.5: The *Government.py* module**

The government sector defines the tax rates that goes into other modules. In the simplest case, we assume that taxation are in constant rates (from the perspective of the firms/consumers). This entails that three tax rates should be supplied by the government sector module:
* $tauD[t,s,n]$: The tax rate levied in the use of inputs such that $PwT[t,s,n] = Peq[t,n]+tauD[t,s,n]$ where $PwT$ is the price with taxes for sectors $s$, good $n$. The revenue from this tax is thus defined as $\sum_{(s,n)\in d\_tauD[s,n]} qD[t,s,n]*tauD[t,s,n]$.
* $tauS[t,s,n]$: The tax rate levied on the output of a sector. This defines the difference between prices before taxes and equilibrium taxes (some variations in how they are specifically used in modules). The revenue from this tax is defined as $\sum_{(s,n)\in d\_tauS[s,n]} qS[t,s,n] * tauS[t,s,n]$.
* $tauLump[t,s]$ for $s\in s\_HH$: The lump sum tax that is charged on households. For other sectors (production, investment), the lump-sum tax is computed as a part of the $tauS[t,s,n]$ tax on the sectors' outputs.

Note that this restricts us to only charge constant rates for supply/demand of any type, i.e. the marginal tax rate = average tax rate. However, nothing prevents us from creating more elaborate functions for how these prices may evolve (as functions e.g. of quantity variables).

### *A simple setup for tax rates:*

In the following we define a simple government sector with the following instruments:
* $tauLump[t,s]$: A lump-sum tax is levied on all sectors except (1) trade and (2) inventory investment sectors.
* $tauS[t,s,n]$: A flat rate charged on the supply of goods.
* $tauD[t,s,n]$: A flat rate charged on the demand of goods.

### *Data requirements:*

Data on the specific tax rates are per default set to 0, unless otherwise specified. In this instance, we read the tax rates in the data section (from the file 'Tax.xlsx'), and added them to the IO data stored in *GE_data*. They were, however, only for the baseline year. In this instance, we assume these are kept constant (for now). Other versions of the government sector may include more elaborate tax rules, thus endogenizing the rates. This should be straightforward to provide.

*settings:*

```python
name_module = 'G'
```

### *Static model:*

```python
gm_static = Government.g_static(GE_data=GE_data.copy(),work_folder=work_folder,**{'data_folder':gams_folder,'name':'g_static'})
```

Run simple baseline model:

```python
gm_static.write_and_run(kwargs_init={'check_variables':True},overwrite=True)
```

*Calibrate:*

```python
gm_static.setstate('SC')
gm_static.reset_settings() # this resets the RunFile and CollectFile settings.
gm_static.write_and_run(overwrite=True) # this overwrites files in the work_folder if they already exists.
db_static = gm_static.model_instances['baseline'].out_db
```

### *Dynamic model:*

```python
gm_G = Government.g_dynamic(GE_data=GE_data.copy(),work_folder=work_folder,gs_v=gs,**{'data_folder':gams_folder,'name':name_module})
gm_G.ivfs(db_static,merge=False)
gm_G.initialize_variables(**{'check_variables': True})
```

```python
GE_G = small_updates.subset_db(GE_data.copy(),gm_G.get('s_G'))
GE_G_t = DataBase.GPM_database()
for var in GE_G.variables_flat:
    GE_G_t[var] = DataBase_wheels.repeat_variable_windex(GE_G.get(var),gm_G.get('t0'))
gm_G.setstate('DC')
GE_G_t = gm_G.slice_exo(GE_G_t,copy=False)
gm_G.calibrate_sneaky(GE_G_t,overwrite=True,**{'n_steps':2,'diff':True})
```

    {'Modelstat': 15.0, 'Solvestat': 1.0}

*Store as pickle to run from at a later point:*

```python
db = gm_G.model_instances['baseline'].out_db
[db.series.__delitem__(sym) for sym in db.symbols if sym not in gm_G.model.database.symbols];
gm_G.model.database.merge_dbs(gm_G.model.database,db,'second')
gm_G.export()
```

    'C:\\Users\\sxj477\\Documents\\GitHub\\GPM_v05\\examples\\gamsmodels\\GE\\gmspython_G'

```python
compare_vars,year = 'tauD',1
ci = DataBase.gpy_symbol(db[compare_vars].rctree_pd(GE_data[compare_vars]).xs(year))
pd.DataFrame({'Baseline': db[compare_vars].rctree_pd(ci).xs(year), 'IO data': GE_data[compare_vars].rctree_pd(ci)}).plot.bar(figsize=(8,6));
```


# Applied Stochastic Processes HW_06

**11510691 程远星$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\AcA}{\mathscr{A}}
\newcommand{\FcF}{\mathscr{F}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}\void^\dagger$**

## Question 1

$\bspace$We consider each of his step, if we consider this as the Markov chain for the drunk man, a $r.v.$ with distribution

$$P\P{X_i} = \begin{cases}
p, & \text{if } X_i = 1 \\
1-p, & \text{if } X_i = -1
\end{cases}$$

$\bspace$then by the strong law of large numbers, we have the expected value of his final step, $S = \sum X_i$ lands on:

$$\Exp\SB{S} = n\Exp\SB{X_i} = n\P{2p-1} \Rightarrow P\CB{\lim_{n\to\infty} \Exp\SB{S} = \infty} = 1, p\neq 0.5$$

$\bspace$meaning that the state $0$ cannot be recurrent.

## Question 2

$\bspace$Suppose the little $p$ in the problem has the same meaning with the capital $P$ in the textbook (please note that next time THANKS), then we have

$$\begin{align}
p_{ij}^{\P{n}} &= P\CB{X_n = j \mid X_0 = i}\\
&= \sum_{m=1}^{\infty}P\CB{X_n=j \mid X_{n-m} = j}\cdot P\CB{X_{n-m} = j,X_k \neq j, 1\leq k < m \mid X_0 = i}\\
&= \sum_{m=1}^{\infty}f_{ij}^{m} \cdot p_{jj}^{\P{n-m}}
\end{align}$$

## Question 3

$\P{1}$

$\bspace$There are two classes, $\CB{0,1}$ and $\CB{2}$. State $2$ is recurrent since ever in, never out. And states $0$ and $1$ are transient since it's possible they move to state $2$ and never come back.

$\P{2}$

$\bspace$Since $T>2$, $X_2 \neq 2$ which also concludes that $X_1 \neq 2$. So that after calculating the $\mathbf{P}^2$

$$\mathbf{P}^2 = \begin{Vmatrix}
0.7 & 0.2 & 0.1 \\
0.3 & 0.5 & 0.2 \\
0 & 0 & 1
\end{Vmatrix}^2 = \begin{Vmatrix}
0.55 & 0.24 & 0.21 \\
0.36 & 0.31 & 0.33 \\
0 & 0 & 1
\end{Vmatrix}$$

$\bspace$Thus the probability to find is $\ffrac{0.55} {0.55+0.24} = \ffrac{55} {79}$

>I just met a problem to find why this method is different and I just feel something is not right but don't know how. Since $T>2$, during the first two steps the $r.v.$ will never enter state $2$ so we have the new transition matrix $\mathbf{P}'$ and its square is:
>
>$$\mathbf{P'}^2 = \begin{Vmatrix}
7/9 & 2/9 \\
3/8 & 5/8
\end{Vmatrix}^2 = \begin{Vmatrix}
223/324 & 101/324 \\
101/192 & 91/192
\end{Vmatrix}$$
>
>$\bspace$So that the probability is
>
>$$P\CB{X_2 = 0\mid X_0 = 0,T >2} = P\CB{X_2 = 0\mid X_0 = 0,X_1 \neq 2,X_2 \neq 2} = \ffrac{223} {324}$$

## Question 4

$\bspace$By the given information we can assert that $P_{ij}^{\P{n}}>0$ for certain $n$. If $n\leq M$ then the proof is finished, otherwise, there're must be at least one state that has been entered for at least twice.

$\bspace$So we just ignore the transitions that happen between that "two" identical states so that during moving from state $i$ to $j$, there's no time we're in the same state. And since there's only $M$ possible states, we conclude that there's still one possible way with less than $M$ transitions that can move from $i$ to $j$.

## Question 5

$$\begin{align}
\;&P\CB{X_n = m \mid X_0 = i, X_k \neq r, k = 1,2,\dots,n} \\[0.8em]
=\;& \ffrac{P\CB{X_n = m, X_k \neq r, k = 1,2,\dots,n \mid X_0 = i}} {P\CB{X_k \neq r, k = 1,2,\dots,n\mid X_0 = i}}\\
=\;& \ffrac{P\CB{X_n = m, X_n \neq r \mid X_0 = i}\cdot \d{\prod_{j=1}^{n-1}P\CB{X_j\neq r\mid X_0 = i}}} {\d{\prod_{j=1}^{n}P\CB{X_j\neq r\mid X_0 = i}}}\\
=\;& \ffrac{P\CB{X_n = m \mid X_0 = i}} {P\CB{X_n \neq r\mid X_0 = i}}, r\neq m,i\\
=\;& \ffrac{P\CB{X_n = m \mid X_0 = i}} {1-P\CB{X_n = r \mid X_0 = i}} \\
=\;& \ffrac{P_{i,m}^n} {1-P_{i,r}^n} = Q_{i,m}^{n}
\end{align}$$

>Can't find a counterexample...😭😭😭

Probability and Statistics/Applied Random Process/HW/HW_06.ipynb


# Grid vs. Random Search Hyperparameter Optimization

## Setup

### Installation

```python
!pip install matbench
!pip install CBFV
!pip install torch
!pip install torchvision
```

    Requirement already satisfied: matbench in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (0.5)
    Requirement already satisfied: matminer==0.7.4 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matbench) (0.7.4)
    Requirement already satisfied: monty==2021.8.17 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matbench) (2021.8.17)
    Requirement already satisfied: scikit-learn==1.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matbench) (1.0)
    Requirement already satisfied: six>=1.16.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (1.16.0)
    Requirement already satisfied: pymongo>=3.12.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (4.0.1)
    Requirement already satisfied: tqdm>=4.62.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (4.62.3)
    Requirement already satisfied: requests>=2.26.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (2.27.1)
    Requirement already satisfied: future>=0.18.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (0.18.2)
    Requirement already satisfied: pymatgen>=2022.0.11 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (2022.2.7)
    Requirement already satisfied: sympy>=1.8 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (1.8)
    Requirement already satisfied: pint>=0.17 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (0.18)
    Requirement already satisfied: pandas>=1.3.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (1.4.0)
    Requirement already satisfied: jsonschema>=3.2.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (3.2.0)
    Requirement already satisfied: numpy>=1.21.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matminer==0.7.4->matbench) (1.22.2)
    Requirement already satisfied: joblib>=0.11 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from scikit-learn==1.0->matbench) (1.1.0)
    Requirement already satisfied: scipy>=1.1.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from scikit-learn==1.0->matbench) (1.7.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from scikit-learn==1.0->matbench) (3.1.0)
    Requirement already satisfied: pyrsistent>=0.14.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from jsonschema>=3.2.0->matminer==0.7.4->matbench) (0.18.0)
    Requirement already satisfied: setuptools in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from jsonschema>=3.2.0->matminer==0.7.4->matbench) (52.0.0.post20210125)
    Requirement already satisfied: attrs>=17.4.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from jsonschema>=3.2.0->matminer==0.7.4->matbench) (21.2.0)
    Requirement already satisfied: python-dateutil>=2.8.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pandas>=1.3.1->matminer==0.7.4->matbench) (2.8.1)
    Requirement already satisfied: pytz>=2020.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pandas>=1.3.1->matminer==0.7.4->matbench) (2021.1)
    Requirement already satisfied: packaging in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pint>=0.17->matminer==0.7.4->matbench) (21.0)
    Requirement already satisfied: uncertainties>=3.1.4 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (3.1.5)
    Requirement already satisfied: spglib>=1.9.9.44 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (1.16.1)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.17.10)
    Requirement already satisfied: matplotlib>=1.5 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (3.4.2)
    Requirement already satisfied: networkx>=2.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (2.5.1)
    Requirement already satisfied: Cython>=0.29.23 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.29.27)
    Requirement already satisfied: plotly>=4.5.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (5.1.0)
    Requirement already satisfied: tabulate in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.8.9)
    Requirement already satisfied: pybtex in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.24.0)
    Requirement already satisfied: palettable>=3.1.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pymatgen>=2022.0.11->matminer==0.7.4->matbench) (3.3.0)
    Requirement already satisfied: cycler>=0.10 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matplotlib>=1.5->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.10.0)
    Requirement already satisfied: pillow>=6.2.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matplotlib>=1.5->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (8.3.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matplotlib>=1.5->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (1.3.1)
    Requirement already satisfied: pyparsing>=2.2.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from matplotlib>=1.5->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (2.4.7)
    Requirement already satisfied: decorator<5,>=4.3 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from networkx>=2.2->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (4.4.2)
    Requirement already satisfied: tenacity>=6.2.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from plotly>=4.5.0->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (7.0.0)
    Requirement already satisfied: certifi>=2017.4.17 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from requests>=2.26.0->matminer==0.7.4->matbench) (2021.10.8)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from requests>=2.26.0->matminer==0.7.4->matbench) (1.26.6)
    Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from requests>=2.26.0->matminer==0.7.4->matbench) (2.0.11)
    Requirement already satisfied: idna<4,>=2.5 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from requests>=2.26.0->matminer==0.7.4->matbench) (2.10)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from ruamel.yaml>=0.15.6->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (0.2.2)
    Requirement already satisfied: mpmath>=0.19 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from sympy>=1.8->matminer==0.7.4->matbench) (1.2.1)
    Requirement already satisfied: colorama in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from tqdm>=4.62.0->matminer==0.7.4->matbench) (0.4.4)
    Requirement already satisfied: PyYAML>=3.01 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pybtex->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (5.4.1)
    Requirement already satisfied: latexcodec>=1.0.4 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pybtex->pymatgen>=2022.0.11->matminer==0.7.4->matbench) (2.0.1)
    Requirement already satisfied: CBFV in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (1.1.0)
    Requirement already satisfied: pytest in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from CBFV) (6.2.5)
    Requirement already satisfied: pandas in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from CBFV) (1.4.0)
    Requirement already satisfied: tqdm in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from CBFV) (4.62.3)
    Requirement already satisfied: numpy in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from CBFV) (1.22.2)
    Requirement already satisfied: python-dateutil>=2.8.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pandas->CBFV) (2.8.1)
    Requirement already satisfied: pytz>=2020.1 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pandas->CBFV) (2021.1)
    Requirement already satisfied: six>=1.5 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from python-dateutil>=2.8.1->pandas->CBFV) (1.16.0)
    Requirement already satisfied: atomicwrites>=1.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (1.4.0)
    Requirement already satisfied: pluggy<2.0,>=0.12 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (0.13.1)
    Requirement already satisfied: attrs>=19.2.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (21.2.0)
    Requirement already satisfied: colorama in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (0.4.4)
    Requirement already satisfied: packaging in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (21.0)
    Requirement already satisfied: iniconfig in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (1.1.1)
    Requirement already satisfied: toml in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (0.10.2)
    Requirement already satisfied: py>=1.8.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from pytest->CBFV) (1.11.0)
    Requirement already satisfied: pyparsing>=2.0.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from packaging->pytest->CBFV) (2.4.7)
    Requirement already satisfied: torch in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (1.10.2)
    Requirement already satisfied: typing-extensions in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from torch) (4.1.1)
    Collecting torchvision
      Downloading torchvision-0.11.3-cp39-cp39-win_amd64.whl (947 kB)
    Requirement already satisfied: torch==1.10.2 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from torchvision) (1.10.2)
    Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from torchvision) (8.3.1)
    Requirement already satisfied: numpy in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from torchvision) (1.22.2)
    Requirement already satisfied: typing-extensions in c:\users\taylo\miniconda3\envs\my_pymatgen\lib\site-packages (from torch==1.10.2->torchvision) (4.1.1)
    Installing collected packages: torchvision
    Successfully installed torchvision-0.11.3

### Imports

```python
import torch
import torch.nn as nn # All neural network modules, nn.Linear, nn.Conv2d, BatchNorm, Loss functions
import torch.optim as optim # For all Optimization algorithms, SGD, Adam, etc.
import torch.nn.functional as F # All functions that don't have any parameters
from torch.utils.data import DataLoader # Gives easier dataset managment and creates mini batches
import torchvision.datasets as datasets # Has standard datasets we can import in a nice and easy way
import torchvision.transforms as transforms # Transformations we can perform on our dataset

from matbench.bench import MatbenchBenchmark
from CBFV.composition import generate_features
import pandas as pd
```

### Data

```python
mb = MatbenchBenchmark(subset=["matbench_expt_is_metal"])
task = list(mb.tasks)[0]
task.load()
fold0 = task.folds[0]
train_inputs, train_outputs = task.get_train_and_val_data(fold0)
test_inputs, test_outputs = task.get_test_data(fold0, include_target=True)
print(train_inputs[0:2], train_outputs[0:2])
print(train_outputs.shape, test_outputs.shape)

```

    2022-02-17 14:00:08 INFO     Initialized benchmark 'matbench_v0.1' with 1 tasks:
    ['matbench_expt_is_metal']
    2022-02-17 14:00:08 INFO     Loading dataset 'matbench_expt_is_metal'...
    2022-02-17 14:00:08 INFO     Dataset 'matbench_expt_is_metal loaded.
    mbid
    mb-expt-is-metal-0001      Ag(AuS)2
    mb-expt-is-metal-0002    Ag(W3Br7)2
    Name: composition, dtype: object mbid
    mb-expt-is-metal-0001    True
    mb-expt-is-metal-0002    True
    Name: is_metal, dtype: bool
    (3936,) (985,)

```python
train_inputs.describe()
```

    count         3936
    unique        3936
    top       Ag(AuS)2
    freq             1
    Name: composition, dtype: object

```python
train_outputs.describe()
```

    count      3936
    unique        2
    top       False
    freq       1976
    Name: is_metal, dtype: object

```python
train_df = pd.DataFrame({"formula": train_inputs, "target": train_outputs})
test_df = pd.DataFrame({"formula": test_inputs, "target": test_outputs})
train_df
```

      formula
      target

      mbid

      mb-expt-is-metal-0001
      Ag(AuS)2
      True

      mb-expt-is-metal-0002
      Ag(W3Br7)2
      True

      mb-expt-is-metal-0003
      Ag0.5Ge1Pb1.75S4
      False

      mb-expt-is-metal-0005
      Ag2BBr
      True

      mb-expt-is-metal-0006
      Ag2BiO3
      True

      ...
      ...
      ...

      mb-expt-is-metal-4916
      ZrSiTe
      True

      mb-expt-is-metal-4917
      ZrTaN3
      False

      mb-expt-is-metal-4918
      ZrTe
      True

      mb-expt-is-metal-4920
      ZrTiF6
      True

      mb-expt-is-metal-4921
      ZrW2
      True

3936 rows × 2 columns

```python
X_train, y_train, _, _ = generate_features(train_df)
X_train
```

    Processing Input Data: 100%|████████████████████████████████████████████████████| 3936/3936 [00:00<00:00, 10725.93it/s]

    	Featurizing Compositions...

    Assigning Features...: 100%|█████████████████████████████████████████████████████| 3936/3936 [00:00<00:00, 8724.69it/s]

    	Creating Pandas Objects...

      avg_Atomic_Number
      avg_Atomic_Weight
      avg_Period
      avg_group
      avg_families
      avg_Metal
      avg_Nonmetal
      avg_Metalliod
      avg_Mendeleev_Number
      avg_l_quantum_number
      ...
      mode_polarizability(A^3)
      mode_Melting_point_(K)
      mode_Boiling_Point_(K)
      mode_Density_(g/mL)
      mode_specific_heat_(J/g_K)_
      mode_heat_of_fusion_(kJ/mol)_
      mode_heat_of_vaporization_(kJ/mol)_
      mode_thermal_conductivity_(W/(m_K))_
      mode_heat_atomization(kJ/mol)
      mode_Cohesive_energy

      0
      47.400000
      113.186656
      4.600000
      13.000000
      5.200000
      0.600000
      0.400000
      0.000000
      74.600000
      1.200000
      ...
      2.900
      385.95
      717.85
      2.07000
      0.128
      1.71750
      9.8000
      0.26900
      279.0
      2.85

      1
      46.714286
      110.931629
      4.619048
      13.571429
      6.666667
      0.333333
      0.666667
      0.000000
      81.000000
      1.238095
      ...
      3.100
      265.95
      331.95
      3.12000
      0.473
      5.28600
      15.4380
      0.12200
      112.0
      1.22

      2
      36.275862
      85.159738
      4.000000
      14.896552
      6.172414
      0.310345
      0.551724
      0.137931
      83.482759
      0.931034
      ...
      2.900
      385.95
      717.85
      2.07000
      0.710
      1.71750
      9.8000
      0.26900
      279.0
      2.85

      3
      33.500000
      76.612850
      4.000000
      13.000000
      5.500000
      0.500000
      0.250000
      0.250000
      74.250000
      0.500000
      ...
      7.900
      1235.15
      2485.15
      10.50000
      0.235
      11.30000
      250.5800
      429.00000
      284.0
      2.95

      4
      33.500000
      78.785828
      3.666667
      14.166667
      5.666667
      0.500000
      0.500000
      0.000000
      79.500000
      0.666667
      ...
      0.793
      54.75
      90.15
      0.00143
      0.920
      0.22259
      3.4099
      0.02674
      249.0
      2.62

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      3931
      35.333333
      82.303167
      4.333333
      11.333333
      5.333333
      0.333333
      0.666667
      0.000000
      70.666667
      1.333333
      ...
      5.400
      722.65
      1262.95
      2.33000
      0.200
      16.90000
      52.5500
      2.35000
      197.0
      2.19

      3932
      26.800000
      62.838424
      3.400000
      10.800000
      5.800000
      0.400000
      0.600000
      0.000000
      67.600000
      1.400000
      ...
      1.100
      63.25
      77.35
      0.00125
      1.040
      0.36040
      2.7928
      0.02598
      473.0
      4.92

      3933
      46.000000
      109.412000
      5.000000
      10.000000
      5.000000
      0.500000
      0.500000
      0.000000
      67.000000
      1.500000
      ...
      5.500
      722.65
      1262.95
      6.24000
      0.200
      16.90000
      52.5500
      2.35000
      197.0
      2.19

      3934
      14.500000
      31.636802
      2.625000
      13.750000
      7.000000
      0.250000
      0.750000
      0.000000
      80.625000
      1.250000
      ...
      0.634
      53.35
      85.05
      0.00170
      0.820
      0.25520
      3.2698
      0.02790
      79.0
      0.84

      3935
      62.666667
      152.968000
      5.666667
      5.333333
      4.000000
      1.000000
      0.000000
      0.000000
      48.666667
      2.000000
      ...
      11.100
      3683.15
      5933.15
      19.30000
      0.130
      35.40000
      824.0000
      174.00000
      849.0
      8.90

3936 rows × 264 columns

```python
X_test, y_test, _, _ = generate_features(test_df)
X_test
```

    Processing Input Data: 100%|██████████████████████████████████████████████████████| 985/985 [00:00<00:00, 12605.81it/s]

    	Featurizing Compositions...

    Assigning Features...: 100%|███████████████████████████████████████████████████████| 985/985 [00:00<00:00, 8309.95it/s]

    	Creating Pandas Objects...

      avg_Atomic_Number
      avg_Atomic_Weight
      avg_Period
      avg_group
      avg_families
      avg_Metal
      avg_Nonmetal
      avg_Metalliod
      avg_Mendeleev_Number
      avg_l_quantum_number
      ...
      mode_polarizability(A^3)
      mode_Melting_point_(K)
      mode_Boiling_Point_(K)
      mode_Density_(g/mL)
      mode_specific_heat_(J/g_K)_
      mode_heat_of_fusion_(kJ/mol)_
      mode_heat_of_vaporization_(kJ/mol)_
      mode_thermal_conductivity_(W/(m_K))_
      mode_heat_atomization(kJ/mol)
      mode_Cohesive_energy

      0
      46.206897
      111.032290
      4.551724
      14.896552
      6.172414
      0.310345
      0.551724
      0.137931
      84.034483
      0.931034
      ...
      3.800
      490.15
      958.15
      4.79000
      0.320
      6.69400
      37.7000
      0.52000
      227.0
      2.46

      1
      25.750000
      57.815474
      3.375000
      14.000000
      5.750000
      0.375000
      0.500000
      0.125000
      78.375000
      0.625000
      ...
      0.793
      54.75
      90.15
      0.00143
      0.920
      0.22259
      3.4099
      0.02674
      249.0
      2.62

      2
      46.750000
      107.506150
      5.000000
      10.750000
      4.000000
      1.000000
      0.000000
      0.000000
      64.250000
      0.500000
      ...
      7.900
      1235.15
      2485.15
      10.50000
      0.235
      11.30000
      250.5800
      429.00000
      284.0
      2.95

      3
      27.125000
      61.084025
      3.500000
      13.125000
      5.500000
      0.500000
      0.500000
      0.000000
      74.875000
      0.750000
      ...
      0.793
      54.75
      90.15
      0.00143
      0.920
      0.22259
      3.4099
      0.02674
      249.0
      2.62

      4
      42.454545
      97.547122
      4.636364
      13.000000
      5.272727
      0.636364
      0.363636
      0.000000
      74.818182
      0.363636
      ...
      7.900
      1235.15
      2485.15
      10.50000
      0.235
      11.30000
      250.5800
      429.00000
      284.0
      2.95

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      980
      24.000000
      51.785333
      3.666667
      12.000000
      6.000000
      0.333333
      0.666667
      0.000000
      73.333333
      1.333333
      ...
      2.900
      385.95
      717.85
      2.07000
      0.710
      1.71750
      9.8000
      0.26900
      279.0
      2.85

      981
      45.000000
      104.684667
      5.000000
      9.000000
      4.666667
      0.666667
      0.333333
      0.000000
      61.666667
      1.666667
      ...
      6.600
      904.15
      2223.15
      6.51000
      0.210
      16.90000
      77.1400
      22.70000
      262.0
      2.75

      982
      37.000000
      85.092000
      4.500000
      10.000000
      5.500000
      0.500000
      0.500000
      0.000000
      66.500000
      1.500000
      ...
      3.800
      490.15
      958.15
      4.79000
      0.270
      6.69400
      37.7000
      0.52000
      227.0
      2.46

      983
      22.666667
      49.131667
      3.666667
      10.666667
      5.333333
      0.333333
      0.666667
      0.000000
      66.666667
      1.333333
      ...
      5.400
      1683.15
      2628.15
      2.33000
      0.710
      50.55000
      384.2200
      148.00000
      452.0
      4.63

      984
      23.000000
      50.745850
      3.750000
      7.000000
      4.750000
      0.750000
      0.250000
      0.000000
      54.250000
      1.750000
      ...
      14.600
      1933.15
      3560.15
      4.54000
      0.520
      15.45000
      421.0000
      21.90000
      470.0
      4.85

985 rows × 264 columns

## construct model

Let's build our nn in pytorch

```python

batch_size = 64
train_dataset = datasets.MNIST(
    root="dataset/",
    train=True,
    transform=transforms.ToTensor(),
    download=True,
)
train_loader = DataLoader(
    dataset=train_dataset, batch_size=batch_size, shuffle=True
)
test_dataset = datasets.MNIST(
    root="dataset/",
    train=False,
    transform=transforms.ToTensor(),
    download=True,
)
test_loader = DataLoader(
    dataset=test_dataset, batch_size=batch_size, shuffle=True
)
```

```python

```

We can do the same grid vs random search with another model, like a decision tree classifier

```python

```

```python

```

## Test

```python

```

## Code Graveyard

```python
# from sklearn.preprocessing import StandardScaler
# scaler = StandardScaler()
# scaler.fit(X_train)
# X_train = scaler.transform(X_train)
# X_test = scaler.transform(X_test)
# X_train
```


```python
%matplotlib inline
```

Synthetic seismograms using the convolutional model
---------------------------------------------------

The simplest way to get a seismogram (in time x offset) is through the
convolutional model

\begin{align}trace(t) = wavelet(t) \ast reflectivity(t)\end{align}

Module :mod:`fatiando.seismic.conv` defines functions for doing this
convolution, calculating the required reflectivity, and converting from depth a
model into time.

```python
import numpy as np
import matplotlib.pyplot as plt
from fatiando.seismic import conv
from fatiando.vis import mpl

# Define the parameters of our depth model
n_samples, n_traces = [600, 100]
velocity = 1500*np.ones((n_samples, n_traces))
# We'll put two interfaces in depth
velocity[150:, :] = 2000
velocity[400:, :] = 3500
dt = 2e-3

# We need to convert the depth model we made above into time
vel_l = conv.depth_2_time(velocity, velocity, dt=dt, dz=1)
# and we'll assume the density is homogeneous
rho_l = 2200*np.ones(np.shape(vel_l))
# With that, we can calculate the reflectivity model in time
rc = conv.reflectivity(vel_l, rho_l)
# and finally perform our convolution
synt = conv.convolutional_model(rc, 30, conv.rickerwave, dt=dt)

# We can use the utility function in fatiando.vis.mpl to plot the seismogram
fig, axes = plt.subplots(1, 2, figsize=(8, 5))

ax = axes[0]
ax.set_title("Velocity model (in depth)")
tmp = ax.imshow(velocity, extent=[0, n_traces, n_samples, 0],
                cmap="copper", aspect='auto', origin='upper')
fig.colorbar(tmp, ax=ax, pad=0, aspect=50)
ax.set_xlabel('Trace')
ax.set_ylabel('Depth (m)')

ax = axes[1]
ax.set_title("Synthetic seismogram")
mpl.seismic_wiggle(synt[:, ::20], dt, scale=1)
mpl.seismic_image(synt, dt, cmap="RdBu_r", aspect='auto')
ax.set_xlabel('Trace')
ax.set_ylabel('Time (s)')
plt.tight_layout()
plt.show()
```


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,y,t = symbols("x h y t")
Image(url= "https://i.imgur.com/B6ERnuf.png")
```

```python
expr = (6/x**4) + (-3/x**5) + 12
def F(x):
    return expr
print(integrate(F(x)))
```

    12*x + (3 - 8*x)/(4*x**4)

```python
Image(url= "https://i.imgur.com/vFxYle2.png")
```

```python

```


```
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```

#from google.colab import drive
#drive.mount('/content/gdrive')
```

```
#Q1. Execute the following statement. What is displayed? What does it mean?
!pwd
```

    /content

```
#Q2: Execute the following statement. What happens?  Examine the left column of your colab page to see what happens.
!git clone https://github.com/fastai/course-v3.git
```

    fatal: destination path 'course-v3' already exists and is not an empty directory.

```
#export
#Q3a: Execute the following statement. Read the error message. Explain what it means.

from exp.nb_02 import *

```

```
#Q3b. You can solve the problem by executing the following statement before the above statement.
# Explain why the following statements can solve this "module not found" problem.
```

```
import sys
sys.path.append('/content/course-v3/nbs/dl2')
sys.path
```

```
import torch.nn.functional as F
```

```

```

## Initial setup

### Data

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=1786)

```
from IPython.display import Image
from six.moves import urllib

opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)

def get_data():
    import os
    import torchvision.datasets as datasets
    root = '../data'
    if not os.path.exists(root):
        os.mkdir(root)

    train_set = datasets.MNIST(root=root, train=True, download=True)
    test_set = datasets.MNIST(root=root, train=False, download=True) #load validation set
    x_train, x_valid = train_set.data.split([50000, 10000])
    y_train, y_valid = train_set.targets.split([50000, 10000])
    return (x_train.view(50000, -1) / 256.0), y_train.float(), (x_valid.view(10000, -1))/ 256.0, y_valid.float()

```

```
#mpl.rcParams['image.cmap'] = 'gray'
```

```
#Q4: when you execute the following statement, where is the downloaded data stored? Examine the left column of your colab page.
x_train,y_train,x_valid,y_valid = get_data()
```

```
#Q5a: Execute the following statement. What does the number displayed mean?
len(x_train)
```

    50000

```
#Q5b: Execute the following statement. What do the numbers displayed refer to?
x_train[0]
```

```
#Q5c: Execute the following statement. What do the numbers displayed refer to?
y_train[0]
```

    tensor(5.)

```
#Q5d. Execute the following statement. What does the number displayed refer to?
x_train[0].shape
```

    torch.Size([784])

```
#Q5e. Execute the following statement. What do the numbers displayed refer to?
x_train.shape
```

    torch.Size([50000, 784])

```
#Q5f. Execute the following statement. What do the numbers displayed refer to?
y_train.shape
```

    torch.Size([50000])

```
#Q6: Display the values of n,m, c, and nh. What are they? For what are they used in the following code?
n,m = x_train.shape
c = y_train.max()+1
nh = 50
```

```
#The following defines Model class, which will be used to create a neural net.
#Q7a: nn.Module is the parent class of class Model. Why do you want to make Model a child class to its parent nn.Module, rather than making Model stand on its own?
#Q7b. See the definition of self.layers field. It contains nn.Linear(n_in,nh). What is the difference between nn.Linear class nn.Linear(n_in,nh)?
#Q7c. Would you think that the weight and bias parameters of the two linear layers are initialized when object nn.Linear(n_in,nh) is constructed? If so, guess why.

class Model(nn.Module):
    def __init__(self, n_in, nh, n_out):
        super().__init__()
        self.layers = [nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out)]

    def __call__(self, x):
        for l in self.layers: x = l(x)
        return x
```

```
#Q8. When the following statement is executed, what happens? Explain by referring to a method in class Model.
model = Model(m, nh, 10)
```

```
#Q9a. When the following statement is executed, what happens? Explain by referring to a method in class Model.
#Q9b. Draw a diagram or a graph by using arrows and boxes, which shows what computation is performed when model(x_train) is executed.
pred = model(x_train)
```

```
#Q10: Execute the following statement. What do the 10 displayed numbers refer to?
pred[0]
```

    tensor([-0.1243, -0.1045, -0.0993, -0.1187, -0.0220,  0.0487,  0.0301, -0.0693,
            -0.2306,  0.0330], grad_fn=<SelectBackward>)

```
#Q11. Execute the following statement. Is the resulting value near to 1? If not, what does it imply?
pred[0].sum()
```

    tensor(-0.6569, grad_fn=<SumBackward0>)

### Softmax function

First, we will need to compute the softmax of our activations. This is defined by:

$$\hbox{softmax(x)}_{i} = \frac{e^{x_{i}}}{e^{x_{0}} + e^{x_{1}} + \cdots + e^{x_{n-1}}}$$

or more concisely:

$$\hbox{softmax(x)}_{i} = \frac{e^{x_{i}}}{\sum_{0 \leq j \leq n-1} e^{x_{j}}}$$

In practice, we will need the log of the softmax when we calculate the loss.

```
def softmax(x): return (x.exp()/(x.exp().sum(-1,keepdim=True)))
```

```
sm_pred = softmax(pred)

```

```
#Q12. Execute the following statement. What do the 10 displayed numbers refer to?
sm_pred[0]
```

    tensor([0.0940, 0.0959, 0.0964, 0.0945, 0.1041, 0.1117, 0.1097, 0.0993, 0.0845,
            0.1100], grad_fn=<SelectBackward>)

```
#Q13. Execute the following statement. What does the displayed numbers mean?
sm_pred[0].sum()
```

    tensor(1.0000, grad_fn=<SumBackward0>)

## Log softmax

```
def log_softmax(x): return (x.exp()/(x.exp().sum(-1,keepdim=True))).log()
```

```
log_sm_pred = log_softmax(pred)
```

```
log_sm_pred[0]
```

    tensor([-2.3646, -2.3449, -2.3396, -2.3591, -2.2624, -2.1917, -2.2103, -2.3097,
            -2.4710, -2.2073], grad_fn=<SelectBackward>)

## Cross Entropy Loss Function.
 Read the following paragraph to understand what the cross entroy loss functioni is.

The difference between  two probabilities:
 https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation

The cross entropy formula takes in two distributions,  $p(x^{(s)})$, the true distribution (defined by the label data), and $\hat{p}(x^{(s)})$, the estimated distribution (predicted by the neural net), defined over the discrete variable $x^{(s)}$  and is given by

$H(p,\hat{p})=−\sum_{s \in B} p(x^{(s)}) \cdot log(\hat{p}(x^{(s)}))$

In general, $ p(x^{(s)}) = [ p_{1} (x^{(s)}), ..., p_{n}(x^{(s)})]$ is a probability distribution over a set of categories.
But since our $ p(x^{(s)})$ are 1-hot encoded, that is, in the form of  $ p(x^{(s)}) =[0,0,..,0,1,0..]$, where the probability of only one category is one and those of the other categories are all zero, this can be rewritten as

 \begin{equation}
 H(p,\hat{p})= -\sum_{s \in B} [ 0*\log(\hat{p}_{1} (x^{(s)}))
 + 1*\log(\hat{p}_{i} (x^{(s)}) ) +..+
 0* \log( \hat{p}_{n} (x^{(s)}) ) ] \\
  = -\sum_{s \in B} \log(\hat{p}_{i(s)} ) (x^{(s)}) )
  \tag{crossEntroyEq}
 \end{equation}

 Te softmax function plays the role of the probability distribution $\hat{p}_{1} (x^{(s)})$.

  Here $i(s)$ is the index of the one-hot  probability distribution $p(x^{(s)})$ where the probability is one.

# integer array indexing

```
#Q14a.  Execute the following statement. What is the role of list [0,1,2] in the statement?  What do the displayed numbers refer to?
log_sm_pred[ [0,1,2]]
```

    tensor([[-2.3646, -2.3449, -2.3396, -2.3591, -2.2624, -2.1917, -2.2103, -2.3097,
             -2.4710, -2.2073],
            [-2.3923, -2.3010, -2.3765, -2.4043, -2.2272, -2.2951, -2.1785, -2.2922,
             -2.5106, -2.1105],
            [-2.3806, -2.3104, -2.3593, -2.4057, -2.1497, -2.3245, -2.2133, -2.2681,
             -2.4566, -2.2012]], grad_fn=<IndexBackward>)

```
#Q14b.  Execute the following statement. What is the role of list [2,4,6] in the statement?  What do the displayed numbers mean? Explain compare the result of Q15b and the result of this statement.
log_sm_pred[[0,1,2], [2,4,6]]
```

    tensor([-2.3396, -2.2272, -2.2133], grad_fn=<IndexBackward>)

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=2081)

```
range(y_train.long().shape[0])
```

    range(0, 50000)

```

```

```
selectOneHot = log_sm_pred[ range(y_train.long().shape[0]), y_train.long() ]
```

```
selectOneHot.shape
```

    torch.Size([50000])

```
#Q15.  What does selectOneHot refer to?
```

```
def nll(input, target):

   return -input[range(target.shape[0]), target].mean()
```

```
y_train.long().shape[0]
```

    50000

```
#Q16. Read nill function and explain how this function computes the result. You need to refer to {CrossEntropyLoss}.
loss = nll(log_sm_pred, y_train.long())
```

```
loss
```

    tensor(2.3240, grad_fn=<NegBackward>)

```
#Q16. Compare function nll() and the formula CrossEntropyEq. Explain that function nll() computes the cross entropy function between
# predicted probability distrubution of the input images and the ground truth (labeled) probability distribution of the input images.
```

## Basic training loop

Basically the training loop repeats over the following steps:
- get the output of the model on a batch of inputs
- compare the output to the labels we have and compute a loss
- calculate the gradients of the loss with respect to every parameter of the model
- update said parameters with those gradients to make them a little bit better

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=2542)

```
loss_func = F.cross_entropy # This built-in Pytorch function  combines log_softmax and nll in a single function.
# In the following code, we will use this function, not the function nll defined above.
```

```
#export
def accuracy(out, yb): return (torch.argmax(out, dim=1)==yb).float().mean()
```

```
bs=1000                  # batch size

xb = x_train[0:bs]     # a mini-batch from x
preds = model(xb)      # predictions
preds[0], preds.shape

#Q17. What does preds[0] refer to?
```

    (tensor([-0.0992, -0.1511, -0.1637, -0.0874,  0.1793, -0.0683, -0.1372,  0.0398,
             -0.1217, -0.1008], grad_fn=<SelectBackward>), torch.Size([500, 10]))

```
yb = y_train[0:bs]
loss = loss_func(preds, yb.long())
loss
#Q18. What does the value of loss refer to?
```

    tensor(0.1827, grad_fn=<NllLossBackward>)

```
accuracy(preds, yb)
#Q19. What does  the value of the above statement refer to? Examine the value. Is it supposed to be equal to one approximately?
# If it is not at the current moment, what is the reason for it?
```

    tensor(0.9460)

# The mechanism for training the network:

(1) computing the graidents of Loss with respect to tensors at each layer, (2) backpropgation: applying the chain rule to compute the gradient vector of the parameters, (3) updating the parameters.

https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/tensor.py

https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/tensor.py

https://stackoverflow.com/questions/57248777/backward-function-in-pytorch

```
lr = 0.5   # learning rate
epochs = 2 # how many epochs to train for
```

```
#Q20. What does the number  (n-1)//bs + 1 refer to?
import pdb
for epoch in range(epochs):
    for i in range((n-1)//bs + 1): # for each batch in the current epoch
#       pdb.set_trace()
        start_i = i*bs
        end_i = start_i+bs

        xb = x_train[start_i:end_i]
        yb = y_train[start_i:end_i]

        ybHat = model(xb)
        #h = ybHat.register_hook( lambda grad: print (grad) )

        loss = loss_func( ybHat, yb.long())

        #ybHat.grad
        loss.backward() # When you call loss.backward(), all it does is compute gradient of loss
                        # w.r.t all the parameters in loss that have requires_grad = True and store them in parameter.grad attribute for every parameter

        #ybHat.grad
        #The following performs what Pytorch function optimizer.step() does:
        # It updates all the weight and bias parameters based on the gradients of loss with respect to the weight and bias parameter
        with torch.no_grad():
            for l in model.layers:
                if hasattr(l, 'weight'):
                    l.weight -= l.weight.grad * lr
                    l.bias   -= l.bias.grad   * lr

                    #
                    # loss.backward() computes the gradient of loss w.r.t. graph LEAVES.
                    #  This function accumulates gradients in the leaves - you might need to zero
                    # we don't care about gradients from the previous batch.
                    # Not zeroing grads would lead to gradient accumulation across batches
                    l.weight.grad.zero_()
                    l.bias.grad.zero_()

       #Q21:  print the loss and the accuracy after training the net using the validation dataset.
       #      Explain how the loss and accuracy change as each batch is used  for training the network model.
       #      Try to convey detailed and specific information about the progress of training the neural net.
       #      Observe the printed data carefully

       #Q22. Explain why you would use validation dataset to check the progress of learning of your network.

        yHatValid = model(x_valid)
        lossValid = loss_func( yHatValid, y_valid.long() )

        print('epoch={0}, batch ={1}:'.format(epoch, i) )
        print('  lossValid=', lossValid )
        print('  accuracyValid = ', accuracy( yHatValid, y_valid) )

    #Q23. Afer each epoch, print the loss and the accuracy of the network by using the training dataset.
    #     Explain the result. Be attentive to the result.

    yHatTrain = model(x_train)
    lossTrain = loss_func( yHatTrain, y_train.long() )

    print('*******************************************************')
    print('epoch={0}:', epoch )
    print('  lossTrain=', lossTrain)
    print('  accuracyTrain = ', accuracy( yHatTrain, y_train) )

    #Q24. Afer each epoch, print the loss and the accuracy of the network by using the validatio  dataset.
    #     Explain the result. Be attentive to the result.

    yHatValid = model(x_valid)
    lossValid = loss_func( yHatValid, y_valid.long() )

    print('*******************************************************')
    print('epoch={0}:', epoch )
    print('  lossValid ', lossValid)
    print('  accuracyValid = ', accuracy( yHatValid, y_valid) )

```

```
loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
```

    (tensor(0.0979, grad_fn=<NllLossBackward>), tensor(0.9375))

## Using parameters and optim

### Parameters

Use `nn.Module.__setattr__` and move relu to functional:

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=2818)

```
class Model(nn.Module):
    def __init__(self, n_in, nh, n_out):
        super().__init__()
        self.l1 = nn.Linear(n_in,nh)
        self.l2 = nn.Linear(nh,n_out)

    def __call__(self, x): return self.l2(F.relu(self.l1(x)))
```

```
model = Model(m, nh, 10)
```

```
for name,l in model.named_children(): print(f"{name}: {l}")
```

    l1: Linear(in_features=784, out_features=50, bias=True)
    l2: Linear(in_features=50, out_features=10, bias=True)

```
model
```

    Model(
      (l1): Linear(in_features=784, out_features=50, bias=True)
      (l2): Linear(in_features=50, out_features=10, bias=True)
    )

```
model.l1
```

    Linear(in_features=784, out_features=50, bias=True)

```
def fit():
    for epoch in range(epochs):
        for i in range((n-1)//bs + 1):
            start_i = i*bs
            end_i = start_i+bs
            xb = x_train[start_i:end_i]
            yb = y_train[start_i:end_i]
            loss = loss_func(model(xb), yb.long())

            loss.backward()
            with torch.no_grad():
                for p in model.parameters(): p -= p.grad * lr
                model.zero_grad()
```

```
fit()
loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
```

    (tensor(0.0097, grad_fn=<NllLossBackward>), tensor(1.))

Behind the scenes, PyTorch overrides the `__setattr__` function in `nn.Module` so that the submodules you define are properly registered as parameters of the model.

```
class DummyModule():
    def __init__(self, n_in, nh, n_out):
        self._modules = {}
        self.l1 = nn.Linear(n_in,nh)
        self.l2 = nn.Linear(nh,n_out)

    def __setattr__(self,k,v):
        if not k.startswith("_"): self._modules[k] = v
        super().__setattr__(k,v)

    def __repr__(self): return f'{self._modules}'

    def parameters(self):
        for l in self._modules.values():
            for p in l.parameters(): yield p
```

```
mdl = DummyModule(m,nh,10)
mdl
```

    {'l1': Linear(in_features=784, out_features=50, bias=True), 'l2': Linear(in_features=50, out_features=10, bias=True)}

```
[o.shape for o in mdl.parameters()]
```

    [torch.Size([50, 784]),
     torch.Size([50]),
     torch.Size([10, 50]),
     torch.Size([10])]

### Registering modules

We can use the original `layers` approach, but we have to register the modules.

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=2997)

```
layers = [nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10)]
```

```
class Model(nn.Module):
    def __init__(self, layers):
        super().__init__()
        self.layers = layers
        for i,l in enumerate(self.layers): self.add_module(f'layer_{i}', l)

    def __call__(self, x):
        for l in self.layers: x = l(x)
        return x
```

```
model = Model(layers)
```

```
model
```

    Model(
      (layer_0): Linear(in_features=784, out_features=50, bias=True)
      (layer_1): ReLU()
      (layer_2): Linear(in_features=50, out_features=10, bias=True)
    )

### nn.ModuleList

`nn.ModuleList` does this for us.

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3173)

```
class SequentialModel(nn.Module):
    def __init__(self, layers):
        super().__init__()
        self.layers = nn.ModuleList(layers)

    def __call__(self, x):
        for l in self.layers: x = l(x)
        return x
```

```
model = SequentialModel(layers)
```

```
model
```

    SequentialModel(
      (layers): ModuleList(
        (0): Linear(in_features=784, out_features=50, bias=True)
        (1): ReLU()
        (2): Linear(in_features=50, out_features=10, bias=True)
      )
    )

```
fit()
loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
```

    (tensor(0.0318, grad_fn=<NllLossBackward>), tensor(1.))

### nn.Sequential

`nn.Sequential` is a convenient class which does the same as the above:

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3199)

```
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
```

```
fit()
loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
```

    (tensor(0.0522, grad_fn=<NllLossBackward>), tensor(1.))

```
nn.Sequential??
```

```
model
```

    Sequential(
      (0): Linear(in_features=784, out_features=50, bias=True)
      (1): ReLU()
      (2): Linear(in_features=50, out_features=10, bias=True)
    )

### optim

Let's replace our previous manually coded optimization step:

```python
with torch.no_grad():
    for p in model.parameters(): p -= p.grad * lr
    model.zero_grad()
```

and instead use just:

```python
opt.step()
opt.zero_grad()
```

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3278)

```
class Optimizer():
    def __init__(self, params, lr=0.5): self.params,self.lr=list(params),lr

    def step(self):
        with torch.no_grad():
            for p in self.params: p -= p.grad * lr

    def zero_grad(self):
        for p in self.params: p.grad.data.zero_()
```

```
model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
```

```
opt = Optimizer(model.parameters())
```

```
for epoch in range(epochs):
    for i in range((n-1)//bs + 1):
        start_i = i*bs
        end_i = start_i+bs
        xb = x_train[start_i:end_i]
        yb = y_train[start_i:end_i]
        pred = model(xb)
        loss = loss_func(pred, yb.long())

        loss.backward()
        opt.step()
        opt.zero_grad()
```

```
loss,acc = loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
loss,acc
```

    (tensor(0.0091, grad_fn=<NllLossBackward>), tensor(1.))

PyTorch already provides this exact functionality in `optim.SGD` (it also handles stuff like momentum, which we'll look at later - except we'll be doing it in a more flexible way!)

```
#export
from torch import optim
```

```
optim.SGD.step??
```

```
def get_model():
    model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))
    return model, optim.SGD(model.parameters(), lr=lr)
```

```
model,opt = get_model()
loss_func(model(xb), yb.long())
```

    tensor(2.3490, grad_fn=<NllLossBackward>)

```
for epoch in range(epochs):
    for i in range((n-1)//bs + 1):
        start_i = i*bs
        end_i = start_i+bs
        xb = x_train[start_i:end_i]
        yb = y_train[start_i:end_i]
        pred = model(xb)
        loss = loss_func(pred, yb.long())

        loss.backward()
        opt.step()
        opt.zero_grad()
```

```
loss,acc = loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
loss,acc
```

    (tensor(0.0048, grad_fn=<NllLossBackward>), tensor(1.))

Randomized tests can be very useful.

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3442)

```
assert acc>0.7
```

## Dataset and DataLoader

### Dataset

It's clunky to iterate through minibatches of x and y values separately:

```python
    xb = x_train[start_i:end_i]
    yb = y_train[start_i:end_i]
```

Instead, let's do these two steps together, by introducing a `Dataset` class:

```python
    xb,yb = train_ds[i*bs : i*bs+bs]
```

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3578)

```
#export
class Dataset():
    def __init__(self, x, y): self.x,self.y = x,y
    def __len__(self): return len(self.x)
    def __getitem__(self, i): return self.x[i],self.y[i]
```

```
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)
assert len(train_ds)==len(x_train)
assert len(valid_ds)==len(x_valid)
```

```
xb,yb = train_ds[0:5]
assert xb.shape==(5,28*28)
assert yb.shape==(5,)
xb,yb
```

    (tensor([[0., 0., 0.,  ..., 0., 0., 0.],
             [0., 0., 0.,  ..., 0., 0., 0.],
             [0., 0., 0.,  ..., 0., 0., 0.],
             [0., 0., 0.,  ..., 0., 0., 0.],
             [0., 0., 0.,  ..., 0., 0., 0.]]), tensor([5., 0., 4., 1., 9.]))

```
model,opt = get_model()
```

```
for epoch in range(epochs):
    for i in range((n-1)//bs + 1):
        xb,yb = train_ds[i*bs : i*bs+bs]
        pred = model(xb)
        loss = loss_func(pred, yb.long())

        loss.backward()
        opt.step()
        opt.zero_grad()
```

```
loss,acc = loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
assert acc>0.7
loss,acc
```

    (tensor(0.0098, grad_fn=<NllLossBackward>), tensor(1.))

### DataLoader

Previously, our loop iterated over batches (xb, yb) like this:

```python
for i in range((n-1)//bs + 1):
    xb,yb = train_ds[i*bs : i*bs+bs]
    ...
```

Let's make our loop much cleaner, using a data loader:

```python
for xb,yb in train_dl:
    ...
```

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3674)

```
class DataLoader():
    def __init__(self, ds, bs): self.ds,self.bs = ds,bs
    def __iter__(self):
        for i in range(0, len(self.ds), self.bs): yield self.ds[i:i+self.bs]
```

```
train_dl = DataLoader(train_ds, bs)
valid_dl = DataLoader(valid_ds, bs)
```

```
xb,yb = next(iter(valid_dl))
assert xb.shape==(bs,28*28)
assert yb.shape==(bs,)
```

```
plt.imshow(xb[0].view(28,28))
yb[0]
```

```
model,opt = get_model()
```

```
def fit():
    for epoch in range(epochs):
        for xb,yb in train_dl:
            pred = model(xb)
            loss = loss_func(pred, yb.long())
            loss.backward()
            opt.step()
            opt.zero_grad()
```

```
fit()
```

```
loss,acc = loss_func(model(xb), yb.long()), accuracy(model(xb), yb)
assert acc>0.7
loss,acc
```

    (tensor(0.0277, grad_fn=<NllLossBackward>), tensor(1.))

### Random sampling

We want our training set to be in a random order, and that order should differ each iteration. But the validation set shouldn't be randomized.

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=3942)

```
class Sampler():
    def __init__(self, ds, bs, shuffle=False):
        self.n,self.bs,self.shuffle = len(ds),bs,shuffle

    def __iter__(self):
        self.idxs = torch.randperm(self.n) if self.shuffle else torch.arange(self.n)
        for i in range(0, self.n, self.bs): yield self.idxs[i:i+self.bs]
```

```
small_ds = Dataset(*train_ds[:10])
```

```
s = Sampler(small_ds,3,False)
[o for o in s]
```

    [tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])]

```
s = Sampler(small_ds,3,True)
[o for o in s]
```

    [tensor([2, 5, 1]), tensor([3, 0, 6]), tensor([7, 8, 4]), tensor([9])]

```
def collate(b):
    xs,ys = zip(*b)
    return torch.stack(xs),torch.stack(ys)

class DataLoader():
    def __init__(self, ds, sampler, collate_fn=collate):
        self.ds,self.sampler,self.collate_fn = ds,sampler,collate_fn

    def __iter__(self):
        for s in self.sampler: yield self.collate_fn([self.ds[i] for i in s])
```

```
train_samp = Sampler(train_ds, bs, shuffle=True)
valid_samp = Sampler(valid_ds, bs, shuffle=False)
```

```
train_dl = DataLoader(train_ds, sampler=train_samp, collate_fn=collate)
valid_dl = DataLoader(valid_ds, sampler=valid_samp, collate_fn=collate)
```

```
xb,yb = next(iter(valid_dl))
plt.imshow(xb[0].view(28,28))
yb[0]
```

```
xb,yb = next(iter(train_dl))
plt.imshow(xb[0].view(28,28))
yb[0]
```

```
xb,yb = next(iter(train_dl))
plt.imshow(xb[0].view(28,28))
yb[0]
```

```
model,opt = get_model()
fit()

loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)
assert acc>0.7
loss,acc
```

### PyTorch DataLoader

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=4171)

```
#export
from torch.utils.data import DataLoader, SequentialSampler, RandomSampler
```

```
train_dl = DataLoader(train_ds, bs, sampler=RandomSampler(train_ds), collate_fn=collate)
valid_dl = DataLoader(valid_ds, bs, sampler=SequentialSampler(valid_ds), collate_fn=collate)
```

```
model,opt = get_model()
fit()
loss_func(model(xb), yb), accuracy(model(xb), yb)
```

PyTorch's defaults work fine for most things however:

```
train_dl = DataLoader(train_ds, bs, shuffle=True, drop_last=True)
valid_dl = DataLoader(valid_ds, bs, shuffle=False)
```

```
model,opt = get_model()
fit()

loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)
assert acc>0.7
loss,acc
```

Note that PyTorch's `DataLoader`, if you pass `num_workers`, will use multiple threads to call your `Dataset`.

## Validation

You **always** should also have a [validation set](http://www.fast.ai/2017/11/13/validation-sets/), in order to identify if you are overfitting.

We will calculate and print the validation loss at the end of each epoch.

(Note that we always call `model.train()` before training, and `model.eval()` before inference, because these are used by layers such as `nn.BatchNorm2d` and `nn.Dropout` to ensure appropriate behaviour for these different phases.)

[Jump_to lesson 9 video](https://course.fast.ai/videos/?lesson=9&t=4260)

```
def fit(epochs, model, loss_func, opt, train_dl, valid_dl):
    for epoch in range(epochs):
        # Handle batchnorm / dropout
        model.train()
#         print(model.training)
        for xb,yb in train_dl:
            loss = loss_func(model(xb), yb.long())
            loss.backward()
            opt.step()
            opt.zero_grad()

        model.eval()
#         print(model.training)
        with torch.no_grad():
            tot_loss,tot_acc = 0.,0.
            for xb,yb in valid_dl:
                pred = model(xb)
                tot_loss += loss_func(pred, yb.long())
                tot_acc  += accuracy (pred,yb)
        nv = len(valid_dl)
        print(epoch, tot_loss/nv, tot_acc/nv)
    return tot_loss/nv, tot_acc/nv
```

*Question*: Are these validation results correct if batch size varies?

`get_dls` returns dataloaders for the training and validation sets:

```
#export
def get_dls(train_ds, valid_ds, bs, **kwargs):
    return (DataLoader(train_ds, batch_size=bs, shuffle=True, **kwargs),
            DataLoader(valid_ds, batch_size=bs*2, **kwargs))
```

Now, our whole process of obtaining the data loaders and fitting the model can be run in 3 lines of code:

```
train_dl,valid_dl = get_dls(train_ds, valid_ds, bs)
model,opt = get_model()
loss,acc = fit(5, model, loss_func, opt, train_dl, valid_dl)
```

```
assert acc>0.9
```

## Export

```
!python notebook2script.py 03_minibatch_training.ipynb
```

```

```


```python
from IPython.core.display import HTML
from IPython.display import Image
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# *Circuitos Elétricos I*

## Semana 1 - Convenções para aplicação das Leis de Kirchhoff na análise de circuitos

### Caso 1

```python
Image("./figures/J1C1.png", width=500)
```

#### Lei de Kirchhoff das tensões (LKT)

Em qualquer malha frechada do circuito $\sum_k v_k = 0$

`Convenção arbitrária (1): ao percorrer a malha, escolha um sinal (+ ou -) para indicar aumentos de tensão e o sinal oposto para indicar quedas de tensão no somatório da LKT.`

Logo, atribuindo o sinal (-) para aumentos de tensão e o sinal (+) para quedas de tensão, ao aplicar a LKT no circuito mostrado acima, temos:

$$
\begin{align}
-10 + v_1 + v_2 &= 0\\
-v_2 + v_3 + v_4 &= 0
\end{align}
$$

#### Lei de Kirchhoff das correntes (LKC)

Em qualquer nó do circuito $\sum_k i_k = 0$

`Convenção arbitrária (2): para o nó em questão, escolha um sinal (+ ou -) para indicar correntes chegando ao nó e o sinal oposto para indicar correntes deixando o nó no somatório da LKT.`

ou, para evitar erros com troca de sinais, simplesmente faça

`Somatório das correntes chegando ao nó igual ao somatório das correntes deixando o nó.`

$$
\begin{align}
i_1 &= i_2 + i_3\\
i_3 &= -0.5~A
\end{align}
$$

#### Lei de Ohm (+convenção passiva)

`Convenção passiva (3): qualquer expressão que relacione as grandezas de tensão e corrente num elemento ideal de dois terminais deve ser escrita de acordo com a convenção passiva.`

A convenção passiva estabelece que:

1. Se o sentido de referência adotado para corrente coincide com a queda de tensão na polaridade de referência ($+ \rightarrow -$), *qualquer expressão envolvendo $v$ e $i$* para o elemento em questão deve ser escrita com **sinal positivo**.

2. Se o sentido de referência adotado para corrente coincide com o aumento de tensão na polaridade de referência ($+ \leftarrow -$), *qualquer expressão envolvendo $v$ e $i$* para o elemento em questão deve ser escrita com **sinal negativo**.

A Lei de Ohm expressa a relação entre tensão, corrente e resistência num resistor ideal. Logo, as expressões da Lei de Ohm devem obedecer a convenção passiva.

Desse modo, podemos escrever as seguintes equações para o circuito acima.

$$
\begin{align}
v_1 &= 10i_1\\
v_2 &= 50i_2\\
v_3 &= 20i_3
\end{align}
$$

Logo:

$$
\begin{align}
-10 + 10i_1 + 50i_2 &= 0\\
-50i_2 -10 + v_4 &= 0\\
i_1 - i_2 &= -0.5
\end{align}
$$

Rearranjando as equações:

$$
\begin{align}
 10i_1 + 50i_2 &= 10\\
-50i_2 + v_4 &= 10\\
i_1 - i_2 &= -0.5
\end{align}
$$

### Solução das equações

```python
import sympy as sp
import numpy as np
```

```python
# define as N variáveis desconhecidas
i1, i2, v4 = sp.symbols('i1, i2, v4')

# define os sistema de N equações
eq1 = sp.Eq()
eq2 = sp.Eq()
eq3 = sp.Eq()

# resolve o sistema
soluc = sp.solve((eq1, eq2, eq3), dict=True)

i1 = np.array([sol[i1] for sol in soluc])
i2 = np.array([sol[i2] for sol in soluc])
v4 = np.array([sol[v4] for sol in soluc])
i3 = -0.5

print('Solução do sistema:\n\n i1 = %.2f A,\n i2 = %.2f A,\n i3 = %.2f A,\n v4 = %.2f V.' %(i1, i2, i3, v4))
```

#### Cálculo das potências

```python
# expressões para a Lei de Ohm (convenção passiva)
v1 =
v2 =
v3 =

# expressões para as potências (convenção passiva)
p10V =
p1   =
p2   =
p3   =
p4   =

print('Potências:\n\n p10V = %.2f W\n p1 = %.2f W,\n p2 = %.2f W,\n p3 = %.2f W,\n p4 = %.2f W\n' %(p10V, p1, p2, p3, p4))
```

```python
# calcula somatório das potências
print('Somatório das potências : %.2f W\n' %(p10V+p1+p2+p3+p4))
```

Simulação do circuito: https://tinyurl.com/yfbwd4vz

### Caso 2

```python
Image("./figures/J1C2.png", width=500)
```

```python
# define as N variáveis desconhecidas
i1, i2, v4 = sp.symbols('i1, i2, v4')

# define os sistema de N equações
eq1 = sp.Eq( )
eq2 = sp.Eq( )
eq3 = sp.Eq( )

# resolve o sistema
soluc = sp.solve((eq1, eq2, eq3), dict=True)

i1 = np.array([sol[i1] for sol in soluc])
i2 = np.array([sol[i2] for sol in soluc])
v4 = np.array([sol[v4] for sol in soluc])
i3 = 0.5

print('Solução do sistema:\n\n i1 = %.2f A,\n i2 = %.2f A,\n i3 = %.2f A,\n v4 = %.2f V.' %(i1, i2, i3, v4))
```

```python
# expressões para a Lei de Ohm (convenção passiva)
v1 =
v2 =
v3 =

# expressões para as potências (convenção passiva)
p10V =
p1   =
p2   =
p3   =
p4   =

print('Potências:\n\n p10V = %.2f W\n p1 = %.2f W,\n p2 = %.2f W,\n p3 = %.2f W,\n p4 = %.2f W\n' %(p10V, p1, p2, p3, p4))
```

### Caso 3

```python
Image("./figures/J1C3.png", width=500)
```

```python
# define as N variáveis desconhecidas
i1, i2, v4 = sp.symbols('i1, i2, v4')

# define os sistema de N equações
eq1 = sp.Eq( )
eq2 = sp.Eq( )
eq3 = sp.Eq( )

# resolve o sistema
soluc = sp.solve((eq1, eq2, eq3), dict=True)

i1 = np.array([sol[i1] for sol in soluc])
i2 = np.array([sol[i2] for sol in soluc])
v4 = np.array([sol[v4] for sol in soluc])
i3 = 0.5

print('Solução do sistema:\n\n i1 = %.2f A,\n i2 = %.2f A,\n i3 = %.2f A,\n v4 = %.2f V.' %(i1, i2, i3, v4))
```

```python
# expressões para a Lei de Ohm (convenção passiva)
v1 =
v2 =
v3 =

# expressões para as potências (convenção passiva)
p10V =
p1   =
p2   =
p3   =
p4   =

print('Potências:\n\n p10V = %.2f W\n p1 = %.2f W,\n p2 = %.2f W,\n p3 = %.2f W,\n p4 = %.2f W\n' %(p10V, p1, p2, p3, p4))
```

Jupyter notebooks/.ipynb_checkpoints/Circuitos Eletricos I - Semana 1-checkpoint.ipynb