Entropy-TRPO Model Weights

PyTorch checkpoints for the comparative study in A Review of Entropy-Based Extensions to Trust Region Policy Optimization.

Repository layout

Each checkpoint directory contains:

File	Description
`policy.pt`	Policy network state dict
`value.pt`	Value network state dict
`config.json`	Training hyperparameters
`metadata.json`	Paper source, variant flags, final metrics

Hub path: {env_id}/{variant}/latest/ (e.g. CartPole-v1/entrpo/latest/).

The repo README is updated automatically during training with a Training progress table (epoch n/N, eval return, best return, KL) from results/summary.md, plus a JSON index of available checkpoints.

Notation

$\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$, GAE advantages $\hat{A}_t$, trust-region radius $\delta$
$\alpha$ — Roostaie advantage entropy; Xu ERO objective entropy (distinct roles, same symbol in each paper's row)
$\beta$ — Xu ERC constraint coefficient (Xu Eq. 49)
$c_{\mathrm{ent}}$ — PPO entropy bonus (Schulman et al., 2017; config field entropy_coef)

Variant definitions

Key	Paper name	Surrogate / constraint
`trpo`	TRPO	$\mathbb{E}[\rho_t \hat{A}t]$; $\bar D{\mathrm{KL}} \le \delta$
`entrpo_entropy`	EnTRPO-Entropy	$\mathbb{E}[\rho_t \tilde{A}t]$, $\tilde{A}t=\hat{A}t+\alpha,\mathcal{H}(\pi{\theta{\text{old}}}(\cdot\|s_t))$ (fixed during step); $\bar D{\mathrm{KL}} \le \delta$
`ero_trpo`	ERO-TRPO	$\mathbb{E}[\rho_t \hat{A}t]+\alpha,\mathbb{E}[\mathcal{H}(\pi_\theta)]$; $\bar D{\mathrm{KL}} \le \delta$
`erc_trpo`	ERC-TRPO	$\mathbb{E}[\rho_t \hat{A}t]$; $\bar D{\mathrm{KL}} \le \delta+\beta,\mathbb{E}[\mathcal{H}(\pi_\theta)]$ (Xu Eq. 49)
`entrpo_buffer`	EnTRPO-Buffer	$\mathbb{E}[\rho_t \hat{A}_t]$ with Roostaie on-policy replay
`entrpo`	EnTRPO	$\mathbb{E}[\rho_t \tilde{A}_t]$ + Roostaie buffer
`ppo`	PPO	$\mathbb{E}[\min(\rho_t \hat{A}_t,\mathrm{clip}(\rho_t)\hat{A}t)]+c{\mathrm{ent}}\mathbb{E}[\mathcal{H}(\pi_\theta)]$

$\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/PPO rows at the candidate policy $\pi_\theta$.

ERO-TRPO implementation: Xu Table 1 — $H\mathbf{d}=\mathbf{g}+\alpha\mathbf{h}$, step size $\eta\le\sqrt{2\delta/(\mathbf{g}+\alpha\mathbf{h})^\top H(\mathbf{g}+\alpha\mathbf{h})}$, Xu line search ($\eta_0=0.5$, strict objective improvement, hard $\bar{D}_{\mathrm{KL}}\le\delta$).

ERC-TRPO implementation: Xu Table 1 — two CG solves ($\mathbf{u},\mathbf{v}$), $\eta\mathbf{u}+\beta\mathbf{v}$, Eq.~(49) acceptance $\bar{D}_{\mathrm{KL}}\le\delta+\beta,\mathbb{E}[\mathcal{H}]$, same Xu line search. Random ERC shares the ERC step; only the acceptance bonus is randomized.

Older Hub folders (trpo_entropy, trpo_buffer, …) remain valid; training resumes from them automatically.

Environments

Environment	Obs / action	Training budget	Hyperparameter source
CartPole-v1	Gymnasium classic control	1M steps	Roostaie + Xu Tables 4--5
Humanoid-v5	348 / 17	$10^6$ steps	PPO/baselines backbone; Xu ERO/ERC proxied from Walker2d
HumanoidStandup-v5	348 / 17	$10^6$ steps	Same backbone; Xu ERO/ERC proxied from BipedalWalker

See HYPERPARAMETERS.md for per-field provenance and paper/results/annex_hyperparameters.tex for tables.

Variants and paper sources

Variant	Paper
`trpo`	Schulman et al. (2015), Trust Region Policy Optimization, ICML
`entrpo_entropy`	Roostaie & Ebadzadeh (2021), EnTRPO — entropy-in-advantage ablation
`entrpo_buffer`	Roostaie & Ebadzadeh (2021), EnTRPO — replay-buffer ablation
`entrpo`	Roostaie & Ebadzadeh (2021), EnTRPO — full method
`ero_trpo`	Xu et al. (2024), ERO-TRPO
`erc_trpo`	Xu et al. (2024), ERC-TRPO
`ppo`	Schulman et al. (2017), Proximal Policy Optimization

See metadata.json in each folder for full author names and URLs.

Usage

Training and evaluation code: GitHub — entropy-trpo (update URL when published).

git clone https://github.com/pre63/entropy-trpo.git
cd entropy-trpo
make setup          # install deps + create .env
# edit .env with HF_TOKEN and HF_REPO_ID
make download-weights
make eval-checkpoints

Citation

@article{entropytrporeview2026,
  title   = {A Review of Entropy-Based Extensions to Trust Region Policy Optimization},
  author  = {Green, Simon},
  journal = {IEEE Transactions},
  year    = {2026}
}

@article{roostaie2021entrpo,
  title   = {EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization},
  author  = {Roostaie, Sahar and Ebadzadeh, Mohammad Mehdi},
  journal = {arXiv:2110.13373},
  year    = {2021}
}

@article{xu2024trpo,
  title   = {Trust region policy optimization via entropy regularization for {Kullback--Leibler} divergence constraint},
  author  = {Xu, Haotian and Xuan, Junyu and Zhang, Guangquan and Lu, Jie},
  journal = {Neurocomputing},
  volume  = {589},
  pages   = {127716},
  year    = {2024}
}

Training progress

Last updated: 2026-07-05 19:43:57 UTC

Xu window — mean eval return over the last W epochs (Xu Table 6 protocol).
Full run — mean eval return over all epochs.
Device: cpu
Config: configs/e4/e4_entrpo_substitute_cartpole_v5.yaml
Jobs complete: 72/72
Running: 0

CartPole-v1 (1M benchmark)

Variant	Status	Epoch	Timesteps	Eval (final)	Best	Xu window	Full run	W	KL
TRPO (s0)	done	1953/1953	999,936	325.5 ± 62.6	500.0	284.9	382.5	326	0.0032
TRPO (s1)	done	1953/1953	999,936	56.1 ± 62.4	500.0	438.6	399.8	326	0.0032
TRPO (s2)	done	1953/1953	999,936	290.1 ± 184.0	500.0	459.8	378.0	326	0.0021
EnTRPO-Entropy (s0)	done	200/200	1,000,000	497.9 ± 6.3	500.0	365.4	372.3	33	0.0035
EnTRPO-Entropy (s1)	done	200/200	1,000,000	420.3 ± 37.0	500.0	448.7	434.1	33	0.0059
EnTRPO-Entropy (s2)	done	200/200	1,000,000	494.9 ± 12.1	500.0	425.4	409.5	33	0.0050
ERO-TRPO (s0)	done	2000/2000	1,000,000	182.8 ± 66.6	232.8	168.1	86.8	333	0.0000
ERO-TRPO (s1)	done	2000/2000	1,000,000	153.6 ± 68.2	280.2	198.5	103.7	333	0.0000
ERO-TRPO (s2)	done	2000/2000	1,000,000	175.8 ± 110.2	254.6	157.7	84.4	333	-0.0000
ERC-TRPO (s0)	done	2000/2000	1,000,000	229.2 ± 81.0	258.3	186.2	97.4	333	-0.0000
ERC-TRPO (s1)	done	2000/2000	1,000,000	22.0 ± 9.3	43.7	25.6	25.7	333	-0.0000
ERC-TRPO (s2)	done	2000/2000	1,000,000	20.3 ± 5.2	41.7	21.7	21.8	333	0.0000
EnTRPO-Buffer (s0)	done	200/200	1,000,000	170.3 ± 41.3	199.7	153.4	157.5	33	0.0091
EnTRPO-Buffer (s1)	done	200/200	1,000,000	309.2 ± 62.2	500.0	330.6	320.8	33	0.0064
EnTRPO-Buffer (s2)	done	200/200	1,000,000	205.0 ± 8.4	485.7	224.4	251.9	33	0.0070
EnTRPO (s0)	done	200/200	1,000,000	101.7 ± 6.1	489.2	106.1	129.7	33	0.0089
EnTRPO (s1)	done	200/200	1,000,000	99.0 ± 22.0	500.0	123.0	213.5	33	0.0076
EnTRPO (s2)	done	200/200	1,000,000	168.9 ± 9.4	500.0	399.5	287.1	33	0.0067
PPO (s0)	done	31250/31250	1,000,000	218.9 ± 84.4	340.9	207.5	150.0	1000	4.6836
PPO (s1)	done	31250/31250	1,000,000	500.0 ± 0.0	500.0	499.5	265.0	1000	0.0824
PPO (s2)	done	31250/31250	1,000,000	134.3 ± 51.4	500.0	299.0	275.5	1000	-0.0170

Humanoid-v5 (1M benchmark)

Variant	Status	Epoch	Timesteps	Eval (final)	Best	Xu window	Full run	W	KL
TRPO (s0)	done	976/976	1,000,448	310.7 ± 62.0	385.1	303.1	253.8	163	0.0060
TRPO (s1)	done	976/976	999,424	321.3 ± 88.9	379.0	309.6	257.8	163	0.0043
TRPO (s2)	done	976/976	999,424	309.2 ± 105.4	365.1	304.2	251.8	163	-0.0009
EnTRPO-Entropy (s0)	done	488/488	999,424	239.5 ± 64.2	332.1	271.8	194.6	81	-0.0000
EnTRPO-Entropy (s1)	done	488/488	999,424	276.8 ± 63.1	333.5	268.0	183.7	81	0.0042
EnTRPO-Entropy (s2)	done	488/488	999,424	249.7 ± 82.2	328.3	268.7	173.1	81	0.0000
ERO-TRPO (s0)	done	976/976	999,424	280.8 ± 101.4	387.9	310.7	262.6	163	-0.0012
ERO-TRPO (s1)	done	976/976	999,424	330.0 ± 83.0	369.2	304.4	253.4	163	0.0046
ERO-TRPO (s2)	done	976/976	999,424	316.5 ± 60.8	420.3	309.0	245.6	163	0.0092
ERC-TRPO (s0)	done	976/976	999,424	277.9 ± 89.4	383.3	302.5	257.0	163	0.8317
ERC-TRPO (s1)	done	976/976	999,424	281.9 ± 85.2	399.6	312.0	258.6	163	0.0051
ERC-TRPO (s2)	done	976/976	999,424	254.2 ± 46.3	392.6	311.4	257.9	163	-0.0025
EnTRPO-Buffer (s0)	done	488/488	999,424	233.2 ± 46.0	314.2	262.7	181.5	81	0.0000
EnTRPO-Buffer (s1)	done	488/488	999,424	293.6 ± 75.7	308.4	257.5	178.3	81	0.0000
EnTRPO-Buffer (s2)	done	488/488	999,424	263.0 ± 78.6	310.7	250.2	161.5	81	0.0082
EnTRPO (s0)	done	488/488	999,424	276.8 ± 52.6	352.6	256.4	186.2	81	0.0000
EnTRPO (s1)	done	488/488	999,424	227.8 ± 31.5	334.7	256.3	174.8	81	0.0089
EnTRPO (s2)	done	488/488	999,424	286.4 ± 68.8	323.3	255.7	163.5	81	0.0093
PPO (s0)	done	1953/1953	999,936	286.4 ± 70.3	411.1	305.6	291.3	326	0.0171
PPO (s1)	done	1953/1953	999,936	324.4 ± 72.3	403.0	309.6	288.9	326	-0.0023
PPO (s2)	done	1953/1953	999,936	291.7 ± 65.1	409.1	307.0	290.9	326	-0.0067

HumanoidStandup-v5 (1M benchmark)

Variant	Status	Epoch	Timesteps	Eval (final)	Best	Xu window	Full run	W	KL
TRPO (s0)	done	976/976	999,424	65564.3 ± 13149.1	75225.9	65080.2	55528.2	163	0.0086
TRPO (s1)	done	976/976	999,424	85331.8 ± 15316.0	93973.1	83992.1	69408.5	163	0.0013
TRPO (s2)	done	976/976	999,424	71929.0 ± 11498.5	81636.4	73421.4	61150.4	163	0.0018
EnTRPO-Entropy (s0)	done	488/488	999,424	64478.0 ± 14627.4	73943.7	61896.3	50554.0	81	0.0022
EnTRPO-Entropy (s1)	done	488/488	999,424	79008.0 ± 7394.7	83421.9	75278.5	59057.8	81	0.0088
EnTRPO-Entropy (s2)	done	488/488	999,424	74557.1 ± 10527.1	75892.7	66491.7	53834.4	81	0.0032
ERO-TRPO (s0)	done	976/976	999,424	45868.3 ± 2401.2	48375.1	45105.9	40929.5	163	0.0005
ERO-TRPO (s1)	done	976/976	999,424	52085.0 ± 6858.4	54183.2	50235.2	45674.6	163	-0.0008
ERO-TRPO (s2)	done	976/976	999,424	54828.2 ± 5440.4	57270.1	53365.4	45660.3	163	0.0006
ERC-TRPO (s0)	done	976/976	999,424	44648.7 ± 1975.6	47619.2	43774.6	40086.5	163	-0.0007
ERC-TRPO (s1)	done	976/976	999,424	53205.4 ± 4787.5	56941.8	52720.3	46702.6	163	-0.0042
ERC-TRPO (s2)	done	976/976	999,424	51636.4 ± 5581.1	56086.9	51601.2	44689.4	163	-0.0001
EnTRPO-Buffer (s0)	done	488/488	999,424	36194.0 ± 1992.8	40890.8	36737.9	37064.1	81	0.0096
EnTRPO-Buffer (s1)	done	488/488	999,424	42289.7 ± 2855.5	46657.7	40983.5	41893.8	81	0.0078
EnTRPO-Buffer (s2)	done	488/488	999,424	37238.8 ± 1760.8	40826.2	37388.7	37541.4	81	0.0058
EnTRPO (s0)	done	488/488	999,424	37475.7 ± 2124.3	40184.2	38075.0	37742.2	81	0.0000
EnTRPO (s1)	done	488/488	999,424	42126.8 ± 1923.1	46326.0	41585.1	41616.1	81	0.0000
EnTRPO (s2)	done	488/488	999,424	36979.4 ± 2180.2	41487.5	38496.5	38440.7	81	0.0000
PPO (s0)	done	1953/1953	999,936	59353.4 ± 8311.7	86651.2	64306.1	67688.9	326	0.1466
PPO (s1)	done	1953/1953	999,936	71828.6 ± 12149.2	86146.6	70013.6	69247.9	326	0.2344
PPO (s2)	done	1953/1953	999,936	73034.0 ± 5080.3	83874.9	72769.7	69345.9	326	-0.0429

CartPole-v5 (E4_CARTPOLE_V5 benchmark)

Variant	Status	Epoch	Timesteps	Eval (final)	Best	Xu window	Full run	W	KL
EnTRPO-Entropy (s0)	done	10/10	50,000	255.7 ± 71.7	263.5	254.9	152.9	2	0.0030
EnTRPO-Entropy (s1)	done	10/10	50,000	326.2 ± 98.8	337.1	305.4	188.7	2	0.0071
EnTRPO-Entropy (s2)	done	10/10	50,000	363.0 ± 105.1	393.6	378.3	181.3	2	0.0062
Binomial EnTRPO (s0)	done	10/10	50,000	323.2 ± 101.5	323.2	288.6	159.7	2	0.0093
Binomial EnTRPO (s1)	done	10/10	50,000	326.2 ± 98.8	337.1	305.4	188.7	2	0.0071
Binomial EnTRPO (s2)	done	10/10	50,000	392.6 ± 101.7	393.6	393.1	184.2	2	0.0061
Constant EnTRPO (s0)	done	9/10	50,000	382.3 ± 74.8	385.0	383.6	209.4	2	0.0054
Constant EnTRPO (s1)	done	10/10	50,000	268.7 ± 103.6	309.0	266.8	175.5	2	0.0054
Constant EnTRPO (s2)	done	10/10	50,000	392.6 ± 101.7	393.6	393.1	184.2	2	0.0061

Available checkpoints

{
  "CartPole-v1": [
    "entrpo",
    "entrpo_buffer",
    "entrpo_entropy",
    "ppo",
    "trpo",
    "trpo_buffer",
    "trpo_entropy"
  ]
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Video Preview

Reinforcement Learning

Paper for pre63/entropy-trpo-weights

EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization

Paper • 2110.13373 • Published Oct 26, 2021