NaN error when running PAIRED and REPAIRED on BipedalWalker env

[wenjunli-0]

When running PAIRED and REPAIRED on BipedalWalker env. After a few thousands of updates, there will be an “nan” error in self.actor_critic.evaluate_actions() method (check this line

dcd/algos/ppo.py

Line 91 in 0411f1f

values, action_log_probs, dist_entropy, _ = self.actor_critic.evaluate_actions(

). The error happens when the Generator is being updated using ppo.

I assume this is caused by some gradient explosion or something. Could you guys check this bug?