Not so much, a timer ticking up is only novel the first time round, and is unrelated to actions taken by the agent. Over multiple plays the agent will learn to ignore it.

EDIT: It could be that the agent will just stand there the first few plays around, enjoying the novelty reward gained from simply watching the timer tick up. Haha

The point is that every time the timer ticks, if you had defined 'novelty' as the bitstring representing the screen, you get a 'new' state. This multiplies against any blinking animations, any moving enemies, any of the agent's moves, any visible scores, etc. You get thousands or millions of unique framebuffer states before the agent has so much as left the first room in _Montezuma's Revenge_. And DQN already is RAM intensive for the experience replay buffer.

Thanks for the extra explanation. It seems I assumed too much about these Deep Q networks, due to some prior knowledge of the neuroscience related to RL. Although I do remember having seen a video about Montezuma's Revenge a week ago or so, where they talked about this exact problem.

Anyway, it would seem to me that novelty functions that would allow the agent to ignore periodic changes in state such as timers going up, can be quite simple. A function that estimates novelty of individual bit values in the bitstring of the gamestate and then aggregates it, could quite easily account for timers, or generally elements changing periodically regardless of agent actions. A baseline novelty reward would seem relatively easy to predict by the agent and thus result in low prediction errors and low reinforcement of actions by agents. This function would have a linear space and time complexity to the length of the gamestate, and fairly naive & simple to use, but would get the job done I think?

P.S Just wanted to thank you for the work you've put into your website, it's very informative and always a great starting point to dive deeper into topics you cover!

You have to come up with something or else the agent will never be able to explore worth a damn in complex domains. Imagine trying to learn to write Haskell programs by typing random gibberish...

'gamestate' is illegal. It's pointless to suppose an agent which has access to the true groundtruth RAM of the Atari games, because that generalizes to vanishingly few other domains. The goal is to create a general agent which can be used elsewhere, such as in recommender systems. (And if you did have access to the raw RAM, that would reduce the problem from an extremely challenging POMDP or harder, to a fully-observed deterministic MDP, because you could then construct a game-tree of each individual RAM state and the possible actions taken in it; in which case, you would use a much faster and more powerful MDP solver like MCTS rather than DQN.)

One can come up with hand-crafted heuristics which might improve over the naive bitstring equality approach, but your suggestion still doesn't do the trick, assuming you could figure out how to meaningfully define 'periodic changes' and teach the NN to ignore them. Imagine a game in which the overall screen lighting varies (perhaps it's set at night or during rain, or perhaps each level has different color themes). As all the bits keep flipping with changes in lighting/intensity, you'd be in about the same place.