so just spitballing: traditional RL treats the rewards as fixed and tries to model and optimize policy over environment to maximize. but in a lot of real world human scenarios, the "reward" isn't very strictly specified, rather it's another complex agent (person) evaluating your output, and often that person can be queried! you don't just dump a whole report to your boss on friday after a 1 sentence request, you know you have uncertainty about what they actually want and what their rubric is, and so tend to ask some high value clarification questions or get feedback on intermediate results. a human would never think "oh if i just reward hack this benchmark's execution environment to get all 100%s my boss will be happy with me", because they model the boss as a complex and capable agent rather than a strict reward spec.
i'm sure there's already been a deep body of research around this and or it collapses into being equivalent to existing paradigms, but what if we had explicitly modeled uncertainty around the reward and capability to query the evaluator agent as part of the training process? obviously current models will do some of this, but seemingly more as "they've been prompted and fine tuned a little to ask clarifying questions", and less "they're explicitly attempting to model each individual user's desires separately as part of execution rollout"
i guess this could be modeled as effectively moving the reward to live inside the environment and keeping the raw RL level reward as "does this chunk of the environment evaluate to positive when passed this other chunk"? i suppose the premise is effectively "theory of mind"
anyone know if stuff like this has been tried and whether it has any meaningful impact or is just adding pointless complexity?
it looks like this is very similar to the very widely cited "Cooperative Inverse Reinforcement Learning" from 2016.... but i notice i haven't heard anything about it in recent years! i wonder why
it looks like AssistanceZero might be the most modern extension of this work? https://arxiv.org/abs/2504.07091