Evaluating the Effectiveness of Reward Modeling of Generative AI Systems

This post summarizes SEAL, a study evaluating reward modeling in RLHF, introducing metrics (feature imprint, alignment resistance, alignment robustness) and reporting experiments with Anthropic preference data and OpenAssistant reward models that show strong target-feature imprints, sensitivity to spoiler features, ~26% alignment resistance where labelers disagreed with humans, and misalignment often due to ambiguous data—highlighting the need to scrutinize both reward models and alignment datasets.