A General Language Assistant as a Laboratory for Alignment

This abstract summarizes research into aligning large language models by evaluating baseline techniques like prompting, comparing training objectives (imitation learning, binary discrimination, ranked preference modeling), finding that ranked preference modeling outperforms and scales more favorably than imitation learning or binary discrimination, and exploring a 'preference model pre-training' stage to improve sample efficiency when finetuning on human preferences.