The increasing integration of Large Language Models (LLMs) into various sectors has recently brought to light the pressing need to align these models with human preferences and implement safeguards against the generation of inappropriate content. This challenge stems from both ethical considerations and practical demands for responsible AI usage. Ethically, there is a growing recognition that the outputs of LLMs must align with laws, societal values, and norms. Practically, the widespread application of LLMs in sensitive contexts necessitates robust measures to prevent the misuse of technology, including content that may circumvent designer-imposed restrictions. 

Researchers highlight the ethical risks in the potential leakage of sensitive medical data inputted into LLMs and the bias in the generated contents. The generative capacity of the LLMs can be misused to disseminate misinformation and disinformation on a massive scale. Despite collective efforts to curtail the generation of undesirable content, innovative misconduct such as prompt injection and other forms of misuse remain prevalent. This scenario underscores the urgency of developing more resilient strategies to harness the transformative potential of LLMs while navigating the complex terrain of safety, ethics, and compliance.

Existing methods focus on actively guiding models to converge to human preference by fine-tuning the model using datasets aligned with human preference and adopting a reward-guided learning procedure. Nonetheless, it is difficult to evaluate the effectiveness of these methods when it comes to predicting the output of LLMs in complex real-world applications with multi-shot questions and constant iterations of model architectures and fine-tuning.