Large language models (LLMs) produce certain output based on the input. How do we determine if the the output is correct or of good quality ? One way to do this is by employing humans who actually read the output and rate it.

This of course is not scalable. Humans are expensive resource and also very slow. To solve this problem. autoraters have been invented. Think of them as AI judges for AI-generated content. Instead of relying solely on human reviewers (which can be slow, expensive, and subjective at scale), autoraters provide a more efficient and consistent way to assess how well an LLM is performing.

Vertex AI AutoSxS: Google Cloud's tool that uses an autorater to compare the quality of responses from different models side-by-side.

In essence, autoraters are becoming increasingly vital tools in the LLM ecosystem, enabling more efficient, scalable, and consistent evaluation of these powerful AI models. They play a key role in driving progress and ensuring the quality of LLM-generated content.