With several chatbots available online, it can become extremely difficult to select the one that meets your needs. Though you can compare any two chatbots manually, it'll take considerable time and effort.

A better and simpler way is to use Chatbot Arena to compare the different LLMs that power popular chatbots. It offers a couple of modes for comparing the various models, which we explain below.

What Is Chatbot Arena?

Created by LMSYS Org, Chatbot Arena is a platform to benchmark various LLMs. It uses the Elo Rating system to rank the various models.

Chatbot Arena offers a couple of ways for users to compare and rate LLMs. Based on the submitted feedback, Chatbot Arena ranks the different LLMs on the public leaderboard. The project is sponsored by HuggingFace, an open-source alternative to ChatGPT.

How to Compare Anonymous LLMs with Chatbot Arena

Chatbot Arena's battle mode lets you compare LLMs anonymously. For instance, you can compare ChatGPT (GPT 3.5) and Claude. This means that Chatbot Arena itself selects any two language models and, without revealing their names, lets you compare them.

As you enter the first prompt, Chatbot Arena fetches responses from both models, presenting them side by side. The platform allows you to regenerate responses (for both LLMs) and clear history to start a different conversation. You can keep asking more questions until you've selected a clear winner.

Then, you can choose if model A is better or B. On selecting the winner, Chatbot Arena reveals the names of both bots. This mode works great as your decision is not affected by your previous perception or popularity of the models. Chatbot Arena also lets you adjust parameters like temperature, Top P, and max output tokens.

How to Compare Selected LLMs with Chatbot Arena

If you want to compare any two specific LLMs, you can switch to Chatbot Arena's side-by-side mode. Other than the fact that you can pick the LLMs yourself, this mode works almost the same as battle mode. You can adjust parameters, regenerate responses, clear history, and select a winner in the end.

However, the number of LLMs available in this mode is limited. You can select different versions of Llama 2, Vicuna, and ChatGLM. Though the popular LLMs, like GPT-4, GPT-3.5, Claude 1, Claude 2, etc., are currently unavailable in this mode, Chatbot Arena does plan to add them.

Compare LLMs Using Chatbot Arena

Whether you're looking to find a suitable chatbot for your needs or just want to test different LLMs, Chatbot Arena is a great platform.

It provides a simplified way of comparing different language models side-by-side. And since it maintains a leaderboard based on users' feedback, you can directly view the rankings of various models without running the tests yourself.