Running a fair side-by-side test

When you finish, you can run the same prompt in two or more tools and compare results fairly.

Same prompt, same conditions

A fair test changes only one thing: the tool. Use exactly the same prompt, copied and pasted, in each. Start each in a fresh chat so earlier messages do not tilt the result. Use the same files or text if the task needs them.

If possible, run each prompt more than once, since answers vary. One lucky or unlucky answer can mislead you. Two or three runs give a better picture of how the tool usually behaves.

Choose real tasks

Pick tasks from your own life, ranging from easy to hard. Include a writing task, an explanation task, a task using your own pasted text, and one with a built-in trap, such as a question that has a false detail. How the tool handles a trap tells you whether it notices problems or just goes along.

Use invented or non-sensitive details when you test. You do not need a real customer’s record to see how a tool writes a reminder. Make up a name and a situation.

Judge with a simple sheet

Score each answer on a few things you care about. Accuracy: are the facts right, as far as you can check? Usefulness: could I use this with little editing? Clarity and tone: does it sound right for my reader? Honesty: does it admit uncertainty, or bluff?

Give a quick score from one to five for each, or just mark better, same, or worse. Write a one-line note. Do not let length fool you. A longer answer is not automatically better; it may only be padded.

Weak versus strong testing

Weak testing: ask each tool a different question, on different days, and decide by gut feeling. Strong testing: “Summarize the following paragraph in three bullet points for a busy manager,” pasted identically into each tool, scored on the same sheet.

Blind scoring helps too. Copy each answer into a document labeled A and B without the tool names, then score them. This reduces the pull of brand loyalty. You may be surprised which answer you prefer once you cannot see where it came from.

Practice now

Pick one real task and run the identical prompt in two different AI tools, each in a fresh chat. Copy both answers into a document labeled A and B and score them without looking at which is which.
Add a trap question, such as a question with a false detail, and see which tool notices. Record whether each flags the mistake or goes along with it.

Use any AI assistant you have. Do not paste private, client or patient information into it.

Check yourself

1. What should stay the same in a fair side-by-side test?

2. Why use a trap question?

3. Why score answers blind?

Ask the AI tutor

Ask anything about this lesson, in any language. The tutor is an AI: it can be wrong, and it never needs your personal details.

Next lesson: Matching tools to tasks

AI-drafted. Checked by an automated validator and a separate AI reviewer; a person has not read every lesson. Translations and tutor answers are AI-made. Found a mistake? Tell us. AI can be wrong. General information only, not professional advice. Not affiliated with any AI company named. Current University is not accredited.