How to Test an AI Concierge Before Your Guests Use It
October 30, 2026 · 4 min read · Mateo Bolívar
To test an AI concierge before your guests use it, build a list of real hotel questions, define what counts as a good answer and review the results with the team that handles those questions today. The test is over when the answers are correct, sensitive cases go to a person and nobody on the team finds an outdated fact.
It takes a few hours and saves you from fixing mistakes in front of a guest. What follows is a simple method to organize that test.
Build a list of test cases
A test case is a specific question with the answer you expect. Testing with questions made up in the office does not work, because guests do not write that way. Ask the front desk for real messages from recent weeks, without personal data, and use them as the base.
To cover what matters, include these types of cases.
- Frequent questions with a stable answer, such as breakfast hours.
- Incomplete or misspelled questions, such as "wat time pool close".
- Questions in other languages. Contler's website says the agent answers in more than 20 languages, so test at least the ones your hotel receives most.
- Questions about something the hotel does not offer, to check that the agent does not invent it.
- Requests that depend on a reservation or on availability.
- Cases that should go to a person, such as a complaint or an emergency.
Thirty or forty cases give you a reasonable base for a first round. That is a suggested starting point, not a rule.
How to test an AI concierge with clear criteria
Before looking at the answers, decide how you will grade them. Otherwise each person on the team judges by their own standard and the review becomes a debate.
Four criteria are usually enough.
- Accuracy. The fact is correct and current.
- Completeness. The answer includes the condition or next step when needed.
- Limits. The agent does not promise something it cannot deliver or invent a service.
- Handoff. Sensitive cases or ones that need a decision reach the team.
Grade each case as correct, fixable or serious. A serious error is one that could hurt the guest or the hotel, such as giving the wrong check-out time or answering an emergency without alerting anyone.
Who should run the test
The test should be run by the people who know the answers. The front desk reviews general information, and each department head reviews their own area, for example restaurant, spa or maintenance.
It helps if someone other than the person who prepared the information asks the questions. Whoever wrote the content tends to ask exactly what they wrote.
What to do with each error
In a first test, many errors come from the information rather than the agent. Before concluding that something does not work, check whether the fact was there, whether it was current and whether it had its full condition.
Log each error with three things, what was asked, what the agent answered and what it should have answered. Fix the content and rerun only the cases that failed, plus a few that already worked to confirm nothing broke.
Illustrative example. A hotel tests the question "can I bring my dog?" and the agent says yes, but does not mention the extra fee or the weight limit. The error is not the agent's, it is the policy that was loaded without those conditions. The text is fixed and tested again.
When it is ready for guests
Define the exit criteria before you start. A reasonable approach is to require zero serious errors, every sensitive case going to a person and sign-off from the front desk and each department head on their answers.
Opening the agent does not end the review. During the first weeks it pays to read real conversations often, because guests always ask something nobody anticipated.
If you worry that the test will stay a demo that never reaches operations, I wrote about that risk in this article on pilots. This method is the practical part, the list you use on testing day.
Frequently asked questions
How many test cases do I need?
It depends on the size of the hotel and how many services it has. Thirty or forty well-chosen cases make a good first round. What matters is covering the types of question, not hitting a number.
Can I test the agent without guests seeing it?
That is the point of this stage. Confirm with Contler how to run the internal test in your case before sharing the channel with guests.
What if the agent answers with something the hotel does not offer?
Treat it as a serious error. Check whether the loaded information leads to that answer and make clear which services do not exist.
How long should the test last?
There is no fixed timeline. It ends when you meet the exit criteria you defined. For a small hotel it can be a matter of days.
Should I repeat the test after launch?
Yes, every time important information changes, such as hours, services or policies. Keep the list of cases so you can reuse it.
Who approves the launch?
Whoever is responsible for guest service, with sign-off from the department heads whose answers the agent handles.
What you can do this week
Ask the front desk for twenty real guest messages and turn them into test cases with their expected answer. If you do not have an agent to test them on yet, you can create your AI Concierge and start with that list.
