For starters, they’re not about uptime or other traditional SLA metrics. You’ve implemented a Conversational AI to satisfy customers, and that is what should be guaranteed. You want to measure the output of the AI and how well it performed at its intended task. In the virtual assistant world, this translates to how often it understands and satisfies your customers correctly. As AI continues to advance, these core metrics will continue to impress.
For NLU accuracy, the amount of time your system will correctly understand the customer, you should expect at least 80%. Now, don’t get caught by the sneaky tactics of some vendors who will only offer a performance number for specific “use cases” — this number should apply to absolutely every customer interaction with your brand about anything at all. The next metric is a percentage of resolved customer issues before a transfer to a live agent is required. This is case dependent but should fall between 25 and 35%. Again, don’t get stuck on specific “use cases”, your customers will come with all of their questions and your virtual assistant should perform well across them all.
As cliche as it may sound, AI is a journey and not a destination. It needs to continuously learn to improve and get better. One analogy we like to use is comparing an AI to a child. Children need a lot of attention, as well as great teachers to help them reach their full potential. If left to themselves, they’ll devolve into something like Lord of the Flies. The key takeaway here is that they do not learn on their own, but rely on great teachers to help them along the way.
Many will argue that self-training AI is on the horizon, but having worked in this space for the last 5 years, we are still many years away from truly automated self-learning in the enterprise market. Black boxes (automated training or deep learning) scare people in this industry, they need 100% control over every message that a customer will receive. So, we’re going to be using supervised learning with people behind the AI training for the foreseeable future.
Any competent vendor can source and install technology. But teaching is something new to the tech industry. It requires a new type of team, a new set of analytics, and a solid set of training tools. Just like any teaching environment, someone needs to set the curriculum and expectations, and continuously teach, monitor, and optimize to constantly improve.
We take pride in being great teachers of the AIs under our care, and have equipped our teams with the best tools, technologies, and AI data library to confidently remain dedicated to our goal of continuous improvement. This assures we can guarantee performance levels for our clients, and provide the white glove service our customers expect.