MindMeld, a conversational AI company, has published The Conversational AI Playbook, a guide outlining the challenges and the steps to be made to create conversational applications.
Lately, a number of major players have come with incipient solutions for building conversational applications. Researchers, science fiction and movies have predicted for a long time that the day will come when people will be able to talk to a machine similarly to another human being. Amazon Alexa, Apple Siri, Google Assistant, Microsoft Cortana or other solutions attempt to serve that need, offering applications and frameworks enabling developers to create apps that are meant to understand human speech and provide answers to basic questions.
MindMeld provides a platform for creating conversational applications based on deep-domain conversational artificial intelligence. They have written a step-by-step playbook with guidance to those interested in writing such applications. The guide includes several principles and best practices they have developed over the years working in this area.
MindMeld outlines a number of possible approaches for building conversational applications:
Rules Engines. With this approach a developer needs to create a large set of rules used to select the appropriate response for a question. It is the quickest solution but the most limited:
Since rule-based frameworks are not intended to provide AI capabilities to parse or classify incoming messages, the developer must code all of the necessary message processing and interaction logic by hand. It is not uncommon for even simple applications to require hundreds of rules to handle the different dialogue states in a typical conversational interface.
The guide lists BotKit and Microsoft Bot Framework as some of the rule-based engines for conversational applications.
NLP Services. Another approach is using cloud-based Natural Language Processing (NLP) services. While providing services for interpreting natural languages, a step necessary in building conversational apps, MindMeld thinks that such services serve an additional purpose:
These services are generally provided by large consumer internet companies to entice developers to upload their training data and thereby help the service provider improve their own conversational AI offerings in the process.
Offerings in this area includes Amazon Lex, Facebook wit.ai, Google api.ai, IBM Watson Conversation, Microsoft LUIS, and Samsung Viv.
Machine Learning. A more sophisticated solution is represented by toolkits relying on machine learning. They provide access to advanced deep-learning algorithms used to interpret data. The guide mentions Apple GraphLab Create, Google TensorFlow, and Microsoft Cognitive Toolkit as possible solutions for creating conversational applications.
Conversational AI Platforms. These are platforms based on machine learning, but optimized for conversational applications. They include tools for “intent classification, entity recognition, entity resolution, question answering, and dialogue management.” Also:
Unlike cloud-based NLP services, Conversational AI platforms do not require that training data be uploaded to a shared cloud infrastructure. Instead, they provide a flexible and versatile platform which ensures that data sets and trained models are locally managed and always remain the intellectual property of the application developer.
MindMeld mentions only their solution as example. Answering our question on what makes MindMeld’s platform different than other solutions, they said:
Services like wit.ai, api.ai, and LUIS are tools which can help engineers without a data science background build simple NLP models. MindMeld is a more advanced machine learning platform which enables data scientists to create far more advanced conversational interfaces. For example, MindMeld enables our customers to create a custom knowledge graph which captures a deep understanding of their content or product catalog. MindMeld can also be used to build advanced NLP models trained on thousands or millions of training examples. MindMeld also provides a state-of-the-art question answering system which can be adapted to any knowledge domain. None of the web-based NLP services (wit.ai, api.ai, LUIS) provide capabilities like this.
The playbook also includes a 10-steps implementation guideline for creating a conversational application using MindMeld’s platform. Some of these steps apply to other solutions too. They are briefly mentioned here:
- Select the right use case. Not all use cases are fit for a conversational application. When selecting a use case one needs to consider if it resembles a real-world human interaction, if it saves user’s time and if it is convenient for the user.
- Script your ideal dialogue interactions. Consider how the user would use the application, what are the possible conversations to be carried including edge case such as asking questions beyond the scope of the application.
- Define the domain, intent, entity and role hierarchy. Determine the basic elements of the natural language to be interpreted.
- Define the dialog state handlers. Determine the dialog steps to be made from the first interaction with a user to the last, providing the final detail required.
- Create the question answerer knowledge base. This is an information repository needed to understand questions and answers.
- Generate representative training data. Create datasets of training data needed by machine learning algorithms to develop a better understanding of the domain and the possible conversations that can be carried.
- Train the natural language processing classifiers. This involves training the NLP engine to recognize the domain, the intent, the entities mentioned by the user and the role played by these entities.
- Implement the language parser. This step involves configuring the language parser to correctly identify the relationship between entities to create the parse tree.
- Optimize question answering performance. Tune the question answerer for best results.
- Deploy trained models to production. Test the application locally then deploy it to production.
For more details on these 10 steps on creating a conversational application, we recommend reading the guide.