Lessons learned from building a commercial bot development platform
•
1 like•8,685 views
Keynote at 2nd International Workshop on Bots in Software Engineering on the lessons learned while building our chatbot platform Xatkit
Report
Share
Lessons learned from building a commercial bot development platform
- 1. Lessons learned from building a commercial bot development platform Jordi Cabot & Gwendal Daniel @softmodeling @xatkit – xatkit.com
- 2. Xatkit is a spin-off of the SOM team
- 3. Our mission We are interested in the broad area of systems and software engineering, especially promoting the rigorous use of models and engineering principles while keeping an eye on the most unpredictable element in any project: the people. Flickr/clement127 Create Better Software Faster
- 5. Bots vs chatbots vs voicebots
- 6. What is exactly a Chatbot? • Chatbot = Bot + Chat interface • Voicebot = Bot + Voice Interface • At the core, a NLU/NLP engine with two key missions – Classify the user utterance (i.e. input text) in one of the intents (set of possible questions/requests) defined in the bot – Identify the parameters (called « entities ») from the utterance Voicebot Chatbot Bot
- 7. Chatbots combine all the complexity of bots with the complexity of NL (typos, trolls, sentiment analysis…)
- 8. Think twice before building a chatbot
- 9. Can you help us create a chatbot to help newcomers to write good bug reports on GitHub? Sure! It’s just a set of questions & answers, this will be pretty simple! Narrator It wasn’t. Once upon a time (we were young and naïve)
- 10. What went wrong? – Many pieces to assemble Input/Output Messaging channels External Platforms NLU Engine
- 11. Chatbots are complex software systems Conversation Logic Text Processing External Services Messaging Platforms Deployment Evolution Maintenance Tests
- 12. You’d like to focus on writing bot rules User Intent Action Parameters If the User Wants To Open Issue Reply « In which repository? » on Slack Platform
- 13. Why Xatkit?
- 15. >100 chatbot platforms • Rasa, BotPress, Chatfuel, Inbenta, Botsify, Flow XO,… • And quite a few NLU providers / engines: DialogFlow, Lex, LUIS, Watson, NLP.js, Stanford…
- 16. … but • Only around 15% are open source • Not flexible (i.e. difficult to integrate other platforms) • NLU Lock-in • Low-level (beyond simple Q&A bots, you quickly end up programming in JS)
- 17. • Grady Booch – history of softwre engineering The entire history of software engineering is that of the rise in levels of abstraction - Grady Booch Also true for AI/chatbots -> Chatbot Modeling to the rescue!
- 18. We were asked to build a bot. We ended up building a bot platform
- 20. Key Concepts
- 21. Model-driven / Low-code solution • Raising the level of abstraction at what chatbots are defined • DSLs to focus on the core logic of the chatbot • Conversation and user interactions • Action computations • Independent from specific implementation technologies • Automating the deployment and execution of the modeled chatbot • Deploy over multiple platforms • Runtime to execute the chatbot logic
- 22. Platforms in Xatkit • An abstraction of any input/output messaging tool and any external service. Platform interfaces are also defined with a DSL • Platforms offer actions (that the bot can execute) and events (whom the bot can subscribe to) • PIM/PSM : Bots can be defined using generic platforms that are then concretely instantiated during the deployment. No need to change the bot
- 23. Xatkit Framework Chatbot Designer Intent Recognition NLU Providers (platform-specific) Platform Package Intent Package Xatkit Modeling Language Chatbot User Instant Messaging Platforms Xatkit Runtime Execution Package uses uses Platform-independent chatbot definition External Services Deployment Configuration Platform Designer Daniel, Cabot, Deruelle, Derras: Xatkit: A Multimodal Low-Code Chatbot Development Framework. IEEE Access 8: 15332-15346 (2020)
- 25. Example platforms Abstract platform Chat actions { PostMessage(message) Reply(message) } } Platform Slack extends Chat actions { PostMessage(message, channel) Reply(message) } }
- 26. Ex1: Stargazer
- 28. Ex2: Talk with your repos from Slack
- 30. Xatkit Tool
- 31. Tool Support • Available on GitHub as an independent organization: https://github.com/xatkit-bot-platform/ • >40 repositories, thousands of commits,… • Open source (EPL v2) • With tutorials and examples to help you get started building bots!
- 32. Reflections / Lessons Learned (May 2018 - )
- 33. From a proof-of-concept to a real tool
- 35. You need to be a Polymath to create a bot platform • Plenty of core bot tech: NLU, Sentiment Analysis, Xtext, Slack/GH/… APIs,…. • But even more non-bot tech: InfluxDB, Nginx, eCharts, React, web sockets, Docker… • Commercial-grade quality involves spending more and more time on “boring” stuff Bruneliere, Hugo, Jordi Cabot, Frédéric Jouault, Massimo Tisi, and Jean Bézivin. "Industrialization of research tools: The atl case." 2010.
- 36. Common challenges to any OSS project • Attracting contributors • Giving support to users that don’t even say thank you • Cost of maintaining external contributions (e.g. connectors) especially when the core is still evolving Being pragmatic (ROI) • We went from having official releases, to core releases to no releases (compile from sources). Moving to Docker images (good compromise) • Also, no support for older versions. • No backward compatibility.
- 37. Technical
- 38. Eat your own chatbot food for a better language • Evolving your DSLs is expensive, try to design a variety of bots to make sure your language is expressive enough for your needs • We had to move from a partial order language to a state machine one (and change the examples, the docs, the editor,…) with it • Set on a language granularity (e.g. live chat primitive) Combine textual and graphical notations • Textual for writing + Graphical for Reading • Do not reinvent the wheel -> We came close to reinvent Java (and moved to xbase)
- 39. Choose your stack carefully • We chose Java, Eclipse/EMF and Xtext as this was our “natural” option • Changing now would imply almost starting from scratch Think long-term • Research prototypes are pure technical debt • MapDB -> InfluxDB • Scalability: multi-user/multi-session bots
- 40. DDoS attacks via a bot • Easy to attack your bot simulating a massive number of utterances • Could also be expensive (e.g. DialogFlow API) Other security concerns • Must provide support for offline chatbots • Privacy of the monitoring data, especially if linked to specific users
- 41. NLP
- 42. NLP engines don’t speak Catalan • You may need to combine more than 1 NLP provider to support less mainstream languages NLP engines are not bilingual • Bots should be able to combine different languages in the same conversation
- 43. Everybody seems interested in having sex with a bot: • Trolls are a huge problem with bots • As a Brand you don’t want to risk going viral because your bot gave the wrong response to a Troll question Importance of intent libraries • To deal with troll questions but also for greetings, thank you, goodbye,… and other common intents you don’t want to define again and again
- 44. Testing chatbots is a largely unexplored area • We use Botium to record conversations to be used as test cases • But we need much more (e.g. automatic generation of test cases) Quality of the intent matching process • It can decrease just because your NLP provider releases a new version • More is not always better (e.g. adding more training sentences may end up conflicting with another intent) • Enumerations (“entities”, e.g. list of cities) make a huge difference
- 45. Business
- 46. End-users do not want to define chatbots: • Doesn’t matter how nice is your inteface • Doesn’t matter how high-level is your language • Independent consultants DO want that (who are you selling to?) End-users just want to have fun give you their data • And hope you’ll transform those excels, FAQs, guides… into a bot • Selling a service vs selling a tool
- 47. Find your niche • You cannot compete with Microsoft or DialogFlow. Nor with Zapier From an Open Core model to a Chatbot as a Service • We didn’t feel there was a market for selling add-ons • Moving to tailor made bots thanks to Xatkit flexibility + ChaaS for premade ones (e.g. Xatkit for WordPress plugin) • DISCLAIMER: We’re still looking for the good niche + business model combination
- 48. Understand what bot owners are looking for • E.g. making sure the color of the Xatkit widget matches the look&feel of the website is more important than any item in your roadmap Help them understand what their visitors want to use the bot for • Importance of the bot monitoring (wrong training sentences, unexpected questions,…) • (Supervised) bot learning as automatic bot learning is risky
- 49. Legal
- 50. You sure you want to create a spin-off? • 1 year of paperwork • University takes a cut of the company and limits your dedication • IP is problematic (also among team members) • It may limit your publication strategy • (Your mileage may vary) Look for win-win situations • Even if the spin-off fails everybody should get something out of it (publications, better CV,…)
- 51. Where do we go from here
- 52. Xatkit as a bot teaching platform • Do you teach any course involving bots, chatbots or NLP? Or DSLs / software modeling ? • And are looking for a use-case or a ready-to-use platform? • We’ll be happy to help • Materials • Assistance • (Remote) seminar
- 53. Xatkit as a research platform • Researching on the impact of (chat)bots in Software Development? Combining a number of different platforms? • Do not reinvent the wheel, adopt Xatkit as base platform • Benefit from all Xatkit features and connectors to have powerful bots with less effort • Spend more time on the core research aspects
- 54. Xatkit as a research platform • I think we all agree that there is plenty of research to be done on • Software engineering for bots • Bots for software engineering • Examples: • Testing for bots • Automatic generation of bots from data source (e.g. open data) • Bots to promote diversity and inclusion (or to detect online harassment) • …