Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

Editor's Notes

1. Thrilled to be here today presenting at JanusCon. Before we start, I wanted to share a picture of my wife and me dancing salsa. Just like in salsa dancing, action-response coordination is important in what we're talking about today. We'll see how WebRTC and LLMs working together could revolutionize RTC. Ready to dive in? Let’s go!
2. R1 small mobile like. Built to interact with LLMs in real time. Camera, mic for input and a touch screen. Caught my attention -> Under the hood technology allowing them to play your spotify account or book a flight…And this seems to be handled by something called LAM (Large Action Model)
3. Symbolic reasoning: <button> element is a button Neural networks: html elements with `click`, `start` or `push` are typically buttons You can think of them as an “action-GPT” or “LLM agents”. Large Action Models is not an agreed upon industry wide term, it is generating a great deal of interest Below are a few concrete examples: - A user could create a “RSVP to a wedding” workflow which might include finding the email invite, RSVPing yes with a note, checking the registry for purchasable items and buying the gift. THIS IS NOT NEW, has been there for 1-2 years
4. Or Imagine a scenario where a user wants to book a hotel room. Instead of having to navigate through multiple websites or apps, the user can simply interact with the bot and provide travel preferences. The bot can then search for available options, compare prices, and make bookings on behalf of the user in real-time.
5. In customer service scenarios, a bot that can perform actions can handle a wide range of tasks such as processing returns, issuing refunds, updating account information, or troubleshooting technical issues. This would reduce the workload on human agents and provide faster resolutions for customers. Note: At this point, user feedback might be required.
6. Personal Shopping Assistant: In the realm of online shopping, a bot that can perform actions can act as a personal shopping assistant. Users can specify their preferences and requirements for certain products, and the bot can automatically search for the best deals, apply discounts or coupons, and even complete the purchase process seamlessly. Smart Home Automation (SMART HOME DEVICES): For homeowners with smart home devices, a bot that can control these devices based on voice commands or predefined preferences would be invaluable. Users can ask the bot to adjust the thermostat, turn on lights…saving time and enhancing convenience. Workflow Automation: In a business context, a bot that can perform actions can automate various workflows and processes, such as data entry, report generation, or task assignment. This would streamline operations, increase efficiency, and free up human employees to focus on more strategic tasks.
7. For bidirectional communication we could just use WebRTC. Spin up a headless browser and connect to Janus as a standard WebRTC client
8. Speaking requires less cognitive effort than typing, especially for complex queries. More human-like Middle point (option2) seemed a good compromise for the demo although wouldn’t have been complicated to do (option3) and makes sense in video calls
9. Websockets can be integrated with other services (e.g., speech-to-text APIs, natural language processing). Most AI SaaS platforms have selected this as the preferred protocol. Implementing websockets is relatively straightforward Usage of speech to text has the benefit of providing a more human-like experience graph LR     A[Audio Capture] -- Websockets --> B(Speech to Text)     B --> C{LLM Prompting}     C --> D(Text Response)     D -- Websockets --> F{Client}     F -- Audio --> A 
10. Usage of RAG (Retrieval Augmented Generation) is key to make this agent assist functionality valuable. A technique that allows you to add knowledge base to the LLM providing more specific answers.
11. As mentioned before LAM are just LLMs that perform actions that go beyond text or audio transcription. The ingredients for the hack I did were: 1) Mind2Web dataset (trained to perform actions in 100s of sites) 2) Large Multimodal Model to capture images and understand them 3) Headless browser to interact with the browser 4) App logic code to translate described actions to the actual browse action. SeeAct open source
12. Cross-Encoder is a type of neural network architecture used in natural language processing tasks, in the context of text pair classification. Its purpose is to evaluate and provide a single similarity score for a pair of input sentences
13. So all put together we have the webrtc client… This is what I will demonstrate in the demo now graph LR     subgraph Server         H <--WebRTC--> M         M(WebRTC Client) -- Audio --> I{User Speaking?}         I --|Yes / Audio|--> J(Speech to Text)         I --|No|--> I         J --|Text| --> P(SeeAct)         P --> Q(Headless Browser)         P --> K{LLM Prompting}         L --|WS or SSE| --> P(SeeAct)         Q --|pass image|--> K     end     O --> L(Text Response)     K --|API Call| --> O{3rd Party LLM API}     G[User] <-- WebRTC --> H(Janus)     Q --|opens and performs actions|--> R(website) graph LR     subgraph Server         H <--WebRTC--> M         M(WebRTC Client) -- Audio --> I{User Speaking?}         I -- Yes / |Audio| --> J(Speech to Text)         J --|Text| --> K{LLM Prompting}         L --|WS or SSE| --> M         L --|Wait| --> N[Wait State]         N --> I     end     O --> L(Text Response)     K --|API Call| --> O{3rd Party API}     G[User] <-- WebRTC --> H(Janus)
14. Particularly the specific frameworks and tools used are…
15. Launch Janus and connect to video room Initialize R-SeeAct service Visualize Browser Actions
16. Transcription: Headless Service Actions: WebRTC Media Server Latencies <300ms So Can Be Disregarded sequenceDiagram     participant MediaServer     participant WebRTC_Bot     participant Whisper     participant GPT4_Turbo     participant HeadlessBrowser     MediaServer->>WebRTC_Bot: Send audio (50ms)     WebRTC_Bot->>Whisper: Capture and send audio     Whisper-->>WebRTC_Bot: Transcribe audio (1000ms)     WebRTC_Bot->>HeadlessBrowser: Initialize and gather HTML elements     WebRTC_Bot->>GPT4_Turbo: Prompt with action goal, HTML elements and image     GPT4_Turbo-->>WebRTC_Bot: Text response with image information (5000+ms)     WebRTC_Bot->>GPT4_Turbo: Next prompt with multichoice and further context     GPT4_Turbo-->>WebRTC_Bot: Final action response (5000+ms)     WebRTC_Bot->>HeadlessBrowser: Execute action, e.g: click button     HeadlessBrowser-->>WebRTC_Bot: Executed action (3000ms)