Step 14: Get the app visual context

We are building a context-aware voice agent: it should differentiate responses depending on what is currently displayed on the page. Let’s update our dialog script to let the user ask about the order details.

• If no items are added to the cart on the page, the voice agent will inform the user about it.

• Otherwise, it will list all items added to the cart.

For this step, we will use Alan AI’s visual state. This predefined object allows you to get information about the app visual context, such as what page is currently open or what data is displayed, and send it to the dialog script.

1. In the app, update the changeOrder() function to call the setVisualState() method.

   With the help of this method, we will send the information about the visual state of the app to the dialog script. To the method, you can pass any object; in our case, it will be the order object we have in the app.

   HTML file

   function changeOrder(item, quantity) {
       let number = (order[item] ? order[item] : 0) + quantity;
       // Removing the item or updating the number of items
       if (number <= 0) {
           delete order[item];
       } else {
           order[item] = number;
       }
       updateCart();
       // Sending the app visual state
       alanBtnInstance.setVisualState({order});
   }
   

2. To the dialog script in Alan AI Studio, add two voice commands:

   Dialog script

   intent(visual(v => !_.isEmpty(v.order)), '(What are)- my order details?', p => {
       p.play('You have ordered:');
       for (let product in p.visual.order) {
           p.play(p.visual.order[product] + " " + product);
       }
   });
   
   intent(visual(v => _.isEmpty(v.order)), '(What are)- my order details?', p => {
       p.play('You have not ordered anything');
   });
   

   The visual state data sent from the app is accessible through Alan AI’s visual object. In these commands, the object is used as a filter: if the object contains any data, the first command is matched. Otherwise, Alan AI matches the second command.

3. To make sure our voice agent is context-aware, we should also update the voice command for removing items from the cart. If the item is not yet in the cart, Alan AI must inform the user about it; otherwise, it will remove the named items.

   In the dialog script, replace the command for removing items with the following one:

   Dialog script

   intent(`(Remove|Delete) $(NUM: 1, 2, 3...) $(ITEM ${itemList}), (please)-`, p => {
       let order = p.visual.order || {};
       const item = p.ITEM.label;
       if (!order[item]) {
           p.play(`${p.ITEM.value} is not in the cart yet`);
       } else {
           let quantity = order[item] ? order[item] : 0;
           let deleteQnty = p.NUM ? p.NUM.value : quantity;
   
           if (quantity - deleteQnty <= 0) {
               p.play(`Removing all ${p.ITEM.value}`);
           } else {
               p.play(`Updating ${p.ITEM.value}`);
           }
           p.play({command: 'updateOrder', item: item, quantity: -deleteQnty});
       }
   });
   

   Here is how this command works: Alan AI checks what is currently available in p.visual.order sent from the app. If the named item is not available, it plays: The item is not in the cart yet. If the named item is added to the order, Alan AI updates the order to remove the desired quantity of items.

Refresh the web page and ask about the order details. Add some items to the cart and ask about the order details again. Then try adding and removing items with voice.