# Google Assistant — Interactive Canvas ### Overview [*Google Assistant*](https://assistant.google.com/#?modal_active=none) *is a personal voice assistant that offers a host of actions and integrations. From sending texts and setting reminders, to ordering coffee and playing music, the 1 million+ actions available suit a wide range of voice command needs.* ### Objectives * What is an interactive canvas * How interactive canvas works * Create an action with interactive canvas ### Let’s Begin During I/O’19 Google announced an Assistant-based-framework called **Interactive canvas** which will allow developers to build full-screen apps (or Actions) on smart displays that bring together touchscreen inputs, voice and visuals. For the time being, games are the only type of Actions they can create for smart displays using Canvas, with more categories coming soon. *HQ University* (a version of [*HQ Trivia*](https://www.engadget.com/2019/04/12/hq-trivia-rogowsky/) that was already available through Assistant) was given as an example of the type of game you’ll be able to play, along with *Jungle Dream*, an interactive word game. You probably shouldn’t expect AAA blockbuster games here, as Canvas uses open web technologies such as HTML, CSS and Javascript, so these Actions seem likely to be akin to web apps. However, it certainly seems you’ll have more options for things to do with your [Nest display](https://www.engadget.com/2019/05/07/google-nest-hub-max-smart-display-assistant-hands-on-price-availability/) in the not-too-distant future. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056827597/024d9244-553e-42f4-8df8-53d0dcbd892c.gif align="left") Source: Google I/O’19 ### What is an interactive canvas Interactive Canvas is a framework built on the Google Assistant that allows developers to add a visual, immersive experience to conversational Actions. This visual layer is an interactive web app that is sent as a response to the user in conversation. Unlike traditional [rich responses](https://developers.google.com/actions/assistant/responses#rich_responses) that exist in-line in the Assistant conversation, the Interactive Canvas web app renders as a full-screen web view. You can use Interactive Canvas if you want to do any of the following in your Action: * Create full-screen visuals * Create custom animations and transitions * Do data visualization * Create custom layouts and GUIs * Implement video playback (videos are not yet fully supported, but may still play in Interactive Canvas) ### How it works Interactive Canvas connects your conversational Action to an interactive web app so that your users can interact with your visual user interface through voice or touch. There are four components to an Action that uses Interactive Canvas 1. Custom Conversational Action 2. Web app 3. Assistant Canvas 4. ImmersiveResponse #### Custom Conversational Action An Action that uses a conversational interface to fulfill user requests. Actions that use Interactive Canvas operate in the same fundamental way as any conversational Action, but use immersive web views (`[ImmersiveResponse](https://developers.google.com/actions/interactivecanvas/reference/immersiveresponse)`) to render responses instead of rich cards or simple text and voice responses. #### Web app A front-end web app with customized visuals that your Action sends as a response to users during a conversation. You build a web app with web standards like HTML, JavaScript, and CSS. `assistantCanvas` lets your web app communicate with your Conversational Action. #### Assistant Canvas JavaScript API that you include in the web app to enable communication between the web app and your conversational Action. #### Immersive Response A response type that defines how the web app should render. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056829772/9d3cc7c9-5278-4f6c-8b7f-088fbbb0a070.png align="left") Source: Google I/O’19 1. The user talks to the device 2. User input is passed to actions on google. 3. Actions on Google then invokes the dialog flow 4. Dialog flow matches the intent based on the user phrase 5. It invokes the associated fulfillment which provides the immersive response 6. As part of the immersive response, a URL is provided along with the response 7. The device loads the associated web app from the URL 8. It also registers some callbacks from the Interactive Canvas API ### Creating an Action in Interactive Canvas Let’s start an action using an interactive canvas #### Create an Actions project Regardless of the Assistant application, you’re building, you will always have to create an Actions project so your app has an underlying organizational unit. Open the [Actions on Google Developer Console](http://console.actions.google.com/) in a new tab. You should be looking at a clean Actions console that resembles the following (If you are a new user (: ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056831483/20636522-f9a6-4d10-886c-92f1184ae92f.png align="left") Click **Add/import project** and agree to Actions on Google’s terms of service when prompted. Click into the Project Name field and Add the project name **Interactive Canvas** Select Games/Fun ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056833147/ca5db4af-cc03-46a8-afea-aea2498975d1.png align="left") Source: Google I/O’19 Select Conversational ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056834702/116c494e-56c2-4b11-aa0a-088f387b6096.png align="left") Source: Google I/O’19 Once Conversational is selected click **Directory Information** in **Deploy** and the app category is selected as Games&Fun once scrolled down you can see a new option ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056836185/90ef67aa-2d5a-41a5-b336-ca24c1a53471.png align="left") Source: Google I/O’19 Once the Checkbox is enabled. Click **Actions** in **Build** ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056838545/a64e71df-5b5c-4f6e-8d96-e943e56a2521.png align="left") Source: Google I/O’19 Click **ADD YOUR FIRST ACTION.** A pop-up dialog will appear in that select **PLAY GAME** and Click **GET STARTED IN DIALOG FLOW** ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056840270/12e2ff22-6ef6-4f9e-b30f-cec61a86806d.png align="left") Source: Google I/O’19 Build better voice apps. Get more articles & interviews from voice technology experts at [voicetechpodcast.com](https://voicetechpodcast.com/) This will take you to the Dialogflow console. click **Allow** when Dialogflow prompts you for permission to access your Google Account. When you land on the Dialogflow account settings page, check the box next to **Yes, I have read and accept the agreement** and click **Accept**. You will now be brought to a Dialogflow agent creation page. Click **CREATE** You can see the default intents like below ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056842791/56d1612a-8261-4fb3-9862-eab599d16627.png align="left") Source: Google I/O’19 once you click **actions.intent.PLAY\_GAME** you can see the events like below ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056844146/0212ed53-94bc-4a34-989d-6f64e2676437.png align="left") Source: Google I/O’19 Remove all the intents which is shown by default and create the below intents * color * fallback * instructions * pause * start * timer * welcome select **color** intent and add the training phrases like below * set tint to blue * change color to red * change the triangle’s color to green * green ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056845700/7ae2691a-65a5-46c1-b137-333946d6c1c9.png align="left") Under fulfillment, section enable webhook for the intent and click **SAVE** ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056847604/55e46590-2051-45aa-97d2-6af5bedc9f84.png align="left") Select **pause** intent and add the below **Training Phrase** ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056848884/6151f0f2-7d1f-4bd6-9dc4-4445db67eecf.png align="left") Under fulfillment, section enable webhook for the intent and click **SAVE** Select **start** intent and add the below Training Phrase ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056850366/9c892683-7077-4693-93c6-8771aa8de422.png align="left") Under fulfillment, section enable webhook for the intent and click **SAVE** Select timer intent and add the below Training Phrase ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056851576/6d9a8641-4b1c-4abc-bd9c-4b2388a99213.png align="left") Select Response in timer intent and add the below code in default response Under fulfillment, section enable webhook for the intent and click **SAVE** For the rest of the intents click fulfillment and enable webhook and click **SAVE** #### Backend Part Create a firebase cloud function. If you are new to cloud function have a look at this [post](https://medium.com/@nidhinkumar/firebase-cloud-function-f4ab77ae2807) In index.js add the below code Replace package.json with the below code Now we will create the html page Create a folder named public and then create files/folders like below * css (folder) * js(folder) * index.html(file) * triangle.svg(svg file) or you can grab the code from this [repository](https://github.com/actions-on-google/dialogflow-interactive-canvas-nodejs/tree/master/public) You deploy your codes and get the webhook url and replace the url in dialogflow ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1698056852962/78734ca5-ee11-4766-a0b7-7ce81345aa7f.jpeg align="left") Scroll down to the bottom of the page and click **Save** in the lower right corner. Now that your Cloud Function has been deployed and your webhook has been properly set up, you can preview the app in the Actions simulator. ### Change your Google permission settings In order to test the application that you built you need to enable the necessary permissions. Open a new tab in your browser and visit the [Activity Controls page](https://myaccount.google.com/activitycontrols). Sign in with your Google credentials if prompted. Ensure that the following permissions are enabled by sliding the toggles and selecting **TURN ON** for the following cards: * Web & App Activity * Device Information * Voice & Audio Activity Now **close** the Activity Controls page. ### Test the application with the Actions simulator Return to the Actions console. Then from the left-hand menu, under the TEST header click **Simulator**. Your output will be like below ### Congratulations! You have learned what is interactive canvas, how it works as well as created a sample action using interactive canvas. It’s time to create more actions with interactive canvas. Good Luck!!! ### References 1. [Dialogflow interactive canvas](https://github.com/actions-on-google/dialogflow-interactive-canvas-nodejs) 2. [Build Interactive Canvas Google I/O’19](https://www.youtube.com/watch?v=J8lsvvJ21Ok) 3. [Actions on Google interactive canvas](https://developers.google.com/actions/interactivecanvas/)