Implement your own voice commands

Voice commands implementation guide

Contexts

A context defines a list of voice commands that could be executed at a given time.

The Handsfree for Web application has loaded a single context to a given moment. For example, at the beginning of a session, the context called "root" is loaded. Most of the relevant commands are set for the "root" context. e.g. "click", "select", "scroll down", "open", "search", etc.

Why we need multiple contexts? imagine the user is writing something in a text box, if the user says "open" we want to type "open" into the box, we don't want to ask to the user which website has to be opened.

Some commands shouldn't be able to be executed all the time, most of the time we'll want to define which are the commands that could be executed.

So, if you want to define a command that could be executed at the beginning of the session you can extend the "root" context in the following way.

Defining hello world command within a context

var myContext = {
    context: 'root', 
    commands: [{
        name: 'say hi',
        action: function () {
            document
                .querySelectorAll('p, a, span, li, button, th, td, h1, h2, h3, h4, h5, h6')
                .forEach(function(el) {
                    el.textContent = 'hello!';
                })
        },
        group: 'Greetings',
        help: 'It shows "hello!" everywhere'
    }]
};
                    

Imagine if we want to implement some commands useful to change the font of a website.

We can define the following commands:

  • Switch font to serif
  • Switch font to sans serif
  • Switch font to monospace

Given these commands we can extend the "root" context with these three commands.

Defining switch font commands within "root" context

var myContext = {
    context: 'root', 
    commands: [{
        name: 'switch font to serif',
        action: function () {
            document.body.style.fontFamily = "serif"
        },
        group: 'Fonts',
        help: 'Switch website font to serif'
    }, {
        name: 'switch font to sans serif',
        action: function () {
            document.body.style.fontFamily = "sans-serif"
        },
        group: 'Fonts',
        help: 'Switch website font to sans-serif'
    }, {
        name: 'switch font to monospace',
        action: function () {
            document.body.style.fontFamily = "monospace"
        },
        group: 'Fonts',
        help: 'Switch website font to monospace'
    }]
};
                    

These commands are fine and they will work for sure, but what happen if we want to do the things a bit different. We want the user be able to choose which font want after say "Switch font". In oder to accomplish that now we have the following commands to implement.

Root context

  • Switch font

Font Picker context

  • Serif
  • Sans Serif
  • Monospace
Defining font picker commands

var rootContext = {
    context: 'root', 
    commands: [{
        name: 'switch font',
        action: function () { },
        group: 'Font Picker',
        help: 'Switch website font',
        switchToContext: 'font-picker'
    }]
};

var fontPickerContext = {
    context: 'font-picker',
    commands: [{
        name: 'switch font to serif',
        action: function () {
            document.body.style.fontFamily = "serif"
        },
        group: 'Fonts',
        help: 'Switch website font to serif',
        switchToContext: 'root'
    }, {
        name: 'switch font to sans serif',
        action: function () {
            document.body.style.fontFamily = "sans-serif"
        },
        group: 'Fonts',
        help: 'Switch website font to sans-serif',
        switchToContext: 'root'
    }, {
        name: 'switch font to monospace',
        action: function () {
            document.body.style.fontFamily = "monospace"
        },
        group: 'Fonts',
        help: 'Switch website font to monospace',
        switchToContext: 'root'
    }]
};
                    

So now we have two contexts, when the user says "switch font" the command loads the context "font-picker", this allows the user to be able to execute the commands "serif", "sans serif" or "monospace". Once one of this commands be pronounced, the font of the website will be changed and the "root" context will be loaded.

We can allow the user to pick several times the font until the command "exit" be executed. For achieve that it's necessary to remove the switchToContext: 'root' property and value at the end of each file picker context's command.

When defining a context the property context and commands are the most important but there are others that help us to implement more complex behaviors.

Property Required Type Description
context yes string Defines the key name of the context
name no string Defaults to []. Defines the context name that will see the user
commands no array Defaults to []. Defines the commands allowed to be executed when the context is active
i18n no object Defaults to {}. Allows to define texts for different languages at context level
setup no function Defaults to function() {}. This function is called before the context is active
teardown no function Defaults to function() {}. This function is called when the context is not longer active
switchOnSelectElement no function Defaults to function() {}. This function is called when the user selects some DOM element, if the function returns true the context will be activated

Next: Modules

Sobre a Ferramenta

Javier Perez UNLP Unigranrio

A extensão Handsfree for Web foi desenvolvido por Javier Pérez, a fim de trazer uma alternativa de navegação web.

O aplicativo foi apresentado como uma implementação de um modelo de navegação na web sem o uso das mãos, descrito no trabalho final de Navegação Mãos-Livres conduzido por comandos de voz para o grau de Ciência da Computação na Universidade Nacional de La Plata.

Colaboradores

Alunos da Universidade Unigranrio enviaram a versão em português da ferramenta.
Agradecimento especial a:

  • Matheus Freitas
  • João Victor Plessim
  • Gabriel Francisco
  • Guilherme Cabral
  • Lucas Hermida