Open Allure DS conversational interface (using Python)

Back at Christmas I was speaking to John Graves about his Open Allure DS PhD project – a conversational interface written in Python. The project has moved wonderfully forward over the past few months, I’ll summarise some of the features of Open Allure here.

Sidenote – if you prefer podcasts then John was recently interviewed by Ron Stephens on Python411 (with a lovely plug for ShowMeDo – cheers John!). The interview runs for an hour and is completely about Open Allure.

The project aims to build a speech and gesture recognising interface, mostly using Python, to provide us with new ways of interacting with the computer. As John states:

Open Allure is a project aimed at developing new ways to share what we know with one another by permitting the collaborative creation and experience of interactive dialogs.

These verbal exchanges give your interaction with the computer a very different quality and permit immediate feedback to help reinforce or reorganize your thinking.

Because voice recognition is still imperfect, the interface also supports making choices by gesture: your webcam watches you as you raise your hand.

Here’s a good overview video:

And here’s a demo showing speech and gesture recognition:

Here John shows the start of a chatbot interface – you can verbally ask the computer to solve math problems and to look-up word definitions:

I particularly like the direction of that latest (0.1d7) release – John has integrated a persona (shown as photos of John in the top-right corner) to visually show the state of the machine. The persona indicates if the machine is listening or responding, this feedback is very common in human-to-human interaction:

Gesture recognition uses a ‘green screen’ approach. The system records a set of images to generate an average background, it then subtracts (using a grey scale image) each new image from the average and applies a threshold. Movement shows up as regions of activity, if these regions occupy a region of interest then an event is generated:

Currently John has the best support for Open Allure on Windows using the Microsoft Speech API (support for Dragon Naturally Speaking is just starting). On the Mac and Linux there is no speech recognition yet. pyGame is used for the vision components. This Google Sheet lists the status of the components for each platform.

John has tested the open source Julius continuous speech recogniser on Linux (shown below) but it isn’t yet integrated into Open Allure:

Do you want to try it yourself? If you have Windows then you should be all set, Linux and Mac have partial support. To see what you need visit the PyPI page.

If you want to contribute then see this list of easy improvements that need to be made and join the project’s Google Group.

Comments are disabled for this post