Terry is a virtual assistant that uses a composite robot learning approach to perform tasks on a computer using the same interface that a person would use. This interface is the operating system graphical user interface (OS GUI), including the peripherals needed to interact with it (the mouse and keyboard).
The composite learning approach attempts to incorporate aspects of three: demonstration, instruction, and reinforcement learning. The fact that Terry performs actions in the same way that a user would means those actions can be taught using demonstration learning. Additionally, a voice/text interface between Terry and the user will be used to both give commands and teach actions using instruction learning. Lastly, the user can watch the actions being carried out on-screen while Terry performs them, and provide feedback and guidance throughout.
The paper is in the docs/ folder, an exported PDF version of this Google doc online.
Here are demonstrations of Terry's major features implemented so far, done on a Mac laptop and a Windows desktop, uploaded to youtube
I got the following icons from fontawesome (link to their license here), which can be found in src/com/terry/res/img/:
- microphone (
fa-microphone)
I used paletton to create the color scheme for Terry at docs/color_scheme.html.