Research focus

2010-02-25T03:37:00.000-08:00

The demo app is up and running. Sadly, through lack of knowledge and resources, I am unable to host this somewhere for readers of this blog to see.

Now I have proved to myself that the underlying concepts of what I am aiming to achieve are possible. The time has come to focus more on the research section of my project.

Things to consider

What are the common tasks a user expects to be able to carry out when using a visual design tool ? As previously stated, although my app demo's editing of a UML class diagram, the creation of a comprehensive and ultimately useful UML modelling tool is out of the scope for this project. The reason for choosing a UML design tool (Class diagrams) for use alongside the Touchless SDK is that in creating and editing even the most basic class diagram, the user demands a wide range of gesture interactions. Should i of simply used Touchless to navigate web-page content then there would only be a need to implement basic click functionality; Either to click buttons or two draw the cursors focus to various text boxes and list box items. In using a visual design tool alongside Touchless, the user would expect ways to click-and-drag, resize, delete, create, update as well as link and unlink elements. This forces my implementation of Touchless to facilitate a much wider range of gesture interactions.

For this reason, much of the material written about what a 'good' uml tool must include is far too detailed for what i require. The OMG, leaders of UML initiatives, detail a list of essential features of a UML design tool in their publication, "UML 2.0 Toolkit". Such features include uni and bi-directional transformation between model and code, language specific semantic checking to ensure that one cannot create something in a model that is impossible to create in the code implementation (multiple inheritance in some cases), version support, collaboration support, as well as supporting not just Class diagrams but many other UML model types. Again, all of these essential features are far away from the scope of this project, I am merely using the notion of modelling a class diagram as a medium to demonstrate the diverse set of gestures a simple web-cam based gesture system could encompass. To this end, i shall attach to this blog a list of the features (gestures essentially) that my tool should and will support.

As well as review of others' research, I feel it would be beneficial and ultimately lead to a more through and complete project if i was to conduct some of my own research into the viability of simple gesture based systems replacing common keyboard and mouse interaction. The way i see it, this can be approached in two ways;

1. Contact development teams directly and ask them if they use UML modelling tools and quiz them about their choice of tool and what it offers them that made them settle. If the teams admit to using UML at some point in their development process but do not use a specific tool (this is the category i suspect most will fall in) then it would not be silly to ask them why they have not chosen a tool to develop with. Hopefully, the answer will be in that the tools take too long to learn how to use or the navigation / construction of diagram elements are more complicated on machine than simply using pencil.

2. Construct a second demo app which first asks users to place coloured shapes over the top of similar coloured areas of the screen and then draw lines linking all areas of the same colour together (maybe without lines crossing). What this would allow me to do is time users performing the same task using keyboard and mouse against those using my gesture system. As well as time, number of errors and a rough estimate of the level of frustration experienced could be measured, giving a good indication as to weather this gesture based system could actually replace and / or improve on current keyboard and mouse scenarios.

The latter approach would significantly widen my test group as participants need know nothing about UML and the issues surrounding it. They simply need to be able to understand the basic tasks of the demo app and be able to compare this to keyboard and mouse usage in the same situation. Also, as I have stressed many times that my projects relationship to a useful UML design tool is negligible, one could class the former approach as futile. If one approach must be chosen I think the latter would be a better choice but hopefully time will allow for both to be carried out.

Day 1: Plans and Thoughts

2010-02-17T17:27:00.000-08:00

Today I finally have everything I need to begin development on a Touchless UML modelling system. This project had originally aimed to develop an online-collaborative UML modelling application, however, as the time given to me to develop this application is very small, I would not be able to develop both the collaboration side of the application along with the diagramming side to any level of quality within 3 months. With the prospect of creating simply another diagramming tool seeming quite boring to me, and the collaborative side alone being little more than a specialised web-service, I began to search for a way of making UML modelling more interesting.

Concepts, concepts, motivation bla bla bla. If your interested in just the project and development side of things, scroll down until the text returns to black once again.

My research and idea took my through constructing neural networks to preempt faults or inconsistencies in class diagrams to using Silverlights powerful graphical abilities to add a third dimension to modelling. Eventually, when considering why I enjoy using my iPhone so much over a computer or other touch-screen devices I found myself philosophising over the gap between our real world and our digital world.

If you think about it, its strange in a way. In 'Todays world' you could say that our digital life is more integrated with our actual life more than ever. My phone can show me my emails on Outlook faster than my computer can usually. I can access Facebook / Twitter ect on both phone and computer and the experience on either is near identical. My computer can tell me who is using their Xbox and if I had the money, my computer could detect im on my way home, run a bath and turn the oven on. I once observed a fridge that detects its low on milk and contacts the local supermarket via internet to order in some more and the other day I was warned of a DVD i had to return to Blockbusters via text as the phone had detected I was close to a store. With such seamless integration then, why can I not interact with elements in my digital life in a way I do with the physical elements of my life ? Any time I wish to inform my computer of information or request information from it, I must use one of its pre-defined methods of communication i.e a mouse or keyboard.

At this point it is clear, I have strayed far from attempting to make the process of UML modelling more accessible to development teams and more to do with the essence of human-computer interaction. Surely one would rather talk than type (except in a few specific occasions)? Surely one would prefer to simply point at or nod towards an icon / web-link to navigate ? Surely one would prefer to flick through ones digital photo album as they do their actual photo-album ? Indeed these transformations in human-computer interaction have been hugely successful in the gaming environment. These are, of course, simply my own opinions but consider what the Wii has done in enabling users perhaps not well acquainted with conventional game controllers to play console games. In all my years of playing games when living with my parents, neither parent ever got to grips with the Mega Drive, Playstation or Xbox controller layouts yet in a matter of minutes they were able to put up a good fight on Wii tennis ! Why ? Because the way in which one informs the tennis racket of a Wii game is much more akin to the real-life version than flicking the flicking of an analogue stick in Xbox or Playstation counterparts. Following on from the Wii, observe how Microsofts Natal system for the Xbox promises to once again revolutionise our interactions with gaming consoles. Why does the Natal system seem to be a step forward from the Wii ? It removes the concept of using a controller to inform a computer system of your intentions entirely. Now you interact with the system as you would with its real-life counterpart.

So back to the project side of things. My intentions are no longer to build a UML modelling tool but via medium of online, UML modelling, investigate and conclude if it is both possible and beneficial to bring this concept of gesture based control into the business domain of computing. If I can build a concept application where one can create, modify and arrange the basic elements of a UML class diagram via the web browser and the means by which the user does this is on par or perhaps surpasses that of conventional keyboard and mouse interaction, then it would not be too much of a stretch to extend this to navigating and manipulating both web and system based data. Perhaps this also has applications with people who are unable to use mouse and keyboard interfaces fully or where it is inappropriate to handle them (medical facilities).

So a few starting points:

I want to develop this application in Silverlight. I have worked with it before and very much enjoyed it and believe that the way in which Silverlight operates would facilitate my aims better than the standard ASP.Net approach.
Interacting with a computer in the absence of a mouse and keyboard is not impossible. Microsoft's multi-touch setup would allow this. However, it requires the user purchased some specialised and expensive hardware. I believe for this form of computer interaction to real 'catch on' to the larger crowd, it would need to be free (or very cheap). Using the Touchless SDK I believe I can achieve similar results to that of a touch-aware screen with the only hardware requirement being a Webcam (the one I am developing with cost £20).
Because of both my desire to develop in Silverlight and my decision on using the Touchless SDK, I will need to use Silverlight 4 (currently in beta release). I am not overly happy about using a beta version of Silverlight for developing such an application but version 4 brings Webcam support, essentially to any Touchless based app. I believe there is a work-around, allowing the use of Silverlight 3 + Expression Encoder but it seems more 'fiddly' and adding another layer of complexity onto the already lightly supported stack of tools I am using seems counter-productive.
Because of the need to develop in Silverlight 4, I will have to use Visual Studios 2010 beta 2 and its Blend counter-part in order to have an environment capable of Silverlight 4 development.
I was put-off my original idea of a collaborative UML tool because developing, from scratch all the logic to handle creating the definitive UML class box, anchoring associations, grouping and moving elements ect would take far too long. Following on from this, the final application will still not be able to facilitate the creation of a useful UML class diagram but merely demonstrate the moving and grouping of classes and perhaps defining associations between classes.
There are other alternatives to using a webcam for gesture based input that still do not rival the cost of an actual touch-screen. There are many demo's available online of using a Wii remote control as a sensor and having it detect the movement of IR sources to interpret mouse movements and gestures. This certainly seems like a more accurate method of interpreting human movements as its accuracy does not reply on lighting conditions or unique and definitive colours. However, webcams often come with computers and some are even built-in. To require of our users to have a working IR sensor hooked up to their machine would be less realistic than requiring a webcam. Should it be clear that using the webcam will not provide an accurate representation of real-life marker positioning then I shall have to use the IR method as a plan B to ensure I produce a working artifact with my project.

If you have read this far, I thank you for your interest and would appreciate your comments. For those of you reading this that are not associated with Bournemouth University, please bear in mind that with exception to this post, all further posts will be much more brief and task orientated. This one is so long and waffle-ey because I am storing my ideas to later write a proposal and introduction. Further posts will be used to track progress and deadlines and to focus efforts on specific tasks.

Touchless UML Modelling

Research focus

Day 1: Plans and Thoughts