Mobile ShopAssist Rainer Wasinger, DFKI GmbH

Description:

The Mobile ShopAssist is a Pocket PC application used to demonstrate the use of natural language technologies within a stereo-typical shopping environment.  Mobile and multimodal input interaction is the central theme of this application. Through the use of a blackboard architecture and modality fusion techniques, users can interact flexibly with both physical real-world objects on a shelf, and their digital-world counterparts shown on a Pocket PC display. Interaction can take place in the form of intra- and extra- gestures, speech, handwriting, and mixed-modal combinations based on the aforementioned modalities. Objects are also capable of initiating a dialog with the user when picked up from the shelf. The application caters for English and German speakers, and it was demonstrated with a product base of digital cameras at CeBIT 2005 and CeBIT 2006.

This work was carried out at the DFKI GmbH and the University of Saarland, under the project COLLATE/COLLATE II (Computational Linguistics and Language Technology for Real Life Applications), which was funded by the German BMBF (01.01.2001 - 31.12.2006). In particular, the work falls under the theme M3I (Mobile and MultiModal Interaction), and it has the goal of bringing people closer to cutting edge language technology.

The Movie:

• Mobile ShopAssist [34 MB]:

Interaction Video Clips:

• Speech & Speech [2.2 MB]
• Speech & Intra-Gesture [2.2 MB]
• Speech & Extra-Gesture [2.5 MB]
• Speech & Intra- & Extra-Gesture (mixed reality) [4.7 MB]
• Intra- & Intra-Gesture [2 MB]
• Intra- & Extra-Gesture [2.5 MB]
• Handwriting & Intra-Gesture [6.8 MB]
• Handwriting & Extra-Gesture [3.1 MB]
   
• Overlaid combinations [15.6 MB]
   
• Anthropomorphized Objects [3.8 MB]
• Anthropomorphized Objects (mixed voices) [4.3 MB]
   
• Integration of the FLUIDUM Search Light:
  Scan (based on optical markers) [13 MB]
  Finding a product [3 MB]
   
• Integration of PADs (Public Associated Displays):
  PAD product interaction [17 MB]
  PAD example of display coverage [2 MB]
  PAD use in a multiple user domain [7 MB]
   
• Integration of the Presentation Output Planner (use of symmetric and context-based templates):
  Symmetric Multimodality [1.5 MB]
  Synchronization (speech and deictic gesture) [2.7 MB]
  Spoken output formats (anthropomorphized): Long (Feature+Object+Value) [2.3 MB], Medium (Feature+Value) [2 MB], Short (Value)
  Spoken output speed (symmetric): Normal [2.2 MB], fast [2 MB]
   

Modality Fusion Architecture:

• MobileShopAssist.pdf

Photo Gallery:

For more information on the Mobile ShopAssist, please see my publications listing, which is now at:
http://sydney.edu.au/engineering/it/~wasinger/