I am broadly interested in Human-Computer Interaction,
especially gesture-based interaction, activity-based computing and rapid prototyping tools.
Gesture Search is an
example of my recent work on gesture-based interaction, which enables gestures as a fast and alternative modality
for mobile interaction. My latest work on activity-based computing and rapid prototyping tools is embodied by
ActivityStudio, a suite of tools that I created
for designing and testing ubicomp applications in situ. It demonstrates how activity, as a larger and
richer context for design, can be supported by tools throughout the design process.
Bio · Yang is a Senior Research Scientist at Google. Before joining Google's research team, Yang was a Research Associate in Computer Science & Engineering at the University of Washington and helped found the DUB (Design:Use:Build), a cross-campus HCI community. He earned a Ph.D. degree in Computer Science from the Chinese Academy of Sciences, and then conducted his postdoctoral research in EECS at the University of California at Berkeley.
Gesture Coder: A Tool for Programming Multi-Touch Gestures by Demonstration
CHI Best Paper Honorable Mention Award
Present a tool that automatically generates code for recognizing each state of multi-touch gestures and invoking corresponding application actions, based on a few gesture examples given by the developer.
Bootstrapping Personal Gesture Shortcuts with the Wisdom of the Crowd and Handwriting Recognition
Contribute the approaches for bootstrapping a user’s personal gesture library, alleviating the need to define most gestures manually.
Gesture Search: Random Access to Smartphone Content
Present a tool for random access of smartphone content by drawing touchscreen gestures. It flattens the UI hierarchy of smartphone interfaces.
Tap, Swipe, or Move: Attentional Demands for Distracted Smartphone Input
Investigated attention demands of motion gestures in comparison with traditional interaction techniques for mobile devices.
Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures
Present Gesture Avatar, a novel interaction technique that allows users to operate existing arbitrary user interfaces using gestures. It leverages the visibility of graphical user interfaces and the casual interaction of gestures. It outperformed prior techniques especially while users are on the go.
Deep Shot: A Framework for Migrating Tasks Across Devices Using Mobile Phone Cameras
Presents a framework for migrating tasks across devices using mobile cameras. It supports two interaction techniques, Deep Shot and Posting, that enabled direct manipulation of information and work states in a multi-device environment.
DoubleFlip: A Motion Gesture Delimiter for Mobile Interaction
Designed a motion gesture for separating intended motion input from ambient motion of mobile phones. A DTW-based recognizer was built to recognize the gesture which had high precision and recall.
User-Defined Motion Gestures for Mobile Interaction
Present the results of a guessability study that elicits end-user motion gestures to invoke commands on a smartphone device, which led to the design of a taxonomy for motion gestures and an end-user inspired motion gesture set.
Experimental Analysis of Touch-Screen Gesture Designs in Mobile Environments
Investigates the impact of situational impairments on touchscreen interaction. Reveals that in the presence of environmental distractions, gestures can offer significant performance gains and reduced attentional load, while performing just as well as soft buttons when the user's attention is fully focused on the phone.
Gesture Search: A Tool for Fast Mobile Data Access
Describes a tool that allows users to access mobile phone data using touch screen gestures. Gesture Search flattens the deep UI hierarchy of mobile user interfaces and learns the mapping from gestures to data items.
FrameWire: A Tool for Automatically Extracting Interaction Logic from Paper Prototyping Tests
Presents a tool for automatically extracting interaction logic from the video recording of paper prototype tests. FrameWire generates interactive prototypes from extracted interaction logic.
Protractor: A Fast and Accurate Gesture Recognizer
Pseudo Code Shipped to the Android Gesture Library
Presents an algorithm for recognizing drawn gestures. Protractor employs a closed-form solution to find the best match of an unknown gesture given a set of templates.
Beyond Pinch and Flick: Enriching Mobile Gesture Interaction
Presents the design of a toolkit for gesture-based interaction for touchscreen mobile phones. Introduces the concept of gesture overlays.
CHI Best Paper Honorable Mention Award Video & Download
Presents a tool that allows designers to incorporate large-scale, long-term human activities as a basis for design, and speeds up ubicomp design by providing integrated support for modeling, prototyping, deployment and in situ testing.
Cascadia: A System for Specifying, Detecting, and Managing RFID Events
Cascadia is a system that provides RFID-based pervasive computing applications with an infrastructure for specifying, extracting and managing meaningful high-level events from raw RFID data.
Design Challenges and Principles for Wizard of Oz Testing of Ubicomp Applications
Presents the $1 algorithm for gesture recognition and a comprehensive study that evaluates $1 against two other popular gesture recognition algorithms: Dynamic Time Wrapping and Rubine Recognizer. The study indicated that the $1 recognizer though simple outperformed its peers in both accuracy and learnability.
Gestures without libraries, toolkits or Training: a $1.00 Recognizer for User Interface Prototypes
Invited to the SIGGRAPH UIST Reprise Session
Presents the $1 algorithm for gesture recognition and a comprehensive study that evaluates $1 against two other popular gesture recognition algorithms: Dynamic Time Wrapping and Rubine Recognizer. The study indicated that the $1 recognizer though simple outperformed its peers in both accuracy and learnability.
BrickRoad: A Light-Weight Tool for Spontaneous Design of Location-Enhanced Applications
Presents a tool for testing location-based behaviors without specifying interaction logic. The tool explores the extreme of Wizard of Oz approaches for designing field-oriented applications, i.e., testing with zero effort beforehand.
Design and Experimental Analysis of Continuous Location Tracking Techniques for Wizard of Oz Testing
Presents various Wizard of Oz techniques for continuously tracking user locations.
Informal Prototyping of Continuous Graphical Interactions by Demonstration
Invited to the SIGGRAPH UIST Reprise Session Video
Presents a tool for creating continuous interactions using examples. Discusses the algorithms for learning continuous interaction behaviors from discrete examples, without using any domain knowledge.
Experimental Analysis of Mode Switching Techniques in Pen-based User Interfaces
Conducted a study to compare different mode switching techniques for pen-based user interfaces. The study revealed that bi-manual based mode switching outperformed other techniques.
Topiary: A Tool for Prototyping Location-Enhanced Applications
Topiary is a tool for rapidly prototyping location-based applications. It introduces a Wizard of Oz approach for testing location-based applications in the field, without requiring a location infrastructure.
Tom Ouyang, Google 2011 Summer Intern
MIT, worked on crowd and HWR-based bootstrapping for gesture recognition
Hao Lu, Google 2010, 2011 Summer Intern
University of Washington, worked on Gesture Avatar, Gesture Coder
Tsung-Hsiang Chang, Google 2010 Summer Intern
MIT, worked on Deep Shot
Jaime Ruiz, Google 2010 Winter Intern
University of Waterloo, worked on double flip & motion gesture design
Evan Welbourne, University of Washington 2006-07
Spring 2008: Guest Lecturer UW iSchool INFO 498: Special Topics: Input & Interaction
Lectured for Information School Professor Jacob Wobbrock on "Pen-Based Interaction"
Autumn 2007: Instructor for UW CSE 373: Data Structures and Algorithms
Taught a Computer Science & Engineering course for 69 undergraduate students