It was a love of toys, a shared appreciation for the intrinsic beauty of physical objects and a recognition of their absence in the daily computer interactions of a world that currently spends most of its time gazing at and touching two-dimensional glass that propelled a team of Microsoft researchers in Cambridge, UK and Redmond, WA to embark on a quest to blur the divide between the digital and physical worlds.
The decidedly tangible fruit of their efforts is Project Zanzibar, a completely new sensing platform in the form of a flexible, portable mat that has the ability to locate, sense and communicate with objects as well as sense a user’s touch. The Project Zanzibar mat combines capacitive sensing and Near Field Communication (NFC) in a novel way, enabling multi-touch and hover gesture input to coexist with physical object manipulation and control. The Project Zanzibar research platform also introduces the power of portability in a tangible user interface because rather than provide its own display, it takes advantage of existing devices such as tablets. Roll it up, stow it and break it out at a picnic or on a train trip. Or in any room in the house that has a screen.
“The question driving us over a number of years was how you make interacting with a computer a very natural, invisible thing,” said Microsoft Senior Researcher Nicolas Villar. “Our fascination was with making the technology invisible so that you get the power of technology without it feeling like you are using technology.”
Spotlight: On-demand video
AI Explainer: Foundation models and the next era of AI
Explore how the transformer architecture, larger models and more data, and in-context learning have helped advance AI from perception to creation.
The scope of their achievements, detailed in a paper to be presented later this month at CHI 2018 in Montreal, is impressive. Supplementing the flexible form-factor Project Zanzibar mat, are hardware design patterns for tangible objects that build on NFC, leveraging and extending the capabilities of the mat.
“We began with a simple thought: what if we could blur the divide between the physical and digital worlds? What if you could play with toys and cards and blocks, while watching your actions come alive onscreen?” – Project Zanzibar Research Team
Any new research platform worthy of the name provides the ability to support a variety of compelling applications. The Project Zanzibar research team obviously intended not to disappoint anyone, developing a set of exciting scenarios in the play and learning spaces, two areas that the team is passionate about and in which the benefits of tangible interaction are well accepted. The team also validated the Project Zanzibar research platform by sharing deployments with professional game developers and incorporated the resulting experiences in creating physical/digital hybrid games.
With the portability and flexibility of the Project Zanzibar mat and its ability to leverage the ubiquity of equally portable screen technology, combined with the prevalence of homes with screens conceivably in every room, the team feels the moment for TUIs is at hand.
Scenarios
In their pursuit of a vision of humans interfacing with computers in which the focus is no longer on the computer or its peripherals, the Project Zanzibar team went the extra mile in imagining and prototyping delightful scenarios that convey a very compelling technological story.
Play
Every physical object in the Project Zanzibar world has a globally unique ID, enabling data to be stored for each unique object. This introduces the concept of a toy or object learning and possessing its own history and the capability to tell its story. Collectible cards can gain value over time. Dolls could conceivably “grow up” alongside a child, with shared experiences from playtime remembered and relatable (that is, replayable) over time.
Movie maker. Off-the-shelf toys were used to demonstrate the flexibility of Project Zanzibar for different types of play activities.
Project Zanzibar invites children to bring their toys into an interactive experience and watch them come alive through sound, visuals and special effects. The team had observed kids spending hours making stop-action movies to share online. They envisioned the Project Zanzibar movie maker scenario to let children tell stories by physically interacting with their toys on the Project Zanzibar mat while at the same generating an animated movie.
“The physical world is such a rich place, abundant with dimensionality and materiality. Our vision is to tap into that richness and imbue it with that digital magic and our hope is that Project Zanzibar is a step in that direction.” – Haiyan Zhang, Project Zanzibar Researcher
Kids can tell stories by manipulating toys and props on the mat to control corresponding graphical avatars, previewing and capturing the result as a movie on the screen of a connected device. Camera and light source objects frame and light the mise-en-scène. Additional props can be used to change the setting of a virtual scene. For example, a palm tree can change the virtual environment to a tropical setting. Movie maker is thus a story authoring and storytelling prototype, envisioned for children aged 5 to 12. In this age group, socio-dramatic or pretend play are prevalent play patterns, with children experimenting with role playing and ascribing unique personalities to their toys.
Toybox. Off-the-shelf toys were used to demonstrate the flexibility of Project Zanzibar for different types of play activities.
The toybox uses sound effects and audio dialogue to complement physical play. The toybox includes figurines for different characters (pirates and naval officers), the combination and proximity of different characters on the mat triggers lines of dialogue from a story — for example, two pirates will banter if placed close to one another, while a pirate will hail an officer with a friendly challenge. Through its globally unique ID, each pirate toy is unique, building on its narrative with each play session.
The sensing ability of the mat can be extended by the toy pieces, for example when the pirate climbs the tower, the mat detects when he reaches the top. Toy pieces can also become inputs; when the pirate fires his cannon, players hear the boom and later a splash when the cannonball lands in the sea.
Learning
The Project Zanzibar team exhibited an equal passion for imagining powerful opportunities for the research platform in the educational space, inspired by the Montessori Method. Developed over the late 19th and early 20th century, Maria Montessori’s theories of child development stressed the role of physical activity in absorbing academic concepts and practical skills. The Montessori method also emphasizes the importance of self-directed activity and adapting a child’s learning environment to his or her developmental level.
“The importance of multi-sensory learning is especially apparent in the Montessori method of teaching, where children use physical objects and self-directed activities for all subjects of learning. We took traditional Montessori exercises for young children and extended them with digital content and feedback using Project Zanzibar.” – the Project Zanzibar Research Team
The team took inspiration from traditional Montessori exercises for young children and extended them with digital content and feedback using Project Zanzibar. For example, the Letter Plates Montessori exercise provides feedback as words are formed and tactile letter-shapes are traced. Paper overlays on the Project Zanzibar mat do not interfere with functionality and provide additional context to the exercise.
Montessori-inspired letter plates with Project Zanzibar mat and tablet
The team envisions a rich family of educational apps including spelling and simple coding exercises using blocks, again with or without screens. By allowing kids to incorporate their tactile, natural world actions and senses when interfacing with computers, learning can conceivably be enhanced.
Implementation: Making the Tangible Real
The number of innovations and implementation decisions present in the Project Zanzibar mat and research platform defy encapsulation in a mere blog article and interested readers are encouraged to download the comprehensive technical paper by team members Villar, Zhang and their Microsoft Research colleagues Daniel Cletheroe, Greg Saul, Christian Holz, Tim Regan, Oscar Salandin, Misha Sra, Hui-Shyeong Yeo, and William Field.
Project Zanzibar incorporates scalable NFC coverage that can localize tag, capacitive sensing to detect object footprints, touch, and hover input and a tracking system that fuses NFC-based and capacitive-based tracking information to report interaction events. Manufacturing innovations led to a rollable mat surface. Hardware design includes patterns for tangible objects in order to track their rotation, harvest power, enable on-object input/output capabilities, extend sensing outside the bounds of the mat surface, and detect the presence and order of stacked objects.
Project Zanzibar Mat: Some Technical Detail
The Project Zanzibar mat prototype is designed for self-contained operation— all sensing and processing happens in the mat itself; high-level interaction events are sent to connected devices via USB or Bluetooth.
The mat incorporates scalable and localized NFC coverage and has the ability to energize and communicate with multiple tags up to 30mm above the surface of the mat. Innovations such as capacitive object footprints and touch and hover recognition detect gestures such as swiping across the mat or drawing shapes in the air.
Additional innovations solve problems in NFC and capacitive coexistence; for example, NFC communication and capacitive sensing must happen concurrently for the touch and object tracking system to work.
The tracking system fuses data from the NFC and capacitive subsystems to generate high-level interaction events.
Design Patterns for Project Zanzibar Tangibles
The Project Zanzibar team strove to ensure experiences that went beyond mere interaction with simple tokens, which offer only presence, location and identity as input. The researchers developed design patterns and hardware reference designs that simplify the creation of tangible objects with a range of flexibility across cost, complexity, and interactive capabilities. Simple NFC sticker tags were determined to support innovative design decisions across rotation, power harvesting, input/output, and extending and stacking.
One reads all this and senses that the moment for tangible user interfaces indeed is upon us. And if the passion of this research can be as inspiring in the outside world, then we may indeed be at a turning point in the realization of natural interactions with computers that lead to the next era of how we think of – or don’t think of! – computers when we are using them to create, learn, play and build.
Project Zanzibar has been a research exploration and the researchers are glad to share its publication with the wider HCI community. Microsoft Research pushes the boundaries of research in diverse areas, although not all projects may be pursued beyond the research prototype stage.
Hardware and devices 
Human-computer interaction