Catching up with Computational Media faculty research at SEMICON West

Katherine Isbister, Aviv Elor, and Eddie Melcer

You don’t usually hear shrieking at SEMICON West, the enormous annual electronics product design and manufacturing association’s annual conference. But it happened twice at this year’s conference. Both came during the “Gaming at the Hardware Boundaries” TechTalk; both during presentations by UC Santa Cruz faculty in the Computational Media department.

Professor Katherine Isbister’s Social Emotional Technology Lab (SET Lab) is looking for ways to use technology to foster relationships between people. The experience of seeing a character or another person’s biometric information change seemed to intensify the empathy their audience would have for them.

The scream came from a recording of a student project (“All the Feels”). The student was curious whether seeing a real-time feed of biometric information might help create a sense of empathy between a player and her Twitch audience. A volunteer wore a galvanic skin response monitor as she traversed the dark landscape of a horror game... Suddenly something spooked her! And she shrieked! And the subsequent spike in heart rate and breathing was displayed to her audience along with the ear-splitting screech.

Conveying vulnerability appears powerful for creating bonds. The SET Lab also created a biometric cloak for Live Action Role Playing (LARP) that would glow in a particular way when a player was experiencing stress or fatigue. The warning lights allowed strangers to approach one another and strike up conversations. Lest a loutish LARPer monopolize the conversation, another project, Lagom, used discrete haptic feedback to invisibly suggest to speakers that it was time to give someone else a chance (or to pipe up!).

Isbister also demonstrated several uses of design within in virtual reality, such as park benches and covered conversation nooks for encouraging conversation; the use of floating, benign pointer monsters to identify speakers within crowds; and the emergence of new forms of etiquette within virtual spaces such as the ritualistic petting of avatars and feeding one another.

The scream in Professor Edward Melcer’s presentation served a different purpose. This was an intentional scream. One controlling the behavior of a computer game. Melcer’s talk described what are known as “alternative controllers,” in other words, he was describing games that did not use a standard console or joystick to operate. These ranged from the confounding -- such as Melcer’s creation: a drink mixing game that requires players to swap points of view while mixing drinks -- to the sublime: an old-fashioned television whose knobs and antenna served as a control panel for a story.

Another project, Liberi, used a modified exercise bike to control a wriggling lizard game to encourage children with cerebral palsy to exercise.

Melcer (who once a research assistant in Isbister’s lab) also demonstrated a block game used for helping children learn how to code. By duplicating the functions of basic code on tangible objects (literal blocks of code to control a robot, for example) they managed to improve retention rates and comprehension.

The last UC Santa Cruz related presentation came at the very end of the conference and was really designed to prevent screaming. Incoming Ph.D. student Aviv Elor described his personal journey into rehabilitative medicine (which began with nasty fall while playing judo which ended his career as an athlete) and his work with Professor Sri Kurniawan’s ASSIST Lab’s “Project Butterfly,” a virtual reality physical rehabilitation game.

Project Butterfly is a game that rewards mimicking physical motion. The user plays a butterfly on a rainy day and movements are designed to dodge falling raindrops. Elor did a live demonstration and showed off the marionette-like exoskeleton developed in the lab.

SEMICON West shows off the latest and greatest in hardware. There were acres of elegant booths and glimmering equipment, including a gilded life-scale model of The IBM Quantum Computer beyond the TechTalks. The content hasn’t running on these powerful machines however hasn’t quite caught up. It remains weird and wonderful and… adolescent?

“We’re definitely in our awkward adolescent phase,” said developer Theresa Duringer. The future of content is under construction here at UC Santa Cruz.