Inspired by the accidents she witnessed as an engineer in the oil fields, the CITRIS principal investigator and UC Santa Cruz scholar of human-centered design puts her expertise to work serving vulnerable populations.
“Since starting my career, I have been working on a variety of assistive technologies, in the forms of games, computer and tablet applications, and virtual reality,” said Sri Kurniawan, professor of computational media at the University of California, Santa Cruz, and a principal investigator (PI) at the Center for Information Technology Research in the Interest of Society and the Banatao Institute (CITRIS). “But it all has one common goal of improving the quality of life for all people.”
Kurniawan joined the Department of Computational Media in the Baskin School of Engineering at UC Santa Cruz in 2007, and she has helped to develop an immersive CAVE facility for creating virtual reality (VR) environments and to launch the country’s first graduate program in serious games in the years since.
Through the Assistive Sociotechnical Solutions for Individuals with Special needs using Technology (ASSIST) Lab, Kurniawan and her students design low-cost, reliable assistive systems that incorporate interdisciplinary best practices and rely heavily on real-world user feedback.
After graduating with a bachelor’s degree in electronics engineering from Satya Wacana Christian University in Indonesia, Kurniawan began her career as a field engineer in the oil industry. While on the job, she noticed how the heavy, dangerous machinery and ineffective user interfaces affected workers, sometimes with tragic consequences.
“Accidents happened,” Kurniawan said. “I saw firsthand how people who got injured in the oil field became physically impaired and lost their jobs. That was my very first call to start working on assistive technology.”
Kurniawan decided to return to school after five years in the field. She received a master’s degree in industrial engineering and engineering management with a focus in human factors from the Hong Kong University of Science and Technology, and then a Ph.D. in industrial and manufacturing engineering with a focus on human-computer interaction at Wayne State University. She spent six years on the faculty at the University of Manchester in the United Kingdom before joining UC Santa Cruz.
Helping stroke survivors regain speech
Not long after arriving on campus, she came across one of the very first calls for proposals for the CITRIS Seed Funding Program. Inspired by its mission to empower UC researchers to solve large-scale societal issues, she decided to pursue a project that had first come to mind back in England: using recycled cellphones to help stroke survivors with the language disorder aphasia regain their speech.
According to Kurniawan, at the time of the project’s inception in 2008, Malaysia was experiencing an extreme shortage of trained speech therapists and needed about 10 times as many clinicians as were available. To help address the imbalance, she had the idea to repurpose old cellphones as rehabilitation tools by developing a robust, user-friendly virtual speech therapy platform, loading it onto the phones and then offering the devices to stroke patients, especially those living in rural areas far from clinics.
Kurniawan teamed up with UC Santa Cruz psychologist Dominic Massaro to create the speech therapy platform using a virtual talking head model Massaro had built and nicknamed Baldi. By animating Baldi to move its lips, tongue and jaw while articulating words in Malay through a text-to-speech synthesizer, the program could coach patients to mimic Baldi’s mouth positioning in a set of speech therapy exercises.
“With the same cellphone, a real human speech therapist could also call the patient — and less frequently than without the platform,” Kurniawan said. “Instead of meeting with a patient three times a week, the speech therapist could call in once a month to assess if a patient has done their homework.”
The speech therapy project also received support from a Microsoft Research gift.
Prioritizing vulnerable populations
Kurniawan has continued to prioritize the well-being of vulnerable populations, from stroke survivors and people with disabilities to people at risk from natural disasters, with five additional CITRIS-funded projects in subsequent years, including one from the campus-specific Interdisciplinary Innovation Program (I2P).
In 2011, she led a CITRIS Seed Award project with colleagues at UC Santa Cruz and UC San Francisco to design a digital childbirth simulation, in the form of a serious game, to train birth partners and nursing midwifery students how to best support laboring parents during delivery.
The following year, she and researchers at UC Santa Cruz and UC Davis Health received a 2012 Seed Award to develop an autonomous telemonitoring robot to help older adults live safely in their own homes for longer.
In 2020, Kurniawan responded to a special CITRIS pandemic-response call for proposals with a project to ensure that older adults, who often have specific dietary needs due to chronic health conditions, could still access healthy food amid stay-at-home orders.
“It was very hard to find fresh food if you didn’t know how to order on apps like Instacart or DoorDash,” Kurniawan said, “because supermarkets were shut down.
“A lot of older people at high risk from COVID-19 were not able to get fresh food, and canned and frozen foods tend to be high in sodium, preservatives and other unhealthy ingredients.”
After consulting with two groups in their local community, Kurniawan and co-investigator Nancy Chen, a professor of anthropology at UC Santa Cruz, chose to collaborate with Teen Kitchen Project, a nonprofit that teaches young people to cook medically tailored meals to fit the nutritional needs of people with illnesses such as cancer, congestive heart failure and diabetes. The organization then delivers the food to the homes of Santa Cruz County residents in need.
“She has a lot of knowledge about a lot of different things. She’s also super friendly, which helps with making connections to people who can give us feedback.”
Alison Crosby
When the UC Santa Cruz team launched their partnership, Teen Kitchen Project lacked a user interface that allowed recipients to select their meals or view nutritional facts about the dishes, such as protein content and calorie count.
Kurniawan and Chen worked on a mobile application called Teen Kitchen Companion, which allows meal recipients to choose meals according to their health needs and taste preferences. Nutritional facts are also listed for each dish, offering recipients full transparency.
“If you don’t know how much sugar or saturated fat you’ve had, long-term conditions like diabetes can be hard to manage,” Kurniawan said. “We thought it would be helpful for users to know exactly what they had for the past week, so that they knew when it was a good time to ‘treat’ themselves.”
Immersive training for wildfire evacuations
Quick on the heels of COVID-19, in August 2020, the CZU Lightning Complex fires burned over 85,000 acres of land in Santa Cruz and San Mateo counties and destroyed nearly 1,500 buildings, necessitating the evacuation of more than 70,000 people, including the entire UC Santa Cruz campus.
“I had a number of colleagues who lost their homes,” Kurniawan said. “They told me that they were really unprepared because the order to evacuate happened in the middle of the night.
“A lot of people were caught off-guard. Some of them couldn’t find their pets and lost them forever.”
Kurniawan had been working on a series of VR simulations and serious games for various purposes. After the fire, she reconnected with her former student Spencer Castro, now an assistant professor of management of complex systems at UC Merced. They discussed the fire risk in their respective locations and the challenges of preparing for a sudden evacuation order, and they decided to design an immersive environment to simulate a wildfire evacuation.
With support from a 2021 CITRIS Seed Award, the team created a VR game that places players in a simulated house and gives them a detailed list of important items — including a dog — to gather before the clock runs down and it’s time to evacuate.
“We chose virtual reality because it’s much more immersive — you’re wearing a headset, and we use the smell of smoke to make it even more realistic,” Kurniawan said. “The idea is that the more realistic the simulation is, the closer your decision-making will be when you are exposed to a real evacuation situation.”
UC Santa Cruz doctoral candidate and ASSIST Lab member Alison Crosby was drawn to the project by her own experience with Northern California wildfires. Soon after she started her fourth year at UC Merced, her hometown, Paradise, was destroyed by the 2018 Camp Fire.
Crosby recalls the confusion after power lines and cell service went down during the fire and the concern that she felt for the family members and friends who were evacuating. She values the expertise and perspective that Kurniawan brings to the work.
“She has a lot of knowledge about a lot of different things, which is great for coming up with ideas,” Crosby said. “She’s also super friendly, which helps with making connections to people like firefighter volunteers who can give us feedback on the simulation.”
A ‘titan’ of assistive tech
Kurniawan also offers her expertise and guidance to fall 2021 CITRIS Foundry venture Immergo Labs, a telehealth company founded by three UC Santa Cruz alumni that uses VR as a platform for physical therapy. She and fellow CITRIS PI Mircea Teodorescu currently serve on the startup’s advising team.
Kurniawan was named a 2023 Titan of Tech by Santa Cruz Works, a nonprofit that works to cultivate a vibrant entrepreneurship ecosystem in Santa Cruz County, and a 2024 Woman of Influence by the Silicon Valley Business Journal. Her nominator for the Santa Cruz Works honor, Immergo Lab co-founder and ASSIST Lab graduate Aviv Elor, called her a “leading voice” and “real visionary” for her commitment to making emerging technologies inclusive.
And her vision for inclusion is intersectional and intentional. As of December, Kurniawan serves as the campus’s senior faculty lead for the Action Collaborative on Transforming Trajectories for Women of Color in Tech, an initiative of the National Academies of Sciences, Engineering, and Medicine that aims to improve tech workforce development for women of color.
Crosby also appreciates Kurniawan’s dedication to building rapport and community.
“The biggest thing I’ve learned from her is to keep connections open,” Crosby said. “It’s good to be on friendly terms with a bunch of people, because getting different perspectives is always helpful for research and technology design.
“It’s a quality that I will implement throughout the rest of my research.”