Researchers Aim to Make Useful Data More Available Via Mixed Reality

Imagine seeing the strength of a Wi-Fi network just by glancing at a building. Or walking down the street and reading customer reviews of businesses and shops that are digitally displayed on each storefront.

This could soon be commonplace with the expanded use of mixed reality (MR), wherein information is overlaid onto your field of view via a headset, clip-ons for glasses, or a handheld device, in order to visually augment the physical world with valuable streams of data.

Niklas Elmqvist, a professor in the College of Information Studies (iSchool), is working with a team to expand the use of MR, building on ideas already in use while exploring challenges that still need to be addressed like biased data and user costs.

With funding from the National Science Foundation, Elmqvist says the project—called DataWorld—is “about piercing the veil to the world of data that's really overlaid on our natural real world.”

“We could think of the data that's being collected about everything around us actually being this bubble in the world itself,” he says.

The DataWorld team includes Elmqvist, Andrea Batch, a fourth-year doctoral student in information studies, and Sungbok Shin, a first-year doctoral student in computer science.

They plan to use the University of Maryland campus as a test location to conduct research, gather data and learn the best ways to make DataWorld scalable for future users.

To build their framework, the DataWorld team has partnered with the Do Good Institute, a campus resource housed in the School of Public Policy that provides opportunities to address significant social issues. Thanks to this partnership, DataWorld will include themes of public safety and historical, civic and environmental awareness.

Because DataWorld does not yet exist, it’s impossible to show users exactly how data will appear in their field of view, the researchers say. But Elmqvist anticipates creating consciously placed, interactive data hierarchies that will be visible but not distracting, and will help users easily navigate through the data that surrounds them.

The team is currently working with data that’s readily available on campus like UMD Department of Public Safety alerts. Next steps include drawing on other data sources such as the Diamondback student newspaper, as well as inviting the campus community to participate by submitting geotagged social media posts, dining hall reviews and more.

The idea for DataWorld grew from Elmqvist’s interest in combining his work in two-dimensional graphics with 3D technology.

Unlike 2D, which is restricting, a 3D environment can be used like a “canvas,” says Emlqvist, who has an appointment in the University of Maryland Institute for Advanced Computer Studies and is director of the Human Computer Interaction Lab. “It's a radically different way of thinking about computing. That's the power of mixed reality,” he says.

The team has some experience studying data analysis in virtual reality, but MR is a new medium for the researchers.

“We’re trying to start from scratch,” Shin says. “We want to test in which setting mixed reality is effective. We’re very eager to solve the question.”

To test their data in MR, they will be using hands-free, head-mounted displays, like the Magic Leap and the Microsoft Hololens2. But MR equipment is expensive. The Magic Leap, for example, retails for around $2,300. This “presents a barrier to people who might want to buy it out of their own personal budget,” says Batch. So they also plan to create a more traditional display, like a smartphone application.

Another challenge is that data is messy. It can be wrong, contain duplicates, include personal bias or be incomplete. Historical data, for example, can leave out or misrepresent marginalized groups. While the team has not yet encountered these problems, Elmqvist recognizes the importance of utilizing the available data ethically.

“You have to be careful,” he says. “Information is usually double-edged, and you can't just say all information is benign.”

—Story by Colleen Curran