A fast-paced game app where players pretend they are baggage screening officers operating airport x-ray scanners has provided researchers with billions of pieces of data in record time, according to a new article published by the American Psychological Association.
The findings, which were published online ahead of print in APA’s Journal of Experimental Psychology: Human Perception and Performance, suggest that crowdsourcing could be a game changer for psychological research, allowing scientists to answer nuanced questions that were previously out of reach.
“Mobile devices offer researchers an exciting new means to crowdsource an experiment using games that are actually tests of cognition or other brain functions,” said Stephen R. Mitroff, PhD, of Duke University, lead author of the report. “Questions that could have taken decades to answer in a laboratory setting, or that could not be realistically answered in a lab, can be examined using big data gathered in a relatively short time.”
To demonstrate the potential of mobile technology to gather data, the researchers used a common perceptual task known as visual search, which involves looking for something (the target) in a visual environment cluttered by extraneous items (distractors). The ability to consciously locate a target amongst a complex array of distractors has been extensively studied over the past 40 years, providing insight into psychological processes such as attention and learning, and contributing to our understanding of diseases ranging from Alzheimer’s to autism.
To develop the visual search experiment, Dr. Mitroff’s team partnered with Kedlin Co., the developer of the popular mobile app game Airport Scanner, which challenges players to identify illegal items in luggage passing through an airport x-ray scanner. Players view one bag at a time and tap their touchscreen to identify banned items. The game includes a logbook of illegal items (e.g., guns, hand grenades, switchblades) and legal items (e.g., ear phones, clothing), and the list expands from a handful of possible target items to hundreds as players progress through the game.
Between January 2013 and November 2014, the game provided anonymous data from more than 2 billion trials from over 7 million mobile devices that Mitroff and his research team at the Duke Center for Cognitive Neuroscience have been analyzing, the report said. Players consent to the data collection when installing the game on their smartphone or other hand-held device.
Ten years worth of data collected in one day
One research question explored through the Airport Scanner game was examining players’ ability to spot items that occurred on screen very rarely, or in less than 0.1 percent of all bags.
“When a target appears only 0.1 percent of the time, you need 1,000 trials to get just a single case of that target. Too many trials would be needed to realistically assess detection of that particular target in a laboratory setting,” Mitroff said. “With the large dataset, we were able to look at hundreds of cases for each of the nearly 30 rarest targets.”
Another experiment found that when two banned items both appeared in a bag, players were more likely to miss the second to-be-found item when the two were different than if they were identical. For example, players were more likely to miss a dynamite stick after finding a gasoline can than they were to miss a dynamite stick after first finding a dynamite stick.
“While using a game interface to assess cognitive abilities is not new to psychological research, mobile technology offers a phenomenal opportunity to examine cognitive processes on a large scale,” Mitroff said. “In 1998, 1 million trials were analyzed by Jeremy M. Wolfe, PhD, of Harvard Medical School, to understand visual search. This was a mind-blowing amount of data that took 10 years to collect. Today, we can collect over a million trials a day through Airport Scanner.”
Advantages & disadvantages of crowdsourcing
Another advantage of crowdsourcing is that it can be a less-expensive means of collecting data automatically and continuously. It also has the ability to mimic real-world situations that are difficult to address in a laboratory, the authors said.
Crowdsourcing research data does have disadvantages, the authors noted. Researchers must have the skills to create a fun game app or they need to partner with a developer. Collecting large amounts of data may not necessarily result in high-quality data, which requires carefully developing research questions to be addressed through the game. And collecting data through crowdsourcing means researchers have no control over who is playing, they said.
Even with those limitations, the authors say the potential uses of crowdsourcing in psychological research could be a watershed opportunity for the field. “At present, we are in the early stages of another breakthrough capable of pushing psychological research forward,” they wrote. “[A]s researchers, we should (responsibly and carefully) look for ways to take advantage of this opportunity.”