This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility: In the first year of her doctoral research, Abigail Grassick collected hours of underwater video as she studied fish communities living on bommies—individual clumps of coral—on the seafloor off the coast of Curaçao. A doctoral candidate in the field of computational biology, she tried using computer vision models to automatically identify behaviors of certain fish species.
But the existing models failed—they couldn't even keep tabs on an individual fish, let alone identify when they were feeding or trying to evade a predator. To help pave the way for better models, Grassick and a team of Cornell researchers and colleagues created WildFin, a data set composed of nine hours of video that they labeled frame by frame with fish behaviors. They hope computer scientists will use the video to develop and test models that better interpret fish behavior, and they invite fellow ecologists to release their own field videos to assist in the effort.
Grassick presented their work, "WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition," at the European Conference on Computer Vision on Sept. 8 in Malmö, Sweden. The work is also published on the arXiv preprint server. Existing models perform poorly because they are trained on very little wildlife footage, said Andrew Hein, associate professor of computational biology in the College of Agriculture and Life Sciences and co-author of the study.
"People are training these computer vision models on internet-scale data. Well, what fraction of internet-scale data are scientific imagery? The answer is, it's a tiny fraction." Better computer vision models would not only allow ecologists to answer their questions more efficiently but also enable researchers to use citizen science videos and crowdsourced data from naturalist platforms to monitor the health of ecosystems and log occurrences of rare species.
"My hope is that we really can unlock the value in citizen science footage," Hein said, "and allow biologists who are going out into all these amazing places around the world to vastly increase the amount of data that we can acquire through these really expensive and time-consuming expeditions." Hein and Grassick collaborated with Jennifer Sun, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science. Sun had successfully used computer vision models to interpret the behaviors of animals in the lab.
But the footage recorded in the wild, with its roving bands of fish, variable lighting and shifting camera angles, proved to be a bigger challenge. "When I first talked to Andrew, I never anticipated how hard his videos were to analyze," Sun said. "He has hundreds of fish, and it's hard to know if it's even the same fish in subsequent frames." Grassick and a team of undergraduates annotated the bommie videos by choosing from a list of behaviors to catalog what the fish were doing in each frame—such as foraging, feeding or fighting.
Extract — continue reading at the source.
