Where is George? Ask this software to look at the crowd

January 14, 2019

George is a zebrafish. Along with Tom and 98 other mates, George swims freely in a laboratory tank at the Champalimaud Centre for the Unknown (CCU), in Lisbon, Portugal. A camera records from above a video of all the animals' comings and goings.

Question: is it possible to identify, from the video images, which fish is which at every moment? Yes, says Gonzalo de Polavieja, principal investigator of the Collective Behavior Lab, who with his team has built a software, called idtracker.ai (where "ai" stands for artificial intelligence, or AI) that can do the job quickly and with extreme accuracy. Their results were published in the journal Nature Methods.

"The ultimate goal of our team is understanding group behavior", points out de Polavieja, whose preferred names for the fish, when he explains his work, are George and Tom. We want to understand how animals in a group decide together and learn together". And for that, it is mandatory to extract very high quality data from the videos (such as position and shape of each of the animals, as well as their individual paths) without mistakes.

Recognizing each individual among dozens of very similar ones would be impossible to accomplish for a human brain - or for that matter, a conventional computer program. "We would just go crazy trying", says de Polavieja. For large crowds, without artificial intelligence in the mix, even a powerful computer might have to run the program for years to get results. And these would probably not be very accurate.

That's where idtracker.ai comes in. The new software, adds de Polavieja, supplies the quality data that is absolutely needed to be able, in a second phase, to understand the rules that drive the animals' collective behavior.

Working in Spain four years ago, before joining the CCU, de Polavieja published, also in Nature Methods, a first version of the software, which did not resort to artificial intelligence. The results were much more modest. "We could track ten animals back then", he says.

De Polavieja and his co-authors, Francisco Romero-Ferrero, Mattia Bergomi, Robert Hinz and Francisco Heras have now tested the new AI version with groups of 30, then 50, then up to 100 zebrafish. "We didn't test more than 100 because our tank is not big enough for that". Nonetheless, using another approach to record the images, they showed that the software can identify up to 150 individual fish with very little loss of accuracy. "I didn't believe we could reach those numbers; it was a surprise", notes de Polavieja. "I thought there wouldn't be enough information in the images."

Deep learning

Idtracker.ai is composed of two so-called deep-learning neural networks and a few more conventional algorithms. A deep-learning neural network is a computer simulation of real networks of neurons in the brain that are capable of learning from experience.

Using the video images of the zebrafish in the tank, the first network in the chain is trained to tell whether every little blob visible in the images corresponds to just one animal or to several.

With this output, the second neural network is then trained to assign a name (or number) to each blob that contains just one fish - in other words, to effectively identify each individual fish. The recognition is based on the unique features of each zebrafish. "People think zebrafish are all alike, but this proves they are in fact all different from each other", notes de Polavieja.

Lastly, two conventional algorithms are used. "One is to gain some certainty about the [few individuals whose] identities are still somewhat uncertain", says de Polavieja "and the other to know which animal is which when their paths cross" - that is, when their trajectories appear superimposed on the video.

The results speak for themselves: it takes about an hour for idtracker.ai to identify each and every one of the 100 zebrafish in the video at all times with almost 100% accuracy! "If you show the network a random part of the video it has never seen before and ask it: 'who is this?', the network will correctly assign the right name - or number - to that fish 99.997% of the time", says de Polavieja. And if you ask it where George, or Tom, or any other zebrafish is at a given moment, it will find it in the crowd almost beyond the shadow of a doubt.

The team also tested the software with fruit flies, medaka fish (Japanese rice fish), ants and mice. It also works, though with smaller numbers of individuals. "Zebrafish are the best" for these studies, says de Polavieja. On the other hand, "mice are more difficult because they tend to cluster and deform".

"This is the first time such high quality data has been obtained for 100 fish", concludes de Polavieja. The team has now used the data supplied by idtracker.ai - which is freely available to everyone from http://www.idtracker.ai - to extract a set of rules that explain, in a large measure, zebrafish behavior in groups. They describe their results in another paper, which they have posted on bioarxiv.org and have submitted for publication in a scientific journal.

As for possible applications, this software and others like it allow (or will allow) for tracking persons, or to identify a given person in a crowd based on information about his or her physical appearance. "There is now a whole industry for this type of software", says de Polavieja. "People are applying these [AI] techniques to develop other similar tracking tools. But before we proved we could do it in animals, it was hard  to believe it was even remotely possible."

Champalimaud Centre for the Unknown

Related Zebrafish Articles from Brightsurf:

Zebrafish embryos help prove what happens to nanoparticles in the blood
What happens to the nanoparticles when they are injected into the bloodstream, for example, to destroy solid tumours?

Social experiences impact zebrafish from an early age
Study in zebrafish demonstrates that early social experiences have an effect on the behaviour of the fish even at an age when they are still not considered ''social''.

How zebrafish maintain efficient and fair foraging behaviours
New insight on how zebrafish achieve near-optimal foraging efficiency and fairness among groups has been published today in the open-access journal eLife.

How the zebrafish got its stripes
Animal patterns are a source of endless fascination, and now researchers at the University Bath have worked out how zebrafish develop their stripes.

Extraordinary regeneration of neurons in zebrafish
Biologists from the University of Bayreuth have discovered a uniquely rapid form of regeneration in injured neurons and their function in the central nervous system of zebrafish.

Zebrafish teach researchers more about atrial fibrillation
Genetic research in zebrafish at the University of Copenhagen has surprised the researchers behind the study.

How decisions unfold in a zebrafish brain
Researchers were able to track the activity of each neuron in the entire brain of zebrafish larvae and reconstruct the unfolding of neuronal events as the animals repeatedly made 'left or right' choices in a behavioral experiment.

'Census' in the zebrafish's brain
Dresden scientists have succeeded in determining the number and type of newly formed neurons in zebrafish; practically conducting a 'census' in their brains.

Zebrafish 'avatars' can help decide who should receive radiotherapy treatment
To date, there is no method for clearly determining whether radiotherapy will be an effective treatment for individual cancer patients.

Special cells contribute to regenerate the heart in Zebrafish
It is already known that zebrafish can flexibly regenerate their hearts after injury.

Read More: Zebrafish News and Zebrafish Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.