Wild Chimpanzee Foundation
Wild Chimpanzee Foundation & DrivenData’s ‘Project Zamba’
Vision: Automated, accurate, and accessible species detection using machine learning on
camera trap videos for conservation.
ZambaCloud is a user-friendly, web application, where users can upload their camera trap videos, which are automatically processed in the cloud, and receive a spreadsheet with which species are most likely present each video, allowing researchers to weed out false triggers and get straight to the videos of species of interest.
Project Overview
Data Challenge
Labeling training data, AI, faster analytics; Custom conservation science platform
Data needs
Cloud infrastructure, Training data
Data practices
Open Source, Predictive modeling, Decision Support, Humans-in-the-loop AI
target users
Conservation Scientists
tech team
Project TEAM
Partnerships & Collaboration
The Wild Chimpanzee Foundation (WCF), have been collaborating with the Max Planck Institute for Evolutionary Anthropology (MPI-EVA), German Center for Integrative Biodiversity Research (iDiv). They worked with DrivenData.org to build ZambaCloud, with support from a lead data scientist and consultation from a technical advisor.
hypothesis
Camera traps are widely used in conservation research to capture images and videos of wildlife without human interference. They are a powerful tool but they are not being used to their fullest potential for species detection and abundance estimation.
“The really motivating question we’re trying to answer is species abundance. Generally we care a lot about whether populations are increasing or decreasing. We want to know if our interventions are working, or if we need to be doing more”
Introduction
There are fewer than 65,000 critically endangered western chimpanzees left in the wild, and those that remain are under threat from:
Wild Chimpanzee Foundation’s (WCF) primary goal is to enhance the survival of the remaining wild chimpanzee populations and their habitat, saving the behavioral diversity of the fascinating species. They operate in the west African countries of Liberia, Guinea, and Ivory Coast. The group conducts biomonitoring to answer the following questions:
READ More:
HOW IT Started:
“ZambaCloud is such a great example of why ML can be useful. It is not replacing the work, but it instead does the job that nobody wants to be doing which is watching videos where nothing is there. Conservationists are not used to working on command line. ZambaCloud runs this under the hood.”
WCF and DrivenData.org have been working together since 2018, when WCF joined a machine learning competition that DrivenData sponsored. Before the competition, WCF had used graduate students to complete the arduous task of watching and labeling countless hours of blank videos interspersed with just a few interesting ones with chimpanzees. They desperately needed to find a faster, less labor intensive way to label these videos, and by the end of competition they had built the first version of their ML model, ZambaCloud.
An important differentiator for ZambaCloud is that it works with camera trap videos rather than images. Videos are becoming more widely used in the field, as they afford a richer capture of animal behavior and are more effective for gathering inputs into ecological models (such as animal sex, size, age, and/or individual identification). There are a number of machine learning tools for detecting animals in camera trap images yet none of the existing tools support video, in part because video classification is a much harder technical problem.
Approach
The team set out to build on the existing functionality in ZambaCloud to make it easier for users to train custom models on new locations or for different species categories of interest (beyond chimps), transforming ZambaCloud from a single purpose application to one that supports the diverse needs of conservationists.
The team’s Tactical roadmap is a detailed timeline of steps taken through the Patrick J. McGovern Foundationa’s accelerator program. Tools used are linked in summary below.
The ZambaCloud platform is open source and their website has interactive tutorials explaining how users can train models to work with their own unique camera trap video datasets.
Approach details
Training Data:
The team worked with 15,000 videos to train an earlier version of the ZambaCloud platform; this time they had access to a dataset of 150,000 videos to train their models on. A challenge with video is that there are so many frames in each video to analyze. This work needs to be done ex-ante to identify the relevant part of a video before classification. The requirements dictate that this has to be both relevant and fast.
“The dataset preparation took a long amount of time. In some ways this was a pivotal point for the team, and they were forced to deprioritize some of the user-facing conversations to complete preparation. In the end, their expanded training dataset led to significant improvements in the accuracy of models.”
“We were initially only using researcher labeled videos. Our crowdsourced labels were pretty good but could hamstring model performance. Our dataset had 125,000 researcher labeled videos. We added another 100,000 that were crowdsourced videos that had high agreement with our standards, which gave us high confidence in the accuracy of the labels.”
An important differentiator for ZambaCloud is that it works with camera trap videos rather than images. Videos are becoming more widely used in the field, as they afford a richer capture of animal behavior and are more effective for gathering inputs into ecological models (such as animal sex, size, age, and/or individual identification). There are a number of machine learning tools for detecting animals in camera trap images yet none of the existing tools support video, in part because video classification is a much harder technical problem.
Predictions:
After running videos through a model on ZambaCloud, users receive a spreadsheet with probabilistic predictions of how likely various species are to be present in the video. There are two main pathways for what happens next:
Blank Detection
This makes it easier to use ZambaCloud on new species and geographies.
Eliminating blanks blog post & videos >
“Videos with no animals often account for greater than 50% of captured videos. Manually weeding out these videos is especially time intensive as it requires watching the full video and carefully screening for animals. Instead, researchers can run the videos gathered from SD cards in the field through Zamba Cloud to determine how likely each video is to be blank. Videos above a certain blank probability threshold can then be confidently tossed out, sight unseen. This can save valuable viewing time, storage space, and data transfer time. We evaluated the models on a set of 30,000 videos from the jungles of Central, West, and East Africa and found that we could remove more than 80% of blanks while losing fewer than 10% of non-blank videos.”
Frame Selection
An object detection approach to find the frames that are most likely to contain an animal, was trained on over 1 million images.
species classification:
Trained on over 250,000 videos, which is likely one of the largest camera trap video datasets used to date for machine learning, yielding a 20 point gain in accuracy over previous models.
“This allows a user to perform a targeted search for animals of interest. For example, if we are studying behavior diversity in chimpanzees, it is useful to find the maximum number of videos showing chimpanzee behavior while minimizing viewing time. By watching the videos in descending order by the probability that they contain a chimpanzee, we can find what we are looking for much more quickly than if we chose videos at random. We find that chimpanzees obey something like an 80-10 rule, where we can find 80% of the videos showing chimpanzees after watching only 10% of the videos. The efficiency of the search depends on the fraction of videos showing the target species, which varies across species and from one location to another. Pangolins are much less common than chimpanzees; less than 0.3% of videos in our dataset contain pangolins. Nevertheless, a targeted search finds almost 80% of the pangolins after viewing less than 7% of the videos. This amounts to a dramatic reduction in the number of hours spent watching camera trap videos.”
Utilizing the cloud infrastructure supported by the Accelerator, the team retrained these models using dramatically more data. Training on a larger and more diverse dataset is one of the highest leverage ways to improve model accuracy across species and locations.
Models were retrained on over 15 times the amount of previously used data, and the newly released models demonstrate over a 20 point gain in top-1 accuracy, along with high precision and recall metrics especially for larger species.
“Accurate models are important, but the work can’t end there. Users need to be able to access these models in a way that makes sense for them. We recognize that most conservationists don’t want to futz around with github or need to write any lines of code. This is why all of the models are available through the point and click interface of ZambaCloud (in addition to the underlying open-source zamba python package).”
data fluency
ZambaCloud is a Wild Chimpanzee Foundation project, supported & maintained by
DrivenData. DrivenData had high data fluency prior to the accelerator program, but the WCF team’s fluency with ML improved significantly, they went from having no experience with AI/ML to running all of the tests for the ZambaCloud platform.
data tools
Data Cleaning . . .
Data cleaning is performed in jupyter notebooks on a local machine and uses libraries such as pandas and cloudpathlib to work with files that are on s3.
Data Transfer . . .
For this we use a bash script run on an ec2 machine. As part of this script, goofys is installed with wget and used to mount s3 bucket as directory. In the bash script, lftp is used to copy files to that directory.
repo organization
We use cookiecutter data science to organize all of our projects. This is a standardized structure that facilitates reproducibility.
python package dev. . . .
Commonly libraries used include typer (for command line interface), pytorch lightning (for modeling), pytest (for testing). Github action workflows are used to run the test suite on all PRs into the main branch. Ffmpeg underlies video loading.
Documentation . . .
Docs are written in markdown and deployed to the site using Netlify.
model training and inference. . .
This is performed on an ec2 machine on which zamba is pip installed. Videos are stored in an s3 bucket and goofys is used to mount the data so we can access it as if it were local to the machine. Tensorboard is used to monitor train loss and validation metrics during training.
Django application development . . .
ZambaCloud is a django web application deployed on Heroku. The compute resources are set up on Azure and managed by Kubernetes.
Evaluation and visualization . . .
Model predictions are written out to CSV files as part of inference. We then use templatized jupyter notebooks to produce visualizations on key metrics such as overall f1 and accuracy scores and species level f1, precision, and recall. For plotting, we generally use matplotlib.
Species Detection Models . . .
European, Time_Detection, Slowfast, Megadetector, YOLOX, densepose for segmentation.
Lessons Learned
Precision and recall are critical elements of a high performance modeling framework.
See how the team quantified these in their blog post
REliability is a tough problem, but an important one
Reliability is one of the hardest but most important things to achieve for machine learning tools to become an integral part of conservationists’ workflows. Through the PJMF’s Data Practice Accelerator, we’ve been able to make Zamba Cloud more robust and resilient.
Carefully consider tradeoffs
Questions that should be knowable are not really knowable in training ML algorithms – you must consider tradeoffs.
“If you have a balanced dataset, you’ll be more likely to get things right. But you also may have rare cases to deal with, and these are often hard to communicate. You want to give better guidance but you can’t always get something clear cut. The PM led our team with tests, datasets, and visuals to clarify where we had incorrect assumptions.”
Aligning on terminology is critical
Make sure your team has standard terms for training set, test set, holdout set, and prediction set. Be consistent and clear in what you mean and use the same language. A lot of times misunderstandings can happen when you are not clear on these.
Frame selection work was a larger lift than anticipated
Rather than just retraining the existing model architecture with a larger set of frames, the team chose to supplement the training set with frames for rare species and also experiment with various model architecture to determine which one gave us the highest accuracy for a given compute time.
“We prioritized this work given that poor frame selection can severely hamstring the performance of species and depth models (i.e., if the animal is not in the selected frames, there is no way to determine the species). However, this meant that we needed to scope down other work. We decided to limit the ZambaCloud work to just the needed fixes to ensure training and inference jobs were working consistently. Unfortunately, we did not get to engage with current users or build out a broader user base, all of which will be important in the future for ensuring the sustainability of ZambaCloud.”
Watch your azure credits
Azure credits underlying Zamba Cloud expired during the course of the grant.
“Thankfully, we were permitted to repurpose some of the AWS credits to cover this, but the approval period caused a delay in our ZambaCloud work. To make up for this, we set up multiple nodes for the model training in December (so multiple models could be trained simultaneously), which made up for lost time but did require more manual monitoring.”.
impact metrics
The ultimate aim is for conservation organizations to have an accessible and accurate automated species detection tool that is routinely used upon collecting videos from the field as well as a tool for distance estimation so population monitoring of multiple species can be done in the same habitats. With ZambaCloud, researchers and conservationists can now eliminate unneeded blank videos and effectively find animals of interest.
Users in Hawaii are using ZambaCloud to look at species abundance in Guam. There is additional interest from scientists studying Alaskan Pikas, and others studying flightless birds in new zealand.
This platform will save researchers countless hours of valuable time, and has the potential to increase the value of camera trap data and improve data management practices for everyone working in biodiversity monitoring.
“We found we could remove more than 80% of blanks while losing fewer than 10% of non-blank videos. This amounts to valuable savings in viewing time, storage space, and data transfer time”
project
SustainabilitY
With the release of the new models on Zamba Cloud, one of the main needs for sustainability is coverage of cloud compute costs. The team are currently working on a sustainability plan for ZambaCloud including:
- Enhanced service tiers (with fees)
- Partnerships with other organizations
- Expanding to commercial customers
WCF has hired a technical resource who is running the open source Zamba python package to process videos for the Guinea country team. Such an embedded staffing model avoids the cloud compute costs and may play a useful bridge role in upskilling conservation teams.
Future Direction
WCF is working to write a technical paper on the development of ZambaCloud.
Depth estimation is the next big challenge in the team’s pipeline, which requires a reference frame in videos to show distance of an animal away from the camera trap. The team had a depth-labeled dataset and ran a challenge on this work, and the resulting model was integrated into their python package, but will need to be retrained on more species.
“A big decision for our future direction is whether we will support training on custom species detection models”
The DrivenData and WCF teams are working on making ZambaCloud more accessible for users with low-bandwidth internet connections with a ‘fast-upload feature’. One of the current challenges with the platform is that uploads often fail because the videos are too large and connections time out.
“The technical work in progress includes browser-side resampling, whereby videos are resampled to a lower frame rate and smaller size on the user’s machine, before being uploaded to the cloud. This work is likely a prerequisite for rolling out a subscription model for field users, since users will only be willing to pay if they can reliably upload their videos.”