In my previous post I introduced the Breast Cancer Challenge Sage is hosting to build predictive models of the disease. The initial conception of this project was as a winner take all competition, with a clear scoring method and single top model as the winner of the big prize: publication in Science Translation Medicine. Compared to some of our other attempts to catalyze scientific collaborations based more on preaching to scientists to share data and methods for the better good of society, this approach seems to have triggered substantially more interest from the community, and action by some of the initial participants.
Our task now is how to best harness this energy to motivate researchers, and also form a community where people not only compete, but also collaborate and build off each other’s work effectively. Many of my recent discussions with the challenge organizers have drawn analogies to the Tour de France, and how multiple dimensions of awards and glory motivate different riders to focus on achieving different objectives in the race. Every year there’s only a handful of guys who can realistically hope to win the yellow jersey, but many other riders that compete to take home other awards.
Multiple Jerseys – In the tour, there’s not only an overall winner of the yellow jersey, but several other jerseys that award different riding styles. The green jersey is awarded by points obtained by sprints to stage and intermediate race points, and the polka-dotted king of the mountains awards the rider who does the best on the major climbs of the tour. Breast cancer is not really a single disease; in reality many different molecular defects give rise to a heterogeneous collection of diseases. A prognostic test, particularly one focused on detailed genetic data, is unlikely to perform well across all types of breast cancer. It would be interesting to award sub-category awards for particular types of cancer, based on things like ER, PR, and HER2 status.
Stage Winners – An awful lot of the tactics of a bike race involve the dynamic between riders who are trying to win an individual stage, and riders or teams that are more concerned with the overall standings. The glory of a stage win is enough to cause riders to attempt to break away from the peloton even though they have no hope of competing for the overall lead. With our challenge we’d like to reward people for submitting ideas early and allowing others to incorporate and modify their models, instead of waiting to the the last minute to submit an entry. One way to do this would be to have defined intermediate points where we had some sort of recognition of the leader. Maybe we could introduce new validation data at set periods to score the models, and then allow contestants to incorporate previous validation data as new training data in subsequent rounds. Another approach we’ve discussed is to reward people for time spent on top of the leaderboard and for the amount of improvement they make over the previous best score. The person who makes many submissions that improve the top score over the competition may have contributed more to the competition than someone who sneaks in a marginally better model right at the end of the challenge.
Visible Leaderboard – In the tour, everyone knows what everyone is doing. Riders have their coaches in their ears, giving them information on what other riders are doing, and the team constantly replan strategy as the race unfolds over several weeks. We’ve got a basic leaderboard up giving real-time feedback on the challenge and it seems to be a good way to engage people. I’m sure with more time and resources we could do a better job of making this resource more exciting for the challengers. Maybe we should pull in contestants Synapse profiles and try to connect the contestants to an audience. Math and statistics might not be as sexy as bike racing, but there’s got to be some cancer survivors out there that would be interested in seeing people compete to better understand the disease.
So what do you think? Do you have any better ideas on how to organize these scientific challenges? We are open to experimentation.