Performance evaluation & metrics

We will assess the participants' performance in two tasks: (1) classification, that is, the presence of the csRNA-Seq signal, binary classification of positive (initiation, 5' read counts above zero) versus negative (no initiation, no reads) genomic positions, and (2) regression (for classification positives only), quantitative prediction of initiation efficiency.
To prevent technical overfitting, only a random half of the test data labels will be used during the online leaderboard stage to compute performance metrics. The other half of the test data will be used in the final evaluation.
Performance metrics: area-under-precision-recall-curve, AUPRC (classification), computed with sklearn (average_precision_score); Spearman correlation (regression), computed with scipy (pearsonr with rankdata, method=dense). The team rankings will be based on log-ranks to aggregate these two scores (as in IBIS, https://ibis.autosome.org) and, consequently, the ranks for each species, to identify the winning teams (gold-silver-bronze medals). Additional metrics might be computed during the post-challenge assessment, but they will not affect the team rankings.