This is the current version of the BioCreative evaluation library including a command line tool to use it; current, official version: 2.3 (use command line option --version to see the version of the script you have installed: bc-evaluate --version. If you have reason to believe that there is a bug with the tool or the library, or any other questions related to it, please contact the author, Florian Leitner.

Updates for major version 2

To update any former version you have installed, you only need to download the file, unpack it, and install it from command line by typing sudo python setup.py install in the directory where you unpacked the download. To check the updated worked, then type bc-evaluate --version and it should show you the correct version (see above).

The following changes have been applied to the initial library (2.0):

• 2.0 to 2.1: adding a command line option for the method detection task (IMT, for BioCreative III) - although this is the same evaluation as the normalization task (INT).
• 2.1 to 2.2: changes the strict column behavior: formerly, you were only allowed to supply the exact number of columns (document ID, annotation, rank, confidence) as used in BC II.5, while now the result file may contain any number of additional columns, e.g., the evidence string used in BioCreative III. To get the old (strict) behavior back, use the new command line option -s/--strict, in which case additional columns will be shown as error messages.
• 2.2 to 2.3: bugfix in the plotting function (thanks to Sérgio Matos for reporting it!), more informative output when an input file is missing.

Introduction

This library is used to evaluate the results of BioCreative II.5 and III with regard to the official BC II.5 evaluation function. The evaluation score is calculated from the AUC (area under curve) of the interpolated precision/recall (iP/R) curve, macro-averaged for IMT, IPT and INT results. The library provides various additional performance calculations which can be generated through the command line tool (see below and the tool's help and documentation). In addition, if you wish to use the library code (license: GPL, latest version), please consult the inline API documentation in the source code.

Installation

You will need to have a working version of Python 2.5 (or 2.6, 2.7) installed to use this package. It imports only on the standard libraries part of any Python base distribution as long as you do not want to use the plotting functionality. In this case, you need to install matplotlib, too.

To run the evaluation after installing the library (see the included README.txt file), you can call it from the command line:

bc-evaluate -h

The -h (or --help) flag will explain you the parameters and options; In-depth explanations can be found by using -d (or --documentation). The tool can evaluate the results for all tasks, ACT, INT, IMT, and IPT by using the corresponding option flag -a/--ACT, -n/--INT, -m/--IMT, -p/--IPT. The default is -n/--INT.

Overview

The tool allows you to explore your results in more detail than just the official evaluation function. By default, it gives you a detailed overview of evaluation results, including recall, precision, and F-score of your data, and all values are reported both micro-, and macro-averaged (the official evaluation function is the macro-averaged AUC iP/R score), except for the ACT task, where there is no macro/micro-averaging, but instead provides calculations for specificity, sensitivity, accuracy, and Matthew's Correlation Coefficient in addition to the AUC iP/R score.

The main arguments when using the library with the command line tool (bc-evaluate) are:

1. one or more result files as tab-separated plain-text (see the BC II.5 evaluation description and the --documentation option of the tool itself for explanations about the format of the result file), and
2. the corresponding gold standard annotation as provided by the BioCreative II.5 Elsevier corpus (either training or test set annotations). Also, homonym ortholog mapping and organism filtering files for this corpus can be found there.

You can download and install the ready made source packages for all operating systems. Please have a look at the README file for instructions on how to install this library.

Homonym ortholog mapping

For the homonym ortholog mappging, please use the files provided in the gold standard, the BioCreative II.5 Elsevier corpus. The homonym ortholog mapping files for this corpus are found in the directories biocreative_II.5_elsevier_corpus/training_set/vocabulary and biocreative_II.5_elsevier_corpus/test_set/vocabulary, respectively, called "uniprot_15.0_homonym_orthologs.tsv" in both cases.

If you would like to use another gold standard with this library (which is no problem as long as you keep to the file formats) and want to do homonym ortholog mapping for that gold standard, you can download a cluster of homonym identity (50% protein sequence identity) clusters for UniProt r15.0 from the Bc II.5 Elsevier corpus page and use this script to extract mappings in the format the evaluation library uses for a list of source proteins (i.e., a list of UniProt accessions used in your gold standard).

Organism Filtering

The organism filtering data is in the download "UniProt 15.0 acc2tax mapping + hom. orth. clusters" on the same page, in the file accession2tax_id.tsv. Note that you do not need the file homonym_identity_clusters.tsv found there unless you want to build your own H.O. mapping files (analogous to the "uniprot_15.0_homonym_orthologs.tsv" files explained above) for another gold standard.

Example Usage

So, for example, if you have your own normalization result file for the BC II.5 test set that contains the default output format (with ranks and confidence scores) - let's say you called it "normalization_result.tsv" - you could evaluate it with the following command:

bc-evaluate -n \
--ho test_set/vocabulary/uniprot_15.0_homonym_orthologs.tsv \
--of uniporot_15.0/accession2tax_id.tsv \
normalization_result.tsv test_set/annotations/normalizations.tsv

Or, in more compact notation (without the full file paths and names): "bc-evaluate -n --ho=ho_file --of=acc2tax_file result_file gs_file". In general, please consult the help (option --help) and documentation (option --documentation) provided by the bc-evaluate exectuable for all the possible options and details about the result file formats.