RSS 2.0
Critical Assessment of Information Extraction in Biology - data sets are available from Resources/Corpora and require registration.

BioCreative II.5

Announcement [2008-11-18]

Together with the publisher Elsevier/FEBS Letters and the biological database MINT the BioCreative organizers are announcing the online evaluation challenge. The BioCreative challenge evaluations consists of a community-wide effort for promoting the development and evaluation of text mining and information extraction systems applied to the biological domain. Previous BioCreative challenges attracted considerable interest not only in the bio text mining community, but also in the bioinformatics and biological database domain, resulting in two special journal issues and useful data resources for the development of biomedical text mining systems. BioCreative II.5 will evaluate real-time text mining capabilities on full text articles and explore future possibilities for author-assisted annotations using information extraction tools. BioCreative II.5 is a community challenge complimentary to the currently ongoing BioNLP '09 Shared Task on Event Extraction.

For general information and the background of BioCreative, please visit the about page here.


The principle of the challenge evaluation is centered around re-creating existing FEBS Letters author- and database-curated annotations (Structured Digital Abstracts, SDAs) using information extraction tools. The results should help conclude if automated IE can be used as basis for human curation of full-text articles. Therefore, to participate in the BioCreative II.5 evaluation, you will be required to have a working Annotation Server (AS) accessible through the BCMS platform to create a realistic, working scenario for the objective. The test set will consist of the full-text articles provided by the FEBS Letters journal, plus some additional undisclosed articles which we will use to validate nobody is providing precomputed (and therefor possibly manually curated) results for the articles. Therefore, ASs have to be able to work with BCMS-supplied text, capable of providing annotations in real-time (time frames of several minutes or less per article), as to minimize any possiblities to do manual annotations on the data.

Interesting new aspects of BioCreative II.5 relate to:

  1. The use of an online MetaServer platform as basic evaluation infrastructure, allowing direct comparison of multiple participating systems.
  2. Collaboration of article authors, publishers and expert human annotators (for the preparation of task resources).
  3. Real-time, sequential data generation by using results from more basic tasks from all participants as input.
  4. Use of full text articles in various formats including XML.
  5. Alignment of various tasks on a common data collection.

Background information on FEBS Letters SDAs can be found here:

An in-depth FEBS special issue on the topic can be found here:


INT: Interactor normalization task
(Given an article extract a set of biological entities, namely mentioned interactor proteins)
Extraction of a ranked list of interactor proteins in terms of their primary UniProt accession number limited to a total maximum of 50 proteins per article. Participating teams are asked to return a ranked list of proteins that are detected as being used to describe an interaction in the article (short: "interactor proteins") in terms of their primary UniProt accession number together with a normalized confidence score (again, range: ]0..1]) or an unique integer rank ([1..50]) for each interactor, as well as an optional list of corresponding protein mentions/passages used for extracting the normalization. This optional list provided by participants will serve in the evaluation phase to analyze problematic normalizations. The evaluation will be based on comparison between the automatically generated results and the manual protein normalization done by the authors and by the database curators. The macro-averaged results will be used to score the performance of the submissions, computing the macro-averaged f-score (based on calculating the f-score per-document and then averaging it across documents).
IPT: Interaction pair task
(Extraction of biological relations, namely pairs of interacting proteins)
Extraction of protein interaction pairs in terms of their primary UniProt accession number limited to a total maximum of 50 interactions per article. Participants are asked to return for each article a list of non-redundant, binary interaction pairs together with a normalized confidence score (range: ]0..1]) or an unique integer rank ([1..50]). Directionality of the interaction will not be considered, thus also redundancy resulting from alternative interaction directionality has to me removed from the training and test set evaluation files. The evaluation will be based on comparison between the automatically generated results and the manual protein interaction pair normalization done by the authors and database curators. The macro-averaged results will be used to score the performance of the submissions, computing the macro-averaged f-score (based on calculating the f-score per-document and then averaging it across documents).
ACT: Article categorization sub-task
(Binary classification of articles as relevant for extracting interaction annotations)
Binary classification of full text articles as relevant for extracting/containing protein-protein interaction annotations together with a confidence score for the classification in the ]0..1] range (excluding zero). This is the document classification task. Evaluation will be based on AUC (area under the ROC curve). Here, full text articles will be provided to the participants, as opposed to the IAS of BC2, where only abstracts were used.


Evaluation will be based on direct comparison of automatically generated predictions by participating systems against a Gold Standard of manually generated annotations, so-called Structured Digital Abstracts (SDAs) generated by the original article authors themselves as well we by expert biological database annotators. In order to assure a more robust evaluation setting and to be able to directly compare and visualize different online predictions the BioCreative MetaServer (BCMS) platform will be used.

Technical Features

Currently, there are 12 research groups world-wide providing results from their Annotation Servers to the public via the BioCreative MetaServer platform. Using this simple-to-implement infrastructure will allow you to directly compare your tool's performance to other Annotation Servers - making evaluation of your new algorithms and pipelines simpler - and will give you a head-start for future BioCreative challenges, which will use the platform for evaluation, thus increasing the visibility of your systems. The organizers will provide the participants with sample scripts, technical resources and all information to implement their Annotation Servers.

The issues to solve are:

  • XML-RPC communication with the BioCreative MetaServer as defined by the platform;
  • Full-text capability (the BCMS will send your AS the text using the XML-RPC service);
  • Real-time annotations (not more than a few minutes per article);

An Annotation Server is simply a “remote procedure call” (RPC) web service that the MetaServer - or any other client, if you wish to make your Annotation Server accessible to the general public - can connect to. That is, an RPC function is no different than any other function/method in regular programming languages, but with the added benefit that the function caller can be a program running on a different computer in another location with Internet access. The clients (MetaServer) request annotations for MEDLINE IDs or, as in the case of this challenge, for full-text from the Annotation Server. The Annotation Server then is expected to respond to the request by returning these annotation results as structured data. Setting up such a RPC web-service is fairly simple;  it consists of a server (which can be your oldest PC in the lab) connected to the Internet running a simple script (we will provide examples in various programming languages to the participants) capable of calling your text-mining pipeline with the text it received as parameter. Finally, the script parses/receives the results of your pipeline and sends them back to the client/MetaServer.

To help participants focus on a particular task, the BCMS will allow downloading results from more basic tasks generated by other ASs, which you can use as input for your own system. E.g., if you want to find interaction pairs (IPT task) only, you will be able to first fetch the article classification (ACT task) and all protein normalizations (INT task).

The BioCreative MetaServer and more information about it can be found here:

Technical background on implementing Annotation Servers can be found here:

The exact specifications for the full-text Annotation Servers will be published on the particpating team’s pages (see registration below).

Registration and Mailing List

To join the BioCreative mailing list:

Additionally to obtain data and get access to all parts of this page (BioCreative copora, workshop proceedings, team registration, etc.):

If you are have a user account, you can register a team for BC II.5 at:

Optionally, you can add any additional members to your team with their email and ask those member to create an account on this website with the same email. Everybody in your team then has access to a team page, where we will publish information, downloads, and results for each participating team at:


  1. Ceol et al., Linking entries in protein interaction database to structured text: The FEBS Letters experiment., FEBS Letters (2008) vol. 582 (8) pp. 1171-7
  2. Gerstein et al., Structured digital abstract makes text mining easy., Nature (2007) May 10;447(7141):142
  3. Hahn et al., Text mining: powering the database revolution., Nature (2007) May 10;447(7141):142
  4. Hirschman et al., Overview of BioCreAtIvE: critical assessment of information extraction for biology., BMC Bioinformatics (2005) vol. 6 Suppl 1 (1471-2105 (Electronic)) pp. S1
  5. Krallinger et al., Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge., Genome biology (2008) vol. 9 Suppl 2 pp. S1
  6. Leitner et al., A text-mining perspective on the requirements for electronically annotated abstracts., FEBS Letters (2008) vol. 582 (8) pp. 1178-81
  7. Leitner et al., Introducing meta-services for biomedical information extraction., Genome biology (2008) vol. 9 Suppl 2 pp. S6