We are pleased to announce that Elsevier B.V. has granted us the privilege of providing the corpus of FEBS Letters articles used during BioCreative II.5 to the scientific community. The official announcement of the corpus' availability was published on the FEBS homepage.

The corpus contains 1190 articles, mostly from 2007 and 2008, both in machine-readable XML format and the UTF-8 special format used during the challenge, as distributed via the BioCreative Meta-Server. All annotations (i.e., the gold standard) used during BioCreative II.5 are contained within the package. Additionally, an archive containing all UniProt 15.0 accession-taxonomic ID mappings as well as a list of clusters of homonym ortholog proteins in UniProt 15.0 can be downloaded here. The clusters were established from UniRef50 r15.0 clusters, intersected with all clusters extracted by using case-insensitive matching of UniProt names (all names available per record, but excluding one-letter names and purely numerical "names"). The taxonomy mapping file can be used in conjunctions with the evaluation library, while the homonym ortholog clusters are provided as reference (limited clusters relevant for each the training and test set only are provided directly through the corpus).

Corpus overview:

• Protein-protein interaction (PPI)-describing articles: 124 (61 in the training set, 63 in the test set)
• Negative articles: 1066 (i.e., articles that do not describe experimentally demonstrated PPIs)
• Training set: 595 FEBS Letters articles from 2008
• Test set: 595 FEBS Letters articles from 2007
• Articles with interaction annotations: 122 (2 positive articles in the test set do not have interaction annotations)
• Protein annotations: UniProt major release 15.0 primary accessions, both as accessions only (normalizations) and as binary interaction pairs.
• Normalizations: Training set: 261; Test set: 252
• Pairs: Training set: 236; Test set: 216

We would like to express our gratitude to Elsevier for granting us the rights to keep providing this significant collection of articles and to the MINT database curators for contributing the annotations.

This is the final version of the BioCreative evaluation library including a command line tool to use it; current version: 2.0a1. This is the first release candidate and it is possible that you might encounter a bug or that some functionality still will be improved after the initial feedback. If you have reason to believe that there is a problem with the tool or the library, or any other questions related to it, please contact the author, Florian Leitner.

This library is used to evaluate the results of BioCreative II.5 with regard to the official BC II.5 evaluation function. The evaluation score is calculated from the AUC (area under curve) of the interpolated precision/recall (iP/R) curve, macro-averaged for IPT ant INT results. The library provides various additional performance calculations which can be generated through the command line tool (see below and the tool's help and documention). In addition, if you wish to use the library directly, please consult the inline documentation.

You will need to have a working version of Python 2.5 (or 2.6, 2.7) installed to use this package. It imports only on the standard libraries part of any Python base distribution as long as you do not want to use the plotting functionality. In this case, you need to install matplotlib, too.

To run the evaluation after installing the library (see the included README.txt file), you can call it from the command line:

bc-evaluate -h

The -h (or --help) flag will explain you the parameters and options; In-depth explanations can be found by using -d (or --documentation). The tool can evaluate the results for all three tasks, ACT, INT, and IPT by using the corresponding option flag -a/--ACT, -n/--INT, -p/--IPT. The default is -n/--INT.

The tool allows you to explore your results in more detail than just the official evaluation function. By default, it gives you a detailed overview of evaluation results, including recall, precision, and F-score of your data, and all values are reported both micro-, and macro-averaged (the official evaluation function is the macro-averaged AUC iP/R score), except for the ACT task, where there is no macro/micro-averaging, but instead provides calculations for specificity, sensitivity, accuracy, and Matthew's Correlation Coefficient in addition to the AUC iP/R score.

The main arguments when using the library with the command line tool (bc-evaluate) are:

1. one or more result files as tab-separated plain-text (see the BC II.5 evaluation description and the --documentation option of the tool itself for explanations about the format of the result file), and
2. the corresponding gold standard annotation as provided by the BioCreative II.5 Elsevier corpus (either training or test set annotations). Also, homonym ortholog mapping and organism filtering files for this corpus can be found there.

You can download and install the ready made source packages for all operating systems. Please have a look at the README file for instructions on how to install this library.

Over the past few weeks, the website has received some small improvements here and there, in an intent to improve its visibility and add more functionality in the light of the upcoming BioCreative III. On a side-note, please do not register teams for BC III through this site currently, the mode of the official team registration for BC III still needs to be determined by the BioCreative III organizers. A quick run-down of the improvements made to the website:

• Downloads now are shown for all items in the news section, and link to the actual entry in the respective topic. However, you still need to have a valid account or be a team member to download Resource or Team files.
• The RSS feeds now show the complete content of published items, including images and downloads.
• There is a new feed for any entry published on this website over the course of the last year to make it easier for you to keep track of all new content. If you want to add this RSS feed to your news-reader, e.g. Google Reader, you can use this link.
• Many invisble administrative improvements for the site's contributers.

We hope you find some of these additions useful!

All speakers who so far have agreed to make their talk presentations available can be downloaded from the drop.io site. This site creates a data "drop" that stays online as long as there is any activity within the last year, meaning that once nobody is interested in the slides for more than that time span, the data will automatically vanish from the Internet.

If you have any problems with downloading the data, please use the contact form.

Special Issue of Transactions in Computational Biology and Bioinformatics

Special Issue on BioCreative II.5

BioCreative (Critical Assessment for Information Extraction in Biology) is the main challenge evaluation in the area of biological text mining, focused on assessment of text mining applied to problems from real biological applications. To date, there have been three BioCreative evaluations; the biological task for the first BioCreative (2003-2004) focused on the automatic extraction and assignment of Gene Ontology (GO) annotations to human proteins based on information from full text articles. The biological task for both BioCreative II (2006-2007) and BioCreative II.5 (2009) focused on the identification of protein-protein interactions, in support for the curation of the IntAct and the MINT protein interaction databases. This series of evaluations has advanced the state of the art in text mining applied to biological problems.

This special issue invites submission of technical papers that describe and analyze the approaches used in the recent BioCreative II.5 evaluation. We encourage papers that describe novel approaches and/or provide analysis of the effectiveness of these approaches in tackling the protein-protein interaction task. In particular, papers must contain novel content not previously published – for guidelines, see here.

Papers will undergo the standard review procedures for TCBB. Authors should indicate their intent to submit a paper by Jan 8, 2010. Please submit your paper to Manuscript Central at cs-ieee.manuscriptcentral.com by this date. As an author, you are responsible for understanding and adhering to our submission guidelines (available online here). Feel free to contact TCBB’s Publication Coordinator at tcbb@computer.org or the guest editors if you have any questions.

Important Dates

1. Manuscript Submission Deadline: Friday, Jan. 08, 2010
2. Initial Decision: Monday, Feb. 15, 2010
3. Notification of Final Acceptance: Friday, Apr. 30, 2010
4. Publication Date: July-September 2010

Guest Editors

• Lynette Hirschman, The MITRE Corporation
• Martin Krallinger, CNIO
• Florian Leitner, CNIO
• Alfonso Valencia, CNIO

The workshop on BioCreative II.5 will be held in Madrid at the CNIO during October 7th and 9th, 2009, sponsored by the ENFIN Network and the EMBRACE Grid Network. It will focus on the tasks created for the challenge, especially the extraction of protein-protein interactions (PPIs), with the background of creating structured annotations for full-text papers. These PPIs were generated by the paper authors themselves and MINT database curators (a PPI DB). There will be invited speakers discussing various topics relevant to Information Extraction in Text Mining and the topic of digital annotations, meta-services and applications in Bioinformatics.

Given the topic of the SDAs and online digital annotations, we are organizing a special session on the first day (Wed, Oct 7th). This "round table" should generally cover the topic of electronic/structured annotations on publications from the viewpoints of the most important stakeholders: the academic world, (pharma) industry, the publishers, and the databases.

We will have at least one prominent speaker from each of these areas followed by a discourse about how all stakeholders can cooperate to overcome issues in the evening and achieve the necessary requirements for creating and maintaining such structured descriptors. Speakers will be:

• Judith Blake (JAX)
• Alan Bridge (SIB)
• Udo Hahn (Uni Jena)
• Ian Harrow (Pfizer UK)
• Adriaan Klinkenberg (Elsevier)

Additionally, there will be a session dedicated to applied systems in text mining related to the issue of digital annotations that will be held in the afternoon of the second day (Thu, Oct 8th). Speakers during this session will be:

• Stefano Bocconi (Elsevier, OKKAM/Word plugin for SDAs)
• Seán O'Donoghue (EMBL, Reflect)
• Robert Hoffmann (MIT, WikiGenes)
• Florian Leitner (CNIO, BCMS)

The workshop will cover presentations from several participant systems of the challenge, the result evaluation, an inter-annotator- and curator to author-agreement study, the SDAs and the FEBS Letters Experiments, and a talk about the BioCreative Meta-Server, as well as some plans on the future of BioCreative (BC III). This also means we will continue in the tradition of former BioCreative challenges and - much like the CASP challenges in Structural Biology - present the final results of all the participants during the workshop. In addition, there will be three keynote speakers:

• Larry Hunter (UCHSC)
• Dietrich Rebholz-Schuhmann (EBI)
• Anna Tramontano (Uni Roma)

Registration

To register for the workshop, please visit the CNIO event page. Please note that the abstract submission is only for the poster sessions, challenge participant system talk entries (and papers) should be made via EasyChair as explained to each of them via mail.

Final Schedule

Wed, 7th of Oct., 8:30-18:30

Registration (08:30-09:00)

• 09:00-09:20 - Workshop Opening (Alfonso Valencia, CNIO)
• 09:20-10:00 - Digital annotations: the publishers (Adriaan Klinkenberg, Elsevier)
• 10:00-10:40 - Digital annotations: the academic world/text miners (Udo Hahn, Uni Jena)
• 10:40-11:20 - Digital annotations: the databases (Judith Blake, JAX)

Coffee Break

• 11:40-12:20 - Digital Annotations: the databases (Alan Bridge, SIB)
• 12:20-13:00 - Digital Annotations: the industry (Ian Harrow, Pfizer UK)

Lunch

• 14:30-15:10 - The FEBS Letters experiment (Gianni Cesareni, MINT & FEBS)
• 15:10-15:50 - The BioCreative II.5 challenge (Martin Krallinger, CNIO)
• 15:50-16:30 - Author- and systems-agreement studies (Lynette Hirschman, MITRE)

Coffe Break

• 17:00-17:40 - BC II.5 Evaluation results (Florian Leitner, CNIO & Lynette Hirschman, MITRE)
• 17:40-18:30 - Discussion on BC II.5 and digital annotations

Thu, 8th of Oct., 09:00-18:30

• 09:00-09:50 - Keynote (Larry Hunter, UCHSC)
• 09:50-10:40 - Particiapnt systems 1

Coffee Break

• 11:10-13:00 - Participant systems 2

Lunch & Posters

• 14:30-15:20 - Keynote (Anna Tramontano)
• 15:20-15:50 - Applied systems: WikiGenes (Robert Hoffmann)
• 15:50-16:20 - Applied systems: Reflect (Sean O'Donoghue)

Coffee Break & Posters

• 16:50-17:20 - Applied systems: OKKAM/Word (Stefano Bocconi)
• 17:20-17:50 - Applied systems: BCMS (Florian Leitner)
• 17:50-18:30 - Discussion on applied systems

Workshop Dinner (20:00 - 22:30)

Fri, 9th of Oct., 09:30-13:00

• 09:30-10:20 - Keynote (Dietrich Rebholz-Schuhmann, EBI)
• 10:20-10:45 - Participant systems 3

Coffee Break

• 11:15-12:05 - Participant systems 4
• 12:05-12:45 - Retrospective & future: BioCreative III (Alfonso Valencia)
• 12:45-13:00 - Workshop closing (Alfonso Valencia)

Links

• Steering and scientific committees
• List of recommended hotels - primary conference hotel (w/ shuttle bus): Abba Plaza Hotel
• Registration page (includes poster abstract submission)
• City map with important locations (CNIO, subway stations, hotels, dinner site, city center, airport)
• Asador Fronton - conference dinner restaurant

The BioCreative II.5 workshop on text-mining based extraction of protein-protein interactions in automated environments as well as in assisting the annotation process for authors and DB curators will be held at the

CNIO - Spanish National Cancer Research Centre in Madrid, Spain

from October, 7th, 13:00 CET, to October, 9th, 12:00 CET.

The main focus will be on presenting recent approaches and improvements in PPI data extraction by the participants and the complete evaluation results of BC II.5. In addition, we intend to have discussions and presentations on the online apporach via the BCMS platform and protocol, creating author-annotations via the ENFIN integration plaform, the FEBS Letters Experiments and Structured Digital Abstracts, and providing an environment to connect publishers and text miners.

The exact schedule for the workshop itself and for the creating of the proceedings will be published over the next few weeks, including the registration fees and the list of invited speakers.

These are the slides for the talks about BioCreative II.5 (06/28/09, BioLINK) and the highlight talk about BioCreative II (06/02/09, ISMB/ECCB), both by Florian Leitner, for download as PDF files from the resources section.

The BioCreative II.5 test phase has been closed! Thank you very much to all participants, for their patience with moving test phase dates, changing training data, and the development of the BCMS infrastructure! And, last but not least, thank you to all online teams that took up the work to create Annotation Servers and therefore made this "online challenge" a reality!

In total, we had ten teams participating online, plus twelve teams (six of which were part of the online challenge) in the offline evaluation for a total of 16 teams. This amounts to the following numbers of officially accepted runs for all three tasks (in total making up 470 MB of uncompressed data):

ONLINE:

For all participant teams an online questionnaire is now available on the team page. We kindly ask every team that took part either in the online or offline evaluation to fill in the form. It consists mainly of checkboxes and most text-based answers are completely optional, so it should take very little time. Thank you!

The test phase start windows are closing in short. The online phase start window will close:

Friday, 29th of May, at 12:00 GMT

And the offline (only) phase will close on:

Monday, 1st of June, at 12:00 GMT (in 4 days)

I.e., by those dates participants must have either told us which of their servers are ready, or have sent us at least one valid training run together with their team ID for the offline participation (online teams can choose to participate offline after completing their online phase).

Due to an error in the SDA of article 10.1016/j.febslet.2008.07.043, the training set data has been updated to version 5. The current correct string in the READMEs describing the release should read:

Training set, May 11, 2009 - release version 5

The issue affects the distribution of unique SwissProt accessions/normalizations (+5) and unique pairs (+6). It also generates a new interaction type in the interaction type classification, although this is not used for the BioCreative II.5 challange. Please download the update from your team page.

The current data distribution therefor is as follows:

• 222 identifiers from SwissProt
• 22 identifiers from TrEMBL

generating a total of

• 61 positive set articles with
• 252 unique normalizations and
• 228 unique PPI pairs (not counting doubles entries in annotated pairs)

Additionally, there are 558 negative articles in the training set for a total of 619 articles, of which 24 negative set articles are empty in the UTF-8 converted full-text articles, as they are actually non-regular publications (i.e., resulting in the 595 articles that online participants get loaded into their Annotation Server queues on the BCMS 1.0 beta site).

Thanks to Zuofeng Li for finding and reporting this error!

Also, the BioCreative MetaServer 1.0 beta has gone online, which the online participants should use to generate their annotations. This tool provides a wealth of functionality to get Annotation Servers up and running:

1. Full configuration of your connection settings and connectivity testing.
2. Error/warning/info log reports about the connection and the data being exchanged.
3. Report generation for the BioCreative II.5 corpora results, including evaluation statistics.
4. Re-running of the annotations for the BioCreative II.5 corpora at any time (except the test phase).

If you still have not informed us about your Annotation Server, want to create one, or need more information on how to install a Web Service Server, please do not hesitate to contact us for help.

We would like to announce that we finally have received all test phase data necessary; This brings the BioCreative II.5 corpus to over 1,000 full-text articles on FEBS Letters supplied by Elsevier, which were all classified by MINT as PPI-containing or not and over 100 articles thereof with PPI annotations.

We want to hereby express our appreciation for all the work and contributions both Elsevier and MINT made - without this cooperation, scientific assessments such as the BioCreative challenges would not be possible - thank you!

For participants, the exact proceedings can be found on their team pages (link here). Anybody else interested in a short overview of the dates can have a look at the Events entry here.

These are screenshots from the Annotation Server administration interface that will be exclusively available to the BioCreative II.5 online challenge participants. Hopefully, it will be online sometime during next week, so participants will have enough time to familiarize themselves with it.

Having this interface should more than make good for the delay in the test phase. It will allow remote administration of all you Annotation Server settings relevant to the BCMS, including error logs, setting public information, and adding additional servers to your current setup.

The screenshots are from the main administration panel ("team homepage") and the dialog for initializing a new server ("server initialization"). As the current development is progressing very well, it is highly likely we will be using this interface during the test phase.

More information will be published next week.

Due to delays with the test data preparation, the test phase has been moved for about two weeks (expected date end of April). Exact dates will be published here and on the mailing list as soon as we have established the final dates for sure.

Table of Contents

1. General Participation
2. Corpus and Data Curation
3. Online Challenge
4. Offline Challenge
5. Test Data/Phase
6. Result Data
7. Evaluation

General Participation [top]

Which resources are allowed in the challenge?

In principal, any resource available to you can be used to create the annotations. However, you can declare that for any number of your submissions you did not use any existing annotations on the articles (MeSH terms, GOA or SwissProt annotations, etc.) to better simulate the most realistic setting in the context of author-based annotations. Also, a questionnaire will be part of your result submission to determine differences in approaches based on the used resources.

Should participants use SwissProt or the entire UniProt space for the protein mappings?

Obviously, it is much more challenging to perform well on the training set when trying to map against the entire UniProt accession space. Additionally, it is also more realistic to map to that larger set, as newly published articles frequently contain proteins which have so far not been annotated by the SwissProt curators. The distribution of SwissProt/TrEMBL accessions in the test set will be as close as possible to the training set, but there will be notable differences in the difficulty of the accession space you choose, which will be accredited in the evaluation.

Can a team participate in both the online and the offline challenge?

Yes, you can choose to submit results for both challenge classes, and in both cases you can submit up to the the five allotted runs (see Test Data/Phase - "How many different runs may a team submit?").

Can teams choose to participate only in a specific task of the challenge?

You are free to choose to only submit results for a specific task (ACT, INT, IPT) for any of your Annotation Servers or offline runs (in both cases simply by submitting only the relevant results).

Corpus and Data Curation [top]

How are the interaction-relevant articles chosen and which interactions are "curateable"?

The must contain direct, physical protein-protein interactions that have been specifically demonstrated in the article.

Can interaction annotations stem from figures or tables?

Yes. Figures often provide experimental evidence information, while for large interaction experiments tables are used to list annotations. Note although that in the context of this challenge these annotations (Interaction Type, Experimental Method, etc.) are not part of the challenge.

Are there cases where a protein interaction is between two proteins from different organisms (e.g. protein A from mouse and protein B from human)?

Yes, although this is not very common, there are cases.

So what kind of interaction pairs should participants extract?

All unique (in terms of the two UniProt accessions), undirected pairs found in the article which are "curatable" (see first question and section Result Data).

Online Challenge [top]

How will the full-text data be sent to the Annotation Servers?

The data will be transferred via XML-RPC and will be structured just as the plain-text files in the training set. If you want to receive XML data, please first read the next answer first. Following XML-RPC guidelines, this means each article will be sent as a list of structs (dictionaries, hashes, associative arrays). For example, the first four blocks from of article 10.1016/j.febslet.2008.01.064 are (shortened):

FIGURE Immunofluorescence analysis ... cells. Scale bar, 20μm.
QUALIFIER Fig. 1

FIGURE (A) Confirmation of the ... with RP2.
QUALIFIER Fig. 2

FIGURE Identification of affixin-binding ... anti-(His)6 antibody.
QUALIFIER Fig. 3

FIGURE Pulldown assay of active ... using the cell lysates.
QUALIFIER Fig. 4

This would be sent as the first four elements of an array where each element contains the struct with the text, the section name, and an optional qualifier and a number in case the section name and qualifier are not unique in the article (i.e., not the case in this example). Here, the array therefore will be:

[{
	"section": "figure",
	"qualifier": "Fig. 1",
	"content": "Immunofluorescence analysis ... cells. Scale bar, 20μm.",
},
{
	"section": "figure",
	"qualifier": "Fig. 2",
	"content": "(A) Confirmation of the ... with RP2.",
},
{
	"section": "figure",
	"qualifier": "Fig. 3",
	"content": "Identification of affixin-binding ... anti-(His)6 antibody.",
},
{
	"section": "figure",
	"qualifier": "Fig. 4",
	"content": "Pulldown assay of active ... using the cell lysates.",
}]

Why is the data preferably sent as structured text and not as XML?

The problem with the XML format is that if you would want to make your Annotation Server system available later in the BCMS platform, the XML format is of hardly any value: you would have to write parsers for each other format that might be used as input and you would have to think about how to handle plain-text input, too. Given that this challenge's setting is to evaluate author-annotation assistance, having plain-text is the more probable starting ground and it has better long-term value, while you do not have to occupy yourself with disambiguating the unicode symbols in the text - which results a huge performance boost if you get the right UTF-8 symbols in our baseline tests for NER, at least. Therefore, the BCMS platform will provide the full-text as structured data for you to use directly without the need to worry about constantly writing and maintaining parsers for various XML formats and entities or recognizing the text structure in pure plain-text data. Only if there are sufficient teams who insist on using the XML format, this format will be made available solely for the purpose of the challenge - but your Annotation Server would not benefit from the unified text structure and parsing the BCMS platform will offer for you and you will have to "reinvent the wheel" yourself each time another XML format is added. In other words, this measure is intended to make life easier for you and not as much limit you to unreasonable settings, but we obviously can provide the XML if there is are several teams interested in receiving the raw data. However, keep in mind that especially given the benefit of the extra data found in the XML, UTF-8 text resource use will be accredited during evaluation.

How much time do the Annotation Servers have to report results for an article?

Mainly to avoid the possibility of manual annotations in the online challenge and to simulate a real-time setting for generating the annotations, the total time for doing all annotations will be short. The current goal is that the servers report their results within 2 days for all articles; as you will receive about the same number of articles as found in the training set, this means a server will have about 5 minutes per article on average to report the results. Given our previous experience with the BCMS system, where any server was able to report results on MEDLINE abstracts in less than 10 seconds, this should be a generous timeframe.
In summary, this means that you will have the ability to set any timeout you wish per article for your Annotation Server, but after the total time span has passed, no further annotation requests will be sent from the MetaServer to the Annotation Servers and therefore no further results will be accepted. This means, if a particular article takes very long to annotate you can either choose to continue trying to report annotations for it or abort the annotation process for that specific article after a certain time-span has elapsed. The web interface will allow you to retry any articles where the annotation process has failed or skip it and continue with the next article in the set while the online test phase is still running.

Offline Challenge [top]

What will the available input formats/files be?

The files you will receive for the test phase will come in two sets, the articles in UTF-8 plain text and in XML format, in both cases the same format as the *.noSDA.utf8 and *.noSDA.xml files in the training set. I.e., the files will not contain the SDAs, neither in XML nor UTF-8 format.

How will the files be structured or ordered?

All files will be anonymized with a unique identifier (i.e., not their DOI identifier) which will be part of the filename minus the suffix (*.noSDA.utf8 and *.noSDA.xml), i.e., every participant will receive all files (negative and positive, interaction containing, articles) without any indication to which of those two sets (negative/positive article set) the file belongs.

In which (file) format should the results be reported?

As plain-text, tab-separated files, exactly as explained in the evaluation documentation. Other formats are currently not available, but if there is a significant amount of teams which would want use another common format (ieXML, etc.) and a conversion script to the official challenge format can be created in due time, we can arrange for that.

How can participants make sure the result file has the correct format?

You can check your files using the evaluation script. If the script returns an evaluation result and does not abort with an error, the format is valid.

Test Data/Phase [top]

What data and information will the participants receive?

For the online challenge, the participants will be sent the full-text (similar to UTF-8) for all articles (positive, interaction-containing articles, as well as negative articles) one-by-one from the MetaServer with no additional information. In the case of the offline challenge, participants will download the articles which will be grouped in two folders - for UTF-8 and XML formated articles - for which the filename of each article (sans the file suffix, just as in the training set) will act as the article identifier. As with the online challenge, no additional information will be provided beyond that. I.e., you will not know a priori for which articles you should generate normalization and pair annotations (also see Evaluation - "Will normalizations or pairs annotated on negative, non-interaction containing articles be evaluated?" about this issue).

Which interaction types and detection methods will the protein interaction pairs have in the test set?

As defined by the vocabulary, which forms part of the training set data, any type and method listed there might form part of the interaction pairs in the test set. However, note that interaction type and detection method annotations are not part of the challenge evaluation itself.

When will a team be allowed to enter the test phase?

Before entering the actual test phase, any participant will have to complete a complete run using the training data to ensure their systems are working and formats are correct. For offline submission this means you will send us the annotations you generated for the files from the training set you received from us. For the online challenge, you will have a web-interface to active the MetaServer, which will then try to query your Annotation Server and keep you informed about errors and problems. Once you complete the submission of valid annotations of the complete training set, you will be ready for the test phase. Additionally, we will use this data to compare performance changes when moving from the training to the test set.

How many different runs may a team submit?

You can set up five different servers (online) or submit up to five different runs (offline) per team. You will be asked to state fundamental differences between each run you submit (other resources used, XML or UTF-8 articles, MeSH terms, machine learning algorithms, possible ACT/INT/IPT task focus, etc.).

How will the test phase proceed?

After all participants have successfully tested their Annotation Servers with the training set, the online annotation of the test set can be started individually for each participant team from the team's web interface. Teams will have approx. two days (see Online Challenge - "How much time do the Annotation Servers have to report results for an article?") to complete the online test phase, finally filling in a survey. After all teams have completed the online phase, the complete article set will be made available for download and the offline phase will start where teams will have again two more days to submit offline results, again together with a survey. After the offline phase has passed, the whole test phase will end and the evaluation phase of the results by the organizers starts.

Result Data [top]

Will non-unique normalization/pair results be allowed?

No. Pairs and normalizations have to be unique. You can test this assertion by using the evaluation script on plain-text formated results - it will abort with an error if your results are not unique.

Will non-existing UniProt accessions be accepted?

No. For the online evaluation, the MetaServer will raise an error if you try to submit an accession that does not exist (NOTE: this has been disabled for the challenge to be comparable to offline results - any valid string is accepted and you have to make sure it is usable - i.e., just as for the offline evaluation). For the offline evaluation, they will although be "accepted", as they have no impact on the actual evaluation process.

What will the survey for result submission be asking from the teams?

For each active server/submitted run the participants will be asked to fill in a survey in addition to the result data itself to complete the test phase. You will be asked if you used any existing annotations, which mapping you attempted (SwissProt/UniProt), if you used the XML files or not, which resources you used, and a general short description of your approach (e.g., machine learning and NLP methods).

How should the results be ranked?

One question was raised about ranking of the results, a slightly modified quote from Jörg Hakenberg:
Could you please provide us with an exemplary ranking for the PPIs in the training data? In the annotated pairs:

10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ES28
10.1016/j.febslet.2008.01.064	Q9HBI1	Q8K4I3
10.1016/j.febslet.2008.01.064	Q13153	P63001
10.1016/j.febslet.2008.01.064	Q13153	P70766
10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ESD7
10.1016/j.febslet.2008.01.064	Q9HBI1	O55222

which of these 6 is the most important (rank 1), which the least (rank 6).
There is no order in the gold standard. Your evaluation result score will be equal no matter in which order you rank these results. If you want to show that a particular section of consecutive, ranked classifications are of equal importance, report them all with the same confidence score. Only the ranking influences results, i.e. (DOI - Acc1 - Acc2 - Rank - Confidence):

10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ES28	1	0.8
10.1016/j.febslet.2008.01.064	Q9HBI1	Q8K4I3	2	0.8
10.1016/j.febslet.2008.01.064	FALSE_P	FALSE_P	3	0.6
10.1016/j.febslet.2008.01.064	Q13153	P63001	4	0.6
10.1016/j.febslet.2008.01.064	Q13153	P70766	5	0.4
10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ESD7	6	0.4
10.1016/j.febslet.2008.01.064	Q9HBI1	O55222	7	0.1

is exactly equal to, for example:

10.1016/j.febslet.2008.01.064	Q13153	P63001	1	0.8
10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ES28	2	0.8
10.1016/j.febslet.2008.01.064	FALSE_P	FALSE_P	3	0.6
10.1016/j.febslet.2008.01.064	Q9HBI1	Q9ESD7	4	0.6
10.1016/j.febslet.2008.01.064	Q13153	P70766	5	0.4
10.1016/j.febslet.2008.01.064	Q9HBI1	O55222	6	0.4
10.1016/j.febslet.2008.01.064	Q9HBI1	Q8K4I3	7	0.1

If you were to compare these two cases with the evaluation script, you would get the same evaluation scores. The only factor influencing the score is how many true positives you have and how many false positives you do not have in your (top influcencing) ranks.

Evaluation [top]

Which evaluation types will there be?

In principal, we will make a big differentiation between online and offline submissions of results and treat the offline results separately, as the goal is to show the online, real-time capability of text mining in the context of author and database-curator annotation. Additionally, given the settings, it is more realistic to use no existing annotations on the articles (MeSH Terms, GOA, etc.), so we will separate these two groups, too. The most prestigious class is therefore the online challenge without using any existing annotations. Finally, for reasons explained in the section ONLINE CHALLENGE - "Why is the data sent as structured text but not as XML?", it will be more difficult, yet more realistic, to generate the annotations from the UTF-8 articles and we will analyze differences on annotations stemming from the text resource (UTF-8 or XML) and mapping resource (SwissProt or UniProt) use, too.

Will normalizations or pairs annotated on negative, non-interaction containing articles be evaluated?

No. Given the setting, an author would know if the article contains that information or not and database curators would have decided already wether or not to annotate the article, so if you do annotate these articles, these classifications will not be counted as false positives against your total.

Finally, we want to remind you that another good soruce of information is the FAQ for the BioCreative II PPI task.

The evaluation script has been published and is available for download from the team pages, while a detailed discussion of the evaluation for BC II.5 (AUC of the interpolated P/R curve) has been added to the site.