Quality Control in Proteomics Workshop
25-27 November 2009
Hinxton, UK

Organisers:

Draft Report

Summary

The ESF Quality Control in Proteomics Workshop was aimed at assembling a critical mass of international participants from academia, industry and scientific journals on the topic of in vitro and in silico quality control procedures for proteomics experiments. The participants also reflected several existing groups that are working around this broader topic in proteomics, such as such as the Fixing Proteomics campaign (www.fixingproteomics.org), the Association of Biomolecular Research Facilities (ABRF), the Human Proteome Organisation (HUPO) Test Samples Initiative, and the HUPO Proteomics Standards Initiative (HUPO PSI). Through lectures and breakout sessions, the 37 participants representing academia, industry, start-ups, and scientific journals discussed and proposed techniques and methods for quality control in proteomics, working towards a draft white paper on the topic. The meeting schedule was based around plenary lectures by invited speakers during the morning sessions, with break-out sessions taking up the afternoon. During the break-out sessions the participants split up into five groups, addressing specific aspects of quality control relating to five main topics: sample and sample processing, instrumentation, workflow, data analysis, and reproducibility and reuse of public data.

The plenary talks provided a very useful overview on existing quality control methods and metrics as employed by academic labs and core facilities, by instrument vendors, and by bioinformaticians. Additionally, representatives from journals provided their perspective on the importance of quality control for data relating to submitted manuscripts. Apart from being highly informative, these sessions also served as a springboard from which to start the constructive discussions in the breakout sessions. The break-out sessions were each moderated by an appointed representative, and this person provided a brief summary of the work performed during the break-out session to close the formal agenda each day. Of course, relevant conversations and discussions continued until well after dinner, as it was arranged that the participants all stayed at the conference hotel, providing ample opportunity to interact and network.

The outcomes so far are threefold. First of all, a Steering Committee has been established to investigate the creation of a formal organisation dedicated to quality control in proteomics, and to organise a follow-up meeting next year, as the participants unanimously agreed that this meeting should be repeated yearly. Second, a Special Issue on quality control in proteomics is being planned for the Wiley journal PROTEOMICS, and the participants, along with others in the field, will be invited to submit manuscripts to this Special Issue. Furthermore, the RSC journal Molecular BioSystems has proposed to publish a meeting report. Third, a whitepaper is currently being drafted that will summarise the current state of the field with regards to quality control, including methods that can be applied today as well as deficiencies in our methodological or technical arsenal. The whitepaper will further provide a perspective on improvements that can be made in the future, and will highlight the overall importance of solid quality control in the maturing field of proteomics mass spectrometry.

Scientific Content

The meeting started off with a talk by Prof. Lennart Martens from Ghent University, Gent, Belgium, that introduced the concept of the workshop, and provided an overview of existing initiatives that already target specific aspects of quality control (QC) in proteomics, including an online QC check during automated batch analyses on mass spectrometers to ensure operational parameters remain met, the HUPO Test Samples initiative’s equimolar twenty-protein sample, and the National Cancer Institute (NCI) CPTAC results with regards to reproducibility and implicit QC metrics. Prof. Martens is active in the ABRF Proteomics Informatics Research Group (iPRG), the HUPO PSI, the HUPO Test Samples Initiative, and the Fixing Proteomics Campaign.

The next talk, by Bruno Domon, ETH Zurich, Switzerland, provided insight into the upcoming HUPO Test Samples Initiative project, dealing with quantitative analyses. Furthermore, Prof. Domon discussed the tradeoffs between different instruments and approaches in proteomics analyses, which will influence the areas where QC will have most impact. The exciting new developments in the field of Selected Reaction Monitoring (SRM, also referred to as Multiple Reaction Monitoring or MRM) were also explained, along with the specific challenges this targeted approach brings. Prof. Domon is active in the HUPO Test Samples Initiative.

The next speaker, Dr. Karl Mechtler from IMP in Vienna, Austria, presented the simple yet powerful CARLA test developed by his group to assess dead volumes in an LC system, and the influence that the chromatography step can have on subsequent analysis on the mass spectrometer. Dr. Mechtler also discussed the work of the ABRF Proteomics Standards research Group (sPRG). This group has contributed several noteworthy studies that have focused on labs’ ability to characterise artificially created samples, ultimately leading to the well-known commercially available UPS-1 and UPS-2 standards. Dr. Mechtkler is a member of the ABRF sPRG group.

Will Dracup from Nonlinear Dynamics Group, UK then presented the results of the first HUPO reproducibility study that focused on the analysis of 2D-gel images. This study found good reproducibility between the various groups, in part because the final analysis was performed on a single software platform. Based on these results, Dr. Dracup proposed that reproducibility of an analysis provides a fundamental QC parameter for any analysis, and that, in an ideal case, all results should be reproduced before being published. This was discussed, as many participants felt that it would often be impractical to make this happen in practice (samples might be limited, shipping of samples to different sites can influence the subsequent measurements, and time might not be available at peer labs to actually perform extensive analyses). He further commented on the direct correlation between the number of samples analysed, and the ability to distinguish significant changes in a study. Interestingly, one of the experiences that Nonlinear has had with the processing of thousands of samples in their software is that a consensus picture can emerge from such aggregated information. As such, a virtual reference could be constructed to serve as a baseline comparator, provided that enough high quality data is released into the public domain. Finally, Dr. Dracup pointed out that the Bell et al. study demonstrated that feedback can improve researchers’ performance, highlighting the importance of QC metrics to provide this sort of objective feedback early on. Dr. Dracup is highly active in the HUPO Industrial Advisory Board (IAB) and has helped launch the Fixing Proteomics educational campaign.

Dr. Matthieu Visser from Philips Research, Eindhoven, the Netherlands, then started off by pointing at three levels of quality in a proteomics study: 1) The quality of the question, i.e. can the hypothesis be answered by the technology used. 2) The quality of the experiment, i.e. have technical and biological replicates been included, and bias excluded e.g. through sample order randomisation. 3) The quality of the data. Regarding this latter aspect, he presented a substantial array of simple metrics that were computed from a relatively small (5000 spectra) pilot experiment with experimental mass spectrometry data from an external supplier. Goal was to inspect the efficiency of e.g. digestion, fragmentation and identification of peptides. Furthermore, Dr. Visser provided additional metrics that can be used in the specific case of quantitative experiments (to assess labelling efficiency and quantification accuracy and dynamic range), and metrics that can be used to compare the performance of different peptide identification algorithms. Overall, Dr. Visser pointed out that performing robust QC based on proteomics pilot data before full-force acquisition is essential. But there is equally a role for QC after full data acquisition, illustrated by an analysis of data redundancy in a large data set (120000 spectra). QC resembles accounting, as it consists primarily of counting, and seeing whether the numbers all add up. This was generally considered a good analogy, but it was also pointed out that a posteriori QC can not rectify a problem early on, and that an entire analysis pipeline needs to be run before metrics become available. This topic was picked up again throughout the workshop, as post-analysis QC was seen as a relatively easy and straightforward approach, but lacking the ability of a priori or in-line QC to halt or pre-empt a time-consuming (and often expensive) analysis altogether if a sample or processing step did not fall within acceptable parameters.

Dr. Katleen Verleysen from Pronota, Gent, Belgium, then introduced an overview of the QC requirements in a biotech start-up company focused on biomarker discovery. Dr. Verleysen illustrated that, as a project matures from the discovery phase over the assay development phase to a clinical study, the QC requirements continue change and evolve. She also pointed out that QC entails careful project planning and that high sample quality is paramount for the entire downstream workflow, again emphasising that a priori methods to assess sample fitness are important future goals for QC, a motion that met with a large amount of approval.

The next speaker, Dr. Hans Vissers from Waters Corporation, Manchester, UK, was the first representative from an instrument vendor to present, and he provided a broad scope of QC as it applies to the manufacturing chain of an instrument, all the way to the acceptance criteria that need to be fulfilled at the customer site. Dr. Vissers also pointed out that for this latter goal, the on-site acceptance test, Waters is now pioneering a total system test that simultaneously verifies HPLC and MS performance. Furthermore, Dr. Vissers presented a data simulator tool developed by Waters that can simulate a complete LC-MS/MS run based on a sequence database and a variety of input parameters. The output of the simulator is a raw mass spectrometer output file as would be obtained from a normal experiment. Such tools could play a role in future tests of data processing and identification software, amongst others. Finally, Dr. Vissers also expanded on some ideas of Dr. Dracup, by touching upon the concept of ‘virtual reproducibility’, in which data acquired in a lab could be compared to similar datasets in the public domain, providing a comparison including reproducibility metrics.

Dr. Sarah Robinson, of Thermo Fisher Scientific, Hemel Hempstead, UK, also a vendor representative, detailed the existing expertise in SRM QC and guidelines from the small molecules field. Dr. Robinson also emphasised the importance of a sufficient number of intra-assay and inter-assay replicates (also referred to as ‘technical’ and ‘biological’ replicates in the life sciences) to ensure significance of the results. Furthermore, she presented data on sample losses due to adhesion or adsorption to vessel walls, and presented results from an inter-lab study that showed that reproducibility can be achieved in SRM assays on clinical samples.

The first speaker of the second day, Prof. Dr. Kathryn Lilley from Cambridge University, Cambridge, UK, covered QC from the point of view of a core facility. Similar to the contributions by Dr. Verleysen earlier, she emphasised the importance of QC throughout the entire process, starting from initial conversations with the facility user or customer about the requested analysis and expected results, all the way to the reporting and archiving of the results. Dr. Lilley also highlighted the importance of economical, small networking and method-sharing meetings to boost the communication of technical and methodological advances in the field, especially since these innovations are often not reported separately in journals. Another topic brought forward by Dr. Lilley touched on the necessity of reliable and economical standards, to be distributed by consumables vendors, to enable QC steps to take place without requiring sample analysis. She also pointed out that a downstream goal of established QC is accreditation testing. Dr. Lilley also presented results from the ABRF Proteomics Research Group (PRG), which initiated whole-workflow studies of mass spectrometry based proteomics analyses. The 2006 and 2007 studies were highlighted specifically, with the former employing an 8-protein sample for quantitative analyses, and the latter consisting of an E. coli lysate spiked with 12 peptides. Perhaps the most interesting result was that performance correlated directly with reported experience. Dr. Lilley is very active in ABRF, and the Fixing Proteomics campaign.

Dr. René Zahedi from the Institute for Analytical Sciences (ISAS), Dortmund, Germany next spoke about the QC protocols that are carried out in an analytical institute. Dr. Zahedi started his talk by defining QC as the means to validate an analytical procedure, and then expanding on this definition by dubbing it the monitoring of systems (or workflows) for day-to-day, between-instrument, and between- user reproducibility, thus linking back to the concept of reproducibility as the ultimate QC test as proposed by Dr. Dracup in his lecture, and the importance of standard operating procedures (SOPs) as highlighted by various speakers. Dr. Zahedi further illustrated the importance of QC by showing the influence of difficult-to-control variables on instrumentation, and highlighted some state-of-the-art QC procedures that can be available in an analytical institute (e.g., electron micrographs of columns). In the closing part of his talk, Dr. Zahedi introduced the various strategies they employ to ensure the quality of their peptide and protein identifications and modification site assignments.

Dr. Ola Forsstrom-Olssen, from Ludesi, Sweden, next presented a highly interesting, freely available application developed by his company to allow any user to perform a semi-automated quality control on their 2D gel image analysis. The software strikes a careful balance between the necessity to automatically call and match gel spots (due to the sheer amount of spots that need to be processed) and the need for manual validation of these automatic assignments for QC purposes. The software first asks the user to specify a number of spots to evaluate. Subsequently, that number of spots will be randomly selected from the data, and will be shown in detail to the user. For each spot, the user has to grade the quality of the assignment. As soon as the manual validation quota is reached, the tool outputs the quality metrics, including a‘combined correctness’ score. Dr. Forsstrom-Olssen is active in the HUPO Industrial Advisory Board.

The following speaker, Dr. Mats Borén of Denator, Sweden, focused on preserving sample quality in proteomics workflows, as the degradation of samples through remaining enzyme activity after lysis is a well-known phenomenon (particularly in plasma). The stabilisation of samples of course ties in directly with the previously highlighted need for high quality samples, and touches upon the related goal of coming up with a priori sample quality metrics.

The next speaker, Prof. Dr. Lukas Käll from Stockholm University, Sweden, specifically talked about quality control applied to automated peptide identifications. The currently popular practice of calculating false discovery rates (FDRs) has the unfortunate issue that it can fluctuate depending on the precise cut-off point taken, and Prof. Käll proposed q values as a stable alternative when sets of identifications are searched. Additionally, rather than assigning each peptide identification with an expectancy value (reflecting the chance of obtaining the score given that it is a random match), it is preferable to rely on posterior error probabilities (PEP, also called a local-FDR). The PEP provides the chance that a peptide of the given score is an incorrect match, and this is a much more useful metric when assessing confidence of a selected identified peptide.

In the next talk, Dr. Sara ten Have from Dundee University, UK presented her views on quality control throughout a complex proteomics pipeline. Dr. ten Have actually performs a rather novel and specific role, as she serves as a ‘proteomics consultant’ in the group, providing detailed guidance about the various nuances of proteomics experiments to biologists. Another topic presented by Dr. ten Have focused on the LIMS system developed in the group, called PepTracker. The use of such a comprehensive data management system has a number of advantages, but most notably, it enables the group to look at results across experiments, providing a de facto super-experiment view.

The final speaker of the day, Marc Vaudel from ISAS, Dortmund, Germany, focused on a very technical, yet equally important topic: the effects of signal processing on the quantification of proteins using mass spectrometry. In particular, Mr. Vaudel focused on the quantification of the errors that can be made during the initial signal processing, and which relate to intricate details of the algorithms such as the peak model used, or the noise detection and filtering algorithms. These errors can relatively easily accumulate to 10 or even 20% of the measurement. The conclusion of Mr. Vaudel was therefore that signal processing should not be treated as a black box, and that quality control steps at these early processing stages are crucial to ensure downstream quality in a data analysis pipeline.

The third day saw Dr. Michael Smith from the Royal Society of Chemistry (RSC), Cambridge, UK as its first speaker. Dr. Smith is Deputy Editor of three journals at RSC Publishing, and presented his views on quality control from a journal editor perspective. The RSC journals have a long-standing history in this topic, as the field of chemistry benefits from long-standing and stringent QC metrics throughout many of its subdisciplines. Dr. Smith illustrated this for the field of small molecule analysis, and showed that mandatory deposition of source data in a centralised public repository, along with (semi-)automated QC steps on the data can be extremely effective in bolstering the overall quality of the work, and subsequent data reuse. Dr. Smith also pointed out that one of the requirements for such an approach to work is the support from publishers, a process that has is now ongoing in proteomics as well.

This was evident in the talk of the next speaker, Prof. Dr. Roz Banks from Leeds University, UK. Prof. Banks is Editor of the Wiley journal Proteomics, and Associate Editor for its sister journal Proteomics Clinical Applications (PCA). Prof. Banks started off by introducing the newly minted publication guidelines for PCA, highlighting the stringent criteria that will need to be met before manuscripts can be considered for review. Prof. Banks also showed data from her own lab, illustrating that outside effects such as temperature can have a strong influence on the sample and the data obtained. This relates to similar observations made by Dr. Zahedi in his talk.

The next speaker, Dr. Juan Pablo Albar of CNB-CSIC and the ProteoRed Consortium in Madrid, Spain, described the comparative inter-laboratory studies performed by the ProteoRed Consortium. These results show that SOPs and QC are crucial for reproducibility, and that inter-lab reproducibility can be achieved if these criteria are met.

The final speaker, Dr. Chris Taylor from EMBLEBI in Cambridge, UK, discussed the relationship between quality control and minimal reporting requirements formulated in the field of proteomics. Dr. Taylor is very active in the HUPO PSI.

Assessment of the results & impact of the event on the future direction of the field

The workshop set four major goals: (i) to build a critical mass of proteomics scientists around the issue of quality control, ensuring tight links with existing initiatives that touch upon the same topic, and wide representation from academia, industry and publishers; (ii) to produce a whitepaper with the conclusions of the workshop, and a view on the future of quality control in proteomics; (iii) to organise a Special Issue in a well-known journal in the field, to bring the topic to the attention of all scientists in the field; and (iv) to investigate the feasibility of forming a consortium or organisation around to topic to take these initial discussions further into standard implementations, educational resources and workshops, and journal guidelines. The final plenary session of the workshop discussed the progress made during the workshop in light of these objectives. First of all, it was widely agreed that the breakout sessions were very successful, and that the notes taken by the different working group chairs already contain a substantial amount of information for the whitepaper. The whitepaper proper was drafted in outline during the workshop, and will now be written up over the next two months, assembling input from the notes made by the working group chairs as well as all individual participants. Furthermore, there was a proposal from the journal Proteomics to work out a Special Issue on ‘Quality Control in Proteomics’ with them. This Special Issue is now being worked on by Dr. Juan Antonio Vizcaíno (EMBLEBI, Cambridge UK), Prof. Dr. Roz Banks, and Prof. Dr. Lennart Martens, who will also act as (guest) editors for the Special Issue. Contributions for this Special Issue will be solicited from the workshop participants, with additional invitations going out to other interested groups as well. Additionally, Dr. Michael Smith requested a meeting report for the workshop be submitted to Molecular BioSystems, one of the journals he is Deputy Editor of. This report will also be written with support from the workshop participants, and we are aiming for submission in mid-January 2010. With the first three goals either achieved or in well underway, the discussion then focused on the feasibility of creating an organisation dedicated to quality control in proteomics. It was agreed to set up a steering committee, consisting of several representative participants, and to appoint several other participants as liaisons with various existing organisations that already touch upon aspects of quality control in proteomics. A tentative name was proposed as well: ‘iQCiP’ (initiative for Quality Control in Proteomics). The Steering Committee consists of eight members, which comprises all participants who volunteered to take up this position. The members are (in alphabetical order): Dr. Juan Pablo Albar, Prof. Dr. Bruno Domon, Prof. Dr. Lukas Käll, Prof. Dr. Karl Mechtler, Dr. Sarah Robinson, Dr. Matthieu Visser, Dr. Juan Antonio Vizcaíno, and Prof. Dr. Lennart Martens as ad hoc secretary. Liaisons to external bodies were also agreed upon, and the following people have taken up liaison roles: Prof. Dr. Kathryn Lilley (ABRF), Prof. Dr. Bruno Domon (HUPO Test Samples) and Dr. Ola Forsstrom-Olssen (HUPO Industrial Advisory Board), Dr. Christian Stephan, Ruhr-Universität Bochum, Germany (HUPO PSI), Paddy Lavery, Nonlinear Dynamics Group, Newcastle-upon-Tyne, UK (Fixing Proteomics Campaign), Dr. Michael Smith (RSC Publishing) and Prof. Dr. Roz Banks (Wiley), Dr. Juan Antonio Vizcaíno (proteomics data repositories and the ProteomExchange Consortium), and Prof. Dr. Lennart Martens (funding agencies, EC and ESF). It was agreed that the first tasks of the Steering Committee are the writing and release of the whitepaper, along with the planning of a follow-up workshop to be held next year. The Steering Committee should by then also look into the establishment of a formal organisation, to be inaugurated at the next (yearly) workshop.

From these outcomes, it is clear that the workshop was indeed well-timed, and managed to assemble a critical mass to take the topic of quality control in proteomics forward and outward. The impact of the field will effectively be maximised through several ways of outreach: (i) a whitepaper will be written up that will summarise the outcome of the discussions that took place during the meeting, and that will provide a vision for the future with regards to quality control; (ii) there is strong progress toward the publication of a meeting report as well as a Special Issue on the topic in two prominent journals in the field; (iii) a Steering Committee with very broad and representative membership has been established, along with liaisons to various existing initiatives, around the topic of quality control in proteomics; and (iv) a follow-up workshop will be planned for next year, to ensure that the momentum is maintained and
extended. With these various initiatives in place, or well advanced in their planning phase, the outcome of the meeting will be highly visible in the field, and from the feedback of the participants, it is clear that there is a strong demand for a more prominent role for quality control in the field of proteomics.

Programme

Day 1  --Wednesday, 25 November 2009
09:30 -10:00 Introduction + workshop concept Prof. Dr. Lennart Martens, UGent/VIB, Belgium
10:00 -10:30 Experiences from ETHZ and the HUPO Test Sample efforts Prof. Dr. Bruno Domon, ETHZ, Switzerland
10:30 -11:00 Experiences of the IMP and the ABRF Proteomics Standards Group Prof. Dr. Karl Mechtler, IMP Vienna, Austria
11:00 -11:20 Coffeebreak
11:20 -11:50 Experiences of the Fixing Proteomics Campaign Dr. Will Dracup, Nonlinear Dynamics, UK
11:50 -12:20 A view to quality control from industry Dr. Matthieu Visser, Philips Research, The Netherlands
12:20 -12:50 Quality control in the biotech start-up Dr. Katleen Verleysen, Pronota, Belgium
Lunch
13:50 -14:20 Vendor Quality Control standards for mass spectrometry (I) Dr. Hans Vissers, Waters Corporation, UK
14:20 -14:50 Vendor Quality Control standards for mass spectrometry (II) Dr. Sarah Robinson, Thermo Fisher Scientific, UK
14:50 -15:00 Delineation of key areas of proteomics experiments affected by quality control
Tentatively: sample extraction, sample separation, sample processing, mass spectrometry
15:00 -15:20 Coffee break
15:20 -17:00 Breakout sessions according to the key areas delineated in the previous session
17:00 -17:30 Progress reports from each breakout session
18:00 -20:00 Dinner and discussions

 Day 2 --Thursday, 26 November 2009
09:00 -09:30 Quality Control in core facilities Prof. Dr. Kathryn Lilley, Cambridge, UK
09:30 -10:00 Quality Control in analytical science Dr. René Zahedi, ISAS Dortmund, Germany
10:00 -10:30 Providing software for quality control in 2D gels Dr. Ola Forsstrom-Olsen, Ludesi, Sweden
10:30 -11:00 Coffee break
11:00 -11:30 Quality control and sample preparation Dr Mats Borén, Denator
11:30 -12:00 Quality control for data analysis in mass spectrometry Prof. Dr. Lukas Käll, Stockholm, Sweden
12:00 -12:30 Quality control in Proteomic Qualitative and Quantitative studies Dr. Sara Ten Have, Dundee, UK
Lunch
13:30 -14:00 Quality control in quantitation studies Marc Vaudel, ISAS Dortmund, Germany
14:00 -14:30 Coffee break
14:30 -17:00 Breakout sessions (continued from previous day)
17:00 -17:30 Progress reports from each breakout session
18:00 -20:00 Dinner and discussions

Day 3 --Friday 27 November 2009
09:00 -09:30 Quality control and journals (I) Dr. Michael Smith, Molecular BioSystems, Cambridge, UK
09:30 -10:00 Quality control and journals (II) Prof. Dr. Roz Banks, PROTEOMICS and PCA, Leeds, UK
10:00 -10:30 Quality control in the ProteoRed Consortium Dr. Juan Pablo Albar, CNB-CSIC/ProteoRed, Madrid, Spain
10:30 -11:00Coffee break
11:00 -11:45 Quality control and minimal reporting guidelines Dr. Chris Taylor, EMBL-EBI, UK
Lunch
13:00 -13:30 Creation of the Proteomics Quality Control Consortium
13:30 -14:00 White paper outline and division of writing tasks by breakout session
14:00 -14:30 Coffee break
14:30 -16:30 Overall discussion and start of writing tasks
16:30 Meeting end