International Journal on Software Tools for Technology Transfer

STTT Cover

Competitions and Challenges (CoCha)

Research Competitions and Challenges can be a driving force in transferring theoretical results into working software tools that demonstrate the state of the art. Competitions provide a controlled comparison and independent evaluation of tool capabilities on subject matter selected by experts within the field. Challenges pose community-oriented research problems and tool capability objectives and may provide exemplar systems on which many teams of researchers may apply their techniques to comparatively assess capabilities, identify community technology gaps, and discover and illustrate synergies between technologies developed by different groups.

This STTT theme area provides a forum for overview articles about new competitions and challenges, progress reports of established efforts, and articles that provide insights about research challenges as a driver of progress within the scientific community.

Competitions: Typical characteristics of tool competitions include (a) a common collection of examples/benchmarks to which tools will be applied, where examples have been curated by experts within the field to exercise different dimensions of tool capabilities, (b) controlled application of tools to examples in a manner that removes ambiguities and uncertainties about tool configuration and enables results to be repeated, (c) independent evaluation of results by expert, and (d) a wide call and participation that helps ensure that competition winners are reflective of the state-of-the-art. Some of the most impactful competitions for formal methods tools have been the SAT competitions (first held in 1992) for satisfiability solvers and the CASC competitions (first held in 1996) for automated theorem provers. Since 2000, the number of dedicated verification competitions has been steadily increasing, also in areas different from verification. Competition victories provide motivation and “bragging rights” to individual research groups. Yet, strong benefits for the entire community include the ability to identify both successful strategies and algorithms as well as weaknesses and capability gaps that the community can address. Yearly competitions enable experts within the community as well as potential consumers of the technology to track the advance of tool capabilities across time.

This STTT theme area seeks Competition-related papers on (but not limited to) the following topics:

Challenge Problems: Challenge problems come in several forms. They may be narratives describing research goals that require broad community action to be met. A well-known example is the Verifying Compiler Grand Challenge of Tony Hoare. They may also be system artifact exemplars with challenge goals for verification or some other aspect of development. An example is the Pacemaker Challenge which presented an industry requirements document for a pacemaker medical device and called on researchers to demonstrate the effectiveness of rigorous development methods for behavior specification, architecture descriptions, implementation, testing/verification, and certification using the real-world description provided by the requirements. The intent of both types of challenges is to spur the community to action around a common, concrete and challenging system – where clear successes in building and assuring the system correspond to significant improvements in the state-of-the-art that are meaningful to a variety of stakeholders.

This STTT theme area seeks Challenge-related papers on (but not limited to) the following topics:

Dirk Beyer (LMU Munich, Germany)
Marieke Huisman (Twente University, Netherlands)
Theme Editors-in-Chief of CoCha

Editorial Board