Code coverage criteria for asynchronous programs

In recent years, asynchronous programming has gained significantly in popularity. Asynchronous software often exhibits complex and error-prone behaviors and should therefore be tested thoroughly. Code coverage has been the most popular metric to assess test suite quality. However, traditional code coverage criteria are not sufficient as a measure of test adequacy for asynchronous applications. In particular, they do not adequately reflect completion, interactions, and error handling of asynchronous operations. This research proposes novel test adequacy criteria for measuring: (i) eventual completion of asynchronous operations in terms of both successful and exceptional execution, (ii) registration of reactions for handling both possible outcomes, and (iii) execution of said reactions through tests. We present JScope, a code coverage tool for automatically measuring these criteria in JavaScript applications and implement it as an interactive plug-in for Visual Studio Code. An evaluation of JScope on 20 JavaScript applications shows that the proposed code coverage criteria can help improve assessment of test adequacy, complementing traditional criteria. Furthermore, an investigation of 15 real GitHub issues concerned with asynchrony demonstrates that the new criteria can help reveal faulty asynchronous behaviors that are untested and are deemed covered by traditional coverage criteria. We also report on a controlled experiment with 12 participants to investigate the usefulness of JScope in realistic settings, demonstrating that it is effective in improving programmers' ability to assess test adequacy and detect untested behavior, and that it can be helpful for debugging.