Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Abstract

RESTful APIs are a type of web service that are widely used in industry. In the past few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real faults were automatically found in a large variety of RESTful APIs. However, usually the analyzed fuzzers treat the APIs as black-box, and no analysis of what is actually covered in these systems is done. Therefore, although these fuzzers are clearly useful for practitioners, we do not know their current limitations and actual effectiveness. Solving this is a necessary step to be able to design better, more efficient, and effective techniques. To address this issue, in this article we compare seven state-of-the-art fuzzers on 18 open source—1 industrial and 1 artificial—RESTful APIs. We then analyze the source code for which parts of these APIs the fuzzers fail to generate tests. This analysis points to clear limitations of these current fuzzers, listing concrete follow-up challenges for the research community.