fuzzr: Fuzz-Testing for R Functions

Matthew Lincoln

2017-05-01

R’s dynamic typing can be both blessing and curse. One drawback is that a function author must decide how to check which inputs should be accepted, and which should throw warnings or errors. fuzzr helps you to check how cleanly and informatively your function responds to a range of unexpected inputs.

Say we build a function intended to a single string and a single integer, repeat the string that number of times, and paste it together using a given delimiter:

library(fuzzr)
# Note that, while we are specifically fuzz testing the 'n' argument, we still # need to provide an 'x' argument to pass along to my_function(). We do not have# to supply a delimiter, as my_function() declares a default value for this# argument.
my_fuzz_results <-fuzz_function(my_function, "n", x =1:3, tests =test_all())
# Produce a data frame summary of the results
fuzz_df <-as.data.frame(my_fuzz_results)
knitr::kable(fuzz_df)

n

x

output

messages

warnings

errors

result_classes

results_index

char_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

1

char_single

1:3

NA

NA

NAs introduced by coercion

invalid ‘times’ argument

NA

2

char_single_blank

1:3

NA

NA

NA

invalid ‘times’ argument

NA

3

char_multiple

1:3

NA

NA

NAs introduced by coercion

invalid ‘times’ argument

NA

4

char_multiple_blank

1:3

NA

NA

NAs introduced by coercion

invalid ‘times’ argument

NA

5

char_with_na

1:3

NA

NA

NAs introduced by coercion

invalid ‘times’ argument

NA

6

char_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

7

char_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

8

int_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

9

int_single

1:3

NA

NA

NA

NA

character

10

int_multiple

1:3

NA

NA

NA

NA

character

11

int_with_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

12

int_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

13

int_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

14

dbl_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

15

dbl_single

1:3

NA

NA

NA

NA

character

16

dbl_mutliple

1:3

NA

NA

NA

NA

character

17

dbl_with_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

18

dbl_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

19

dbl_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

20

fctr_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

21

fctr_single

1:3

NA

NA

NA

NA

character

22

fctr_multiple

1:3

NA

NA

NA

NA

character

23

fctr_with_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

24

fctr_missing_levels

1:3

NA

NA

NA

NA

character

25

fctr_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

26

fctr_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

27

lgl_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

28

lgl_single

1:3

NA

NA

NA

NA

character

29

lgl_mutliple

1:3

NA

NA

NA

NA

character

30

lgl_with_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

31

lgl_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

32

lgl_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

33

date_single

1:3

NA

NA

NA

NA

character

34

date_multiple

1:3

NA

NA

NA

invalid ‘times’ argument

NA

35

date_with_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

36

date_single_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

37

date_all_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

38

raw_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

39

raw_char

1:3

NA

NA

NA

NA

character

40

raw_na

1:3

NA

NA

NA

invalid ‘times’ argument

NA

41

df_complete

1:3

NA

NA

NA

(list) object cannot be coerced to type ‘integer’

NA

42

df_empty

1:3

NA

NA

NA

invalid ‘times’ argument

NA

43

df_one_row

1:3

NA

NA

NA

invalid ‘times’ argument

NA

44

df_one_col

1:3

NA

NA

NA

invalid ‘times’ argument

NA

45

df_with_na

1:3

NA

NA

NA

(list) object cannot be coerced to type ‘integer’

NA

46

null_value

1:3

NA

NA

NA

invalid ‘times’ argument

NA

47

Almost all the unexpected values for n throw the fairly generic warning invalid 'times' argument, which really comes from the rep function within my_function. Some types, like doubles, factors, and even dates (!) don’t throw errors, but instead return a result. We can check the value of that result with fuzz_value(), and the call originating it with fuzz_call(), both of which search for the first test result that matches a regex of the test name. The argument should match the name of the argument tested with in fuzz_function:

Fuzzing multiple arguments

fuzz_function works by mapping several test inputs over one argument of a function while keeping the other arguments static. p_fuzz_function lets you specify a battery of tests for each variable as a named list of named lists. Every test combination is then run. These tests can be specified using the provided functions like test_char, or with variable inputs you provide. Remember that each test condition must, itself, be named.