Step 5: More about grading functions for functions

Warning This step is ahead of the current version of [learn-ocaml].

Different grading functions for functions

There are 2 main grading functions:

test_function_<nb_args>_againt_solution: the usual. Test the student code against a given solution written in the solution.ml file.
test_function_<nb_args>: compare the student code to a series of tests where both inputs and expected outputs are given. Note that you still need to write a solution in solution.ml to build your exercise session since learn-ocaml build tests your grader with solution.ml as the student copy.

A few words about grading function for functions

Grading functions for functions return a global report concatening 4 specific ones:

report resulting of comparison between student and solution outputs. The function ~test is used to build this report. By default, ~test uses the structural equality to compare outputs.
report resulting of comparison between student and solution standard outputs. The function ~test_stdout is used to build this report. By default, ~test_stdout is set to ignore standard output and returns an empty report.
report resulting of comparison between student and solution standard errors. The function ~test_stderr is used to build this report. By default, ~test_stderr is set to ignore standard error and returns an empty report.
report resulting of the result of function ~after and returns an empty report by default.

`test_function_<nb_args>_against_solution`

Signature for unary function

val test_function_1_against_solution :
  ?gen: int ->
  ?test: 'b tester ->
  ?test_stdout: io_tester ->
  ?test_stderr: io_tester ->
  ?before_reference : ('a -> unit) ->
  ?before_user : ('a -> unit) ->
  ?after : ('a -> ('b * string * string) -> ('b * string * string) -> Learnocaml_report.report) ->
  ?sampler : (unit -> 'a) ->
  ('a -> 'b) Ty.ty -> string -> 'a list -> Learnocaml_report.item

Mandatory arguments [`test_function_1_againt_solution_1 ty name tests`]:

ty: type of the tested function specified for the given tests. It must not contain type variables (i.e. 'a, 'b etc..), match the type of the tests and be compatible with the solution.
name: name of the tested function in the student code and in solution.ml
tests: list of tested inputs.

`test_function_<nb_args>`

Signature of grading function for unary function

val test_function_1 :
  ?test: 'b tester ->
  ?test_stdout: io_tester ->
  ?test_stderr: io_tester ->
  ?before : ('a -> unit) ->
  ?after : ('a -> ('b * string * string) -> ('b * string * string) -> Learnocaml_report.report) ->
  ('a -> 'b) Ty.ty -> string -> ('a * 'b * string * string) list -> Learnocaml_report.item

Mandatory arguments [`test_function_1 ty name tests`]:

ty: type of the tested function specified for the given tests. It must not contain type variables (i.e. 'a, 'b etc..), match the type of the tests and be compatible with the solution.
name: name of the tested function in the student code and in solution.ml
(in, out, stderr, stdout) :: _ as tests: list of tuples where the first element in is the argument passed to the tested function (for a three-arguments functions tests will instead matched (in1, in2, in3, out, stderr, stdout)). The second element out is the expected output. stderr and stdout are the expected strings in standard output and error output respectively.

Optional arguments of grading functions

gen: number of automatically generated tests. See step-3 and step-4 for more information.
sampler: used to define sampler for automatically generating inputs for tests. See step-3 and step-4 for more information.
test: is used to redefine the function which compare the output of the student function and the output of the solution. See WIP for more information.
test_stdout: is used to redefine the function which compare the standard output channel of the student function and the one of the solution. See WIP for more information.
test_stderr: is used to redefine the function which compare the standard error channel of the student function and the one of the solution. See WIP for more information.
after: is used to redefine a function which is called with the current tested inputs, the student result and the solution result and returns a report. Enables for example to inspect references introduced with ~before, ~before_user or ~before_reference and build an appropriate report. See WIP for more information.
before_reference: is used to redefine a function called right before the application of the current tested inputs to the solution. This enables for example to introduce a reference or make a side effect before each test. See WIP for more information.
before_user: is used to redefine a function called right before the application the current tested inputs to the student function. This enables to introduce a reference or make a side effect between solution evaluation and student function evaluation. See WIP for more information.
before: same as before_reference for test_function_<nb_args>. Since no solution is evaluated, there is no need to distinguish between before or after solution evaluation.

Examples

Note: only trivial examples can be found here. For more advanced examples, see the corresponding tutorials.

Identity

This is a classical example of an unary function, with a user-defined sampler.

let exercise_1 =
  test_function_1_against_solution
  [%ty: int -> int]
  "identity"
  ~gen:10
  ~sampler:(fun () -> Random.int 42)
  [0 ; 42]

With test_function_1:

let exercise_2 =
  test_function_1
    [%ty: int -> int]
    "identity"
    [0, 0, "", "" ;
    42, 42, "", ""]  (* List of tests *)

Note that since the default comparison function for standard output and error actually ignore them, we could put anything as expected outputs in the tests list.

Hello world

Hello world is also very classical, however grading this function is a little more tricky since by default standard output and error output are ignored (meaning the student standard/error output can be different from the solution ones without causing a failure).

let exercise_2 =
  test_function_1_against_solution
    [%ty: unit -> unit]
    ~test:test_ignore
    ~test_stdout:io_test_equals
    "hello"
    ~gen:0
    [()]

let exercise_4 =
  test_function_1
  [%ty: unit -> unit]
  ~test:test_ignore
  ~test_stdout:io_test_equals
  "hello"
  [(), (), "Hello world!", ""]

The function test_ignore for optional argument test enables to ignore the student and solution output. On the contrary, io_test_equals for test_stdout enables to compare standard outputs with the structural equality.