=begin pod

=TITLE Testing

=SUBTITLE Writing and running tests in Perl 6

Testing code is an integral part of software development.  For instance, how
can one be sure the code works if there isn't some way to verify it behaves
as desired?  This is the role of tests: to provide automated, repeatable
verifications of code behaviour.

In Perl 6, the C<Test> module provides a testing framework similar to the
traditional Perl 5 L<Test::More|http://perldoc.perl.org/Test/More.html>
module.  Thus, anyone familiar with C<Test::More> (and friends) should be
comfortable with Perl 6's C<Test> module.  The C<Test> module is the testing
module used by the official spectest suite.

The testing functions emit output conforming to the L<Test Anything
Protocol|http://testanything.org>.

=head1 Writing tests

As with any Perl project, the tests live under the C<t> directory in the
project's base directory.

A typical test file looks something like this:

    use v6;
    use Test;      # a Standard module included with Rakudo
    use lib 'lib';

    plan $num-tests;

    # .... tests

We ensure that we're using Perl 6, via the C<use v6> pragma, then we load
the C<Test> module and specify where our libraries are.  We then specify how
many tests we I<plan> to run (such that the testing framework can tell us if
more or fewer tests were run than we expected) and when finished with the
tests, we tell the framework that we are I<done> testing.

=head1 Running tests

One can run tests individually by specifying the test filename on the
command line:

    $ perl6 t/test-filename.t

Or via the L<prove|http://perldoc.perl.org/prove.html> command from Perl 5,
where one specifies C<perl6> as the executable to run the tests:

    $ prove --exec perl6 -r t

=head1 Test plans

=item X<plan($count)|plan>

Specify the total number of tests. Usually written at the beginning of a
test file.

    plan 15;   # expect to run 15 tests

In C<subtest>s, C<plan> is used to specify the total number of tests within
the subtest.

If a C<plan> is used, it is not necessary to denote the end of testing with
C<done-testing>.

=item X<done-testing>

Specify that testing has finished.  Use this function when you do not as yet
have a C<plan> for the number of tests to run.  A C<plan> is thus not
required when using C<done-testing>.

=head1 Testing return values

The C<Test> module exports various functions that check the return value of a
given expression, and produce standardized test output accordingly.

In practice, the expression will often be a call to a function or method that
you want to unit-test.

=head2 By truthiness

=item X<ok($value, $description?), nok($condition, $description?)|ok,nok>

The C<ok> function marks a test as passed if the given C<$value>
evaluates to C<True> in a boolean context. The C<nok> function marks
a test as passed if the given value evaluates to C<False>. Both functions
accept an optional C<$description> of the test.

    ok  $response.success, 'HTTP response was successful';
    nok $query.error,      'Query completed without error';

In principle you could use C<ok> for every kind of comparison test, by
including the comparison in the exression passed to C<$value>:

    ok factorial(6) == 720, 'Factorial - small integer';

However, where possible it is better to use one of the specialized comparison
test functions below, because they can print more helpful diagnostics ouput in
case the comparison fails.

=head2 By string comparison

=item X<is($value, $expected, $description?)|is>

Marks a test as passed if C<$value> and C<$expected> compare positively with
the L<eq operator|/routine/eq>. The function accepts an optional
C<$description> of the test.

    is $pdf-document.author, "Joe", 'Retrieving the author field';
    is factorial(6),         720,   'Factorial - small integer';

As a special case, if C<$expected> is a type object, the comparison uses the
C<===> operator instead:

    my Int $a;
    is $a, Int, 'The variable $a is an unassigned Int';

=item X<isnt($value, $expected, $description?)|isnt>

Marks a test as passed if C<$value> and C<$expected> are B<not> equal using
the same rules as C<is()>.  The function accepts an optional C<$description>
of the test.

    isnt pi, 3, 'The constant π is not equal to 3';

    my Int $a = 23;
    $a = Nil;
    isnt $a, Nil, 'Nil should not survive being put in a container';

=head2 By approximate numeric comparison

=item X<is-approx($value, $expected, $description?)|is-approx>

Marks a test as passed if the C<$value> and C<$expected> numerical values
are within C<1e-5> of each other.  The function accepts an optional
C<$description> of the test.

    my $eulers-constant-approx = 2.71828;
    is-approx $eulers-constant-approx, e, "approximate Euler's constant";

=item X<is-approx($value, $expected, $tol, $description?)|is-approx>

Marks a test as passed if the C<$value> and C<$expected> numerical values
are within C<$tol> of each other.  The function accepts an optional
C<$description> of the test.  For example, test for an approximation of pi:

    my $pi-approx = 3.1416;
    my $pi-tol = 0.0001;
    is-approx $pi-approx, pi, $pi-tol, "approximate Pi";
 
=head2 By structural comparison

=item X<is-deeply($value, $expected, $description?)|is-deeply>

Marks a test as passed if C<$value> and C<$expected> compare positively with
the L<eqv operator|/routine/eqv>. This is the best way to check for equality of
(deep) data structures. The function accepts an optional C<$description> of
the test.

=begin code
use v6;
use Test;
plan 1;

sub count-chars(Str() $x) {
    my %chars;
    %chars{$_}++ for $x.comb;
    return %chars;
}
my %expected =
    f   => 1,
    l   => 1,
    e   => 3,
    c   => 1,
    ;
is-deeply count-chars('fleece'), %expected, 'count-chars works on "fleece"';
=end code

Note that string comparison here would have been fragile, because hashes are
not sorted, and converting a L<Hash|/type/Hash> to a string loses information.

=head2 By arbitrary comparison

=item X<cmp-ok($value, $comparison, $expected, $description?)|cmp-ok>

Compares C<$value> and C<$expected> with the given C<$comparison> operator and
passes the test if the comparison yields a C<True> value.  For ease of use,
operators may be passed as strings, such as C<'=='> or C<'~~'>.  The function
accepts an optional C<$description> of the test.

    cmp-ok 'my spelling is apperling', '~~', /perl/, "bad speller";

=head2 By object type

=item X<isa-ok($value, $expected-type, $description?)|isa-ok>

Marks a test as passed if the given object C<$value> is, or inherits from, the
given C<$expected-type>.  For convenience, types may also be specified as a
string.  The function accepts an optional C<$description> of the test.

    class Womble {}
    class GreatUncleBulgaria is Womble {}
    my $womble = GreatUncleBulgaria.new;

    isa-ok $womble, Womble, "Great Uncle Bulgaria is a womble";
    isa-ok $womble, 'Womble';     # equivalent

=head2 By method name

=item X<can-ok($variable, $method-name, $description?)|can-ok>

Marks a test as passed if the given C<$variable> can run the given
C<$method-name>.  The function accepts an optional C<$description>.  For
instance:

    my $womble = Womble.new;

    # with automatically generated test description
    can-ok $womble, 'collect-rubbish';
    #  => An object of type 'Womble' can do the method 'collect-rubbish'

    # with human-generated test description
    can-ok $womble, 'collect-rubbish', "Wombles can collect rubbish";
    #  => Wombles can collect rubbish

=head2 By role

=item X<does-ok($variable, $role, $description?)|does-ok>

Marks a test as passed if the given C<$variable> can do the given C<$role>.
The function accepts an optional C<$description> of the test.

    # create a Womble who can invent
    role Invent {
        method brainstorm { say "Aha!" }
    }
    class Womble {}
    class Tobermory is Womble does Invent {}

    # ... and later in the tests
    use Test;

    my $tobermory = Tobermory.new;

    # with automatically generated test description
    does-ok $tobermory, Invent;
    #  => The object does role Type

    does-ok $tobermory, Invent, "Tobermory can invent";
    #  => Tobermory can invent

=head2 By regex

=item X<like($value, $expected-regex, $description?)|like>

    like 'foo', /fo/, 'foo looks like fo';

Marks a test as passed if the C<$value>, when coerced to a string, matches the
C<$expected-regex>.  The function accepts an optional C<$description> of the
test.

=item X<unlike($value, $expected-regex, $description?)|unlike>

    unlike 'foo', /bar/, 'foo does not look like bar';

Marks a test as passed if the C<$value>, when coerced to a string, does B<not>
match the C<$expected-regex>.  The function accepts an optional C<$description>
of the test.

=head1 Testing modules

=item X<use-ok($module)|use-ok>

Marks a test as passed if the given C<$module> is correctly loaded.

    use-ok 'Full::Qualified::ModuleName';

=head1 Testing exceptions

=item X<dies-ok($code, $description?)|dies-ok>

Marks a test as passed if the given C<$code> throws an exception.

The function accepts an optional C<$description> of the test.

=begin code
sub saruman(Bool :$ents-destroy-isengard) {
    die "Killed by Wormtongue" if $ents-destroy-isengard;
}

dies-ok { saruman(ents-destroy-isengard => True) }, "Saruman dies";
=end code

=item X<lives-ok($code, $description?)|lives-ok>

Marks a test as passed if the given C<$code> B<does not> throw an
exception.

The function accepts an optional C<$description> of the test.

=begin code
sub frodo(Bool :$destroys-ring) {
    die "Oops, that wasn't supposed to happen" unless $destroys-ring;
}

lives-ok { frodo(destroys-ring => True) }, "Frodo survivies";
=end code

=item X<eval-dies-ok($string, $description?)|eval-dies-ok>

Marks a test as passed if the given C<$string> throws an exception when
C<eval>ed as code.

The function accepts an optional C<$description> of the test.

=begin code
eval-dies-ok q[my $joffrey = "nasty";
               die "bye bye Ned" if $joffrey ~~ /nasty/],
    "Ned Stark dies";
=end code

=item X<eval-lives-ok($string, $description?)|eval-lives-ok>

Marks a test as passed if the given C<$string> B<does not> throw an
exception when C<eval>ed as code.

The function accepts an optional C<$description> of the test.

=begin code
eval-lives-ok q[my $daenerys-burns = False;
                die "Oops, Khaleesi now ashes" if $daenerys-burns],
    "Dany is blood of the dragon";
=end code

=item X<throws-like($code, $expected-exception, $description?, *%matcher)|throws-like>

Marks a test as passed if the given C<$code> throws the specific exception
C<$expected-exception>. The code C<$code> may be specified as something
C<Callable>, or as a string to be C<EVAL>ed. The exception may be specified
as a type object, or as a string containing its type name.

If an exception was thrown, it will also try to match the matcher hash, in
which the key is the name of the method to be called on the exception, and
the value is the value it should have to pass. For example:

=begin code
sub frodo { fail "Oops. Frodo dies" unless $destroys-ring };
throws-like { frodo }, Exception, message => /dies/;
=end code

=comment Someone needs to clarify the preceding paragraph, because I have
         no idea what it is talking about... ;)

The function accepts an optional C<$description> of the test.

Please note that you can only use the C<EVAL> form if you are not referencing
any symbols in the surrounding scope.  If you are, you should encapsulate
your string with a block and an EVAL.  For instance:

    throws-like { EVAL q[ fac("foo") ] }, X::TypeCheck::Argument;

=head1 Grouping tests

The result of a group of subtests is only C<ok> if all subtests are C<ok>.

=item X<subtest(&subtests, $description?)|subtest>

The C<subtest> function executes the given block, consisting of usually more
than one test, possibly including a C<plan> or C<done>.  It will pass the
test only if B<all> tests in the block, pass.  The function accepts an
optional C<$description> of the test.

=begin code
class Womble {}

class GreatUncleBulgaria is Womble {
    has $.location = "Wimbledon Common";
    has $.spectacles = True;
}

subtest {
    my $womble = GreatUncleBulgaria.new;

    isa-ok $womble,            Womble,             "Correct type";
    is     $womble.location,   "Wimbledon Common", "Correct location";
    ok     $womble.spectacles,                     "Correct eyewear";

}, "Check Great Uncle Bulgaria";
=end code

=head1 Skipping tests

Sometimes tests just aren't ready to be run, for instance a feature might
not yet be implemented, in which case tests can be marked as C<todo>.  Or it
could be the case that a given feature only works on a particular platform -
in which case one would C<skip> the test on other platforms.

=item X<todo($reason, $count = 1)|todo>

Mark C<$count> tests as TODO, giving a C<$reason> as to why.  By default
only one test will be marked TODO.

    sub my-custom-pi { 3 };

    todo 'not yet precise enough';         # Mark the test as TODO.
    is my-custom-pi(), pi, 'my-custom-pi'; # Run the test, but don't report
                                           # failure in test harness.

=item X<skip($reason, $count = 1)|skip>

Skip C<$count> tests, giving a C<$reason> as to why.  By default only one
test will be skipped.  Use such functionality when a test (or tests) would
die if run.

=begin code
if $*KERNEL ~~ 'linux' {
    is num-forward-slashes("/a/b"),             2;
    is num-forward-slashes(normalize("/a//b")), 2;
}
else {
    skip "Can't use forward slashes on Windows", 2;
}
=end code

=item X<skip-rest($reason?)|skip-rest>

Skip the remaining tests.  If the remainder of the tests in the test file
would all fail due to some condition, use this function to skip them,
providing an optional C<$reason> as to why.

=begin code
unless $location ~~ "Wimbledon Common" {
    skip-rest "We can't womble, the remaining tests will fail";
    exit;
}

# tests requiring functional wombling
ok womble();
# ...
=end code

Note that C<skip-rest> requires a C<plan> to be set, otherwise the
C<skip-rest> call will throw an error. If no plan was set, you can use
C<done> to indicate the end of the test run (though of course the test
summary will not count skipped tests).

=head1 Manual control

If the convenience functionality documented above does not suit your needs,
you can instead use the following functions to direct the test harness
output manually.

=item X<pass($description?), flunk($description?)|pass>

The C<pass> function marks a test as passed.  C<flunk> marks a test as
B<not> passed.  Both functions accept an optional test C<$description>.

    pass "Actually, this test has passed";

    flunk "But this one hasn't passed";

As the Perl 5 documentation for C<Test::More> notes, such constructs are
used when one has a complicated test condition.  As such, they should be
used very, very sparingly.

=item X<diag($message)|diag>

Display diagnostic information in a TAP-compatible manner on the standard
error stream.  It is usually used when a particular test has failed to
provide information that the test itself did not provide.  Or it can be used
to provide visual markers on how the testing of a test-file is progressing
(which can be important when doing stress testing).

    diag "Yay!  The tests got to here!";

=end pod

# vim: expandtab shiftwidth=4 ft=perl6
