=begin pod

=TITLE Grammars

=SUBTITLE Parsing and interpreting text

Grammars are a powerful tool used to destructure text and often to return
data structures that have been created by interpreting that text.

For example, Perl 6 is parsed and executed using a Perl 6-style grammar.

An example that's more practical to the common Perl 6 user is the
L<JSON::Tiny module|https://github.com/moritz/json>, which can deserialize
any valid JSON file, however the deserializing code is written in less than
100 lines of simple, extensible code.

If you didn't like grammar in school, don't let that scare you off grammars.
Grammars allow you to group regexes, just as classes allow you to group
methods of regular code.

=head1 X<Named Regexes|declarator,regex;declarator,token;declarator,rule>

The main ingredient of grammars is named L<regexes|/language/regexes>.
While the syntax of L<Perl 6 Regexes|/language/regexes> is outside the scope
of this document, I<named> regexes have a special syntax, similar to
subroutine definitions:N<In fact, named regexes can even take extra
arguments, using the same syntax as subroutine parameter lists>

    =begin code :allow<B>
    my B<regex number {> \d+ [ \. \d+ ]? B<}>
    =end code

In this case, we have to specify that the regex is lexically scoped using
the C<my> keyword, because named regexes are normally used within grammars.

Being named gives us the advantage of being able to easily reuse the regex
elsewhere:

    =begin code :allow<B>
    say "32.51" ~~ B<&number>;
    say "15 + 4.5" ~~ /B<< <number> >>\s* '+' \s*B<< <number> >>/
    =end code

B<C<regex>> isn't the only declarator for named regexes -- in fact, it's the
least common. Most of the time, the B<C<token>> or B<C<rule>> declarators
are used. These are both I<ratcheting>, which means that the match engine
won't back up and try again if it fails to match something. This will
usually do what you want, but isn't appropriate for all cases:

    =begin code :allow<B>
    my regex works-but-slow { .+ q }
    my token fails-but-fast { .+ q }
    my $s = 'Tokens won\'t backtrack, which makes them fail quicker!';
    say so $s ~~ &works-but-slow; # True
    say so $s ~~ &fails-but-fast; # False, the entire string get taken by the .+
    =end code

The only difference between the C<token> and C<rule> declarators is that the
C<rule> declarator causes L<C<:sigspace>|/language/regexes#Sigspace> to go
into effect for the Regex:

    =begin code :allow<B>
    my token non-space-y { 'once' 'upon' 'a' 'time' }
    my rule space-y { 'once' 'upon' 'a' 'time' }
    say 'onceuponatime'    ~~ &non-space-y;
    say 'once upon a time' ~~ &space-y;
    =end code

=head1 X<Creating Grammars|class,Grammar;declarator,grammar>

=SUBTITLE Group of named regexes that form a formal grammar

L<Grammar|/type/Grammar> is the superclass that classes automatically get when
they are declared with the C<grammar> keyword instead of C<class>. Grammars
should only be used to parse text; if you wish to extract complex data, an
L<action object|/language/grammars#Action_Objects> is recommended to be used in
conjunction with the grammar.

=head1 Action Objects

A successful grammar match gives you a parse tree of L<Match|/type/Match>
objects, and the deeper that match tree gets, and the more branches in the
grammar are, the harder it becomes to navigate the match tree to get the
information you are actually interested in.

To avoid the need for diving deep into a match tree, you can supply an
I<actions> object. After each successful parse of a named rule in your
grammar, it tries to call a method of the same name as the grammar rule,
giving it the newly created L<Match|/type/Match> object as a positional
argument. If no such method exists, it is skipped.

Here is a contrived example of a grammar and actions in action:

=begin code
use v6;

grammar TestGrammar {
    token TOP { ^ \d+ $ }
}

class TestActions {
    method TOP($/) {
        $/.make(2 + $/);
    }
}

my $actions = TestActions.new;
my $match = TestGrammar.parse('40', :$actions);
say $match;         # ｢40｣
say $match.made;    # 42
=end code

An instance of C<TestActions> is passed as named argument C<actions> to the
L<parse> call, and when token C<TOP> has matched
successfully, it automatically calls method C<TOP>, passing the match object
as an argument.

To make it clear that the argument is a match object, the example uses C<$/>
as a parameter name to the action method, though that's just a handy
convention, nothing intrinsic. C<$match> would have worked too. (Though using
C<$/> does give the advantage of providing C<< $<capture> >> as a shortcut
for C<< $/<capture> >>).

A slightly more involved example follows:

=begin code
use v6;

grammar KeyValuePairs {
    token TOP {
        [<pair> \n+]*
    }
    token ws { \h* }

    rule pair {
        <key=.identifier> '=' <value=.identifier>
    }
    token identifier {
        \w+
    }
}

class KeyValuePairsActions {
    method identifier($/) { $/.make: ~$/                          }
    method pair      ($/) { $/.make: $<key>.made => $<value>.made }
    method TOP       ($/) { $/.make: $<pair>».made                }
}

my  $res = KeyValuePairs.parse(q:to/EOI/, :actions(KeyValuePairsActions)).made;
    second=b
    hits=42
    perl=6
    EOI

for @$res -> $p {
    say "Key: $p.key()\tValue: $p.value()";
}
=end code

This produces the following output:

=begin code
Key: second     Value: b
Key: hits       Value: 42
Key: perl       Value: 6
=end code

Rule C<pair>, which parsed a pair separated by an equals sign, aliases the two
calls to token C<identifier> to separate capture names to make them available
more easily and intuitively. The corresponding action method constructs a
L<Pair|/type/Pair> object, and uses the C<.made> property of the sub match
objects. So it (like the action method C<TOP> too) exploits the fact that
action methods for submatches are called before those of the calling/outer
regex. So action methods are called in
L<post-order|https://en.wikipedia.org/wiki/Tree_traversal#Post-order>.

The action method C<TOP> simply collects all the objects that were C<.made> by
the multiple matches of the C<pair> rule, and returns them in a list.

Also note that C<KeyValuePairsActions> was passed as a type object to method
C<parse>, which was possible because none of the action methods use attributes
(which would only be available in an instance).

In other cases, action methods might want to keep state in attributes. Then of
course you must pass an instance to method parse.

Note that C<token> C<ws> is special: when C<:sigspace> is enabled (and it is
when we are using C<rule>), it replaces certain whitespace sequences. This is
why the spaces around the equals sign in C<rule pair> work just fine and why
the whitespace before closing C<}> does not gobble up the newlines looked for
in C<token TOP>.

=end pod
