=begin pod :kind("Language") :subkind("Language") :category("tutorial")

=TITLE CompUnits and where to find them

=SUBTITLE How and when Raku modules are compiled, where they are stored, and how to access them in compiled form.

=head1 Overview

Programs in Raku, as a member of the Perl language family, tend at the top level to be more at the interpreted
end of the interpreted-compiled spectrum. In this tutorial, an 'interpreted' program means that the source code,
namely the human-readable text such as C<say 'hello world';>, is immediately processed by the C<Raku> program into code
that can be executed by the computer, with any intermediate stages being stored in memory.

A compiled program, by contrast, is one where the human readable source is first processed into machine-executable code
and some form of this code is stored 'on disc'. In order to execute the program, the machine-readable version is loaded
into memory and then run by the computer.

Both compiled and interpreted forms have advantages. Briefly, interpreted programs can be 'whipped up' quickly and
the source changed quickly. Compiled programs can be complex and take a significant time to pre-process into machine-readable
code, but then running them is much faster for a user, who only 'sees' the loading and running time, not the compilation
time.

C<Raku> has both paradigms. At the B<top level> a Raku program is interpreted, but if code that is separated out into a
Module will be compiled and the preprocessed version is then loaded when necessary. In practice, Modules that have been
written by the community will only need to be precompiled once by a user when they are 'installed', for example by a
Module manager such as C<zef>. Then they can be C<use>d  by a developer in her own program. The effect is to make C<Raku>
top level programs run quickly.

One of the great strengths of the C<Perl> family of languages was the ability to integrate a whole ecosystem of modules
written by competent programmers into a small program. This strength was widely copied and is now the norm for all
languages. C<Raku> takes integration even further, making it relatively easy for C<Raku> programs to incorporate system
libraries written in other languages into C<Raku> programs, see L<Native Call|/language/nativecall>.

The experience from C<Perl> and other languages is that the distributive nature of Modules generate several practical difficulties:
=item a popular module may go through several iterations as the API gets improved, without a guarantee that there is
backward compatibility. So, if a program relies on some specific function or return, then there has to be a way to
specify the B<Version>.
=item a module may have been written by Bob, a very competent programmer, who moves on in life, leaving the module unmaintained,
so Alice takes over. This means that the same module, with the same name, and the same general API may have have two
versions in the wild. Alternatively, two developers (e.g., Alice and Bob) who initially cooperated on a module, then part company about its
development. Consequently, it sometimes is necessary for there to be a way to define the B<Auth> of the module.
=item a module may be enhanced over time and the maintainer keeps two versions up to date, but with different APIs. So it is
may be necessary to define the B<API> required.
=item when developing a new program a developer may want to have the modules written by both Alice and Bob installed locally.
So it is not possible simply to have only one version of a module with a single name installed.

C<Raku> enables all of these possibilities, allowing for multiple versions, multiple authorities, and multiple APIs to be present,
installed, and available locally. The way classes and modules can be accessed with specific attributes
is explained L<elsewhere|/language/typesystem#Versioning_and_authorship>. This tutorial is about how C<Raku> handles these
possibilities.

=head1 Introduction

Before considering the C<Raku> framework, let's have a look at how languages like C<Perl> or C<Python> handle module
installation and loading.

=begin code :lang<text>
ACME::Foo::Bar -> ACME/Foo/Bar.pm
os.path -> os/path.py
=end code

In those languages, module names have a 1:1 relation with filesystem paths.
We simply replace the double colons or periods with slashes and add a C<.pm> or C<.py>.

Note that these are relative paths.
Both C<Python> and C<Perl> use a list of include paths, to complete these paths.
In C<Perl> they are available in the global C<@INC> array.

=begin code :lang<text>
@INC

/usr/lib/perl5/site_perl/5.22.1/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.22.1/
/usr/lib/perl5/vendor_perl/5.22.1/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.22.1/
/usr/lib/perl5/5.22.1/x86_64-linux-thread-multi
/usr/lib/perl5/5.22.1/
=end code

Each of these include directories is checked for whether it contains a relative path determined from the module name.
If the shoe fits, the file is loaded.

Of course that's a bit of a simplified version.
Both languages support caching compiled versions of modules.
So instead of just the C<.pm> file C<Perl> first looks for a C<.pmc> file.
And C<Python> first looks for C<.pyc> files.

Module installation in both cases means mostly copying files into locations determined by the same simple mapping. The
system is easy to explain, easy to understand, simple and robust.

=head2 Why change?

Why would C<Raku> need another framework? The reason is there are features that those languages lack, namely:
=item Unicode module names
=item Modules published under the same names by different authors
=item Having multiple versions of a module installed

The set of 26 Latin characters is too restrictive for virtually all real modern languages, including English, which
have diacritics for many commonly-used words.

With a 1:1 relation between module names and filesystem paths, you enter a world of pain
once you try to support Unicode on multiple platforms and filesystems.

Then there's sharing module names between multiple authors. This one may or may not work out well in practice.
I can imagine using it for example for publishing a module with some fix until the original author includes
the fix in the "official" version.

Finally there's multiple versions. Usually people who need certain versions of modules reach for local::lib or
containers or some home grown workarounds. They all have their own disadvantages. None of them would be necessary
if applications could just say, hey I need good old, trusty version 2.9 or maybe a bug fix release of that branch.

If you had any hopes of continuing using the simple name mapping solution, you probably gave up at the
versioning requirement. Because, how would you find version 3.2 of a module when looking for a 2.9 or higher?

Popular ideas included collecting information about installed modules in JSON files but when those turned out to be
toe-nail growing slow, text files were replace by putting the metadata into SQLite databases.
However, these ideas can be easily shot down by introducing another requirement: distribution packages.

Packages for Linux distributions are mostly just archives containing some files plus some metadata.
Ideally the process of installing such a package means just unpacking the files and updating the central package database.
Uninstalling means deleting the files installed this way and again updating the package database.
Changing existing files on install and uninstall makes packagers' lives much harder, so we really want to avoid that.
Also the names of the installed files may not depend on what was previously installed.
We must know at the time of packaging what the names are going to be.

=head2 Long names

=begin code :lang<text>
Foo::Bar:auth<cpan:nine>:ver<0.3>:api<1>
=end code

Step 0 in getting us back out of this mess is to define a long name.
A full module name in C<Raku> consists of the short-name, auth, version and API

At the same time, the thing you install is usually not a single module but a distribution which probably contains one or more modules.
Distribution names work just the same way as module names.
Indeed, distributions often will just be called after their main module.
An important property of distributions is that they are immutable.
C<< Foo:auth<cpan:nine>:ver<0.3>:api<1> >> will always be the name for exactly the same code.

=head2 $*REPO

In C<Perl> and C<Python> you deal with include paths pointing to filesystem directories.
In C<Raku> we call such directories "repositories" and each of these repositories is governed by an object that does the
C<CompUnit::Repository> role.
Instead of an C<B<@INC>> array, there's the C<$*REPO> variable.
It contains a single repository object.
This object has a B<next-repo> attribute that may contain another repository.
In other words: repositories are managed as a I<linked list>.
The important difference to the traditional array is, that when going through the list, each object has a say in whether
to pass along a request to the next-repo or not.
C<Raku> sets up a standard set of repositories, i.e. the "core", "vendor" and "site" repositories, just like you know them from C<Perl>.
In addition, we set up a "home" repository for the current user.

Repositories must implement the C<need> method.
A C<use> or C<require> statement in C<Raku> code is basically translated to a call to C<B<$*REPO>>'s C<need> method.
This method may in turn delegate the request to the next-repo.

=begin code :preamble<class CompUnit::Store {};> :method
role CompUnit::Repository {
    has CompUnit::Repository $.next is rw;

    method need(CompUnit::DependencySpecification $spec,
                CompUnit::PrecompilationRepository $precomp,
                CompUnit::Store :@precomp-stores
                --> CompUnit:D
                )
        { ... }
    method loaded(
                --> Iterable
                )
        { ... }

    method id( --> Str )
        { ... }
}
=end code

=head2 Repositories

Rakudo comes with several classes that can be used for repositories.
The most important ones are C<CompUnit::Repository::FileSystem> and C<CompUnit::Repository::Installation>.
The FileSystem repo is meant to be used during module development and actually works just like C<Perl> when
looking for a module.
It doesn't support versions or C<auth>s and simply maps the short-name to a filesystem path.

The Installation repository is where the real smarts are. When requesting a module, you will usually either do it
via its exact long name, or you say something along the lines of "give me a module that matches this filter."
Such a filter is given by way of a C<CompUnit::DependencySpecification> object which has fields for
=item short-name,
=item auth-matcher,
=item version-matcher and
=item api-matcher.

When looking through candidates, the Installation repository will smartmatch a module's long name against this
DependencySpecification or rather the individual fields against the individual matchers.
Thus a matcher may be some concrete value, a version range, or even a regex (though an arbitrary regex, such as C<.*>,
would not produce a useful result, but something like C<3.20.1+> will only find candidates higher than 3.20.1).

Loading the metadata of all installed distributions would be prohibitively slow. The current implementation of
the C<Raku> framework uses
the filesystem as a kind of database. However, another implementation may use another strategy. The following description
shows how one implementation works and is included here to illustrate what is happening.

We store not only a distribution's files but also create indices for speeding up lookups.
One of these indices comes in the form of directories named after the short-name of installed modules.
However most of the filesystems in common use today cannot handle Unicode names, so we cannot just use
module names directly.
This is where the now infamous SHA-1 hashes enter the game.
The directory names are the ASCII encoded SHA-1 hashes of the UTF-8 encoded module short-names.

In these directories we find one file per distribution that contains a module with a matching short name.
These files again contain the ID of the dist and the other fields that make up the long name: auth, version, and api.
So by reading these files we have a usually short list of auth-version-api triplets which we can match against our
DependencySpecification.
We end up with the winning distribution's ID, which we use to look up the metadata, stored in a JSON encoded file.
This metadata contains the name of the file in the sources/ directory containing the requested module's code.
This is what we can load.

Finding names for source files is again a bit tricky, as there's still the Unicode issue and in addition the same
relative file names may be used by different installed distributions (think versions).
So for now at least, we use SHA-1 hashes of the long-names.

=head2 Resources

=begin code :lang<text>
%?RESOURCES
%?RESOURCES<libraries/p5helper>
%?RESOURCES<icons/foo.png>
%?RESOURCES<schema.sql>

Foo
|___ lib
|     |____ Foo.rakumod
|
|___ resources
      |___ schema.sql
      |
      |___ libraries
            |____ p5helper
            |        |___
            |___ icons
                     |___ foo.png

=end code

It's not only source files that are stored and found this way.
Distributions may also contain arbitrary resource files.
These could be images, language files or shared libraries that are compiled on installation.
They can be accessed from within the module through the C<%?RESOURCES> hash

As long as you stick to the standard layout conventions for distributions, this even works during development
without installing anything.

A nice result of this architecture is that it's fairly easy to create special purpose repositories.

=head2 Dependencies

Luckily precompilation at least works quite well in most cases. Yet it comes with its own set of challenges.
Loading a single module is easy.
The fun starts when a module has dependencies and those dependencies have again dependencies of their own.

When loading a precompiled file in C<Raku> we need to load the precompiled files of all its dependencies, too.
And those dependencies B<must> be precompiled, we cannot load them from source files.
Even worse, the precomp files of the dependencies B<must> be exactly the same files we used for precompiling our
module in the first place.

To top it off, precompiled files work only with the exact C<Raku> binary, that was used for compilation.

All of that would still be quite manageable if it weren't for an additional requirement: as a user you expect a new
version of a module you just installed to be actually used, don't you?

In other words: if you upgrade a dependency of a precompiled module, we have to detect this and precompile the module
again with the new dependency.

=head2 Precomp stores

Now remember that while we have a standard repository chain, the user may prepend additional repositories by way of
C<-I> on the command line or "use lib" in the code.

These repositories may contain the dependencies of precompiled modules.

Our first solution to this riddle was that each repository gets its own precomp store where precompiled files are stored.
We only ever load precomp files from the precomp store of the very first repository in the chain because this is the
only repository that has direct or at least indirect access to all the candidates.

If this repository is a FileSystem repository, we create a precomp store in a C<.precomp> directory.

While being the safe option, this has the consequence that whenever you use a new repository, we will start out
without access to precompiled files.

Instead, we will precompile the modules used when they are first loaded.

=head2 Credit

This tutorial is based on a C<niner> L<talk|http://niner.name/talks/A%20look%20behind%20the%20curtains%20-%20module%20loading%20in%20Perl%206/>.

=end pod
