docs: Document how to use onrun + memory_set.

[goaljobs.git] / goaljobs-reference.pod

=encoding utf8

=head1 NAME

goaljobs-reference - reference documentation for writing goaljobs scripts

=head1 SUMMARY

 open Unix
 open Printf
 open Goaljobs
 
 let goal name args... =
   target (condition);
   require (goal1 ...);
   require (goal2 ...);
   (* code to implement the goal *)
 
 let goal all () =
   require (name args...)
 
 every 30 minutes (
   fun () ->
     require (goal1 ...)
 )

=head1 DESCRIPTION

Goaljobs is a flexible build system and business rules manager similar
to make and cron, but much more powerful.  You can use it to automate
many complex tasks that have multiple steps (even with manual steps)
that have to be carried out in dependency order.

For a tutorial-like introduction to goaljobs, see
L<http://rwmj.wordpress.com/tag/goaljobs/>

For examples, see the C<examples/> directory in the source and
L<http://git.annexia.org/?p=goals.git;a=summary>

For reference documentation on how to write scripts, see below.

Note this man page does not cover the whole Goaljobs API.  To read
about the Goaljobs API, look for the file C<goaljobs.mli> (in the
source code or installed as part of the goaljobs package), or see the
HTML documentation installed as part of the goaljobs package.

=head1 THE SCRIPT FILE

The script file should usually start with opening these modules (none
of this are required, it's just useful to have them open):

 open Unix
 open Printf
 open Goaljobs

This is followed by goals and/or functions and/or top-level OCaml
statements and/or C<every> statements (a.k.a periodic jobs).

You can use multiple script files to make up a goaljobs program.  You
have to list them in dependency order on the goaljobs command line
(earlier files required by later files), the same way that the OCaml
compiler works.  So usually you end up writing:

 goaljobs utils.ml another_library.ml script.ml

where C<script.ml> requires the functions from the utils/library.
Note that circular dependencies are not possible.

=head1 GOALS

Each goal should have the following basic form:

 let goal name args... =
   target (condition);
   require (goal1 ...);
   require (goal2 ...);
   (* code to implement the goal *)

There is no hard-and-fast rule about this.  In particular you can put
arbitrary OCaml statements anywhere inside a goal (since a goal is
just a special form of OCaml function), but sticking to this overall
plan is a good idea.

There should be zero or one target.  Multiple target statements should
not be used in a goal.  The target should come as early as possible,
and the target condition should be as simple and fast to evaluate as
is practical (see L</THE MEMORY> below).

There should be zero or any number of C<require> statements.  Each
require statement should name a single goal (with optional parameters
for that goal).

After that should come the code that implements the goal, which might
be, for example, a series of shell commands, but could even be
user-interactive.

As with ordinary OCaml functions, you can define goals recursively
or with mutual recursion using:

 let rec goal1 args... =
   ...
 and goal2 args... =
   ...
 and goal3 args... =
   ...

A goal can also have no arguments:

 let goal all () =
   ...

This defines the common goal called C<all>, which acts the same way as
C<make all>, ie. if you run the program without any arguments, it will
run the C<all> goal if one exists.

=head2 PUBLISHING GOALS

If a goal is "published" it means it is available to be run directly
from the command line.  All no-arg goals are published by default.
You do not need to do anything special for them.  For example:

 let goal clean () = sh "rm *~"

can be used on the command line:

 ./myscript clean

For goals which take any parameters, you have to define a small code
snippet that converts command line arguments to goal parameters (the
reason has to do with OCaml being strongly typed, and because goal
parameters might not all be strings).

 let goal compile program sources =
   target (more_recent [program] sources);
   ...
 
 let () =
   publish "compile" (
     fun args ->
       let program = List.hd args in
       let sources = List.tl args in
       require (compiled program sources)
   )

Then you can use:

 ./myscript compile program main.c utils.c

=head1 TARGET AND REQUIRE

The target is promise or contract that you make that the given
condition I<will> be true when the goal has finished running.

In the first example, the target is that the C<o_file> (object) exists
and is newer than the C<c_file> (source).  The goal meets that target
by running the C compiler (C<cc>) which, if it succeeds, will ensure
that the object file exists and is newer than the source file.

 let goal compiled c_file =
   let o_file = change_file_extension "o" c_file in
   target (more_recent [o_file] [c_file]);
 
   sh "
     cd $builddir
     cc -c %s -o %s
   " c_file o_file

In the second example, the goal requires that several files have been
compiled (C<require (compiled ...)>) before it can link the final
program:

 let goal built program source =
   target (more_recent [program] [source]);
 
   require (compiled source);
 
   let object = change_file_extension "o" source in
   sh "
     cd $builddir
     cc %s -o %s
   " object program

=head1 SPECIAL VALUES INSIDE GOALS

=head2 goalname

Inside goals, you can use C<goalname> to get the name of the goal, ie:

 let goal foo () =
   printf "my name is %s\n" goalname

would print:

 my name is foo

=head2 goalloc

Inside goals, you can use C<goalloc> to get a printable source
location of the goal, ie:

 let goal foo () =
   printf "%s\n" goalloc

would print:

 File "source.ml", line 2, characters 13-71 (end at line 3, character 23)

Note that the actual string format depends on the internal OCaml
function C<Loc.to_string> so it might change in future.

=head2 onfail, onsuccess, onrun

Inside goals you can register function(s) which run if the goal
completes successfully (C<onsuccess>), if the goal completes
successfully after running to the end (C<onrun>), or if the goal fails
(C<onfail>).

For example:

 let goal built () =
   onfail (fun _ -> eprintf "goal '%s' failed\n" goalname);
   sh "
     cc -o program main.o
   "

If the shell command (or another part of the goal) fails, then this
would print out:

 goal 'built' failed

The single parameter passed to C<onfail> is the exception that was
thrown.

Note that the helper function C<Goaljobs.mailto> is a useful function
to call from an C<onfail> handler:

 let from = "me@example.com"
 let to_ = "you@example.com"
 let logfile = log_program_output ()
 
 let goal built () =
   onfail (fun _ ->
             let subject = sprintf "goal: %s: BUILD FAILED" goalname in
             mailto ~from ~subject ~attach:[logfile] to_);
   sh "
     cc -o program main.o
   "

C<onsuccess> and C<onrun> are slightly different from C<onfail> and
from each other:

C<onsuccess> functions can be called if a C<target> condition is met
and the rest of the goal is short-circuited.  C<onrun> will only be
called if all the instructions in the goal actually run and succeed.

The single unit C<()> parameter is passed to the C<onsuccess> and
C<onrun> functions.

You can register as many functions as you want for each handler.  The
order in which the functions are called is not defined.

=head1 PERIODIC JOBS

If you want to have a goal that runs when some outside event happens
you have three choices: Manually run the script (this is basically
what C<make> forces you to do).  Have some sort of hook that runs the
script (eg. a git hook).  Or use a periodic job to poll for an event
or change.

Periodic jobs run regularly to poll for an outside event or change.
If a script has periodic jobs, then it runs continuously (or until you
kill it).

An example of a script that checks for new git commits and when it
sees one it will ensure it passes the tests:

 let repo = Sys.getenv "HOME" // "repo"
 
 let goal git_commit_tested commit =
   let key = sprintf "repo-tested-%s" commit in
   target (memory_exists key);
   onrun (fun () -> memory_set key "1");
 
   sh "
     git clone %s test
     cd test
     ./configure
     make
     make check
   " repo_url
 
 every 30 minutes (fun () ->
   let commit = shout "cd %s && git rev-parse HEAD" repo in
   (* Require that this commit has been tested. *)
   require (git_commit_tested commit)
 )

Some notes about the above example: Firstly only the current HEAD
commit is required to be tested.  This is because older commits are
irrelevant and because if they failed the test before there is not
point retesting them (commits are immutable).  Secondly we use the
Memory to remember that we have successfully tested a commit.  This is
what stops the program from repeatedly testing the same commit.

=head1 SHELL

You can call out to the Unix shell using one of the functions
C<Goaljobs.sh>, C<Goaljobs.shout> or C<Goaljobs.shlines>.  (These
functions are documented in the C<goaljobs.mli> file / API
documentation.)

C<sh> runs the command(s).  C<shout> collects the output of the
command (to stdout only) and returns it as a single string.
C<shlines> collects the output and returns it as a list of lines.

C<sh>, C<shout>, C<shlines> work like printf.  ie. You can substitute
variables using C<%s>, C<%d> and so on.  For example:

 sh "rsync foo-%s.tar.gz example.com:/html/" version

Each shell runs in a new temporary directory.  The temporary directory
and all its contents is deleted after the shell exits.  If you want to
save any data, C<cd> somewhere.  If you don't want the temporary
directory creation, use C<~tmpdir:false>.

The environment variable C<$builddir> is exported to the script.  This
is the current directory when the goaljobs program was started.

Each invocation of C<sh> (etc) is a single shell (this is slightly
different from how C<make> works).  For example:

 sh "
   package=foo-%s
   tarball=$package.tar.gz
   cp $HOME/$tarball .
   tar zxf $tarball
   cd $package
   ./configure
   make
 " version

The shell error mode is set such that if any single command
returns an error then the C<sh> function as a whole exits with
an error.  Write:

 command ||:

to ignore the result of a command.

C</bin/sh> is used unless you set C<Goaljobs.shell> to some other
value.  Note that the environment variable C<SHELL> is I<never> used.

=head1 THE MEMORY

"The Memory" is key/value storage which persists across goaljobs
sessions.  It is stored in the file C<$HOME/.goaljobs-memory> (which is
a binary file, but you can delete it if you want).

The Memory is locked during accesses, so it is safe to read or write
it from multiple parallel goaljobs sessions.

Keys and values are strings.  The keys should be globally unique, so
it is suggested you use some application-specific prefix.  eg:
C<myapp-key>

A common pattern is:

let goal tested version =
  let key = "myapp-tested-" ^ version in
  target (memory_exists key);
  onrun (fun () -> memory_set key "1");
 
  (* some code to test this version *)

Note in that example the value C<1> is arbitrary.  You just want to
store I<any> value so that a later call to C<memory_exists> will
succeed.

For information about C<Goaljobs.memory_*> APIs see the
C<goaljobs.mli> file / API documentation.

=head1 FILES

=over 4

=item C</bin/sh>

This is the default shell used by C<sh*> APIs.  You can change
the shell by setting the C<Goaljobs.shell> reference.

=item C<curl>

The curl program (on the path) is used to check for and download
URLs by APIs such as C<Goaljobs.url_exists>.

=item C<~/.goaljobs-memory>

Persistent key/value store used when you use the C<Goaljobs.memory_*>
APIs.

=back

=head1 ENVIRONMENT VARIABLES

=over 4

=item B<builddir>

This environment variable is set to the current directory when the
goals program starts, and is available in goals, shell scripts, etc.

=back

=head1 SEE ALSO

L<goaljobs(1)>

=head1 AUTHORS

Richard W.M. Jones <rjones @ redhat . com>

=head1 COPYRIGHT

(C) Copyright 2013 Red Hat Inc.,

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.