Persist variables to file (~/.whenjobs/variables).

[whenjobs.git] / tools / whenjobs.pod

=encoding utf8

=head1 NAME

whenjobs - A powerful but simple cron replacement

=head1 SYNOPSIS

Editing the jobs script:

 whenjobs -e | --edit
 whenjobs -l | --list

Get and set variables:

 whenjobs --get variable
 whenjobs --set variable=value [variable=value ...]
 whenjobs --variables

Start and stop the per-user daemon:

 whenjobs --daemon-start
 whenjobs --daemon-stop
 whenjobs --daemon-status
 whenjobs --daemon-restart

Examine running jobs:

 whenjobs --jobs
 whenjobs --cancel serial
 whenjobs --start "name"
 whenjobs --tail serial

=head1 DESCRIPTION

Whenjobs is a powerful but simple replacement for cron.  It lets you
run jobs periodically like cron, but it also lets you trigger jobs to
run when user-defined variables are set or change value.

Periodic jobs are written like this:

 every 10 minutes :
 <<
   # Get the current load average.
   load=`awk '{print $1}' /proc/loadavg`
   whenjobs --set --type float load=$load
 >>

When-statements let you create jobs that run based on variables set
elsewhere:

 when load >= 6 :
 <<
   mail -s "ALERT: high load average: $load" $LOGNAME < /dev/null
 >>

(When statements are "edge-triggered", meaning that this job will only
run when the load goes from under 6 to E<ge> 6).

Like L<crontab(5)>, whenjobs are controlled by a jobs file which can
be edited from the command line:

 $ whenjobs -e

Whenjobs uses a daemon called L<whenjobsd(8)>.  Unlike crond, this
daemon runs as the same user.  Each user who wants to use whenjobs
starts their own daemon:

 $ whenjobs --daemon-start

You can also have the daemon start as you when the machine boots by
adding the following line to a boot file such as C</etc/rc.local>.
Replace C<username> with your username:

 su username -c /usr/sbin/whenjobsd

Variables are the key to expressing dependencies between whenjobs.
Variables are stored (per-user) in the daemon.  You can use the
command line tool to examine and set variables:

 $ whenjobs --variables
 JOBSERIAL=297
 libguestfs_build_local=1.17.16
 libguestfs_commit=7e32d892d76a31f55e2a4151902623b9949e3efa
 libguestfs_dist=1.17.16
 libguestfs_release=1.17.16
 libguestfs_stable_build_local=1.16.10
 libguestfs_stable_commit=27433a0a335301441b1eb6244ba425c2c44b2d99
 libguestfs_stable_dist=1.16.10
 libguestfs_stable_release=1.16.10
 libguestfs_stable_version=1.16.10
 libguestfs_version=1.17.16
 $ whenjobs --set cat=sushi
 $ whenjobs --get cat
 sushi

Note: The act of setting a variable (using I<--set>) can trigger jobs
to run.

You can also list out what jobs are running:

 $ whenjobs --jobs
 287 libguestfs-stable: fedora 16
    running in: /tmp/whenjobsa2afc44fd757465f95438309f1a51609
    started at: 2012-03-13 10:59:37

and you can 'tail' the output of running jobs which is useful for
debugging:

 $ whenjobs --tail 287
 Uploading: 147496271972717053d46b82a07435ca  libguestfs-1.16.10.tar.gz

You can start and cancel jobs manually:

 $ whenjobs --start 'libguestfs: poll'
 $ whenjobs --cancel 287

=head1 OPTIONS

=over 4

=item B<--cancel> serial

Cancel the job with the given serial number.

Use I<--jobs> to list running jobs along with their serial numbers.
The serial number is also available in the job script (as
C<$JOBSERIAL>) and in the log file.

=item B<--daemon-start>

=item B<--daemon-stop>

Start and stop the per-user daemon.

=item B<--daemon-status>

Prints the status of the daemon: C<up> or C<down>.

=item B<--daemon-restart>

Restart the daemon.  (If it is not running, then this command
starts it).

=item B<-e>

=item B<--edit>

Edit the jobs script.  If you make changes to the jobs script, then it
is automatically uploaded to the daemon.

The C<$EDITOR> environment variable is used for editing.  If not set,
C<vi> is used.

=item B<--get> variable

Print the value of a variable.

=item B<-help>

=item B<--help>

Display brief usage and exit.

=item B<--job-names>

List the names of all loaded jobs (whether they are running or not).
Use I<--jobs> to list running jobs.

=item B<--jobs>

List all running jobs.

Note that it is possible for the same job to be running more than once
(for example, a periodic job that takes longer than the period to run).

=item B<-l>

=item B<--list>

List the jobs script.

=item B<--lib> directory

Set the library directory which needs to contain the auxiliary files
C<pa_when.cmo> and C<whenlib.cma>.  Normally you do not need to
specify this.  However if you are running whenjobs without installing
it, then you need to point this to the C<lib/> directory from the
source, eg:

 whenjobs --lib $builddir/lib -e

=item B<--set> variable=value [variable=value ...]

=item B<--type> bool|int|float|string|unit

I<--set> sets the variable named C<variable> to the new C<value>.  The
variable is created if it does not already exist.  Note that setting a
variable can cause jobs to run immediately.

To unset a variable, set it to the empty string like this:

 whenjobs --set var=

By default variables are strings.  You can also set the type of a
variable when setting it by adding the optional I<--type> parameter.
The I<--type> parameter should come I<before> the variable
declaration, like this:

 whenjobs --set --type int free_space=10000

See the discussion of variable types in the L</REFERENCE> section
below.

You can set multiple variables.  When setting multiple variables in a
single command, the values are all changed in a single atomic
operation.

 whenjobs --set cat=sushi food=fish

When using I<--type> and multiple variables, the type changes the
remaining command line parameters until the next I<--type>, eg:

 whenjobs --set cat=sushi \
     --type float weight=3.5 \
     --type string food=fish

(C<cat> and C<food> are strings, and C<weight> is a float).

=item B<--start> "job name"

Start the job immediately and unconditionally.

This runs the job even if its normal preconditions are not met.  This
may cause unexpected results, so use with caution.

=item B<--tail> serial

Tail the output of the running job identified by its serial number.
Use the I<--jobs> flag to get a list of running jobs.

=item B<--test> variable=value [variable=value ...]

This works the same way as the I<--set> option, but the difference is
that the variables are not set.  Instead, it lists out the jobs that
I<would> run, I<if> the variables were updated to these new values.

The variables are not actually updated, and the jobs are not actually
run.

The output is a list of job names that would run.

=item B<--upload>

Compile the jobs file(s) and upload it to the daemon, without editing.
Note that the I<--edit> option does this automatically.  Furthermore,
when the daemon is started it checks for a jobs script and loads it if
found.

See also L</MULTIPLE JOBS FILES> below.

=item B<--variables>

Display all the variables and their values, in the format C<name=value>.

=item B<-V>

=item B<--version>

Display the name and version of the program and exit.

=item B<--whisper> variable=value [variable=value ...]

This works the same way as the I<--set> option, but with the
difference that jobs' when clauses are not reevaluated.  In other
words, the variables are set, but "quietly" so as not to trigger any
jobs to run.

Note that this can lead to some unexpected results: one case is a
job such as:

 when changed a || changed b : << ... >>

If C<a> is changed using I<--whisper>, then the job will not run.

But later on, if C<b> is set but to the same value that it already has
(ie. not changed), the job will run because the whole when-clause is
reevaluated and C<a> is found to have changed since the last run of
the job.

=back

=head1 REFERENCE

A whenjobs file consists of a series of one or more "every" or "when"
statements.

Comments in the file can be written using C<(* ... *)>.  Comments
may be nested.

Shell script fragments are written using C<E<lt>E<lt> ... E<gt>E<gt>>.
Within shell script fragments, use C<#> for comments (as in ordinary
shell scripts).  Because C<E<gt>E<gt>> has a special meaning, it
cannot be used in the shell script (ie. for redirection).  You have to
write C<E<gt>\E<gt>> instead which is replaced with C<E<gt>E<gt>> when
the shell script is parsed.

=head2 EVERY STATEMENTS (PERIODIC JOBS)

An every statement has the form:

 every <period> :
 <<
   # shell script
 >>

where C<E<lt>periodE<gt>> is a I<period expression>, which may take
one of the forms below.  Don't forget the colon character between the
period expression and the shell script.

An every statement is a job which runs periodically.

=head3 PERIOD EXPRESSIONS

=over 4

=item B<every second>

The job runs every second.

=item B<every minute>

The job runs every minute.

=item B<every hour>

The job runs every hour.

=item B<every day>

The job runs every day, at midnight UTC.

=item B<every week>

The job runs every week, on a Thursday at midnight UTC.

=item B<every month>

The job runs every month, on the first of the month at midnight UTC.

=item B<every year>

The job runs every year, on the first day of the year at midnight UTC.

=item B<every decade>

=item B<every century>

=item B<every millenium>

The job runs every 10, 100 or 1000 years.

=item B<every I<N> seconds>

The job runs every I<N> seconds (I<N> is any number E<ge> 1).

=item B<every I<N> minutes>

The job runs every I<N> minutes.

=item B<every I<N> hours>

The job runs every I<N> hours.

=item B<every I<N> days>

The job runs every I<N> days.

=item B<every I<N> weeks>

The job runs every I<N> weeks.

=item B<every I<N> months>

The job runs every I<N> months.

=item B<every I<N> years>

=item B<every I<N> decades>

=item B<every I<N> centuries>

=item B<every I<N> millenia>

The job runs every I<N>, I<10*N>, I<100*N> or I<1000*N> years.

=back

=head2 WHEN STATEMENTS (DEPENDENT JOBS)

A when statement has the form:

 when <expr> :
 <<
   # shell script
 >>

where C<E<lt>exprE<gt>> is a I<when expression>, described below.
Don't forget the colon character between the expression and the
shell script.

A when statement is a job which runs when the conditions described in
its when-expression become true.

When jobs are I<edge triggered>.  This means that they run when the
condition changes from false to true (or in the case where the
expression has not been evaluated before, when it evaluates initially
to true).

=head3 WHEN EXPRESSIONS

When expressions are fully recursive expressions constructed from the
following elements:

=over 4

=item I<expr> B<&&> I<expr>

=item I<expr> B<||> I<expr>

The boolean "and" or "or" of the two sub-expressions.

=item I<expr> B<E<lt>> I<expr>

=item I<expr> B<E<lt>=> I<expr>

=item I<expr> B<==> I<expr>

=item I<expr> B<E<gt>=> I<expr>

=item I<expr> B<E<gt>> I<expr>

The two sub-expressions are evaluated and the usual comparison
operator is performed.

If the sub-expressions are numeric, then numeric comparison is done.
If either sub-expression is non-numeric, then both expressions are
converted (if necessary) to strings and string comparison is done.

=item I<expr> B<E<lt>E<gt>> I<expr>

=item I<expr> B<!=> I<expr>

Either form can be used to test the two expressions for inequality.

=item B<!> I<expr>

Boolean negative of I<expr>.

=item I<expr> B<+> I<expr>

For numeric sub-expressions, this performs addition.

If both sub-expressions are strings, this performs string
concatenation.

Other types give an error.

=item I<expr> B<-> I<expr>

=item I<expr> B<*> I<expr>

=item I<expr> B</> I<expr>

=item I<expr> B<mod> I<expr>

Both sub-expressions are evaluated, and if both are numeric, then the
result is subtraction, multiplication, division or modulo.

Other types give an error.  Note that I<mod> really is an infix
operator.

=item B<len> I<expr>

If I<expr> is a string, this returns the length of the string.

=item I<variable>

The value of the named variable.

Previously undefined variables are automatically initialized to the
empty string.

=item B<prev> I<variable>

The I<previous> value of the named variable.  This means, the value
that it had last time this when-job ran.

If the when-job has not run yet, then this returns C<"">.

Job state is preserved across file reloads, but I<only> for jobs that
are explicitly named.  If you find that jobs using C<prev>, C<changes>
etc are running unnecessarily when the jobs file is edited or
uploaded, try giving the jobs an explicit name.

=item B<changes> I<variable>

If the named variable has changed since this job last ran, then this
evaluates to true, else false.

This is the same as writing C<prev variable != variable>.

=item B<increases> I<variable>

If the named variable has changed and increased since this job last
ran, then this evaluates to true, else false.

This is the same as writing C<prev variable E<lt> variable>.

=item B<decreases> I<variable>

If the named variable has changed and decreased since this job last
ran, then this evaluates to true, else false.

This is the same as writing C<prev variable E<gt> variable>.

B<Note:> There is a subtle gotcha with the I<decreases> operator: The
first time the expression is evaluated, the job has (by definition)
not yet run.  Therefore C<prev variable> evaluates to C<""> (see
definition of I<prev> above).  Since it is always true that

 "" < anything

the I<decreases> operator evaluates to false, and since this usually
means the job does not run, the operator always evaluates to false.

To fix this, ensure that the variable is initialized (see
L</SETTING THE INITIAL VALUE OF VARIABLES> below).

=item B<reloaded ()>

This evaluates to true the first time the expression is evaluated
after the jobs file has been reloaded or the daemon restarted.
Thereafter it evaluates to false.

Don't use this to initialize variables: it won't do what you mean.

=item B<false>

=item B<true>

Constants that evaluate to boolean false or true respectively.

=item I<"any string">

Any string.

In a boolean context, the empty string evaluates to false, and
non-empty strings evaluate to true.

=item I<N>

Any integer.  (Arbitrarily large integers are supported.)

In a boolean context, 0 evaluates to false, and non-zero evaluates to
true.

=item I<N.>

=item I<.N>

=item I<N.N>

=item I<N.NeN>

Any floating point number.

In a boolean context, 0 evaluates to false, and non-zero evaluates to
true.

=back

=head2 SHELL SCRIPTS

The code between C<E<lt>E<lt> ... E<gt>E<gt>> is a shell script.  It
is executed using C<$SHELL>, or if that environment variable is not
set then C</bin/sh>.

=head3 SHELL SCRIPT VARIABLES

Every variable that has been set (using the whenjobs I<--set> option)
is exported to the script, so you can simply get the value of any
variable by writing C<$name>.

In addition, there are some special variables available:

=over 4

=item C<$JOBNAME>

The name of the job.  If the job has been named explicitly, then that
name is available through this variable, else it will be some implicit
name like C<job$1>.

=item C<$JOBSERIAL>

The serial number of the job.  This is simply a variable that
increments each time a job is run, and is unique to that run of the
job.

=back

Other environment variables such as C<$HOME>, C<$LOGNAME> etc are
available as normal.

=head3 SHELL SCRIPT TEMPORARY CURRENT DIRECTORY

The shell script runs with its current directory set to a temporary
directory.  The temporary directory is removed when the shell script
exits.  Therefore you can write temporary files here without worrying
about cleaning them up.

If you want to store permanent state, then you have to save it to a
well-known directory, eg. C<$HOME>, C</var> etc.

=head3 SHELL SCRIPT USER

The shell script runs as the ordinary user.  It has no special
privileges.

=head2 JOB NAMES

Jobs are given implicit names (C<job$1>, C<job$2> etc.).  You can also
name jobs explicitly by preceeding the "every" or "when" statement
with C<job "name">:

 job "poll source"
 every 10 seconds :
 <<
   # ...
 >>

The job name is passed to the shell script in the C<$JOBNAME>
environment variable.

=head2 OCAML EXPRESSIONS

As well as simple "every" and "when" expressions, advanced users may
want to use arbitrary OCaml expressions, functions, etc in the jobs
script.  These are useful for factoring common code or strings, for
setting the initial values of variables, or for defining pre and post
functions.

A simple example of an OCaml expression is:

 let prefix = "daily_"
 
 job (prefix ^ "virus_scan")
 every day :
 <<
   # ...
 >>
 
 job (prefix ^ "disk_check")
 every day :
 <<
   # ...
 >>

which creates two jobs called C<"daily_virus_scan"> and
C<"daily_disk_check"> (C<^> is the OCaml string concatenation
operator).

OCaml expressions have access to a library of functions called
B<Whentools> which is described below.  It lets you set variables,
create jobs algorithmically, etc.

The OCaml expressions run once, when the jobs file is being loaded or
reloaded.

=head3 SETTING THE INITIAL VALUE OF VARIABLES

Variables are created when they are referenced, and until set they
have the value empty string (just like the shell).  Across file
reloads, the previous values of variables are preserved.

To initialize a variable to a known value when the jobs file is
loaded, call one of the C<Whentools.set_variable*> functions as in
this example:

 let () =
   Whentools.set_variable "name" "Richard";
   Whentools.set_variable_int "counter" 0

=head3 PRE FUNCTIONS

Before a job runs, you can arrange that a C<pre> function is called.
This function may decide not to run the job (by returning C<false>).

One use for this is to prevent a particular job from running if there
is already an instance of the same job running:

 job "only one"
 pre (Whentools.one ())
 every 10 seconds :
 <<
   # Takes longer than 10 seconds to run, but 'Whentools.one ()'
   # will ensure only one is ever running.
   sleep 11
 >>

When using pre functions, jobs must be given an explicit name, ie.
you must use the C<job> statement.

A number of pre functions are available in the library; see below.

You can also write your own post functions (in OCaml).  The function
is passed one argument which is a C<Whentools.preinfo> struct, defined
below.  It should return a boolean: C<true> if the job should run, and
C<false> if the job should not run.

Note that a fresh serial number (see L</JOBSERIAL>) is assigned to
each run, whether or not the job actually runs because of
preconditions.

=head3 POST FUNCTIONS

After a job runs, you can control what happens to its output by
writing a C<post> function.  To write a post function you have to
name the job (ie. have an explicit C<job> statement).  Put C<post ...>
after the job name like this:

 job "poll source"
 post (Whentools.mailto "you@example.com")
 every 10 seconds :
 <<
   # ...
 >>

A number of post functions are available in the library; see below.

You can also write your own post functions (in OCaml).  The
function is passed one argument which is a C<Whentools.result> struct,
defined below.

=head3 WHENTOOLS LIBRARY

=head4 Functions

=over 4

=item B<Whentools.mailto> [I<~only_on_failure:true>]
[I<~from:from_address>] I<email_address> I<result>

This built-in post function sends the result of the script by email to
the given email address.

If the optional C<~only_on_failure:true> flag is set, then it is only
sent out if the script failed.

If the optional C<~from> flag is set, then the from address is set
accordingly.  This is sometimes needed when sending mail.

Note the C<result> parameter is passed implicitly by the daemon.  You
do not need to add it.

Here are some examples of using the mailto function:

 job "ex.1"
 post (Whentools.mailto "you@example.com")
 every 10 seconds :
 <<
   # do something
 >>

 job "ex.2"
 post (Whentools.mailto ~only_on_failure:true
                        "you@example.com")
 every 10 seconds :
 <<
   # do something
 >>

 let from = "me@example.com"
 let to_addr = "you@example.com"
 
 job "ex.3"
 post (Whentools.mailto ~from to_addr)
 every 10 seconds :
 <<
   # do something
 >>

=item B<Whentools.max> I<n>

This built-in pre function ensures that a maximum of I<n> instances of
the job are running.

It checks the list of running jobs, and if I<n> or more instances are
already running, then it returns C<false>, which ensures that the new
job is not started.

=item B<Whentools.one> I<()>

This built-in pre function ensures that only one instance of the job
is running.  It is the same as calling:

 Whentools.max 1

=item B<Whentools.set_variable> I<name> I<string>

Set variable I<name> to the string.

=item B<Whentools.set_variable_bool> I<name> I<b>

Set variable I<name> to the boolean value I<b>.

=item B<Whentools.set_variable_int> I<name> I<i>

Set variable I<name> to the integer value I<i>.

=item B<Whentools.set_variable_string> I<name> I<s>

Set variable I<name> to the string value <s>.  This is
the same as I<Whentools.set_variable>.

=item B<Whentools.set_variable_float> I<name> I<f>

Set variable I<name> to the floating point value I<f>.

=back

=head4 Structures

=over 4

=item B<Whentools.preinfo>

This structure is passed to pre functions.  It has the following
fields:

 type preinfo = {
   pi_job_name : string;           # Job name.
   pi_serial : Big_int.big_int;    # Job serial number.
   pi_variables : (string * variable) list; # Variables set in job.
   pi_running : preinfo_running_job list;   # List of running jobs.
 }
 and preinfo_running_job = {
   pirun_job_name : string;        # Running job name.
   pirun_serial : Big_int.big_int; # Running job serial number.
   pirun_start_time : float;       # Running job start time.
   pirun_pid : int;                # Running job process ID.
 }

=item B<Whentools.result>

This structure is passed to post functions.  It has the following
fields:

 type result = {
   res_job_name : string;  # job name
   res_serial : big_int;   # job serial (same as $JOBSERIAL)
   res_code : int;         # return code from the shell script
   res_tmpdir : string;    # temporary directory script ran in
   res_output : string;    # filename of stdout/stderr output
   res_start_time : float; # when the job started
 }

=back

=head1 MULTIPLE JOBS FILES

The whenjobs I<-e> and I<-l> options edit and list a file called
C<$HOME/.whenjobs/jobs.ml>.

You can also edit C<$HOME/.whenjobs/jobs.ml> by other means (eg.  your
own editor).  After editing, to recompile and upload it, use:

 whenjobs --upload

When you have lots of jobs, it is convenient to split the jobs across
multiple files.  Any C<*.ml> files located in C<$HOME/.whenjobs> can
be used (with some restrictions on filenames -- see below).  These are
compiled and loaded into the daemon using the I<--upload> command.

To create multiple jobs files, you cannot use the I<-e> or I<-l>
options.  Instead you have to create them yourself in
C<$HOME/.whenjobs>, and when you have finished creating or editing
them, upload them.

=head2 FILENAME RESTRICTIONS ON JOBS FILES

In OCaml, a file called C<jobs.ml> corresponds to an OCaml module
called C<Jobs> (note the capitalization).  OCaml module names can only
contain ASCII alphanumeric characters, underscore, and C<'> (single
quote), and they must begin with an alphabetic character.  The same
rules apply to jobs files.

Furthermore, various OCaml module names are reserved (eg. C<Map>,
C<List>).  It is therefore better to prefix any names with something
specific to your application.

Examples of legal filenames are:

 foo.ml
 app_foo.ml
 app_123.ml
 jobs.ml

Examples of illegal filenames are:

 ann.txt    # must end with .ml
 123.ml     # must begin with alphabetic
 app!.ml    # must contain alphanumeric or underscore
 app-foo.ml # must contain alphanumeric or underscore
 map.ml     # reserved module name

=head1 FILES



=head1 ENVIRONMENT VARIABLES



=head1 SEE ALSO

L<whenjobsd(8)>

=head1 AUTHOR

Richard W.M. Jones L<http://people.redhat.com/~rjones/>

=head1 COPYRIGHT

Copyright (C) 2012 Red Hat Inc.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.