5 goaljobs-reference - reference documentation for writing goaljobs scripts
13 let goal name args... =
17 (* code to implement the goal *)
20 require (name args...)
29 Goaljobs is a flexible build system and business rules manager similar
30 to make and cron, but much more powerful. You can use it to automate
31 many complex tasks that have multiple steps (even with manual steps)
32 that have to be carried out in dependency order.
34 For a tutorial-like introduction to goaljobs, see
35 L<http://rwmj.wordpress.com/tag/goaljobs/>
37 For examples, see the C<examples/> directory in the source and
38 L<http://git.annexia.org/?p=goals.git;a=summary>
40 For reference documentation on how to write scripts, see below.
42 Note this man page does not cover the whole Goaljobs API. To read
43 about the Goaljobs API, look for the file C<goaljobs.mli> (in the
44 source code or installed as part of the goaljobs package), or see the
45 HTML documentation installed as part of the goaljobs package.
47 =head1 THE SCRIPT FILE
49 The script file should usually start with opening these modules (none
50 of this are required, it's just useful to have them open):
56 This is followed by goals and/or functions and/or top-level OCaml
57 statements and/or C<every> statements (a.k.a periodic jobs).
59 You can use multiple script files to make up a goaljobs program. You
60 have to list them in dependency order on the goaljobs command line
61 (earlier files required by later files), the same way that the OCaml
62 compiler works. So usually you end up writing:
64 goaljobs utils.ml another_library.ml script.ml
66 where C<script.ml> requires the functions from the utils/library.
67 Note that circular dependencies are not possible.
71 Each goal should have the following basic form:
73 let goal name args... =
77 (* code to implement the goal *)
79 There is no hard-and-fast rule about this. In particular you can put
80 arbitrary OCaml statements anywhere inside a goal (since a goal is
81 just a special form of OCaml function), but sticking to this overall
84 There should be zero or one target. Multiple target statements should
85 not be used in a goal. The target should come as early as possible,
86 and the target condition should be as simple and fast to evaluate as
87 is practical (see L</THE MEMORY> below).
89 There should be zero or any number of C<require> statements. Each
90 require statement should name a single goal (with optional parameters
93 After that should come the code that implements the goal, which might
94 be, for example, a series of shell commands, but could even be
97 As with ordinary OCaml functions, you can define goals recursively
98 or with mutual recursion using:
100 let rec goal1 args... =
107 A goal can also have no arguments:
112 This defines the common goal called C<all>, which acts the same way as
113 C<make all>, ie. if you run the program without any arguments, it will
114 run the C<all> goal if one exists.
116 =head2 PUBLISHING GOALS
118 If a goal is "published" it means it is available to be run directly
119 from the command line. All no-arg goals are published by default.
120 You do not need to do anything special for them. For example:
122 let goal clean () = sh "rm *~"
124 can be used on the command line:
128 For goals which take any parameters, you have to define a small code
129 snippet that converts command line arguments to goal parameters (the
130 reason has to do with OCaml being strongly typed, and because goal
131 parameters might not all be strings).
133 let goal compile program sources =
134 target (more_recent [program] sources);
140 let program = List.hd args in
141 let sources = List.tl args in
142 require (compiled program sources)
147 ./myscript compile program main.c utils.c
149 =head1 TARGET AND REQUIRE
151 The target is promise or contract that you make that the given
152 condition I<will> be true when the goal has finished running.
154 In the first example, the target is that the C<o_file> (object) exists
155 and is newer than the C<c_file> (source). The goal meets that target
156 by running the C compiler (C<cc>) which, if it succeeds, will ensure
157 that the object file exists and is newer than the source file.
159 let goal compiled c_file =
160 let o_file = change_file_extension "o" c_file in
161 target (more_recent [o_file] [c_file]);
168 In the second example, the goal requires that several files have been
169 compiled (C<require (compiled ...)>) before it can link the final
172 let goal built program source =
173 target (more_recent [program] [source]);
175 require (compiled source);
177 let object = change_file_extension "o" source in
185 If you want to have a goal that runs when some outside event happens
186 you have three choices: Manually run the script (this is basically
187 what C<make> forces you to do). Have some sort of hook that runs the
188 script (eg. a git hook). Or use a periodic job to poll for an event
191 Periodic jobs run regularly to poll for an outside event or change.
192 If a script has periodic jobs, then it runs continuously (or until you
195 An example of a script that checks for new git commits and when it
196 sees one it will ensure it passes the tests:
198 let repo = Sys.getenv "HOME" // "repo"
200 let goal git_commit_tested commit =
201 let key = sprintf "repo-tested-%s" commit in
202 target (memory_exists key);
212 (* Record that this commit was tested successfully. *)
215 every 30 minutes (fun () ->
216 let commit = shout "cd %s && git rev-parse HEAD" repo in
217 (* Require that this commit has been tested. *)
218 require (git_commit_tested commit)
221 Some notes about the above example: Firstly only the current HEAD
222 commit is required to be tested. This is because older commits are
223 irrelevant and because if they failed the test before there is not
224 point retesting them (commits are immutable). Secondly we use the
225 Memory to remember that we have successfully tested a commit. This is
226 what stops the program from repeatedly testing the same commit.
230 You can call out to the Unix shell using one of the functions
231 C<Goaljobs.sh>, C<Goaljobs.shout> or C<Goaljobs.shlines>. (These
232 functions are documented in the C<goaljobs.mli> file / API
235 C<sh> runs the command(s). C<shout> collects the output of the
236 command (to stdout only) and returns it as a single string.
237 C<shlines> collects the output and returns it as a list of lines.
239 C<sh>, C<shout>, C<shlines> work like printf. ie. You can substitute
240 variables using C<%s>, C<%d> and so on. For example:
242 sh "rsync foo-%s.tar.gz example.com:/html/" version
244 Each shell runs in a new temporary directory. The temporary directory
245 and all its contents is deleted after the shell exits. If you want to
246 save any data, C<cd> somewhere. If you don't want the temporary
247 directory creation, use C<~tmpdir:false>.
249 The environment variable C<$builddir> is exported to the script. This
250 is the current directory when the goaljobs program was started.
252 Each invocation of C<sh> (etc) is a single shell (this is slightly
253 different from how C<make> works). For example:
257 tarball=$package.tar.gz
265 The shell error mode is set such that if any single command
266 returns an error then the C<sh> function as a whole exits with
271 to ignore the result of a command.
273 C</bin/sh> is used unless you set C<Goaljobs.shell> to some other
274 value. Note that the environment variable C<SHELL> is I<never> used.
278 "The Memory" is key/value storage which persists across goaljobs
279 sessions. It is stored in the file C<$HOME/.goaljobs-memory> (which is
280 a binary file, but you can delete it if you want).
282 The Memory is locked during accesses, so it is safe to read or write
283 it from multiple parallel goaljobs sessions.
285 Keys and values are strings. The keys should be globally unique, so
286 it is suggested you use some application-specific prefix. eg:
291 let goal tested version =
292 let key = "myapp-tested-" ^ version in
293 target (memory_exists key);
295 (* some code to test this version *)
299 Note in that example the value C<1> is arbitrary. You just want to
300 store I<any> value so that a later call to C<memory_exists> will
303 For information about C<Goaljobs.memory_*> APIs see the
304 C<goaljobs.mli> file / API documentation.
312 This is the default shell used by C<sh*> APIs. You can change
313 the shell by setting the C<Goaljobs.shell> reference.
317 The curl program (on the path) is used to check for and download
318 URLs by APIs such as C<Goaljobs.url_exists>.
320 =item C<~/.goaljobs-memory>
322 Persistent key/value store used when you use the C<Goaljobs.memory_*>
327 =head1 ENVIRONMENT VARIABLES
333 This environment variable is set to the current directory when the
334 goals program starts, and is available in goals, shell scripts, etc.
344 Richard W.M. Jones <rjones @ redhat . com>
348 (C) Copyright 2013 Red Hat Inc.,
350 This program is free software; you can redistribute it and/or modify
351 it under the terms of the GNU General Public License as published by
352 the Free Software Foundation; either version 2 of the License, or
353 (at your option) any later version.
355 This program is distributed in the hope that it will be useful,
356 but WITHOUT ANY WARRANTY; without even the implied warranty of
357 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
358 GNU General Public License for more details.
360 You should have received a copy of the GNU General Public License
361 along with this program; if not, write to the Free Software
362 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.