1 TITLE: "goals" is a new tool which generalizes "make"
3 This talk is *not* about several things. It's *not* about build
4 tools. It's *not* about how autoconf sucks or the best tool
5 to build Java software. It's *not* about package management,
6 continuous integration, package ecosystems or anything like that.
8 It's about one tool which is over 40 years old: MAKE. Designed by
9 Stuart Feldman in 1976.
11 Make is a great tool! It's easy to get started, intuitive,
12 and wildly successful. If I critize make it's not because I
13 think it's a bad tool, just that we could do even better if
14 we addressed some shortcomings.
21 Only one predicate for solving dependencies.
23 "If the target file doesn't exist, or if it's older than
24 one of the dependencies, then run this recipe."
26 If you think for a little while you'll see that other
27 predicates are possible:
29 eg: URL, newer than any file (not all files),
30 Koji build, comparing checksums, test with skip
38 If "test" happens to be created, your tests will stop running
41 Experienced users know they can use the .PHONY directive to
44 Make doesn't check that when you run a rule that it actually
45 creates the file that it says it creates.
46 The new tool called goals does check this.
48 It also points to a fundamental issue: the target
49 is overloaded to mean either a file or a rule name.
53 SINGLE PARAMETER PROBLEM:
55 Make recipes can be parameterized, provided you only need
58 If a single parameter is useful, it's not outlandish to
59 imagine that having two parameters could be useful, or
67 How do you quote a file with spaces?
68 https://stackoverflow.com/questions/15004278/allow-space-in-target-of-gcc-makefile
70 For a tool whose main job is running shell commands
71 it has quite a lot of sharp edges when running shell commands.
78 - Individual commands are passed to separate shells
80 - $ has meaning to both shell and make
82 - Invisible whitespace is meaningful
87 Let's talk about shell scripting first, because that's easiest to fix:
89 target: foo.o bar.o "target": "foo.o", "bar.o" {
90 ${CC} ${CFLAGS} $< -o $@ %CC %CFLAGS %< -o %@
93 The new tool uses a real LALR(1) parser. Filenames have to be
94 enclosed in quotes, code always appears in curly braces, % is
95 the magic character for variables leaving $ for the shell to use,
96 commands in the code block run under a single shell (but any
97 command returning an error is still fatal), and the formatting
98 is free-form so there's no special whitespace.
102 Goals can optionally be given names and parameters:
104 goal all = : "target"
107 "target" : "foo.o", "bar.o" { ... }
109 goal compile (name) =
110 "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ }
113 You can run a goal in two ways. The "make way" is to find
114 the target that matches the given filename. "foo.o" matches
115 "%name.o" and so we know to run the compile goal. But
116 if you want you can also run compile ("bar") directly:
121 "target" : "foo.o", compile ("bar") { ... }
123 goal compile (name) =
124 "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ }
128 Predicates are special rules that we can use to change how we
129 determine if a goal needs to be rebuilt. When you see a
130 filename string, there's an implicit predicate called is-file, so
131 these are equivalent, because when goals sees a bare string
132 but it wants a predicate it implicitly uses is-file.
134 "target" : "foo.o", "bar.o" { ... }
136 is-file("target") : is-file("foo.o"), is-file("bar.o") { ... }
139 Apart from is-file being the default predicate, it's not built
140 into goals. In fact is-file is defined in the goals standard
141 library. The special @{...} code section means the code
142 doesn't print verbosely when its running. And "exit 99"
143 is used by the predicate to indicate that the target needs
144 to be rebuilt, but other than that it's all written in
145 ordinary shell script:
147 predicate is-file (filename) = @{
148 test -f %filename || exit 99
150 test %filename -ot "$f" && exit 99 ||:
156 And you can of course write other predicates in shell script.
157 Here's a predicate for running test suites. This predicate lets
158 you skip a test by setting an environment variable.
160 predicate is-test (script) = @{
161 # Check if SKIP variable is set.
163 echo -n SKIP_%script |
167 if test "${!skip_var}" = "1"; then exit 0; fi
168 if test %goals_final_check; then exit 0; else exit 99; fi
171 You can use the predicate like this. There's quite a lot to unpack
172 in this example, but I'll just say that the wildcard function
173 expands to a list of files, and the wrap function changes them
174 from a list of strings into a list of is-test predicates.
176 let tests = wrap ("is-test", wildcard ("test-*.sh"))
177 goal check () = : tests
178 goal run (script) = is-test(script) : { ./%script }
181 Another predicate we use is called is-koji-built, which I
182 use for mass rebuilding Fedora packages in dependency order.
183 I won't go into the full definition of is-koji-built since
184 interfacing with Koji is quite complicated, but you can
185 write a mass rebuild tool in goals fairly easily:
187 [ DISCUSSION OF FEDORA-OCAML-REBUILD IN SLIDES ]
191 We saw a couple of standard functions in the test example -
192 "wildcard" and "wrap". In make there are many built in functions.
193 In goals, all functions are defined in a standard library and
194 written in the goals language plus shell script. Here's the
195 definition of the wildcard function, this is the actual
196 code you're running if you use the wildcard function in a
197 Goalfile. You can see that functions can take zero, one, or
198 more parameters, and they can return strings or arbitrary Goalfile
201 function wildcard (wc) returning strings = @{
204 for f in $wc; do echo "$f"; done
208 In fact goals consists of a native core language parser and runtime
209 for evaluating the language, building the dependency graph and
210 executing jobs in parallel. But around this small core is a
211 large standard library which is written in the goals language
212 plus shell script. So in a sense goals is bootstrapped from
213 a smaller core into a larger ecosystem, which makes it quite
214 different from "make".
218 COMPUTER SCIENCY THINGS
219 (not necessary to know this)
221 There are some interesting parallels between the goals language
222 and programming languages that I want to highlight. Not least
223 because they point to future ways we might explore this space.
226 - Goals are functions + dependency solving
228 goal clean () = { rm -f *~ }
230 goal all () = : "program"
232 goal link = "program" : "foo.o" { %CC %CFLAGS %< -o %@ }
235 - Predicates are constructors
236 - Targets are patterns
238 is-file ("%name.o") : ... match name with
239 | File (name + ".o") -> compile name
242 - Goal "functions" may be called by name or by pattern,
243 which is unusual. Is there another programming language
248 - But our pattern matcher is very naive, could it be more complex?
249 What would that mean?
251 SCREENSHOT OF ZINC PAPER
254 - Dependencies have implicit & operator, could we use | and ! operators?
255 What would that mean? Build targets in several different ways?
256 Fallback if a tool isn't available?
262 - Types other than strings. Int and bool would be useful. However
263 this also implies that we should do type inference and/or checking.
267 goal build (project, bool release = true) = ...
271 - Anonymous functions
272 Any code section is potentially a closure
274 let hello = { echo "hello" }
276 let f = function (name, version) { CODE }
280 - Cover some of the other tools that are better makes.
281 (Note I'm not covering all DAG solvers like Ninja unless they
282 bring something new to the table)
284 * Plan 9 mk - has alternate build predicates
286 * https://github.com/casey/just - has functions and multi-parameter rules
288 * http://www.crufty.net/help/sjg/bmake-meta-mode.htm
289 - alternative route to finding deps by snooping on file
290 accesses at the kernel level
291 * https://github.com/apenwarr/redo - similar to above
293 * https://web.archive.org/web/20001027183954/http://software-carpentry.codesourcery.com/entries/build/Tromey/Tromey.html
299 - Explain predicates better, Kashyap found it confusing.