TITLE: "goals" is a new tool which generalizes "make" This talk is *not* about several things. It's *not* about build tools. It's *not* about how autoconf sucks or the best tool to build Java software. It's *not* about package management, continuous integration, package ecosystems or anything like that. It's about one tool which is over 40 years old: MAKE. Designed by Stuart Feldman in 1976. Make is a great tool! It's easy to get started, intuitive, and wildly successful. If I critize make it's not because I think it's a bad tool, just that we could do even better if we addressed some shortcomings. TACTIC PROBLEM: Only one tactic for solving dependencies. "If the target file doesn't exist, or if it's older than one of the dependencies, then run this recipe." If you think for a little while you'll see that other tactics are possible: eg: URL, newer than any file (not all files), Koji build, comparing checksums, test with skip PHONY FILE PROBLEM: "test" is not a file. If "test" happens to be created, your tests will stop running silently. Experienced users know they can use the .PHONY directive to avoid this. Make doesn't check that when you run a rule that it actually creates the file that it says it creates. The new tool called goals does check this. It also points to a fundamental issue: the target is overloaded to mean either a file or a rule name. SINGLE PARAMETER PROBLEM: Make recipes can be parameterized, provided you only need a single parameter. If a single parameter is useful, it's not outlandish to imagine that having two parameters could be useful, or even more. SHELL PROBLEM: How do you quote a file with spaces? https://stackoverflow.com/questions/15004278/allow-space-in-target-of-gcc-makefile For a tool whose main job is running shell commands it has quite a lot of sharp edges when running shell commands. dest/target: deps cd dest var=1 echo $var > target - Individual commands are passed to separate shells - $ has meaning to both shell and make - Invisible whitespace is meaningful Let's talk about shell scripting first, because that's easiest to fix: target: foo.o bar.o "target": "foo.o", "bar.o" { ${CC} ${CFLAGS} $< -o $@ %CC %CFLAGS %< -o %@ } The new tool uses a real LALR(1) parser. Filenames have to be enclosed in quotes, code always appears in curly braces, % is the magic character for variables leaving $ for the shell to use, commands in the code block run under a single shell (but any command returning an error is still fatal), and the formatting is free-form so there's no special whitespace. Goals can optionally be given names and parameters: goal all = : "target" goal link = "target" : "foo.o", "bar.o" { ... } goal compile (name) = "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ } You can run a goal in two ways. The "make way" is to find the target that matches the given filename. "foo.o" matches "%name.o" and so we know to run the compile goal. But if you want you can also run compile ("bar") directly: goal all = : link goal link = "target" : "foo.o", compile ("bar") { ... } goal compile (name) = "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ } Tactics are special rules that we can use to change how we determine if a goal needs to be rebuilt. When you see a filename string, there's an implicit tactic called *file, so these are equivalent, because when goals sees a bare string but it wants a tactic it implicitly uses *file. "target" : "foo.o", "bar.o" { ... } *file("target") : *file("foo.o"), *file("bar.o") { ... } Apart from *file being the default tactic, it's not built into goals. In fact *file is defined in the goals standard library. The special @{...} code section means the code doesn't print verbosely when its running. And "exit 99" is used by the tactic to indicate that the target needs to be rebuilt, but other than that it's all written in ordinary shell script: tactic *file (filename) = @{ test -f %filename || exit 99 for f in %<; do test %filename -ot "$f" && exit 99 ||: done } And you can of course write other tactics in shell script. Here's a tactic for running test suites. This tactic lets you skip a test by setting an environment variable. tactic *test (script) = @{ # Check if SKIP variable is set. skip_var=$( echo -n SKIP_%script | tr 'a-z' 'A-Z' | tr -c 'A-Z0-9' '_' ) if test "${!skip_var}" = "1"; then exit 0; fi if test %goals_final_check; then exit 0; else exit 99; fi } You can use the tactic like this. There's quite a lot to unpack in this example, but I'll just say that the wildcard function expands to a list of files, and the wrap function changes them from a list of strings into a list of *test tactics. let tests = wrap ("*test", wildcard ("test-*.sh")) goal check () = : tests goal run (script) = *test(script) : { ./%script } Another tactic we use is called *koji-built, which I use for mass rebuilding Fedora packages in dependency order. I won't go into the full definition of *koji-built since interfacing with Koji is quite complicated, but you can write a mass rebuild tool in goals fairly easily: [ DISCUSSION OF FEDORA-OCAML-REBUILD IN SLIDES ] We saw a couple of standard functions in the test example - "wildcard" and "wrap". In make there are many built in functions. In goals, all functions are defined in a standard library and written in the goals language plus shell script. Here's the definition of the wildcard function, this is the actual code you're running if you use the wildcard function in a Goalfile. You can see that functions can take zero, one, or more parameters, and they can return strings or arbitrary Goalfile expressions. function wildcard (wc) returning strings = @{ shopt -s nullglob wc=%wc for f in $wc; do echo "$f"; done } In fact goals consists of a native core language parser and runtime for evaluating the language, building the dependency graph and executing jobs in parallel. But around this small core is a large standard library which is written in the goals language plus shell script. So in a sense goals is bootstrapped from a smaller core into a larger ecosystem, which makes it quite different from "make". COMPUTER SCIENCY THINGS (not necessary to know this) There are some interesting parallels between the goals language and programming languages that I want to highlight. Not least because they point to future ways we might explore this space. - Goals are functions + dependency solving goal clean () = { rm -f *~ } goal all () = : "program" goal link = "program" : "foo.o" { %CC %CFLAGS %< -o %@ } - Tactics are constructors - Targets are patterns *file ("%name.o") : ... match name with | File (name + ".o") -> compile name | ... - Goal "functions" may be called by name or by pattern, which is unusual. Is there another programming language which does this? (Prolog actually) - But our pattern matcher is very naive, could it be more complex? What would that mean? SCREENSHOT OF ZINC PAPER - Dependencies have implicit & operator, could we use | and ! operators? What would that mean? Build targets in several different ways? Fallback if a tool isn't available? TO-DO - Types other than strings. Int and bool would be useful. However this also implies that we should do type inference and/or checking. - Default arguments goal build (project, bool release = true) = ... build ("foo") build ("foo", false) - Anonymous functions Any code section is potentially a closure let hello = { echo "hello" } let f = function (name, version) { CODE } f ("goals", "0.1")