From: Richard W.M. Jones Date: Sat, 11 Jan 2020 15:16:07 +0000 (+0000) Subject: Add notes for talk on goals. X-Git-Url: http://git.annexia.org/?a=commitdiff_plain;h=92a5a3f603374f88037129ac1a1456a2c250c4ef;p=libguestfs-talks.git Add notes for talk on goals. --- diff --git a/2020-goals/notes.txt b/2020-goals/notes.txt new file mode 100644 index 0000000..68604c6 --- /dev/null +++ b/2020-goals/notes.txt @@ -0,0 +1,208 @@ +TITLE: "goals" is a new tool which generalizes "make" + +This talk is *not* about several things. It's *not* about build +tools. It's *not* about how autoconf sucks or the best tool +to build Java software. It's *not* about package management, +continuous integration, package ecosystems or anything like that. + +It's about one tool which is over 40 years old: MAKE. Designed by +Stuart Feldman in 1976. + + + +TACTIC PROBLEM: + - Only one tactic. + - Others are possible, + eg: URL, newer than any file (not all files), + Koji build, comparing checksums, test with skip + +PHONY FILE PROBLEM: + - "all" is not like a file + +SINGLE PARAMETER PROBLEM: + - %.o: %c only allows a single parameter + +SHELL PROBLEM: + - How do you quote a file with spaces? + https://stackoverflow.com/questions/15004278/allow-space-in-target-of-gcc-makefile + For a tool whose main job is running shell commands + it has quite a lot of problems with shell commands: + dest/target: deps + cd dest + var=1 + echo $var > target + - Invisible whitespace is meaningful + - Individual commands are passed to separate shells + - $ has meaning to both shell and make + + + + +Let's talk about shell scripting first, because that's easiest to fix: + + target: foo.o bar.o "target": "foo.o", "bar.o" { + $CC $CFLAGS $< -o $@ %CC %CFLAGS %< -o %@ + } + +The new tool uses a real LALR(1) parser. Filenames have to be +enclosed in quotes, code always appears in curly braces, % is +the magic character for variables leaving $ for the shell to use, +commands in the code block run under a single shell (but any +command returning an error is still fatal), and the formatting +is free-form so there's no special whitespace. + + + +Goals can optionally be given names and parameters: + + goal all = : "target" + + goal link = + "target" : "foo.o", "bar.o" { ... } + + goal compile (name) = + "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ } + + +You can run a goal in two ways. The "make way" is to find +the target that matches the given filename. "foo.o" matches +"%name.o" and so we know to run the compile goal. But +if you want you can also run compile ("bar") directly: + + goal all = : link + + goal link = + "target" : "foo.o", compile ("bar") { ... } + + goal compile (name) = + "%name.o" : "%name.c", "dep.h" { %CC %CFLAGS -c $^ -o $@ } + + + +Tactics are special rules that we can use to change how we +determine if a goal needs to be rebuilt. When you see a +filename string, there's an implicit tactic called *file, so +these are equivalent, because when goals sees a bare string +but it wants a tactic it implicitly uses *file. + + "target" : "foo.o", "bar.o" { ... } + + *file("target") : *file("foo.o"), *file("bar.o") { ... } + + +Apart from *file being the default tactic, it's not built +into goals. In fact *file is defined in the goals standard +library. The special @{...} code section means the code +doesn't print verbosely when its running. And "exit 99" +is used by the tactic to indicate that the target needs +to be rebuilt, but other than that it's all written in +ordinary shell script: + + tactic *file (filename) = @{ + test -f %filename || exit 99 + for f in %<; do + test %filename -ot "$f" && exit 99 ||: + done + } + + + +And you can of course write other tactics in shell script. +Here's a tactic for running test suites. This tactic lets +you skip a test by setting an environment variable. + + tactic *test (script) = @{ + # Check if SKIP variable is set. + skip_var=$( + echo -n SKIP_%script | + tr 'a-z' 'A-Z' | + tr -c 'A-Z0-9' '_' + ) + if test "${!skip_var}" = "1"; then exit 0; fi + if test %goals_final_check; then exit 0; else exit 99; fi + } + +You can use the tactic like this. There's quite a lot to unpack +in this example, but I'll just say that the wildcard function +expands to a list of files, and the wrap function changes them +from a list of strings into a list of *test tactics. + + let tests = wrap ("*test", wildcard ("test-*.sh")) + goal check () = : tests + goal test (script) = *test(name) : { ./%name } + + +Another tactic we use is called *built-in-koji, which I +use for mass rebuilding Fedora packages in dependency order. +I won't go into the full definition of *built-in-koji since +interfacing with Koji is quite complicated, but you can +write a mass rebuild tool in goals fairly easily: + +[SHOW OUTLINE FROM fedora-ocaml-rebuild/Goalfile] + + +We saw a couple of standard functions in the test example - +"wildcard" and "wrap". In make there are many built in functions. +In goals, all functions are defined in a standard library and +written in the goals language plus shell script. Here's the +definition of the wildcard function, this is the actual +code you're running if you use the wildcard function in a +Goalfile. You can see that functions can take zero, one, or +more parameters, and they can return strings or arbitrary Goalfile +expressions. + + function wildcard (wc) returning strings = @{ + shopt -s nullglob + wc=%wc + for f in $wc; do echo "$f"; done + } + + +In fact goals consists of a native core language parser and runtime +for evaluating the language, building the dependency graph and +executing jobs in parallel. But around this small core is a +large standard library which is written in the goals language +plus shell script. So in a sense goals is bootstrapped from +a smaller core into a larger ecosystem, which makes it quite +different from "make". + + + +COMPUTER SCIENCY THINGS +(not necessary to know this) + +There are some interesting parallels between the goals language +and programming languages that I want to highlight. Not least +because they point to future ways we might explore this space. + + + - Goals are functions + dependency solving + + goal clean () = { rm -f *~ } + + goal all () = : "program" + + goal link = "program" : "foo.o" { %CC %CFLAGS %< -o %@ } + + + - Tactics are constructors + - Targets are patterns + + *file ("%name.o") : ... match name with + | File (name + ".o") -> compile name + | ... + + - But our pattern matcher is very naive, could it be more complex? + What would that mean? +SCREENSHOT OF ZINC PAPER + + + - Goal "functions" may be called by name or by pattern, + which is unusual. Is there another programming language + which does this? + (Prolog actually) + + + - Dependencies have implicit & operator, could we use | and ! operators? + What would that mean? Build targets in several different ways? + Fallback if a tool isn't available?