From 07670525fc00bfb9aa0ba4da160d0c7453c857c2 Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" Date: Sun, 19 Apr 2020 18:59:52 +0100 Subject: [PATCH] Add full README describing how this works. --- README | 130 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-) diff --git a/README b/README index 9ceeea0..79f5dc2 100644 --- a/README +++ b/README @@ -1,5 +1,127 @@ -Perform a complete Fedora OCaml rebuild, in build order. -Requires the goals tool found here: -http://git.annexia.org/?p=goals.git;a=summary +*** DO NOT RUN THIS without first reading the code and this README *** -DON'T RUN THIS without first reading the code. +This "script" is used to perform a complete Fedora rebuild +of all OCaml packages, in build-requires order. + +It requires the goals tool found here: + + http://git.annexia.org/?p=goals.git;a=summary + +You should usually use the version of goals compiled from that git +repository (using the ./run script in the top directory) because we +are still evolving goals and this script together. + +In the video below I go into an explanation of how the goals tool +works, and I definitely recommend watching it. There is one +relatively minor change we made to goals since this video was +produced: tactics are now called "is-..." rather than using the +asterisk (so for example "is-koji-built" instead of "*koji-built"). +Apart from that it generally still works as described in the video. + + https://rwmj.wordpress.com/2020/01/14/goals-an-experimental-new-tool-which-generalizes-make/ + +Everything in this git repo is driven from the single "Goalfile". +However it uses functions and tactics from the goals standard library, +especially goals.git/stdlib/fedora.gl. It's not really necessary to +understand the library in detail, but keep this in mind in case you +find a function or tactic in this Goalfile which doesn't appear to be +defined anywhere. + +You will need all Fedora OCaml packages checked out locally (or at +least, all the ones which you want to rebuild). To find a list of all +OCaml packages, look in Koji here: + + https://koji.fedoraproject.org/koji/search?match=glob&type=package&terms=ocaml* + +and also look at the "other-packages" definition in Goalfile. + +The Goalfile lets you set the path to where you have checked out these +Fedora packages. I use "~/d/fedora". + +You will need to either be a Fedora Proven Packager, or have at least +commit access to each OCaml package that you intend to rebuild, since +building them involves pushing a Release bump. + +Fedora builds are generally done into side tags, and this Goalfile can +either build into a side tag or into a main Fedora branch (eg. +"f99-build"). For major rebuilds it is Fedora policy to use a side +tag. You can choose by setting "fedora-tag" at the top of the +Goalfile. + +You will also need to decide on a unique string which identifies the +build. It should be a string which will never be reused (so don't use +something generic like "OCaml rebuild."). Also the string must not +appear in the existing spec files already. This is because we grep +the spec file to decide if the package has already been rebuilt. + +You can ignore packages by setting fedora-ignored. We usually use +this for packages which are broken and we are unable to fix +immediately. + +Packages must not have cyclic dependencies. There are two packages in +particular which need manual attention: ocaml-menhir and ocaml-dune. +Both have "bootstrap" settings which you have to enable (see their +specfiles). Also you will have to remember to come back after the +full rebuild, disable the bootstrap setting, and build those packages +again. + +So you're ready to do an OCaml rebuild? Good, but first you must +build the ocaml (compiler) package manually. We don't automate this +step (although we could) because we assume that the main reason we do +an OCaml rebuild in the first place is because we want to update the +compiler to a new version. + +When you're ready to do a rebuild: + +(1) Make sure your Kerberos key is refreshed. If it expires during + the process then all following builds will fail, which sucks. + + kinit username@FEDORAPROJECT.ORG + +(2) Similarly you may need to make sure your Fedora SSH key hasn't + expired and ssh-agent is working. + +(3) You probably want to set up screen/tmux if you are using a remote + server because the whole rebuild will take quite a long time, + possibly 1-2 days. + +(4) Assuming you have goals checked out and compiled in an adjacent + directory, do: + + ../goals/run goals -k + +The -k option "keeps going" on failure (rebuilding all packages that +don't depend on failed packages). This is a personal preference: Do +you want to get as much done as possible, or do you want to fix things +as soon as possible? + +You may also want to try the -j N option to limit (or increase) the +number of parallel builds that will be run. The default of goals is +to use the same number of cores as your local machine, which is not +very useful since you will be doing the actual builds on the Fedora +koji servers. + +As the script is running, builds will fly by because it's doing many +parallel builds, but also there will be periods where it appears to +have hung - these are caused by us waiting for Koji to build the +newRepo. + +The script will run for a long time, and (inevitably I'm afraid) fail. +Pay careful attention to the failures, printed at the end, and also to +Koji and email messages that you get. + +The script as written at the moment will only attempt each build once. +If a build fails you MUST fix it yourself (or add the package and +dependencies to the ignored list). An earlier version of the script +did in fact keep retrying failed builds, but we found that wasn't very +helpful. When fixing and rebuilding a package, don't forget to +rebuild it into the side tag (if using side tags) not into the main +Fedora branches! + +After the rebuild completes, don't forget to rebuild the bootstrap +packages manually. Then you will need to create a Bodhi update for +the side tag. + +Good luck! + +Richard Jones, 19th April 2020 -- 1.8.3.1