--- title: Most Makefiles Should .DELETE_ON_ERROR description: _Make_ has been used extensively for forty years and offers incremental builds, parallelization, and a declarative syntax. In this post we'll take a look at how the `.DELETE_ON_ERROR` special target helps eliminate possible downstream problems in your _makefiles_. You'll also come to understand why most _makefiles_ should include it. author: - J. David Giese needs_pigments: true topics: - Software --- # Topic The utility of the `.DELETE_ON_ERROR` makefile special target. # Audience People who are already familiar with Make, but may have forgotten some details about the terminology. Also, assume the audience is familiar with the linux command line (e.g., what an exit status is). # Keywords - DICOM (will mention parsing the DICOM standard as an example use case) - Expert (will say this is an expert tip for using Make) - Software Development - Regulatory Documentation (will mention this as an example use case) # Research - Refreshed on GNU Make lingo: https://www.gnu.org/software/make/manual/html_node/Rules.html#Rules - Rule includes - Target - Prerequisites - Recipe - Errors in Makefiles: https://www.gnu.org/software/make/manual/html_node/Errors.html#Errors - Interrupts: https://www.gnu.org/software/make/manual/html_node/Interrupts.html#Interrupts - Recipe syntax: https://www.gnu.org/software/make/manual/html_node/Recipe-Syntax.html#Recipe-Syntax - After a few simple transformations, each line is passed to the set shell "as-is" - How Recipes are executed: - https://www.gnu.org/software/make/manual/html_node/Execution.html#Execution - Each line is executed in a new shell (/bin/sh by default) - Why make is useful: - https://medium.com/@jolson88/its-time-for-makefiles-to-make-a-comeback-36cbc358bb0a - One thing I really like about this is that things only run if they need to. You don't even need an incremental compiler to make it possible. If source files haven't been updated, there is no need to regenerate the target files. Make knows this by comparing the last modified times of the source files compared to the target files. - https://www.cs.virginia.edu/~dww4s/articles/build_systems.html - At their core, build systems are a functional based languages mapping a set of source resources (in most cases, files) to a target (executable). The primary assumption of the build system is that each of the build actions are idempotent. That is, each invocation of a build command with the same input and options will create the same output. This assumption allows the build system to memoize the actions that is has already performed, and only perform build actions on resources that have changed. - When are redirection locations generated? - Before the command is run; Bash man page # Outline - The .DELETE\_ON\_ERROR special target - Hook: Most makefiles would be better with the `.DELETE_ON_ERROR` special target - Paraphrase description + motiviation of the option - Summarize rest of article - Make refresher - Refresher on Rule, Target, Prerequisites, Recipe - How recipes are executed - Special targets - When the problem can occur - Bugs in prerequisites - Bugs in scripts that are not prerequisites - Scripts that are not idempotent (bc they make network requests) - GNU quote about `.DELETE_ON_ERROR` - Why you may not want to include it: GNU - Why you may not want to include it: precious files - Continue Learning - Link to some places where we use it at Innolitis - Point to man file - Make resources # Figures - A diagram illustrating the parts of a rule. # Title Expert Make Tip: Use `.DELETE_ON_ERROR` # Grammarly I ran the script through grammarly; we don't have an editor yet so I skipped this step for now. ## _Makefile_ Lingo refresher A _makefile_ consists of one or more *rules*. Here's an example with one rule: ```makefile languages.csv: projects.json python extract_languages.py projects.json > languages.csv ``` The first line informs us that languages.csv, the target, depends on projects.json, its only prerequisite. The next line—the rule's recipe—is a script that creates the target from its prerequisites. Recipes often consist of multiple lines. ![Key makefile terminology](/img/articles/make-delete-on-error/makefile-lingo.svg) ## Special targets Certain names that begin with a period have special meaning if they appear as targets. For example, consider a rule having `.PHONY` as its target name. The rule prerequisites will be considered phony targets and will always have their recipes run, regardless of their last modified times. `.PHONY` is one example of a special target that configures how _make_ treats its prerequisites. Others act as global configuration. For example, the `.DELETE_ON_ERROR` and `.ONE_SHELL` special targets act globally. If we added `.DELETE_ON_ERROR` to our _makefile_ example it would look as follows: ```makefile .DELETE_ON_ERROR: languages.csv: projects.json python extract_languages.py projects.json > languages.csv ``` A trailing colon signals that `.DELETE_ON_ERROR` is the target of a rule, even though it has no prerequisites and no recipe. ## How _Make_ runs recipes _Make_ processes a rule when it compares the last modification time of the target to that of its prerequisites. If a prerequisite is newer than the target (or doesn't exist), then _make_ runs the recipe. _Make_ executes each recipe line in a new shell, one by one, and quits if an invocation exits with a non-zero status. If the target file was altered prior to exiting, then its last modified time will indicate that it's up to date—even though it's likely corrupt or incomplete. Thus, next time _make_ runs it won't update the target file. This behavior is often confusing and may lead to malformed build artifacts. But if the _makefile_ includes the `.DELETE_ON_ERROR` special target, then _make_ deletes the target file if a recipe has an error. This behaviour solves several problems. ## Problem 1: rerunning _Make_ with invalid targets Imagine that you introduce a syntax error while updating projects.json. You run our example _makefile_ and extract_languages.py has a non-zero exit status—but not before the shell's output redirection truncates language.csv and updates its last modified time. Thus languages.csv will be empty, although _make_ considers it up to date. Fixing the syntax error in projects.json increments its last modified time and informs _make_ that it needs to rebuild languages.csv. However, imagine this rule is part of a complex _makefile_ which was run in parallel. It's easy to imagine a developer—especially one not familiar with the project—rerunning _make_ without fixing the error. If an empty languages.csv didn't cause problems downstream, _make_ would successfully complete, possibly without the developer realizing the build targets are invalid. Using `.DELETE_ON_ERROR` prevents this from occurring. ## Problem 2: defects in missing dependencies Now imagine there is a bug in extract_languages.py. Since that isn't a prerequisite, _make_ won't recognize that languages.csv is out of date even if you fix the bug. Therefore, once you do you'll need to manually delete languages.csv to have _make_ build it again. The ideal solution is to include your scripts as prerequisites. Doing so protects you in this situation while also ensuring that _make_ rebuilds your targets if your scripts are updated. However, maintaining script dependencies can be challenging (extract_languages.py may depend on many other python files). Using `.DELETE_ON_ERROR` is a practical alternative. It avoids the need to manually clean invalid targets while not forcing you to maintain your scripts as prerequisites. ## Problem 3: flaky recipes Finally, imagine that extract_languages.py makes an HTTP request to a web server. If the server is unavailable, languages.csv will be invalid. Once it's back online, you would need to manually delete languages.csv before _make_ would run the recipe again. Using `.DELETE_ON_ERROR` ensures you don't need to manually clean targets constructed from flaky recipes. ## Does `.DELETE_ON_ERROR` have downsides? The _GNU Make Manual_ says this about deleting target files after a recipe error: > This is almost always what you want make to do, but it is not historical practice; so for compatibility, you must explicitly request it. After considering the problems `.DELETE_ON_ERROR` solves, it's reasonable to ask if there are situations when you wouldn't want to include it in your _makefiles_. The main reason to not use `.DELETE_ON_ERROR` is that it's a GNU _make_ extension. GNU _make_ is quite pervasive, but you may not want to rely on it if you build on multiple platforms. (See this excellent tutorial about [writing portable _makefiles_](https://nullprogram.com/blog/2017/08/20/).) You also may not want to enable `.DELETE_ON_ERROR` if you want to keep the target files even after an error occurs (e.g., imagine a target takes weeks to compute). Here it may be appropriate to use `.DELETE_ON_ERROR`, but to list these “precious” targets as prerequisites of the .PRECIOUS special target. This informs _make_ not to delete them even if there is an error in the recipe. ## Continue learning Being easy to read and well written, Innolitics recommends the _[GNU Make Manual](https://www.gnu.org/software/make/manual/html_node/index.html#Top)_ if you want to learn more about _make_. Its pages that discuss [special targets](https://www.gnu.org/software/make/manual/html_node/Special-Targets.html) and [errors in recipes](https://www.gnu.org/software/make/manual/html_node/Errors.html#Errors) are especially relevant to this article. John Graham-Cumming wrote a book about GNU _Make_; much of the book originated from [his blog posts on GNU _Make_](https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html), which are still available and are useful. How could we improve our sample _makefile_? Hint: check out the [automatic variables](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html) page. At Innolitics, our medical imaging software development team uses _make_ to [build regulatory documents](https://github.com/innolitics/rdm/blob/master/rdm/init_files/Makefile), to transform the DICOM standard into a [condensed JSON format](https://github.com/innolitics/dicom-standard/blob/master/dicom_standard/Makefile) (for our [standard browser](https://dicom.innolitics.com/ciods)), and other tasks.