Machina addendum

James Amundson
Fermi National Accelerator Laboratory

Appendix: The sins of make

The purpose of the Machina design document is to propose a new build system, not to complain about the deficiencies of an old one. Nevertheless, the ubiquity of make has affected the general consciousness of the software building process. The purpose of this section is to encourage the reader to step back and consider the difficulties in building software and discover how many of them are limitations of make itself.

Make and sources

A typical makefile does not refer to sources at all. Generic rules in make depend on pattern matching, i.e., .o files depend on .c files. While it is possible to write rules where particular objects depend on particular sources, it comes at a great cost. To do so, the developer must relinquish all access to make's built-in knowledge on how to compile files. She also loses all ability to describe the build process in any generality: a separate rule must be specified for each file to be compiled. Here are two real world complications that arise because of the way make deals with sources:

  1. It is impossible to deal with multiple file extensions. In designing C++, Stroustrup explicitly declined to specify a standard file extension for C++ source files. As a result there are many different file extensions in common use. It is impossible in make to specify a list of possible endings for source files. As a result, developers of generic makefiles must make separate rules for each files extension. In the end, they can only hope that they have found them all. Making it easy for developers to add their own extensions is nontrivial.
  2. Extra source files can get in the way. In the process of writing a C++ wrapper for a legacy Fortran library, I found that running the Fortran sources through f2c was an invaluable aid in understanding calling conventions. I once wasted an afternoon of work because I ran legacy.f through f2c to produce legacy.c and left legacy.c in my source directory. Subsequent changes to legacy.f did not seem to affect my final result. It took me a long time to discover that the built-in rule for .c files was being used by make to generate legacy.o. Since my makefile only referred to legacy.o, there was no way for make to tell what I meant.
Machina improves on this situation by focusing on the source files, not the intermediate files. The developer tells Machina which language the sources are written in. Extensions can be arbitrary. The control is in the hands of the developer

Make and translation units

Take the simple example used in the translation unit section in the main text. The c file foo.c includes the header file foo.h. These files are compiled to create foo.o. Invariably, this relationship is modeled using make as follows.


This relationship is at once simple, accurate, and extremely difficult to maintain. It is simple because it requires no new concepts in make. It is accurate because changes in either foo.h or foo.c represent a change in the translation unit, which correctly triggers an update of foo.o. It is a maintenance disaster because there is no good way to keep the dependency relationship itself up-to-date. Here are some possibilities:

  1. Enter the information manually in the makefile. Not only is this approach inconvenient for the programmer, it is demonstrably difficult to maintain. The contents of the translation unit are already entered in the files themeselves through include statements. Requiring this information to be updated separately in the makefile means we have introduced multiple points of maintenance, inviting error.
  2. Automatically extract the information from foo.c. This is the usual approach. There are several ways to go about extracting the information. The only foolproof approach is to extract the information each time the makefile is parsed. This is dramatically inefficient. A much more efficient approach is to discover the the dependency on foo.h only when foo.c is compiled, then to save this information. This is the approach taken by automake. It can cause make to fail if the headers are moved or deleted between compiles. Many users are familiar with the following problem: A programmer creates a package on his machine, types ``make clean'', then tars it and sends it to novice (or not-so novice) user on a different machine. The user tries to compile the files on his machine, only to be greeted by the message ``unable to make /strange-long-pathname/foo.h''. The problem here being that the original programmer forgot to type ``make distclean'' before distributing his package. The burden of dealing with translation units has been passed along to the users, in a bewildering way.
Make lacks a workable method for dealing with non-trivial translation units, even though they are the fundamental components of almost any project. The Machina solution to this problem is described in detail in the text.

Make and output

Building almost any non-trivial project in make produces reams of output. This output is remarkable in that it manages to be neither machine readable nor human readable. It is not machine readable because it is completely free form. I have seen (and even worked on) various projects to parse the output of make, but they all end up being very complicated and very fragile. It would seem logical that free form output would be human readable, but the raw volume of completely undifferentiated output overwhelms all but the most determined reader. Because make simply copies the commands it invokes to the screen, the more sophisticated the makefile, the more difficult the output is to read.

Machina solves this problem by offering a variety of levels of verbosity. By default, the user sees only a brief message at each step unless an error has occurred; only commands producing errors are displayed in their entirety. Furthermore, Machina provides a system by which compiler output can be parsed and sent to the user in a meaningful way. All output is available in either human-readable or machine-readable form.

Make and modularity

Make has no concept of modularity. The most common workaround for this problem is the use of recursive makefiles. The pitfalls of this approach are well documented in Peter Miller's paper Recursive Make Considered Harmful, <>. A large project I am involved with here at Fermilab contains over 800,000 lines of make. Dealing with that is painful, whether it is through recursive make, or Miller's include-it-all-in-one-giant-makefile approach.

It is interesting to see how large free software projects deal with this problem. The GNOME project is a good example. A frequent complaint on the part of new users of GNOME is the difficulty in getting it installed on their systems. The installation instructions at <> list 12 base, 3 core application and 16 additional packages, for a total of 31 packages. The instructions say:

Download all the source files in order, and use the standard ./configure; make; make install command sequence to compile and install each package. If you choose to specify a -prefix option to the ./configure command, please specify the same one for all the packages - current GNOME limitations prevent full functionality unless this is done.
These instructions constitute a special kind of recursive makefile. It is special because instead of doing the work with make, it does the work with a human typing on a keyboard. There has to be a better way.

In Machina, there is a better way.

Make is a rule-based system

Make is a rule-based, or production system. Rule-based systems have a long history in artificial intelligence research. Their strength lies in their ability to make inferences. Their weaknesses, however, are many. For a discussion, see, e.g., the chapter on production systems in The Handbook of artificial intelligence, edited by Avron Barr and Edward A. Feigenbaum, Addison-Wesley, 1986. Although the list of problems given there will resonate with anyone who has spent much time with make, one problem stands out above the rest: opacity. I have never met anyone who needed an inference system to tell him how to build his project. I have, however, met many people who do not know what make is doing when it tries to build their projects.

Machina is not a rule-based system.

James Amundson 2000-03-31