Airs – Ian Lance Taylor

Now, a cute kid story.

My daughter asked me why some people used to wear hair shirts and the like. I tried to explain about mortification of the flesh as a way to improve the spirit, how people tried to ignore their physical body to pursue a closer relationship with God. This naturally led to a discussion of the difference between salvation by faith and salvation by works. I’ve been reading a world history book to her (A Little History of the World, by E. H. Gombrich). We recently covered the Reformation and the associated religious wars. I explained that one of the key doctrinal differences between Martin Luther and the Catholic Church was the idea that faith is what matters, not good deeds.

My daughter said that she felt that God would favor good deeds rather than faith, so I asked why. I thought she would say something about how it’s good to make other people happy, but what she actually said was that if you only focus on faith you will serve as an inspiration to other people to only focus on faith, and that that will lead people to using the aforementioned hair shirts, and that God would surely not want you to encourage other people to hurt themselves. I’m not sure this entirely holds up, philosophically speaking, but it was a very interesting idea for me because I would never have thought of it myself.

While I am often surprised by what other people think, I have after all spoken with my daughter almost every day of her life. While I often don’t know what she is going to do, this is probably the first time she has surprised me with a truly new thought, one that I would not have thought myself. It’s a strange feeling.

Later, in discussing the way that Martin Luther advocated removing the priesthood as an intermediary between people and God, she said that evidently Luther was the inventor of the phrase “Oh My God!”

During the Reagan administration, the U.S. reduced tax rates and increased defense spending. The national debt as a percentage of overall GDP increased from 32.5% to 53.1% (Reagan called this increase in debt the “greatest disappointment” of his presidency). During the first Bush administration, it continued to rise, reaching 66.1%. During the Clinton administration, the government raised taxes, the economy grew, and defense spending was reduced somewhat; the debt decreased to 56.4% of GDP. During the second Bush administration, again taxes were reduced and defense spending was increased; the debt increased to 83.4% of GDP.

Today fiscal conservatives are arguing that the high levels of debt require that government spending be reduced. At the same time, the plan put forward by Republican representative Paul Ryan, and strongly supported by the Republican House, calls for more tax cuts and higher defense spending. While it’s understood that his plan will not be adopted, it’s hard to see how it can be a serious proposal for debt reduction.

It’s clear that the U.S. has a high level of debt due largely to past steps of reducing taxes while increasing spending. One can argue details back and forth quite a bit, but it’s also clear that the debt has increased significantly under Republican administrations. Fiscal conservatives now argue that the high level of debt shows that the U.S. can not afford social programs like Social Security and Medicare. But while one can argue about increasing health care costs, history suggests that that simply isn’t true. What is true is that the U.S. can not steadily cut taxes without cutting spending.

It’s perfectly consistent to say that the U.S. should be a low-tax, low-service country. But arguments about debt which don’t mention the possibility of tax increases are not telling the whole truth about how the U.S. got into its current situation. What has happened, intentionally or not, is that tax cuts are being leveraged to reduce spending on social programs.

Incidentally, I think most people agree that governments should use tax money to invest in infrastructure. It’s generally most efficient to let the government build and maintain roads and bridges, as they require a large investment and the payback is indirect. I think one could make a good argument that health care is another form of infrastructural investment, an investment in people, which is most efficiently done by government.

Sorry for the long delay. Anyhow, I wrote that so that I could write this.

DejaGNU is the test harness used by gcc, gdb, the GNU binutils, and probably other programs as well. Frankly, it’s a disaster. The documentation is weak, the implementation is complex and confusing, it’s slow, it does not support running tests in parallel, it’s hard to use. It has exactly two things in its favor, and they are powerful. The first is that it mostly works. The second is that people have written many different board support packages which let it test cross-compilers on simulators and real hardware.

DejaGNU was initially written at Cygnus by Rob Savoye as a way to test gdb. I don’t recall if there was any gdb testsuite prior to DejaGNU, but if there was it was largely useless. Because gdb was a command-line program, the idea for DejaGNU was to write a test harness which could run gdb, send it commands, and examine the resulting output. That was the first mistake. It meant that all the gdb tests were required to look for syntactic details of the output which were irrelevant to the test. Tests for gdb revolve around making sure that gdb stops in the right place and can print local variables and do backtraces and so forth. That is a lot of output which DejaGNU matches using regexps. I think it would have been smarter to put the effort into adding a test harness internally to gdb itself, so that a program could query gdb’s state. This could have evolved into the MI output format which would up getting added to gdb anyhow. Or it could have evolved into a library interface for gdb, something which would have been very useful for IDEs and other purposes and still does not really exist.

Anyhow, once the decision was made to test gdb as a pure command line program, Rob looked for a program which could do that. He came across expect. The tag line of the expect paper from 1990 is “Curing Those Uncontrollable Fits of Interaction.” The expect program does a nice job of that: if you have a program that you can only interact with manually, expect lets you write a program to interact with it instead. So expect is a nice choice when you need to work with a program you don’t control. In our case, we did control gdb; choosing expect was a hack to save time modifying gdb, a hack we are still paying for nearly 20 years later.

Expect uses an embedded Tcl interpreter, so expect programs are Tcl programs. This is a good use of Tcl: it means that expect has a full programming language for writing scripts. Since interacting with other programs is all about strings, it’s perfectly reasonable to use Tcl, which is also all about strings. The consequence for DejaGNU, though, is that DejaGNU is written in Tcl.

Once Cygnus was using DejaGNU as a test harness for gdb, it seemed natural to use it as a test harness for gcc as well. But of course gcc is not an interactive program, so the advantages of using expect no longer applied. The disadvantages of Tcl remained intact.

Cygnus specialized in cross-compilers, so DejaGNU grew the ability to build programs for target boards and run them there, using various different communication mechanisms. None of this had much to do with expect or Tcl, but it was all written in Tcl because that was the mechanism available. All this support is the main reason it is difficult to move away from DejaGNU today.

At least on a native system, it’s natural to want to run tests in parallel. That’s only become more important over the years. Unfortunately, Tcl doesn’t support threads (there is a thread extension available these days, but it is written in such a way that it would have to be integrated into expect before DejaGNU could use it). It’s easy enough to write Tcl code to start gcc a bunch of times, but it’s much harder to write Tcl code to examine those results. The gcc testsuite does now run in parallel, but it does so by manually creating subsets of tests and invoking DejaGNU multiple times in parallel to run those subsets. This works but is hardly optimal.

As a highly dynamic interpreted language, Tcl is relatively slow. The expect program is quite clever and sets up pseudo terminals in order to properly interact with general programs, and effort that is useless when testing a simple program like gcc. A significant amount of the CPU time taken by a gcc testsuite run is for expect, time which is largely wasted.

The DejaGNU code is complex and hard to read. This is not entirely the fault of Tcl, but Tcl is partly to blame as DejaGNU struggles with namespace issues. Function and variable names are constructed at runtime to avoid namespace collisions, which makes it very hard to figure out what code will run. It plays games like having tests return the name of the function to run to report whether the test succeed (e.g., return “pass” to invoke the function named pass), which sounds almost clever until you realize that there is nothing which prevents you from returning an invalid value.

I could continue with more specific horrors from DejaGNU, but those are in principle fixable. I hope that the earlier points show that DejaGNU itself is broken by design. We need to move away from it.

Unfortunately DejaGNU’s large knowledge base of how to run programs on embedded systems, a knowledge base which is largely represented in hand-written Tcl code, is very hard to get around. Some of these scripts can be automatically translated into a better test harness, in that they simply set flags for various tools and set a communication mechanism. Many others will require hand conversion.

Unfortunately DejaGNU has more or less blighted the world of free test harnesses. There is CodeSourcery’s qmtest program, but I don’t know how widely that is used. Fortunately, test harness need not be particularly complex. I don’t think it would be that hard for a thoughtful person to replace DejaGNU for gcc testing, and I think the benefits would be manifold. Replacing it for gdb testing would be harder, as the gdb tests rely more on string matching. As I mentioned above that is in itself a bug, but it means that recreating the tests is hard.

There are various test scripts which are built around DejaGNU’s log files, basically attempts to parse human readable information. Those will have to change for any new test harness.

Although I don’t have a good alternative, I hope I have at least demonstrated that DejaGNU must go. Effort put into working with DejaGNU is effort wasted.