Airs – Ian Lance Taylor

Archive Alignment

gold normally simply mmaps input files and reads the data directly from the mapped memory. In general this requires that all the data structures be properly aligned in the file, which is guaranteed by the ELF standard. The x86, of course, does not require proper alignment for memory loads and stores, so this was never tested. When David Miller tested on a SPARC host, he saw crashes.

It turns out that the problem was object files in archives. Archives only guarantee an alignment to 16-bit boundaries. ELF data structures require 32-bit or 64-bit boundaries (the latter for 64-bit ELF). Thus it is possible for an object file stored in an archive to be misaligned with respect to mmap.

I fixed this in gold in the obvious way. But it suggests that changing the archive format to enforce 64-bit alignment could give better linker performance. Unfortunately the archive format, which dates back the old a.out format, is quite simple. The size of the object file is stored, but not the size that it takes up in the archive. And there is no global table of contents pointing to each object file. So a general archive file reader must walk through the archive file one object file at a time. Archive file readers must align to 16-bit boundaries as they go. There is no provision for recording the amount of required alignment.

The only way I see to get optimal performance would be to actually define a new archive format, with a new magic string. It’s not clear that the compatibility hassles would be worth it.

Posted

in

by

Ian Lance Taylor

Tags:

Comments

9 responses to “Archive Alignment”

  1. BenHutchings Avatar
    BenHutchings

    I’m not sure what the “obvious way” to deal with this when reading is, as I can think of at least 2 possible solutions: malloc() and read() headers to fix alignment, or use the packed attribute.

    Rather than change the archive format, you could ensure alignment when writing by inserting dummy files of appropriate length with names that should never be matched, such as “./.”.

  2. ncm Avatar
    ncm

    It seems like you could fix ar so it always aligns on 16-byte boundaries. That wouldn’t help linking with existing .a files, but the number of them will naturally decline. That also wouldn’t improve the time spent reading them, but you wouldn’t need to memmove the contents of up-to-date archives.

    Seems, too, that you could make ranlib do something useful again, by making it insert a global table of contents at the beginning, and insert
    padding to align the objects.

    Neither of these seems to demand a new magic number.

  3. Ian Lance Taylor Avatar
    Ian Lance Taylor

    BenHutchings: for gold using the packed attribute would be a bad idea, as it would mean that in every case we use byte reads rather than whole word reads. That would make everything slower. What I do is malloc and read or memcpy depending on whether the data is available.

    Inserting dummy files is a good idea, though it would be problematic when people do “ar x”. Some Makefiles do in fact do that to merge several archives into one. Unless of course you modify the ar program the ignore the dummy files. But then you’re not that far off from a different magic number.

    ncm: Fixing ar to change the alignment gives you a backward compatibility problem, in that old ar programs will not be able to read your new archives. That would be bad. ar just doesn’t give you any way t o insert padding. Files are just concatenated in an archive, with file headers in between, aligned to a 16-bit boundary. There is no way to say that a particular archive requires additional alignment. The ar program simply expects the next fixed-size file header to be on a 16-bit boundary after the end of the file. Perhaps I just don’t see what you are getting at.

  4. ncm Avatar
    ncm

    Is there any particular reason why the size of the object as stored in the archive needs to be exactly the size of the file that was appended? If you extracted the files and they came out a bit larger, how would it matter? (Who extracts files from .a archives, any more, anyway?) If you thought it really important to preserve sizes, by default, you could have a “–align” option discoverable by configure.

    Maybe ar could recognize ELF files being appended, and make any added padding part of the ELF layout. Finally, when extracting files from an archive with a prepended table of contents, they could be truncated or otherwise altered back to their original size/layout.

    Or maybe I don’t understand the problem well enough. Is it that components of an ELF object are aligned properly assuming the beginning of the file is aligned, but ar would tend to misalign by the size of its file header?

  5. Ian Lance Taylor Avatar
    Ian Lance Taylor

    I hadn’t thought of simply making the object files larger. You’re right; that could be done safely by simply appending a few null bytes. It would cause no harm.

    The ar file header is 60 bytes. Since that is not a multiple of 8, it is a problem on 64-bit systems. Fortunately, in any case that matters the first entry in the archive will be the symbol table, which could be safely extended as the number of symbols is explicitly recorded.

    Thanks for the suggestions!

    (Who extracts files from .a archives? Mostly scripts which combine multiple .a archives into a single one. newlib is an example of a program which does this.)

  6. ncm Avatar
    ncm

    I suppose ranlib could make fake symbol table entries to record the positions of the files.

  7. Ian Lance Taylor Avatar
    Ian Lance Taylor

    I think that would require changing a bunch of existing tools to ignore those entries, though.

  8. ncm Avatar
    ncm

    Do existing tools care about symbols they’re not interested in? No code linking against the archive would make any references to those symbols.

    It might be simpler to add a dummy object file for the map.

  9. Ian Lance Taylor Avatar
    Ian Lance Taylor

    I’m leery of adding new symbols to the symbol table without explicitly ignoring them, because people do so many crazy things.

Leave a Reply