Response files for your Python scripts

By | May 6, 2011

Many moons ago I started working on a toolchain for my home projects. Since I had seen some really elegant things done with Opus Make back in the day, I decided to go old-skool and write my tools as command-line utilities. However, owing to the fact that it isn’t back-in-the-day, I also decided to use Python to get the job done.

Now Python, in addition to being just generally awesome, really works well for command-line tools. So well, in fact, that it has no less than three classes in various stages of deprecation to handle the command-line arguments. But one thing I couldn’t figure out how to get any of them to do was accept a response file.

If you didn’t grow up in a cave and you’re wondering WTF a response file is, it’s simply a text file that contains command-line options that you would pass to a tool in lieu of specifying all the options by hand. They are an outgrowth of a day when the shell in your OS could only feed so many characters to your *argv[], but at the same time offer a really slick way to data-drive your command-line tools. In fact, these things still exist in the wild. Check out the docs for MSBuild for a little Microsoft response file action.

But back to the topic.

A tool that accepts response files will generally take the file with an @ in front of it on the command line just like it says in the Microsoft link. They also have been known to allow comments in some fashion. So we’ll be jotting down a series of like requirements for what our command-line tools should do to accept response files.

  1. A tool should look for the @ symbol when parsing its options.
  2. Treat the string following the @ as a filename and attempt to open it.
  3. Read the file and insert its guts it back into the argument steam, in order.
  4. Ignore lines commented out with a # token at the start.
  5. Support recursive response files for better modularization.

Fortunately, those are easy requirements to satisfy using standard Python stuff. Let’s look at the code for a function that will generate an argument list with the data from a response file. (This is also a carefully crafted opportunity for me to make sure that my syntax highlighter actually installed properly.)

import shlex

def ParseWithResponseFile(args):
    commands = []

    for arg in args:
        if arg[0] == '@':
                inFile = open(arg[1:])

                for line in inFile.readlines():
                    tokens = shlex.split(line, True)

            except IOError as desc:

    return commands

So this function meets all of our requirements. The only clever part about this is using shlex to tokenize each line read from the file. The advantage of this is that we get free support for a comment character (our # in this case, but it can be set to others). It also extends the command list recursively with the parsed lines of the response file.

We can now use the function something like this:

parser = argparse.ArgumentParser()    # Add stuff to constructor

# Fill in parser options and settings and junk...

(options, args) = parser.parse_args(ParseWithResponseFile(sys.argv[1:]))

All that leaves is for us to nifty up a response file for our new tools:

# ImgConvert_PS3.rsp -- PS3 specific args for converting textures

--compression DXT5
--maxsize 1024
--filter quadratic

And there you have it.

(P.S. — I realize that the recursion presents a potential “10 GOTO 10” problem. If you need a higher level of user protection just rewrite the recursion and throw an error if the same response file is used more than once. This isn’t a problem for me.)

Leave Your Comment

Your email will not be published or shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>