Trade-off between usability and clean design

Question

I have designed our in-house hardware testing framework. My goal is to ultimately release the framework into the public domain. Early on, my foremost design criterion was to provide a powerful yet easy-to-use test API for test authors. But I didn't want to neglect the test operators either (since they are normally not the same people as those writing the tests).

This hardware testing framework is therefore two things for two different types of users:

• For test authors it is a Python package that provides basic building blocks such as Test and TestSuite classes along with several abstraction layers for physically interacting with the device under test and for inserting records into a remote database.
• For test operators it is a standalone Python script ("installed" via setuptools) acting as a runner for a test suite written by a test author. It literally compile()s a Python script, then exec()s the resulting code.

In other words, test authors write my_suite.py as follows:

from ht_framework import Test, TestSuite, ...

test1 = Test(...)
test2 = Test(...)
suite = TestSuite(test1, test2, ...)
suite.execute()

While test operators launch my_suite.py as follows:

ht_framework my_suite.py --quiet --some_arg some_value

The ht_framework standalone script, apart from executing test suites, also controls things specific to the tasks performed by a test operator. For instance, it asks the operator whether a single failing test should be repeated (if the operator forgot to push a button at the right moment for instance) or whether the whole suite should be repeated (when the operator replaces the device under test with a new one of the same type).

My question

So far so good. Here is the part that I'm second-guessing: since I didn't want test authors to be bothered by internal design intricacies, I wanted them to be able to simply write my_suite.py as a normal Python script. But because the ht_framework standalone compiles and executes the code in my_suite.py, the test suite is like a black box into which I have to painfully inject configuration parameters such as verbosity, paths to firmware binaries or other command-line arguments.

What is your take on this? Do you think...

• ...that the whole configuration should done via a settings file, with no command-line arguments accepted whatsoever? (I did consider this, but sometimes you really don't feel like editing a file just to set verbosity on/off. The framework started off like this actually, when there wasn't a ht_framework standalone.)
• ...that the design should force test authors to provide a rigid configuration upfront? (Almost as above, no standalone and more stuff for test authors to write.)
• ...there is a way to "poke" through the Python code generated by compile() (via AST maybe?) and sprinkle some if/else statements here and there?
• ...about something not listed above? :)

Thanks in advance for your input.