I am amazed by the way APIs of some machine learning packages are designed, namely Chainer's and Pytorch's "define-by-run" API. Even Optuna, a hyper parameter tuning library has "define-by-run" API.
How to achieve such design of API? Can anyone guide me towards it with some code?
Define-by-run is just a neat application of symbolic programming. Python does not support symbolic programming the way that Lisp, Mathematica, or R do. But Python can emulate it with operator overloading.
The idea is that an expression x + y will not compute the addition of two variables, but produce an object that represents the addition of two variables. We can later manipulate this object as a data structure. In the context of ANN, it is particularly important that we can perform back-propagation.
As a simpler example, let's consider such a symbolic system that supports partial derivatives. First, we need a base class for symbolic operations:
class Symbolic:
def eval(self, bindings: Dict[str, float]) -> float:
raise NotImplementedError
def derivative(self, x: Variable) -> Symbolic:
raise NotImplementedError
def __add__(self, other) -> Symbolic:
return Addition.of(self, symbolify(other))
def __radd__(self, other) -> Symbolic:
return Addition.of(symbolify(other), self)
def __mul__(self, other) -> Symbolic:
return Multiplication.of(self, symbolify(other))
def __rmul__(self, other) -> Symbolic:
return Multiplication.of(symbolify(other), self)
def symbolify(x) -> Symbolic:
if isinstance(x, Symbolic):
return x
if isinstance(x, (float, int)):
return Constant(float(x))
raise TypeError
We can create variables that are not yet assigned a value:
class Variable(Symbolic):
def __init__(self, name: str) -> None:
self.name = name
def __repr__(self) -> str:
return self.name
def eval(self, bindings: Dict[str, float]) -> float:
return bindings[self.name]
def derivative(self, x: Variable) -> Symbolic:
if self.name == x.name:
return Constant(1)
else:
return Constant(0)
We can create constants with fixed values:
class Constant(Symbolic):
def __init__(self, value: float) -> None:
self.value = value
def eval(self, bindings) -> float:
return self.value
def __repr__(self) -> str:
return str(self.value)
def derivative(self, x: Variable) -> Symbolic:
return Constant(0)
And we can create a symbolic object representing addition and multiplication, with some rules for simplification.
class Addition(Symbolic):
def __init__(self, left: Symbolic, right: Symbolic) -> None:
self.left = left
self.right = right
def __repr__(self) -> str:
return f"({self.left!r} + {self.right!r})"
@staticmethod
def of(left: Symbolic, right: Symbolic) -> Symbolic:
if isinstance(left, Constant) and left.value == 0:
return right
if isinstance(right, Constant) and right.value == 0:
return left
return Addition(left, right)
def eval(self, bindings) -> float:
return self.left.eval(bindings) + self.right.eval(bindings)
def derivative(self, x: Variable) -> Symbolic:
return self.left.derivative(x) + self.right.derivative(x)
class Multiplication(Symbolic):
def __init__(self, left: Symbolic, right: Symbolic) -> None:
self.left = left
self.right = right
def __repr__(self) -> str:
return f"({self.left!r} * {self.right!r})"
@staticmethod
def of(left: Symbolic, right: Symbolic) -> Symbolic:
if isinstance(left, Constant):
if left.value == 0:
return left
elif left.value == 1:
return right
if isinstance(right, Constant):
if right.value == 0:
return right
elif right.value == 1:
return left
return Multiplication(left, right)
def eval(self, bindings) -> float:
return self.left.eval(bindings) * self.right.eval(bindings)
def derivative(self, x: Variable) -> Symbolic:
# product rule
return self.left.derivative(x) * self.right \
+ self.left * self.right.derivative(x)
We now have sufficient infrastructure to define and manipulate a symbolic expression:
>>> x = Variable('x')
>>> y = Variable('y')
>>> expr = x + 2 * y
>>> expr
(x + (2.0 * y))
>>> expr.eval(dict(x=3, y=5))
13.0
>>> dy = expr.derivative(y)
>>> dy
2.0
>>> (x * y).derivative(x)
y
So this isn't magic, just lots of tedious code. There are limitations of this approach, e.g. such symbolic expressions don't work well with control flow: you can't assign a true or false value to these expressions until they are evaluated.
You mention Optuna which doesn't have this limitation, since it doesn't overload Python expressions as Chainer or Numpy would do. Instead, the programmer explicitly requests values from a context which can then be stored:
def objective(trial):
x = trial.suggest_uniform('x', 0, 10)
y = trial.suggest_uniform('y', -10, 10)
# x and y are normal Python numbers
if y > x:
return x + 2 * y
else:
return 3 + x * y
study = optuna.create_study()
study.optimize(objective)
After the objective is executed, the trial contains a black-box view onto the result, for example:
trial_input = {
'x': 3.0,
'y': 5.0,
}
trial_output = {
'result': 13.0,
}
The optimizer does not know about the algebraic relationship between the various values, but can build a black-box model about their interactions. After many iterations, the optimizer might “learn” about the correct interactions and will thus be able to find the optimal values for x and y.
