Python memoising/deferred lookup property decorator

Question

Recently I've gone through an existing code base containing many classes where instance attributes reflect values stored in a database. I've refactored a lot of these attributes to have their database lookups be deferred, ie. not be initialised in the constructor but only upon first read. These attributes do not change over the lifetime of the instance, but they're a real bottleneck to calculate that first time and only really accessed for special cases. Hence they can also be cached after they've been retrieved from the database (this therefore fits the definition of memoisation where the input is simply "no input").

I find myself typing the following snippet of code over and over again for various attributes across various classes:

class testA(object):

  def __init__(self):
    self._a = None
    self._b = None

  @property
  def a(self):
    if self._a is None:
      # Calculate the attribute now
      self._a = 7
    return self._a

  @property
  def b(self):
    #etc

Is there an existing decorator to do this already in Python that I'm simply unaware of? Or, is there a reasonably simple way to define a decorator that does this?

I'm working under Python 2.5, but 2.6 answers might still be interesting if they are significantly different.

Note

This question was asked before Python included a lot of ready-made decorators for this. I have updated it only to correct terminology.

Answer 1

Here is an example implementation of a lazy property decorator:

import functools

def lazyprop(fn):
    attr_name = '_lazy_' + fn.__name__

    @property
    @functools.wraps(fn)
    def _lazyprop(self):
        if not hasattr(self, attr_name):
            setattr(self, attr_name, fn(self))
        return getattr(self, attr_name)

    return _lazyprop


class Test(object):

    @lazyprop
    def a(self):
        print 'generating "a"'
        return range(5)

Interactive session:

>>> t = Test()
>>> t.__dict__
{}
>>> t.a
generating "a"
[0, 1, 2, 3, 4]
>>> t.__dict__
{'_lazy_a': [0, 1, 2, 3, 4]}
>>> t.a
[0, 1, 2, 3, 4]

Answer 2

I wrote this one for myself... To be used for true one-time calculated lazy properties. I like it because it avoids sticking extra attributes on objects, and once activated does not waste time checking for attribute presence, etc.:

import functools

class lazy_property(object):
    '''
    meant to be used for lazy evaluation of an object attribute.
    property should represent non-mutable data, as it replaces itself.
    '''

    def __init__(self, fget):
        self.fget = fget

        # copy the getter function's docstring and other attributes
        functools.update_wrapper(self, fget)

    def __get__(self, obj, cls):
        if obj is None:
            return self

        value = self.fget(obj)
        setattr(obj, self.fget.__name__, value)
        return value


class Test(object):

    @lazy_property
    def results(self):
        calcs = 1  # Do a lot of calculation here
        return calcs

Note: The lazy_property class is a non-data descriptor, which means it is read-only. Adding a __set__ method would prevent it from working correctly.

Answer 3

For all sorts of great utilities I'm using boltons.

As part of that library you have cachedproperty:

from boltons.cacheutils import cachedproperty

class Foo(object):
    def __init__(self):
        self.value = 4

    @cachedproperty
    def cached_prop(self):
        self.value += 1
        return self.value


f = Foo()
print(f.value)  # initial value
print(f.cached_prop)  # cached property is calculated
f.value = 1
print(f.cached_prop)  # same value for the cached property - it isn't calculated again
print(f.value)  # the backing value is different (it's essentially unrelated value)

Answer 4

property is a class. A descriptor to be exact. Simply derive from it and implement the desired behavior.

class lazyproperty(property):
   ....

class testA(object):
   ....
  a = lazyproperty('_a')
  b = lazyproperty('_b')

Answer 5

Here's a callable that takes an optional timeout argument, in the __call__ you could also copy over the __name__, __doc__, __module__ from func's namespace:

import time

class Lazyproperty(object):

    def __init__(self, timeout=None):
        self.timeout = timeout
        self._cache = {}

    def __call__(self, func):
        self.func = func
        return self

    def __get__(self, obj, objcls):
        if obj not in self._cache or \
          (self.timeout and time.time() - self._cache[key][1] > self.timeout):
            self._cache[obj] = (self.func(obj), time.time())
        return self._cache[obj]

ex:

class Foo(object):

    @Lazyproperty(10)
    def bar(self):
        print('calculating')
        return 'bar'

>>> x = Foo()
>>> print(x.bar)
calculating
bar
>>> print(x.bar)
bar
...(waiting 10 seconds)...
>>> print(x.bar)
calculating
bar

Answer 6

They added exactly what you're looking for in python 3.8

Transform a method of a class into a property whose value is computed once and then cached as a normal attribute for the life of the instance. Similar to property(), with the addition of caching.

Use it just like @property :

@cached_property
def a(self):
    self._a = 7
    return self._a

Answer 7

What you really want is the reify (source linked!) decorator from Pyramid:

Use as a class method decorator. It operates almost exactly like the Python @property decorator, but it puts the result of the method it decorates into the instance dict after the first call, effectively replacing the function it decorates with an instance variable. It is, in Python parlance, a non-data descriptor. The following is an example and its usage:

>>> from pyramid.decorator import reify

>>> class Foo(object):
...     @reify
...     def jammy(self):
...         print('jammy called')
...         return 1

>>> f = Foo()
>>> v = f.jammy
jammy called
>>> print(v)
1
>>> f.jammy
1
>>> # jammy func not called the second time; it replaced itself with 1
>>> # Note: reassignment is possible
>>> f.jammy = 2
>>> f.jammy
2

Answer 8

There is a mix up of terms and/or confusion of concepts both in question and in answers so far.

Lazy evaluation just means that something is evaluated at runtime at the last possible moment when a value is needed. The standard @property decorator does just that.(*) The decorated function is evaluated only and every time you need the value of that property. (see wikipedia article about lazy evaluation)

(*)Actually a true lazy evaluation (compare e.g. haskell) is very hard to achieve in python (and results in code which is far from idiomatic).

Memoization is the correct term for what the asker seems to be looking for. Pure functions that do not depend on side effects for return value evaluation can be safely memoized and there is actually a decorator in functools @functools.lru_cache so no need for writing own decorators unless you need specialized behavior.

Answer 9

You can do this nice and easily by building a class from Python native property:

class cached_property(property):
    def __init__(self, func, name=None, doc=None):
        self.__name__ = name or func.__name__
        self.__module__ = func.__module__
        self.__doc__ = doc or func.__doc__
        self.func = func

    def __set__(self, obj, value):
        obj.__dict__[self.__name__] = value

    def __get__(self, obj, type=None):
        if obj is None:
            return self
        value = obj.__dict__.get(self.__name__, None)
        if value is None:
            value = self.func(obj)
            obj.__dict__[self.__name__] = value
        return value

We can use this property class like regular class property ( It's also support item assignment as you can see)

class SampleClass():
    @cached_property
    def cached_property(self):
        print('I am calculating value')
        return 'My calculated value'


c = SampleClass()
print(c.cached_property)
print(c.cached_property)
c.cached_property = 2
print(c.cached_property)
print(c.cached_property)

Value only calculated first time and after that we used our saved value

Output:

I am calculating value
My calculated value
My calculated value
2
2

Answer 10

I agree with @jason When I think about lazy evaluation, Asyncio immediately comes to mind. The possibility of delaying the expensive calculation till the last minute is the sole benefit of lazy evaluation.

Caching / memozition on the other hand could be beneficial but on the expense that the calculation is static and won't change with time / inputs.

A practice I often do for expensive calculations of these sorts is to calculate then cache with TTL.