1

I'm trying to iterate a bunch of data ranges at once, to get the combinations of all their values.

The number of ranges can differ, but I have them collected in a list.

Is there a way to iterate them using list comprehension or a similar clean, pythonic way?

This is what I mean by iterating together:

[print(i, j) for i in r1 for j in r2]

So that's a simple example with two known ranges, but what I need is more like

[print(i, j, ...) for i in r1 for j in r2 for k in r3...]

Note: i don't just need a list of number combinations, the iterators are my own iterator class which works similarly to range() but allows me to also get current state without calling next(), which would alter the state.

My iterator class sets its value back to the start on StopIteration, so it can be looped through more than once.

Here you can see the class:

@dataclass
class Range:
    start: float
    end: float
    step: float = field(default=1)
    includeEnd: bool = field(default=True)
    
    def __post_init__(self):
        self.value = self.start
        
    def __next__(self):
        v = self.value
        end = v > self.end if self.includeEnd else v >= self.end
        if not end:
            self.value += self.step
            return v
        else:
            self.value = self.start
            raise StopIteration

    def __iter__(self):
        return self
10
  • 4
    You're looking for itertools.product().
    – Tim Peters
    Commented Jul 2, 2021 at 2:58
  • 1
    If "r2" is an iterator (but not an iterable) it will be exhausted after the first "i" (outer for-iteration) and won't produce anything for the next "i". Commented Jul 2, 2021 at 2:58
  • @Julien I didn't see where he said combinations, I thought he wanted parallel.
    – Barmar
    Commented Jul 2, 2021 at 3:00
  • As an aside, range is not an iterator. Commented Jul 2, 2021 at 3:26
  • 1
    You still want itertools.product() ;-) It doesn't care whether your pass it iterators or iterables. It materializes each argument's sequence once at the start, into internal tuples, then iterates as many times as needed across those internal tuples.
    – Tim Peters
    Commented Jul 2, 2021 at 3:39

1 Answer 1

2

But how would you get the product of n iterators using itertools.product(), when you have a list of n iterators?

itertools.product(*the_list). Nothing special about product() there. The leading * is general Python syntax for treating a list (more generally, an iterable) as a sequence of individual arguments.

>>> from itertools import product
>>> args = [range(2), range(3), (i**2 for i in [5, 9])]
>>> args
[range(0, 2), range(0, 3), <generator object <genexpr> at 0x000001E2E7A710B0>]
>>> for x in product(*args):
...     print(x)
    
(0, 0, 25)
(0, 0, 81)
(0, 1, 25)
(0, 1, 81)
(0, 2, 25)
(0, 2, 81)
(1, 0, 25)
(1, 0, 81)
(1, 1, 25)
(1, 1, 81)
(1, 2, 25)
(1, 2, 81)
4
  • This almost works, the only problem is that itertools.product seems to iterate on a copy of my iterator objects, whereas I'd like the objects themselves to have their current value modified. Is there any way to fix that? Commented Jul 2, 2021 at 8:13
  • Temp solution: assigning the values back to the ranges as I iterate. pastebin.com/winMNr6x Commented Jul 2, 2021 at 8:31
  • No, as already said in a comment, product() deliberately constructs temporary tuples at the start and iterates over them thereafter. There is no support anywhere in Python for distinguishing between iterables that can, and cannot, be restarted. As in the example I gave, generator expressions cannot be - they're forever dead after iterating over them once. That's why product() iterates over each argument only once at the very start, and iterates over those copies instad.
    – Tim Peters
    Commented Jul 2, 2021 at 16:34
  • BTW, you can get at the internal tuples product() uses, and the current index into each, by calling .__reduce__() on the iterator product() returns. But that's an internal implementation detail (used for pickling), so you would use it at your own risk. In practice, though, the __reduce__() implementation here is unlikely to change.
    – Tim Peters
    Commented Jul 2, 2021 at 20:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.