Iterators in Python

Let's get practical. Imagine that in an interview you were asked to implement an analog of the range function. Let's try to write one quickly.

PythonCopy

class Range:
    def __init__(self, stop_value: int):
        self.current = -1
        self.stop_value = stop_value - 1

    def __iter__(self):
        return RangeIterator(self)

class RangeIterator:
    def __init__(self, container):
        self.container = container

    def __next__(self):
        if self.container.current < self.container.stop_value:
            self.container.current += 1
            return self.container.current
        raise StopIteration

You now have a first working version of the code. Let's run it to make sure.

PythonCopy

_range = Range(5)
for i in _range:
    print(i)

As a result you get the numbers from 0 to 4 on separate lines — all good. Right away comes the question: «So how does it work? Tell us in more detail». Many people fall through on this question because they don't know how to answer it. To avoid failing the interview, let's dig deeper into how iterators are actually built in Python.

How iterators work

Let's start with the range class. Inside it implements the magic method __iter__. It marks the object of this class as iterable, that is, you can work with it in a for loop. It's also said that __iter__ returns an iterable object.

For the code to actually hand out new data from range, you need to implement a corresponding function. It is precisely called an iterator. RangeIterator is the iterator for the range class. Any iterator implements the magic function __next__, in which it must hand out new values for the objects of the range class. If you've reached the end of the set of values, the StopIteration exception is raised.

But can the code above be simplified somehow? Yes, it can.

PythonCopy

class Range2:
    def __init__(self, stop_value: int):
        self.current = -1
        self.stop_value = stop_value - 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.current < self.stop_value:
            self.current += 1
            return self.current
        raise StopIteration

In Python you can declare objects of a class to be both iterable and iterators. This is convenient, but from the standpoint of application design principles such an object has two traits: it is an iterator and at the same time performs some logic of its own. In the Python world this is acceptable, but in some other languages you may be misunderstood. Stay vigilant!

The for loop under the hood

It's also worth looking at how the for loop works under the hood.

PythonCopy

iterable = Range2(5)
iterator = iterable.__iter__()
while True:
    try:
        value = iterator.__next__()
        print(value)
    except StopIteration:
        break

Or a bit simpler.

PythonCopy

iterable = Range2(5)
iterator = iter(iterable)
while True:
    try:
        value = next(iterator)
        print(value)
    except StopIteration:
        break

Many built-in standard functions in Python are iterators. You've already met the range iterator earlier. Let's look at other options you may come across while working with code.

Enumerate

The function creates and adds a sequential number to the elements of the passed collection. The returned result is a tuple of the index and the collection element.

PythonCopy

my_list = ["a", "b", "c", "d", "e"]
enum_list = enumerate(my_list)
next(enum_list)
next(enum_list)

# >>> (0, "a")
# >>> (1, "b")

for idx, val in enumerate(my_list):
    print(idx, val)

# output
# >>> 0 "a"
# >>> 1 "b"
# >>> 2 "c"
# >>> 3 "d"
# >>> 4 "e"

Zip

Creates a tuple from the passed pairs of values.

PythonCopy

values = [1, 2, 3, 4, 5]
keys = ['a', 'b', 'c', 'd', 'e']
matches = zip(keys, values)
next(matches)

# output
# >>> ('a', 1)

Open

The function opens a file descriptor for reading line by line.

PythonCopy

file_data = open("my_file.txt")
next(file_data)

# >>> 'bar\n'

next(file_data)

# >>> 'baz\n'

There are a great many iterators in Python. They defer doing the work until the next element is requested with next. The program block runs in so-called «lazy» mode.