Intermediate Python: Dunder-Methods, Decorators, and Context Managers

From Wikipedia:

In computer programming, operator overloading, sometimes termed operator ad hoc polymorphism, is a specific case of polymorphism, where different operators have different implementations depending on their arguments

More specifically to Python, operator overloading allows the programmer to hook custom classes into fundamental operators of the language like “+” or “==”. Some specific examples of this are accessing NumPy arrays using slice notation (via the dunder method __getitem__), or the fact that string equality means an element-wise comparison of the code points in a unicode string while object equality means comparison of each object’s id() (via the dunder method __eq__). Generally speaking, operator overloading in Python is performed by implementing dunder methods for a class, which define how the class interacts with various operators.

Dunder methods are a powerful way to make certain operations more “pythonic”; however, they should be used with care. A warning from the Python data model:

When implementing a class that emulates any built-in type, it is important that the emulation only be implemented to the degree that it makes sense for the object being modelled. For example, some sequences may work well with retrieval of individual elements, but extracting a slice may not make sense.

A good rule of thumb for when to use operator overloading: someone who does not have access to the class definition should be able to guess what is happening when they see the class used with an operator. If it is not clear what “+” means for class MyAvatar, then it is probably not a good use of __add__. However, it may be obvious what “+” means for class ComplexNumber. Operator overloading should always be used to make operations simpler to understand, not just more concise.

(As a side note, don’t forget that Python has built-in support for complex numbers, so there’s no need to write a class ComplexNumber!)

dir, when called with an object as its argument, “returns an alphabetized list of names comprising (some of) the attributes of the given objects, and of attributes reachble from it.” A more in-depth description of dir can be found in the function’s docstring, accessible via dir.__doc__.

• __add__: Implements the behavior of this object for the + operator.
• __class__: Returns the class of which this object is an instance.
• __contains__: Implements the boolean check somevar in self (i.e. is some value contained in the collection)
• __delattr__: Implements the behavior for the delattr() function, which as the name implies, deletes an attribute from an object.
• __delitem__: Implements behavior for del self[x], which deletes the item at position x within self.
• __dir__: Describes what is listed out when dir() is called. Note that we used dir to get this list of dunder methods in the first place; objects can override what is shown to dir, so some attributes may not be listed by using this method.
• __doc__: Returns the docstring for the object.
• __eq__: Implements the boolean check self == some_other.
• __format__:
• __ge__: Implements “greater than or equal” comparison
• __getattribute__: Implements behavior when getattribute(self, 'someattribute') is called
• __getitem__: Implements indexing behavior, i.e. self[x].
• __gt__: Implementa “greater than” comparison.
• __hash__: Returns a unique hash of the object. Some objects cannot, by nature, have a unique hash that does not change through the lifetime of the object. list is one of these; you’ll notice that [].__hash__ is actually None.
• __iadd__: Implements +=, which performs in-place addition.
• __imul__ Implements *=, in-place multiplication
• __init__: The method called when an object is first created.
• __init_subclass__: A method called when a class is subclassed.
• __iter__: Implements iterator behavior for self, allowing the object to be used in for loops.
• __le__: Implementation of “less than or equal”
• __len__: Returns the object length when accessed via len(self).
• __lt__: Implements “less than”
• __mul__: Implements multiplication *.
• __ne__: Implements the not-equal check self != someval. Usually can just be the logical not of __eq__.
• __new__: A class method called when creating a new instance of that class.
• __repr__: Returns the official string representation of the object (i.e. for printing with print(self))
• __reversed__: Returns a reverse iterator.
• __rmul__: “Reflected” multiplication, where the right-hand operator is first. This is a fallback method called if the left-hand operator cannot perform normal multiplication.
• __setattr__: Implements behavior of setattr(self, 'attr_name', 'attr_value')
• __setitem__: Implements assigning to an index (i.e. self[x] = 5)
• __sizeof__: Specific to lists; returns the size of the list in memory (bytes).
• __str__: Returns a string form of the object when called as str(self).
• __subclasshook__: A method that can be used to alter the result of issubclass(self, SomeClass).

Dunder methods allow customization of essentially every aspect of an object in Python. They should be used carefully in order to preserve readability and minimize unnecessary complexity.

For print #1: [1, 2, 8, 88]

For print #2: <a StopIteration error being raised>

This is because spreading the generator’s iterations to an array expends all of the yield keywords. When the iterator tries to find the next yield in order to return its value and can’t, the error is thrown.

The context manager allows the explicit opening and closing of external resources. This is particularly important in Python since the timing of garbage collection of a reference is implementation-specific and not guaranteed. From the Python data model:

Some objects contain references to “external” resources such as open files or windows. It is understood that these resources are freed when the object is garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a close() method. Programs are strongly recommended to explicitly close such objects. The ‘try…finally’ statement and the ‘with’ statement provide convenient ways to do this.

Context managers are also more generally useful for any kind of operation that requires defined setup and teardown steps.

A context manager can be created by defining a class with __enter__ and __exit__ methods. Alternatively, it can be created with a generator function decorated with the @contextmanager decorator. The latter approach is usually preferred since it is visually simpler to understand.

Here’s how you would implement the context manager using a class:

class TemporaryOverride:
    def __init__(self, key, new_value):
        self.key = key
        self.new_value = new_value
        self.original_value = None

    def __enter__(self):
        self.original_value = GLOBAL_CONFIG[self.key]
        GLOBAL_CONFIG[self.key] = self.new_value

    def __exit__(self, exc_type, exc_value, traceback):
        GLOBAL_CONFIG[self.key] = self.original_value

Here’s how you would do it using the contextlib.contextmanager decorator:

from contextlib import contextmanager

@contextmanager
def temporary_override(key, new_value):
    old_value = GLOBAL_CONFIG[key]
    GLOBAL_CONFIG[key] = new_value
    yield
    GLOBAL_CONFIG[key] = old_value

Here’s the updated context manager that uses a class:

class TemporaryOverride:
    def __init__(self, key, new_value):
        self.key = key
        self.new_value = new_value
        self.original_value = None

    def __enter__(self):
        self.original_value = GLOBAL_CONFIG[self.key]
        GLOBAL_CONFIG[self.key] = self.new_value

    def __exit__(self, exc_type, exc_value, traceback):
        GLOBAL_CONFIG[self.key] = self.original_value

Here’s the updated version that uses contextlib.contextmanager:

@contextmanager
def temporary_override(key, new_value):
    old_value = GLOBAL_CONFIG[key]
    GLOBAL_CONFIG[key] = new_value
    try:
        yield
    finally:
        GLOBAL_CONFIG[key] = old_value

def context_manager(funk):

def wrapper(*args, **kwargs):
    class ContextManager:
        def __init__(self, func, *args, **kwargs):
            self.func = func(*args, **kwargs)

        def __enter__(self):
            next(self.func)

        def __exit__(self, exc_type, exc_value, traceback):
            next(self.func)

    return ContextManager(funk, *args, **kwargs)

return wrapper

The actual Python source code is here. Our context_manager takes in the function as a parameter. Next it defines a wrapper function and declare that it is allowed to take in any number of arguments and keyword arguments. This gives us access to the arguments passed to funk. We then define our boilerplate context manager, per usual, with some minor tweaks. Upon initialization it will take in a function, it will also take in an unspecified amount of arguments and key arguments as well for the same reason as the Wrapper function. We will declare a key within our context management objects to save the value of the invoked function that was passed as a parameter; we also pass these unspecified arguments and keyword arguments into the function. Now we see and can get a better understanding as to why our context manager needs to be passed a generator function And why is that generator should be set up in a particular way. When the __enter__ m-method is invoked, we will iterate our generator to the yield. We then iterate our generator once more when the __exit__ dunder-method is invoked To run the code after the yield, our “closing code”, if you will.