Python Tricks: A Buffet of Awesome Python Features

Book cover

No need to know everything

Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up.

Patterns for Cleaner Python

Avoid currency rounding issues by using an int and set price in cents

Assert Statement

Internal self-checks for your program
proper use of assertions is to inform developers about unrecoverable errors in program
they are not intended to signal expected error (FileNotFound) from which a user can recover
excellent debugging tool - if you reach assertion, you have a bug

Common Pitfalls

security risks
possibility to write useless assertions

1. Don’t use asserts for data validation

assert can be globally disabled (-0 -00 commands)
validate data with regular if-else statements and raise exceptions on errors

2. Asserts that never fail

passing a tuple as first argument always evaluates to true - remember truthy values
this will never fail:

1

assert(1 == 2, 'This should fail.')

Always make sure your tests can actually fail

Comma Placement

list/dict/set - end all of your lines with a comma

1

names = ['Peter', 'Alice', 'John',]

splitting it into new lines is an excellent method for seeing what was changed (git)

1
2
3
4
5


names = [
		 'Peter',
		 'Alice',
		 'John',
]

leaving a comma avoids strings from getting merged

Context Managers & the with statement

with statement - write cleaner and more readable code
simplify common resource management patterns
encapsulates standard try/finally statements

Context Managers

protocol, that your objects needs to follow in order to support the with statement
add __enter__ and __exit__ methods to an object if you want it to function as a context manager - these methods will be called in resource management cycle

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class ManagedFile:
	def __init__(self, name):
		self.name = name

	def __enter__(self):
		self.file = open(self.name, 'w')
		return self.file

	def __exit__(self, exc_type, exc_val, exc_tb):
		if self.file:
			self.file.close()

now we can use

1
2
3


with ManagedFile('hello.txt') as f:
	f.write('hello, word')
	f.write('bye now')

you can also use @contextmanager decorator for this purpose

Underscores, Dunders & More

1
2
3
4
5
6


# These are conventions, Python Interpreter doesn't check them
_var # var meant for internal use 
var_ # next most fitting var name (first is taken) - class_
__var # Here Interpreter actualle steps in (name mangling) - it rewrites attribute name to avoid naming conflicts in subclasses
__var__ # Interpreter doesn't care - these are used for special/magic methods
_ # temporary or insignificant variable (for _ in range(10)...)

_ is also good for unpacking when there are values in the middle for which you don’t care

1
2


car = ('red', 'auto', 12, 3812.4)
color, _, _, mileage = car

_ is also handy when constructing objects on the fly and assigning values without naming them first

1
2
3
4
5
6


>>> list()
>>> _.append(1)
>>> _.append(2)
>>> _.append(3)
>>> _
[1, 2, 3]

Truth About String Formatting

Python offers 4-ways to format strings

1
2
3
4


errno = 50159747054
name = 'Bob'
# We want to output:
'Hey Bob, there is a 0xbadc.ffee error!'

Old-style formatting:

'Hello, %s' % name - %s replace this as string; '%x' % errno - %x as int

New-style formatting

format() function

1
2


>>> 'Hello, {}'.format(name)
'Hello, Bob'

Literal String Interpolation

f-Strings f'Hello, {name}!'
supports also expressions

Template Strings

simpler and less powerful, but in some cases very useful
useful when it comes to safety

1
2
3
4


from string import Template
t = Template('Hey, $name!')
t.substitute(name=name)
'Hey, Bob!'

When to use which method?

If your format strings are user-supplied, use Template Strings to avoid security issues. Otherwise, use f-Strings.

Effective Functions

Python’s Functions are First-Class

First-class objects: ability to assign them t variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions.

1
2


def yell(text):
	return text.upper() + '!'

Functions are Objects

all data in Python is represented by objects
because a function is an object, you can assign it to a variable
bark = yell
bark now points to the original yell function - bark('woof')
function objects and their names are two separate concerns
you can del yell - Name is not defined, but bark still works
bark.__name__ ‘yell’
name identifier is a debugging aid - A variable pointing to a function and the function itself are really two separate concerns

Functions can be stored in Data Structures

1

funcs = [bark, str.lower, str.capitalize]

Functions can be passed to other Functions

1
2
3


def greet(func):
	greeting = func('Hi, I am a Python program')
	print(greeting)

You can influence the resulting greeting by passing in different functions.

1
2
3
4


>>> greet(bark)
>>> 'HI, I AM A PYTHON PROGRAM!'
>>> greet(whisper)
>>> 'hi, i am a python program...'

The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around behavior in your programs.

Higher-order functions - functions that accept other functions as arguments - key principle in Functional programming style

Classical example of higher-order function in Python - map()

1
2


>>> list(map(bark, ['hello', 'hey', 'hi'])) # apply bark() to each item
>>> ['HELLO!', 'HEY!', 'HI!']

Functions can be Nested

Python allows functions to be defined inside other functions
also known as nested functions / inner functions

1
2
3
4
5
6
7


def speak(text):
	def whisper(t):
		return t.lower() + '...'
	return whisper

>>> speak('Hello, World')
>>> 'hello, word...'

Everytime you call speak, it defines a new inner function whisper and then calls it immediately after.

whisper does not exist outside speak!

Functions can not only accept behaviors through arguments, but they can also return behaviors.

Functions can capture Local State

it gets even crazier
inner functions can also capture and carry some of the parent function’s state
closure - remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope

Objects can behave like Functions

all functions are objects, but not all objects are functions
you can make objects callable - meaning, you can use () on them
__call__ method

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


class Adder:
	def __init__(self, n):
		self.n = n

	def __call__(self, x):
		return self.n + x

>>> plus_3 = Adder(3)
>>> plus_3(4)
>>> 7

not all objects will be callable - you can check with built-in function callable()

Lambdas are Single-Expression Functions

lambda - declare small anonymous function
lambda functions behave like regular function defined with def

1
2
3


>>> add = lambda x, y: x + y
>>> add(5, 3)
8

Function expression

1
2


>>> (lambda x, y: x + y)(5, 3)
8

you don’t have to bind a function to a name
you define it and call it immediately
Lambda functions are restricted to single expression - no statements, annotations, no return statement
there is implicit return statement - automatic
just like nested functions, lambdas also work as lexical closures - function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that scope

When to use Lambda functions

they should be used sparingly and with care
use them when you are expected to supply a function object

1

>>> sorted(my_list, key=lambda x: x[1])

it looks cool, but readability is a critical aspect of well-designed code

The power of Decorators

Decorator - extend & modify the behavior of a callable without permanently modifying the callable itself

How are they useful? Example:
- you have 30 functions, something needs to change very quickly (short deadline) - no need to rewrite them all
- you need to add input/output logging to them
- ideal use case for a decorator
- understanding decorators here will be difference between high blood-pressure and staying (relatively) calm :)
Very useful when you need to extend function with some generic functionality, such as:
- logging
- authentication
- instrumentation/timing functions
- rate-limiting
- catching, and more

Decorator basics

decorate/wrap another function and execute code before and after the wrapped function runs

Decorator is a callable that takes a callable as input and returns another callable

1
2
3


# The most basic example of a decorator
def null_decorator(func): # takes in function
	return func # returns a function

using @decorator and what happens in the background:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


def greet():
	return 'Hello!'

greet = null_decorator(greet)
>>> greet()
'Hello!'

@null_decorator
def greet():
	return 'Hello!'

using @null_decorator in front is the same as defining the function first and then running it through the decorator
@ syntax decorates the function immediately at definition time - this makes it hard to access the undecorated function, that’s why sometimes you might prefer to decorate the function manually
Let’s take a look at more useful decorator, one that converts result to upper case

1
2
3
4
5
6


def uppercase(func):
	def wrapper():
		original_result = func()
		modified_result = original_result.upper()
		return modified_result
	return wrapper

Note how, up until now, the decorated function has never been executed. Actually calling the input function at this point wouldn’t make any sense - you’ll want the decorator to be able to modify the behavior of its input function when it eventually gets called

1
2
3
4
5
6


@uppercase
def greet():
	return 'Hello!'

>>> greet()
'HELLO!'

our decorator returns a different function object when it decorates a function

1
2
3
4
5
6
7
8


>>> greet
<function greet at 0x10e9f0950>

>>> null_decorator(greet)
<function greet at 0x10e9f0950> # The same object

>>> uppercase(greet)
<function uppercase.<locals>.wrapper at 0x76da02f28> # Different object

decorators modify the behavior of a callable through a wrapper closure, so you don’t have to permanently modify the original

Applying multiple decorators to a function

you can apply more than 1 decorator to a function
decorators are applied from bottom to top

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


def strong(func):
	def wrapper():
		return '<strong>' + func() + '</strong>'
	return wrapper

def emphasis(func):
	def wrapper():
		return '<em>' + func() + '</em>'
	return wrapper

@strong # Second
@emphasis # This will be used first
def greet():
	return 'Hello!'

>>> greet()
'<strong><em>Hello!</em></strong>'

# In the background the chain looks like this
decorated_greet = strong(emphasis(greet))

deep levels of stacking may affect performance

Decorating Functions that Accept Arguments

*args, **kwargs come handy here

1
2
3
4


def proxy(func):
	def wrapper(*args, **kwargs): 
		return func(*args, **kwargs) # This is unpacking - wrapper unpacks the args, kwargs into  original function
	return wrapper

Debugging Decorators

it is challenging to debug decorators
Python brings in functools.wraps decorator included in Standard library for this purpose

*Args & **Kwargs

very useful feature in Python
they allow a function to accept optional arguments - you can create flexible APIs
using * & ** is required, naming is convention

1
2
3
4
5
6


def foo(required, *args, **kwargs):
	print(required)
	if args: # These get collected as a tuple because of '*' prefix
		print(args)
	if kwargs: # These get collected as a dict because of '**' prefix
		print(kwargs)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


>>> foo()
TypeError:
"foo() missing 1 required positional arg: 'required'"
>>> foo('hello')
'hello'
>>> foo('hello', 1, 2, 3)
'hello'
(1, 2, 3)
>>> foo('hello', 1, 2, 3, key1='value', key2=999)
'hello'
(1, 2, 3)
{'key1': 'value', 'key2': 999}

you can pass args, kwargs into other functions with unpacking
you can also modify them before passing them along

1
2
3
4


def foo(x, *args, **kwargs):
	kwargs['name'] = 'Alice'
	new_args = args + ('extra', )
	bar (x, *new_args, **kwargs)

this is useful for subclassing and writing wrapper functions (decorators)
typically though, you would use this when overriding behavior in some external class which you don’t control

Function Argument Unpacking

1
2
3
4
5
6
7


vectors = [1, 0, 1]

def print_vectors(x, y, z):
	print(f'<{x}, {y}, {z}>')

print_vectors(*vectors) # unpack list with *
<1, 0, 1>

1
2
3


vectors_dict = {'y': 0, 'z': 1, 'x': 1}
print_vectors(**vectors_dict) # unpack dictionary with ** (values); * for keys
<1, 0, 1>

Nothing to Return Here

Python adds an implicit return None statement to the end of any function - no need to explicitly mention it

1
2
3
4


def foo(value):
	if value:
		return value
	# No need to return None otherwise - that's default

When to use this future?
- function doesn’t return anything? Leave out return statement
- it comes down to readability

Classes & Object-Oriented Programming

Object comparisons

‘is’ vs ‘==’
== checks for equality
is checks for identity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


a = [1, 2, 3]
b = a

>>> a
[1, 2, 3]
>>> b
[1, 2, 3]
>>> a == b # They do look the same
True
>>> a is b # And they also point to the same object
True

# Let's create a copy
c = list(a)
>>> c
[1, 2, 3]
>>> a == c # They still look identical
True
>>> a is c # But they do not point to the same object anymore
False

An is expression evaluates to True if two variables point to the same (identical) object.

An == expression evaluates to True if the objects referred to by the variables are equal (have the same contents)

Every class needs a repr, str methods

these methods convert objects to strings

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Car:

	def __init__(self, color, mileage):
		self.color = color
		self.mileage = mileage
		
	def __str__(self):
		return f'STR: a {self.color} car' # Print out instance now returns this string

	def __repr__(self):
		return f'REPR: a {self.color} car' # Calling instance uses repr  method

>>> bmw = Car('blue', 2000)
>>> print(bmw)
'STR: a blue car' # From __str__
>>> bmw
'REPR: a blue car' # From __repr__

main differences:
- repr should be intended for developers - with debugging info
- str should be purely readable representation of object - for user
always add at least __repr__ method to a class

Custom Exception Classes

Excellent way to write more readable & maintainable code

1
2
3
4


# validate an input string
def validate(name):
	if len(name) < 10:
		raise ValueError # Works, but this is way too generic - doesn't say much

Create custom error for name validation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


class NameTooShortError(ValueError): # Inherit from root Exception class or specific exceptions
	pass

def validate(name):
	if len(name) < 10:
		raise NameTooShortError(name) # Now we clearly know what went wrong

>>> validate('tom')
...
NameTooShortError: tom

When publicly releasing a package, or creating a reusable module - good practice is to create a custom exception base class and derive from that base class

1
2
3
4
5


class BaseValidationError(ValueError):
	pass

class NameTooShortError(BaseValidatonError):
	pass

Cloning Objects for Fun and Profit

Assignment statements do not create copies of objects - they bind names to an object
One option - call factory function on the original object (doesn’t work on custom objects and the copy is shallow)

1
2
3


new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)

Shallow vs Deep Copying

Shallow: constructing a new collection object and then populating it with references to the child objects found in the original - only one level deep

1
2
3
4
5
6


xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
ys = list(xs) # Shallow copy

xs.append(['new sublist']) # Will be inserted into xs but not into ys
xs[1][0] = 'X' # inserts X into original as well as into the copy - because it's a shallow copy 
# if it was deep copy, X would go only in to the original xs object

Deep: copying process is recursive. Also deeper levels are copied. Both objects are fully independent

Making Deep Copies

Python’s import module

1
2
3
4
5


import copy
xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
zs = copy.deepcopy(xs)
# now altering original xs object doesn't affect new zs deepcopy at all
# you can also create Shallow copy via copy.copy() method

there is more to copying objects - they can have methods such as __copy__() and __deepcopy__() to control how they are copied

Abstract Base Classes

ABCs ensure that derived classes implement particular methods from the base class at instantiation time
using ABCs can help avoid bugs and make class hierarchies easier to maintain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


from abc import ABCMeta abstractmethod

class Base(metaclass=ABCMeta):
	@abstractmethod
	def foo(self):
		pass

	@abstractmethod
	def bar(self):
		pass

class Concrete(Base):
	def foo(self):
		pass

	# forgetting bar(), but still behaves as expected and creates the correct class hierarchy

additional useful benefit: Subclasses of Base raise a TypeError at instantiation time whenever we forget to implement any abstract methods

Namedtuples

alternative to defining a class manually with other interesting features
“extension” of the built-in tuple data type
issues with regular tuples:
- items can only be indexed via number
- it’s hard to ensure that two tuples have the same number of fields and the same properties stored on them
namedtuples allow to access item through a unique identifier

1
2
3
4
5
6
7
8
9


from collections import namedtuple
Car = namedtuple('Car', 'color mileage') # function calls .split() on field names in the background so what happens is Car = namedtuple('Car', ['color', 'mileage'])
# you can also pass a list directly

>>> my_car = Car('red', 2000)
>>> my_car.color # or my_car[0]
'red'
>>> my_car.mileage # or my_car[1]
2000

namedtuples are a memory efficient shortcut to defining an immutable class in Python manually

Built-in Helper Methods

each namedtuple instance comes with helper methods
all start with underscore: _fields
- _asdict() # return dict
- _replace() # create shallow copy
- _make() # create new instance from a sequence or iterable

When to use Namedtuples?

Clean up your code
express intentions more clearly

Class vs Instance Variable Pitfalls

there are not only class/instance methods, but also class/instance variables

Class Variables

declared inside the class definition, but outside of any instance method
are part of the class itself
all instances share them

Instance Variables

always tied to a particular instance
stored on each individual instance

1
2
3
4
5


class Dog:
	num_legs = 4 # Class variable

	def __init__(self, name):
		self.name = name # Instance variable

class variables can be accessed directly through the class (Dog.num_legs), but instance variables only through the instance: Dog.name - AttributeError
Count how many times an object was instantiated over the lifetime:

1
2
3
4
5


class CountedObject:
	num_instances = 0

	def __init__(self):
		self.__class__.num_instances += 1

Class variables can be shadowed by instance variables of the same name - it’s easy to accidentally override class variables - easy to introduce bugs

Instance, Class, and Static Methods

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class MyClass:
	def method(self): # self naming is convention, position must be first
		return 'instance method called', self

	@classmethod
	def classmethod(cls): # cls naming is convention, position must be first
		return 'class method called', cls

	@staticmethod
	def staticmethod():
		return 'static method called'

Instance Methods

regular instance method
can also modify class via self.__class__ attribute

1
2
3
4
5
6
7


obj = MyClass()
obj.method() # In background *self* does this: MyClass.method(obj) - passes object as an argument
('instance method called', <MyClass instance at 0x11a2>)

# Calling instance method on the Class itself raises TypeError
MyClass.method()
TypeError: """unbound method method() must be called with MyClass instance as first argument (got nothing instead)""" # because Python can not populate self argument

Class Methods

@classmethod
instead of self, they take cls parameter that points to the class and not the instance
since it has access only to the cls argument, it can not modify instance state (this would require self)

1
2


obj.classmethod() 
('class method called', <MyClass at 0x11a2>) # No access to the MyClass instance, but only to the MyClass object

Static Methods

@staticmethod
no cls/self parameter (can accept other arbitrary parameters)
can not modify object or class state
they are primarily a way to namespace your methods

1
2


obj.staticmethod() # Can be called on an instance
('static method called')

Common Data Structures in Python

Data structure - fundamental construct around which you build your program
Python ships with a lot of built-in data structures, but prefers “human” naming, so it can be a bit unclear what is the purpose of a particular data structure (compared to Java for example, where a List is either a LinkedList or an ArrayList)

Dictionaries, Maps & Hashtables

dict - fundamental data structure, key-value pairs
dictionaries are also often called maps, hashmaps, lookup tables or associative arrays
dictionaries allow you to quickly find the information associated with a given key

The dict data type

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


phonebook = {
			 'bob': 7387,
			 'alice': 3719,
			 'jack': 7052,
}

squares = {x: x * x for x in range(6)} # dict comprehension

>>> phonebook['alice']
3719
>>> squares
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

keys can be of any hashable type - hashable object has a hash value which never changes during its lifetime, and it can be compared to other objects
O(1) time complexity for lookup, insert, update, and delete operations in the average case
besides ‘plain’ dict objects, Python’s standard library also includes a specialized dictionary implementations (OrderedDict, defaultdict, ChainMap….)

Array Data Structures

an array - another fundamental data structure
Arrays consist of fixed-size data records that allow each element to be efficiently located based on its index
since arrays store information in adjoining blocks of memory, they’re considered contiguous data structure (as opposed to linked data structure like linked lists, for example)
performance - it’s very fast to look up an element in an array given the element’s index - O(1)
Python comes with several array-like data structures

list - Mutable Dynamic Arrays

lists are dynamic - items can be added or removed and memory will be automatically relocated
in Python, everything is an object - lists can hold arbitrary elements, including functions (downside - whole structure takes up more space)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


>>> arr = ['one', 'two', 'three']
>>> arr[0]
one
>>> arr # Lists have a nice __repr__
['one', 'two', 'three']
>>> arr[1] = 'hello' # Lists are mutable
>>> arr
['one', 'hello', 'three']
>>> del arr[1]
>>> arr
['one', 'three']
>>> arr.append(23)
>>> arr
['one', 'three', 23]

tuple - Immutable Containers

tuples are immutable - elements can’t be added or removed dynamically, all elements in a tuple must be defined at a creation time
tuples can also hold elements of arbitrary data types

1
2
3
4


>>> tup = 1, 2, 3
>>> tup + (23) # adding elements creates a copy of the tuple
>>> tup
(1, 2, 3, 34)

array.array - Basic Typed Arrays

basic C-style data types like bytes, 32-bit integers, floating point numbers and so on
similar to lists, however: they are “typed arrays” constrained to a single data type - therefore, they are more space-efficient

1
2
3
4
5
6
7


import array
arr = array.array('f', (1.0, 1.5, 2.0, 2.5))
>>> arr[1]
1.5
# Arrays are "typed"
>>> arr[1] = 'hello'
TypeError: "must be real number, not str"

str - Immutable Arrays of Unicode Characters

strings - textual data - immutable sequences of Unicode Characters
immutable - modifying a string requires creating a modified copy

1
2
3
4
5
6
7


>>> arr = 'abcd'
>>> arr[1]
'b'
>>> arr[1] = 'e' # immutable
TypeError: "'str' object does not support item assignment"
>>> list('abcd') # Get a mutable representation
['a', 'b', 'c', 'd'] 

Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up.#

Patterns for Cleaner Python#

Assert Statement#

Common Pitfalls#

1. Don’t use asserts for data validation#

2. Asserts that never fail#

Comma Placement#

Context Managers & the with statement#

Context Managers#

Underscores, Dunders & More#

Truth About String Formatting#

Old-style formatting:#

New-style formatting#

Literal String Interpolation#

Template Strings#

When to use which method?#

Effective Functions#

Python’s Functions are First-Class#

Functions are Objects#

Functions can be stored in Data Structures#

Functions can be passed to other Functions#

Functions can be Nested#

Functions can capture Local State#

Objects can behave like Functions#

Lambdas are Single-Expression Functions#

Function expression#

When to use Lambda functions#

The power of Decorators#

Decorator basics#

Applying multiple decorators to a function#

Decorating Functions that Accept Arguments#

Debugging Decorators#

*Args & **Kwargs#

Function Argument Unpacking#

Nothing to Return Here#

Classes & Object-Oriented Programming#

Object comparisons#

Every class needs a __repr__, __str__ methods#

Custom Exception Classes#

Cloning Objects for Fun and Profit#

Shallow vs Deep Copying#

Making Deep Copies#

Abstract Base Classes#

Namedtuples#

Built-in Helper Methods#

When to use Namedtuples?#

Class vs Instance Variable Pitfalls#

Class Variables#

Instance Variables#

Instance, Class, and Static Methods#

Instance Methods#

Class Methods#

Static Methods#

Common Data Structures in Python#

Dictionaries, Maps & Hashtables#

The dict data type#

Array Data Structures#

list - Mutable Dynamic Arrays#

tuple - Immutable Containers#

array.array - Basic Typed Arrays#

str - Immutable Arrays of Unicode Characters#