Book cover

No need to know everything

Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up.

Patterns for Cleaner Python

  • Avoid currency rounding issues by using an int and set price in cents

Assert Statement

  • Internal self-checks for your program
  • proper use of assertions is to inform developers about unrecoverable errors in program
  • they are not intended to signal expected error (FileNotFound) from which a user can recover
  • excellent debugging tool - if you reach assertion, you have a bug
Common Pitfalls
  1. security risks
  2. possibility to write useless assertions
1. Don’t use asserts for data validation
  • assert can be globally disabled (-0 -00 commands)
  • validate data with regular if-else statements and raise exceptions on errors
2. Asserts that never fail
  • passing a tuple as first argument always evaluates to true - remember truthy values
  • this will never fail:
1
assert(1 == 2, 'This should fail.')
  • Always make sure your tests can actually fail

Comma Placement

  • list/dict/set - end all of your lines with a comma
1
names = ['Peter', 'Alice', 'John',]
  • splitting it into new lines is an excellent method for seeing what was changed (git)
1
2
3
4
5
names = [
		 'Peter',
		 'Alice',
		 'John',
]
  • leaving a comma avoids strings from getting merged

Context Managers & the with statement

  • with statement - write cleaner and more readable code
  • simplify common resource management patterns
  • encapsulates standard try/finally statements
Context Managers
  • protocol, that your objects needs to follow in order to support the with statement
  • add __enter__ and __exit__ methods to an object if you want it to function as a context manager - these methods will be called in resource management cycle
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
class ManagedFile:
	def __init__(self, name):
		self.name = name

	def __enter__(self):
		self.file = open(self.name, 'w')
		return self.file

	def __exit__(self, exc_type, exc_val, exc_tb):
		if self.file:
			self.file.close()
  • now we can use
1
2
3
with ManagedFile('hello.txt') as f:
	f.write('hello, word')
	f.write('bye now')
  • you can also use @contextmanager decorator for this purpose

Underscores, Dunders & More

1
2
3
4
5
6
# These are conventions, Python Interpreter doesn't check them
_var # var meant for internal use 
var_ # next most fitting var name (first is taken) - class_
__var # Here Interpreter actualle steps in (name mangling) - it rewrites attribute name to avoid naming conflicts in subclasses
__var__ # Interpreter doesn't care - these are used for special/magic methods
_ # temporary or insignificant variable (for _ in range(10)...)
  • _ is also good for unpacking when there are values in the middle for which you don’t care
1
2
car = ('red', 'auto', 12, 3812.4)
color, _, _, mileage = car
  • _ is also handy when constructing objects on the fly and assigning values without naming them first
1
2
3
4
5
6
>>> list()
>>> _.append(1)
>>> _.append(2)
>>> _.append(3)
>>> _
[1, 2, 3]

Truth About String Formatting

  • Python offers 4-ways to format strings
1
2
3
4
errno = 50159747054
name = 'Bob'
# We want to output:
'Hey Bob, there is a 0xbadc.ffee error!'
Old-style formatting:
  • 'Hello, %s' % name - %s replace this as string; '%x' % errno - %x as int
New-style formatting
  • format() function
1
2
>>> 'Hello, {}'.format(name)
'Hello, Bob'
Literal String Interpolation
  • f-Strings f'Hello, {name}!'
  • supports also expressions
Template Strings
  • simpler and less powerful, but in some cases very useful
  • useful when it comes to safety
1
2
3
4
from string import Template
t = Template('Hey, $name!')
t.substitute(name=name)
'Hey, Bob!'
When to use which method?

If your format strings are user-supplied, use Template Strings to avoid security issues. Otherwise, use f-Strings.

Effective Functions

Python’s Functions are First-Class

First-class objects: ability to assign them t variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions.

1
2
def yell(text):
	return text.upper() + '!'

Functions are Objects

  • all data in Python is represented by objects
  • because a function is an object, you can assign it to a variable
  • bark = yell
  • bark now points to the original yell function - bark('woof')
  • function objects and their names are two separate concerns
  • you can del yell - Name is not defined, but bark still works
  • bark.__name__ ‘yell’
  • name identifier is a debugging aid - A variable pointing to a function and the function itself are really two separate concerns
Functions can be stored in Data Structures
1
funcs = [bark, str.lower, str.capitalize]
Functions can be passed to other Functions
1
2
3
def greet(func):
	greeting = func('Hi, I am a Python program')
	print(greeting)

You can influence the resulting greeting by passing in different functions.

1
2
3
4
>>> greet(bark)
>>> 'HI, I AM A PYTHON PROGRAM!'
>>> greet(whisper)
>>> 'hi, i am a python program...'

The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around behavior in your programs.

  • Higher-order functions - functions that accept other functions as arguments - key principle in Functional programming style

Classical example of higher-order function in Python - map()

1
2
>>> list(map(bark, ['hello', 'hey', 'hi'])) # apply bark() to each item
>>> ['HELLO!', 'HEY!', 'HI!']
Functions can be Nested
  • Python allows functions to be defined inside other functions
  • also known as nested functions / inner functions
1
2
3
4
5
6
7
def speak(text):
	def whisper(t):
		return t.lower() + '...'
	return whisper

>>> speak('Hello, World')
>>> 'hello, word...'

Everytime you call speak, it defines a new inner function whisper and then calls it immediately after.

  • whisper does not exist outside speak!

Functions can not only accept behaviors through arguments, but they can also return behaviors.

Functions can capture Local State
  • it gets even crazier
  • inner functions can also capture and carry some of the parent function’s state
  • closure - remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope
Objects can behave like Functions
  • all functions are objects, but not all objects are functions
  • you can make objects callable - meaning, you can use () on them
  • __call__ method
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
class Adder:
	def __init__(self, n):
		self.n = n

	def __call__(self, x):
		return self.n + x

>>> plus_3 = Adder(3)
>>> plus_3(4)
>>> 7
  • not all objects will be callable - you can check with built-in function callable()

Lambdas are Single-Expression Functions

  • lambda - declare small anonymous function
  • lambda functions behave like regular function defined with def
1
2
3
>>> add = lambda x, y: x + y
>>> add(5, 3)
8
Function expression
1
2
>>> (lambda x, y: x + y)(5, 3)
8
  • you don’t have to bind a function to a name
  • you define it and call it immediately
  • Lambda functions are restricted to single expression - no statements, annotations, no return statement
  • there is implicit return statement - automatic
  • just like nested functions, lambdas also work as lexical closures - function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that scope
When to use Lambda functions
  • they should be used sparingly and with care
  • use them when you are expected to supply a function object
1
>>> sorted(my_list, key=lambda x: x[1])
  • it looks cool, but readability is a critical aspect of well-designed code

The power of Decorators

Decorator - extend & modify the behavior of a callable without permanently modifying the callable itself

  • How are they useful? Example:
    • you have 30 functions, something needs to change very quickly (short deadline) - no need to rewrite them all
    • you need to add input/output logging to them
    • ideal use case for a decorator
    • understanding decorators here will be difference between high blood-pressure and staying (relatively) calm :)
  • Very useful when you need to extend function with some generic functionality, such as:
    • logging
    • authentication
    • instrumentation/timing functions
    • rate-limiting
    • catching, and more
Decorator basics
  • decorate/wrap another function and execute code before and after the wrapped function runs

Decorator is a callable that takes a callable as input and returns another callable

1
2
3
# The most basic example of a decorator
def null_decorator(func): # takes in function
	return func # returns a function
  • using @decorator and what happens in the background:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def greet():
	return 'Hello!'

greet = null_decorator(greet)
>>> greet()
'Hello!'

@null_decorator
def greet():
	return 'Hello!'
  • using @null_decorator in front is the same as defining the function first and then running it through the decorator
  • @ syntax decorates the function immediately at definition time - this makes it hard to access the undecorated function, that’s why sometimes you might prefer to decorate the function manually
  • Let’s take a look at more useful decorator, one that converts result to upper case
1
2
3
4
5
6
def uppercase(func):
	def wrapper():
		original_result = func()
		modified_result = original_result.upper()
		return modified_result
	return wrapper

Note how, up until now, the decorated function has never been executed. Actually calling the input function at this point wouldn’t make any sense - you’ll want the decorator to be able to modify the behavior of its input function when it eventually gets called

1
2
3
4
5
6
@uppercase
def greet():
	return 'Hello!'

>>> greet()
'HELLO!'
  • our decorator returns a different function object when it decorates a function
1
2
3
4
5
6
7
8
>>> greet
<function greet at 0x10e9f0950>

>>> null_decorator(greet)
<function greet at 0x10e9f0950> # The same object

>>> uppercase(greet)
<function uppercase.<locals>.wrapper at 0x76da02f28> # Different object
  • decorators modify the behavior of a callable through a wrapper closure, so you don’t have to permanently modify the original
Applying multiple decorators to a function
  • you can apply more than 1 decorator to a function
  • decorators are applied from bottom to top
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
def strong(func):
	def wrapper():
		return '<strong>' + func() + '</strong>'
	return wrapper

def emphasis(func):
	def wrapper():
		return '<em>' + func() + '</em>'
	return wrapper

@strong # Second
@emphasis # This will be used first
def greet():
	return 'Hello!'

>>> greet()
'<strong><em>Hello!</em></strong>'

# In the background the chain looks like this
decorated_greet = strong(emphasis(greet))
  • deep levels of stacking may affect performance
Decorating Functions that Accept Arguments
  • *args, **kwargs come handy here
1
2
3
4
def proxy(func):
	def wrapper(*args, **kwargs): 
		return func(*args, **kwargs) # This is unpacking - wrapper unpacks the args, kwargs into  original function
	return wrapper
Debugging Decorators
  • it is challenging to debug decorators
  • Python brings in functools.wraps decorator included in Standard library for this purpose

*Args & **Kwargs

  • very useful feature in Python
  • they allow a function to accept optional arguments - you can create flexible APIs
  • using * & ** is required, naming is convention
1
2
3
4
5
6
def foo(required, *args, **kwargs):
	print(required)
	if args: # These get collected as a tuple because of '*' prefix
		print(args)
	if kwargs: # These get collected as a dict because of '**' prefix
		print(kwargs)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
>>> foo()
TypeError:
"foo() missing 1 required positional arg: 'required'"
>>> foo('hello')
'hello'
>>> foo('hello', 1, 2, 3)
'hello'
(1, 2, 3)
>>> foo('hello', 1, 2, 3, key1='value', key2=999)
'hello'
(1, 2, 3)
{'key1': 'value', 'key2': 999}
  • you can pass args, kwargs into other functions with unpacking
  • you can also modify them before passing them along
1
2
3
4
def foo(x, *args, **kwargs):
	kwargs['name'] = 'Alice'
	new_args = args + ('extra', )
	bar (x, *new_args, **kwargs)
  • this is useful for subclassing and writing wrapper functions (decorators)
  • typically though, you would use this when overriding behavior in some external class which you don’t control

Function Argument Unpacking

1
2
3
4
5
6
7
vectors = [1, 0, 1]

def print_vectors(x, y, z):
	print(f'<{x}, {y}, {z}>')

print_vectors(*vectors) # unpack list with *
<1, 0, 1>
1
2
3
vectors_dict = {'y': 0, 'z': 1, 'x': 1}
print_vectors(**vectors_dict) # unpack dictionary with ** (values); * for keys
<1, 0, 1>

Nothing to Return Here

  • Python adds an implicit return None statement to the end of any function - no need to explicitly mention it
1
2
3
4
def foo(value):
	if value:
		return value
	# No need to return None otherwise - that's default
  • When to use this future?
    • function doesn’t return anything? Leave out return statement
    • it comes down to readability

Classes & Object-Oriented Programming

Object comparisons

  • is’ vs ‘==
  • == checks for equality
  • is checks for identity
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
a = [1, 2, 3]
b = a

>>> a
[1, 2, 3]
>>> b
[1, 2, 3]
>>> a == b # They do look the same
True
>>> a is b # And they also point to the same object
True

# Let's create a copy
c = list(a)
>>> c
[1, 2, 3]
>>> a == c # They still look identical
True
>>> a is c # But they do not point to the same object anymore
False

An is expression evaluates to True if two variables point to the same (identical) object.

An == expression evaluates to True if the objects referred to by the variables are equal (have the same contents)

Every class needs a __repr__, __str__ methods

  • these methods convert objects to strings
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
class Car:

	def __init__(self, color, mileage):
		self.color = color
		self.mileage = mileage
		
	def __str__(self):
		return f'STR: a {self.color} car' # Print out instance now returns this string

	def __repr__(self):
		return f'REPR: a {self.color} car' # Calling instance uses repr  method

>>> bmw = Car('blue', 2000)
>>> print(bmw)
'STR: a blue car' # From __str__
>>> bmw
'REPR: a blue car' # From __repr__
  • main differences:
    • repr should be intended for developers - with debugging info
    • str should be purely readable representation of object - for user
  • always add at least __repr__ method to a class

Custom Exception Classes

  • Excellent way to write more readable & maintainable code
1
2
3
4
# validate an input string
def validate(name):
	if len(name) < 10:
		raise ValueError # Works, but this is way too generic - doesn't say much
  • Create custom error for name validation
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
class NameTooShortError(ValueError): # Inherit from root Exception class or specific exceptions
	pass

def validate(name):
	if len(name) < 10:
		raise NameTooShortError(name) # Now we clearly know what went wrong

>>> validate('tom')
...
NameTooShortError: tom
  • When publicly releasing a package, or creating a reusable module - good practice is to create a custom exception base class and derive from that base class
1
2
3
4
5
class BaseValidationError(ValueError):
	pass

class NameTooShortError(BaseValidatonError):
	pass

Cloning Objects for Fun and Profit

  • Assignment statements do not create copies of objects - they bind names to an object
  • One option - call factory function on the original object (doesn’t work on custom objects and the copy is shallow)
1
2
3
new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)
Shallow vs Deep Copying
  • Shallow: constructing a new collection object and then populating it with references to the child objects found in the original - only one level deep
1
2
3
4
5
6
xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
ys = list(xs) # Shallow copy

xs.append(['new sublist']) # Will be inserted into xs but not into ys
xs[1][0] = 'X' # inserts X into original as well as into the copy - because it's a shallow copy 
# if it was deep copy, X would go only in to the original xs object
  • Deep: copying process is recursive. Also deeper levels are copied. Both objects are fully independent
Making Deep Copies
  • Python’s import module
1
2
3
4
5
import copy
xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
zs = copy.deepcopy(xs)
# now altering original xs object doesn't affect new zs deepcopy at all
# you can also create Shallow copy via copy.copy() method
  • there is more to copying objects - they can have methods such as __copy__() and __deepcopy__() to control how they are copied

Abstract Base Classes

  • ABCs ensure that derived classes implement particular methods from the base class at instantiation time
  • using ABCs can help avoid bugs and make class hierarchies easier to maintain
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from abc import ABCMeta abstractmethod

class Base(metaclass=ABCMeta):
	@abstractmethod
	def foo(self):
		pass

	@abstractmethod
	def bar(self):
		pass

class Concrete(Base):
	def foo(self):
		pass

	# forgetting bar(), but still behaves as expected and creates the correct class hierarchy
  • additional useful benefit: Subclasses of Base raise a TypeError at instantiation time whenever we forget to implement any abstract methods

Namedtuples

  • alternative to defining a class manually with other interesting features
  • “extension” of the built-in tuple data type
  • issues with regular tuples:
    • items can only be indexed via number
    • it’s hard to ensure that two tuples have the same number of fields and the same properties stored on them
  • namedtuples allow to access item through a unique identifier
1
2
3
4
5
6
7
8
9
from collections import namedtuple
Car = namedtuple('Car', 'color mileage') # function calls .split() on field names in the background so what happens is Car = namedtuple('Car', ['color', 'mileage'])
# you can also pass a list directly

>>> my_car = Car('red', 2000)
>>> my_car.color # or my_car[0]
'red'
>>> my_car.mileage # or my_car[1]
2000
  • namedtuples are a memory efficient shortcut to defining an immutable class in Python manually
Built-in Helper Methods
  • each namedtuple instance comes with helper methods
  • all start with underscore: _fields
    • _asdict() # return dict
    • _replace() # create shallow copy
    • _make() # create new instance from a sequence or iterable
When to use Namedtuples?
  • Clean up your code
  • express intentions more clearly

Class vs Instance Variable Pitfalls

  • there are not only class/instance methods, but also class/instance variables
Class Variables
  • declared inside the class definition, but outside of any instance method
  • are part of the class itself
  • all instances share them
Instance Variables
  • always tied to a particular instance
  • stored on each individual instance
1
2
3
4
5
class Dog:
	num_legs = 4 # Class variable

	def __init__(self, name):
		self.name = name # Instance variable
  • class variables can be accessed directly through the class (Dog.num_legs), but instance variables only through the instance: Dog.name - AttributeError
  • Count how many times an object was instantiated over the lifetime:
1
2
3
4
5
class CountedObject:
	num_instances = 0

	def __init__(self):
		self.__class__.num_instances += 1
  • Class variables can be shadowed by instance variables of the same name - it’s easy to accidentally override class variables - easy to introduce bugs

Instance, Class, and Static Methods

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
class MyClass:
	def method(self): # self naming is convention, position must be first
		return 'instance method called', self

	@classmethod
	def classmethod(cls): # cls naming is convention, position must be first
		return 'class method called', cls

	@staticmethod
	def staticmethod():
		return 'static method called'
Instance Methods
  • regular instance method
  • can also modify class via self.__class__ attribute
1
2
3
4
5
6
7
obj = MyClass()
obj.method() # In background *self* does this: MyClass.method(obj) - passes object as an argument
('instance method called', <MyClass instance at 0x11a2>)

# Calling instance method on the Class itself raises TypeError
MyClass.method()
TypeError: """unbound method method() must be called with MyClass instance as first argument (got nothing instead)""" # because Python can not populate self argument
Class Methods
  • @classmethod
  • instead of self, they take cls parameter that points to the class and not the instance
  • since it has access only to the cls argument, it can not modify instance state (this would require self)
1
2
obj.classmethod() 
('class method called', <MyClass at 0x11a2>) # No access to the MyClass instance, but only to the MyClass object
Static Methods
  • @staticmethod
  • no cls/self parameter (can accept other arbitrary parameters)
  • can not modify object or class state
  • they are primarily a way to namespace your methods
1
2
obj.staticmethod() # Can be called on an instance
('static method called')

Common Data Structures in Python

  • Data structure - fundamental construct around which you build your program
  • Python ships with a lot of built-in data structures, but prefers “human” naming, so it can be a bit unclear what is the purpose of a particular data structure (compared to Java for example, where a List is either a LinkedList or an ArrayList)

Dictionaries, Maps & Hashtables

  • dict - fundamental data structure, key-value pairs
  • dictionaries are also often called maps, hashmaps, lookup tables or associative arrays
  • dictionaries allow you to quickly find the information associated with a given key
The dict data type
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
phonebook = {
			 'bob': 7387,
			 'alice': 3719,
			 'jack': 7052,
}

squares = {x: x * x for x in range(6)} # dict comprehension

>>> phonebook['alice']
3719
>>> squares
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}
  • keys can be of any hashable type - hashable object has a hash value which never changes during its lifetime, and it can be compared to other objects
  • O(1) time complexity for lookup, insert, update, and delete operations in the average case
  • besides ‘plain’ dict objects, Python’s standard library also includes a specialized dictionary implementations (OrderedDict, defaultdict, ChainMap….)

Array Data Structures

  • an array - another fundamental data structure
  • Arrays consist of fixed-size data records that allow each element to be efficiently located based on its index
  • since arrays store information in adjoining blocks of memory, they’re considered contiguous data structure (as opposed to linked data structure like linked lists, for example)
  • performance - it’s very fast to look up an element in an array given the element’s index - O(1)
  • Python comes with several array-like data structures
list - Mutable Dynamic Arrays
  • lists are dynamic - items can be added or removed and memory will be automatically relocated
  • in Python, everything is an object - lists can hold arbitrary elements, including functions (downside - whole structure takes up more space)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
>>> arr = ['one', 'two', 'three']
>>> arr[0]
one
>>> arr # Lists have a nice __repr__
['one', 'two', 'three']
>>> arr[1] = 'hello' # Lists are mutable
>>> arr
['one', 'hello', 'three']
>>> del arr[1]
>>> arr
['one', 'three']
>>> arr.append(23)
>>> arr
['one', 'three', 23]
tuple - Immutable Containers
  • tuples are immutable - elements can’t be added or removed dynamically, all elements in a tuple must be defined at a creation time
  • tuples can also hold elements of arbitrary data types
1
2
3
4
>>> tup = 1, 2, 3
>>> tup + (23) # adding elements creates a copy of the tuple
>>> tup
(1, 2, 3, 34)
array.array - Basic Typed Arrays
  • basic C-style data types like bytes, 32-bit integers, floating point numbers and so on
  • similar to lists, however: they are “typed arrays” constrained to a single data type - therefore, they are more space-efficient
1
2
3
4
5
6
7
import array
arr = array.array('f', (1.0, 1.5, 2.0, 2.5))
>>> arr[1]
1.5
# Arrays are "typed"
>>> arr[1] = 'hello'
TypeError: "must be real number, not str"
str - Immutable Arrays of Unicode Characters
  • strings - textual data - immutable sequences of Unicode Characters
  • immutable - modifying a string requires creating a modified copy
1
2
3
4
5
6
7
>>> arr = 'abcd'
>>> arr[1]
'b'
>>> arr[1] = 'e' # immutable
TypeError: "'str' object does not support item assignment"
>>> list('abcd') # Get a mutable representation
['a', 'b', 'c', 'd']