
No need to know everything
Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up.#
Patterns for Cleaner Python#
- Avoid currency rounding issues by using an int and set price in cents
Assert Statement#
- Internal self-checks for your program
- proper use of assertions is to inform developers about unrecoverable errors in program
- they are not intended to signal expected error (FileNotFound) from which a user can recover
- excellent debugging tool - if you reach assertion, you have a bug
Common Pitfalls#
- security risks
- possibility to write useless assertions
1. Don’t use asserts for data validation#
- assert can be globally disabled (-0 -00 commands)
- validate data with regular if-else statements and raise exceptions on errors
2. Asserts that never fail#
- passing a tuple as first argument always evaluates to true - remember truthy values
- this will never fail:
1
|
assert(1 == 2, 'This should fail.')
|
- Always make sure your tests can actually fail
Comma Placement#
- list/dict/set - end all of your lines with a comma
1
|
names = ['Peter', 'Alice', 'John',]
|
- splitting it into new lines is an excellent method for seeing what was changed (git)
1
2
3
4
5
|
names = [
'Peter',
'Alice',
'John',
]
|
- leaving a comma avoids strings from getting merged
Context Managers & the with statement#
- with statement - write cleaner and more readable code
- simplify common resource management patterns
- encapsulates standard try/finally statements
Context Managers#
- protocol, that your objects needs to follow in order to support the with statement
- add __enter__ and __exit__ methods to an object if you want it to function as a context manager - these methods will be called in resource management cycle
1
2
3
4
5
6
7
8
9
10
11
|
class ManagedFile:
def __init__(self, name):
self.name = name
def __enter__(self):
self.file = open(self.name, 'w')
return self.file
def __exit__(self, exc_type, exc_val, exc_tb):
if self.file:
self.file.close()
|
1
2
3
|
with ManagedFile('hello.txt') as f:
f.write('hello, word')
f.write('bye now')
|
- you can also use @contextmanager decorator for this purpose
Underscores, Dunders & More#
1
2
3
4
5
6
|
# These are conventions, Python Interpreter doesn't check them
_var # var meant for internal use
var_ # next most fitting var name (first is taken) - class_
__var # Here Interpreter actualle steps in (name mangling) - it rewrites attribute name to avoid naming conflicts in subclasses
__var__ # Interpreter doesn't care - these are used for special/magic methods
_ # temporary or insignificant variable (for _ in range(10)...)
|
- _ is also good for unpacking when there are values in the middle for which you don’t care
1
2
|
car = ('red', 'auto', 12, 3812.4)
color, _, _, mileage = car
|
- _ is also handy when constructing objects on the fly and assigning values without naming them first
1
2
3
4
5
6
|
>>> list()
>>> _.append(1)
>>> _.append(2)
>>> _.append(3)
>>> _
[1, 2, 3]
|
- Python offers 4-ways to format strings
1
2
3
4
|
errno = 50159747054
name = 'Bob'
# We want to output:
'Hey Bob, there is a 0xbadc.ffee error!'
|
'Hello, %s' % name
- %s replace this as string; '%x' % errno
- %x as int
1
2
|
>>> 'Hello, {}'.format(name)
'Hello, Bob'
|
Literal String Interpolation#
- f-Strings
f'Hello, {name}!'
- supports also expressions
Template Strings#
- simpler and less powerful, but in some cases very useful
- useful when it comes to safety
1
2
3
4
|
from string import Template
t = Template('Hey, $name!')
t.substitute(name=name)
'Hey, Bob!'
|
When to use which method?#
If your format strings are user-supplied, use Template Strings to avoid security issues. Otherwise, use f-Strings.
Effective Functions#
Python’s Functions are First-Class#
First-class objects: ability to assign them t variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions.
1
2
|
def yell(text):
return text.upper() + '!'
|
Functions are Objects#
- all data in Python is represented by objects
- because a function is an object, you can assign it to a variable
bark = yell
- bark now points to the original yell function -
bark('woof')
- function objects and their names are two separate concerns
- you can
del yell
- Name is not defined, but bark
still works
bark.__name__
‘yell’
- name identifier is a debugging aid - A variable pointing to a function and the function itself are really two separate concerns
Functions can be stored in Data Structures#
1
|
funcs = [bark, str.lower, str.capitalize]
|
Functions can be passed to other Functions#
1
2
3
|
def greet(func):
greeting = func('Hi, I am a Python program')
print(greeting)
|
You can influence the resulting greeting by passing in different functions.
1
2
3
4
|
>>> greet(bark)
>>> 'HI, I AM A PYTHON PROGRAM!'
>>> greet(whisper)
>>> 'hi, i am a python program...'
|
The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around behavior in your programs.
- Higher-order functions - functions that accept other functions as arguments - key principle in Functional programming style
Classical example of higher-order function in Python - map()
1
2
|
>>> list(map(bark, ['hello', 'hey', 'hi'])) # apply bark() to each item
>>> ['HELLO!', 'HEY!', 'HI!']
|
Functions can be Nested#
- Python allows functions to be defined inside other functions
- also known as nested functions / inner functions
1
2
3
4
5
6
7
|
def speak(text):
def whisper(t):
return t.lower() + '...'
return whisper
>>> speak('Hello, World')
>>> 'hello, word...'
|
Everytime you call speak, it defines a new inner function whisper and then calls it immediately after.
- whisper does not exist outside speak!
Functions can not only accept behaviors through arguments, but they can also return behaviors.
Functions can capture Local State#
- it gets even crazier
- inner functions can also capture and carry some of the parent function’s state
- closure - remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope
Objects can behave like Functions#
- all functions are objects, but not all objects are functions
- you can make objects callable - meaning, you can use () on them
- __call__ method
1
2
3
4
5
6
7
8
9
10
|
class Adder:
def __init__(self, n):
self.n = n
def __call__(self, x):
return self.n + x
>>> plus_3 = Adder(3)
>>> plus_3(4)
>>> 7
|
- not all objects will be callable - you can check with built-in function callable()
Lambdas are Single-Expression Functions#
- lambda - declare small anonymous function
- lambda functions behave like regular function defined with
def
1
2
3
|
>>> add = lambda x, y: x + y
>>> add(5, 3)
8
|
Function expression#
1
2
|
>>> (lambda x, y: x + y)(5, 3)
8
|
- you don’t have to bind a function to a name
- you define it and call it immediately
- Lambda functions are restricted to single expression - no statements, annotations, no return statement
- there is implicit return statement - automatic
- just like nested functions, lambdas also work as lexical closures - function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that scope
When to use Lambda functions#
- they should be used sparingly and with care
- use them when you are expected to supply a function object
1
|
>>> sorted(my_list, key=lambda x: x[1])
|
- it looks cool, but readability is a critical aspect of well-designed code
The power of Decorators#
Decorator - extend & modify the behavior of a callable without permanently modifying the callable itself
- How are they useful? Example:
- you have 30 functions, something needs to change very quickly (short deadline) - no need to rewrite them all
- you need to add input/output logging to them
- ideal use case for a decorator
- understanding decorators here will be difference between high blood-pressure and staying (relatively) calm :)
- Very useful when you need to extend function with some generic functionality, such as:
- logging
- authentication
- instrumentation/timing functions
- rate-limiting
- catching, and more
Decorator basics#
- decorate/wrap another function and execute code before and after the wrapped function runs
Decorator is a callable that takes a callable as input and returns another callable
1
2
3
|
# The most basic example of a decorator
def null_decorator(func): # takes in function
return func # returns a function
|
- using
@decorator
and what happens in the background:
1
2
3
4
5
6
7
8
9
10
|
def greet():
return 'Hello!'
greet = null_decorator(greet)
>>> greet()
'Hello!'
@null_decorator
def greet():
return 'Hello!'
|
- using
@null_decorator
in front is the same as defining the function first and then running it through the decorator
- @ syntax decorates the function immediately at definition time - this makes it hard to access the undecorated function, that’s why sometimes you might prefer to decorate the function manually
- Let’s take a look at more useful decorator, one that converts result to upper case
1
2
3
4
5
6
|
def uppercase(func):
def wrapper():
original_result = func()
modified_result = original_result.upper()
return modified_result
return wrapper
|
Note how, up until now, the decorated function has never been executed. Actually calling the input function at this point wouldn’t make any sense - you’ll want the decorator to be able to modify the behavior of its input function when it eventually gets called
1
2
3
4
5
6
|
@uppercase
def greet():
return 'Hello!'
>>> greet()
'HELLO!'
|
- our decorator returns a different function object when it decorates a function
1
2
3
4
5
6
7
8
|
>>> greet
<function greet at 0x10e9f0950>
>>> null_decorator(greet)
<function greet at 0x10e9f0950> # The same object
>>> uppercase(greet)
<function uppercase.<locals>.wrapper at 0x76da02f28> # Different object
|
- decorators modify the behavior of a callable through a wrapper closure, so you don’t have to permanently modify the original
Applying multiple decorators to a function#
- you can apply more than 1 decorator to a function
- decorators are applied from bottom to top
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
def strong(func):
def wrapper():
return '<strong>' + func() + '</strong>'
return wrapper
def emphasis(func):
def wrapper():
return '<em>' + func() + '</em>'
return wrapper
@strong # Second
@emphasis # This will be used first
def greet():
return 'Hello!'
>>> greet()
'<strong><em>Hello!</em></strong>'
# In the background the chain looks like this
decorated_greet = strong(emphasis(greet))
|
- deep levels of stacking may affect performance
Decorating Functions that Accept Arguments#
- *args, **kwargs come handy here
1
2
3
4
|
def proxy(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs) # This is unpacking - wrapper unpacks the args, kwargs into original function
return wrapper
|
Debugging Decorators#
- it is challenging to debug decorators
- Python brings in functools.wraps decorator included in Standard library for this purpose
*Args & **Kwargs#
- very useful feature in Python
- they allow a function to accept optional arguments - you can create flexible APIs
- using * & ** is required, naming is convention
1
2
3
4
5
6
|
def foo(required, *args, **kwargs):
print(required)
if args: # These get collected as a tuple because of '*' prefix
print(args)
if kwargs: # These get collected as a dict because of '**' prefix
print(kwargs)
|
1
2
3
4
5
6
7
8
9
10
11
12
|
>>> foo()
TypeError:
"foo() missing 1 required positional arg: 'required'"
>>> foo('hello')
'hello'
>>> foo('hello', 1, 2, 3)
'hello'
(1, 2, 3)
>>> foo('hello', 1, 2, 3, key1='value', key2=999)
'hello'
(1, 2, 3)
{'key1': 'value', 'key2': 999}
|
- you can pass args, kwargs into other functions with unpacking
- you can also modify them before passing them along
1
2
3
4
|
def foo(x, *args, **kwargs):
kwargs['name'] = 'Alice'
new_args = args + ('extra', )
bar (x, *new_args, **kwargs)
|
- this is useful for subclassing and writing wrapper functions (decorators)
- typically though, you would use this when overriding behavior in some external class which you don’t control
Function Argument Unpacking#
1
2
3
4
5
6
7
|
vectors = [1, 0, 1]
def print_vectors(x, y, z):
print(f'<{x}, {y}, {z}>')
print_vectors(*vectors) # unpack list with *
<1, 0, 1>
|
1
2
3
|
vectors_dict = {'y': 0, 'z': 1, 'x': 1}
print_vectors(**vectors_dict) # unpack dictionary with ** (values); * for keys
<1, 0, 1>
|
Nothing to Return Here#
- Python adds an implicit return None statement to the end of any function - no need to explicitly mention it
1
2
3
4
|
def foo(value):
if value:
return value
# No need to return None otherwise - that's default
|
- When to use this future?
- function doesn’t return anything? Leave out return statement
- it comes down to readability
Classes & Object-Oriented Programming#
Object comparisons#
- ‘is’ vs ‘==’
- == checks for equality
- is checks for identity
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
a = [1, 2, 3]
b = a
>>> a
[1, 2, 3]
>>> b
[1, 2, 3]
>>> a == b # They do look the same
True
>>> a is b # And they also point to the same object
True
# Let's create a copy
c = list(a)
>>> c
[1, 2, 3]
>>> a == c # They still look identical
True
>>> a is c # But they do not point to the same object anymore
False
|
An is expression evaluates to True if two variables point to the same (identical) object.
An == expression evaluates to True if the objects referred to by the variables are equal (have the same contents)
Every class needs a __repr__, __str__ methods#
- these methods convert objects to strings
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
class Car:
def __init__(self, color, mileage):
self.color = color
self.mileage = mileage
def __str__(self):
return f'STR: a {self.color} car' # Print out instance now returns this string
def __repr__(self):
return f'REPR: a {self.color} car' # Calling instance uses repr method
>>> bmw = Car('blue', 2000)
>>> print(bmw)
'STR: a blue car' # From __str__
>>> bmw
'REPR: a blue car' # From __repr__
|
- main differences:
- repr should be intended for developers - with debugging info
- str should be purely readable representation of object - for user
- always add at least __repr__ method to a class
Custom Exception Classes#
- Excellent way to write more readable & maintainable code
1
2
3
4
|
# validate an input string
def validate(name):
if len(name) < 10:
raise ValueError # Works, but this is way too generic - doesn't say much
|
- Create custom error for name validation
1
2
3
4
5
6
7
8
9
10
|
class NameTooShortError(ValueError): # Inherit from root Exception class or specific exceptions
pass
def validate(name):
if len(name) < 10:
raise NameTooShortError(name) # Now we clearly know what went wrong
>>> validate('tom')
...
NameTooShortError: tom
|
- When publicly releasing a package, or creating a reusable module - good practice is to create a custom exception base class and derive from that base class
1
2
3
4
5
|
class BaseValidationError(ValueError):
pass
class NameTooShortError(BaseValidatonError):
pass
|
Cloning Objects for Fun and Profit#
- Assignment statements do not create copies of objects - they bind names to an object
- One option - call factory function on the original object (doesn’t work on custom objects and the copy is shallow)
1
2
3
|
new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)
|
Shallow vs Deep Copying#
- Shallow: constructing a new collection object and then populating it with references to the child objects found in the original - only one level deep
1
2
3
4
5
6
|
xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
ys = list(xs) # Shallow copy
xs.append(['new sublist']) # Will be inserted into xs but not into ys
xs[1][0] = 'X' # inserts X into original as well as into the copy - because it's a shallow copy
# if it was deep copy, X would go only in to the original xs object
|
- Deep: copying process is recursive. Also deeper levels are copied. Both objects are fully independent
Making Deep Copies#
1
2
3
4
5
|
import copy
xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
zs = copy.deepcopy(xs)
# now altering original xs object doesn't affect new zs deepcopy at all
# you can also create Shallow copy via copy.copy() method
|
- there is more to copying objects - they can have methods such as __copy__() and __deepcopy__() to control how they are copied
Abstract Base Classes#
- ABCs ensure that derived classes implement particular methods from the base class at instantiation time
- using ABCs can help avoid bugs and make class hierarchies easier to maintain
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
from abc import ABCMeta abstractmethod
class Base(metaclass=ABCMeta):
@abstractmethod
def foo(self):
pass
@abstractmethod
def bar(self):
pass
class Concrete(Base):
def foo(self):
pass
# forgetting bar(), but still behaves as expected and creates the correct class hierarchy
|
- additional useful benefit: Subclasses of Base raise a TypeError at instantiation time whenever we forget to implement any abstract methods
Namedtuples#
- alternative to defining a class manually with other interesting features
- “extension” of the built-in tuple data type
- issues with regular tuples:
- items can only be indexed via number
- it’s hard to ensure that two tuples have the same number of fields and the same properties stored on them
- namedtuples allow to access item through a unique identifier
1
2
3
4
5
6
7
8
9
|
from collections import namedtuple
Car = namedtuple('Car', 'color mileage') # function calls .split() on field names in the background so what happens is Car = namedtuple('Car', ['color', 'mileage'])
# you can also pass a list directly
>>> my_car = Car('red', 2000)
>>> my_car.color # or my_car[0]
'red'
>>> my_car.mileage # or my_car[1]
2000
|
- namedtuples are a memory efficient shortcut to defining an immutable class in Python manually
Built-in Helper Methods#
- each namedtuple instance comes with helper methods
- all start with underscore: _fields
- _asdict() # return dict
- _replace() # create shallow copy
- _make() # create new instance from a sequence or iterable
When to use Namedtuples?#
- Clean up your code
- express intentions more clearly
Class vs Instance Variable Pitfalls#
- there are not only class/instance methods, but also class/instance variables
Class Variables#
- declared inside the class definition, but outside of any instance method
- are part of the class itself
- all instances share them
Instance Variables#
- always tied to a particular instance
- stored on each individual instance
1
2
3
4
5
|
class Dog:
num_legs = 4 # Class variable
def __init__(self, name):
self.name = name # Instance variable
|
- class variables can be accessed directly through the class (Dog.num_legs), but instance variables only through the instance: Dog.name - AttributeError
- Count how many times an object was instantiated over the lifetime:
1
2
3
4
5
|
class CountedObject:
num_instances = 0
def __init__(self):
self.__class__.num_instances += 1
|
- Class variables can be shadowed by instance variables of the same name - it’s easy to accidentally override class variables - easy to introduce bugs
Instance, Class, and Static Methods#
1
2
3
4
5
6
7
8
9
10
11
|
class MyClass:
def method(self): # self naming is convention, position must be first
return 'instance method called', self
@classmethod
def classmethod(cls): # cls naming is convention, position must be first
return 'class method called', cls
@staticmethod
def staticmethod():
return 'static method called'
|
Instance Methods#
- regular instance method
- can also modify class via self.__class__ attribute
1
2
3
4
5
6
7
|
obj = MyClass()
obj.method() # In background *self* does this: MyClass.method(obj) - passes object as an argument
('instance method called', <MyClass instance at 0x11a2>)
# Calling instance method on the Class itself raises TypeError
MyClass.method()
TypeError: """unbound method method() must be called with MyClass instance as first argument (got nothing instead)""" # because Python can not populate self argument
|
Class Methods#
- @classmethod
- instead of self, they take cls parameter that points to the class and not the instance
- since it has access only to the cls argument, it can not modify instance state (this would require self)
1
2
|
obj.classmethod()
('class method called', <MyClass at 0x11a2>) # No access to the MyClass instance, but only to the MyClass object
|
Static Methods#
- @staticmethod
- no cls/self parameter (can accept other arbitrary parameters)
- can not modify object or class state
- they are primarily a way to namespace your methods
1
2
|
obj.staticmethod() # Can be called on an instance
('static method called')
|
Common Data Structures in Python#
- Data structure - fundamental construct around which you build your program
- Python ships with a lot of built-in data structures, but prefers “human” naming, so it can be a bit unclear what is the purpose of a particular data structure (compared to Java for example, where a List is either a LinkedList or an ArrayList)
Dictionaries, Maps & Hashtables#
- dict - fundamental data structure, key-value pairs
- dictionaries are also often called maps, hashmaps, lookup tables or associative arrays
- dictionaries allow you to quickly find the information associated with a given key
The dict data type#
1
2
3
4
5
6
7
8
9
10
11
12
|
phonebook = {
'bob': 7387,
'alice': 3719,
'jack': 7052,
}
squares = {x: x * x for x in range(6)} # dict comprehension
>>> phonebook['alice']
3719
>>> squares
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}
|
- keys can be of any hashable type - hashable object has a hash value which never changes during its lifetime, and it can be compared to other objects
- O(1) time complexity for lookup, insert, update, and delete operations in the average case
- besides ‘plain’ dict objects, Python’s standard library also includes a specialized dictionary implementations (OrderedDict, defaultdict, ChainMap….)
Array Data Structures#
- an array - another fundamental data structure
- Arrays consist of fixed-size data records that allow each element to be efficiently located based on its index
- since arrays store information in adjoining blocks of memory, they’re considered contiguous data structure (as opposed to linked data structure like linked lists, for example)
- performance - it’s very fast to look up an element in an array given the element’s index - O(1)
- Python comes with several array-like data structures
list - Mutable Dynamic Arrays#
- lists are dynamic - items can be added or removed and memory will be automatically relocated
- in Python, everything is an object - lists can hold arbitrary elements, including functions (downside - whole structure takes up more space)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
>>> arr = ['one', 'two', 'three']
>>> arr[0]
one
>>> arr # Lists have a nice __repr__
['one', 'two', 'three']
>>> arr[1] = 'hello' # Lists are mutable
>>> arr
['one', 'hello', 'three']
>>> del arr[1]
>>> arr
['one', 'three']
>>> arr.append(23)
>>> arr
['one', 'three', 23]
|
tuple - Immutable Containers#
- tuples are immutable - elements can’t be added or removed dynamically, all elements in a tuple must be defined at a creation time
- tuples can also hold elements of arbitrary data types
1
2
3
4
|
>>> tup = 1, 2, 3
>>> tup + (23) # adding elements creates a copy of the tuple
>>> tup
(1, 2, 3, 34)
|
array.array - Basic Typed Arrays#
- basic C-style data types like bytes, 32-bit integers, floating point numbers and so on
- similar to lists, however: they are “typed arrays” constrained to a single data type - therefore, they are more space-efficient
1
2
3
4
5
6
7
|
import array
arr = array.array('f', (1.0, 1.5, 2.0, 2.5))
>>> arr[1]
1.5
# Arrays are "typed"
>>> arr[1] = 'hello'
TypeError: "must be real number, not str"
|
str - Immutable Arrays of Unicode Characters#
- strings - textual data - immutable sequences of Unicode Characters
- immutable - modifying a string requires creating a modified copy
1
2
3
4
5
6
7
|
>>> arr = 'abcd'
>>> arr[1]
'b'
>>> arr[1] = 'e' # immutable
TypeError: "'str' object does not support item assignment"
>>> list('abcd') # Get a mutable representation
['a', 'b', 'c', 'd']
|