Working with Python Collections and Iterables
Python’s iterables and functional programming tools like map()
, filter()
, and reduce()
enable efficient data processing. In this guide, you’ll learn to master iterators, lambda functions, and the itertools
module for advanced workflows.
Iterables vs. Iterators
Definitions
- Iterable: An object that can return its elements one at a time (e.g., lists, tuples, strings).
- Iterator: An object representing a data stream. It implements
__next__()
to fetch items.
Using iter()
and next()
my_list = [10, 20, 30]
iterator = iter(my_list) # Convert iterable to iterator
print(next(iterator)) # 10
print(next(iterator)) # 20
Built-In Functions
map()
: Apply a Function to Items
# Double each number in a list
numbers = [1, 2, 3]
doubled = map(lambda x: x * 2, numbers) # Returns iterator
print(list(doubled)) # [2, 4, 6]
filter()
: Select Items Conditionally
# Filter even numbers
numbers = [1, 2, 3, 4, 5]
evens = filter(lambda x: x % 2 == 0, numbers)
print(list(evens)) # [2, 4]
reduce()
: Aggregate Values
from functools import reduce
# Sum all numbers
total = reduce(lambda x, y: x + y, [1, 2, 3, 4])
print(total) # 10
Lambda Functions
Anonymous functions for concise one-liners:
# Syntax: lambda arguments: expression
add = lambda x, y: x + y
print(add(5, 3)) # 8
The itertools
Module
Advanced tools for working with iterables:
1. chain()
: Combine Iterables
from itertools import chain
list1 = [1, 2]
list2 = ['a', 'b']
all_items = chain(list1, list2) # 1, 2, 'a', 'b'
2. cycle()
: Repeat Indefinitely
from itertools import cycle
count = 0
for item in cycle(['a', 'b']):
print(item) # a, b, a, b, ...
count += 1
if count == 4: break
3. islice()
: Slice Iterators
from itertools import islice
numbers = range(10)
sliced = islice(numbers, 2, 6) # 2, 3, 4, 5
4. groupby()
: Group by Key
from itertools import groupby
data = [("a", 1), ("a", 2), ("b", 3)]
for key, group in groupby(data, lambda x: x[0]):
print(key, list(group)) # a [('a',1), ('a',2)], b [('b',3)]
Practice Work
Exercise 1: Convert Loop to map()
Convert this loop to use map()
:
numbers = [1, 2, 3]
squared = []
for num in numbers:
squared.append(num ** 2)
squared = list(map(lambda x: x**2, numbers))
Exercise 2: Filter with Lambda
Use filter()
to extract words longer than 3 characters:
words = ["apple", "cat", "dog", "elephant"]
long_words = list(filter(lambda x: len(x) > 3, words))
Exercise 3: Use groupby()
Group numbers by even/odd:
numbers = [1, 2, 3, 4, 5, 6]
from itertools import groupby
sorted_numbers = sorted(numbers, key=lambda x: x % 2)
for key, group in groupby(sorted_numbers, lambda x: x % 2):
print("Even" if key == 0 else "Odd", list(group))
Workshop: Real-World Applications
Workshop 1: Data Pipeline
Process log files using itertools
:
from itertools import chain
# Read multiple log files
logs = chain(open("log1.txt"), open("log2.txt"))
errors = [line for line in logs if "ERROR" in line]
Workshop 2: Aggregate Sales Data
Use reduce()
to calculate total sales:
from functools import reduce
sales = [100, 200, 150]
total_sales = reduce(lambda x, y: x + y, sales)
Best Practices
- Prefer List Comprehensions for Simplicity: Use
map()
/filter()
when readability improves. - Avoid Complex Lambdas: Use named functions for multi-step logic.
- Use itertools for Memory Efficiency: Iterators save memory with large datasets.
Conclusion
Mastering iterables, functional tools, and itertools
unlocks efficient data processing in Python. Practice with the exercises to streamline your workflows!
Next Steps: Explore Python Generators or Data Classes.