Programing with Iterators and Generators

Prerequisites: Programmers should have had several months experience programming in Perl and some experience working on group projects.

Sometimes you'll write a function that takes too long to run because it produces too much useful information. A search function might locate ten thousand matches. A database query might return ten million records. What can you do?

Perl provides a simple and familiar model for dealing with such problems: The filehandle! Instead of reading every file in one giant gulp, we can use filehandles to staunch the flow of information, trickling it into the program in manageable gulps.

In this class, we'll see several important modules, such as File::Find and DBI, which use this approach. We'll go inside these modules and see how to implement filehandle-like data structures and objects ourselves. We'll see how to write functions that suspend themselves and then pick up later where they left off. We'll learn how to take long-running slow functions and convert them into speedy filehandle-like data generator objects.

This fundamental technique is a mainstay of programmers in other languages, but isn't as well-known as it should be. You'll be amazed at how many difficult problems become simple when you unleash the power of filehandles.

• Introduction:
  • Filehandles are Iterators
  • Iterators are Objects
  • Common Examples of Iterators: readdir, each, and DBI
    
• Homemade iterators
  • Examples:
    • File tree scanning
    • Permutations
    • Genomic Sequence Generator
    • Filehandle Iterators
    • A Flat-File Database
    • Searching Databases Backwards
    • Random Number Generation
      
• Alternatives to Iterators
  
• Filters and Transforms: map and grep for iterators
• Recursively-Constructed Iterators
• The Semipredicate Problem
• Alternative Interfaces to Iterators: Tied scalars and handles
  
• An Extended Example: Web Spiders