How This Book Is Organized

There are several facets of spidering: searching for and finding the information, gleaning and gathering, repurposing and repackaging, and sharing it with peers. On the flip side, there's also ensuring that your own site's information is spider-friendly. We've divided this book into six parts:

Chapter 1

Before setting out, it behooves you to make sure you understand all the basics, philosophies, considerations, and issues. This chapter is an overview of all you should know before you build and unleash your spiders on the world.

Chapter 2

The most useful tools in your spidering toolbox are often those you didn't have to write yourself. There are modules galore, written by prominent members of the programming community and free for borrowing, repurposing, and using as a foundation for your own work. This chapter introduces you to the tools you'll need to program for the Web, spidering and scraping in the most efficient and appropriate manner possible.

Chapter 3

Hey, nobody lives by text alone. We'll show you some great resources for media files and some cool ways to go about getting them. From current comic strips to old movies from the Library of Congress, there are some wonderful finds for your media library.

Chapter 4

Collections of data are going online at an unbelievable pace. While getting just the data you want isn't as easy as just scraping some words off a web page, we'll show you how to get to just the information you're after, combine it with data from other sites, and repurpose it in just the way you need.

Chapter 5

Oh, the Web is always a-changin'. We'll show you how to keep your data current, mirror collections to your local hard drive, and schedule your spiders to run on a regular basis.

Chapter 6

Maybe you're just as interested in being spidered as spidering. Perhaps you have a collection others might find useful. We'll show you how to share your own data in ways that make it easier for those downstream to spider you—avoiding many of the headaches we discuss in this book!