Wget

Wget 是个稳定的下载工具，用它甚至可以镜像整个网站，通过 info wget 可以得到完整的使用说明。用 wget 下载整个网页的命令如下

wget -r -p -np -k http://www.kernel.org
wget -r -x --user-agent=Mozilla http://www.faqs.org/docs/artu/

--spider

When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there. For example, you can use Wget to check your bookmarks:

wget --spider --force-html -i bookmarks.html

This feature needs much more work for Wget to get close to the functionality of real web spiders.

http://www.gnu.org/software/wget/