From: David ‘Bombe’ Roden <bombe@pterodactylus.net>
Date: Mon, 16 Mar 2009 23:09:47 +0000 (+0100)
Subject: Remember crawled pages and don’t crawl them again.
X-Git-Url: https://git.pterodactylus.net/?a=commitdiff_plain;h=83a84def68ebc8d094bc9837ec72b5fc5cfdd782;hp=83a84def68ebc8d094bc9837ec72b5fc5cfdd782;p=arachne.git

Remember crawled pages and don’t crawl them again.
Don’t add pages that are already scheduled for crawling.
---