Added
- Add HCCrawler.defaultArgs() method.
- Emit
requestretriedevent.
changed
- Use
cacheoption not only for remembering already requested URLs but for request queue for distributed environments. - Moved
onSuccess,onErrorandmaxDepthoptions from HCCrawler.connect() and HCCrawler.launch() to crawler.queue().