Skip to content
GitLab

Obey robot txt

Merged yujiosaka requested to merge obey_robot_txt into master
  • Support obeyRobotsTxt option
  • Support skipDuplicates option
  • Make cache option to be required
  • Added docs for exporter option
  • Modify BaseCache interface

Fixes: https://github.com/yujiosaka/headless-chrome-crawler/issues/11