Spidering

Web Spidering

zaproxy | Kali Linux ToolsKali Linux

spiderfoot | Kali Linux ToolsKali Linux

Burp

Enable proxy and browse a site normally. You you will see the complete site details.

Mirroring a Website

Wget

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent certifiedhacker.com

HTTrack Website Copier - Free Software Offline Browser (GNU GPL)www.httrack.com

httrack http://target.ine.local -O target.html

Other tools

Website data scrappers

ParseHub | Free web scraping - The most powerful web scraperwww.parsehub.com