SpiderFoot – Automated Intelligence Gathering Tool.

Learn about SpiderFoot, a powerful Opensource intelligence gathering tool that automates reconnaissance process and delivers actionable information.

Introduction

SpiderFoot is an open source tool for online intelligence gathering and footprinting. It automates the task footprinting a given target like the details of IP address, domain name, hostname or network subnet etc.

Key uses of SpiderFoot – An Open Source Intelligence Automation Tool (OSINT)

• For pen-testers- automating the entire reconnaissance stage.
• Gathering threat intelligence -about suspicious IPs or site names caught in logs or SIEM tools.
• For corporate security teams -what sensitive information is openly exposed to the outside world.

Features Of SpiderFoot

SpiderFoot has plenty of features, including the following:

• Utilizes a lot of different data sources, some of them being SHODAN, RIPE, Whois, PasteBin, Google, SANS and more.
• Designed for intelligent and maximum data extraction through various modules.
• Cross platform support- Runs on Linux and Windows and it is open source.
• User-friendly visualizations -Built-in JavaScript-based visualizations or export to GEXF/CSV for use in other tools with easy to use, easy to navigate web-based UI.
• Highly configurable modular design with modules written in python so the level of intrusiveness and functionality can be defined.
• SQLite back-end support - scan results are stored in a local SQLite database so you can play with your data to your heart's content.
• Simultaneous scans-Each footprint scan runs as its own thread so you can perform footprinting of many different targets simultaneously.

Getting Started with SpiderFoot

Before installation make sure the following Pre-Requisites are met.

Linux/BSD/Solaris

SpiderFoot is written in Python (2.7), so to run on Linux/Solaris/FreeBSD/etc. you need Python 2.7 installed, in addition to the lxml, netaddr, M2Crypto, CherryPy and Mako modules.
To install the dependencies using PIP, run the following:
~$ pip install lxml netaddr M2Crypto cherrypy mako
Other modules such as MetaPDF, SOCKS and more are included in the SpiderFoot package, so you don't need to install them separately.

Windows

SpiderFoot for Windows is a compiled executable file, and so all dependencies are packaged with it.
No third party tools/libraries need to be installed, not even Python.

Installation

Installing SpiderFoot is literally as simple as unpacking the distribution tar.gz/zip file.

Linux/BSD/Solaris

To install SpiderFoot on Linux/Solaris/FreeBSD/etc. you only need to un-targz the package, as follows:
~$ tar zxvf spiderfoot-X.X.X-src.tar.gz
~$ cd spiderfoot-X.X.X
~/spiderfoot-X.X.X$

Windows

Unzip the distribution ZIP file and run it.

Starting SpiderFoot

Linux/BSD/Solaris

To run SpiderFoot, simply execute sf.py from the directory you extracted SpiderFoot into:
~/spiderfoot-X.X.X$ python ./sf
Once executed, a web server will be started, which by default will listen on 127.0.0.1:5001. You can then use the web-browser of your choice by browsing to http://127.0.0.1:5001.

If you wish to make SpiderFoot accessible from another system, for example running it on a server and controlling it remotely, then you can specify an external IP for SpiderFoot to bind to, or use 0.0.0.0 so that it binds to all addresses, including 127.0.0.1:
~/spiderfoot-X.X.X$ python ./sf 0.0.0.0:5001
If port 5001 is used by another application on your system, you can change the port:
~/spiderfoot-X.X.X$ python ./sf 127.0.0.1:9999

Windows

SpiderFoot for Windows comes as a pre-packaged executable, with no need to install any dependencies.
For now, there is no installer wizard, so all that's needed is to unzip the package into a directory (e.g. C:\SpiderFoot) and run sf.exe:
C:\SpiderFoot>sf.exe
As with Linux, you can also specify the IP and port to bind to:
C:\SpiderFoot>sf.exe 0.0.0.0:9999


Detailed step by instructions for installing, configuring and running a scan using SpiderFoot can be found on the documentation page http://www.spiderfoot.net/documentation/

Conclusion

SpiderFoot is a reconnaissance tool that automates the data collection process for penetration testing. It collects and analyses data from a variety of open-source intelligence sources. The tool is flexible and scalable, allowing it to be used for a variety of use cases. Its automated data collection and analysis approach can save security professionals time and streamline their testing processes.