Introduction: The Foundation of Cybersecurity
Effective inventory management is the foundation upon which robust cybersecurity and risk management programs are built. The old truism remains valid - you can't secure what you don't know exists. Organizations frequently struggle with maintaining accurate inventories of their digital assets, creating significant security blind spots that leave them vulnerable to attacks.
This tutorial addresses a critical need in the cybersecurity ecosystem: the ability to objectively validate that vendors and organizations are performing adequate inventory management. Using free, unintrusive tools available to anyone, you can ethically assess the thoroughness of public asset inventory practices (with permission, of course). This capability is particularly valuable for security professionals evaluating vendor risk.
By following this tutorial, you'll learn how to validate that the foundation of an organization's security program is solid before building additional controls on top.
Before You Dive In: Legal and Ethical Considerations
A great way to practice this material is to target a public asset that you personally own and control. For example, to create this tutorial I targeted my own website which is hosted on AWS. But before trying these methods yourself, there are important boundaries to respect, even for assets you own:
- Hosting Provider Terms: Your web hosting agreement may have specific clauses about security scanning. Review these before proceeding.
- Shared Hosting Caution: If your website shares server resources with others, aggressive scanning could impact innocent bystanders.
- AWS-Specific Policies: For CloudFront/S3-hosted websites, AWS has particular rules regarding security testing:
- Some basic scanning is permitted without approval
- More intensive tests may require explicit permission
- Aggressive scanning might trigger AWS's protection mechanisms, potentially limiting your own access
The golden rule is to document everything you do, scan during low-traffic periods, and respect rate limits to avoid triggering defensive measures.
Prerequisites: Your Security Testing Toolkit
Here's what you'll need:
- Kali Linux
- Python 3.x
- Shodan API key
- At least one public domain to target (legally and ethically)
How to install Kali Linux is outside the scope of this tutorial, but documentation for that can be found at www.kali.org.
Nmap is a powerful open-source network scanner that comes pre-installed on Kali Linux that is used for discovering hosts and services on computer networks, creating a "map" of the network. Masscan, also pre-installed on Kali Linux, is an Internet-scale port scanner, designed for high-speed scanning of large networks with a specialized focus on raw packet performance. Amass, developed by OWASP (Open Web Application Security Project), performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques. Shodan is a web-accessible search engine for Internet-connected devices that allows users to find specific types of devices, vulnerable systems, and gather intelligence about network infrastructure. Shodan is not included in Kali Linux. It is a web application with a CLI. The free tier includes basic search functionality with limited results, while the paid tier provides more comprehensive scanning capabilities and higher query limits.
Accessing Shodan
Sign Up: Visit https://shodan.io and create a free account API Key: After registration, find your API key in your account settings
CLI Installation:
# Install pip if needed
sudo apt update
sudo apt install python3-pip
# Install the virtual environment package if needed
sudo apt install python3-venv
# Create and activate the virtual environment
python3 -m venv ~/shodan-env
source ~/shodan-env/bin/activate
# Now install shodan within the virtual environment
pip install shodan
# Initialize your API key
shodan init YOUR_API_KEY_HERE
It should return Successfully Initialized
The Tutorial: Mapping Your Digital Territory
Step 1: Gather Basic Website Information
First, let's identify your website's basic footprint:
# Get your website's IP address
host yourdomain.com
# Get DNS information
dig yourdomain.com
# Store the IP for later use (work smarter, not harder!)
export TARGET_IP=$(dig +short yourdomain.com)
echo $TARGET_IP
Think of this as creating a basic map before the expedition. You're identifying the coordinates and street address of your digital property.
Step 2: Initial Reconnaissance with Nmap
Time to see what services are visible from the outside:
# Format the IPs for Nmap by saving the IPs to a file and scan using the file
echo $TARGET_IP > target_ips.txt
# Basic port scan
sudo nmap -sV -iL target_ips.txt | tee initial_scan.txt
The -sV flag identifies service versions. Other arguments, like -A can enable OS detection, version detection, script scanning, and traceroute.
These scans reveal what services are running and potentially vulnerable. To take the property analogy one step further, this is checking which doors and windows are visible from the street.
Step 3: Shodan Intelligence Gathering
Shodan shows what information is already publicly indexed about your website. Many of Shodan's rich functionality are disabled at the free tier, however some are available:
# Search for your domain
shodan domain yourdomain.com
# Search for your IP directly
shodan host $TARGET_IP
# Look for historical data (what changed over time?)
shodan host $TARGET_IP --history
This is akin to checking what information a curious stranger could easily find about your property without even visiting it (blueprint, home inspection report, etc.)
Step 4: Advanced Port Scanning with Masscan
Let's use Masscan's speed to discover ports that might be missed by other tools:
# Full port scan of your website (at a reasonable rate)
sudo masscan -p1-65535 $TARGET_IP --rate=1000 -oJ masscan_results.json
# Compare with Nmap results (spot any differences!)
cat masscan_results.json | jq '.[]| .ports[].port' | sort -n > masscan_ports.txt
grep "open" detailed_scan.txt | grep -v "filtered" | awk '{print $1}' | sort -n > nmap_ports.txt
diff masscan_ports.txt nmap_ports.txt
Why use both Nmap and Masscan? They're like different detectives with unique investigation styles:
- Different technologies: Nmap uses TCP/IP fingerprinting; Masscan uses asynchronous transmission
- Speed vs. depth: Masscan is fast but less detailed; Nmap provides richer context
- Cross-verification: Discrepancies between tools often reveal interesting security insights
- Edge case detection: Some services respond differently to different scanning techniques
This combination gives you both breadth and depth in your reconnaissance.
Step 5: Domain Intelligence with Amass
Now let's discover subdomains and related information:
# Passive reconnaissance
amass enum -passive -d yourdomain.com -o amass_passive.txt
# More active enumeration
amass enum -active -d yourdomain.com -o amass_active.txt
This reveals your domain's broader attack surface—like discovering not just your main property but all the connected outbuildings that someone might target.
Step 6: Creating Your Assessment Report
Time to compile everything into a comprehensive report:
# Create report directory
mkdir website_assessment
cd website_assessment
# Move all results to report directory
mv ../initial_scan.txt ../detailed_scan.txt ../masscan_results.json ../amass_*.txt ./
# Generate a basic report
echo "Website Security Assessment Report" > assessment_report.txt
echo "Target: yourdomain.com ($TARGET_IP)" >> assessment_report.txt
echo "Date: $(date)" >> assessment_report.txt
echo "----------------------------------------" >> assessment_report.txt
# Add open ports and services
echo "OPEN PORTS AND SERVICES:" >> assessment_report.txt
grep "open" detailed_scan.txt | grep -v "filtered" >> assessment_report.txt
echo "----------------------------------------" >> assessment_report.txt
# Add discovered subdomains
echo "DISCOVERED SUBDOMAINS:" >> assessment_report.txt
cat amass_*.txt | sort | uniq >> assessment_report.txt
echo "----------------------------------------" >> assessment_report.txt
# Add Shodan intelligence
echo "SHODAN INTELLIGENCE:" >> assessment_report.txt
shodan host $TARGET_IP | grep -v "Last update" >> assessment_report.txt
Conclusion: Putting It All Together
Congratulations! By completing these steps, you've either validated that your assets are secure, or you've found weaknesses that need to be addressed. Either way, it's better than not knowing. This assessment gives you valuable insights into:
- Visible services: What ports and services are exposed to the internet
- Domain footprint: The broader attack surface including subdomains
- Public intelligence: What information is already being collected by adversaries about the assets
- Potential vulnerabilities: Areas that might require additional hardening
This knowledge allows you to make informed security decisions before potential attackers discover weaknesses.
Remember, in security, what you don't know CAN hurt you. By regularly validating public assets, you're seeing yourself as others see you online, and addressing vulnerabilities before they become incidents.
Now go forth and scan responsibly. Your future uncompromised self will thank you!