In this post, we delve into the utilization of projectdiscovery’s tools—nuclei, naabu, and subfinder—for conducting a comprehensive scan of the internet to identify security vulnerabilities. The IP addresses associated with these domains are sourced from pre-compiled lists provided by kaeferjaeger. Additionally, we provide recommendations on the most efficient approaches to streamline your workflow.

Theory

Tools

1. subfinder
2. naabu
3. nuclei
4. ip-ranges

Download domains

You can either scan the internet using zmap, then use zgrab to grab domains and then proceed to grep out all the domains and use that or you can visit kaeferjaeger.gay and download all of those and grep the domains yourself. However the list only uses popular cloud providers such as Google, Digital Ocean, Microsoft, Amazon and Oracle.

Grep domains

It’s better to output all `*.txt`` into one big file and then grep that file:

cd ~/sni_ip_ranges; 
cat *.txt > phatboi.txt
grep -E -o '[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+(\.[a-zA-Z]{2,})' ~/sni_ip_ranges/phatboi.txt > domains.txt

Enumerate subdomains

For a more comprehensive scan enumerate all domains for their subdomains:

subfinder -dL gov_domains.txt -silent -o government_domains.txt

Port scan domains

For an even more comprehensive scan port scan the domains before vuln scanning them:

naabu -l government_domains.txt -silent -o government_domains_final.txt

Vulnerability scan hosts file for critical and high severity vulns only

Finally vuln scan the domains and wait a while. We specify critical and high severity vulnerabilities to later shell:

nuclei -l government_domains_final.txt -s critical,high -silent -o vuln_gov_domains.txt

Advices

Install naabu

Installing naabu requires two preparations which are not specified in Install manual of naabu. Before installing naabu, do the following:

sudo apt install build-essential -y
export CGO_ENABLED=1

This are preparations steps for compiling pcap for golang. Then

sudo apt install -y libpcap-dev
go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest

Optimization for ssh closed connection

The vulnerability scan is conducted on bulletproof servers, known for their anonymity and resilience against abuse reports. Consequently, you operate remotely using SSH. However, without utilizing the screen utility to safeguard against disconnections, you might encounter a significant issue: running a scan with tools like nuclei or naabu in silent mode provides no indication of the scan’s progress when the SSH connection terminates abruptly. As a result, you’ll be required to restart the scan from the beginning.

Personally, I choose not to use the screen utility. I enjoy employing “differentials”: in the event of a minor loss of work, I find it manageable to redo the small amount lost. Hence, I segment all substantial text files into numerous smaller files and execute tools on these fragments. For instance, if chunk X, composed of ten lines extracted from a 1000-line large file, remains incomplete due to a disruption in the connection, I can effortlessly rerun those ten lines.

Example

Let’s consider the following command:

naabu -l government_domains.txt -silent -o government_domains_final.txt

This command operates on a significantly large file, government_domains.txt, comprising many thousands of lines. While executing silently, the tool doesn’t display progress in the output file; it only contains successful results. For instance, if the initial file had 10,000 lines and the tool scanned the first 1000 lines without discovering any open ports, the output file would remain empty. Now, if the SSH connection breaks and upon reconnecting, finding an empty output file, you’d need to rerun the tool without any prior information. However, the tool had already scanned the first 1000 lines and found no open ports. Thus, running the tool again necessitates rescanning these initial lines.

Alternatively, let’s imagine splitting the input file, government_domains.txt, into 100 files, each containing 10 lines. In this scenario, the tool would generate 100 output files, signifying the absence of open ports in the first 100 chunks. In case of a disconnection, resuming the scan from bulk 101 would be feasible, avoiding redundant rescanning of the initial 1000 lines.

To solve that problem we use Linux split utility installed on most distributions by default.

Here are the command line options of split utility:

Usage: split [OPTION]... [FILE [PREFIX]]
Output pieces of FILE to PREFIXaa, PREFIXab, ...;
default size is 1000 lines, and default PREFIX is 'x'.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   generate suffixes of length N (default 2)
      --additional-suffix=SUFFIX  append an additional SUFFIX to file names
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of records per output file
  -d                      use numeric suffixes starting at 0, not alphabetic
      --numeric-suffixes[=FROM]  same as -d, but allow setting the start value
  -x                      use hex suffixes starting at 0, not alphabetic
      --hex-suffixes[=FROM]  same as -x, but allow setting the start value
  -e, --elide-empty-files  do not generate empty output files with '-n'
      --filter=COMMAND    write to shell COMMAND; file name is $FILE
  -l, --lines=NUMBER      put NUMBER lines/records per output file
  -n, --number=CHUNKS     generate CHUNKS output files; see explanation below
  -t, --separator=SEP     use SEP instead of newline as the record separator;
                            '\0' (zero) specifies the NUL character
  -u, --unbuffered        immediately copy input to output with '-n r/...'
      --verbose           print a diagnostic just before each
                            output file is opened
      --help     display this help and exit
      --version  output version information and exit

The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).
Binary prefixes can be used, too: KiB=K, MiB=M, and so on.

CHUNKS may be:
  N       split into N files based on size of input
  K/N     output Kth of N to stdout
  l/N     split into N files without splitting lines/records
  l/K/N   output Kth of N to stdout without splitting lines/records
  r/N     like 'l' but use round robin distribution
  r/K/N   likewise but only output Kth of N to stdout

Practice

Download ip range file

First, we get oracle’s hosted ips

Extract domains

We use the following script to get domains from the ip range file we got:

#!/bin/bash

ipfile_path=$1

cat $ipfile_path | sort | uniq | \
	awk -F' -- ' 'match($2, /\[(.*?)\]/,a) {print a[1]}' | \
	awk '{print $NF}' | \
	grep '\.org$' | \
	awk -F'.' '{if (NF>1) print $(NF-1) "." $NF}' | \
	sort | uniq

This script produces output for .org domains. Put the script into extract.sh and save the output:

./extract.sh ips/oracle/ipv4_merged_sni.txt > ips/oracle/org/domains.txt

Split

We got 1395 lines in the domains.txt:

cd ips/oracle/org/
wc -l domains.txt
1395

We split the file to chunks of 10 lines:

split --suffix-length=3 --numeric-suffixes=100 -l 10 domains.txt 

which results in

domains.txt  x104  x109  x114  x119  x124  x129  x134  x139  x144  x149  x154  x159  x164  x169  x174  x179  x184  x189  x194  x199  x204  x209  x214  x219  x224  x229  x234  x239
x100         x105  x110  x115  x120  x125  x130  x135  x140  x145  x150  x155  x160  x165  x170  x175  x180  x185  x190  x195  x200  x205  x210  x215  x220  x225  x230  x235
x101         x106  x111  x116  x121  x126  x131  x136  x141  x146  x151  x156  x161  x166  x171  x176  x181  x186  x191  x196  x201  x206  x211  x216  x221  x226  x231  x236
x102         x107  x112  x117  x122  x127  x132  x137  x142  x147  x152  x157  x162  x167  x172  x177  x182  x187  x192  x197  x202  x207  x212  x217  x222  x227  x232  x237
x103         x108  x113  x118  x123  x128  x133  x138  x143  x148  x153  x158  x163  x168  x173  x178  x183  x188  x193  x198  x203  x208  x213  x218  x223  x228  x233  x238

The content of x100:

00x00.org
034548.org
0509xnxx.org
0971.org
099c.org
0n0n.org
0to0.org
11cd.org
12354.org
13jk.org

Finding subdomains

We are going to find the subdomains of the domains in x* files:

for i in $(seq 100 239); do echo Working on x$i; subfinder -dL x$i -silent -o s$i; done

The subdomains will be saved in s* files.

Now, suppose disconnection happens. We reconnect and see there are s100, s101 and s102 files. That means the disconnection happened when x102 was processed. So we re-run one liner from x102:

for i in $(seq 102 239); do echo Working on x$i; subfinder -dL x$i -silent -o s$i; done

Example of the output of the one liner:

Working on x100
www.00x00.org
arm.00x00.org
00x00.org
cpanel.00x00.org
mail.00x00.org
webdisk.00x00.org
webmail.00x00.org
www.034548.org
0509xnxx.org
whm.0971.org
www.0971.org
mail.0971.org
webmail.0971.org
0971.org
dashboard.apps.099c.org
099c.org
wiki.apps.099c.org
vps01.099c.org
traefik.apps.099c.org
gitea.apps.099c.org
ip.099c.org
apps.099c.org
app.099c.org
mail.099c.org
home.099c.org
int.099c.org
nginx.home.099c.org
s.0n0n.org
0n0n.org

After all subdomains are discovered, we unite them:

cat s* > subdomains.txt

There are 76937 lines ini subdomains.txt file, here is a fragment of the file:

webmail.00x00.org
www.00x00.org
arm.00x00.org
00x00.org
cpanel.00x00.org
mail.00x00.org
webdisk.00x00.org
www.034548.org
0509xnxx.org
webmail.0971.org
0971.org
whm.0971.org
www.0971.org
mail.0971.org
099c.org
wiki.apps.099c.org
dashboard.apps.099c.org
traefik.apps.099c.org
gitea.apps.099c.org
ip.099c.org
apps.099c.org
app.099c.org
vps01.099c.org
home.099c.org
mail.099c.org
nginx.home.099c.org
int.099c.org
s.0n0n.org
0n0n.org
drive.0to0.org
nc.0to0.org
0to0.org
www.11cd.org
statics.11cd.org

Note, the subdomains are already sorted.

Port dicovery

For port scanning we use naabu tool and we want to use the same princple of differential jobs. We split subdomains.txt file to many of 10 lines.

split --additional-suffix=m --suffix-length=4 --numeric-suffixes=1000 -l 10 subdomains.txt 

We got xNm files with N=[1000,1001,…,8693], that is 7693 files of 10 lines or 76930 subdomains to conduct port scanning for them. The scan is pretty fast.

We use the following one liner:

for i in$(seq 1000 8693); do echo "Work on x${i}m:"; naabu -l x${i}m -silent -o ports${i}.txt; done

Here is an example of ports1000.txt:

webmail.0971.org:22
webmail.0971.org:143
webmail.0971.org:443
webmail.0971.org:80
webmail.0971.org:587
webmail.0971.org:25
webmail.0971.org:993
webmail.0971.org:995
webmail.0971.org:21
webmail.0971.org:465
webmail.0971.org:110
www.034548.org:443
www.034548.org:80
arm.00x00.org:443
arm.00x00.org:22
arm.00x00.org:111
arm.00x00.org:8443
0509xnxx.org:3306
0509xnxx.org:443
0509xnxx.org:21
0509xnxx.org:22
0509xnxx.org:80

Vulnerability scanning

Now that we have all the information stored in the portsN.txt files, we face a decision: either wait until all the files are ready or initiate scanning on each of the portsN.txt files. However, it’s important to note that some of the portsN.txt files might contain more than 100 lines, especially for hosts with numerous open ports. Consequently, if a disconnection occurs, a significant portion of the work could be lost. Given that the nuclei scanning process is lengthy, losing even a small part of its results is highly undesirable.

So we wait, and when all portsN.txt are available, we unite them:

cat ports*.txt > allports.txt 

We have got 147618 lines in allports.txt file.

Then we split allports.txt to 10-line files as usual:

split --additional-suffix=p --suffix-length=5 --numeric-suffixes=10000 -l 10 allports.txt 

and we got xNp files with N=[10000, 24059]

An example of the x10000p file:

webmail.0971.org:22
webmail.0971.org:143
webmail.0971.org:443
webmail.0971.org:80
webmail.0971.org:587
webmail.0971.org:25
webmail.0971.org:993
webmail.0971.org:995
webmail.0971.org:21
webmail.0971.org:465

Finally, we can start our vulnerability scan with the following one-liner:

for i in $(seq 10000 24059); do echo "Work on x${i}p:"; nuclei -l x${i}p -silent -s high,critical -o vul${i}; done

In this one-liner we search for vulnerabilities of high and critical level.

What next?

We utilize nuclei to scan for high and critical vulnerabilities. However, merely discovering vulnerabilities isn’t enough; we must ascertain if they are genuine or false positives. Even when confirmed as actual vulnerabilities, understanding their usability is crucial. It’s worth noting that many reported vulnerabilities, although they may indeed exist within the scanned system, are exceedingly difficult or practically impossible to exploit. In the upcoming part of this series, we’ll explore some vulnerabilities that are straightforward and practical to address, contrasting them with others that are essentially a waste of time.

See also

• Vulnerability databases
• Practical single domain vulnerability scanning using nuclei, naabu and subfinder
• Non-arabic african population is willing to invest in high-risk crypto and forex platforms (research)
• Demystifying Crypto and Forex CFD Trading: Platforms, FTDs, and Leads Explained
• Tech SEO Pro video course by Kristina Azarenko