Take a list of domains and probe for working HTTP and HTTPS servers

Related tags

Networkinghttprobe
Overview

httprobe

Take a list of domains and probe for working http and https servers.

Install

▶ go get -u github.com/tomnomnom/httprobe

Basic Usage

httprobe accepts line-delimited domains on stdin:

▶ cat recon/example/domains.txt
example.com
example.edu
example.net
▶ cat recon/example/domains.txt | httprobe
http://example.com
http://example.net
http://example.edu
https://example.com
https://example.edu
https://example.net

Extra Probes

By default httprobe checks for HTTP on port 80 and HTTPS on port 443. You can add additional probes with the -p flag by specifying a protocol and port pair:

▶ cat domains.txt | httprobe -p http:81 -p https:8443

Concurrency

You can set the concurrency level with the -c flag:

▶ cat domains.txt | httprobe -c 50

Timeout

You can change the timeout by using the -t flag and specifying a timeout in milliseconds:

▶ cat domains.txt | httprobe -t 20000

Skipping Default Probes

If you don't want to probe for HTTP on port 80 or HTTPS on port 443, you can use the -s flag. You'll need to specify the probes you do want using the -p flag:

▶ cat domains.txt | httprobe -s -p https:8443

Prefer HTTPS

Sometimes you don't care about checking HTTP if HTTPS is working. You can do that with the --prefer-https flag:

▶ cat domains.txt | httprobe --prefer-https

Docker

Build the docker container:

▶ docker build -t httprobe .

Run the container, passing the contents of a file into stdin of the process inside the container. -i is required to correctly map stdin into the container and to the httprobe binary.

▶ cat domains.txt | docker run -i httprobe 
   

   
Comments
  • Install command not work for macOS Mojave

    Install command not work for macOS Mojave

    I re-install go both from official and with brew installer. But there is no difference.

    After command : go get -u github.com/tomnomnom/httprobe execute nothing is change. There is no output in terminal also.

    opened by cyb3rsalih 8
  • Not download httprobe

    Not download httprobe

    Hi,

    i have a problem with the Httprobe because not download..

    sorry for my english XD

    order for install..

    Apt install golang (working done) now go get -u github.com/tomnomnom/httprobe (not error message, but dont download anything..)

    attached screenshots.

    https://ibb.co/c19P9Gb

    https://ibb.co/QQYXKfF

    upload photos from ImgBB.

    opened by densdiego 3
  • Max concurrency?

    Max concurrency?

    Hello tomnomnom!

    I have a short question, does httprobe have a max value for concurrency?

    i am running it against +200k subdomains and sometimes even more, and it is a little bit slow with -c 50 provided. Is there a max value for the -c flag? any other method to speed this up without breaking everything?

    Have a good day.

    opened by marcelo321 2
  • Httprobe installation issue

    Httprobe installation issue

    go version go1.6.2 linux/amd64

    PATH is set as follows:

    export GOPATH=$HOME/go
    export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
    
    ~/go$ go get -u github.com/tomnomnom/httprobe
    # github.com/tomnomnom/httprobe
    src/github.com/tomnomnom/httprobe/main.go:57: unknown http.Transport field 'MaxIdleConns' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:58: unknown http.Transport field 'IdleConnTimeout' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:64: unknown http.Transport field 'DialContext' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:68: undefined: http.ErrUseLastResponse
    
    opened by tacticthreat 2
  • Adding port templates

    Adding port templates

    Currently, if you want to use httprobe and scan additional ports you have to use -p http:<PORT> which is fine for a few ports but for heavy scanning could become tedious.

    I added two port templates to allow scanning for non-standard HTTP ports more easily. The idea is taken from aquatone.

    The templates I implemented are:

    • xlarge for 80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082, 2087, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711, 4712, 4993, 5000, 5104, 5108, 5800, 6543, 7000, 7396, 7474, 8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8088, 8090, 8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8443, 8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090, 9091, 9200, 9443, 9800, 9981, 12443, 16080, 18091, 18092, 20720, 28017
    • large for 80, 81, 443, 591, 2082, 2087, 2095, 2096, 3000, 8000, 8001, 8008, 8080, 8083, 8443, 8834, 8888

    So if you want to use the xlarge template for example:

    echo api.twitter.com | ./httprobe -p xlarge
    http://api.twitter.com
    https://api.twitter.com
    
    opened by RiccardoAncarani 2
  • Unable to run httprobe from script

    Unable to run httprobe from script

    Hello,

    I have been using httprobe for a while and everything is working fine so I decided to add it to my script, when I run the script I get an httprobe command not found. error. Every other tool in the script ( amass, nmap etc) is working fine and I can still use httprobe perfectly from the terminal in any directory (including the directory the script is in). Any thoughts? Thank you.

    opened by MathematicsUnicorn 1
  • [Feature Request] Additional argument for printing hosts which weren't alive/responded

    [Feature Request] Additional argument for printing hosts which weren't alive/responded

    Hi @tomnomnom,

    Thanks for the amazing tool, It really helps in HTTP probing.

    As mentioned in the title, can we please have an additional argument if a user wants to print out the hosts which didn't respond or were down during the probing? Would help out in domains that point to a STALE record which is terminated etc.

    opened by Anon-Exploiter 1
  • cannot run --prefer-https

    cannot run --prefer-https

    cat abc.txt | httprobe --prefer-https -c 50

    the above command is showing error

    cat abc.txt | httprobe
    #this command works fine though

    I want the result to be either http or https and not both and if https is available I dont want http

    opened by juniorhero 1
  • Read Full Response and Remove Defer Statement

    Read Full Response and Remove Defer Statement

    The isListening function does not reads http bodies which may & do lead to error saying Too many open file descriptors as the connection does not gets closed. Adding an io.Copy to copy the response to ioutil.Discard solves the problem.

    The second thing i did was remove the usage of defer from that code path as calling defer is more expensive than a simple statement.

    opened by Ice3man543 1
  • Automatically figure out whether to prepend http protocol to url or not

    Automatically figure out whether to prepend http protocol to url or not

    Currently, the input url must be in the format:

    google.com
    xyz.com
    

    Pushing code to automatically prepend the HTTP protocol if it is not present, or if its present don't error out, continue with that url.

    All the below URL's are valid input now

    http://google.com
    google.com
    https://google.com
    
    opened by adwaithks 0
  • Better -p Option Formatting

    Better -p Option Formatting

    Hi Tom,

    This pull request addresses the issue that you proposed about specifying a large number of ports being inconvenient given how values passed to the -p option are formatted (#3). The modifications that I made to httprobe allow the following to work:

    $ cat domains.txt | httprobe -p 1234
    $ cat domains.txt | httprobe -p 1234,4321
    $ cat domains.txt | httprobe -p large
    $ cat domains.txt | httprobe -p large,1234
    $ cat domains.txt | httprobe -p http:1234 -p https:1234,4321
    $ cat domains.txt | httprobe -p http:large -p https:xlarge,1234
    

    Determining which ports to probe on each of HTTP and HTTPS now happens only once (when the flags are parsed). Prior to this, httprobe would run a switch case for each host which is a waste of a few CPU cycles (I say this sarcastically :D).

    Also note that I made the relevant changes to the README.md file (including what's mentioned in #23).

    Best Regards, Adam

    opened by NULLHE4D 0
  • add status code filter and follow redirects

    add status code filter and follow redirects

    To avoid getting massive 404 pages on large scopes I added two new flags:

    • -f: Filter the given status code
    • - --follow-redirect Use the default redirection policy, following up to a maximum of 10 times
    opened by itasahobby 0
  • go get: installing executables with 'go get' in module mode is deprecated. 	Use 'go install pkg@version' instead.

    go get: installing executables with 'go get' in module mode is deprecated. Use 'go install [email protected]' instead.

    Help on how to install because even after using the command: go install github.com/tomnomnom/[email protected]

    The is no response from the httprobe in terminal returns: httprobe: command not found

    opened by ZedTechInfo 3
  • Update README.md

    Update README.md

    Installing executables with "go get" in module mode is deprecated. "go install [email protected]" should be used instead. For more information, see https://golang.org/doc/go-get-install-deprecation

    opened by adilsoybali 1
Owner
Tom Hudson
Open-source tool maker, trainer, talker, fixer, eater, not really a sheep. He/him.
Tom Hudson
Very simple and tiny file sharing service written in python

Simple File sharing tool Sharing client usage (You will need to use python3 for linux) main.py --send -f file/path Optionnal flags : pwd : set a passw

2 Jul 26, 2022
ExtDNS synchronizes labeled records in docker-compose with DNS providers.

ExtDNS for docker-compose ExtDNS synchronizes labeled records in docker-compose with DNS providers. Inspired by External DNS, ExtDNS makes resources d

DNTSK 6 Dec 24, 2022
ProtOSINT is a Python script that helps you investigate Protonmail accounts and ProtonVPN IP addresses

ProtOSINT ProtOSINT is a Python script that helps you investigate ProtonMail accounts and ProtonVPN IP addresses. Description This tool can help you i

pixelbubble 249 Dec 23, 2022
A working cloudflare uam bypass !!

Dark Utilities - Cloudflare Uam Bypass Our Website https://over-spam.space/ ! Additional Informations The proxies type are http,https ... You need fas

Inplex-sys 26 Dec 14, 2022
Web-server with a parser, connection to DBMS, and the Hugging Face.

Final_Project Web-server with parser, connection to DBMS and the Hugging Face. Team: Aisha Bazylzhanova(SE-2004), Arysbay Dastan(SE-2004) Installation

Aisha Bazylzhanova 2 Nov 18, 2021
Py script to aid in setting up the boot chime in OpenCore.

BootChime Py script to aid in setting up the boot chime in OpenCore. It does so by helping you locate your IOHDACodecDevices, IOHDACodecAddress values

CorpNewt 7 Sep 19, 2022
Slowloris is basically an HTTP Denial of Service attack that affects threaded servers.

slowrise-ddos-tool What is Slowloris? Slowloris is basically an HTTP Denial of S

DEMON cat 4 Jun 19, 2022
An HTML interface for finetuning the sync map output from aeneas

finetuneas 3.0 finetuneas is a simple HTML interface for fine tuning sync maps output by aeneas Version 3.0 Easier adjusting time: following cells wil

Firat Özdemir 50 Mar 12, 2022
🎥 PYnema is a simple UDP server written in python, allows you to watch downloaded videos.

🎥 PYnema is a simple UDP server written in python, allows you to watch downloaded videos.

Jan Kupczyk 1 Jan 16, 2022
euserv auto-renew script - A Python script which can help you renew your free EUserv IPv6 VPS.

eu_ex eu_ex means EUserv_extend. A Python script which can help you renew your free EUserv IPv6 VPS. This Script can check the VPS amount in your acco

A beam of light 92 Jan 25, 2022
A simple Tor switcher script switches tor nodes in interval of time

Tor_Switcher A simple Tor switcher script switches tor nodes in interval of time This script will switch tor nodes in every interval of time that you

d4rk sh4d0w 2 Nov 15, 2021
This program ingests a Cisco "sh ip arp" as a text file and produces the list of vendors seen in the file

IP-ARP-Vendor_lookup This program ingests a Cisco "sh ip arp" as a text file and produces the list of vendors seen in the file Why? Answers the questi

Stew Alexander 1 Dec 24, 2022
IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

invasion 3 Feb 18, 2022
Openconnect VPN RPi Gateway

Openconnect-VPN-RPi-Gateway See the blog (Chinese) for how to build an Openconne

Zhongze Tang 2 Jan 30, 2022
Python implementation of the IPv8 layer provide authenticated communication with privacy

Python implementation of the IPv8 layer provide authenticated communication with privacy

203 Oct 26, 2022
Easy to use gRPC-web client in python

pyease-grpc Easy to use gRPC-web client in python Tutorial This package provides a requests like interface to make calls to gRPC-Web servers.

Sudipto Chandra 4 Dec 03, 2022
A protocol or procedure that connects an ever-changing IP address to a fixed physical machine address

p0znMITM ARP Poisoning Tool What is ARP? Address Resolution Protocol (ARP) is a protocol or procedure that connects an ever-changing IP address to a f

Furkan OZKAN 9 Sep 18, 2022
Readable, simple and fast asynchronous non-blocking network apps

Fast and readable async non-blocking network apps Netius is a Python network library that can be used for the rapid creation of asynchronous non-block

Hive Solutions 120 Nov 20, 2022
MQTT Explorer - MQTT Subscriber client to explore topic hierarchies

mqtt-explorer MQTT Explorer - MQTT Subscriber client to explore topic hierarchies Overview The MQTT Explorer subscriber client is designed to explore

Gambit Communications, Inc. 4 Jun 19, 2022
Simulate Attacks With Mininet And Hping3

Miniattack Simulate Attacks With Mininet And Hping3 It measures network load with bwm-ng when the net is under attack and plots the result. This demo

Geraked 3 Oct 03, 2022