Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

Overview

SpiderFoot Neo4j Tools

Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

A big graph

Step 1: Installation

NOTE: This installs the sfgraph command-line utility

$ pip install spiderfoot-neo4j

Step 2: Start Neo4j

NOTE: Docker must first be installed

$ docker run --rm --name sfgraph -v "$(pwd)/neo4j_database:/data" -e 'NEO4J_AUTH=neo4j/CHANGETHISIFYOURENOTZUCK' -e 'NEO4JLABS_PLUGINS=["apoc", "graph-data-science"]' -e 'NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.*' -p "7474:7474" -p "7687:7687" neo4j

Step 3: Import Scans

Spiderfoot scan ID in web browser

$ sfgraph path_to/spiderfoot.db -s   ...

Step 4: Browse Spiderfoot Data in Neo4j

Visit http://127.0.0.1:7474 and log in with neo4j/CHANGETHISIFYOURENOTZUCK Spiderfoot data in Neo4j

Step 5 (Optional): Use cool algorithms to find new targets

The --suggest option will rank nodes based on their connectedness in the graph. This is perfect for finding closely-related affiliates (child companies, etc.) to scan and add to the graph. By default, Harmonic Centrality is used, but others such as PageRank can be specified with --closeness-algorithm

$ sfgraph --suggest DOMAIN_NAME

Closeness scores

Example CYPHER Queries

() RETURN p # shortest path to all INTERNET_NAMEs from seed domain MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p # match only primary targets (non-affiliates) MATCH (n {scanned: true}) return n # match only affiliates MATCH (n {affiliate: true}) return n ">
# match all INTERNET_NAMEs
MATCH (n:INTERNET_NAME) RETURN n

# match multiple event types
MATCH (n) WHERE n:INTERNET_NAME OR n:DOMAIN_NAME OR n:EMAILADDR RETURN n

# match by attribute
MATCH (n {data: "evilcorp.com"}) RETURN n

# match by spiderfoot module (relationship)
MATCH p=()-[r:WHOIS]->() RETURN p

# shortest path to all INTERNET_NAMEs from seed domain
MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p

# match only primary targets (non-affiliates)
MATCH (n {scanned: true}) return n

# match only affiliates
MATCH (n {affiliate: true}) return n

CLI Help

sfgraph [-h] [-db SQLITEDB] [-s SCANS [SCANS ...]] [--uri URI] [-u USERNAME] [-p PASSWORD] [--clear] [--suggest SUGGEST]
               [--closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -db SQLITEDB, --sqlitedb SQLITEDB
                        Spiderfoot sqlite database
  -s SCANS [SCANS ...], --scans SCANS [SCANS ...]
                        scan IDs to import
  --uri URI             Neo4j database URI (default: bolt://127.0.0.1:7687)
  -u USERNAME, --username USERNAME
                        Neo4j username (default: neo4j)
  -p PASSWORD, --password PASSWORD
                        Neo4j password
  --clear               Wipe the Neo4j database
  --suggest SUGGEST     Suggest targets of this type (e.g. DOMAIN_NAME) based on their connectedness in the graph
  --closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}
                        Algorithm to use when suggesting targets
  -v, -d, --debug       Verbose / debug
Owner
Black Lantern Security
Security Organization
Black Lantern Security
Missing data visualization module for Python.

missingno Messy datasets? Missing values? missingno provides a small toolset of flexible and easy-to-use missing data visualizations and utilities tha

Aleksey Bilogur 3.4k Dec 29, 2022
Editor and Presenter for Manim Generated Content.

Editor and Presenter for Manim Generated Content. Take a look at the Working Example. More information can be found on the documentation. These Browse

Manim Community 149 Dec 29, 2022
Debugging, monitoring and visualization for Python Machine Learning and Data Science

Welcome to TensorWatch TensorWatch is a debugging and visualization tool designed for data science, deep learning and reinforcement learning from Micr

Microsoft 3.3k Dec 27, 2022
基于python爬虫爬取COVID-19爆发开始至今全球疫情数据并利用Echarts对数据进行分析与多样化展示。

COVID-19-Epidemic-Map 基于python爬虫爬取COVID-19爆发开始至今全球疫情数据并利用Echarts对数据进行分析与多样化展示。 觉得项目还不错的话欢迎给一个star! 项目的源码可以正常运行,各个库的版本、数据库的建表语句、运行过程中遇到的坑以及解决方式在笔记.md中都

31 Dec 15, 2022
Simple and lightweight Spotify Overlay written in Python.

Simple Spotify Overlay This is a simple yet powerful Spotify Overlay. About I have been looking for something like this ever since I got Spotify. I th

27 Sep 03, 2022
Geocoding library for Python.

geopy geopy is a Python client for several popular geocoding web services. geopy makes it easy for Python developers to locate the coordinates of addr

geopy 3.8k Jan 02, 2023
With Holoviews, your data visualizes itself.

HoloViews Stop plotting your data - annotate your data and let it visualize itself. HoloViews is an open-source Python library designed to make data a

HoloViz 2.3k Jan 02, 2023
Matplotlib tutorial for beginner

matplotlib is probably the single most used Python package for 2D-graphics. It provides both a very quick way to visualize data from Python and publication-quality figures in many formats. We are goi

Nicolas P. Rougier 2.6k Dec 28, 2022
Streamlit dashboard examples - Twitter cashtags, StockTwits, WSB, Charts, SQL Pattern Scanner

streamlit-dashboards Streamlit dashboard examples - Twitter cashtags, StockTwits, WSB, Charts, SQL Pattern Scanner Tutorial Video https://ww

122 Dec 21, 2022
Altair extension for saving charts in a variety of formats.

Altair Saver This packge provides extensions to Altair for saving charts to a variety of output types. Supported output formats are: .json/.vl.json: V

Altair 85 Dec 09, 2022
A tool for creating SVG timelines from simple JSON input.

A tool for creating SVG timelines from simple JSON input.

Jason Reisman 432 Dec 30, 2022
Draw interactive NetworkX graphs with Altair

nx_altair Draw NetworkX graphs with Altair nx_altair offers a similar draw API to NetworkX but returns Altair Charts instead. If you'd like to contrib

Zachary Sailer 206 Dec 12, 2022
plotly scatterplots which show molecule images on hover!

molplotly Plotly scatterplots which show molecule images on hovering over the datapoints! Required packages: pandas rdkit jupyter_dash ➡️ See example.

150 Dec 28, 2022
1900-2016 Olympic Data Analysis in Python by plotting different graphs

🔥 Olympics Data Analysis 🔥 In Data Science field, there is a big topic before creating a model for future prediction is Data Analysis. We can find o

Sayan Roy 1 Feb 06, 2022
A python script to visualise explain plans as a graph using graphviz

README Needs to be improved Prerequisites Need to have graphiz installed on the machine. Refer to https://graphviz.readthedocs.io/en/stable/manual.htm

Edward Mallia 1 Sep 28, 2021
clock_plot provides a simple way to visualize timeseries data, mapping 24 hours onto the 360 degrees of a polar plot

clock_plot clock_plot provides a simple way to visualize timeseries data mapping 24 hours onto the 360 degrees of a polar plot. For usage, please see

12 Aug 24, 2022
These data visualizations were created for my introductory computer science course using Python

Homework 2: Matplotlib and Data Visualization Overview These data visualizations were created for my introductory computer science course using Python

Sophia Huang 12 Oct 20, 2022
Create animated and pretty Pandas Dataframe or Pandas Series

Rich DataFrame Create animated and pretty Pandas Dataframe or Pandas Series, as shown below: Installation pip install rich-dataframe Usage Minimal exa

Khuyen Tran 92 Dec 26, 2022
Simple Inkscape Scripting

Simple Inkscape Scripting Description In the Inkscape vector-drawing program, how would you go about drawing 100 diamonds, each with a random color an

Scott Pakin 140 Dec 27, 2022
Bokeh Plotting Backend for Pandas and GeoPandas

Pandas-Bokeh provides a Bokeh plotting backend for Pandas, GeoPandas and Pyspark DataFrames, similar to the already existing Visualization feature of

Patrik Hlobil 822 Jan 07, 2023