Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service

Last update: Dec 20, 2022

Overview

hashlookup-forensic-analyser

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

Usage

usage: hashlookup-analyser.py [-h] [-v] [-d DIR] [--print-all] [--print-unknown] [--include-stats]

Analyse a forensic target to find and report files found and not found in hashlookup CIRCL public service

optional arguments:
  -h, --help         show this help message and exit
  -v, --verbose      Verbose output
  -d DIR, --dir DIR  Directory to analyse
  --print-all        Print all files result including known and unknown
  --print-unknown    Print all files unknown to hashlookup service
  --include-stats    Include statistics in the CSV export

Example

[email protected] ~/git/hashlookup-forensic-analyser/bin $ python3 hashlookup-analyser.py --print-all -d /usr/local/bin/ --include-stats
unknown,/usr/local/bin/octopress
unknown,/usr/local/bin/safe_yaml
unknown,/usr/local/bin/bayes.rb
unknown,/usr/local/bin/redcarpet
unknown,/usr/local/bin/listen
unknown,/usr/local/bin/f2py
unknown,/usr/local/bin/f2py3.8
unknown,/usr/local/bin/tabulate
unknown,/usr/local/bin/jekyll
unknown,/usr/local/bin/pdf2txt.py
unknown,/usr/local/bin/rougify
unknown,/usr/local/bin/summarize.rb
unknown,/usr/local/bin/camelot
unknown,/usr/local/bin/kramdown
unknown,/usr/local/bin/posix-spawn-benchmark
unknown,/usr/local/bin/f2py3
unknown,/usr/local/bin/__pycache__/dumppdf.cpython-38.pyc
unknown,/usr/local/bin/__pycache__/pdf2txt.cpython-38.pyc
known,/usr/local/bin/scss
known,/usr/local/bin/sass-convert
known,/usr/local/bin/dumppdf.py
known,/usr/local/bin/sass
stats,Analysed directory /usr/local/bin/ on kolmogorov running Linux-5.10.0-1045-oem-x86_64-with-glibc2.29 at 2021-10-03 10:09:18.254424+00:00- Found 4 on hashlookup.circl.lu - Unknown files 18 - Excluded files 0

License

The software is open source software released under the "Simplified BSD License".

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Comments

Added some python tests using github actions
added some tests in github actions such as :

> bandit --recursive --skip B106,B110,B404,B602,B603,B607,B303 > black --check > flake8 . --count --ignore=B,E203,E722,W503,W605 --max-complexity=28 --max-line-length=220 --show-source --statistics > isort --check-only --profile black > mypy --allow-redefinition --ignore-missing-imports --install-types --non-interactive

changed the Module name platform to pl because the variable platform has the same name and a test was failed. bin/hashlookup-analyser.py:20: error: Incompatible types in assignment (expression has type "str", variable has type Module)
opened by signorrayan 4
Stream processing and cached/async lookups?
I wonder if this script might be enhanced for use cases handling some or all of:

large numbers of local objects

high redundancy across execution runs or local folders in a single run

low bandwidth uplinks

@adulau: In case there is interest I am happy to provide a minimally invasive pull request (have to implement before ... of course).

Questions:

Which python version is targeted as bottom? Is it 3.6+ or 3.8 or ...? I do find indicators in the source but not declaration ... I assume it is a version every analyst has on their OS but that I do not know :wink:

Could one use a bulk query endpoint as per "Bulk search of SHA-1 hashes"?

Is this binary linux version compiled from the python source per nuitka, pythran et al. or is this build from some other source language hosted elsewhere?

If it is compiled from python source I would need to know how, so I can provide a compilable version directly and no reviewer needs to hint me at all things breaking in that compilation.
enhancement
opened by sthagen 4
Add support for multiple bloom filter files

This PR adds support for multiple bloom filters via the "--bloomfilters x y z" argument. (cfr https://github.com/hashlookup/hashlookup-forensic-analyser/issues/10)

opened by wllm-rbnt 2

Consistent string interpolation

Why?

Use one and only one approach for string interpolation.

What?

All but one interpolation tasks for strings are solved per f-strings. The requests get call uses the good all format method on strings for construction of the address parameter.

How?

This is just a proposal with single line scope that shall ease maintenance and thus maybe a diff is enough to either implement or kindly ignore the change suggested:

diff --git a/bin/hashlookup-analyser.py b/bin/hashlookup-analyser.py
index f59dfec..e9e1845 100644
--- a/bin/hashlookup-analyser.py
+++ b/bin/hashlookup-analyser.py
@@ -36,7 +36,7 @@ if not args.dir:
 def lookup(value=None):
     if value is None:
         return False
-    r = requests.get('https://hashlookup.circl.lu/lookup/sha1/{}'.format(value), headers=headers)
+    r = requests.get(f'https://hashlookup.circl.lu/lookup/sha1/{value}', headers=headers)
     return r.json()

opened by sthagen 2

Stuck at named pipes

file=57C8EDB95DF3F0AD4EE2DC2B8CFD4157, mode=4607, finfo=os.stat_result(st_mode=4607, st_ino=10480, st_dev=1793, st_nlink=2, st_uid=0, st_gid=0, st_size=0, st_atime=1612118667, st_mtime=1612118667, st_ctime=1612118667)

file 57C8EDB95DF3F0AD4EE2DC2B8CFD4157 57C8EDB95DF3F0AD4EE2DC2B8CFD4157: fifo (named pipe)

opened by michael-hamm 1
Missing dependency declaration of `pytz`
Why?

Without additionally installing pytz from pypi.org per pip the script does fail to start.

What?

The pytzpackage is not part of the standard install of python but instead hosted on the Python Package Index at pypi.org/project/pytz

Example failure:

$ python bin/hashlookup-analyser.py --include-stats -d . -v Traceback (most recent call last): File "bin/hashlookup-analyser.py", line 11, in <module> import pytz ModuleNotFoundError: No module named 'pytz'

How?

Add the pytz dependency to the requirements file and maybe add an install section to the documentation.

The latter might go like this:

Install

$ pip install -r REQUIREMENTS
opened by sthagen 1
Add option to list files that have known hash, but for which the filename doesn't match any of the known filenames for that hash

Some attack techniques replace a 'known' file by another 'known' file, allowing them to exploit some processing flow that triggers the binary at the target location

This kind of scenario could be detected with this proposed new functionality

opened by Wachizungu 0
[Idea] Option to unarchive

This is not an easy suggestion and it may also introduce risks but having an option to get archived files to get extracted recursively & hashed, I thought could also be an interesting add to hashlookup :)

(E.g of lib - https://githubplus.com/enzok/sflock or 7zip etc.)
enhancement

opened by Maijin 5

Releases(v1.1)

v1.1(Jul 9, 2022)

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

This release includes the ability to load multiple Bloom filters at the same time. Thanks @wllm-rbnt for the contribution.
Source code(tar.gz)
Source code(zip)
v1.0(May 7, 2022)
A new version of the hashlookup-forensic-analyser has been released including a new --live-linux option to verify running process on a Linux machine if these are known or unknown binaries on hashlookup service.

This version 1.0 is considered as stable following the feedback received by the different users. Don't hesitate to provide feedback or ideas by opening issues.

New --live-linux option

python3 hashlookup-analyser.py --cache --print-unknown --live-linux. The option is actually reading the /proc directory to find all running processes and due to the fact the ./exe file in proc directory is the actual file, you can quickly find the known/unkown files from the running processes.

Complete changelog available below:

v1.0 (2022-05-07)

New

[progress] disable progress option --disable-progress as progress is by default nowadays. [Alexandre Dulaunoy]

[live_linux] new --live-linux option to display known and unknown processes running from a running linux instance. [Alexandre Dulaunoy]

The process hashes are read from the /proc/PID/exe which is the actual file running before being loaded in memory.

Changes

[lint] increase code complexity. [Alexandre Dulaunoy]

[code] black -S [Alexandre Dulaunoy]

[hash] skip bandit check on weak hash. [Alexandre Dulaunoy]

[hashlib] the library is not used for security per se but for forensic and use SHA1 as the lookup service gives more potential results in SHA1 (due to old NSRL dataset) [Alexandre Dulaunoy]

[file] black -S [Alexandre Dulaunoy]

[doc] updated with new options. [Alexandre Dulaunoy]

Fix

[action] remove mypy too many FP ref -> https://github.com/python/mypy/issues/12246. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.9(Feb 18, 2022)
hashlookup-forensic-analyser version 0.9 released with MIME type statistics

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup.

This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

For the curious, 📖 A sample summary report is available.

New

[report] add MIME type statistics. [Alexandre Dulaunoy]

[feature] stats about mime_type added. [Alexandre Dulaunoy]

Changes

[doc] fix the sample report link. [Alexandre Dulaunoy]

[hashlookup-forensic-analyser] add mermaid pie chart for mime types. [Alexandre Dulaunoy]

[doc] sample report updated. [Alexandre Dulaunoy]

[doc] sample report updated. [Alexandre Dulaunoy]

[requirements] filemagic added. [Alexandre Dulaunoy]

Fix

[hashlookup-analyser] blake -S [Alexandre Dulaunoy]

[hashlookup-analyser] add missing type annotation. [Alexandre Dulaunoy]

Other

Set theme jekyll-theme-dinky. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.8(Jan 2, 2022)
hashlookup-forensic-analyser version 0.8 released including a report functionality

A new --report option added to generate a report directory including a markdown summary and a JSON export of the results. A sample report in Markdown is available. The JSON includes all the found and unknown files discovered.

New

[hashlookup] --report option added to generate a report directory including a markdown summary and a JSON export of the results. [Alexandre Dulaunoy]

Changes

[doc] cleanup. [Alexandre Dulaunoy]

[doc] add sample gist page of a summary report. [Alexandre Dulaunoy]

Fix

[hashlookup] fix headline report typo. [Alexandre Dulaunoy]

[hashlookup] shadowed variable removed. [Alexandre Dulaunoy]

[hashlookup] shadowed loop variable removed. [Alexandre Dulaunoy]

Other

Update README.md. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.7(Dec 28, 2021)
hashlookup-forensic-analyser version 0.7 released including various bugs fixed

Changes

[hashlookup] if a file cannot be read (e.g. Permission denied) for hashing. [Alexandre Dulaunoy]

Those are skipped, accounted in the nonanalysed files.

[ci] updated. [Alexandre Dulaunoy]

Fix

[bug] better handling of filename and CSV export (fix issue with comma separated filenames) [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)

v0.6(Dec 24, 2021)

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

usage: hashlookup-analyser.py [-h] [-v] [--extended-debug] [--progress] [-d DIR] [--print-all] [--print-unknown] [--include-stats] [--format FORMAT] [--cache] [--bloomfilter BLOOMFILTER]

Analyse a forensic target to find and report files found and not found in hashlookup CIRCL public service.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  --extended-debug      Debug file processed along with the mode and type.
  --progress            Pring progress of the file lookup on stderr.
  -d DIR, --dir DIR     Directory to analyse
  --print-all           Print all files result including known and unknown
  --print-unknown       Print all files unknown to hashlookup service
  --include-stats       Include statistics in the CSV export
  --format FORMAT       Output format (default is CSV)
  --cache               Enable local cache of known and unknown hashes in /tmp/hashlookup-forensic-analyser
  --bloomfilter BLOOMFILTER
                        Specify filename of a bloomfilter in DCSO bloomfilter format

New

[hashlookup] --progress option to display the number of files analysed, excluded, unknown and found in hashlookup. [Alexandre Dulaunoy]

and a completely useless spinner added too.

This fixes #7

Changes

[hashlookup] enable the spinner by default and log on stderr. [Alexandre Dulaunoy]
[doc] README updated with new options. [Alexandre Dulaunoy]
[bin] requests missing. [Alexandre Dulaunoy]
[import] order is important. [Alexandre Dulaunoy]
[workflow] long lines again. [Alexandre Dulaunoy]
[formatting] black -S --target-version py38 [Alexandre Dulaunoy]
[hashlookup] --extended-debug option added to output the filetype. [Alexandre Dulaunoy]

only useful for debugging
[LICENSE] added. [Alexandre Dulaunoy]

Fix

[bin] typo fixed. [Alexandre Dulaunoy]
[hashlookup] exclude char and block device - catch incorrect fstat on size. [Alexandre Dulaunoy]
[chg] excluded named pipe - FIFO. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)

v0.5(Dec 14, 2021)
hashlookup-forensic-analyser version 0.5

Bloom filter support

If you don't want to share your lookups online and do faster lookup, hashlookup provides a bloom filter to download.

The file is around 700MB and can be stored locally in your home directory. hashlookup-analyser works in the same way, --bloomfilter option allows to specify the filename locatoon of the bloom filter.

python3 bin/hashlookup-analyser.py --bloomfilter <yourdir>/hashlookup-full.bloom --include-stats -d /bin

v0.5 (2021-12-14)

New

[doc] add a quick documentation about the bloom filter feature. [Alexandre Dulaunoy]

[bloomfilter] use a bloomfilter source instead of the live request. [Alexandre Dulaunoy]

[email protected] ~/git/hashlookup-forensic-analyser/bin (main)$ python3 hashlookup-analyser.py --cache -d /home/adulau/sample/ --print-unknown --bloomfilter /tmp/x.bloom --include-stats hashlookup_result,filename,sha-1,size unknown,/home/adulau/sample/xxx,278DFA419DC12FB8CD9785A873956885BBE11779,17 unknown,/home/adulau/sample/vi,DFA8A9B6417CE3C43D91B0E69E68C26A0230DD48,3194152 stats,Analysed directory /home/adulau/sample/ on kolmogorov running Linux-5.10.0-1052-oem-x86_64-with-glibc2.29 at 2021-12-06 06:48:56.892009+00:00- Found 1 on hashlookup.circl.lu (hashlookup-blomfilter)- Unknown files 2 - Excluded files 0

New: [gh] based on the pull-request #6 from @signorrayan and feedback from @sthagen. [Alexandre Dulaunoy]

Changes

[workflow] we have big screens nowadays. [Alexandre Dulaunoy]

[hashlookup] black -S --target-version py38 [Alexandre Dulaunoy]

[doc] --bloomfilter option added. [Alexandre Dulaunoy]

[doc] updated with new --cache option. [Alexandre Dulaunoy]

[bin] CSV now includes SHA-1 of the files. [Alexandre Dulaunoy]

[bin] new --cache feature added. [Alexandre Dulaunoy]

if cache is called, this will create a file per hash in a fixed directory. at each run, if --cache is called, it will check the existence of a file. The directory contains the JSON result saved from the previous hashlookup query of the server.

[README] a status badge added. [Alexandre Dulaunoy]

[bin] make isort happy. [Alexandre Dulaunoy]

[bin] make black happy. [Alexandre Dulaunoy]

[bin] consistency update fix #4. [Alexandre Dulaunoy]

Fix

[doc] bloom filter link fixed. [Alexandre Dulaunoy]

[bloomfilter] bug fix as bloomfilter just returns a True or False. [Alexandre Dulaunoy]

[bin] remove ambiguity on file handle type. [Alexandre Dulaunoy]

[bin] BinaryIO and not TextIO while reading file. [Alexandre Dulaunoy]

[bin] blake formatting (need a post-hook ;-) [Alexandre Dulaunoy]

[bin] skip type test on empty dict. [Alexandre Dulaunoy]

[bin] incorrect version. [Alexandre Dulaunoy]

[bin] as referenced in #6 - platform assignment can lead to some confusion. [Alexandre Dulaunoy]

[REQUIREMENTS] pytz dep was missing fix #5. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.2(Oct 16, 2021)
hashlookup-forensic-analyser is a script to analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

What's Changed

Clarification regarding the CSV output format

Size of the file analysed added in the output

Improved performance - reduce lines by @signorrayan in https://github.com/hashlookup/hashlookup-forensic-analyser/pull/1

Pre-built Linux binary of the script added (if Python is not present on the target system to be analysed)

New Contributors

@signorrayan made their first contribution in https://github.com/hashlookup/hashlookup-forensic-analyser/pull/1

Screenshot

Full Changelog: https://github.com/hashlookup/hashlookup-forensic-analyser/compare/v0.1...v0.2
Source code(tar.gz)
Source code(zip)
v0.1(Oct 3, 2021)

hashlookup forensic analyser - version 0.1 release

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation. The CIRCL public service already includes multiple sources such as Linux distribution, NIST NSRL and many others.

The tool is written in Python 3 using a minimal set of dependencies.

Source code(tar.gz)
Source code(zip)

Owner

hashlookup

GitHub Repository

A small Python Script To get all levels of subdomains from a list

getlevels A small Python Script To get all levels of subdomains Easily get 1st level, 2nd level, 3rd level, 4th level .... nth level subdomains Usag

9 Feb 15, 2022

PySharpSphere - Inspired by SharpSphere, just another python version

PySharpSphere Inspired by SharpSphere, just another python version. Installation python3 setup.py install Features Support control both Linux and Wind

191 Dec 22, 2022

This Repository is an up-to-date version of Harvard nlp's Legacy code and a Refactoring of the jupyter notebook version as a shell script version.

17 Sep 25, 2022

A small POC plugin for launching dumpulator emulation within IDA, passing it addresses from your IDA view using the context menu.

Dumpulator-IDA Currently proof-of-concept This project is a small POC plugin for launching dumpulator emulation within IDA, passing it addresses from

9 Sep 21, 2022

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service

Related tags

Overview

hashlookup-forensic-analyser

Usage

Example

License

Comments

Why?

What?

How?

Why?

What?

How?

Install

Releases(v1.1)

v1.1(Jul 9, 2022)

v1.0(May 7, 2022)

New --live-linux option

v1.0 (2022-05-07)

New

Changes

Fix

v0.9(Feb 18, 2022)

hashlookup-forensic-analyser version 0.9 released with MIME type statistics

New

Changes

Fix

Other

v0.8(Jan 2, 2022)

hashlookup-forensic-analyser version 0.8 released including a report functionality

New

Changes

Fix

Other

v0.7(Dec 28, 2021)

hashlookup-forensic-analyser version 0.7 released including various bugs fixed

Changes

Fix

v0.6(Dec 24, 2021)

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

New

Changes

Fix

v0.5(Dec 14, 2021)

hashlookup-forensic-analyser version 0.5

Bloom filter support

v0.5 (2021-12-14)

New

Changes

Fix

v0.2(Oct 16, 2021)

What's Changed

New Contributors

Screenshot

v0.1(Oct 3, 2021)

hashlookup forensic analyser - version 0.1 release

Owner

hashlookup

A small Python Script To get all levels of subdomains from a list

PySharpSphere - Inspired by SharpSphere, just another python version

This Repository is an up-to-date version of Harvard nlp's Legacy code and a Refactoring of the jupyter notebook version as a shell script version.

A small POC plugin for launching dumpulator emulation within IDA, passing it addresses from your IDA view using the context menu.

Python bindings to LibreSSL library

client attack remotely , this script was written for educational purposes only

A simple password generator using Python Tkinter.

A simple multi-threaded distributed SSH brute-forcing tool written in Python.

⛤Keylogger Generator for Windows written in Python⛤

👑 Discovery Header DoD Bug-Bounty

Sudo Baron Samedit Exploit

web指纹识别工具

带回显版本的漏洞利用脚本

A Python wrapper around the OpenSSL library

Tools Crack Fb Terbaru

🔐 A simple command-line password manager.

Proof of concept of CVE-2022-21907 Double Free in http.sys driver, triggering a kernel crash on IIS servers

Webpack自动化信息收集

Encrypted Python Password Manager

Raphael is a vulnerability scanning tool based on Python3.

New `--live-linux` option