An enum member is not populated properly when the resulting string has a special character - only tested with a preceding - (e.g. -path), unsure if position matters or if other characters cause the issue. Assuming it is IDA rejecting the string as a valid enum member name.

Sample (BlackMatter v2): https://www.virustotal.com/gui/file/520bd9ed608c668810971dbd51184c6a29819674280b018dc4027bc38fc42e57

Function 0x411464 contains hashes for command line parameter strings that reproduce the issue. The algorithm is add_ror13. The hashes are correctly identified by HashDB, but the string is not added to the hashdb_strings enum.

Tested versions: main branch at 4d73263cffac975e1e47e67eb7fb55bfb85416a2 and release version 1.4.0

IDA Version: Version 7.6.210427 macOS arm64

Python: 3.9.7

Steps to reproduce:

1. Load malshare sample (from OALabs YT Video) https://malshare.com/sample.php?action=detail&hash=132fa71af952927e1961f735e68ae38a3305e7ae8d7197c170d071f74db60d1c

2. Right click on hash 992892395 in sub_4010D7 Select "HashDB Lookup"

Result: IDA crashes with "Oops! internal error 40217 occurred. "

When performing requests the UI (especially with large IAT scans) the UI is frozen and IDA is unusable while the requests are performed. To fix this we should use asyncio's Futures.

Example combined with add_done_callback:

More elegant alternatives would include the use of async libraries such as:

• aiohttp,
• requests-futures,
• grequests, and others.

Popped up when there's more than 1 hash table that matched the hash.

HashDB: Hunting for a hash algorithm, please wait! Timeout: 15 seconds.
CRITICAL:root:hunt_algorithm_request errored: Traceback (most recent call last):
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 333, in __wrapper
    self.__done_callback(*result)
TypeError: hunt_algorithm_done() takes from 0 to 1 positional arguments but 2 were given

ERROR: HashDB hash scan failed: Traceback (most recent call last):
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 333, in __wrapper
    self.__done_callback(*result)
TypeError: hunt_algorithm_done() takes from 0 to 1 positional arguments but 2 were given

ERROR:root:Caught an unexpected exception: TypeError, raising.
Traceback (most recent call last):
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 333, in __wrapper
    self.__done_callback(*result)
TypeError: hunt_algorithm_done() takes from 0 to 1 positional arguments but 2 were given
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\sang\AppData\Local\Programs\Python\Python37\Lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 385, in run
    super().run()
  File "C:\Users\sang\AppData\Local\Programs\Python\Python37\Lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 344, in __wrapper
    raise exception
  File "C:/Users/sang/Downloads/IDA Pro 7.5.20.1028 SP3 Portable + All decompilers (Windows)/IDA Pro 7.5.20.1028 SP3 Portable/plugins/hashdb.py", line 333, in __wrapper
    self.__done_callback(*result)
TypeError: hunt_algorithm_done() takes from 0 to 1 positional arguments but 2 were given

Hi,

Thank you for sharing this fantastic plugin!

I was using hashdb in latest version of ida pro on win 7 x64 python 3.8 the following line of code threw the above error.

HASHDB_REQUEST_TIMEOUT: int | float = 15 # Limit to 15 seconds

I realised its because python 3.8 doesn't support use of '|' as an Union operator.

I guess the work around is to modify the code to fall back to "Union". as I prefer to work with win 7 and the last of python it supports is 3.8 :(

Just wanted to let you guys know in case someone else faced the same issue.

This PR fixes issue #34.

Detailed description:

• added a new import ida_typeinf for typeinf related APIs,
• added a new helper function get_existing_enum_values which checks if an enum already exists and parses its values using the IDA API (ida_typeinf.get_named_type, ida_typeinf.idc_print_type),
• add_enums will now check for existing enum values before parsing the provided hash list,
• add_enums now manually corrects strings with dots in their names (e.g. ntdll.dll -> ntdll_dll),
• add_enums now uses ida_name.get_name_ea to check if a name already exists in the database (previously used ida_enum.add_enum_member which checked this automatically)
• add_enums no longer expects a 3rd function parameter because the enum size now depends on the database type or the algorithm size (64-bit databases will always generate 64-bit enums),
• add_enums: replaced the method used when creating an enum (massive performance improvements); using ida_typeinf.idc_parse_types and ida_typeinf.import_type.

Possible (but unlikely) bugs:

• get_existing_enum_values will not parse values properly on expected integer bases (e.g. if for some reason the values aren't in decimal or hexadecimal form),
• because we no longer use ida_enum.add_enum_member to append enum members we lose some internal corrections that IDA performs on the enum names (e.g. replacing dots with underscores); this has to be corrected manually from now on.

Disclaimer:

I've only tested the plugin on two samples (one 32-bit and one 64-bit). Strings with invalid names (e.g. -path) haven't been tested yet, but should work (look at #31). TLDR; more testing required before we can push a release.

Steps to reproduce:

1. Just placed the script at the plugins and ran refresh nothing special

Stack trace:

{"user_data": {"platform": "win32", "python_version": "3.8.2", "plugin_version": "1.8.0", "ida": {"kernel_version": "7.6", "bits": 32}}, "exception_data": {"exception_type": "UnboundLocalError", "exception_value": "local variable 'algorithms' referenced before assignment"}, "frames": [{"frame_index": 2, "line_number": 570, "function_name": "OnBtnRefresh", "line": "sorted_algorithms = sorted(algorithms, key = lambda algorithm: algorithm[0].lower())", "locals": {"self": "<__plugins__hashdb.hashdb_settings_t object at 0x000002BA20DAA2B0>", "code": "0", "api_url": "'https://hashdb.openanalysis.net'"}}]}

Error Log

  bytes   pages size description
--------- ----- ---- --------------------------------------------
   524288    64 8192 allocating memory for b-tree...
   278528    34 8192 allocating memory for virtual array...
   262144    32 8192 allocating memory for name pointers...
-----------------------------------------------------------------
  1064960            total memory allocated

Loading processor module C:\Program Files\IDA Pro 7.6\procs\pc.dll for metapc...Initializing processor module metapc...OK
Loading type libraries...
Autoanalysis subsystem has been initialized.
....

  Please check the Edit/Plugins menu for more informaton.
================================================================================
   _   _           _    ____________ 
  | | | |         | |   |  _  \ ___ \ 
  | |_| | __ _ ___| |__ | | | | |_/ /
  |  _  |/ _` / __| '_ \| | | | ___ \ 
  | | | | (_| \__ \ | | | |/ /| |_/ /
  \_| |_/\__,_|___/_| |_|___/ \____/ 


HashDB v1.8.0 by @herrcore

HashDB search shortcut key is Alt+`
================================================================================
No saved HashDB configuration
---------------------------------------------------------------------------------------
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] 
IDAPython v7.4.0 final (serial 0) (c) The IDAPython Team <[email protected]>
---------------------------------------------------------------------------------------
ERROR: HashDB API request failed: HTTPSConnectionPool(host='hashdb.openanalysis.net', port=443): Max retries exceeded with url: /hash (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)')))
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 237, in 'calling callback function'
  File "C:\Program Files\IDA Pro 7.6\python\3\ida_kernwin.py", line 9792, in helper_cb
    r = self.handler(button_code)
  File "C:/Program Files/IDA Pro 7.6/plugins/hashdb.py", line 570, in OnBtnRefresh
    sorted_algorithms = sorted(algorithms, key = lambda algorithm: algorithm[0].lower())
UnboundLocalError: local variable 'algorithms' referenced before assignment
ERROR: HashDB API request failed: HTTPSConnectionPool(host='hashdb.openanalysis.net', port=443): Max retries exceeded with url: /hash (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)')))
HashDB settings saved
HashDB: HashDB terminated...
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 237, in 'calling callback function'
  File "C:\Program Files\IDA Pro 7.6\python\3\ida_kernwin.py", line 9792, in helper_cb
    r = self.handler(button_code)
  File "C:/Program Files/IDA Pro 7.6/plugins/hashdb.py", line 570, in OnBtnRefresh
    sorted_algorithms = sorted(algorithms, key = lambda algorithm: algorithm[0].lower())
UnboundLocalError: local variable 'algorithms' referenced before assignment
HashDB configuration cancelled!

• add_enums now takes one additional argument which has the default value of hdb,
• if the suffix isn't empty it's appended to the member_name.

fixes #23

Previously we used features from PEP 604, which means our minimum version support was Py 3.10. I've replaced all of the instances of union types with typing.Union, and checked the the language support with the vermin module. At the moment the minimum required version is Py 3.6 (because of fstrings).

This PR is in reference to issue #13. For detailed information please read the commit messages.

Known bugs:

• 32-bit IDA can't add 64-bit hash values to enums (ida_enum.add_enum_member is broken, I'll contact Hex-Rays about it).

Previously, when scanning for hashed dynamic imports we would check if the database is 32 or 64-bit, which could break hash resolution if the binary was 64-bit, but the hash was 32-bit (see OALabs/hashdb-ida#5).

To fix this issue I've added/removed/changed the following:

• The plugin now has a new constant (HASHDB_ALGORITHM_SIZE) to keep track of the size of the hashing algorithm,
• the hashdb_settings_t and hunt_result_form_t dialogues now show the size of the hashing algorithm in bits;
• Removed code duplicates when setting the HASHDB_ALGORITHM to avoid errors in the future - set_algorithm contains all the necessary checks,
• added a check for an edge-case where we would print an empty string in the output window if the API returned an empty string (see an example here) - this would break code generation (enums, etc.),
• added a few TODOs regarding error handling and Python 3 syntax usage due to multiple uncertainties (@herrcore please resolve).

Unfortunately at the moment the plugin does a second API request when executing hunt_algorithm, as the API only responds with the algorithm's name (perhaps we can fix this) - we would also need the size and parse it using the added determine_algorithm_size helper function.

The plugin has been tested with the following samples (provided by @herrcore):

• 32-bit: 1, 2, 3
• 64-bit: 1, a private sample to test 64-bit hashes when scanning for a hashed IAT.

Create a set of manual (ugh) tests that developers can use to test their changes. Since these tests have to be manual we should try to standardize the steps as best as possible.

Example.

• Download binary (Malshare link)
• Open binary in IDA
  • ✅ Did plugin load properly?
• In disassembly window jump to address xxxxxxx and add immediate value xxxxx as XOR key
  • ✅ Was the XOR key correctly set?
• In disassembly window jump to address xxxxxxx and right click on immediate value xxxxxx and run Hash Hunt Algorithm
  • ✅ Was the algorithm xxxxx correctly returned from HashDB?
• In disassembly window jump to address xxxxxxx and select bytes from address xxxxxxx to xxxxxxx
• Right click on selection and choose IAT Scan
  • ✅ Was each DWORD in the selected range correctly identified in HashDB and converted to an enum?

Currently we use get_highlight to pull the selected constant from IDA but this is an issue when the constant is negative. The negative sign is not highlighted in IDA so we don't grab it and misinterpret the constant as positive.

This can be handled elegantly in the disassembly by checking the operands at the selected address for both negative and positive values of the selected constant to determine which one to use... however this doesn't really work when the value is selected in the pseudocode window.

TLDR Issue When we use get_highlight in the pseudocode window to select a constant how can we match that constant with the actual value in the IDA microcode/pseudocode representation? This is my lack of understanding of how to manipulate the pseudocode from python... example code would be much appreciated 🙏

I'm not sure how to use ida python to convert a constant to an enum in the IDA pseudocode view... easy enough to do in the disassembly view but I'm not sure how to find the constant in the decompiled code?

Help appreciated 🙏