Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Last update: Mar 15, 2022

Overview

FeatureHasher

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Note, this requires Jina>=2.2.4.

Example

Here I use FeatureHasher to hash each sentence of Pride and Prejudice into a 128-dim vector, and then use .match to find top-K similar sentences.

from jina import Document, DocumentArray, Flow

# load 
   
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').convert_uri_to_text()

# cut into non-empty sentences store in a DA
da = DocumentArray(Document(text=s.strip()) for s in d.text.split('\n') if s.strip())

# use FeatureHasher in a Flow
f = Flow().add(uses='jinahub://FeatureHasher')

embed_da = DocumentArray()
with f:
    f.post('/', da, on_done=lambda req: embed_da.extend(req.docs), show_progress=True)

print('self-matching...')
embed_da.match(embed_da, exclude_self=True, limit=5, normalization=(1, 0))
print('total sentences: ', len(embed_da))
for d in embed_da:
    print(d.text)
    for m in d.matches:
        print(m.scores['cosine'], m.text)
    input()

           [email protected][I]:🎉 Flow is ready to use!
	🔗 Protocol: 		GRPC
	🏠 Local access:	0.0.0.0:52628
	🔒 Private network:	192.168.178.31:52628
	🌐 Public address:	217.70.138.123:52628
⠹       DONE ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 100% ETA: 0 seconds 40 steps done in 1 second
total sentences:  12153
The Project Gutenberg eBook of Pride and Prejudice, by Jane Austen

   
     *** END OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

    
      *** START OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

     
       production, promotion and distribution of Project Gutenberg-tm

      
        Pride and Prejudice

       
         By Jane Austen This eBook is for the use of anyone anywhere in the United States and 
        
          This eBook is for the use of anyone anywhere in the United States and 
         
           by the awkwardness of the application, and at length wholly 
          
            Elizabeth passed the chief of the night in her sister’s room, and 
           
             the happiest memories in the world. Nothing of the past was 
            
              charities and charitable donations in all 50 states of the United

In practice, you can implement matching and storing via an indexer inside Flow. This example is only for demo purpose so any non-feature hashing related ops are implemented outside the Flow to avoid distraction.

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

Owner

Jina AI

ADExplorerSnapshot.py is an AD Explorer snapshot ingestor for BloodHound.

Bandit is a tool designed to find common security issues in Python code.

Source code for "A Two-Stream AMR-enhanced Model for Document-level Event Argument Extraction" @ NAACL 2022

Simple tool to create passwords.

自动化爆破子域名，并遍历所有端口寻找http服务，并使用crawlergo、dirsearch、xray等工具扫描并集成报告；支持动态添加扫描到的域名至任务；

Exploiting CVE-2021-44228 in vCenter for remote code execution and more

Find exposed API keys based on RegEx and get exploitation methods for some of keys that are found

SCodeScanner stands for Source Code scanner where the user can scans the source code for finding the Critical Vulnerabilities.

POC for CVE-2022-1388

Remote control your Greenbone Vulnerability Manager (GVM)

Lnkbomb - Malicious shortcut generator for collecting NTLM hashes from insecure file shares

Simulating Log4j Remote Code Execution (RCE) vulnerability in a flask web server using python's logging library with custom formatter that simulates lookup substitution by executing remote exploit code.

D-810 is an IDA Pro plugin which can be used to deobfuscate code at decompilation time by modifying IDA Pro microcode.

Internal network honeypot for detecting if an attacker or insider threat scans your network for log4j CVE-2021-44228

Dark-Fb No Login 100% safe

Directory Traversal in Afterlogic webmail aurora and pro

Dapunta Multi Brute Force Facebook - Crack Facebook With Login - Free

LittleBrother is a simple parental control application monitoring specific processes on Linux hosts to monitor and limit the play time of children.

Mad Spammer is a python webhook spammer which is very easy and safe to use.

hackinsta: a program to hack instagram