The evaluator covering all of the metrics required by tasks within the DUE Benchmark.

Last update: Jan 21, 2022

Related tags

Overview

DUE Evaluator

The repository contains the evaluator covering all of the metrics required by tasks within the DUE Benchmark, i.e., set-based F1 (for KIE), ANLS (used in document VQA), accuracy (including variant used in WTQ), as well as group-based ANLS we proposed for KIE problems with structured output.

Usage

The deval command will be available after the package installation. Every time, it is required to provide input and output files (both in the DU-Schema format) using -o and -r parameters.

Other settings are task-specific and limited to metric (-m) and optional case-insensitiveness (-i). Recommended values of these are:

Dataset	Metric	Case insensitive
DocVQA, InfographicsVQA	ANLS	Yes
Kleister Charity, DeepForm	F1	Yes
PapersWithCode	GROUP-ANLS	Yes
WikiTableQuestions	WTQ	No (handled by metric itself)
TabFact	F1 (obtained value will be equal to Accuracy)	No

Owner

DUE Benchmark

The benchmark consisting of both available and reformulated datasets to measure the end-to-end capabilities of systems in real-world scenarios.

GitHub Repository

Descriptor Vector Exchange

Descriptor Vector Exchange This repo provides code for learning dense landmarks without supervision. Our approach is described in the ICCV 2019 paper

74 Nov 29, 2022

An advanced multi-threaded, multi-client python reverse shell for hacking linux systems. There's still more work to do so feel free to help out with the development. Disclaimer: This reverse shell should only be used in the lawful, remote administration of authorized systems. Accessing a computer network without authorization or permission is illegal.

PwnLnX An advanced multi-threaded, multi-client python reverse shell for hacking linux systems. There's still more work to do so feel free to help out

212 Dec 24, 2022

The best, free, all in one, multichecking, pentesting utility

58 Jan 03, 2023

Pytest-typechecker - Pytest plugin to test how type checkers respond to code

pytest-typechecker this is a plugin for pytest that allows you to create tests t

2 Aug 20, 2022

Language-agnostic HTTP API Testing Tool

Dredd — HTTP API Testing Framework Dredd is a language-agnostic command-line tool for validating API description document against backend implementati

4k Jan 05, 2023

Python version of the Playwright testing and automation library.

🎭 Playwright for Python Docs | API Playwright is a Python library to automate Chromium, Firefox and WebKit browsers with a single API. Playwright del

7.8k Jan 02, 2023

Automatically mock your HTTP interactions to simplify and speed up testing

VCR.py 📼 This is a Python version of Ruby's VCR library. Source code https://github.com/kevin1024/vcrpy Documentation https://vcrpy.readthedocs.io/ R

2.3k Jan 01, 2023

输入Google Hacking语句，自动调用Chrome浏览器爬取结果

Google-Hacking-Crawler 该脚本可输入Google Hacking语句，自动调用Chrome浏览器爬取结果环境配置 python -m pip install -r requirements.txt 下载Chrome浏览器

4 Jun 21, 2022

:game_die: Pytest plugin to randomly order tests and control random.seed

pytest-randomly Pytest plugin to randomly order tests and control random.seed. Features All of these features are on by default but can be disabled wi

471 Dec 30, 2022

a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)

pytest-sugar pytest-sugar is a plugin for pytest that shows failures and errors instantly and shows a progress bar. Requirements You will need the fol

963 Dec 28, 2022

WomboAI Art Generator

WomboAI Art Generator Automate AI art generation using wombot.art. Also integrated into SnailBot for you to try out. Setup Install Python Go to the py

7 Dec 03, 2022

The Penetration Testers Framework (PTF) is a way for modular support for up-to-date tools.

The PenTesters Framework (PTF) is a Python script designed for Debian/Ubuntu/ArchLinux based distributions to create a similar and familiar distribution for Penetration Testing

4.5k Dec 28, 2022

The successor to nose, based on unittest2

Welcome to nose2 nose2 is the successor to nose. It's unittest with plugins. nose2 is a new project and does not support all of the features of nose.

736 Dec 16, 2022

WIP SAT benchmarking tooling, written with only my personal use in mind.

SAT Benchmarking Some early work in progress tooling for running benchmarks and keeping track of the results when working on SAT solvers and related t

1 Dec 26, 2021

GitHub action for AppSweep Mobile Application Security Testing

GitHub action for AppSweep can be used to continuously integrate app scanning using AppSweep into your Android app build process

14 Oct 06, 2022

UUM Merit Form Filler is a web automation which helps automate entering a matric number to the UUM system in order for participants to obtain a merit

About UUM Merit Form Filler UUM Merit Form Filler is a web automation which helps automate entering a matric number to the UUM system in order for par

3 May 31, 2022