Automatically download and crop key information from the arxiv daily paper. (cpu version)

Related tags

DownloaderFocusAX
Overview

FocusAX

按关键词筛选arxiv每日最新paper或从arxiv搜索。

  • 自动下载、获取摘要、自动截取文中表格和图片。

安装必要的环境

  • 安装 paddle
# GPU安装
python3 -m pip install paddlepaddle-gpu==2.1.1 -i https://mirror.baidu.com/pypi/simple

# CPU安装
 python3 -m pip install paddlepaddle==2.1.1 -i https://mirror.baidu.com/pypi/simple
  • 安装 Layout-Parser
=2.2"">
pip3 install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
pip install "paddleocr>=2.2"
  • 按照其他必要的包
pip3 install -r requirements.txt
  • 下载模型权重
  • PubLayNet 下载解压后放置在paperparse目录下。目录结构如下
FocusAX
    - paperparse
        - ppyolov2_r50vd_dcn_365e_publaynet
            - inference.pdiparams
            - inference.pdiparams.info
            - inference.pdmodel
        - ...
    - downloader
        - ...
    - utils
        - ...
    - configs.py
    - focus_daily.py
    - focus_search.py
    - README.py
    - ...

使用教程

  • configs.py :程序参数配置文件
# =============== 网络代理 ================
# proxy = None # 不使用代理
proxy = {"http": "socks5://127.0.0.1:8080", "https": "socks5://127.0.0.1:8080"}
# =============== 保存文件根目录 ================
root_path = "./arxiv"
# =============== DNN模型推理配置信息 ================
threshold = 0.5
enable_mkldnn = True
enforce_cpu = True
thread_num = 4
  • focus_daily.py :按关键字过滤arxiv daily上的文章(仅当日)
if __name__ == '__main__':
    key_words = ['GAN'] # 要包含的关键词
    subject_words = ['ML', 'CV', 'AI']  # 要包含的类别
    start_parse(key_words, subject_words, needPDF=True, needZip=False)
  • focus_search.py :按关键字在arxiv检索
start_parse('Keyword')
  • root_path 目录中将创建新的文件夹保存结果

效果图

每个文件夹中的abs.md文件保留的是当前pdf的介绍,使用Typora等markdown编辑器打开。 image

image

ps:论文排版不规范会导致截图混乱。

其他

Owner
HeoLis
Interesting in generate methods.
HeoLis
YouTube to MP3 or 4, you get to choose...

UTubeToMP YouTube to MP3 or 4, you get to choose... If you don't wanna git clone andor dont wanna install python. Here: Repl.it Instructions: Pretty s

1 Jan 29, 2022
Download the resources of the Blue Archive easily!

blue-archive-bundle-downloader Download the resources of the Blue Archive easily! Known issue In Windows It works only if the console is "fullscreen"

Ryu juheon 7 Apr 08, 2022
Fast TikTok NO Watermark Video Downloader (username or url)

💎 TD [ TikDown v4 ] Star ⭐ if you want more Discord Server * discord.gg/onlp | Waxor#9999 Why not open source anymore ? * BECAUSE PEOPLE SKID, STEA

Tekky 26 Dec 01, 2022
A Celery application to collect data, download media and extract information from social media APIs

Project IBEX A Celery application to collect data, download media and extract information from social media APIs. Requirements You must have a Redis D

ibex 4 Dec 15, 2022
lo2: Simple youtube-dl web frontend

Simple youtube-dl web frontend

Denis Volk 22 Jun 03, 2022
Gogoanime-dl - Gogoanime downloader for downloading anime.

gogoanime-dl With this script, you can download episodes of your favorite anime from Gogoanime. The current site that's developed against is https://w

1 Jan 06, 2022
This simple Python script allows you to download songs on Telegram🌸❤️😁

SongsDownloaderTgBot 📺 YouTube Song Downloader Bot For Telegram 🔮 3X Fast Telethon Based Bot ⚜ Open Source Bot 👨🏻‍💻 Demo : 𝗔𝗻𝗻𝗶𝗲 - 𝗘𝗹𝗶𝘇?

Sehath Perera 23 Dec 03, 2022
YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric

YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric How to Install on Linux: sudo apt install python3 python3-pip git pip install pytube git

7 Nov 24, 2022
Yahoo! Finance next gen python 3 / pandas market data downloader

Yahoo! Finance-ng python3 / pandas market data downloader Ever since Yahoo! finance decommissioned their historical data API, many programs that relie

Pedro Larroy 7 Dec 09, 2022
Download clips from youtube videos with a few clicks and a GUI!

YouClip v2.0.0 Table Of Contents: What Is YouClip Installation Usage Stuff To Fix Changelog What Is YouClip? ! IMPORTANT: The source files are a total

ador 2 Oct 05, 2021
Youtube Downloader by PyTube é uma ferramenta simples com interface gráfica e escrito em python para baixar vídeos e playlists do youtube...

YouTube Downloader by PyTube O que é o YouTube Downloader by PyTube? YouTube Downloader by PyTube é um software simples para baixar vídeos no YouTube

Elizeu Barbosa Abreu 5 Jul 30, 2022
A toolkit to automatically crawl the paper list and download paper pdfs of ACL Ahthology.

ACL-Anthology-Crawler A toolkit to automatically crawl the paper list and download paper pdfs of ACL Anthology

Ray GG 9 Oct 09, 2022
A standalone pytube wrapper for downloading individual videos from YouTube.

pytube-runner This is a Python CLI script for downloading individual videos from YouTube. The pytube project is the core of this runner, so naturally

Shiva 2 Jun 21, 2022
Tool to get Canvas cover videos from Spotify tracks.

Spotify Canvas Downloader Tool to get Canvas cover videos from Spotify tracks. ✨ Try it out Building Clone the repository git clone https://github.com

Gabriel 35 Dec 28, 2022
Ebook downloader built using python

ebook-downloader Getting Started Open a terminal and run the following commands. git clone github.com/georgemunyoro/ebook-downloader cd ./ebook-downlo

George Munyoro 1 Oct 19, 2021
Apple Music Animated Artwork Fetcher

A python script for downloading the animated artwork of an Apple Music album.

bunny 46 Jan 03, 2023
YouTube-Video-Downloader - Download Youtube Videos for free.

YouTube-Video-Downloader Download Youtube Videos for free. Installing Dependencies:- Windows pip install pytube Mac/Linux pip3 install pytube Clonin

Xception Inc. 1 Jan 01, 2022
Python script to download (TCR) genes from IMGT/GENE-DB

IMGTgeneDL 0.1.0 Jamie Heather | CCR @ MGH | 2021 This script provides an alternative way to access TCR and IG genes stored in IMGT/GENE-DB. It's prim

Jamie Heather 1 Mar 30, 2022
A tool written in Python to download all Snapmaps content from a specific location.

snapmap-archiver A tool written in Python to download all Snapmaps content from a specific location.

46 Dec 09, 2022
A manga download script written in python.

manga-dlp python script to download mangas Description A manga download script written in python. It only supports mangadex.org for now. But support f

Ivan Schaller 15 Nov 28, 2022