Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

Overview

Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

forthebadge made-with-python

windows python

screenshot

Finding and downloading deleted Tweets takes a lot of time. Thankfully, with this tool, it becomes a piece of cake! πŸŽ‚

Twayback is a portmanteau of Twitter and the Wayback Machine. Enter your desired Twitter username, and let Twayback do the rest!

Requirements

  • Python 3
  • waybackpack
    • Install: pip install waybackpack or pip3 install waybackpack
    • Link to repo

Features

  • Can download some or all of a user's archived deleted Tweets.
  • Allows custom time range to narrow search for deleted Tweets archived between two dates.
  • Differentiates between accounts that are active, suspended, or don't/no longer exist.
  • Lets you know if a target handle's archived Tweets have been excluded from the Wayback Machine.

Usage

twayback -u USERNAME [OPTIONS]
Example: twayback -u jack

-u, --username        Specify target user's Twitter handle
-from, --fromdate     Narrow search for deleted Tweets *archived* on and after this date
                      (can be combined with -to)
                      (format YYMMDD)
-to, --todate         Narrow search for deleted Tweets *archived* on and before this date
                      (can be combined with -from)
                      (format YYMMDD)

Installation

For Windows only

  1. Download the latest EXE file.
  2. Launch Command Prompt in the EXE file's directory.
  3. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For Windows, Linux, and macOS

  1. Download the latest ZIP file.
  2. Extract ZIP file to a directory of your choice.
  3. Open terminal in that directory.
  4. Run the command pip install -r requirements.txt.
  5. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For more information, check out the Usage section above.

Things to keep in mind

  • Quality of the HTML files depends on how the Wayback Machine saved them. Some are better than others.
  • This tool is best for text. You might have some luck with photos. You cannot download videos.
  • By definition, if an account is suspended or no longer exists, all their Tweets would be considered deleted.
  • Custom date range is not about when Tweets were made, but rather when they were archived. For example, a Tweet from 2011 may have been archived today.

Future plans

  • GUI. This is a biggie. I don't know shit about Python, let alone GUI. But I'm hoping I can design one using Tkinter Designer. But I don't know how I can link actions to buttons and shit like that, that stuff is super foreign to me, so any help is appreciated, it would mean so much.

Plenty of thanks to jsvine for his amazing work on waybackpack. Without it, this tool cannot work nearly as well.

I hope you enjoy my little script. Please use it for good. Whatever you are, be a good one.

Comments
  • AttributeError: 'NoneType' object has no attribute 'getText'

    AttributeError: 'NoneType' object has no attribute 'getText'

    I got two errors:

    1. The first error occured after grabbing links from wayback machine UnboundLocalError: local variable 'wayback_id' referenced before assignment

    2. Second, the error occured when I typed 'text' or 'both' AttributeError: 'NoneType' object has no attribute 'getText'

    please help, thanks.

    bug good first issue 
    opened by adrn-mm 9
  • Error in the process

    Error in the process

    Hello,

    Thanks for the app which helps save a lot of time. I have tried with several users and in various ways and it does not work. I pass the screenshots of the errors.

    Thank you very much.

    image image

    opened by barripdmx 8
  • Twayback Partial Re-Write

    Twayback Partial Re-Write

    Hello,

    We recently came across your project, and thought we could contribute by re-writing some parts of the code. An effort was made to keep the logic and structure of the code the same.

    By our metrics, we have achieved a speed-up of around 20-30% for accounts with over 1000 tweets. We don't have any actual tests written for the script however, so we have attached it here for review.

    A substantial change was moving from Selenium to Playwright. Users need to run 'playwright install' to install the playwright browsers before running the script.

    Another smaller, but notable change is that due to the nature of the as_completed function, it is not possible to show any indicator of progress during the gathering of statuses, and the order in which the tweets are returned is scrambled due to the script processing website information based on what web request finishes first.

    If the style & format of the re-write is acceptable, and the changes proposed are not considered to be critical, we will proceed with creating a pull request.

    twayback2.txt

    opened by AccentuSoft 5
  • Parsing fails when encountering text in cyrillic

    Parsing fails when encountering text in cyrillic

    Hi, when I try to search for deleted tweets from an account that uses cyrillic, the process fails with the following exception:

    Parsing text...: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4/4 [00:07<00:00, 2.00s/it] Traceback (most recent call last): File "twayback.py", line 184, in File "encodings\cp1250.py", line 19, in encode UnicodeEncodeError: 'charmap' codec can't encode character '\u0430' in position 0: character maps to [20476] Failed to execute script 'twayback' due to unhandled exception!

    Anything that could be done on my end? Thanks,

    opened by Traut89now 5
  • KeyError: 'closest' when parsing accounts with larger number of tweets.

    KeyError: 'closest' when parsing accounts with larger number of tweets.

    Hi,

    after the latest update, the tool seems to be working perfectly when scraping accounts with lower number of tweets archived, but when I tried accounts with 1000+ tweets archived, the process failed with: Traceback (most recent call last): File "C:\15tway\twayback B\twayback.py", line 112, in wayback_url = (jsonResponse['archived_snapshots']['closest']['url']) KeyError: 'closest'

    as also seen below on the screenshots.

    image

    opened by Traut89now 4
  • No deleted tweets have been found

    No deleted tweets have been found

    Hi! Does anyone else, after they pull one search of an user, get an "No deleted tweets have been found" when they try to look for another one? I even try with the same one I did first and it pulls the same thing. I even close and open again the terminal but nothing, got the same issue. Was anyone run into this?

    opened by sofiemmc 3
  • Cleaned up code a bit

    Cleaned up code a bit

    No changes to functionality. I went through and added some more comments, renamed variables to be more descriptive and combined some redundancy with list creation

    opened by humandecoded 2
  • Feature proxy

    Feature proxy

    to avoid rate limiting for large groups of tweets added the ability to use proxies with our GET requests. User will need to provide a list of proxy URLs for the script to randomly pull from

    opened by humandecoded 1
  • Twitter Rate Limiting

    Twitter Rate Limiting

    It looks like twitter is throwing up 429 (too many requests) after the first 900 or so hits. According to their API documentation they limit to around 900 hits per 15 minutes. Although we are not using the API, it seems they are putting similar limiting in place per IP.

    I've begun working on a branch that lets users plug in a list of proxies the script will rotate through to avoid 429. This does make the tool less approachable but at this point it's not working on large groups of tweets. Will keep everyone updated

    enhancement 
    opened by humandecoded 1
  • Adding more Detailed information

    Adding more Detailed information

    I am submitting this PR so that newbie cyber people & OSINT Analysts with no python experience can better understand installation and usage.

    • There is no information about using the command git clone in order to clone the repository
    • No information in regards to changing directory | cd
    • No information in regards to the command pip3 install -r requirements.txt
    • No information in regards to the command python3 twayback.py -u username (if we put ourselves in the shoes of anyone without any python knowledge, they will just write in their terminal twayback.py which won't be of any use, it's typically something I would have done in the past as when reading installation & usage instructions I would just copy paste without trying to understand. 🀦

    Hope this helps a little, awesome tool & thanks for putting it out there !

    opened by C3n7ral051nt4g3ncy 1
  • UPDATE requirements.txt

    UPDATE requirements.txt

    Module aiohttp is required

    Traceback (most recent call last):
      File "/home/gdhindii/twayback/twayback.py", line 18, in <module>
        from aiohttp import ClientSession, TCPConnector
    ModuleNotFoundError: No module named 'aiohttp'
    
    opened by sam5epi0l 1
  • Proxy File List

    Proxy File List

    I am not sure this is an issue or just lack of knowledge from my part, I rather suspect it is the latter, however, I am attempting to recover tweets from an account that has a couple of thousand deleted tweets. I eventually encounter a rate limit error as is to be expected. I saw the way to handle requests like these is to use a proxy file. Now I have a .txt file that I added inside the twayback folder with a list of proxies and formatted according to the guidelines:

    url:port url:port url:port

    and so on. I keep getting error for each proxy server so Im not sure if Im doing this right and what the issue could be. It will stay stuck at 0%, switch between proxies until it eventually just gives up with a "" error.

    My request looks like this:

    python3 twayback.py --proxy-file proxyfiles.txt -u USERNAME

    Am I doing this correctly?

    opened by DirkGaston 2
  • it doesn't work anymore

    it doesn't work anymore

    once I launch the app it closes itself, it was working fine before the last update I tried downloading a pervious version and the same thing happened I tried it on 2 devices and it didn't work on any of them

    opened by unaufhaltbar 4
Releases(10-16-22)
Owner
He/Him. Good-for-nothing GitHubber.
A standalone pytube wrapper for downloading individual videos from YouTube.

pytube-runner This is a Python CLI script for downloading individual videos from YouTube. The pytube project is the core of this runner, so naturally

Shiva 2 Jun 21, 2022
Simple Youtube Video Downloader

Simple Youtube Video Downloader Download Youtube video using link and Will output result in D:/ (You can change the path in main.py file) Installation

Hansen Gianto 1 Oct 28, 2021
Desktop utility to download images/videos/music/text from various websites, and more

Desktop utility to download images/videos/music/text from various websites, and more

Kurt Bestor 11.2k Jan 08, 2023
Download images where login is required using har python and js

이미지 λ‹€μš΄λ‘œλ“œ(har, python, js μ‚¬μš©) 둜그인이 ν•„μš”ν•œ μ‚¬μ΄νŠΈμ—μ„œ DevTools둜 이미지λ₯Ό λ‹€μš΄λ°›λŠ” 방법은 쑰금 κΉŒλ‹€λ‘œμ› λ‹€. κ°€μž₯ μ‰½κ²Œ ν•  수 μžˆλŠ” 방법을 μ°Ύμ•„λ³΄μ•˜λ‹€. μ‚¬μš©λ²• F12λ₯Ό 눌러 DevToolsλ₯Ό μ‹€ν–‰ Network νƒ­μœΌλ‘œ 이동 νŽ˜μ΄μ§€ μƒˆλ‘œκ³ μΉ¨

0 Jul 22, 2022
Downloads yiffer.xyz comics as images

yiffer-dl Downloads comics as images from yiffer.xyz.

Maxim 2 Mar 20, 2022
A downloader for Cave Story written in Python

Cave Story Downloader This is a downloader for Cave Story written in Python. Thi

Imsad2 2 Feb 16, 2022
Download your Spotify playlists and songs along with album art and metadata

spotDL Download your Spotify playlists and songs along with album art and metadata The fastest, easiest, and most accurate command-line music download

10.6k Jan 03, 2023
the best video downloader for terminals (currently only compatible with Linux and Windows)

the best video downloader for terminals (currently only compatible with Linux and Windows)

Amaral 2 Oct 14, 2021
Download the resources of the Blue Archive easily!

blue-archive-bundle-downloader Download the resources of the Blue Archive easily! Known issue In Windows It works only if the console is "fullscreen"

Ryu juheon 7 Apr 08, 2022
A Telegram bot to download Subtitle for movies and tv shows.

Subtitle Downloader Bot A Telegram bot to download Subtitle for movies and tv shows. Host on Heroku Configuring Environments API_HASH : Your Telegram

Joy Biswas 15 Nov 12, 2022
A tool to download program information from Bugcrowd, for use by researchers to compare programs they are eligible to participate in

Description bcstats is a tool which allows Bugcrowd researchers to download information about all accessible programs (public and private) into a sing

19 Oct 13, 2022
Utility for downloading works from AO3 (Archive Of Our Own)

froyo A small graphical application for batch downloading works from Archive Of Our Own (AO3). Curate a fic repo of your own today :) Features Batch d

flux 24 Dec 09, 2022
Application Updater using an download link

Application-Updater This tool will update your app using an storage link

ExtremeDev 1 Dec 20, 2021
Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

DeltaLaboratory 2 Nov 16, 2022
Python code to crawl computer vision papers from top CV conferences. Currently it supports CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, SIGGRAPH

Python code to crawl computer vision papers from top CV conferences. Currently it supports CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, SIGGRAPH. It leverages selenium, a website testing framework to crawl

Xiaoyang Huang 39 Nov 21, 2022
Simple package for Sublime Text 4; download URL's for local viewing and editing

URLDownloader This is a simple example package that allows you to easily download the contents of any web URL to edit locally. Given a URL, the packag

Terence Martin 3 Mar 05, 2022
PyQt5 simple files , youtube videos and youtube playlist downloader

PyQt5 simple files , youtube videos and youtube playlist downloader

AmirHossein Mohammadi 7 Jul 25, 2022
Will load an SRC page, logged in with Firefox's cookies imported, and delete all comments from every run

SRCCommentsAutoDeleter Will load an SRC page, logged in with a support browser's cookies, and delete all comments from every run Config is all done in

3 Oct 29, 2021
Youtube_dl_helper - A hacky python script meant to automate the process of downloading mp3 files from YouTube using youtube-dl library

youtube_dl_helper A helper program meant to automate the process of downloading mp3 files from YouTube using youtube-dl library Dependencies In order

Guilherme Bittencourt de Borba 1 Jan 04, 2022
Download h3t4y for later read

h3nt4y_dl Download h3nt4y for later read TαΊ£i h3nt4y về đọc thΓ΄i nΓ o cΓ‘c bαΊ‘n Ζ‘iiiiiiii! (TαΊ£i tα»« h**taivn nhΓ©) Usage: python get_that_ht4i.py New versio

1 Dec 03, 2021