This is an example of building a video Question-Answer system using Jina.

Related tags

Videoexample-video-qa
Overview

example-video-search

This is an example of building a video Question-Answer system using Jina.

The index data is subtitle files of YouTube videos. After indexing, you can query with questions in natural langurage and retrieve the related video together with the timestamp that the corresponding answer appears.

Prerequisites

pip install -r requirements.txt

Usage

By default, we index the subtitle file, toy-data/zvXkQkqd2I8.vtt

python app.py -m index

Query with questions,

python app.py -m query

To start the video UI, run the following codes and open http://localhost:3000/video/ in your browser.

git clone https://github.com/jina-ai/jina-ui.git
cd jina-ui
git checkout showcase-video-search
yarn install
yarn jinajs build
yarn showcases dev

How it works

The index flow is as below. The sentences are extracted from the subtitle file. The meta information of the sentences are stored in the LMDBStorage. In the other pathway, the sentences of the subtitles are encoded by the DPRTextEncoder and afterwards the result embeddings are stored in the SimpleIndexer.

The query flow is as shown below.

  1. The input query is a question which is encoded into embeddings by using DPRTextEncoder.
  2. The embedding of the query question is used to retrieve the sentences from SimpleIndexer.
  3. After retrieving the indexed sentences, we use MatchExpander to extend the questions so that the sentences from the neighboring timestamps are considered as candidate sentences as well.
  4. The candidate sentences retrieved from the above steps have no text or meta information. We retrieve the meta information from LMDBStorage.
  5. Rank the candidate sentences and extract the exact answers from the sentences by using DPRReaderRanker.
  6. Get the timestamp and video uri information about the answer candidates with Text2Frame

How to index my own data? [W.I.P.]

  1. download the subtitle files
youtube-dl --write-auto-sub --skip-download https://www.youtube.com/watch\?v\=zvXkQkqd2I8 -o data/zvXkQkqd2I8
  1. run the following
python app.py -m index -d data
python app.py -m query
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
Help for manipulating the plex-media-server transcode on the raspberry pi

raspi-plex-transcode Help for manipulating the plex-media-server transcode on the raspberry pi Ensure hardware decoding works and your firmware is up

10 Sep 29, 2022
This application makes a webrtc video call with jitsi meet signaling

gstreamer-jitsi-meet This application makes a webrtc video call with jitsi meet signaling. Other end can be any jitsi meet app or web app. It doesn't

Linh 7 Apr 26, 2022
Media player custom component which works with MQTT.

Media player custom component which works with MQTT. I designed this to specifically work with a ESP32 which i used to control a speakercraft amp.

2 Feb 10, 2022
Stream-Cli application that allow you to play your favorite movies from the terminal

Stream-Cli application that allow you to play your favorite movies from the terminal

redouane 380 Jan 08, 2023
KonomiTV: Kind and Optimized Next brOadcast watching systeM Infrastructure for TV

備考・注意事項 現在 α 版で、まだ実験的なプロダクトです。通常利用には耐えないでしょうし、サポートもできません。 安定しているとは到底言いがたい品質ですが、それでも構わない方のみ導入してください。 使い方などの説明も用意できていないため、自力でトラブルに対処できるエンジニアの方以外には現状おすすめ

tsukumi 244 Dec 31, 2022
A Simple Telegram Bot By @Tellybots to add Subtitle Files in Video

Video-subtitle-merger A Simple Telegram Bot By @Tellybots to add Subtitle Files in Video Features Force Sub Button Added Soon Support Media Type Such

6 Dec 31, 2021
A platform which give you info about the newest video on a channel

youtube A platform which give you info about the newest video on a channel. This uses web scraping, a better implementation will be to use the API. BR

Custom components for Home Assistant 36 Sep 29, 2022
Add a "flame" effect on each hand's index onto a video stream.

Add a "flame" effect on each hand's index onto a video stream. recording.webm.mov This script is just a quick hack, it's a bit of glue between mediapi

Paul Willot 7 Sep 15, 2022
Video-stream - A telegram video stream bot repo

This is a Telegram Video stream Bot. Binary Tech 💫 Features stream videos downl

silentz lk 1 Feb 02, 2022
Lightweight, zero-dependency proxy and storage RTSP server

python-rtsp-server Python-rtsp-server is a lightweight, zero-dependency proxy and storage server for several IP-cameras and multiple clients. Features

Vlad 39 Nov 23, 2022
Tautulli - A Python based monitoring and tracking tool for Plex Media Server.

Tautulli A python based web application for monitoring, analytics and notifications for Plex Media Server. This project is based on code from Headphon

Tautulli 4.7k Jan 07, 2023
Vigia-youtube - The YouTube Watch bot is able to monitor channels on Google's video platform

Vigia do YouTube O bot Vigia do YouTube é capaz de monitorar canais na plataform

Alessandro Feitosa Jr 10 Oct 03, 2022
All the code in these repos was created and explained by HashLips on the main YouTube channel.

Welcome to HashLips 👄 All the code in these repos was created and explained by HashLips on the main YouTube channel. To find out more please visit: ?

HashLips 6.7k Jan 06, 2023
Use ZWO astronomy camera as an IP camera.

ZWO Astronomy Camera as IP Camera Astronomy cameras are known for their high sensitivity and flexibility on whether to have IR pass through and bayer

Yan Wang 9 Oct 15, 2022
Splat a video into a mosaic by sampling a frame at regular intervals

Splat a video into a mosaic by sampling a frame at regular intervals. Useful for seeing the changes over time of an entire video or movie.

Ryan Fox 4 Oct 16, 2022
Jio TV Server - Watch TV right from your laptop

Jio-PyServer Jio TV - Python Server Watch TV right from your laptop! Requirements: Python 3.X Internet Access A Jio Account Known Issues: Channel Stre

Elvis Tony 11 Apr 05, 2022
Zaid Vc Player Bot For Telegram

Zaid Vc Player Bot For Telegram

1 Dec 04, 2021
Video stream image stacking -- live version

video stream image stacking v2 -- live version A very simple streamed video image stacking code! Version 2.1 left mouse click to select a small region

Chakravarthy Mathiazhagan 1 Jan 03, 2022
plumi video sharing

December 2017 update We are moving tickets from the Plumi tracker (trac.plumi.org) here, for historical reasons. Plumi video sharing system Plumi is a

Plumi 111 Dec 15, 2022
An easy to use GUI based video to image sequence converter (and vice versa).

Vdo & Img Conversion Tools This is a quick conversion tool made with python that can save you a lot of time. With this tool you can extract image sequ

Akash Bora 3 Sep 18, 2022