FG-transformer-TTS Fine-grained style control in transformer-based text-to-speech synthesis

Last update: Dec 30, 2022

Related tags

Overview

LST-TTS

Official implementation for the paper Fine-grained style control in transformer-based text-to-speech synthesis. Submitted to ICASSP 2022. Audio samples/demo for our system can be accessed here

Setting up submodules

git submodule update --init --recursive

Get the waveglow vocoder checkpoint from here (This is from the NVIDIA official WaveGlow repo).

Setup environment

See docker/Dockerfile for the packages need to be installed.

Dataset preprocessing

LJSpeech

python preprocess_LJSpeech.py --datadir LJSpeechDir --outputdir OutputDir

VCTK

Get the leading and trailing scilence marks from this repo, and put vctk-silences.0.92.txt in your VCTK dataset directory.

python preprocess_VCTK.py --datadir VCTKDir --outputdir Output_Train_Dir

python preprocess_VCTK.py --datadir VCTKDir --outputdir Output_Test_Dir --make_test_set

--make_test_set: specify this flag to process the speakers in the test set, otherwise only process training speakers.

Training

LJSpeech

python train_TTS.py --precision 16 \
                    --datadir FeatureDir \
                    --vocoder_ckpt_path WaveGlowCKPT_PATH \
                    --sampledir SampleDir \
                    --batch_size 128 \
                    --check_val_every_n_epoch 50 \
                    --use_guided_attn \
                    --training_step 250000 \
                    --n_guided_steps 250000 \
                    --saving_path Output_CKPT_DIR \
                    --datatype LJSpeech \
                    [--distributed]

--distributed: enable DDP multi-GPU training
--batch_size: batch size per GPU, scale down if you train with multi-GPU and want to keep the same batch size
--check_val_every_n_epoch: sample and validate every n epoch
--datadir: output directory of the preprocess scripts

VCTK

python train_TTS.py --precision 16 \
                    --datadir FeatureDir \
                    --vocoder_ckpt_path WaveGlowCKPT_PATH \
                    --sampledir SampleDir \
                    --batch_size 64 \
                    --check_val_every_n_epoch 50 \
                    --use_guided_attn \
                    --training_step 150000 \
                    --n_guided_steps 150000 \
                    --etts_checkpoint LJSpeech_Model_CKPT \
                    --saving_path Output_CKPT_DIR \
                    --datatype VCTK \
                    [--distributed]

--etts_checkpoint: the checkpoint path of pretrained model (on LJ Speech)

Synthesis

We provide examples for synthesis of the system in synthesis.py, you can adjust this script to your own usage. Example to run synthesis.py:

python synthesis.py --etts_checkpoint VCTK_Model_CKPT \
                    --sampledir SampleDir \
                    --datatype VCTK \
                    --vocoder_ckpt_path WaveGlowCKPT_PATH

FG-transformer-TTS Fine-grained style control in transformer-based text-to-speech synthesis

Related tags

Overview

LST-TTS

Setting up submodules

Setup environment

Dataset preprocessing

LJSpeech

VCTK

Training

LJSpeech

VCTK

Synthesis

Owner

Li-Wei Chen

🔥🔥High-Performance Face Recognition Library on PaddlePaddle & PyTorch🔥🔥

A PyTorch implementation of DenseNet.

Pytorch Implementation of DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis (TTS Extension)

CCPD: a diverse and well-annotated dataset for license plate detection and recognition

Ray tracing of a Schwarzschild black hole written entirely in TensorFlow.

Python Algorithm Interview Book Review

Code for the CVPR 2021 paper: Understanding Failures of Deep Networks via Robust Feature Extraction

A Moonraker plug-in for real-time compensation of frame thermal expansion

Pytorch implementations of popular off-policy multi-agent reinforcement learning algorithms, including QMix, VDN, MADDPG, and MATD3.

Implementation of Hire-MLP: Vision MLP via Hierarchical Rearrangement and An Image Patch is a Wave: Phase-Aware Vision MLP.

DI-HPC is an acceleration operator component for general algorithm modules in reinforcement learning algorithms

The official code for paper "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling".

Revisiting Weakly Supervised Pre-Training of Visual Perception Models

Tensorflow 2.x implementation of Panoramic BlitzNet for object detection and semantic segmentation on indoor panoramic images.

SpanNER: Named EntityRe-/Recognition as Span Prediction

Official repository for the ICCV 2021 paper: UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model.

Pytorch Geometric Tutorials

ChainerRL is a deep reinforcement learning library built on top of Chainer.

This code is an unofficial implementation of HiFiSinger.

Official repository for "On Generating Transferable Targeted Perturbations" (ICCV 2021)