Daft-Exprt - PyTorch Implementation
PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis
The validation logs up to 70K of synthesized mel and alignment are shown below (VCTK_val_p237-088).
Quickstart
DATASET refers to the names of datasets such as VCTK in the following documents.
Dependencies
You can install the Python dependencies with
pip3 install -r requirements.txt
Also, Dockerfile is provided for Docker users.
Inference
You have to download the pretrained models and put them in output/ckpt/DATASET/.
For a multi-speaker TTS, run
python3 synthesize.py --text "YOUR_DESIRED_TEXT" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset DATASET --ref_audio REF_AUDIO
to synthesize speech with the style of input audio at REF_AUDIO. The dictionary of learned speakers can be found at preprocessed_data/VCTK/speakers.json, and the generated utterances will be put in output/result/.
Batch Inference
Batch inference is also supported, try
python3 synthesize.py --source preprocessed_data/DATASET/val.txt --restore_step RESTORE_STEP --mode batch --dataset DATASET
to synthesize all utterances consuming themselves as a reference audio in preprocessed_data/DATASET/val.txt.
Controllability
The pitch/volume/speaking rate of the synthesized utterances can be controlled by specifying the desired pitch/energy/duration ratios. For example, one can increase the speaking rate by 20 % and decrease the volume by 20 % by
python3 synthesize.py --text "YOUR_DESIRED_TEXT" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset DATASET --ref_audio REF_AUDIO --duration_control 0.8 --energy_control 0.8
Training
Datasets
The supported datasets are
- VCTK: The CSTR VCTK Corpus includes speech data uttered by 110 English speakers (multi-speaker TTS) with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.
- Any of multi-speaker TTS dataset (e.g., LibriTTS) can be added following VCTK.
Preprocessing
-
For a multi-speaker TTS with external speaker embedder, download ResCNN Softmax+Triplet pretrained model of philipperemy's DeepSpeaker for the speaker embedding and locate it in
./deepspeaker/pretrained_models/. -
Run
python3 prepare_align.py --dataset DATASETfor some preparations.
For the forced alignment, Montreal Forced Aligner (MFA) is used to obtain the alignments between the utterances and the phoneme sequences. Pre-extracted alignments for the datasets are provided here. You have to unzip the files in
preprocessed_data/DATASET/TextGrid/. Alternately, you can run the aligner by yourself.After that, run the preprocessing script by
python3 preprocess.py --dataset DATASET
Training
Train your model with
python3 train.py --dataset DATASET
TensorBoard
Use
tensorboard --logdir output/log
to serve TensorBoard on your localhost. The loss curves, synthesized mel-spectrograms, and audios are shown.
Implementation Issues
- RangeParameterPredictor is built with BiLSTM rather than a single linear layer with softplus() activation (it is however implemented and named as 'range_param_predictor_paper' in GaussianUpsampling).
- Use
32batch size instead of48due to memory issues. - Use log duration instead of normal duration.
- Follow FastSpeech2 for the preprocess of pitch and energy.
- Two options for embedding for the multi-speaker TTS setting: training speaker embedder from scratch or using a pre-trained philipperemy's DeepSpeaker model (as STYLER did). You can toggle it by setting the config (between
'none'and'DeepSpeaker'). - DeepSpeaker on VCTK dataset shows clear identification among speakers. The following figure shows the T-SNE plot of extracted speaker embedding.
- For vocoder, HiFi-GAN and MelGAN are supported.
Citation
@misc{lee2021daft_exprt,
author = {Lee, Keon},
title = {Daft-Exprt},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Daft-Exprt}}
}
References
- keonlee9420's WaveGrad2 for GaussianUpsampling and RangeParameterPredictor
- keonlee9420's STYLER for the (domain) adversarial training of SpeakerClassifier
- keonlee9420's StyleSpeech for reference auido interface
- FiLM: Visual Reasoning with a General Conditioning Layer
- TADAM: Task dependent adaptive metric for improved few-shot learning
![FileNotFoundError: [Errno 2] No such file or directory: './preprocessed_data/VCTK/spker_embed/p317-spker_embed.npy'](https://avatars.githubusercontent.com/u/18510825?v=4)
17 Dec 12, 2022
75 Dec 12, 2022
1.8k Dec 30, 2022
41 Oct 29, 2022
54 Nov 21, 2022
44 Nov 20, 2022
21 Mar 03, 2022
41 Dec 12, 2022
83 Dec 21, 2022
10 Nov 14, 2022
4 Nov 27, 2022
1.7k Dec 22, 2022
43 Nov 08, 2022
35 Sep 14, 2022
541 Nov 27, 2022
432 Dec 17, 2022
7 Sep 29, 2022
2.4k Dec 29, 2022
44 Oct 20, 2022
158 Dec 31, 2022