Neural Style and MSG-Net

Last update: Dec 21, 2022

Overview

PyTorch-Style-Transfer

This repo provides PyTorch Implementation of MSG-Net (ours) and Neural Style (Gatys et al. CVPR 2016), which has been included by ModelDepot. We also provide Torch implementation and MXNet implementation.

Tabe of content

Real-time Style Transfer using MSG-Net
- Stylize Images using Pre-trained Model
- Train Your Own MSG-Net Model
Slow Neural Style Transfer

MSG-Net

Multi-style Generative Network for Real-time Transfer [arXiv] [project]
Hang Zhang, Kristin Dana

@article{zhang2017multistyle,
	title={Multi-style Generative Network for Real-time Transfer},
	author={Zhang, Hang and Dana, Kristin},
	journal={arXiv preprint arXiv:1703.06953},
	year={2017}
}

Stylize Images Using Pre-trained MSG-Net

Download the pre-trained model

git clone [email protected]:zhanghang1989/PyTorch-Style-Transfer.git
cd PyTorch-Style-Transfer/experiments
bash models/download_model.sh

Camera Demo

python camera_demo.py demo --model models/21styles.model

Test the model

python main.py eval --content-image images/content/venice-boat.jpg --style-image images/21styles/candy.jpg --model models/21styles.model --content-size 1024

If you don't have a GPU, simply set --cuda=0. For a different style, set --style-image path/to/style. If you would to stylize your own photo, change the --content-image path/to/your/photo. More options:
- --content-image: path to content image you want to stylize.
- --style-image: path to style image (typically covered during the training).
- --model: path to the pre-trained model to be used for stylizing the image.
- --output-image: path for saving the output image.
- --content-size: the content image size to test on.
- --cuda: set it to 1 for running on GPU, 0 for CPU.

Train Your Own MSG-Net Model

Download the COCO dataset
```
bash dataset/download_dataset.sh
```
Train the model
```
python main.py train --epochs 4
```

If you would like to customize styles, set --style-folder path/to/your/styles. More options:
- --style-folder: path to the folder style images.
- --vgg-model-dir: path to folder where the vgg model will be downloaded.
- --save-model-dir: path to folder where trained model will be saved.
- --cuda: set it to 1 for running on GPU, 0 for CPU.

Neural Style

Image Style Transfer Using Convolutional Neural Networks by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.

python main.py optim --content-image images/content/venice-boat.jpg --style-image images/21styles/candy.jpg

--content-image: path to content image.
--style-image: path to style image.
--output-image: path for saving the output image.
--content-size: the content image size to test on.
--style-size: the style image size to test on.
--cuda: set it to 1 for running on GPU, 0 for CPU.

Acknowledgement

The code benefits from outstanding prior work and their implementations including:

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images by Ulyanov et al. ICML 2016. (code)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution by Johnson et al. ECCV 2016 (code) and its pytorch implementation code by Abhishek.
Image Style Transfer Using Convolutional Neural Networks by Gatys et al. CVPR 2016 and its torch implementation code by Johnson.

Comments

training new model

@zhanghang1989 I trained a model with three style images. Now, I see eight .model files. Can you please tell me which .model file to use OR how to integrate them to single model file.

Thanks Akash

opened by akashdexati 7
Unable to resume training

Hey,

So I started training a model, but seeing how long it was going to take I wanted to double check I could successfully resume training.

I ran: python3 main.py train --epochs 4 --style-folder images/xmas-styles/ --save-model-dir trained_models/ until it generated the first checkpoint, then I ran python3 main.py train --epochs 4 --style-folder images/xmas-styles/ --save-model-dir trained_models/ --resume trained_models/Epoch_0iters_8000_Sat_Dec__9_18\:10\:43_2017_1.0_5.0.model and waiting for the first feedback report, which was Sat Dec 9 18:17:09 2017 Epoch 1: [2000/123287] content: 254020.831359 style: 1666218.549250 total: 1920239.380609 so it appeared to not have resumed at all.

Also slight side question... Say I train with --epochs 4 til I get final model... If I were to use the last checkpoint before final to resume, but set --epochs 5 or higher, would that work correctly and just keep going through to 5 epochs before generating another final, and have no issues etc?

opened by pingu2k4 6
Temporal coherence?

Have you tried some technique for temporal coherence? If not, would you mind if I ask which one would you recommend or would like to try.

Keep up the good work.

opened by rraallvv 3

vgg16.t7 unhashable type: 'numpy.ndarray'

It's been a while since the last vgg16 issue i found on this "Issues".

So i download the vgg16.t7 from the paper quoted in this github. And i run this command "python main.py train --epochs 4 --style-folder images/ownstyles --save-model-dir own_models --cuda 1" i have put the vgg16.t7 into models folder, it's been detected correctly. However, the following problem happened.

Traceback (most recent call last):
  File "main.py", line 295, in <module>
    main()
  File "main.py", line 41, in main
    train(args)
  File "main.py", line 135, in train
    utils.init_vgg16(args.vgg_model_dir)
  File "C:\Users\user\Prepwork\Cap Project\PyTorch-Multi-Style-Transfer\experiments\utils.py", line 100, in init_vgg16
    vgglua = load_lua(os.path.join(model_folder, 'vgg16.t7'))
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 424, in load
    return reader.read_obj()
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 370, in read_obj
    obj._obj = self.read_obj()
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 385, in read_obj
    k = self.read_obj()
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 386, in read_obj
    v = self.read_obj()
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 370, in read_obj
    obj._obj = self.read_obj()
  File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 387, in read_obj
    obj[k] = v
TypeError: unhashable type: 'numpy.ndarray'

Is there anyway i can fix this? i found in other thread they said replace with another one, but i could not find another one other than from stanford.

Thanks!

opened by fuddyduddy 2

Fix colab notebook
Hi. Made some changes to notebook:

fixed RuntimeError #21, #32, that was fixed in #31 and #37, but not for msgnet.ipynb;

removed unused import torch.nn.functional;

prettified according to pep8;

changed os.system('wget ...') to direct calling !wget ... without importing os module.

Tested in colab (run all), the notebook works as expected without errors.
opened by amrzv 1
Establish Docker directory
What: Establishes a Docker directory with Dockerfile and run script

Why: The original repo was written for an outdated version of PyTorch, which makes it hard to run on modern systems without conflicting with updated versions of the dependencies.

Build the container with

cd Docker docker build -t style-transfer .
opened by ss32 1

Fix compatibility issues with torch==1.1.0

[x] torch.utils.serialization has been removed. torchfile.load should be used instead.
[x] Fix the following error according to suggestions by @alvinwan

RuntimeError: Error(s) in loading state_dict for Net:
	Unexpected running stats buffer(s) "model1.1.running_mean" and "model1.1.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.

opened by jianchao-li 1

set default values

Hi,

I try run the camera.py with the arguments discribed in the docs , but fail because inside the code dont have values for args.demo_size and img.copy too Whats the default values for set these variables?

Thank you

opened by gledsoul 1
Super Slow at optim on linux Mint

Have this on a fresh install of linux Mint. I'm running the example, 'python main.py optim --content-image images/content/venice-boat.jpg --style-image images/21styles/candy.jpg' and its taking FOREVER to do anything. I used to have it working at a decent speed on Ubuntu on the same hardware.

When inspecting GPU and CPU usage, I see it start off with minimal GPU usage, and huge CPU usage. it slowly increases GPU usage over time until it has enough and then completes the rest in around the same time as before. As an example, it takes around 8 minutes to figure out that there isn't enough VRAM for the selected image size, whereas previously on my Ubuntu installation that would take a matter of seconds. Any idea why it would take so much longer on Mint? And what I can do to remedy this?

opened by pingu2k4 1

"TypeError: 'torch.FloatTensor' object is not callable" running demo on CPU

Sorry if I'm missing something, I'm unfamiliar with PyTorch. I'm running the demo on CPU on a Mac and getting the following error:

  File "camera_demo.py", line 93, in <module>
    main()
  File "camera_demo.py", line 90, in main
    run_demo(args, mirror=True)
  File "camera_demo.py", line 60, in run_demo
    simg = style_v.data().numpy()
TypeError: 'torch.FloatTensor' object is not callable

Thanks.

opened by Carmezim 1

optim with normal RAM?

Hi,

So I spent around 24 hours so far training a model on my style images, got the results out by using the model on eval and so far they're not great. When I use the optim function with the styles however the results are pretty decent, however I am limited by my VRAM which is 6GB as to what size images I can output. Having a lot more RAM available, I was hoping I could do pretty decently sized images, but it seems that I can only get much larger images with eval. Does eval use normal RAM instead of VRAM?

I will continue training my model so that I can use eval in the future, whether I can do larger images with optim or not, but no idea how much more training is required to make it anywhere near a respectable result.

What sort of overall loss value should I be aiming for? Does the number of style images make a difference to what I should expect?

opened by pingu2k4 1

Error Training TypeError: 'NoneType' object is not callable

I was able to get my environment setup successfully to run eval; however, now, trying train I'm running into an issue. Not sure if it's a syntax issues or if something else is going on? You help is greatly appreciated.


#!/bin/bash
#SBATCH --job-name=train-pytorch
#SBATCH --mail-type=END,FAIL
#SBATCH [email protected]
#SBATCH --ntasks=1
#SBATCH --time=00:10:00
#SBATCH --mem=8000
#SBATCH --gres=gpu:p100:2
#SBATCH --cpus-per-task=6
#SBATCH --output=%x_%j.log
#SBATCH --error=%x_%j.err

source ~/scratch/moldach/PyTorch-Style-Transfer/experiments/venv/bin/activate

python main.py train \
  --epochs 4 \
  --style-folder /scratch/moldach/PyTorch-Style-Transfer/experiments/images/9styles \
  --vgg-model-dir vgg-model/ \
  --save-model-dir checkpoint/

/scratch/moldach/first-order-model/venv/lib/python3.6/site-packages/torchvision/transforms/transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
  "please use transforms.Resize instead.")
Traceback (most recent call last):
  File "main.py", line 295, in <module>
    main()
  File "main.py", line 41, in main
    train(args)
  File "main.py", line 135, in train
    utils.init_vgg16(args.vgg_model_dir)
  File "/scratch/moldach/PyTorch-Style-Transfer/experiments/utils.py", line 102, in init_vgg16
    for (src, dst) in zip(vgglua.parameters()[0], vgg.parameters()):
TypeError: 'NoneType' object is not callable

`pip freeze`:

$ pip freeze
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx2
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic
cffi==1.11.5
cloudpickle==0.5.3
cycler==0.10.0
dask==0.18.2
dataclasses==0.8
decorator==4.4.2
future==0.18.2
imageio==2.9.0
imageio-ffmpeg==0.4.3
kiwisolver==1.3.1
matplotlib==3.3.4
networkx==2.5
numpy==1.19.1
pandas==0.23.4
Pillow==8.1.2
pycparser==2.18
pygit==0.1
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2018.5
PyWavelets==1.1.1
PyYAML==5.1
scikit-image==0.17.2
scikit-learn==0.19.2
scipy==1.4.1
six==1.15.0
tifffile==2020.9.3
toolz==0.9.0
torch==1.7.0
torchfile==0.1.0
torchvision==0.2.1
tqdm==4.24.0
typing-extensions==3.7.4.3

opened by moldach 4

Color produced by eval doesn't match demo

Hi ! Thanks for sharing the code. I've ran the eval program using the defaults provided and I noticed the color tends to be much dimmer than what is shown on the homepage here. Is there something that I am missing? The command I used was

python main.py --style-image ./images/21styles/udnie.jpg --content-image ./images/content/venice-boat.jpg

opened by clarng 1
struct.error: unpack requires a buffer of 4 bytes

Dear author, Thank you so much for sharing a useful code. I able to run your evaluation code, but face the following error during runing of training code: File "main.py", line 41, in main train(args) File "main.py", line 135, in train utils.init_vgg16(args.vgg_model_dir) File "/home2/st118370/models/PyTorch-Multi-Style-Transfer/experiments/utils.py", line 100, in init_vgg16 vgglua = load_lua(os.path.join(model_folder, 'vgg16.t7')) File "/home2/st118370/anaconda3/envs/pytorch-py3/lib/python3.7/site-packages/torchfile.py", line 424, in load return reader.read_obj() File "/home2/st118370/anaconda3/envs/pytorch-py3/lib/python3.7/site-packages/torchfile.py", line 310, in read_obj typeidx = self.read_int() File "/home2/st118370/anaconda3/envs/pytorch-py3/lib/python3.7/site-packages/torchfile.py", line 277, in read_int return self._read('i')[0] File "/home2/st118370/anaconda3/envs/pytorch-py3/lib/python3.7/site-packages/torchfile.py", line 271, in _read return struct.unpack(fmt, self.f.read(sz)) struct.error: unpack requires a buffer of 4 bytes

how can i resolve this problem? kindly guide. thanks

opened by MFarooqAit 1
vgg16.t7 unhashable type: 'numpy.ndarray

hi

I have put the vgg16.t7 into models folder, it's been detected correctly. However, the following problem happened.

Traceback (most recent call last): File "main.py", line 295, in main() File "main.py", line 41, in main train(args) File "main.py", line 135, in train utils.init_vgg16(args.vgg_model_dir) File "C:\Users\user\Prepwork\Cap Project\PyTorch-Multi-Style-Transfer\experiments\utils.py", line 100, in init_vgg16 vgglua = load_lua(os.path.join(model_folder, 'vgg16.t7')) File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 424, in load return reader.read_obj() File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 370, in read_obj obj._obj = self.read_obj() File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 385, in read_obj k = self.read_obj() File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 386, in read_obj v = self.read_obj() File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 370, in read_obj obj._obj = self.read_obj() File "C:\Users\user\anaconda3\envs\FTDS\lib\site-packages\torchfile.py", line 387, in read_obj obj[k] = v TypeError: unhashable type: 'numpy.ndarray'

It does't work for pytorch-1.0.0 and 1.4.0, and giving the same error, how to deal with it? thanks !

opened by Gavin-Evans 13
Different brush stroke size

In your paper you wrote about the ability to train the model with different sizes of the style images to later get control over the brush stroke size. Did you implement this in either the pytorch or torch implementation? Greetings and keep up the great work

opened by lpiribauer 0

Releases(v0.1)

v0.1(May 12, 2022)

Add pretrained model weights
Source code(tar.gz)
Source code(zip)
21styles.model(9.16 MB)

Owner

Hang Zhang

Hang Zhang (张航)

GitHub Repository http://hangzh.com/PyTorch-Style-Transfer/

OpenL3: Open-source deep audio and image embeddings

OpenL3 OpenL3 is an open-source Python library for computing deep audio and image embeddings. Please refer to the documentation for detailed instructi

326 Jan 02, 2023

Implementation of Pooling by Sliced-Wasserstein Embedding (NeurIPS 2021)

PSWE: Pooling by Sliced-Wasserstein Embedding (NeurIPS 2021) PSWE is a permutation-invariant feature aggregation/pooling method based on sliced-Wasser

3 May 06, 2022

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

2 Oct 04, 2021

An open source implementation of CLIP.

OpenCLIP Welcome to an open source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training). The goal of this repository is to enable

2.7k Dec 31, 2022

CLIP (Contrastive Language–Image Pre-training) for Italian

Italian CLIP CLIP (Radford et al., 2021) is a multimodal model that can learn to represent images and text jointly in the same space. In this project,

114 Dec 29, 2022

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

The ImageNet-CoG Benchmark Project Website Paper (arXiv) Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalizatio

23 Oct 09, 2022

Official code for "InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization" (ICLR 2020, spotlight)

InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization Authors: Fan-yun Sun, Jordan Hoffm

232 Dec 28, 2022

Experimental Python implementation of OpenVINO Inference Engine (very slow, limited functionality). All codes are written in Python. Easy to read and modify.

PyOpenVINO - An Experimental Python Implementation of OpenVINO Inference Engine (minimum-set) Description The PyOpenVINO is a spin-off product from my

7 Oct 31, 2022

A script helps the user to update Linux and Mac systems through the terminal

Description This script helps the user to update Linux and Mac systems through the terminal. All the user has to install some requirements and then ru

2 Jan 23, 2022

U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection

The code for our newly accepted paper in Pattern Recognition 2020: "U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection."

6.5k Jan 09, 2023

Efficient Householder transformation in PyTorch

Efficient Householder Transformation in PyTorch This repository implements the Householder transformation algorithm for calculating orthogonal matrice

49 Nov 20, 2022

A Python module for the generation and training of an entry-level feedforward neural network.

ff-neural-network A Python module for the generation and training of an entry-level feedforward neural network. This repository serves as a repurposin

2 Jan 31, 2022

Code and data of the ACL 2021 paper: Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

MetaAdaptRank This repository provides the implementation of meta-learning to reweight synthetic weak supervision data described in the paper Few-Shot

5 Jun 16, 2022

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Neural Attention Distillation This is an implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep

84 Jan 04, 2023