time-consuming of the FID computation Hi, I want to know why the FID computation is very slow. When I calculate the FID of 28000 images, it sometimes got stuck and spent almost one day or more to calculate once! Is there any idea to help me fix this problem? Thanks!

Using the updated weights still get wrong result. See this repo https://github.com/AtlantixJJ/PytorchInceptionV3 for detail.

Run this code (need to store some image in data/cifar10_test, or go to the repo above):

"""
A script to test Pytorch and Tensorflow InceptionV3 have consistent behavior.
"""
import sys, argparse, os, pathlib
sys.path.insert(0, ".")
import numpy as np
import tensorflow as tf
import torch
from inception_modified import inception_v3
from PIL import Image

parser = argparse.ArgumentParser()
parser.add_argument("--load_path", default="", help="The path to changed pytorch inceptionv3 weight. Run change_statedict.py to obtain.")
args = parser.parse_args()

def check_or_download_inception(inception_path):
    ''' Checks if the path to the inception file is valid, or downloads
        the file if it is not present. '''
    INCEPTION_URL = 'http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz'
    if inception_path is None:
        inception_path = '/tmp'
    inception_path = pathlib.Path(inception_path)
    model_file = inception_path / 'classify_image_graph_def.pb'
    if not model_file.exists():
        print("Downloading Inception model")
        from urllib import request
        import tarfile
        fn, _ = request.urlretrieve(INCEPTION_URL)
        with tarfile.open(fn, mode='r') as f:
            f.extract('classify_image_graph_def.pb', str(model_file.parent))
    return str(model_file)

def torch2numpy(x):
    return x.detach().cpu().numpy().transpose(0, 2, 3, 1)

torch.backends.cudnn.benchmark = True
torch.manual_seed(1)
torch.cuda.manual_seed(1)

data_dir = "data/cifar10_test/"
imgs_pil = [Image.open(open(data_dir + s, "rb")).resize((299,299), Image.BILINEAR) for s in os.listdir(data_dir)]
imgs = [np.asarray(img).astype("float32") for img in imgs_pil]
x_arr = np.array(imgs)
x_arr_tf = x_arr
# TF InceptionV3 graph use [0, 255] scale image
feed = {'FID_Inception_Net/ExpandDims:0': x_arr_tf}
# This is identical to TF image transformation
x_arr_torch = x_arr / 255. #(x_arr - 128) * 0.0078125
x_torch = torch.from_numpy(x_arr_torch.transpose(0, 3, 1, 2)).float().cuda()

model = inception_v3(pretrained=True, aux_logits=False, transform_input=False)
if len(args.load_path) > 1:
    # default: pretrained/inception_v3_google.pth
    print("=> Get changed weight from %s" % args.load_path)
    try:
        model.load_state_dict(torch.load(args.load_path))
    except RuntimeError:
        pass
model.cuda()
model.eval()

if x_torch.size(2) != 299:
    import torch.nn.functional as F
    x_torch = F.interpolate(x_torch,
            size=(299, 299),
            mode='bilinear',
            align_corners=False)
features = model.get_feature(x_torch)
feature_pytorch = features[-1].detach().cpu().numpy()
if len(feature_pytorch.shape) == 4:
    feature_pytorch = feature_pytorch[:, :, 0, 0]

inception_path = check_or_download_inception("pretrained")
with tf.gfile.FastGFile("pretrained/classify_image_graph_def.pb", 'rb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())
    _ = tf.import_graph_def( graph_def, name='FID_Inception_Net')
    
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

layername = "FID_Inception_Net/pool_3:0"
layer = sess.graph.get_tensor_by_name(layername)
ops = layer.graph.get_operations()
for op_idx, op in enumerate(ops):
    for o in op.outputs:
        shape = o.get_shape()
        if shape._dims != []:
            shape = [s.value for s in shape]
            new_shape = []
            for j, s in enumerate(shape):
                if s == 1 and j == 0:
                    new_shape.append(None)
                else:
                    new_shape.append(s)
            # print(o.name, shape, new_shape)
            o.__dict__['_shape_val'] = tf.TensorShape(new_shape)

tensor_list = [n.name for n in tf.get_default_graph().as_graph_def().node]

target_layer_names = ["FID_Inception_Net/Mul:0", "FID_Inception_Net/conv:0", "FID_Inception_Net/pool_3:0"]
target_layers = [sess.graph.get_tensor_by_name(l) for l in target_layer_names]

sess.run(tf.global_variables_initializer())
res = sess.run(target_layers, feed)
x_tf = res[0]
feature_tensorflow = res[-1][:, 0, 0, :]

print("=> Pytorch pool3:")
print(feature_pytorch[0][:6])
print("=> Tensorflow pool3:")
print(feature_tensorflow[0][:6])
print("=> Mean abs difference")
print(np.abs(feature_pytorch - feature_tensorflow).mean())

def get_tf_layer(name):
    return sess.run(sess.graph.get_tensor_by_name(name + ':0'), feed)

result:

=> Pytorch pool3:
[0.42730308 0.00819586 0.27243498 0.2880235  0.10205843 0.05626289]
=> Tensorflow pool3:
[0.13085267 0.5260418  0.22931635 0.02919772 0.2439549  0.50965333]
=> Mean abs difference
0.34774715

I'm calculating FID score for a small dataset - folder1 - 1341 images and folder 2 - 1000 images with parameters as --dims 768. Please see below for the error-

100%|██████████| 10/10 [00:11<00:00, 1.11s/it]Warning: batch size is bigger than the data size. Setting batch size to data size

ValueError Traceback (most recent call last) /usr/lib/python3.6/runpy.py in run_module(mod_name, init_globals, run_name, alter_sys) 203 run_name = mod_name 204 if alter_sys: --> 205 return _run_module_code(code, init_globals, run_name, mod_spec) 206 else: 207 # Leave the sys module alone

7 frames /content/drive/My Drive/pytorch-fid-master/pytorch_fid/fid_score.py in get_activations(files, model, batch_size, dims, cuda) 100 pred_arr = np.empty((len(files), dims)) 101 --> 102 for i in tqdm(range(0, len(files), batch_size)): 103 start = i 104 end = i + batch_size

ValueError: range() arg 3 must not be zero /usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2590: UserWarning: Unknown failure executing module: <pytorch_fid> warn('Unknown failure executing module: <%s>' % mod_name)

Hi, Have you compared it with the official TF implementation? Are the score significantly different? Are there pitfalls one should be about when using it that might get significantly different results than TF?

Thanks, and thank you for writing that code.

Hello all, I used following command to calculate FID distance from two different folders. : ./fid_score.py /data/vision/pytorch-CycleGAN-and-pix2pix/datasets/synthia2kitti_ex1/testAB /data/vision/pytorch-CycleGAN-and-pix2pix/datasets/synthia2kitti_ex1/testB

I've come across the error as follows. :Traceback (most recent call last): File "./fid_score.py", line 262, in args.dims) File "./fid_score.py", line 249, in calculate_fid_given_paths dims, cuda) File "./fid_score.py", line 223, in _compute_statistics_of_path imgs = imgs.transpose((0, 3, 1, 2)) ValueError: axes don't match array

Please give me any comment. Thank you.

By definition, the FID score has a minimum bound of 0 but I got -4.8e-5 when I calculated the distance of a dataset to itself. Obviously, this is very close to zero. Nevertheless, can't think of a reason why would this be non-zero from your implementation. Any ideas?

Hi,

First of all, thanks for the nice implementation. Could you explain why you normalized inputs like here? The normalized results seems different from below.

# mean and var for ImageNet
mean = (0.485, 0.456, 0.406)
std = (0.229, 0.224, 0.225)

transform = transforms.Compose([
                    transforms.ToTensor(),
                    transforms.Normalize(mean=mean, std=std)])

Best, Yunjey

Since I want to change the size of the network input image, for example, the input is a rectangular image, do I need to retrain the parameters? If so, how to achieve this? If not, should we just change 229229 to the required size?just like 256176。Hope someone can help to answer, thank you very much

Hi,

first and foremost thank you very much for your work! I have an installation problem: The shell gives me following output: ERROR: Command errored out with exit status 1: command: 'C:\Users\bibi\anaconda3\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\bibi\AppData\Local\Temp\pip-install-ma3x6o5s\pytorch-fid_20cb67d1201c4ad9ae07a1faa01a3199\setup.py'"'"'; file='"'"'C:\Users\bibi\AppData\Local\Temp\pip-install-ma3x6o5s\pytorch-fid_20cb67d1201c4ad9ae07a1faa01a3199\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\bibi\AppData\Local\Temp\pip-pip-egg-info-y5m73gya' cwd: C:\Users\bibi\AppData\Local\Temp\pip-install-ma3x6o5s\pytorch-fid_20cb67d1201c4ad9ae07a1faa01a3199 Complete output (9 lines): Traceback (most recent call last): File "", line 1, in File "C:\Users\bibi\AppData\Local\Temp\pip-install-ma3x6o5s\pytorch-fid_20cb67d1201c4ad9ae07a1faa01a3199\setup.py", line 34, in packages=setuptools.find_packages(where='src/'), File "C:\Users\bibi\anaconda3\lib\site-packages\setuptools_init_.py", line 64, in find convert_path(where), File "C:\Users\bibi\anaconda3\lib\distutils\util.py", line 112, in convert_path raise ValueError("path '%s' cannot end with '/'" % pathname) ValueError: path 'src/' cannot end with '/'

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Can you please help?

Hello,

Is there a special reason as to why you choose num_classes=1008 instead of using the default value 1000 when creating the model. Refer to: https://github.com/mseitzer/pytorch-fid/blob/master/pytorch_fid/inception.py#L193

Best, Jan

Hello,

By using Pytorch's data loading pipeline, we can benefit of parallel and asynchronous data loading.

With the same batch size, My FID computation times goes from ~18 mins to ~3 mins.

Hello, It would be useful to compute the likelihood of a single sample under a specified target distribution (LLID) and I believe this is a significant missing feature.

I create a pull request to add the implementation of a function to compute the likelihood between a sample under a target distribution.

Usage

The usage is very similar to when you compute the FID score between two Gaussian multivariate distribution. To compute the LLID score between a sample and a dataset, where images of the sample and the target dateset are contained in an individual folder:

python -m pytorch_llid path/to/target/dataset path/to/sample

I double checked the dataset and there are no broken images.

Searching on Baidu, It seems that the error occurs when loading large images. As a result, the source code has to be modified to solve this problem.

Thanks for sharing.

The calculation of fid is too slow. The calculation for linalg.sqrtm(sigma1.dot(sigma2), disp=False) takes several minutes. How to speed it?

Hi, Recently, there is some research still evaluating their model by Inception Score, and issue #20 also mentions it. I create a pull request to add the implementation of computing the inception score.

Change

• implement the calculate_inception_score . In this implementation, I used the weight of Inception-V3 from the Pytorch team to predict the probs of images (to validate my implementation in the test phase).
• add --inception-score as store_true argument and --splits with default is 10 (just like the original code).

Test

I tested it with CIFAR-10 (pytorch-fid/src/pytorch-fid/test.py), in this paper, the mean and std of CIFAR-10 are 9.737 and 0.1489. In this PR, the test result return inception score 9.6728±0.1499. (the difference may come from Torch or NumPy version).

Problems

• While testing, I found out that IS is really hard to use, not only because of the issues from the paper above. Because the predicted results then have to be split into groups, the arrangement of data is also a problem (if shuffle=True in dataloader, the result will be different).
• Different from FID, IS is computed individually for each dataset, so the input only needs one path. The function is taking the first path as input and ignoring the second, I wonder is there any more flexible way to fix this.

Hi, Thanks for your work, I had one doubt regarding FID Score. Now if I want to train the GAN to generate the COCO alike datasets, then can I use the pretrained Inception net for that?

Logically, it should not be the case as the backbone training is itself very different from COCO like datasets.

Also, instead of InceptionNet, can I take any backbone trained on object detection on COCO or VOC and then take that as my network for the evaluation of the same?

Thanking you in advance