Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.

A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.

What should I do?

hello, thanks for you amazing work.

I would like to ask, what kind of changes do I need to apply in order to perform a stream from camera and detect a pose of object e.g pedestrians. Do I need to re-train your model as well?

Thanks for answer in an advance

Hey Nicholasli1995,

Thanks for putting together this awesome repo, I really appreciate how thoroughly documented the setup is.

Would it be possible to get a few hints as to how to run the model on a single image as opposed to a directory?

I see that the data_loader in 'inference' is nearly directly from PyTorch, so it's probably my lack of experience with PyTorch that has me confused.

Separately, I also tried making my own directory and referencing it in my .yaml file, the following command still gave me an error about referencing the KITTI dataset

python .\inference.py --cfg "../configs/single_inference_test.yml" --visualize True --batch_to_show 1

I was a little bit surprised to see "training" in the file not found path as I'm not asking to train -- I would expect to have to place an analogous 'test.txt' in this directory with the single test image filename, but I don't think that's currently the issue.

Thanks! Clayton

Thank you for sharing your great code. :smiley_cat:

What is the license for this model? I'd like to cite it to the repository I'm working on if possible, but I want to post the license correctly. https://github.com/PINTO0309/PINTO_model_zoo

Thank you.

Thanks for your great work. However, I have some confusion in “Reproduce results on the test split”. If I understand correctly, the inputs to the model are 2D bounding boxes which located in “../resources/test_boxes” and the outputs should be 3D bounding boxes which placed in “../output/submission/data”. However, the results I got were not 3D bounding boxes, as follows: The results do not contain 3D information. Is it because I made a mistake somewhere?

Hello, firstly I would like to thanks for an amazing work and that you made it public with great documentation in order to reproduce and re train the network.

Today I wanted to train your network according to training - stage2. However, by running the code I got an error as following

$ python train_IGRs.py --cfg "../configs/KITTI_train_IGRs.yml"

=> init weights from normal distribution
=> loading pretrained model ../resources/start_point.pth

Total Parameters: 63,978,471
----------------------------------------------------------------------------------------------------------------------------------
Total Multiply Adds (For Convolution and Linear Layers only): 19.573845863342285 GFLOPs
----------------------------------------------------------------------------------------------------------------------------------
Number of Layers
Conv2d : 306 layers   BatchNorm2d : 304 layers   ReLU : 269 layers   Bottleneck : 4 layers   BasicBlock : 108 layers   Upsample : 28 layers   HighResolutionModule : 8 layers   Sigmoid : 1 layers   
Initializing KITTI train set, please wait...
Found prepared keypoints at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car'].npy
Found prepared instance_ids at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car']_ids.npy
Found prepared rotations at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car']_rots.npy
Initialization finished for KITTI train set
Initializing KITTI valid set, please wait...
Found prepared keypoints at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car'].npy
Found prepared instance_ids at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car']_ids.npy
Found prepared rotations at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car']_rots.npy
Initialization finished for KITTI valid set
Traceback (most recent call last):
  File "train_IGRs.py", line 159, in <module>
    main()
  File "train_IGRs.py", line 154, in main
    train(model, model_settings, GPUs, cfgs, logger, final_output_dir)
  File "train_IGRs.py", line 89, in train
    trainer.train(train_dataset=train_dataset, 
  File "../libs/trainer/trainer.py", line 154, in train
    evaluator = Evaluator(cfgs['training_settings']['eval_metrics'], 
  File "../libs/metric/criterions.py", line 546, in __init__
    self.metrics.append(eval(metric + '(cfgs=cfgs, num_joints=num_joints)'))
  File "<string>", line 1, in <module>
  File "../libs/metric/criterions.py", line 183, in __init__
    self.arg_max = cfgs['testing_settings']['arg_max']
KeyError: 'arg_max

It seems that arg_max key is missing in the KITTI_train_IGRs.yml

How to arg_max parameter should be defined?

Thanks for answer in advance.

Hello @Nicholasli1995 thank you for your implementation. I have several questions.

1. What split (train and validation split) did you use for this code? is it subcnn split?
2. during the inference using testing set do you use ROI from the object detection algorithm?
3. Is it still use the calibration or this system can automatically generate it?
4. How to produce the similar table as in paper? I see that you use AOS, AP, also something like Easy, Medium, Hard. However when I tried to use validation set evaluation and use kitti_eval_offline I only got some plots, and statistic informations in txt and final result as here.

python inference.py --cfg "../configs/try_KITTI_inference:test_submission.yml"

Wrote prediction file at ../result/gt_box_test/data/007025.txt
Warning: 007025.png not included in detected images!
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_detection.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_orientation.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_detection_ground.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_detection_3d.pdf'.
Thank you for participating in our evaluation!
Loading detections...
number of files for evaluation: 1634
  done.
save ../result/submission/plot/car_detection.txt
car_detection AP: 93.934082 84.951851 67.594185
save ../result/submission/plot/car_orientation.txt
car_orientation AP: 93.664360 84.683617 67.141930
save ../result/submission/plot/car_detection_ground.txt
car_detection_ground AP: 41.456226 31.880611 24.876181
save ../result/submission/plot/car_detection_3d.txt
car_detection_3d AP: 29.933027 24.017235 18.650723
Your evaluation results are available at:
../result/submission

It only produce AP no AOS. Is it because you use testing set then evaluate directly on KITTI evaluation systems so they will provide you the detail result? Suppose we want to measure in Validation set, which value must I use? because there are many information such as car_detection AP car_orientation AP car_detection_ground AP car_detection_3d AP. So in specific how to produce table 1 as in paper if we use validation set?

-Thank you-

Thank you for share your huge work ! Can the output coordinates of the model be directly visualized, does any transformation is needed to display them in the original image?

Hello, Thank you for open-sourcing this amazing project!

I have a question about the convention for the transformation of the 3D box. EgoNet only produces an egocentric pose (i.e. camera coordinates) corresponding to the rotation between the 3D box extracted from the keypoints and a template 3D box. We also have a translation corresponding to the first point in kpts_3D_pred, here.

To better understand the coordinate systems involved I'm doing the following experiment:

1. Create a template 3D bounding box following this, in the canonical pose.
2. Rotate it with the rotation matrix given by EgoNet, this one

After doing these two steps, I still need one translation to place the 3D box in space (in the camera system). The question is, what translation should I use? Is it the one corresponding to the first point in kpts_3d_pred?

Thank you for your time

Thank you for the work you have done, and open source the code. I would like to ask how to use your code, will only 2D annotation box pictures with labels, generate 3D coordinate box used for detection tasks? I have a problem now, Is it necessary to have this file? I tried to modify it and got an error. I want to know what it does {{{ Download the resources folder and unzip its contents. Place the resource folder at ${EgoNet_DIR}/resources }}} English is translated by machine, If there is any improper wording, please forgive me Thanks you

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?