While examining evaluate_semantic_instance.py I found a possible bug:

https://github.com/ScanNet/ScanNet/blob/fc255fefa3918a21930681ce7c52502a4447c228/BenchmarkScripts/3d_evaluation/evaluate_semantic_instance.py#L267

I believe that the gt_ids are label_id*1000+inst_id, however the VALID_CLASS_IDS are just label_ids, so the bool_void will always be all zeros. Besides the current formula seems to compute the non-void mask. The right way to compute bool_void is probably:

bool_void = np.logical_not(np.in1d((gt_ids/1000).astype(np.int32), VALID_CLASS_IDS))

This bug may not affect the result too much but should be fixed.

I am reading _vh_clean.aggregation.json for a per point annotation, however, for scene train/ 0002_00, I got the following error: index 307925 is out of bounds for axis 1 with size 297362. It seems that the index stored in aggregation.json exceeds the total number of points.

I am using https://github.com/ScanNet/ScanNet/blob/1cc5149d30d248b776039c76ae8864e63df18831/SensReader/python/reader.py to read sens files. But I ran into the following errors for scene0207_02 only, while other scenes are fine to read and dump data.

Traceback (most recent call last):
  File "./SensReader/python/reader.py", line 79, in <module>
    main()
  File "./SensReader/python/reader.py", line 35, in main
    sd.export_depth_images(os.path.join(opt.output_path, 'depth'))
  File "/home/zhongad/3D_workspace/dataset/ScanNet/SensReader/python/SensorData.py", line 78, in export_depth_images
    depth_data = self.frames[f].decompress_depth(self.depth_compression_type)
  File "/home/zhongad/3D_workspace/dataset/ScanNet/SensReader/python/SensorData.py", line 26, in decompress_depth
    return self.decompress_depth_zlib()
  File "/home/zhongad/3D_workspace/dataset/ScanNet/SensReader/python/SensorData.py", line 31, in decompress_depth_zlib
    return zlib.decompress(self.depth_data)
zlib.error: Error -3 while decompressing data: invalid distance code

Does anyone know how to solve this? P.S. I downloaded the dataset in Nov 2021.

Hello,

I am trying to compute a combined point cloud by aggregating depth maps across all cameras. After unprojecting depth [X_i,Y_i,Z_i] = unproject(depth_i), each point cloud needs to be transformed by the camera pose to get the world coordinates - [X_i,Y_i,Z_i].transform(camera_pose_i). Similarly for frame j: [X_j,Y_j,Z_j].transform(camera_pose_j). I was expecting that projecting these two point clouds would lead to perfectly aligned clouds. Am I missing something?

I noticed that there are some camera poses in the dataset with invalid data, with matrices only containing inf values. It’s not a big issue, because this only applies to a small portion of frames, but the issue should at least be documented (I didn’t read about this so far).

Further, I think that many sequences don’t have timestamps, but sometimes random frames seem to have random stamps. Maybe the value wasn’t always initialised?

Here is a small program that can highlight these issues

#include <iostream>
#include "../external/scannet/sensorData.h"

using namespace std;

tuple<bool,bool> analyze_sens(const string& input, bool verbose = false){

    // Input
    if(verbose){
        cout << "Loading data ... ";
        cout.flush();
    }
    ml::SensorData sd(input);
    if(verbose){
        cout << "done!" << endl;
        cout << sd << endl;
    }

    // Stats
    bool ts_d_monotonic = true;
    bool ts_c_monotonic = true;
    bool ts_d_available = false;
    bool ts_c_available = false;
    uint64_t ts_d_last = 0;
    uint64_t ts_c_last = 0;

    bool has_illegal_transformation = false;

    for (size_t i = 0; i < sd.m_frames.size(); i++) {

        // Test timestamps
        const ml::SensorData::RGBDFrame& frame = sd.m_frames[i];
        uint64_t t_d = frame.getTimeStampDepth();
        uint64_t t_c = frame.getTimeStampColor();
        if (t_d > 0) ts_d_available = true;
        if (t_c > 0) ts_c_available = true;
        if (t_d < ts_d_last) ts_d_monotonic = false;
        if (t_c < ts_c_last) ts_c_monotonic = false;
        ts_d_last = t_d;
        ts_c_last = t_c;

        // Test poses
        ml::mat4f t = frame.getCameraToWorld();
        if(t.matrix[15] != 1 || t.matrix[14] != 0 || t.matrix[13] != 0 || t.matrix[12] != 0){
            has_illegal_transformation = true;
            if(verbose)
                cout << "Found illegal transformation at frame " << to_string(i) << ": ["
                     << t.matrix[0] << ", " << t.matrix[1] << ", " << t.matrix[2] << ", " <<t.matrix[3] << "]["
                     << t.matrix[4] << ", " << t.matrix[5] << ", " << t.matrix[6] << ", " <<t.matrix[7] << "]["
                     << t.matrix[8] << ", " << t.matrix[9] << ", " << t.matrix[10] << ", " <<t.matrix[11] << "]["
                     << t.matrix[12] << ", " << t.matrix[13] << ", " << t.matrix[14] << ", " <<t.matrix[15] << "]]" << endl;
        }
    }

    if(verbose){
        cout << "Depth timestamps are monotonic: " << (ts_d_monotonic ? "\x1B[32m yes" : "\x1B[31m no") << "\x1B[0m \n";
        cout << "RGB   timestamps are monotonic: " << (ts_c_monotonic ? "\x1B[32m yes" : "\x1B[31m no") << "\x1B[0m \n";
        cout << "Depth timestamps are available: " << (ts_d_available ? "\x1B[32m yes" : "\x1B[31m no") << "\x1B[0m \n";
        cout << "RGB   timestamps are available: " << (ts_c_available ? "\x1B[32m yes" : "\x1B[31m no") << "\x1B[0m \n";
        cout << "All  camera  poses  were legal: " << (!has_illegal_transformation ? "\x1B[32m yes" : "\x1B[31m no") << "\x1B[0m \n";
        cout << endl;
    }
    return make_tuple(!ts_d_monotonic || !ts_c_monotonic, has_illegal_transformation);
}

int main(int argc, char* argv[])
{
    if(argc < 2 || argc > 3) {
        cout << "A tool to analyse scannet *.sens data.\n\n"
                "Error, invalid arguments.\n"
                "Mandatory: input *.sens file / input *.txt file\n"
                "Optional path to dataset dir (if txt is provided)."
             << endl;
        return 1;
    }

    // Input data
    string filename = argv[1];
    if(filename.substr(filename.find_last_of(".") + 1) == "txt"){
        // Analyse many sens files
        string sequence_name;
        string root = (argc == 3 ? argv[2] : "");
        ifstream in_stream(filename);
        while (getline(in_stream, sequence_name)){
            cout << "Checking " << sequence_name << "...";
            cout.flush();
            tuple<bool,bool> r = analyze_sens(root + "/" + sequence_name + "/" + sequence_name  + ".sens");

            if(get<0>(r))
                cout << "\x1B[31m Timestamp issue \x1B[0m";
            else
                cout << "\x1B[32m Timestamps good \x1B[0m";

            if(get<1>(r))
                cout << "\x1B[31m Pose issue \x1B[0m" << endl;
            else
                cout << "\x1B[32m Poses good \x1B[0m" << endl;
        }
        in_stream.close();
    } else {
        // Analyse single sens files
        analyze_sens(filename, true);
    }


    return 0;
}

If you run this script on sequence scene0003_01.sens for example, you get a list of invalid poses, like:

Found illegal transformation at frame 1054: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1071: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1079: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1080: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1081: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1082: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1083: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1084: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1085: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1086: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1087: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]] Found illegal transformation at frame 1088: [-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf][-inf, -inf, -inf, -inf]]

Sequence scene0066_00.sens on the other hand has issues with timestamps. You can also run the tool on a list of sequences, giving you output like:

Checking scene0000_00... Timestamps good Poses good Checking scene0000_01... Timestamps good Poses good Checking scene0000_02... Timestamps good Poses good Checking scene0001_00... Timestamps good Pose issue Checking scene0001_01... Timestamps good Pose issue Checking scene0002_00... Timestamps good Poses good Checking scene0002_01... Timestamps good Pose issue Checking scene0003_00... Timestamps good Pose issue Checking scene0003_01... Timestamps good Pose issue Checking scene0003_02... Timestamps good Pose issue Checking scene0004_00... Timestamps good Poses good Checking scene0005_00... Timestamps good Poses good ...

Thanks for providing this great dataset by the way!

Dears,

Thanks for your amazing work!

While I am trying to download the data on a new machine I faced the following error "IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)" Despite I downloaded it two days ago successfully

Thanks in advance!

when i download scannet datasets, i met one error. please help me, thanks very much.

$ python -V Python 3.7.2 $ python download-scannet.py -o ./Scannet_Dataset/ By pressing any key to continue you confirm that you have agreed to the ScanNet terms of use as described at: http://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf

Press any key to continue, or CTRL-C to exit. Traceback (most recent call last): File "download-scannet.py", line 233, in if name == "main": main() File "download-scannet.py", line 142, in main key = raw_input('') NameError: name 'raw_input' is not defined

Hi there, I want to access Scannet benchmark with link http://kaldir.vc.in.tum.de/scannet_benchmark but failed. I tried with different IP to access but all failed, so could you check if there is anything wrong on your server side?

Hi,

When I used the python downloading code to download the dataset in March, it has color, depth, intrinsic, pose folder with the .sen files, today when I tried to download more date, the color, depth, intrinsic, pose folders cannot be downloaded with .sen files, even cannot without any --type specification, is there any update to the dataset downloader?

I am trying to map the RGBD images into a 3D point cloud. But I have struggled for a long time to align the views. The point cloud always looks like this (this point cloud contain two views, and they are not aligned):

Am I doing anything wrong? This is my code:

import matplotlib.pyplot as plt
import numpy as np
import os
import random
from PIL import Image
from mpl_toolkits.mplot3d import Axes3D
from sensor_data import SensorData

# data extracted using SensorData.py
DATA_PATH = ...
RGB_PATH = ... 
DEPTH_PATH = ...  
POSE_PATH = ... 
MAX_INDEX = 100 # only load the first 100 for testing

def load_matrix_from_txt(path, shape=(4, 4)):
    with open(path) as f:
        txt = f.readlines()
    txt = ''.join(txt).replace('\n', ' ')
    matrix = [float(v) for v in txt.split()]
    return np.array(matrix).reshape(shape)

intrinsic_depth = load_matrix_from_txt(os.path.join(DATA_PATH, 'intrinsic_depth.txt'))
poses = [load_matrix_from_txt(os.path.join(POSE_PATH, f'{i}.txt')) for i in range(MAX_INDEX)]

def load_image(path):
    image = Image.open(path)
    return np.array(image)

rgb_images = [load_image(os.path.join(RGB_PATH, f'{i}.jpg')) for i in range(MAX_INDEX)]
depth_images = [load_image(os.path.join(DEPTH_PATH, f'{i}.png'))for i in range(MAX_INDEX)]

# map a rgbd pixel into 3D space. 
# should be the same as SensorReader
def convert_from_uvd(u, v, d, intr, pose):
    
    # const mat4f intrinsicInv = m_calibrationDepth.m_intrinsic.getInverse();
    intrinsic_inv = np.linalg.inv(intr)
    
    # float d = (float)depth[i]/m_depthShift;
    d /= 1000
    
    # vec3f cameraPos = (intrinsicInv*vec4f((float)x*d, (float)y*d, d, 0.0f)).getVec3();
    camera_pos = (intrinsic_inv.dot(np.array([u*d, v*d, d, 0.0])))[:3]
    
    # vec3f worldPos = transform * cameraPos;
    xyzw = pose[:, :3].dot(camera_pos) + pose[:, 3]
    world_pos = xyzw[:3] / xyzw[3]
    
    return world_pos

x_data, y_data, z_data, c_data = [], [], [], []

for idx in [20, 50]: # take two views as a test
    d = depth_images[idx]
    c = rgb_images[idx]
    p = poses[idx]
    
    # map each pixel into 3D space
    for i in range(d.shape[0]):
        for j in range(d.shape[1]):
            x, y, z = convert_from_uvd(i, j, d[i, j], intrinsic_depth, p)
                
            x_data.append(x)
            y_data.append(y)
            z_data.append(z)
            
            ci = int(i * c.shape[0] / d.shape[0])
            cj = int(j * c.shape[1] / d.shape[1])
            c_data.append(c[ci, cj] / 255.0)

# plot the 3D point cloud
def plot_3d(xdata, ydata, zdata, color=None, b_min=-3, b_max=3, view=(45, 45)):
    fig, ax = plt.subplots(subplot_kw={"projection": "3d"}, dpi=200)
    ax.view_init(view[0], view[1])

    ax.set_xlim(b_min, b_max)
    ax.set_ylim(b_min, b_max)
    ax.set_zlim(b_min, b_max)

    ax.scatter3D(xdata, ydata, zdata, c=color, cmap='rgb', s=0.1)

    plt.show()
    
plot_3d(x_data, y_data, z_data, color=c_data)

For the convert_from_uvd function, I also tried the out-of-the-box algorithm implemented by Pytorch3D, which also resulted in the same problem.

from pytorch3d.renderer import PerspectiveCameras
import torch

def convert_from_uvd(u, v, d, intr, pose):
    d /= 1000
    camera = PerspectiveCameras(
        focal_length=torch.Tensor([[intr[0, 0], intr[1, 1]]]),
        principal_point=torch.Tensor([[intr[0, 2], intr[1, 2]]]),
        R=torch.from_numpy(pose[:3, :3]).unsqueeze(0),
        T=torch.from_numpy(pose[:3, 3]).unsqueeze(0)
    )
    p = camera.unproject_points(torch.Tensor([u, v, d]).unsqueeze(0))
    
    return p.tolist()[0]

When I want to align depth to color used by intrinc and colortodepth extrinc, there are some problems. scene0001_00/scene0001_00.txt didn't have DepthtoColorExtric.

Bumps setuptools from 61.2.0 to 65.5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps wheel from 0.37.1 to 0.38.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

The backup captcha is missing at the bottom of the registration webpage, which makes the registration fail the security question. I' eager for any advice, please.

Bumps certifi from 2022.5.18.1 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hello,

When running the Python Dataloader with the following command

python reader.py --filename tmpc4jdhmmf --output_path 'data/images' --export_color_images

I receive this error. struct.error: unpack requires a string argument of length 383260

Is there something I'm doing wrong in the command? Appreciate the support!

Prompt the following sentence: Hmm... can't access this page The page at https://kaldir.vc.in.tum.de/scannet_benchmark/submission_upload?id=1405 may have a problem or has moved permanently to a new URL.