I have a S3 storage as output of my EPT config file.
The input is about 2920 LAZ files which are added 100 by 100 (thank to the
limit parameter) so I run 30 times the process and the log in saved in 30 different log files.
I done all that two times: once with a classic S3 storage (processing n1), and once with a performant one (processing n2).
The EPT is created and looks fine, however I have some issues in the log as "failure" or "errors":
Examples of failure (during the process n2):
logs/log_pq_16.log-01:04:30 - 50% - 39,524,412,339 - 36,766 (0) M/h - 0W - 0R - 0A
logs/log_pq_16.log:Failure #1: Failed to put Semis_2021_0884_6265_LA93_IGN69.json
logs/log_pq_16.log:Failure #1: Failed to put Semis_2021_0884_6268_LA93_IGN69.json
logs/log_pq_16.log-01:04:40 - 50% - 39,524,412,339 - 36,672 (0) M/h - 0W - 0R - 0A
logs/log_pq_17.log-Adding 1622 - ready/Semis_2021_0916_6275_LA93_IGN69.laz
logs/log_pq_17.log:Failure #1: Failed to put 15-26672-8714-16342.laz
logs/log_pq_17.log-13:10 - 50% - 40,076,353,288 - 182,626 (3,148) M/h - 290W - 165R - 1420A
Example of error (before the process n2) :
logs/log_pq_30.log- - ready/Semis_2021_0893_6252_LA93_IGN69.laz: Could not read from pocfluxhd/FXX/ept-data/15-26191-8384-16338.laz
logs/log_pq_30.log- - ready/Semis_2021_0896_6267_LA93_IGN69.laz: Could not read from pocfluxhd/FXX/ept-data/14-13129-4349-8170.laz
logs/log_pq_30.log- - ready/Semis_2021_0913_6274_LA93_IGN69.laz: Could not read from pocfluxhd/FXX/ept-data/15-26610-8839-16347.laz
logs/log_pq_30.log- - ready/Semis_2021_0917_6251_LA93_IGN69.laz: Could not read from pocfluxhd/FXX/ept-data/15-26694-8353-16354.laz
logs/log_pq_30.log-Adding 2900 - ready/Semis_2021_0937_6272_LA93_IGN69.laz
What is import is that if one file appears in this list for the log file
n, it will always appears in all the following ones.
To finish, all the not readable files or the not put files do exist on the S3 storage (n2):
2022/08/30 22:43:24 11.7K FXX/ept-sources/Semis_2021_0884_6268_LA93_IGN69.json
2022/08/30 22:43:24 11.6K FXX/ept-sources/Semis_2021_0884_6265_LA93_IGN69.json
2022/08/30 07:15:23 325.4K FXX/ept-data/15-26672-8714-16342.laz
2022/08/29 19:55:44 461.5K FXX/ept-data/14-13129-4349-8170.laz
2022/08/29 18:01:48 557 FXX/ept-data/15-26191-8384-16338.laz
2022/08/30 05:30:13 557 FXX/ept-data/15-26610-8839-16347.laz
2022/08/30 07:39:26 557 FXX/ept-data/15-26694-8353-16354.laz
(61,573 points for the first one, 0 for the other...)
I have a lot less issues with the performant one (failures: 38/129; errors: 4/54; respectively p2/p1).
The errors files and the failure files are not the same.
I think that the "failure" issue is not a big deal as the file is transferred (and the file are the same on both processing). May be Entwine tries several times and this is just a king of warning?
However, the "Errors" issue is a more problematic: the files are not the same in p1 and p2 (file with issue are smaller or empty). Is there a way of completing the no valid files?
Is there any possibilities of validating the generated EPT data?