I got a number of warning messages when running slow5tools merge on a directory of blow5 files. I was wondering if this is a big problem or whether I can ignore it?

[merge_main::WARNING] In file Calu/Calu_I_2hpi/FAO66427_4fd309c9_127.blow5, read_group 0 has a different number of header attributes than what the processed files had

When I ran slow5tools f2s on the original data I did see the following warning a number of times

[search_and_warn::WARNING] slow5tools-v0.3.0: The attribute 'pore_type' is empty and will be stored in the SLOW5 header.
[search_and_warn::WARNING] slow5tools-v0.3.0: The attribute 'pore_type' is empty and will be stored in the SLOW5 header. This warning is suppressed now onwards.

are these two warnings related?

I am running the command as follows:

slow5tools f2s ${dir_name}/fast5 -d "${outDir}" Where ${dir_name} is the input directory and ${outDir} the output directory.

But shortly, it gives me this error:

list_all_items] Looking for '*.fast5' files in dir_name/fast5
[f2s_main] 3245 fast5 files found - took 0.391s
[f2s_main] Just before forking, peak RAM = 0.000 GB
[f2s_iop] 8 proceses will be used.
[f2s_iop] Spawning 8 I/O processes to circumvent HDF hell.
[f2s_child_worker::ERROR] Bad fast5: Fast5 file 'dir_name/fast5/P5a_279.fast5' could not be opened or is corrupted.
[f2s_iop] Child process 1969843 exited with status=1.

I appreciate your feedback/suggestions, Medhat

Hello @hasindu2008,

Thank you for developing slow5tools - it's been really useful to compress and store ONT data!

Apologies if this is a naïve question but I thought it was possible to basecall blow5 files using Guppy. I have successfully converted my fast5 files to slow5, and am trying to basecall, however I get the following error:

2022-08-01 18:11:09.698375 [guppy/message] ONT Guppy basecalling software version 6.2.1+6588110, minimap2 version 2.22-r1101
config file:        /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/softwares/ont-guppy-cpu/data/dna_r9.4.1_450bps_modbases_5mc_cg_hac.cfg
model file:         /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/softwares/ont-guppy-cpu/data/template_r9.4.1_450bps_hac.jsn
input path:         /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/ONT/Mouse_Whole_Genome/1_raw/20200807_1632_MC-110214_0_add313_506ffc5b/barcode10/blow5
save path:          /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/ONT/Mouse_Whole_Genome/2_basecalled/20200807_1632_MC-110214_0_add313_506ffc5b/barcode10
chunk size:         2000
chunks per runner:  256
minimum qscore:     9
records per file:   4000
num basecallers:    1
cpu mode:           ON
threads per caller: 4

alignment file:     /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/reference_2019/mm10.fa
alignment type:     auto

Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA.
The EULA may be found in /gpfs/mrc0/projects/Research_Project-MRC148213/sl693/softwares/ont-guppy-cpu/bin
2022-08-01 18:11:09.698970 [guppy/info] crashpad_handler not supported on this platform.
2022-08-01 18:13:19.554000 [guppy/message] Full alignment will be performed.
2022-08-01 18:13:19.595740 [guppy/message] Found 0 fast5 files to process.
2022-08-01 18:13:19.596981 [guppy/message] Init time: 129876 ms
2022-08-01 18:13:19.697374 [guppy/message] Caller time: 100 ms, Samples called: 0, samples/s: 0
2022-08-01 18:13:19.697410 [guppy/message] Finishing up any open output files.
2022-08-01 18:13:19.779375 [guppy/message] Basecalling completed successfully.

With the commands (slow5tools 0.5.1):

slow5tools f2s <input_dir> -d <output_dir> -p 8
guppy_basecaller -i <output_dir> -s <output_basecalled_dir> -c dna_r9.4.1_450bps_modbases_5mc_cg_hac.cfg --bam_out --recursive --align_ref mm10.fasta

Am I missing something/argument or is basecall only possible with the original raw fast5 files?

Thank you, Szi Kay

I converted a test fast5 file to slow5 then back to fast5. I thought that by using the --lossless flag that the final fast5 would be the same as the original; however, it appears that I am loosing data during the conversion.

Is it possible to preserve all of the data from the original fast5?

Commands:

# Convert fast5 to slow5
slow5tools f2s --lossless true -o slow.slow5 original.fast5

# Convert slow5 back to fast5
slow5tools s2f -o new.fast5 slow.slow5

Resulting file sizes original.fast5 97M slow.slow5 116M new.fast5 61M

Hi there,

I am testing slow5tools on a few datasets, converting fast5 to blow5 (slow5tools f2s) For most datasets it works well (~37% smaller file sizes) but unfortunately for some datasets I am running into the following warnings etc

[search_and_warn::WARNING] slow5tools-v0.3.0: Weird fast5: Attribute file_version/read_001445a6-1ee9-4e18-90c3-7afee025d1b3 in AKH_1a/AKH_1a_8.fast5 is unexpected. This warning is suppressed now onwards. [search_and_warn::WARNING] slow5tools-v0.3.0: Weird fast5: Attribute previous_read_id/PreviousReadInfo in AKH_1a/AKH_1a_8.fast5 is unexpected. This warning is suppressed now onwards. [search_and_warn::WARNING] slow5tools-v0.3.0: Weird fast5: Attribute previous_read_number/PreviousReadInfo in AKH_1a/AKH_1a_8.fast5 is unexpected. This warning is suppressed now onwards. [fast5_group_itr::ERROR] Bad fast5: run_id is missing in the AKH_1a/AKH_1a_8.fast5 in read_id 001445a6-1ee9-4e18-90c3-7afee025d1b3. [read_fast5::ERROR] Bad fast5: Could not iterate over the read groups in the fast5 file AKH_1a/AKH_1a_8.fast5. [f2s_child_worker::ERROR] Bad fast5: Could not read contents of the fast5 file 'AKH_1a/AKH_1a_8.fast5'.

So it appears the first real issue is that the run_id is missing in these files and this only appears to occur in my older fast5 datasets (data from around 2017-2018), where it seems that the fast5 format didn't contain run_id info. Even using the --allow option doesn't help These files can be manipulated using h5dump, ont_fast5_api and basecalled etc so it is not that the files are corrupted in some way Plus by manually checking the files with h5dump I can see the difference in fast5 structure between files that worked with slow5tools and those that didn't (including but not limited to the missing run_id information)

If there is not a run_id present could one just be placed in it's absence (a randomly generated name as is done)? considering the --allow option appears to choose the first run_id anyway. Or in general just to handle older fast5 files?

Thanks a lot

I got this error when using f2s in v0.6.0.

Bad fast5: Attribute read_008c8f1f-f32f-4b23-ab6f-1ef4f4531a20/pore_type in /nfs/research/zi/mbhall/tech_wars/data/madagascar/nanopore/raw_data/md_tb_reseq_2019_5/multi_fast5s/BC76_r0b1235_0.fast5 is duplicated and has two different values in two different places. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.

Here is the fast5 that caused it. BC76_r0b1235_0.fast5.gz

I have a few runs that give this error : (converted from fast5 to blow5 then merged into a single file)

# slow5tools stats merged.blow5
file path       merged.blow5
file version    0.2.0
file format     BLOW5
record compression method       zlib
signal compression method       svb-zd
number of read groups   2
number of auxiliary fields      6
auxiliary fields        end_reason,channel_number,median_before,read_number,start_mux,start_time
[stats_main] counting number of slow5 records...
[slow5_get_next_mem::ERROR] Failed to allocate memory: Cannot allocate memory At src/slow5.c:3240
[stats_main::ERROR] Error reading the file.

The machine isn't under any memory pressure. (total 128 GB memory) and slow5tool doesn't seem to be using much memory either during the run. (it looks to be CPU bound) The merged.blow5 is about 387 GB large.

Is this a bug or somewhat expected with larger blow5 files ? I use the stats to get the "number of records" to compare with as a sanity check, if there is an other method i'm open to suggestions :)

Thanks!

Thanks for the cool tools! I think SLOW5 is a great idea and hope that ONT follows suit. I started to play around a little with slow5tools with the thought of converting my large number of projects from folders of FAST5 files to single BLOW5 format -- I initially thought I could save some space (it's mentioned a 25% reduction in file size).

This seems to be true for older FAST5 datasets, for example:

$ du -sh old_project_fast5
100G    old_project_fast5
$ du -sh old_project.blow5
93G     old_project.blow5

So here I saved 7 GB by converting to BLOW5, which isn't 25% but still decent.

I then noticed that the newer VBZ compressed files are actually quite a bit smaller than BLOW5, presumably due to the compression, for example:

$ du -sh newer_project_fast5_vbz
28G     newer_project_fast5_vbz
$ slow5tools f2s newer_project_fast5_vbz -d newer_project_blow5 -p 12
$ du -sh newer_project_blow5
36G     newer_project_blow5

In this case, converting the VBZ FAST5 files to BLOW5 format actually increased the size substantially. So it seems that the VBZ compression method that ONT has rolled out recently does a lot in saving space. Is it possible to use VBZ compression for BLOW5 format instead of zlib?

Hello, it's me again! So we have a new minion run using the updated minknow, then when I usedslow5tools to convert the fast5 files I just got these errors:

[list_all_items] Looking for '*.fast5' files in FAS16606_pass_13cc46f5_0.fast5
[f2s_main] 1 fast5 files found - took 0.000s
[f2s_main] Just before forking, peak RAM = 0.000 GB
[f2s_iop] 1 proceses will be used.
[fast5_attribute_itr::ERROR] Bad fast5: Attribute Raw/num_minknow_events in FAS16606_pass_13cc46f5_0.fast5 is unexpected. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.
[read_fast5::ERROR] Bad fast5: Could not iterate over the read groups in the fast5 file FAS16606_pass_13cc46f5_0.fast5.
[f2s_child_worker::ERROR] Could not read contents of the fast5 file 'FAS16606_pass_13cc46f5_0.fast5'.

I don't know if it's caused by the new minknow or what, but I can use h5py to open them. Could you help fix this? Thanks.

I can also send over our fast5 if you need it.

Hi folks.

I'm very happy to see that you worked on this project and published it. I hope the folks in Oxford pick up some of what you showed and takes advantage of these findings.

I am running some tests using version 0.3.0.

I have the following datasets based on a single .fast5 file containing 4000 reads of genomic ONT promethion reads.

dataset | file method | file size (kb) -- | -- | -- 1 | original zlib fast5 | 3027586 2 | input: 1, f2s zlib record and svb-zd signal compression | 1839624 3 | input: 1, f2s zstd record and svb-zd signal compression | 1771464 4 | input: 2, s2f | 2701116 5 | input: 3 s2f | 2701116

The .fast5->blow5->.fast5 results in 4 and 5 resulted in the exact same file independent of the compression params, but both files differ in size from the original.

I've been poking around with h5diff to verify that everything in my original fast5 is still recovered in the resulting .fast5 after roundtripping through blow5. I can't seem to get it to report differences, or to confirm that the file contents are the same. Can you share the approach you used to compare .fast5 file contents? I also tried basecalling multiple times with guppy, but the non-deterministic manner in which the basecalling is done limits the utility here.

Also, is the size difference between 1, 4 and 5 possibly just due to a difference in compression? I don't see any information describing if the .fast5->blow5->.fast5 resulting file is zlib, vbz, or otherwise compressed.

thanks Richard

Hello there, I am using pyslow5 for my task and I found there's one column called "end_reason" in my slow5 file, but I couldn't access it.
It is not in the keys of s5.seq_reads(aux = 'all') or s5_data = s5.seq_reads(aux = ["start_time", "read_number", "start_mux", "median_before", "end_reason", "channel_number"])...

Using slow5tools v0.3.0

The input file had an unexpected field (end_reason) which triggered a warning on conversion to blow5.

Trying to recreate the fast5 from the blow5 file resulted in:

[list_all_items] Looking for '*low5' files in test_blow5/ [s2f_main] 1 files found - took 0.001s [s2f_iop] 1 proceses will be used [s2f_child_worker] Converting test_blow5//FAL37440_pass_c4fa58d7_179.blow5 to fast5 [slow5_get_aux_enum_labels::ERROR] No enum auxiliary type exists. At src/slow5.c:1181 [slow5_get_aux_enum_labels::ERROR] Exiting on error. At src/slow5.c:1181

If you can provide an email I can send an example file.

Data were generated in May 2020 using GridION - 19.12.6 - Guppy Version was 3.2.10