fast_io

fast_io is a new C++20 library for extremely fast input/output and aims to replace iostream and cstdio. It is header-only (module only in the future) for easy inclusion in your project. It requires a capable C++20 compiler supporting concepts.

QQ group

1076846339

Discord URL.

You can directly ask me questions in Discord.

https://discord.gg/wYfh8kk

What does "fast" mean in fast_io?

It does not necessary mean it will be faster than everything else. (Or it would be named as fastest_io. Of course those fmt trolls would claim that.) It means it is significantly faster than stdio.h and iostream for all cases. fast means faster than stdio.h and iostream.

Hello World

#include<fast_io.h>

int main()
{
	print("Hello World!\n");
}

Examples

Deprecated examples are in https://bitbucket.org/ejsvifq_mabmip/fast_io_deprecated

However, they might not work current fast_io since fast_io keeps changing.

This i18n repo stores the i18n source files. https://bitbucket.org/ejsvifq_mabmip/fast_io_i18n

Online docs

https://ewindy.gitee.io/fast_io_rst/index.html

Compiler Support

• GCC 11 or GCC 12
• VS 19.29.29917
• Clang 14
• Intel(R) oneAPI DPC++ Compiler 2021.2.0 (2021.2.0.20210317)
• Winegcc

C++ standard library support

• GCC libstdc++
• LLVM libc++
• MSVC STL

Architecture Support

• alpha
• i686
• x86_64
• arm
• aarch64
• riscv
• riscv64
• powerpc64
• wasm32-wasi
• mips-abi32
• mips-abin32
• mips-abi64
• avr
• ia64
• All architectures should work. (Just i do not have time to build all cross-toolchains to test them one by one)

Platform Support

• Windows 11
• Windows 10
• Windows Server 2019
• Linux
• BSD platforms (Including FreeBSD, NetBSD, OpenBSD)
• Apple Darwin (MacOS)
• Apple IOS
• Microsoft DOS
• FreeDOS
• Barebone (No any operating system.)
• Cygwin/MSYS2
• Android
• Wine
• Wine GCC
• ReactOS
• Windows Runtime
• Nintendo DS (devkitPro)
• Webassembly
• Managarm
• Vinix
• limine Bootloader
• AVR
• Banana
• LemonOS
• LLVM raw_fd_ostream
• Windows Kernel Driver
• Linux Kernel Module
• Any other platform (if you want me to support it, tell me, including your own OS)

Kernel Driver Support

• Windows NT kernel Driver
• Linux kernel Module

Legacy Windows Version Support (need define _WIN32_WINNT and _WIN32_WINDOWS version by yourself)

• Windows 95
• Windows 95 Plus!
• Windows Nashville
• Microsoft Memphis
• Windows 98 FE
• Windows 98 SE
• Windows ME
• Microsoft Neptune
• Windows NT
• Windows Odyssey
• Windows 2000
• Windows Whistler
• Windows XP
• Windows Server 2003
• Windows Longhorn
• Windows Vista
• Windows Server 2008
• Windows Vienna
• Windows 7
• Windows 8
• Windows 8.1
• Windows Home Server 2011
• Windows Server 2012
• Windows Server 2016

Windows 95 or 95 Plus! Support

You can enable -O3 but you cannot not enable -Ofast. Also need to install msvcrt.dll

Legacy Windows Version Support with DJGPP toolchain

• Windows 1.0
• Windows 2.0
• Windows 3.0
• Windows 3.1
• Microsoft Chicago
• Windows 95
• Windows 95 Plus!
• Microsoft Memphis
• Windows 98 FE
• Windows 98 SE
• Windows ME

Supported libc FILE* hacks platforms

• glibc
• MSVCRT
• Universal CRT
• Universal CRT with MSVC debugging
• Wine's Emulated MSVCRT
• Wine's Emulated UCRT
• ReactOS Emulated MSVCRT
• ReactOS Emulated UCRT
• BSD libc/Apple Darwin
• MUSL libc
• DJGPP
• Newlib
• Bionic(Android)
• uclibc-ng
• wasi libc
• mlibc (this is a new libc which still contains a lot of bugs currently)
• avrlibc
• picolibc

Supported C++ standard library std::streambuf/std::filebuf hacks platforms

• GCC libstdc++
• LLVM libc++
• MSVC STL

Remote Debugger Support

• WinDbg
• Visual Studio Debugger
• DbgViewer
• Android ADB

Special Support

• LLVM Address Sanitizer
• Full freestanding support (which means we do not have any headers besides freestanding headers. Tested with GCC's --disable-hosted-libstdcxx without any libc, including newlib)

Design Goal

A general purpose I/O library to replace stdio.h and iostream

• As close to system call as possible.
• No default locale. It is optional.
• Correctly deal with EBCDIC exec-charset (where no libc correctly deals it).
• OOP is not allowed. Yes to C with Concepts. No to C with Classes.

Safe

• No easily misused stuff like std::endl
• No internal iomanip states (since it creates security issues)
• Providing RAII for C FILE*&POSIX fd&win32 HANDLE
• No dual error reporting mechanism. Exception as the ONLY error reporting mechanism.
• No std::error_code and std::system_error
• No useless bounds checking
• No format string nonsense
• Addresss Sanitizer special code for protecting memory safety issues.

Easy to use

• Unicode/UTF-8 support
• Support POSIX iconv. You can use fast_io for your coding conversion.
• Compatible with C stdio and C++ iostream
• Binary serialization for trivially copyable types and C++ standard library containers
• All fast_io devices can be natively put in C++ containers. std::vector<fast_io::obuf_file> is valid
• Basic/Lua/Python/etc format (print, scan). No support to C and C++ since they are security hazards.
• Static I/O manipulator
• Provide APIs to expose the internal implementation of C FILE* and C++ stream.

Customizability

• Exception Safe & Exception neutral
• Native Handle Interface
• Extremely easy to support custom devices
• std::mutex mutex stream
• Compilation time open mode parse. Supports C style open mode and C++ style open mode.
• No traits_type and EOF
• Dynamic Type Support
• Multi Process
• Memory map
• debugging IO
• GUI debugging IO
• Freestanding mode
• Round-trip floating point algorithm
• Network
• Intrinsic SHA-1, Intrinsic HMAC-SHA1, Intrinsic SHA-256, Intrinsic HMAC-SHA256, SHA-512, HMAC-SHA512
• zlib compression/decompression support
• Non crypto hash algorithms support. Jenkins Hash
• Filesystem
• OpenSSL BIO, Qt QFile, MFC CFile support

Codecvt for File I/O support, including correctly dealing with execution charset of them

• UTF-8
• UTF-16LE
• UTF-16BE
• UTF-32LE
• UTF-32BE
• GB18030
• UTF-EBCDIC
• posix iconv is also supported

Post C++20 Plan

1. Module support
2. Coroutine support for async IO
3. Improve and refactor code once Zero-overhead deterministic exceptions are added to the standard

Possible Improvements

2. Interfaces for cloud computing algorithms like MapReduce

I'm aiming for this to be in the C++ standard library in the future :)

Getting Started

Please see examples in the examples folder.

compile option:

g++ -o example example.cc -Ofast -std=c++20 -s -flto -march=native

Documentation

See Wiki Page: https://gitee.com/qabeowjbtkwb/fast_io/wikis

Benchmarks

1. I/O 10M integers

Goal: Print out ten million integers from 0 to 10M to file. Then reopen that file to scan back.

All benchmarks are in benchmarks/0000.10m_size_t/unit.

Notice: I modified libstdc++'s BUFSIZ 1048576 due to BUFSIZE is too small (512 bytes) for MinGW-W64, or it performs horribly.

Rust language is 10x slower than fast_io. + binary bloat and itoa library is still extremely slow and usable for me. It is at least 3x slower than fast_io.

Run the same test on MSVC 19.26.28805.

Run the same test on GCC 11. glibc + libstdc++

You can see fast_io can also boost the performance of existing facilities for 10x! Yes, it can even improve FILE* and fstream's performance for 10x depending on platforms since I use concepts to abstract them all. fmtlib actually slows down I/O performance.

2. Output 10M double in round-trip mode with Ryu algorithm

We only perform this test for MSVC since only msvc's charconv implements it. Yes. fast_io defeats msvc's charconv for over 20% for running the same algorithm.

All benchmarks are in benchmarks/0001.10m_double/charconv.

Run the same test on MSVC 19.26.28805.

3. Raw I/O Performance

All benchmarks are in benchmarks/0014.file_io/file_io.

Output 100000000x "Hello World\n"

Notice: I modified libstdc++'s std::filebuf's BUFSIZ to 1048576 due to BUFSIZE is too small (512 bytes) for MinGW-W64 or it performs horribly.

4. Binary Size Just use the benchmark in benchmarks/0014.file_io/file_io. Dude, you should avoid stream as plague tbh. It is not healthy.

5. Code Convert

Generate 100000000 🚄 emojis by using the program in benchmarks/0020.utf/fill_nc.cc

Benchmarks are in examples/0043.iconv Universal iconv. (UTF-8 to GB18030 as an example) iconv test:

UTF8->UTF16LE

Benchmarks are in examples/0022.utf

iconv test:

UTF8->UTF32LE

Benchmarks are in examples/0022.utf

iconv test:

Credit

This project is made possible by referencing from other open-source projects. (I won't take their code directly. They are reimplemented by myself to fit the purpose of this library or it might have integration issues.)