STXXL: Standard Template Library for Extra Large Data Sets

Overview
DESCRIPTION

STXXL is an implementation of the C++ standard template library STL for
external memory (out-of-core) computations, i. e. STXXL implements containers
and algorithms that can process huge volumes of data that only fit on disks.
While the closeness to the STL supports ease of use and compatibility with
existing applications, another design priority is high performance.


DOCUMENTATION

See the Doxygen documentation for installation manual and programmer
documentation: http://stxxl.sourceforge.net


LICENSE TERMS

STXXL is distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or copy at
http://www.boost.org/LICENSE_1_0.txt)


BUGS AND QUESTIONS

If you find any bugs or have any questions, please visit the forums at

http://sourceforge.net/projects/stxxl/forums
Issues
  • No Mingw support?

    No Mingw support?

    I tried to compile stxxl under mingw-64 (from mingw-builds). Had one error in lib/io/wfs_file_base.cpp , added naive fix

    @@ -73,12 +73,11 @@ static HANDLE open_file_impl(const std::string& filename, int mode)
             dwFlagsAndAttributes |= FILE_FLAG_NO_BUFFERING;
             // TODO: try also FILE_FLAG_WRITE_THROUGH option ?
     #else
             if (mode & file::REQUIRE_DIRECT) {
                 STXXL_ERRMSG("Error: open()ing " << filename << " with DIRECT mode required, but the system does not support it.");
    -            file_des = INVALID_HANDLE_VALUE;
    -            return;
    +            return INVALID_HANDLE_VALUE;
             }
             else {
                 STXXL_MSG("Warning: open()ing " << filename << " without DIRECT mode, as the system does not support it.");
             }
     #endif
    

    Then it compiled successfully but runs with multiple problems: does not support DIRECT mode (MSVC does):

    [STXXL-MSG] Warning: open()ing C:\Users\ak\AppData\Local\Temp\stxxl.tmp without DIRECT mode, as the system does not support it.
    

    and often fails with mutex

    [extractor] Confirming/Writing used nodes     ... terminate called after throwing an instance of 'stxxl::resource_error'
      what():  Error in stxxl::mutex::~mutex() : pthread_mutex_destroy(&m_mutex) : Resource device
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    

    (tried to compile and run https://github.com/DennisOSRM/Project-OSRM software)

    Just wanted to know, is there any plans to support mingw compiler?

    opened by alex85k 23
  • Support newer Mingw

    Support newer Mingw

    It seems I have accdentally fixed MINGW support :)

    It would be nice to have it "out of the box". The change tells cmake to use STD threads and DIRECT io by default on MINGW.

    Tested on http://sourceforge.net/projects/mingwbuilds/files/host-windows/releases/4.7.3/64-bit/threads-posix/sjlj/

    opened by alex85k 8
  • Does stxxl's containers support embed

    Does stxxl's containers support embed "std::vector" into user-defined type?

    Hi all,

    Greetings from me!

    I write a simple program using stxxl's vector:

    #include <stxxl/vector>
    #include <vector>
    #include <iostream>
    
    class Foo
    {
    public:
        std::vector<double> vec;
        Foo()
        {
        }
    };
    
    int main()
    {
        typedef stxxl::VECTOR_GENERATOR<Foo>::result vector;
        vector my_vector;
        std::cout.precision(10);
    
        for (int i = 0; i < 1024 * 1024 * 1024; i++)
        {
            my_vector.push_back(Foo());
            my_vector[i].vec.push_back(i);
        }
    
    
        std::cout << my_vector[99].vec[0] << std::endl;
    
        return 0;
    }
    

    You can see the user-defined type Foo contains a std::vector. I expect the running result is 99, but in fact the output is 1073737827 . I guess the stxxl's vector did "resize" operation, but not sure.

    So does stxxl's containers support embed std::vector into user-defined type? Thanks very much in advance!

    Best Regards Nan Xiao

    opened by NanXiao 4
  • stxxl 1.3.1: crash after unsuccessful resize of stxxl::vector

    stxxl 1.3.1: crash after unsuccessful resize of stxxl::vector

    If stxxl::vector::resize(n) throws an exception due to insufficient disk space the vector object will crash in it's destructor. Looking at the crash in a debugger I see the stack like this: block_manager::delete_blocks(begin, end) -> block_manager::delete_block(bid) -> bid.is_managed() // here and the reason for the crash is that bid.storage is null. The problem is gone after I changed https://github.com/stxxl/stxxl/blob/1.3.1/include/stxxl/bits/mng/mng.h#L208 line 208 from if (!bid.is_managed()) to if (!bid.valid() || !bid.is_managed()) Would be great to know if this problem is fixed in 1.4.x?

    opened by datjko 4
  • stxxl::vector does not work if used as dynamic array

    stxxl::vector does not work if used as dynamic array

    Hi,

    I plan to use your library within my research projects, but I found a bug (or misusage) that prevents me from using it.

    The bug concerns the fact that vector cannot be used unless the size is specified beforehand (which is impossible in my case).

    To reproduce the bug, I wrote the following program:

    #include <iostream>
    #include <cstdlib>
    #include <stxxl/vector>
    
    typedef struct Triple {
    long t1, t2, t3;
    } Triple;
    
    typedef stxxl::VECTOR_GENERATOR<Triple>::result vector_type;
    
    int main(int argc, const char** argv) {
    vector_type vector;
    vector_type::bufwriter_type writer(vector);
    for(int i = 0; i < 1000000000; ++i) {
        std::cout << "Inserting element " << i << std::endl;
        Triple t;
        t.t1 = rand();
        t.t2 = rand();
        t.t3 = rand();
        writer << t;
    }
    writer.finish();
    }
    

    This program is supposed to create a very large array of Triple objects (three longs). Now, if I launch it the following happens:

    Inserting element 0 Inserting element 1 ... Inserting element 174761 Inserting element 174762 Segmentation fault: 11

    I'm currently using macos 10.9, stxxl v1.4.0 (prerelease/Release) and g++ --version is:

    Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix

    Thanks very much!

    bug 
    opened by jrbn 4
  • Syscall param write(buf) points to uninitialised byte(s)

    Syscall param write(buf) points to uninitialised byte(s)

    Hello,

    i have spent hours of trying to debug one class i am using stxxl::vector in. If i substitute with STL vector everything works. All variables are initialized, i dont know what is memcheck complaining about. Thank you for your help.

    //these are properties of EdgeIndex class
    stxxl::vector<WayInfo> way_info;
     sparse_hash_map<uint64_t, uint32_t> way_info_index; //OSM way id -> index to way_info
    
    //i am trying to do 
    void DiskSerializer::insertWayToIndex(const osmium::Way &way)
    {
    	WayInfo way_info;
    	//add way nodes
    	for (const osmium::NodeRef& nr : way.nodes())
    	{
    		OSMNodeID id = nr.ref();
    		way_info.nodes.push_back(id);
    	}
    	OSMWayID way_id = way.id();
    	edge_index->insertWayInfo(way_id, way_info);
    };
    void DiskEdgeIndex::insertWayInfo(OSMWayID &way_id, WayInfo &way_inf)
    {
        pair<uint64_t,uint32_t> way_id_index = make_pair(way_id,way_info.size());
        way_info.push_back(way_inf);    
        way_info_index.insert(way_id_index);
    };
    

    The stack trace

    STXXL-MSG] Disk '/home/doma/Desktop/test_files/monaco-latest.osm.pbf.stxxl' is allocated, space: 10240 MiB, I/O implementation: syscall direct=on queue=0 devid=0 unlink_on_open
    [STXXL-ERRMSG] stxxl::aligned_alloc: disabling realloc()
    ==2627== Thread 7:
    ==2627== Syscall param write(buf) points to uninitialised byte(s)
    ==2627==    at 0x52A068D: ??? (syscall-template.S:84)
    ==2627==    by 0x3AF83A: stxxl::syscall_file::serve(void*, unsigned long long, unsigned long long, stxxl::request_interface::request_type) (syscall_file.cpp:92)
    ==2627==    by 0x3C5007: stxxl::serving_request::serve() (serving_request.cpp:57)
    ==2627==    by 0x3C1FA9: stxxl::request_queue_impl_qwqr::worker(void*) (request_queue_impl_qwqr.cpp:162)
    ==2627==    by 0x5297709: start_thread (pthread_create.c:333)
    ==2627==    by 0x63470AE: clone (clone.S:105)
    ==2627==  Address 0xfe0eff8 is 2,101,176 bytes inside a block of size 67,112,976 alloc'd
    ==2627==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==2627==    by 0x2F14CC: void* stxxl::aligned_alloc<4096ul>(unsigned long, unsigned long) (aligned_alloc.h:58)
    ==2627==    by 0x2EDC13: stxxl::typed_block<2097152u, WayInfo, 0u, void>::operator new[](unsigned long) (typed_block.h:322)
    ==2627==    by 0x2E8BAC: stxxl::simple_vector<stxxl::typed_block<2097152u, WayInfo, 0u, void> >::simple_vector(unsigned long) (simple_vector.h:70)
    ==2627==    by 0x2E443F: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::allocate_page_cache() const (vector.h:1011)
    ==2627==    by 0x2E15FF: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::vector(unsigned long long, unsigned long long) (vector.h:966)
    ==2627==    by 0x2DCC13: DiskEdgeIndex::DiskEdgeIndex() (DiskEdgeIndex.cpp:4)
    ==2627==    by 0x30BA2D: main (parser.cpp:106)
    ==2627==  Uninitialised value was created by a heap allocation
    ==2627==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==2627==    by 0x2F14CC: void* stxxl::aligned_alloc<4096ul>(unsigned long, unsigned long) (aligned_alloc.h:58)
    ==2627==    by 0x2EDC13: stxxl::typed_block<2097152u, WayInfo, 0u, void>::operator new[](unsigned long) (typed_block.h:322)
    ==2627==    by 0x2E8BAC: stxxl::simple_vector<stxxl::typed_block<2097152u, WayInfo, 0u, void> >::simple_vector(unsigned long) (simple_vector.h:70)
    ==2627==    by 0x2E443F: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::allocate_page_cache() const (vector.h:1011)
    ==2627==    by 0x2E15FF: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::vector(unsigned long long, unsigned long long) (vector.h:966)
    ==2627==    by 0x2DCC13: DiskEdgeIndex::DiskEdgeIndex() (DiskEdgeIndex.cpp:4)
    ==2627==    by 0x30BA2D: main (parser.cpp:106)
    ==2627== 
    ==2627== 
    ==2627== HEAP SUMMARY:
    ==2627==     in use at exit: 271,491 bytes in 2,803 blocks
    ==2627==   total heap usage: 104,163 allocs, 101,360 frees, 552,918,871 bytes allocated
    ==2627== 
    ==2627== LEAK SUMMARY:
    ==2627==    definitely lost: 3,040 bytes in 25 blocks
    ==2627==    indirectly lost: 1,943 bytes in 34 blocks
    ==2627==      possibly lost: 912 bytes in 3 blocks
    ==2627==    still reachable: 265,596 bytes in 2,741 blocks
    ==2627==                       of which reachable via heuristic:
    ==2627==                         newarray           : 1,560 bytes in 3 blocks
    ==2627==         suppressed: 0 bytes in 0 blocks
    ==2627== Rerun with --leak-check=full to see details of leaked memory
    ==2627== 
    ==2627== For counts of detected and suppressed errors, rerun with: -v
    ==2627== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
    
    opened by ghost 3
  • Next release?

    Next release?

    I notice that v1.4.1 is the latest release, but there have been over 350 commits to master since: https://github.com/stxxl/stxxl/releases/tag/1.4.1.

    Are there any plans for a new official release or an anticipated date?

    opened by springmeyer 3
  • spin_lock misses the default constructor on newer MSVC

    spin_lock misses the default constructor on newer MSVC

    When compiling under new Visual Studio 2015 RC I am getting the error about missing default constructor in spin_lock class. It is really turned off for MSC_VER 1800 and 1900 . Should it be returned in #else block?

    https://github.com/stxxl/stxxl/blob/master/include/stxxl/bits/common/mutex.h#L167

    opened by alex85k 3
  • Deprecated hash_map header

    Deprecated hash_map header

    Newer Visual Studio 2015 RC stopped compilig deprecated hash_map header

    C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\hash_map(17): error C2338: <hash_map> is deprecated and will be REMOVED. Please use <unordered_map>. You can define _SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\winbuilds\stxxl\build\tools\stxxl_tool.vcxproj]
    

    I guess that <unordered_map> should be activated in https://github.com/stxxl/stxxl/blob/master/include/stxxl/bits/compat/hash_map.h#L24 for _MSC_VER >= 1900 . Not sure about the best solution, so there is no PR.

    opened by alex85k 3
  • Fix for DllMain() initialization deadlock

    Fix for DllMain() initialization deadlock

    [Only affects Windows - Visual Studio configuration]

    In the scenario when stxxl is being initialized in an DllMain() the application may or may mot deadlock, this patch fixes that.

    opened by ghost 3
  • can the container be loaded back from file created by stxxl

    can the container be loaded back from file created by stxxl

    Write the index stxxl::config * cfg = stxxl::config::get_instance(); stxxl::disk_config disk1("disk=/tmp/lsh.index, 10 GiB, syscall unlink"); disk1.direct = stxxl::disk_config::DIRECT_ON; // force O_DIRECT disk1.delete_on_exit = false; cfg->add_disk(disk1);

    What could be the possible way to read the file "/tmp/lsh.index" ?

    opened by cahuja1992 2
  • Guard definition of log2 on versions of Visual C++ that provide it.

    Guard definition of log2 on versions of Visual C++ that provide it.

    Resolves:

    C:\PROGRA~1\MICROS~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\arm64\cl.exe   /TP -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -D_SCL_SECURE_NO_WARNINGS -ID:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\include -ID:\buildtrees\stxxl\arm64-windows-rel\include /nologo /DWIN32 /D_WINDOWS /W4 /utf-8 /GR /EHsc /MP  /wd4127 /wd4290 /wd4250 /wd4512 /wd4355 -openmp /MD /O2 /Oi /Gy /DNDEBUG /Z7 /showIncludes /Fotools\CMakeFiles\stxxl_tool.dir\benchmark_sort.cpp.obj /Fdtools\CMakeFiles\stxxl_tool.dir\ /FS -c D:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\tools\benchmark_sort.cpp
    D:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\include\stxxl/bits/msvc_compatibility.h(23): error C2169: 'log2': intrinsic function, cannot be defined
    ninja: build stopped: subcommand failed.
    
    opened by BillyONeal 0
  • Is STXXL still maintained?

    Is STXXL still maintained?

    Hi, We are using stxxl as a submodule in our project QLEVER. stxxl currently only builds with warnings in C++17 mode, and does not build at all in C++20 mode (using G++ 10). I am able to locally fix the issues and would like to contribute them here as a PR, however I am not sure if this project is still active.

    • It's been almost 3 years since the last activity
    • @bingmann who seems to be the main contributor to this system maintains his own fork of stxxl , which has diverged quite a bit from this repository, but also there the last commit is 2 years old.

    Is stxxl still being actively maintained, and if yes, on which repository and by whom?

    opened by joka921 0
  • [Build] Build failure with boost-system

    [Build] Build failure with boost-system

    Hi guys, today I got a regression when building stxxl:

    D:\installed\x86-windows\include\boost/system/detail/system_category_message_win32.hpp(53): error C2039: '_snprintf': is not a member of 'std'
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\include\mutex(30): note: see declaration of 'std'
    

    When I dig into this issue, I found that even if I set USE_BOOST to OFF, if HAVE_STD_THREAD or HAVE_STD_MUTEX is OFF, boost will be forced to be enabled [1]. Before using boost, stxxl defined snprintf as _snprintf [2], which caused the std::snprintf search statement in boost-system to fail [3].

    So I think that's a source bug.

    See boost official reply https://github.com/boostorg/system/issues/32.

    [1]. https://github.com/stxxl/stxxl/blob/b9e44f0ecba7d7111fbb33f3330c3e53f2b75236/CMakeLists.txt#L347-L352 [2]. https://github.com/stxxl/stxxl/blob/b9e44f0ecba7d7111fbb33f3330c3e53f2b75236/include/stxxl/bits/msvc_compatibility.h#L28 [3]. https://github.com/boostorg/system/blob/689e09e86ff7f4943a1c8236d316ba4608ec3142/include/boost/system/detail/system_category_message_win32.hpp#L53

    opened by JackBoosY 0
  • Build fixes for C++17 (due to errors in Visual Studio)

    Build fixes for C++17 (due to errors in Visual Studio)

    There are several std classes/functions that are removed in C++17 and causes build errors. This PR removes and replaces them as necessary. In addition there are a few build warnings (in particular recommending enum class) that are fixed as well.

    Please check that this causes no problems/changes on Linux

    opened by ssaue 0
  • Fixed C++17 compilation.

    Fixed C++17 compilation.

    I've update the code for handling with success C++17 standard. I've removed references to std::binary_function, std::random_shuffle and std::bind2nd. I've also fixed a __cplusplus issue with Visual Studio 2019 and now it creates unique_ptr.

    opened by jepessen 0
Releases(1.4.1)
  • 1.4.1(Jun 15, 2015)

    Version 1.4.1 (29 October 2014)

    • support kernel based asynchronous I/O on Linux (new file type "linuxaio"), which exploits Native Command Queuing (NCQ) if available. disable/enable with the define STXXL_FILE_LINUXAIO 0/1 via cmake
    • adding new disk_config entry device_id, which specifies the physical device id of the "disk" used during prefetching sequence calculations. This used to be identical with the queue id, however, for linuxaio there is only one queue; thus the distinction had to be made. In a default config, no changes are necessary, as the device_id parameter is automatically enumerated.
    • adding stxxl::binary_buffer which can be used for compact serialization and reading via a stxxl::binary_reader cursor interface.
    • stxxl::unordered_map is a hash map, backed by external memory. It probably only works well when lots of internal memory is used to buffer access to a relatively small working set. Then, however, fast direct in-memory item access can be used.
    • stxxl::external_shared_ptr is a proxy class to allow use of shared_ptr classes inside stxxl containers. Reference counts are kept in memory, while data may be swapped out to disk.
    • removing struct default_completion_handler, using a NULL pointer in default complete handler instead, since otherwise a default_completion_handler objects is cloned for each_io_request! Using a NULL pointer avoids superfluous new/delete work on the heap.
    • minor changes:
      • disable TPIE benchmarks by default, removing a warning.
      • compilation and tests work under MinGW 64-bit with special threads.
      • fixed compilation on 32-bit systems, everything is -Wconversion safe.
      • adding 32-bit and 64-bit cmdline_parser::add_bytes() variants.
      • removing all double underscores.
      • use atomic_counted_object in class file for request reference counting.
      • adding local/test2.cpp containing a stxxl::sorter example.
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.4.1-doxygen.tar.gz(6.44 MB)
    stxxl-1.4.1.tar.gz(1.33 MB)
  • 1.4.0(Jun 15, 2015)

    Version 1.4.0 (12 December 2013)

    • Reorganized Directory Hierarchy
      • separating old component based directories into lib/ (libstxxl source files), tests/ (for tests cases), examples/ (for tutorial and example code) and tools/ (for stxxl_tool and more benchmarks).
    • CMake build system for cross-platform compilation
      • replaced adhoc Makefiles with CMake build system
      • CMake scripts can also output MSVC project files and more.
      • all component tests must register with CMake's test system -> run make test
      • generates pkg-config and CMake's project config files
    • Greatly Improved Doxygen Documentation
      • integrated old LaTeX tutorial
      • newly written tutorials for almost all STXXL containers and algorithms
      • embedded design documentation from Roman Dementiev's PhD thesis
      • new compilation instructions for CMake build system
    • Improved .stxxl Config Options for File I/O Implementations
      • looking for .stxxl in more places than just the current directory (see the docs for which places exactly).
      • parse file size with SI and IEC units, like "10GiB" in disk= config lines.
      • parse additional file I/O implementation options.
      • support for "unlink_on_open" (keep file descriptor but unlink file) and "delete_at_exit" options.
      • support for "direct=off/try/on" to turn off O_DIRECT for tmpfs and similar.
      • the default open mode is now direct=try -> prints a warning if O_DIRECT is unavailable on the system.
    • .stxxl Config Options Represented as disk_config Class
      • all disk= config lines are parsed into disk_config objects.
      • allow API calls to configure disks via stxxl::config::add_disk()
      • delay config file load till config::initialize() is called by block_manager
    • Command line parser stxxl::cmdline_parser
      • stand-alone command line parser with ability to parse SI and IEC units for "bytes sizes".
      • automatically output nicely formatted help for all options and flags.
      • using the cmdline_parser for all stxxl_tool subtools.
    • Created stxxl_tool from multiple small tools and benchmarks
      • stxxl_tool is a collection of subtools, previous scattered around in the directory hierarchy.
      • goal is to have one tool installed in binary distribution together with libstxxl library and includes.
      • primary tool for standardized platform speed tests.
      • provide good command line help on these subtools via cmdline_parser.
    • Adding skew3 as "real-world" STXXL application to examples subdirectory.
      • enhanced integrated stxxl::tuple construction for this application.
    • Enable use of raw block devices on Linux
      • these do provide more stable I/O throughput than even XFS.
      • disable growing of these devices
    • Most platform configuration is now done by CMake and saved into the <stxxl/bits/config.h> header file
    • Enhanced stxxl::block_manager to count maximum, current and total number of bytes allocated in external memory.
    • Support Visual Studio 2012 and 2013 without Boost by using std::thread
      • VS 2012 and 2013 support STL threads, mutexes, condition_variables and more, which are basically modeled after the Boost libraries. Thus we can simply uses these included libraries instead of Boost.
      • Fixed bad runtime deadlock bug occurring when joining std::threads after main() exits. The bug persists in VS 2012 and 2013.
    • Implementing counting_ptr reference counted objects
      • used like boost::intrusive_ptr with a base class.
      • reimplementing request_ptr using just counting_ptr.
      • (optionally) use atomic operations for thread-safe reference counting
      • applying counting_ptr to stream::sorted_runs to avoid extra copying
    • New STXXL container "stxxl::sequence"
      • basically a deque container without random access to blocks.
      • efficient {push/pop}_{front/back} at both ends.
      • access to block sequence via stream and reverse_stream constructions
    • New STXXL container "stxxl::sorter"
      • combination of stream::runs_creator and stream::runs_merger into a convenient sorting container.
    • Added vector_bufreader and vector_bufwriter to overlapping I/O and computation when accessing a stxxl::vector in scanning mode.
      • documented problems of scanning in a vector
      • C++11 range loop access support to stxxl::vector via vector_bufreader.
    • Added uint40 and uint48 types for integers with five and six bytes.
    • Adding STXXL_CHECK macros to always check outcomes in tests.
    • Renaming mng/mng.{h,cpp} -> mng/block_manager.{h,cpp}.
    • Reworking version.h
      • add integer version like 10400 for 1.4.0
      • add get_version_string() for longer version string with features
      • check_library_version() for mismatch between headers and libraries
    • Reworking error_handling.h to throw much more useful exceptions with errno information and more.
    • Various file renames (which are only possible at major version changes).
    • Fixing bug in shrinking vector with vector::resize(n,true).
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.4.0-doxygen.tar.gz(10.83 MB)
    stxxl-1.4.0.tar.gz(1.35 MB)
  • 1.3.1(Jun 15, 2015)

    Version 1.3.1 (10 March 2011)

    • Possibly breaking changes
      • No longer open syscall files with O_SYNC|O_DSYNC|O_RSYNC by default when doing direct I/O, to avoid a write performance decrease starting with Linux kernel version 2.6.32. A stxxl::file::SYNC flag has been added to allow forcing the O_*SYNC flags when opening files.
      • Lock files by default (if implemented for this file type), can by disabled via stxxl::file::NO_LOCK.
      • block_prefetcher now takes a completion_handler instead of a raw function pointer. Furthermore, the completion_handler is already executed after the read operation completes, not only when the issuer waits for it to complete. The I/O thread issues the call to the completion_handler, so for multiple disks, this may happen concurrently.
    • Internal changes, not user-visible
      • Change handling of (empty) filler elements in typed_block to fix mismatching struct size in nested blocks.
      • Removed debugmon which was very limited and disabled for a long time.
    • Bugfixes
      • Allow prefetch schedule computation of vectors that are bound to a file.
      • Fix race growing files during concurrent block allocation.
      • Allow reading a block that spans over end-of-file, fill remainder with zeroes.
      • Crash at program termination when using global stxxl containers.
      • Enable syscall_file to read/write >=2GiB of data in a single I/O operation.
    • New public headers: stxxl/stats, stxxl/request
    • Parallel mode can be switched on for internal STXXL computation selectively (i.e. without enabling it for the non-stxxl part of an application), by setting USE_PARALLEL_MODE=yes in make.settings.local.
    • Platform Support
      • add ICPC 12.0, works with both MCSTL (needs libstdc++ from GCC 4.2) and parallel mode (needs libstdc++ from GCC 4.3 (not 4.4/4.5))
      • add CLANG++ 2.8
      • add MSVC 2010/Windows 7
      • allow the library name generated by MSVC to be changed by setting LIBNAME and redefining STXXL_LIBNAME (defaults to "stxxl")
    • Under-the-hood improvements
      • code cleanup
      • I/O-layer: renaming and reorganization of the request implementation
    • Documentation updates
    • Several new test programs
    • Several test programs improved
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.3.1.tar.gz(900.47 KB)
    stxxl-1.3.1.zip(1.07 MB)
  • 1.3.0(Jun 15, 2015)

    Version 1.3.0 (12 August 2010)

    • New file types
      • fileperblock_syscall/fileperblock_mmap/fileperblock_boostfd/fileperblock_wincall: Use a separate file for each block, which is accessed by means of the underlying file type. The "filename" given is used as a prefix of the block file names. The advantage of these file types is that unused disk space is freed earlier on the file system level.
      • wbtl_file: Do library-based write-combining (good for writing small blocks onto SSDs)
    • I/O subsystem
      • separate the disk number of a file (which queue to put requests in) from the fact that blocks for this file are allocated via the block manager (disk number -1 otherwise).
      • separate wait time counting for read and write I/Os
      • wait times can be logged to a separate log file (compile with -DSTXXL_WAIT_LOG_ENABLED and set STXXLWAITLOGFILE in the environment)
    • Parallel PQ
      • priority_queue now supports parallelized operations utilizing the libstdc++ parallel mode
    • Other new functionality
      • file requests can now be canceled (success not guaranteed), completion handlers are called anyway.
      • log file locations are now configurable by the environment variables STXXLLOGFILE and STXXLERRLOGFILE
      • single queue I/O scheduling
      • reverse_iterator added to containers types vector, deque and map
      • autogrow files (specified in .stxxl by a size of 0), will be deleted on normal program termination
      • add infrastructure to build multiple binaries with different CXXFLAGS from a single source
      • overwriting deleted memfile regions with uninitialized data is now optional (STXXL_CLEAR_FREE_MEMFILE_MEM)
      • read_write_pool that combines prefetch_pool with write_pool and ensures cache coherency
      • add a replaceable exit handler implementation, can be overwritten e.g. for library inclusion
    • Many, many bug fixes, in particular concerning
      • priority queue
      • optimal prefetch schedule
      • possible race condition while creating singleton instances
      • random_shuffle()
    • Platform Support
      • add GCC 4.4 (parallel mode features not working in GCC 4.3 can now be used)
      • add GCC 4.5
      • initial support for GCC 4.6
      • add ICPC 11.x
      • add Boost 1.42
      • add FreeBSD
      • drop Solaris
    • New benchmarks
      • io/benchmark_disks: more command line paramaters for finer testing granularity
      • measure hard-disk and flash-disk combined performance and determine best block size pairs (io/benchmark_disk_and_flash)
      • benchmark using regular STXXL configuration (benchmark_configured_disks)
    • Possibly breaking changes
      • API name changes vector::touch -> vector::block_externally_updated
      • Template parameter changes stream::sorted_runs, trigger_entry, trigger_entry_cmp, trigger_entry_iterator
      • Priority of write changes
      • Other name changes (considered internal) typed_block<>::has_filler -> !typed_block<>::has_only_data file::delete_region -> file::discard vector::alloc_strategy -> vector::alloc_strategy_type
      • stxxl::sort(..., M) and stxxl::stream::sort(..., M) now adhere to to the memory limit M more strictly and throw errors instead of overallocating
      • Execute completion handler before notifying waiters, so far after.
    • Deprecated methods:
      • stxxl::stats::reset(), stxxl::stats::_reset_io_wait_time(): use stxxl::stats_data instead to store snapshots of the counters and compute differences
    • Under-the-hood improvements
      • I/O layer cleanup
      • aligned_alloc
    • Doxy now also lists undocumented members
    • stop requiring STXXL_ROOT to be a hard coded absolute path set in make.settings.local, a default of CURDIR works fine
    • document #defines in defines.h
      • switch on/off file types with STXXL_HAVE_xxx_FILE (to avoid platform compatibility problems)
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.3.0.tar.gz(740.76 KB)
    stxxl-1.3.0.zip(929.95 KB)
  • 1.2.1(Jun 15, 2015)

    Version 1.2.1 (14 August 2008)

    • add support for the libstdc++ parallel_mode (successor of MCSTL), new make targets: library_g++_pmode, tests_g++_pmode, clean_g++_pmode (requires g++ 4.3.2 or later)
    • new stxxl file type stxxl::mem_file (for debugging purposes only), helps debugging with valgrind because no memory cell ever leaves valgrind's control
    • properly destroy the singleton objects (block manager, disk queue threads, logger, stats, config, ...) at program exit
    • fixed a bug (recursive locking) in recursive block allocation
    • added a test program for recursive block allocation
    • sensible dependencies for nmake (MSVC): only rebuild files as necessary
    • improve performance of createdisks
    • human-readable output for I/O stats
    • replace hard coded min/max values by std::numeric_limits<>::min/max in examples
    • fix a case where only half of the available memory was used during recursive merging
    • verify stxxl::set_seed() functionality in tests
    • remove stxxl disk files created from default configuration (no .stxxl file found) and other temporary files at program end
    • stop using deprecated functions, switch to the replacements instead
    • unify the mutex lock handling by introducing stxxl::scoped_mutex_lock
    • unify the I/O wait time counting to work like read/write time counting
    • simplify I/O time counting with scoped_{read,write,wait}_timer
    • add some more tests
    • more code cleanup + reformatting
    • move some more code to the library
    • reduce some include dependencies
    • build system tuning
    • propagate region deletion (when blocks are freed) to the file level
    • fix problem in sorters where blocks were released too early
    • specialize is_sorted() to use const_vector_iterators, no extra writes
    • add c++0x style const_iterator cbegin()/cend() to all containers
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.2.1.tar.gz(664.21 KB)
    stxxl-1.2.1.zip(809.73 KB)
  • 1.2.0(Jun 15, 2015)

    Version 1.2.0 (05 July 2008)

    • made the block management layer thread-safe
    • made all size_types unsigned
    • stxxl::priority_queue
      • fixed a rare assertion
      • fixed a race condition by using proper block hinting
      • insert_queue: replaced std::priority_queue with a special implementation internal_priority_queue that allows for fast exporting of all elements
      • even more bugs and inefficiencies fixed
      • significant speed improvements
    • random number generators are now all seedable, should allow redoing identical program runs for debugging purposes
    • stxxl::noncopyable, inspired by boost::noncopyable: inheriting from this class forbids copying of objects when this is undesirable
      • used in all classes that had implemented their own variants previously
    • stxxl::vector, all sorting functions: replaced two2one_dim_array_row_adapter with array_of_sequences_iterator which is much faster, especially if blocks have padding
    • if required, verify that the sentinels satisfy a strict weak ordering
    • stxxl::vector: several operations sped up, several more implemented
    • fix existing allocation strategies and add experimental support for distinguishing between regular disks and flash devices
    • stxxl::stable_ksort
      • add experimental warning, some parts are not yet implemented
      • fixed an off-by-one error in memory allocation vs. usage
    • btree: fuse last two nodes/leaves if possible, rebalancing can fail
    • btree tests: ensure uniqueness of test data if required
    • reduce function call overhead of stxxl::debugmon if it's not activated
    • add public interface headers: stxxl/types, stxxl/aligned_alloc
    • add compatibility wrappers for standard extensions hash_map, hash_set, auto_ptr
    • MCSTL is only supported with g++ 4.2 and icpc 10.x
    • lots of small bugfixes
    • made the TPIE, BDB and LEDA_SM tests compile again
    • general code cleanup
      • fixed most compiler warnings
      • elimination of duplicate and unused code
      • cleaned up and sanitized debugging macros
      • no more 'using namespace std' and 'using namespace stxxl'
      • fixed ambiguities noted by g++ 4.3
      • unify the #include directives
      • add/unify/cleanup copyright headers
    • general code reformatting (uncrustify)
    • add support for new compiler releases
    • portability fixes for different platforms
    • implemented file truncation on windows platforms
    • build system
      • lots of small modifications
      • now only requires GNU make 3.78 or later
      • check whether STXXL_ROOT has been set correctly and if unset, try autoconfiguration by creating make.settings.local with defaults
      • improved and simplified boost support for posix systems
      • Mac OS X support
    • changed all tests so that they behave well in their default parameters, system assumptions and return values and can be run from a script
      • use aligned_alloc/aligned_dealloc appropriately
    • added some more test programs
    • add misc/run_all_tests that runs all tests with sensible parameters, optionally via valgrind
    • checked all tests with valgrind
      • fixed use of uninitialized memory when writing to normal files
      • (optionally) initialize typed_blocks and payload data in test structs to suppress (most) uninitialized write errors when writing to disk files
      • fix mismatched new/delete in mcstl
    • update install and usage instructions
    • spell checked sources and documentation
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.2.0.tar.gz(658.20 KB)
    stxxl-1.2.0.zip(799.16 KB)
  • 1.1.0(Jun 15, 2015)

    Version 1.1.0 (31 July 2007)

    • stxxl is now hosted on SourceForge: http://stxxl.sourceforge.net/
    • Restructured the source layout:
      • includes moves to include/
      • introduced some public headers: stxxl.h, stxxl/algorithm, stxxl/bits, stxxl/deque, stxxl/io, stxxl/ksort, stxxl/mallocstats, stxxl/map, stxxl/mng, stxxl/priority_queue, stxxl/queue, stxxl/random, stxxl/scan, stxxl/sort, stxxl/stable_ksort, stxxl/stack, stxxl/stream, stxxl/timer, stxxl/vector
      • the header "stxxl" is no longer available, please use "stxxl.h" instead
      • the use of any other (internal) header is discouraged, additional public headers can be added as the need arises
    • Overhauled the build system:
      • merged configuration files, simplified option tuning
      • support object files and binaries with and without mcstl support existing in parallel
      • the library build creates stxxl.mk which can be included in an applications Makefile to set the correct compiler/linker switches for stxxl
      • similarly mcstxxl.mk is created for a stxxl compiled with mcstl support
      • add dependency tracking and improve parallelism during build
    • compiler support matrix: (for an up-to-date list, please see the doxygen documentation) compiler | stxxl stxxl + mcstl --------------+------------------------ GCC 4.3 | x x GCC 4.2 | x x GCC 4.1 | x n/a GCC 4.0 | x n/a GCC 3.4 | x n/a GCC 3.3 | o n/a GCC 2.95 | - n/a ICPC 9.1.051 | x x¹ ICPC 10.0.025 | x x¹ MSVC 2005 8.0 | x n/a x = full support o = partial support - = unsupported n/a = compiler does not support OpenMP which is needed by MCSTL ¹ = does not work with STL GCC 4.2.0 (ICPC bug), workaround: the first include in the program must be "stxxl/bits/common/intel_compatibility.h"
    • pipelined stream::sort, stream::runs_creator and stream::runs_merger are parallelized using MCSTL
    • obsolete files removed
    • fixed include dependencies
    • fixed lots of warnings
    • fixed lots of small bugs
    • add header compilation tests
    • stxxl::vector: implemented some more functions
    • const char* stxxl::get_version_string(): new function
    • comparators inherit from std::binary_function<>
    • cleanup, reformat the code to a consistent format (Thanks, uncrustify!)
    Source code(tar.gz)
    Source code(zip)
    stxxl-1.1.0.tar.gz(567.09 KB)
    stxxl-1.1.0.zip(674.57 KB)
Owner
STXXL
STXXL: Standard Template Library for Extra Large Data Sets
STXXL
FSD-Template - A template UE4.25 project for BP modding.

FSD-Template Project generated by Archengius' UE4 Template Generator. Reflected C++ classes generated by CheatingMuppet & Archengius' UE4SS UHT Genera

null 8 Jun 18, 2022
Libft is an individual project at 42 that requires us to re-create some standard C library functions including some additional ones that can be used later to build a library of useful functions for the rest of the program.

?? Index What is Libft? List of Functions Technologies ✨ What is Libft? Libft is an individual project at 42 that requires us to re-create some standa

Paulo Rafael Ramalho 7 Jan 17, 2022
Thrust is a C++ parallel programming library which resembles the C++ Standard Library.

Thrust: Code at the speed of light Thrust is a C++ parallel programming library which resembles the C++ Standard Library. Thrust's high-level interfac

NVIDIA Corporation 4k Jun 22, 2022
jkds is a modern header-only C++20 library that complements the standard library.

jkds is a modern header-only C++20 library that complements the standard library. It provides generic atypical data structures, ergonomic functional programming abstractions, and then some.

Alberto Schiabel 5 May 24, 2022
Bionic BSD-3-ClauseBionic - Google's standard library, developed for Android. BSD-3-Clause

bionic bionic is Android's C library, math library, and dynamic linker. Using bionic as an app developer See the user documentation. Working on bionic

Android Open Source Project 547 Jun 23, 2022
CloudABI's standard C library

NOTE: This project is unmaintained CloudABI is no longer being maintained. It was an awesome experiment, but it never got enough traction to be sustai

Nuxi 272 Jun 7, 2022
libcu++: The C++ Standard Library for Your Entire System

libcu++, the NVIDIA C++ Standard Library, is the C++ Standard Library for your entire system. It provides a heterogeneous implementation of the C++ Standard Library that can be used in and between CPU and GPU code.

NVIDIA Corporation 2k Jun 26, 2022
An open source standard C library that includes useful functions && (Reimplementation of libc functions + own functions).

?? LIBFT-42 : Artistic view of LIBC: ?? HOW DOES IT FEEL HAVING YOUR OWN LIB: SUBJECT : ENGLISH PDF ℹ️ What is LIBFT : This project aims to code a C l

Abdessamad Laamimi 10 Mar 24, 2022
Your standard library for metaprogramming

Boost.Hana Your standard library for metaprogramming Overview #include <boost/hana.hpp> #include <cassert> #include <string> namespace hana = boost::h

Boost.org 1.4k Jun 23, 2022
Reimplementation of some of the Standard C Library functions.

42-libft Reimplementation of some of the Standard C Library functions. This repository contains some of the standard library C functions. List of avai

Lavrenova Maria 5 Jan 26, 2022
MSVC's implementation of the C++ Standard Library.

Microsoft's C++ Standard Library This is the official repository for Microsoft's implementation of the C++ Standard Library (also known as the STL), w

Microsoft 7.9k Jun 27, 2022
mlibc is a C standard library

mlibc is a C standard library Official Discord server: https://discord.gg/7WB6Ur3 Design of the library Directory Purpose options/ (More or less) OS-i

The Managarm Project 495 Jun 20, 2022
A standard conforming C++20 implementation of std::optional.

A standard conforming C++20 implementation of std::optional.

null 29 Feb 11, 2022
RTL - Reactive Template Library for C++

RTL - Reactive Template Library It solves all my C++ animation problems -- me Using RTL Lazy, caching, reactive variables

Lorents Odin Gravås 3 Dec 22, 2021
Modern C++ generic header-only template library.

nytl A lightweight and generic header-only template library for C++17. Includes various utility of all kind that i needed across multiple projects: Ex

Jan 93 Jun 20, 2022
A template C project using CMAKE, logging library and basic memory handling.

C Project template Aim of this Repository is to create a template repository for C executable projects with following properties: Cmake project Loggin

Aditya Singh Rathore 6 May 23, 2022
this lib with 26 template container and 10 kinds of algorithm, it is a good lib for study and usage

simple stl this lib simplify the achievement detail of common container, but add the container variety, the whole code partily follow Google Style. Em

wujiazheng 7 Mar 10, 2022
This is the template for peripheral projects using nRF5 SDK

NRF5_SDK template project About This is the template for peripheral projects using nRF5 SDK Getting Started To get a local copy up and running follow

Tai Nguyen 1 Nov 27, 2021
Code::Blocks template for custom launcher executable.

Launcher Code::Blocks template for custom launcher executables. This is a basic Code::Blocks project for creating authentic Windows executables. Inclu

Federico Cappelletti 1 Feb 5, 2022