Darshan I/O characterization tool

Overview

Darshan is a lightweight I/O characterization tool that transparently captures I/O access pattern information from HPC applications. Darshan can be used to tune applications for increased scientific productivity or to gain insight into trends in large-scale computing systems.

Please see the Darshan web page for more in-depth news and documentation.

The Darshan source tree is divided into two main parts:

  • darshan-runtime: to be installed on systems where you intend to instrument MPI applications. See darshan-runtime/doc/darshan-runtime.txt for installation instructions.

  • darshan-util: to be installed on systems where you intend to analyze log files produced by darshan-runtime. See darshan-util/doc/darshan-util.txt for installation instructions.

The darshan-test directory contains various test harnesses, benchmarks, patches, and unsupported utilites that are mainly of interest to Darshan developers.

Issues
  • Darshan Python Package - [merged]

    Darshan Python Package - [merged]

    In GitLab by @jakobluettgau on Aug 1, 2019, 10:46

    Merges python-package -> master

    Adds a python package including helpers to release to PyPI to the darshan utils source tree under: darshan-utils/pydarshan

    There is a Makefile with all sorts of useful targets, from testing, to documentation to coverage as well as releasing to PyPI.

    make test
    make coverage
    make lint
    ...
    make install
    ...
    make release
    
    • The tests are currently stubs and use py.test because that seems to be the general recommendation for new projects, but that is easy to switch over to nose if that helps with borrowing from pytokio.

    • I have added an experimental darshan.plots submodule, which uses matplotlib to do the access size histogram as well as the I/O operation summary.

    • I have added docstrings to all the CFFI bindings as well as some general instructions which can build easily using make docs / make show-docs. (Just ensure pip install -r requirements_dev.txt)

    • License?: I have blanked out more or less all the license information, but these need to populated when releasing to PyPI.

    • I had some issues, when accessing the STDIO counters multiple times, which would kill or segfault the execution, which I had no time yet to look into in more detail.

    • The code and scripts to generate a CFFI friendly header is currently included in darshan-util/pydarshan/darshan/data/generate_headers

    Example Notebook/Scripts:

    There is a example directory which includes a Jupyter-Notebook which demonstrates how to use the current interface to interact with the logs, as well as the plotting at the end of the notebook.

    gitlab merge request 
    opened by shanedsnyder 62
  • WIP, ENH: draft library code for

    WIP, ENH: draft library code for "Data Access by Category"

    The draft "library code" is currently under version control in this PR, while the (rougher) plotting code that uses the data munging utilities in the lib code is placed below the fold and was used to produce the sample plots below vs. @carns draft drawing. I don't actually know if we need any of this if @jakobluettgau aggregation/counting code does something similar though.

    Edit: the plotting code is mostly under version control now, so can run it with something like this:

    import os
    
    import darshan
    from darshan.experimental.plots import data_access_by_filesystem
    
    if __name__ == '__main__':
        # produce sample plots for some of the logs
        # available in our test suite
        root_path = '/Users/treddy/github_projects/darshan/darshan-util/pydarshan/tests/input'
        #log_files = ['sample-dxt-simple.darshan', 'sample.darshan', 'sample-goodost.darshan']
        log_files = ['sample-goodost.darshan']
        for log_file in log_files:
            log_path = os.path.join(root_path, log_file)
            data_access_by_filesystem.plot_with_log_file(log_file_path=log_path,
                                                         plot_filename=log_file + '.png')
    
    

    image

    galen_anonymous_1 darshan_data_access_by_category galen_anonymous_2 darshan_data_access_by_category sample-dxt-simple darshan_data_access_by_category sample-goodost darshan_data_access_by_category sample darshan_data_access_by_category treddy_mpi-io-test_id545458_6-3-43062-1247823192538596076_13 darshan_data_access_by_category

    enhancement pydarshan 
    opened by tylerjereddy 60
  • WIP: PyDarshan: Adds heatmap plotting code for DXT tracing logs

    WIP: PyDarshan: Adds heatmap plotting code for DXT tracing logs

    DXT heatmap plotting code for potential use in updated job summary. Primarily uses numpy, seaborn, matplotlib for plotting, which are all pip installable.

    Summary

    At the moment it takes in logs with DXT_POSIX data, concatenates all of the read/write segment dataframes together, then uses that data (along with the respective rank data) to create a 2D histogram, and plot a heat map with marginal bar graphs. Depending on which operations you include, it will plot only the read segment data, the write segment data, or both. It does not distinguish between operations. Each bin (in the heatmap) is populated with the data sum and/or proportionate data sum for all IO events read/written during the time spanned by the bin.The color bar was selected based on the fact that it is non-white at the minimum value so areas without data are visible.

    How to run

    See: https://github.com/darshan-hpc/darshan/pull/396#issuecomment-879202907

    TODO

    • [x] Fix average time plotting -- need to span from start time to end time for accurate IO representation
    • [ ] ~~Find a way to aggregate data from other modules, if applicable~~
    • [ ] ~~Differentiate between read/write data in heat map~~
    • [x] Include number of bins in plot somewhere so you can get an idea of bandwidth (i.e. 100 bins w/ 100 s run time = 1 s/bin)
    • [ ] Figure out if/how to utilize the figure title. At the moment it just states which modules are being used (may want to remove it entirely for the job summary)
    • [x] Finish documentation/comments
    enhancement pydarshan 
    opened by nawtrey 46
  • Python Package: Rename mode switch to dtype to select datatype. Simplify interface log_get_record. Have lustre records include OSTs. Add library version check. - [merged]

    Python Package: Rename mode switch to dtype to select datatype. Simplify interface log_get_record. Have lustre records include OSTs. Add library version check. - [merged]

    In GitLab by @jakobluettgau on Sep 3, 2020, 12:53

    Merges python-package -> master

    Some updates streamlining the interface and a bugfix:

    • Rename mode switch to dtype to select datatype (numpy, dict, pandas). E.g., log_get_record(log, 'POSIX', dtype='pandas')
    • Simplify interface log_get_record. Deprecates log_get_posix_record, etc.
    • Include version check to warn about library incompatability.

    Fix:

    • Lustre records did not unpack list of OSTs.
    gitlab merge request 
    opened by shanedsnyder 29
  • use automake and libtool

    use automake and libtool

    Re-creating @wkliao's PR from our GitLab repo (https://xgitlab.cels.anl.gov/darshan/darshan/-/merge_requests/77) here to continue to track Darshan's transition to automake/libtool.

    opened by shanedsnyder 23
  • WIP: darshan instrumentation for apps that do not use MPI - [closed]

    WIP: darshan instrumentation for apps that do not use MPI - [closed]

    In GitLab by @glennklockwood on Nov 25, 2018, 00:38

    Merges no-mpi -> master

    This branch of Darshan now supports instrumenting serial applications that don't use MPI! This is still a work in progress, but it currently works and produces new insights so I'd like to review this code now with the Darshan wizards and make any changes to style, approach, nomenclature, etc now before hardening the code.

    Most of the work was factoring MPI out of darshan-core and all of the modules' finalization functions so that instead of calling PMPI_Reduce() and the like, darshan_mpi_reduce() is called, and Darshan either passes directly to PMPI_Reduce() if MPI is being used or a stub function otherwise.

    At present one must export DARSHAN_NOMPI to make Darshan spin up/down before entering and after leaving main(); this allows a single libdarshan.so to function with both MPI applications (in which case Darshan behaves identically to how it always has) and non-MPI applications (in which case Darshan is initialized/shutdown twice, but idempotently).

    So to actually test this out,

    1. Build Darshan as normal
    2. LD_PRELOAD libdarshan.so and define DARSHAN_NOMPI=1 in the runtime environment to enable glibc hooks for non-MPI apps

    For example,

    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ make
    cc -DDARSHAN_CONFIG_H=\"darshan-runtime-config.h\" -I . -I ../ -I . -I./../ -g -O2  -D_LARGEFILE64_SOURCE -DDARSHAN_LUSTRE -c lib/darshan-core-init-finalize.c -o lib/darshan-core-init-finalize.o
    cc -DDARSHAN_CONFIG_H=\"darshan-runtime-config.h\" -I . -I ../ -I . -I./../ -g -O2  -D_LARGEFILE64_SOURCE -DDARSHAN_LUSTRE -c lib/darshan-core.c -o lib/darshan-core.o
    ...
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ LD_PRELOAD=$PWD/lib/libdarshan.so DARSHAN_NOMPI=1 cp -v lib/libdarshan.so DELETEME
    'lib/libdarshan.so' -> 'DELETEME'
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ ls -lrt *.darshan
    -r-------- 1 glock glock 1399 Nov 26 14:20 glock_cp_id51423_11-26-51639-2202167712509143483_1.darshan
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ darshan-parser glock_cp_id51423_11-26-51639-2202167712509143483_1.darshan
    ...
    POSIX	-1	10398549382890127017	POSIX_BYTES_READ	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_BYTES_WRITTEN	443088	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_MAX_BYTE_READ	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_MAX_BYTE_WRITTEN	443087	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    ...
    POSIX	-1	10398549382890127017	POSIX_ACCESS1_ACCESS	131072	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS2_ACCESS	49872	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS3_ACCESS	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS4_ACCESS	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS1_COUNT	3	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS2_COUNT	1	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    

    I envisioned implementing this in two phases:

    Phase 1: Still must link against MPI (as in the Cray case) but can force non-MPI mode using an env variable

    • (done) provide serial+mpi abstraction to replace bare PMPI_* calls in Darshan
    • (done) make double-initialization of Darshan impossible so enabling serial mode on an MPI app simply ignores MPI profiling
    • (done) allows us to use a single compiled library on both MPI and non-MPI applications
    • (done) can implement a Darshan-specific subset of MPI functionality when MPI is not initialized
    • (done) use GNU C's __attribute__((constructor)), atexit(), and signal handling (I actually opted to forego atexit() and instead use both glibc constructor and destructor for consistent behavior)

    Phase 2: Serial-only version of Darshan to avoid pulling in all of MPI when a non-MPI application is run

    • must create a separate serial-mode build of Darshan that does not link against MPI at all
    • must create a complete MPI stub library; can we get Sandia/Steve Plimpton to relicense the LAMMPS one to Argonne under non-GPL terms? Otherwise we can just clean-room our own; much of the hard work is already done, as the LAMMPS stub library did not have MPI-IO.
    • figure out the correct environmental integration to ensure this version is linked in the absence of MPI but the full MPI version is included when appropriate.

    Phase 2 may only be necessary for static linking since Phase 1 happily preloads in front of non-MPI applications and still works fine.

    I've confirmed the following works:

    • non-MPI applications (F77, C)
    • enabling DARSHAN_NOMPI with an actual MPI application (each MPI process generates its own independent Darshan log as if it were a serial application in this case)
    • regular MPI app without DARSHAN_NOMPI

    but there are a number of open issues/questions:

    • move stubs into their own .c/.h? what to call it? darshan-mpi.h is already pulled out, but I couldn't see how make dist is done to include it in the source package
    • verify DARSHAN_INTERNAL_TIMING still works
    • verify that OpenMPI works
    • why is POSIX_MMAP -1 for a test job?
    • verify that Python works (can do this with some of pytokio's I/O-heavy tools)
    • verify that DARSHAN_NOMPI with HDF5 actually works
    • verify OpenMP/pthreads and thread safety (should work)
    • static linking
    • does DXT still work?

    There's also two known issues:

    • MPI_Type_* cannot be cleanly wrapped for both MPI and non-MPI because there is always a risk that derived types' MPI_Datatype values will collide with an MPI implementation's representation of built-in types. Darshan works around this by instead wrapping the entire process of deriving a datatype and then performing the collective so that the serial version does not need to call MPI_Type_* at all. The only proper solution I can think of would be to make a fully MPI-independent Darshan so that Darshan's MPI stubs own the representation of both builtin MPI_Datatypes and derived ones.
    • Profiling a do-nothing command like python --version creates a Darshan log that causes darshan-parser to throw an assertion (darshan-parser: darshan-logutils.c:1367: darshan_log_libz_read: Assertion 'state->dz.size > 0' failed.). This might be because the log is completely empty, but I wasn't sure.

    PS: this will resolve #173

    gitlab merge request 
    opened by shanedsnyder 21
  • deadlock with Cray compiler on CLE6 inside tcmalloc/mmap

    deadlock with Cray compiler on CLE6 inside tcmalloc/mmap

    In GitLab by @glennklockwood on Oct 17, 2016, 17:39

    The AMG mini-app reproducibly deadlocks as soon as it is started under certain conditions on NERSC Cori. The problem only occurs on Cray Linux Environment v6 (CLE6) and with the Cray compiler environment, and it occurs on both Haswell and Knight's Landing processors.

    The problem manifests as all processes going to sleep; attaching a debugger reveals that the processes are stuck at a futex within an mmap call. This mmap is wrapped by Darshan, and when AMG is compiled in the absence of Darshan, it executes normally.

    That being said, there are issues we have been seeing with mmap in CLE6 with hugepages enabled that are independent of this issue, and it isn't clear if they are related. The fact that this problem manifests even without Cray hugepages enabled suggests it is not, but they may share a common root cause.

    A tarball containing the AMG source code and this bug report can be found at NERSC in /global/project/projectdirs/m888/glock/amg-deadlock.tar.gz. Unfortunately the problem does not manifest on Edison (CLE5), so reproducing this problem without access to Cori may be difficult.

    Reproducing on Cori/Knight's Landing

    Build AMG with

    module swap PrgEnv-intel PrgEnv-cray
    module swap craype-haswell craype-mic-knl
    module load darshan/3.0.1
    
    make clean
    make -j 32
    

    Then request an allocation with

    salloc -N 1 -p regular_knl -t 30:00
    

    and run with

    srun -n 64 ./amg2013-cori-knl -laplace -P 4 4 4 -n 150 150 150 -solver 2
    

    Reproducing on Cori/Haswell

    Build AMG with

    module swap PrgEnv-intel PrgEnv-cray
    module load darshan/3.0.1
    
    make clean
    make -j 32
    

    Then request an allocation with

    salloc -N 1 -p regular -t 30:00
    

    and run with

    srun -n 32 ./amg2013-cori-hsw -laplace -P 4 4 2 -n 150 150 150 -solver 2 &
    

    Diagnosis

    The problem manifests as all MPI processes falling asleep almost immediately after job launch. Nothing appears on stdout, and ps indicates that no forward progress is happening:

    [email protected]:/global/cscratch1/sd/glock/corip2/AMG.knl/run$ ps -U glock ux
    USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    glock    257982  0.1  0.0  23896  3956 pts/0    Ss   11:34   0:00 /bin/bash
    glock    258166  0.7  0.0 368216  8888 pts/0    Sl   11:36   0:00 srun --ntasks 64 --cpu_bind=cores numactl --preferred 1 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 
    glock    258167  0.0  0.0  97880  1084 pts/0    S    11:36   0:00 srun --ntasks 64 --cpu_bind=cores numactl --preferred 1 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 
    glock    258180  7.5  0.0 548072  1372 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258181  7.6  0.0 554212  1048 ?        R    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258182  7.9  0.0 554212  1036 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258183  8.0  0.0 554212  1044 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258184  8.0  0.0 554212  1028 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258185  8.2  0.0 554212  1052 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    ...
    glock    258282  0.0  0.0  35440  1576 pts/0    R+   11:37   0:00 ps -U glock ux
    

    The deadlock occurs here (example taken from running on KNL):

    (gdb) bt
    #0  0x00000000204419ec in sys_futex ()
    #1  0x0000000020441b13 in base::internal::SpinLockDelay(int volatile*, int, int) ()
    #2  0x0000000020441ec5 in SpinLock::SlowLock() ()
    #3  0x00000000204e8702 in tc_calloc ()
    #4  0x0000000020115440 in __wrap_mmap ()
    #5  0x000000002043c1f8 in HugetlbSysAllocator::Alloc(unsigned long, unsigned long*, unsigned long) ()
    #6  0x000000002043bd60 in TCMalloc_SystemAlloc(unsigned long, unsigned long*, unsigned long) ()
    #7  0x000000002043d9e9 in tcmalloc::PageHeap::GrowHeap(unsigned long) [clone .part.5] ()
    #8  0x000000002043dd3b in tcmalloc::PageHeap::New(unsigned long) ()
    #9  0x00000000204393f8 in (anonymous namespace)::do_memalign(unsigned long, unsigned long) ()
    #10 0x00000000204ec04c in tc_posix_memalign ()
    #11 0x0000000020107734 in hypre_CAlloc () at hypre_memory.c:135
    #12 0x000000002004808b in GenerateLaplacian () at par_laplace.c:166
    #13 0x000000002001bc0b in BuildParLaplacian () at amg2013.c:2851
    #14 0x0000000020019ee3 in main () at amg2013.c:1754
    

    It is worth noting that this deadlock happens even if AMG is compiled in the absence of Cray's hugepages module. The stack trace is the same in both cases and on both Haswell and KNL processors.

    Scope

    The problem is limited to the Cray compiler on CLE6 (Cori). It does not appear with Intel compilers or on CLE5 (Edison).

    System | Compiler | HugePages | Darshan Version | Works? | ------------ |:--------:|:---------:|:---------------:|:------:| Cori/KNL | Cray | no | 3.0.1 | NO | Cori/Haswell | Cray | no | 3.0.1 | NO | Cori/KNL | Cray | 8 MB | 3.0.1 | NO | Cori/Haswell | Cray | 8 MB | 3.0.1 | NO | Cori/KNL | Intel | 8 MB | 3.0.1 | yes | Cori/Haswell | Intel | 8 MB | 3.0.1 | yes | Edison | Cray | 8 MB | 2.3.1 | yes | Edison | Intel | 8 MB | 3.1.1 | yes |

    opened by shanedsnyder 20
  • add wrapper for fileno( ) function

    add wrapper for fileno( ) function

    In GitLab by @carns on Oct 8, 2018, 13:21

    This function can be used to hand off from stdio (FILE *) to posix (file descriptor) instrumentation. This pattern is used by HipMer.

    opened by shanedsnyder 19
  • ENH: Add log path retrieval machinery

    ENH: Add log path retrieval machinery

    Description

    • Add module log_utils.py containing new function get_log_path for easy retrieval of logs from either local sources or darshan-logs repo, using only the filename

    • Fixes #566

    TODO

    • [ ] Decide on module location
    • [ ] Finish documentation for private functions
    • [ ] Add tests
    enhancement pydarshan 
    opened by nawtrey 18
  • ENH: Add I/O cost figure module and tests

    ENH: Add I/O cost figure module and tests

    • Add module for I/O cost figure

    • Add testing module to cover I/O cost functions

    • Add import to __init__.py to ease in importing all plotting functions in summary report code

    • Contributes to #465

    enhancement pydarshan 
    opened by nawtrey 18
  • WIP, ENH: Add initial darshan job summary PDF report

    WIP, ENH: Add initial darshan job summary PDF report

    • Add module, darshan/cli/summary.py, to generate darshan job summary PDF report

    To generate a report: python -m darshan summary ~/path/to/log.darshan

    (will save report in current working directory as report_<filename>.pdf)

    enhancement pydarshan 
    opened by nawtrey 17
  • TST: shims for new logs

    TST: shims for new logs

    • test_main_all_logs_repo_files() has been adjusted to allow the testsuite to pass when the additional logs from https://github.com/darshan-hpc/darshan-logs/pull/39 are included

    • in short, when the log minor version is greater than or equal to 4, we need to account for the default presence of the runtime HEATMAP module; if the major version ever bumps from 3 to 4 we'll need some more shims, but perhaps that can be deferred for now

    • for completeness, I checked that the adjusted test + new logs correctly detect a regression when gh-764 is reverted:

    FAILED tests/test_report.py::test_jobid_type_all_logs_repo_files[/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/darshan_logs/release_logs/mpi-io-test-x86_64-3.0.0.darshan]
    FAILED tests/test_summary.py::test_main_all_logs_repo_files[/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/darshan_logs/release_logs/mpi-io-test-x86_64-3.0.1.darshan]
    FAILED tests/test_summary.py::test_main_all_logs_repo_files[/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/darshan_logs/release_logs/mpi-io-test-x86_64-3.0.0.darshan]
    FAILED tests/test_report.py::test_jobid_type_all_logs_repo_files[/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/darshan_logs/release_logs/mpi-io-test-x86_64-3.0.1.darshan]
    

    The failures are a bit harsh because they are in the C layer, but anyway the point gets across that something really bad happened in log_lookup_name_records.

    replacing crashed worker gw7
    gw0 [448] / gw1 [448] / gw2 [448] / gw3 [448] / gw8 [448] / gw9 [448] / gw6 [448] / gw10 [448]................................................Fatal Python error: Aborted
    
    Thread 0x00007000049d2000 (most recent call first):
      File "/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/execnet/gateway_base.py", line 400 in read
      File "/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/execnet/gateway_base.py", line 432 in from_io
      File "/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
      File "/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/execnet/gateway_base.py", line 220 in run
      File "/Users/treddy/python_venvs/python_395_darshan/lib/python3.9/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
    
    Current thread 0x0000000108448e00 (most recent call first):
      File "/Users/treddy/github_projects/darshan/darshan-util/pydarshan/darshan/backend/cffi_backend.py", line 274 in log_lookup_name_records
    
    pydarshan testing 
    opened by tylerjereddy 0
  • PyDarshan testing should make use of suite of release logs in `darshan-test/example-output`

    PyDarshan testing should make use of suite of release logs in `darshan-test/example-output`

    Every release/pre-release, we add a new example log file to this directory:

    [email protected] ~/software/darshan/darshan (main) $ ls darshan-test/example-output/
    mpi-io-test-ppc64-3.0.0.darshan  mpi-io-test-ppc64-3.1.7.darshan   mpi-io-test-x86_64-3.1.5.darshan
    mpi-io-test-ppc64-3.0.1.darshan  mpi-io-test-ppc64-3.1.8.darshan   mpi-io-test-x86_64-3.1.6.darshan
    mpi-io-test-ppc64-3.1.0.darshan  mpi-io-test-x86_64-3.0.0.darshan  mpi-io-test-x86_64-3.1.7.darshan
    mpi-io-test-ppc64-3.1.1.darshan  mpi-io-test-x86_64-3.0.1.darshan  mpi-io-test-x86_64-3.1.8.darshan
    mpi-io-test-ppc64-3.1.2.darshan  mpi-io-test-x86_64-3.1.0.darshan  mpi-io-test-x86_64-3.2.0.darshan
    mpi-io-test-ppc64-3.1.3.darshan  mpi-io-test-x86_64-3.1.1.darshan  mpi-io-test-x86_64-3.2.1.darshan
    mpi-io-test-ppc64-3.1.4.darshan  mpi-io-test-x86_64-3.1.2.darshan  mpi-io-test-x86_64-3.3.0.darshan
    mpi-io-test-ppc64-3.1.5.darshan  mpi-io-test-x86_64-3.1.3.darshan  mpi-io-test-x86_64-3.3.1.darshan
    mpi-io-test-ppc64-3.1.6.darshan  mpi-io-test-x86_64-3.1.4.darshan  mpi-io-test-x86_64-3.4.0-pre1.darshan
    

    It would be nice to have at least some simple tests iterate over each of these logs, as it's useful to understand whether we ever introduce any backwards compatibility issues. For example, #764 fixes a regression that only affects PyDarshan for logfiles generated with Darshan version 3.0.0, which we don't have a PyDarshan test log for at the moment

    I'm not sure what's the best way to incoporate these logs into our existing log searching infrastructure, that's probably a question for @tylerjereddy. If it's problematic that the logs exist above pydarshan directory level, then maybe it's possible to symlink them or something?

    pydarshan testing 
    opened by shanedsnyder 1
  • PyDarshan: Easily extract OST related information from DXT logs

    PyDarshan: Easily extract OST related information from DXT logs

    I was looking at the documentation/code, and I do not seem to be able to get the OST information for each access collected by the DXT module using PyDarshan, similarly to what is presented when using the command line parser.

    After talking with Share, I understood that this information could be computed based on the stripe width and size but it might be good to have an easier way to extract that without having the end-user manually compute the OST ID by themselves.

    enhancement pydarshan 
    opened by jeanbez 1
  • WIP, TST: Regression test for runtime integer overflow in HEATMAP

    WIP, TST: Regression test for runtime integer overflow in HEATMAP

    • this builds on the new testing infrastructure drafted in gh-753, which should be reviewed/dealt with before this PR

    • add a C-language regression test based on Phil's comment here: https://github.com/darshan-hpc/darshan/pull/737#issuecomment-1126148539

    • at the time of writing, however, the test passes--there is no evidence of overflow--I'm open to suggestions from the "C folks" on the team..

    testing CI 
    opened by tylerjereddy 10
  • TST: add regression test for issue 743

    TST: add regression test for issue 743

    • add a regression test for gh-743--the test reproduces the runtime segfault described there in the CI
    • this builds on the new "functional testing" infrastructure proposed in gh-753, so that PR should be reviewed/dealt with first
    testing CI 
    opened by tylerjereddy 0
  • use HDF5 macro `H5_VERSION_GE` rather than our own autoconf logic

    use HDF5 macro `H5_VERSION_GE` rather than our own autoconf logic

    We currently have autoconf logic that determines whether HDF5 version is 1.10+ or not. That happens to be sufficient for our current instrumentation code, but HDF5 library does have a lot of API changes across versions we may need to protect against (e.g., see 981c315aee2f9dba685fd5264c1d651542623901).

    HDF5 has macros (H5_VERSION_GE, H5_VERSION_LE, etc.) we can use that give us complete access to major/minor/bugfix release numbers rather than writing them ourselves, we should just switch to those to simplify.

    opened by shanedsnyder 0
a tool to count accesses to member variables in c++ programs

access_profiler access_profiler is a heavy-weight class field access profiler, implemented as C++ library. to use this profiler, include "access_profi

Arvid Norberg 68 May 31, 2022
Updated version of Silicos-it's shape-based alignment tool

shape-it Description Code for shape-it with openbabel3 and rdkit INSTALL Following example is the basic way to install the tool: git clone https://git

RDKit 22 Apr 27, 2022
This is a tool for software engineers to view,record and analyse data(sensor data and module data) In the process of software development.

![Contributors][Huang Jianyu] Statement 由于工具源码在网上公开,除使用部分开源项目代码外,其余代码均来自我个人,工具本身不包含公司的知识产权,所有与公司有关的内容均从软件包中移除,软件发布遵循Apache协议,任何人均可下载进行修改使用,如使用过程中出现任何问

HuangJianyu 34 May 5, 2022
Kernel file/process/object tool

kt Kernel file/process/object tool killav bypass av dump lsass basic vs2019 + cpp + wdk usage(64-bit only) kdu -map sys.sys kt -F -d c:\windows\notepa

null 58 Jun 15, 2022
Minimal tool for measuring cost of mode switch

CPU mode switch statistics The mode-switch-stat tool measures the cost of CPU mode switch, the round trip between user and kernel mode. At present, th

Steven Cheng 12 Feb 22, 2022
a undetectable tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our qqgroup is 703156427

a undetectable ios root access tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our

null 58 Nov 22, 2021
An experimental tool to estimate the similarity between all pairs of contigs

This is an experimental tool to estimate the approximate distances between all pairs of unitigs. It takes a GFA or FASTA file as input and outputs a T

Heng Li 33 Mar 16, 2022
Powerful automated tool for reverse engineering Unity IL2CPP binaries

Powerful automated tool for reverse engineering Unity IL2CPP binaries

Katy 1.9k Jun 24, 2022
Simple tool to visualize and amplify mouse movements

mousemic Simple tool to visualize and amplify mouse movements. This utility uses a high-level X11 Api so is not really more sensitive than your screen

Alfredo Ortega 41 Dec 21, 2021
A lightweight ARM reverse engineering tool.

eydis A lightweight (basic and slow) ARM reverse engineering tool. I. Requierements macOS/Linux, Basics compiling tools, The SQLite3 + readline framew

Yui Aioi 19 Feb 16, 2022
Stack-based texture generation tool written in C99!

Stack-based texture generation tool written in C99! Brought to you by @zaklaus and contributors Introduction zpl.texed is a cross-platform stack-based

zpl | pushing the boundaries of simplicity. 17 May 1, 2022
Matryoshka loader is a tool that red team operators can leverage to generate shellcode for Microsoft Office document phishing payloads.

Overview Matryoshka loader is a tool that red team operators can leverage to generate shellcode for an egghunter to bypass size-limitations and perfor

Praetorian 24 Jun 26, 2022
A shellcode crypto-packing tool for PoC (used with msfvenom payloads)

crypter A shellcode crypto-packing tool for PoC (used with msfvenom/binary payloads) This tool is for proof of concept only - please use responsibly.

ripmeep 11 May 30, 2022
MDE is a model extraction tool that converts Destiny 2 dynamic models into fbx files supporting textures, skeletons, and all provided vertex data.

MDE is a model extraction tool that converts Destiny 2 dynamic models into fbx files. A dynamic model is one that is animated or is spawned in during the game.

Montague 31 Jun 20, 2022
Fast and lightweight username lookup tool inspired by sherlock.

Lightweight username lookup inspired by Sherlock. Created in C++. Features Works on 250+ websites Fast, easy to use and compact How to use $ scout.exe

eternity 9 Jan 21, 2022
A Simple tool to execute shellcode with the ability to detect mouse movement

Noobi A Simple tool to execute shellcode with the ability to detect mouse movement Features: Sandbox evasion through detecting mouse movement and chec

null 10 Feb 20, 2022
sent is a simple plaintext presentation tool.

sent is a simple plaintext presentation tool. sent does not need latex, libreoffice or any other fancy file format, it uses plaintext files and png im

Injamul Mohammad Mollah 3 Jun 13, 2021
A tool to kill antimalware protected processes

Backstab Kill EDR Protected Processes Have these local admin credentials but the EDR is standing in the way? Unhooking or direct syscalls are not work

Yasser 768 Jun 21, 2022
A simple tool that aims to efficiently and quickly parse the outputs of web scraping tools like gau

massurl is a simple tool that aims to parse the outputs of tools like gau, and extract the parameters for each URL, remove duplicates and do it all very quickly. Because web scraping tools' outputs can get very large very quickly, it is nice to have a tool that parses them and and outputs something clean and easy to read.

Fr1nge 14 Mar 22, 2022