Darshan I/O characterization tool

Overview

Darshan is a lightweight I/O characterization tool that transparently captures I/O access pattern information from HPC applications. Darshan can be used to tune applications for increased scientific productivity or to gain insight into trends in large-scale computing systems.

Please see the Darshan web page for more in-depth news and documentation.

The Darshan source tree is divided into two main parts:

  • darshan-runtime: to be installed on systems where you intend to instrument MPI applications. See darshan-runtime/doc/darshan-runtime.txt for installation instructions.

  • darshan-util: to be installed on systems where you intend to analyze log files produced by darshan-runtime. See darshan-util/doc/darshan-util.txt for installation instructions.

The darshan-test directory contains various test harnesses, benchmarks, patches, and unsupported utilites that are mainly of interest to Darshan developers.

Comments
  • Darshan Python Package - [merged]

    Darshan Python Package - [merged]

    In GitLab by @jakobluettgau on Aug 1, 2019, 10:46

    Merges python-package -> master

    Adds a python package including helpers to release to PyPI to the darshan utils source tree under: darshan-utils/pydarshan

    There is a Makefile with all sorts of useful targets, from testing, to documentation to coverage as well as releasing to PyPI.

    make test
    make coverage
    make lint
    ...
    make install
    ...
    make release
    
    • The tests are currently stubs and use py.test because that seems to be the general recommendation for new projects, but that is easy to switch over to nose if that helps with borrowing from pytokio.

    • I have added an experimental darshan.plots submodule, which uses matplotlib to do the access size histogram as well as the I/O operation summary.

    • I have added docstrings to all the CFFI bindings as well as some general instructions which can build easily using make docs / make show-docs. (Just ensure pip install -r requirements_dev.txt)

    • License?: I have blanked out more or less all the license information, but these need to populated when releasing to PyPI.

    • I had some issues, when accessing the STDIO counters multiple times, which would kill or segfault the execution, which I had no time yet to look into in more detail.

    • The code and scripts to generate a CFFI friendly header is currently included in darshan-util/pydarshan/darshan/data/generate_headers

    Example Notebook/Scripts:

    There is a example directory which includes a Jupyter-Notebook which demonstrates how to use the current interface to interact with the logs, as well as the plotting at the end of the notebook.

    gitlab merge request 
    opened by shanedsnyder 62
  • WIP, ENH: draft library code for

    WIP, ENH: draft library code for "Data Access by Category"

    The draft "library code" is currently under version control in this PR, while the (rougher) plotting code that uses the data munging utilities in the lib code is placed below the fold and was used to produce the sample plots below vs. @carns draft drawing. I don't actually know if we need any of this if @jakobluettgau aggregation/counting code does something similar though.

    Edit: the plotting code is mostly under version control now, so can run it with something like this:

    import os
    
    import darshan
    from darshan.experimental.plots import data_access_by_filesystem
    
    if __name__ == '__main__':
        # produce sample plots for some of the logs
        # available in our test suite
        root_path = '/Users/treddy/github_projects/darshan/darshan-util/pydarshan/tests/input'
        #log_files = ['sample-dxt-simple.darshan', 'sample.darshan', 'sample-goodost.darshan']
        log_files = ['sample-goodost.darshan']
        for log_file in log_files:
            log_path = os.path.join(root_path, log_file)
            data_access_by_filesystem.plot_with_log_file(log_file_path=log_path,
                                                         plot_filename=log_file + '.png')
    
    

    image

    galen_anonymous_1 darshan_data_access_by_category galen_anonymous_2 darshan_data_access_by_category sample-dxt-simple darshan_data_access_by_category sample-goodost darshan_data_access_by_category sample darshan_data_access_by_category treddy_mpi-io-test_id545458_6-3-43062-1247823192538596076_13 darshan_data_access_by_category

    enhancement pydarshan 
    opened by tylerjereddy 60
  • PnetCDF module

    PnetCDF module

    Add/update the PnetCDF module.

    • the default is changed to --disable-pnetcdf-mod
    • command-line option --with-pnetcdf is added
    • All PnetCDF APIs are included.
    • Internal counters and timestamps are now separated for files and variables respectively
    pydarshan 
    opened by wkliao 59
  • WIP: PyDarshan: Adds heatmap plotting code for DXT tracing logs

    WIP: PyDarshan: Adds heatmap plotting code for DXT tracing logs

    DXT heatmap plotting code for potential use in updated job summary. Primarily uses numpy, seaborn, matplotlib for plotting, which are all pip installable.

    Summary

    At the moment it takes in logs with DXT_POSIX data, concatenates all of the read/write segment dataframes together, then uses that data (along with the respective rank data) to create a 2D histogram, and plot a heat map with marginal bar graphs. Depending on which operations you include, it will plot only the read segment data, the write segment data, or both. It does not distinguish between operations. Each bin (in the heatmap) is populated with the data sum and/or proportionate data sum for all IO events read/written during the time spanned by the bin.The color bar was selected based on the fact that it is non-white at the minimum value so areas without data are visible.

    How to run

    See: https://github.com/darshan-hpc/darshan/pull/396#issuecomment-879202907

    TODO

    • [x] Fix average time plotting -- need to span from start time to end time for accurate IO representation
    • [ ] ~~Find a way to aggregate data from other modules, if applicable~~
    • [ ] ~~Differentiate between read/write data in heat map~~
    • [x] Include number of bins in plot somewhere so you can get an idea of bandwidth (i.e. 100 bins w/ 100 s run time = 1 s/bin)
    • [ ] Figure out if/how to utilize the figure title. At the moment it just states which modules are being used (may want to remove it entirely for the job summary)
    • [x] Finish documentation/comments
    enhancement pydarshan 
    opened by nawtrey 46
  • Python Package: Rename mode switch to dtype to select datatype. Simplify interface log_get_record. Have lustre records include OSTs. Add library version check. - [merged]

    Python Package: Rename mode switch to dtype to select datatype. Simplify interface log_get_record. Have lustre records include OSTs. Add library version check. - [merged]

    In GitLab by @jakobluettgau on Sep 3, 2020, 12:53

    Merges python-package -> master

    Some updates streamlining the interface and a bugfix:

    • Rename mode switch to dtype to select datatype (numpy, dict, pandas). E.g., log_get_record(log, 'POSIX', dtype='pandas')
    • Simplify interface log_get_record. Deprecates log_get_posix_record, etc.
    • Include version check to warn about library incompatability.

    Fix:

    • Lustre records did not unpack list of OSTs.
    gitlab merge request 
    opened by shanedsnyder 29
  • use automake and libtool

    use automake and libtool

    Re-creating @wkliao's PR from our GitLab repo (https://xgitlab.cels.anl.gov/darshan/darshan/-/merge_requests/77) here to continue to track Darshan's transition to automake/libtool.

    opened by shanedsnyder 23
  • ENH: improved precision of darshan job start/end times

    ENH: improved precision of darshan job start/end times

    The existing changes already included in the PR:

    • add struct timespec start/end times to job struct
    • add custom MPI reduction ops for max/min values of timespecs
    • add logutils code for extracting start/end timespecs
    • update some analysis utilities to use new timespec values to calculate floating point app runtimes

    TODO items:

    • [x] remove duplicate calculation of start_time/end_time in job header, and rely solely on start_ts/end_ts
    • [x] update PyDarshan and CFFI bindings for changes
    • [x] update darshan-merge utility
    • [x] update darshan-diff utility
    • [x] update darshan-util docs to show new output format
    pydarshan CI 
    opened by shanedsnyder 21
  • WIP: darshan instrumentation for apps that do not use MPI - [closed]

    WIP: darshan instrumentation for apps that do not use MPI - [closed]

    In GitLab by @glennklockwood on Nov 25, 2018, 00:38

    Merges no-mpi -> master

    This branch of Darshan now supports instrumenting serial applications that don't use MPI! This is still a work in progress, but it currently works and produces new insights so I'd like to review this code now with the Darshan wizards and make any changes to style, approach, nomenclature, etc now before hardening the code.

    Most of the work was factoring MPI out of darshan-core and all of the modules' finalization functions so that instead of calling PMPI_Reduce() and the like, darshan_mpi_reduce() is called, and Darshan either passes directly to PMPI_Reduce() if MPI is being used or a stub function otherwise.

    At present one must export DARSHAN_NOMPI to make Darshan spin up/down before entering and after leaving main(); this allows a single libdarshan.so to function with both MPI applications (in which case Darshan behaves identically to how it always has) and non-MPI applications (in which case Darshan is initialized/shutdown twice, but idempotently).

    So to actually test this out,

    1. Build Darshan as normal
    2. LD_PRELOAD libdarshan.so and define DARSHAN_NOMPI=1 in the runtime environment to enable glibc hooks for non-MPI apps

    For example,

    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ make
    cc -DDARSHAN_CONFIG_H=\"darshan-runtime-config.h\" -I . -I ../ -I . -I./../ -g -O2  -D_LARGEFILE64_SOURCE -DDARSHAN_LUSTRE -c lib/darshan-core-init-finalize.c -o lib/darshan-core-init-finalize.o
    cc -DDARSHAN_CONFIG_H=\"darshan-runtime-config.h\" -I . -I ../ -I . -I./../ -g -O2  -D_LARGEFILE64_SOURCE -DDARSHAN_LUSTRE -c lib/darshan-core.c -o lib/darshan-core.o
    ...
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ LD_PRELOAD=$PWD/lib/libdarshan.so DARSHAN_NOMPI=1 cp -v lib/libdarshan.so DELETEME
    'lib/libdarshan.so' -> 'DELETEME'
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ ls -lrt *.darshan
    -r-------- 1 glock glock 1399 Nov 26 14:20 glock_cp_id51423_11-26-51639-2202167712509143483_1.darshan
    
    (haswell)[email protected]:~/src/git/darshan-dev/darshan-runtime$ darshan-parser glock_cp_id51423_11-26-51639-2202167712509143483_1.darshan
    ...
    POSIX	-1	10398549382890127017	POSIX_BYTES_READ	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_BYTES_WRITTEN	443088	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_MAX_BYTE_READ	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_MAX_BYTE_WRITTEN	443087	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    ...
    POSIX	-1	10398549382890127017	POSIX_ACCESS1_ACCESS	131072	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS2_ACCESS	49872	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS3_ACCESS	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS4_ACCESS	0	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS1_COUNT	3	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    POSIX	-1	10398549382890127017	POSIX_ACCESS2_COUNT	1	/global/u2/g/glock/src/git/darshan-dev/darshan-runtime/DELETEME	/global/u2	gpfs
    

    I envisioned implementing this in two phases:

    Phase 1: Still must link against MPI (as in the Cray case) but can force non-MPI mode using an env variable

    • (done) provide serial+mpi abstraction to replace bare PMPI_* calls in Darshan
    • (done) make double-initialization of Darshan impossible so enabling serial mode on an MPI app simply ignores MPI profiling
    • (done) allows us to use a single compiled library on both MPI and non-MPI applications
    • (done) can implement a Darshan-specific subset of MPI functionality when MPI is not initialized
    • (done) use GNU C's __attribute__((constructor)), atexit(), and signal handling (I actually opted to forego atexit() and instead use both glibc constructor and destructor for consistent behavior)

    Phase 2: Serial-only version of Darshan to avoid pulling in all of MPI when a non-MPI application is run

    • must create a separate serial-mode build of Darshan that does not link against MPI at all
    • must create a complete MPI stub library; can we get Sandia/Steve Plimpton to relicense the LAMMPS one to Argonne under non-GPL terms? Otherwise we can just clean-room our own; much of the hard work is already done, as the LAMMPS stub library did not have MPI-IO.
    • figure out the correct environmental integration to ensure this version is linked in the absence of MPI but the full MPI version is included when appropriate.

    Phase 2 may only be necessary for static linking since Phase 1 happily preloads in front of non-MPI applications and still works fine.

    I've confirmed the following works:

    • non-MPI applications (F77, C)
    • enabling DARSHAN_NOMPI with an actual MPI application (each MPI process generates its own independent Darshan log as if it were a serial application in this case)
    • regular MPI app without DARSHAN_NOMPI

    but there are a number of open issues/questions:

    • move stubs into their own .c/.h? what to call it? darshan-mpi.h is already pulled out, but I couldn't see how make dist is done to include it in the source package
    • verify DARSHAN_INTERNAL_TIMING still works
    • verify that OpenMPI works
    • why is POSIX_MMAP -1 for a test job?
    • verify that Python works (can do this with some of pytokio's I/O-heavy tools)
    • verify that DARSHAN_NOMPI with HDF5 actually works
    • verify OpenMP/pthreads and thread safety (should work)
    • static linking
    • does DXT still work?

    There's also two known issues:

    • MPI_Type_* cannot be cleanly wrapped for both MPI and non-MPI because there is always a risk that derived types' MPI_Datatype values will collide with an MPI implementation's representation of built-in types. Darshan works around this by instead wrapping the entire process of deriving a datatype and then performing the collective so that the serial version does not need to call MPI_Type_* at all. The only proper solution I can think of would be to make a fully MPI-independent Darshan so that Darshan's MPI stubs own the representation of both builtin MPI_Datatypes and derived ones.
    • Profiling a do-nothing command like python --version creates a Darshan log that causes darshan-parser to throw an assertion (darshan-parser: darshan-logutils.c:1367: darshan_log_libz_read: Assertion 'state->dz.size > 0' failed.). This might be because the log is completely empty, but I wasn't sure.

    PS: this will resolve #173

    gitlab merge request 
    opened by shanedsnyder 21
  • deadlock with Cray compiler on CLE6 inside tcmalloc/mmap

    deadlock with Cray compiler on CLE6 inside tcmalloc/mmap

    In GitLab by @glennklockwood on Oct 17, 2016, 17:39

    The AMG mini-app reproducibly deadlocks as soon as it is started under certain conditions on NERSC Cori. The problem only occurs on Cray Linux Environment v6 (CLE6) and with the Cray compiler environment, and it occurs on both Haswell and Knight's Landing processors.

    The problem manifests as all processes going to sleep; attaching a debugger reveals that the processes are stuck at a futex within an mmap call. This mmap is wrapped by Darshan, and when AMG is compiled in the absence of Darshan, it executes normally.

    That being said, there are issues we have been seeing with mmap in CLE6 with hugepages enabled that are independent of this issue, and it isn't clear if they are related. The fact that this problem manifests even without Cray hugepages enabled suggests it is not, but they may share a common root cause.

    A tarball containing the AMG source code and this bug report can be found at NERSC in /global/project/projectdirs/m888/glock/amg-deadlock.tar.gz. Unfortunately the problem does not manifest on Edison (CLE5), so reproducing this problem without access to Cori may be difficult.

    Reproducing on Cori/Knight's Landing

    Build AMG with

    module swap PrgEnv-intel PrgEnv-cray
    module swap craype-haswell craype-mic-knl
    module load darshan/3.0.1
    
    make clean
    make -j 32
    

    Then request an allocation with

    salloc -N 1 -p regular_knl -t 30:00
    

    and run with

    srun -n 64 ./amg2013-cori-knl -laplace -P 4 4 4 -n 150 150 150 -solver 2
    

    Reproducing on Cori/Haswell

    Build AMG with

    module swap PrgEnv-intel PrgEnv-cray
    module load darshan/3.0.1
    
    make clean
    make -j 32
    

    Then request an allocation with

    salloc -N 1 -p regular -t 30:00
    

    and run with

    srun -n 32 ./amg2013-cori-hsw -laplace -P 4 4 2 -n 150 150 150 -solver 2 &
    

    Diagnosis

    The problem manifests as all MPI processes falling asleep almost immediately after job launch. Nothing appears on stdout, and ps indicates that no forward progress is happening:

    [email protected]:/global/cscratch1/sd/glock/corip2/AMG.knl/run$ ps -U glock ux
    USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    glock    257982  0.1  0.0  23896  3956 pts/0    Ss   11:34   0:00 /bin/bash
    glock    258166  0.7  0.0 368216  8888 pts/0    Sl   11:36   0:00 srun --ntasks 64 --cpu_bind=cores numactl --preferred 1 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 
    glock    258167  0.0  0.0  97880  1084 pts/0    S    11:36   0:00 srun --ntasks 64 --cpu_bind=cores numactl --preferred 1 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 
    glock    258180  7.5  0.0 548072  1372 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258181  7.6  0.0 554212  1048 ?        R    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258182  7.9  0.0 554212  1036 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258183  8.0  0.0 554212  1044 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258184  8.0  0.0 554212  1028 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    glock    258185  8.2  0.0 554212  1052 ?        S    11:36   0:02 ./amg2013 -laplace -P 4 4 4 -n 150 150 150 -solver 2
    ...
    glock    258282  0.0  0.0  35440  1576 pts/0    R+   11:37   0:00 ps -U glock ux
    

    The deadlock occurs here (example taken from running on KNL):

    (gdb) bt
    #0  0x00000000204419ec in sys_futex ()
    #1  0x0000000020441b13 in base::internal::SpinLockDelay(int volatile*, int, int) ()
    #2  0x0000000020441ec5 in SpinLock::SlowLock() ()
    #3  0x00000000204e8702 in tc_calloc ()
    #4  0x0000000020115440 in __wrap_mmap ()
    #5  0x000000002043c1f8 in HugetlbSysAllocator::Alloc(unsigned long, unsigned long*, unsigned long) ()
    #6  0x000000002043bd60 in TCMalloc_SystemAlloc(unsigned long, unsigned long*, unsigned long) ()
    #7  0x000000002043d9e9 in tcmalloc::PageHeap::GrowHeap(unsigned long) [clone .part.5] ()
    #8  0x000000002043dd3b in tcmalloc::PageHeap::New(unsigned long) ()
    #9  0x00000000204393f8 in (anonymous namespace)::do_memalign(unsigned long, unsigned long) ()
    #10 0x00000000204ec04c in tc_posix_memalign ()
    #11 0x0000000020107734 in hypre_CAlloc () at hypre_memory.c:135
    #12 0x000000002004808b in GenerateLaplacian () at par_laplace.c:166
    #13 0x000000002001bc0b in BuildParLaplacian () at amg2013.c:2851
    #14 0x0000000020019ee3 in main () at amg2013.c:1754
    

    It is worth noting that this deadlock happens even if AMG is compiled in the absence of Cray's hugepages module. The stack trace is the same in both cases and on both Haswell and KNL processors.

    Scope

    The problem is limited to the Cray compiler on CLE6 (Cori). It does not appear with Intel compilers or on CLE5 (Edison).

    System | Compiler | HugePages | Darshan Version | Works? | ------------ |:--------:|:---------:|:---------------:|:------:| Cori/KNL | Cray | no | 3.0.1 | NO | Cori/Haswell | Cray | no | 3.0.1 | NO | Cori/KNL | Cray | 8 MB | 3.0.1 | NO | Cori/Haswell | Cray | 8 MB | 3.0.1 | NO | Cori/KNL | Intel | 8 MB | 3.0.1 | yes | Cori/Haswell | Intel | 8 MB | 3.0.1 | yes | Edison | Cray | 8 MB | 2.3.1 | yes | Edison | Intel | 8 MB | 3.1.1 | yes |

    opened by shanedsnyder 20
  • add wrapper for fileno( ) function

    add wrapper for fileno( ) function

    In GitLab by @carns on Oct 8, 2018, 13:21

    This function can be used to hand off from stdio (FILE *) to posix (file descriptor) instrumentation. This pattern is used by HipMer.

    opened by shanedsnyder 19
  • MPI_Type_size() is called with MPI_DATATYPE_NULL and breaks some applications

    MPI_Type_size() is called with MPI_DATATYPE_NULL and breaks some applications

    Hi, a user at NERSC reported that darshan is calling the MPI_Type_size() function with MPI_DATATYPE_NULL; they protect all calls to MPI_Type_size in their application with

    if (mpi_datatype == MPI_DATATYPE_NULL) {
      size = 0;
    } else {
      size = MPI_Type_size(mpi_datatype);
    }
    

    And he suggests to do the same in Darshan.

    The following is an example traceback, but he couldn't provide more details, so I don't have a piece of code to reproduce the issue:

    3: Test command: /usr/bin/srun "-n" "5" "/path/to/test/app"
    3: Test timeout computed to be: 600
    3: Rank 1 [Tue Mar 22 10:03:09 2022] [c1-0c0s6n1] Fatal error in PMPI_Type_size: Invalid datatype, error stack:
    3: PMPI_Type_size(119): MPI_Type_size(MPI_DATATYPE_NULL) failed
    3: PMPI_Type_size(69).: Datatype for argument datatype is a null datatype
    3: Rank 2 [Tue Mar 22 10:03:09 2022] [c1-0c0s6n1] Fatal error in PMPI_Type_size: Invalid datatype, error stack:
    3: PMPI_Type_size(119): MPI_Type_size(MPI_DATATYPE_NULL) failed
    3: PMPI_Type_size(69).: Datatype for argument datatype is a null datatype
    3: Rank 3 [Tue Mar 22 10:03:09 2022] [c1-0c0s6n1] Fatal error in PMPI_Type_size: Invalid datatype, error stack:
    3: PMPI_Type_size(119): MPI_Type_size(MPI_DATATYPE_NULL) failed
    3: PMPI_Type_size(69).: Datatype for argument datatype is a null datatype
    3: srun: error: nid00217: tasks 1-3: Aborted
    3: srun: launch/slurm: _step_signal: Terminating StepId=56572294.33
    3: slurmstepd: error: *** STEP 56572294.33 ON nid00217 CANCELLED AT 2022-03-22T10:03:11 ***
    3: srun: error: nid00217: tasks 0,4: Terminated
    3: srun: Force Terminated StepId=56572294.33
    1/1 Test #3: test_rearr .......................***Failed 5.43 sec
    

    I only see occurrences of PMPI_Type_size in the darshan source code; I built darshan with support to autoperf with --enable-apmpi-mod. Let me know if you want more details from the user, I can probably put you in contact if needed.

    Edit: last week we did a maintenance on Cori, after which darshan was upgraded from 3.2.1 to 3.3.1, and the user seems to have been experiencing this only recently, so maybe the issue is in a recent commit after 3.2.1?

    opened by bebosudo 18
  • bug fix: when MPI is disabled

    bug fix: when MPI is disabled

    These 2 APIs contain an argument of type MPI_Datatype.

       static void darshan_core_reduce_min_time(
           void* in_time_v, void* inout_time_v,
           int *len, MPI_Datatype *datatype);
       static void darshan_core_reduce_max_time(
           void* in_time_v, void* inout_time_v,
           int *len, MPI_Datatype *datatype);
    
    opened by wkliao 1
  • CI: run `make distcheck` for runtime CI workflow

    CI: run `make distcheck` for runtime CI workflow

    Building off suggestion from @wkliao in #844

    These changes allow us to build runtime and util portions of Darshan with different options (i.e., util with AutoPerf, runtime without AutoPerf) and test each independently with make distcheck. It seems to work fine for me locally, but will see what the CI says.

    CI 
    opened by shanedsnyder 1
  • CI: Python 3.10 on ubuntu-latest issue

    CI: Python 3.10 on ubuntu-latest issue

    As discussed in gh-830, there is a mysterious "hang" of the pytest suite with Python 3.10 and ubuntu-latest GitHub actions runner on that branch. Logging in to the runner via ssh allows execution via pytest as expected, so it is pretty unusual/weird. Furthermore, in https://github.com/tylerjereddy/darshan/pull/23 I checked that pytest 6.2.5 and Python 3.10.0 had no effect on the hang (vs. the newer versions of those libs in use for the original hang).

    I'll keep this issue open as a reference, but since I couldn't reproduce even in the runner itself when working interactively, nor in act with Python 3.10 and ubuntu-latest, there seems to be no sane way to debug this. It looks like switching to the latest Ubuntu LTS release, using the ubuntu-22.04 tag alleviates the issue so I'll probably try applying that to Shane's PR and referencing this issue in a comment.

    Unless we get related user reports, I wouldn't suggest putting any more time on this one though...

    pydarshan CI 
    opened by tylerjereddy 3
  • WIP, DEBUG: Python derived metrics

    WIP, DEBUG: Python derived metrics

    • this is a Python/pandas-only version of doing some simple derived metrics; I don't think we'll actually do this, but I was exploring a bit because of the difficulties in gh-839

    • this matches pretty well with the perl based reports for total bytes, but even simple cases can sometimes disagree on bandwidth per gh-847, so now I'm curious what is going on

    • one problem with doing this is that we'd have the same algorithms implemented in two different languages; the advantages include:

      • not reading all the records in a second time, one at a time, crossing the CFFI boundary each time (because all the records are already stored in the DataFrame from the first time we did this)
      • easier to debug/maintain because bounds checking/no segfaults, etc.
      • likely easier to regex-filter directly on the pandas data structures than with interop to C/CFFI for i.e., custom file derived metrics
    pydarshan 
    opened by tylerjereddy 1
  • Is autoperf-apmpi-ld-opts used?

    Is autoperf-apmpi-ld-opts used?

    I wonder if autoperf-apmpi-ld-opts is still used by darshan-runtime. When I removed all the lines referring to it in darshan-runtime/share/ld-opts/Makefile.am, make check ran just fine.

    https://github.com/darshan-hpc/darshan/blob/cb20d4929728bcb6a94533de79ab088b725bf9e1/darshan-runtime/share/ld-opts/Makefile.am#L31-L38

    opened by wkliao 0
Owner
null
a tool to count accesses to member variables in c++ programs

access_profiler access_profiler is a heavy-weight class field access profiler, implemented as C++ library. to use this profiler, include "access_profi

Arvid Norberg 68 May 31, 2022
Updated version of Silicos-it's shape-based alignment tool

shape-it Description Code for shape-it with openbabel3 and rdkit INSTALL Following example is the basic way to install the tool: git clone https://git

RDKit 24 Nov 3, 2022
This is a tool for software engineers to view,record and analyse data(sensor data and module data) In the process of software development.

![Contributors][Huang Jianyu] Statement 由于工具源码在网上公开,除使用部分开源项目代码外,其余代码均来自我个人,工具本身不包含公司的知识产权,所有与公司有关的内容均从软件包中移除,软件发布遵循Apache协议,任何人均可下载进行修改使用,如使用过程中出现任何问

HuangJianyu 34 May 5, 2022
Kernel file/process/object tool

kt Kernel file/process/object tool killav bypass av dump lsass basic vs2019 + cpp + wdk usage(64-bit only) kdu -map sys.sys kt -F -d c:\windows\notepa

null 62 Nov 30, 2022
Minimal tool for measuring cost of mode switch

CPU mode switch statistics The mode-switch-stat tool measures the cost of CPU mode switch, the round trip between user and kernel mode. At present, th

Steven Cheng 12 Feb 22, 2022
a undetectable tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our qqgroup is 703156427

a undetectable ios root access tool by modify odyssey, support sign disable & dylib injection, test on iphoneX(13.5.1 expolit by FreeTheSandbox), our

null 58 Nov 22, 2021
An experimental tool to estimate the similarity between all pairs of contigs

This is an experimental tool to estimate the approximate distances between all pairs of unitigs. It takes a GFA or FASTA file as input and outputs a T

Heng Li 33 Mar 16, 2022
Powerful automated tool for reverse engineering Unity IL2CPP binaries

Powerful automated tool for reverse engineering Unity IL2CPP binaries

Katy 2k Nov 26, 2022
Simple tool to visualize and amplify mouse movements

mousemic Simple tool to visualize and amplify mouse movements. This utility uses a high-level X11 Api so is not really more sensitive than your screen

Alfredo Ortega 40 Nov 25, 2022
A lightweight ARM reverse engineering tool.

eydis A lightweight (basic and slow) ARM reverse engineering tool. I. Requierements macOS/Linux, Basics compiling tools, The SQLite3 + readline framew

Yui Aioi 18 Aug 15, 2022
Stack-based texture generation tool written in C99!

Stack-based texture generation tool written in C99! Brought to you by @zaklaus and contributors Introduction zpl.texed is a cross-platform stack-based

zpl | pushing the boundaries of simplicity. 19 Sep 10, 2022
Matryoshka loader is a tool that red team operators can leverage to generate shellcode for Microsoft Office document phishing payloads.

Overview Matryoshka loader is a tool that red team operators can leverage to generate shellcode for an egghunter to bypass size-limitations and perfor

Praetorian 25 Oct 28, 2022
A shellcode crypto-packing tool for PoC (used with msfvenom payloads)

crypter A shellcode crypto-packing tool for PoC (used with msfvenom/binary payloads) This tool is for proof of concept only - please use responsibly.

ripmeep 11 Jul 30, 2022
MDE is a model extraction tool that converts Destiny 2 dynamic models into fbx files supporting textures, skeletons, and all provided vertex data.

MDE is a model extraction tool that converts Destiny 2 dynamic models into fbx files. A dynamic model is one that is animated or is spawned in during the game.

Montague 33 Sep 2, 2022
Fast and lightweight username lookup tool inspired by sherlock.

Lightweight username lookup inspired by Sherlock. Created in C++. Features Works on 250+ websites Fast, easy to use and compact How to use $ scout.exe

eternity 10 Jun 23, 2022
A Simple tool to execute shellcode with the ability to detect mouse movement

Noobi A Simple tool to execute shellcode with the ability to detect mouse movement Features: Sandbox evasion through detecting mouse movement and chec

null 10 Nov 9, 2022
sent is a simple plaintext presentation tool.

sent is a simple plaintext presentation tool. sent does not need latex, libreoffice or any other fancy file format, it uses plaintext files and png im

Injamul Mohammad Mollah 4 Sep 16, 2022
A tool to kill antimalware protected processes

Backstab Kill EDR Protected Processes Have these local admin credentials but the EDR is standing in the way? Unhooking or direct syscalls are not work

Yasser 1k Nov 22, 2022
A simple tool that aims to efficiently and quickly parse the outputs of web scraping tools like gau

massurl is a simple tool that aims to parse the outputs of tools like gau, and extract the parameters for each URL, remove duplicates and do it all very quickly. Because web scraping tools' outputs can get very large very quickly, it is nice to have a tool that parses them and and outputs something clean and easy to read.

Fr1nge 13 Jul 24, 2022