Suite of C++ libraries for radio astronomy data processing

Overview

Casacore

A suite of C++ libraries for radio astronomy data processing.

Installation

Debian / Ubuntu

Casacore is now part of Debian and Ubuntu, use apt to search for the package you need:

$ apt-get update
$ apt-get search casacore

MacOS

Use Homebrew with the ska-sa tap to install casacore:

$ brew tap ska-sa/tap
$ brew install casacore

Docker

We now also publish docker images for each github branch on quay.io:

$ docker pull quay.io/casacore/casacore:master

Building from source

Getting the source code

The casacore source code is maintained on github.

You can obtain it using:

$ git clone https://github.com/casacore/casacore

Requirements

To compile casacore you need to meet the following requirements:

  • cmake
  • gfortran
  • g++
  • flex
  • bison
  • blas
  • lapack
  • cfitsio (3.181 or later)
  • wcslib (4.20 or later)
  • sofa (optional, only for testing casacore measures)
  • fftw3
  • hdf5 (optional)
  • numpy (optional)
  • boost-python (optional)
  • ncurses (optional)

On Debian / Ubuntu you can install these with:

$ sudo apt-get install build-essential cmake gfortran g++ libncurses5-dev \
   libreadline-dev flex bison libblas-dev liblapacke-dev libcfitsio-dev \
   wcslib-dev libfftw3-dev

and the optional libraries:

$ sudo apt-get install libhdf5-serial-dev python-numpy \
    libboost-python-dev libpython3.4-dev libpython2.7-dev

On CentOS7 you can install these with:

$ sudo yum install cmake cmake-gui gcc-gfortran gcc-c++ flex bison \
       blas blas-devel  lapack lapack-devel cfitsio cfitsio-devel \
       wcslib wcslib-devel ncurses ncurses-devel readline readline-devel\
       python-devel boost boost-devel fftw fftw-devel hdf5 hdf5-devel\
       numpy boost-python 

Obtaining measures data

Various parts of casacore require measures data, which requires regular updating. You can obtain the WSRT measures archive from the ASTRON FTP server:

ftp://ftp.astron.nl/outgoing/Measures/

Extract this somewhere on a permanent location on your filesystem.

Compilation

In the casacore source folder run:

mkdir build
cd build
cmake ..
make 
make install

there are various flags available to cmake to enable and disable options:

$ cmake -DDATA_DIR=/usr/share/casacore/data -DUSE_OPENMP=ON \
    -DUSE_HDF5=ON -DBUILD_PYTHON=ON -DUSE_THREADS=ON

The DATA_DIR should point to the location where you extracted the measures data. Special variables %CASAROOT% and %CASAHOME% can be used here, which can be set at run time through the .casarc file.

You can build python3 support using -DBUILD_PYTHON3=ON. Note that CMake may have problems detecting the correct python3 libraries and headers, so probably you need to set them manually. For example:

-DPYTHON3_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so
-DPYTHON3_INCLUDE_DIR=/usr/include/python3.4

To configure Python2 specific settings use:

PYTHON2_EXECUTABLE
PYTHON2_LIBRARY
PYTHON2_INCLUDE_DIR

To configure Python3 specific settings use:

PYTHON3_EXECUTABLE
PYTHON3_LIBRARY
PYTHON3_INCLUDE_DIR

If you run into problems with boost libraries, try setting -DBoost_NO_BOOST_CMAKE=True. This will be necessary if you have the libraries from NRAO casa in your PATH or LD_LIBRARY_PATH.

Documentation

http://casacore.github.io/casacore

Problems & bugs

If you have any issues compiling or using casacore, please open an issue on the issue tracker on github.

If you have patches please open a pull request. Your contributions are more than welcome! But to maintain a high code quality we have written a contribution manual, please read that first.

Issues
  • Replace tcsh script measuresdata.csh with /bin/sh script

    Replace tcsh script measuresdata.csh with /bin/sh script

    The new script is modelled almost after the tcsh script, but uses curl to get the files from remote.

    This pull request removes tcsh from the dependencies of casacore.

    opened by olebole 37
  • Change of behaviour for Regex::match()

    Change of behaviour for Regex::match()

    Pull request #881 (authored by @gervandiepen and committed by @tammojan) introduced an (unintended?) change in behaviour in Regex::match(). The documentation says:

    Test if the regular expression matches (part of) string s.

    The return value gives the length of the matching string part, or String::npos if there is no match or an error. The string has len characters and the test starts at position pos. The string may contain null characters. Negative p is allowed to match at end.

    However, since the std::regex_match() algorithm that is now used to implement this functionality only matches entire strings, so partial matches are no longer found. The correspond regex matching functions in the String class also get this changed behaviour.

    This breaks (for example) the CASA SingleDishSkyCal class which tries to match observing modes using a regex like:

    "^OBSERVE_TARGET#ON_SOURCE"

    which is supposed to match all observing modes that start with

    "OBSERVE_TARGET#ON_SOURCE"

    this can be fixed of course by tweaking the regex to:

    "^OBSERVE_TARGET#ON_SOURCE.*"

    but I'm not sure it is reasonable to expect applications to make such changes as it may e difficult to find all relevant cases in the code and in some cases the regex may come from user input that isn't controlled by the programmer.

    At the very least the documentation should be adjusted if the change of behaviour is intentional.

    opened by kettenis 28
  • Release casacore 2.0.2?

    Release casacore 2.0.2?

    I noticed in the last week's release 4.4 of NRAO Casa that its CMakeLists.txt contains the line

    casa_find( CASACORE
               VERSION 2.0.2
    

    ( see https://svn.cv.nrao.edu/view/casa/branches/release-4_4/code/CMakeLists.txt?revision=33147 )

    It would be nice if this release could actually exist and be compatible with Casa 4.4.

    This may need some ad-hoc fix for #113 since the requested functionality was already implemented on a fork of casacore ( https://svn.cv.nrao.edu/view/casa/branches/release-4_4/casacore/ ).

    opened by tammojan 27
  • CAS-8783 Parallel stats

    CAS-8783 Parallel stats

    • Refactor, create ClassicalStatistics::_increment() method for updating loop counters
    • Implement StatisticsUtilities::combine() for combining statistics derived from different data sets to get a set of statistics that applies to all those data sets.
    • implementation of ParallelClassicalStatistics for parallel support in _getStatistics(). There are other avenues to investigate, but this implementation does improve performance on computing stats on large images by a factor of 1.5 to 3 depending on the use case.
    • add support for parallel computation of stats to LatticeStatistics (default behavior is unchanged; objects must be explicitly configured to use parallel computation).
    • reformat files, changing tabs to spaces.
    opened by dmehring 26
  • Install issues on Mac OS X

    Install issues on Mac OS X

    I'm trying to compile casacore on mac OS X by linking against conda-installed libraries. This may not be entirely sane, but I wanted to post my efforts here and ask if there's anything critical I've missed.

    My cmake command involved a lot of flags:

    PATH=~/anaconda/envs/astropy35/bin/:~/anaconda/envs/astropy27/bin/:$PATH cmake .. -DUSE_FFTW3=ON -DBUILD_PYTHON3=on -DUSE_HDF5=ON -DBUILD_PYTHON=ON -DUSE_THREADS=ON -DUSE_OPENMP=ON -DPYTHON3_LIBRARY=/Users/adam/anaconda/envs/astropy35/lib/python3.5/config-3.5m/libpython3.5m.a -DPYTHON3_INCLUDE_DIR=/Users/adam/anaconda/envs/astropy35/include/python3.5m/ -DBOOST_ROOT=/Users/adam/anaconda/envs/astropy35/
    

    and looked mostly successful up to the end, when it failed. The error started around this line.

    My cmake command revealed that I'm missing SOFA libraries and CASA data, but I don't think either of these things should affect compilation as above.

    Any help would be appreciated.

    opened by keflavich 22
  • Enhancing parallel IO in Casacore Table Data System

    Enhancing parallel IO in Casacore Table Data System

    I don't know if this is the best place but I was looking to some discussions on enhancing parallel IO for the table system as well as Measurementsets.

    A few years ago I wrote a parallel storage manager for the casacore table system based on ADIOS (https://github.com/SKA-ScienceDataProcessor/AdiosStMan.git) when I was at ICRAR, which was put into the SKA-SDP as an early investigation into SKA scale measurementset experiments. When @gervandiepen was visiting us at ICRAR we talked about merging this AdiosStMan into the master but I was worried about the code quality because it was only a prototype rather than a production software. @gervandiepen was also worried about introducing MPI dependency to casacore. So eventually we dropped the plan.

    Recently I have been working on updating the Adios Storage Manager using our new software ADIOS 2.0 (https://github.com/ornladios/ADIOS2) that we are developing at Oak Ridge National Lab under the US Department of Energy's Exa-scale Computing program. The motivation behind this is that we are planning on running a full-scale SKA 1 data pipeline simulation, which may require writing data into a single measurementset table from multiple processes. If this is successful then we can re-think about improving the code quality and merge our efforts into the master.

    However, the current parallel storage manager design is more like a hack than a generic solution because we couldn't find parallel support in casacore's table layer API. As a result, we need to hack user applications and make them believe that each MPI rank is writing to its own independent table, while the Adios Storage Manger does dirty tricks underneath and actually writes data into a single column file. This has worked so far but it makes user applications very ugly and less maintainable.

    I have roughly read through the table layer code trying to think of a solution. If we can add some MPI-aware code in the procedures of creating a table, making only Rank 0 actually write table metadata file, while other ranks hold the table metadata in memory without writing to files, this would make the table system much more friendly to parallel storage managers as well as MPI incorporated applications.

    I am wondering if this is possible or if there is any other suggestions or plans on enhancing parallel IO for casacore. Looking forward to any discussions or suggestions. Thank you.

    enhancement 
    opened by JasonRuonanWang 21
  • Discussion: Adding C++11 features

    Discussion: Adding C++11 features

    We have to start thinking about introducing more C++11 features in Casacore. I can think of a few:

    1. 'Manually' instantiate some often used templates like Array, ScalarColumn, etc. C++11 has a new feature to do so. It makes compilation much faster, because the template does not need to be instantiated everywhere it is used. It should also make the libraries (much) smaller.
    2. Add Array&& arguments to ArrayMath. In this way less temporaries have to be created when doing e.g. a_b+c because c can be added to the temporary result of a_b. The cost is that every operator needs 4 functions instead of 1, but they are very simple. However, I'm not entirely sure if it is worth the trouble.
    3. Use std::regex in the Regex class; it can only be done if fully compatible.

    I have not thought any further, but I'm sure there are more things that could be done. Maybe a student could do these things.

    A very important issue is whether we think all sites using casacore will have compilers supporting C++11. I don't think that is the case, so for the time being all changes need to be #ifdef-ed.

    I'd like to hear your thoughts. Cheers, Ger

    opened by gervandiepen 20
  • Use FFTW instead of FFTPack in Fits-IDI to MS table converter

    Use FFTW instead of FFTPack in Fits-IDI to MS table converter

    The FitsIDItoMS class is the only class that was still dependent on the FFTPack Fortran code. By rewriting it to use FFTW, the FFTPack code can be removed. This required implementing a wrapper for the real even FFT in the FFTW class, as that was not available.

    The FFTW class should be rewritten to have a more sensible interface. I've added some TODO comments for that. Despite the use of a mutex, the FFTW wrapper is also not thread safe.

    This is related to the work in #1024 and #1017.

    opened by aroffringa 19
  • Cache invalidation deadlock with user locks

    Cache invalidation deadlock with user locks

    While debugging a deadlock in dask-ms I noticed that when a running all calls to python-casacore in a single threaded executor pool the library deadlocks with no exception when putcol is called without setting a writable user lock. It always seems to hang on a ISMColumn operation. The bug likely affects only this storage manager:

    #0  0x00007f12c8d3c6d1 in casacore::ISMColumn::getScalarColumnCellsdoubleV (this=0x7f12c006f1c0, rownrs=..., values=0x7f12c4a77670) at /scratch/bhugo/oms-subset/casacore/tables/DataMan/ISMColumn.cc:464
    #1  0x00007f12c8d86ad8 in casacore::StManColumn::getScalarColumnCellsV (this=0x7f12c006f1c0, rownrs=..., dataPtr=0x7f12c4a77670) at /scratch/bhugo/oms-subset/casacore/tables/DataMan/StManColumn.cc:227
    #2  0x00007f12c8c7bbf2 in casacore::ScalarColumnData<double>::getScalarColumnCells (this=0x7f12c0066960, rownrs=..., val=0x7f12c4a77670)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScaColData.tcc:151
    #3  0x00007f12c8c99375 in casacore::ScalarColumn<double>::getColumnCells (this=0x7f12c4a77450, rownrs=..., vec=..., resize=false)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:205
    #4  0x00007f12c8c990d4 in casacore::ScalarColumn<double>::getColumnRange (this=0x7f12c4a77450, rowRange=..., vec=..., resize=false)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:180
    #5  0x00007f12c8c9921b in casacore::ScalarColumn<double>::getColumnRange (this=0x7f12c4a77450, rowRange=...) at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:165
    #6  0x00007f12c8cd343c in casacore::TableProxy::getValueFromTable (this=0x7f12c0001b30, colName=..., rownr=100, nrow=100, incr=1, isCell=false)
        at /scratch/bhugo/oms-subset/casacore/tables/Tables/TableProxy.cc:2245
    #7  0x00007f12c8cca084 in casacore::TableProxy::getColumn (this=0x7f12c0001b30, columnName=..., row=100, nrow=100, incr=1) at /scratch/bhugo/oms-subset/casacore/tables/Tables/TableProxy.cc:1149
    #8  0x00007f12ca5f1970 in boost::python::detail::invoke<boost::python::to_python_value<casacore::ValueHolder const&>, casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::arg_from_python<casacore::TableProxy&>, boost::python::arg_from_python<casacore::String const&>, boost::python::arg_from_python<int>, boost::python::arg_from_python<int>, boost::python::arg_from_python<int> > (ac3=..., ac2=..., ac1=..., ac0=..., tc=<synthetic pointer>, f=
        @0x21865c8: (casacore::ValueHolder (casacore::TableProxy::*)(casacore::TableProxy * const, const casacore::String &, int, int, int)) 0x7f12c8cc9fdc <casacore::TableProxy::getColumn(casacore::String const&, int, int, int)>, rc=...) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/invoke.hpp:86
    #9  boost::python::detail::caller_arity<5u>::impl<casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::default_call_policies, boost::mpl::vector6<casacore::ValueHolder, casacore::TableProxy&, casacore::String const&, int, int, int> >::operator() (args_=<optimized out>, this=0x21865c8) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/caller.hpp:238
    #10 boost::python::objects::caller_py_function_impl<boost::python::detail::caller<casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::default_call_policies, boost::mpl::vector6<casacore::ValueHolder, casacore::TableProxy&, casacore::String const&, int, int, int> > >::operator() (this=0x21865c0, args=<optimized out>, kw=<optimized out>)
        at /scratch/bhugo/oms-subset/opt/include/boost/python/object/py_function.hpp:38
    #11 0x00007f12c853b00d in boost::python::objects::function::call(_object*, _object*) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #12 0x00007f12c853b208 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #13 0x00007f12c8543263 in boost::python::detail::exception_handler::operator()(boost::function0<void> const&) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #14 0x00007f12c82f3b6a in boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>::operator() (this=0x2190e08, handler=..., f=..., 
        translate=0x7f12c82f2fbd <casacore::python::translate_iterexcp(casacore::IterError const&)>) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/translate_exception.hpp:46
    #15 0x00007f12c82f38b5 in boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> >::operator()<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=0x2190e10, f=..., a=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:388
    #16 0x00007f12c82f36c1 in boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=0x2190e08, a1=..., a2=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:1318
    #17 0x00007f12c82f352d in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (
        function_obj_ptr=..., a0=..., a1=...) at /scratch/bhugo/oms-subset/opt/include/boost/function/function_template.hpp:137
    #18 0x00007f12c8543238 in boost::python::detail::exception_handler::operator()(boost::function0<void> const&) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #19 0x00007f12c82f3b6a in boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>::operator() (this=0x2164e08, handler=..., f=..., 
        translate=0x7f12c82f2fbd <casacore::python::translate_iterexcp(casacore::IterError const&)>) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/translate_exception.hpp:46
    #20 0x00007f12c82f38b5 in boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> >::operator()<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=0x2164e10, f=..., a=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:388
    #21 0x00007f12c82f36c1 in boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=0x2164e08, a1=..., a2=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:1318
    #22 0x00007f12c82f352d in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (
        function_obj_ptr=..., a0=..., a1=...) at /scratch/bhugo/oms-subset/opt/include/boost/function/function_template.hpp:137
    #23 0x00007f12c854301d in boost::python::handle_exception_impl(boost::function0<void>) () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #24 0x00007f12c8538409 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #25 0x00000000005c3bd7 in PyObject_Call () at ../Objects/abstract.c:2165
    #26 0x00000000005354af in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7f12c4a77ed0, func=<optimized out>) at ../Python/ceval.c:4936
    

    The minimum steps to reproduce the issue are the following. Note the deliberate bug induced in line 30. I'm very careful not to submit calls (even open and close calls) in anything other than the single executor thread:

      1 from __future__ import print_function
      2 
      3 import argparse
      4 import numpy as np
      5 import pyrap.tables as pt
      6 from concurrent.futures import ThreadPoolExecutor
      7 import threading
      8 def create_parser():
      9     p = argparse.ArgumentParser()
     10     p.add_argument("ms")
     11     p.add_argument("-rc", "--row-chunks", default=100, type=int)
     12     p.add_argument("-nc", "--num-consumers", default=64, type=int)
     13     return p
     14 
     15 args = create_parser().parse_args()
     16 
     17 futs = []
     18 
     19 consumer = ThreadPoolExecutor(max_workers=args.num_consumers)
     20 producer = ThreadPoolExecutor(max_workers=1) # ensure we don't interact with CASAcore in more than one thread
     21 
     22 # get future to the table on thread 1 of the data producer queue
     23 tbl_hnd = producer.submit(pt.table, args.ms, readonly=False, lockoptions="user")
     24 
     25 def _proxy_table_method(handle, method, write, *args, **kwargs):
     26     print("Running CC method '{}' on thread {}".format(method, threading.get_ident()))
     27 
     28     table = handle.result()
     29 
     30     table.lock(write=not write)
     31 
     32     try:
     33         result = getattr(table, method)(*args, **kwargs)
     34 
     35         if write:
     36             table.flush()
     37     finally:
     38         if method != "close":
     39             table.unlock()
     40 
     41     return result
     42 
     43 nrows = producer.submit(_proxy_table_method, tbl_hnd, "nrows", False).result()
     44 print("{} rows in this MS".format(nrows))
     45 
     46 
     47 def _dummy_rw(colname, r, e):
     48     """ Dummy method to read and write using user locks """
     49     def _rw_batch(colname, r, e):
     50         print("Submitting read write batch for rows {} through {}".format(r, e))
     51         nddat = _proxy_table_method(tbl_hnd, "getcol", False, colname, startrow=r, nrow=e-r)
     52         # whatever ops to be done here
     53         _proxy_table_method(tbl_hnd, "putcol", True, colname, nddat, startrow=r, nrow=e-r)
     54         return nddat
     55     datshp = producer.submit(_rw_batch, colname, r, e).result().shape
     56     return r, e-r, datshp
     57 
     58 # Chunk up the data nice and fine so there are a lot of calls to CC
     59 for r in range(0, nrows, args.row_chunks):
     60     e = min(r + args.row_chunks, nrows)
     61     futs.append(consumer.submit(_dummy_rw,"TIME", r, e))
     62     futs.append(consumer.submit(_dummy_rw,"FLAG", r, e))
     63 
     64 # ensure all rw ops finish
     65 for f in futs:
     66     f.result()
     67 
     68 print(">>>>>>>>>>>>FINISHED<<<<<<<<<<<")
     69 for f in futs:
     70     sr, nr, res = f.result()
     71     print(sr, nr, res)
     72 
     73 import atexit
     74 atexit.register(lambda: producer.submit(_proxy_table_method, tbl_hnd, "close", False).result())
    

    with output as follows:

    > $ python3 test.py msdir/subset-1563189318-20M.ms
    Successful read/write open of user-locked table msdir/subset-1563189318-20M.ms: 24 columns, 6368888 rows
    Running CC method 'nrows' on thread 139718585456384
    6368888 rows in this MS
    Submitting read write batch for rows 0 through 100
    Running CC method 'getcol' on thread 139718585456384
    Running CC method 'putcol' on thread 139718585456384
    Submitting read write batch for rows 0 through 100
    Running CC method 'getcol' on thread 139718585456384
    Running CC method 'putcol' on thread 139718585456384
    Submitting read write batch for rows 100 through 200
    Running CC method 'getcol' on thread 139718585456384
    

    Note that I get the expected user exception when executing in the parent thread:

    In [5]: t = tbl("msdir/subset-1563189318-20M.ms", readonly=False, lockoptions="user")
    Successful read/write open of user-locked table msdir/subset-1563189318-20M.ms: 24 columns, 6368888 rows
    
    In [6]: t.lock(write=True)
    
    In [7]: time = t.getcol("TIME")
    
    In [8]: t.unlock()
    
    In [9]: t.lock(write=False)
    
    In [10]: t.putcol("TIME", time)
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-10-d8534bd8f3fa> in <module>()
    ----> 1 t.putcol("TIME", time)
    
    /scratch/bhugo/oms-subset/opt/home/hugo/.local/lib/python3.5/site-packages/casacore/tables/table.py in putcol(self, columnname, value, startrow, nrow, rowincr)
       1152
       1153         """
    -> 1154         self._putcol(columnname, startrow, nrow, rowincr, value)
       1155
       1156     def putvarcol(self, columnname, value, startrow=0, nrow=-1, rowincr=1):
    
    RuntimeError: ColumnSet::doLock: table /scratch/bhugo/oms-subset/msdir/subset-1563189318-20M.ms should be locked when using UserLocking
    

    I've compiled casacore master (#2f45a1a7fa8474ffc) from source with debug symbols and installed the source distribution of python-casacore==3.1.1. We are running python 3.5.

    opened by bennahugo 19
  • CAS-8447 enable parallel processing using omp

    CAS-8447 enable parallel processing using omp

    I had made some methods protected in a previous commit that was merged, but now these can be re-made private since I was able to provide a multi-threaded implementation using openmp in MSMetaData, obviating the need for a derived class.

    opened by dmehring 19
  • Fast way to update DATA and WEIGHT columns of  a MS

    Fast way to update DATA and WEIGHT columns of a MS

    Hello everyone,

    This is a question more than a issue. I am using casacore to develop an image reconstruction algorithm and I am currently using casacore to read and write MS files.

    Suppose that I have a data structures like

    field[f].visibilities[i][sto].Vm[h].x

    field[f].visibilities[i][sto].Vm[h].y

    which are my model visibilities, and f is the index for fields, i is the index for frequencies (yep, we get rid of spw, so i is the index of each frequency), sto is the index of the stokes and h is the index of each visibility with field f, frequency i and stoke sto.

    What would be the fastest way to update column DATA ?

    There is an obvious choice like iterating all the samples, but that wouldn't be convenient, and would be extremely slow for big datasets. Can be a column be updated by another table created and updated by a query?

    Any response is appreciated.

    Best regards.

    opened by miguelcarcamov 18
  • segfault in coordinate module

    segfault in coordinate module

    While looking at some casa C++ tests I have found a segfault that can be reproduced with a small code snippet in Debian 11. This snippet is part of one of the imaging casa C++ tests:

    #include <casacore/measures/Measures/Stokes.h>
    #include <casacore/coordinates/Coordinates/CoordinateSystem.h>
    #include <casacore/coordinates/Coordinates/DirectionCoordinate.h>
    #include <casacore/coordinates/Coordinates/SpectralCoordinate.h>
    #include <casacore/coordinates/Coordinates/StokesCoordinate.h>
    #include <casacore/casa/namespace.h>
    
    using namespace std;
    using namespace casacore;
    
    Int main(/*int argc, char **argv*/){
    
       Matrix<Double> xform(2,2);
       xform = 0.0;
       xform.diagonal() = 1.0;
       DirectionCoordinate dc(MDirection::J2000, Projection::SIN, Quantity(20.0,"deg"), Quantity(20.0, "deg"),
                              Quantity(0.5, "arcsec"), Quantity(0.5,"arcsec"),
                              xform, 50.0, 50.0, 999.0,
                              999.0);
       Vector<Int> whichStokes(1, Stokes::I);
       StokesCoordinate stc(whichStokes);
       SpectralCoordinate spc(MFrequency::LSRK, 1.5e9, 1e6, 0.0 , 1.420405752E9);
       CoordinateSystem cs;
       cs.addCoordinate(dc);
       cs.addCoordinate(stc);
    
       exit(0);
    }
    
    

    Valgrind reports the following errors:

    ==1432366== Invalid read of size 4
    ==1432366==    at 0x49D8B25: __exchange_and_add (atomicity.h:50)
    ==1432366==    by 0x49D8B25: __exchange_and_add_dispatch (atomicity.h:84)
    ==1432366==    by 0x49D8B25: _M_release (shared_ptr_base.h:155)
    ==1432366==    by 0x49D8B25: ~__shared_count (shared_ptr_base.h:733)
    ==1432366==    by 0x49D8B25: ~__shared_ptr (shared_ptr_base.h:1183)
    ==1432366==    by 0x49D8B25: ~shared_ptr (shared_ptr.h:121)
    ==1432366==    by 0x49D8B25: casacore::Array<double, std::allocator<double> >::~Array() (Array.tcc:190)
    ==1432366==    by 0x10A5FC: main (Quantum.h:271)
    ==1432366==  Address 0x80ba109 is 25 bytes after a block of size 32 in arena "client"
    ==1432366== 
    ==1432366== Invalid free() / delete / delete[] / realloc()
    ==1432366==    at 0x483A58B: operator delete[](void*) (vg_replace_malloc.c:649)
    ==1432366==    by 0x49B90D9: casacore::ArrayBase::~ArrayBase() (ArrayBase.cc:153)
    ==1432366==    by 0x10A5FC: main (Quantum.h:271)
    ==1432366==  Address 0x2 is not stack'd, malloc'd or (recently) free'd
    ==1432366== 
    ==1432366== Warning: set address range perms: large range [0x8233040, 0x4886de40) (undefined)
    ==1432366== Invalid read of size 1
    ==1432366==    at 0x483F8DE: memmove (vg_replace_strmem.c:1270)
    ==1432366==    by 0x4A1FD90: __copy_m<long int> (stl_algobase.h:426)
    ==1432366==    by 0x4A1FD90: __copy_move_a2<false, long int*, long int*> (stl_algobase.h:472)
    ==1432366==    by 0x4A1FD90: __copy_move_a1<false, long int*, long int*> (stl_algobase.h:506)
    ==1432366==    by 0x4A1FD90: __copy_move_a<false, long int*, long int*> (stl_algobase.h:513)
    ==1432366==    by 0x4A1FD90: copy<long int*, long int*> (stl_algobase.h:569)
    ==1432366==    by 0x4A1FD90: __copy_n<long int*, long unsigned int, long int*> (stl_algo.h:750)
    ==1432366==    by 0x4A1FD90: copy_n<long int*, long unsigned int, long int*> (stl_algo.h:777)
    ==1432366==    by 0x4A1FD90: casacore::IPosition::operator=(casacore::IPosition const&) (IPosition.cc:269)
    ==1432366==    by 0x49BA436: casacore::ArrayBase::copyVectorHelper(casacore::ArrayBase const&) (ArrayBase.cc:594)
    ==1432366==    by 0x4D79A24: casacore::Vector<casacore::String, std::allocator<casacore::String> >::assign_conforming_implementation(casacore::Vector<casacore::String, std::allocator<casacore::String> > const&, std::integral_constant<bool, true>) (Vector.tcc:249)
    ==1432366==    by 0x4D72C2E: assign_conforming (Vector.h:185)
    ==1432366==    by 0x4D72C2E: operator= (Vector.h:195)
    ==1432366==    by 0x4D72C2E: casacore::DirectionCoordinate::copy(casacore::DirectionCoordinate const&) (DirectionCoordinate.cc:2189)
    ==1432366==    by 0x4D72FA4: casacore::DirectionCoordinate::DirectionCoordinate(casacore::DirectionCoordinate const&) (DirectionCoordinate.cc:220)
    ==1432366==    by 0x4D73032: casacore::DirectionCoordinate::clone() const (DirectionCoordinate.cc:1173)
    ==1432366==    by 0x4D2224C: casacore::CoordinateSystem::addCoordinate(casacore::Coordinate const&) (CoordinateSystem.cc:194)
    ==1432366==    by 0x10A6F7: main (tGridFT.cc:24)
    ==1432366==  Address 0x1fff001000 is not stack'd, malloc'd or (recently) free'd
    ==1432366== 
    ==1432366== 
    ==1432366== Process terminating with default action of signal 11 (SIGSEGV)
    ==1432366==  Access not within mapped region at address 0x1FFF001000
    ==1432366==    at 0x483F8DE: memmove (vg_replace_strmem.c:1270)
    ==1432366==    by 0x4A1FD90: __copy_m<long int> (stl_algobase.h:426)
    ==1432366==    by 0x4A1FD90: __copy_move_a2<false, long int*, long int*> (stl_algobase.h:472)
    ==1432366==    by 0x4A1FD90: __copy_move_a1<false, long int*, long int*> (stl_algobase.h:506)
    ==1432366==    by 0x4A1FD90: __copy_move_a<false, long int*, long int*> (stl_algobase.h:513)
    ==1432366==    by 0x4A1FD90: copy<long int*, long int*> (stl_algobase.h:569)
    ==1432366==    by 0x4A1FD90: __copy_n<long int*, long unsigned int, long int*> (stl_algo.h:750)
    ==1432366==    by 0x4A1FD90: copy_n<long int*, long unsigned int, long int*> (stl_algo.h:777)
    ==1432366==    by 0x4A1FD90: casacore::IPosition::operator=(casacore::IPosition const&) (IPosition.cc:269)
    ==1432366==    by 0x49BA436: casacore::ArrayBase::copyVectorHelper(casacore::ArrayBase const&) (ArrayBase.cc:594)
    ==1432366==    by 0x4D79A24: casacore::Vector<casacore::String, std::allocator<casacore::String> >::assign_conforming_implementation(casacore::Vector<casacore::String, std::allocator<casacore::String> > const&, std::integral_constant<bool, true>) (Vector.tcc:249)
    ==1432366==    by 0x4D72C2E: assign_conforming (Vector.h:185)
    ==1432366==    by 0x4D72C2E: operator= (Vector.h:195)
    ==1432366==    by 0x4D72C2E: casacore::DirectionCoordinate::copy(casacore::DirectionCoordinate const&) (DirectionCoordinate.cc:2189)
    ==1432366==    by 0x4D72FA4: casacore::DirectionCoordinate::DirectionCoordinate(casacore::DirectionCoordinate const&) (DirectionCoordinate.cc:220)
    ==1432366==    by 0x4D73032: casacore::DirectionCoordinate::clone() const (DirectionCoordinate.cc:1173)
    ==1432366==    by 0x4D2224C: casacore::CoordinateSystem::addCoordinate(casacore::Coordinate const&) (CoordinateSystem.cc:194)
    ==1432366==    by 0x10A6F7: main (tGridFT.cc:24)
    ==1432366==  If you believe this happened as a result of a stack
    ==1432366==  overflow in your program's main thread (unlikely but
    ==1432366==  possible), you can try to increase the size of the
    ==1432366==  main thread stack using the --main-stacksize= flag.
    ==1432366==  The main thread stack size used in this run was 8388608.
    
    

    If one removes the creation of object spc then there is another valgrind report, in this case in the getenv() function. But I think that's a side effect of the first error reported by valgrind, namely the invalid read of size 4.

    I was getting this error on a Debian 11 system. The weird thing is that this cannot be reproduced on a Fedora 35 system.

    opened by cquike 0
  • Speedup frame conversion

    Speedup frame conversion

    Changes aim to speedup frequency frame conversion by reducing memory allocation and deallocation. I'm sorry that many whitespace changes are included, especially in Coordinate.cc. Major changes of Coordinate.cc is L977-1036.

    I carried out hot spot analysis on the imaging (convolutional gridding) of single dish data using CASA's tsdimaging task. Test data contains about 70000 rows with 2048 channels and 2 polarizations. Roughly 10^8 frequency conversions are required. The screenshots below are the result of hot spot analysis of original code. You can see that significant fraction of CPU time was spent for operator new (please see Summary figure), which is very likely to accompany extra CPU time to execute constructor code. Indeed, constructor was one of the bottleneck of frequency conversion. The Breakdown figure for the original code shows that about 23% of CPU time was spent for frequency conversion (MCFrequency::doConvert), which competes to the CPU time for gridding (ggridsd_), and about half of frequency conversion was construction/destruction or copy of MVPosition and MVDirection objects. This is really inefficient.

    In the improved code, I tried to reduce using operator new (Coordinate.cc) and to avoid using constructor/destructor to update another object (MCFrequency.cc and MeasMath.cc). You can see in the figures for branch code that CPU time for operator new was less than half of the original analysis, and fraction of the CPU time for frequency conversion was reduced to 15%, which means performance of frequency conversion was significantly improved although it didn't reached to double-speed. In terms of total CPU time, improvement was about 10% (actually performance was improved about 20% including the changes in casa6 repo).

    The change is currently a trade-off between performance and the "beauty" of the object oriented programming because the change somehow breaks encapsulation of the object by exposing implementation detail of MVDirection and MVPosition outside these classes. But 10% improvement in terms of total execution time is quite beneficial.

    Hot spot analysis of original code

    Summary

    conversion-benchmark-summary-original

    Breakdown

    conversion-benchmark-breakdown-original

    How spot analysis of branch code

    Summary

    conversion-benchmark-summary-tuned

    Breakdown

    conversion-benchmark-breakdown-tuned

    opened by tnakazato 2
  • Use shared_ptr instead of own reference counting of BaseTable

    Use shared_ptr instead of own reference counting of BaseTable

    The Table object used its own (old) way of counting BaseTable references. It has been replaced by shared_ptr. However, it is somewhat tricky because not all BaseTable references in DataManager objects cannot be counted (otherwise mutual referencing occurs). Therefore the actual pointer in a Table object is still a raw pointer to BaseTable. A weak_ptr (in the BaseTable object itself) it used to created shared_ptr as needed.

    The code has been run through valgrind without any error or leak.

    opened by gervandiepen 0
  • Remove notice and link

    Remove notice and link

    According to @gervandiepen (in https://github.com/casacore/casacore/issues/896) this functionality is unused. This PR changes the behaviour of RecordFieldPtr. Some tests in Record and RecordField explicitly check for this behaviour, and therefore fail currently, but all other tests still succeed. It should be checked if this is ok before merging.

    This is #1096, but on the casacore repos instead of my own.

    opened by aroffringa 6
  • WIP #1163

    WIP #1163

    @gervandiepen This fixes the issue we had with tables not being thread safe

    I did not fix it as you suggested in the ticket but rather made the tablecache a multiton depending on the pid and tid. This replaces what you had in plaintable by not having a single tablecache static (class) object. Previously the underlying caching and buckets associated with the shared table system induced a segfault (edit: when removing the python-based single threading via the GIL)

    Rather now the first time a thread executes an open operation the plaintable will create a tablecache for use for the duration of the lifetime of that thread.

    Using @JSKenyon and @sjperkins fixes python-casacore #209 dropping the gil (which now should not be necessary because the solution makes the casacore table caching system thread safe) I get the following on a small dataset

    (venvcc) > $ python threaded_tableread.py mk64.Lwide.0.5hr.10s.856mhz.ms 50                                 
    Reading MS using a single thread: 61.677s
    Again: Reading MS using a single thread: 60.933s
    Reading MS using threads and pid multiton TableCache: 42.342s
    Reading MS using processes and pid multiton TableCache: 39.944s
    

    using the following snippet

    import numpy as np
    import threading
    import pyrap.tables as pt
    import os
    import concurrent.futures as cf
    import multiprocessing
    import time
    import sys
    
    
    def threaded_getcolnp(ms_path, colname, startrow, nrow):
    
        tid = threading.get_ident()
    
        with pt.table(ms_path, lockoptions="user", ack=False) as ms:
    
            ref_row = ms.getcol(colname, nrow=1)  # Read an example row.
            ref_dims = ref_row.shape[1:]
            ref_dtype = ref_row.dtype
    
            out = np.empty((nrow, *ref_dims), dtype=ref_dtype)  # Preallocate output.
    
            ms.getcolnp(colname, out, startrow=startrow, nrow=nrow)
    
        return  # We don't want to profile the cost of returning the result.
    
    
    if __name__ == "__main__":
    
        ms_path = sys.argv[1]
        no_times = int(sys.argv[2])
        column = "DATA"
        nchunk = 40  # Partition the read into this many chunks.
        nworker = 4  # Number of threads/processes.
    
        table = pt.table(ms_path, ack=False)
        total_nrow = table.nrows()
        table.close()
    
        nrow = int(np.ceil(total_nrow / nchunk))  # nrow per chunk.
    
        starts = np.repeat(np.arange(0, total_nrow, nrow), no_times)
        np.random.shuffle(starts)
        t0 = time.time()
        result = [threaded_getcolnp(ms_path, column, row, nrow) for row in starts]
        print(f"Reading MS using a single thread: {time.time() - t0:.3f}s")
    
    
        t0 = time.time()
        result = [threaded_getcolnp(ms_path, column, row, nrow) for row in starts]
        print(f"Again: Reading MS using a single thread: {time.time() - t0:.3f}s")
    
        t0 = time.time()
        with cf.ThreadPoolExecutor(max_workers=nworker) as tpe:
            futures = [
                tpe.submit(
                    threaded_getcolnp,
                    ms_path,
                    column,
                    row,
                    nrow
                )
                for row in starts
            ]
        print(f"Reading MS using threads and pid multiton TableCache: {time.time() - t0:.3f}s")
    
        mp_context = multiprocessing.get_context("spawn")  # Forking breaks things.
    
        t0 = time.time()
        with cf.ProcessPoolExecutor(max_workers=nworker) as ppe:
            futures = [
                ppe.submit(
                    threaded_getcolnp,
                    ms_path,
                    column,
                    row,
                    nrow
                )
                for row in starts
            ]
        print(f"Reading MS using processes and pid multiton TableCache: {time.time() - t0:.3f}s")
    

    Of course one should use the appropriate locking when trying to do reading and writing between threads or processes as before.

    opened by bennahugo 25
Releases(v3.5.0)
  • v3.5.0(Jul 5, 2022)

    3.5.0

    Highlights

    Apart from many bug fixes and modernizations, a few new features were introduced:

    • TaQL now supports copying columns to new columns (e.g. to make a backup of the FLAG column)
    • Non-zero coordinates in JPL frames, e.g. the SUN frame, are now interpreted as an offset in right ascension and declination. Previously, coordinates in a JPL coordinate frame were ignored.
    • The Dysco storage manager is now built by default as part of casacore.

    General

    • Improve continuous integration (#1180, #1182), move from Travis CI to Github Actions (#1086, #1097, #1098, #1161)
    • Modernizations: remove Mutex (#1095, #1127, #1128), typing system (#1172, #1176)
    • Allow building with newer compilers (#1134, #1137, #1145, #1206, #1208, #1210, #1211)
    • Build python3 by default, not python2 (#1209)

    MeasurementSet

    • Improve reading of DataDescriptionId (#1103)
    • Compute feed information (#1104) and field attributes (#1109) on demand
    • UVFits improvements: support X-Y mounts (#1115), allow large antenna numbers (#1144), fix handling FITS-IDI GAIN_CURVE (#1151)
    • Fix precision in T+dT syntax (#1118)

    Measures

    • Allow offsets to JPL coordinates (#1160)

    Tables

    • Cache iteration boundaries while sorting tables (#1106)
    • Use RefTable for iteration (#1108)
    • Adios2 related changes: #1110, #1116, #1121, #1148
    • Make Dysco part of casacore (#1117, #1125, #1146, #1166)
    • Add TaQL commands LIKE, COPYCOLUMN and DROPTABLE (#1154, #1169)
    • Fix one bug in rownr_t migration for TiledDataStMan (#1156)
    • Improve documentation (#1192)

    Images

    • Support CARTA opening images with a custom handler (#1158)
    • Allow easier python handling of beams (#1184)
    Source code(tar.gz)
    Source code(zip)
  • v3.4.0(Feb 19, 2021)

    3.4.0

    General

    • Major modernization of Arrays, bringing it more in line with C++11 (#1012)
    • Use std::regex (#1072)
    • Remove many warnings (#1024, #1036, #1062)
    • Use FFTW as default FFT provider (#1029, #1047, #1049)
    • UVFITS improvements (#1033, #1040, #1064, #1066, #1067)
    • Support for 64-bit row numbers.
    Source code(tar.gz)
    Source code(zip)
  • v3.3.0(Apr 15, 2020)

    3.3.0

    General

    • Replace implementation of complex trig functions by std:: (#1010)
    • Fix bug SciMath / StatsFramework where parameters were passed incorrectly (#984)

    Tables

    • Pass sorting algorithm optio nthrough to table iterator (#992)
    • Support for ADIOS2 (#1006)
    • Add .casarc option to disable table locking (#1002)

    Measurement Set / UVFits

    • Fix antenna swapping in MSConcat (#977)
    • Merge PROCESSOR table in MSConcat (#1004)
    • Fix antenna positions in importing old VLA UVFITS (#1005)

    Images

    • More efficient WCS batch coordinate lookups (#390
    Source code(tar.gz)
    Source code(zip)
  • v3.2.1(Nov 22, 2019)

    3.2.1

    This release is identical to 3.2.0 but increments the SOVERSION.

    3.2

    General

    • Remove some unused functionality (#854, #947)
    • Prevent setting OpenMP num_threads to zero (#962)

    Tables

    • Fix locking issue with IncrementalStorageManager (#970, #974)
    • Various improvements to TaQL
    • Various ADIOS2 improvements (#943, #966)

    Measures

    • Update hardcoded URLs for fetching measures data (#975)

    FITS

    • Correctly convert Nasmyth mount types (#946, #954)
    Source code(tar.gz)
    Source code(zip)
  • v3.2.0(Nov 19, 2019)

    3.2

    General

    • Remove some unused functionality (#854, #947)
    • Prevent setting OpenMP num_threads to zero (#962)

    Tables

    • Fix locking issue with IncrementalStorageManager (#970, #974)
    • Various improvements to TaQL
    • Various ADIOS2 improvements (#943, #966)

    Measures

    • Update hardcoded URLs for fetching measures data (#975)

    FITS

    • Correctly convert Nasmyth mount types (#946, #954)
    Source code(tar.gz)
    Source code(zip)
  • v3.1.2(Jul 11, 2019)

  • v3.1.1(Jun 5, 2019)

    3.1.1

    General

    • Only use DataManager.get/putSlice if possible (#901)
    • User docker for Linux build on Travis for CI (#856)
    • improve unicode support (#853)
    • Changed MadfmFunc so that it inherits from ArrayFunctorBase (#905)
    • Fix memory leak in TableProxy::getColumnDescription (#900)
    • Allow colon and fix recursive bison parsing (#894)
    • ObjectID's hash function behavior fix (#897)
    • Add a public getter for the mask array to LCRegionFixed (#895)
    • Replace STL-like containers by their STL counterparts (#890)

    Python support

    • Boost 1.67 Python components require a Python version suffix (#844)
    • Handle numpy Unicode arrays (#912)
    • Fix memory leak when converting Python unicode to casa string (#910)
    • Possible leak bug (#908)
    • Always use TpInt64 for PyLong (#916)
    • Improve CMake FindPython (#922)

    Build system

    • Travis macOS fails on Python3 OSX build system (#778)
    • Switch travis to docker /xenial (#847)
    Source code(tar.gz)
    Source code(zip)
  • v3.1.0(Apr 6, 2019)

    Fix bug, thread issues and added some new features.

    General

    • Removed many warnings issued by newer compiler versions (#798, #809, #819, #820, #866, #883)
    • Further improved thread-safety (#817, #869, #877, #886)
    • Added variance and standard deviation of complex numbers (#851)

    Tables

    • Improved parallel storage manager based on ADIOS2
    • Support of Int64 table columns (#859)
    • Added O_DIRECT support when using MultiFile (#885)
    • TaQL improvements (#813, #851, #867)
    • Small CTDS fixes (#840, #868)

    Measurement Set

    • Small improvements (#858, #864, #872)
    • Better handling of FITS-IDI data bit sampling (#836)

    Images

    • Support for non-chunked HDF5 data sets (#879, #880)
    • Fixed count overflow on very large images (#849)
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Nov 15, 2018)

    3.0.0

    General

    • Building with C++11 is now required
    • Improved thread-safety of statics (#775)
    • Improve statistics code (#785, #776, #662, #663)

    Tables

    • Avoid growing the size of tables when overwriting data (#768)
    • TaQL support for radial velocity, earth magnetic field and Doppler (#750)
    • Added performance tests for tables (#658)

    Measurement Set

    • Improvements to MSConcat (#779, #701)
    • Better handling of FITS-IDI flag history (#748)

    Images

    • Better support for double precision images (#752, #751)

    Other

    Source code(tar.gz)
    Source code(zip)
  • v2.4.1(Dec 13, 2017)

    2.4.1

    General

    • This release fixes a problem with TableIterator introduced in v2.4.0. This issue would cause problems with LOFAR's DPPP when msin.starttime was set. (#681)
    Source code(tar.gz)
    Source code(zip)
  • v2.4.0(Nov 1, 2017)

    2.4.0

    General

    • Bug fixes and improvements
    • This version can be used to compile CASA 5.0
    • Improved error checking when parsing dates/times (#619)
    • Statistics: allow data provider to specify number of threads (#645)

    MS

    • Disable caching of MS main table columns (#597)
    • Properly copy the SIGMA_SPECTRUM column, if present (#599)
    • Implement getTimesForSpws (#600)

    FITS and FITS-IDI

    • Implement digital corrections for DiFX/VLBA (#602)
    • Several improvements for WEIGHTS when reading FITS-IDI (#590, #608)
    • Parallactic angle calculation for Nasmyth mounts (#627)
    • Modernised matrix syntax in FITS files now follows standard 3.0 (#606)
    Source code(tar.gz)
    Source code(zip)
  • v2.3.0(May 17, 2017)

    2.3.0

    General

    • Bug fixes and improvements
    • Improved installation documentation in README
    • Add an option to use Ccache (-DUseCcache) (#549)
    • Add some statistics functionality (#569)

    Python

    • Make some of the TableProxy functionality publicly available (#559)
    • Make version checking from python (or plain C) possible (#583)

    TaQL

    • Fix transpose in TaQL (#563)
    • Added functions delay and uvwapp to mscal (#562)

    FITS

    • MSFitsOutput now writes ant diams (#536)
    • Improvements to FITS-IDI conversion (#538, #590, #579)
    Source code(tar.gz)
    Source code(zip)
  • v2.2.0(Dec 1, 2016)

    2.2.0

    General

    • Lots of bug fixes and improvements
    • Tests are not built by default anymore, use -DBUILD_TESTING=True to build them
    • Building with C++11 is now default, use -DCXX11=False if you do not have a recent compiler (#533)
    • Added JSON support (#506)

    TaQL

    • Major improvements in TaQL, such as masked arrays, new commands ALTER TABLE, SHOW TABLE, HELP (#388)

    Images

    • ImageConcat and ImageExpr now use JSON export (#517)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Mar 18, 2016)

    2.1.0

    General

    • Lots of code improvement, optimization and added tests

    Measures

    • The default search path for measures data is now smaller. The path can be set at cmake time by specifying -DDATA_DIR=/path/to/data. This path can contain %CASAROOT% which is expanded at run time, to support relocatable installations. The measures path can also still be set in .casarc. (#277)
    • Inserted fix to leap second handling problem (#290)
    • Various coordinate performance improvements (#258)

    Tables

    • Renamed showtable to showtableinfo (#320)
    • Added several TaQL functions (#229)

    MS

    • Improve testing of MSSummary (#330)
    • Many improvements for the MS related functions (#318, #291, #228, #208, ...)
    • Fix multithreaded MS creation (#298)

    Python

    • Build python bindings by default (not for CASA build)
    • Add experimental Python3 support. Read the README for instructions (#280)
    • Changed behavior from 2.0.3: Use PYTHON2_* configuration options to set your python2 interpreter. (#280)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.3(Jul 24, 2015)

    2.0.3

    NOTE: this release is not ABI compatible with release 2.0.1

    General

    • Merge of CASA work into casacore, CASA should compile with casacore now
    • Under-the-hood optimizations, like using allocator features (#132)
    • Building with gcc 5 is now possible (#166)
    • Compatibility with more versions of wcslib and cfitsio
    • SOFA is now an external dependency (#105)

    Tables

    • Arrays can now sometimes be reshaped while keeping allocated memory (#113)

    MS

    • Selection of baselines with a regexp between stations is now possible (#99)
    • Several new methods to MSMetaData (#138)

    Bug fixes

    • Fix a bug which caused an error with LOFAR measurement sets with the LOFAR tool msoverview (#140)
    • Fix a bug where TaQL would not write output when an expression was used (#184)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Apr 10, 2015)

  • v2.0.0(Apr 9, 2015)

    2.0.0

    Changes since 1.7.0

    General

    • Fixed all build problems for GCC and clang on various platforms (up to OS-X 10.10)
    • Fixed all valgrind issues found in test programs
    • Made the statics in all of casacore thread-safe. Similar to libstdc++ the user is responsible for thread-safety when using casacore ojbects.
    • Added casacore/ to the #include path of all header files
    • Changed namespace from casa to casacore (but still #defined as casa for time being)
    • Split Tables, MeasurementSets and Lattices into smaller parts
    • The 3 changes above give rise to backward incompatibilities. Scripts in casacore/changescripts (notably updateall) can be used to fix client code.

    Tables

    • Added GROUPBY/HAVING/aggregation function to TaQL
    • Optimized TaQL's IN operator (linear time for integer sets)
    • Added table tracing (class TableTrace)
    • Fixed TiledStMan issue where BucketCache was not shrinked
    • Added MultiFile option to combine table files in a single one

    Components

    • This module has been removed (too CASA specific)

    Images

    • Added support for beam per frequency channel and Stokes
    • Added persistency for ImageExpr and ImageCocat objects

    Python

    • Moved converters from pyrap to casacore/Python
    Source code(tar.gz)
    Source code(zip)
POCO C++ Libraries are powerful cross-platform C++ libraries for building network

The POCO C++ Libraries are powerful cross-platform C++ libraries for building network- and internet-based applications that run on desktop, server, mobile, IoT, and embedded systems.

POCO C++ Libraries 6.2k Aug 2, 2022
Mod - MASTERS of DATA, a course about videogames data processing and optimization

MASTERS of DATA Welcome to MASTERS of DATA. A course oriented to Technical Designers, Technical Artists and any game developer that wants to understan

Ray 32 Apr 5, 2022
Minipops alternative firmware for Music Thing Modular Radio Music

RMoxy Minipops drummer alternative firmware for Music Thing Modular Radio Music module The repository for the Radio Music module by Tom Whitwell Kits

Johan Berglund 6 Apr 6, 2022
The Express LRS Handset project is first and foremost an experimental test bed for pushing the envelope with what can be done with the ELRS radio link

The Express LRS Handset project is first and foremost an experimental test bed for pushing the envelope with what can be done with the ELRS radio link

ExpressLRS 21 May 23, 2022
FM radio tuner application for Raspberry Pi OS

GTK FM Tuner GTK FM Tuner is a lightweight, GUI base FM radio tuner application for Raspberry Pi OS. This application uses GTK to provide the UI of th

Dilshan R Jayakody 14 Apr 25, 2022
QN8035 based Raspberry Pi FM radio receiver

QN8035 based Raspberry Pi FM radio receiver. QN8035 is a single-chip stereo FM radio receiver IC designed by the Quintic Corporation. This receiver su

Dilshan R Jayakody 7 Apr 17, 2022
New version of the well known ESP32 Radio. Now optional I2S output!

ESP32Radio-V2 New version of the well known ESP32 Radio. Now optional I2S output! Compile time configuration in config.h. Do not forget to upload the

Ed Smallenburg 42 Jul 19, 2022
An ATSC 3.0 Transmitter for GNU Radio

# Copyright 2021 Ron Economos # # This file is part of gr-atsc3 # # gr-atsc3 is free software; you can redistribute it and/or modify # it under the te

Ron Economos 11 Mar 16, 2022
Decoded Information from Radio Emissions for Windows Or Linux Fans

Dire Wolf is a software "soundcard" AX.25 packet modem/TNC and APRS encoder/decoder. It can be used stand-alone to observe APRS traffic, as a tracker, digipeater, APRStt gateway, or Internet Gateway (IGate).

null 1.1k Aug 5, 2022
AX.25 protocol (packet radio and APRS) interface / bridging / switching / terminal related software for Linux

CB3ROB-AX25-TOOLS-LINUX 2021-11-23T00:58:49Z (TUESDAY) cb3rob-kiss-tcp-attach.c links ax0 interface to KISS-TCP TNC, soundmodem (direwolf), or multipl

CB3ROB TACTICAL SYSTEMS 1 Jan 1, 2022
A set of tutorial projects for creating a simple digital radio receiver based on the STM32G431KB microcontroller

simple-radio Обучающие проекты по созданию простого цифрового радиоприемника на базе микроконтроллера STM32G431KB. Разработка программ выполнялась в W

null 8 Jul 7, 2022
2.4 GHz LORA based telemetry and radio link for remote controlled vehicles

mLRS This is the mLRS project. The goal is an open source 2.4 GHz LORA-based high-performance long-range radio link, which provides fully transparent

null 44 Aug 8, 2022
the ares multi-system console emulation suite

ares ares is a multi-system emulator that began development on October 14th, 2004. It is a descendent of higan and bsnes, and focuses on accuracy and

higan 315 Aug 3, 2022
Test suite for evaluating C code on 42 Basecamp

Joãonette Test suite based on Moulinette, made by "Iniciativa João" ⛺️ Setup Clone this repo ( I suggest cloning to the root and calling it joaonette

Lucas Kuhn 12 Apr 19, 2022
Violent Fungus is a command and control (C2) software suite, providing red teams post-exploitation persistence and other juicy stuff.

Violent Fungus is a command and control (C2) software suite, providing red teams post-exploitation persistence and other juicy stuff.

Chris Humphries 33 Jul 3, 2022
the ares multi-system console emulation suite

ares is a multi-system emulator that began development on October 14th, 2004. It is a descendent of higan and bsnes, and focuses on accuracy and prese

ares 315 Jul 30, 2022
A combined suite of utilities for exporting images to retro formats.

ImageTools A combined suite of utilities for exporting images to retro formats. It was developed for use on Windows but might compile on other systems

David Walters 4 Dec 6, 2021
AWS Ambit Scenario Designer for Unreal Engine 4 (Ambit) is a suite of tools to streamline content creation at scale for autonomous vehicle and robotics simulation applications.

AWS Ambit Scenario Designer for Unreal Engine 4 Welcome to AWS Ambit Scenario Designer for Unreal Engine 4 (Ambit), a suite of tools to streamline 3D

AWS Samples 47 Jul 24, 2022
Blender is the free and open source 3D creation suite

Blender is the free and open source 3D creation suite. It supports the entirety of the 3D pipeline-modeling, rigging, animation, simulation, rendering, compositing, motion tracking and video editing.

Blender 6.2k Aug 1, 2022