Suite of C++ libraries for radio astronomy data processing

Overview

Casacore

A suite of C++ libraries for radio astronomy data processing.

Installation

Debian / Ubuntu

Casacore is now part of Debian and Ubuntu, use apt to search for the package you need:

$ apt-get update
$ apt-get search casacore

MacOS

Use Homebrew with the ska-sa tap to install casacore:

$ brew tap ska-sa/tap
$ brew install casacore

Docker

We now also publish docker images for each github branch on quay.io:

$ docker pull quay.io/casacore/casacore:master

Building from source

Getting the source code

The casacore source code is maintained on github.

You can obtain it using:

$ git clone https://github.com/casacore/casacore

Requirements

To compile casacore you need to meet the following requirements:

  • cmake
  • gfortran
  • g++
  • flex
  • bison
  • blas
  • lapack
  • cfitsio (3.181 or later)
  • wcslib (4.20 or later)
  • sofa (optional, only for testing casacore measures)
  • fftw3
  • hdf5 (optional)
  • numpy (optional)
  • boost-python (optional)
  • ncurses (optional)

On Debian / Ubuntu you can install these with:

$ sudo apt-get install build-essential cmake gfortran g++ libncurses5-dev \
   libreadline-dev flex bison libblas-dev liblapacke-dev libcfitsio-dev \
   wcslib-dev libfftw3-dev

and the optional libraries:

$ sudo apt-get install libhdf5-serial-dev python-numpy \
    libboost-python-dev libpython3.4-dev libpython2.7-dev

On CentOS7 you can install these with:

$ sudo yum install cmake cmake-gui gcc-gfortran gcc-c++ flex bison \
       blas blas-devel  lapack lapack-devel cfitsio cfitsio-devel \
       wcslib wcslib-devel ncurses ncurses-devel readline readline-devel\
       python-devel boost boost-devel fftw fftw-devel hdf5 hdf5-devel\
       numpy boost-python 

Obtaining measures data

Various parts of casacore require measures data, which requires regular updating. You can obtain the WSRT measures archive from the ASTRON FTP server:

ftp://ftp.astron.nl/outgoing/Measures/

Extract this somewhere on a permanent location on your filesystem.

Compilation

In the casacore source folder run:

mkdir build
cd build
cmake ..
make 
make install

there are various flags available to cmake to enable and disable options:

$ cmake -DDATA_DIR=/usr/share/casacore/data -DUSE_OPENMP=ON \
    -DUSE_HDF5=ON -DBUILD_PYTHON=ON -DUSE_THREADS=ON

The DATA_DIR should point to the location where you extracted the measures data. Special variables %CASAROOT% and %CASAHOME% can be used here, which can be set at run time through the .casarc file.

You can build python3 support using -DBUILD_PYTHON3=ON. Note that CMake may have problems detecting the correct python3 libraries and headers, so probably you need to set them manually. For example:

-DPYTHON3_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so
-DPYTHON3_INCLUDE_DIR=/usr/include/python3.4

To configure Python2 specific settings use:

PYTHON2_EXECUTABLE
PYTHON2_LIBRARY
PYTHON2_INCLUDE_DIR

To configure Python3 specific settings use:

PYTHON3_EXECUTABLE
PYTHON3_LIBRARY
PYTHON3_INCLUDE_DIR

If you run into problems with boost libraries, try setting -DBoost_NO_BOOST_CMAKE=True. This will be necessary if you have the libraries from NRAO casa in your PATH or LD_LIBRARY_PATH.

Documentation

http://casacore.github.io/casacore

Problems & bugs

If you have any issues compiling or using casacore, please open an issue on the issue tracker on github.

If you have patches please open a pull request. Your contributions are more than welcome! But to maintain a high code quality we have written a contribution manual, please read that first.

Comments
  • Replace tcsh script measuresdata.csh with /bin/sh script

    Replace tcsh script measuresdata.csh with /bin/sh script

    The new script is modelled almost after the tcsh script, but uses curl to get the files from remote.

    This pull request removes tcsh from the dependencies of casacore.

    opened by olebole 37
  • Change of behaviour for Regex::match()

    Change of behaviour for Regex::match()

    Pull request #881 (authored by @gervandiepen and committed by @tammojan) introduced an (unintended?) change in behaviour in Regex::match(). The documentation says:

    Test if the regular expression matches (part of) string s.

    The return value gives the length of the matching string part, or String::npos if there is no match or an error. The string has len characters and the test starts at position pos. The string may contain null characters. Negative p is allowed to match at end.

    However, since the std::regex_match() algorithm that is now used to implement this functionality only matches entire strings, so partial matches are no longer found. The correspond regex matching functions in the String class also get this changed behaviour.

    This breaks (for example) the CASA SingleDishSkyCal class which tries to match observing modes using a regex like:

    "^OBSERVE_TARGET#ON_SOURCE"

    which is supposed to match all observing modes that start with

    "OBSERVE_TARGET#ON_SOURCE"

    this can be fixed of course by tweaking the regex to:

    "^OBSERVE_TARGET#ON_SOURCE.*"

    but I'm not sure it is reasonable to expect applications to make such changes as it may e difficult to find all relevant cases in the code and in some cases the regex may come from user input that isn't controlled by the programmer.

    At the very least the documentation should be adjusted if the change of behaviour is intentional.

    opened by kettenis 28
  • Release casacore 2.0.2?

    Release casacore 2.0.2?

    I noticed in the last week's release 4.4 of NRAO Casa that its CMakeLists.txt contains the line

    casa_find( CASACORE
               VERSION 2.0.2
    

    ( see https://svn.cv.nrao.edu/view/casa/branches/release-4_4/code/CMakeLists.txt?revision=33147 )

    It would be nice if this release could actually exist and be compatible with Casa 4.4.

    This may need some ad-hoc fix for #113 since the requested functionality was already implemented on a fork of casacore ( https://svn.cv.nrao.edu/view/casa/branches/release-4_4/casacore/ ).

    opened by tammojan 27
  • Question about fields on different frames (FK5, ICRS)

    Question about fields on different frames (FK5, ICRS)

    Hello,

    I have just noticed that some datasets might have fields with different frames. One field could be in FK5 and other one could be in ICRS. Is there any taql query of casacore code that allows to retrieve all fields with the same frame using casacore?

    Cheers, Miguel

    opened by miguelcarcamov 26
  • CAS-8783 Parallel stats

    CAS-8783 Parallel stats

    • Refactor, create ClassicalStatistics::_increment() method for updating loop counters
    • Implement StatisticsUtilities::combine() for combining statistics derived from different data sets to get a set of statistics that applies to all those data sets.
    • implementation of ParallelClassicalStatistics for parallel support in _getStatistics(). There are other avenues to investigate, but this implementation does improve performance on computing stats on large images by a factor of 1.5 to 3 depending on the use case.
    • add support for parallel computation of stats to LatticeStatistics (default behavior is unchanged; objects must be explicitly configured to use parallel computation).
    • reformat files, changing tabs to spaces.
    opened by dmehring 26
  • Install issues on Mac OS X

    Install issues on Mac OS X

    I'm trying to compile casacore on mac OS X by linking against conda-installed libraries. This may not be entirely sane, but I wanted to post my efforts here and ask if there's anything critical I've missed.

    My cmake command involved a lot of flags:

    PATH=~/anaconda/envs/astropy35/bin/:~/anaconda/envs/astropy27/bin/:$PATH cmake .. -DUSE_FFTW3=ON -DBUILD_PYTHON3=on -DUSE_HDF5=ON -DBUILD_PYTHON=ON -DUSE_THREADS=ON -DUSE_OPENMP=ON -DPYTHON3_LIBRARY=/Users/adam/anaconda/envs/astropy35/lib/python3.5/config-3.5m/libpython3.5m.a -DPYTHON3_INCLUDE_DIR=/Users/adam/anaconda/envs/astropy35/include/python3.5m/ -DBOOST_ROOT=/Users/adam/anaconda/envs/astropy35/
    

    and looked mostly successful up to the end, when it failed. The error started around this line.

    My cmake command revealed that I'm missing SOFA libraries and CASA data, but I don't think either of these things should affect compilation as above.

    Any help would be appreciated.

    opened by keflavich 22
  • Enhancing parallel IO in Casacore Table Data System

    Enhancing parallel IO in Casacore Table Data System

    I don't know if this is the best place but I was looking to some discussions on enhancing parallel IO for the table system as well as Measurementsets.

    A few years ago I wrote a parallel storage manager for the casacore table system based on ADIOS (https://github.com/SKA-ScienceDataProcessor/AdiosStMan.git) when I was at ICRAR, which was put into the SKA-SDP as an early investigation into SKA scale measurementset experiments. When @gervandiepen was visiting us at ICRAR we talked about merging this AdiosStMan into the master but I was worried about the code quality because it was only a prototype rather than a production software. @gervandiepen was also worried about introducing MPI dependency to casacore. So eventually we dropped the plan.

    Recently I have been working on updating the Adios Storage Manager using our new software ADIOS 2.0 (https://github.com/ornladios/ADIOS2) that we are developing at Oak Ridge National Lab under the US Department of Energy's Exa-scale Computing program. The motivation behind this is that we are planning on running a full-scale SKA 1 data pipeline simulation, which may require writing data into a single measurementset table from multiple processes. If this is successful then we can re-think about improving the code quality and merge our efforts into the master.

    However, the current parallel storage manager design is more like a hack than a generic solution because we couldn't find parallel support in casacore's table layer API. As a result, we need to hack user applications and make them believe that each MPI rank is writing to its own independent table, while the Adios Storage Manger does dirty tricks underneath and actually writes data into a single column file. This has worked so far but it makes user applications very ugly and less maintainable.

    I have roughly read through the table layer code trying to think of a solution. If we can add some MPI-aware code in the procedures of creating a table, making only Rank 0 actually write table metadata file, while other ranks hold the table metadata in memory without writing to files, this would make the table system much more friendly to parallel storage managers as well as MPI incorporated applications.

    I am wondering if this is possible or if there is any other suggestions or plans on enhancing parallel IO for casacore. Looking forward to any discussions or suggestions. Thank you.

    enhancement 
    opened by JasonRuonanWang 21
  • Discussion: Adding C++11 features

    Discussion: Adding C++11 features

    We have to start thinking about introducing more C++11 features in Casacore. I can think of a few:

    1. 'Manually' instantiate some often used templates like Array, ScalarColumn, etc. C++11 has a new feature to do so. It makes compilation much faster, because the template does not need to be instantiated everywhere it is used. It should also make the libraries (much) smaller.
    2. Add Array&& arguments to ArrayMath. In this way less temporaries have to be created when doing e.g. a_b+c because c can be added to the temporary result of a_b. The cost is that every operator needs 4 functions instead of 1, but they are very simple. However, I'm not entirely sure if it is worth the trouble.
    3. Use std::regex in the Regex class; it can only be done if fully compatible.

    I have not thought any further, but I'm sure there are more things that could be done. Maybe a student could do these things.

    A very important issue is whether we think all sites using casacore will have compilers supporting C++11. I don't think that is the case, so for the time being all changes need to be #ifdef-ed.

    I'd like to hear your thoughts. Cheers, Ger

    opened by gervandiepen 20
  • Use FFTW instead of FFTPack in Fits-IDI to MS table converter

    Use FFTW instead of FFTPack in Fits-IDI to MS table converter

    The FitsIDItoMS class is the only class that was still dependent on the FFTPack Fortran code. By rewriting it to use FFTW, the FFTPack code can be removed. This required implementing a wrapper for the real even FFT in the FFTW class, as that was not available.

    The FFTW class should be rewritten to have a more sensible interface. I've added some TODO comments for that. Despite the use of a mutex, the FFTW wrapper is also not thread safe.

    This is related to the work in #1024 and #1017.

    opened by aroffringa 19
  • Cache invalidation deadlock with user locks

    Cache invalidation deadlock with user locks

    While debugging a deadlock in dask-ms I noticed that when a running all calls to python-casacore in a single threaded executor pool the library deadlocks with no exception when putcol is called without setting a writable user lock. It always seems to hang on a ISMColumn operation. The bug likely affects only this storage manager:

    #0  0x00007f12c8d3c6d1 in casacore::ISMColumn::getScalarColumnCellsdoubleV (this=0x7f12c006f1c0, rownrs=..., values=0x7f12c4a77670) at /scratch/bhugo/oms-subset/casacore/tables/DataMan/ISMColumn.cc:464
    #1  0x00007f12c8d86ad8 in casacore::StManColumn::getScalarColumnCellsV (this=0x7f12c006f1c0, rownrs=..., dataPtr=0x7f12c4a77670) at /scratch/bhugo/oms-subset/casacore/tables/DataMan/StManColumn.cc:227
    #2  0x00007f12c8c7bbf2 in casacore::ScalarColumnData<double>::getScalarColumnCells (this=0x7f12c0066960, rownrs=..., val=0x7f12c4a77670)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScaColData.tcc:151
    #3  0x00007f12c8c99375 in casacore::ScalarColumn<double>::getColumnCells (this=0x7f12c4a77450, rownrs=..., vec=..., resize=false)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:205
    #4  0x00007f12c8c990d4 in casacore::ScalarColumn<double>::getColumnRange (this=0x7f12c4a77450, rowRange=..., vec=..., resize=false)
        at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:180
    #5  0x00007f12c8c9921b in casacore::ScalarColumn<double>::getColumnRange (this=0x7f12c4a77450, rowRange=...) at /scratch/bhugo/oms-subset/casacore/casacore/tables/Tables/ScalarColumn.tcc:165
    #6  0x00007f12c8cd343c in casacore::TableProxy::getValueFromTable (this=0x7f12c0001b30, colName=..., rownr=100, nrow=100, incr=1, isCell=false)
        at /scratch/bhugo/oms-subset/casacore/tables/Tables/TableProxy.cc:2245
    #7  0x00007f12c8cca084 in casacore::TableProxy::getColumn (this=0x7f12c0001b30, columnName=..., row=100, nrow=100, incr=1) at /scratch/bhugo/oms-subset/casacore/tables/Tables/TableProxy.cc:1149
    #8  0x00007f12ca5f1970 in boost::python::detail::invoke<boost::python::to_python_value<casacore::ValueHolder const&>, casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::arg_from_python<casacore::TableProxy&>, boost::python::arg_from_python<casacore::String const&>, boost::python::arg_from_python<int>, boost::python::arg_from_python<int>, boost::python::arg_from_python<int> > (ac3=..., ac2=..., ac1=..., ac0=..., tc=<synthetic pointer>, f=
        @0x21865c8: (casacore::ValueHolder (casacore::TableProxy::*)(casacore::TableProxy * const, const casacore::String &, int, int, int)) 0x7f12c8cc9fdc <casacore::TableProxy::getColumn(casacore::String const&, int, int, int)>, rc=...) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/invoke.hpp:86
    #9  boost::python::detail::caller_arity<5u>::impl<casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::default_call_policies, boost::mpl::vector6<casacore::ValueHolder, casacore::TableProxy&, casacore::String const&, int, int, int> >::operator() (args_=<optimized out>, this=0x21865c8) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/caller.hpp:238
    #10 boost::python::objects::caller_py_function_impl<boost::python::detail::caller<casacore::ValueHolder (casacore::TableProxy::*)(casacore::String const&, int, int, int), boost::python::default_call_policies, boost::mpl::vector6<casacore::ValueHolder, casacore::TableProxy&, casacore::String const&, int, int, int> > >::operator() (this=0x21865c0, args=<optimized out>, kw=<optimized out>)
        at /scratch/bhugo/oms-subset/opt/include/boost/python/object/py_function.hpp:38
    #11 0x00007f12c853b00d in boost::python::objects::function::call(_object*, _object*) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #12 0x00007f12c853b208 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #13 0x00007f12c8543263 in boost::python::detail::exception_handler::operator()(boost::function0<void> const&) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #14 0x00007f12c82f3b6a in boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>::operator() (this=0x2190e08, handler=..., f=..., 
        translate=0x7f12c82f2fbd <casacore::python::translate_iterexcp(casacore::IterError const&)>) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/translate_exception.hpp:46
    #15 0x00007f12c82f38b5 in boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> >::operator()<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=0x2190e10, f=..., a=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:388
    #16 0x00007f12c82f36c1 in boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=0x2190e08, a1=..., a2=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:1318
    #17 0x00007f12c82f352d in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (
        function_obj_ptr=..., a0=..., a1=...) at /scratch/bhugo/oms-subset/opt/include/boost/function/function_template.hpp:137
    #18 0x00007f12c8543238 in boost::python::detail::exception_handler::operator()(boost::function0<void> const&) const () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #19 0x00007f12c82f3b6a in boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>::operator() (this=0x2164e08, handler=..., f=..., 
        translate=0x7f12c82f2fbd <casacore::python::translate_iterexcp(casacore::IterError const&)>) at /scratch/bhugo/oms-subset/opt/include/boost/python/detail/translate_exception.hpp:46
    #20 0x00007f12c82f38b5 in boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> >::operator()<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=0x2164e10, f=..., a=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:388
    #21 0x00007f12c82f36c1 in boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=0x2164e08, a1=..., a2=...)
        at /scratch/bhugo/oms-subset/opt/include/boost/bind/bind.hpp:1318
    #22 0x00007f12c82f352d in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<casacore::IterError, void (*)(casacore::IterError const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(casacore::IterError const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (
        function_obj_ptr=..., a0=..., a1=...) at /scratch/bhugo/oms-subset/opt/include/boost/function/function_template.hpp:137
    #23 0x00007f12c854301d in boost::python::handle_exception_impl(boost::function0<void>) () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #24 0x00007f12c8538409 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
    #25 0x00000000005c3bd7 in PyObject_Call () at ../Objects/abstract.c:2165
    #26 0x00000000005354af in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7f12c4a77ed0, func=<optimized out>) at ../Python/ceval.c:4936
    

    The minimum steps to reproduce the issue are the following. Note the deliberate bug induced in line 30. I'm very careful not to submit calls (even open and close calls) in anything other than the single executor thread:

      1 from __future__ import print_function
      2 
      3 import argparse
      4 import numpy as np
      5 import pyrap.tables as pt
      6 from concurrent.futures import ThreadPoolExecutor
      7 import threading
      8 def create_parser():
      9     p = argparse.ArgumentParser()
     10     p.add_argument("ms")
     11     p.add_argument("-rc", "--row-chunks", default=100, type=int)
     12     p.add_argument("-nc", "--num-consumers", default=64, type=int)
     13     return p
     14 
     15 args = create_parser().parse_args()
     16 
     17 futs = []
     18 
     19 consumer = ThreadPoolExecutor(max_workers=args.num_consumers)
     20 producer = ThreadPoolExecutor(max_workers=1) # ensure we don't interact with CASAcore in more than one thread
     21 
     22 # get future to the table on thread 1 of the data producer queue
     23 tbl_hnd = producer.submit(pt.table, args.ms, readonly=False, lockoptions="user")
     24 
     25 def _proxy_table_method(handle, method, write, *args, **kwargs):
     26     print("Running CC method '{}' on thread {}".format(method, threading.get_ident()))
     27 
     28     table = handle.result()
     29 
     30     table.lock(write=not write)
     31 
     32     try:
     33         result = getattr(table, method)(*args, **kwargs)
     34 
     35         if write:
     36             table.flush()
     37     finally:
     38         if method != "close":
     39             table.unlock()
     40 
     41     return result
     42 
     43 nrows = producer.submit(_proxy_table_method, tbl_hnd, "nrows", False).result()
     44 print("{} rows in this MS".format(nrows))
     45 
     46 
     47 def _dummy_rw(colname, r, e):
     48     """ Dummy method to read and write using user locks """
     49     def _rw_batch(colname, r, e):
     50         print("Submitting read write batch for rows {} through {}".format(r, e))
     51         nddat = _proxy_table_method(tbl_hnd, "getcol", False, colname, startrow=r, nrow=e-r)
     52         # whatever ops to be done here
     53         _proxy_table_method(tbl_hnd, "putcol", True, colname, nddat, startrow=r, nrow=e-r)
     54         return nddat
     55     datshp = producer.submit(_rw_batch, colname, r, e).result().shape
     56     return r, e-r, datshp
     57 
     58 # Chunk up the data nice and fine so there are a lot of calls to CC
     59 for r in range(0, nrows, args.row_chunks):
     60     e = min(r + args.row_chunks, nrows)
     61     futs.append(consumer.submit(_dummy_rw,"TIME", r, e))
     62     futs.append(consumer.submit(_dummy_rw,"FLAG", r, e))
     63 
     64 # ensure all rw ops finish
     65 for f in futs:
     66     f.result()
     67 
     68 print(">>>>>>>>>>>>FINISHED<<<<<<<<<<<")
     69 for f in futs:
     70     sr, nr, res = f.result()
     71     print(sr, nr, res)
     72 
     73 import atexit
     74 atexit.register(lambda: producer.submit(_proxy_table_method, tbl_hnd, "close", False).result())
    

    with output as follows:

    > $ python3 test.py msdir/subset-1563189318-20M.ms
    Successful read/write open of user-locked table msdir/subset-1563189318-20M.ms: 24 columns, 6368888 rows
    Running CC method 'nrows' on thread 139718585456384
    6368888 rows in this MS
    Submitting read write batch for rows 0 through 100
    Running CC method 'getcol' on thread 139718585456384
    Running CC method 'putcol' on thread 139718585456384
    Submitting read write batch for rows 0 through 100
    Running CC method 'getcol' on thread 139718585456384
    Running CC method 'putcol' on thread 139718585456384
    Submitting read write batch for rows 100 through 200
    Running CC method 'getcol' on thread 139718585456384
    

    Note that I get the expected user exception when executing in the parent thread:

    In [5]: t = tbl("msdir/subset-1563189318-20M.ms", readonly=False, lockoptions="user")
    Successful read/write open of user-locked table msdir/subset-1563189318-20M.ms: 24 columns, 6368888 rows
    
    In [6]: t.lock(write=True)
    
    In [7]: time = t.getcol("TIME")
    
    In [8]: t.unlock()
    
    In [9]: t.lock(write=False)
    
    In [10]: t.putcol("TIME", time)
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-10-d8534bd8f3fa> in <module>()
    ----> 1 t.putcol("TIME", time)
    
    /scratch/bhugo/oms-subset/opt/home/hugo/.local/lib/python3.5/site-packages/casacore/tables/table.py in putcol(self, columnname, value, startrow, nrow, rowincr)
       1152
       1153         """
    -> 1154         self._putcol(columnname, startrow, nrow, rowincr, value)
       1155
       1156     def putvarcol(self, columnname, value, startrow=0, nrow=-1, rowincr=1):
    
    RuntimeError: ColumnSet::doLock: table /scratch/bhugo/oms-subset/msdir/subset-1563189318-20M.ms should be locked when using UserLocking
    

    I've compiled casacore master (#2f45a1a7fa8474ffc) from source with debug symbols and installed the source distribution of python-casacore==3.1.1. We are running python 3.5.

    opened by bennahugo 19
  • CAS-8447 enable parallel processing using omp

    CAS-8447 enable parallel processing using omp

    I had made some methods protected in a previous commit that was merged, but now these can be re-made private since I was able to provide a multi-threaded implementation using openmp in MSMetaData, obviating the need for a derived class.

    opened by dmehring 19
  • MSLister: prevent undefined behavior from 'log10(

    MSLister: prevent undefined behavior from 'log10("negative value") casted to int' that produces crashes in the CASA task listvis with newer GCCs

    The cast of the result from rint() to (Int) in this line: https://github.com/casacore/casacore/blob/990a7e0e4108d1b39534aa918f675fff27baf4d3/ms/MSOper/MSLister.cc#L909 produces an undefined behavior that with more modern GCCs and the optimization flag -ftree-vrp enabled (it is included in -O2) makes the MSLister crash in a CASA task test.

    Besides the potential crash that happens when (max(phase)+0.5 < 0), dependent on how the undefined cast behaves in practice, the code is miscalculating that "order" of the phase array, as it assumes that the largest values in (absolute value) are positive.

    This patch prevents that log10("negative value") by adding an abs() around the max(), and also uses the same function in all the lines that make a similar calculation for other MS columns / arrays derived from MS columns.


    Explanation of the crash:

    When working on an updated build system for CASA we noticed crashes in one of the tests of the CASA task 'listvis', listvis_test1.test2. So far we hadn't seen crashes for this in official CASA builds, but with more modern GCCs (jumping from 5.3 to 8.5 or later) and having -ftree-vrp optimizaton flag enabled, the crash happens as follows, in this line: https://github.com/casacore/casacore/blob/990a7e0e4108d1b39534aa918f675fff27baf4d3/ms/MSOper/MSLister.cc#L909 for the selection of phase values in that test, max(phase) = -89.3596, which produces a "nan" log10 result that then propagates to the max(). Then the undefined behavior (+ -ftree-vrp = "enable Value Range Propagation on trees") turns into the following: in the version of max being used for the max(1, (Int)...), from casa/BasicMath/Math.h https://github.com/casacore/casacore/blob/990a7e0e4108d1b39534aa918f675fff27baf4d3/casa/BasicMath/Math.h#L158 That "a > b" comparison (1 > "nan casted to int = ~2147483648"), when inlined yields false, then max returns "b", which is then casted to uInt 2147483648. This finally produces an out-of-bound access segfault further down in MSLister::listData((), here: https://github.com/casacore/casacore/blob/990a7e0e4108d1b39534aa918f675fff27baf4d3/ms/MSOper/MSLister.cc#L980-L981 (where wVis_p includes oPhase_p).

    With older GCCs or with optimization level < O2 (or using -fno-tree-vrp to disable this particular optimization option) the (a>b) comparison is true and that max returns 1, which is a wrong result (as estimate of the number of digits needed to format the output) but doesn't lead to a crash. The latter behavior is what we had so far with older GCCs.

    To circumvent this issue temporarily, CASA compiles casacore with -fno-tree-vrp. Once the patch included here is applied to MSLister, the -fno-tree-vrp can be removed and all tests of task listvis pass without issues. This funny interaction between the undefined (Int) cast and -ftree-vrp can be reproduced with the attached small example: min_max_nan.cc.gz

    opened by FedeMPouzols 0
  • First stage of join clause in TaQL

    First stage of join clause in TaQL

    This is the first part of the changes for support of the JOIN clause in TaQL. This part contains the changes in the scanner/parser and the code to handle the result from the parser and store it in a tree. Handling the result tree is done in a separate PR.

    The syntax of the join looks like:

    select t1.col1, t2.col2, ... from table table1 t1 JOIN table2 t2 ON condition JOIN table3 t3 ON condition ... WHERE ...

    Theta joins (implicit joins in the SELECT clause) are not supported. They are discouraged in SQL.

    opened by gervandiepen 0
  • Generalize pow(array,scalar)

    Generalize pow(array,scalar)

    In ArrayMath the pow function is defined for array,array and scalar,array for any T. However, pos(array,scalar) is only defined for (T,double). Generalize that to (T,T). It also makes ExprFuncNodeArray more clean.

    opened by gervandiepen 0
  • Casacore on mac os monterey

    Casacore on mac os monterey

    I want to install casacore on mac os monterey and i have this error

    Failed to build python-casacore Installing collected packages: python-casacore Running setup.py install for python-casacore ... error error: subprocess-exited-with-error

    × Running setup.py install for python-casacore did not run successfully. │ exit code: 1 ╰─> [77 lines of output]

    Can you help to fix this error ?

    opened by abdallahhamini 0
  • Add pre-compiled header support

    Add pre-compiled header support

    Using pre-compiled headers speeds up the build depending on what goes into the headers, the build type, the compiler itself, etc. For some numbers see https://github.com/casacore/casacore/pull/1236/commits/809c898b4d80d489f3be4857704f9e413f448cc9.

    The PCHs created here consist on the top-level headers for each of the modules, and apply both to the sources going into the shared libraries and to app and test executables, which covers already a good starting base. In the casa library in particular this caused a couple of issues due to function overload problems, which are fixed in this PR. This exercise also found some duplicated code that I removed.

    More work can probably be done to speed things up further, but let's see how this plays out in CI first.

    opened by rtobar 2
Releases(v3.5.0)
  • v3.5.0(Jul 5, 2022)

    3.5.0

    Highlights

    Apart from many bug fixes and modernizations, a few new features were introduced:

    • TaQL now supports copying columns to new columns (e.g. to make a backup of the FLAG column)
    • Non-zero coordinates in JPL frames, e.g. the SUN frame, are now interpreted as an offset in right ascension and declination. Previously, coordinates in a JPL coordinate frame were ignored.
    • The Dysco storage manager is now built by default as part of casacore.

    General

    • Improve continuous integration (#1180, #1182), move from Travis CI to Github Actions (#1086, #1097, #1098, #1161)
    • Modernizations: remove Mutex (#1095, #1127, #1128), typing system (#1172, #1176)
    • Allow building with newer compilers (#1134, #1137, #1145, #1206, #1208, #1210, #1211)
    • Build python3 by default, not python2 (#1209)

    MeasurementSet

    • Improve reading of DataDescriptionId (#1103)
    • Compute feed information (#1104) and field attributes (#1109) on demand
    • UVFits improvements: support X-Y mounts (#1115), allow large antenna numbers (#1144), fix handling FITS-IDI GAIN_CURVE (#1151)
    • Fix precision in T+dT syntax (#1118)

    Measures

    • Allow offsets to JPL coordinates (#1160)

    Tables

    • Cache iteration boundaries while sorting tables (#1106)
    • Use RefTable for iteration (#1108)
    • Adios2 related changes: #1110, #1116, #1121, #1148
    • Make Dysco part of casacore (#1117, #1125, #1146, #1166)
    • Add TaQL commands LIKE, COPYCOLUMN and DROPTABLE (#1154, #1169)
    • Fix one bug in rownr_t migration for TiledDataStMan (#1156)
    • Improve documentation (#1192)

    Images

    • Support CARTA opening images with a custom handler (#1158)
    • Allow easier python handling of beams (#1184)
    Source code(tar.gz)
    Source code(zip)
  • v3.4.0(Feb 19, 2021)

    3.4.0

    General

    • Major modernization of Arrays, bringing it more in line with C++11 (#1012)
    • Use std::regex (#1072)
    • Remove many warnings (#1024, #1036, #1062)
    • Use FFTW as default FFT provider (#1029, #1047, #1049)
    • UVFITS improvements (#1033, #1040, #1064, #1066, #1067)
    • Support for 64-bit row numbers.
    Source code(tar.gz)
    Source code(zip)
  • v3.3.0(Apr 15, 2020)

    3.3.0

    General

    • Replace implementation of complex trig functions by std:: (#1010)
    • Fix bug SciMath / StatsFramework where parameters were passed incorrectly (#984)

    Tables

    • Pass sorting algorithm optio nthrough to table iterator (#992)
    • Support for ADIOS2 (#1006)
    • Add .casarc option to disable table locking (#1002)

    Measurement Set / UVFits

    • Fix antenna swapping in MSConcat (#977)
    • Merge PROCESSOR table in MSConcat (#1004)
    • Fix antenna positions in importing old VLA UVFITS (#1005)

    Images

    • More efficient WCS batch coordinate lookups (#390
    Source code(tar.gz)
    Source code(zip)
  • v3.2.1(Nov 22, 2019)

    3.2.1

    This release is identical to 3.2.0 but increments the SOVERSION.

    3.2

    General

    • Remove some unused functionality (#854, #947)
    • Prevent setting OpenMP num_threads to zero (#962)

    Tables

    • Fix locking issue with IncrementalStorageManager (#970, #974)
    • Various improvements to TaQL
    • Various ADIOS2 improvements (#943, #966)

    Measures

    • Update hardcoded URLs for fetching measures data (#975)

    FITS

    • Correctly convert Nasmyth mount types (#946, #954)
    Source code(tar.gz)
    Source code(zip)
  • v3.2.0(Nov 19, 2019)

    3.2

    General

    • Remove some unused functionality (#854, #947)
    • Prevent setting OpenMP num_threads to zero (#962)

    Tables

    • Fix locking issue with IncrementalStorageManager (#970, #974)
    • Various improvements to TaQL
    • Various ADIOS2 improvements (#943, #966)

    Measures

    • Update hardcoded URLs for fetching measures data (#975)

    FITS

    • Correctly convert Nasmyth mount types (#946, #954)
    Source code(tar.gz)
    Source code(zip)
  • v3.1.2(Jul 11, 2019)

  • v3.1.1(Jun 5, 2019)

    3.1.1

    General

    • Only use DataManager.get/putSlice if possible (#901)
    • User docker for Linux build on Travis for CI (#856)
    • improve unicode support (#853)
    • Changed MadfmFunc so that it inherits from ArrayFunctorBase (#905)
    • Fix memory leak in TableProxy::getColumnDescription (#900)
    • Allow colon and fix recursive bison parsing (#894)
    • ObjectID's hash function behavior fix (#897)
    • Add a public getter for the mask array to LCRegionFixed (#895)
    • Replace STL-like containers by their STL counterparts (#890)

    Python support

    • Boost 1.67 Python components require a Python version suffix (#844)
    • Handle numpy Unicode arrays (#912)
    • Fix memory leak when converting Python unicode to casa string (#910)
    • Possible leak bug (#908)
    • Always use TpInt64 for PyLong (#916)
    • Improve CMake FindPython (#922)

    Build system

    • Travis macOS fails on Python3 OSX build system (#778)
    • Switch travis to docker /xenial (#847)
    Source code(tar.gz)
    Source code(zip)
  • v3.1.0(Apr 6, 2019)

    Fix bug, thread issues and added some new features.

    General

    • Removed many warnings issued by newer compiler versions (#798, #809, #819, #820, #866, #883)
    • Further improved thread-safety (#817, #869, #877, #886)
    • Added variance and standard deviation of complex numbers (#851)

    Tables

    • Improved parallel storage manager based on ADIOS2
    • Support of Int64 table columns (#859)
    • Added O_DIRECT support when using MultiFile (#885)
    • TaQL improvements (#813, #851, #867)
    • Small CTDS fixes (#840, #868)

    Measurement Set

    • Small improvements (#858, #864, #872)
    • Better handling of FITS-IDI data bit sampling (#836)

    Images

    • Support for non-chunked HDF5 data sets (#879, #880)
    • Fixed count overflow on very large images (#849)
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Nov 15, 2018)

    3.0.0

    General

    • Building with C++11 is now required
    • Improved thread-safety of statics (#775)
    • Improve statistics code (#785, #776, #662, #663)

    Tables

    • Avoid growing the size of tables when overwriting data (#768)
    • TaQL support for radial velocity, earth magnetic field and Doppler (#750)
    • Added performance tests for tables (#658)

    Measurement Set

    • Improvements to MSConcat (#779, #701)
    • Better handling of FITS-IDI flag history (#748)

    Images

    • Better support for double precision images (#752, #751)

    Other

    Source code(tar.gz)
    Source code(zip)
  • v2.4.1(Dec 13, 2017)

    2.4.1

    General

    • This release fixes a problem with TableIterator introduced in v2.4.0. This issue would cause problems with LOFAR's DPPP when msin.starttime was set. (#681)
    Source code(tar.gz)
    Source code(zip)
  • v2.4.0(Nov 1, 2017)

    2.4.0

    General

    • Bug fixes and improvements
    • This version can be used to compile CASA 5.0
    • Improved error checking when parsing dates/times (#619)
    • Statistics: allow data provider to specify number of threads (#645)

    MS

    • Disable caching of MS main table columns (#597)
    • Properly copy the SIGMA_SPECTRUM column, if present (#599)
    • Implement getTimesForSpws (#600)

    FITS and FITS-IDI

    • Implement digital corrections for DiFX/VLBA (#602)
    • Several improvements for WEIGHTS when reading FITS-IDI (#590, #608)
    • Parallactic angle calculation for Nasmyth mounts (#627)
    • Modernised matrix syntax in FITS files now follows standard 3.0 (#606)
    Source code(tar.gz)
    Source code(zip)
  • v2.3.0(May 17, 2017)

    2.3.0

    General

    • Bug fixes and improvements
    • Improved installation documentation in README
    • Add an option to use Ccache (-DUseCcache) (#549)
    • Add some statistics functionality (#569)

    Python

    • Make some of the TableProxy functionality publicly available (#559)
    • Make version checking from python (or plain C) possible (#583)

    TaQL

    • Fix transpose in TaQL (#563)
    • Added functions delay and uvwapp to mscal (#562)

    FITS

    • MSFitsOutput now writes ant diams (#536)
    • Improvements to FITS-IDI conversion (#538, #590, #579)
    Source code(tar.gz)
    Source code(zip)
  • v2.2.0(Dec 1, 2016)

    2.2.0

    General

    • Lots of bug fixes and improvements
    • Tests are not built by default anymore, use -DBUILD_TESTING=True to build them
    • Building with C++11 is now default, use -DCXX11=False if you do not have a recent compiler (#533)
    • Added JSON support (#506)

    TaQL

    • Major improvements in TaQL, such as masked arrays, new commands ALTER TABLE, SHOW TABLE, HELP (#388)

    Images

    • ImageConcat and ImageExpr now use JSON export (#517)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Mar 18, 2016)

    2.1.0

    General

    • Lots of code improvement, optimization and added tests

    Measures

    • The default search path for measures data is now smaller. The path can be set at cmake time by specifying -DDATA_DIR=/path/to/data. This path can contain %CASAROOT% which is expanded at run time, to support relocatable installations. The measures path can also still be set in .casarc. (#277)
    • Inserted fix to leap second handling problem (#290)
    • Various coordinate performance improvements (#258)

    Tables

    • Renamed showtable to showtableinfo (#320)
    • Added several TaQL functions (#229)

    MS

    • Improve testing of MSSummary (#330)
    • Many improvements for the MS related functions (#318, #291, #228, #208, ...)
    • Fix multithreaded MS creation (#298)

    Python

    • Build python bindings by default (not for CASA build)
    • Add experimental Python3 support. Read the README for instructions (#280)
    • Changed behavior from 2.0.3: Use PYTHON2_* configuration options to set your python2 interpreter. (#280)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.3(Jul 24, 2015)

    2.0.3

    NOTE: this release is not ABI compatible with release 2.0.1

    General

    • Merge of CASA work into casacore, CASA should compile with casacore now
    • Under-the-hood optimizations, like using allocator features (#132)
    • Building with gcc 5 is now possible (#166)
    • Compatibility with more versions of wcslib and cfitsio
    • SOFA is now an external dependency (#105)

    Tables

    • Arrays can now sometimes be reshaped while keeping allocated memory (#113)

    MS

    • Selection of baselines with a regexp between stations is now possible (#99)
    • Several new methods to MSMetaData (#138)

    Bug fixes

    • Fix a bug which caused an error with LOFAR measurement sets with the LOFAR tool msoverview (#140)
    • Fix a bug where TaQL would not write output when an expression was used (#184)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Apr 10, 2015)

  • v2.0.0(Apr 9, 2015)

    2.0.0

    Changes since 1.7.0

    General

    • Fixed all build problems for GCC and clang on various platforms (up to OS-X 10.10)
    • Fixed all valgrind issues found in test programs
    • Made the statics in all of casacore thread-safe. Similar to libstdc++ the user is responsible for thread-safety when using casacore ojbects.
    • Added casacore/ to the #include path of all header files
    • Changed namespace from casa to casacore (but still #defined as casa for time being)
    • Split Tables, MeasurementSets and Lattices into smaller parts
    • The 3 changes above give rise to backward incompatibilities. Scripts in casacore/changescripts (notably updateall) can be used to fix client code.

    Tables

    • Added GROUPBY/HAVING/aggregation function to TaQL
    • Optimized TaQL's IN operator (linear time for integer sets)
    • Added table tracing (class TableTrace)
    • Fixed TiledStMan issue where BucketCache was not shrinked
    • Added MultiFile option to combine table files in a single one

    Components

    • This module has been removed (too CASA specific)

    Images

    • Added support for beam per frequency channel and Stokes
    • Added persistency for ImageExpr and ImageCocat objects

    Python

    • Moved converters from pyrap to casacore/Python
    Source code(tar.gz)
    Source code(zip)
POCO C++ Libraries are powerful cross-platform C++ libraries for building network

The POCO C++ Libraries are powerful cross-platform C++ libraries for building network- and internet-based applications that run on desktop, server, mobile, IoT, and embedded systems.

POCO C++ Libraries 6.5k Nov 29, 2022
Mod - MASTERS of DATA, a course about videogames data processing and optimization

MASTERS of DATA Welcome to MASTERS of DATA. A course oriented to Technical Designers, Technical Artists and any game developer that wants to understan

Ray 34 Nov 4, 2022
Minipops alternative firmware for Music Thing Modular Radio Music

RMoxy Minipops drummer alternative firmware for Music Thing Modular Radio Music module The repository for the Radio Music module by Tom Whitwell Kits

Johan Berglund 7 Nov 6, 2022
The Express LRS Handset project is first and foremost an experimental test bed for pushing the envelope with what can be done with the ELRS radio link

The Express LRS Handset project is first and foremost an experimental test bed for pushing the envelope with what can be done with the ELRS radio link

ExpressLRS 24 Oct 19, 2022
FM radio tuner application for Raspberry Pi OS

GTK FM Tuner GTK FM Tuner is a lightweight, GUI base FM radio tuner application for Raspberry Pi OS. This application uses GTK to provide the UI of th

Dilshan R Jayakody 17 Nov 3, 2022
QN8035 based Raspberry Pi FM radio receiver

QN8035 based Raspberry Pi FM radio receiver. QN8035 is a single-chip stereo FM radio receiver IC designed by the Quintic Corporation. This receiver su

Dilshan R Jayakody 8 Oct 6, 2022
New version of the well known ESP32 Radio. Now optional I2S output!

ESP32Radio-V2 New version of the well known ESP32 Radio. Now optional I2S output! Compile time configuration in config.h. Do not forget to upload the

Ed Smallenburg 51 Nov 3, 2022
An ATSC 3.0 Transmitter for GNU Radio

# Copyright 2021 Ron Economos # # This file is part of gr-atsc3 # # gr-atsc3 is free software; you can redistribute it and/or modify # it under the te

Ron Economos 14 Oct 20, 2022
Decoded Information from Radio Emissions for Windows Or Linux Fans

Dire Wolf is a software "soundcard" AX.25 packet modem/TNC and APRS encoder/decoder. It can be used stand-alone to observe APRS traffic, as a tracker, digipeater, APRStt gateway, or Internet Gateway (IGate).

null 1.2k Nov 21, 2022
AX.25 protocol (packet radio and APRS) interface / bridging / switching / terminal related software for Linux

CB3ROB-AX25-TOOLS-LINUX 2021-11-23T00:58:49Z (TUESDAY) cb3rob-kiss-tcp-attach.c links ax0 interface to KISS-TCP TNC, soundmodem (direwolf), or multipl

CB3ROB TACTICAL SYSTEMS 1 Jan 1, 2022
A set of tutorial projects for creating a simple digital radio receiver based on the STM32G431KB microcontroller

simple-radio Обучающие проекты по созданию простого цифрового радиоприемника на базе микроконтроллера STM32G431KB. Разработка программ выполнялась в W

null 9 Sep 21, 2022
2.4 GHz LORA based telemetry and radio link for remote controlled vehicles

mLRS This is the mLRS project. The goal is an open source 2.4 GHz LORA-based high-performance long-range radio link, which provides fully transparent

null 61 Nov 22, 2022
SatDump - A generic satellite data processing software.

SatDump A generic satellite data processing software. Thanks Mnux for the icon! Still WIP Usage First of all, as with any program using volk, running

Altillimity 363 Nov 21, 2022
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow Powering In-Memory Analytics Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enabl

The Apache Software Foundation 10.7k Nov 27, 2022
the ares multi-system console emulation suite

ares ares is a multi-system emulator that began development on October 14th, 2004. It is a descendent of higan and bsnes, and focuses on accuracy and

higan 395 Nov 23, 2022
Test suite for evaluating C code on 42 Basecamp

Joãonette Test suite based on Moulinette, made by "Iniciativa João" ⛺️ Setup Clone this repo ( I suggest cloning to the root and calling it joaonette

Lucas Kuhn 14 Sep 29, 2022
Violent Fungus is a command and control (C2) software suite, providing red teams post-exploitation persistence and other juicy stuff.

Violent Fungus is a command and control (C2) software suite, providing red teams post-exploitation persistence and other juicy stuff.

Chris Humphries 34 Sep 7, 2022
the ares multi-system console emulation suite

ares is a multi-system emulator that began development on October 14th, 2004. It is a descendent of higan and bsnes, and focuses on accuracy and prese

ares 394 Nov 22, 2022
A combined suite of utilities for exporting images to retro formats.

ImageTools A combined suite of utilities for exporting images to retro formats. It was developed for use on Windows but might compile on other systems

David Walters 4 Dec 6, 2021