Parsing Expression Grammar Template Library

Overview

Welcome to the PEGTL

Windows CI macOS CI Linux CI Android CI
clang-analyze clang-tidy Sanitizer CodeQL Code Coverage

The Parsing Expression Grammar Template Library (PEGTL) is a zero-dependency C++ header-only parser combinator library for creating parsers according to a Parsing Expression Grammar (PEG).

Documentation

Contact

Join us on Discord

For questions and suggestions regarding the PEGTL, success or failure stories, and any other kind of feedback, please feel free to join our Discord server, open a discussion, an issue or a pull request on GitHub or contact the authors at taocpp(at)icemx.net.

Introduction

Grammars are written as regular C++ code, created with template programming (not template meta programming), i.e. nested template instantiations that naturally correspond to the inductive definition of PEGs (and other parser-combinator approaches).

A comprehensive set of parser rules that can be combined and extended by the user is included, as are mechanisms for debugging grammars, and for attaching user-defined actions to grammar rules. Here is an example of how a PEG grammar rule is implemented as C++ class with the PEGTL.

// PEG rule for integers consisting of a non-empty
// sequence of digits with an optional sign:

// sign ::= '+' / '-'
// integer ::= sign? digit+

// The same parsing rule implemented with the PEGTL:

using namespace tao::pegtl;

struct sign : one< '+', '-' > {};
struct integer : seq< opt< sign >, plus< digit > > {};

PEGs are superficially similar to Context-Free Grammars (CFGs), however the more deterministic nature of PEGs gives rise to some very important differences. The included grammar analysis finds several typical errors in PEGs, including left recursion.

Design

The PEGTL is designed to be "lean and mean", the core library consists of approximately 6000 lines of code. Emphasis is on simplicity and efficiency, preferring a well-tuned simple approach over complicated optimisations.

The PEGTL is mostly concerned with parsing combinators and grammar rules, and with giving the user of the library (the possibility of) full control over all other aspects of a parsing run. Whether/which actions are taken, and whether/which data structures are created during a parsing run, is entirely up to the user.

Included are some examples for typical situation like unescaping escape sequences in strings, building a generic JSON data structure, and on-the-fly evaluation of arithmetic expressions.

Through the use of template programming and template specialisations it is possible to write a grammar once, and use it in multiple ways with different (semantic) actions in different (or the same) parsing runs.

With the PEG formalism, the separation into lexer and parser stages is usually dropped -- everything is done in a single grammar. The rules are expressed in C++ as template instantiations, and it is the compiler's task to optimise PEGTL grammars.

Status

Each commit is automatically tested with multiple architectures, operating systems, compilers, and versions thereof.

Each commit is checked with GCC's and Clang's sanitizers, Clang's Static Analyzer, and clang-tidy. Additionally, we use CodeQL to scan for (security) issues.

Code coverage is automatically measured and the unit tests cover 100% of the core library code (for releases).

Releases are done in accordance with Semantic Versioning. Incompatible API changes are only allowed to occur between major versions.

Thank You

In appreciation of all contributions here are the people that have directly contributed to the PEGTL and/or its development.

amphaal anand-bala andoma barbieri bjoe bwagner cdiggins clausklein delpinux dkopecek gene-hightower irrequietus jedelbo joelfrederico johelegp jovermann jubnzv kelvinhammond kneth kuzmas lambdafu lichray michael-brade mkrupcale newproggie obiwahn ohanar pauloscustodio pleroux0 quadfault ras0219 redmercury robertcampion samhocevar sanssecours sgbeal skyrich62 studoot svenjo wickedmic wravery zhihaoy

The Art of C++

The PEGTL is part of The Art of C++.

colinh d-frey uilianries

License

Open Source Initiative

The PEGTL is certified Open Source software. It may be used for any purpose, including commercial purposes, at absolutely no cost. It is distributed under the terms of the MIT license reproduced here.

Copyright (c) 2007-2021 Dr. Colin Hirsch and Daniel Frey

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Issues
  • Switch license from MIT to BSL-1.0

    Switch license from MIT to BSL-1.0

    Dear contributors,

    we would like to switch from the MIT license to the Boost Software License, Version 1.0. Both are very similar with one crucial difference: The attribution clause of the BSL-1.0 has an exception for binaries. I'm not a lawyer, so do your own research if you want to get into the details.

    The motivation for our change is to allow the use of the PEGTL in more environments, especially when it is difficult or even impossible for the binary to output the license. Think pacemaker-like device, but please no one use the PEGTL in pacemakers... 😅

    In order to do the switch properly, we need the consent from everyone who contributed something that is meeting a certain threshold of originality. That threshold is required for your work to fall under copyright. As mentioned I am not a lawyer and I'd like to avoid having to judge your contributions. Therefore, I am simply asking you to give your permission to switch the license from MIT to BSL-1.0. You can do so by replying to this issue and clearly stating that you agree to the license change.

    As we are also planning on switching our other projects, and if you have contributed to those other projects, it would be helpful if you could explicitly mention in your answer whether you give your permission only for the PEGTL or for all your contributions to our project, "The Art of C++".

    Please also reply if you do not give permission to switch. It is better for us to have an explicit answer than having to guess.

    Thank you for your contributions!

    The Art of C++ Team: @d-frey @ColinH @uilianries

    Pinging everyone from GitHub's list of contributors, in the order they appeared from GitHub:

    @wravery @pleroux0 @Bjoe @studoot @sanssecours @samhocevar @joelfrederico @kelvinhammond @pauloscustodio @zhihaoy @redmercury @kuzmas @ras0219 @Amphaal @robertcampion @michael-brade @barbieri @delpinux @hutorny @andoma @lichray @jbomanson @mkrupcale @quadfault @bwagner @jubnzv @lambdafu @dkopecek @JohelEGP

    question feedback please! 
    opened by d-frey 28
  • Fix or nix Android CI builds

    Fix or nix Android CI builds

    Currently, the Andriod builds are broken due to using std::filesystem::path. On some non-Android builds this was fixed by adding -llibstdc++fs, on some builds it just worked. Android, however, seems to be too old in some cases and I wouldn't know how to add the above option (in case it is required and helps). Also, the version we are using now seem to be a bit outdated.

    We therefore need to either fix (and possibly modernize the build) or I will have to remove them (and most likely Android is then broken).

    Any help is appreciated, @Bjoe ;)

    bug help wanted 
    opened by d-frey 21
  • Support Conan

    Support Conan

    Related issue: #111

    I dont have permission to create branches on taocpp, because I have member role. But no problem, I just forked the project.

    This PR just update the support to use Conan + PEGTL

    I added a job for Travis and Appeveyor to validate on both platforms

    Why I copied the files instead to use cmake helper?
    The cmake helper requires settings.compiler when run over Windows, but this package is header-only and no settings is required. In the future I'll provide FindPEGTL.cmake file, so don't worry.

    The next step is update Travis settings. I don't have permission to change the project settings on Travis :( But you need to add CONAN_LOGIN_USERNAME as taocpp-user, and CONAN_PASSWORD with Key API provided by Bintray.

    Regards!

    opened by uilianries 21
  • more useful tracer

    more useful tracer

    [ provides tracer with easier color toggle and more detailed location reporting ]

    this expands the tracer to provide explicit color control for easy configuration for terminal/file output which can be useful when redirecting to file via terminal > or >>

    # --no-color is not interpereted by pegtl itself
    #   it is instead interperated by test_trace.cpp main
    #   and causes the no_color variant to be called instead
    $ ./test_trace --no-color > out.txt
    
        trace_standard // defaults to trace_standard_color
        trace_standard_color
        trace_standard_no_color
    
        trace_complete // defaults to trace_complete_color
        trace_complete_color
        trace_complete_no_color
    

    sample configuration

    // default color output
    tao::pegtl::complete_trace<grammar, action>(in, data);
    
    // color output
    tao::pegtl::complete_trace_color<grammar, action>(in, data);
    
    // no color output
    tao::pegtl::complete_trace_no_color<grammar, action>(in, data);
    

    this also expands the tracer to include the source line and cursor position at which the rule is being matched against

    in the previous tracer, this is the current output

    sample output

            position GLSL preprocessor stage 3:1:1
    #1      GLSL_PREPROCESSOR::grammar
    #2        GLSL_PREPROCESSOR::reset_preprocessor_state
              apply
              success
    #3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
    #4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #5            GLSL_PREPROCESSOR::whitespaces_and_newlines
    #6              GLSL_PREPROCESSOR::spaces
                    failure
    #7              GLSL_PREPROCESSOR::tabs
                    failure
    #8              GLSL_PREPROCESSOR::newlines
                    failure
                  failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
    #9            GLSL_PREPROCESSOR::identifier
    #10             tao::pegtl::ascii::identifier_first
                    success
                    position GLSL preprocessor stage 3:1:2
    #11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
    #12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                      success
                      position GLSL preprocessor stage 3:1:4
                    success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
                  apply
                  success #9 GLSL_PREPROCESSOR::identifier
                success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #14           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #15             GLSL_PREPROCESSOR::spaces
                    success
                    position GLSL preprocessor stage 3:1:5
    #16             GLSL_PREPROCESSOR::spaces
                    failure
    #17             GLSL_PREPROCESSOR::tabs
                    failure
    #18             GLSL_PREPROCESSOR::newlines
                    failure
                  apply
                  success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
                success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #20           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #21             GLSL_PREPROCESSOR::spaces
                    failure
    #22             GLSL_PREPROCESSOR::tabs
                    failure
    #23             GLSL_PREPROCESSOR::newlines
                    failure
                  failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
    #24           GLSL_PREPROCESSOR::identifier
    #25             tao::pegtl::ascii::identifier_first
                    failure
                  failure #24 GLSL_PREPROCESSOR::identifier
                failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
              success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
    #26       tao::pegtl::eof
              failure
            failure #1 GLSL_PREPROCESSOR::grammar
            position GLSL preprocessor stage 3:1:1
    

    this makes debugging difficult as you cannot see the input stream, nor the input that is about to processed, and instead have to try to decipher and track the given line numbers and columns to try to make sense of where your grammar may wrong

    this can be difficult especially if the input is memory or a stream such as stdin or similar or any other input in which you do not easily have access to the source being parsed

    so i added the current source and current cursor position underneath the source position

            position GLSL preprocessor stage 3:1:1
            source int 6 main
                   ^
    

    in the above, the grammar will attempt to match starting at i

            position GLSL preprocessor stage 3:1:3
            source int 6 main
                     ^
    

    in the above, the grammar will attempt to match starting at t

    sample output

            position GLSL preprocessor stage 3:1:1
            source int 6 main
                   ^
    #1      GLSL_PREPROCESSOR::grammar
    #2        GLSL_PREPROCESSOR::reset_preprocessor_state
              apply
              success
    #3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
    #4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #5            GLSL_PREPROCESSOR::whitespaces_and_newlines
    #6              GLSL_PREPROCESSOR::spaces
                    failure
    #7              GLSL_PREPROCESSOR::tabs
                    failure
    #8              GLSL_PREPROCESSOR::newlines
                    failure
                  failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
    #9            GLSL_PREPROCESSOR::identifier
    #10             tao::pegtl::ascii::identifier_first
                    success
                    position GLSL preprocessor stage 3:1:2
                    source int 6 main
                            ^
    #11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
    #12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                      success
                      position GLSL preprocessor stage 3:1:4
                      source int 6 main
                                ^
                    success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
                  apply
                  success #9 GLSL_PREPROCESSOR::identifier
                success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #14           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #15             GLSL_PREPROCESSOR::spaces
                    success
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #16             GLSL_PREPROCESSOR::spaces
                    failure
    #17             GLSL_PREPROCESSOR::tabs
                    failure
    #18             GLSL_PREPROCESSOR::newlines
                    failure
                  apply
                  success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
                success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #20           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #21             GLSL_PREPROCESSOR::spaces
                    failure
    #22             GLSL_PREPROCESSOR::tabs
                    failure
    #23             GLSL_PREPROCESSOR::newlines
                    failure
                  failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
    #24           GLSL_PREPROCESSOR::identifier
    #25             tao::pegtl::ascii::identifier_first
                    failure
                  failure #24 GLSL_PREPROCESSOR::identifier
                failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
              success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
    #26       tao::pegtl::eof
              failure
            failure #1 GLSL_PREPROCESSOR::grammar
            position GLSL preprocessor stage 3:1:1
            source int 6 main
                   ^
    

    this makes debugging grammar easier as you can now see what the input stream looks like and exactly where in the stream the grammar is attempting to match

    i did consider adding the same information to both success and failure

    sample output

            position GLSL preprocessor stage 3:1:1
            source int 6 main
                   ^
    #1      GLSL_PREPROCESSOR::grammar
    #2        GLSL_PREPROCESSOR::reset_preprocessor_state
              apply
              success
              position GLSL preprocessor stage 3:1:1
              source int 6 main
                     ^
    #3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
    #4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #5            GLSL_PREPROCESSOR::whitespaces_and_newlines
    #6              GLSL_PREPROCESSOR::spaces
                    failure
                    position GLSL preprocessor stage 3:1:1
                    source int 6 main
                           ^
    #7              GLSL_PREPROCESSOR::tabs
                    failure
                    position GLSL preprocessor stage 3:1:1
                    source int 6 main
                           ^
    #8              GLSL_PREPROCESSOR::newlines
                    failure
                    position GLSL preprocessor stage 3:1:1
                    source int 6 main
                           ^
                  failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
                  position GLSL preprocessor stage 3:1:1
                  source int 6 main
                         ^
    #9            GLSL_PREPROCESSOR::identifier
    #10             tao::pegtl::ascii::identifier_first
                    success
                    position GLSL preprocessor stage 3:1:2
                    source int 6 main
                            ^
    #11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
    #12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                      success
                      position GLSL preprocessor stage 3:1:4
                      source int 6 main
                                ^
                    success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
                    position GLSL preprocessor stage 3:1:4
                    source int 6 main
                              ^
                  apply
                  success #9 GLSL_PREPROCESSOR::identifier
                  position GLSL preprocessor stage 3:1:4
                  source int 6 main
                            ^
                success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
                position GLSL preprocessor stage 3:1:4
                source int 6 main
                          ^
    #13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #14           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #15             GLSL_PREPROCESSOR::spaces
                    success
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #16             GLSL_PREPROCESSOR::spaces
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #17             GLSL_PREPROCESSOR::tabs
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #18             GLSL_PREPROCESSOR::newlines
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
                  apply
                  success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
                  position GLSL preprocessor stage 3:1:5
                  source int 6 main
                             ^
                success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
    #19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
    #20           GLSL_PREPROCESSOR::whitespaces_and_newlines
    #21             GLSL_PREPROCESSOR::spaces
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #22             GLSL_PREPROCESSOR::tabs
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
    #23             GLSL_PREPROCESSOR::newlines
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
                  failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
                  position GLSL preprocessor stage 3:1:5
                  source int 6 main
                             ^
    #24           GLSL_PREPROCESSOR::identifier
    #25             tao::pegtl::ascii::identifier_first
                    failure
                    position GLSL preprocessor stage 3:1:5
                    source int 6 main
                               ^
                  failure #24 GLSL_PREPROCESSOR::identifier
                  position GLSL preprocessor stage 3:1:5
                  source int 6 main
                             ^
                failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
              success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
              position GLSL preprocessor stage 3:1:5
              source int 6 main
                         ^
    #26       tao::pegtl::eof
              failure
              position GLSL preprocessor stage 3:1:5
              source int 6 main
                         ^
            failure #1 GLSL_PREPROCESSOR::grammar
            position GLSL preprocessor stage 3:1:1
            source int 6 main
                   ^
    

    but as you can see it kinda spams the log with info you already know and at this point it is largely duplicate info for most rules that fail to match, so i stuck with only showing it when ever the position changes, which is and was the default

    Source

    namespace tao::pegtl
    {
        template< bool HideInternal = false, bool UseColor = true, std::size_t IndentIncrement = 2, std::size_t InitialIndent = 8 >
        struct tracer_traits
        {
            template< typename Rule >
            static constexpr bool enable = ( HideInternal ? tao::pegtl::normal< Rule >::enable : true );
    
            static constexpr std::size_t initial_indent = InitialIndent;
            static constexpr std::size_t indent_increment = IndentIncrement;
    
            static constexpr std::string_view ansi_reset = UseColor ? "\033[m" : "";
            static constexpr std::string_view ansi_rule = UseColor ? "\033[36m" : "";
            static constexpr std::string_view ansi_hide = UseColor ? "\033[37m" : "";
    
            static constexpr std::string_view ansi_position = UseColor ? "\033[1;34m" : "";
            static constexpr std::string_view ansi_success = UseColor ? "\033[32m" : "";
            static constexpr std::string_view ansi_failure = UseColor ? "\033[31m" : "";
            static constexpr std::string_view ansi_raise = UseColor ? "\033[1;31m" : "";
            static constexpr std::string_view ansi_unwind = UseColor ? "\033[31m" : "";
            static constexpr std::string_view ansi_apply = UseColor ? "\033[1;36m" : "";
        };
    
        using standard_tracer_traits_color = tracer_traits< true, true >;
        using standard_tracer_traits_no_color = tracer_traits< true, false >;
        using complete_tracer_traits_color = tracer_traits< false, true >;
        using complete_tracer_traits_no_color = tracer_traits< false, false >;
    
        template< typename TracerTraits >
        struct tracer
        {
            const std::ios_base::fmtflags m_flags;
            std::size_t m_count = 0;
            std::vector< std::size_t > m_stack;
            tao::pegtl::position m_position;
    
            template< typename Rule >
            static constexpr bool enable = TracerTraits::template enable< Rule >;
    
            template< typename ParseInput >
            explicit tracer( const ParseInput& in )
                    : m_flags( std::cerr.flags() ),
                      m_position( in.position() )
            {
                std::cerr << std::left;
                print_position(in);
            }
    
            tracer( const tracer& ) = delete;
            tracer( tracer&& ) = delete;
    
            ~tracer()
            {
                std::cerr.flags( m_flags );
            }
    
            tracer& operator=( const tracer& ) = delete;
            tracer& operator=( tracer&& ) = delete;
    
            [[nodiscard]] std::size_t indent() const noexcept
            {
                return TracerTraits::initial_indent + TracerTraits::indent_increment * m_stack.size();
            }
    
            template <typename ParseInput>
            void print_position(const ParseInput& in) const
            {
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_position << "position" << TracerTraits::ansi_reset << ' ' << m_position << '\n';
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_position << "source" << TracerTraits::ansi_reset << " " << in.line_at(m_position) << "\n";
                std::cerr << std::setw( indent() + 6 + m_position.column ) << ' ' << '^' << '\n';
            }
    
            template <typename ParseInput>
            void update_position( const ParseInput& in )
            {
                const tao::pegtl::position& p = in.position();
                if( m_position != p ) {
                    m_position = p;
                    print_position(in);
                }
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void start( const ParseInput& /*unused*/, States&&... /*unused*/ )
            {
                std::cerr << '#' << std::setw( indent() - 1 ) << ++m_count << TracerTraits::ansi_rule << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset << '\n';
                m_stack.push_back( m_count );
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void success( const ParseInput& in, States&&... /*unused*/ )
            {
                const auto prev = m_stack.back();
                m_stack.pop_back();
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_success << "success" << TracerTraits::ansi_reset;
                if( m_count != prev ) {
                    std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
                }
                std::cerr << '\n';
                update_position( in );
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void failure( const ParseInput& in, States&&... /*unused*/ )
            {
                const auto prev = m_stack.back();
                m_stack.pop_back();
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_failure << "failure" << TracerTraits::ansi_reset;
                if( m_count != prev ) {
                    std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
                }
                std::cerr << '\n';
                update_position( in );
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void raise( const ParseInput& /*unused*/, States&&... /*unused*/ )
            {
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_raise << "raise" << TracerTraits::ansi_reset << ' ' << TracerTraits::ansi_rule << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset << '\n';
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void unwind( const ParseInput& in, States&&... /*unused*/ )
            {
                const auto prev = m_stack.back();
                m_stack.pop_back();
                std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_unwind << "unwind" << TracerTraits::ansi_reset;
                if( m_count != prev ) {
                    std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
                }
                std::cerr << '\n';
                update_position( in );
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void apply( const ParseInput& /*unused*/, States&&... /*unused*/ )
            {
                std::cerr << std::setw( static_cast< int >( indent() - TracerTraits::indent_increment ) ) << ' ' << TracerTraits::ansi_apply << "apply" << TracerTraits::ansi_reset << '\n';
            }
    
            template< typename Rule, typename ParseInput, typename... States >
            void apply0( const ParseInput& /*unused*/, States&&... /*unused*/ )
            {
                std::cerr << std::setw( static_cast< int >( indent() - TracerTraits::indent_increment ) ) << ' ' << TracerTraits::ansi_apply << "apply0" << TracerTraits::ansi_reset << '\n';
            }
    
            template< typename Rule,
                    template< typename... > class Action = tao::pegtl::nothing,
                    template< typename... > class Control = tao::pegtl::normal,
                    typename ParseInput,
                    typename... States >
            bool parse( ParseInput&& in, States&&... st )
            {
                return tao::pegtl::parse< Rule, Action, tao::pegtl::state_control< Control >::template type >( in, st..., *this );
            }
        };
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool standard_trace_color( ParseInput&& in, States&&... st )
        {
            tracer< standard_tracer_traits_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool standard_trace( ParseInput&& in, States&&... st )
        {
            tracer< standard_tracer_traits_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool standard_trace_no_color( ParseInput&& in, States&&... st )
        {
            tracer< standard_tracer_traits_no_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool complete_trace_color( ParseInput&& in, States&&... st )
        {
            tracer< complete_tracer_traits_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool complete_trace( ParseInput&& in, States&&... st )
        {
            tracer< complete_tracer_traits_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool complete_trace_no_color( ParseInput&& in, States&&... st )
        {
            tracer< complete_tracer_traits_no_color > tr( in );
            return tr.parse< Rule, Action, Control >( in, st... );
        }
    
        template< typename Tracer >
        struct trace
                : tao::pegtl::maybe_nothing
        {
            template< typename Rule,
                    tao::pegtl::apply_mode A,
                    tao::pegtl::rewind_mode M,
                    template< typename... >
                    class Action,
                    template< typename... >
                    class Control,
                    typename ParseInput,
                    typename... States >
            [[nodiscard]] static bool match( ParseInput& in, States&&... st )
            {
                if constexpr( sizeof...( st ) == 0 ) {
                    return tao::pegtl::match< Rule, A, M, Action, tao::pegtl::state_control< Control >::template type >( in, st..., Tracer( in ) );
                }
                else if constexpr( !std::is_same_v< std::tuple_element_t< sizeof...( st ) - 1, std::tuple< States... > >, Tracer& > ) {
                    return tao::pegtl::match< Rule, A, M, Action, tao::pegtl::state_control< Control >::template type >( in, st..., Tracer( in ) );
                }
                else {
                    return tao::pegtl::match< Rule, A, M, Action, Control >( in, st... );
                }
            }
        };
    
        using trace_standard_color = trace< tracer< standard_tracer_traits_color > >;
        using trace_standard = trace_standard_color;
        using trace_standard_no_color = trace< tracer< standard_tracer_traits_no_color > >;
        using trace_complete_color = trace< tracer< complete_tracer_traits_color > >;
        using trace_complete = trace_complete_color;
        using trace_complete_no_color = trace< tracer< complete_tracer_traits_no_color > >;
    
    }  // namespace tao::pegtl
    
    enhancement 
    opened by mgood7123 17
  • Need hierarchical action handling

    Need hierarchical action handling

    I'm writing a parser for the Verilog language using PEGTL and wanted to share a problem I'm running into and a possible solution. This is my first time using PEG-style parsing, although I am far from a novice at parsing, having done work for my PhD thesis on approximate parsing in a network security context (google "flowsifter").

    I can appreciate the simplicity and elegance (and incredible run-time efficiency) of having actions associated with each rule run when that rule is matched; this combines with the control system to make a very powerful way to write code that gets triggered by parsing. But I'm particularly annoyed at how unnecessarily difficult it is to write side-effect heavy code that works correctly without these pieces of code having a hierarchical context to work in.

    In the best parsers I've worked with (and written), when part of the grammar matches, it can return a value to the next level up, and that value can be used as part of constructing the result of that higher-level rule's parse result. This ends up resulting in a sequence of function calls that are nested in exactly the same structure as the parse tree of the text being parsed. In PEGTL, I can decide exactly what code runs when a particular rule matches, but I get no nesting of function calls, I only get a flat space of code executions. Of course it's possible to implement that hierarchy by using an explicit stack and pushing and popping from the stack as rules are matched, but this has a high degree of complexity and is prone to user error. The expression parser example in this repository even goes as far as using a stack of stacks to handle parentheses in expressions.

    There's got to be a reasonable way to stitch together rules and actions in such a way that actions can return a value and the action for a rule receives/can access values produced by child rules. Maybe this will require each child rule to have an action that returns a value; that seems a reasonable price to pay for this feature.

    enhancement 
    opened by thelema 17
  • Replace try/catch/rethrow in unwind with unwind_guard

    Replace try/catch/rethrow in unwind with unwind_guard

    I noticed while investigating https://github.com/microsoft/cppgraphqlgen/issues/222 that parse_tree implements unwind on the make_control<>::state_handler<> struct, and this results in creating a try/catch/rethrow block around every rule match in match_control_unwind. Every try/catch/rethrow seems to be consuming significant stack space, so if you raise/throw an exception from a relatively shallow (< 20) nested rule, it often exhausts the stack.

    This change makes it so parse_tree will only implement unwind on the Control type wrapper if either the Control or the Node type implements it. When parsing into a tree that doesn't implement either of them, the exception will be thrown once all the way from the initial raise to the caller of parse_tree::parse.

    Since the unit tests use the default node which inherits basic_node, they still implement unwind. The unit tests also wrap the rule in a try_catch_type handler, though, so the exception doesn't bubble all the way up. Before I nailed down the constexpr bool definitions, I accidentally suppressed unwind for basic_node types in the unit tests as well, and the lack of stack.pop_back() calls resulted in the assert(stack.size() == 1) firing for the unit test. So that tells me this is incompatible with try_catch_type. That's the only thing I think might block it.

    enhancement 
    opened by wravery 16
  • line_at not finding line ending when inside of trace.hpp

    line_at not finding line ending when inside of trace.hpp

    im using the following

    pegtl::memory_input in(spec, "Specification test");
    

    but in.line_at( p ) does not seem to correctly acquire the line

    #5        tao::pegtl::ascii::any]
              success
              position Specification test:2:4
              source VERTEX_CHAIN {
        vec4 position;
        vec4 textureCoordinates;
    };
    
    FRAGMENT_CHAIN {
        vec4 color;
    }
    
    

    even if i change the Eol from eol::cr_crlf to eol::lf for all input classes it does not make a difference

    question 
    opened by mgood7123 16
  • Added support for fixed-length list.

    Added support for fixed-length list.

    I made a list which supports both dynamic and fixed length. It should be backward compatible.

    Also, the ifdef was added in rules.hpp for demonstration purpose. This should be removed after some discussion.

    opened by qawbecrdtey 15
  • Add final argument for internal `parse_tree` state to `make_control` functions

    Add final argument for internal `parse_tree` state to `make_control` functions

    Since we know that make_control is going to be given the internal state argument as its last argument from parse_tree::parse, we can explicitly take it as the function argument. This removes the need for using tuple methods and allows us to only pass on the user-provided state arguments to the underlying base Control class. Thus, any external Action or Control functions will also not have to accept the internal parse_tree state argument.

    The Control match function is special, however, because it must use make_control< Rule >::match rather than Control< Rule >::match so that it can correctly invoke and pass the internal parse_tree state to the other make_control< Rule >::(start|success|failure|raise|apply0|apply) functions through duseltronik.

    Note that this PR also results in performance improvements on the order of 5-10% for some simple parse_tree calculator benchmarks I've run. This is likely due to the reduced complexity by not using the tuple tie and get functions to create a tuple to extract the last variadic template argument.

    enhancement 
    opened by mkrupcale 14
  • Automatically fallback to older versions of std::filesystem

    Automatically fallback to older versions of std::filesystem

    This is addressing the discussion in #216. Along with the build and code changes I created a GitHub Actions CI/PR workflow to test it on Ubuntu 16.04 and 18.04, with gcc 7 and 8 on each of them. The 2 versions of Ubuntu work more or less the same way, but the gcc 7 jobs fall back to std::experimental::filesystem. The gcc 8 jobs detect that they need to link with -lstdc++fs, but they still use std::filesystem.

    We might be able to add another job to do an Android build and test out the Boost.Filesystem fallback, which would also fix the original problem with #216.

    opened by wravery 13
  • add missing virtual dtors

    add missing virtual dtors

    Fixes clang warnings /usr/include/c++/v1/memory:3710:5: error: destructor called on non-final 'examples::null_json' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor] _data.second().~_Tp(); ^

    Signed-off-by: Khem Raj [email protected]

    opened by kraj 13
  • Error propagation

    Error propagation

    would it be possible to modify parse_error to support error probogation?

    i have a basic version working

    parse_error.hpp

    namespace TAO_PEGTL_NAMESPACE
    {
       namespace internal
       {
          class parse_error
          {
          private:
             std::string m_src;
    
    // ...
    
             explicit parse_error( const char* where, const char* msg )
                : m_src(where), m_msg( msg )
             {}
    
             explicit parse_error( const std::string& where, const char* msg )
                : m_src(where), m_msg( msg )
             {}
    
             [[nodiscard]] const char* where() const noexcept
             {
                return m_src.c_str();
             }
    
    // ...
    
          };
    
       }  // namespace internal
    
       class parse_error
          : public std::runtime_error
       {
    // ...
    
          parse_error( const char* where, const char* msg, position p )
             : std::runtime_error( msg ),
               m_impl( std::make_shared< internal::parse_error >(where, msg ) )
          {
             m_impl->add_position( std::move( p ) );
          }
    
          parse_error( const std::string& where, const char* msg, position p )
             : std::runtime_error( msg ),
               m_impl( std::make_shared< internal::parse_error >(where.c_str(), msg ) )
          {
             m_impl->add_position( std::move( p ) );
          }
    
          parse_error( const std::string_view& where, const char* msg, position p )
             : std::runtime_error( msg ),
               m_impl( std::make_shared< internal::parse_error >(std::string(where.data(), where.size()), msg ) )
          {
             m_impl->add_position( std::move( p ) );
          }
    
          parse_error( const char* where, const std::string& msg, position p )
             : parse_error( where, msg.c_str(), std::move( p ) )
          {}
    
          parse_error( const std::string& where, const std::string& msg, position p )
             : parse_error( where.c_str(), msg.c_str(), std::move( p ) )
          {}
    
          parse_error( const std::string_view& where, const std::string& msg, position p )
             : parse_error( std::string(where.data(), where.size()), msg.c_str(), std::move( p ) )
          {}
    
          template< typename ParseInput >
          parse_error( const char* msg, const ParseInput& in )
             : parse_error( in.line_at(in.position()), msg, in.position() )
          {}
    
          template< typename ParseInput >
          parse_error( const std::string& msg, const ParseInput& in )
             : parse_error( in.line_at(in.position()), msg, in.position() )
          {}
    
    // ...
    
          [[nodiscard]] const char* where() const noexcept
          {
             return m_impl->where();
          }
    
    // ...
    
       };
    
    }  // namespace TAO_PEGTL_NAMESPACE
    

    action_input.hpp

    // ...
          [[nodiscard]] std::string_view line_at( const TAO_PEGTL_NAMESPACE::position& p ) const noexcept
          {
             return input().line_at(p);
          }
    // ...
    

    the source line is obtainable via e.where() and should replace in.line_at(p) or in.line_at(e.positions().front())

    the source line is automatically set when given a ParseInput and can be set manually via where prefix as where, msg, pos instead of msg, pos

    such enables correct source line probogation when sub-parsing inside of actions

    #61                           GLSL_PREPROCESSOR::spaces
                                  success
                                  position GLSL preprocessor macro expansion:1:10
                                  source bar A (4 + (2 * x))()
                                                  ^
    // ...
    #97                         tao::pegtl::raise_message<'I', 'n', 'v', 'a', 'l', 'i', 'd', ' ', 'c', 'h', 'a', 'r', 'a', 'c', 't', 'e', 'r', ' ', 'f', 'o', 'u', 'n', 'd'>
                                  raise tao::pegtl::raise_message<'I', 'n', 'v', 'a', 'l', 'i', 'd', ' ', 'c', 'h', 'a', 'r', 'a', 'c', 't', 'e', 'r', ' ', 'f', 'o', 'u', 'n', 'd'>
                                unwind
                              unwind #83 GLSL_PREPROCESSOR::function_call_arg
                            unwind #82 GLSL_PREPROCESSOR::function_call_arg_and_optional_comma
                          unwind #78 tao::pegtl::sor<tao::pegtl::seq<tao::pegtl::at<tao::pegtl::eof>, tao::pegtl::raise_message<'U', 'n', 't', 'e', 'r', 'm', 'i', 'n', 'a', 't', 'e', 'd', ' ', 'f', 'u', 'n', 'c', 't', 'i', 'o', 'n', ' ', 'p', 'a', 'r', 'e', 'n', 't', 'h', 'e', 's', 'i', 's', ',', ' ', 'e', 'x', 'p', 'e', 'c', 't', 'e', 'd', ' ', '\'', ')', '\'', ' ', 't', 'o', ' ', 'm', 'a', 't', 'c', 'h', ' ', '\'', '(', '\''> >, GLSL_PREPROCESSOR::function_call_arg_and_optional_comma>
                        unwind #75 tao::pegtl::until<tao::pegtl::at<GLSL_PREPROCESSOR::function_end>, tao::pegtl::sor<tao::pegtl::seq<tao::pegtl::at<tao::pegtl::eof>, tao::pegtl::raise_message<'U', 'n', 't', 'e', 'r', 'm', 'i', 'n', 'a', 't', 'e', 'd', ' ', 'f', 'u', 'n', 'c', 't', 'i', 'o', 'n', ' ', 'p', 'a', 'r', 'e', 'n', 't', 'h', 'e', 's', 'i', 's', ',', ' ', 'e', 'x', 'p', 'e', 'c', 't', 'e', 'd', ' ', '\'', ')', '\'', ' ', 't', 'o', ' ', 'm', 'a', 't', 'c', 'h', ' ', '\'', '(', '\''> >, GLSL_PREPROCESSOR::function_call_arg_and_optional_comma> >
                      unwind #46 GLSL_PREPROCESSOR::function_call_parens
                    unwind #35 GLSL_PREPROCESSOR::function_call_actual
                  unwind #34 GLSL_PREPROCESSOR::function_call
                unwind #30 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces, GLSL_PREPROCESSOR::function_call, GLSL_PREPROCESSOR::identifier, tao::pegtl::ascii::any>
                position GLSL preprocessor macro expansion:1:5
                source bar A (4 + (2 * x))()
                           ^
              unwind #2 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces, GLSL_PREPROCESSOR::function_call, GLSL_PREPROCESSOR::identifier, tao::pegtl::ascii::any>>
            unwind #1 GLSL_PREPROCESSOR::grammar___
            position GLSL preprocessor macro expansion:1:1
            source bar A (4 + (2 * x))()
                   ^
                unwind
              unwind #11 GLSL_PREPROCESSOR::function_call_actual
              position GLSL preprocessor function expansion:1:1
              source foo(bar)
                     ^
    GLSL preprocessor macro expansion:1:10: Invalid character found
    bar A (4 + (2 * x))()
             ^
    
        } catch (const pegtl::parse_error &e) {
          const auto p = e.positions().front();
          std:cerr << e.what() << '\n'
               << e.where() << '\n'
               << std::setw(p.column) << '^' << std::endl;
        }
    
    question 
    opened by mgood7123 2
Releases(3.2.7)
  • 3.2.7(Jul 14, 2022)

  • 3.2.6(Jun 29, 2022)

  • 3.2.5(Feb 5, 2022)

  • 3.2.4(Feb 3, 2022)

  • 3.2.3(Feb 3, 2022)

  • 3.2.2(Oct 22, 2021)

    • Added rule odigit for octal digits.
    • Enabled default-constructed state in state<>, change_state<>, and change_action_and_state<>.
    • Changed rules in tao/pegtl/contrib/integer.hpp to not throw by default.
    • Added tao/pegtl/contrib/separated_seq.hpp.
    • Added tao/pegtl/contrib/iri.hpp grammar for IRIs.
    • Added tao/pegtl/contrib/proto3.hpp grammar for protocol buffer v3.
    Source code(tar.gz)
    Source code(zip)
  • 3.2.1(Jul 31, 2021)

  • 3.2.0(Jan 15, 2021)

    • Added support for disabling exceptions with -fno-exceptions.
    • Improved efficiency of parse tree nodes.
    • Fixed namespace issue with tao::pegtl::demangle<T>() (was: tao::demangle<T>()).
    Source code(tar.gz)
    Source code(zip)
  • 3.1.0(Dec 17, 2020)

    • Made analyze() more verbose by default to aid finding the rule cycles.
    • Added parse_nested() overload that accepts a position as first argument.
    • Added some experimental and undocumented contrib features and their infrastructure.
    • Improved CMake support for <filesystem> fallbacks and alternatives.
      • Re-enabled support for GCC 7.
      • Automatically link with libstdc++fs or libc++fs as needed.
      • Added automatic fallback from std::filesystem to std::experimental::filesystem.
      • Added manual fallback from std::filesystem to boost::filesystem.
      • Thank you Beman Dawes!
    • Converted continuous integration infrastructure to GitHub Actions.
    Source code(tar.gz)
    Source code(zip)
  • 3.0.0(Nov 28, 2020)

    • Use the migration guide when updating.
    • Infrastructure
      • Updated required C++ standard to C++17.
      • Updated required CMake version to 3.8.
      • The macro TAO_PEGTL_NAMESPACE now contains the fully qualified namespace, e.g. tao::pegtl.
      • Added [[nodiscard]] or [[noreturn]] to most non-void functions.
    • Meta-Data Layer
      • Replaced analysis_t with more general and complete rule_t and subs_t.
      • Added functions to visit all rules of a grammar.
      • Added functions to measure rule coverage of a parsing run.
      • Moved the analysis function and header to contrib.
    • Error Handling
      • Replaced tao::pegtl::input_error with std::system_error and std::filesystem::filesystem_error.
      • Added must_if<>
        • Allows to define custom error messages for global errors.
        • Adds a non-intrusive way to define global parse errors for a grammar retroactively.
    • Demangling
      • Removed the need for RTTI.
        • Some broken/unknown compilers will use RTTI as a fallback, without demangling.
      • Moved tao::pegtl::internal::demangle<T>() to tao::demangle<T>().
      • Improved generated code to be shorter and more efficient.
    • Parse Tree
      • Removed the need for RTTI.
    • Other
      • Changed std::string to std::filesystem::path for filename parameters.
      • Renamed byte_in_line to column and use 1-based counting.
      • Moved rule eolf from inline namespace tao::pegtl::ascii to tao::pegtl.
      • Changed rules in tao/pegtl/contrib/integer.hpp to not accept redundant leading zeros.
      • Added rules to tao/pegtl/contrib/integer.hpp that test unsigned values against a maximum.
      • Demoted UTF-16 and UTF-32 support to contrib.
      • Demoted UINT-8, UINT-16, UINT-32 and UINT-64 support to contrib.
      • Folded contrib/counter.hpp into json_count.cpp, count is superceded by coverage.
      • Removed right padding from contrib/json.hpp's value.
    • Cleanup
      • Removed option of state's S::success() to have an extended signature to get access to the current apply_mode, rewind_mode, action- and control class (template).
      • Removed compatibility macros starting with TAOCPP_PEGTL_.
      • Removed compatibility uppercase enumerators.
      • Removed compatibility peek_byte() member functions.
      • Removed compatibility header changes.hpp from contrib.
    Source code(tar.gz)
    Source code(zip)
  • 2.8.3(Apr 22, 2020)

    • Fixed excessive read-ahead with incremental inputs.
    • Added state manipulators remove_first_state, remove_last_states, rotate_states_right, rotate_states_left, and reverse_states to contrib.
    • Reduced the number of intermediate parse tree nodes.
    Source code(tar.gz)
    Source code(zip)
  • 2.8.2(Apr 4, 2020)

  • 2.8.1(Aug 6, 2019)

    • Added fallback symbol demangling if RTTI is disabled.
    • Fixed missing string_input<> in amalgamated header.
    • Fixed discard_input* actions to properly forward the apply mode.
    • Fixed contrib HTTP grammar for chunked data.
    Source code(tar.gz)
    Source code(zip)
  • 2.8.0(Apr 9, 2019)

    • Use the migration guide when updating.
    • Changed enumerators to lowercase.
      • Renamed tracking_mode::IMMEDIATE to tracking_mode::eager.
      • Compatibility enumerators with uppercase names are still included.
        • Will be removed in version 3.0.0.
    • Renamed peek_byte() to peek_uint8().
      • Compatibility member functions with previous names are still included.
        • Will be removed in version 3.0.0.
    • Allowed actions to implement match.
    • Made deriving action class templates from nothing optional.
    • Added debug tools require_apply and require_apply0.
    • Added combinator class rematch.
    • Improved the Parse Tree / AST interface to mostly hide its internal state.
    • Added new action-based helpers change_*.hpp.
      • The control-based helpers in contrib/changes.hpp are still included.
        • Will be removed in version 3.0.0.
    • Added new action-based helpers disable_action.hpp and enable_action.hpp.
    • Added new action-based helpers discard_input.hpp, discard_input_on_success.hpp, and discard_input_on_failure.hpp.
    • Added Clang Static Analyzer to the CI build.
    • Added new Makefile target amalgamate to generate a single-header version of the PEGTL.
    • Added support for Universal Windows Platform (UWP).
    Source code(tar.gz)
    Source code(zip)
  • 2.7.1(Sep 29, 2018)

    • Added new ASCII convenience rule forty_two.
    • Added experimental if_then rule.
    • Simplified how parse tree nodes can be selected.
    • Reduced the number of intermediate parse tree nodes.
    • Allowed an action class template to be used with the parse tree.
    Source code(tar.gz)
    Source code(zip)
  • 2.7.0(Jul 31, 2018)

  • 2.6.1(Jul 22, 2018)

  • 2.6.0(Jun 23, 2018)

    • Added Conan packages.
    • Fixed the UTF-8 decoder to no longer accept UTF-16 surrogates.
    • Fixed the UTF-16 decoder to no longer accept UTF-16 unmatched surrogates.
    • Fixed the UTF-32 "decoder" to no longer accept UTF-16 surrogates.
    • Fixed pegtl/contrib/unescape.hh to no longer accept unmatched surrogates.
    • Optimised convenience rule two.
    • Added new convenience rule three.
    Source code(tar.gz)
    Source code(zip)
  • 2.5.2(May 31, 2018)

  • 2.5.1(May 14, 2018)

  • 2.5.0(May 1, 2018)

    • Added rules to match Unicode properties via ICU to contrib.
    • Improved the Parse Tree / AST interface.
    • Fixed parse tree node generation to correctly remove intermediate nodes.
    • Added big- and little-endian support to the UTF-16 and UTF-32 rules.
    • Added rules for UINT-8 and big- and little-endian UINT-16, UINT-32 and UINT-64.
    • Added function to memory_input<> to obtain the line around a position.
    • Added function to memory_input<> to start again from the beginning.
    • Added example for Python-style indentation-aware grammars.
    • Added examples for regular, context-free, and context-sensitive grammars.
    • Added example for how to parse with a symbol table.
    • Added automated testing with Clang 6.
    • Added automated testing with Clang's -fms-extensions.
    • Fixed build with Clang when -fms-extensions is used (clang-cl).
    Source code(tar.gz)
    Source code(zip)
  • 2.4.0(Feb 17, 2018)

    • Improved and documented the Parse Tree / AST support.
    • Changed prefix of all macros from TAOCPP_PEGTL_ to TAO_PEGTL_. Compatibility macros with the old names are provided, they will be removed in version 3.0.
    • Added a deleted overload to prevent creating a memory_input<> from a temporary std::string.
    Source code(tar.gz)
    Source code(zip)
  • 2.3.4(Feb 8, 2018)

  • 2.3.3(Jan 1, 2018)

  • 2.3.2(Dec 16, 2017)

  • 2.3.1(Dec 14, 2017)

  • 2.3.0(Dec 11, 2017)

    • Added constructor to read_input<> that accepts a FILE*, see issue #78.
    • Enhanced apply<>, apply0<> and if_apply<> to support apply()/apply0()-methods returning boolean values.
    • Simplified implementation of raw_string, the optional Contents... rules' apply()/apply0()-methods are now called with the original states.
    • Fixed the tracer to work with apply()/apply0()-methods returning boolean values. (Thanks Joel Frederico)
    • Fixed, simplified and improved examples/parse_tree.cpp.
    Source code(tar.gz)
    Source code(zip)
  • 2.2.2(Nov 22, 2017)

    • Fixed missing call to the control class' failure()-method when a rule with an apply()-method with a boolean return type fails.
    • Fixed string handling in examples/abnf2pegtl.cc.
    • Simplified/improved Android build.

    (this is a re-release of 2.2.1 to fix the version number reported by CMake and version.hpp)

    Source code(tar.gz)
    Source code(zip)
  • 2.2.1(Nov 22, 2017)

    • Fixed missing call to the control class' failure()-method when a rule with an apply()-method with a boolean return type fails.
    • Fixed string handling in examples/abnf2pegtl.cc.
    • Simplified/improved Android build.
    Source code(tar.gz)
    Source code(zip)
  • 2.2.0(Sep 24, 2017)

    • Added possibility for actions' apply() or apply0()-methods to return a bool which is then used to determine overall success or failure of the rule to which such an action was attached.
    • Added <tao/pegtl/contrib/parse_tree.hpp> and the examples/parse_tree.cpp application that shows how to build a parse tree. The example goes beyond a traditional parse tree and demonstrates how to select which nodes to include in the parse tree and how to transform the nodes into an AST-like structure.
    • Added bom rules for UTF-8, UTF-16 and UTF-32.
    • Added some missing includes for config.hpp.
    • Added automated testing with Clang 5.
    • Added automated testing with Xcode 9.
    Source code(tar.gz)
    Source code(zip)
Owner
The Art of C++
A collection of high-quality C++ libraries
The Art of C++
BNFLite is a C++ template library for lightweight flexible grammar parsers

BNFLite is a C++ template library for lightweight flexible grammar parsers. BNFLite offers creative approach when the developer can specify a language for further parsing directly in the C++ code. Moreover, such "specifications" are executable now!

Alexander S 59 Jul 27, 2022
This is a C plus plus coding template for Compitative programming. This template is very optimized for the Online Judgment

C-plusplus-compitative-Programming-Template Tech We Used C++ Features Easy to compile Easy debug facility Analysised and optimized base template Steps

Alan Binu 15 Jan 27, 2022
OpenGL Template Engine - a C++ OpenGL graphics engine which aimed to be a simple startup template for 3D OpenGL projects.

OpenGL Template Engine is a C++ OpenGL graphics engine which aimed to be a simple startup template for 3D OpenGL projects. This is the template I personally use for my own projects and provides me with the general OpenGL 3D render setup with model import and UI.

Marcus Nesse Madland 2 May 16, 2022
A portable fork of the high-performance regular expression matching library

Vectorscan? A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD is 100% functional, and Power VSX are in developm

VectorCamp 205 Aug 4, 2022
Tree-sitter grammar for comment tags like TODO, FIXME(user).

Tree-sitter grammar for comment tags like TODO:, FIXME(user):, etc. Useful to be embedded inside comments.

Santos Gallegos 66 Aug 2, 2022
tree-sitter grammar for emacs lisp

Tree-sitter Grammar for Emacs Lisp A simple tree-sitter grammar for elisp. Syntax supported: Atoms (integers, floats, strings, characters, symbols) Li

Wilfred Hughes 20 Jun 17, 2022
Header-only ECMAScript (JavaScript) compatible regular expression engine

SRELL (std::regex-like library) is a regular expression template library for C++ and has native support for UTF-8, UTF-16, and UTF-32. This is up-to-d

Dmitry Atamanov 4 Mar 11, 2022
Love 6's Regular Expression Engine. Support Concat/Select/Closure Basic function. Hope u can enjoy this tiny engine :)

Regex_Engine Love 6's Blog Website: https://love6.blog.csdn.net/ Love 6's Regular Expression Engine Hope u can love my tiny regex engine :) maybe a fe

Love6 2 May 24, 2022
Node.js bindings for the Mathematical Expression Toolkit

ExprTk.js This is the Node.js bindings for ExprTk (Github) by @ArashPartow ExprTk.js supports both synchronous and asynchronous background execution o

Momtchil Momtchev 6 Jul 23, 2022
A C++ expression -> x64 JIT

NativeJIT NativeJIT is an open-source cross-platform library for high-performance just-in-time compilation of expressions involving C data structures.

null 1.1k Aug 6, 2022
A single-header C/C++ library for parsing and evaluation of arithmetic expressions

ceval A C/C++ header for parsing and evaluation of arithmetic expressions. [README file is almost identical to that of the ceval library] Functions ac

e_t 7 Apr 14, 2022
A single-header C/C++ library for parsing and evaluation of arithmetic expressions

ceval A C/C++ header for parsing and evaluation of arithmetic expressions. [README file is almost identical to that of the ceval library] Functions ac

e_t 7 Apr 14, 2022
An 802.11 Frame Generation and Parsing Library in C

libwifi 802.11 Parsing / Generation library Build Status OS Architecture Linux x86_64 What is this? libwifi is a C library with a permissive license f

null 29 Aug 8, 2022
Just a basic mini library for parsing simple files that only have variables written and with Lua extension.

C++ Parser Lua file config Just a basic mini library for parsing simple files that only have variables written and with Lua extension. Note: At the mo

Marcos Oliveira 3 Dec 26, 2021
ZSV/lib: a fast CSV parsing library and standalone utility

Please note: this code is still alpha / pre-production. Everything here should be considered preliminary. If you like ZSVlib, please give it a star! Z

null 64 Jul 9, 2022
A C/C++ library for parsing and evaluation of arithmetic expressions.

ceval A C/C++ header for parsing and evaluation of arithmetic expressions. Functions accessibe from main() Function Argument(s) Return Value ceval_res

e_t 4 Jul 20, 2022
A lightweight C++14 parsing library for tmx map files created with the Tiled map editor

tmxlite Description A lightweight C++14 parsing library for tmx map files created with the Tiled map editor. Requires no external linking, all depende

Matt Styles 317 Aug 6, 2022
This project contains a library for C++ AST parsing, metaprogramming and reflection

Meta C++ This project contains a library for C++ AST parsing, metaprogramming and reflection. Also included is a tool for generating the necessary met

Keith Hammond 75 Jun 26, 2022
Boilerplate-free YAML parsing for C++

AutoYAML AutoYAML is a Clang/LLVM/LibTooling-based program that automatically generates yaml-cpp conversion code for C++ record types. Usage As a moti

Timo Nicolai 10 Nov 12, 2021