Parsing Expression Grammar Template Library

Last update: Jan 8, 2023

Overview

Welcome to the PEGTL

The Parsing Expression Grammar Template Library (PEGTL) is a zero-dependency C++ header-only parser combinator library for creating parsers according to a Parsing Expression Grammar (PEG).

Documentation

Changelog
Version 3.x (requires C++17)
Version 2.x (requires C++11)
Version 1.x (requires C++11)

Contact

For questions and suggestions regarding the PEGTL, success or failure stories, and any other kind of feedback, please feel free to join our Discord server, open a discussion, an issue or a pull request on GitHub or contact the authors at taocpp(at)icemx.net.

Introduction

Grammars are written as regular C++ code, created with template programming (not template meta programming), i.e. nested template instantiations that naturally correspond to the inductive definition of PEGs (and other parser-combinator approaches).

A comprehensive set of parser rules that can be combined and extended by the user is included, as are mechanisms for debugging grammars, and for attaching user-defined actions to grammar rules. Here is an example of how a PEG grammar rule is implemented as C++ class with the PEGTL.

// PEG rule for integers consisting of a non-empty
// sequence of digits with an optional sign:

// sign ::= '+' / '-'
// integer ::= sign? digit+

// The same parsing rule implemented with the PEGTL:

using namespace tao::pegtl;

struct sign : one< '+', '-' > {};
struct integer : seq< opt< sign >, plus< digit > > {};

PEGs are superficially similar to Context-Free Grammars (CFGs), however the more deterministic nature of PEGs gives rise to some very important differences. The included grammar analysis finds several typical errors in PEGs, including left recursion.

Design

The PEGTL is designed to be "lean and mean", the core library consists of approximately 6000 lines of code. Emphasis is on simplicity and efficiency, preferring a well-tuned simple approach over complicated optimisations.

The PEGTL is mostly concerned with parsing combinators and grammar rules, and with giving the user of the library (the possibility of) full control over all other aspects of a parsing run. Whether/which actions are taken, and whether/which data structures are created during a parsing run, is entirely up to the user.

Included are some examples for typical situation like unescaping escape sequences in strings, building a generic JSON data structure, and on-the-fly evaluation of arithmetic expressions.

Through the use of template programming and template specialisations it is possible to write a grammar once, and use it in multiple ways with different (semantic) actions in different (or the same) parsing runs.

With the PEG formalism, the separation into lexer and parser stages is usually dropped -- everything is done in a single grammar. The rules are expressed in C++ as template instantiations, and it is the compiler's task to optimise PEGTL grammars.

Status

Each commit is automatically tested with multiple architectures, operating systems, compilers, and versions thereof.

Each commit is checked with GCC's and Clang's sanitizers, Clang's Static Analyzer, and clang-tidy. Additionally, we use CodeQL to scan for (security) issues.

Code coverage is automatically measured and the unit tests cover 100% of the core library code (for releases).

Releases are done in accordance with Semantic Versioning. Incompatible API changes are only allowed to occur between major versions.

Thank You

In appreciation of all contributions here are the people that have directly contributed to the PEGTL and/or its development.

The Art of C++

The PEGTL is part of The Art of C++.

License

The PEGTL is certified Open Source software. It may be used for any purpose, including commercial purposes, at absolutely no cost. It is distributed under the terms of the MIT license reproduced here.

Copyright (c) 2007-2021 Dr. Colin Hirsch and Daniel Frey

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Comments

Switch license from MIT to BSL-1.0

Dear contributors,

we would like to switch from the MIT license to the Boost Software License, Version 1.0. Both are very similar with one crucial difference: The attribution clause of the BSL-1.0 has an exception for binaries. I'm not a lawyer, so do your own research if you want to get into the details.

The motivation for our change is to allow the use of the PEGTL in more environments, especially when it is difficult or even impossible for the binary to output the license. Think pacemaker-like device, but please no one use the PEGTL in pacemakers... 😅

In order to do the switch properly, we need the consent from everyone who contributed something that is meeting a certain threshold of originality. That threshold is required for your work to fall under copyright. As mentioned I am not a lawyer and I'd like to avoid having to judge your contributions. Therefore, I am simply asking you to give your permission to switch the license from MIT to BSL-1.0. You can do so by replying to this issue and clearly stating that you agree to the license change.

As we are also planning on switching our other projects, and if you have contributed to those other projects, it would be helpful if you could explicitly mention in your answer whether you give your permission only for the PEGTL or for all your contributions to our project, "The Art of C++".

Please also reply if you do not give permission to switch. It is better for us to have an explicit answer than having to guess.

Thank you for your contributions!

The Art of C++ Team: @d-frey @ColinH @uilianries

Pinging everyone from GitHub's list of contributors, in the order they appeared from GitHub:

@wravery @pleroux0 @Bjoe @studoot @sanssecours @samhocevar @joelfrederico @kelvinhammond @pauloscustodio @zhihaoy @redmercury @kuzmas @ras0219 @Amphaal @robertcampion @michael-brade @barbieri @delpinux @hutorny @andoma @lichray @jbomanson @mkrupcale @quadfault @bwagner @jubnzv @lambdafu @dkopecek @JohelEGP
question feedback please!

opened by d-frey 28
Fix or nix Android CI builds

Currently, the Andriod builds are broken due to using std::filesystem::path. On some non-Android builds this was fixed by adding -llibstdc++fs, on some builds it just worked. Android, however, seems to be too old in some cases and I wouldn't know how to add the above option (in case it is required and helps). Also, the version we are using now seem to be a bit outdated.

We therefore need to either fix (and possibly modernize the build) or I will have to remove them (and most likely Android is then broken).

Any help is appreciated, @Bjoe ;)
bug help wanted

opened by d-frey 21
Support Conan

Related issue: #111

I dont have permission to create branches on taocpp, because I have member role. But no problem, I just forked the project.

This PR just update the support to use Conan + PEGTL

I added a job for Travis and Appeveyor to validate on both platforms

Why I copied the files instead to use cmake helper?
The cmake helper requires settings.compiler when run over Windows, but this package is header-only and no settings is required. In the future I'll provide FindPEGTL.cmake file, so don't worry.

The next step is update Travis settings. I don't have permission to change the project settings on Travis :( But you need to add CONAN_LOGIN_USERNAME as taocpp-user, and CONAN_PASSWORD with Key API provided by Bintray.

Regards!

opened by uilianries 21

more useful tracer

[ provides tracer with easier color toggle and more detailed location reporting ]

this expands the tracer to provide explicit color control for easy configuration for terminal/file output which can be useful when redirecting to file via terminal > or >>

# --no-color is not interpereted by pegtl itself
#   it is instead interperated by test_trace.cpp main
#   and causes the no_color variant to be called instead
$ ./test_trace --no-color > out.txt

    trace_standard // defaults to trace_standard_color
    trace_standard_color
    trace_standard_no_color

    trace_complete // defaults to trace_complete_color
    trace_complete_color
    trace_complete_no_color

sample configuration

// default color output
tao::pegtl::complete_trace<grammar, action>(in, data);

// color output
tao::pegtl::complete_trace_color<grammar, action>(in, data);

// no color output
tao::pegtl::complete_trace_no_color<grammar, action>(in, data);

this also expands the tracer to include the source line and cursor position at which the rule is being matched against

in the previous tracer, this is the current output

sample output

        position GLSL preprocessor stage 3:1:1
#1      GLSL_PREPROCESSOR::grammar
#2        GLSL_PREPROCESSOR::reset_preprocessor_state
          apply
          success
#3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
#4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#5            GLSL_PREPROCESSOR::whitespaces_and_newlines
#6              GLSL_PREPROCESSOR::spaces
                failure
#7              GLSL_PREPROCESSOR::tabs
                failure
#8              GLSL_PREPROCESSOR::newlines
                failure
              failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
#9            GLSL_PREPROCESSOR::identifier
#10             tao::pegtl::ascii::identifier_first
                success
                position GLSL preprocessor stage 3:1:2
#11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
#12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                  success
                  position GLSL preprocessor stage 3:1:4
                success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
              apply
              success #9 GLSL_PREPROCESSOR::identifier
            success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#14           GLSL_PREPROCESSOR::whitespaces_and_newlines
#15             GLSL_PREPROCESSOR::spaces
                success
                position GLSL preprocessor stage 3:1:5
#16             GLSL_PREPROCESSOR::spaces
                failure
#17             GLSL_PREPROCESSOR::tabs
                failure
#18             GLSL_PREPROCESSOR::newlines
                failure
              apply
              success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
            success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#20           GLSL_PREPROCESSOR::whitespaces_and_newlines
#21             GLSL_PREPROCESSOR::spaces
                failure
#22             GLSL_PREPROCESSOR::tabs
                failure
#23             GLSL_PREPROCESSOR::newlines
                failure
              failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
#24           GLSL_PREPROCESSOR::identifier
#25             tao::pegtl::ascii::identifier_first
                failure
              failure #24 GLSL_PREPROCESSOR::identifier
            failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
          success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
#26       tao::pegtl::eof
          failure
        failure #1 GLSL_PREPROCESSOR::grammar
        position GLSL preprocessor stage 3:1:1

this makes debugging difficult as you cannot see the input stream, nor the input that is about to processed, and instead have to try to decipher and track the given line numbers and columns to try to make sense of where your grammar may wrong

this can be difficult especially if the input is memory or a stream such as stdin or similar or any other input in which you do not easily have access to the source being parsed

so i added the current source and current cursor position underneath the source position

        position GLSL preprocessor stage 3:1:1
        source int 6 main
               ^

in the above, the grammar will attempt to match starting at i

        position GLSL preprocessor stage 3:1:3
        source int 6 main
                 ^

in the above, the grammar will attempt to match starting at t

sample output

        position GLSL preprocessor stage 3:1:1
        source int 6 main
               ^
#1      GLSL_PREPROCESSOR::grammar
#2        GLSL_PREPROCESSOR::reset_preprocessor_state
          apply
          success
#3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
#4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#5            GLSL_PREPROCESSOR::whitespaces_and_newlines
#6              GLSL_PREPROCESSOR::spaces
                failure
#7              GLSL_PREPROCESSOR::tabs
                failure
#8              GLSL_PREPROCESSOR::newlines
                failure
              failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
#9            GLSL_PREPROCESSOR::identifier
#10             tao::pegtl::ascii::identifier_first
                success
                position GLSL preprocessor stage 3:1:2
                source int 6 main
                        ^
#11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
#12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                  success
                  position GLSL preprocessor stage 3:1:4
                  source int 6 main
                            ^
                success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
              apply
              success #9 GLSL_PREPROCESSOR::identifier
            success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#14           GLSL_PREPROCESSOR::whitespaces_and_newlines
#15             GLSL_PREPROCESSOR::spaces
                success
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#16             GLSL_PREPROCESSOR::spaces
                failure
#17             GLSL_PREPROCESSOR::tabs
                failure
#18             GLSL_PREPROCESSOR::newlines
                failure
              apply
              success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
            success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#20           GLSL_PREPROCESSOR::whitespaces_and_newlines
#21             GLSL_PREPROCESSOR::spaces
                failure
#22             GLSL_PREPROCESSOR::tabs
                failure
#23             GLSL_PREPROCESSOR::newlines
                failure
              failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
#24           GLSL_PREPROCESSOR::identifier
#25             tao::pegtl::ascii::identifier_first
                failure
              failure #24 GLSL_PREPROCESSOR::identifier
            failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
          success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
#26       tao::pegtl::eof
          failure
        failure #1 GLSL_PREPROCESSOR::grammar
        position GLSL preprocessor stage 3:1:1
        source int 6 main
               ^

this makes debugging grammar easier as you can now see what the input stream looks like and exactly where in the stream the grammar is attempting to match

i did consider adding the same information to both success and failure

sample output

        position GLSL preprocessor stage 3:1:1
        source int 6 main
               ^
#1      GLSL_PREPROCESSOR::grammar
#2        GLSL_PREPROCESSOR::reset_preprocessor_state
          apply
          success
          position GLSL preprocessor stage 3:1:1
          source int 6 main
                 ^
#3        tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
#4          tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#5            GLSL_PREPROCESSOR::whitespaces_and_newlines
#6              GLSL_PREPROCESSOR::spaces
                failure
                position GLSL preprocessor stage 3:1:1
                source int 6 main
                       ^
#7              GLSL_PREPROCESSOR::tabs
                failure
                position GLSL preprocessor stage 3:1:1
                source int 6 main
                       ^
#8              GLSL_PREPROCESSOR::newlines
                failure
                position GLSL preprocessor stage 3:1:1
                source int 6 main
                       ^
              failure #5 GLSL_PREPROCESSOR::whitespaces_and_newlines
              position GLSL preprocessor stage 3:1:1
              source int 6 main
                     ^
#9            GLSL_PREPROCESSOR::identifier
#10             tao::pegtl::ascii::identifier_first
                success
                position GLSL preprocessor stage 3:1:2
                source int 6 main
                        ^
#11             tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
#12               tao::pegtl::skip<tao::pegtl::ascii::identifier_other>
                  success
                  position GLSL preprocessor stage 3:1:4
                  source int 6 main
                            ^
                success #11 tao::pegtl::opt<tao::pegtl::skip<tao::pegtl::ascii::identifier_other> >
                position GLSL preprocessor stage 3:1:4
                source int 6 main
                          ^
              apply
              success #9 GLSL_PREPROCESSOR::identifier
              position GLSL preprocessor stage 3:1:4
              source int 6 main
                        ^
            success #4 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
            position GLSL preprocessor stage 3:1:4
            source int 6 main
                      ^
#13         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#14           GLSL_PREPROCESSOR::whitespaces_and_newlines
#15             GLSL_PREPROCESSOR::spaces
                success
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#16             GLSL_PREPROCESSOR::spaces
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#17             GLSL_PREPROCESSOR::tabs
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#18             GLSL_PREPROCESSOR::newlines
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
              apply
              success #14 GLSL_PREPROCESSOR::whitespaces_and_newlines
              position GLSL preprocessor stage 3:1:5
              source int 6 main
                         ^
            success #13 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
            position GLSL preprocessor stage 3:1:5
            source int 6 main
                       ^
#19         tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
#20           GLSL_PREPROCESSOR::whitespaces_and_newlines
#21             GLSL_PREPROCESSOR::spaces
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#22             GLSL_PREPROCESSOR::tabs
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
#23             GLSL_PREPROCESSOR::newlines
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
              failure #20 GLSL_PREPROCESSOR::whitespaces_and_newlines
              position GLSL preprocessor stage 3:1:5
              source int 6 main
                         ^
#24           GLSL_PREPROCESSOR::identifier
#25             tao::pegtl::ascii::identifier_first
                failure
                position GLSL preprocessor stage 3:1:5
                source int 6 main
                           ^
              failure #24 GLSL_PREPROCESSOR::identifier
              position GLSL preprocessor stage 3:1:5
              source int 6 main
                         ^
            failure #19 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>
            position GLSL preprocessor stage 3:1:5
            source int 6 main
                       ^
          success #3 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces_and_newlines, GLSL_PREPROCESSOR::identifier>>
          position GLSL preprocessor stage 3:1:5
          source int 6 main
                     ^
#26       tao::pegtl::eof
          failure
          position GLSL preprocessor stage 3:1:5
          source int 6 main
                     ^
        failure #1 GLSL_PREPROCESSOR::grammar
        position GLSL preprocessor stage 3:1:1
        source int 6 main
               ^

but as you can see it kinda spams the log with info you already know and at this point it is largely duplicate info for most rules that fail to match, so i stuck with only showing it when ever the position changes, which is and was the default

Source

namespace tao::pegtl
{
    template< bool HideInternal = false, bool UseColor = true, std::size_t IndentIncrement = 2, std::size_t InitialIndent = 8 >
    struct tracer_traits
    {
        template< typename Rule >
        static constexpr bool enable = ( HideInternal ? tao::pegtl::normal< Rule >::enable : true );

        static constexpr std::size_t initial_indent = InitialIndent;
        static constexpr std::size_t indent_increment = IndentIncrement;

        static constexpr std::string_view ansi_reset = UseColor ? "\033[m" : "";
        static constexpr std::string_view ansi_rule = UseColor ? "\033[36m" : "";
        static constexpr std::string_view ansi_hide = UseColor ? "\033[37m" : "";

        static constexpr std::string_view ansi_position = UseColor ? "\033[1;34m" : "";
        static constexpr std::string_view ansi_success = UseColor ? "\033[32m" : "";
        static constexpr std::string_view ansi_failure = UseColor ? "\033[31m" : "";
        static constexpr std::string_view ansi_raise = UseColor ? "\033[1;31m" : "";
        static constexpr std::string_view ansi_unwind = UseColor ? "\033[31m" : "";
        static constexpr std::string_view ansi_apply = UseColor ? "\033[1;36m" : "";
    };

    using standard_tracer_traits_color = tracer_traits< true, true >;
    using standard_tracer_traits_no_color = tracer_traits< true, false >;
    using complete_tracer_traits_color = tracer_traits< false, true >;
    using complete_tracer_traits_no_color = tracer_traits< false, false >;

    template< typename TracerTraits >
    struct tracer
    {
        const std::ios_base::fmtflags m_flags;
        std::size_t m_count = 0;
        std::vector< std::size_t > m_stack;
        tao::pegtl::position m_position;

        template< typename Rule >
        static constexpr bool enable = TracerTraits::template enable< Rule >;

        template< typename ParseInput >
        explicit tracer( const ParseInput& in )
                : m_flags( std::cerr.flags() ),
                  m_position( in.position() )
        {
            std::cerr << std::left;
            print_position(in);
        }

        tracer( const tracer& ) = delete;
        tracer( tracer&& ) = delete;

        ~tracer()
        {
            std::cerr.flags( m_flags );
        }

        tracer& operator=( const tracer& ) = delete;
        tracer& operator=( tracer&& ) = delete;

        [[nodiscard]] std::size_t indent() const noexcept
        {
            return TracerTraits::initial_indent + TracerTraits::indent_increment * m_stack.size();
        }

        template <typename ParseInput>
        void print_position(const ParseInput& in) const
        {
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_position << "position" << TracerTraits::ansi_reset << ' ' << m_position << '\n';
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_position << "source" << TracerTraits::ansi_reset << " " << in.line_at(m_position) << "\n";
            std::cerr << std::setw( indent() + 6 + m_position.column ) << ' ' << '^' << '\n';
        }

        template <typename ParseInput>
        void update_position( const ParseInput& in )
        {
            const tao::pegtl::position& p = in.position();
            if( m_position != p ) {
                m_position = p;
                print_position(in);
            }
        }

        template< typename Rule, typename ParseInput, typename... States >
        void start( const ParseInput& /*unused*/, States&&... /*unused*/ )
        {
            std::cerr << '#' << std::setw( indent() - 1 ) << ++m_count << TracerTraits::ansi_rule << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset << '\n';
            m_stack.push_back( m_count );
        }

        template< typename Rule, typename ParseInput, typename... States >
        void success( const ParseInput& in, States&&... /*unused*/ )
        {
            const auto prev = m_stack.back();
            m_stack.pop_back();
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_success << "success" << TracerTraits::ansi_reset;
            if( m_count != prev ) {
                std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
            }
            std::cerr << '\n';
            update_position( in );
        }

        template< typename Rule, typename ParseInput, typename... States >
        void failure( const ParseInput& in, States&&... /*unused*/ )
        {
            const auto prev = m_stack.back();
            m_stack.pop_back();
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_failure << "failure" << TracerTraits::ansi_reset;
            if( m_count != prev ) {
                std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
            }
            std::cerr << '\n';
            update_position( in );
        }

        template< typename Rule, typename ParseInput, typename... States >
        void raise( const ParseInput& /*unused*/, States&&... /*unused*/ )
        {
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_raise << "raise" << TracerTraits::ansi_reset << ' ' << TracerTraits::ansi_rule << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset << '\n';
        }

        template< typename Rule, typename ParseInput, typename... States >
        void unwind( const ParseInput& in, States&&... /*unused*/ )
        {
            const auto prev = m_stack.back();
            m_stack.pop_back();
            std::cerr << std::setw( indent() ) << ' ' << TracerTraits::ansi_unwind << "unwind" << TracerTraits::ansi_reset;
            if( m_count != prev ) {
                std::cerr << " #" << prev << ' ' << TracerTraits::ansi_hide << tao::pegtl::demangle< Rule >() << TracerTraits::ansi_reset;
            }
            std::cerr << '\n';
            update_position( in );
        }

        template< typename Rule, typename ParseInput, typename... States >
        void apply( const ParseInput& /*unused*/, States&&... /*unused*/ )
        {
            std::cerr << std::setw( static_cast< int >( indent() - TracerTraits::indent_increment ) ) << ' ' << TracerTraits::ansi_apply << "apply" << TracerTraits::ansi_reset << '\n';
        }

        template< typename Rule, typename ParseInput, typename... States >
        void apply0( const ParseInput& /*unused*/, States&&... /*unused*/ )
        {
            std::cerr << std::setw( static_cast< int >( indent() - TracerTraits::indent_increment ) ) << ' ' << TracerTraits::ansi_apply << "apply0" << TracerTraits::ansi_reset << '\n';
        }

        template< typename Rule,
                template< typename... > class Action = tao::pegtl::nothing,
                template< typename... > class Control = tao::pegtl::normal,
                typename ParseInput,
                typename... States >
        bool parse( ParseInput&& in, States&&... st )
        {
            return tao::pegtl::parse< Rule, Action, tao::pegtl::state_control< Control >::template type >( in, st..., *this );
        }
    };

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool standard_trace_color( ParseInput&& in, States&&... st )
    {
        tracer< standard_tracer_traits_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool standard_trace( ParseInput&& in, States&&... st )
    {
        tracer< standard_tracer_traits_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool standard_trace_no_color( ParseInput&& in, States&&... st )
    {
        tracer< standard_tracer_traits_no_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool complete_trace_color( ParseInput&& in, States&&... st )
    {
        tracer< complete_tracer_traits_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool complete_trace( ParseInput&& in, States&&... st )
    {
        tracer< complete_tracer_traits_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Rule,
            template< typename... > class Action = tao::pegtl::nothing,
            template< typename... > class Control = tao::pegtl::normal,
            typename ParseInput,
            typename... States >
    bool complete_trace_no_color( ParseInput&& in, States&&... st )
    {
        tracer< complete_tracer_traits_no_color > tr( in );
        return tr.parse< Rule, Action, Control >( in, st... );
    }

    template< typename Tracer >
    struct trace
            : tao::pegtl::maybe_nothing
    {
        template< typename Rule,
                tao::pegtl::apply_mode A,
                tao::pegtl::rewind_mode M,
                template< typename... >
                class Action,
                template< typename... >
                class Control,
                typename ParseInput,
                typename... States >
        [[nodiscard]] static bool match( ParseInput& in, States&&... st )
        {
            if constexpr( sizeof...( st ) == 0 ) {
                return tao::pegtl::match< Rule, A, M, Action, tao::pegtl::state_control< Control >::template type >( in, st..., Tracer( in ) );
            }
            else if constexpr( !std::is_same_v< std::tuple_element_t< sizeof...( st ) - 1, std::tuple< States... > >, Tracer& > ) {
                return tao::pegtl::match< Rule, A, M, Action, tao::pegtl::state_control< Control >::template type >( in, st..., Tracer( in ) );
            }
            else {
                return tao::pegtl::match< Rule, A, M, Action, Control >( in, st... );
            }
        }
    };

    using trace_standard_color = trace< tracer< standard_tracer_traits_color > >;
    using trace_standard = trace_standard_color;
    using trace_standard_no_color = trace< tracer< standard_tracer_traits_no_color > >;
    using trace_complete_color = trace< tracer< complete_tracer_traits_color > >;
    using trace_complete = trace_complete_color;
    using trace_complete_no_color = trace< tracer< complete_tracer_traits_no_color > >;

}  // namespace tao::pegtl

enhancement

opened by mgood7123 17

Need hierarchical action handling

I'm writing a parser for the Verilog language using PEGTL and wanted to share a problem I'm running into and a possible solution. This is my first time using PEG-style parsing, although I am far from a novice at parsing, having done work for my PhD thesis on approximate parsing in a network security context (google "flowsifter").

I can appreciate the simplicity and elegance (and incredible run-time efficiency) of having actions associated with each rule run when that rule is matched; this combines with the control system to make a very powerful way to write code that gets triggered by parsing. But I'm particularly annoyed at how unnecessarily difficult it is to write side-effect heavy code that works correctly without these pieces of code having a hierarchical context to work in.

In the best parsers I've worked with (and written), when part of the grammar matches, it can return a value to the next level up, and that value can be used as part of constructing the result of that higher-level rule's parse result. This ends up resulting in a sequence of function calls that are nested in exactly the same structure as the parse tree of the text being parsed. In PEGTL, I can decide exactly what code runs when a particular rule matches, but I get no nesting of function calls, I only get a flat space of code executions. Of course it's possible to implement that hierarchy by using an explicit stack and pushing and popping from the stack as rules are matched, but this has a high degree of complexity and is prone to user error. The expression parser example in this repository even goes as far as using a stack of stacks to handle parentheses in expressions.

There's got to be a reasonable way to stitch together rules and actions in such a way that actions can return a value and the action for a rule receives/can access values produced by child rules. Maybe this will require each child rule to have an action that returns a value; that seems a reasonable price to pay for this feature.
enhancement

opened by thelema 17
Replace try/catch/rethrow in unwind with unwind_guard

I noticed while investigating https://github.com/microsoft/cppgraphqlgen/issues/222 that parse_tree implements unwind on the make_control<>::state_handler<> struct, and this results in creating a try/catch/rethrow block around every rule match in match_control_unwind. Every try/catch/rethrow seems to be consuming significant stack space, so if you raise/throw an exception from a relatively shallow (< 20) nested rule, it often exhausts the stack.

This change makes it so parse_tree will only implement unwind on the Control type wrapper if either the Control or the Node type implements it. When parsing into a tree that doesn't implement either of them, the exception will be thrown once all the way from the initial raise to the caller of parse_tree::parse.

Since the unit tests use the default node which inherits basic_node, they still implement unwind. The unit tests also wrap the rule in a try_catch_type handler, though, so the exception doesn't bubble all the way up. Before I nailed down the constexpr bool definitions, I accidentally suppressed unwind for basic_node types in the unit tests as well, and the lack of stack.pop_back() calls resulted in the assert(stack.size() == 1) firing for the unit test. So that tells me this is incompatible with try_catch_type. That's the only thing I think might block it.
enhancement

opened by wravery 16

line_at not finding line ending when inside of trace.hpp

im using the following

pegtl::memory_input in(spec, "Specification test");

but in.line_at( p ) does not seem to correctly acquire the line

#5        tao::pegtl::ascii::any]
          success
          position Specification test:2:4
          source VERTEX_CHAIN {
    vec4 position;
    vec4 textureCoordinates;
};

FRAGMENT_CHAIN {
    vec4 color;
}

even if i change the Eol from eol::cr_crlf to eol::lf for all input classes it does not make a difference

question

opened by mgood7123 16

Added support for fixed-length list.

I made a list which supports both dynamic and fixed length. It should be backward compatible.

Also, the ifdef was added in rules.hpp for demonstration purpose. This should be removed after some discussion.

opened by qawbecrdtey 15
Add final argument for internal `parse_tree` state to `make_control` functions

Since we know that make_control is going to be given the internal state argument as its last argument from parse_tree::parse, we can explicitly take it as the function argument. This removes the need for using tuple methods and allows us to only pass on the user-provided state arguments to the underlying base Control class. Thus, any external Action or Control functions will also not have to accept the internal parse_tree state argument.

The Control match function is special, however, because it must use make_control< Rule >::match rather than Control< Rule >::match so that it can correctly invoke and pass the internal parse_tree state to the other make_control< Rule >::(start|success|failure|raise|apply0|apply) functions through duseltronik.

Note that this PR also results in performance improvements on the order of 5-10% for some simple parse_tree calculator benchmarks I've run. This is likely due to the reduced complexity by not using the tuple tie and get functions to create a tuple to extract the last variadic template argument.
enhancement

opened by mkrupcale 14
Automatically fallback to older versions of std::filesystem

This is addressing the discussion in #216. Along with the build and code changes I created a GitHub Actions CI/PR workflow to test it on Ubuntu 16.04 and 18.04, with gcc 7 and 8 on each of them. The 2 versions of Ubuntu work more or less the same way, but the gcc 7 jobs fall back to std::experimental::filesystem. The gcc 8 jobs detect that they need to link with -lstdc++fs, but they still use std::filesystem.

We might be able to add another job to do an Android build and test out the Boost.Filesystem fallback, which would also fix the original problem with #216.

opened by wravery 13
add missing virtual dtors

Fixes clang warnings /usr/include/c++/v1/memory:3710:5: error: destructor called on non-final 'examples::null_json' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor] _data.second().~_Tp(); ^

Signed-off-by: Khem Raj [email protected]

opened by kraj 13

Why do I have an infinite loop?

I am currently trying to write a parser for the Franca IDL. For some reason the grammar has an infinite loop. But I cannot figure out whats wrong with it. I cannot see any cycle in the grammar.

struct FrancaIdentifier
  : tao::pegtl::star<
      tao::pegtl::sor<
        tao::pegtl::alnum, tao::pegtl::one<'_'>
      >
    >
  { };

struct FrancaTypeIdentifier
  : tao::pegtl::star<
      tao::pegtl::sor<
        tao::pegtl::alnum, tao::pegtl::one<'_'>, tao::pegtl::one<'.'>
      >
    >
  { };

struct FrancaComment
  : tao::pegtl::sor<
      tao::pegtl::seq<
        tao::pegtl::two<'/'>, tao::pegtl::until<tao::pegtl::eolf>
      >,
      tao::pegtl::seq<
        tao::pegtl::string<'/','*'>, tao::pegtl::until<tao::pegtl::string<'*','/'>>
      >
    >
  { };

struct FrancaIgnore
  : tao::pegtl::sor<
      FrancaComment, tao::pegtl::space
    >
  { };

template<typename Rule>
struct FrancaPadded
  : tao::pegtl::pad<Rule,
      FrancaIgnore
    >
  { };

struct FrancaArgument
  : tao::pegtl::seq<
      FrancaPadded<FrancaTypeIdentifier>, FrancaPadded<FrancaIdentifier>
    >
  { };

template<typename Keyword>
struct FrancaArgumentsList
  : tao::pegtl::seq<
      FrancaPadded<Keyword>,
      tao::pegtl::one<'{'>,
        tao::pegtl::plus<FrancaArgument>,
      tao::pegtl::one<'}'>
    >
  { };

struct FrancaInputArguments
  : FrancaArgumentsList<tao::pegtl::keyword<'i','n'>>
  { };

struct FrancaOutputArguments
  : FrancaArgumentsList<tao::pegtl::keyword<'o','u','t'>>
  { };

struct FrancaMethod
  : tao::pegtl::seq<
      FrancaPadded<tao::pegtl::keyword<'m','e','t','h','o','d'>>,
      FrancaPadded<FrancaIdentifier>,
      tao::pegtl::one<'{'>,
      tao::pegtl::opt<FrancaInputArguments>,
      tao::pegtl::opt<FrancaOutputArguments>,
      tao::pegtl::one<'}'>
    >
  { };

struct FrancaInterface
  : tao::pegtl::seq<
      FrancaPadded<tao::pegtl::keyword<'i','n','t','e','r','f','a','c','e'>>,
      tao::pegtl::one<'{'>,
      tao::pegtl::plus<tao::pegtl::sor<FrancaMethod>>,
      tao::pegtl::one<'}'>>
  { };

struct FrancaGrammar
  : tao::pegtl::seq<
      FrancaInterface, tao::pegtl::eof
    >
  { };

The analyze function prints the following:

WARNING: Possible cycle without progress at rule tao::pegtl::opt<tao::pegtl::plus<franca::FrancaArgument> >
- involved (transformed) rule: tao::pegtl::opt<tao::pegtl::plus<franca::FrancaArgument> >
- involved (transformed) rule: tao::pegtl::plus<franca::FrancaArgument>

But I don't see such a rule in my grammar definition. Would be happy if someone can help me out.

question

opened by dprogm 4

parse_tree needs to be optimized

I wrote a script language parser half a year ago. At that time, it took 12 seconds to parse 300000 lines of 27mb code. The efficiency was too low. I tried to optimize parse_tree yesterday, the efficiency is increased by 200%, and it only takes 4 seconds to complete the parsing. Although the efficiency cannot meet my requirements, this optimization scheme should be told to you

parse_tree2.zip
question

opened by w4454962 8

Error propagation

would it be possible to modify parse_error to support error probogation?

i have a basic version working

parse_error.hpp

namespace TAO_PEGTL_NAMESPACE
{
   namespace internal
   {
      class parse_error
      {
      private:
         std::string m_src;

// ...

         explicit parse_error( const char* where, const char* msg )
            : m_src(where), m_msg( msg )
         {}

         explicit parse_error( const std::string& where, const char* msg )
            : m_src(where), m_msg( msg )
         {}

         [[nodiscard]] const char* where() const noexcept
         {
            return m_src.c_str();
         }

// ...

      };

   }  // namespace internal

   class parse_error
      : public std::runtime_error
   {
// ...

      parse_error( const char* where, const char* msg, position p )
         : std::runtime_error( msg ),
           m_impl( std::make_shared< internal::parse_error >(where, msg ) )
      {
         m_impl->add_position( std::move( p ) );
      }

      parse_error( const std::string& where, const char* msg, position p )
         : std::runtime_error( msg ),
           m_impl( std::make_shared< internal::parse_error >(where.c_str(), msg ) )
      {
         m_impl->add_position( std::move( p ) );
      }

      parse_error( const std::string_view& where, const char* msg, position p )
         : std::runtime_error( msg ),
           m_impl( std::make_shared< internal::parse_error >(std::string(where.data(), where.size()), msg ) )
      {
         m_impl->add_position( std::move( p ) );
      }

      parse_error( const char* where, const std::string& msg, position p )
         : parse_error( where, msg.c_str(), std::move( p ) )
      {}

      parse_error( const std::string& where, const std::string& msg, position p )
         : parse_error( where.c_str(), msg.c_str(), std::move( p ) )
      {}

      parse_error( const std::string_view& where, const std::string& msg, position p )
         : parse_error( std::string(where.data(), where.size()), msg.c_str(), std::move( p ) )
      {}

      template< typename ParseInput >
      parse_error( const char* msg, const ParseInput& in )
         : parse_error( in.line_at(in.position()), msg, in.position() )
      {}

      template< typename ParseInput >
      parse_error( const std::string& msg, const ParseInput& in )
         : parse_error( in.line_at(in.position()), msg, in.position() )
      {}

// ...

      [[nodiscard]] const char* where() const noexcept
      {
         return m_impl->where();
      }

// ...

   };

}  // namespace TAO_PEGTL_NAMESPACE

action_input.hpp

// ...
      [[nodiscard]] std::string_view line_at( const TAO_PEGTL_NAMESPACE::position& p ) const noexcept
      {
         return input().line_at(p);
      }
// ...

the source line is obtainable via e.where() and should replace in.line_at(p) or in.line_at(e.positions().front())

the source line is automatically set when given a ParseInput and can be set manually via where prefix as where, msg, pos instead of msg, pos

such enables correct source line probogation when sub-parsing inside of actions

#61                           GLSL_PREPROCESSOR::spaces
                              success
                              position GLSL preprocessor macro expansion:1:10
                              source bar A (4 + (2 * x))()
                                              ^
// ...
#97                         tao::pegtl::raise_message<'I', 'n', 'v', 'a', 'l', 'i', 'd', ' ', 'c', 'h', 'a', 'r', 'a', 'c', 't', 'e', 'r', ' ', 'f', 'o', 'u', 'n', 'd'>
                              raise tao::pegtl::raise_message<'I', 'n', 'v', 'a', 'l', 'i', 'd', ' ', 'c', 'h', 'a', 'r', 'a', 'c', 't', 'e', 'r', ' ', 'f', 'o', 'u', 'n', 'd'>
                            unwind
                          unwind #83 GLSL_PREPROCESSOR::function_call_arg
                        unwind #82 GLSL_PREPROCESSOR::function_call_arg_and_optional_comma
                      unwind #78 tao::pegtl::sor<tao::pegtl::seq<tao::pegtl::at<tao::pegtl::eof>, tao::pegtl::raise_message<'U', 'n', 't', 'e', 'r', 'm', 'i', 'n', 'a', 't', 'e', 'd', ' ', 'f', 'u', 'n', 'c', 't', 'i', 'o', 'n', ' ', 'p', 'a', 'r', 'e', 'n', 't', 'h', 'e', 's', 'i', 's', ',', ' ', 'e', 'x', 'p', 'e', 'c', 't', 'e', 'd', ' ', '\'', ')', '\'', ' ', 't', 'o', ' ', 'm', 'a', 't', 'c', 'h', ' ', '\'', '(', '\''> >, GLSL_PREPROCESSOR::function_call_arg_and_optional_comma>
                    unwind #75 tao::pegtl::until<tao::pegtl::at<GLSL_PREPROCESSOR::function_end>, tao::pegtl::sor<tao::pegtl::seq<tao::pegtl::at<tao::pegtl::eof>, tao::pegtl::raise_message<'U', 'n', 't', 'e', 'r', 'm', 'i', 'n', 'a', 't', 'e', 'd', ' ', 'f', 'u', 'n', 'c', 't', 'i', 'o', 'n', ' ', 'p', 'a', 'r', 'e', 'n', 't', 'h', 'e', 's', 'i', 's', ',', ' ', 'e', 'x', 'p', 'e', 'c', 't', 'e', 'd', ' ', '\'', ')', '\'', ' ', 't', 'o', ' ', 'm', 'a', 't', 'c', 'h', ' ', '\'', '(', '\''> >, GLSL_PREPROCESSOR::function_call_arg_and_optional_comma> >
                  unwind #46 GLSL_PREPROCESSOR::function_call_parens
                unwind #35 GLSL_PREPROCESSOR::function_call_actual
              unwind #34 GLSL_PREPROCESSOR::function_call
            unwind #30 tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces, GLSL_PREPROCESSOR::function_call, GLSL_PREPROCESSOR::identifier, tao::pegtl::ascii::any>
            position GLSL preprocessor macro expansion:1:5
            source bar A (4 + (2 * x))()
                       ^
          unwind #2 tao::pegtl::plus<tao::pegtl::sor<GLSL_PREPROCESSOR::whitespaces, GLSL_PREPROCESSOR::function_call, GLSL_PREPROCESSOR::identifier, tao::pegtl::ascii::any>>
        unwind #1 GLSL_PREPROCESSOR::grammar___
        position GLSL preprocessor macro expansion:1:1
        source bar A (4 + (2 * x))()
               ^
            unwind
          unwind #11 GLSL_PREPROCESSOR::function_call_actual
          position GLSL preprocessor function expansion:1:1
          source foo(bar)
                 ^
GLSL preprocessor macro expansion:1:10: Invalid character found
bar A (4 + (2 * x))()
         ^

    } catch (const pegtl::parse_error &e) {
      const auto p = e.positions().front();
      std:cerr << e.what() << '\n'
           << e.where() << '\n'
           << std::setw(p.column) << '^' << std::endl;
    }

question

opened by mgood7123 2

Releases(3.2.7)

3.2.7(Jul 14, 2022)
Removed superfluous deprecated include.

Source code(tar.gz)
Source code(zip)
3.2.6(Jun 29, 2022)
Made unwind() optional for parse tree nodes.

Fixed demangle() for MSVC, again.

Fixed demangle() for GCC 12.

Source code(tar.gz)
Source code(zip)
3.2.5(Feb 5, 2022)
Added missing include for fallback demangle() implementations.

Source code(tar.gz)
Source code(zip)
3.2.4(Feb 3, 2022)
Fixed version.hpp.

Source code(tar.gz)
Source code(zip)
3.2.3(Feb 3, 2022)
Fixed static_assert in demangle() with recent MSVC.

Source code(tar.gz)
Source code(zip)
3.2.2(Oct 22, 2021)
Added rule odigit for octal digits.

Enabled default-constructed state in state<>, change_state<>, and change_action_and_state<>.

Changed rules in tao/pegtl/contrib/integer.hpp to not throw by default.

Added tao/pegtl/contrib/separated_seq.hpp.

Added tao/pegtl/contrib/iri.hpp grammar for IRIs.

Added tao/pegtl/contrib/proto3.hpp grammar for protocol buffer v3.

Source code(tar.gz)
Source code(zip)
3.2.1(Jul 31, 2021)
Added an optional limiter to guard against infinite recursion.

Fixed CMake export error.

Improved compile time efficiency.

Source code(tar.gz)
Source code(zip)
3.2.0(Jan 15, 2021)
Added support for disabling exceptions with -fno-exceptions.

Improved efficiency of parse tree nodes.

Fixed namespace issue with tao::pegtl::demangle<T>() (was: tao::demangle<T>()).

Source code(tar.gz)
Source code(zip)
3.1.0(Dec 17, 2020)
Made analyze() more verbose by default to aid finding the rule cycles.

Added parse_nested() overload that accepts a position as first argument.

Added some experimental and undocumented contrib features and their infrastructure.

Improved CMake support for <filesystem> fallbacks and alternatives.

Re-enabled support for GCC 7.

Automatically link with libstdc++fs or libc++fs as needed.

Added automatic fallback from std::filesystem to std::experimental::filesystem.

Added manual fallback from std::filesystem to boost::filesystem.

Thank you Beman Dawes!

Converted continuous integration infrastructure to GitHub Actions.

Source code(tar.gz)
Source code(zip)
3.0.0(Nov 28, 2020)
Use the migration guide when updating.

Infrastructure

Updated required C++ standard to C++17.

Updated required CMake version to 3.8.

The macro TAO_PEGTL_NAMESPACE now contains the fully qualified namespace, e.g. tao::pegtl.

Added [[nodiscard]] or [[noreturn]] to most non-void functions.

Meta-Data Layer

Replaced analysis_t with more general and complete rule_t and subs_t.

Added functions to visit all rules of a grammar.

Added functions to measure rule coverage of a parsing run.

Moved the analysis function and header to contrib.

Error Handling

Replaced tao::pegtl::input_error with std::system_error and std::filesystem::filesystem_error.

Added must_if<>

Allows to define custom error messages for global errors.

Adds a non-intrusive way to define global parse errors for a grammar retroactively.

Demangling

Removed the need for RTTI.

Some broken/unknown compilers will use RTTI as a fallback, without demangling.

Moved tao::pegtl::internal::demangle<T>() to tao::demangle<T>().

Improved generated code to be shorter and more efficient.

Parse Tree

Removed the need for RTTI.

Other

Changed std::string to std::filesystem::path for filename parameters.

Renamed byte_in_line to column and use 1-based counting.

Moved rule eolf from inline namespace tao::pegtl::ascii to tao::pegtl.

Changed rules in tao/pegtl/contrib/integer.hpp to not accept redundant leading zeros.

Added rules to tao/pegtl/contrib/integer.hpp that test unsigned values against a maximum.

Demoted UTF-16 and UTF-32 support to contrib.

Demoted UINT-8, UINT-16, UINT-32 and UINT-64 support to contrib.

Folded contrib/counter.hpp into json_count.cpp, count is superceded by coverage.

Removed right padding from contrib/json.hpp's value.

Cleanup

Removed option of state's S::success() to have an extended signature to get access to the current apply_mode, rewind_mode, action- and control class (template).

Removed compatibility macros starting with TAOCPP_PEGTL_.

Removed compatibility uppercase enumerators.

Removed compatibility peek_byte() member functions.

Removed compatibility header changes.hpp from contrib.

Source code(tar.gz)
Source code(zip)
2.8.3(Apr 22, 2020)
Fixed excessive read-ahead with incremental inputs.

Added state manipulators remove_first_state, remove_last_states, rotate_states_right, rotate_states_left, and reverse_states to contrib.

Reduced the number of intermediate parse tree nodes.

Source code(tar.gz)
Source code(zip)
2.8.2(Apr 4, 2020)
Fixed parse tree node generation to correctly remove intermediate nodes.

Source code(tar.gz)
Source code(zip)
2.8.1(Aug 6, 2019)
Added fallback symbol demangling if RTTI is disabled.

Fixed missing string_input<> in amalgamated header.

Fixed discard_input* actions to properly forward the apply mode.

Fixed contrib HTTP grammar for chunked data.

Source code(tar.gz)
Source code(zip)
2.8.0(Apr 9, 2019)
Use the migration guide when updating.

Changed enumerators to lowercase.

Renamed tracking_mode::IMMEDIATE to tracking_mode::eager.

Compatibility enumerators with uppercase names are still included.

Will be removed in version 3.0.0.

Renamed peek_byte() to peek_uint8().

Compatibility member functions with previous names are still included.

Will be removed in version 3.0.0.

Allowed actions to implement match.

Made deriving action class templates from nothing optional.

Added debug tools require_apply and require_apply0.

Added combinator class rematch.

Improved the Parse Tree / AST interface to mostly hide its internal state.

Added new action-based helpers change_*.hpp.

The control-based helpers in contrib/changes.hpp are still included.

Will be removed in version 3.0.0.

Added new action-based helpers disable_action.hpp and enable_action.hpp.

Added new action-based helpers discard_input.hpp, discard_input_on_success.hpp, and discard_input_on_failure.hpp.

Added Clang Static Analyzer to the CI build.

Added new Makefile target amalgamate to generate a single-header version of the PEGTL.

Added support for Universal Windows Platform (UWP).

Source code(tar.gz)
Source code(zip)
2.7.1(Sep 29, 2018)
Added new ASCII convenience rule forty_two.

Added experimental if_then rule.

Simplified how parse tree nodes can be selected.

Reduced the number of intermediate parse tree nodes.

Allowed an action class template to be used with the parse tree.

Source code(tar.gz)
Source code(zip)
2.7.0(Jul 31, 2018)
Added mmap_file<> support for Windows.

Added deduction guides for the input classes when compiling with C++17.

Source code(tar.gz)
Source code(zip)
2.6.1(Jul 22, 2018)
Fixed endianness detection in test program.

Source code(tar.gz)
Source code(zip)
2.6.0(Jun 23, 2018)
Added Conan packages.

Fixed the UTF-8 decoder to no longer accept UTF-16 surrogates.

Fixed the UTF-16 decoder to no longer accept UTF-16 unmatched surrogates.

Fixed the UTF-32 "decoder" to no longer accept UTF-16 surrogates.

Fixed pegtl/contrib/unescape.hh to no longer accept unmatched surrogates.

Optimised convenience rule two.

Added new convenience rule three.

Source code(tar.gz)
Source code(zip)
2.5.2(May 31, 2018)
Fixed opt and until to work as documented in some rare edge cases.

Used opt_must and star_must to optimise some included grammars.

Source code(tar.gz)
Source code(zip)
2.5.1(May 14, 2018)
Added new convenience rule opt_must.

Optimised convenience rule if_must.

Fixed examples to compile with Visual Studio and MinGW.

Added automated testing with GCC 8.

Source code(tar.gz)
Source code(zip)
2.5.0(May 1, 2018)
Added rules to match Unicode properties via ICU to contrib.

Improved the Parse Tree / AST interface.

Fixed parse tree node generation to correctly remove intermediate nodes.

Added big- and little-endian support to the UTF-16 and UTF-32 rules.

Added rules for UINT-8 and big- and little-endian UINT-16, UINT-32 and UINT-64.

Added function to memory_input<> to obtain the line around a position.

Added function to memory_input<> to start again from the beginning.

Added example for Python-style indentation-aware grammars.

Added examples for regular, context-free, and context-sensitive grammars.

Added example for how to parse with a symbol table.

Added automated testing with Clang 6.

Added automated testing with Clang's -fms-extensions.

Fixed build with Clang when -fms-extensions is used (clang-cl).

Source code(tar.gz)
Source code(zip)
2.4.0(Feb 17, 2018)
Improved and documented the Parse Tree / AST support.

Changed prefix of all macros from TAOCPP_PEGTL_ to TAO_PEGTL_. Compatibility macros with the old names are provided, they will be removed in version 3.0.

Added a deleted overload to prevent creating a memory_input<> from a temporary std::string.

Source code(tar.gz)
Source code(zip)
2.3.4(Feb 8, 2018)
Fixed build on older systems where O_CLOEXEC is not available.

Added automated testing with Android 6.0 and 7.0.

Source code(tar.gz)
Source code(zip)
2.3.3(Jan 1, 2018)
Added more noexcept-specifications.

Fixed most clang-tidy-issues.

Source code(tar.gz)
Source code(zip)
2.3.2(Dec 16, 2017)
Worked around a Visual Studio 15.5 bug.

Source code(tar.gz)
Source code(zip)
2.3.1(Dec 14, 2017)
Fixed linkage of tao::pegtl::internal::file_open.

Improved error message for missing source parameter of string_input<>.

Source code(tar.gz)
Source code(zip)
2.3.0(Dec 11, 2017)
Added constructor to read_input<> that accepts a FILE*, see issue #78.

Enhanced apply<>, apply0<> and if_apply<> to support apply()/apply0()-methods returning boolean values.

Simplified implementation of raw_string, the optional Contents... rules' apply()/apply0()-methods are now called with the original states.

Fixed the tracer to work with apply()/apply0()-methods returning boolean values. (Thanks Joel Frederico)

Fixed, simplified and improved examples/parse_tree.cpp.

Source code(tar.gz)
Source code(zip)
2.2.2(Nov 22, 2017)
Fixed missing call to the control class' failure()-method when a rule with an apply()-method with a boolean return type fails.

Fixed string handling in examples/abnf2pegtl.cc.

Simplified/improved Android build.

(this is a re-release of 2.2.1 to fix the version number reported by CMake and version.hpp)
Source code(tar.gz)
Source code(zip)
2.2.1(Nov 22, 2017)
Fixed missing call to the control class' failure()-method when a rule with an apply()-method with a boolean return type fails.

Fixed string handling in examples/abnf2pegtl.cc.

Simplified/improved Android build.

Source code(tar.gz)
Source code(zip)
2.2.0(Sep 24, 2017)
Added possibility for actions' apply() or apply0()-methods to return a bool which is then used to determine overall success or failure of the rule to which such an action was attached.

Added <tao/pegtl/contrib/parse_tree.hpp> and the examples/parse_tree.cpp application that shows how to build a parse tree. The example goes beyond a traditional parse tree and demonstrates how to select which nodes to include in the parse tree and how to transform the nodes into an AST-like structure.

Added bom rules for UTF-8, UTF-16 and UTF-32.

Added some missing includes for config.hpp.

Added automated testing with Clang 5.

Added automated testing with Xcode 9.

Source code(tar.gz)
Source code(zip)