Today's the day. The decision has been made. The parser - although cool, simply wasn't enough for a format like Neorg. Neorg needs a decent chunk of custom lexing. I will try and implement as many things as I can via regex rules and will only use the lexer as a last resort (aka the good way of doing things).

Things we need:

• If a token cannot be detected then mark it as a paragraph
• Detecting *bold*, /italic/, _underline_, [these](links) and all the other modifiers present in the specification.
• Properly identifying escape sequences
• Distinguishing attached vs detached modifiers
• Making sure the parser doesn't crash every 3.5ms :P
• The ability for the parser to create a true structured syntax tree. For example, the current parser generates trees similar to this:

  heading1
    leading_whitespace
    heading1_prefix
    paragraph_segment
  paragraph
    paragraph_segment
        words
  

  Since the paragraph is usually underneath the heading you would logically want it to look like so:

    heading1
      etc.
      paragraph
          paragraph_segment
             words
  

Excited to start this new endeavor. I'll start proper work on the parser tomorrow. If you have any experience with TreeSitter and wanna hop along for the ride and help out then please do! :)

I am running windows, and I have this after following the documentation:

Console output:

[nvim-treesitter] [0/1] Downloading...
[nvim-treesitter] [0/1] Checking out locked revision
[nvim-treesitter] [0/1] Compiling...
LINK : warning LNK4044: unrecognized option '/Z-reserved-lib-stdc++'; ignored^M
   Creating library parser.lib and object parser.exp^M
parser-d99b79.o : error LNK2001: unresolved external symbol tree_sitter_norg_external_scanner_create^M
parser-d99b79.o : error LNK2001: unresolved external symbol tree_sitter_norg_external_scanner_destroy^M
parser-d99b79.o : error LNK2001: unresolved external symbol tree_sitter_norg_external_scanner_scan^M
parser-d99b79.o : error LNK2001: unresolved external symbol tree_sitter_norg_external_scanner_serialize^M
parser-d99b79.o : error LNK2001: unresolved external symbol tree_sitter_norg_external_scanner_deserialize^M
parser.so : fatal error LNK1120: 5 unresolved externals^M
nvim-treesitter[norg]: Error during compilation
clang: error: linker command failed with exit code 1120 (use -v to see invocation)^M

I am pretty sure it use to work before, but for some reason not any more, and I couldn't make sense of what the issue is:

init.lua section:

local parser_configs = require('nvim-treesitter.parsers').get_parser_configs()

parser_configs.norg = {
    install_info = {
        url = "https://github.com/vhyrro/tree-sitter-norg",
        files = { "src/parser.c" },
        branch = "main"
    },
}

require'nvim-treesitter.configs'.setup {
    ensure_installed = 'all', -- one of "all", "maintained" (parsers with maintainers), or a list of languages
    ignore_install = { "fortran" }, -- List of parsers to ignore installing
    highlight = {
        enable = true -- false will disable the whole extension
    },
    playground = {
        enable = true,
        disable = {},
        updatetime = 25, -- Debounced time for highlighting nodes in the playground from source code
        persist_queries = false -- Whether the query persists across vim sessions
    },
    autotag = {enable = true},
    rainbow = {enable = true}
    -- refactor = {highlight_definitions = {enable = true}}
}

Yo!

I use doom-nvim and .norg i not working on neither of my old intel macbook pro or my new M1 macbook air.

M1 startup error

nvim-treesitter[norg]: Error during compilation
Undefined symbols for architecture arm64:
  "_tree_sitter_norg_external_scanner_create", referenced from:
      _tree_sitter_norg.language in parser-29db6d.o
  "_tree_sitter_norg_external_scanner_deserialize", referenced from:
      _tree_sitter_norg.language in parser-29db6d.o
  "_tree_sitter_norg_external_scanner_destroy", referenced from:
      _tree_sitter_norg.language in parser-29db6d.o
  "_tree_sitter_norg_external_scanner_scan", referenced from:
      _tree_sitter_norg.language in parser-29db6d.o
  "_tree_sitter_norg_external_scanner_serialize", referenced from:
      _tree_sitter_norg.language in parser-29db6d.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Intel startup error

nvim-treesitter[norg]: Error during compilation
Undefined symbols for architecture x86_64:
  "_tree_sitter_norg_external_scanner_create", referenced from:
      _language.0 in cc3MwxxW.o
  "_tree_sitter_norg_external_scanner_deserialize", referenced from:
      _language.0 in cc3MwxxW.o
  "_tree_sitter_norg_external_scanner_destroy", referenced from:
      _language.0 in cc3MwxxW.o
  "_tree_sitter_norg_external_scanner_scan", referenced from:
      _language.0 in cc3MwxxW.o
  "_tree_sitter_norg_external_scanner_serialize", referenced from:
      _language.0 in cc3MwxxW.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status

Adds the ranged element syntax discussed on Discord.

• [x] change spoiler syntax from | to !
• [x] implement ranged attached modifiers

You can now use the range-modifier `|` to write |*ranged markup like this
which has the benefit

of allowing arbitrary whitespace inside of it      *|.

Ranged markup elements are placed into the paragraph in which they start.
They can contain other in-line syntax like more markup, links, etc.
No detached modifiers, headings, etc. are allowed inside of them.

• [x] support things like |*/ ranged bold + italic /*|
• [x] implement general ranged element frames

- |
  If you want to include more elements inside of a single detached modifier you can use a `||` ranged element.
  This allows you to for example include code blocks inside list items:

  @code lua
  print("Hello world!")
  @end
  ---

I guess this is more an enquiry than an issue, but I was wondering how realistic it would be to use this parser as the base for a markdown parser. Norg markup seems to be closely related to markdown and markdown is currently missing a tree-sitter parser. A proper markdown parser would be very useful, especially if you have markdown documents with lots of code blocks (e.g. rmarkdown).

Using the latest nixpkgs-neorg-overlay flake results in "error: attribute 'name' missing' (full '--show-trace' output attached).

Pinning the inputs "norg" and "norg-meta" to the commits preceding Define outputs for each system and Define outputs for each system respectively, fixes the issue.

neorg-overlay = {
    url = "github:nvim-neorg/nixpkgs-neorg-overlay";
    inputs.nixpkgs.follows = "nixpkgs";
    inputs.norg.url = "github:nvim-neorg/tree-sitter-norg?rev=5d9c76b5c9927955f7c5d5d946397584e307f69f";
    inputs.norg-meta.url = "github:nvim-neorg/tree-sitter-norg-meta?rev=e93dcbc56a472649547cfc288f10ae4a93ef8795";
};

I'm happy to debug this further on my machine but I'll need some help as I don't have that much experience with nix yet.

@bandithedoge @pschyska

trace.txt

Follow-up from https://github.com/nvim-neorg/nixpkgs-neorg-overlay/issues/1#issuecomment-1266974818.

I realized it basically undoes https://github.com/nvim-neorg/tree-sitter-norg/commit/ed37d4c571104c64e233a319fc366ede75ee56d4. By merging in defaultPackage again, it would remove e.g. defaultPackage.x86_64-linux which is what is blocking me. What was the reasoning for that change? I don't think, in flakes parlance, there can't be a "generic" package as the result is system-dependant.

That said, users shouldn't have too much exposure to the system -- nix build ., nix develop ., etc. should "just work" unless you intend to cross compile, or compile with emulation[^1].

If that's alright, I have corresponding changes for tree-sitter-norg-{meta,table}.

[^1] What I'm doing for aarch64-linux is having the following nixos config...

  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

...so that I can say:

  $ nix build .#packages.aarch64-linux.tree-sitter-norg
  $ file result/parser
  result/parser: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

This adds support for attached modifiers.

:warning: this is very much WIP!

Open tasks:

• [x] rewrite check_attached with int32 instead of unsigned char
• [x] allow attached modifiers within headings (currently they immediately cause the heading to end)
• [x] permit nested attached modifiers
• this will likely come with some caveat of a limited nesting level support...

We also still need to add the following new attached modifiers:

• [x] +: inline comment
• [x] $: inline math
• [x] =: variable reference

Hi, I experience the following errors when opening any .norg file and during :checkhealth nvim-treesitter:

nvim-treesitter: require("nvim-treesitter.health").check()
========================================================================
## Installation
  - WARNING: `tree-sitter` executable not found (parser generator, only needed for :TSInstallFromGrammar, not required for :TSInstall)
  - OK: `node` found v10.19.0 (only needed for :TSInstallFromGrammar)
  - OK: `git` executable found.
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
    Version: cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
  - OK: Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

## Parser/Features H L F I J
...
  - norg           x . ✓ . ✓
...

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang}

## The following errors have been detected:
  - ERROR: norg(highlights): /usr/local/share/nvim/runtime/lua/vim/treesitter/query.lua:218: query: invalid field at position 7227 for language norg
    norg(highlights) is concatenated from the following files:
    | [ERROR]:"/home/[username]/.local/share/nvim/plugged/neorg/queries/norg/highlights.scm", failed to load: /usr/local/share/nvim/runtime/lua/vim/treesitter/query.lua:218: query: invalid field at position 7227 for language norg

If I run TSUpdate norg as suggested, I get the info "Parsers are up-to-date!". Tried with neovim 0.7.2 and 0.8.0-dev. I'm using neorg 0.0.12. Any idea what is the issue?

Reproduction example:

* Heading
  - \
      Text
      ---
This text no longer belongs in the heading even though it should.

The --- paragraph delimiter should end the indent segment, but it instead terminates the heading.

Treesitter considers Cyrillic н letter (the analog of Latin n) at the beginning of the word as NeorgMarkupVariableDelimiter and if there is another one somewhere at the end of the word, everything between these two letters inside one paragraph becomes a variable.

For example:

Я тоже не знал, пока в моей жизни не появились проклятые рыбки. Жизнь аквариумиста - это один сплошной стресс.

theHamsta@ from nvim-treesitter/nvim-treesitter recommended me to forward this issue here.

Describe the bug

I’m not able to install the norg parser. It never finishes compiling.

To Reproduce

1. Open up nvim
2. Run :TSInstall norg

Expected behavior

I expected the norg parser to be installed within a few seconds

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()                                                                           
========================================================================                                                             
## Installation                                                                                                                      
  - WARNING: `tree-sitter` executable not found (parser generator, only needed for :TSInstallFromGrammar, not required for :           TSInstall)                                                                                                                         
  - OK: `node` found v14.18.1 (only needed for :TSInstallFromGrammar)                                                                
  - OK: `git` executable found.                                                                                                      
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }                                          
    Version: gcc (GCC) 10.3.0                                                                                                        
  - OK: Neovim was compiled with tree-sitter runtime ABI version 13 (required >=13). Parsers must be compatible with runtime ABI.    
                                                                                                                                     
## Parser/Features H L F I J                                                                                                         
  - nix            ✓ ✓ ✓ . ✓                                                                                                         
  - clojure        ✓ ✓ ✓ . ✓                                                                                                         
                                                                                                                                     
  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections                                                                    
         +) multiple parsers found, only one will be used                                                                            
         x) errors found in the query, try to run :TSUpdate {lang}

Output of nvim --version

NVIM v0.6.1
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by nixbld

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "
/nix/store/29bl3zlp96r1w8d0ii3zcim1p9fdk1xm-neovim-unwrapped-0.6.1/share/nvim
"

Run :checkhealth for more info

Additional context

No response

Consolidates the work on:

• the formal syntax specification (#29)
• the breaking changes and paragraph parsing refactoring (#30)
• the inline link targets (#25)

Although it is disallowed for such syntax to be valid:

{
* My Link
}

and

[
my description
]

This syntax is allowed:

{* My
link}

and

[my
description]

But since both elements allow only a paragraph_segment any newlines are treated as erroneous, therefore breaking things like Neorg's hop module.

I have the following in my vim config

local parser_config = require 'nvim-treesitter.parsers'.get_parser_configs()

parser_config.norg = {
  install_info =
    { url    = 'https://github.com/vhyrro/tree-sitter-norg'
    , files  = { 'src/parser.c', 'src/scanner.cc' }
    , branch = 'main'
  }
}

When I run TSUpdate, I get this error:

nvim-treesitter[norg]: Error during compilation
src/scanner.cc:97:39: error: expected expression
        return std::vector<TokenType>({ lhs, static_cast<TokenType>(rhs) });
                                      ^
src/scanner.cc:100:27: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
    std::vector<TokenType>&& operator|(std::vector<TokenType>&& lhs, TokenType rhs)
                          ^
src/scanner.cc:100:62: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
    std::vector<TokenType>&& operator|(std::vector<TokenType>&& lhs, TokenType rhs)
                                                             ^
src/scanner.cc:327:41: error: expected ';' at end of declaration list
    std::vector<size_t>& get_tag_stack() noexcept { return m_TagStack; }
                                        ^
                                        ;
src/scanner.cc:556:27: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    TokenType m_LastToken = NONE;
                          ^
src/scanner.cc:559:26: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    size_t m_ParsedChars = 0, m_IndentationLevel = 0;
                         ^
src/scanner.cc:559:50: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    size_t m_ParsedChars = 0, m_IndentationLevel = 0;
                                                 ^
src/scanner.cc:565:54: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    const std::array<int32_t, 6> s_DetachedModifiers = { '*', '-', '>', '|', '=', '~' };
                                                     ^
src/scanner.cc:121:13: error: use of undeclared identifier 'advance'
            advance(lexer);
            ^
src/scanner.cc:131:13: error: use of undeclared identifier 'advance'
            advance(lexer);
            ^
src/scanner.cc:141:17: error: use of undeclared identifier 'advance'
                advance(lexer);
                ^
src/scanner.cc:148:21: error: use of undeclared identifier 'advance'
                    advance(lexer);
                    ^
src/scanner.cc:162:13: error: use of undeclared identifier 'advance'
            advance(lexer);
            ^
src/scanner.cc:167:48: error: use of undeclared identifier 'm_Current'
                if (lexer->lookahead == ']' && m_Current != '\\')
                                               ^
src/scanner.cc:169:21: error: use of undeclared identifier 'advance'
                    advance(lexer);
                    ^
src/scanner.cc:141:17: error: use of undeclared identifier 'advance'
                advance(lexer);
                ^
src/scanner.cc:148:21: error: use of undeclared identifier 'advance'
                    advance(lexer);
                    ^
src/scanner.cc:162:13: error: use of undeclared identifier 'advance'
            advance(lexer);
            ^
src/scanner.cc:167:48: error: use of undeclared identifier 'm_Current'
                if (lexer->lookahead == ']' && m_Current != '\\')
                                               ^
src/scanner.cc:169:21: error: use of undeclared identifier 'advance'
                    advance(lexer);
                    ^
src/scanner.cc:177:17: error: use of undeclared identifier 'advance'
                advance(lexer);
                ^
src/scanner.cc:184:20: error: use of undeclared identifier 'check_link'
            return check_link(lexer);
                   ^
src/scanner.cc:188:13: error: use of undeclared identifier 'advance'
            advance(lexer);
            ^
src/scanner.cc:206:17: error: use of undeclared identifier 'skip'
                skip(lexer);
                ^
src/scanner.cc:211:17: error: use of undeclared identifier 'advance'
                advance(lexer);
                ^
src/scanner.cc:217:21: error: use of undeclared identifier 'advance'
                    advance(lexer);
                    ^
src/scanner.cc:220:25: error: use of undeclared identifier 'advance'
                        advance(lexer);
                        ^
src/scanner.cc:223:29: error: use of undeclared identifier 'advance'
                            advance(lexer);
                            ^
src/scanner.cc:227:37: error: use of undeclared identifier 'advance'
                                    advance(lexer);
                                    ^
src/scanner.cc:252:25: error: use of undeclared identifier 'advance'
                        advance(lexer);
                        ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
6 warnings and 20 errors generated.

From :checkhealth nvim_treesitter

  health#nvim_treesitter#check
  ========================================================================
  ## Installation
    - OK: `tree-sitter` found  0.20.0 (parser generator, only needed for :TSInstallFromGrammar)
    - OK: `node` found v16.8.0 (only needed for :TSInstallFromGrammar)
    - OK: `git` executable found.
    - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl" }
    - OK: Neovim was compiled with tree-sitter runtime ABI version 13 (required >=13). Parsers must be compatible with runtime ABI.

And cc --version

Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin