With the changes in the current markup-redo branch, I'd like to pause before integration (primarily understanding orgmode.nvim uses the main branch) to have some discussion about the work here, and compile a 1.0 checklist.

Changes to markup

To start, thoughts on changes in the markup-redo branch. The first big one is that all markup parsing is completely removed, and in place of processing text as hidden nodes, whitespace delimited words are now parsed as (expr) nodes. An expression here is parsed first for looking for ascii symbols as hidden nodes, then letters as "str", then numbers as "num", and finally any remaining symbols as "sym".

The reasoning for this set up is first that anything parsed in the parser is unchangeable, so markup was difficult to customize, and if a user wanted to add another link style (such as markdown-style links), they were unable to do so. Now, they just need to write queries and no modification for the parser is required.

For example, a (*b1 cdef*) is parsed to (expr) (expr) (expr) and when looking at hidden nodes, (expr "str") (expr "(" "*" "str" "num") (expr "str" "*" ")"). Which we can query for in a variety of ways. Unfortunately, a major caveat for this is the fact that tree-sitter queries capture exactly one node (see tree-sitter/tree-sitter#1508), so capturing alone (without writing a directive or callback of some sort) will not be able to highlight markdown as is. Additionally, we need to account for nested expressions.

The other major caveat is that anonymous nodes are unable to be anchored (tree-sitter/tree-sitter#1461), so we can't make the queries as precise as I would like right now, but single node queries can just use the #match? predicate so I'm not worried about that.

The positive side of this is that because the markup characters will be hidden nodes, they are easily queried for and whatever algorithm is applied will generally only need to look at very few nodes. Any language should be able to do that very quickly. Queries to find possible pairs are in the branch in the markup.scm file, and I'm working the kinks out of a lua implementation of this pseudocode:

def markup(nodes):
    # assumes sorted list of nodes
    # all nodes should be from a single (paragraph), (itemtext), or (item)

    seeking = []
    markup = []
    for node in nodes:
        if node.type == start:
            seeking.append((node.type, node))
        elif node.type in seeking:
            ix = seeking.index(node.type)
            stnode = seeking[ix][1]
            if validate(node, stnode):  # check the pre/post markup or whatever else, can be modified as desired
                # stop seeking everything after the node we just matched: *a /b*    Like this '/', when we just matched the * after the b
                seeking = seeking[:ix-1]
                # If we complete a verbatim-style markup, we need to purge everything interior of it
                markup.append({'type': node.type, 'range': (stnode.start, node.end)})
    return markup

The details of this algorithm will change depending on exactly which node is captured (The symbol vs (expr)) and whether or not we're using match predicate in advance. My example does not do that, but after writing my lua example I'm thinking it makes sense to do so.

Note how easily constructed the markup queries are via some generative code for customization.

Lastly, queries/algorithms are needed for latex fragments, subscript and superscript, and bracketed expressions for the *scripts.

Regex patterns

I still need to work through the list of changes I made, but one change I've made/am still making to the markup branch is I've tried replacing specific patterns as often as possible with some variant of (expr). I think using queries to determine if a propery plan name is used, for example, is much cleaner than throwing a parser error. And this way it's easier to change languages. I think there are limits though, and it might be nice to simply support different languages in the parser directly (for example, END, PROPERTIES, etc.). That could be read from a file, or simply left for others to re-compile to their own language.

Additionally, queries are highlighting, so allowing more things to be specified as queries nicely can be pretty cool. For example, I might only ever use TITLE and FILETAGS directives, so I could consider highlighting those nicely and any other patterns as an error.

A good example here is timestamp contents. Right now, I've hard coded possible regex expressions. Should that just be queriable? That could be nice if people want different formats. On the other hand, having fields and nodes directly in the tree is really nice, and I think in timestamps there are few enough formats that we can just support all of them in the parser.

Fields and nodes/aliases

When writing the parser I didn't add a lot of fields because it was constantly in so much flux. But they're useful even if they link directly to a named node with the same name, because access via a name can be a lot more convenient than named nodes, even if there is a small number of nodes.

I've added a large number of (name) nodes, and a few others, and many fields. I'll try to compile a list later today. When thinking about where I've added nodes, one of the things I was thinking about was incremental selection via nodes, I just want it to make sense. For example, tag -> taglist -> item -> headline -> section.

Versioning

These changes were a lot more than I'd like to do in a single commit or merge in the future, but since I was writing queries as I went, I kept finding small changes that make using the parser a lot more straightforward. I don't work on coding projects that are public really, so I don't think much about this, so this is kind of a "Yeah, why didn't you do that sooner?" section.

So AFTER 1.0.0, I'll be using consistent conventional commits in the future and an auto-updating semantic versioning system based on that. With major.minor.patch versioning: fix changes increment patch, feat changes increment minor, and any breaking change (appending a ! to the scope) will increment major versioning. I just want to make sure anything using the parser has a tag that they can link to so changes can be made to main without breaking dependent projects.

Specific questions

• Are there any missing/hidden aliases or fields that would be useful?
• Are the names of aliases and fields sensible?
• Should newlines be part of (body)? (Newlines before body in (section) and in (document)) So, should an empty section have a body, basically, or should a body exist only if there is an element?
• If we have text that is a paragraph followed by a footnote after a new line, is that parsed in emacs as a footnote reference or a definition?
• Should (_element) be available to listitems and in drawer contents, or should those just be expressions? I'm under the impression that whether or not elements are in a list and drawer are customizable options for orgmode, and I prefer to keep the parser simple as possible. So it makes sense to me to inject an org parser in the itemtext if a user wants it.
• Should (taglist) be a named node?

1.0.0 checklist

I want to expand upon this list (some items should be multiple), but really quickly:

• [x] Revisit all tests - A lot are out of date
• ~~[ ] Write tests for queries~~
• [x] Fix some regexes to better query for: :block: is the whole name, the goal would be ':', (name), ':'. (easier highlighting ':'s)
• ~~[ ] Cleanup table precedences. Yikes. (I don't care anymore)~~
• ~~[ ] Add semantic versioning git hook script~~
• [x] Revisit the readme build instructions
• [x] Revisit the npm dependencies (could really use some help here, no idea what I'm doing there)
• [x] Add plan entries
• [x] Add newlines to (contents) and (body)
• [x] Fix failing headline test

Thoughts? Anything else I'm missing?

Hi 👋🏿 ,

I've been refactoring my plugin https://github.com/akinsho/org-bullets.nvim which is similar to the emacs bullets plugin. I recently switched to identifying nodes to conceal using treesitter (your parser) and that's been working really well except for this issue I just hit.

I've noticed that using [X] will return one expr node which I can #match/#eq against whereas an empty check box i.e. [ ] return 3 expression nodes for each character so matching fails.

I'm guessing this is a bug or workaround of some sort but would be great to have the whole thing returned as a node so I can conceal it.

This PR adds support for src and dynamic blocks in list items. I got a report here about this. I tested on emacs orgmode, and it properly supports these blocks in list items. ~~I created an anonymous node for this for the sake of DRY, but if you think it needs to be done differently, let me know.~~ I set it up to go only after new line.

I got an issue reported on my repo that markup is not working in the headlines, which works fine in Emacs orgmode. That's currently an issue because Vim's syntax doesn't allow overlapping of two hl groups (Example below):

I tried adding a markup to the headline item here, but it caused some different issues which I'm not sure how to address. My current workaround is just to use Vim's syntax for markup, and let headline level highlights be handled through treesitter highlights, which makes it work ok:

Only downside is that it requires enabling Vim's syntax highlighting (additional_vim_regex_highlighting). What are your thoughts on this? How hard would it be to add it to headline item?

I know that you removed this few commits ago, but I don't think there is a way to get a chunk of the capture, even with the custom directive, because treesitter highlighter does not even respect the #offset directive. Since todo keywords are configurable and dynamic (both keywords and their colors), this allows to add highlights for the first word in headline if it's matching the configuration.

Edit: Here's how I do it with this solution: https://github.com/kristijanhusak/orgmode.nvim/blob/feature/ts/lua/orgmode/colors/todo_highlighter.lua#L30-L34

Hi,

Today I started playing with tables, and noticed one thing I can't really figure out why it's happening.

Having this content:

  | Test | Foo  | Bar    |
  |------+------+--------|
  | Test |      | Seven  |
  | dda  | test |        |
  |      |      |        |
  | dda  | test |        |

Generates this:

(document [0, 2] - [6, 0]
  (body [0, 2] - [6, 0]
    (table [0, 2] - [2, 0]
      (row [0, 2] - [1, 0]
        (cell [0, 2] - [0, 8])
        (cell [0, 9] - [0, 14])
        (cell [0, 16] - [0, 21])))
    (table [2, 2] - [6, 0]
      (row [2, 2] - [3, 0]
        (cell [2, 2] - [2, 8])
        (cell [2, 9] - [2, 10])
        (cell [2, 16] - [2, 23]))
      (row [3, 2] - [4, 0]
        (cell [3, 2] - [3, 7])
        (cell [3, 9] - [3, 15])
        (cell [3, 16] - [3, 17]))
      (row [4, 2] - [5, 0]
        (cell [4, 2] - [4, 3])
        (cell [4, 9] - [4, 10])
        (cell [4, 16] - [4, 17]))
      (row [5, 2] - [6, 0]
        (cell [5, 2] - [5, 7])
        (cell [5, 9] - [5, 15])
        (cell [5, 16] - [5, 17])))))

Also, if I add one more hr rule to the table like this:

  | Test | Foo  | Bar    |
  |------+------+--------|
  | Test |      | Seven  |
  | dda  | test |        |
  |      |      |        |
  |------+------+--------|
  | dda  | test |        |

Generates this:

(document [0, 2] - [7, 0]
  (body [0, 2] - [7, 0]
    (table [0, 2] - [3, 0]
      (row [0, 2] - [1, 0]
        (cell [0, 2] - [0, 8])
        (cell [0, 9] - [0, 14])
        (cell [0, 16] - [0, 21]))
      (row [2, 2] - [3, 0]
        (cell [2, 2] - [2, 8])
        (cell [2, 9] - [2, 10])
        (cell [2, 16] - [2, 23])))
    (table [3, 2] - [7, 0]
      (row [3, 2] - [4, 0]
        (cell [3, 2] - [3, 7])
        (cell [3, 9] - [3, 15])
        (cell [3, 16] - [3, 17]))
      (row [4, 2] - [5, 0]
        (cell [4, 2] - [4, 3])
        (cell [4, 9] - [4, 10])
        (cell [4, 16] - [4, 17]))
      (row [6, 2] - [7, 0]
        (cell [6, 2] - [6, 7])
        (cell [6, 9] - [6, 15])
        (cell [6, 16] - [6, 17])))))

I would not expect horizontal rule to split the table in any way. I think this should be single table with multiple rows. Looking at grammar doesn't give me a clue, because there it looks like it can either be a row with valid cells, or hr rule. I can't figure out why it would cause this.

Also, is there a chance to expose the horizontal rule node itself? It's much easier to collect it while parsing the tree instead of figuring it out manually, since I want to introduce reformatting the table on edit, and I also potentially need to shrink/expand the hr rule line.

Hello, I'm using neovim with the orgmode plugin and treesitter. When opening the treesitter playground, it looks like the links are not recognized, they just appear as "expr" inside a paragraph. Is it a problem on my machine or is it not implemented? If I wrote on the wrong place, then I'm sorry, but it's kind of hard to understand which packages are used...

Hey,

I just got an issue report on orgmode.nvim repo that found a problem with parsing things that are "escaped". For example, with this content:

* TODO Test
  This \[is error]

I get this:

(document [0, 0] - [2, 0]
  (ERROR [0, 0] - [2, 0]
    (headline [0, 0] - [0, 11]
      (stars [0, 0] - [0, 1])
      (item [0, 2] - [0, 11]))))
test.org	0 ms	(ERROR [0, 0] - [2, 0])

Backslash before the [ causes the issue.

Similar thing happens with dot for example:

* TODO Test
  This \. Test

Output:

(document [0, 0] - [2, 0]
  (section [0, 0] - [2, 0]
    (headline [0, 0] - [0, 11]
      (stars [0, 0] - [0, 1])
      (item [0, 2] - [0, 11]))
    (body [1, 2] - [2, 0]
      (paragraph [1, 2] - [2, 0]
        (ERROR [1, 7] - [1, 9]
          (ERROR [1, 8] - [1, 9]))))))
test.org	0 ms	(ERROR [1, 7] - [1, 9])

Latter seems more specific than former. Is this because it looks like a regex?

Original issue reported here: https://github.com/nvim-orgmode/orgmode/issues/362.

Org content:

* TODO Test
	 DEADLINE: <2022-09-10 Sat 15:21>

Expected:

(document [0, 0] - [2, 0]
  subsection: (section [0, 0] - [2, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 11]
        (expr [0, 2] - [0, 6])
        (expr [0, 7] - [0, 11])))
    plan: (plan [1, 2] - [2, 0]
      (entry [1, 2] - [1, 34]
        name: (entry_name [1, 2] - [1, 10])
        timestamp: (timestamp [1, 12] - [1, 34]
          date: (date [1, 13] - [1, 23])
          day: (day [1, 24] - [1, 27])
          time: (time [1, 28] - [1, 33]))))))

Got:

(document [0, 0] - [2, 0]
  subsection: (section [0, 0] - [2, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 11]
        (expr [0, 2] - [0, 6])
        (expr [0, 7] - [0, 11])))
    body: (body [1, 0] - [2, 0]
      (paragraph [1, 0] - [2, 0]
        (expr [1, 0] - [1, 1])
        (expr [1, 2] - [1, 11])
        (expr [1, 12] - [1, 23])
        (expr [1, 24] - [1, 27])
        (expr [1, 28] - [1, 34])))))

I'm new to using org mode in Neovim (and Neovim itself...) so I could be doing something wrong but I'm getting the following error when trying to install the org treesitter parser:

Neovim version: 0.7.2

I simplified my init.vim down to just the Plug install command for treesitter:

call plug#begin('~/.config/nvim/plugged')
Plug 'nvim-treesitter/nvim-treesitter', { 'do': ':TSUpdate' }
call plug#end()

Command in Neovim: :TSInstall org

This project is awesome, thanks for working on it!

Hi,

I got an issue about LaTeX support, and wanted to check with you what is actually supported? I don't have any LaTeX experience so I can hardly figure it out. From what I tested, only something like this works:

\begin{align}
2x - 5y &= 8 \\
3x + 9y &= -12
\end{align}

This is an example from here https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Environments, but without the * in the align, since that causes error in parsing. Other examples seems to parse like a regular paragraph.

running npm -g install [email protected]:milisims/tree-sitter-org.git I get

npm ERR! code 7
npm ERR! path /usr/local/lib/node_modules/tree-sitter-org
npm ERR! command failed
npm ERR! command sh -c prebuild-install || node-gyp rebuild
npm ERR! sh: line 1: prebuild-install: command not found
npm ERR! shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | x64
npm ERR! gyp ERR! UNCAUGHT EXCEPTION 
npm ERR! gyp ERR! stack Error: ENOENT: no such file or directory, uv_cwd
npm ERR! gyp ERR! stack     at process.wrappedCwd [as cwd] (node:internal/bootstrap/switches/does_own_process_state:126:28)
npm ERR! gyp ERR! stack     at setopts (/usr/lib64/node_modules/npm17/node_modules/glob/common.js:89:21)
npm ERR! gyp ERR! stack     at new Glob (/usr/lib64/node_modules/npm17/node_modules/glob/glob.js:132:3)
npm ERR! gyp ERR! stack     at Function.glob.hasMagic (/usr/lib64/node_modules/npm17/node_modules/glob/glob.js:98:11)
npm ERR! gyp ERR! stack     at rimraf (/usr/lib64/node_modules/npm17/node_modules/rimraf/rimraf.js:106:36)
npm ERR! gyp ERR! stack     at clean (/usr/lib64/node_modules/npm17/node_modules/node-gyp/lib/clean.js:11:3)
npm ERR! gyp ERR! stack     at Object.self.commands.<computed> [as clean] (/usr/lib64/node_modules/npm17/node_modules/node-gyp/lib/node-gyp.js:41:37)
npm ERR! gyp ERR! stack     at run (/usr/lib64/node_modules/npm17/node_modules/node-gyp/bin/node-gyp.js:80:30)
npm ERR! gyp ERR! stack     at processTicksAndRejections (node:internal/process/task_queues:78:11)
npm ERR! gyp ERR! System Linux 5.19.2-1-default
npm ERR! gyp ERR! command "/usr/bin/node17" "/usr/lib64/node_modules/npm17/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
npm ERR! node:internal/bootstrap/switches/does_own_process_state:126
npm ERR!     cachedCwd = rawMethods.cwd();
npm ERR!                            ^
npm ERR! 
npm ERR! Error: ENOENT: no such file or directory, uv_cwd
npm ERR!     at process.wrappedCwd [as cwd] (node:internal/bootstrap/switches/does_own_process_state:126:28)
npm ERR!     at errorMessage (/usr/lib64/node_modules/npm17/node_modules/node-gyp/bin/node-gyp.js:127:28)
npm ERR!     at issueMessage (/usr/lib64/node_modules/npm17/node_modules/node-gyp/bin/node-gyp.js:133:3)
npm ERR!     at process.<anonymous> (/usr/lib64/node_modules/npm17/node_modules/node-gyp/bin/node-gyp.js:117:3)
npm ERR!     at process.emit (node:events:527:28)
npm ERR!     at process._fatalException (node:internal/process/execution:167:25) {
npm ERR!   errno: -2,
npm ERR!   code: 'ENOENT',
npm ERR!   syscall: 'uv_cwd'
npm ERR! }
npm ERR! 
npm ERR! Node.js v17.7.1

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-09-13T15_59_20_845Z-debug-0.log

This PR will flatten (expr) in paragraph, contents, description & add (nl) nodes.

(expr) nodes, which previously contained a sequence of anonymous "str", "num", and "sym" nodes, are replaced with a corresponding sequence of (str), (num), and (sym) nodes. In cases where there's one (expr) (like block names, properties, directive names, etc.) the (expr) node still exists (but will contain named nodes instead of anonymous nodes). For example, a block starting with #+begin_ab3 is parsed as (expr (str) (num)). In (paragraph), (item), (fndef (definition)), there's now just a sequence of (str), (num), (sym), and (nl) nodes. Well, no (nl)s in (item).

Note that (sym ":") still works for ascii symbols like expr previously did, so we don't need to check for ascii symbols explicitly in a predicate. This makes querying for those symbols quite fast, since they're part of the AST and don't require a predicate to check.

In (paragraph), (fndef (description)), (contents) which is in drawer, block, dynamic_block, and latex_env, newlines are now given a node: (nl)

This will resolve #31 and #26 by enabling queries for single line items: Fixed width area:

(paragraph . (sym ":") @fixed_width_start [(str)(num)(sym)]* @fixed_width_text (nl)) ; matches first line
(paragraph (nl) (sym ":") @fixed_width_start [(str)(num)(sym)]* @fixed_width_text (nl)) ; matches every other line

I tried some combinations of anchors and I couldn't get this down to one pattern to match the first + every other line.

For #31, sexp diary entries will require a predicate / some effort if you want to support multiline expressions as emacs' orgmode does, but single line support is straightforward as above. For the multiline version, lua-match? with something like %b() would be helpful, if you're using neovim.

I am not really sure but from the first look, this grammar does not implement parsing sexp diary entries.

Would you consider adding support for that (or at least accepting PR with this addition)?

I have noticed that the grammar implemented here does not support "Fixed width areas" elements which basically are paragraphs that start with the colon character followed by space.

These elements are usually used to denote code evaluation output. For instance:

#+begin_src python
1 + 1
#+end_src

#+results: 
: 2

Is this a deliberate choice? It would be nice to have such support.

Hi, cool project. How about a Makefile to build and install org.so? This would simplify the installation section in the README a bit and only recompile if sources have changed...