[asciidoc-lang-dev] How do we proceed?

[David Jencks <david.a.jencks@xxxxxxxxx>]

I’m thrilled that after a long wait and an unbelievable amount of mostly invisible work by Dan and others this mailing list is finally open!

There are a lot of things I’d like to bring up or participate in and I’m wondering what the most appropriate way might be.
I’ll mention 5 that seem to me to be in different categories so we have something concrete to think about.
Some aspects I wonder about  are:

* What is the most appropriate place to bring each issue up?
This mailing list?
An issue on the Eclipse GitLab instance?
* Where will discussion take place? 
(same choices, AFAICT)
* What is the basis for concluding the issue? 
Consensus?
Lazy consensus?
* How and where is the conclusion recorded?
* To what extent is there are commit-then-review vs review-then-commit? 
(This is obviously an inaccurate description in the age of Git, but still I think worth thinking about).
* Do we want some way to categorize these types of issues or discussions?

I think for all of these I’m pretty much in favor of discussion on this list, followed by possibly opening an issue, recording the main points and agreement from the discussion.
I really like Apache’s idea of lazy consensus with a 72 hour window.

To be clear, I’m not trying to discuss any of these issues on this thread, just asking how to bring them up and the location of the path the discussion might take.

1. Open block syntax: ‘~’ as the open block delimiter.
 This is a new behavior that will not change any existing behavior, and I believe happens to be universally waited for :-).

2. Macro attribute list parsing.
I recently tried to accurately document this and couldn’t :-).
I’m also not sure the current rules can be implemented by a normal (e.g. Antlr4) lexer in any reasonable way.
I would like the treatment of quotes to be simple, easy to understand, easy to explain, and easy to parse.
For instance, `Unescaped quotes indicate attribute boundaries`.
(This may be a much larger change than desirable, but it satisfies my criteria).
This is a change to definitely existing but perhaps effectively undocumented behavior.

3. “File system” independence.
I would like the spec to establish the principle that all references outside the document being parsed are uninterpreted tokens as far as the core Asciidoc processor itself is concerned, and are delegated to some sort of extension (with a default implementation probably referring to the file system).
As background, Antora currently has a converter whose sole task is to intercept some of these tokens and interpret them with their Antora meaning rather than the built in meaning.
Somewhat similarly , jekyll-asicidoc does not currently recognize “permalink” attributes, because a similar mechanism would be needed and no one has dared to write it yet.
As far as Asciidoctor is concerned, this would be an internal architectural change only visible to those who wish to take advantage of it: by default there would be no changed behavior.
It would be changed behavior for people writing code that uses Asciidoctor (e.g. Antora, jekyll-asciidoc).

4. “If it looks like an extension, it’s an extension” (except, presumably, for `include::`)
There is a lot of built-in behavior that is triggered by syntax that looks like a block macro or inline macro.
I’d like these to all be implemented through the extension mechanism, or at least appear to the outside as if they are: in particular, a  registered user extension can replace any of these.
Theoretically this shouldn’t affect the default behavior of an Asciidoc processor, but it does allow users more possibilities.

5. Grammar!
How and where are we going to discuss what format our grammar will take and how to develop it?
AsciiDoc is not context-free, so a completely generated parser is not likely to be possible.
I’m not an expert, but after some thought, I think it’s likely that a mostly-generated Antlr 4 parser is possible, with considerable communication between the lexer and parser.
This would be pretty nice, because the generated parts can be generated into Java, Javascript, Go, C++ and I think (unofficially) Ruby.
Since there are projects such as the Java AsciiDoc implementation proposal, grammar development could happen only there but I think it would be better to have the discussion and files from which the code is generated here.
if we start with something like a partially generated Antlr 4 parser, how will the source files and necessary hand-coded bits turn into the spec?
As far as I know this is effectively new territory exploring where AsciiDoc will be in the future.

Thanks,
David Jencks