Adding New Language Support

I am putting comments based on email threads in hopes they can make it into the main context above after which the specific comments can be removed.

What top level Netbeans packages/classes are involved in developing a LSP language?

• LSP SPI project found in this project/context
  • Netbeans\ide\api.lsp\src\org\netbeans\spi\lsp\

• Java LSP Server Implementation found in this project/context
  • netbeans\java\java.lsp.server ( https://github.com/apache/netbeans/tree/master/java/java.lsp.server ) which is heavily used in VSCode NetBeans Language Server.

Mailing List References

• https://lists.apache.org/thread.html/r004212da38a55a7779b58ed03e851e6f017b150abe43d7a868bea236%40%3Cdev.netbeans.apache.org%3E
• https://lists.apache.org/thread.html/rd64d4c59ba02339d0a7744b56f569effca6c6ea41f1d214a67ff10cb%40%3Cdev.netbeans.apache.org%3E

There are different ways to implement new languages which includes:

   - JavaCC <https://javacc.github.io/javacc/> is the old way
   - ANTLR <https://www.antlr.org/> is the modern way, but it is not easy to keep the many languages up-to-date; there are a number of tutorials on how to create a NB plugin using ANTLR, many outdated, with the best one in the Apache NetBeans book
   - LSP <https://langserver.org/> seems to be gaining ground, 
   <https://blogs.apache.org/netbeans/entry/lsp-client-demo-ba-sh>.

I was hoping for some guidance and/or pointers to documentation or tutorials on how to add new Language support to Netbeans.

I understand there being movement towards using LSP rather than full language support within a give language module without LSP. Is the preferred route to use LSP or native support? And what is the pros/con of each approach?

For native support...

I found some older articles (1) which provides some details but wasn’t sure if this is still the “best practice” in current Netbeans or if there is another preferred way or even new tutorials. There are also some older documents on the old Netbeans site (4)(5)(6) (not sure where on the new site so providing archived versions) but now sure how up to date these are.

For LSP support...
I find a discussion on similar implementation on a new language here (2) which provides some guidance as well as another’s related to the VS Code LSP activities (3)

It seems for LSP the general consensus is to look at the cpplite and Typescript implementations as good examples of LSP based.

So what other newer guidance is available and/or recommended?

Reference
(1) https://dzone.com/articles/quickstart-guide-language-supp

(2) https://lists.apache.org/x/list.html?dev@netbeans.apache.org:lte=1y:Textmate%20or%20LSP%20for%20Cucumber

(3) https://lists.apache.org/x/thread.html/rf84a830a545d0f3bfb39a07d3f7fcf33ca7ab28deb82bf6551f54e44@%3Cdev.netbeans.apache.org%3E

(4) https://web.archive.org/web/20210118012355/http://wiki.netbeans.org/How_to_create_support_for_a_new_language

(5) https://web.archive.org/web/20210117160954/https://platform.netbeans.org/tutorials/nbm-javacc-lexer.html

(6) https://web.archive.org/web/20210117160955/https://platform.netbeans.org/tutorials/nbm-javacc-parser.html

if you build it you're under control. If someone else builds the LSP server for you then you're not under control, but you 
don't have to build it.

Note also that "implementing a new language" (adding support for it in NetBeans, if I understand your question correctly) is much more than lexing and parsing the language.

You may want to add a specific project type, interact with language specific tools (Makefiles, pom.xml or whatever), add wizards, editor 
folding, editor hints, etc.

From a Netbeans project creation perspectives details defined here https://dzone.com/articles/quickstart-guide-language-supp is a good starting point

LSP Spec: https://microsoft.github.io/language-server-protocol/specification

Old wiki site provided legacy details on how to create a new language https://web.archive.org/web/20210118012355/http://wiki.netbeans.org/How_to_create_support_for_a_new_language

This in turn indicates 

The replacement tutorials for all the text that follows below are:

* NetBeans JavaCC Lexer Tutorial http://platform.netbeans.org/tutorials/nbm-javacc-lexer.html (Web Archive https://web.archive.org/web/20210117160954/https://platform.netbeans.org/tutorials/nbm-javacc-lexer.html )

* NetBeans JavaCC Parser Tutorial http://platform.netbeans.org/tutorials/nbm-javacc-parser.html (Web Archivew https://web.archive.org/web/20210117160955/https://platform.netbeans.org/tutorials/nbm-javacc-parser.html )

From Antonio:

Maintaining an ANTLR grammar over the years is difficult and time consuming (see for instance all the changes to the NetBeans C++ grammar in NetBeans [1]).

LSP servers, when mantained by knowledgeable teams, may give better results with less effort. Apple, for instance, decided to move to clangd (from libclang) a few years back ([2]).

For simpler languages/DSLs I think Eclipse's XText 
https://www.eclipse.org/Xtext/ outperforms JetBrain's MPS 
https://www.jetbrains.com/mps/ (the generated parsers could then be 
reused in NetBeans).


[1] https://github.com/emilianbold/netbeans-releases/commits/master/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/cppparser.g
[2] https://lists.llvm.org/pipermail/cfe-dev/2018-April/057668.html

From Chris L:

What is needed here for every language is:

- Semantic syntax highlighting (more than with textmate I think)
- Formatting (custom implementation or LSP?)
- Refactoring (renaming, find usages, go to, etc. – LSP needed)
- Hints/Fixes/Suggestions, etc – LSP needed + custom implementation if LSP doesn’t have that much
- Braces matching, automatic closing of braces (custom implementation?)
- Code completion – LSP needed
- Code templates – custom implementation
- Showing errors of the Syntax or Control flow – LSP needed

So as an alternative to ANTLR which of course is good, we Need those stuff for textmate too. Where we can all implement the stuff based on textmate files and not only on g and g4 lexer and parser. If possible.

From Chris Lentz (https://lists.apache.org/x/thread.html/r4e0767b237d64cbb1a15f19b88550c94ec0b2e246371dcd0e321ec8e@%3Cdev.netbeans.apache.org%3E)

- Textmate is used for

o Syntax highlighting

o Syntax errors

o Braces matching and automatically closing after type open brace

o Indentation and auto indentation on enter

o Folding code blocks

- LSP is used for

o Code completion

o Hints

o Occurencies

o Formatting

Examples of Implementations:

• From Jan Lahoda, Example of Bash LSP integration available https://blogs.apache.org/netbeans/entry/lsp-client-demo-ba-sh
• Java LSP Server Implementation found in this project/context
  • netbeans\java\java.lsp.server ( https://github.com/apache/netbeans/tree/master/java/java.lsp.server ) which is heavily used in VSCode NetBeans Language Server.
• From Matthias ( https://lists.apache.org/x/thread.html/rc1bba83ec7e56de4722402d8b9c28c55c0cd9ac4cf0a9ebbe59e6b4d@%3Cdev.netbeans.apache.org%3E )
  • I suggest to have a look at the integration of TypeScript or cpplite inside the codebase. That are from my POV the two reference projects.
  • The core point is to look at how programmatic/dynamic registration is possible inside a netbeans feature and then figure out, how you can do that from your target language. For example in the TypeScript Editor, the integration of "Comment code" was missing. The issue could be tracked down into the fact, that the TypeScript Editor code did not  provide a custom EditorKit, that returned the right mimetype. The EditorKit was defined an then registered in the MimeLookup of the typescript mimetype and picked up by the netbeans infrastructure.
  • Most probably a hint provider could be registered similarly. You'd need to check how the hints are gathered and where/how contributors are found, but it should be doable.
• Example for Cucumber https://lists.apache.org/x/list.html?dev@netbeans.apache.org:lte=1y:Textmate%20or%20LSP%20for%20Cucumber
• Old site on how to create a new language https://web.archive.org/web/20210118012355/http://wiki.netbeans.org/How_to_create_support_for_a_new_language

ANTLR Based:

the best 'tutorial' is chapter 11 of Pro Apache NetBeans <https://www.amazon.com/Pro-Apache-NetBeans-Building-Applications/dp/1484253698> book, 
even though it doesn't use the modern trend of TextMate and LSP, but it is based on ANTLR.

LSP Server Implementors

https://microsoft.github.io/language-server-protocol/implementors/servers/

And leverage the tm grammars from some place like here

https://github.com/Chris2011/netbeans-textmate-files

From Chris L:

So use a Textmate language from wherever you want. I just made a little repo as you already mentioned to collect all of them to use. 

And then the LSP for that language, if it exists.  

I would prefer to have it inside NetBeans, because we are more flexible with the APIs making friends if needed or taking whatever is needed.

On the other hand I like 3rd Party pllugins more because of different release cycles. If we encounter a bug today, I can fix it today and 
release it today. Not possible with NetBeans if it is not a real blocker bug. Problem is again the problem that not all APIs are public 
to use them within your 3rd party plugin.

When adding a new language, should this generally be done as a separate plugin or be integrated within the netbeans codebase itself? 

I guess this could go either way but assume making an external plugin allows more flexibility to management of it, 
avoids growing the netbeans codebase significantly, but has a little less visibility up front (and not part of
“Netbeans” proper) with the need to manage things separately from the netbeans master repository

A third party plugin only make sense if all needed APIs inside NetBeans are public and can be used from the 3rd-party-plugin which is not often the case. See this ticket for the nbts plugin: https://github.com/Everlaw/nbts/issues/81#issuecomment-272748787