New features in Neovim 0.5

Neovim 0.5, the fifth major version of the Neovim editor, which descends from the venerable vi editor by way of Vim, was released on July 2. This release is the culmination of almost two years of work, and it comes with some major features that aim to modernize the editing experience significantly. Highlights include native support for the Language Server Protocol (LSP), which enables advanced editing features for a wide variety of languages, improvements to its Lua APIs for configuration and plugins, and better syntax highlighting using Tree-sitter. Overall, the 0.5 release is a solid upgrade for the editor; the improvements should please the existing fan base and potentially draw in new users and contributors to the project.

The Neovim project was started by Thiago Padilha in 2014 shortly after his patch to introduce multi-threading capabilities to Vim was rejected without much in the way of feedback. This event was the major trigger that led Padilha to create this fork, with the explicit aim of improving the usability, maintainability, and extensibility of Vim while facilitating a more open and welcoming environment.

A built-in LSP client

The Language Server Protocol is an open-source specification that standardizes programming language features across different source code editors and integrated development environments (IDEs). It facilitates communication between code-editing tools (clients), and locally running language servers to provide language-specific smarts such as auto-completion, find-and-replace, go-to-definition, diagnostics, and refactoring assistance.

Prior to the development of LSP, the work of providing support for a programming language had to be implemented for each IDE or text editor, either directly in the code, or through its extension system, which led to varying levels of support across language and editor combinations. The LSP standard enables the decoupling of language services from the editor into a self-contained piece so that language communities can concentrate on building a single server that has a deep understanding of a language. Other tools can then provide advanced capabilities for any programming language simply by integrating with the existing language servers.

While it was already possible to use LSP in Neovim with the help of third-party plugins, the 0.5 release adds native LSP support to Neovim for the first time. The introduction of LSP in Neovim allows the editor to act as a client, informing a language server about user actions (such as executing a "go-to-definition" command); the server answers the request with the appropriate information, which could be the location of the definition for the symbol under the cursor. That will allow the editor to navigate to the specified location in the file or project.

The interface provided by the Neovim LSP client is a general one, so it does not support all of the features that are available in third-party LSP plugins (e.g. auto-completion). It was built to be extensible, though, so it includes a Lua framework that allows plugins to add features not currently supported in the Neovim core. Setting up individual language servers for the editor can be done using the nvim-lspconfig plugin, which helps with the launching and initialization of language servers that are currently installed on the system. Note that language servers are not provided by Neovim or nvim-lspconfig, they must be installed separately. There is a long list of LSP servers supported by the nvim-lspconfig plugin.

Lua integration

Initial support for the Lua programming language in Neovim landed in the 0.2.1 release in 2017. It has seen continued development and deeper integration in the editor since then, most notably with the addition of a Neovim standard library for Lua in the 0.4 release in 2019. The Neovim developers expect Lua to become a first-class scripting language in the editor, thus providing an alternative to VimL, which is the scripting language inherited from Vim. Neovim 0.5 takes big strides toward the realization of this goal by improving the Lua API and adding init.lua as an alternative to init.vim for configuring the editor.

A good explanation of the rationale behind the decision to embed Lua in Neovim can be found in a video of a talk by Justin M. Keyes, a lead maintainer for the project. In summary, Lua is a more approachable language than VimL due to its simplicity and ease of embedding. It is also an order of magnitude faster than VimL. Neovim supports Lua 5.1, which was released in 2006, rather than more recent versions of Lua, such as 5.3 or 5.4 (released 2015 and 2020 respectively), mostly due to LuaJIT, which only supports Lua 5.1. The motivation for maintaining compatibility with LuaJIT stems from its significant performance advantages over the standard Lua compiler.

Adding Lua to Neovim has made it easier to extend the capabilities of the editor and contribute to its core code, especially for users who have been put off by VimL, which is not a language that is used outside of Vim. Since Lua is also heavily used for scripting video games and for extending other programs written in a variety of languages (C, C++, Java, etc.), there is an abundance of resources available for learning the language, along with examples that show how to use it to interact with APIs from other languages. This wealth of information on Lua makes it possible for new plugin authors and aspiring Neovim contributors to get up to speed with the language quickly.

The Lua support in Neovim has led to it becoming the preferred language for how newer Neovim features, such as the LSP client, are being exposed. Using these APIs can only be done with Lua, since VimL cannot be used to interact with them. However, VimL support in Neovim is not going anywhere, and the Neovim developers do not anticipate any reason to deprecate it, so migrating an existing init.vim configuration to init.lua, or porting a VimL plugin to Lua for the sake of it is completely optional at this time. The only caveat is that using these Neovim APIs (such as LSP or Tree-sitter) in an init.vim configuration or VimL plugins can only be done by embedding some Lua snippets within the existing VimL code.

Although deeper Lua integration is seen as one of the main achievements of the 0.5 release, not all of the reactions toward the push to supplant VimL in the editor core have been positive. There is some concern that the emphasis on Lua APIs, and Lua-only plugins, will lead to a split in the plugin community where an increasing number of plugins will be Neovim-only (as opposed to supporting both Vim and Neovim). Also, an improved and not entirely backward-compatible version of VimL (currently referred to as Vim9) is under active development by Vim creator Bram Moolenaar and other Vim contributors. It is not entirely clear whether the Neovim maintainers plan to support Vim9, since they are more invested in Lua. At the time of this writing, there are already several Lua plugins that work only in Neovim, and a handful of Vim9 plugins that work only in Vim. It is therefore easy to speculate that the ecosystems for both projects may diverge significantly in the near future as there are currently no plans to bring a similar level of Lua integration into Vim.

Tree-sitter

Tree-sitter is a new parsing system that aims to replace the limited, regular-expression-based, code-analysis capabilities that are prevalent in current developer tools. It is a high-performance parser generator that can build parsers to create an incremental syntax tree for a source file, and can efficiently update the syntax tree in realtime as the file is being edited. In Neovim 0.5, support for Tree-sitter has been added to the editor core, although it is currently classed as experimental due to some known bugs along with performance issues for large files. The expectation is that it will become stable in the next major release (0.6), which should be expected in a year or two judging from past releases.

Using Tree-sitter in Neovim makes it possible for the editor to understand the code in a source file as a tree of programming language constructs (such as variables, functions, types, keywords, etc.), and use that information to handle those constructs consistently. When a Tree-sitter parser is installed and enabled for a specific language, the editor's syntax highlighting will be based on the syntax trees it provides; this results in improvements to the use of color to outline the structure of the code more clearly. In particular, object fields, function names, keywords, types, and variables will be highlighted more consistently throughout the file.

Tree-sitter is also able to do incremental parsing, which keeps the syntax tree up to date as the code is being edited. This puts an end to the practice of re-parsing an entire file from scratch in order to update its syntax highlighting after a change is made, which is currently the case with regular-expression-based highlighting systems. That leads to significant speed improvements.

Tree-sitter has been lauded for its improved syntax-highlighting capabilities, but it also enables the definition of language-aware text objects better suited to editing code than what is provided by default in the editor. The nvim-treesitter-textobjects module allows the creation of text objects for constructs like classes, functions, parameters, conditionals, and more, which can be manipulated just as easily as words or sentences. Several examples of the Tree-sitter-based highlighting can be seen in the gallery for the nvim-treesitter repository.

Wrapping up

The features above make up the bulk of this release, but Neovim 0.5 also includes improvements and bug fixes to the user interface, as well as smaller features such as support for remote plugins written in Perl 5.22+ on Unix platforms. It is also worth mentioning that around 1000 Vim patches were merged in this release, updating various aspects of the editor. The full list of changes, fixes and refinements can be seen in the release notes linked above.

The Neovim project uses GitHub issues to track all feature and bug requests, so a list of closed issues for the 0.5 milestone is available for a further exploration of the changes that made it into this release. The planning for subsequent releases is detailed on the project's roadmap page, while priorities are tracked through GitHub milestones. Contributions from the community are welcome, of course, and the project maintainers may be reached via Gitter, Matrix, or the #neovim room on irc.libera.chat.