Vim's 25th anniversary and the release of Vim 8

2016 was a big year for project anniversaries. The Linux kernel, of course, turned 25. And Vim, that other iconic text editor, also celebrated its 25th anniversary. Bram Moolenaar, Vim's benevolent dictator for life, marked the occasion at the end of last year with a short note on the project's web site:

Vim's ancestry goes back even further. Vim, or "Vi IMproved", was started as a clone of the venerable vi editor, which was created by Bill Joy in 1976. Vi has its own complicated lineage: vi was the visual mode of Joy's own ex editor, which in turn can be traced back to Ken Thompson's ed editor from Bell Labs. O'Reilly's Learning the vi and Vim Editors has a complete family tree of the vi editors. Vim's 25th anniversary also coincides with the official release of version 8 on September 12, 2016. Vim 8 is the first major release since the appearance of version 7 in May 2006. Even the last point release, 7.4, was released back in August 2013.

Moolenaar happened to mark the occasion with a presentation [YouTube] at Google's Zurich office in December. He talked about the project's history as well as some of the decisions that went into the latest release. Moolenaar acknowledged he's been slow making new releases, but he believes the deliberate pace has been good for the project: "In the beginning it was easier to add new features," he said. "The further you get the harder it is to add a feature that's actually adding something... I always hesitate to add new features unless it's clear that it's good."

At last, Vim 8 has arrived. The release contains several new features, including some functionality that users have been waiting to see appear in Vim.

Packages

Managing Vim extensions used to require a separate plugin (such as pathogen) to be able to load other plugins automatically when the editor starts. Vim can now do this on its own. Users just need to move their plugins into a directory with their other configuration files. Vim 8 defines a new variable packpath, which points to $HOME/.vim/pack/*/start. This is now the default plugin directory for Vim. Plugins that use this directory structure will automatically be loaded at startup.

The introduction of packpath also includes new commands to activate plugins manually after Vim starts:

• :packadd will add a plugin from packpath
• :packloadall will load all packages under packpath

Vim 8 mirrors the directory structure many Vim users are already accustomed to. Vim 8 does not offer a package manager, but only provides the built-in framework for users to drop in their own plugins. Extensions added to the packpath folder can be managed with version control alongside the rest of a user's configuration files.

Asynchronous plugins

Vim has always been a single-threaded application, which meant that calling an external command from within Vim would block the editor for the duration of the command. While the core editor remains single-threaded, some changes have been made to allow the editor to proceed while external commands are running. Vim 8 introduces changes to its scripting language that enable communication between the core editor and other processes along with changes to start and stop those processes. Although these features are primarily of interest to plugin developers, the addition of interprocess communication and job control will have direct benefits for Vim's end users.

For example, Vim already has a built-in make command. And it's also possible to call the operating system's external make command. But both of these uses block the editor until the command finishes.

Vim 8 allows plugin developers to asynchronously execute commands in a separate process and pass the results back to Vim upon completion. Plugins that make use of the new API will no longer block Vim during long computations. For example, this means that end users can now run make or grep commands from within Vim and still be able to continue using the editor while the background job runs.

Two new additions to Vimscript make this possible. Channels allow the editor to exchange messages with other processes running in the background. Messages passed over the channels are encoded in JSON. And the job control addition can be used to start and stop background processes and communicate with running processes through channels.

Existing plugins will need to be patched to take advantage of the new features. One way to try out the new API in Vim 8 is with the new AsyncRun plugin. Just drop it into your packpath directory and you can run any shell command as a background job and still capture the output within Vim.

With Vim 8, there are now three ways to run make as an external command:

• :make

  Run Vim's built-in make. This will block the editor.

• :!make

  Run the operating system's make command. Prepending ! to a command tells Vim to run a shell command. Even though make is running in a separate process, Vim is still blocked while it waits for make to return.

• :AsyncRun make

  Use the AsyncRun external plugin to run the operating system's make command. This is new in Vim 8 and requires a plugin that takes advantage of the job control API. This is just like :!make, but now Vim doesn't have to wait for the result of make to return.

The asynchronous features are only available to plugins and do not change the core editor. Adding thread support to Vim is challenging for historical reasons, Moolenaar said in his presentation at Google: "Vim is really single threaded. Nearly all the code depends on that. Making it multi-threaded would be a really big, difficult thing with a lot of bugs. It's a lot easier if you run something in a different process and then add interprocess communication."

User demand for a multi-threaded Vim (e.g. a patch for supporting multiple threads) and its rejection is what prompted the Neovim fork in January 2014. In his presentation, Moolenaar concedes that Neovim did create some pressure to add a way to handle asynchronous jobs. Ultimately, the new feature was a compromise to give plugin authors the features they wanted without having to change Vim's fundamental design.

"Did they [Neovim] influence Vim development? A little bit. They found some bugs and fixed them and then gave them back to me," Moolenaar said in the presentation. "Of course jobs and channels are one thing where plugin authors told me 'hey this thing is something Neovim is working on and we should really have it'. I didn't take the Neovim implementation because I didn't like it that much."

A few sundry additions

Users who are new to Vim will have a better out of box experience with Vim 8. If no configuration file is present, Vim now loads a default configuration with some friendlier settings. Vim 8 now starts with vi compatibility disabled. This ensures new users will have access to Vim's extended feature set and can start using plugins right out of the box.

The default settings also include:

• Command history and auto-completion
• Enabling syntax highlighting
• Automatic file-type detection
• Saving the file position on exit

In addition to asynchronous support, Vim 8 also introduces a few other additions to the Vimscript language. Vimscript already has references to functions. Vim extends this with partials, which allow a calling function to bind the arguments for a referenced function. This can be useful for callbacks on channels, for example. Vim also introduces lambdas to allow defining anonymous functions to pass as arguments. In addition, closures are now available so that lambda functions can use the variables in the scope in which they are defined.

Users who work inside Vim with split windows can now rely on window identifiers remaining consistent when new splits are added. In Vim 8, split windows now have a unique ID separate from their window number. The graphical version of Vim now supports GTK+ 3. Users in older desktop environments should not be impacted: Vim 8 will still fallback to GTK+ 2. And Windows users have gained support for DirectX.

Anyone who has waited too long to update their computer should be aware that Vim 8 removes support for several old platforms, including MS-DOS, Windows 95, and OS/2.

Don't forget to smile

Finally, users who are burned out after a long day of hacking should try out Vim's new :smile command. (Technically, this feature was added to Vim 7.4 on the last day of 2015.) The changelog for this feature simply states that "Vim users are not always happy.", so this patch was added as a solution to that problem.

More details can be found in the full list of changes for Vim 8. Vim's built in documentation includes more detailed instructions on how to use the new features, as well as lots of information for plugin authors on changes to the scripting language. You can get Vim directly from the project web site, although the latest version has already started appearing in the package managers of many distributions.