The lexer, or
defines how the contents of a file is broken into tokens.
The lexer serves as a foundation for nearly all of the features of custom language plugins, from basic syntax highlighting to advanced code analysis features.
The API for the lexer is defined by the
The IDE invokes the lexer in three main contexts, and the plugin can provide different lexer implementations for these contexts:
Syntax highlighting: The lexer is returned from the implementation of the
SyntaxHighlighterFactoryinterface which is registered in the
Building the syntax tree of a file: the lexer is expected to be returned from
ParserDefinition.createLexer(), and the
ParserDefinitioninterface is registered in the
Building the index of the words contained in the file: if the lexer-based words scanner implementation is used, the lexer is passed to the
The lexer used for syntax highlighting can be invoked incrementally to process only the changed part of a file, whereas lexers used in other contexts are always called to process an entire file, or a complete language construction embedded in a file in a different language.
A lexer that can be used incrementally may need to return its state, which means the context corresponding to each position in a file.
For example, a
could have separate states for top level context, comment context and string literal context.
An important requirement for a syntax highlighting lexer is that its state must be represented by a single integer number returned from
That state will be passed to the
method, along with the start offset of the fragment to process, when lexing is resumed from the middle of a file.
Lexers used in other contexts can always return
0 from the
The easiest way to create a lexer for a custom language plugin is to use JFlex.
adapt JFlex lexers to the IntelliJ Platform Lexer API.
We have a
patched version of JFlex
that can be used with the lexer skeleton file located at tools/lexer/idea-flex.skeleton in the
IntelliJ IDEA Community Edition
source to create lexers compatible with
The patched version of JFlex provides a new command line option
--charat which changes the JFlex generated code so that it works with the IntelliJ Platform skeleton.
--charat option passes the source data for lexing as a
and not as an array of characters.
For developing lexers using JFlex, the GrammarKit plugin can be useful. It provides syntax highlighting and other useful features for editing JFlex files.
Types of tokens for lexers are defined by instances of
A number of token types common for all languages are defined in the
Custom language plugins should reuse these token types wherever applicable.
For all other token types, the plugin needs to create new
instances and associate with the language in which the token type is used.
instance should be returned every time a particular token type is encountered by the lexer.
An important feature which can be implemented at lexer level is mixing languages within a file, for example, embedding fragments of Java code in some template language.
If a language supports embedding its fragments in another language, it needs to define the chameleon token types for different types of fragments which can be embedded, and these token types need to implement the
The lexer of the enclosing language needs to return the entire fragment of the embedded language as a single chameleon token, of the type defined by the embedded language.
To parse the contents of the chameleon token, the IDE will call the parser of the embedded language through a call to