The Spirit.Lex library implements several components on top of possibly different lexer generator libraries. It exposes a pair of iterators, which, when dereferenced, return a stream of tokens generated from the underlying character stream. The generated tokens are based on the token definitions supplied by the user.
Currently, Spirit.Lex is built on top of Ben Hansons excellent Lexertl library (which is a proposed Boost library). Lexertl provides the necessary functionality to build state machines based on a set of supplied regular expressions. But Spirit.Lex is not restricted to be used with Lexertl. We expect it to be usable in conjunction with any other lexical scanner generator library, all what needs to be implemented is a set of wrapper objects exposing a well defined interface as described in this documentation.
For the sake of clarity all examples in this documentation assume Spirit.Lex to be used on top of Lexertl.
Building a lexer using Spirit.Lex is highly configurable, where most of this configuration is done at compile time. Almost all of the configurable parameters have generally useful default values, allowing project startup to be a easy and straightforward task. Here is a (non-complete) list of features you can tweak to adjust the generated lexer instance to the actual needs:
Special care has been taken during the development of the library that optimal code will be generated regardless of the configuration options selected.
The series of tutorial examples of this section will guide you through some common use cases helping to understand the big picture. The first two quick start examples (Lex Quickstart 1 - A word counter using Spirit.Lex and Lex Quickstart 2 - A better word counter using Spirit.Lex) introduce the Spirit.Lex library while building two standalone applications, not being connected to or depending on any other part of Spirit. The section Lex Quickstart 3 - Counting Words Using a Parser demonstrates how to use a lexer in conjunction with a parser (where obviously the parser is built using Spirit.Qi).