Boost C++ Libraries of the most highly regarded and expertly designed C++ library projects in the world. Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

This is the documentation for an old version of boost. Click here for the latest Boost documentation.

C++ Boost

TokenizerFunction Concept

A TokenizerFunction is a functor whose purpose is to parse a given sequence until exactly 1 token has been found or the end is reached. It then updates the token, and informs the caller of the location in the sequence of the next element immediately after the last element of the sequence that was parsed for the current token.

Refinement of

Assignable, CopyConstructable


X A type that is a model of TokenizerFunction
func Object of type X
tok Object of Token
next iterator that points to the first unparsed element of the sequence being parsed
end iterator that points to the past the end of the sequence being parsed


A token is the result of parsing a sequence.

Valid expressions

In addition to the expression in Assignable and CopyConstructable the following expressions are valid

Name Expression Return type
Functor func(next, end, tok) bool
reset reset() void

Expression semantics

In addition to the expression semantics in Assignable and CopyConstructable, TokenizerFunction has the following expression semantcs

Name Expression Precondition Semantics Postcondition
operator() func(next, end, tok) next and end are valid iterators to the same sequence. next is a reference the function is free to modify. tok is constructed. The return value indicates whether a new token was found in the sequence [next,end) If the return value is true, the new token is assigned to tok. next is always updated to the position where parsing should start on the subsequent call.
reset reset() None Clears out all state variables that are used by the object in parsing the current sequence. A new sequence to parse can be given.

Complexity guarantees

No guarantees. Models of TokenizerFunction are free to define their own complexity






Valid HTML 4.01 Transitional

Revised 25 December, 2006

Copyright © 2001 John R. Bandela

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at