Boost.Locale provides a boundary analysis tool, allowing you to split text into characters, words, or sentences, and find appropriate places for line breaks.
Boost.Locale provides 2 major classes for boundary analysis:
Each of the classes above use an iterator type as template parameter. Both of these classes accept in their constructor:
Each of them provide a members
find() that allow to iterate over the selected segments or boundaries in the text or find a location of a segment or boundary for given iterator.
Convenience a typedefs like ssegment_index or wcboundary_point_index provided as well, where "w", "u16" and "u32" prefixes define a character type
char32_t and "c" and "s" prefixes define whether
CharType const * are used.
The text segments analysis is done using segment_index class.
It provides a bidirectional iterator that returns segment object. The segment object represents a pair of iterators that define this segment and a rule according to which it was selected. It can be automatically converted to
To perform boundary analysis, we first create an index object and then iterate over it:
"To", " ", "be", " ", "or", " ", "not", " ", "to", " ", "be", ",", " ", "that", " ", "is", " ", "the", " ", "question", ".",
This sentence "生きるか死ぬか、それが問題だ。" (from Tatoeba database) would be split into following segments in
ja_JP.UTF-8 (Japanese) locale:
"生", "きるか", "死", "ぬか", "、", "それが", "問題", "だ", "。",
The boundary analysis that is done by Boost.Locale is much more complicated then just splitting the text according to white space characters, even thou it is not perfect.
By default segment_index's iterator return each text segment defined by two boundary points regardless the way they were selected. Thus in the example above we could see text segments like "." or " " that were selected as words.
For example, by calling
Before starting the iteration process, specify a selection mask that fetches: numbers, letter, Kana letters and ideographic characters ignoring all non-word related characters like white space or punctuation marks.
So the code:
"To", "be", "or", "not", "to", "be", "that", "is", "the", "question",
And the for given text="生きるか死ぬか、それが問題だ。" and rule(word_ideo), the example above would print.
"生", "死", "問題",
You can access specific rules the segments where selected it using segment::rule() member function. Using a bit-mask of rules.
Segment 生 contains: ideographic characters Segment きるか contains: kana characters Segment 死 contains: ideographic characters Segment ぬか contains: kana characters Segment 、 contains: white space or punctuation marks Segment それが contains: kana characters Segment 問題 contains: ideographic characters Segment だ contains: kana characters Segment 。 contains: white space or punctuation marks
One important things that should be noted that each segment is defined by a pair of boundaries and the rule of its ending point defines if it is selected or not.
In some cases it may be not what we actually look like.
For example we have a text:
Hello! How are you?
And we want to fetch all sentences from the text.
The sentence rules have two options:
Naturally to ignore sentence separators we would call segment_index::rule(rule_type v) with sentence_term parameter and then run the iterator.
However we would get the expected segments:
Sentence [Hello! ] Sentence [are you? ]
The reason is that "How\n" is still considered a sentence but selected by different rule.
This behavior can be changed by setting segment_index::full_select(bool) to
true. It would force iterator to join the current segment with all previous segments that may not fit the required rule.
So we add this line:
Right after "map.rule(sentence_term);" and get expected output:
Sentence [Hello! ] Sentence [How are you? ]
Sometimes it is useful to find a segment that some specific iterator is pointing on.
For example a user had clicked at specific point, we want to select a word on this location.
This function returns the iterator to the segmet such that p points to.
if the iterator lays inside the segment this segment returned. If the segment does not fit the selection rules, then the segment following requested position is returned.
The boundary_point_index is similar to segment_index in its interface but as a different role. Instead of returning text chunks (segments, it returns boundary_point object that represents a position in text - a base iterator used that is used for iteration of the source text C++ characters. The boundary_point object also provides a rule() member function that defines a rule this boundary was selected according to.
Lets see an example of selecting first two sentences from a text:
First two sentences are: First sentence. Second sentence!
Lets change an example above a little:
If we run our program as is on the sample above we would get:
First two sentences are: First sentence. Second
Which is not something that we really expected. As the "Second\n" is considered an independent sentence that was separated by a line separator "Line Feed".
However, we can set set a rule sentence_term and the iterator would use only boundary points that are created by a sentence terminators like ".!?".
So by adding:
Right after the generation of the index we would get the desired output:
First two sentences are: First sentence. Second sentence!
You can also use boundary_point::rule() member function to learn about the reason this boundary point was created by comparing it with an appropriate mask.
Would give the following output:
There is a sentence terminator: [First sentence. |Second sentence! Third one?] There is a sentence separator: [First sentence. Second |sentence! Third one?] There is a sentence terminator: [First sentence. Second sentence! |Third one?] There is a sentence terminator: [First sentence. Second sentence! Third one?|]
Sometimes it is useful to find a specific boundary point according to given iterator.
It would return an iterator to a boundary point on p's location or at the location following it if p does not point to appropriate position.
For example, for word boundary analysis:
For example if we want to select 6 words around specific boundary point we can use following code:
That would print:
be or not to be, that