unicode_ident.hpp File Reference

UTF-8-aware identifier building blocks for the classic (Bertini 1) Qi grammar. More...

#include <string>
#include <boost/spirit/include/qi.hpp>
#include <boost/regex/pending/unicode_iterator.hpp>
#include "bertini2/naming.hpp"

Include dependency graph for unicode_ident.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	bertini
namespace	bertini::parsing
namespace	bertini::parsing::classic

Detailed Description

UTF-8-aware identifier building blocks for the classic (Bertini 1) Qi grammar.

The classic parser keeps byte iterators (std::string::const_iterator) everywhere – the symbol tables (qi::symbols<char,...>) and error handling are all keyed on char. To accept Unicode letters (Ω, α, CJK, ...) as variable names without churning that whole pipeline, we make only the identifier rule UTF-8-aware: code points are transiently decoded to UTF-32 via boost::u8_to_u32_iterator and classified by Unicode character properties (never the platform locale / wchar_t, which is the cp1252/latin-1 footgun). Because the source string is already UTF-8, a matched identifier's raw bytes are* the desired std::string attribute.

Two self-contained Spirit primitives are provided so composition never needs a unary/binary Spirit operator applied to a bare custom primitive (which requires the terminal machinery): utf8_identifier_parser matches a whole identifier and exposes it as a std::string; utf8_ident_boundary_parser is a zero-width negative lookahead used to stop a known symbol from matching a prefix of a longer identifier.