Preprocessing tokens

A token is the basic lexical unit of the language. All source input must be formed into valid tokens by translation phase seven. Preprocessing tokens (pp-tokens) are a superset of regular tokens. Preprocessing tokens allow the source file to contain non-token character sequences that constitute valid preprocessing tokens during translation. There are four categories of preprocessing tokens:

• Header filenames, meant to be taken as a single token.

• Preprocessing numbers

• All other single characters that are not otherwise (regular) tokens. See the example in ``Preprocessing numbers''.

• Identifiers, numeric constants, character constants, string literals, operators, and punctuators.

Preprocessing numbers

• A preprocessing number is made up of a digit, optionally preceded by a period, and may be followed by letters, underscores, digits, periods, and any one of: e+ e- E+ E-.

• Preprocessing numbers include all valid number tokens, plus some that are not valid number tokens. For example, in the macro definition:

  #define R 2e ## 3
  

  the preprocessing number 2e is not a valid number. However, the preprocessing operator ## will ``paste'' it together with the preprocessing number 3 when R is replaced, resulting in the preprocessing number 2e3, which is a valid number.