update tokenizer comment

pull/55/head
Donald Pipowitch 7 years ago committed by GitHub
parent bc2898670e
commit 80288d2ad2

@ -122,7 +122,7 @@
* Syntactic Analysis. * Syntactic Analysis.
* *
* 1. *Lexical Analysis* takes the raw code and splits it apart into these things * 1. *Lexical Analysis* takes the raw code and splits it apart into these things
* called tokens by a thing called a tokenizer (or lexer). * called tokens by a thing called a tokenizer (or lexer or scanner).
* *
* Tokens are an array of tiny little objects that describe an isolated piece * Tokens are an array of tiny little objects that describe an isolated piece
* of the syntax. They could be numbers, labels, punctuation, operators, * of the syntax. They could be numbers, labels, punctuation, operators,

Loading…
Cancel
Save