9e1ccfcd4c
Patch fixes issue of not ignoring UNICODE format characters in match
and peek methods, adds explicit assertions checks for code assumptions
and makes handling of ASCII '\r', '\n' and UNICODE U+2028, U+2029 line
ends uniform.
It was rather tricky to fix format character issue and I spend some
time figuring out what TokenStream assumes about LineBuffer that
breaks my initial thoughts on the patch in cases like very long
sequences of format characters that do not fit in the buffer. I
fixed that but it made the code rather unclear so I put explicit
checks for assumptions/preconditions to help with debugging.
I added Context.check flag to turn on/off these checks and
Context.codeBug to throw an exception in case of check violations,
and also modified UintMap to use them instead of the private
flags there.
It would be nice to add some tests about format characters to the test
suite with checks similar to "eval('1 =\u200C= 1') == true" and
"eval('.\u200C1') == 0.1".
git-svn-id: svn://10.0.0.236/trunk@102914 18797224-902f-48f8-a5cc-f745e15eee43