Better error reporting for malformed literals

[yav]

The problem is illustrated by this example:

Cryptol> 0b12

[error] at <interactive>:1:1--1:4:
  Type mismatch:
    Expected type: Integer -> ?it`945
    Inferred type: [1]
  where
  ?it`945 is the type of '(expression)' at <interactive>:1:1--1:5

We could probably solve it by allowing more digits in the lexer's regexp, but doing a validation after and reporting
and error if malformed digits are found.

[robdockins]

Yeah, once we see the start of a literal, we should probably munch all of the characters until the next whitespace, then validate after the fact.

[yav]

Maybe only munch the alphanumeric characters though, as 1+1 is probably ok, and also comments are fine.

[robdockins]

Ah, yes, until whitespace is too much.

[brianhuffman]

This is kind of fun:

Cryptol> (+) 0b012
0x3

[brianhuffman]

Another thing that the parser ought to reject:

Cryptol> (+) 2z where z = 0x3
0x5

Or for twice the fun, we can combine these tricks together:

Cryptol> (+) 2z where z = (+) 0b012
0x1

[brianhuffman]

I was thinking that maybe we could just replace all the numeric regexps in Lexer.x with a single one, like this:

@num = [0-9] @id_next*

So basically, whenever we see a digit, we munch as many alphanumeric chars as we can; then we could parse and validate those strings later.

However, the various floating point literals make that a bit complicated. Ordinarily we would treat an operator symbol like . or - or + as belonging to a separate lexeme from an adjacent alphanumeric char. But if the surrounding characters form a valid floating point literal, then we want to treat the . or - or + as part of the same lexeme as the surrounding characters.

So now I'm wondering whether we should address this issue by just adding a right context to some of the lexer rules(https://www.haskell.org/alex/doc/html/alex-files.html#contexts). For example, we require that @num8 only matches the regexp "0o" @digits8 in contexts where it is not immediately followed by another alphanumeric character. (We'd probably want a similar restriction on all of the rules for numeric literals.)

[brianhuffman]

I tried the right-context approach; while it was easy to implement, and it rejects the inputs that we want to reject, it doesn't give very good error messages:

Cryptol> (+) 2z where z = 3

Parse error at <interactive>:1:5--1:6
  unrecognized character:2
Cryptol> (+) 0b012

Parse error at <interactive>:1:5--1:9
  unrecognized character:0

I think that reporting a malformed numeric literal with an "unrecognized character" error message is not really acceptable. Instead of using right-contexts, I think the right approach is to append something like @id_next* to the end of the regexp for each kind of numeric literal, and validate them afterward; this way we can customize the error messages to whatever we want.