Quantcast

json and numbers - bug or feature?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

json and numbers - bug or feature?

Jolle Carlestam-2
Johan discovered an issue with json_deserialize. Turns out if you try to deserialize a number (integer or decimal) it will fail. As in return nothing.

This is the test code Johan tried with
json_deserialize(json_serialize(1234))
whereas this works
json_deserialize(json_serialize('1234'))

I’ve tracked the issue to this code:
define json_deserialize(ibytes::bytes)::any => {
        #ibytes->removeLeading(bom_utf8);

//============================================================================
// Reset marker on provided bytes
        #ibytes->marker = 0
//............................................................................
       
        Local(temp) = #ibytes->export8bits;
        If(#temp == 91); // [
                Return(json_consume_array(#ibytes));
        Else(#temp == 123); // {
                Return(json_consume_object(#ibytes));
        else(#temp == 34) // "
                return json_consume_string(#ibytes)
        /If;
}

The export8bits snippet it looks for are either 91 as in [ indicating that the following is an array. Or 123 as in { for a map/object and 34 as in ” to confirm that it is a string.

But, json_serialize(1234) will return
1234
and json_serialize('1234’) will return
”1234”

As a result, the first will not trigger anything when called with json_deserialize since the first byte is none of the options but the second one will.

Is this expected? I could not tell by reading the specs for json that I found online. A test in Chrome though showed that it was fine with a json object 1234. Deserializing that produced 1234 instead of Lassos void.
Try
console.info(JSON.parse(1234))

I am also not json spec savvy enough to come up with the proper solution, if one is to be implemented at all. My suggestion would be to send any remaining #ibytes to json_consume_string. Is that good or bad?

HDB
Jolle

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: json and numbers - bug or feature?

Brad Lindsay
Jolle,

Can you do something like this:

define json_deserialize(value::integer) => #value
define json_deserialize(value::decimal) => #value

I’m not at a spot I can test that real quick, but wouldn’t using
multiple dispatch in this way fix the issue?

Brad


On July 1, 2016 at 2:21:08 PM, Jolle Carlestam ([hidden email]) wrote:

> Johan discovered an issue with json_deserialize. Turns out if you try to deserialize
> a number (integer or decimal) it will fail. As in return nothing.
>
> This is the test code Johan tried with
> json_deserialize(json_serialize(1234))
> whereas this works
> json_deserialize(json_serialize('1234'))
>
> I’ve tracked the issue to this code:
> define json_deserialize(ibytes::bytes)::any => {
> #ibytes->removeLeading(bom_utf8);
>
> //============================================================================
> // Reset marker on provided bytes
> #ibytes->marker = 0
> //............................................................................
>
> Local(temp) = #ibytes->export8bits;
> If(#temp == 91); // [
> Return(json_consume_array(#ibytes));
> Else(#temp == 123); // {
> Return(json_consume_object(#ibytes));
> else(#temp == 34) // "
> return json_consume_string(#ibytes)
> /If;
> }
>
> The export8bits snippet it looks for are either 91 as in [ indicating that the following
> is an array. Or 123 as in { for a map/object and 34 as in ” to confirm that it is a string.
>
> But, json_serialize(1234) will return
> 1234
> and json_serialize('1234’) will return
> ”1234”
>
> As a result, the first will not trigger anything when called with json_deserialize since
> the first byte is none of the options but the second one will.
>
> Is this expected? I could not tell by reading the specs for json that I found online. A test
> in Chrome though showed that it was fine with a json object 1234. Deserializing that produced
> 1234 instead of Lassos void.
> Try
> console.info(JSON.parse(1234))
>
> I am also not json spec savvy enough to come up with the proper solution, if one is to be implemented
> at all. My suggestion would be to send any remaining #ibytes to json_consume_string.
> Is that good or bad?
>
> HDB
> Jolle
>
> #############################################################
>
> This message is sent to you because you are subscribed to
> the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to:
> Send administrative queries to

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: json and numbers - bug or feature?

Jolle Carlestam-2
1 juli 2016 kl. 21:16 skrev Brad Lindsay <[hidden email]>:

>
> Jolle,
>
> Can you do something like this:
>
> define json_deserialize(value::integer) => #value
> define json_deserialize(value::decimal) => #value
>
> I’m not at a spot I can test that real quick, but wouldn’t using
> multiple dispatch in this way fix the issue?
>
> Brad

Good try, Brad. But no, does not seem to solve the issue. Thing is that
json_serialize(1234) -> type
returns string.

Still not savvy enough to judge if that is right or wrong.

HDB
Jolle

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: json and numbers - bug or feature?

Trevor Borgmeier
An integer alone is valid json.

http://www.json.org/



On 7/1/16 4:12 PM, Jolle Carlestam wrote:

> 1 juli 2016 kl. 21:16 skrev Brad Lindsay <[hidden email]>:
>> Jolle,
>>
>> Can you do something like this:
>>
>> define json_deserialize(value::integer) => #value
>> define json_deserialize(value::decimal) => #value
>>
>> I’m not at a spot I can test that real quick, but wouldn’t using
>> multiple dispatch in this way fix the issue?
>>
>> Brad
> Good try, Brad. But no, does not seem to solve the issue. Thing is that
> json_serialize(1234) -> type
> returns string.
>
> Still not savvy enough to judge if that is right or wrong.
>
> HDB
> Jolle
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>    the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


ɹǝıǝɯƃɹoq ɹoʌǝɹʇ


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: json and numbers - bug or feature?

Brad Lindsay
In reply to this post by Jolle Carlestam-2
On July 1, 2016 at 5:12:41 PM, Jolle Carlestam ([hidden email]) wrote:

> 1 juli 2016 kl. 21:16 skrev Brad Lindsay :
> >
> > Jolle,
> >
> > Can you do something like this:
> >
> > define json_deserialize(value::integer) => #value
> > define json_deserialize(value::decimal) => #value
> >
> > I’m not at a spot I can test that real quick, but wouldn’t using
> > multiple dispatch in this way fix the issue?
> >
> > Brad
>
> Good try, Brad. But no, does not seem to solve the issue. Thing is that
> json_serialize(1234) -> type
> returns string.
>
> Still not savvy enough to judge if that is right or wrong.

That is correct. From http://json.org: "JSON is a *text* format that
is completely language independent” (emphasis added by me). From that
description I would expect serialized JSON data to be either bytes or
a string.

Brad

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: json and numbers - bug or feature?

Jolle Carlestam-2
Some more digging. (What else am I supposed to do when it rains?)

I have shifted focus to json_decode and json_encode. After all that is the replacement calls that are superior to json_deserialize and json_serialize.

Turns out that they too share the same bug. But also that the code reveals the author thought about it, only didn’t test it. After some tweaking in that code I have a fix in place.
For anyone using json_decode here’s the proposed replacement that you should install:

define json_decode => type {

        data private stack
        data private exit

        public onCreate(s::string) => {
                #s->size == 0?
                        return void
                local(stack = array,
                        result = .deserialize(#s, 1, #s->size, #stack))
                #stack->size != 0?
                        fail(-1, 'Unterminated JSON object: ' + #stack)
                return #result // return something not self
        }

        protected handlePop() => .handlePop(.stack->pop)

        protected handlePop(obj) => {
                .stack->size > 0?
                        return .handleNested(.stack->top, #obj)
                .exit = #obj
                return #obj
        }

        protected handleNested(top::pair, obj) => {
                #top->second = #obj
                return .handleNested(.stack->pop&top, #top)
        }
        protected handleNested(top::map, obj::pair) => #top->insert(#obj->first = #obj->second)
        protected handleNested(top::map, obj) => { .stack->push(pair(#obj, void)) }
        protected handleNested(top::array, obj) => #top->insert(#obj)
        protected handleNested(errorObj, obj) => fail('')

        protected readString(src::string, start::integer) => {
                local(esc = false)

                // #start is pointing at opening quote
                #start += 1

                local(subStart = #start)

                local(char)
                {
                        #char = #src->integer(#start)
                        #start += 1
                        #char == json_quote_double and not #esc?
                                returnHome

                        #esc = (#char == json_back_slash and not #esc)

                        currentCapture->restart
                }()

                local(sub = #src->sub(#subStart, #start - #subStart - 1)->unescape)

                return (:#sub, #start - 1)
        }

        protected readNumber(src::string, start::integer) => {

                local(
                        subStart = #start,
                        needPeriod = true,
                        needExp = true,
                        inNumber = true,
                        sizeOk = true,
                        srcsize = #src -> size,
                        char
                )

                #src->integer(#start) == json_negative?
                        #start += 1

                while(#inNumber) => {
                        if(#start <= #srcsize) => {
                                #char = #src->integer(#start)
                        else
                                #sizeOk = false
                        }

                        match(true) => {
                                case(#sizeOk and #src->isDigit(#start))
                                        // do nothing
                                case(#needPeriod and #char == json_period)
                                        #needPeriod = false

                                case(#needExp and (#char == json_e_upper or #char == json_e_lower))
                                        #needExp = false
                                        #needPeriod = false

                                case
                                        #inNumber = false
                        }
                        #start += 1

                }

                local(sub = #src->sub(#subStart, #start - #subStart - 1))

                #needExp and #needPeriod?
                        return (:integer(#sub), #start-2)

                return (:decimal(#sub), #start-2)
        }

        public deserialize(s::string,
                                        start::integer,
                                        end::integer,
                                        stack::array) => {
                .stack = #stack
                .exit = null

                local(top, char)

                {// main loop

                        #start > #end?
                                return .exit or #stack->top
                        #char = #s->integer(#start)

                        match(#char) => {

                                case(json_open_object)
                                        //json_debug('json_open_object')
                                        #stack->push(map)

                                case(json_open_array)
                                        //json_debug('json_open_array')
                                        #stack->push(array)

                                case(json_close_object)
                                        //json_debug('json_close_object')
                                        .handlePop()

                                case(json_close_array)
                                        //json_debug('json_close_array')
                                        .handlePop()

                                case(json_colon)
                                        //json_debug('json_colon')
                                        // valid inside a map. top object must be a pair
                                        not #stack->top->isa(::pair)?
                                                fail(-1, 'Invalid colon location at ' + #start + ' ' + #stack)
                                        // do nothing

                                case(json_comma)
                                        //json_debug('json_comma')
                                        #stack->top->isa(::pair)?
                                                fail(-1, 'Invalid comma location at ' + #start + ' ' + #stack)

                                case(json_quote_double) // start new string
                                        //json_debug('json_quote_double')
                                        local(newS, start) = .readString(#s, #start)

                                        .handlePop(#newS)

                                case(json_white_space, json_lf, json_cr)
                                        // inter-element white space. ignored
                                        //json_debug('json_white_space')

                                case// number, true, false, null
                                        if (#char == json_negative or #s->isDigit(#start)) => {
                                                //json_debug('json_number')
                                                local(newS, start) = .readNumber(#s, #start)
                                                .handlePop(#newS)

                                        else(#char == json_t_lower)
                                                #s->sub(#start, 4) != 'true'?
                                                        fail(-1, 'Invalid character at location ' + #start + ' ' + #stack)
                                                #start += 3
                                                .handlePop(true)

                                        else(#char == json_f_lower)
                                                #s->sub(#start, 5) != 'false'?
                                                        fail(-1, 'Invalid character at location ' + #start + ' ' + #stack)
                                                #start += 4
                                                .handlePop(false)

                                        else(#char == json_n_lower)
                                                #s->sub(#start, 4) != 'null'?
                                                        fail(-1, 'Invalid character at location ' + #start + ' ' + #stack)
                                                #start += 3 // fixing #7920. Was 4 but when 4 does not read closing brace
                                                .handlePop(null)

                                        else
                                                stdoutnl('Unknown char: ' + #char)

                                        }
                        }
                        #start += 1
                        currentCapture->restart
                }()
        }
}

Summary, if you are using json_decode / json_encode make sure to apply this fix.
If you’re using json_deserialize / json_serialize replace all calls for json_deserialize / json_serialize with json_decode / json_encode. It is faster and more efficient. AND apply the fix…

HDB
Jolle

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Loading...