Even the simplest json can not be parsed on FreeBSD or OS X

[mulle-nat]

curl -O https://raw.githubusercontent.com/step-/JSON.awk/master/JSON.awk
cat <<EOF > x.json
{
    "name": "latest"
}
EOF
awk -f JSON.awk x.json 

produces

x.json: expected <string> but got <"> at input token 2
{ <<">> 
x.json: expected <value> but got <a> at input token 1
<<a>> m e ": " l a t e s t " 
invalid: x.json
expected <string> but got <"> at input token 2
{ <<">> 
expected <value> but got <a> at input token 1
<<a>> m e ": " l a t e s t "

---

This is on FreeBSD with awk version 20121220 (FreeBSD) as well as on Darwin with awk version 20070501.

[step-]

Thank you. I can reproduce it using the FreeBSD-11.2-STABLE VM. I will be able to take a deeper look in a few days because I'm currently travelling.

[step-]

JSON.awk fails to parse the example you provided because the FreeBSD awk implementation behaves differently from gawk, mawk and busybox awk. I will suggest some options to work around this issue. This discussion should apply equally to Darwin, although I have no means to test that platform.

Unlike the other awk flavors that I mentioned above, FreeBSD awk seems not to support assigning a regular expression to the built-in RS (record separator) variable. It instead treats the first character of RS as the literal separator, and silently ignores all subsequent characters in the string. This behavior works against the code that purposefully assigns RS to an impossible-to-match expression. Consequently, the input record that JSON.awk attempts to tokenize is incomplete, hence the final error message.

Specifically, for the json file you provided (x.json), FreeBSD awk breaks the input record on the first "n" in "name" because "n" is the first character of RS. Then JSON.awk attempts to parse {" as valid json, which results in error message x.json: expected <string> but got <"> at input token 2. All the other awk flavors that I mentioned do not break file x.json into multiple records. Therefore JSON.awk attempts to parse the whole file and succeeds in doing so.

Possible Work-arounds

If you don't want to change JSON.awk source code, you can install gawk (recommended), mawk (fastest) or busybox awk (smallest, slowest) and run JSON.awk with that.

Versions tested on FreeBSD-11.2-STABLE

• awk 20121220 (FreeBSD) -- doesn't work for JSON.awk
• gawk 4.1.4 API 1.1
• mawk 1.3.4.20161120 see also
• busybox awk 1.26.2 2018-10-03

If you need to use FreeBSD awk, knowing its limitation, you need to change the source code. Set the record separator to a single character that doesn't appear anywhere in the input file -- you must know that in advance. For instance, you could set <backspace> as the record separator by replacing the entire line with:

RS = "\b"

Note that theoretically any Unicode codepoint is valid input (with proper escaping) in Json, so you might find it difficult to determine a character that is guaranteed not to exist across all possible input streams. However, practically most input data will not include certain control characters or high-bit characters or higher codepoints.

[mulle-nat]

As I understand now, what the problem is, I can make the decision not to try harder with JSON.awk. I have this tool called mulle-fetch where I'd like to start parsing github API. It needs to work with a standard awk, for portability reasons. So I will use JSON.sh instead, for less hassle.

Thanks for the detailed reply, it was very much appreciated.

[step-]

OK, thank you for your report, and good luck with your project!

[mohd-akram]

Changing RS does not fix the issue:

x.json: expected <:> but got <}> at input token 3
{ "name": "latest" <<}>>  

[step-]

Changing RS does not fix the issue:

x.json: expected <:> but got <}> at input token 3
{ "name": "latest" <<}>>  

Thank you for your comment. What RS string did you use? What platform, OS version, awk flavor and version, input data produced the above error message?

[mohd-akram]

I tried a few, '\b', '\t', '\030'. Btw those aren't allowed in JSON unescaped so the RS value shouldn't be an obstacle to macOS compatibility. I'm on macOS 10.15.6. The sources of awk used by macOS can be found here. You can reproduce by downloading awk-24.40.1, then cd src && make. The awk binary will be a.out. I ran the same command as the OP with the same awk version.

The macOS (and FreeBSD) awk is an old version of onetrueawk. Strangely, the latest version of onetrueawk works fine.

[step-]

@mohd-akram can you please test 7550234 and tell me if the old onetrueawk version works now? Thank you.

[mohd-akram]

After changing RS, It works! Thanks! Perhaps you can change RS to "\036" which is the ASCII record separator and shouldn't exist in any valid JSON.

[step-]

@mohd-akram, thank you for testing.

Perhaps you can change RS to "\036" which is the ASCII record separator and shouldn't exist in any valid JSON.

Hmm, it's a good idea but probably I won't change it because of this. Even though JSON.awk doesn't support streaming JSON now I don't want to preclude this avenue for the future. On the other hand, your suggestion made me think that I could change RS to a string that starts with a control character (not RS or \t, \f, \r, \n, space). This change would make no different on platforms other than MacOS and FreeBSD running onetrueawk, where only the first character of RS counts as record separator. On these platforms the addition of a control character should eliminate #10.

[mohd-akram]

Good point. I used your awesome project for building an awk for JSON. I patched it slightly to use RS="\1", and to allow newline-delimited JSON (which might be of use) in addition to some project-specific changes. You can see the patch in the src folder. Thanks again for the fix!

[step-]

@mohd-akram, your jawk is very cool! May I link to it from my "Projects known to use JSON.awk" README section?

I added a comment to your patch file, if you would please answer.

One more question; have you considered using JSON.awk's callback interface for your project instead of a full patch? I'm not suggesting you shouldn't use a patch, just curious to get some feedback to improve JSON.awk's callback interface.

[mohd-akram]

Thank you! It would be great to have it linked! I tried the callback interface briefly and I noticed that I had to implement quite a few callbacks to get it working, and I really only needed to nudge the output a bit so writing a patch was faster, and I could prototype quickly. Now that I've mostly decided on the interface and have a bunch of tests written, it might be worth looking into it again. I think most of my changes can be done using callbacks. One thing that would probably require some minor changes to JSON.awk is allowing a stream of multiple JSON objects. It would also probably be more complicated for me to get the length of the array, and the list of keys in an object. Btw the link to "library" and callbacks.awk in that doc seem to be broken.