ADR 0012: A lone UTF-16 surrogate makes a request malformed and a ruleset invalid

Status: accepted

ContextCopy Anchor Link

JSON can carry a UTF-16 code unit from D800 to DFFF that is not part of a pair, as an escape ("\ud800"). It is not a code point, and the runtimes read it differently: Go's encoding/json replaced it with U+FFFD, while Python, JavaScript and Java kept it. So len, matches and equality could answer the same request differently in two runtimes, which breaks "evaluated identically". The Python compiler also crashed on a member name holding one, while computing the checksum. I-JSON (RFC 7493) already forbids such strings.

DecisionCopy Anchor Link

• Refuse it. A string or member name holding a lone surrogate, at any depth in a member that specification section 8 lists, makes the request malformed: RequestError (400) in every runtime and BAD_REQUEST from an engine. It is checked after the shape and depth checks and before anything is evaluated. Members a request does not define stay ignored.
• A ruleset holding one does not load (SCHEMA_INVALID, checked once the schema passes), for the child and for its parent.
• A surrogate pair is one character, as before.
• Go keeps the escape in ParseJSON. encoding/json turns a lone surrogate into U+FFFD before any check can see it, so ParseJSON keeps it as the three bytes UTF-8 would give that code unit (bytes valid UTF-8 never contains), the checks look for those bytes, and Marshal writes them back as the escape.

ConsequencesCopy Anchor Link

• Every runtime refuses the same input at the same point; conformance has 7 protocol cases and 4 load-error cases for it.
• A client that sent a lone surrogate and got an answer now gets a 400. Released while the packages are alpha.
• Go only refuses it when the JSON was read with ParseJSON. A Go caller that decodes with encoding/json (or json.Decoder.UseNumber()) already holds U+FFFD, so the request is accepted and evaluated with U+FFFD. The Go README and the rules-cascade-go skill say that ParseJSON is required for this guarantee.
• The Go JSON layer has a slower path for string tokens that contain a \ud/\uD escape; other strings take the same path as before.

Alternatives consideredCopy Anchor Link

• Replace it with U+FFFD everywhere. Identical results, but the engine would evaluate data the caller did not send, and a rule would see a character the client never typed.
• Leave it undefined. No work, but the runtimes keep disagreeing, and the conformance suite could not say which answer is right.