Two examples are evidence, not a contract

Comparing two JSON responses can reveal a useful change: an identifier became a string, a field disappeared, or a nested object gained a member. That observation is not yet proof that an API contract changed. One sample may represent a rare branch, a partially populated record, a different API version, or data altered by a test fixture. Before calling a difference a breaking change, identify where each sample came from, what request produced it, and which version of the producer was running.

The JSON Shape Diff in Benchlight compares the observed structure of two values. It reports added paths, removed paths, and paths whose value types differ. Values themselves are intentionally ignored. If the same path contains “blue” in one response and “green” in another, the structural report stays quiet. If the value changes from a string to an object, the report calls attention to the path. That division lets a reviewer focus on shape before examining domain-level meaning.

Establish comparable samples first

Label samples with the conditions that produced them: endpoint version, request parameters, account state, locale, feature flags, and a safe capture date. Do not compare a list response with a detail response unless the purpose is specifically to understand their difference. Where a response is conditional, collect one sample for each meaningful branch. A field absent for a user without a profile may be expected; a field absent for a fully provisioned user may indicate a defect.

Redact sensitive values while preserving their JSON types. Replacing a secret string with another non-sensitive string keeps the comparison focused on shape. Replacing it with null creates a false type change. Remove unique identifiers only when doing so will not make two important records appear identical. Store fixtures in a place with the same access controls expected for other test data, and avoid committing personal data just because an example is convenient.

Arrays need special care. An array’s items may be heterogeneous, empty in one sample, or contain only one representative item. A shape tool can only report structures that were actually observed. If one sample contains an empty list, there is no item shape to infer. If a sample contains one object, it does not establish that all later elements have the same members. The report should therefore be read as “this sample showed these paths,” not “the service guarantees every future record has this shape.”

Classify each difference by its likely impact

For every changed path, ask four questions. Is the path required by any consumer? Does the change alter the JSON type? Can older clients ignore the new member? Does the producer still send the old representation to any supported version? A newly added optional field is often easier for a tolerant client to accept than a removed field or a type change, but compatibility depends on the actual clients and validation rules. A number serialized as a string can change sorting, arithmetic, validation, and language-level parsing behavior.

Distinguish missing from null. “Missing” means no member is present. “Null” means the member exists and has the JSON null value. An API can use null to mean known absence, redaction, not-yet-computed, or something else defined by its contract. Replacing a missing property with null is not automatically more compatible. Similarly, an empty array may communicate “there are zero results,” while a missing collection may mean “this version does not provide it.” Ask the owner of the contract to define the intended semantics.

Renames are often represented as one removed path and one added path. A structural diff cannot know that profile.fullName and user.name represent the same concept. It also cannot decide whether a nested object should be flattened. Record semantic migrations separately, with a mapping and a deprecation window if the producer and consumers are versioned independently. A structural tool should not invent such relationships from similar spelling.

Promote stable expectations into a schema

When an interface is important enough to rely on, encode the expectations instead of keeping them only in screenshots or chat. JSON Schema’s getting-started guide demonstrates describing object properties and the data types expected for those properties. A real schema can also express requirements and constraints that a sample diff cannot, such as whether a property is required or whether an array item must match a particular shape.

Make version choices explicit. JSON Schema has multiple drafts, and tooling behavior is tied to which vocabulary and draft a validator supports. Write down the selected draft and test the schema with the validator used by the service. Include representative valid and invalid fixtures: missing required properties, unexpected null values, values with the wrong type, boundary lengths or ranges, and additional properties when those are restricted. A schema is useful only if it reflects the consumer-facing contract rather than one engineer’s guess.

Do not overconstrain fields that are expected to evolve. If every unknown property is rejected without a versioning plan, a harmless producer addition can break a strict client. If every property is optional and arbitrary, the schema may give little protection. Decide which layer owns forward compatibility. Some producers guarantee a closed object; others permit additional fields while promising stable meaning for documented ones. The schema should express that choice, and contract tests should verify the actual serializer against it.

Work a migration question from two small records

These invented samples describe the same kind of record before and after a proposed serializer change. Enter the first as Earlier sample and the second as Newer sample:

{"id":12,"profile":{"name":"Mira"},"tags":[]}

{"id":"12","profile":null,"tags":[]}

The local comparison reports three changed paths. At $["id"], number becomes string. At $["profile"], object becomes null. The nested $["profile"]["name"] path is removed because there is no longer an object containing that member. The empty tags array is unchanged. Its unchanged appearance does not supply evidence about the type of a future tag item.

Give each observation a separate decision. If a client performs arithmetic on id, changing its type requires more than updating example text. If a client reads profile.name unconditionally, the null branch needs a defined response, such as displaying an absent-profile state. A consumer that already handles null may need no implementation change. Those outcomes come from the consumer's behavior and supported interface, not the number of rows in the diff report.

For a schema discussion, specifying properties alone does not require a member to exist. Put genuinely mandatory names in required; separately allow null in the property's type only when null has defined meaning. Select a JSON Schema draft that the actual validator supports. Then test an omitted profile, an explicit null profile, and a populated profile against the chosen rule. These three fixtures settle different questions and should not be compressed into a single supposedly representative response.

Finish the review with a concrete migration record: which representation is supported, which consumers were checked, and which serializer test enforces it. A renamed field also needs an explicit old-to-new mapping. This procedure turns a visible difference into a decision that a producer and consumer can implement consistently.