Stepped JSON Reader

A webservice rarely returns a whole dataset in one response. More often the first request returns a list, of order ids or of URLs, and you have to fetch the detail behind each one separately.

Stepping makes the JSON Reader do that for you. It issues a request per record and assembles the responses into a single dataset, with the parent-child structure intact.

Stepping needs the http(s) Url controller

Stepping needs the http(s) Url controller. This page is about reading from a webservice. A reader on the File, Email or Transaction controller has no Url For Steppable Reader field.

Everything else about the reader — the entry point, transaction JPaths, the field grid, hoisting — works as it does on the JSON Reader page. This page covers only what stepping adds.

Url For Steppable Reader

This field turns stepping on. It appears on the Field Mapping tab, on every transaction, once the controller is http(s) Url.

A transaction steps only when its own Url For Steppable Reader has a value. If it is empty, the reader reads that transaction from the response its parent already fetched, as it would without stepping. A reader can therefore step at the top and not below it, or step only at one child. The two modes below are the two places a value can go.

The token spelling a URL needs

You parameterise the URL with the values of fields, written %[FieldName].

A bare token ends at a space, so a URL needs the bracketed form

Use the bracketed form in a URL. You can write a token either as %[FieldName] or bare as %FieldName, but a bare token ends at the next space, and only at a space. URLs do not contain spaces, so

/admin/order-%SYS.ITEM/lines

takes the field name to be SYS.ITEM/lines, finds no such field, and fails or substitutes nothing. /admin/order-%[SYS.ITEM]/lines works as intended. IMan reports an unclosed bracket as a parse error naming its position.

Seed list

The first request returns a list of values, and the reader makes a request of its own from each value.

1. The initial request is the one the controller's Query Url describes. The reader makes it once.
2. The JSON Entry Point picks the value out of that response. Without stepping it would pick out the records. Here it picks out the single value that identifies each record. An entry point of /orders/order_id yields one order id per order.
3. Each value is substituted into the top transaction's Url For Steppable Reader through %[SYS.ITEM], and the reader makes a request for each. The value can be the whole URL or, as here, a fragment of one.
4. Each response is read as one top-level record. The transaction's own JPath is the entry point into that response, and the JPaths of its fields and child transactions are relative to it.

Example

Query Url:

/admin/ordersList?fulfillment_status=unshipped

which returns a summary list:

{ "orders": [ { "order_id": 101 }, { "order_id": 103 } ] }

JSON Entry Point: /orders/order_id

Url For Steppable Reader, on the top transaction:

/admin/order-%[SYS.ITEM]

The reader then requests /admin/order-101 and /admin/order-103 in turn, and each returns a whole order:

{
  "order_id": 101,
  "customer": "ABC001",
  "line_items": [
    { "item_id": "A1-103/0", "qty": 5, "price": 12.10 }
  ]
}

The top transaction's JPath is the entry point into that response. Relative to it, order_id reads the order id, and a child transaction over the line items has a Transaction JPath of line_items[].

Parent to child

This mode applies the same idea one level down. Any field of a parent record can build the request that fetches its children.

The seed list can offer only one value, through SYS.ITEM. Parent-to-child can use any field of the parent, referred to by its own name, so you can build a request from an id, a date and a currency together.

For each record in the parent, the reader makes a request from the child's Url For Steppable Reader and inserts the response as that record's children. The child's Transaction JPath is the entry point into its own response, exactly as the top transaction's is into the seed response.

/admin/order-%[order_id]/payments on a Payment child therefore produces /admin/order-101/payments for order 101.

Using both

You can use both modes in one reader. A reader can seed its top level from a summary list and then step again from each order to its payments, because each transaction's own Url For Steppable Reader decides whether it steps.

When a stepped reader returns less than you expected, check this first. A transaction whose Url For Steppable Reader is empty is not broken. It reads from the response above it.