CurlWithPagination

Description

Makes one or more REST API calls and returns the responses received from all the pages requested from the server. Requests are made either anonymous or with basic authentication.

Syntax

CurlWithPagination(url, method[, headers[, request_body[, username[, password[, pagination_scheme[, sleep_interval[, max_pages]]]]]]])

Arguments

  url - HTTP or HTTPS url for the request.

  method - HTTP method, such as GET, POST, PUT, DELETE.

  headers (optional) - A json string that identifies the header key and value in the following manner: '{"header1" : "value1" , "header2" : "value2"}' 

  request_body (optional) - The string body of the request (most relevant in the case of POST)

  username (optional) - The user name for basic authentication. Can be empty string if the site does not require authentication.

  password (optional) - The password for basic authentication. Can be empty string if the site does not require authentication.

  pagination_scheme (optional) - The pagination method to use, see options below. If empty, the function will try to detect the pagination method according to the url's domain. See the supported domains here.

  sleep_interval (optional) - number of milliseconds to wait between requests. Default is 0.

  max_pages (optional) - stop after X number of pages. Default is unlimited, 0 or 1 means single page call.

Examples

CurlWithPagination('https://api.github.com/repos/integrate.io ETL/integrate.io ETL-api-doc-v2/events','GET')

Return value datatype

The returned value is a bag of map objects that contains the following keys: 

  • status - response code (integer)
  • body - the body of the response (string)
  • headers - map object of response headers

Notes

Supported pagination methods are:

  • Automatic - default, same as null or empty string. Detects pagination method by domain.
  • NoPagination - Simply make one request, don't follow pagination.
  • LinkHeader - Conforms with RFC 5988.

Impact of null value

The url and method arguments are required. The rest are optional and default to null.