Using components: Rest API Source

 

Using the Rest API source provides access to read data from HTTP endpoints such as Rest Web Services. Use the Rest API source component to define the authentication method, request parameters, and response fields to use in the package.

Authentication

  • None - No authentication
  • Basic - Use basic authentication (user and/or password).
  • Connection - Use an Integrate.io ETL connection to handle authentication automatically.

Request and response

Request

  • method - The HTTP method. Select GET (default), POST, PATCH or PUT.
  • URL - Enter the URL of your source endpoint.
  • Headers - Optionally add request headers (names and values)
  • Body - Optionally add a request body string

Pagination

Check Use pagination to have the source component make paginated requests. See here for list of supported API endpoints in addition to the standard link headers .

  • Pagination scheme - The method to paginate through response data (e.g. page number, offset+limit, cursor, link headers) and how to detect the final page.
    • Automatic (Default) - the source detects the pagination scheme to use by the URL host/path.
    • Jira - Use with hosted Jira installations.
    • Magento - Use with hosted Magento installations.
    • BigCommerce - Use with hosted BigCommerce installations.
    • Elasticsearch - Use with hosted Elasticsearch installations.
    • Link headers - the fallback option which will be used with the default automatic as well.
  • Sleep interval between paginated requests (ms) - use to throttle API requests to avoid rate limitations. Default is 0 (no sleep).
  • Maximum paginated requests - use to control the maximum number of pages for the source to go through. Default is unlimited, 0 or 1 means a single request will be made.

Response

Select the response type to determine how to parse the response from the API endpoint.

  • Raw - returns 3 fields: status code (integer), body (string) and headers (map).
  • JSON - Response is processed using a JSONPath expression.
    You can use a custom JSONPath expression to extract nested objects and arrays or the presets for object and array. Using the object preset, the response fields are the keys of the JSON object. Using the array preset, the response fields are the keys of the JSON objects within the array and each object is returned as a record in the component's output.
  • Line Delimited JSON - Response is a single JSON object per line. Note that the response as a whole is not a valid JSON. The response fields are the keys of the JSON objects and each object is processed as a record.
  • Line Delimited raw data - Response is broken to record per line. Useful when data contains records delimited by new line.

Schema

After defining the request and the response, select the fields to use in the source.

Known Limitations

When using a $ sign in the call, it needs to be escaped with back slash - \

Please note - it will run successfully at runtime, but will fail with a preview of the API source component or component preview in the flow.