> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# ETL: SugarCRM Source
> How to configure the SugarCRM component to read CRM data (Accounts, Contacts, Leads, and custom modules) in your Integrate.io ETL pipeline.
Use the SugarCRM source component to read records from any module in your SugarCRM instance, including standard modules (Accounts, Contacts, Leads, Opportunities, Cases, Tasks, Calls, Meetings, etc.) and any custom modules your tenant has, and ingest them into your [Integrate.io](http://integrate.io/) ETL pipeline. The connector authenticates against the SugarCRM REST API v11\_24 using your SugarCRM username and password, fetches an OAuth token internally on every job, and supports both full and incremental loads.
## Connection Setup
Create a SugarCRM connection from **Connections → New connection → SugarCRM**.
| Field | Description |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Name | Display name for the connection inside Integrate.io |
| Instance URL | The base URL of your SugarCRM instance, e.g. `https://yourcompany.sugarondemand.com`. Do not include `/rest/...`; Integrate.io appends the API path automatically. |
| Username | The SugarCRM user the connector authenticates as. This user's role determines which modules and records the pipeline can read. |
| Password | The password for that SugarCRM user. Integrate.io uses this once per pipeline run to obtain a short-lived OAuth token from `/rest/v11_24/oauth2/token`; the password itself is never sent to SugarCRM modules. |
Use a dedicated **integration user** with read-only access to the modules you want to extract, rather than a regular user account. This isolates pipeline activity in Sugar's audit log and avoids breakage when a real user changes their password or leaves.
After filling the form, click **Test connection**. A successful test confirms that:
1. The instance URL resolves and `/rest/v11_24/oauth2/token` accepted the username + password
2. The user has at least read access to the `me` endpoint
If the test fails with `invalid_grant`, double-check the password and confirm the user can log in to SugarCRM's web UI directly. SugarCRM's password-grant OAuth flow uses the same credentials.
## Source Properties
The source component is configured in Step 02 of the component editor.
### Source Table (Object)
The dropdown lists every SugarCRM module the authenticated user can see. This includes:
* **Curated standard modules.** Exposed by default with rich datetime field metadata so incremental loading works out of the box
* **Discovered custom modules.** Anything else the user can see on the Sugar instance, queried from `/rest/v11_24/metadata?type_filter=full_module_list` once per pipeline run and merged into the picker
#### Curated standard modules
| Object | SugarCRM module | Datetime fields available for incremental load |
| :------------ | :-------------- | :------------------------------------------------------------ |
| accounts | Accounts | `date_entered`, `date_modified` |
| contacts | Contacts | `date_entered`, `date_modified` |
| leads | Leads | `date_entered`, `date_modified` |
| opportunities | Opportunities | `date_entered`, `date_modified`, `date_closed` |
| cases | Cases | `date_entered`, `date_modified` |
| notes | Notes | `date_entered`, `date_modified` |
| tasks | Tasks | `date_entered`, `date_modified`, `date_start`, `date_due` |
| calls | Calls | `date_entered`, `date_modified`, `date_start`, `date_end` |
| meetings | Meetings | `date_entered`, `date_modified`, `date_start`, `date_end` |
| emails | Emails | `date_entered`, `date_modified`, `date_sent` |
| campaigns | Campaigns | `date_entered`, `date_modified`, `start_date`, `end_date` |
| users | Users | `date_entered`, `date_modified` |
| documents | Documents | `date_entered`, `date_modified` |
| quotes | Quotes | `date_entered`, `date_modified`, `date_quote_expected_closed` |
| contracts | Contracts | `date_entered`, `date_modified`, `start_date`, `end_date` |
| bugs | Bugs | `date_entered`, `date_modified` |
| prospects | Prospects | `date_entered`, `date_modified` |
#### Custom modules
If your SugarCRM tenant has custom modules (e.g. `Subsc_Subscriptions`, `My_Custom_Module`), they appear in the dropdown under the lowercased module name (`subsc_subscriptions`, `my_custom_module`). The connector hits the SugarCRM REST endpoint at `/rest/v11_24/{ModuleName}` and exposes `date_entered` and `date_modified` as the default incremental fields. These are inherited from `SugarBean`, the base class every Sugar module extends, so they're present on every record by default.
Click the **Refresh** button next to the dropdown to re-query SugarCRM's module list. This is useful after an admin adds a new custom module or grants the integration user access to additional modules.
### Load Type
Select how records are loaded on each pipeline run:
* **Full Load.** Fetches all records for the selected object on every run.
* **Incremental Load.** Fetches only records created or updated after a reference date. Recommended for scheduled pipelines once the initial historical load is complete, so you don't re-process records that haven't changed.
### Incremental Load Settings
When **Incremental Load** is selected, three additional fields appear:
| Field | Description |
| :-------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Sync date field | The datetime field used to filter records server-side. Defaults to `date_modified` for most modules. You can pick any datetime field listed for the object (for example `date_closed` on Opportunities or `date_start` on Meetings), and SugarCRM will filter on that field instead. The connector substitutes the picked field into the API filter `filter[0][{field}][$gte]=...`, so jobs work consistently regardless of which datetime field you choose. |
| Load records | The filter direction. `newer than` / `on or after` fetches records with `{field} >= reference date`. `older than` / `on or before` fetches records with `{field} <= reference date`. |
| Reference date | The source of the date value. `Fixed Date` uses a specific calendar date, useful for one-off backfills. `Variable` uses a system or custom variable; the recommended value for scheduled pipelines is `$package_last_successful_job_submission_timestamp`, which automatically advances after each successful run so each run picks up only what changed since the previous run finished. |
SugarCRM expects ISO 8601 dates. The connector passes the date value directly to Sugar as provided. If you use a system variable like `$package_last_successful_job_submission_timestamp`, Integrate.io fills it in with a UTC ISO 8601 timestamp automatically.
## Field Flattening
The SugarCRM REST API returns nested JSON for some fields, including relate links, address objects, and custom field groups. The connector automatically **flattens** these into individual columns using underscore-separated naming so each leaf value lands in its own column instead of a serialized JSON blob.
For example, a relate field that the API returns as:
```json theme={null}
{
"assigned_user_link": {
"name": "Jane Doe",
"id": "abc-123"
}
}
```
is flattened into two columns: `assigned_user_link_name` and `assigned_user_link_id`. This lets you map the columns directly to your destination schema without writing a downstream transformation to parse JSON.
## Schema
After configuring the source properties, the **Schema** section (Step 03) displays all available fields with their detected data types.
Common SugarCRM field types and how they map:
| SugarCRM type | Integrate.io type | Notes |
| :--------------------------------------------- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| varchar / text / name / phone / email | string | All string-shaped fields |
| int / id | string | SugarCRM IDs are 36-char UUIDs returned as strings |
| datetime / date | string | ISO 8601 strings (e.g. `2026-04-28T18:01:12+10:00`); cast to `datetime` in a downstream Select transformation if your destination needs typed timestamps |
| bool | boolean | |
| decimal / float / currency | double | |
| enum / multienum | string | Multi-select enums are returned as comma-separated strings |
| relate (assigned\_user\_id, account\_id, etc.) | string | The ID of the related record. The display name is in the corresponding `*_link_name` field after flattening. |
Use the field selector to choose which columns to include in your pipeline. You can rename fields with aliases and override the detected data type as needed.
## Best Practices
* **Use a dedicated integration user** with the minimum module-level role needed. Avoid using a real person's account. If their password changes, every pipeline breaks at the next run.
* **Start with full load, switch to incremental** once the destination is seeded with historical data. Use `date_modified` as the sync field for most modules, as it covers both inserts and updates. Only switch to `date_entered` if you specifically want inserts only.
* **Pick the right datetime field for the module's semantics.** `date_modified` catches every change. `date_closed` on Opportunities only fires when the opportunity stage moves to Closed Won/Lost, which is useful for daily revenue reporting but skips opportunities still in flight. `date_start` on Calls/Meetings reflects when the activity is scheduled, not when it was logged.
* **Custom module datetime fields**: discovered custom modules default to `date_entered` and `date_modified` only. If your custom module has additional date fields you want to filter on (e.g. `effective_date_c`), pick the field manually in the dropdown. The connector will still build a valid filter.
* **Refresh the object list** after Sugar admins add or rename modules. The list is cached for 1 hour per connection; the **Refresh** button bypasses the cache.
## FAQ
**Q: Why is my custom module not in the dropdown?**
The connector lists modules using the SugarCRM `/rest/v11_24/metadata?type_filter=full_module_list` endpoint, which only returns modules the authenticated user can see. If the integration user doesn't have a role granting access to that module, it won't appear. Have a SugarCRM admin grant the user a role with read access to the module, then click **Refresh** in the source dropdown.
**Q: I picked `date_entered` for incremental load, but the pipeline returns data older than my reference date. What's wrong?**
This was a bug in earlier versions of the connector. The filter was hard-coded for `date_modified`, so when you picked a different field the filter was silently dropped and SugarCRM returned all records. The current connector substitutes the picked field into the filter automatically, so the issue should be resolved. If you're still seeing it, check the pipeline log for the line `Incremental date filter for API:` and confirm it shows `filter[0][date_entered][$gte]=...` (not `date_entered=...`). If the wrapper is missing, your job is running against an older Pig JAR. Contact support to upgrade.
**Q: What SugarCRM API version does the connector use?**
`v11_24`. This applies to both the OAuth token endpoint (`/rest/v11_24/oauth2/token`) and all data endpoints (`/rest/v11_24/{Module}`). SugarCRM's `v11` is an alias for the latest minor, but the connector pins to a specific version for stability across SugarCRM tenant upgrades.
**Q: How does pagination work?**
The connector uses SugarCRM's offset-based pagination (`offset=` + `max_num=1000` per page). Up to 10,000 pages are followed per object per run, which translates to 10 million records, enough for any single full load. The schema-discovery and data-preview steps cap at 50 records to keep the UI responsive.
**Q: Is there a rate limit?**
The connector throttles outbound requests to 15 requests per second by default and retries up to 3 times on HTTP 429 / 503 with exponential backoff capped at 60 seconds. SugarCRM's per-tenant API quotas are typically much higher, so this rarely matters in practice, but if you run many concurrent pipelines against the same tenant you may want to stagger them.
## Related