> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL: Creating a Workflow

> Create workflow packages in Integrate.io ETL to orchestrate multi-step tasks. Define sequential and conditional task execution with error handling.

<Frame>
  <iframe src="https://fast.wistia.com/embed/iframe/78fd4qurqq" allowfullscreen playinline class="wistia_embed" width="640" height="360" />
</Frame>

## To create a workflow:

<Steps>
  <Step>
    On the main menu, click **Packages**.
  </Step>

  <Step>
    Click **New package**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-1.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=1c6ae52eb82c243ac4637a2d4cde9914" alt="Packages menu with New package button" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-1.png" />
    </Frame>
  </Step>

  <Step>
    Optionally, set a name and/or description
  </Step>

  <Step>
    Select **workflow** option from Type dropdown
  </Step>

  <Step>
    Click **Create package** to create workflow package.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-2.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=e23920094b48e14979422419d41a41ac" alt="New package dialog with workflow type selected" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-2.png" />
    </Frame>
  </Step>

  <Step>
    Click  **+ Add Task** button.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-3.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=67e45db4e32b4830a8fca27b7b2bba28" alt="Add Task button on the workflow canvas" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-3.png" />
    </Frame>
  </Step>

  <Step>
    Choose task.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-4.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=0695f75ad1418c0cb8cca13057666c1a" alt="Task type selection menu with available task options" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-4.png" />
    </Frame>
  </Step>

  <Step>
    Add additional tasks as required
  </Step>

  <Step>
    Connect tasks to create sequence of task execution. Click the connect icon on the dotted line to set the execution condition:

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-5.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=54bf89d20310c51dedacff76a0d93962" alt="Workflow tasks connected with execution condition options" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-5.png" />
    </Frame>

    * **On success (default)** - task will be executed once the preceding task was executed successfully
    * **On failure** - task will be executed once the preceding task execution failed
    * **On completion** - task will be executed once the preceding task completed, regardless to the completion status (failed/succeeded)
  </Step>
</Steps>

## Execute SQL task

<Steps>
  <Step>
    Select your DB connection.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-6.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=22c5d15230281f8254b3a3417e1aca65" alt="Execute SQL task with database connection selector" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-6.png" />
    </Frame>
  </Step>

  <Step>
    Write the SQL query that should be executed, and select the query result type from **Result type** dropdown. You can test the query by clicking **Test Query**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-7.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=00d02883499924a108ec45b3738d04d9" alt="SQL query editor with result type dropdown and Test Query button" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-7.png" />
    </Frame>
  </Step>

  <Step>
    You can assign the return value to a workflow variable.
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

<Note>
  **Note**:

  BigQuery connections uses standard SQL within Execute SQL task
</Note>

## Run package task

<Steps>
  <Step>
    Select the package to run

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-8.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=a6ecf5b8edd7101e8558a1ef8a9406c3" alt="Run package task with package selector dropdown" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-8.png" />
    </Frame>
  </Step>

  <Step>
    Optionally, you may edit the dataflow variables if you want to override the values with workflow variables.
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

## File mover task

<Steps>
  <Step>
    Select a file storage connection as the source of the file(s)
  </Step>

  <Step>
    Input source bucket (if connection requires a bucket) as well as the source path

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-9.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=2e57063368106e4eb90877bd86cc7516" alt="File mover task source connection and path configuration" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-9.png" />
    </Frame>
  </Step>

  <Step>
    You can enable following options on the source connection:

    * Delete files from source
    * Fail if the source is empty
  </Step>

  <Step>
    Select a file storage connection as the destination of the file(s)
  </Step>

  <Step>
    Input destination bucket (if connection requires a bucket) as well as the destination path. Any directories/folders in the path must already exist. If you wish to change the name of the file, define the path all the way to the file name and extension - for example, directory\_name/new\_file\_name.csv

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-10.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=19ca79a836233f0d8d6d38f86b21d1f6" alt="File mover task destination connection and path configuration" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-10.png" />
    </Frame>
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

<Note>
  **Note**:

  Package variable (\`\$\`) and Wildcard file pattern (\`\*\`) can be applied as bucket or path name.
</Note>

## Using Variables in Workflows

User variables can be defined at the workflow package level and can be used for both the Execute SQL Task and the Run Package Task.

1. **Execute SQL Task**

   * Variables can also be assigned values by the Execute SQL task. This is useful if you want to have dynamic values on your variable and use it later on.
     * When using variables in a SQL query, enclose the variable within curly brackets (i.e: `'${var\_name}'`).
     * Example of using a variable within the Execute SQL task query:\
       DELETE FROM event\
       WHERE `date>='${START\_DATE}'`\
       AND `date<='${END\_DATE}'`

2. **Run Package Task**
   * Workflow package level variables can be used to override dataflow level variables. Here's an example:

     <Frame>
       <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-11.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=cf33b9a6b58473155fbf7255ab4a0199" alt="Workflow variables overriding dataflow package variables" width="1200" height="639" data-path="images/creating-packages/creating-a-workflow/image-11.png" />
     </Frame>

     <Note>
       **Note:**

       Take note that we address package variables regularly as **\$variable\_name** which is different from the way we used it on Execute SQL Task above.
     </Note>

     * If both workflow and dataflow variables have the same name, you will still have to assign the workflow variable to the dataflow variable.
     * If a task dataflow uses a variable that isn't defined at the dataflow level but is assigned a value at the workflow level, the dataflow task will use the workflow variable value.
