"Filter columns" node

This node allows you to keep or remove columns that match the rule you define in the configuration of the node. In result, if you use the data transformation rule that contains this node, the system removes or keeps the matching columns as a part of data transformation in the workflow.

Node configuration

1. In the upper right corner of the pop-up, click the icon.
2. Select one of the following options:
   • Remove columns - This option deletes columns that match the conditions you define in the further steps.
   • Keep columns - This option keeps only the columns that match the conditions you define in the further steps. The columns that don’t match the conditions are removed.
3. From the dropdown list, select one of the logical operators:
   • Equal
   • Not equal
   • Ends with
   • Starts with
   • Contain
   • Not contain
4. In the text field, enter the value.
5. To add more conditions, click Add condition.
6. Repeat steps 1-4.
7. To define what happens if rows contain errors, select one of the options available in the Error handling section.
8. Confirm the settings by clicking Apply.

Handle errors

You can define what to do with failed values (for example, as a result of Jinja rendering or in other cases) in cells where data type modification failed. In the Error handling section, select one of the following options:

• Skip row if error occurred - It omits the row when an exception occurs.
• Skip if result is null or empty string - It omits rows with empty strings and null (which may be the result of rendering Jinja)
• Stop further transformation - The transformation stops at the row with an error. The data before the line that caused the error is transformed (and imported if the transformation rule is used in the Data Transformation node).
• Insert null if error occurred - Wrong values are replaced with null. If you use such a transformation for the import of profiles, the existing value of an attribute is cleared and replaced with the null value.
• Insert empty string if error occurred - Wrong value is removed and the cell remains empty.

Example of use

While importing a list of transactions, you can edit out unnecessary columns. In this example, these are any columns that include an underscore in their name (import_type, payment_info) and columns whose name is exactly customers.

price,customer,orderId,payment_info,date,import_type
10.99,John Doe,123345,cash,12.03.2022,toSynerise
4.99,Dwight Addams,122345,cash,12.03.2022,toSynerise
8.99,Andrew Malone,112345,cash,12.03.2022,toSynerise

1. Start the diagram with Data Input.
2. In the configuration of the node, upload a sample file.
3. Add the Filter columns node.
4. In the configuration of the node:
   1. Select Remove columns.
   2. Add the following conditions:
      • Contain _
      • Equal customer

      

   3. Confirm by clicking Apply.
5. Add the Data Output node.
   Result: You can use this transformation in the Automation workflow to import or export files with the structure presented in this example.