2024-02-28 - Landing to History - allow users to override column list¶

Release¶

Status: Available

Type: DataOps

Date: 2024-02-28

Problem¶

If landing contains a column we dont want to load the current template doesnt let us remove it as its an autogenerated rule that brings through all columns .

Solution¶

Allow users to use the “Output Field” rule for landing to hisotry to select the columns they want to persist into history, and exclude attributes they dont want loaded.

Leverage the Magic¶

Leverage our extensible jinja template pattern by updating the pattern that creates the column names.

Magician Partner¶

{% if config.standard_displayed_columns -%}
   {# user is overriding the default pattern #}
   {% for column in config.standard_displayed_columns.split(';') %}
     {{ column|trim }}{{ ", " if column  }}
   {% endfor %}
 {% else %}
   {# default pattern - use source columns #}
   {% for column in config.source_columns %}
     {% if column|trim|lower == 'event_id' %}
       {{ "event_id as event_id_," }}
     {% else %}
       {{ column|trim ~ ", "}}
     {% endif %}
   {% endfor %}

{% endif %}

Last Refreshed¶

Doc Refreshed: 2024-05-20