Table Of Contents
Table Of Contents

2024-02-28 - Landing to History - allow users to override column list

Release

Status: Available

Type: DataOps

Date: 2024-02-28

Problem

If landing contains a column we dont want to load the current template doesnt let us remove it as its an autogenerated rule that brings through all columns .

Solution

Allow users to use the “Output Field” rule for landing to hisotry to select the columns they want to persist into history, and exclude attributes they dont want loaded.

Leverage the Magic

Leverage our extensible jinja template pattern by updating the pattern that creates the column names.

Magician Partner

{% if config.standard_displayed_columns -%}
   {# user is overriding the default pattern #}
   {% for column in config.standard_displayed_columns.split(';') %}
     {{ column|trim }}{{ ", " if column  }}
   {% endfor %}
 {% else %}
   {# default pattern - use source columns #}
   {% for column in config.source_columns %}
     {% if column|trim|lower == 'event_id' %}
       {{ "event_id as event_id_," }}
     {% else %}
       {{ column|trim ~ ", "}}
     {% endif %}
   {% endfor %}

{% endif %}

Last Refreshed

Doc Refreshed: 2024-05-20