Data engineering: ELT – a faster, more fluid data platform

But is it the right choice for your organisation's data needs? The sheer speed and volume of data these days is staggering. Many organisations have databases overwhelmed with data and it’s become imperative to the success of many businesses (both financial and operational) to find an effective and scalable way to process and analyse it — also known as data warehousing.

While data cleansing is an important part of data warehousing, without a scalable data platform that is prepared for the potential growth in your data needs, ‘cleaning up’ won’t solve the problem of ‘too much data’ for long.

ETL vs ELT: what you need to know

ETL (extract, transform, load) has been the traditional approach for data analytics and warehousing for the last couple of decades. But with the introduction of cloud technologies, ELT (extract, load, transform), also known as ‘ingest, store, transform’, is a more modern approach to data processing that offers some organisations more speed and fluidity.

What’s the difference?

ETL: data is extracted from a source system, transformed on a secondary processing server and loaded into a destination system. Every time you want to update a data point, you must go back to the source to extract, transform then load all the data.
ELT: data is extracted from a source system, loaded into a destination system and transformed inside the destination system. If you want to update your data, the platform only looks for the data points you want to update, not all the data.

The challenges of more data and source types

According to a whitepaper published by Aberdeen and Sisense, surveyy respondents use an average of 30 unique data sources regularly, and 40% of respondents analyse unstructured data from both internal and external sources.

Doing that efficiently requires a platform that supports structured and unstructured data from multiple sources and in large volumes.

The advantages of ELT

While it might seem like just re-ordering the steps of the more traditional ETL approach, loading data to its destination before transforming it does have several advantages, including:

Data is loaded and transformed simultaneously. Transforming data on a separate server before the loading process slows down the process of data ingestion, resulting in a slow-running platform.
Less time required for data maintenance. With ETL, a secondary processing server adds to the maintenance burden. With ELT, the original data is intact and already loaded should additional transformation be necessary.
Raw data is loaded directly into the destination system and can be re-queried endlessly. The raw data retention of ELT creates a rich historical archive for generating business intelligence. As goals and strategies change, BI teams can re-query raw data to develop new transformations using comprehensive datasets.
More flexible, accessible and scalable. This is especially relevant for ingesting large amounts of data, processing data sets that contain both structured and unstructured data and developing diverse business intelligence.

What makes ELT the right choice?

Speed and efficiency. ELT technology can handle a substantial amount of data and load structured and unstructured data into the target warehouse, making it the best approach when your organisation needs quick access to available data.
Data security. Direct loading of data requires more privacy safeguards, transforming the data on an as-needed basis for analysis. ELT can redact and remove sensitive information before putting it into the data warehouse or cloud server, making it easier to satisfy GDPR, HIPAA, and CCPA compliance standards.
A simplified data stack costs less. The ELT process enables you to load information without transformation, saving the high initial costs of data processing.

ETL or ELT: it all comes down to your business needs

Choosing the right approach requires a deep understanding of your organisation’s data needs — now and in the future — so you can ultimately empower your teams to identify and analyse all your data in a way that will help you make well-informed business decisions.

To become a data driven organisation, there are many components — Architecture, Engineering, Warehousing, and Analytics — that, when working in harmony, improve performance outcomes for organisations.

Is there still a place for ETL?

The short answer is yes. It’s ideal for:

Small data sets with complicated transformation requirements (where speed isn’t required)
Computer-intensive transformations, systems with legacy architectures or data workflows that require manipulation before entering a target system.