As part of week three of Data School, we are covering Data Engineering Fundamentals. One of the most interesting aspects I heard about was data orchestration. Data orchestration coordinates four main parts of the data lifecycle: ingestion, workflow management, serving data, and observability. If the data lifecycle is a symphony, data orchestration is the conductor- it sets the pace and tells each instrument when to play. Orchestration also provides an efficient means to manage failures and retries in the data pipeline. In our example, we could fail gracefully by pre-determining how many times to retry querying the database for our orders and customer data if the first request fails. We could also raise an alert in slack or via email instead.
This form of automation and coordination works for all types of data architecture like batch processing, streaming, or hybrid architecture. For batch processing orchestration can be used to start workflows on a given time cadence, or as the result of a pre-determined external trigger. This way a monthly report could be generated at the touch of a button. In a streaming use case, orchestration might be used to raise real time alerts to fraud, or to provide predictive maintenance to sensors based on their health.
Another strength of data orchestration is standardizing the data format. Initial cleaning steps like deduplication and masking sensitive data can be automated through orchestration. This leaves the data cleaner and ready for downstream transformation. However, orchestration shouldn’t be confused with data integration. A helpful way of thinking about the difference is that data integration focuses on how to connect data sources, while orchestration manages the outcomes of those connections.
Now that we’ve discussed the advantages of data orchestration, it’s important to discuss potential pitfalls. Some pitfalls include:
· Increased complexity – While orchestration intends to reduce the complexity and coordination needed in the data lifecycle, it does mean incorporating another tool, as well as configuring it to properly handle all use cases, including rare or infrequent edge cases. The logic associated can quickly grow unwieldly if not implemented with careful attention.
· Increased Maintenance – Data orchestration tools aren’t meant to be “set and forget”. Regular maintenance is necessary to ensure the pipeline is working effectively and efficiently and again addresses edge cases.
· Data Quality Issues – Automating ingestion and data processing steps could introduce data quality issues if there are no guardrails or sanity checks in place.
There are many tools created for data orchestration. Apache Airflow is a python based open-source orchestrator. Since it treats workflows as code, users can version control their orchestration workflows. It also uses a graph structure to easily visualize workflows and their dependencies.
If open source isn’t the desired solution, Snowflake and Databricks are two of the most utilized paid data orchestration providers. Each allows for native orchestration using their own tools (Snowflake Tasks and Streams or Databricks Jobs) as well as running Apache Airflow jobs on each platform. Snowflake tasks also allow you to use the AFTER keyword to chain orchestration steps into a multi-step pipeline, as well as customized configuration through dynamic parameters. This pipeline can also be viewed as a graph. Databricks Jobs feature both code and no-code implementations. In addition, their history as a Big Data provider means their orchestration is equipped to manage all kinds of workflows including AI and machine learning. They also incorporate AI powered error handling.
The advantages of running native orchestration jobs are that the workflows can stay consistent and in the same ecosystem if you are already a client of either vendor. In addition, if you aren’t it opens extensive data engineering ecosystems with each vendor. As a Snowflake or Databricks customer you could create native data pipelines and then orchestrate them using the associated tool.
In summary, data orchestration can automate many data processes into a simplified multi-step pipeline that runs predictably, and surfaces alerts when necessary. It applies to any type of data architecture including batch processing and streaming. There are several providers to choose from, including open source and paid vendors like Snowflake and Databricks.