Billie's Innovative Use of Apache Airflow and Fivetran for Cost-Effective Warehousing

Billie.io is a fintech startup based in Berlin that is reinventing the way businesses handle payments. Through Billie, Small and Medium Enterprises (SMEs) can get instant financing for each invoice, eliminating the need to wait 90 days to get paid by their customers. Billie.io also outsources the collections process and coverage of default risk, providing a comprehensive financial solution for businesses. The company is built on the speed of its innovation, with a data architecture that is just as innovative as its business model. Since its founding in February 2017, Billie.io has been building an archetype for data-driven businesses to emulate.

Billie.io, a Berlin-based fintech startup, is revolutionizing the way businesses handle payments by providing instant financing for invoices and outsourcing the collections process and default risk coverage. However, the company faced a significant challenge in managing its data architecture. The company needed a solution that could handle the Extract, Load, and Transformation (ELT) process of their production database to the data warehouse efficiently and cost-effectively. The company also needed to avoid latency problems or Service Level Agreement (SLA) issues and prevent transformations from occurring too early. Furthermore, the company wanted to have fine-grain control over when things happen and awareness on tasks that comprise pipelines, their dependencies, and their execution.

To address these challenges, Billie.io implemented Apache Airflow, a community-managed platform that allows for complex orchestration and automation of workflows. The platform gave Billie's team fine-grain control over when things happen and awareness on tasks that comprise pipelines, their dependencies, and their execution. The company also used Fivetran to ingest data to Snowflake. The Fivetran connectors were used as-is, but Airflow was essential in the ELT process of their production database to the data warehouse. The company treated each of the steps as 'segments', which could then be dynamically scheduled or managed through Airflow. For example, the Fivetran segment, or 'Extract' and 'Load', could be scheduled and run independently of the transformation layer. This approach allowed the company to avoid latency problems or SLA issues, or even to prevent transformations from occurring too early. Additionally, Billie.io used Airflow and Fivetran’s Operators, Sensors, and Hooks to create complex and valuable workflows for the business.

• The implementation of Apache Airflow and Fivetran resulted in significant operational benefits for Billie.io. The company gained fine-grain control over when things happen and awareness on tasks that comprise pipelines, their dependencies, and their execution. This allowed for efficient data transformations across incoming data sources. The use of Airflow and Fivetran’s Operators, Sensors, and Hooks enabled the creation of complex and valuable workflows for the business. The flexibility of Airflow's scheduling allowed Billie to easily and dynamically change when a FivetranOperator is called, giving them fine control of their data warehousing costs. Furthermore, the company was able to ensure that data is fully synced and available before running a reporting process on top of Snowflake.

• Billie.io was able to save up to 20% on warehousing costs.
• The company was able to reduce the frequency of running its most important data pipelines from every five minutes during business hours to every two hours outside of business hours, significantly cutting the Snowflake compute resources used.
• A simple shift from a five-minute to a two-hour sync resulted in substantial long-term cost savings for the business.