Microservices are an infamous topic in Software, but will it be the same for Data Engineering and Machine Learning? — Microservices are a divisive topic, in the last decade Cloud applications have moved from massive monoliths to decentralised components which can operate independently from one another. …

We believe Metaflow is the optimal tool for easy implementation of complex Data Science workloads while keeping the rigour of Software Engineering structure and validation. Today, we explain why. — In previous articles, we’ve discussed some of the benefits of using the Aero Platform. These benefits however wouldn’t be possible without the core underlying technologies we use. …

There is no best Data Orchestration Platform. Part 1: From an Infrastructure Perspective From Twilio to Kubernetes, there is a mish-mash of tools to help with Data Orchestration. Let’s define a set of criteria for choosing the best for your use case. There is now a wealth of data orchestration…

Dealing with deployment, monitoring and debugging of production workflows is a pain. Aero solves this simply by providing a using consistent environment and rich monitoring UI. — Aero is a platform built to take responsibility for infrastructure, security and orchestration away from developers and allow them to focus on adding value, be that for a personal project or a company. In this series of blog posts showcasing Aero, we are building an ingest pipeline for the GDELT…

Dealing with large amounts of data is normally a pain, requiring lots of optimisations. Today we show that isn't always the case, how how Aero can help you achieve your data processing goals. — Aero is a platform built to take responsibility for infrastructure, security and orchestration away from developers and allow them to focus on adding value, be that for a personal project or a company. In this series of blog posts showcasing Aero, we are building an ingest pipeline for the GDELT…

Aero is a platform built to take responsibility for infrastructure, security and orchestration away from developers and allow them to focus on adding value, be that for a personal project or a company. It’s built on top of the open-source standard of Metaflow [1], which allows developers to run workflows…

A review of the ML/DS development ecosystem: From a Software Engineers perspective — FYI: Everything I talk about concerning Machine Learning is also true of any Data Science work; while a Data Scientist’s work might be focussed internally, if it’s used to make decisions for the business then it should be reliable and dependable. Software development has come a long way in the…

Running large-scale ETL Jobs without an army of developers behind you — ETL — or Extract, Transform, Load — is a common pattern for processing incoming data. It allows efficient use of resources by bunching the “transform” into a single bulk operation, often making it far easier to develop and maintain than its stream processing counterpart. It also lends itself well to…