New B2B commerce study
Download now
Intuitive and flexible APIs for big data climate research
140x speedup with optimizing data processing algorithms
Imagine you have the technical understanding, financial resources, and the right people in your organization, but timely cloud-based processing and flawless analysis of satellite data pose challenges for your team; challenges that may be due to the sheer amount or the heterogeneity of your data, or due to the lack of parallelization within your current python-based processing pipelines.
In close collaboration with researchers and EO-data specialists worldwide, the data engineering experts of Cloudflight analyze and identify possible bottlenecks in data-processing pipelines, optimize data science workflows, and develop intuitive APIs enabling the transformation of pure data into comprehensive information.
The challenge
Overcoming volume, variability, and veracity challenges
For several decades, atmospheric researchers wrote thousands of lines of Fortran code that was always varied and complex, with the goal of achieving the best possible results without ever considering the run time. If we were to apply the original algorithm, the same data for a 100-minute satellite orbit would turn into months' worth of waiting for results – so there is no way to think of a productive usage.
Satellites constantly generate data, resulting in big amounts of data that need to be stored, accessed, and processed. This can range from just few GB to multiple TB a day (full constellations).
However, the process is not over even after the data has been processed. The results need to be scientifically validated. This step requires manual steps which can only be done with the expertise of a scientist.
The solution
Tailored cloud solutions transform EO data handling
We focus on our strengths in data engineering and development instead of relying on rigid solutions; we equip researchers and experts with tailor-made cloud-based solutions for processing large earth observation data and corresponding pipelines.
By optimizing algorithms, and fostering parallelization and automation, we effectively strengthen our project partners. Using our expertise and effort, externally developed complex algorithms are shaped as part of comprehensive processing chains, enabling e.g., the detection of man-made aerosols and atmospheric changes seen from ground station or satellites.
Researchers and other user groups feel comfortable navigating web-based interactive computing platforms such as the Jupyter Hub environment and even performing small-scale data processing either in their own Jupyter Lab instances or by sending processings to our customized Nebari data platform.
We are proud that developing intuitive and flexible APIs has led Cloudflight to support contributors at the forefront of this research area.
Current projects
We have worked on >20 projects with ESA so far
We are currently working on the demonstration of an integrated approach for the validation and exploitation of atmospheric missions (DIVA) on behalf of ESA. Additionally, we are developing the land surface reflectance auxiliary dataset from Sentinel-3 optical instruments to support operational atmospheric processors (S3-LSR) on behalf of EUMETSAT.
Furthermore, we are involved in the Sentinel 4 Level 2 operational processor component development (Sentinel-4 L2OP) project, which we are undertaking on behalf of both ESA and EUMETSAT. We are also contributing to the scientific framework for the development of aerosol products (SCAERP) for EUMETSAT. Lastly, we are working on the development of advanced clouds, aerosols, and water vapor products (CAWA) for ESA.
Our partner
The European Space Agency (ESA) is Europe’s gateway to space. Its mission is to shape the development of Europe’s space capability and ensure that investment in space continues to deliver benefits to the citizens of Europe and the world.
