Processing 4 Petabytes of Data from Space

Development of the world’s best (and most complex) algorithm for aerosol detection.

Observing global climate events

In collaboration with the GRASP team, Cloudflight focuses on the GRASP algorithm, the most advanced algorithm for aerosol and surface retrieval algorithm. One of the main goals is to parallelize the calculations to that, the algorithm is fast enough to compute aerosol particles in the atmosphere. The resulting data is used for a better understanding of global climate development.

Michael Aspetsberger, Industry Focus Leader Aerospace at Cloudflight

Michael Aspetsberger
AeroSpace Industry Focus Lead
Cloudflight

0

processing (virtual) machines

0 PB

of storage

0k

lines of framework code developed by us

Challenge

For several decades, atmospheric researchers wrote thousands of lines of Fortran code that was always varied and complex, with the goal of achieving the best possible results without ever considering the run time. If the original algorithm was to to be applied  the same data for a 100-minute satellite orbit, one would have to wait months for the result – so there is no way to think of a productive use.

At the same time, modern satellite instruments generate large amounts of data – 24/7 over their entire lifetime. So even before thinking about processing data, accessing and storing it is a challenge in itself.

But even after the data has been processed, the process is not over. The results need to be scientifically validated. This step requires manual steps which can only be done with the expertise of a scientist.

Idea

In this project, we support the scientists working on GRASP in improving the algorithm. To overcome the problems of lack of computing power, challenging data access and complex validation procedures, our approach is to shift the development environment of the scientists to the resources. Additionally, a software framework can be developed and reused for different processes.

Solution

We provide the GRASP team with a cloud-based development environment in which they can easily run GRASP on our data archive and evaluate the generated results using a tailor-made framework we developed together with them.

In order to scale GRASP to huge amounts of data, we have optimized it in terms of runtime and memory requirements. In addition, we built up our own cluster infrastructure and implemented a decentralized processing system.

How Cloudflight can help in wildfires detection

Air quality can be monitored by analyzing the development of aerosol particles in the atmosphere.
We are able to detect wildfires and distinguish them from other aerosols, such as harmless clouds. This is done by identifying a high concentration of smoke-trail particles.
In this study case, you can see the results of identified forest/wildfires forming up (shown up as bright red spots, see circle) during September 2020 in the area of Paraguay.
The image describes the retrieved values for Aerosol Optical Depth at 490nm; while values near 0 correspond to an extremely clean atmosphere, values over 0.5 represent higher concentrations of aerosols, which in this case correspond to the identified wildfires, reaching values up to 1.

Input data from Sentinel 3 satellite (OLCI instrument, part of the Copernicus program from ESA) is processed using GRASP algorithm at 1 km resolution per pixel, resulting in Aerosol Optical Depth at 490nm.

Cloudflight develops custom software solutions, with a focus on high-performance computing, scalability and reliability. Regarding this project, Cloudflight has been optimizing the algorithm, providing a working environment, processing data and writing supporting code.

GRASP SAS provides services in every area of remote sensing, especially in the observations of earth atmosphere and surface. One specific activity of the company is the valorization of research development, by providing extensive services in utilization and adaptation of (Generalized Retrieval of Atmosphere and Surface Properties) GRASP open-source code for processing diverse remote sensing observations.

Vision

  • Would you also like to be able to speed up calculations?
  • Would you like to to run GRASP on your own data?
  • You want to buy an existing data product?
  • We look forward to develop a plan for the future together!
Michael Aspetsberger, Industry Focus Leader Aerospace at Cloudflight

Michael Aspetsberger
AeroSpace Industry Focus Lead
Cloudflight

Sign up to our monthly wrap-up and get access to exclusive material and updates about our technologies, research content and events. You can always adjust your settings.

Menu