Blog #portfolio

CoGeoTIFF Research on Space Tech SaaS Platform at Pixxel

Research notes on Cloud Optimized GeoTIFFs at Pixxel: the tile servers that exist, what it takes to build a product on COGs, Sentinel conversion cases, and AWS cost estimates.

Published
Updated
Reading
5 min
Author
Sudhanva Narayana

Introduction

GeoTIFF is a public domain metadata standard. It lets you put georeferencing information inside a TIFF file. This information can include the map projection, the coordinate system, and all other data that sets the exact spatial reference for the file. But these files are large, and it is difficult to serve large files over the internet (HTTP). If you do serve them, the infrastructure costs more.

CoGeoTIFF

The only purpose of a co-geotiff (Cloud Optimized GeoTIFF) is to help developers serve TIFF images over HTTP. It is a regular TIFF file that went through a compression step. A COG uses HTTP GET range requests. With these requests, a client asks for only the parts of the file that it needs.

Creation of CoGeoTiff

To convert GeoTiff files to CoGeoTiffs, use the GDAL or Rio CoGeo CLI tools. You can also read about other challenges.

Initial Development

The CoGeotiff project and Vincent Sarago developed CoGeotiff. But their tools only help you convert one image to another. A cogeotiff alone is not enough.

Rio Tiler

This tool helps you get information out of a CoGeotiff. You can read its documentation and examples.

Tile Server

A tile server serves geoinformation metadata. It usually sends JSON over HTTP. Mapbox developed an early tile server, Sentinel Tiler. It is only a wrapper around RioTiler in a Flask application.

Marblecutter

Marble Cutter is another tile server. Its developer first built it to run on AWS Lambdas.

Remote Pixel Tiler

The tile server by Remote Pixel is another example. It comes with its own examples.

CoGeo Tiler

CoGeo Tiler is another tile server from Vincent. It is ready for deployment to AWS Lambda.

Titiler - Latest WIP

Titiler is a lightweight service. Its only goal is to make map tiles dynamically from Cloud Optimized GeoTIFF. It is also ready for deployment and use in production.

Build A Product

The examples above are tile servers. At best, they are servers that spit out JSON. To render maps in a browser, you must also use a map layer such as OpenLayers, Leaflet.js, or Geotiff.js, or a combination of them.

Example Products

These examples are stale, and nobody maintains them now. But they show well how to serve cogeotiff with a UI:

Conversion from Sentinel 2 L1C to Sentinel 2 L2A

Case 1 - Tiff to COG conversion

Most of the raw data is in tiff format. If we keep the original raw data, we must convert these tiffs to COG for later processes, such as client-side image rendering. This can cause data replication, because we then store both tiff and cog. Tiff and cog have the same properties, and cog keeps all the metadata of the tiff files. So COG is the ideal solution.

Case 2 - COG to COG conversion

COG is a compressed tiff with all the properties of tiff. If the data is already in COG, the format has no effect on later processes such as machine learning. So COG storage is an advantage for both storage space and later processing.

Storage, Networking, Processing

The Sentinel-2 scenes on AWS are in JPEG2000, not in Cloud Optimized format. For a partial read of the JPEG2000 dataset, GDAL (the rasterio backend library) must send many GET requests and transfer a lot of data. The only solution is to ask the user to select the boundary to download, and then convert those JP2 files to COG. We can log each region or tile that a user requested or converted before. Then we do not need to convert it again.

  • Worst case: We convert everything to COG. This gist says that, hypothetically, a conversion of every tile costs around <$100,000.
  • Best case: We convert only the required tiles, and we do not convert tiles or images a second time.

Downloading from S3: A researcher decides to download 1 tile each of Sentinel 1, Sentinel 2 L1, and Sentinel 2 L2. Assume that an S1 tile is 7.5GB and the S2 L1 and L2 tiles are 4.5GB each. Then the total download size is 16.5 GB. We create a pipeline on Glue. It extracts these files, finds fourteen bandwidths of images, and transforms all of them to CoGeoTiff or another format. Each conversion takes 10 seconds (per image). Lambda triggers this Glue pipeline. The pipeline stores the converted files in AWS S3 again. Quota - 16.5GB internet download bandwidth ~= 7 * 16.5 = ₹289 Transformation assuming each image is 1GB in size - 10 seconds * 14 images * 3 satellites = 420 seconds ~= 7 minutes ~= ₹35 Lambda - assuming after free quota - ₹15 Total Pricing = 289 + 35 + 15 = ₹340

Downloading from Copernicus: A developer decides to download one hundred tiles of Sentinel-1 images. Assume that each tile is 7.5GB on average. Copernicus limits each user to two downloads, so we must use a queue. Each download takes around 10 minutes. Assume that the free quota is over. Then the total size is 7.5GB * 100 = 750GB, with 350 queues. We create a pipeline on Glue. It extracts these files, finds fourteen bandwidths of images, and transforms all of them to CoGeoTiff or another format. Each conversion takes 10 seconds (per image). Lambda triggers this Glue pipeline. The pipeline stores the converted files in AWS S3 again. Quota - 750GB internet bandwidth - Copernicus egress and AWS Ingress is free Transformation assuming each image is 1GB in size - 10 seconds * 14 images * 100 tiles * 1 satellite = 140 seconds * 100 ~= 3 hr ~= ₹105 SQS for 350 queues ~= ₹30 Total Pricing = 105 + 30 = ₹135

In both cases, the output format should be COG, because COG fits well with the rest of the pipeline.

AWS Pricing

Title Quantity Quota Cost (INR) Cost (USD)
S3 POST, PUT - 1000 req ₹0.38 $0.005
S3 GET - 1000 req ₹0.030 $0.0004
S3 Data transfer outbound 1 GB Monthly ₹7 $0.09
AWS Lambda 1 GB Monthly for 1M compute seconds and 2M requests ~₹1200 $15
AWS Glue ETL 1 GB Monthly for 1hr ₹35 $0.44
AWS SQS - 1m req ₹30 $0.4

Sentinel L2A to ML Case

Case 1 - Client-Side Rendering of COG

This works when the client and the server use COGs to coordinate data delivery. A COG can deliver only the information that the current view needs. This approach needs more work before it operates smoothly. But most JS stacks run the same JavaScript code in the browser and on the server.

Case 2 - COG to COG conversion

The input format, Tiff or Cog, has no effect on the ML model. The two formats are almost the same, so the training process and the time to convergence stay the same.

Doing everything on-demand and through an API then opens many possibilities to tailor the data more to how the end-user wants it. The first of these was to enable ‘clipping’, which lets a user request just the geometry they care about, instead of trying to select the scenes that overlap with their area of interest. This will evolve to enable co-registration of images, application of TOA, atmospheric correction, surface reflectance, and eventually full analytic processing of images with operations like band math to create indices and even computer vision-based object detection.

Storage, Networking, Processing

JPEG2000 storage costs less than COG storage. But someone must pay to access and process the data. If you store the data and use it to provide services, COG is probably a better long-term solution.

Short answer to the question: there is no such thing as an Ultimate data format, in the real world there are plenty of good data formats. At the end of the day, it relies on what you want the user to do.

COG Client-Side Rendering

Size: 50 Tb.

  • Storage: 50 Tb * 1000 * 0.023 = 86,000 Rs / month*
  • Data access: (1M * 5 (GET requests) / 1000) * 0.004 = Rs. 1500*
  • Processing time (1GB AWS Lambda): (1 second * 1M * 1GB / 1000) * 0.00001667 $ = Rs 2000*

A tile read from a COG is at least 3-4 times faster than a tile read from JPEG2000

Cost: 86k + 1500 + 2000 = Rs. 90,000 (*Estimated +/- 10k for processing, network transfer)

Future

A new ecosystem of geospatial algorithms can run fully on the client side. It uses a newer HTTP feature called Byte Serving. Hotosm’s ML Enabler lets you apply ML algorithms to maps directly in the browser. Development Seed’s ML Enabler post explains how the tool makes it easier to collect and organize AI-derived map information. Soon, the work to apply ML to maps will decrease. The project is not open source yet. The team plans to release it when the work moves from a research project to a more stable tool.

Conclusion

To build a custom tile server, you need a good knowledge of OpenLayers, Leaflet, or GeoTiff.js. You must also be able to request information from the server (instant updates, like React.js state) and render it on the client side. Most of the images are in AWS S3 buckets, so you can serve them directly from there. But some of them are not in tiff format. Those images can increase the conversion and storage costs.

See https://pixxel.space.

Footnotes

Theory behind COG

Cloud Native Architecture

Related work

Production ML context

See how this topic connects to production ML systems, infrastructure, and inference.

Keep reading

All posts

Real-Time Transformer Prediction at Autodesk

#machine-learning #ai #portfolio

Chat with SQLite Database

#portfolio #ai #openai

LangChain Chat and Chroma Embeddings Integration

#ai #openai #portfolio