Blog #portfolio

CoGeoTIFF Research on Space Tech SaaS Platform at Pixxel

Engineering research memo from my time at Pixxel evaluating Cloud Optimized GeoTIFFs (COGs), dynamic tile servers, Sentinel-2 conversion pipelines, and AWS unit economics.

Published
Updated
Reading
7 min
Author
Sudhanva Narayana

Introduction

A GeoTIFF is a TIFF raster with embedded georeferencing metadata (map projection, coordinate reference system, affine transform, and ground control points) so GIS tools know where each pixel sits on Earth. Raw multi-band satellite scenes are several gigabytes per file, so downloading whole GeoTIFFs over HTTP to render a web map is too slow and expensive.

When we were designing the satellite imagery platform at Pixxel, I wrote this internal research memo to evaluate Cloud Optimized GeoTIFFs (COGs), compare open-source dynamic tile servers, and estimate the AWS cost of converting and serving Sentinel-1 and Sentinel-2 scenes on demand.

CoGeoTIFF

A Cloud Optimized GeoTIFF (COG) is a standard GeoTIFF whose internal byte layout is organized for HTTP range reads. It stores internal tiles and precomputed low-resolution overviews (pyramids) at known byte offsets near the header. A client or tile server can send an HTTP GET with a Range: bytes=... header and fetch only the bounding box and zoom level currently in the user’s viewport, without downloading the rest of the multi-gigabyte scene from S3.

Creation of CoGeoTiff

You can convert standard GeoTIFFs into validated COGs using either GDAL (gdal_translate -of COG) or Development Seed’s rio-cogeo CLI plugin. Vincent Sarago’s post Do you really want people using your data? covers the layout details such as internal block sizes and overview compression.

Initial Development

The COG tooling grew out of the cogeotiff organization and Vincent Sarago’s work. Once rasters are in COG format, you also need a reader and a tile server that turn byte-range reads into map tiles.

Rio Tiler

rio-tiler is the Python library (built on Rasterio and GDAL) that reads slippy-map tiles (/{z}/{x}/{y}), bounding-box crops, and band statistics from remote COGs over S3 or HTTP.

Tile Server

A dynamic tile server sits between an S3 bucket of COGs and the browser map client, serving TileJSON metadata and rendering XYZ raster tiles on the fly:

  • Sentinel Tiler (Mapbox): One of the earliest implementations, a Flask wrapper around rio-tiler for Sentinel scenes.
  • Marblecutter: An early virtual-tiling server built to run on AWS Lambda.
  • Remote Pixel Tiler: A serverless Lambda tiler with reference notebooks and public viewer demos.
  • cogeo-tiler: Development Seed’s AWS Lambda tile server built on rio-tiler.
  • TiTiler: The FastAPI successor to cogeo-tiler. It serves dynamic XYZ tiles, STAC mosaics, and band-math expressions, and is the best option in this list.

Build A Product

A dynamic tile server only gives you HTTP endpoints returning PNG/WebP tiles and TileJSON metadata. To turn that into an interactive product in the browser, you pair TiTiler with a client-side WebGL/canvas map engine like OpenLayers, Leaflet, Mapbox GL / MapLibre, or GeoTIFF.js for direct client-side array inspection.

Example Products

A few older demos and templates that illustrate the end-to-end UX of serving COGs to a web map:

Conversion from Sentinel 2 L1C to Sentinel 2 L2A

Case 1 - Tiff to COG conversion

Storing both an uncompressed raw TIFF and a separate COG copy for web tiling doubles the S3 storage bill. Because a lossless COG preserves every band, CRS tag, and pixel value of the source GeoTIFF, we can store the archive directly as COG and drop the raw TIFFs.

Case 2 - COG to COG conversion

Downstream ML training and band-math pipelines read COGs through Rasterio and GDAL at the same speed (or faster on windowed reads) as standard GeoTIFFs, so one COG copy works for both the web map and the ML pipeline.

Storage, Networking, Processing

Public Sentinel-2 archives on AWS are stored in JPEG2000 (.jp2), not COG. Doing partial HTTP range reads against JPEG2000 over S3 forces GDAL to issue dozens of small GET requests and pull down far more bytes than needed.

That leaves two strategies:

  • Worst case (convert the entire archive up front): As Vincent Sarago estimated in this cost breakdown gist, blindly converting the entire Sentinel archive to COG costs under ~$100,000, which is cheap for a space agency, though expensive for a startup.
  • Best case (lazy, on-demand conversion with caching): Convert only the AOI/tiles a user actually requests, cache the resulting COG in our own S3 bucket, and record the tile ID in a catalog so no scene is ever converted twice.

Here is the back-of-the-envelope AWS unit math (in 2021 INR/USD) for two typical workflows:

Scenario A: Pulling from AWS S3: A researcher requests 1 tile each of Sentinel-1 (7.5 GB), Sentinel-2 L1C (4.5 GB), and Sentinel-2 L2A (4.5 GB), totaling 16.5 GB. A Lambda triggers an AWS Glue job that extracts the 14 spectral bands, converts each to COG (~10 seconds per ~1 GB band), and writes the COGs back to S3:

  • Bandwidth quota (16.5 GB transfer): ~7 × 16.5 ≈ ₹289
  • Glue ETL conversion (10s × 14 bands × 3 scenes = 420s ≈ 7 min): ~₹35
  • Lambda orchestration (post free-tier): ~₹15
  • Total run cost: 289 + 35 + 15 = ₹340

Scenario B: Pulling from Copernicus Open Access Hub: A pipeline pulls 100 Sentinel-1 tiles (100 × 7.5 GB = 750 GB). Because Copernicus caps concurrent downloads at 2 per account, we fan requests through an SQS queue (350 queue batches) into Glue and write the resulting COGs to S3:

  • Bandwidth (750 GB): Copernicus egress and AWS S3 ingress are both free
  • Glue ETL conversion (10s × 14 bands × 100 tiles ≈ 3.8 hours): ~₹105
  • SQS (350 queue batches): ~₹30
  • Total run cost: 105 + 30 = ₹135

In both flows, writing the output as COG means every subsequent map view or ML crop reads straight from S3 byte ranges.

AWS Pricing

Service Quantity Unit / Quota Cost (INR) Cost (USD)
S3 POST, PUT - per 1,000 requests ₹0.38 $0.005
S3 GET - per 1,000 requests ₹0.030 $0.0004
S3 Data transfer outbound 1 GB Monthly ₹7 $0.09
AWS Lambda 1 GB Monthly for 1M compute seconds and 2M requests ~₹1200 $15
AWS Glue ETL 1 GB Monthly for 1 hr ₹35 $0.44
AWS SQS - per 1M requests ₹30 $0.4

Sentinel L2A to ML Case

Case 1 - Client-Side Rendering of COG

When the browser speaks COG directly (via geotiff.js) or through a thin serverless tiler, the client pulls only the overview blocks needed for the current viewport. Because modern JS geospatial stacks share projection and tiling math between Node and the browser, you can even push lightweight band math (like NDVI) to the client.

Case 2 - COG to COG conversion

For model training and batch inference, reading from COG instead of raw GeoTIFF does not change model convergence or accuracy. The pixel tensors are identical, and windowed reads over S3 are faster.

Doing everything on-demand and through an API then opens many possibilities to tailor the data more to how the end-user wants it. The first of these was to enable ‘clipping’, which lets a user request just the geometry they care about, instead of trying to select the scenes that overlap with their area of interest. This will evolve to enable co-registration of images, application of TOA, atmospheric correction, surface reflectance, and eventually full analytic processing of images with operations like band math to create indices and even computer vision-based object detection.

Storage, Networking, Processing

JPEG2000 files are slightly smaller at rest than deflate- or LZW-compressed COGs, but they cost more in S3 GET requests and decode time whenever a tile server or ML worker reads a sub-region. If a scene is viewed or analyzed more than once, COG comes out ahead.

COG Client-Side Rendering (50 TB Archive Estimate)

Assuming a 50 TB working archive of COGs on S3 serving 1M tile requests per month:

  • S3 storage: 50 TB × 1000 × $0.023 ≈ ₹86,000 / month
  • S3 GET requests: (1M × 5 range GETs / 1000) × $0.0004 ≈ ₹1,500
  • Lambda tiler compute (1 GB, ~1s avg): ₹2,000

Note: Reading a slippy-map tile from a COG is 3–4× faster than decoding the same region from JPEG2000.

Estimated monthly total: ₹86,000 + ₹1,500 + ₹2,000 ≈ ₹90,000 / month (±₹10k depending on egress and cache hit rates).

Future

Projects like HOTOSM’s ML Enabler (covered in Development Seed’s write-up) connect ML inference directly to live tile streams for tasks like building footprint detection and band-index math.

What We Chose at Pixxel

For Pixxel’s platform, storing Cloud Optimized GeoTIFFs on S3 behind a FastAPI/TiTiler service and OpenLayers/GeoTIFF.js in the browser lets both the web map and our regional GPU inference pipelines read from the same S3 objects, while converting external scenes on demand only when a user requests them.

See https://pixxel.space and the companion multi-regional geospatial ML case study.

Footnotes

Theory behind COG

Cloud Native Architecture

Related work

Production ML context

See how this topic connects to production ML systems, infrastructure, and inference.

Keep reading

All posts

Real-Time Transformer Prediction at Autodesk

#machine-learning #ai #portfolio

Chat with SQLite Database

#portfolio #ai #openai

LangChain Chat and Chroma Embeddings Integration

#ai #openai #portfolio