From 138,000 Scattered Files to One: Migrating Mine Site Map Overlays to PMTiles

The Problem

The Dust Ranger dashboard includes a heatmap feature that overlays high-resolution aerial imagery of mine sites on top of Google Maps. Users toggle these overlays to see up-to-date site layouts beneath the dust monitoring data points — a critical visual reference when interpreting where dust concentrations are highest relative to haul roads, crushers, and stockpiles.

The original approach was straightforward: take a georeferenced aerial image, slice it into XYZ map tiles using gdal2tiles.py, and upload every individual PNG file to Cloudflare R2. A Cloudflare Worker would then serve each tile on demand.

For the Marandoo mine site alone, this produced roughly 138,000 PNG files totaling 1.7 GB. The approach worked, but it came with real operational pain:

• Uploading was slow. Pushing 138K small objects to R2 took a long time and was error-prone.
• Cleanup was painful. Deleting or replacing an overlay meant iterating through thousands of objects.
• Adding new sites scaled linearly. Every new mine site meant repeating the same bulk upload process.
• No compression. The tiles were stored as raw PNGs — no opportunity to leverage modern image formats like WebP.

We needed a solution that preserved the existing front-end tile URL format (so the Google Maps ImageMapType integration wouldn't change) while collapsing the storage and deployment complexity down to something manageable.

Why PMTiles

PMTiles is a single-file archive format for map tiles. Instead of storing each tile as a separate object, all tiles are packed into one file with an internal directory that maps z/x/y coordinates to byte ranges. A client — or in our case, a Cloudflare Worker — can retrieve any individual tile by issuing an HTTP range request against the single file.

This was a natural fit for our setup:

• One file per mine site. Marandoo goes from 138K files to marandoo.pmtiles. Gudai Darri becomes gudai-darri.pmtiles.
• WebP compression. We convert tiles from PNG to WebP during the packaging step, cutting storage by roughly 90%.
• Unchanged front-end. The Worker still responds to /{site}/{z}/{x}/{y}.png requests — the Google Maps integration doesn't know or care that the backend changed.
• Trivial deployment. Adding a new site means uploading one file to R2 and adding a config entry.

The Conversion Pipeline

We built a reusable shell script (scripts/convert-tiles-to-pmtiles.sh) that takes a directory of TMS-structured PNG tiles and produces a single .pmtiles file in three steps:

PNG tiles → WebP tiles → MBTiles → PMTiles

Step 1: PNG to WebP

The script walks the tile directory and batch-converts every PNG to WebP using cwebp at quality 80, running 8 parallel jobs:

find "${TILES_DIR}" -name "*.png" | while read -r png; do
    REL="${png#${TILES_DIR}/}"
    WEBP="${WEBP_DIR}/${REL%.png}.webp"
    echo "cwebp -q ${QUALITY} -quiet \"${png}\" -o \"${WEBP}\""
done | xargs -P 8 -I {} bash -c '{}'

This is the biggest win in terms of file size. WebP at quality 80 retains visual fidelity for aerial imagery while dramatically reducing the per-tile size.

Step 2: WebP to MBTiles

The WebP tiles are packed into an MBTiles file (a SQLite database) using mb-util:

mb-util "${WEBP_DIR}" "${MBTILES_FILE}" --image_format=webp --scheme=tms

MBTiles is an intermediate format — it gives us a single-file container that the pmtiles CLI can then convert.

Step 3: MBTiles to PMTiles

The final conversion uses the pmtiles CLI:

pmtiles convert "${MBTILES_FILE}" "${PMTILES_FILE}"

The output is a single file optimized for HTTP range request access. For Marandoo, the result was 172 MB — down from 1.7 GB of raw PNGs. Gudai Darri came out at 11 MB.

The script is idempotent: it skips steps if intermediate outputs already exist, so you can re-run it safely. Usage is simple:

./scripts/convert-tiles-to-pmtiles.sh path/to/tiles site-name [quality]

Required Tools

• cwebp — brew install webp
• mb-util — pip install mbutil
• pmtiles — brew install pmtiles or npm install -g pmtiles

The Cloudflare Worker

The Worker (workers/map-tiles/src/index.ts) is the bridge between the front-end's tile requests and the PMTiles files sitting in R2. It was completely rewritten for this migration.

R2Source: Connecting PMTiles to R2

The PMTiles JavaScript library expects a Source interface — an object that can fetch arbitrary byte ranges from the archive. We implemented a custom R2Source class that maps this to R2's range request API:

class R2Source implements Source {
  private bucket: R2Bucket;
  private key: string;

  constructor(bucket: R2Bucket, key: string) {
    this.bucket = bucket;
    this.key = key;
  }

  getKey(): string {
    return this.key;
  }

  async getBytes(offset: number, length: number): Promise<RangeResponse> {
    const obj = await this.bucket.get(this.key, {
      range: { offset, length },
    });
    if (!obj) throw new Error(`R2 object not found: ${this.key}`);
    const data = await obj.arrayBuffer();
    return { data, etag: obj.etag, cacheControl: "public, max-age=86400" };
  }
}

When the PMTiles library needs to read the internal directory or fetch a specific tile, it calls getBytes with an offset and length. The R2Source translates this into an R2 get() call with a range option — no need to download the entire file.

Request Flow

The Worker handles incoming requests through a clean pipeline:

1. Parse the URL. Extract site, z, x, and y from /{site}/{z}/{x}/{y}.png using a regex.

2. Get or create a PMTiles instance. PMTiles instances are cached per site in a Map to avoid re-initializing on every request within the same Worker isolate.

3. Convert coordinates. The front-end uses TMS convention (Y-axis flipped), but PMTiles uses XYZ. The conversion is a single line:

   const xyzY = (1 << z) - 1 - tmsY;

4. Fetch the tile. Call pm.getZxy(z, x, xyzY) to retrieve the tile data from the PMTiles archive.

5. Detect content type. Read the tile format from the PMTiles header and map it to the correct MIME type (WebP, PNG, JPEG, etc.).

6. Return with cache headers. Tiles are immutable, so we set Cache-Control: public, max-age=31536000, immutable — a full year.

Backward Compatibility

Not every site was migrated at the same time. The Worker includes a fallback path: if the PMTiles lookup throws (typically because {site}.pmtiles doesn't exist in R2), it falls back to fetching the tile as a scattered file at {site}/{z}/{x}/{y}.png. This allowed us to migrate sites incrementally without any downtime.

try {
  const pm = getPMTiles(env.TILES, site);
  // ... fetch from PMTiles
} catch {
  return serveLegacyTile(env.TILES, path, corsHeaders);
}

CORS

The Worker restricts cross-origin access to two origins:

const ALLOWED_ORIGINS = [
  "https://dashboard.dustac.com.au",
  "http://localhost:3000",
];

It checks the Origin header and only returns Access-Control-Allow-Origin if the origin is whitelisted. Requests from other origins still get tile data (it's not secret), but browsers won't allow cross-origin use of the response.

Front-End Integration

The front-end integration lives in the heatmap page component and a small configuration module. The key design goal was to keep the overlay system declarative: define overlays in a config file, and the component handles the rest.

Overlay Configuration

Each mine site overlay is defined in src/features/heatmap/overlayConfig.ts:

export type TileMapOverlay = {
  type: "tile";
  id: string;
  name: string;
  mineSiteId: string;
  tileBasePath: string;
  minZoom: number;
  maxZoom: number;
  bounds: MapOverlayBounds;
};

The bounds field is critical — it defines the geographic rectangle that the overlay covers. Tiles outside these bounds are skipped entirely, avoiding unnecessary network requests.

Currently two sites are configured:

Site	PMTiles Size	Zoom Levels
Marandoo	172 MB	14–20
Gudai Darri	11 MB	14–20

Tile Rendering with Google Maps ImageMapType

The overlay is rendered using google.maps.ImageMapType, which lets you define a custom getTileUrl function. This function is called by Google Maps for every visible tile at the current zoom level:

const tileOverlay = new googleMaps.ImageMapType({
  getTileUrl: (coord, zoom) => {
    if (zoom < minZoom || zoom > maxZoom) return null;

    const n = 1 << zoom;
    const tmsY = n - 1 - coord.y;

    // Calculate tile geographic bounds
    const tileLngWest = (coord.x / n) * 360 - 180;
    const tileLngEast = ((coord.x + 1) / n) * 360 - 180;
    const tileLatNorth =
      (Math.atan(Math.sinh(Math.PI * (1 - (2 * coord.y) / n))) * 180) /
      Math.PI;
    const tileLatSouth =
      (Math.atan(Math.sinh(Math.PI * (1 - (2 * (coord.y + 1)) / n))) * 180) /
      Math.PI;

    // Skip tiles outside overlay bounds
    if (
      tileLatSouth > bounds.north ||
      tileLatNorth < bounds.south ||
      tileLngWest > bounds.east ||
      tileLngEast < bounds.west
    ) {
      return null;
    }

    return `${tileBasePath}/${zoom}/${coord.x}/${tmsY}.png`;
  },
  tileSize: new googleMaps.Size(256, 256),
  opacity: controls.overlayOpacity,
});

A few things worth noting:

• Bounds checking. For each tile, the function converts the tile's x/y coordinates to latitude/longitude ranges using Mercator projection math. If the tile falls entirely outside the overlay's geographic bounds, it returns null — Google Maps won't request that tile at all.

• TMS Y-axis. Google Maps provides tile coordinates in XYZ convention (Y increases downward), but our tiles are stored in TMS convention (Y increases upward). The conversion tmsY = n - 1 - coord.y handles this.

• Opacity control. The overlay opacity is set at creation time and can be updated dynamically via setOpacity() when the user adjusts the slider.

Layer Order

The rendering stack from bottom to top is:

1. Google Maps satellite imagery (base layer)
2. Mine site tile overlay (ImageMapType)
3. Canvas heatmap overlay (dust monitoring data points)

This ensures the aerial imagery provides context without obscuring the actual dust data.

UI Controls

When an overlay is available for the selected mine site, the control panel shows:

• A toggle switch to show/hide the overlay
• An opacity slider (10%–100%, 5% increments)
• The overlay name for reference

The overlay defaults to visible at 75% opacity — enough to see the site layout while still letting the satellite base map show through.

The Georeferencing Pipeline

Before tiles can be created, the raw aerial image needs to be georeferenced — aligned to real-world coordinates. This is a manual process done in QGIS:

1. Load the aerial image into QGIS as a raster layer.
2. Select ground control points (GCPs) — identifiable features (road intersections, building corners) that can be matched between the aerial image and the satellite base map.
3. Apply a polynomial transformation to warp the image to the correct geographic position.
4. Export as GeoTIFF with the coordinate reference system embedded.
5. Generate tiles using gdal2tiles.py with TMS scheme.
6. Convert to PMTiles using the conversion script described above.

The full pipeline from raw image to deployed overlay:

Aerial image
  → QGIS georeferencing (GCPs + polynomial transform)
  → GeoTIFF
  → gdal2tiles.py (TMS PNG tiles)
  → convert-tiles-to-pmtiles.sh (PNG → WebP → MBTiles → PMTiles)
  → wrangler r2 object put (upload to R2)
  → Add entry in overlayConfig.ts

Adding a New Mine Site

With the infrastructure in place, adding a new mine site overlay is a repeatable process:

1. Georeference the aerial image in QGIS and export as GeoTIFF.
2. Generate TMS tiles with gdal2tiles.py.
3. Run the conversion script:

   ./scripts/convert-tiles-to-pmtiles.sh path/to/tiles site-name

4. Upload the single PMTiles file to R2:

   npx wrangler r2 object put map-tiles/site-name.pmtiles \
     --file=build/pmtiles/site-name.pmtiles --remote

5. Add an entry in overlayConfig.ts with the site's bounds, zoom range, and mine site ID.

No Worker changes needed. No bulk file uploads. One file, one config entry.

Results

Metric	Before	After
Files per site (Marandoo)	~138,000 PNGs	1 PMTiles file
Storage (Marandoo)	1.7 GB	172 MB
Image format	PNG	WebP (quality 80)
Deployment	Bulk upload 138K objects	Upload 1 file
Adding a new site	Repeat bulk upload	Run script + upload 1 file
Rollback	Delete 138K objects	Delete 1 file
Front-end changes	None required	None required

The migration reduced storage by roughly 90%, turned a multi-hour deployment process into a single upload command, and made the system trivially extensible to new mine sites. The front-end URL format stayed identical, so the Google Maps integration required zero changes — the Worker transparently serves WebP tiles from PMTiles while the browser still requests /{site}/{z}/{x}/{y}.png.