{{ site.title }}

{{ theme.darkModeSwitchLabel || 'Appearance' }}

PMTiles Migration Implementation Plan ​

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Migrate Marandoo tile overlay from 138K scattered PNG files to a single PMTiles file with WebP compression, served via Cloudflare Worker.

Architecture: PNG tiles → WebP conversion → MBTiles packaging → PMTiles conversion. Cloudflare Worker reads PMTiles from R2 via range requests, returns individual tiles. Front-end URL format unchanged.

Tech Stack: cwebp (installed), mb-util (pip), pmtiles CLI (npm), pmtiles JS library, Cloudflare Workers, R2

Task 1: Install Required Tools ​

Step 1: Install mb-util and pmtiles CLI

bash
pip install mbutil
npm install -g pmtiles

Step 2: Verify installations

bash
mb-util --help
pmtiles --help

Expected: Both commands show help output.

Step 3: Commit — N/A (no code changes)

Task 2: Create Tile Conversion Script ​

Files:

  • Create: scripts/convert-tiles-to-pmtiles.sh

Step 1: Create the conversion script

Create scripts/convert-tiles-to-pmtiles.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

# Usage: ./scripts/convert-tiles-to-pmtiles.sh <tiles-dir> <site-name>
# Example: ./scripts/convert-tiles-to-pmtiles.sh public/overlays/Marandoo/tiles marandoo
#
# Converts a TMS tile directory of PNGs into a single .pmtiles file with WebP compression.
# Requirements: cwebp, mb-util, pmtiles

TILES_DIR="${1:?Usage: $0 <tiles-dir> <site-name>}"
SITE_NAME="${2:?Usage: $0 <tiles-dir> <site-name>}"
QUALITY="${3:-80}"

OUTPUT_DIR="$(pwd)/build/pmtiles"
WEBP_DIR="${OUTPUT_DIR}/${SITE_NAME}-webp"
MBTILES_FILE="${OUTPUT_DIR}/${SITE_NAME}.mbtiles"
PMTILES_FILE="${OUTPUT_DIR}/${SITE_NAME}.pmtiles"

echo "=== PMTiles Conversion ==="
echo "Source:  ${TILES_DIR}"
echo "Site:    ${SITE_NAME}"
echo "Quality: ${QUALITY}"
echo "Output:  ${PMTILES_FILE}"
echo ""

# Check required tools
for cmd in cwebp mb-util pmtiles; do
  if ! command -v "$cmd" &>/dev/null; then
    echo "Error: $cmd is not installed."
    exit 1
  fi
done

mkdir -p "${OUTPUT_DIR}"

# --- Step 1: Convert PNG to WebP ---
echo ">>> Step 1/3: Converting PNG → WebP (quality ${QUALITY})..."

if [ -d "${WEBP_DIR}" ]; then
  echo "WebP directory already exists, skipping conversion. Delete ${WEBP_DIR} to reconvert."
else
  mkdir -p "${WEBP_DIR}"

  # Copy directory structure (zoom/x/ directories)
  (cd "${TILES_DIR}" && find . -type d) | while read -r dir; do
    mkdir -p "${WEBP_DIR}/${dir}"
  done

  # Convert PNGs in parallel (8 jobs)
  TOTAL=$(find "${TILES_DIR}" -name "*.png" | wc -l | tr -d ' ')
  COUNT=0

  find "${TILES_DIR}" -name "*.png" | while read -r png; do
    REL="${png#${TILES_DIR}/}"
    WEBP="${WEBP_DIR}/${REL%.png}.webp"
    echo "cwebp -q ${QUALITY} -quiet \"${png}\" -o \"${WEBP}\""
    COUNT=$((COUNT + 1))
    if [ $((COUNT % 10000)) -eq 0 ]; then
      echo "# Progress: ${COUNT}/${TOTAL}" >&2
    fi
  done | xargs -P 8 -I {} bash -c '{}'

  echo "WebP conversion complete."
fi

WEBP_COUNT=$(find "${WEBP_DIR}" -name "*.webp" | wc -l | tr -d ' ')
WEBP_SIZE=$(du -sh "${WEBP_DIR}" | cut -f1)
echo "WebP tiles: ${WEBP_COUNT} files, ${WEBP_SIZE}"

# --- Step 2: Pack into MBTiles ---
echo ""
echo ">>> Step 2/3: Packing WebP tiles → MBTiles..."

if [ -f "${MBTILES_FILE}" ]; then
  echo "MBTiles file already exists, skipping. Delete ${MBTILES_FILE} to repack."
else
  # mb-util expects TMS scheme by default, which matches our directory structure
  mb-util "${WEBP_DIR}" "${MBTILES_FILE}" --image_format=webp --scheme=tms
  echo "MBTiles packing complete."
fi

MBTILES_SIZE=$(du -sh "${MBTILES_FILE}" | cut -f1)
echo "MBTiles: ${MBTILES_SIZE}"

# --- Step 3: Convert to PMTiles ---
echo ""
echo ">>> Step 3/3: Converting MBTiles → PMTiles..."

if [ -f "${PMTILES_FILE}" ]; then
  echo "PMTiles file already exists, deleting for fresh conversion."
  rm "${PMTILES_FILE}"
fi

pmtiles convert "${MBTILES_FILE}" "${PMTILES_FILE}"

PMTILES_SIZE=$(du -sh "${PMTILES_FILE}" | cut -f1)
echo "PMTiles: ${PMTILES_SIZE}"

echo ""
echo "=== Done ==="
echo "Output: ${PMTILES_FILE}"
echo ""
echo "Upload to R2:"
echo "  wrangler r2 object put map-tiles/${SITE_NAME}.pmtiles --file=${PMTILES_FILE}"

Step 2: Make executable

bash
chmod +x scripts/convert-tiles-to-pmtiles.sh

Step 3: Commit

bash
git add scripts/convert-tiles-to-pmtiles.sh
git commit -m "feat: add tile-to-pmtiles conversion script"

Task 3: Run Conversion for Marandoo ​

Step 1: Run the conversion script

bash
./scripts/convert-tiles-to-pmtiles.sh public/overlays/Marandoo/tiles marandoo

This will take a while (~10-20 minutes for 138K tiles). Output goes to build/pmtiles/.

Expected output:

  • build/pmtiles/marandoo-webp/ — WebP tiles (~300-400MB)
  • build/pmtiles/marandoo.mbtiles — intermediate MBTiles
  • build/pmtiles/marandoo.pmtiles — final PMTiles file (~200-300MB)

Step 2: Verify the PMTiles file

bash
pmtiles show build/pmtiles/marandoo.pmtiles

Expected: Shows metadata including zoom levels 14-20, tile count, tile type webp.

Step 3: Add build output to .gitignore

Append to .gitignore:

# PMTiles build output
build/pmtiles/

Step 4: Commit

bash
git add .gitignore
git commit -m "chore: add pmtiles build output to gitignore"

Task 4: Rewrite Cloudflare Worker with PMTiles Support ​

Files:

  • Modify: workers/map-tiles/wrangler.toml
  • Rewrite: workers/map-tiles/src/index.ts
  • Create: workers/map-tiles/package.json
  • Create: workers/map-tiles/tsconfig.json

Step 1: Initialize package.json and install pmtiles

bash
cd workers/map-tiles
cat > package.json << 'EOF'
{
  "name": "map-tiles-worker",
  "private": true,
  "scripts": {
    "dev": "wrangler dev",
    "deploy": "wrangler deploy"
  },
  "dependencies": {
    "pmtiles": "^4.2.1"
  },
  "devDependencies": {
    "@cloudflare/workers-types": "^4.20250214.0",
    "wrangler": "^4.17.0"
  }
}
EOF
npm install

Step 2: Create tsconfig.json

bash
cat > tsconfig.json << 'EOF'
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ES2022",
    "moduleResolution": "bundler",
    "strict": true,
    "types": ["@cloudflare/workers-types"]
  },
  "include": ["src"]
}
EOF

Step 3: Rewrite wrangler.toml

Replace workers/map-tiles/wrangler.toml with:

toml
name = "map-tiles"
main = "src/index.ts"
compatibility_date = "2024-12-01"

[[r2_buckets]]
binding = "TILES"
bucket_name = "map-tiles"

(No change needed — keeping as-is.)

Step 4: Rewrite the Worker

Replace workers/map-tiles/src/index.ts with:

typescript
import { PMTiles, RangeResponse, Source, TileType } from "pmtiles";

interface Env {
  TILES: R2Bucket;
}

const ALLOWED_ORIGINS = [
  "https://dashboard.dustac.com.au",
  "http://localhost:3000",
];

// Tile path: /{site}/{z}/{x}/{y}.png
const TILE_PATH_RE = /^([\w-]+)\/(\d+)\/(\d+)\/(\d+)\.png$/;

// Content-Type mapping from PMTiles TileType
const TILE_CONTENT_TYPE: Record<number, string> = {
  [TileType.Png]: "image/png",
  [TileType.Webp]: "image/webp",
  [TileType.Jpeg]: "image/jpeg",
  [TileType.Avif]: "image/avif",
  [TileType.Mvt]: "application/vnd.mapbox-vector-tile",
};

/**
 * Custom PMTiles Source that reads from Cloudflare R2 via range requests.
 */
class R2Source implements Source {
  private bucket: R2Bucket;
  private key: string;

  constructor(bucket: R2Bucket, key: string) {
    this.bucket = bucket;
    this.key = key;
  }

  getKey(): string {
    return this.key;
  }

  async getBytes(
    offset: number,
    length: number
  ): Promise<RangeResponse> {
    const obj = await this.bucket.get(this.key, {
      range: { offset, length },
    });

    if (!obj) {
      throw new Error(`R2 object not found: ${this.key}`);
    }

    const data = await obj.arrayBuffer();
    return {
      data,
      etag: obj.etag,
      cacheControl: "public, max-age=86400",
    };
  }
}

// Cache PMTiles instances per site to reuse across requests in the same isolate
const pmtilesCache = new Map<string, PMTiles>();

function getPMTiles(bucket: R2Bucket, site: string): PMTiles {
  let pm = pmtilesCache.get(site);
  if (!pm) {
    pm = new PMTiles(new R2Source(bucket, `${site}.pmtiles`));
    pmtilesCache.set(site, pm);
  }
  return pm;
}

function getCorsHeaders(request: Request): Record<string, string> {
  const origin = request.headers.get("Origin") || "";
  if (ALLOWED_ORIGINS.includes(origin)) {
    return {
      "Access-Control-Allow-Origin": origin,
      "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS",
      Vary: "Origin",
    };
  }
  return {};
}

/**
 * Fallback: serve tile from scattered R2 files (backward compatibility).
 */
async function serveLegacyTile(
  bucket: R2Bucket,
  path: string,
  corsHeaders: Record<string, string>
): Promise<Response> {
  const object = await bucket.get(path);
  if (!object) {
    return new Response("Not Found", { status: 404 });
  }
  return new Response(object.body, {
    headers: {
      "Cache-Control": "public, max-age=31536000, immutable",
      "Content-Type": "image/png",
      ETag: object.httpEtag,
      ...corsHeaders,
    },
  });
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const corsHeaders = getCorsHeaders(request);

    if (request.method === "OPTIONS") {
      return new Response(null, { status: 204, headers: corsHeaders });
    }

    if (request.method !== "GET" && request.method !== "HEAD") {
      return new Response("Method Not Allowed", { status: 405 });
    }

    const url = new URL(request.url);
    const path = url.pathname.slice(1); // remove leading /

    const match = path.match(TILE_PATH_RE);
    if (!match) {
      return new Response("Not Found", { status: 404 });
    }

    const [, site, zStr, xStr, yStr] = match;
    const z = parseInt(zStr, 10);
    const x = parseInt(xStr, 10);
    const tmsY = parseInt(yStr, 10);

    // Try PMTiles first
    try {
      const pm = getPMTiles(env.TILES, site);
      const header = await pm.getHeader();

      // Convert TMS Y to XYZ Y (PMTiles uses XYZ/ZXY scheme)
      const xyzY = (1 << z) - 1 - tmsY;

      const tile = await pm.getZxy(z, x, xyzY);

      if (!tile) {
        return new Response("Not Found", { status: 404 });
      }

      const contentType =
        TILE_CONTENT_TYPE[header.tileType] || "application/octet-stream";

      return new Response(tile.data, {
        headers: {
          "Cache-Control": "public, max-age=31536000, immutable",
          "Content-Type": contentType,
          ...corsHeaders,
        },
      });
    } catch {
      // PMTiles file not found — fall back to legacy scattered tiles
      return serveLegacyTile(env.TILES, path, corsHeaders);
    }
  },
} satisfies ExportedHandler<Env>;

Step 5: Verify Worker builds

bash
cd workers/map-tiles
npx wrangler deploy --dry-run

Expected: Build succeeds without errors.

Step 6: Commit

bash
cd workers/map-tiles
git add -A
git commit -m "feat: rewrite worker with pmtiles support and CORS restriction"

Task 5: Update overlayConfig.ts maxZoom ​

Files:

  • Modify: src/features/heatmap/overlayConfig.ts (line 66)

Step 1: Change maxZoom from 18 to 20

In src/features/heatmap/overlayConfig.ts, change line 66:

typescript
// Before
maxZoom: 18,

// After
maxZoom: 20,

Also update the comment on line 58:

typescript
// Before
// Marandoo overlay - XYZ tiles (zoom 14-18)

// After
// Marandoo overlay - XYZ tiles (zoom 14-20)

Step 2: Commit

bash
git add src/features/heatmap/overlayConfig.ts
git commit -m "feat: extend Marandoo overlay to zoom level 20"

Task 6: Upload PMTiles to R2 and Deploy Worker ​

Step 1: Upload PMTiles file to R2

bash
cd /Users/jackqin/Projects/Dashboard
npx wrangler r2 object put map-tiles/marandoo.pmtiles --file=build/pmtiles/marandoo.pmtiles

Step 2: Deploy the Worker

bash
cd workers/map-tiles
npx wrangler deploy

Step 3: Verify with curl

Test a known tile (zoom 14, first x-directory):

bash
# Test PMTiles tile serving
curl -I "https://map-tiles.fudong.workers.dev/marandoo/14/13567/4566.png"

Expected headers:

  • HTTP/2 200
  • Content-Type: image/webp
  • Cache-Control: public, max-age=31536000, immutable

Test CORS:

bash
curl -I -H "Origin: https://dashboard.dustac.com.au" "https://map-tiles.fudong.workers.dev/marandoo/14/13567/4566.png"

Expected: Access-Control-Allow-Origin: https://dashboard.dustac.com.au

bash
curl -I -H "Origin: https://evil.com" "https://map-tiles.fudong.workers.dev/marandoo/14/13567/4566.png"

Expected: No Access-Control-Allow-Origin header.

Step 4: Test in browser

Open http://localhost:3000, navigate to Marandoo heatmap, verify tile overlay renders correctly at all zoom levels 14-20.

Task 7: Cleanup (Optional) ​

Step 1: Remove scattered tiles from R2

After confirming everything works in production:

bash
# List files to verify
npx wrangler r2 object list map-tiles --prefix=marandoo/ --limit=10

# Delete scattered tiles (run in batches if needed)
# This may require a script to list and delete all objects under marandoo/

Step 2: Consider removing local PNG tiles from git

The public/overlays/Marandoo/tiles/ directory (1.7GB) can be removed from the repo if it's tracked. Keep the conversion script and source GeoTIFF as the source of truth.