{{ site.title }}

{{ theme.darkModeSwitchLabel || 'Appearance' }}

Migration: Remove Legacy flowMeter Field ​

Date: 2026-01-15
Type: Breaking Change
Status: Completed

Summary ​

Removed all legacy flowMeter string fields from the flow meter and calibration systems. The system now exclusively uses asset_id with JOIN to data_assets table to retrieve display_id for display purposes.

Motivation ​

  • Eliminate data redundancy and inconsistency
  • Enforce referential integrity through foreign keys
  • Standardize on asset-based identification system
  • Simplify data model and reduce maintenance burden

Changes Made ​

1. Type Definitions (src/features/flow-meter/types.ts) ​

Before:

typescript
export type FlowMeterRecord = {
  site: string;
  flowMeter: string; // DEPRECATED
  assetId?: string;
  assetDisplayId?: string;
  // ...
};

export type DustLocRefill = {
  site: string;
  flowMeter?: string; // DEPRECATED
  assetId?: string;
  assetDisplayId?: string;
  // ...
};

export type FlowMeterCalibration = {
  site: string;
  flowMeter?: string; // legacy fallback
  assetId?: string;
  // ...
};

After:

typescript
export type FlowMeterRecord = {
  site: string;
  assetId: string;  // Now required
  assetDisplayId: string;  // Now required
  // ...
};

export type DustLocRefill = {
  site: string;
  assetId: string;  // Now required
  assetDisplayId: string;  // Now required
  // ...
};

export type FlowMeterCalibration = {
  site: string;
  assetId: string;  // Now required
  assetDisplayId: string;  // Now required
  // ...
};

2. Database Service (src/features/flow-meter/services/databaseService.ts) ​

Updated Mapping Functions: ​

  • mapFlowMeterRow() - Now only uses assetId and assetDisplayId
  • mapDustLocRefillRow() - Removed flowMeter field
  • mapFlowMeterCalibrationRow() - Removed flowMeter field, added assetDisplayId

Updated Query Functions (All now JOIN with data_assets): ​

  • fetchAllFlowMeterData() - JOINs data_assets for display_id
  • fetchFlowMeterDataByDateRange() - Added JOIN
  • fetchAllRefills() - Already had JOIN, removed fallback
  • fetchRefillsByDateRange() - Added JOIN
  • fetchRefillsBySite() - Added JOIN
  • fetchAllCalibrations() - Added JOIN

Updated Insert/Update Functions: ​

  • insertRefill() - Removed flow_meter parameter, asset_id now required
  • updateRefill() - Removed flow_meter parameter
  • insertCalibration() - Removed flow_meter parameter, asset_id now required
  • updateCalibration() - Removed flow_meter parameter
  • insertFlowMeterData() - Changed from flow_meter to asset_id

3. UI Components ​

AddRefillModal (src/features/flow-meter/components/AddRefillModal.tsx) ​

  • Removed flowMeter state and legacy flow meter dropdown
  • Simplified to only show asset selection
  • Shows error message if no assets available for site
  • onSubmit interface updated to require assetId

EditRefillModal (src/features/flow-meter/components/EditRefillModal.tsx) ​

  • Removed flowMeter state
  • Flow meter now read-only (cannot be changed after creation)
  • Site selection disabled during edit
  • Displays assetDisplayId in read-only field
  • onSubmit interface updated to require assetId

SiteSummaryCard (src/features/flow-meter/components/SiteSummaryCard.tsx) ​

  • Filtering logic updated to only use assetDisplayId
  • Removed flowMeter fallback

CalibrationModal (src/features/flow-meter/components/CalibrationModal.tsx) ​

  • Removed flowMeter field handling
  • assetId now required (validation added)
  • onSubmit interface updated

4. Page Components ​

Refill Management (src/app/(admin)/(pages)/refill-management/index.tsx) ​

  • Table displays only assetDisplayId
  • handleAddRefill updated to pass assetId
  • handleEditRefill updated to pass assetId
  • Removed flowMeterRecords prop from EditRefillModal

Flow Meter Index (src/app/(admin)/(pages)/flow-meter/index.tsx) ​

  • Filter logic updated to use assetDisplayId exclusively
  • Table columns display assetDisplayId

Calibration Management (src/app/(admin)/(pages)/calibration-management/index.tsx) ​

  • Search updated to use assetDisplayId
  • Total assets calculation simplified
  • handleAddCalibration requires assetId
  • handleEditCalibration requires assetId
  • Table displays assetDisplayId

5. Dashboard Components ​

CalibrationReminder (src/app/(admin)/(dashboards)/index/components/CalibrationReminder.tsx) ​

  • Key generation uses only assetId
  • Display label uses assetDisplayId

useDashboardCalibrationCompliance Hook ​

  • Updated to use assetId for keys
  • Display label uses assetDisplayId

6. Data Services ​

dataAggregation.ts ​

  • Deduplication key updated to use assetId instead of flowMeter

csvParser.ts ​

  • Updated CSV format to expect 6 columns:
    1. site
    2. assetId
    3. assetDisplayId
    4. litres
    5. dateTime
    6. scrapedAt
  • Removed flowMeter column

dataImporter.ts ​

  • Import logic updated to use asset_id instead of flow_meter
  • Both batch import functions updated

Database Schema Changes ​

The migration assumes the following database schema changes have been applied:

  1. data_flow_meters table:

    • flow_meter column deprecated (or removed)
    • asset_id column is required with FK to data_assets(asset_id)

  2. ops_dustloc_refills table:

    • flow_meter column deprecated (or removed)
    • asset_id column is required with FK to data_assets(asset_id)

  3. ops_flow_calibrations table:

    • flow_meter column deprecated (or removed)
    • asset_id column is required with FK to data_assets(asset_id)

Data Migration Required ​

IMPORTANT: Before deploying this change, a data migration must be run to:

  1. Populate asset_id for all existing records that only have flow_meter
  2. Create corresponding data_assets entries if they don't exist
  3. Verify all foreign key constraints are satisfied

Breaking Changes ​

  1. CSV Import Format Changed:

    • Old format: site, flowMeter, litres, dateTime, scrapedAt
    • New format: site, assetId, assetDisplayId, litres, dateTime, scrapedAt

  2. API Interfaces Changed:

    • All refill/calibration creation/update functions now require assetId
    • flowMeter parameter removed from all interfaces

  3. Component Props Changed:

    • Components no longer accept or display flowMeter field
    • assetId and assetDisplayId are now required

Validation ​

Build status: ✅ SUCCESS

bash
pnpm build:check
# Exit code: 0

Testing Checklist ​

  • [ ] Add new refill with asset selection
  • [ ] Edit existing refill (verify flow meter is read-only)
  • [ ] Add new calibration with asset selection
  • [ ] Edit existing calibration
  • [ ] View refill management table
  • [ ] View calibration management table
  • [ ] View flow meter page with filtering
  • [ ] Check dashboard calibration reminders
  • [ ] Import CSV with new format
  • [ ] Verify data aggregation works correctly

Files Modified ​

Type Definitions: ​

  • src/features/flow-meter/types.ts

Database Services: ​

  • src/features/flow-meter/services/databaseService.ts
  • src/features/flow-meter/services/dataAggregation.ts
  • src/features/flow-meter/services/csvParser.ts
  • src/features/flow-meter/services/dataImporter.ts

Components: ​

  • src/features/flow-meter/components/AddRefillModal.tsx
  • src/features/flow-meter/components/EditRefillModal.tsx
  • src/features/flow-meter/components/SiteSummaryCard.tsx
  • src/features/flow-meter/components/CalibrationModal.tsx

Pages: ​

  • src/app/(admin)/(pages)/refill-management/index.tsx
  • src/app/(admin)/(pages)/flow-meter/index.tsx
  • src/app/(admin)/(pages)/calibration-management/index.tsx

Dashboard: ​

  • src/app/(admin)/(dashboards)/index/components/CalibrationReminder.tsx
  • src/features/dashboard/hooks/useDashboardCalibrationCompliance.ts

Rollback Plan ​

If issues are discovered:

  1. Revert all code changes (git revert)
  2. Do NOT rollback database schema changes (keep asset_id columns)
  3. Re-apply data migration to restore flowMeter values if needed
  4. Consider gradual migration approach instead

Benefits ​

  1. Data Integrity: Foreign key constraints prevent orphaned records
  2. Single Source of Truth: Asset display names managed centrally
  3. Simplified Logic: No more fallback chains or duplicate display logic
  4. Type Safety: Required fields prevent null/undefined issues
  5. Maintainability: Cleaner codebase without legacy code paths

Notes ​

  • The migration requires database schema changes to be applied first
  • All existing records must have valid asset_id values before deployment
  • CSV import files must be updated to new format
  • Any external integrations using the old format will break