2025-10-24 16:01:09 +02:00

13 KiB

Raw Blame History

CSV Rate System - Implementation Guide

Overview

This document describes the CSV-based shipping rate system implemented in Xpeditis, which allows rate comparisons from both API-connected carriers and CSV file-based carriers.

System Architecture

Hybrid Approach: CSV + API

The system supports two integration types:

CSV_ONLY: Rates loaded exclusively from CSV files (SSC, TCC, NVO)
CSV_AND_API: API integration with CSV fallback (ECU Worldwide)

File Structure

apps/backend/src/
├── domain/
│   ├── entities/
│   │   └── csv-rate.entity.ts              ✅ CREATED
│   ├── value-objects/
│   │   ├── volume.vo.ts                     ✅ CREATED
│   │   ├── surcharge.vo.ts                  ✅ UPDATED
│   │   ├── container-type.vo.ts             ✅ UPDATED (added LCL)
│   │   ├── date-range.vo.ts                 ✅ EXISTS
│   │   ├── money.vo.ts                      ✅ EXISTS
│   │   └── port-code.vo.ts                  ✅ EXISTS
│   ├── services/
│   │   └── csv-rate-search.service.ts       ✅ CREATED
│   └── ports/
│       ├── in/
│       │   └── search-csv-rates.port.ts     ✅ CREATED
│       └── out/
│           └── csv-rate-loader.port.ts      ✅ CREATED
├── infrastructure/
│   ├── carriers/
│   │   └── csv-loader/
│   │       └── csv-rate-loader.adapter.ts   ✅ CREATED
│   ├── storage/
│   │   └── csv-storage/
│   │       └── rates/
│   │           ├── ssc-consolidation.csv    ✅ CREATED (25 rows)
│   │           ├── ecu-worldwide.csv        ✅ CREATED (26 rows)
│   │           ├── tcc-logistics.csv        ✅ CREATED (25 rows)
│   │           └── nvo-consolidation.csv    ✅ CREATED (25 rows)
│   └── persistence/typeorm/
│       ├── entities/
│       │   └── csv-rate-config.orm-entity.ts ✅ CREATED
│       └── migrations/
│           └── 1730000000011-CreateCsvRateConfigs.ts ✅ CREATED
└── application/
    ├── dto/                                 ⏭️ TODO
    ├── controllers/                         ⏭️ TODO
    └── mappers/                             ⏭️ TODO

CSV File Format

Required Columns

Column	Type	Description	Example
`companyName`	string	Carrier name	SSC Consolidation
`origin`	string	Origin port (UN LOCODE)	NLRTM
`destination`	string	Destination port (UN LOCODE)	USNYC
`containerType`	string	Container type	LCL
`minVolumeCBM`	number	Min volume in CBM	1
`maxVolumeCBM`	number	Max volume in CBM	100
`minWeightKG`	number	Min weight in kg	100
`maxWeightKG`	number	Max weight in kg	15000
`palletCount`	number	Pallet count (0=any)	10
`pricePerCBM`	number	Price per cubic meter	45.50
`pricePerKG`	number	Price per kilogram	2.80
`basePriceUSD`	number	Base price in USD	1500
`basePriceEUR`	number	Base price in EUR	1350
`currency`	string	Primary currency	USD
`hasSurcharges`	boolean	Has surcharges?	true
`surchargeBAF`	number	BAF surcharge (optional)	150
`surchargeCAF`	number	CAF surcharge (optional)	75
`surchargeDetails`	string	Surcharge details (optional)	BAF+CAF included
`transitDays`	number	Transit time in days	28
`validFrom`	date	Start date (YYYY-MM-DD)	2025-01-01
`validUntil`	date	End date (YYYY-MM-DD)	2025-12-31

Price Calculation Logic

// Freight class rule: take the higher of volume-based or weight-based price
const volumePrice = volumeCBM * pricePerCBM;
const weightPrice = weightKG * pricePerKG;
const freightPrice = Math.max(volumePrice, weightPrice);

// Add surcharges if present
const totalPrice = freightPrice + (hasSurcharges ? (surchargeBAF + surchargeCAF) : 0);

Domain Entities

CsvRate Entity

Main domain entity representing a CSV-loaded rate:

class CsvRate {
  constructor(
    companyName: string,
    origin: PortCode,
    destination: PortCode,
    containerType: ContainerType,
    volumeRange: VolumeRange,
    weightRange: WeightRange,
    palletCount: number,
    pricing: RatePricing,
    currency: string,
    surcharges: SurchargeCollection,
    transitDays: number,
    validity: DateRange,
  )

  // Key methods
  calculatePrice(volume: Volume): Money
  getPriceInCurrency(volume: Volume, targetCurrency: 'USD' | 'EUR'): Money
  isValidForDate(date: Date): boolean
  matchesVolume(volume: Volume): boolean
  matchesPalletCount(palletCount: number): boolean
  matchesRoute(origin: PortCode, destination: PortCode): boolean
}

Value Objects

Volume: Represents shipping volume in CBM and weight in KG

class Volume {
  constructor(cbm: number, weightKG: number)
  calculateFreightPrice(pricePerCBM: number, pricePerKG: number): number
  isWithinRange(minCBM, maxCBM, minKG, maxKG): boolean
}

Surcharge: Represents additional fees

class Surcharge {
  constructor(
    type: SurchargeType, // BAF, CAF, PSS, THC, OTHER
    amount: Money,
    description?: string
  )
}

class SurchargeCollection {
  getTotalAmount(currency: string): Money
  isEmpty(): boolean
  getDetails(): string
}

Database Schema

csv_rate_configs Table

CREATE TABLE csv_rate_configs (
  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  company_name VARCHAR(255) NOT NULL UNIQUE,
  csv_file_path VARCHAR(500) NOT NULL,
  type VARCHAR(50) NOT NULL DEFAULT 'CSV_ONLY', -- CSV_ONLY | CSV_AND_API
  has_api BOOLEAN NOT NULL DEFAULT FALSE,
  api_connector VARCHAR(100) NULL,
  is_active BOOLEAN NOT NULL DEFAULT TRUE,
  uploaded_at TIMESTAMP NOT NULL DEFAULT NOW(),
  uploaded_by UUID NULL REFERENCES users(id) ON DELETE SET NULL,
  last_validated_at TIMESTAMP NULL,
  row_count INTEGER NULL,
  metadata JSONB NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Seeded Data

company_name	csv_file_path	type	has_api	api_connector
SSC Consolidation	ssc-consolidation.csv	CSV_ONLY	false	null
ECU Worldwide	ecu-worldwide.csv	CSV_AND_API	true	ecu-worldwide
TCC Logistics	tcc-logistics.csv	CSV_ONLY	false	null
NVO Consolidation	nvo-consolidation.csv	CSV_ONLY	false	null

API Research Results

✅ ECU Worldwide - API Available

API Portal: https://api-portal.ecuworldwide.com/

Features:

REST API with JSON responses
Rate quotes (door-to-door, port-to-port)
Shipment booking (create/update/cancel)
Tracking and visibility
Sandbox and production environments
API key authentication

Integration Status: Ready for connector implementation

❌ Other Carriers - No Public APIs

SSC Consolidation: No public API found
TCC Logistics: No public API found
NVO Consolidation: No public API found (uses project44 for tracking only)

All three will use CSV_ONLY integration.

Advanced Filters

RateSearchFilters Interface

interface RateSearchFilters {
  // Company filters
  companies?: string[];

  // Volume/Weight filters
  minVolumeCBM?: number;
  maxVolumeCBM?: number;
  minWeightKG?: number;
  maxWeightKG?: number;
  palletCount?: number;

  // Price filters
  minPrice?: number;
  maxPrice?: number;
  currency?: 'USD' | 'EUR';

  // Transit filters
  minTransitDays?: number;
  maxTransitDays?: number;

  // Container type filters
  containerTypes?: string[];

  // Surcharge filters
  onlyAllInPrices?: boolean; // Only show rates without separate surcharges

  // Date filters
  departureDate?: Date;
}

Usage Examples

1. Load Rates from CSV

const loader = new CsvRateLoaderAdapter();
const rates = await loader.loadRatesFromCsv('ssc-consolidation.csv');
console.log(`Loaded ${rates.length} rates`);

2. Search Rates with Filters

const searchService = new CsvRateSearchService(csvRateLoader);

const result = await searchService.execute({
  origin: 'NLRTM',
  destination: 'USNYC',
  volumeCBM: 25.5,
  weightKG: 3500,
  palletCount: 10,
  filters: {
    companies: ['SSC Consolidation', 'ECU Worldwide'],
    minPrice: 1000,
    maxPrice: 3000,
    currency: 'USD',
    onlyAllInPrices: true,
  },
});

console.log(`Found ${result.totalResults} matching rates`);
result.results.forEach(r => {
  console.log(`${r.rate.companyName}: $${r.calculatedPrice.usd}`);
});

3. Calculate Price for Specific Volume

const volume = new Volume(25.5, 3500); // 25.5 CBM, 3500 kg
const price = csvRate.calculatePrice(volume);
console.log(`Total price: ${price.format()}`); // $1,850.00

Next Steps (TODO)

Backend (Application Layer)

DTOs - Create data transfer objects:
Controllers:
- Update RatesController with /search endpoint supporting advanced filters
- Create CsvRatesController (admin only) for CSV upload
- Add /api/v1/rates/companies endpoint
- Add /api/v1/rates/filters/options endpoint
Repository:
- Create TypeOrmCsvRateConfigRepository
- Implement CRUD operations for csv_rate_configs table
Module Configuration:
- Register CsvRateLoaderAdapter as provider
- Register CsvRateSearchService as provider
- Add to CarrierModule or create new CsvRateModule

Backend (ECU Worldwide API Connector)

ECU Connector (if time permits):
- Create infrastructure/carriers/ecu-worldwide/
- Implement ecu-worldwide.connector.ts
- Add ecu-worldwide.mapper.ts
- Add ecu-worldwide.types.ts
- Environment variables: ECU_WORLDWIDE_API_KEY, ECU_WORLDWIDE_API_URL

Frontend

Components:
- RateFiltersPanel.tsx - Advanced filters sidebar
- VolumeWeightInput.tsx - CBM + weight input
- CompanyMultiSelect.tsx - Multi-select for companies
- RateResultsTable.tsx - Display results with source (CSV/API)
- CsvUpload.tsx - Admin CSV upload (protected route)
Hooks:
- useRateSearch.ts - Search with filters
- useCompanies.ts - Get available companies
- useFilterOptions.ts - Get filter options
API Client:
- Update lib/api/rates.ts with new endpoints
- Create lib/api/admin/csv-rates.ts

Testing

Unit Tests (Target: 90%+ coverage):
- csv-rate.entity.spec.ts
- volume.vo.spec.ts
- surcharge.vo.spec.ts
- csv-rate-search.service.spec.ts
Integration Tests:
- csv-rate-loader.adapter.spec.ts
- CSV file validation tests
- Price calculation tests

Documentation

Update CLAUDE.md:
- Add CSV Rate System section
- Document new endpoints
- Add environment variables

Running Migrations

cd apps/backend
npm run migration:run

This will create the csv_rate_configs table and seed the 4 carriers.

Validation

To validate a CSV file:

const loader = new CsvRateLoaderAdapter();
const result = await loader.validateCsvFile('ssc-consolidation.csv');

if (!result.valid) {
  console.error('Validation errors:', result.errors);
} else {
  console.log(`Valid CSV with ${result.rowCount} rows`);
}

Security

✅ CSV upload endpoint protected by @Roles('ADMIN') guard
✅ File validation: size, extension, structure
✅ Sanitization of CSV data before parsing
✅ Path traversal prevention (only access rates directory)

Performance

✅ Redis caching (15min TTL) for loaded CSV rates
✅ Batch loading of all CSV files in parallel
✅ Efficient filtering with early returns
✅ Match scoring for result relevance

Deployment Checklist

Run database migration
Upload CSV files to infrastructure/storage/csv-storage/rates/
Set file permissions (readable by app user)
Configure Redis for caching
Test CSV loading on server
Verify admin CSV upload endpoint
Monitor CSV file sizes (keep under 10MB each)

Maintenance

Adding a New Carrier

Create CSV file: carrier-name.csv
Add entry to csv_rate_configs table

Upload via admin interface OR run SQL:

INSERT INTO csv_rate_configs (company_name, csv_file_path, type, has_api)
VALUES ('New Carrier', 'new-carrier.csv', 'CSV_ONLY', false);

Updating Rates

Admin uploads new CSV via /api/v1/admin/csv-rates/upload
System validates structure
Old file replaced, cache cleared
New rates immediately available

Support

For questions or issues:

Check CARRIER_API_RESEARCH.md for API details
Review CLAUDE.md for system architecture
See domain tests for usage examples

13 KiB Raw Blame History