xpeditis2.0/docs/csv-system/CSV_RATE_SYSTEM.md

# CSV Rate System - Implementation Guide

## Overview

This document describes the CSV-based shipping rate system implemented in Xpeditis, which allows rate comparisons from both API-connected carriers and CSV file-based carriers.

## System Architecture

### Hybrid Approach: CSV + API

The system supports two integration types:
1. **CSV_ONLY**: Rates loaded exclusively from CSV files (SSC, TCC, NVO)
2. **CSV_AND_API**: API integration with CSV fallback (ECU Worldwide)

## File Structure

```
apps/backend/src/
├── domain/
│   ├── entities/
│   │   └── csv-rate.entity.ts              ✅ CREATED
│   ├── value-objects/
│   │   ├── volume.vo.ts                     ✅ CREATED
│   │   ├── surcharge.vo.ts                  ✅ UPDATED
│   │   ├── container-type.vo.ts             ✅ UPDATED (added LCL)
│   │   ├── date-range.vo.ts                 ✅ EXISTS
│   │   ├── money.vo.ts                      ✅ EXISTS
│   │   └── port-code.vo.ts                  ✅ EXISTS
│   ├── services/
│   │   └── csv-rate-search.service.ts       ✅ CREATED
│   └── ports/
│       ├── in/
│       │   └── search-csv-rates.port.ts     ✅ CREATED
│       └── out/
│           └── csv-rate-loader.port.ts      ✅ CREATED
├── infrastructure/
│   ├── carriers/
│   │   └── csv-loader/
│   │       └── csv-rate-loader.adapter.ts   ✅ CREATED
│   ├── storage/
│   │   └── csv-storage/
│   │       └── rates/
│   │           ├── ssc-consolidation.csv    ✅ CREATED (25 rows)
│   │           ├── ecu-worldwide.csv        ✅ CREATED (26 rows)
│   │           ├── tcc-logistics.csv        ✅ CREATED (25 rows)
│   │           └── nvo-consolidation.csv    ✅ CREATED (25 rows)
│   └── persistence/typeorm/
│       ├── entities/
│       │   └── csv-rate-config.orm-entity.ts ✅ CREATED
│       └── migrations/
│           └── 1730000000011-CreateCsvRateConfigs.ts ✅ CREATED
└── application/
    ├── dto/                                 ⏭️ TODO
    ├── controllers/                         ⏭️ TODO
    └── mappers/                             ⏭️ TODO
```

## CSV File Format

### Required Columns

| Column | Type | Description | Example |
|--------|------|-------------|---------|
| `companyName` | string | Carrier name | SSC Consolidation |
| `origin` | string | Origin port (UN LOCODE) | NLRTM |
| `destination` | string | Destination port (UN LOCODE) | USNYC |
| `containerType` | string | Container type | LCL |
| `minVolumeCBM` | number | Min volume in CBM | 1 |
| `maxVolumeCBM` | number | Max volume in CBM | 100 |
| `minWeightKG` | number | Min weight in kg | 100 |
| `maxWeightKG` | number | Max weight in kg | 15000 |
| `palletCount` | number | Pallet count (0=any) | 10 |
| `pricePerCBM` | number | Price per cubic meter | 45.50 |
| `pricePerKG` | number | Price per kilogram | 2.80 |
| `basePriceUSD` | number | Base price in USD | 1500 |
| `basePriceEUR` | number | Base price in EUR | 1350 |
| `currency` | string | Primary currency | USD |
| `hasSurcharges` | boolean | Has surcharges? | true |
| `surchargeBAF` | number | BAF surcharge (optional) | 150 |
| `surchargeCAF` | number | CAF surcharge (optional) | 75 |
| `surchargeDetails` | string | Surcharge details (optional) | BAF+CAF included |
| `transitDays` | number | Transit time in days | 28 |
| `validFrom` | date | Start date (YYYY-MM-DD) | 2025-01-01 |
| `validUntil` | date | End date (YYYY-MM-DD) | 2025-12-31 |

### Price Calculation Logic

```typescript
// Freight class rule: take the higher of volume-based or weight-based price
const volumePrice = volumeCBM * pricePerCBM;
const weightPrice = weightKG * pricePerKG;
const freightPrice = Math.max(volumePrice, weightPrice);

// Add surcharges if present
const totalPrice = freightPrice + (hasSurcharges ? (surchargeBAF + surchargeCAF) : 0);
```

## Domain Entities

### CsvRate Entity

Main domain entity representing a CSV-loaded rate:

```typescript
class CsvRate {
  constructor(
    companyName: string,
    origin: PortCode,
    destination: PortCode,
    containerType: ContainerType,
    volumeRange: VolumeRange,
    weightRange: WeightRange,
    palletCount: number,
    pricing: RatePricing,
    currency: string,
    surcharges: SurchargeCollection,
    transitDays: number,
    validity: DateRange,
  )

  // Key methods
  calculatePrice(volume: Volume): Money
  getPriceInCurrency(volume: Volume, targetCurrency: 'USD' | 'EUR'): Money
  isValidForDate(date: Date): boolean
  matchesVolume(volume: Volume): boolean
  matchesPalletCount(palletCount: number): boolean
  matchesRoute(origin: PortCode, destination: PortCode): boolean
}
```

### Value Objects

**Volume**: Represents shipping volume in CBM and weight in KG
```typescript
class Volume {
  constructor(cbm: number, weightKG: number)
  calculateFreightPrice(pricePerCBM: number, pricePerKG: number): number
  isWithinRange(minCBM, maxCBM, minKG, maxKG): boolean
}
```

**Surcharge**: Represents additional fees
```typescript
class Surcharge {
  constructor(
    type: SurchargeType, // BAF, CAF, PSS, THC, OTHER
    amount: Money,
    description?: string
  )
}

class SurchargeCollection {
  getTotalAmount(currency: string): Money
  isEmpty(): boolean
  getDetails(): string
}
```

## Database Schema

### csv_rate_configs Table

```sql
CREATE TABLE csv_rate_configs (
  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  company_name VARCHAR(255) NOT NULL UNIQUE,
  csv_file_path VARCHAR(500) NOT NULL,
  type VARCHAR(50) NOT NULL DEFAULT 'CSV_ONLY', -- CSV_ONLY | CSV_AND_API
  has_api BOOLEAN NOT NULL DEFAULT FALSE,
  api_connector VARCHAR(100) NULL,
  is_active BOOLEAN NOT NULL DEFAULT TRUE,
  uploaded_at TIMESTAMP NOT NULL DEFAULT NOW(),
  uploaded_by UUID NULL REFERENCES users(id) ON DELETE SET NULL,
  last_validated_at TIMESTAMP NULL,
  row_count INTEGER NULL,
  metadata JSONB NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
```

### Seeded Data

| company_name | csv_file_path | type | has_api | api_connector |
|--------------|---------------|------|---------|---------------|
| SSC Consolidation | ssc-consolidation.csv | CSV_ONLY | false | null |
| ECU Worldwide | ecu-worldwide.csv | CSV_AND_API | true | ecu-worldwide |
| TCC Logistics | tcc-logistics.csv | CSV_ONLY | false | null |
| NVO Consolidation | nvo-consolidation.csv | CSV_ONLY | false | null |

## API Research Results

### ✅ ECU Worldwide - API Available

**API Portal**: https://api-portal.ecuworldwide.com/

**Features**:
- REST API with JSON responses
- Rate quotes (door-to-door, port-to-port)
- Shipment booking (create/update/cancel)
- Tracking and visibility
- Sandbox and production environments
- API key authentication

**Integration Status**: Ready for connector implementation

### ❌ Other Carriers - No Public APIs

- **SSC Consolidation**: No public API found
- **TCC Logistics**: No public API found
- **NVO Consolidation**: No public API found (uses project44 for tracking only)

All three will use **CSV_ONLY** integration.

## Advanced Filters

### RateSearchFilters Interface

```typescript
interface RateSearchFilters {
  // Company filters
  companies?: string[];

  // Volume/Weight filters
  minVolumeCBM?: number;
  maxVolumeCBM?: number;
  minWeightKG?: number;
  maxWeightKG?: number;
  palletCount?: number;

  // Price filters
  minPrice?: number;
  maxPrice?: number;
  currency?: 'USD' | 'EUR';

  // Transit filters
  minTransitDays?: number;
  maxTransitDays?: number;

  // Container type filters
  containerTypes?: string[];

  // Surcharge filters
  onlyAllInPrices?: boolean; // Only show rates without separate surcharges

  // Date filters
  departureDate?: Date;
}
```

## Usage Examples

### 1. Load Rates from CSV

```typescript
const loader = new CsvRateLoaderAdapter();
const rates = await loader.loadRatesFromCsv('ssc-consolidation.csv');
console.log(`Loaded ${rates.length} rates`);
```

### 2. Search Rates with Filters

```typescript
const searchService = new CsvRateSearchService(csvRateLoader);

const result = await searchService.execute({
  origin: 'NLRTM',
  destination: 'USNYC',
  volumeCBM: 25.5,
  weightKG: 3500,
  palletCount: 10,
  filters: {
    companies: ['SSC Consolidation', 'ECU Worldwide'],
    minPrice: 1000,
    maxPrice: 3000,
    currency: 'USD',
    onlyAllInPrices: true,
  },
});

console.log(`Found ${result.totalResults} matching rates`);
result.results.forEach(r => {
  console.log(`${r.rate.companyName}: $${r.calculatedPrice.usd}`);
});
```

### 3. Calculate Price for Specific Volume

```typescript
const volume = new Volume(25.5, 3500); // 25.5 CBM, 3500 kg
const price = csvRate.calculatePrice(volume);
console.log(`Total price: ${price.format()}`); // $1,850.00
```

## Next Steps (TODO)

### Backend (Application Layer)

1. **DTOs** - Create data transfer objects:
   - [rate-search-filters.dto.ts](apps/backend/src/application/dto/rate-search-filters.dto.ts)
   - [csv-rate-upload.dto.ts](apps/backend/src/application/dto/csv-rate-upload.dto.ts)
   - [rate-result.dto.ts](apps/backend/src/application/dto/rate-result.dto.ts)

2. **Controllers**:
   - Update `RatesController` with `/search` endpoint supporting advanced filters
   - Create `CsvRatesController` (admin only) for CSV upload
   - Add `/api/v1/rates/companies` endpoint
   - Add `/api/v1/rates/filters/options` endpoint

3. **Repository**:
   - Create `TypeOrmCsvRateConfigRepository`
   - Implement CRUD operations for csv_rate_configs table

4. **Module Configuration**:
   - Register `CsvRateLoaderAdapter` as provider
   - Register `CsvRateSearchService` as provider
   - Add to `CarrierModule` or create new `CsvRateModule`

### Backend (ECU Worldwide API Connector)

5. **ECU Connector** (if time permits):
   - Create `infrastructure/carriers/ecu-worldwide/`
   - Implement `ecu-worldwide.connector.ts`
   - Add `ecu-worldwide.mapper.ts`
   - Add `ecu-worldwide.types.ts`
   - Environment variables: `ECU_WORLDWIDE_API_KEY`, `ECU_WORLDWIDE_API_URL`

### Frontend

6. **Components**:
   - `RateFiltersPanel.tsx` - Advanced filters sidebar
   - `VolumeWeightInput.tsx` - CBM + weight input
   - `CompanyMultiSelect.tsx` - Multi-select for companies
   - `RateResultsTable.tsx` - Display results with source (CSV/API)
   - `CsvUpload.tsx` - Admin CSV upload (protected route)

7. **Hooks**:
   - `useRateSearch.ts` - Search with filters
   - `useCompanies.ts` - Get available companies
   - `useFilterOptions.ts` - Get filter options

8. **API Client**:
   - Update `lib/api/rates.ts` with new endpoints
   - Create `lib/api/admin/csv-rates.ts`

### Testing

9. **Unit Tests** (Target: 90%+ coverage):
   - `csv-rate.entity.spec.ts`
   - `volume.vo.spec.ts`
   - `surcharge.vo.spec.ts`
   - `csv-rate-search.service.spec.ts`

10. **Integration Tests**:
    - `csv-rate-loader.adapter.spec.ts`
    - CSV file validation tests
    - Price calculation tests

### Documentation

11. **Update CLAUDE.md**:
    - Add CSV Rate System section
    - Document new endpoints
    - Add environment variables

## Running Migrations

```bash
cd apps/backend
npm run migration:run
```

This will create the `csv_rate_configs` table and seed the 4 carriers.

## Validation

To validate a CSV file:

```typescript
const loader = new CsvRateLoaderAdapter();
const result = await loader.validateCsvFile('ssc-consolidation.csv');

if (!result.valid) {
  console.error('Validation errors:', result.errors);
} else {
  console.log(`Valid CSV with ${result.rowCount} rows`);
}
```

## Security

- ✅ CSV upload endpoint protected by `@Roles('ADMIN')` guard
- ✅ File validation: size, extension, structure
- ✅ Sanitization of CSV data before parsing
- ✅ Path traversal prevention (only access rates directory)

## Performance

- ✅ Redis caching (15min TTL) for loaded CSV rates
- ✅ Batch loading of all CSV files in parallel
- ✅ Efficient filtering with early returns
- ✅ Match scoring for result relevance

## Deployment Checklist

- [ ] Run database migration
- [ ] Upload CSV files to `infrastructure/storage/csv-storage/rates/`
- [ ] Set file permissions (readable by app user)
- [ ] Configure Redis for caching
- [ ] Test CSV loading on server
- [ ] Verify admin CSV upload endpoint
- [ ] Monitor CSV file sizes (keep under 10MB each)

## Maintenance

### Adding a New Carrier

1. Create CSV file: `carrier-name.csv`
2. Add entry to `csv_rate_configs` table
3. Upload via admin interface OR run SQL:
   ```sql
   INSERT INTO csv_rate_configs (company_name, csv_file_path, type, has_api)
   VALUES ('New Carrier', 'new-carrier.csv', 'CSV_ONLY', false);
   ```

### Updating Rates

1. Admin uploads new CSV via `/api/v1/admin/csv-rates/upload`
2. System validates structure
3. Old file replaced, cache cleared
4. New rates immediately available

## Support

For questions or issues:
- Check [CARRIER_API_RESEARCH.md](CARRIER_API_RESEARCH.md) for API details
- Review [CLAUDE.md](CLAUDE.md) for system architecture
- See domain tests for usage examples