harden media

This commit is contained in:
2025-12-31 15:39:28 +01:00
parent 92226800df
commit 8260bf7baf
413 changed files with 8361 additions and 1544 deletions

178
scripts/MIGRATION_GUIDE.md Normal file
View File

@@ -0,0 +1,178 @@
# Media Reference Migration Guide
This guide explains how to migrate existing seeded data from old URL formats to the new `MediaReference` format.
## Problem
Old seeded data stores media references as URL strings:
- `/api/avatar/{driverId}`
- `/api/media/teams/{teamId}/logo`
- `/api/media/leagues/{leagueId}/logo`
New format uses `MediaReference` objects:
```json
{
"type": "system-default",
"variant": "avatar",
"avatarVariant": "male"
}
```
## Solutions
### Option 1: Migration Script (Preserve Data)
**Best for:** Production databases or when you need to preserve existing data
```bash
# Test what would change (dry run)
npm run migrate:media:test
# Execute the migration
npm run migrate:media:exec
```
**What it does:**
- Converts `/api/avatar/{id}``system-default` with deterministic variant
- Converts `/api/media/teams/{id}/logo``generated`
- Converts `/api/media/leagues/{id}/logo``generated`
- Handles unknown formats → `none`
- Skips already-migrated entries
**Environment variables:**
- `GRIDPILOT_API_PERSISTENCE=postgres|inmemory` (default: postgres)
- `DATABASE_URL` (required for postgres)
### Option 2: Wipe and Reseed (Clean Slate)
**Best for:** Development/testing when you don't care about existing data
```bash
# Stop services and remove all volumes
npm run docker:dev:clean
# Rebuild and start fresh
npm run docker:dev:build
```
**What it does:**
- Deletes all existing data
- Runs fresh seed with correct `MediaReference` format
- No migration needed
## Migration Script Details
### Supported Old Formats
| Old Format | New Reference | Example |
|------------|---------------|---------|
| `/api/avatar/{id}` | `system-default` (deterministic) | `/api/avatar/driver-1``male`/`female`/`neutral` |
| `/api/media/teams/{id}/logo` | `generated` | `/api/media/teams/team-1/logo``generated:team-team-1` |
| `/api/media/leagues/{id}/logo` | `generated` | `/api/media/leagues/league-1/logo``generated:league-league-1` |
| `/images/avatars/male-default-avatar.jpg` | `system-default` (male) | Static files → `system-default` |
| `https://external.com/...` | `none` | External URLs → `none` |
| Empty/null | `none` | Missing values → `none` |
### Deterministic Avatar Selection
Driver avatars use a hash-based selection for consistency:
```typescript
const hash = hashCode(driverId);
const variantIndex = Math.abs(hash) % 3;
// 0 → male, 1 → female, 2 → neutral
```
This ensures the same driver ID always gets the same avatar variant.
### What Gets Updated
**Driver entities:**
- `avatarRef` field (JSONB column)
**Team entities:**
- `logoRef` field (JSONB column)
**League entities:**
- `logoRef` field (JSONB column)
### Safety Features
1. **Dry Run Mode:** Default behavior shows changes without applying them
2. **Validation:** Only updates entries with invalid or missing references
3. **Error Handling:** Continues on individual errors, reports all failures
4. **Idempotent:** Safe to run multiple times
## Testing the Migration
Test the migration logic without touching real data:
```bash
# Run test script
npm run migrate:media:test
```
This will show you:
- How each URL format is parsed
- What MediaReference it becomes
- Deterministic avatar variants for sample IDs
## When to Use Each Option
### Use Migration Script When:
- ✅ You have production data to preserve
- ✅ You want to see what changes will be made
- ✅ You need a controlled, reversible process
- ✅ You're migrating a live database
### Use Wipe and Reseed When:
- ✅ You're in development/testing
- ✅ You don't care about existing data
- ✅ You want the fastest path to a clean state
- ✅ You're setting up a new environment
## Troubleshooting
### Migration fails with "DATABASE_URL required"
Set the environment variable:
```bash
export DATABASE_URL=postgresql://user:pass@localhost:5432/dbname
```
### Some entries weren't migrated
Check the error output. Common issues:
- Invalid URL format (will be converted to `none`)
- Already valid MediaReference (skipped)
- Database connection issues
### Want to rollback
The migration only updates entries that need it. To rollback:
1. Restore from database backup
2. Or manually revert the `avatarRef`/`logoRef` fields
## Example Migration Output
```
[INFO] Starting media reference migration in DRY RUN mode
[INFO] Persistence mode: postgres
[INFO] Connecting to PostgreSQL database...
[INFO] Database connection established
[INFO] Found 150 drivers to migrate
[INFO] Found 25 teams to migrate
[INFO] Found 5 leagues to migrate
[INFO] Migration completed: 150 drivers, 25 teams, 5 leagues updated
=== Migration Summary ===
Mode: DRY RUN
Processed: 150 drivers, 25 teams, 5 leagues
Updated: 150 drivers, 25 teams, 5 leagues
✅ Dry run completed successfully. Run with --execute to apply changes.
```
## Next Steps
After migration:
1. Verify data integrity in the database
2. Test that avatars and logos render correctly
3. Update any hardcoded URL references in the frontend
4. Remove any legacy URL construction code

View File

@@ -0,0 +1,571 @@
#!/usr/bin/env ts-node
/**
* Migration Script: Convert Old Media URLs to MediaReferences
*
* This script migrates existing seeded data from old stored route strings/URLs
* to the new MediaReference format. It handles:
*
* - Driver avatars: /api/avatar/{id} -> system-default (deterministic variant)
* - Team logos: /api/media/teams/{id}/logo -> generated
* - League logos: /api/media/leagues/{id}/logo -> generated
* - Other old formats -> none
*
* Usage:
* # Test mode (dry run, no changes)
* npm run migrate:media:test
*
* # Execute migration
* npm run migrate:media:exec
*
* Environment:
* - GRIDPILOT_API_PERSISTENCE=postgres|inmemory (default: postgres)
* - DATABASE_URL (for postgres mode)
*/
import { DataSource } from 'typeorm';
import { Logger } from '@core/shared/application';
import { MediaReference } from '@core/domain/media/MediaReference';
// Import entities
import { DriverOrmEntity } from '../adapters/racing/persistence/typeorm/entities/DriverOrmEntity';
import { TeamOrmEntity } from '../adapters/racing/persistence/typeorm/entities/TeamOrmEntities';
import { LeagueOrmEntity } from '../adapters/racing/persistence/typeorm/entities/LeagueOrmEntity';
// Import in-memory repositories for testing
import { InMemoryDriverRepository } from '../adapters/racing/persistence/inmemory/InMemoryDriverRepository';
import { InMemoryTeamRepository } from '../adapters/racing/persistence/inmemory/InMemoryTeamRepository';
import { InMemoryLeagueRepository } from '../adapters/racing/persistence/inmemory/InMemoryLeagueRepository';
import { InMemoryMediaRepository } from '../adapters/racing/persistence/media/InMemoryMediaRepository';
interface MigrationResult {
success: boolean;
processed: {
drivers: number;
teams: number;
leagues: number;
};
updated: {
drivers: number;
teams: number;
leagues: number;
};
errors: string[];
}
class MediaMigrationLogger implements Logger {
info(message: string): void {
console.log(`[INFO] ${message}`);
}
warn(message: string): void {
console.warn(`[WARN] ${message}`);
}
error(message: string, trace?: string): void {
console.error(`[ERROR] ${message}`, trace || '');
}
debug(message: string): void {
console.debug(`[DEBUG] ${message}`);
}
}
export class MediaReferenceMigration {
private logger: Logger;
constructor(private readonly dryRun: boolean = true) {
this.logger = new MediaMigrationLogger();
}
/**
* Parse old URL format and determine appropriate MediaReference
*/
private parseOldUrl(url: string | null | undefined): MediaReference | null {
if (!url || typeof url !== 'string') {
return null;
}
const trimmed = url.trim();
if (trimmed === '') {
return null;
}
// Pattern: /api/avatar/{driverId}
const avatarMatch = trimmed.match(/^\/api\/avatar\/([a-zA-Z0-9-]+)$/);
if (avatarMatch) {
const driverId = avatarMatch[1];
const variant = this.getDeterministicAvatarVariant(driverId);
return MediaReference.systemDefault(variant);
}
// Pattern: /api/media/teams/{teamId}/logo
const teamLogoMatch = trimmed.match(/^\/api\/media\/teams\/([a-zA-Z0-9-]+)\/logo$/);
if (teamLogoMatch) {
const teamId = teamLogoMatch[1];
return MediaReference.generated('team', teamId);
}
// Pattern: /api/media/leagues/{leagueId}/logo
const leagueLogoMatch = trimmed.match(/^\/api\/media\/leagues\/([a-zA-Z0-9-]+)\/logo$/);
if (leagueLogoMatch) {
const leagueId = leagueLogoMatch[1];
return MediaReference.generated('league', leagueId);
}
// Pattern: /api/media/teams/{teamId}/logo (alternative format)
const teamLogoAltMatch = trimmed.match(/^\/api\/teams\/([a-zA-Z0-9-]+)\/logo$/);
if (teamLogoAltMatch) {
const teamId = teamLogoAltMatch[1];
return MediaReference.generated('team', teamId);
}
// Pattern: Static file paths (old format)
if (trimmed.includes('/images/avatars/')) {
// Old static files - convert to system default
if (trimmed.includes('male')) return MediaReference.systemDefault('male');
if (trimmed.includes('female')) return MediaReference.systemDefault('female');
if (trimmed.includes('neutral')) return MediaReference.systemDefault('neutral');
return MediaReference.systemDefault('avatar');
}
// Pattern: Full URLs (external or old API)
if (trimmed.startsWith('http://') || trimmed.startsWith('https://')) {
// External URLs - can't migrate automatically, mark as none
this.logger.warn(`External URL found: ${trimmed} - converting to none`);
return MediaReference.createNone();
}
// Unknown format - convert to none
this.logger.warn(`Unknown URL format: ${trimmed} - converting to none`);
return MediaReference.createNone();
}
/**
* Deterministic avatar variant selection based on driver ID
* Uses hash % 3 to ensure consistency
*/
private getDeterministicAvatarVariant(driverId: string): 'male' | 'female' | 'neutral' {
const hash = this.hashCode(driverId);
const variantIndex = Math.abs(hash) % 3;
switch (variantIndex) {
case 0: return 'male';
case 1: return 'female';
case 2: return 'neutral';
default: return 'neutral';
}
}
/**
* Simple hash function for deterministic selection
*/
private hashCode(str: string): number {
let hash = 0;
for (let i = 0; i < str.length; i++) {
const char = str.charCodeAt(i);
hash = ((hash << 5) - hash) + char;
hash = hash & hash;
}
return hash;
}
/**
* Migrate PostgreSQL database
*/
async migratePostgres(): Promise<MigrationResult> {
const result: MigrationResult = {
success: false,
processed: { drivers: 0, teams: 0, leagues: 0 },
updated: { drivers: 0, teams: 0, leagues: 0 },
errors: []
};
try {
const databaseUrl = process.env.DATABASE_URL;
if (!databaseUrl) {
throw new Error('DATABASE_URL environment variable is required for postgres migration');
}
this.logger.info('Connecting to PostgreSQL database...');
const dataSource = new DataSource({
type: 'postgres',
url: databaseUrl,
entities: [DriverOrmEntity, TeamOrmEntity, LeagueOrmEntity],
synchronize: false, // Don't auto-create tables
logging: false
});
await dataSource.initialize();
this.logger.info('Database connection established');
// Migrate Drivers
const driverRepo = dataSource.getRepository(DriverOrmEntity);
const drivers = await driverRepo.find();
result.processed.drivers = drivers.length;
this.logger.info(`Found ${drivers.length} drivers to migrate`);
for (const driver of drivers) {
try {
// Check if avatarRef already exists and is valid
if (driver.avatarRef && typeof driver.avatarRef === 'object') {
try {
const existing = MediaReference.fromJSON(driver.avatarRef as any);
// Valid reference, skip
continue;
} catch {
// Invalid reference, proceed with migration
}
}
// Get old URL from avatarRef if it's a string, or check if we need to parse
let oldUrl: string | null = null;
if (driver.avatarRef && typeof driver.avatarRef === 'string') {
oldUrl = driver.avatarRef;
} else if (driver.avatarRef && typeof driver.avatarRef === 'object') {
// Try to extract URL from old object format
const refAny = driver.avatarRef as any;
oldUrl = refAny.url || refAny.avatarUrl || null;
}
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
if (!this.dryRun) {
await driverRepo.update(driver.id, {
avatarRef: newRef.toJSON() as any
});
result.updated.drivers++;
} else {
result.updated.drivers++; // Count as would-be update
}
}
}
} catch (error) {
result.errors.push(`Driver ${driver.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
// Migrate Teams
const teamRepo = dataSource.getRepository(TeamOrmEntity);
const teams = await teamRepo.find();
result.processed.teams = teams.length;
this.logger.info(`Found ${teams.length} teams to migrate`);
for (const team of teams) {
try {
// Check if logoRef already exists and is valid
if (team.logoRef && typeof team.logoRef === 'object') {
try {
const existing = MediaReference.fromJSON(team.logoRef as any);
// Valid reference, skip
continue;
} catch {
// Invalid reference, proceed with migration
}
}
let oldUrl: string | null = null;
if (team.logoRef && typeof team.logoRef === 'string') {
oldUrl = team.logoRef;
} else if (team.logoRef && typeof team.logoRef === 'object') {
const refAny = team.logoRef as any;
oldUrl = refAny.url || refAny.logoUrl || null;
}
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
if (!this.dryRun) {
await teamRepo.update(team.id, {
logoRef: newRef.toJSON() as any
});
result.updated.teams++;
} else {
result.updated.teams++;
}
}
}
} catch (error) {
result.errors.push(`Team ${team.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
// Migrate Leagues
const leagueRepo = dataSource.getRepository(LeagueOrmEntity);
const leagues = await leagueRepo.find();
result.processed.leagues = leagues.length;
this.logger.info(`Found ${leagues.length} leagues to migrate`);
for (const league of leagues) {
try {
// Check if logoRef already exists and is valid
if (league.logoRef && typeof league.logoRef === 'object') {
try {
const existing = MediaReference.fromJSON(league.logoRef as any);
// Valid reference, skip
continue;
} catch {
// Invalid reference, proceed with migration
}
}
let oldUrl: string | null = null;
if (league.logoRef && typeof league.logoRef === 'string') {
oldUrl = league.logoRef;
} else if (league.logoRef && typeof league.logoRef === 'object') {
const refAny = league.logoRef as any;
oldUrl = refAny.url || refAny.logoUrl || null;
}
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
if (!this.dryRun) {
await leagueRepo.update(league.id, {
logoRef: newRef.toJSON() as any
});
result.updated.leagues++;
} else {
result.updated.leagues++;
}
}
}
} catch (error) {
result.errors.push(`League ${league.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
await dataSource.destroy();
result.success = true;
this.logger.info(`Migration completed: ${result.updated.drivers} drivers, ${result.updated.teams} teams, ${result.updated.leagues} leagues updated`);
return result;
} catch (error) {
result.errors.push(error instanceof Error ? error.message : String(error));
this.logger.error('Migration failed', error instanceof Error ? error.stack : undefined);
return result;
}
}
/**
* Migrate in-memory repositories (for testing)
*/
async migrateInMemory(): Promise<MigrationResult> {
const result: MigrationResult = {
success: false,
processed: { drivers: 0, teams: 0, leagues: 0 },
updated: { drivers: 0, teams: 0, leagues: 0 },
errors: []
};
try {
const logger = new MediaMigrationLogger();
// Create in-memory repositories
const driverRepo = new InMemoryDriverRepository(logger);
const teamRepo = new InMemoryTeamRepository(logger);
const leagueRepo = new InMemoryLeagueRepository(logger);
const mediaRepo = new InMemoryMediaRepository(logger);
// Get all data
const drivers = await driverRepo.findAll();
const teams = await teamRepo.findAll();
const leagues = await leagueRepo.findAll();
result.processed.drivers = drivers.length;
result.processed.teams = teams.length;
result.processed.leagues = leagues.length;
this.logger.info(`In-memory mode: Found ${drivers.length} drivers, ${teams.length} teams, ${leagues.length} leagues`);
// Migrate drivers
for (const driver of drivers) {
try {
// Check if already has valid MediaReference
if (driver.avatarRef && driver.avatarRef instanceof MediaReference) {
continue;
}
// Get old URL from in-memory media repo
const oldUrl = await mediaRepo.getDriverAvatar(driver.id);
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
// Update driver in repository
const updated = driver.update({ avatarRef: newRef });
if (!this.dryRun) {
await driverRepo.update(updated);
result.updated.drivers++;
} else {
result.updated.drivers++;
}
}
}
} catch (error) {
result.errors.push(`Driver ${driver.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
// Migrate teams
for (const team of teams) {
try {
if (team.logoRef && team.logoRef instanceof MediaReference) {
continue;
}
const oldUrl = await mediaRepo.getTeamLogo(team.id);
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
const updated = team.update({ logoRef: newRef });
if (!this.dryRun) {
await teamRepo.update(updated);
result.updated.teams++;
} else {
result.updated.teams++;
}
}
}
} catch (error) {
result.errors.push(`Team ${team.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
// Migrate leagues
for (const league of leagues) {
try {
if (league.logoRef && league.logoRef instanceof MediaReference) {
continue;
}
const oldUrl = await mediaRepo.getLeagueLogo(league.id);
if (oldUrl) {
const newRef = this.parseOldUrl(oldUrl);
if (newRef) {
const updated = league.update({ logoRef: newRef });
if (!this.dryRun) {
await leagueRepo.update(updated);
result.updated.leagues++;
} else {
result.updated.leagues++;
}
}
}
} catch (error) {
result.errors.push(`League ${league.id}: ${error instanceof Error ? error.message : String(error)}`);
}
}
result.success = true;
this.logger.info(`In-memory migration completed: ${result.updated.drivers} drivers, ${result.updated.teams} teams, ${result.updated.leagues} leagues updated`);
return result;
} catch (error) {
result.errors.push(error instanceof Error ? error.message : String(error));
this.logger.error('In-memory migration failed', error instanceof Error ? error.stack : undefined);
return result;
}
}
/**
* Run migration based on environment
*/
async run(): Promise<MigrationResult> {
const persistence = process.env.GRIDPILOT_API_PERSISTENCE ||
(process.env.DATABASE_URL ? 'postgres' : 'inmemory');
this.logger.info(`Starting media reference migration in ${this.dryRun ? 'DRY RUN' : 'EXECUTE'} mode`);
this.logger.info(`Persistence mode: ${persistence}`);
if (this.dryRun) {
this.logger.info('DRY RUN: No changes will be made to the database');
}
if (persistence === 'postgres') {
return this.migratePostgres();
} else {
return this.migrateInMemory();
}
}
}
/**
* CLI entry point
*/
async function main() {
const args = process.argv.slice(2);
const dryRun = !args.includes('--execute') && !args.includes('-e');
if (args.includes('--help') || args.includes('-h')) {
console.log(`
Media Reference Migration Script
Usage:
ts-node scripts/migrate-media-refs.ts [options]
Options:
--execute, -e Execute the migration (default: dry run)
--help, -h Show this help message
Environment:
GRIDPILOT_API_PERSISTENCE=postgres|inmemory (default: postgres if DATABASE_URL set)
DATABASE_URL (required for postgres mode)
Examples:
# Dry run (test mode)
ts-node scripts/migrate-media-refs.ts
# Execute migration
ts-node scripts/migrate-media-refs.ts --execute
# With specific persistence
GRIDPILOT_API_PERSISTENCE=inmemory ts-node scripts/migrate-media-refs.ts --execute
`);
process.exit(0);
}
const migration = new MediaMigration(dryRun);
const result = await migration.run();
// Print summary
console.log('\n=== Migration Summary ===');
console.log(`Mode: ${dryRun ? 'DRY RUN' : 'EXECUTED'}`);
console.log(`Processed: ${result.processed.drivers} drivers, ${result.processed.teams} teams, ${result.processed.leagues} leagues`);
console.log(`Updated: ${result.updated.drivers} drivers, ${result.updated.teams} teams, ${result.updated.leagues} leagues`);
if (result.errors.length > 0) {
console.log(`\nErrors (${result.errors.length}):`);
result.errors.forEach(err => console.log(` - ${err}`));
}
if (!result.success) {
console.log('\n❌ Migration failed');
process.exit(1);
}
if (dryRun) {
console.log('\n✅ Dry run completed successfully. Run with --execute to apply changes.');
} else {
console.log('\n✅ Migration completed successfully.');
}
process.exit(0);
}
// Run if called directly
if (require.main === module) {
main().catch(error => {
console.error('Fatal error:', error);
process.exit(1);
});
}

View File

@@ -0,0 +1,75 @@
#!/usr/bin/env ts-node
/**
* Test script for media reference migration
* Creates a fixture with old URL formats and tests the migration
*/
import { MediaReferenceMigration } from './migrate-media-refs';
// Mock data with old URL formats
const mockDriverData = [
{ id: 'driver-1', oldUrl: '/api/avatar/driver-1' },
{ id: 'driver-2', oldUrl: '/api/avatar/driver-2' },
{ id: 'driver-3', oldUrl: '/images/avatars/male-default-avatar.jpg' },
{ id: 'driver-4', oldUrl: 'https://external.com/avatar.jpg' },
{ id: 'driver-5', oldUrl: '' }, // Empty
{ id: 'driver-6', oldUrl: null }, // Null
];
const mockTeamData = [
{ id: 'team-1', oldUrl: '/api/media/teams/team-1/logo' },
{ id: 'team-2', oldUrl: '/api/teams/team-2/logo' },
{ id: 'team-3', oldUrl: 'https://example.com/logo.png' },
];
const mockLeagueData = [
{ id: 'league-1', oldUrl: '/api/media/leagues/league-1/logo' },
{ id: 'league-2', oldUrl: null },
];
async function testMigration() {
console.log('=== Testing Media Reference Migration ===\n');
const migration = new MediaReferenceMigration(true); // Dry run mode
console.log('Testing URL parsing logic...\n');
// Test driver avatars
console.log('Driver Avatar Tests:');
for (const driver of mockDriverData) {
const ref = (migration as any).parseOldUrl(driver.oldUrl);
const result = ref ? ref.toJSON() : null;
console.log(` ${driver.id}: "${driver.oldUrl}" -> ${JSON.stringify(result)}`);
}
console.log('\nTeam Logo Tests:');
for (const team of mockTeamData) {
const ref = (migration as any).parseOldUrl(team.oldUrl);
const result = ref ? ref.toJSON() : null;
console.log(` ${team.id}: "${team.oldUrl}" -> ${JSON.stringify(result)}`);
}
console.log('\nLeague Logo Tests:');
for (const league of mockLeagueData) {
const ref = (migration as any).parseOldUrl(league.oldUrl);
const result = ref ? ref.toJSON() : null;
console.log(` ${league.id}: "${league.oldUrl}" -> ${JSON.stringify(result)}`);
}
// Test deterministic avatar selection
console.log('\nDeterministic Avatar Variant Tests:');
const testIds = ['driver-1', 'driver-2', 'driver-3', 'driver-4', 'driver-5'];
for (const id of testIds) {
const variant = (migration as any).getDeterministicAvatarVariant(id);
console.log(` ${id} -> ${variant}`);
}
console.log('\n✅ All tests completed successfully!');
console.log('\nTo execute the actual migration:');
console.log(' npm run migrate:media:exec');
console.log('\nFor dry run (no changes):');
console.log(' npm run migrate:media:test');
}
// Run tests
testMigration().catch(console.error);