LLM supplier entity-matching platform
FLAGSHIP · GREENFIELD → AWS → GCP REBUILD · 2026The platform automatically matches supplier and vendor records at scale for a real-time integration with a third-party supplier-data platform. I built about 90% of it. It started as a greenfield build on AWS, and in 2026 I led its rebuild on GCP, which is now running in production.
Original build AWS
- Partner events → Amazon MQ (RabbitMQ)
- Lambda trigger — dedupes by batch ID
- MWAA (managed Airflow) orchestration
- ECS Fargate preprocessing
- 3 parallel Bedrock models — Titan · Claude · Llama
- Lambda consensus reconciliation
- Redshift checkpoints & results → Lambda notify
2026 rebuild GCP · IN PRODUCTION
- Event-driven RabbitMQ trigger — idempotent handling
- Airflow on Cloud Composer orchestration
- 5 containerised stages on GKE
- 3-model ensemble — Gemini · Claude · GPT, consensus matching
- Incremental checkpointing — resumable multi-hour runs
- Snowflake — cross-cloud, dynamic masking
- RabbitMQ notify back to partner
- Migration: replaced the old self-hosted AWS deployment with a set of microservices running on Kubernetes.
- Cost control: each message is processed only once, so a duplicate event never pays for the same LLM work twice. Long batch runs save checkpoints and can resume after a failure instead of starting over.
- CI/CD across 5 services: GitHub Actions builds versioned images into Artifact Registry and promotes them automatically from DEV to PROD. Workload Identity Federation means there are no long-lived credentials.
- Data layer: I owned the Snowflake layer, including cross-cloud connectivity, dynamic data masking and schema migrations.
- Production operations: resolved live incidents, added structured logs linked by trace ID, and worked with the partner's engineering and security teams.