# Kashif Sohail — full profile > Markdown rendering of https://kashifsohail.com/ for LLMs and agents. Summary: https://kashifsohail.com/llms.txt SENIOR DATA PLATFORM ENGINEER · CONTRACT / B2B # Cloud data platforms and production LLM pipelines on GCP and AWS. I'm Kash. I design and build data platforms, including warehouses, orchestration, streaming and LLM pipelines, for European and US teams. I own the architecture and the client relationship, and a small team helps with delivery. Available for B2B contracts · EU & US East Coast hours [me@kashifsohail.com](mailto:me@kashifsohail.com) See the work GCP + AWS — production platforms on both clouds 3-model — LLM ensemble running in production 1,450+ — member tech community I founded B2B — contracting through KashEdge LLC (US) ## Selected work Client work is described without naming the client. ### LLM supplier entity-matching platform FLAGSHIP · GREENFIELD → AWS → GCP REBUILD · 2026 The platform automatically matches supplier and vendor records at scale for a real-time integration with a third-party supplier-data platform. I built about 90% of it. It started as a greenfield build on AWS, and in 2026 I led its rebuild on GCP, which is now running in production. #### Original build AWS - Partner events → Amazon MQ (RabbitMQ) - Lambda trigger — dedupes by batch ID - MWAA (managed Airflow) orchestration - ECS Fargate preprocessing - 3 parallel Bedrock models — Titan · Claude · Llama - Lambda consensus reconciliation - Redshift checkpoints & results → Lambda notify #### 2026 rebuild GCP · IN PRODUCTION - Event-driven RabbitMQ trigger — idempotent handling - Airflow on Cloud Composer orchestration - 5 containerised stages on GKE - 3-model ensemble — Gemini · Claude · GPT, consensus matching - Incremental checkpointing — resumable multi-hour runs - Snowflake — cross-cloud, dynamic masking - RabbitMQ notify back to partner - Migration: replaced the old self-hosted AWS deployment with a set of microservices running on Kubernetes. - Cost control: each message is processed only once, so a duplicate event never pays for the same LLM work twice. Long batch runs save checkpoints and can resume after a failure instead of starting over. - CI/CD across 5 services: GitHub Actions builds versioned images into Artifact Registry and promotes them automatically from DEV to PROD. Workload Identity Federation means there are no long-lived credentials. - Data layer: I owned the Snowflake layer, including cross-cloud connectivity, dynamic data masking and schema migrations. - Production operations: resolved live incidents, added structured logs linked by trace ID, and worked with the partner's engineering and security teams. GKECloud ComposerAirflowBedrockGeminiClaudeGPTRabbitMQSnowflakeRedshiftGitHub Actions ### GCP data platform for a music streaming company CONTRACT · PRODUCTION DATA PLATFORM A production data platform on Google Cloud. Pipelines are generated from YAML config, so adding a new source means writing config, not code. A custom data-quality framework checks the data as it moves through. ClickHouse serves the creator-facing analytics. - Creator Studio analytics: ClickHouse serves sub-second queries over high-cardinality creator metrics. - Ingestion: Spark jobs on Dataproc bring in app data from Firebase and AppsFlyer and land it in BigQuery. - Config-driven DAGs: YAML definitions generate the Cloud Composer (Airflow) pipelines automatically. - Data quality: I built a custom framework of quality checks that runs as part of the pipelines. - Modelling & operations: dbt on BigQuery with event data from Pub/Sub, and Airflow failures alert the team in Slack. BigQueryCloud ComposerDataprocSparkClickHousedbtPub/SubFirebaseAppsFlyer ### Social media ingestion & normalisation platform CONTRACT · MEDIA INTELLIGENCE A production platform for a narrative-intelligence company. It pulls social and news content from many sources into one standard schema, enriches it, and lands it in Delta tables. Those tables power a searchable feed of narratives for analysts. - Sources: Meltwater (Twitter/X, Facebook, Instagram, Weibo, Douyin, WeChat and news), Reddit, Telegram and Webz. - Normalisation: one schema covering authors, hashtags, media types and empty fields, including Arabic and Chinese, Japanese and Korean content. - Enrichment: fills in missing link details and validates engagement metrics. - Reliability: alerts when jobs time out, handles very large engagement values, recovers from 404s when filling in Reddit links, and checks that channel tables exist before writing. - Operations: managed the Databricks workspace and bundle configuration, and the Argo/Docker CI triggers. PySparkDatabricksDelta LakePythonArgoDocker ## More platform work Production pipelines, integrations, infrastructure and AI applications from client and contract work, including 40+ production pipelines built on Airflow, Dagster and dbt. CONTRACT · US DELIVERY-TECH UNICORN ### Snowflake data platform & CDC pipelines I led the design and delivery of a Snowflake platform that brings together databases, APIs and mobile app events. Dagster runs ingestion and transformation on Azure Kubernetes Service (AKS). I rebuilt the PostgreSQL replication with Meltano to read the database's change log (log-based CDC) and built a fault-tolerant FTP ingestion service in Docker. Dashboards run on Cube.js and Looker. TERRAFORM · SNOWFLAKE · GITHUB ACTIONS ### Infrastructure as code for data platforms Snowflake resources, roles and permissions managed through version-controlled Terraform and CI/CD, so infrastructure changes are repeatable instead of manual. GCP · BIGQUERY · AIRFLOW · CLICKHOUSE ### GCP data integration & analytics platform Brings operational databases, APIs and marketing platforms into BigQuery. Airflow on Cloud Composer orchestrates ingestion and transformation for reporting and BI. CLICKHOUSE · BIGQUERY · REVERSE ETL ### ClickHouse analytics & reverse ETL Processes user-level data in ClickHouse and sends it back to downstream applications through reverse ETL. The work covered table design, query optimisation and moving data between analytical systems. PYTHON · REST APIS · FIVETRAN ### Multi-tenant e-commerce analytics Custom connectors and reusable ingestion logic for Shopify, Meta Ads, Google Analytics, Uplisting and payment providers. They feed analytics platforms that generate insights automatically for product and commercial teams. CAREEM · 2021 · EMR · KAFKA · HIVE · PRESTO ### AWS data platform & self-serve portal Batch pipelines run with Airflow and Spark on EMR. Real-time CDC streams through Maxwell, Kafka and Spark Streaming into a Hive warehouse on S3, queried with Presto and reported in Redash. I built a self-serve portal that manages 500+ streaming and 700+ batch jobs without command-line access, with provisioning through CloudFormation. LANGGRAPH · DAGSTER · GRAFANA ### DataOps agent A multi-agent LangGraph system for a delivery-tech client. It watches Dagster pipelines, has an LLM read failure logs, and fixes failures itself by scaling compute and restarting jobs. A second flow handles PostgreSQL replication: when Grafana alerts that the database's change log has grown too large, the agent stops the Dagster job, removes the stuck replication slot and starts a clean partial refresh. No one needs to be paged for it. LANGCHAIN · FASTAPI · PINECONE · AWS ### Legal AI search engine A question-answering system for legal content. It covers the full RAG pipeline, from ingesting documents to generating answers with an LLM, and runs on AWS with CI/CD, auto-scaling and model versioning in MLflow. LLM AGENT · AUTOMOTIVE ### Automotive inventory agent A conversational agent that answers natural-language questions about live inventory, drawing on data pipelines from several sources. PYTHON · ETL FRAMEWORK ### Transflux A Python ETL framework of reusable extract, transform and load components, so new integrations aren't written as one-off scripts. ## Architecture & AI prototypes Designs, demos and early-stage concepts. Each card says what stage it reached. ARCHITECTURE / DEMO ### Modern data lakehouse Apache Iceberg tables on S3, with Silver and Gold layers modelled in dbt, Dagster for orchestration and DuckDB for queries. Superset and Cube.js were planned for dashboards and the semantic layer. AI PROTOTYPE ### Quran semantic search Semantic search over the Quran using embeddings, so questions in plain language find the relevant verses. STARTUP CONCEPT · INITIAL ARCHITECT ### Pakistani e-commerce price aggregator An architecture to collect listings from Pakistani retailers, standardise their catalogues and match the same product across stores. Price history and drops are modelled in a column-oriented analytics database such as ClickHouse. ## Open source I've worked in the Meltano and Singer ecosystem for years. These are projects I've published or am building now. SINGER SDK · IN PROGRESS ### tap-workday A modern Workday tap built on the Singer SDK, with support for both RaaS reports and REST/WQL. I designed it after comparing the existing Workday connectors in Meltano, Fivetran, Airbyte and Stitch. MELTANO · DAGSTER · IN PROGRESS ### Meltano self-service platform UI A self-service interface on top of Meltano that edits meltano.yml, so the config file stays the source of truth. Dagster runs the pipelines, and the UI triggers reloads through Dagster's GraphQL API. [ SINGER SDK · 2026 ### tap-absorb ↗ A Singer tap for Absorb LMS. It extracts from the Integration API v2 into Snowflake, BigQuery, Postgres or any other Singer target. ](https://github.com/kashif-se/tap-absorb) SINGER SDK · E-COMMERCE ### tap-nuvemshop A Singer tap for the Nuvemshop (Tiendanube) REST API, a Latin American e-commerce platform. It syncs store data incrementally from a configurable start date. [ MELTANO SDK · BIGQUERY ### target-bigquery ↗ Extended the Meltano BigQuery target so users can override the default partition column and control how streams map to table names. ](https://github.com/kashif-se/target-bigquery) [ AIRFLOW · ★ 11 ### airflow-google-chat-integration ↗ Sends Airflow DAG task success and failure alerts to Google Chat. ](https://github.com/kashif-se/airflow-google-chat-integration) [ DOCKER · ORACLE · ★ 4 ### docker-windows-python-oracle-instant-client ↗ A Windows Server Core container with Python and Oracle Instant Client 19 preinstalled, for running Python pipelines against Oracle databases on Windows hosts. ](https://github.com/kashif-se/docker-windows-python-oracle-instant-client) [ DOCKER · JUPYTER · ★ 2 ### docker-ipyjulia-notebooks ↗ A single container with Python, Julia, PyJulia, IJulia and Jupyter, for notebooks that mix Python and Julia. ](https://github.com/kashif-se/docker-ipyjulia-notebooks) ## Stack The tools I use in production. ### CLOUD — GCP BigQueryCloud ComposerPub/SubGKEArtifact Registry ### CLOUD — AWS MWAALambdaECS FargateBedrockRedshiftAmazon MQS3IAMSecrets ManagerCloudWatchX-RayCodePipelineECR ### DATA ENGINEERING ETL / ELTChange data captureAPI & DB integrationsData modellingReverse ETLPipeline observability ### WAREHOUSING & LAKEHOUSE SnowflakeBigQueryClickHouseRedshiftPostgreSQLDatabricksDelta LakeUnity CatalogApache IcebergNessieTrino / PrestoHiveSQL ServerMySQLDuckDB ### ORCHESTRATION & TRANSFORM AirflowCloud ComposerMWAADagsterdbtMeltanodltFivetranSpark ### STREAMING & MESSAGING KafkaSpark StreamingPub/SubRabbitMQActiveMQELK Stack ### AI / LLM LangChainLangGraphRAGEmbeddingsVector searchPineconeWeaviateMCPMulti-LLM ensembles ### ML PyTorchTensorFlowHugging FaceMLflowPlotly ### INFRA & DEVOPS TerraformCloudFormationAzureKubernetesDockerGitHub ActionsJenkinsGitLabn8nWorkload Identity FederationDEV→PROD CI/CD ### LANGUAGES & MORE PythonSQLJavaNode.jsBashFastAPIFlaskCelerySinger SDKFlutterReact ## Experience I started out as a software engineer and have worked in data since 2018. I usually join as the founding or lead data engineer. ONGOING ### Founder & principal engineer — [KashEdge LLC](https://kashedge.net) A boutique data platform studio. I own the architecture and client relationships, and a team handles execution. FEB 2022 – NOW ### Principal Data Platform Engineer — independent consultant Founding or lead data engineer for a US delivery-tech unicorn, a music streaming platform, a legal tech startup, a cybersecurity company, a media-intelligence company and a UK fintech consultancy, where in 2026 I delivered the supplier entity-matching platform described above. I own the architecture, the choice of stack and the roadmap from day one. The platforms I designed on AWS and GCP use Snowflake and BigQuery warehouses, batch and CDC streaming ingestion, access control managed in Terraform, and monitoring that handles billions of events a day. I set the platform engineering standards (CI/CD, secrets management, Prometheus monitoring, runbooks) and mentor junior engineers. AUG 2021 – FEB 2022 ### Data Engineer II — Careem (Uber) Part of the DataOps team running Careem's AWS data platform at ride-hail scale: 500+ streaming and 700+ batch jobs in production. I built Spark and Python pipelines with Airflow and Jenkins, and shipped a self-serve portal for scheduling jobs from MySQL, PostgreSQL, SQL Server and DynamoDB. I also managed infrastructure in Terraform and set up monitoring in Grafana with on-call alerts through VictorOps. OCT 2019 – AUG 2021 ### Data Engineer — Easypaisa (Telenor Microfinance Bank) Built a cloud data-as-a-service platform from scratch for Pakistan's largest digital payments system, one of the first big data platforms in the country's fintech sector. Spark jobs brought AppsFlyer, Google Analytics and Firebase data into Hadoop to give senior management a complete view of the product. I was the technical contact for supplying app data to the State Bank of Pakistan for regulatory and fraud work, and worked with data scientists on credit-scoring and churn models. DEC 2018 – OCT 2019 ### Big Data Engineer — SlashNext Designed ScrapiXcel, a large distributed web-scraping platform built on Selenium, Celery, Redis and RabbitMQ that scaled across Docker clusters. I owned the full pipeline: storage in Cassandra, MongoDB and Elasticsearch, processing in Spark, and analytics in a new Greenplum warehouse. AUG 2016 – DEC 2018 ### Software Engineer — Expertflow Technical owner and lead developer of the Generic CRM Connector. I led the Expertflow Chat Solution, including its chat server, media routing engine, and customer–bot–agent routing over Kafka and WebRTC. I also worked on the Cisco UCCX Media Routing Engine (Java, C#, Node.js, Apache Camel, ActiveMQ, Azure). ## Community & leadership In 2023 I founded Sahiwal Tech Community (STC) to help students, developers, freelancers and tech professionals in Sahiwal, Pakistan learn, connect and grow. It's run by volunteers. We hold conferences, meetups, hands-on workshops and mentoring on AI, cloud, software engineering and open source, linking local talent with industry and the global tech world. 1,450+ — community members 1,100+ — attendees across major events 6 — major events since Dec 2025 ~40% — women at our GitHub events ### Events I've organised OCT 2026 · 500+ ATTENDEES ### AWS Student Community Day Sahiwal 2026 I led the organisation of this full day of AI, cloud, data platform, agentic-AI security and career sessions, which included an expert panel on AI trends. District Government Sahiwal supported the event. I also spoke there on "Data Platforms Decoded", an AWS-first walkthrough of modern data platforms, and wrote it up for AWS Builder Center. [Event highlights ↗](https://www.instagram.com/p/Dey-szxAT4G/) [Talk reel ↗](https://www.instagram.com/p/DeuFhwGPX92/) [Builder Center article ↗](https://builder.aws.com/content/3KaigkKAZjtEp3kB901sChqXfgb/data-platforms-decoded-five-jobs-two-real-aws-builds-and-500-builders-in-sahiwal) SEP 2026 · 300+ AUDIENCE ### Dev Days Sahiwal 2026, with GitHub Sessions on GitHub Copilot, context engineering, the Model Context Protocol (MCP) and browser-extension development, given by experienced engineers. It was held at Jinnah Hall with District Government Sahiwal, and the Deputy Commissioner of Sahiwal addressed the audience. [Highlights ↗](https://www.instagram.com/p/DdvXL3cDaix/) 2026 · 140+ PARTICIPANTS ### GitHub Copilot Dev Days Sahiwal An introduction to AI-assisted software development, with sessions led by GitHub Campus Experts. About 40% of participants were women. [Highlights ↗](https://www.instagram.com/p/DXtrkSfDQcv/) [Reel ↗](https://www.instagram.com/reel/DXzHY5pPuGZ/) 2026 · 100+ ATTENDEES ### Sahiwal Agentic AI Meetup Hands-on agentic AI workshops with GitHub Campus Experts and practitioners. About 40% of participants were women. [Highlights ↗](https://www.instagram.com/p/DZu-FQxjblW/) 2026 ### Tech Iftaar A Ramadan get-together where students, professionals and volunteers could connect outside technical sessions. [Highlights ↗](https://www.instagram.com/p/DV1C8hmF0rS/) DEC 2025 · 60+ PARTICIPANTS ### Sahiwal Tech Conference & Networking Event Brought together company leaders, freelancers, students and professionals to network and share knowledge. [Highlights ↗](https://www.instagram.com/p/DXFFxz3DZPq/) [Event post ↗](https://www.facebook.com/share/p/18CVECUKhM/) ### Talks & guest lectures OCT 2026 ### DPSC Sahiwal Talks for intermediate students on building a career in the AI era. [Session reel ↗](https://www.instagram.com/p/DerclbRPHWs/) FEB 2026 ### Building a Strong Foundation Before Chasing AI — Notion Elevate, CUI Sahiwal Why fundamentals still matter, and how students and engineers can use AI responsibly. The event was organised by Notion CUI Sahiwal, and the students asked lots of curious questions. [Session reel ↗](https://www.instagram.com/p/DUpkHovCOUs/) [Event post ↗](https://www.instagram.com/p/DUsb53CiMet/) SEP 2024 · 250+ STUDENTS ### A Road Map to Success, CUI Sahiwal Career roadmap session hosted by the COMSATS Skill Development Society. [Event post ↗](https://www.facebook.com/share/p/1KcBF7S2HF/) 2024 ### Your First Step Into Data Platforms — Google DevFest Sahiwal An introduction to data platforms at the Sahiwal edition of DevFest. [Talk post ↗](https://www.instagram.com/p/DDrmpO7Cahs/) [Event post ↗](https://www.facebook.com/share/p/1DYGQqyREj/) DEVHACK HACKATHON ### The Power of Remote Work, CUI Sahiwal Remote jobs: how hiring works, where the opportunities are, and the challenges. [Event post ↗](https://www.facebook.com/share/p/1KNhRKFKuc/) OCT 2022 · PANEL ### Open Source Day panel talk, CUI Sahiwal Alumni panel on open source, hosted by the Computer Science Society. [Event post ↗](https://www.facebook.com/share/p/1b4Kr8qLUH/) TALK SERIES · 4 UNIVERSITIES ### Data Science for Everyone An introduction to data science for students of any background, organised with student societies at Quaid-i-Azam University's IT department (IIT Society), Arid Agriculture University's UIIT (Aridian Array Software Society), Air University Multan and CUI Sahiwal (Computer Science Society, Oct 2020). [QAU Islamabad ↗](https://www.facebook.com/share/p/1JmpHZhaYj/) [Arid Agriculture University ↗](https://www.facebook.com/share/p/1CNRLktwxi/) [Air University Multan ↗](https://www.facebook.com/share/p/1DAzAqiFj6/) [CUI Sahiwal ↗](https://www.facebook.com/share/p/17zFCx7kby/) MAY 2019 · TRAINING ### Applied Data Science, Riphah International University Lahore Hands-on applied data science training. [Event post ↗](https://www.facebook.com/share/p/1AP2ZomNSH/) ALUMNI SESSION ### Learn with Industry Expert: A Day with Alumni, CUI Sahiwal Hosted by the Computer Science Society. [Event post ↗](https://www.facebook.com/share/p/1DGGoajDC7/) WORKSHOP ### Building real-time web apps with Node.js, CUI Sahiwal Hands-on workshop. I received a certificate of appreciation from the Head of the Computer Science department. [Event post ↗](https://www.facebook.com/share/p/1HJUj1mkpJ/) ### Mentoring & placements SINCE UNIVERSITY ### Preparing graduates for their first roles I work with students as they graduate on career preparation and mock interviews, and I refer them wherever I can. Graduates I've prepared now work at Expertflow, Arbisoft, Constellation Software (USA), SABIC (KSA), AddoAI and Mountainise. ### Recognition & roles AWARD ### COMSATS Alumni Excellence Award: Social Impact & Community Service Recognised for my community building, youth mentorship, technology engagement and social impact. [Receiving the award ↗](https://www.instagram.com/p/DZe3GoHvS_B/) [Ceremony post ↗](https://www.instagram.com/p/Daa0nqgv7c5/) ADVISORY · SINCE 2024 ### Industrial Advisory Board, Department of Computer Science COMSATS University Islamabad, Sahiwal Campus. I attend the board's meetings and give feedback on improving the curriculum, linking academia with industry. ECOSYSTEM ### Partners & collaborators GitHub Campus Experts, AWS Community Builders, universities, industry professionals and District Government Sahiwal. Talent exists everywhere, and opportunity should too. LEARN · BUILD · CONNECT ## Products Apps I've built for underserved markets and communities. ### WakeelsApp FOUNDER · 2024–NOW An AI-powered case-management assistant for lawyers in Pakistan. Lawyers track cases, hearings and deadlines and get AI legal research in Urdu and English. Clients can find a verified lawyer through the public directory. The Android app is live on Google Play, and iOS is in beta. I built the platform myself: a Flutter app, a Flask and Postgres API with Celery, a Next.js web app and back-office, court data synced from 36 Punjab districts, and Grafana monitoring. The app is built around law chambers with multiple users. Next on the AI roadmap: legal research grounded in judgments and statutes (RAG), vector search to find similar cases, and suggestions of relevant law sections. FlutterFlaskPostgreSQLCeleryNext.jsLLMs [wakeels.app ↗](https://wakeels.app) [Google Play ↗](https://play.google.com/store/apps/details?id=com.wakeels.app) [ FLUTTER · FASTAPI ### Altasreef ↗ An Arabic morphology learning app, live on Google Play. I built the Flutter app, the FastAPI backend and the iOS release. ](https://play.google.com/store/apps/details?id=org.haqnawaz.altasreef) ## Education MS · 2017 – 2019 ### Data Science — Information Technology University (ITU), Lahore Thesis: predicting extreme conditions in light-sport aircraft engines using machine learning. MICROMASTERS · 2017 – 2018 ### Data Science — UC San Diego (edX) Graduate-level programme in statistics, Python for data science, machine learning and big data analytics. BS · 2012 – 2016 ### Computer Science — COMSATS University Islamabad, Sahiwal Teaching assistant for Digital Logic Design, Object-Oriented Programming and Data Structures. Campus Bronze Medalist. I won team and individual places in inter- and intra-university programming competitions and founded student societies. In my final year I was selected as the campus's first Microsoft Student Partner and helped students add cloud and modern tech skills to their CVs. ### Certifications 2021 ### Microsoft Certified: Azure Fundamentals 2020 ### Certified Data Domain Practitioner — Ant Financial 2018 ### Big Data Analytics Using Spark — UC San Diego (edX) AVAILABLE FOR B2B CONTRACTS ## Let's scope your platform. I work remotely, with strong overlap with European and US East Coast hours. Honest scoping I agree on clear boundaries and an estimate before committing. Milestone-based Fixed-price milestones, so you know the cost upfront. B2B-ready I contract corp-to-corp through [KashEdge LLC](https://kashedge.net) (US). [me@kashifsohail.com](mailto:me@kashifsohail.com) [KashEdge ↗](https://kashedge.net) [LinkedIn](https://www.linkedin.com/in/kshfse/) [GitHub](https://github.com/kashif-se) [X / Twitter](https://x.com/kshfse) © 2026 Kashif Sohail · [KashEdge LLC](https://kashedge.net) Remote · EU & US East Coast hours