Snowcap DatalabTarun Kumar — Data & AI ArchitectureSelected work
Client names are withheld by default — what's below is real, anonymized engagement structure. Five are client engagements I delivered; four are illustrative reference scenarios, modeled on publicly documented industry implementation patterns rather than my own client work, included to show how I'd approach a comparable problem. Each is labeled.
Work delivered for real clients, anonymized
Company names are withheld; architecture, decisions, and outcomes are not.
Unifying SAP and Salesforce contract data on a governed Databricks lakehouse
Context
A global manufacturer of industrial and technology products running SAP for core order and contract management alongside Salesforce (Apttus CLM, SMAX, CCRM) for service-contract and entitlement workflows — two systems of record, two data models, no unified governed view of what a customer was entitled to at any point in time.
Architecture problem
Reconciling SAP's sales-document chain against Salesforce's contract lifecycle objects into consistent entities without losing either domain's meaning, delivering near-real-time entitlement data to a downstream Contract API without exposing either source system directly, and supporting SOX-auditable, fully reprocessable daily runs where any prior day's contract state has to be reproducible on demand.
Decisions I made
Kept the two domains — SAP-rooted product contracts, Salesforce-rooted service contracts — as parallel Silver-layer notebook sets sharing common Bronze/Gold conventions, rather than forcing a premature unification. Decoupled the Contract API from both source systems entirely via a Kafka change-data-capture chain. Staged the Unity Catalog governance migration behind an environment/domain-scoped toggle instead of one high-risk cutover event.
What I built / changed
An Azure Data Factory + Databricks Bronze/Silver/Gold medallion pipeline on ADLS Gen2; ten-plus Silver transformation notebooks implementing the domain-specific contract, entitlement, and discount models; a five-stage Kafka CDC chain from change-tracking through Contract API delivery, with event-volume guard thresholds protecting downstream systems from abnormal batches; and the CSV mapping-sheet validation gate that runs before every execution.
Result
A single governed Gold layer replaced what were previously siloed, source-specific extracts. The pipeline runs on a daily automated cadence with a full SOX-auditable run history and explicit reprocessing controls, and the Contract API gets near-real-time entitlement visibility instead of batch-only visibility.
Technologies
Databricks · Azure Data Factory · Delta Lake · Unity Catalog · Kafka · PySpark · SAP · Salesforce
Identity synchronization and service-principal risk analytics on Databricks
Context
A global, diversified industrial and technology manufacturer with a large Databricks estate governed under Unity Catalog, and a growing population of service principals (SPNs) provisioned automatically via enterprise identity federation — with no consolidated view of which ones held admin privileges, which were dormant, or which had been granted access outside governed groups.
Architecture problem
The platform has no native "SPN created" audit event — service principals arrive via identity federation, not a Databricks-native provisioning flow — and the job-run audit trail records identity by an opaque federated GUID rather than display name, so ownership and activity had to be reconstructed from indirect signals before any risk model could be built.
Decisions I made
Built a dedicated GUID-to-display-name mapping table as the missing link, rather than working around the gap per-query. Separated "risk tier" — a governance-policy waterfall of HIGH/MEDIUM/LOW — from a distinct, additive anomaly score across three severity tiers, since conflating governance category with operational priority produces a model that's either too coarse to act on or too noisy to trust.
What I built / changed
A parameterized IAM sync job merging PostgreSQL role, persona, and business-segment data into four Unity Catalog Delta tables with service-account masking; and a four-notebook SPN Fleet pipeline — inventory, privilege map, activity summary, planned sensitivity enrichment — feeding a Databricks Lakeview dashboard, plus the written methodology reference used to onboard the security team.
Result
A single governed source of truth for user/role/persona state inside Unity Catalog, and the first consolidated, scored inventory of the platform's service-principal population — with a documented, explainable remediation workflow (CRITICAL → HIGH → MEDIUM) giving the security team a prioritized queue instead of an undifferentiated identity list.
Technologies
Databricks · Unity Catalog · Delta Lake · PySpark · PostgreSQL · AWS Secrets Manager
Tokenizing PII in real time on a Kafka-to-Snowflake streaming pipeline
Context
A veterinary pharmacy and prescription-fulfillment platform serving thousands of veterinary practices (9,000+) across the country, generating tens of millions of order and prescription events a month that needed to reach Snowflake for downstream analytics — without ever letting raw PII transit the pipeline in plaintext, and without per-row SaaS connector pricing at that event volume.
Architecture problem
A managed SaaS connector had no architecturally sound place to inject tokenization before data left the source — its only tokenization point is a post-load Snowflake transformation, by which point the data has already transited the vendor's infrastructure unprotected. The alternative had to hit a sub-2-minute latency target while supporting multiple tokenization methods — reversible, irreversible hash, format-preserving — in one framework.
Decisions I made
Tokenize at the very first hop, before Kafka, with the protection policy embedded in the Avro schema itself so every producer and consumer team inherits it automatically instead of configuring DLP per team. Batch DLP calls at 100 records per call rather than one at a time, cutting per-record cost by roughly two orders of magnitude. Chose GCP Managed Schema Registry over Confluent Cloud, and Managed Kafka Connect's GCS Sink over a custom Dataflow job — both documented, evidence-based calls, not defaults.
What I built / changed
StreamShield — a production Python SDK with a producer/consumer facade, a schema-driven DLP tokenizer/detokenizer, and a token cache — plus the Terraform modules reproducing the full GCP environment, and 63 automated tests ahead of the first rollout. Delivery runs an event-driven GCS → Pub/Sub → Snowpipe chain with manual offset commits and a dead-letter-topic design for the failure path.
Result
A structural guarantee that raw PII/PCI never transits Kafka, storage, or Snowflake in plaintext. Steady-state end-to-end latency of roughly 60–90 seconds against a sub-2-minute target, and per-row SaaS-connector pricing avoided entirely at 9,000+ practices and tens of millions of monthly events.
Technologies
Python · Kafka (GCP Managed) · Google Cloud DLP · Cloud KMS · Managed Kafka Connect · Snowflake (Snowpipe) · Terraform
Consolidating global SAP supply chain data on an event-driven Databricks lakehouse
Context
A multinational manufacturing and distribution enterprise running SAP ECC, HANA, and S/4HANA concurrently across multiple global regions — with an active HANA-to-S/4 migration in flight — serving 2,000+ daily users across demand planning, inventory, supplier operations, and logistics.
Architecture problem
Harmonizing three structurally different SAP generations into one canonical model without losing region-specific meaning, ingesting only newly arrived delta files with exactly-once guarantees across many regions landing files asynchronously, and balancing centralized governance against decentralized domain-analytics ownership without re-fragmenting the platform.
Decisions I made
Event-driven ingestion (S3 PUT → EventBridge → Auto Loader) instead of scheduled polling; Delta Lake MERGE-based CDC instead of full reloads; and a hub-and-spoke workspace model — one governed Gold contract, many independently operated spoke workspaces — so regional teams could move independently without becoming a bottleneck on the central platform team.
What I built / changed
The Bronze/Silver/Gold medallion pipeline itself; the versioned transformation and mapping-layer framework absorbing the in-flight SAP migration; certified Gold business models for inventory turnover, OTIF, lead-time variance, forecast accuracy, and supplier performance; and a cost/performance observability layer tracking job, compute, and storage growth.
Result
Certified Gold-layer models consumed by 2,000+ daily users across regions and functions; exactly-once, event-driven ingestion replacing scheduled extraction across multiple global markets; a repeatable transformation framework absorbing the SAP HANA → S/4 migration without breaking existing Gold consumers.
Technologies
AWS · Databricks · Delta Lake · Unity Catalog · Auto Loader · Kafka · PySpark · Amazon EventBridge
Introducing a production GenAI system on top of an existing manufacturing data platform
Context
A technology organization running distributed engineering and delivery teams whose project knowledge was scattered across four disconnected systems of record — source control, documentation, task tracking, and support ticketing — with strict data-compliance requirements that ruled out routing everything through a single third-party hosted LLM API.
Architecture problem
Capturing both semantic similarity and structural relationship information — which commit closed which ticket, which document describes which service — in one retrieval layer; supporting bring-your-own-model and bring-your-own-cloud requirements without forking the platform per deployment; and measuring retrieval quality rigorously enough to trust the system in production, not just in a demo.
Decisions I made
A dual-index design — a vector store for semantic retrieval, a graph store for structural retrieval — fused at query time, rather than picking one. A distinct, decoupled model-serving layer (Bedrock, SageMaker, custom) so backend swaps never touch retrieval or serving. Evaluation and guardrails built in as first-class pipeline stages from day one, not bolted on after a working prototype.
What I built / changed
The BYOM/BYOC platform architecture end to end — ingestion connectors feeding a Milvus vector index and a Neo4j graph index, a multi-stage retrieval and evaluation pipeline (precision@k, recall, MRR, NDCG), ML-powered PII and content guardrails, and a horizontally scalable FastAPI/Redis/Celery serving layer fronting LangChain/LangGraph agentic orchestration.
Result
A single natural-language interface giving end-to-end project visibility across four previously disconnected systems, 100% data-compliance maintained across BYOM/BYOC deployment options without forking the platform per backend, and a production evaluation framework giving retrieval quality a measurable, trackable baseline instead of a one-time demo impression.
Technologies
RAG · LangGraph · FastAPI · Redis · Celery · Milvus · Neo4j · AWS Bedrock · SageMaker
How I'd approach problems I haven't billed hours on — yet
These four are not my client work. Each is modeled on a publicly documented industry implementation pattern and used here to show my architectural thinking against a comparable problem, with industry-reported figures clearly marked as such.
Reducing on-shelf out-of-stocks with lakehouse-based demand forecasting
Modeled on a publicly documented Databricks Lakehouse solution pattern for on-shelf availability — not a client engagement of mine. Included to show how I'd approach a comparable problem.
Context
Large-format, multi-SKU retailers lose an estimated 8% of on-shelf availability on average — about one in thirteen products unavailable at any moment — with some studies putting the figure as high as one in three shopping trips. The traditional fallback, physical shelf audits, doesn't scale against hundreds of thousands of SKUs serviced nearly around the clock.
Architecture problem
Reconciling inventory and point-of-sale data at a consistent per-period grain across an enormous SKU × location matrix, then telling a genuine stockout apart from a slow-mover's natural zero-sales day, phantom inventory that doesn't match reality, and safety-stock thresholds miscalibrated in either direction — all while serving both data science and merchandising teams off one platform.
Decisions I'd make
Use Delta Lake as the single reconciliation ledger between inventory and POS rather than reconciling downstream in a BI tool; split modeling logic into three composable stages instead of one monolithic notebook; pair a classical seasonal model (Prophet) with a fast, explainable baseline so operations teams get a defensible signal rather than a black box. I'd also make the zero-sales flagging threshold a per-velocity-class parameter rather than one global constant — a single threshold under- or over-flags fast vs. slow movers.
What this looks like built
A three-notebook accelerator pattern — data preparation and reconciliation, out-of-stock detection (phantom inventory, safety-stock violations, statistically significant zero-sales runs), and on-shelf availability scoring — running on a single Lakehouse platform with a SQL Analytics layer serving BI tools directly, no separate warehouse.
Result
Industry-reported figures from the source pattern: a 2% improvement in on-shelf availability is associated with a 1% increase in retail sales, and the addressable problem is estimated at up to $1 trillion in global retail sales.
Technologies
Databricks · Delta Lake · PySpark/SQL · Prophet · SQL Analytics · Tableau · Power BI
Grid-scale distributed energy load forecasting at daily cadence
Modeled on a publicly documented large-scale Azure Databricks forecasting implementation for a European electricity distribution network — not a client engagement of mine. Included to show how I'd approach a comparable problem.
Context
A large regional electricity distribution operator planning medium- and high-voltage grid capacity around conservative static safety margins — a real, near-term load forecast at the individual-installation level was the missing ingredient for unlocking hosting-capacity headroom without new physical build-out.
Architecture problem
Forecasting roughly 200,000 individual time series daily at 15-minute granularity — not a handful of aggregate curves — where a naive nightly retrain-everything approach doesn't fit an operational batch window, weather data has to be spatially matched to the right installation, and model drift has to be tracked without an unaffordable retraining cadence.
Decisions I'd make
Distribute per-series model fitting with pandas UDFs, repartitioned by series key with Adaptive Query Execution explicitly disabled — AQE's automatic partition coalescing fights you when each task's unit of work is intentionally small and independent. Select a daily "champion" model per series, from an ensemble of Elastic Net, LightGBM, and Prophet, by prior-day error rather than one globally tuned model — and set an annual, not monthly, retraining cadence backed by a daily metrics workflow that gives early warning if drift outpaces it.
What this looks like built
Bronze/Silver ETL landing smart-meter data, a weather nearest-neighbor join, and channel aggregation; a three-model ensemble with Bayesian (Optuna) hyperparameter search over sliding-window cross-validation; daily per-series champion selection for inference.
Result
Industry-reported figures from the source pattern: ~200,000 load diagrams forecast daily, a ~14.5 billion-record training corpus, ~3-hour daily inference on a 40-node cluster, and a reported potential to increase grid hosting capacity by 20%+ in some cases.
Technologies
Azure Databricks · Delta Lake · PySpark · pandas UDFs · LightGBM · Prophet · Optuna · Azure Data Factory · Azure DevOps
Turning a client-facing report from prototype to production
Modeled on a publicly documented Databricks implementation pattern from a demand-side advertising platform — not a client engagement of mine. Included to show how I'd approach a comparable problem.
Context
A demand-side advertising platform running at billions of daily tracked events, where the internal report lifecycle typically stalled between "compelling proof of concept" and "reliable, always-current product feature" — not for lack of a good idea, but because the data engineering underneath a scrappy prototype rarely survives contact with live production data.
Architecture problem
Computing a statistical index across user-segment membership and site-visit data at combinatorial scale — one user can belong to many segments and visit many sites — keeping a rolling 30-day membership state fresh without reprocessing full history daily, and writing frequently refreshed results into a relational database backing a live client-facing app without downtime.
Decisions I'd make
Solve the genuinely irregular nested-join logic in Spark's lower-level RDD API, then hand off to DataFrames the moment the data becomes regular again, rather than forcing an inherently ragged transformation into SQL. Use a stage/rename table-swap pattern for every write into the live-serving table instead of a blocking overwrite.
What this looks like built
A per-user key/value "mini feature store" (segment membership as sequence files), joined and aggregated into the report's index, written via the Spark JDBC connector into PostgreSQL — the whole backend running from a single Databricks notebook and job.
Result
Industry-reported from the source pattern: the pipeline condenses hundreds of terabytes of rolling data into a single, always-current report, on a platform tracking 4+ billion daily events, with the entire backend consolidated into one notebook and job — no separate cluster-ops tooling.
Technologies
Databricks · Apache Spark (RDD + DataFrame) · PySpark · Amazon S3 · PostgreSQL (RDS) · Spark JDBC
Breaking down data silos: democratizing data science
Modeled on a publicly documented implementation pattern at a large, long-established education technology company — not a client engagement of mine. Included to show how I'd approach a comparable problem.
Context
A large, long-established education technology company where real data science capability existed but was locked inside per-team silos — getting research-ready data meant one-off, manually PII-scrubbed flat-file extracts that went stale almost immediately.
Architecture problem
Exposing fresh, current data to researchers securely without repeated manual extraction, operationalizing a research-notebook model — a student-attrition classifier — against live production data without a heavy bespoke MLOps build, and doing it all while an expensive, always-on leased Spark footprint was being phased out.
Decisions I'd make
Re-architect the access path itself — secure read-replica connections, encrypted object storage, one shared compute layer — rather than treating "democratization" as a permissions change. Give every persona (engineer, researcher, analyst) one governed path to the data instead of a bespoke pipe per team.
What this looks like built
Secure, encrypted read-replica connections into the analytics platform; encrypted S3/Parquet storage queryable via SparkSQL by researchers and analysts alike; and an operational notebook writing live model predictions back into the same platform researchers already used — closing the loop from research to production.
Result
Industry-reported from the source pattern: legacy always-on leased Spark infrastructure was decommissioned in favor of dynamic, on-demand provisioning, reported to cut analytics infrastructure operating cost by roughly 30%.
Technologies
Spark-based unified analytics platform · Amazon S3 · SparkSQL · Single sign-on · Job-scheduler model operationalization
Have a similar problem?
Let's talk through what's actually going on before deciding what to do about it.