Platforms

Streaming & ingestion

Getting data in: Kafka at telecom scale, NiFi flows from files, databases and streams, and change-data-capture on a government platform.

More detail

Kafka Production
Designed and configured a Kafka setup of around 150 nodes in a single cluster, using config groups and ZooKeeper znodes, and monitored it with Kafka-HQ and Grafana.
NiFi Production
Configured HDF, CFM and CDP NiFi clusters for batch and streaming flows from HDFS, Kafka, SFTP and SQL sources into Hive, Azure ADLS and Elasticsearch, with content and provenance repositories sized for the volume.

→ Data platforms: Kafka and NiFi land events and files in the lake.
← Observability & reliability: Brokers and flows are monitored and alerted on.

Data platforms

The centre of my work: Cloudera and Hadoop platforms I've administered, migrated, secured and handed over since 2015.

More detail

Cloudera CDP Production
Migrated a telecom data lake from HDP to CDP, ran the bank's analytics platform on it, and now install, secure, upgrade and hand over CDP clusters for clients. Cloudera CDP Private Cloud Base certified.
Hadoop (HDFS, YARN, Hive, Ozone) Production
Ten years administering Hadoop. At Jio that meant NameNode federation, HDFS erasure coding and Ozone. Earlier work covered Hortonworks clusters, and more recently a documented open-source Hadoop build with Kerberos.

← Streaming & ingestion: Kafka and NiFi land events and files in the lake.
→ Processing & workflows: Spark jobs read raw lake data and write curated layers back.
← CI/CD & automation: Ansible checks and configures every host before and after a cluster install.
→ Cloud & infrastructure: Hadoop workloads re-platformed onto Azure managed services.
← Security & governance: Kerberos, Ranger and TLS protect data and access at every layer.

Processing & workflows

Turning raw data into curated tables: Spark and PySpark pipelines, scheduled and retried by Oozie and Airflow.

More detail

Airflow Production
Part of the big-data delivery toolchain I designed and configured at Jio. Airflow handled workflow management, Jenkins built artefacts, Rundeck managed deployments and Nexus versioned the builds. Airflow's job was dependency-aware scheduling of platform and data workflows.
Oozie Production
Scheduled a government platform's production PySpark use cases as Oozie workflows, so operators could monitor and rerun them with tooling they already knew.
Spark & PySpark Production
Ran Spark at platform level, packaged Spark jobs as containers at the bank, and built the layered PySpark pipelines behind a government platform's production use cases.

← Data platforms: Spark jobs read raw lake data and write curated layers back.
→ AI/ML enablement: Curated, governed data is what ML workloads consume.
→ Containers & orchestration: Spark jobs packaged as containers carry their own dependencies onto YARN or Kubernetes.

AI/ML enablement

The platform side of AI: running containerised ML workflows and Cloudera Machine Learning on governed data. Experiments are labelled as such.

More detail

Governed AI access (exploration) Experiment
Designing AI access to enterprise data through the platform's own policies, lineage and audit, shown in proposals and an interactive demo.
Cloudera Machine Learning Platform support
Administer Cloudera Machine Learning alongside CDP for data science teams.
Containerised ML workflows Platform support
Deployed containerised AI/ML model workflows on the CDP Private Cloud cluster for the analytics team, using the same container and CI/CD path as other jobs.

← Processing & workflows: Curated, governed data is what ML workloads consume.
← Containers & orchestration: Model workflows run as containers on CDP Private Cloud.

Security & governance

Kerberos, Ranger, Knox and TLS on every cluster, plus audit controls such as alerting on privileged logins.

More detail

Kerberos, Ranger, Knox & TLS Production
Kerberised and hardened clusters: MIT Kerberos, SSSD with LDAP/AD, TLS and Auto-TLS, Ranger policies, Knox gateways and Atlas tags. Wrote an alerting service for privileged cluster-manager logins.

→ Data platforms: Kerberos, Ranger and TLS protect data and access at every layer.
→ CI/CD & automation: Security scans are a mandatory pipeline stage.
← Observability & reliability: Privileged logins raise an alert and leave an audit trail.

CI/CD & automation

Pipelines and configuration as code: GitHub Actions, Jenkins, Azure DevOps and Ansible, with quality and security gates.

More detail

Ansible Production
Configuration management for Linux and Windows. Wrote separate Ansible validate and remediate roles that check about 40 settings per host before a Cloudera install, plus an all-to-all port matrix.
Azure DevOps Production
Implemented continuous integration and release automation with Azure DevOps alongside Jenkins.
GitHub Actions Production
Designed CI/CD for a bank's big-data estate with SonarQube quality gates, Checkmarx scans and Nexus artefacts, plus on-demand self-hosted runners. At BBI.ai, GitHub Actions triggers Ansible readiness checks.
Jenkins Production
Built Jenkins pipelines for Maven and SBT artefacts and administered Jenkins: agents, distributed builds, plug-ins and job configuration.
Python & Bash Production
Platform maintenance scripts, a systemd alerting service, certificate (CSR) tooling and database bootstrap scripts.
SonarQube, Checkmarx & Nexus Production
Quality gates, security scanning and versioned artefacts as mandatory pipeline stages for big-data code.

→ Containers & orchestration: Pipelines build, scan and publish the images that containers run.
→ Data platforms: Ansible checks and configures every host before and after a cluster install.
← Security & governance: Security scans are a mandatory pipeline stage.
← Observability & reliability: Readiness checks and runbooks gate every change.

Containers & orchestration

Docker and Podman for big-data jobs on YARN, Kubernetes support (CKA 2020), and workloads packaged for OpenShift.

More detail

Docker & Podman Production
Built a framework to run Spark and big-data jobs in Docker/Podman containers scheduled by YARN on CDP, and moved server-based workflows onto it. Used Podman for container-in-container GitHub Actions runners.
Kubernetes Platform support
Supported production Kubernetes clusters and ran proofs of concept at Jio. At the bank I orchestrated containerised big-data jobs through YARN and Kubernetes. Certified Kubernetes Administrator (CKA) in 2020; the certification expired in 2023.
Red Hat OpenShift Platform support
Containerised Spark and big-data workflows with Docker/Podman for deployment and orchestration on OpenShift and Kubernetes. OpenShift administration: training in progress.

← Processing & workflows: Spark jobs packaged as containers carry their own dependencies onto YARN or Kubernetes.
← CI/CD & automation: Pipelines build, scan and publish the images that containers run.
→ AI/ML enablement: Model workflows run as containers on CDP Private Cloud.
← Cloud & infrastructure: Cloud and virtual infrastructure host the runtimes.

Cloud & infrastructure

Azure re-platforming, hypervisor migration and the RHEL hosts underneath it all.

More detail

Azure data services Production
Worked on re-platforming a three-tier Hadoop application to Synapse, Blob Storage, Azure SQL and Event Hubs. Ran NiFi flows into Azure ADLS. Worked with ExpressRoute connectivity.
Terraform Experiment
Modular Terraform landing zone for an Azure analytics demo: ADLS Gen2, Synapse, Data Factory, Databricks, HDInsight and private endpoints.
RHEL, VMware & Nutanix Production
Build and harden RHEL hosts for Cloudera installs, and moved a live VM between hypervisors in a migration proof of concept.

→ Containers & orchestration: Cloud and virtual infrastructure host the runtimes.
← Data platforms: Hadoop workloads re-platformed onto Azure managed services.

Observability & reliability

Monitoring, alerting, readiness checks, runbooks and rollback plans that keep large platforms predictable.

More detail

Grafana, Prometheus & alerting Production
Monitored a large data lake with Check-MK, Grafana and Prometheus, Dr. Elephant for Spark, Kafka-HQ, ElastAlert and a custom KDC monitor.
Runbooks, readiness & rollback Production
Gated upgrade runbooks with validation and rollback steps, readiness checks before installs, and reconciliation checks before sign-off.

→ CI/CD & automation: Readiness checks and runbooks gate every change.
→ Streaming & ingestion: Brokers and flows are monitored and alerted on.
→ Security & governance: Privileged logins raise an alert and leave an audit trail.

Engineering constellation

Streaming & ingestion

Data platforms

Processing & workflows

AI/ML enablement

Security & governance

CI/CD & automation

Containers & orchestration

Cloud & infrastructure

Observability & reliability

From source to serving

Data arrives

Landed untouched

Arrival triggers work

Code arrives through gates

Spark transforms

Layered storage

Served to consumers

Secured at every layer

Watched end to end