Principal Data Architect & DevOps Lead

ProductionOwn tooling · built during BBI.ai client engagements2025

Automated readiness checks for Cloudera installs

Ansible roles that check, and only on request fix, every host before a cluster install, then report the results as a spreadsheet.

RunnerHostsCSV

Run the readiness checks

Illustrative · synthetic dataSynthetic hosts and invented results.

HostPackagesOS settingsDNS and FQDNTime syncJavaPort matrixDisk space
demo-mn01
demo-mn02
demo-wk01
demo-wk02
demo-wk03
demo-wk04

Press “Run checks” to validate six synthetic hosts. Checks only read; nothing changes until you opt in to remediation.

The run as text
  • demo-mn01: ready
  • demo-mn02: fails DNS and FQDN
  • demo-wk01: fails Java
  • demo-wk02: fails OS settings
  • demo-wk03: fails Port matrix
  • demo-wk04: unreachable, still reported as a row

Remediation (opt-in) fixes: OS settings, Java. Other failures are flagged for the network team.

Major challenges

  1. ~40 checks

    Every host, every time

    About 40 facts per host, written to one CSV row per host, including unreachable ones.

  2. All-to-all

    Firewall gaps before install

    A port matrix covering every CDP service port between every pair of hosts.

  3. Read-only

    Safe on client estates

    Validate by default; remediation runs only when asked and fails early without privilege.

  4. 1 click

    Run by anyone

    A GitHub Actions dropdown runs any check role on a self-hosted runner, secrets from GitHub Secrets.

My contribution

Sole author of both generations, used on client engagements.

Tools used

  • Ansible
  • GitHub Actions
  • Ansible Vault
  • Python
  • Bash
  • RHEL 9
  • nmap

Outcomes

  • About 40 checks per host, written to one CSV row per host, including hosts that couldn't be reached.
  • An all-to-all port matrix covering every CDP service port.
  • Separate validate and remediate roles, so a check never changes a host by accident.
ArchitectureAnonymised component diagram

Trigger

  • GitHub Actions (manual dispatch)Choose a role from a dropdown
  • Self-hosted runnerVault password from GitHub Secrets

Validate

  • Packages and OS settings
  • Network, DNS and directory
  • All-to-all port matrix
  • About 40 host factsCPU, memory, disks, Java, time sync

Evidence

  • CSV report per host
  • Job summary diagram

Remediate (opt-in)

  • Remediate roleFails early without privilege
Two generations of the tool. The first added checks per area; the second split validate from remediate.
Read the flows as text
  • GitHub Actions (manual dispatch) → Self-hosted runner
  • Self-hosted runner → Packages and OS settings
  • Self-hosted runner → Network, DNS and directory
  • Self-hosted runner → All-to-all port matrix
  • Self-hosted runner → About 40 host facts
  • Packages and OS settings → CSV report per host
  • Network, DNS and directory → CSV report per host
  • All-to-all port matrix → CSV report per host
  • About 40 host facts → CSV report per host
  • CSV report per host → Job summary diagram
  • CSV report per host → Remediate role (on request)
Full case studyProblem, decisions, implementation, rollout

Problem and constraints

Every Cloudera install starts with the same question: are these servers actually ready? That means kernel settings, packages, users, DNS, time sync, directory and database connectivity, and open ports between every pair of hosts. Checking all of that by hand across dozens of servers for each client is slow and easy to get wrong.

  • Client estates. A readiness check must never change a host unless someone explicitly asks it to.
  • Readable evidence. Project managers and client teams need results they can read without Ansible.
  • Secrets. Directory and database checks need credentials, which must stay out of the repository.

My role and the team’s

I wrote both generations of the tool myself and used them on client engagements.

Key decisions and trade-offs

  • Validate and remediate are separate roles. The default run only checks, and fixes are opt-in. Running twice is slower, but safety on someone else’s estate matters more.
  • CSV over a dashboard. A spreadsheet is less impressive, but it can be emailed, filtered and attached to a sign-off.
  • One row even for unreachable hosts. A missing host shows up as a failed row instead of silently disappearing from the report.

Implementation

First generation. Four Ansible roles covering packages, OS services, network and DNS (including directory binds and database connections), and an all-to-all port matrix built from a definition of every CDP service port. A GitHub Actions workflow lets an operator pick a role from a dropdown. It runs on a self-hosted runner, injects the vault password from GitHub Secrets and renders a diagram in the job summary.

Second generation. For RHEL 9 and CDP 7.1.9:

  • A validate role gathers about 40 facts per host: CPU, memory, OS, kernel, cgroups, file systems and free space, Java, Python, time sync, SELinux, firewall, IPv6, DNS, swappiness, transparent huge pages, ulimits, SSH reachability, repository and database reachability, packages and disk layout.
  • A remediate role runs in “safe mode”. It fails early if privilege escalation doesn’t work, installs Java 11 and sets JAVA_HOME, and applies the kernel, firewall, SELinux, swappiness, transparent-huge-pages and limits settings Cloudera expects.

Companion scripts bootstrap the service databases and generate certificate requests with SANs for every host.

Testing and rollout

Developed against lab hosts before use on client estates. The validate-only default means a first run on a new estate is always read-only.

Verified outcomes

Readiness checks that used to be a manual checklist now run as a repeatable job and produce a report, and the tool is reused across engagements.

Domains CI/CD & automation · Data platforms · Observability & reliability · Security & governance

Trace it in the constellation

Code The repositories hold client-specific details, so they aren't linked yet. A cleaned-up public version is planned.

Try:

↑ ↓ to move · Enter to open · Esc to close