Automating an HDP Cluster with Puppet

What Is It?

A Puppet module from 2018–2019 that automates deploying an Ambari-managed Hortonworks Data Platform (HDP) cluster: installs and configures the Ambari server and agents, sets up PostgreSQL as the backing metastore for Hive and Oozie, and generates a self-signed certificate so the Ambari API runs over HTTPS instead of plain HTTP.

Chef, then Puppet. This wasn't the first time I automated an HDP deployment — I'd done it before with Chef, and this Puppet module was the second pass at the same problem: get Ambari installed, configured, and pointed at a database, without touching a server by hand. The Chef-era version didn't survive to today; this Puppet one did.

The Stack

Built With

Puppet — module structure, Jenkins-built and installed onto a Puppet master
Ambari — cluster management for HDP, driven via its setup CLI and REST API
PostgreSQL — backing metastore for Hive and Oozie, provisioned with dedicated users per service
OpenSSL — self-signed certificate so the Ambari API runs over HTTPS
Jenkins — lint (puppet-lint), build the module package, then scp + ssh install it onto the Puppet master

What Actually Happens on a Node

Broken down by manifest, roughly in the order it runs:

Every Node (`hdp::default`)

Disable Python's SSL certificate verification — needed because the Ambari agent's Python-based tooling would otherwise reject the self-signed cert set up later.
Install java-1.8.0-openjdk.
Pull in Hortonworks' Ambari yum repo (2.5.1.0, CentOS 7).

Worker Nodes (`hdp::agent`)

Install ambari-agent.
Rewrite the agent's config so it points at the Ambari server's hostname instead of localhost.
Force PROTOCOL_TLSv1_2 in the agent's security settings.
Start and enable the ambari-agent service.

The Ambari Server (`hdp::server`)

Install ambari-server and run its silent setup.
Download the PostgreSQL JDBC driver and point Ambari at it instead of the default embedded database.
Override the default API port.
Generate a self-signed cert — write out the openssl req prompts as a canned answers file, then genrsa → req → self-sign, and enable HTTPS on the Ambari API with it.
Start the ambari-server service.
Create dedicated PostgreSQL users and databases for Hive and Oozie.

Scala, Spark, and This

This overlaps with when I was deep into Scala and Spark — the 2016 USA election analysis with Spark GraphX is from a couple years earlier, but HDP and Spark were the same world: standing up and babysitting a cluster was the unglamorous prerequisite to actually being able to run Spark jobs on one. Also see the earlier note on Chef from around the same era, for the tool this module eventually replaced.

The source is open, at jane-natucci/my-puppet-hdp.