Originally published at https://puppet.com/blog/hiera-hierarchies-and-custom-facts-everyone-needs on 8 July 2016
---
---
NOTE: This article is targeted at versions of Puppet which utilize Hiera 3 or higher and Facter 3 or higher — generally Puppet 4, or Puppet Enterprise 2015.2 and later versions. Some of this information will also apply to older versions.
For the last year and a half, I've been representing Puppet as the technical solutions engineer covering all the accounts headquartered in Silicon Valley. This has been a fantastic opportunity to evangelize configuration management to clients both new and old. One of the areas I've noticed every new Puppet user runs into quite quickly is how to utilize Hiera effectively to manage separation of data from code. On a fresh install of Puppet Enterprise 2015.2 or greater the hierarchy is pretty simple:
On the plus side, Hiera, like nearly all of Puppet, is very customizable and can be tweaked to each individual organization's needs. For new adopters, all of that power can be confusing. I've noticed I nearly always recommend the same ideas, so it seemed only fair to share those as a blog.
Hiera is powerful, but it has some limitations. Notably, there is a functional limit to the number of layers which can be added until performance begins to take a hit, and in Puppet Enterprise versions before 2016.2, only one hierarchy could be used at a time for the entire compile master.
Because of this, it makes sense to focus the hierarchy on generic concepts instead of specifically referencing unique items of the business unit or workflow. The hierarchy which I've recommended the most is along the lines of this:
The first objection I usually hear is something along the lines of, "But we need a layer for X." To that, I challenge clients to consider whether they really need the additional layer, or if it fits into one of the existing layers. The vast majority of the differentiation they want tends to fit into the application layer. Having a ton of different applications with some data overlap is preferable to having too many layers with too little differentiation.
The conversation moves quickly from there to how to create the facts for %{::team}, %{::application} and %{::datacenter}. While there isn't a universal answer for how to create these facts, often one of a few possible methods will solve this problem for the vast majority of organizations.
Ultimately, the goal is to find a way to programmatically determine the answer to, "What is the X for this node?" To do this, we look to the pieces of information that are already a part of or attached to the node. I'll outline these approaches below.
These examples are custom Ruby facts; they can be added into any module in the
The following code creates Facter facts for the EC2 region and the EC2 tags (credit to Chris Barker and Adrien Thebo for this code snippet):
Generally, facts files work best when they are created by the provisioning system at initial provisioning. Most workflows with VMware vRealize Automation or Cisco UCS Director use this method to pass information from the provisioning system to Puppet. It's relatively easy in any of these systems to create a file with the appropriate values on the provisioned system, then install Puppet and let it handle the rest. An example of a file that might get created is below:
facts.txt
Chris Matteson is a senior technical solutions engineer at Puppet.
For the last year and a half, I've been representing Puppet as the technical solutions engineer covering all the accounts headquartered in Silicon Valley. This has been a fantastic opportunity to evangelize configuration management to clients both new and old. One of the areas I've noticed every new Puppet user runs into quite quickly is how to utilize Hiera effectively to manage separation of data from code. On a fresh install of Puppet Enterprise 2015.2 or greater the hierarchy is pretty simple:
:hierarchy:
- nodes/%{::trusted.certname}
- commonOn the plus side, Hiera, like nearly all of Puppet, is very customizable and can be tweaked to each individual organization's needs. For new adopters, all of that power can be confusing. I've noticed I nearly always recommend the same ideas, so it seemed only fair to share those as a blog.
Hiera is powerful, but it has some limitations. Notably, there is a functional limit to the number of layers which can be added until performance begins to take a hit, and in Puppet Enterprise versions before 2016.2, only one hierarchy could be used at a time for the entire compile master.
Because of this, it makes sense to focus the hierarchy on generic concepts instead of specifically referencing unique items of the business unit or workflow. The hierarchy which I've recommended the most is along the lines of this:
:hierarchy:
- nodes/%{::trusted.certname}
- team/%{::team}
- application/%{::application}
- datacenter/%{::datacenter}
- common- The name of the node
- Who owns the node
- What the node does
- Where the node is
- Metadata common to all nodes
The first objection I usually hear is something along the lines of, "But we need a layer for X." To that, I challenge clients to consider whether they really need the additional layer, or if it fits into one of the existing layers. The vast majority of the differentiation they want tends to fit into the application layer. Having a ton of different applications with some data overlap is preferable to having too many layers with too little differentiation.
The conversation moves quickly from there to how to create the facts for %{::team}, %{::application} and %{::datacenter}. While there isn't a universal answer for how to create these facts, often one of a few possible methods will solve this problem for the vast majority of organizations.
Ultimately, the goal is to find a way to programmatically determine the answer to, "What is the X for this node?" To do this, we look to the pieces of information that are already a part of or attached to the node. I'll outline these approaches below.
Parse existing fact
Sysadmins have been attaching metadata to servers for a very long time in the form of hostnames. Many organizations still place information such as data center, application and team in the system's name. Facter by default already creates a fact for hostname, so we can parse that existing fact to generate new facts.These examples are custom Ruby facts; they can be added into any module in the
<module>/lib/facter directory as .rb files, and will be copied to all of the nodes via pluginsync and executed. The first example here simply takes the first four characters and turns them into a new fact.Facter.add(:datacenter) do
setcode do
Facter.value(:hostname)[0..3]
end
endFacter.value(:hostname)[4..7]Facter.value(:hostname)[4..-1]Facter.value(:hostname)[5..-1][/(.*?)(\-|\z)/,1]Match value to table
This is ugly, but sometimes it's the best option, particularly when the only way to determine the data center is via IP address. The shortcoming of this approach is that it requires the fact to be updated whenever there is a new potential value. Utilizing a known fact such as the IP network and case statements, we can match up with a value such as data center. Where avoidable, this shouldn't be used just to replace shorthand metadata with full names, as it adds unnecessary complication (such as matching pdx in the hostname and replacing with portland):Facter.add(:datacenter) do
setcode do
network=Facter.value(:network)
case network
when '10.0.2.0'
'portland'
when '10.0.3.0'
'sydney'
when '192.168.0.0'
'home'
else
'unknown'
end
end
endRead metadata
AWS allows VMs to be tagged with metadata that can be read elsewhere. Often, these types of tags are created already for the purpose of charging back the cost of the node to the group responsible for them. This metadata can be turned into custom facts and used with Puppet as well.The following code creates Facter facts for the EC2 region and the EC2 tags (credit to Chris Barker and Adrien Thebo for this code snippet):
Facter.add(:ec2_region) do
confine do
Facter.value(:ec2_metadata)
end
setcode do
region = Facter.value(:ec2_metadata)['placement']['availability-zone'][0..-2]
region
end
end
Facter.add(:ec2_tags) do
confine do
begin
require 'aws-sdk-core'
true
rescue LoadError
false
end
end
confine do
Facter.value(:ec2_metadata)['iam']['info']
end
setcode do
instance_id = Facter.value('ec2_metadata')['instance-id']
region = Facter.value(:ec2_metadata)['placement']['availability-zone'][0..-2]
ec2 = Aws::EC2::Client.new(region: region)
instance = ec2.describe_instances(instance_ids: [instance_id])
tags = instance.reservations[0].instances[0].tags
taghash = { }
tags.each do |tag|
taghash[tag['key'].downcase] = tag['value'].downcase
end
taghash
end
endWrite a custom fact script
Any script that returns a value can be turned into a custom fact. With a Ruby fact, it’s simply a matter of wrapping the script in a setcode and exec statement:Facter.add('hardware_platform') do
setcode do
Facter::Core::Execution.exec('/bin/uname --hardware-platform')
end
endfacts.d folder of any module. Pluginsync will copy the fact, the same as for facts in the lib/facter directory of a module. A simple example:#!/usr/bin/bash
echo testfact=fluffyDrop facts file
When there is no programmatic way to determine the appropriate value, Facter supports the creation of this type of metadata via fact files in/etc/puppetlabs/facter/facts.d. (Note: This folder isn’t pre-created on a default Puppet Enterprise install.)
Many organizations actually prefer this method over the deterministic methods above, because it avoids potential collisions. These files can be in yaml, json or txt format. There can even be executable scripts in this directory, as long as they return key value pairs. Generally I consider txt to be easiest as it's simply:key=valueGenerally, facts files work best when they are created by the provisioning system at initial provisioning. Most workflows with VMware vRealize Automation or Cisco UCS Director use this method to pass information from the provisioning system to Puppet. It's relatively easy in any of these systems to create a file with the appropriate values on the provisioned system, then install Puppet and let it handle the rest. An example of a file that might get created is below:
facts.txt
datacenter=portland
application=doc
team=TSEConclusion
Hiera is one of the most powerful parts of Puppet for enabling reusability of code, but custom facts are a critical component you need to take advantage of that capability. Starting with a sane but simple hierarchy and building a few simple custom facts can greatly accelerate the ability to adopt Puppet across your organization. It's quite likely that you'll need to mix and match, and tweak the examples here to fulfill your organization's needs, but my hope is that this blog will provide sufficient guidance to get you started with Hiera and custom facts.Chris Matteson is a senior technical solutions engineer at Puppet.
Learn more
- Custom fact documentation: https://docs.puppet.com/facter/3.1/custom_facts.html
- Hiera documentation: https://docs.puppet.com/hiera/3.0/
- How Swisscom automates Hiera data creation to help build custom clouds efficiently.
- Download Puppet Enterprise: https://puppet.com/download-puppet-enterprise
