What It’s Actually Like to Build a Puppet Module

[0:00]Ben Ford: Hello again everybody. Welcome to today's episode of the Pulling the Strings podcast, as always, powered by Puppet. My name's Ben Ford, I'm our developer relations director here at Puppet, and I'm pretty active in the community as @binford2k. We may have run into one another every now and then. Today we're talking with Yeshua. They've been working in open source and web development and other interesting, fascinating things well over 10 years and I see that there's a big list of some projects that you've been involved with. So I'm curious to know... I've never actually worked on an IBMI machine. Could you tell me a little bit about that? 

[0:55]Yeshua Hall: Sure, yeah, and I had never even heard of IBMI in my early career. I started with being a full on Linux geek for a long time, and while I was living in Sioux Falls, South Dakota looking for a job, one of the best offerings I got was at this IBMI shop. And again, even when I applied to that, I hadn't heard of the IBMI, but- 

[1:29]Ben Ford: I bet they laughed a little bit and they said, don't worry, we'll teach you. 

[1:31]Yeshua Hall: Yeah, basically. That's basically how it ended up. And it was really funny because when I first got there and everything, you get this... They showed me this archaic, I shouldn't say that word, but IBMI people might get upset, but what they call the green screen, which is the terminals that people use at banks and stuff. And it's just, it's this really practical business driven machine that uses a language called RPG to produce reports and stuff based on finances, et cetera. So IBMI's are very ubiquitous, especially in the business world. The government runs on them, local government has IBMI's, a lot of hospitals, insurance companies, even railroads, et cetera, they all usually have an IBMI driving the business aspect of everything. So it was really fascinating when I got there, the types of systems I got to work on. My previous jobs before that were what I like to call WordPress assembly lines. 

[2:52]Ben Ford: Oh yeah, I'm familiar with that. 

[2:53]Yeshua Hall: Yeah. And those can get really tedious and repetitive and it was really funto... A lot of IBMI stuff is dealing with integrations from the bare bones business world to newer types of technology and software as a service. So it was a lot of fun once I got into it and understood it and then around that same time that I joined up on it, open source became a huge deal on IBMI. And I like to put it this way, when I was younger and started my work in software, I've always been a big proponent of open source, but I always felt like, man, I wish I could have been part of the wave back in the day, part of Linus Torvalds and all of them kind of fighting the system. 

[3:52]Ben Ford: That's wild. 

[3:54]Yeshua Hall: If you look at the history, how many waves we've gone through and realizing how many that we've actually lived through and didn't quite realize at the time. And that's exactly the opportunity I got with IBMI. They were fresh on the open source scene. So it was a lot of fun helping them get acquainted and get open source going. 

[4:17]Ben Ford: You had your list of things that you did is making it more open source feeling, making it by integrating Bash and using an actual familiar command line interpreter. What was that like? 

[4:29]Yeshua Hall: Yeah, so the default at the time you got the corn shell, if anyone's familiar, the sea shell and it had the original born shell as well, which Bash comes from. 

[4:44]Ben Ford: I tried using CSH for a little while. I tried because you can write shell scripts in this very sea like language and it- 

[4:51]Yeshua Hall: Yep. 

[4:51]Ben Ford: Yeah, it just didn't work out. 

[4:54]Yeshua Hall: Yeah, it's got a lot of intricacies that are different. If it was just you could write cool stuff and see, that would be great, but there were a lot... Stuff wasn't very standardized when it came out, so it took liberties. But yeah, they eventually got Bash on the machine, but they just gave you bash no configuration. So for those that haven't ever done that, you Bash all by itself. You don't have read line, you don't have tab completion, you don't have a history configured. It's just basically a shell that you can interact with commands and that's it. So just like everybody has their Linux dot files available if they're nerdy enough, I created an RPM that IBMI people can install and use it to manage dot files for users and it will configure all those things that I just said. So that basically when you SSH into an IBMI, it feels like you're using Ubuntu basically. There's not much different other than system commands and database interaction. Other than that, you feel like you're on Ubuntu. 

[6:09]Ben Ford: Maybe IBMI ought to consider integrating that themselves. Yeah? 

[6:13]Yeshua Hall: It would be... So it's interesting that you bring that up. I've talked even before I released my doc files, I talked to them about that a lot. It all boiled down to licensing because IBMI's proprietary system that's very commercial and enterprise. They were worried, they were like, sure we can ship Bash, but they were worried about shipping their own configuration and stuff with Bash and they were worried about the legalities even though I personally don't think there's anything that could happen, they didn't want to risk it. So that's why that doesn't exist. 

[6:48]Ben Ford: I mean at least they were paying attention to open source licensing. A lot of companies don't. 

[6:51]Yeshua Hall: Yeah, I know, right? I was actually really happy about that because I was like, well that's good. You're being cognizant of what you're doing. That's good. 

[7:00]Ben Ford: So now at Perforce, you're a senior solutions architect and means you get to build really cool stuff for your customers. What's that job like? 

[7:11]Yeshua Hall: Yeah, so a lot of it has to do with proof of concepts around our Zen PHP or just PHPLTS offerings and stuff. And I get to do some R and D, research and development, et cetera, which is one of the reasons I got to play around with Puppet a lot. But yeah, I basically help customers understand how our products can fit into their full stack and sometimes that involves also architecting the stack they will be using and such. Hence the title. 

[7:50]Ben Ford: Right on. Well, that brings us up to our actual topic of the podcast here and that's the Puppet module that we've worked on together. So I had the pleasure of mentoring and showing Yeshua around a little bit as they were first introduced to the world of Puppet and how it worked and how you managed systems and that ended up resulting in some modules that managed PHP and some contributions to an upstream project and whatnot. Could you maybe talk a little bit about how that worked and what that experience was like from your perspective? 

[8:29]Yeshua Hall: Oh yeah. Well Ben was super helpful obviously being in the position you're in, and basically pointed me around a documentation and stuff. Now to be transparent, I started with wrapping my head around Puppet Bolt, which is just a really fantastic introductory to the Puppet world. I feel like a lot of other complex systems could benefit from a... I don't even know what you just sort of a dumbed town. 

[9:06]Ben Ford: It's like the on-ramp is much lower, it's you can go from zero to something really, really quickly and then you can just follow the steps as you learn more, right? 

[9:16]Yeshua Hall: Yeah. It integrates so well with stuff that you already have laying around. So if you've already built deployment scripts in Bash or any other Python whatever you want, it's really easy to just plug and play. You can just define a puppet task around... A Bolt task, sorry, around that script with the parameters and such that you want and you just run a Bolt command and it goes, does that. It also at the base is agentless, which I think lowers the on-ramping a lot. So you're basically just streamlining the commands that you would've been running previously, but within Bolt. 

[10:06]Ben Ford: You can take that same thing and just replicate it across your whole infrastructure. 

[10:10]Yeshua Hall: Exactly. Plus you can string them together. That's the word that was escaping me. So especially if you made your scripts really modular, you could build a billion different things in Bolt using each of them. So that was a ton of fun. I got a reasonable template going with the scripts and stuff that I already had and that helped me wrap my head around what my goal really was, which was to build some Puppet modules that handled that. And Ben was really helpful at basically pointing out how Puppet can handle most of the things I was doing in scripts. So I was able to abstract those away and just have my Puppet language handle the systems that I needed. And yeah, it's really easy to modify things, really easy to maintain, really easy to contribute back to modules that's like the community PHP module. It was really easy in my opinion, to just sort of modify it a little bit to fit with Zend PHP. 

[11:24]Ben Ford: And one of the... Actually a real quick tangent, you talked about building things with Bolt and stacking a whole bunch of things together and recombining it and whatnot. Just an aside, one of the things that the solution architects here at Puppet worked on is a Bolt based installer for PE, so installer and manager. So now the next generation of our installer is actually going to be run by Bolt, so it'll be orchestrated across your whole infrastructure and whatnot. 

[11:52]Yeshua Hall: That's awesome. 

[11:54]Ben Ford: Super, super cool. And it's really powerful and flexible and lets you kind of recombine how the system is set up. So depending on whether you have just one server or if you have several compilers behind a load balancer, it'll be able to do a lot of that just kind of transparently. 

[12:10]Yeshua Hall: That's awesome. 

[12:11]Ben Ford: Yeah, totally unrelated to our conversation, but you said that and I was like.. It is super powerful. Let me talk about why. So one of the first things that we did after we figured out what the needs were for this module and why you were working on it and what it was for is identify that there was this community PHP module that our vox populi group has been maintaining. And so rather than reinvent everything yourself, you shifted gears to look at that module and see if you could help take that module to the next level, do we need it. What was that experience like? 

[12:52]Yeshua Hall: Getting into the code was fairly straightforward. I like the way Puppet's naming schemes and stuff is all very straightforward and self-documenting, which is a great thing about the language. And so I just dug around and there were basically just a few places because the way the Puppet module for PHP works is it just basically figures out what operating system you're running on and installs the proper repositories, binary repositories necessary and runs through the configurations that you explained that it needs. So that boiled down to me needing to create... I had to create a Zend module to define our repositories for the binaries and another module that installs and manages Zend HQ, one of the enterprise solutions that comes with our PHPLTS because that would've muddied up the Puppet PHP module. So it was really great. Once I got those modules created, all I had to do was add a couple of flags to the PHP Puppet module that would basically say, use the Zend repositories instead and here's some credentials if you need the credentials to get access to the repository. 

And that was it. Besides that, I had to obviously fill in the tests for those different variants and that was another thing I really liked. I feel like it probably has more to do with Ruby maybe or something. But the testing... For the first time in my career, I actually liked writing unit tests, so if that tells you anything. 

[14:59]Ben Ford: That's really neat to hear. We kind of get down on ourselves a little bit because the testing ecosystem is so complex and it's like there's so many things you can do, but it's useful to be reminded that the base of what it is if you don't need to do all the weird wacky things, is actually not that bad. 

[15:16]Yeshua Hall: It's straightforward. 

[15:17]Ben Ford: Yeah. I like the R spec testing language and I like how we've adapted it into Puppet. It makes it pretty descriptive and the syntax is pretty clear just to say, this is what I expect there to be. So I also writing a unit test in for Puppet. PDK itself makes a lot of it easy too. 

[15:35]Yeshua Hall: PDK is awesome. 

[15:37]Ben Ford: Was that your first exposure to the PDK? 

[15:41]Yeshua Hall: Yeah, it was for my modules. Unfortunately, the Puppet PHP module isn't yet PDK ready, but I know that they're working on it. 

[15:52]Ben Ford: Oh, that would be good. That would be useful. 

[15:54]Yeshua Hall: Yeah. 

[15:54]Ben Ford: Maybe we can help them along a little bit. 

[15:56]Yeshua Hall: Yeah. 

[15:57]Ben Ford: Kind of a meta question here. You've talked about how so many of these things are pretty intuitive and whatnot. Is there anything that you found that wasn't intuitive or maybe something that you would've liked to know before you got started or any big major roadblocks or anything that you stumbled into? I like that that's a hard question. 

[16:18]Yeshua Hall: Yeah, it actually is a tough question. I mean, the documentation is just so I would say it's in my top three, maybe even my top. Documentation is just so straightforward and tells you what you need to know and it's not muddied. The examples are fantastic too. Y'all have some great examples in the documentation that cover pretty much every piece that you could do. 

[16:43]Ben Ford: They are a pretty excellent team. 

[16:45]Yeshua Hall: I would say... It took me a while to not understand that PDK was relatively newish and that some modules at the moment aren't PDK standardized. So it took... I was messing around for two or three days trying to get PDK running, and then I finally realized, oh, it has to be PDK— 

[17:10]Ben Ford: It has to be ready for it. 

[17:10] Yeshua Hall: Yeah. It has to be ready for PDK before you can use it. But I mean, besides that, I didn't really run into anything. 

[17:20]Ben Ford: [inaudible 00:17:19] to help them clarify that or maybe even make some of the error messages that the PDK gives. 

[17:26]Yeshua Hall: Yeah, maybe that, yeah. The error messages would be nice. 

[17:29]Ben Ford: Cool, cool. Well, what do you think you would recommend for other people who want to try and take these same steps that you did? Is there a pathway that is the easiest or do you have any tips or tricks for them? 

[17:42]Yeshua Hall: Yeah, definitely just try modifying... I'm trying to think of do a lot of people already have pipelines yet, but getting your already running scripts and stuff into Puppet Bolt is a fantastic way to mess around and understand the ecosystem. And then from there it's basically just a matter of... Because the thing is when you're using Bolt without the apply command, it's just basically scripting and then you can mess around with the apply command, which is basically it'll run out there to your target server. And is it setting up the full agent or just sort of— 

[18:28]Ben Ford: Yeah, it’ll— 

[18:29]Yeshua Hall: minified?  

[18:30]Ben Ford: So when you use the apply sub command in Bolt, it basically manages the installation of the Puppet agent for you and sort of does it in the background. So you don't need to know or care. And then it will take that bit of Puppet code that you provide and let that agent apply it for you. 

[18:49]Yeshua Hall: Okay. Yeah. 

[18:50]Ben Ford: It's really cool. You can take scripts and you can do part of your orchestration with some scripts and then part of it with a little bit of Puppet code and then maybe finish up with, I don't know, a database schema migration or something in another script and you can just blend all these things together. 

[19:06]Yeshua Hall: Yeah, yeah. And that's the beautiful thing about it and that's what helped me wrap my head around what Puppet could and could not do for me. You know what I mean? Because I was stringing together each piece, and at first I had no apply sub commands, and then I just slowly started migrating all that I was doing into apply sub commands. And it makes a surprising difference because when you're doing those applies and stuff, it's actually checking if that thing already exists and may or may not do it depending on... It's just basically fixing any configuration drift and making sure everything was installed and stuff, but it's not going to do it twice, which was really nice. 

[19:57]Ben Ford: That's the cool thing about the declarative model. I always make the joke that it's doing a blueprint for your house or something. You don't need to tell your contractor, go build this wall and then this wall and then this wall. You just say like, "Hey, this is what I want it to look like. Go figure out how to make it." 

[20:14]Yeshua Hall: Right. Yeah. 

[20:16]Ben Ford: All right. Well as we close up, I think kind of the thread that I'm hearing, the most important part is that learning to write Puppet code is surprisingly easy and it's like, especially if you use some of the things that are out there, if you take a look at Bolt and use that to slowly adopt some of your existing code or your existing practices and just make that OnRamp to Puppet a lot shallower, it's really easy to pick up over time how to build things and it might even be fun. It sounds like you had a lot of fun doing this. And then the second thing I'm hearing is that contributing code, rather than building your own code, going into an existing ecosystem and finding something that you can improve often it's a really, really fast way of getting to a solution rather than the not invented here, building the whole thing all by yourself. Would that kind of sum up what we talked about you think? 

[21:11]Yeshua Hall: Exactly that. Yep. And I did have a ton of fun. Puppet has... It's quickly risen to my favorite configuration management. I like it a lot. 

[21:24]Ben Ford: Well, that is really cool to hear. I suspect that we'll be working together a lot more in the future. Well, and that's a wrap for today. So thank you all for being here with us and thank you Yeshua for being here on the Pulling the Strings podcast powered by Puppet.