IBM i DevOps TechTalk
Moving to Git – The Build #8

R.B. – Welcome to IBM i DevOps TechTalk, where we discuss key topics and questions with ARCAD experts. I’m Ray Bernardi and I’ll be your host today. We’ll be listening to Alan Ashley, a senior Solutions Architect here at ARCAD software, and discuss the build process that you need for Git to work with the IBM i. I will be discussing that with Jeff Tickner.
He’s our CTO from North America. Jeff has a world of knowledge in the area of Git and how it relates to the IBM i. Let’s listen in.

A.A. – So in the past we’ve heard you talked about features and branches and releases, but I keep hearing you mention this, and I actually heard in one of your comments sessions where you keep talking about the build. What is the build in this aspect of things and isn’t it just a compile?

J.T. – So yes, you’re absolutely right, Alan. There’s a lot of interest in this. I do sessions that come in all the time. I’ve been doing them for years, really, about the build and about Bob and how ARCAD does its build. And so when we started out working with Source Control, we didn’t have a very efficient build and it was slow.
And so one of the challenges, in fact the name of one of my common sessions was I want just the changes. It’s actually pretty hard to build just the changes with Git.

A.A. – So, Jeff, why is it so hard to get the changes? I mean you can boom right from Git?

J.T. – You’d think so but the problem is that Git is from the open systems world and we’re trying to use it in the IBM i. So if I’m a Java developer, when I do a build, what do I build? Everything. I want to see the changes. But I want to build everything. And in the IBM i, we don’t have that choice in building everything.
We could, but who wants to wait all night to see that ACL compile? So the build is a lot more. It’s a build engine and that build engine is trying to retrieve just the changes. I want to compile the source. I changed the course and I want to manage the dependencies which are unique to the IBM i.

Now Bob (Better Object Build), IBM has been supporting that tool, but it’s Makefile based, so it is a make engine that reads a makefile that tells it how to build your objects and what to build, what are the dependencies. But it’s a text file, a very interesting syntactical text file and somebody has to create it and has to maintain it. You need 3 to 5 lines of text for each object. So if I have 10,000 or I’ve had customers, as many as 50,000 members, that’s a pretty good sized text file that somebody has to create and then maintain it as your object relationships change and as you add new objects. So Bob is a nice idea. It’s a working technical demonstration, but it really doesn’t scale that well. ARCAD use their own method to retrieve the changes and we’ve tried several different ways. We started out doing a diff. So that’s a function of Git. I want to see my changes get diff. I say what branches, It tells me what’s changed. We started out that way and as we scaled up our customers didn’t like it.

So if I have a big project and I have 500 changes and I have maybe 2000 recompiles, the diff always tells me the same thing. The diff is between master and my branch here all the differences. There are always the same. I always have to compile 2500 objects for that big project. So we had to redo our build.

We’ve actually rewritten our build engine three times. We’ve been doing this for nine years, so we’re using tags right now. So when you do a build, we’re going to put a tag on the branch and then we essentially do the changes since that last successful build and then manage the dependencies, of course, for those changes.
So if I had that big project, I had to fix ACL in test. I’m compiling just that ACL. The build is pretty quick. There’s lots of places in our processes where we have optimized the process. We used to clone the whole repo. And when you go out in the real world and you had 50,000 members, takes a long time to clone 50,000 members to the IFS, even if you’re just getting the changes, it’s still comparing them all.

Now we do a sparse clone, essentially, which isn’t very easy. It’s a lot faster and we get just the source that changed. So we’ve done a lot of optimization for this process, but the build is the key. If you don’t have an automated build, it’s really hard to leverage the other powerful aspects of Git, which is the merge, the Gitflow, merging one branch into another.
I don’t want to merge my only merge to be back to master to main because it’s too late and going back to main I’m ready to go to production. I don’t want to find out then that I have merge conflicts. I have to change source or that I’m going to do merging hot fixes or something like that and have a significant impact to my functionality.
We want to shift to the left. We want the developer to be able to grab those changes when they’re still working in their library and unit test those and see that we want that two way integration, but then we want to build those changes and when the developer pushes them, we want to automatically build them and put them in a test so we can test them.
If the source changes, test it.

A.A. – I can see where a build is a little bit more than just a compile because there’s a lot that seems to go into this. And you know, in reading some of the things you’ve said in the past, I can really see why IBM came to ARCAD to use our build and our build process in the development of Merlin.

J.T. – Yes, we’ve worked with IBM for a long time, starting back in the RTC days, but we are the only choice. When it came to a build, they looked at Bob. They actually offered Bob as an option in Merlin, but they also know that it’s going to be hard to scale it up for a enterprise company. And we had a tried and tested process.
Our build is used by the largest IBM’s customer in the world, so it makes sense to come to us. The cool thing is that in Merlin we developed a personal build, and that personal build allows the developer to, very quickly in their sandbox, compile their current changes and dependencies to check them out in a library. Because in Merlin, they’re not working in a library, they’re working in the Merlin container. So that was a big plus there as well to have that personal build, something that we had wanted and had trouble with developing for a while.

A.A. – With VS Code that the ARCAD archived extensions now handle personal builds. Is that right?.

J.T. – Yes. So we actually took the personal build we developed for Merlin and we started adding it to our products, starting with the VS Code extensions, and we’re hoping to roll it out to the rest of our IDE support as well. Personal build is super powerful. Developers don’t want to have to micromanage stuff. They want things just to be taken care of.
They have more important things to do than worry about which objects got compiled, and so on. They want an accurate, optimized build process so they can get right into unit testing.

A.A. – So let me just ask this question because how is this personal build different than maybe kicking off a build against a feature as I’m developing it, a feature branch?

J.T. – So the idea behind the personal build is that you are isolating the developer while they’re trying things out. That’s the point of a sandbox. So a sandbox is kind of like a developer library. I can work in it and I can do things and not bother anybody else. When I get to the point of a feature, depending on your workflow or GitFlow, you may have more than one developer working on a feature.
And so if I push, my code changes up and the build doesn’t work, neither does theirs anymore. So the personal build is a way of providing a test for the developer to say: are my code changes able to compile? And the interesting thing is in Merlin and VS Code, we’re actually creating a branch on the fly, a sandbox branch, for the developer to use so they know their stuff compiles before they push it up to a common branch where other developers are exposed to it and might be impacted by it.
So it really adds efficiency to the development process and the use again. That’s why personal builds are so popular in the rest of the open system world. It’s just been a challenge to bring it to the IBM i in our case doing it.

A.A. – So this really ties, this personal build really almost helps move shift left because now you’re not even interfering with other developers. You can find your individual problems faster and more efficiently and allows you to move into the feature with more confidence as you go through the workflow.

J.T. – You don’t have to micromanage your compiles. I’ve been making incremental changes to different objects. Do I have to worry about which one I compile, dependencies and so on. It’s all taken care of, so it adds another dimension of efficiency under the process. But with that level of isolation, you want to identify the problems as early as possible, you want to be able to unit test because the earlier you find a defect in the code, the cheaper it is to fix. We certainly don’t want a defect getting all the way out to production because then it’s really expensive. And our automated builds are integration with automation tools like Jenkins and now Azure – we have plugins for Azure – really allow our customer base to automatically build and deploy to test any changes based on their requirements, their workflow, so they can quickly test and we can message from our deployment tool out to QA and say there’s new changes in there.
We can identify the task so they can see a test plan that’s linked with a task so that allows better integration between the different teams and it means less work for the developer to manage those processes and make sure that the QA team knows there’s changes in the target environment, etc..

R.B. – As usual, that’s a lot of information. Thank you, Alan. Thank you, Jeff. It sounds like the build process is really the key to using Git on an IBM i. I mean, putting the source within the Git repository is one thing. We’ve discussed that in prior podcasts, but understanding the IBM i, its architecture and being able to perform an automatic build, that’s another thing.
That’s where the ARCAD metadata repository comes in. I mean, what I just heard was modern development following a git workflow. And again, the benefits there are the automation bringing DevOps to your IBM i developers, they act just like your open developers. You’ve got one workflow, you’ve got one toolset. The IBM i developers, they become agile. You’re going to save time, you’re more efficient and you should save money as well.