The Origins of Bubbly
Why did we roll up our sleeves to build a kick-ass product?
Published on Wed Feb 17 2021
Authors: Valocode, Jacob Larfors, Ben Marsden, Oliver Frolovs, Nate Ham
Welcome to Bubbly Babble
A podcast discussing the challenges around continuous integration, continuous delivery and release readiness. In our first episode the Bubbly development team talk about the challenges around release readiness, having the confidence that your software is indeed ready to deploy. What are the challenges, and how do we manage them, and why it led them to develop a new tool to tackle this.
Episode 1 - The Origins of Bubbly
During this episode we discuss
- Why we built Bubbly [03:46]
- What are challenges around release readiness [04:48]
- What are the available solutions [09:31]
- How do I set something up in Bubbly? [11:13]
- defining a schema
- defining a data pipeline
- extracting data
- How do I visualise and use the data aggregated by Bubbly? [14:23]
- Why we wrote Bubbly in Golang [21:47]
Jacob: [00:00:00] Welcome everyone to the first episode of Bubbly Babble which is our podcast that we’ve started about Bubbly, which is an open source tool that we have now been working on for a little while and are soon releasing to the open source world. And we’ll be trying with pilot customers. So it’s really an exciting time for us. And yeah, this, this podcast is called The Origins of Bubbly and why we decided to create a tool. So before we get into it, I guess we could give a quick round of introductions to who’s here. So you hear our voices and know who’s talking. I’m Jacob and yeah, I guess I’ll be mostly hosting this podcast. Okay, so we’ve got we’ve got three lovely people with us in this podcast. I say we, I’m the only one talking right now. I and three other people are in this podcast. So let’s say hello to everybody. So, Ben, you want to give a quick introduction about yourself and also name your three favorite technologies. And we promise we won’t judge.
Ben: [00:01:06] Thanks Jacob. Yep. Hi everyone. So I’m Ben. I was a DevOps infra engineer at Verifa, and now I work as a full stack developer at Bubbly. Three top technologies. That’s a tough one, but I would have to be a bit of a cop-out and say, Go, Terraform and Kubernetes.
Jacob: [00:01:27] So three times Go then, because they’re all written in Go.
Ben: [00:01:31] Exactly. Yeah.
Jacob: [00:01:35] Cool. Okay. Sounds great. Thanks for that. And we’ve also got Oliver here.
Oliver: [00:01:41] Hello. I’m Oliver, I’m a junior member of the team. I’m The Nocturnal Gopher because I program in Go, that is one of my favorite tools as well. I’ll be different and I’ll say that my other two favorite things are laser cutting and 3d printing.
Jacob: [00:01:56] Are those technologies or those activities?
Oliver: [00:02:01] Technologies. Well, 3d printing is a technology. Yes. It’s a set of technologies if you want to be precise.
Jacob: [00:02:10] All right. Cool.
Nate: [00:02:13] Hi, I’m Nate I’m Software Architect here on Bubbly. I have a background in server side technologies. My three favorite technologies are Go, GraphQL and DataGrip.
Jacob: [00:02:24] Cool. Thanks Nate. And yeah, finally my name is Jacob and I’ve been a consultant for quite some time doing DevOps things, Terraform, Cloud , CI/CD, lots of different things, and now taking a product lead type role on, Bubbly, which is really exciting. So yeah. And I guess I should do my three favorite technologies as well, which is kind of tough, but I could be easy and say Terraform, Kubernetes and Golang as well. I mean, are we doing this by what we enjoy? Oh. And Bubbly somebody is posting! But yeah, I would say somewhere in that realm . Container orchestration and automation and these kinds of things are really things that I enjoy. So. All right. So yeah. Quick background as to why we ended up creating a tool and a little bit about Verifa as well. Well you probably heard a few times being mentioned from the introductions, Verifa and continuous delivery and this kind of thing. So Verifa is a pure services company offering help, advice and actually doing work for our customers around continuous integration, continuous delivery, DevOps, Cloud. So basically lots of Kubernetes, lots of Terraform , lots of Jenkins still. And tackling and solving these problems, and providing a knowledge base of information across the different consultants that we have. And yeah. How we got to where we are now is that we’ve been working with different customers in different domains, but we always come across the same sort of challenge in our work. And that is the challenge of release readiness , moving from sort of CI to CD and bringing visibility and understanding into what’s going on there. So, that is the primary reason for Bubbly existing. And I guess, to fully explain that, you need to consider some of the differences between continuous integration and continuous delivery. And if you ask different people, you’re probably going to get a different response. But I think what we have agreed on as an easy way to distinguish these is that if you can put something into a CI pipeline easily, then it’s probably CI. But if it’s something that isn’t easily put in some kind of CI pipeline, then it’s probably CD. So let’s take some examples. If you have like a software build, well you can put that in a CI pipeline it’s probably C I . Running unit tests, again, CI. Running some kind of automated tests, probably CI again . Running some kind of security scans, yep, probably CI. Maybe if we get a little bit more advanced, it could be something like scanning for open source packages and these sorts of things, probably still CI, because you can run tools and put them in a CI pipeline. So if we start talking about some CD activities, continuous delivery activities, like what do you need to do, together with these kinds of tools or tests that you run in your pipeline? There could be things like making sure that all your tasks in your task management are closed. If you have filed any bugs , if you have features that have been created in JIRA or something, checking that those have been closed. So that’s not something that usually falls inside of the CI pipeline, but is probably an activity as part of your release process. You might be creating some kind of documentation. These could be like internal artifacts with test reports, with something else that you might need for compliance reasons. They could also be the actual end user documentation, making sure that gets updated and handled. And I guess at some point in the process as well, you’re going to get together and run through some kind of checklist to say that, have we ran the tests? Yeah. And what are the results? Well, they’re good. And have we closed all the JIRA tasks? Yeah. Looks good. You know, you’re going to continue through this process and in the end, it’s going to come down to a decision about confidence. Do you collectively have the confidence in the data that you have to say that we are ready for release . And then yeah, release your software, whether that be through a channel like partners or something, or whether it’s deploying to your Kubernetes environment in the cloud. I mean, this might all happen in a continuous deployment pipeline and will be automated, which is great for you, but not really possible for everybody and, and maybe not a goal for everybody as well. But this question about confidence is really what we are trying to tackle with Bubbly. And I guess the problem there is data. Even within the same company, there might be different teams that are working with different technologies. You might have some of your team developing in C++ others in C sharp, some in, I don’t know, Golang. They’re going to be using different tools, different testing frameworks , maybe even a slightly different branching and development process. And how do you get this data into the same sort of format, into the same sort of process, into the same sort of way, so that you can get this confidence measure? That’s really quite a challenging thing to do. Our experience has been that some companies might use tools like SonarQube for, for quality, where they put static analysis and unit test data into one place. They might then create some kind of testing reports or they might have a tool for capturing the test automation results. They might then have a commercial tool for managing open source licenses and security vulnerabilities. And just these three things would be three completely separate databases, three completely separate dashboards. And yeah, it would be a manual process of going through and checking those. Obviously you could put some scripts in your pipeline to say, do we have any major CVEs or do we have any major license violations? You could create some kind of simple check and you’re in your CI pipelines for that. But at the end of the day, you’re going to need to go around these three different databases and have a look. So, what are the solutions to this? At least what we have seen is that while people don’t solve it, which is quite common, they just continue and check these things manually. A more common implementation though is to use something like Elastic Search or InfluxDB, or maybe even Prometheus to some extent, some kind of database ,or Postgres or something like this. And then some guys would pick up their favorite tools, favorite languages, and maybe start hacking away with Python and building some kind of like data streaming thing to put the data into these databases and create some dashboards maybe with Grafana or Kibana, or something else. But I think what a lot of people don’t see with this is that there’s a lot of effort involved in this process in the end. And well, yeah, you’re kind of reinventing the wheel each time. So, this is where we’re hoping Bubbly will come in and enable people to quickly build data pipelines, and build dashboards, and get the data that they want, and be able to use it for this kind of confidence. All right, does that kind of summarize the problem? Do you think guys, sorry, I went on a bit of a rant there …
Ben: [00:09:37] I think so. Yeah. I think it does a really good job.
Jacob: [00:09:41] Cool. Okay. Then I guess let’s maybe have some general questions about Bubbly. Ben, maybe you want to take this one . Yeah, how do I work with Bubbly? Like, how do I code in Bubbly? How do I set something up?
Ben: [00:09:55] Yeah, sure. I can take this one. So I think we really wanted to keep it as native as possible coming from an infrastructure or devops background, we had a couple of things. We knew we loved Terraform, and we knew we loved Kubernetes. Now for those who don’t know, Terraform is written in HCL, which is HashiCorp’s DSL basically. And it’s a way of defining infrastructure declaratively. So we took that inspiration and we pretty much allow you to write Bubbly pipelines in the same DSL. So as HCL code. The second, as I mentioned, was Kubernetes. So if you speak to anyone who uses Kubernetes, you know, that they love the API. So what we wanted to do when building out Bubbly was again, keep this as native as possible. And so Bubbly is built upon the same building blocks as Kubernetes. It uses resources. So things like , extract the process by which you pull data from somewhere, is defined as an extract resource. And the idea for this is basically that if you know how to write Kubernetes manifests, you know how to write Bubbly pipelines and Bubbly resources.
Jacob: [00:11:04] Cool. So I guess then you just mentioned two quite big technologies there. Would it be fair to make the assessment that if I know how to code HCL and Terraform, I will know how to code in Bubbly. And if I know how Kubernetes works, then probably I have an understanding of how the Bubbly command line tool and server will work.
Is that like a fair assessment?
Ben: [00:11:26] I think that’s definitely a fair assessment. As time goes on, if there’s any case where that isn’t strictly true. I think we’re definitely driving ourselves in that direction.
Jacob: [00:11:36] Yeah. Cool. Thanks. All right. So we talked about data pipelines and coding data pipelines. I guess there might be people listening who don’t really know what a data pipeline is. Do you want to explain a data pipeline in a really simple term, so it’s not like a crazy word that sounds really sophisticated.
Ben: [00:11:58] I can do my best. I don’t pretend to be a data engineer or data scientist. And I think this took quite a while for us to realize, we went back and forth with our own interpretation of a data pipeline. But in the end, we settled very strictly on ETL. So for those who aren’t aware ETL is basically the process of extracting data, transforming it in some way, and then loading it to some separate location, be it a database or other warehouse or whatnot. And this is exactly where we basically settled on data pipelining ourselves. So we have an extract resource, a transform resource, and a load resource. And we’ve kind of, again, kept this as simple as possible so that people coming from different domains can just jump onto Bubbly and see exactly what they need to do in order to create a pipeline.
Jacob: [00:12:50] Awesome, so yeah, if we go back to the original problem that we were discussing . I think we had the idea of SonarQube and some test automation tool in some OSS tool. So how does this fit in? If I have these, maybe I have JIRA as well or something, I have like more tools. What is the idea here, that I basically write data pipelines for each of these tools and extract the data and put it into Bubbly? And actually, what does that mean, to put it into Bubbly? Where does the data end up?
Ben: [00:13:25] Good question. So we basically have the concept of a schema, which is something that customers themselves will define according to their needs. And this is basically a declarative way of defining your underlying database structure. So the idea would be that you then write, extract, transform, load pipelines, data pipelines. Then the end goal is to push data into an underlying database that is inherent to that schema basically. Yeah, I think that summarizes things.
Jacob: [00:13:58] Yeah. That’s a really nice summary. Okay. So I have a bunch of tools that I’m already using in my process. I don’t have to change anything and I want to define the data that I want to solve my problem of like release readiness. So I create my schema to map that. And then I write data pipelines and populate the data that is going to help me bring confidence to my release.
Ben: [00:14:22] Exactly.
Jacob: [00:14:23] Okay. Cool. That’s awesome. Yeah, that’s really nice. So I guess I’ll switch over to Oliver then. Given this whole pipeline process works great and I’ve defined my schema, written the pipelines, and I have now my data , I guess the next question would be that, how do I actually use that data? Do you want to give a quick intro to how we would go about doing that?
Oliver: [00:14:48] Yes. So in regards to displaying the data we have a number of options available. One of which would be the built-in Bubbly UI, which is a web interface. It is configurable by a customer using HCL. Again, HashiCorp HCL. It’s familiar technology and it’s made of widgets, which can display various aspects of the data, according to the Bubbly schema, also defined by the customer. We also don’t want to hoard all these data just in our own database. We have a GraphQL interface to access it from any tool that can be graph field data. So that could be your business intelligence tools. That could be some custom script, which puts the data into somewhere else, or just queries the parts of it which are relevant to the task in hand. So, basically two options available are the built-in UI, which can do nice dashboards to your needs. And also there is this general GraphQL interface which lets you work with Bubbly as if it were a graphical source, like a GraphQL database.
Jacob: [00:16:00] Cool. Thanks, Oliver. I have two questions about that. One question would be going back to the problem again, we already have these different tools, these different dashboards that people are using. Doesn’t introducing another tool, like the Bubbly UI, so introducing yet another dashboard . Is there a reason behind that? Like, what is the idea behind creating yet another UI, another dashboard. And why not use something like Grafana?
Oliver: [00:16:31] Yes, specifically in case of Grafana. Grafana is a time series renderer, so it displays time series. You can wrestle it into displaying information, which is not time series, but that’s not its intended purpose, you’d be swimming against the current. It’s not a convenient thing to do and it’s not very maintainable. And Grafana is not very good at building complex hierarchies or pivot tables. So any information which is kind of a people table, Grafana is not really good at this, outside of its intended purpose. But having mentioned Grafana, you can also, if you have simple needs, you can run GraphQL queries against Bubbly. So Grafana has a GraphQL plugin and any idea that you can express in GraphQL, you can source into your Grafana dashboard as well. Yes. So the aim with our own custom UI was to give the customers an easy way to programmatically describe the interfaces as called, basically using HCL from the rich widgets, which do only the things that you want doing and not much more. So you’re not trying to fiddle some complex tool to do the task in hand, you have the right tool for the job.
Jacob: [00:17:49] Nice. Okay. So if I want to put some data into Bubbly and show a section of that data, I could easily create a panel in GraphQL to do that. And that’s perfectly fine. But then if I want to start representing this hierarchical data, like if we take a project level, is the project ready for release? This might now be associated with a whole variety of different types of data, you know, testing, OSS, security , tasks being closed , documents being created, this sort of stuff. That’s more like hierarchical data in a way that might then be a challenge in Grafana to fully present.
Oliver: [00:18:26] Yes. Yes, because it’s not meant for that. There are some ways to go around it, but they’re all very hackey and that’s just not the intended purpose for Grafana as a tool. It’s a very wonderful tool to display time series and associated information, but not for exploring complex hierarchies or building pivot tables.
Jacob: [00:18:44] Yeah. Okay. Nice. That’s a nice summary. And yeah, I had one more question about the GraphQL interface, and I guess anybody who’s listening, who doesn’t know what graph QL is. I’m probably going to do a very bad attempt right now, explaining GraphQL.
But everybody should be familiar with what a rest API is. So you send a request to a server, to an endpoint, and you get a response back. So GraphQL is an HTTP post type and you send a text query and this text query can be quite highly customized and configurable to only return the data that you actually need.
And can also be used for things like mutating the data. So you can create new data, you can update data and you can do things like subscriptions as well. So it is a query language, in my opinion. I mean, a rest API, if you want to do something with that rest API, you have to code it in the backend.
Whereas with GraphQL, you can provide a GraphQL API, and then the user has some flexibility and control over the queries that they write which is pretty cool. But yeah, if that was a really bad explanation, then please chime in!
But yeah, my question was going to be this GraphQL that is created , Ben mentioned earlier, by the schema that we defined. So how does this GraphQL interface work? Is it created from the schema or do I need to do something extra to it to use it?
Oliver: [00:20:19] The GraphQL schema is created automatically from the database schema, which models the release readiness process for a specific customer. So we’re not making a lot of assumptions about what your release readiness process looks like. We’re just offering the tools to model it. So once you model it as a database schema, basically what we call a Bubbly schema, GraphQL schema is generated automatically and using modern GraphQL tools, you can even explore that schema in an introspective way. So you can ask what fields and what objects it supports. So it’s almost effortless. It’s actually effortless to have a GraphQL schema for your data, because once you get the model of your release readiness process, the GraphQL schema is just there for you to explore. With whatever tool you want to use for exploring the data. Even Grafana with a GraphQL plugin can be used, or our built in UI will have built in facilities for exploring the data in interactive ways like a REPL type thing.
Jacob: [00:21:31] REPL being the ability to write code in a browser and execute it.
Oliver: [00:21:39] Read, execute, print. And what’s the L for? It’s a loop. Loop, yeah
Jacob: [00:21:47] Alright. Nice. And we had Nate on the call as well, but he’s a recent father and so he has obligations, let’s put it that way. He had to jump off now. But our question for Nate was going to be what is Bubbly written i n? Well,we were hoping he’d go off on a rant about how much we love Go. And I guess the three of us can make the best effort at that now. So yeah, I’ll say that Bubbly is primarily written in Go, Golang. At least the backend is. We drew a lot of inspiration from the HashiCorp tools. This is kind of like a single executable, a single binary , nice architecture, Golang’s a simple language. But extremely powerful and yeah, just a really nice experience to code in. And I don’t think we regret that decision at all , right guys.
Ben: [00:22:36] Not at all, no.
Jacob: [00:22:36] Did Golang even exist 10 years ago? I don’t know, but were you secretly a developer at Google working on the language before it was public or ..?
Oliver: [00:22:50] No, no, no. Well, ..
Jacob: [00:22:52] When was Golang released?
Oliver: [00:22:52] That was a long while ago.
Jacob: [00:22:54] I think it was 2011, wasn’t it?
Ben: [00:22:56] I think it was 10 years ago.
Jacob: [00:22:58] Yeah. Wow. It’s 2021, isn’t it. That’s a bit of a fright! Okay. Yeah. And I guess the community around Golang as well is something that we really like. And, and I guess we should give a shout out to Go Time, which has been like , I can’t remember when Ben, you were the one who discovered Go Time, which is a podcast for, well around Go. But I think it’s just quite interesting anyway. So anybody who is developing in Go and listening to this and doesn’t know about Go Time, I really recommend you go and check it out. I don’t know what kind of activities we’ve been doing whilst listening to Go Time. At least when I Hoover nowadays, I have Go Time on.
Ben: [00:23:38] Some cross country skiing the other day for me,
Oliver: [00:23:43] I go for long walks and I listen to Go Time, during walking because I struggle to maintain concentration for extended periods of time listening to something. But I found that when I go for long walks outside, like in nature, it works for me. I really enjoy people discussing technical subjects, listening to it.
Jacob: [00:24:05] Cool. So now we have it as a personal or personal team, team personal… what’s a team personal goal, a personal goal with everyone in the team to appear on a Go Time episode once and talk about bubbly when this thing grows a bit and becomes bigger. So yeah, hopefully, well, hopefully you’ll hear us there talking about Bubbly. Yeah, that pretty much wraps up what we were going to talk about today. So I guess in summary, there’s a problem of continuous delivery and confidence in release and lots of data. So we have created this thing called Bubbly, which is going to be an open source tool, and we’re open sourcing it very soon , which will allow you to write data pipelines, to read data from these different data sources, convert it into the schema that you define, and load it to a single database. So it’s all collected in one place that has a GraphQL interface in front of it. And then you can use things like Grafana, if you want to. And we will also have a Bubbly UI, which is built to tackle some of the challenges with using Grafana. Things like hierarchical data and getting the like big picture rather than creating simple graphs, panels, this kind of thing. And yeah, it’s mostly written and Go and we love Go. Was that a fair summary? I’m getting thumbs up on the video, which people can’t see, but yeah. Cool. So yeah. Thanks a lot, Ben and Oliver for being on this podcast and, well, thank you Nate, as well, who’s gone. I guess when he listens to this afterwards, he’ll be grateful that we thanked him.
Oliver: [00:25:46] Always a pleasure.
Jacob: [00:25:50] Awesome. And yeah, stay tuned for episode two, which will come in, I guess, two to four weeks around that time frame. We don’t know what the topic will be yet, but yeah, we’ll try and keep it fun and interesting. So, all right. Take care.
Sign up for the Bubbly Bulletin
Keep up to date on the latest news, releases and features and fun stuff as Bubbly redefines Release Readiness.