A presentation at Deserted Island DevOps in April 2020 in by Jacquie Grindrod
There’s been so many great talks today, & some hard acts to follow but I’m here and I’m going to do my best! Good morning, Good afternoon or Good Evening to you, wherever you’re tuning in from today. Wave This talk will be a little like one of those movies where the end comes first. We will not be going to space, at least, not today. Today, We’re on a deserted island & the Dodo Aerospace program hasn’t made it off the ground yet.
This talk started with an internal announcement that HashiCorp Nomad’s engineering team would be doing a Hackweek. I’d been looking for new technical work to dig into as well as excuses to work closer with the Nomad team. This felt like the perfect opportunity! I kept hearing about it in our meetings and was already getting serious FOMO, so I asked if I could join & was delighted when I got the go ahead.
Before we continue, here’s a little bit about me. I recently moved a role as a Developer Advocate at HashiCorp. Prior to that, I spent ~5 years as a DevOps Practitioner & most of those were in the Digital Healthcare space. I am not a game dev nor am I a space expert. I’d like to walk you through the last 2 weeks or so of my life, the story of this game development and the experiences had along the way.
A teammate pitched me an idea about building a game which would visually demonstrate Raft. Raft is a consensus algorithm for fault tolerance and performance used by HashiCorp’s Nomad & Consul products. I didn’t know much about Raft and I really wasn’t sure about the idea of building a video game in 3 days but it met my goals of new tech & working with a teammate so I dove in with her. Which leads us to…
SpaceHack Written by someone who has never built a game or been to space, SpaceHack is a game about collaboratively making it to space during a time of unprecedented isolation. Our goal was to create a game where we work together to face challenges but also explore our surroundings with our community. We took inspiration from various games including JackBox, SpaceTeam & NetHack. From JackBox we took the concepts of quick, short lived party games flexible for a range of players. From SpaceTeam, we took the concepts of needing to work together to share insight on how to clear levels. From NetHack, we liked the RogueLike concept of exploring dynamically generated levels within a world created around you. Going into development, we came up with 2 major goals for the game.
My family & some friends live 20 hours away from me in the best of times, and now most of my friends might as well be that far. My teammate Jasmine uses Minecraft to spend time with her family that she can’t visit in person right now. Making something that could help us to connect with them despite distances was important to us, so it became one of our project goals to build something that supported remote multiplayer.
We also wanted to focus on creating an Uplifting Experience. We didn’t want to create a game that would leave our users (or us) stressed out & frustrated.
Those goals felt like they naturally relied some key devops concepts if we wanted to build them. Scalable Infra & infrastructure as Code both felt imperative for us to handle dynamically generated rooms for an arbitrary number of players. Build & Deployment consistency felt necessary for both us as the developers to have an uplifting experience & for our players to have a consistent experience. THere’s nothing more frustrating than playing a game with people and encountering game ruining bugs, to find out it’s your operating system and you can’t do much about it.
Which brings us to version 1 of our game. At this point, we were playing Nethack to get a feel for terminal based games & we were pretty sure we wanted to build one ourselves.
After scouring the internet for Go based terminal game frameworks, we came across TermLoop. It had very promising looking games built on it such as Go Mans Sky. We didn’t really know what we were getting into but we were optimistic that it was going to be awesome. We wanted to make spaceship quadrants based off of XKCD’s up Goer Five comic, with different player roles assigned to these We had tons of fun ideas we wanted to try and a document full of ideas for item types, player roles and how those would fit into the spaceship quadrants & gameplay.
But we didn’t make it very far. While TermLoop has some excellent examples out there built on it, it just wasn’t fitting our immediate needs and super short timeframe. Some problems we ran into were: Term color. It was green on my but the pink is what it looked like in Jasmine’s terminal. We hadn’t considered how terminal setups would impact player experience. Scene connectivity - We got this welcome screen to show up, but we couldn’t get it to connect to the game screen no matter what we tried. The game screen was from a tutorial, so right off the bat you would be right if you asked “but wait, isn’t that a field with a lake? I thought we were in space?” Finally We weren’t able to figure out how we would connect multiple private laptop terminal sessions together to share a game state. So a halfway in, we pivoted.
While scouring the internet, I found a promising 42 minute long Tutorial on how to build a simple rogue like game with autogenerating maps & levels. This seemed like a great win that was too good to be true. Spoiler: It was too good to be true. 42 minutes quickly turned into almost 50 hours of development time, BUT it worked!
At least half of that went into understanding tile maps & tilesets. There’s a lot of different options for creating TileMaps. Those are the rooms and maps that my character navigates through. Some options were custom hand placing tiles, creating auto tile maps (which I initially couldn’t figure out) or procedurally generating the rooms. But first, we had to FIND tile sets. The image on the right was my first attempt at creating a custom room using individual tiles. I was a little bit too enthusiastic about how neat it was going to look.
There were a lot of bumps along the way, happy to talk about them in Q&A. One of our first goal guided decisions was between aesthetic rooms and function. Ultimately, our goals of remote play and ability to scale led us to choose the procedurally generated rooms and levels. We believed this would make it easier to allow different numbers of players and different rooms in addition to keeping each run of the game different. Honestly, this led to some interesting troubleshooting scenarios for me, because even I didn’t know what it would look like or where things would spawn!
Another decision we made based on our goals was whether or not to include enemies & combat. Traditionally, Roguelike games also come with enemies and combat. We felt that adversarial mobs created stress in the user and this didn’t align with our goal of creating an uplifting experience. I certainly didn’t feel very good getting into this screen and going ?? Where did they all come from?? Why are they in the walls??
Originally, we thought our Pipeline would look something like this. Local Development, Push to Github. On a PR merge into master, Jenkins-X would run and build the artifacts. We would then deploy these to a Nomad/Consul cluster on AWS.
Here’s a little more on what that cluster looks like. We wanted to use Packer & Terraform to provision up a cluster of AWS EC2 instances running Nomad & Consul. This piece is actually done! We would have a room scheduler job to handle that would handle the creations of rooms & joining. We’d also have a game room server to be our source of truth for the game’s state & environment which our clients (or players) would connect to. It was either this or diving into Peer to Peer game development which neither of us had a mental model of what that would look like. Remember, this is our first time!
GitHub -> GitLab -> Export -> Pages Expectations are great, but sometimes getting the right pieces in the right place at the right time just isn’t possible. Our reality looked like this: A hacked together pipeline based on Gitlab CI because that was the first & clearest example I could find for our project. Kudos to the community for it working within the first 3 tries. We would code locally, then push that to a branch in Github. When the PR was merged into Master, I would then pull from origin master & push to gitlab master. This would trigger the pipeline that would export our web build and deploy to Gitlab Pages. Not ideal, but it worked! Which brings us to the next chapter in this story.
Bugs. Bugs EVERRRYWHERE. Completing the Tutorial left me further along than I ever expected to get with game dev, but also suddenly off the training wheels. We found the official documentation hard to work with which made learning on our own to build things quite difficult & we still had big goals for what we wanted to finish!
We wanted to personalize our game. We’d use Items to support our goal of remote collaboration. This meant we needed an incentive for players to work together. So we came with a strategy where a player will see 2 items they need to get each level, but they might not be spawned on their map.This meant they would need to describe the objects they see to their teammates so their teammates could find it to progress as a team. To emphasis communication, we also added “unnecessary” items. If you picked these up, they will lower your score & possibly give you a game over. We thought that from a “story” perspective, it would make sense. If your sysadmin in the control room sees the engine room is over heating and has Magical Coolant as a key item then perhaps the engineer should find and use it to bring down the temperature. If they do something else instead & the temperature is left to climb, it should hamper your progress. This was also to deter players from picking up everything in the level to proceed as fast as possible. After TileMaps, Items were the next hardest to implement & came with some interesting bugs.
Round 1 of Item Issues included items stacking with items, spawning where they shouldn’t, and items that were visible but did nothing. It was crucial that we fix this to stick with auto generated rooms & assets to keep maps & levels that would scale well. We found this to be a combination of a few problems: Our function to reset the level didn’t always get called due to a code scope issue. This was why items were sometimes in space or visible but unusable. This was an easy fix - We moved the function to a place where it would be called properly everytime. However, sometimes items still spawned where they shouldn’t, like the hallway. This would mean if your room was blocked by an unnecessary item at the start, you would instantly lose the game and couldn’t do anything about it. Not very uplifting. We ended up working around this one by adding an additional tile for the room floors & forcing items to spawn on that.
Round 2 of item issues was the sole focus of one of my streams. No matter what I tried, I just couldn’t get it working. Even though all the other items were working, it would spawn in space or the walls. Sometimes it would be in a room & be unusable. And even others, it would drop in the first room, work, but it was invisible. It was a long 3.5 hours of questioning myself and why I was streaming myself failing over and over and over again. But something good came out of this! A fellow streamer was watching at the time, and asked if he could try something new & make a pixel art item for the game. He did & even though I still hadn’t fixed the lightbulb, it was heartwarming & made this experience worth it.
The next game personalization we tackled was adding a timer. We also wanted to add a little pressure for players to work together quickly. We hoped that the timer is long enough that it’s okay to take your time, but completing levels faster leads to a bigger score bonus. We did this in the spirit of making it quick, fun, social games that you can run and then rerun with your friends.
Occasionally, something like this will happen where the room you need will spawn right beside another room, making key items or portals inaccessible. We suspect it has something to do with the algorithm handling room generation & pathfinding and likely needing an adjustment to require at least 1 tile between rooms. Again, not an uplifting experience but we’re working on it!
The final personalization we tackled was getting Exports to work. We want our game to be accessible to different kinds of users. Someone like my mom would probably prefer to run a web client vs downloading a game from steam or as a binary for her computer. Getting the exports to work was another rollercoaster. It deployed without the timer, items or level count and was more or less unplayable. It turns out we didn’t have scenes for those things so the compiler was treating them as orphaned and not including them.
That covers most of our journey these past 2 weeks. So what’s next?
First we got some initial feedback. This is a small snippet of the feedback we received but covers a range of what we got. It ranged from the good, the bad & the ugly. We’ve already taken some of this feedback and done some things to help users out such as adding instructions.
Networking Multiplayer Lobby & Character Selection Running it in the cloud with Nomad/Consul
First and foremost: Let’s give ourselves the space to be gentle to ourselves. It’s hard when we’re always sprinting towards the next deliverable but things just aren’t lining up. It’s hard to pick up new things & run. I’m still working on this too. I kept stressing about little details not being right & not getting things done in time to show you today. It’s so important to give ourselves the space to have fun with our learning & reflect on how far we’ve come.
I’ve talked a bit about tools/processes, but not so much about culture today. I think Tools & Processes are a part of the big picture, but to me, the heart of DevOps is bridging systems and people together into a community that works together to solve problems. I came into this hoping to work with at least one other person. I didn’t expect to have someone from the streaming community volunteer to try something new & help out, & I was delighted by the chance to actively develop & work with a friend I haven’t had the chance to code with before. This game really brought parts of my life together that normally don’t intersect & I think that’s part of the beauty of a DevOps mindset. Whether you’re a designer, support, SysOps, Security, a Dev. Or something else. We’re here, solving a problem, together.
And finally, Empathetic documentation is the key to growing out community. Without going into too much detail, I struggled a lot with GoDot’s documentation. It wasn’t easy to follow and made a lot of assumptions about knowledge & steps I didn’t know I needed. I think we, as a community, can help people of other backgrounds & depths of experience by writing documentation in an approachable way. Get review from those who are different than you. Something that seems simple to you, might not be simple to others. Let’s set up documentation in a way that helps others to succeed. While I was disappointed with Godot’s docs, I was thrilled with the community’s tutorials and help guides. Almost all of my progress is due to those. So for those listening today, or who might see this later, I’d like to ask you to do this: If you’re thinking of writing a guide or a tutorial, or something to help but maybe not so sure if you should: Please do. We need you. We need your experiences.
Just to recap - My name’s Jacquie, I’m a Developer Advocate at HashiCorp. I’m looking forward to continuing this project & sharing what I learn during it. You can reach me on Github as GogoCoco (there’s a story here) or Twitch/Twitter/Discord as DevOpsJacquie. Again, I am not a space expert. But if my talk has sparked any curiosity for about about how space actually works,
I strongly recommend you check out Sophia’s content. She’s an actual space expert who does cosmology & astroparticle theory & has tons of cool content on Dark Matter.
In an unexpected twist ending, we made it to space! The one thing I want you to walk away with today is this: remember that in times where things feel dark & hard, we are not powerless. YOU are the light that can help to make things smoother, better or brighter for those around you. Thank you for coming to listen to my talk & I look forward to any questions & discussion points you may have for me.
