Learn from our mistakes in upgrading Keycloak — dependency deadlocks and undocumented hacks made us burn in dependency hell

Building a tech product out of nothing is simple and hard at the same time.

Simple, because no legacy tech is around, and you can start on the green field.

Hard, because you don’t know yet what product you will need once you reach 10, 20, 30, or even more customers.

At Yonder, we went live with our product in 2019 with our first customer. Five years later, we serve nearly 30 B2B and enterprise customers with the same product — albeit with many added features. Below the hood, the architecture is still the same. Some parts of the architecture will change shortly, but the larger challenge is managing dependencies when core packages need to be upgraded. That’s what this article is about.

Upgrading Keycloak

We use Keycloak for authentication and authorization. That means users log in to our system through Keycloak, and they get their permissions for certain features in the application through Keycloak. Not enough, some of our customers connect their single-sign-on infrastructure to Keycloak, so that they can use their organization’s Microsoft 365 credentials to log in to our system. It’s even possible to map permissions from a customer’s active directory to our system via Keycloak.

Can you smell the complexity? It’s a mixed blessing — some of our customers wouldn’t be customers if these functionalities weren’t available, but upgrading such a component at the heart of your system is a nightmare.

Nevertheless, it has to be done at some point for security and compatibility issues.

Here is what we learned from this piece of work.

Learning #1: Don’t Wait for Too Long

Frankly, we waited for too long before we upgraded Keycloak. Not because we didn’t want to upgrade it, but because of the complexity described above.

Because we waited for too long, there were some dependencies on other frameworks: The new Keycloak version required an upgrade to Spring Boot, another core component we use in our backend application. Not an issue in itself, but it created a deadlock on other bug fixes that depended on the upgraded version of Spring Boot: We couldn’t ship those bug fixes before we upgraded Keycloak — the old Keycloak version wasn’t compatible with the new Spring Boot version, and the new Keycloak version wasn’t compatible with the old Spring Boot version. Grmpf.

Learning #2: Don’t Modify Core Packages

Because of that deadlock, we decided to leap ahead and test all the customer Keycloak configurations on the Staging environment, where both Spring Boot and Keycloak were upgraded already.

The active directory mappers gave us lots of trouble, mainly because the new Keycloak version included a key feature that was missing in the old Keycloak version. It took me a long call to an ex-employee to find out what had been done to overcome that shortcoming at the time, and it took me another long evening in GitLab to find the relevant commit from 2021. Believe it or not, the missing feature was injected by our development team into the core Keycloak package at the time — solving the problem then, but creating a dependency downstream.

Learning #3: Document Your System

Injecting features into core packages and doing hacks are fine, provided that they are properly documented for future team members to know what you did in the days of old. How should a new DevOps engineer know that he will kill an essential feature by a version upgrade because you injected it into the old version but didn’t document it? We could have saved lots of time by documenting the feature injection properly.

Conclusion

We were able to circumvent all the issues and upgrade Keycloak on our production system, but we paid a hefty price in terms of resources invested and post-release hotfixes needed.

Life for us and our customers would have been easier, had we followed the three learnings described above.

Promised, no more core package modifications anymore, and everything is documented now.