Alpaca - How workday handles with SSL/TLS connections
Jessica Rangel31:502023
Workday chose to build its own certificate authority, not buy one
A third-party certificate solution would have cost Workday a hundred thousand dollars a year in 2014, for a server fleet that has since grown tenfold. Jessica Rangel, Senior DevOps in Workday's security product teams, explains why the company built Alpaca, an internal certificate authority that has issued more than 30 million certificates. For anyone securing microservices across clouds, the question is how one compromised cluster stays contained.
Learn how layered certificates let one revocation isolate a compromised cluster, zone or environment
Jessica Rangel, Senior DevOps at Workday, explains how Workday built Alpaca, an internal certificate authority that secures TLS connections and service identity across thousands of microservices on multiple platforms. She covers PKI fundamentals, the build, borrow or buy decision, a layered trust model designed so a single revocation contains a compromise, and the challenges of running a certificate authority in future.
“And that is the key when you have these kind of layers, if you want to revoke the certificates, your chain or everything that's under that certificate are going to get revoked as well.”
“So the private keys for a certificate authority is the most important thing.”
What you will learn
Explain how public and private keys encrypt messages and why a certificate authority is needed to vouch for identity.
Weigh building, borrowing or buying a certificate authority using licensing cost, integration effort, documentation and adoption.
Design a layered certificate hierarchy so revoking one certificate cuts off a compromised server, cluster, zone or environment.
Use an authoritative service list and identity certificates with JSON web tokens to authorize service-to-service calls.
Anticipate the hard problems of an internal CA: offline private keys, conflicting compliance regimes and new encryption methods.
Key takeaways from the speaker
The decisions to invest to have your own certificate authority
How to support encrypted connections between apis in a multi platforms architecture
The challenges to adapt a certificate authority to support future request
Details
Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
31:50
Published
June 2023
Questions this talk answers
Why did Workday build its own certificate authority instead of buying one?
In 2014 a third-party solution cost about 100K dollars a year for 20K servers and still needed custom development to fit Workday's architecture. Open source options at the time lacked documentation and adoption, so Workday decided in 2015 to build its own.
How does a layered certificate chain help contain a breach?
Each layer (actor per server, Uber actor per cluster, issuing CA per zone, policy per environment type) can be revoked, and everything under it is revoked too. The rest of the infrastructure then stops trusting the compromised server, cluster, zone or environment.
How do services at Workday prove who they are to each other?
Each service gets an identity certificate from Alpaca and uses it to create a JSON web token. The receiving service can accept it or check back with Alpaca that the certificate is valid and not revoked before processing the request.
What stops an attacker's pod from getting a valid certificate?
Platforms must name the service each certificate is for, and Alpaca only issues certificates for services on its authoritative service list. A pod with an unknown name cannot obtain one.
What are the main future challenges of running an internal certificate authority?
Automating around private keys that should be kept offline without weakening security, meeting compliance standards such as FedRAMP and GDPR that can conflict, and adopting new encryption methods and technologies like mutual TLS.
About this talk
With the advent of public and private clouds, Kubernetes and microservices, Workday faced a real challenge to leverage an authentication solution and maintain confidentiality across different platforms and thousands of servers and APIs. Alpaca is our in-house certificate authority solution, to enable dev teams and acquired companies to securely create and manage certificates. Successfully served more than 20M internal digital certificates, we will briefly mention the PKI and JWT fundamentals, decision behind to create our own solution, key functionalities, technical choices, aspects of design, implementation, but also deployment and maintenance.
The real cost of buying a certificate authority (79 s)
Revoke one certificate, cut off a hacked server (61 s)
The private key dilemma: automation vs security (84 s)
Speaker 1: Presentation today, and I'm very happy that you guys, girls, actually choose to come and see my presentation about security. So my name is Jessica Rangel, and today I'm going to talk about Alpaca and how Workday handles with TLS and SSL connections. So in our agenda today, I'm going to mention a little bit what it's Workday and introduce myself. I'm going to cover some key concepts about what's PKI and CA. Also, I'm going to mention exactly what it's Alpaca, the decisions behind to build, borrow, or buy this kind of solution, our key features, and also some feature challenges.
So just introducing myself, my name is Jessica Rangel. I am originally from Brazil, but I'm based in Dublin, Ireland. I'm senior DevOps at Workday, and there I am part of the security product teams. Speaker 1: Overall, I have more than 15 years working on IT. So if you have any question about Workday, or if you want to talk about security, you can connect it with me on my LinkedIn, and I would love to keep in touch with you.
Speaker 1: So covering a little bit about Workday, so Workday, it's an American company. We do systems that are cloud-based for finance, HR, and planning. Speaker 1: Nowadays, we are leader of the Gartner for HSM systems, so it's basically HR, and we also are Gartner leader for the ERP. We have 90,500 customers, and a few of them are Netflix, Google, Walmart.
Speaker 1: So because those huge customers and these huge companies trust Workday to handle with their most precious data, that's like finance or HR, for Workday security, it's one of our top priorities. Speaker 1: Workday also has behind the scenes thousands of applications that are supported for hundreds of teams and also acquired companies. Speaker 1: So if you feel interesting to see the positions that we have open, and we have positions for everything that's related to IT, in this presentation, I put a link where you can check.
Speaker 1: And also you can connect with me on the LinkedIn to have more details about how it works, how it is working at Workday, because yeah, I really like to work there. Speaker 1: Now, going to the mentioning a little bit about the concepts, so then before we start to talk, I would like to explain actually a little bit, a little brief.
Speaker 1: How the public key infrastructure works. Speaker 1: So the PKI basically it's in a way to encrypt your message if you want to send that security to a friend of yours or to a recipient. Speaker 1: So how you do that the PKI creates two keys, the public and the private key. Speaker 1: So let's say that you want to send a message to your friend and this message, this message would be encrypted.
Speaker 1: So no one besides your friend would have access. Speaker 1: So what you can do it's using PKI. Speaker 1: You can encrypt your message with your friend's public key. Speaker 1: And because he's the only one that has the private key, he's going to be the only one that's able to decrypt the message and then read the information that is there.
Speaker 1: If someone fetches them this message that's encrypted with the public key, they're not going to have access because this is going to be encrypted and only the private key can decrypt the message. Speaker 1: And now commenting a little bit about the certificate authority. Speaker 1: Actually, if you think about PKI that I just mentioned, so whatever has access to the private key would be able to decrypt the message, right?
Speaker 1: So how can I be sure that only my friend has the private key and he didn't share with someone else? Speaker 1: So imagine that in the Internet, we have it's a very hard time to actually make sure that websites are who they are right because hackers can create can mimic a very. Speaker 1: Can can mimic bank websites and you don't know if you can trust that website is actually your bank or not.
Speaker 1: So because of that for the Internet, we have a third party or organization or a component called certificate authority that as you can see in this image of the certificate authority. Speaker 1: It's an organization or a company that would trust and you can trust on the certificate authority that they in this image. Speaker 1: Alisi is actually the owner of the public key that it's related to Alisi and only her has the private key.
Speaker 1: In the Internet, we have many external certificate authority. Speaker 1: So this and this other graph, it's basically saying that Komodo that's a company that creates certificates and is also a certificate authority holds like 45% of this. Speaker 1: SSLs certificates over the Internet. Speaker 1: So basically Komodo is a third party company that that's is going to say that that bank that that were all came from the certificate that associated to that is actually from from that company that you were believing.
Speaker 1: Then thinking about it. Speaker 1: So how do we do we enable TLS and SSL for thousands of microservices? Speaker 1: So more than 100 teams deployed also in different platforms like Kubernetes, OpenStack, physical server. Speaker 1: So Workday has had this problem. Speaker 1: And we solve that with Alpaca. Speaker 1: So then mention a little bit about Alpaca.
Speaker 1: So first of all, it's not an animal, right? Speaker 1: Actually, we managed to create an acronym. Speaker 1: Acronymus that is stands for asynchronous low latency platform automation certificate authority. Speaker 1: So more as known as Alpaca. Speaker 1: So Alpaca, it's an internal certificate authority solution that it's using at Workday and see and it's the only solution that's authorized to issue manage validate and renew digital certificates in our infrastructure.
Speaker 1: But to create Alpaca, we didn't create that from what we had in our heads. Speaker 1: Basically, we follow up some best practice that are described in the CPS and also in the RFC. Speaker 1: So all the certificate authorities even being internally or externally has to follow these rules for the RFC also because they mentioned a few security items that you have to implement when you have the certificate authority.
Speaker 1: And nowadays Alpaca, it's responsible to issuing more than 30 million internal certificates so far for Workday. Speaker 1: And here it's a little bit about the decision that we had to between to build borrow or buy a solution for the certificate authority. Speaker 1: So back in 2014, we started to try to fix this issue to have SSL and TLS internally for the communication of.
Speaker 1: The microservices and the microservices and the services that we had. Speaker 1: And back on that time, we saw a very high cost to use a 30 party solution. Speaker 1: So basically it was a hundred K dollars per year to create certificates for 20 K servers on that time. Speaker 1: And also on top of that Workday has to.
Speaker 1: To use a little bit of developer development on top of it to create the technical fit for our architecture, because a 30 party solution doesn't come perfectly for your architecture. Speaker 1: You have to create some kind of connectors and etc. Speaker 1: So you have to invest some development hours, and these would be a little bit on top of this high cost of the licensing.
Speaker 1: If you imagine that now Workday has actually 10 times more than the servers that they have that we have 2014. Speaker 1: Also, we evaluated some open source solutions. Speaker 1: And the main reason that we didn't went for any all that was because we check it documentation. Speaker 1: We check it adoption for other companies.
Speaker 1: And back on that time, I was one year before of the last encrypt and any of the solutions inspired confidence for us because of the lack of the documentation because of the language that was used or because of the lack of. Speaker 1: Other companies and other people using actually the open sourcing and supporting that.
Speaker 1: So then in 2015 we we just decided to build our own certificate authority solution based on the PKI. Speaker 1: And also taking in account that the solution would be fit for our technical requests. Speaker 1: We need to reach out to our performance thresholds and also our use cases that Workday has in the infrastructure.
Speaker 1: A few keys. Speaker 1: A few key features about Alpaca is our trust model of the certificate. Speaker 1: I'm going to give you more details how that works in the next slide. Speaker 1: Also authoritative list service list. Speaker 1: So basically we have a list of services that can use Alpaca. Speaker 1: And I'm going to mention also in the preview in the next slide a little bit more about it.
Speaker 1: We have our trust store. Speaker 1: We have identity and security certificate. Speaker 1: I'm going to give you more details later. Speaker 1: And also we have an API for Alpaca. Speaker 1: So going for the trust model. Speaker 1: That is actually what makes Alpaca different from Last Encrypt and any other external certificate authority that we have in the internet.
Speaker 1: Usually if you get a certificate to put in your website and that's a public website, you would have three certificates. Speaker 1: You have the certificate for your web server. Speaker 1: You have the intermediate certificate and the root certificate, but Workday created more layers on top of it that I'm going to explain why. Speaker 1: And it's going to make sense for you why we have like that.
Speaker 1: So first of all, starting from the bottom of this image, we have the identity and security endpoint certificate. Speaker 1: These two certificates are going to be available inside of your server or inside of your pod. Speaker 1: And it's actually what your application are going to use in the daily work. Speaker 1: Identity is a certificate that you're going to that that's going to tell to the other services that you are who you are.
Speaker 1: And the security endpoint is the certificate for the HTTPS. Speaker 1: I'm going to give you more details in the next slide. Speaker 1: What's important here is your server also has a certificated called actor. Speaker 1: So that actor actually isn't a certificate to validate that your server is something that we trust inside of our infrastructure.
Speaker 1: So then imagine that, for example, if your server get hacked or get leaked for for any reason. Speaker 1: So if we would revoke the certificate that are related to the actor or for your server, then the identity and the security endpoint certificate are going to also be revoked. Speaker 1: And that is the key when you have these kind of layers, if you want to revoke the certificates, your chain or everything that's under that certificate are going to get revoked as well.
Speaker 1: So on top of the actor, we have another certificate that's called Uber actor. Speaker 1: And the Uber actor is in a certificate that it's installed in your cluster, in your platform and through the Uber actor, the platform can send a request to Alpaca and get the actor certificate for the server, install that for you.
Speaker 1: And then the server with the actor certificate, the server, it's able to then connect to Alpaca and request the identity and security endpoint certificate for your application. Speaker 1: So again, these Uber actor, they are created per cluster, per platform. Speaker 1: So imagine that I have three clusters of Kubernetes, each of them would have an Uber actor.
Speaker 1: And if one cluster get compromised, I can just revoke the Uber actor and everything that all the certificates that are inside of that cluster are going to get revoked at anyone else. Speaker 1: In our infrastructure are going to trust on that service because we know that they were compromised. Speaker 1: On top of the Uber actor, we also have the issuing see way CA.
Speaker 1: So as I mentioned, work day, it's a cloud based. Speaker 1: So usually our systems are deployed in different zones in AWS and Google. Speaker 1: So then each zone would have a different issuing see way. Speaker 1: And again, imagine it that the whole zone was compromised for, for any reason. Speaker 1: I can revoking the issuing see way for that zone.
Speaker 1: And then everything else that are there are going to get revoked as well. Speaker 1: So then it's with that it, we make it sure that, for example, if you have any issue. Speaker 1: With that environment, we can revoke and add the rest of the cluster. Speaker 1: The rest of the environment are not going to trust on that, on that region anymore.
Speaker 1: On top of the issue issue way, we have the policy and the policy. Speaker 1: It's basically created for the type of the environment that we have. Speaker 1: So then again, imagine it that your whole development environment that works in different regions get compromised. Speaker 1: Some someone got your private key. Speaker 1: So because of that, if something happens like that, we have the layers so you can revoke everything that under and then.
Speaker 1: Basically, the rest of the components that we have in your infrastructure are not going to trust on that type of environment anymore. Speaker 1: And you can work safely and be secure that that environment that was compromised. Speaker 1: No one is going to receive or accept any request from there. Speaker 1: On top of the CA policy, then we have the root CA and the root CA here.
Speaker 1: And for any certificate authority works to trust in the certificate authority. Speaker 1: So by default, if you have an internal certificate your browser or your server, they are not going to trust in Alpaca or any internal certificate if you don't install the root CA. Speaker 1: So you have to install it and for work today, we have only one certificate and this certificate is broadcast everywhere.
Speaker 1: So then if you check internally one URL to your browser, you're going to have that green locker and in your browser. Speaker 1: Then going over to the other key features. Speaker 1: So what is authoritative test service list and what is identity certificate? Speaker 1: They are kind of related. Speaker 1: So when I mentioned about the Uber actor and mentioned that the platform is able to request certificates to Alpaca.
Speaker 1: The platform cannot request any certificate. Speaker 1: They have they have to tell which services is for each which service that certificate was requests. Speaker 1: And if this service is not described on Alpaca, they cannot get a certificate. Speaker 1: That is kind of a security feature as well, because imagine it that a hacker.
Speaker 1: It's going to bootstrap a pod inside of your cluster and you even know that he's there and he's he's going to use like a name. Speaker 1: I don't know, call it DNA DNS if the DNS name is not described in our service list and it's not going to be there because Workday is very creative for names of services.
Speaker 1: They are not going to be able to get the certificates. Speaker 1: So then this authoritative service list, it works also with another barrier of security to grant the creation of the certificates in our infrastructure. Speaker 1: Also, this service list work with identity. Speaker 1: So at Workday, if two applications want to communicate between each other and want to exchange some information, Speaker 1: We use chase on web token for authentication and authorization.
Speaker 1: And these Jason web token are going to use this identity certificate and in this identity certificate is going to have the name of the server. Speaker 1: So then imagine it that you have an an end point A to services service A and service B and in your service B your end point. Speaker 1: It's saying that you would only receive a data would be authorized only for service A because the service A has has the identity certificate from all pocket.
Speaker 1: When you send this request, the service B is accepted that and authorize you to do to communicate with them. Speaker 1: So that's the reason that we have the identity certificate in our in our infrastructure and the authoritative service list. Speaker 1: Also, we have the security endpoint. Speaker 1: So this one is the is the is the is the is the certificate that's common.
Speaker 1: That's common for people. Speaker 1: So you install the security point in your Apache or your tone cat and then you are going to expose your end point over HTTPS or any other protocol that you want to do encrypted traffic. Speaker 1: Then also, I mentioned about the trust store. Speaker 1: So trust store is a package that that my team controls that has the root CA inside and then it's installed everywhere.
Speaker 1: So we also are responsible for that. Speaker 1: So if we have a new root certificate is or we need to renew or if this root certificate could be internal or external, we also we we put in this package and you distribute this everywhere. Speaker 1: All the servers all the laptops have must to have this package and to be able to communicate over encrypted protocol.
Speaker 1: And at the end, we have also EPIs. Speaker 1: So the only way that we can make service teams that are developers and platform works with you is basically having the EPI because then they can develop some solution on their side or they can automate their side. Speaker 1: And Alpaca has an API. Speaker 1: So for example, for Kubernetes clusters, the guys that take care of the Kubernetes cluster created an operator to be able to request the certificates to Alpaca and mounted the certificates inside of the pod.
Speaker 1: But also with the EPIs, we create a CLI and libraries. Speaker 1: So the CLI actually the develop the developers can use local in your laptop and use the certificates if they want to test on their application and etc. Speaker 1: And the library also you can import the library to talk with our API and this library would be inside of your code for you to use it.
Speaker 1: So the EPI actually make the Alpaca be very successful by the other teams who take Alpaca to use it because we have this EPI actually and they don't need to do stuff manually to request the certificates, for example. Speaker 1: And then actually how this works, not just like this an education purpose, but it's almost close to the what we have in the reality.
Speaker 1: So at work day, so in the orange box, we have Alpaca. Speaker 1: So, and then we have in the yellow box, we have a platform that would be like Kubernetes. Speaker 1: This Kubernetes, they have to have an Uber actor and also imagine it that this Uber actor has the authoritative service list for the service A and service B.
Speaker 1: So then the Kubernetes using the Uber actor is able to fetch two actors certificates for Alpaca and mounting this actor certificates for the pod. Speaker 1: Then the pod with the service A goes to Alpaca as well and requested the identity and the security endpoint. Speaker 1: The same happens with the service B that connects to Alpaca requests the two certificates and with the security endpoint, they are going to expose the HTTPS.
Speaker 1: And then imagine that service A wants to send a post request to an endpoint in service B. Speaker 1: And the service B in this endpoint is is listed there that only would accept it on a message or a data from their service A. Speaker 1: So then when the service when the service A send these requests to the service B, they create a JSON web token with the identity certificate.
Speaker 1: Proving that Alpaca says that he is the service A as he, they are mentioned that they are. Speaker 1: So then service B when receive this request, they can do two things. Speaker 1: They can just accept as it is, or they can look back Alpaca and ask like this certificate. Speaker 1: It's revogated or should I trust in these kind of source.
Speaker 1: If they'll pocket say that, yes, the certificate was created by me and you can trust the source in the look back. Speaker 1: Let's say that then service B would accept the request. Speaker 1: And you would process the message that was sent by the service A. Speaker 1: So this is how actually it works with the trust model.
Speaker 1: And it's in a practical way to use Alpaca internally. Speaker 1: And then going towards the end of my presentation. Speaker 1: What's the future challenges when you have a search internal certificate authority. Speaker 1: So first of all, it's handled with the private keys. Speaker 1: So the private keys for a certificate authority is the most important thing.
Speaker 1: If you leak these private keys, you're going to have a lot of trouble. Speaker 1: And if you follow the RFC, they mentioned that these private keys has to be stored in an offline way. Speaker 1: So in an offline laptop or in an offline server. Speaker 1: So, and the main thing is like how we automate to handle this private key if they are offline, you know.
Speaker 1: So this is one of the things that we have the with the certificate authority, because if you automate it too much, someone can bypass your automation can actually explore and then you can get hacked. Speaker 1: So then this is one of the challenge that that we see in our in our team. Speaker 1: It's like how we automate it without losing the security without being less secure.
Speaker 1: Also working at Workday, we have to follow a lot of compliance and privacy standards. Speaker 1: So, and that is also a challenge because the compliance is, for example, FedRAMP that's for American government and the GDPR and others that we also have to follow some time. Speaker 1: Sometimes they can collide between each other. Speaker 1: So how you make sure that you will have that you are compliance and following the compliance and following the privacies that are requested.
Speaker 1: And also in the feature, I can see that we we might have also to be compliance per segment and also per region. Speaker 1: So let's say that you're going to have something like FedRAMP for you and other regions like Asia. Speaker 1: Or per segment. Speaker 1: So this is also an issue because they can collide between each other.
Speaker 1: Also, another challenge is to use new encryption methods. Speaker 1: I didn't mention about it, but when you create private and public key, you're going to create a hash and to create that hash. Speaker 1: You have to use an algorithm RCA, AES, for example, and now we have elliptical curve. Speaker 1: So then you have to update this new encryption methods for your certificate.
Speaker 1: Also, we have new technologies like Muto TLS, for example, that you have to adapt your application. Speaker 1: And you know, with chat GPT nowadays, who knows which kind of new technologies we're going to have for the SSL and TLS side. Speaker 1: Right. Speaker 1: As I mentioned to hear Alpaca so far was a very successful solution that we deployed.
Speaker 1: But if you're interested to see what happened when a certificate authority lose their private key and which kind of damage that can be done. Speaker 1: I put a link on this presentation that I'm going to share later, and it's in a podcast episode mentioned about GGG notar. Speaker 1: That was a company that had the private key leaked and how hard it was for other companies that were trusting in this certificate authority.
Speaker 1: So if you want to hear also about the not happy path, I invite you to listen this podcast because it's a very interesting story. Speaker 1: Okay, we are almost over 30 minutes of my presentation. I would like to thank you all so much for being here and listening. Speaker 1: What I have to say if you have any question about what I mentioned regarding to the certificate authority SSL TLS or even about Workday, you can drop your message and I'm happy to answer now.
Speaker 1: Thank you. Speaker 1: Okay, I don't see any questions for now. So I think that I'm going to close the session. *music*
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why build your own certificate authority? Lessons from Workday's Alpaca
Jessica Rangel shows how Workday secures TLS and service identity across thousands of microservices with an in-house certificate authority.
In short
Workday built its own certificate authority, Alpaca, because buying one was expensive, still needed custom integration, and open source options did not yet inspire confidence. Jessica Rangel, Senior DevOps at Workday, explains how Alpaca's layered trust model lets the team revoke a single certificate and cut off a compromised server, cluster, cloud zone or whole environment, and why automation, compliance and new encryption methods remain the hard problems ahead.
Key takeaways
The decisions behind investing in your own certificate authority
How to support encrypted connections between APIs in a multi-platform architecture
The challenges of adapting a certificate authority to support future requests
Why a certificate authority matters
Rangel opens with the basics of public key infrastructure. A sender encrypts a message with the recipient's public key, and only the holder of the matching private key can read it. As she puts it: "And because he's the only one that has the private key, he's going to be the only one that's able to decrypt the message and then read the information that is there." (3:53)
That raises a trust problem: how can anyone be sure the right party holds the private key, especially on an internet where attackers can mimic a bank's website? Certificate authorities solve this by vouching that a public key belongs to who it claims to. Rangel notes that, as she cited, Comodo holds around 45% of SSL certificates on the internet. Workday faced the same question internally at scale: how to enable TLS and SSL for thousands of microservices, run by more than 100 teams on Kubernetes, OpenStack and physical servers.
Build, borrow or buy
Alpaca (an acronym for asynchronous low latency platform automation certificate authority) is the only solution authorized to issue, manage, validate and renew digital certificates in Workday's infrastructure, and it follows the practices set out in the relevant RFCs and its certification practice statement. Rangel says it has issued more than 30 million internal certificates so far.
The decision to build dates back to 2014. A third-party product would have cost about 100K dollars a year for 20K servers at the time, and Workday's fleet is now, as she cited, ten times larger. Licensing was only part of the cost: "So you have to invest some development hours, and these would be a little bit on top of this high cost of the licensing." (9:16) Open source alternatives were evaluated too, but this was a year before Let's Encrypt, and none had the documentation, language fit or community adoption the team needed. In 2015 Workday decided to build a solution matched to its performance thresholds and use cases.
A layered trust model that contains compromise
A public website typically uses three certificates: server, intermediate and root. Alpaca adds more layers. Each server or pod holds an identity certificate and a security endpoint certificate (for HTTPS), plus an actor certificate that marks the server itself as trusted. Above that sits an Uber actor per cluster or platform, an issuing CA per cloud zone in AWS and Google, a policy CA per environment type, and a single root CA distributed everywhere.
The point of the layers is surgical revocation. "And that is the key when you have these kind of layers, if you want to revoke the certificates, your chain or everything that's under that certificate are going to get revoked as well." (13:33) If a Kubernetes cluster is compromised, revoking its Uber actor makes the rest of the infrastructure stop trusting everything inside it. The same logic applies to a whole zone or a whole development environment.
Identity, service lists and APIs
Platforms cannot request arbitrary certificates. They must name the service, and Alpaca checks it against an authoritative service list: "And if this service is not described on Alpaca, they cannot get a certificate." (18:23) An attacker who launches a pod under an unknown name gets nothing.
The identity certificate carries the service name and is used to build JSON web tokens for authentication and authorization between services. In Rangel's walkthrough, service A sends a request to an endpoint on service B that only accepts service A. Service B can accept the token as is or check back with Alpaca that the certificate is valid and not revoked. A trust store package, maintained by her team, puts the root CA on every server and laptop.
Adoption depended on automation. Alpaca exposes an API, a CLI for developers testing locally, and libraries to embed in code. The Kubernetes team, for example, built an operator that requests certificates from Alpaca and mounts them inside pods.
The challenges ahead
Rangel is candid about what remains hard. "So the private keys for a certificate authority is the most important thing." (26:15) The RFCs say those keys should be kept offline, yet too much automation creates paths an attacker could exploit. The open question for her team: "It's like how we automate it without losing the security without being less secure." (27:08)
Compliance is another pressure, with standards such as FedRAMP and GDPR that can collide, and the prospect of requirements per region and per segment. New encryption methods such as elliptic curve algorithms and technologies such as mutual TLS also require ongoing adaptation. For the unhappy path, she points listeners to a podcast episode about DigiNotar, a certificate authority whose private key leaked.
Watch the talk
Alpaca - How workday handles with SSL/TLS connections, Jessica Rangel, 32 minutes: https://www.womentech.net/video/alpaca-how-workday-handles-ssltls-connections
About the speaker
Jessica Rangel is Senior Devops at Workday, based in Dublin, Ireland. She has more than 14 years of experience in IT and loves tech events, connecting with people and solving complex problems.
Questions this talk answers
Why did Workday build its own certificate authority instead of buying one?
How does a layered certificate chain help contain a breach?
How do services at Workday prove who they are to each other?
What stops an attacker's pod from getting a valid certificate?
What are the main future challenges of running an internal certificate authority?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
PKI
JSON Web Token (JWT)
Build, borrow or buy
RFC
Certification practice statement (CPS)
FedRAMP
GDPR
Mutual TLS
Resources mentioned
Podcast episode about the DigiNotar private key leak (linked in the speaker's slides)
Workday open positions link (in the speaker's slides)
Cite this talk
Jessica Rangel. "Alpaca - How workday handles with SSL/TLS connections." WomenTech Network, 2023. https://www.womentech.net/video/alpaca-how-workday-handles-ssltls-connections