0:0031:50

Alpaca - How workday handles with SSL/TLS connections

Jessica Rangel 31:50 2023

Workday chose to build its own certificate authority, not buy one

A third-party certificate solution would have cost Workday a hundred thousand dollars a year in 2014, for a server fleet that has since grown tenfold. Jessica Rangel, Senior DevOps in Workday's security product teams, explains why the company built Alpaca, an internal certificate authority that has issued more than 30 million certificates. For anyone securing microservices across clouds, the question is how one compromised cluster stays contained.

Learn how layered certificates let one revocation isolate a compromised cluster, zone or environment

Watch on YouTube

In short

Jessica Rangel, Senior DevOps at Workday, explains how Workday built Alpaca, an internal certificate authority that secures TLS connections and service identity across thousands of microservices on multiple platforms. She covers PKI fundamentals, the build, borrow or buy decision, a layered trust model designed so a single revocation contains a compromise, and the challenges of running a certificate authority in future.

“And that is the key when you have these kind of layers, if you want to revoke the certificates, your chain or everything that's under that certificate are going to get revoked as well.”
“So the private keys for a certificate authority is the most important thing.”

What you will learn

  • Explain how public and private keys encrypt messages and why a certificate authority is needed to vouch for identity.
  • Weigh building, borrowing or buying a certificate authority using licensing cost, integration effort, documentation and adoption.
  • Design a layered certificate hierarchy so revoking one certificate cuts off a compromised server, cluster, zone or environment.
  • Use an authoritative service list and identity certificates with JSON web tokens to authorize service-to-service calls.
  • Anticipate the hard problems of an internal CA: offline private keys, conflicting compliance regimes and new encryption methods.

Key takeaways from the speaker

  • The decisions to invest to have your own certificate authority
  • How to support encrypted connections between apis in a multi platforms architecture
  • The challenges to adapt a certificate authority to support future request

Details

Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
31:50
Published
June 2023

Questions this talk answers

Why did Workday build its own certificate authority instead of buying one?

In 2014 a third-party solution cost about 100K dollars a year for 20K servers and still needed custom development to fit Workday's architecture. Open source options at the time lacked documentation and adoption, so Workday decided in 2015 to build its own.

How does a layered certificate chain help contain a breach?

Each layer (actor per server, Uber actor per cluster, issuing CA per zone, policy per environment type) can be revoked, and everything under it is revoked too. The rest of the infrastructure then stops trusting the compromised server, cluster, zone or environment.

How do services at Workday prove who they are to each other?

Each service gets an identity certificate from Alpaca and uses it to create a JSON web token. The receiving service can accept it or check back with Alpaca that the certificate is valid and not revoked before processing the request.

What stops an attacker's pod from getting a valid certificate?

Platforms must name the service each certificate is for, and Alpaca only issues certificates for services on its authoritative service list. A pod with an unknown name cannot obtain one.

What are the main future challenges of running an internal certificate authority?

Automating around private keys that should be kept offline without weakening security, meeting compliance standards such as FedRAMP and GDPR that can conflict, and adopting new encryption methods and technologies like mutual TLS.

About this talk

With the advent of public and private clouds, Kubernetes and microservices, Workday faced a real challenge to leverage an authentication solution and maintain confidentiality across different platforms and thousands of servers and APIs. Alpaca is our in-house certificate authority solution, to enable dev teams and acquired companies to securely create and manage certificates. Successfully served more than 20M internal digital certificates, we will briefly mention the PKI and JWT fundamentals, decision behind to create our own solution, key functionalities, technical choices, aspects of design, implementation, but also deployment and maintenance.

Topics

Related talks

Browse the library