Services. So today, I wanted to share what I've learned about building distributed systems, how they are evolving, and what it takes them to scale them responsibly.So so in the agenda, we'll be covering the evolution of architectures, the challenges which we face in scaling distributed systems, a short primer on cell based architecture, how we are leveraging AI to automate some of the, some of the scaling aspects in distributed systems, a short, brief, introduction to edge computing, and then I'll open it up for q and a.

So distributed systems are the backbone of modern tech, which power everything from cloud services, AI AIML applications, ecommerce, and IoT. But scaling them today isn't just about handling more load. It's about scaling them in a manner that enables seamless integration across teams and services, how we maintain security, and how we evolve it, in a way that, ensures agility for adding new features and also avoids long term operational debt. The question is, how do you build systems that can scale and continue to work for years from now? Let's take a short look at, the evolution of applications and, you know, how scaling approaches have evolved. We started with the, monolithic architecture, which was a large unified application where all the logic live in a single code base and typically ran as one deployable unit.

While monoliths could be horizontal scale, but by adding like, for example, by adding more instances behind our load balancer. But typically, we have worked on vertically scaling them by just moving them to bigger instances. However, we do run into limits even with horizontal scaling because core components within this application like databases or shared caches often become centralized bottlenecks. And as traffic grew, these shared dependencies made it hard to scale these applications effectively. Also, it made it hard to add new features, like and maintaining these applications. So to address those limits, we moved towards microservice architecture, which was decomposing an application into independently deployable services. With this, we gained fine grained control where each component could scale based on its own load and evolve independently.

This brought agility but also increased the number of moving parts which teams had to manage. And managing this communication, maintaining consistency and observability across these services became a challenge. The next step in this evolution was adopting cloud native patterns. So these emphasize on stateless ephemeral services running in containerized in container orchestrated by platforms like Kubernetes or even serverless environments where the infrastructure is completely abstracted away. So this gave a bunch of benefits. It gave us agility, in and elasticity where as the services came, adding new serve, servers was very easy, and it was also faster to recover, and, also, the scaling happened automatically. But this also adds a little, complexity in how we design these, services to work together, how the failure management handling works, and how the interprocess communication works.

So moving on, let's talk about what are the challenges in scaling, distributed system. One of the challenges is, biggest challenge is data consistency, specifically if you view through the lens of the CAP theorem. What the CAP theorem, says is in the presence of a network partition, a distributed system has to choose between consistency or availability, and it's difficult to achieve both. So at scale, partitions are inevitable. So many systems many distributed systems choose to have high availability by going with the approach of eventual consistency. So this works well for many use cases, but it also places the burden on the developers to handle this data anomaly and design with eventual consistency in mind. Another key challenge is latency. As user expectations have increasing, even milliseconds matter.

Like, you know, scaling pushes the systems, across data center boundaries, making it harder to achieve those low latency responses. To address those latency concerns, we add caching layers or CDNs and edge computing into place, which helps address the latency concern, but also adds its own set of complexity into the system. The next challenge which I would like to look into is fault tolerance. So more node means more failures. Designing for failure, it becomes very essential. We need to invest in redundancy, failover mechanisms. We need systems where we can do chaos engineering where whereas we introduce failures into the system and see how our systems react and make sure that they react and automatically detect, resolve, and address the issue in a, faster manner. So, adding monitoring, detection, and all the message, resiliency systems becomes very essential when you're building distributed systems.

Without a strong automation, strong monitoring and detection, scaling can actually slow things down because teams are running blind, and they don't have visibility into what is happening within the system and when systems are hitting scaling limits. So let's look briefly into what cell based architecture is. So the cell based architecture comes from the concept of a bulkhead in a ship where vertical partition walls subdivide the ship into watertight comp compartments. So these bulkheads are critical. They prevent flooding from one, one, container to another. So that way they limit the blast radius of any leaks happening, and they also provide structural integrity into the ship. Similarly, in distributed systems, cells act as those isolated components. So these are fault isolated boundaries that contain workloads within a limited scope.

If a failure occurs within one cell, it's confined to that cell. Components in other cell continue to operate normally. This model provides not just fault isolation but also operational independence and forms the foundation for building resilient and scalable architecture. So now let's look into let's contrast it with the traditional web server. So in case of a traditional web server, as client traffic increases, we typically scale out the load balancer, add more application servers. We increase the capacity on the database layer. But this introduces several challenges where components become tightly coupled. They share infrastructure, which turns into a bottleneck, and failure in one, component can basically cascade across the entire system.

So even with horizontal scaling where, you know, we can just keep adding servers, we do run into limits where there are limits on the number of connections which you can make to the database layer. So the more pressure we pay place on the shared component, it increases the complexity and blast radius of potential failures. So let's see how cell based architecture tackles this all. So your instead of scaling one last service, we split them into multiple fully self contained cells, each capable of handling a subset of the traffic. At the front, we have a cell router layer, which is the thinnest possible layer, which can be at a regional or regional level. So this layer is responsible for routing traffic.

So when the client traffic comes in, it basically figures out which tenant or which cell that customer traffic needs to go to and sends it there. So now you can think of each cell being its own service. Like, it it would have its own load balancer layer. It would have a compute layer of servers, servers, and a data store layer. So there's no shared infrastructure where that means, none of these components within the cell are shared with other cells. So this means there's no noisy neighbors, no cross cell dependency, and also it reduces the blast radius. So this provides us with workload isolation where failure in one cell does not, impact other cells. Other cells can continue providing the traffic.

It provides us with testability where we can test a given component to the cell limit. Like, we can with the given cell, you can basically, make sure we scale the system to its limit in our test environment and see how the system behaves. Beyond the cell's limit to scale, you can just do horizontal scaling where you keep adding cells, and that's how the cell based architecture will scale scales. This is a very common, architecture pattern which we use in AWS and multiple servers, and it's used for scaling distributed systems. Next up, let's look at how we leverage AI to, for smarter automation within distributed systems. So, traditionally, we relied on fixed scaling rules like, you know, adding capacity when the CPU hits a given threshold, but that approach is reactive and often inefficient.

With AI powered predictive scaling, this predictive scaling algorithm learns historical traffic patterns and user behavior based on the data which is available, and it uses it to forecast the demand and scale ahead of time. So that means that even before the demand comes, you can actually scale up your fee fee to handle the increased load. And then asset, demand, tapers off, you basically get your fleet back to the normal baseline. So this help, this helps with optimizing cost and ensuring the best performance for our customers, of for our customers for the traffic which is coming in. On the reliability side, AI driven anomaly detection continuously monitors system metrics to understand what normal looks like. So it will try to find if there is any unusual latency or traffic drip that might indicate emerging problems.

It will flag that changes, and this enables, the operators to provide a faster response and help with incidents before it and cause big customer impact. So in short, AI enables distributed systems to scale more intelligently and recover more quickly. And, also, similar AI monitoring can be used for different systems. And as, we are getting more exposed, there are so many use cases which are unlocked, which would otherwise take a lot of operational complexity to achieve. So last up, we'll talk about edge computing. So edge computing is transforming how we build low latency systems. Instead of sending all the data to the cloud, we move it closer to this, source. Whether it is the smart self driving car or smart IoT devices or other application, we want to make sure these low compute resources devices, get the necessary low latency benefits of edge computing.

So we have a edge layer which is very close to the device layer. This provides us with low latency. It supports real time decision where the data agreed across this across the devices can be used to make real time quick decisions. And it unlocks new, use cases where these devices can operate without having to connect with the cloud layer, which will add a significant latency to the service. So this is a powerful shift and redefining how proximity is used in distributed system design. And, and it also considers how these devices have low bandwidth resources. So they can rely on the edge servers for all of the bandwidth processing and the low latency needs. So