The Yin and the Yang of managing complex technical projects
Nadia Clifford19:182023
Complex tech projects can't be controlled, only kept in balance
On a typical complexity graph, predictability plummets from 100% to zero at just seven variables, and a modern tech initiative has far more. That is why squeezing complexity into the neat boxes of milestones or even Agile sprints so often fails. Nadia Clifford, a senior engineering program director whose work spans thousands of IoT devices on New York City subway trains, turns to a 5,000-year-old philosophy to ask what should replace control.
Get five practical balances, from mind mapping to t-shirt sizing, for steering complex projects
Nadia Clifford argues that modern technology initiatives meet the scientific definition of complex systems, so linear plans and rigid control are bound to disappoint. Drawing on complexity theory and the Taoist idea of yin and yang, she shows how to manage projects by balancing complementary forces such as structure and flow, detail and vision, scope and time, and execution and reflection.
“Predictability and control fall dramatically as complexity rises, which essentially means it's impossible to know with any certainty the effects of a single action in a complex system.”
“And we have literally shortened projects by several months just because we allowed ideas to flow.”
What you will learn
Recognize why a technology initiative meets the scientific definition of a complex system and why tight control fails.
Open new development with a mind mapping session before moving into structured planning.
Estimate annual roadmaps with high-level t-shirt sizes and save low-level detail for execution.
Decide explicitly as an organization whether time or scope flexes when the two collide.
Build reflection into team and leadership rhythms instead of executing nonstop.
Key takeaways from the speaker
Understanding why complex technology is so difficult to manage and why a lot of methodologies fail
Understanding of a new mindful approach to managing complex technology initiatives
Ideas on balancing structure and flow, such as disciplined execution vs space for innovation
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
19:18
Published
June 2023
Questions this talk answers
Why are complex technology projects so hard to manage?
A tech initiative's software, infrastructure, hardware and people are diverse, connected, interdependent and adaptable, so the system's behavior cannot be inferred from its parts. Predictability falls sharply as variables increase, making rigorous control impossible.
What does yin and yang have to do with project management?
Yin and yang describes complementary opposites in dynamic balance, and imbalance creates stress. Clifford applies it to pairs such as structure and flow, detail and vision, scope and time, and execution and reflection.
How should teams estimate an annual roadmap when details are unknown?
Clifford has teams give high-level relative t-shirt size estimates, framed as gut feel rather than precision. Low-level detail comes later, when the team is ready to execute.
What should happen when scope and time conflict?
The organization needs a crucial conversation about which one it can adjust. If the timeline is hard, scope should be negotiable, and if scope is non-negotiable, the timeline should move.
Does balance mean keeping everything at 50/50?
No. Dynamic balance flows from 80/20 to 70/30 to 50/50, and the natural healthy state is for the balance to move.
About this talk
Nadia Clifford is a seasoned technology delivery leader. She has worked on highly complex projects - from a biotech science system to a crypto platform to the largest IoT network on public transit. As a result, Nadia knows firsthand the pitfalls of traditional linear thinking of fitting high complexity into the neat boxes of project milestones or even Agile sprints.
In this talk, she reveals the secrets of approaching complexity in technology (and beyond) using the ancient philosophy of the Yin and the Yang, balancing Structure and Flow. Find out how this new old mindset can help you see complexity and risk management in a completely new light.
Why predictability collapses at seven variables (79 s)
Mind mapping that cut months off projects (88 s)
Stop exhaling: build in time to reflect (61 s)
Nadia Clifford: All right. Hello, everyone. I'm Nadia Clifford. I'm a senior director of engineering program management at Outfront Media. Thank you so much for joining my session today. I am delighted to be here and honored to have this opportunity to present my thoughts on managing complexity mindfully using the yin and yang philosophy. I have been very interested in complex systems all my career. I've been fortunate to work with some cutting edge tech. And as we all know, tech careers can get very stressful. A few years ago, I was seriously burned out and I had to find a way to manage stress in my life. And that eventually put me on a journey to mindfulness.
Nadia Clifford: And at some point I realized that if I applied these ancient philosophies of yin and yang to that I was learning about to my work and not just my personal life. Nadia Clifford: And to the way I approach technology initiatives things become much more manageable efficient less stressful and not just for me personally, but for everyone working on these projects.
Nadia Clifford: So I did not want to keep this discovery secret and I really would love others to benefit from what I realized so here we are. Nadia Clifford: So we can all appreciate how the complexity of technology has increased exponentially. Nadia Clifford: Even over the last two decades or so and that can be easily shown in the example of our own individual careers. So if we look at my first job in 1997.
Nadia Clifford: I had to build a Microsoft access application to computerize a manual process. It was paper facts swivel chair operation and I hope I'm not the only person here who remembers what a facts machine looked like, but here's a little reminder for you. Nadia Clifford: And just to compare to my current job. I'm working on a network of thousands of cloud enabled smart IoT devices and they run on New York City subway trains, which means underground with spotty Wi Fi with fluctuating power supply.
Nadia Clifford: And nonetheless, they're connected to the cloud they're synchronized and we manage them remotely. So if you just compare the two the scale and the level of complexity are vastly vastly different. Nadia Clifford: And so as complexity of text attack has been increasing the management methods have also been evolving. Nadia Clifford: We started with waterfall in the 70s, then we had agile manifesto that was introduced in 2001, but it has been 20 years.
Nadia Clifford: And if we think how much complexity has increased since 2001 it makes one pause and think, is it time to expand that mindset again. Nadia Clifford: So let's talk about it. What makes complexity so difficult to manage. I'm sure we've all been through situations when we changed something small in a system over here and it went.
Nadia Clifford: Kaboom in a big way over there. Right. So why does it happen. Nadia Clifford: So, first, we're going to get a little bit nerdy and I'm going to use science specifically complexity theory to explain why modern tech can be so difficult to manage. Nadia Clifford: And then I'm going to introduce a very different concept, Yin and Yang, and this is an idea that is 5000 years old and it stems from Taoism.
Nadia Clifford: And we will explore what it can teach us about managing complexity and then i'm going to offer some practical ways to apply this new old thinking in your work. Nadia Clifford: So now let's look at how science defines what a complex system is so according to the complexity theory, a system can be considered complex if it consists of entities that are.
Nadia Clifford: Diverse connected interdependent and adaptable. Nadia Clifford: So now that we defined a complex system let's see how that definition compares to a large technology initiative. Nadia Clifford: But first let's see what kinds of entities we may have in a complex technology initiative. Nadia Clifford: So we have multiple software components some are written in house some of third party libraries we have back and front end we have infrastructure components such as servers balance loaders.
Nadia Clifford: Network work when we have hardware devices firmware and above all, we have people and people can also be organized into entities such as teams departments vendors customers and users. Nadia Clifford: You get the idea. Nadia Clifford: So are these entities diverse. Nadia Clifford: Yes, of course. Nadia Clifford: Are they connected. Nadia Clifford: Yes, they're all part of our ecosystem.
Nadia Clifford: Are they interdependent. Nadia Clifford: Absolutely teams can depend on each other. Nadia Clifford: Front end depends on back end. Nadia Clifford: And application behavior depends on user behavior, etc. Nadia Clifford: And are they adaptable. Nadia Clifford: Yes, as simple as an if then statement in your code, if a happens, then do this else do something else.
Nadia Clifford: This is basic adaptability. Nadia Clifford: And if we talk about people, then adaptability just seems almost infinite. Nadia Clifford: So we can safely conclude that attack initiative meets the definition over complex system, according to science. Nadia Clifford: So now that we have an overview of the system we're ready for a little bit more detail. Nadia Clifford: So what do scientists say about complex systems system properties cannot be inferred from its components.
Nadia Clifford: What it means, we cannot tell how the system is going to behave as a whole, even if we know its individual components system may not respond. Nadia Clifford: According to defined rules, even if we just focus on people right and not tech, we have social norms, we have processes, but we can never predict with 100% certainty how someone is going to act.
Nadia Clifford: Response to change can be unpredictable whether they're local within the system or global outside of the system system into entities can modify the system itself. Nadia Clifford: And it's self organizing because of adaptability so basically it takes a life of its own so rigorously controlling a complex system is impossible. Nadia Clifford: So this is a typical graph for complex system we're essentially plotting predictability against number of variables and predictability starts at 100% when number of variables is one.
Nadia Clifford: And then it plummets to zero when number of variables is just seven. Nadia Clifford: So if you think back on that word cloud that I shown of entities in the tech initiative, we have way more than seven and each of those things has variables within it. Nadia Clifford: So let's summarize. Nadia Clifford: Predictability and control fall dramatically as complexity rises, which essentially means it's impossible to know with any certainty the effects of a single action in a complex system.
Nadia Clifford: Complexity increases with a number of variables, which means that larger chunks of work are even less predictable. Nadia Clifford: So now that we've established the problem with complexity let's talk about mindfulness leave each moment practice being mindful be present it hold on I have to take this I love this picture. Nadia Clifford: So it's not just tech right our lives have also been getting more and more complex in the recent decades and as complexity rises so does rise the human stress.
Nadia Clifford: And it's no wonder that as a society, we have started turning to mindfulness and isn't it interesting how mindfulness starts to come into the mainstream. Nadia Clifford: At the same time as the complexity of our collective lives starts increasing and these concepts are thousands of years old and they have withstood the test of time very well they're really good.
Nadia Clifford: At managing the most complex system of all, which is life itself they're simple powerful yet not very easy to implement and doesn't that sound familiar. Nadia Clifford: Ever tried implementing agile. Nadia Clifford: So I don't claim. Nadia Clifford: To be an expert in ancient philosophies, but I have been extremely interested in one concept and how it applies to our modern lives, which is the concept of yin and yang.
Nadia Clifford: And this is the image that illustrates it it comes from Daoism and is almost 5000 years old. Nadia Clifford: And it says that opposites that are complementary to each other. Nadia Clifford: And that form a dynamic whole and they give rise to each other and they are part of each other and basically balancing the two is the ultimate goal.
Nadia Clifford: And when these energies are out of balance that creates stress and hardship. Nadia Clifford: So let's look at some of the examples of yin and yang energies, you will see that they're not really opposites they complement each other. Nadia Clifford: We cannot have light without dark we cannot give something without someone else receiving there's no fullness without an emptiness to fill and there's no exhale without an inhale.
Nadia Clifford: And this image really doesn't do this concept justice so let's take a look at a video. Nadia Clifford: So notice how these two energies are not static at all they're constantly moving and replacing each other they're in a dynamic balance. Nadia Clifford: And when these inner energies are not in a state of a natural balance that creates stress, for example, imagine maximizing light and dark at the same time.
Nadia Clifford: It's hard to do but possible right so maybe we blink a bright light on and off in a dark room and how long would you be able to stay calm in a room like that that be very stressful right. Nadia Clifford: So now compare that with experiencing a sunrise when the amount of light is increasing, while the amount of dark is decreasing it's a peaceful experience that is a natural dynamic balance it may be a crude example, but it illustrates the concept.
Nadia Clifford: So now i'm asking a radical question can this ancient philosophy be applied to a complex system. Nadia Clifford: And I found a quote from a renowned expert in complexity theory and the author of at home in the universe, Stuart Kaufman. Nadia Clifford: The faith of all complex adapting systems in the biosphere from single cells to economies is to evolve to a natural state between order and chaos, a grand compromise between structure and surprise.
Nadia Clifford: And doesn't that sound a lot like the concept of yin and young and the idea of an ultimate balance between the two. Nadia Clifford: So let's apply the concept of balance between yin and young to complex technology projects. Nadia Clifford: I picked out several complimentary energies and I hope that will help you discover how you could get better at managing complexity and therefore managing stress.
Nadia Clifford: And it's been proven many times over that humans and systems perform better longer when not under stress. Nadia Clifford: So applying structure moves the system into a more ordered state away from a complex state. Nadia Clifford: It could be a good rigorous process it could be disciplined agile ceremonies. Nadia Clifford: But then, when we have a lot of structure that means the flow diminishes so an unconstrained flow of ideas and creativity may be something we need to innovate.
Nadia Clifford: If we apply the idea of balancing yin and young, then perhaps there are times when we need to let go of the structure and let the ideas flow and emerge. Nadia Clifford: I love starting new development with a mind mapping session that's a collaborative brainstorm to build out a visual tree of ideas, a great way to uncover hidden barriers and open up a flow of ideas.
Nadia Clifford: And we have literally shortened projects by several months just because we allowed ideas to flow. Nadia Clifford: And only after mind mapping exercise would we move into a structured planning process balancing structure and flow. Nadia Clifford: Let's look at balancing low level detail with visioning I often run into situations when the leadership team wants to have an annual roadmap sketched out.
Nadia Clifford: But development teams are too worried about figuring out all the details before they say how long something would take in this case, I always have the teams provide high level relative sizing estimates. Nadia Clifford: Let go of the details let's zoom up so is it small medium extra extra large you set the expectations that these t shirt estimates are you know God feel rather than precision.
Nadia Clifford: But then, when you're ready to execute that's when you focus on the low level detail that's when you drill into the specifics and narrow down your time horizon. Nadia Clifford: Scope and time in my experience people often do not understand that these two are opposing energies in the typical development team, you have two forces that are naturally pushing against each other.
Nadia Clifford: Product management pushing for more features and development pushing for more time to build quality. Nadia Clifford: When that picture becomes out of balance that's when we get into trouble. Nadia Clifford: I have mentioned t shirt sizing for an annual roadmap that is usually based on high level understanding of scope and essentially creates a high level time boundary.
Nadia Clifford: But the more we dig into the specific initiative, the better sense we get on the true scope of it and potential timing of delivery, and this is when it's critical to have those crucial conversations as an organization. Nadia Clifford: What are we in a position to adjust time or scope if your timeline is hard your scope is new it should be negotiable if your scope is non negotiable your timeline should be moving it's a natural balance of things.
Nadia Clifford: So this is one of my favorites in our modern hustle culture, we don't usually allow enough time for reflection we're usually executing on exhale forgetting to inhale. Nadia Clifford: pause and reflect and that's when we lose our balance and become very stressed this applies not only to individuals, but also to entire teams and organizations, we cannot always be hustling we have to balance that with periods of reflection and strategizing.
Nadia Clifford: And it's not only scrum teams that need to have those moments of reflection and their retros executive leadership teams can benefit from those even more as that has the potential to affect the entire organization. Nadia Clifford: And one more important thing to know that dynamic balance doesn't mean things are always 50 50 it means that they flow from 80 20 to 70 30 to 50 50 etc.
Nadia Clifford: The natural healthy state is for the balance to move. Nadia Clifford: There's no time for thinking we have to make a management decision haven't we all been there. Nadia Clifford: So most of my career I've been a change agent I know how hard change can be I also know how hard it can be to accept a situation you desperately want to change.
Nadia Clifford: That's another opportunity to fine tune that balance to know when it's appropriate to push for change and maybe when it's time to step back assess and respect what is in place and works already. Nadia Clifford: Whenever I start a new job, I see so many things that I would love to jump in and improve. Nadia Clifford: I literally have to restrain myself to wait and see and sure enough, after I spend maybe a few weeks trying to understand the present realities of this organization.
Nadia Clifford: I much better position to understand why things are done in a certain way and i'm much wiser and how to move the needle in the direction I wanted to move. Nadia Clifford: So when we become aware of these opposing and at the same time complimentary energies we become much wiser at purposefully shifting this balance in our work environments.
Nadia Clifford: And thank you very much for attending today, I hope you have some takeaways maybe you've thought of some imbalances in your own systems. Nadia Clifford: Please do reach out to me, let me know what you think and if you want to chat about it further find me on linkedin you can either scan this qr code or you can just search for nadia clifford and thank you very much for your time today.
Nadia Clifford: And I hope you enjoy the rest of the conference. Nadia Clifford:
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why complex tech projects need balance, not more control
Nadia Clifford combines complexity theory with the ancient idea of yin and yang to offer a calmer way to run complex technology initiatives.
In short
Nadia Clifford argues that modern technology initiatives behave like complex systems, so trying to control them tightly with linear plans is bound to disappoint. Drawing on complexity theory and the Taoist idea of yin and yang, she proposes managing projects by consciously balancing complementary forces: structure and flow, detail and vision, scope and time, and action and reflection. The result, in her experience, is work that is more manageable, more efficient and less stressful for everyone involved.
Key takeaways
Understand why complex technology is so difficult to manage and why a lot of methodologies fail.
Understand a new, mindful approach to managing complex technology initiatives.
Get ideas on balancing structure and flow, such as disciplined execution versus space for innovation.
Why complexity defeats control
Clifford opens with a contrast from her own career. Her first job, in 1997, was building a Microsoft Access application to computerize a paper and fax process. Today she works on a network of thousands of cloud-enabled smart IoT devices running on New York City subway trains, underground, with spotty Wi-Fi and a fluctuating power supply, yet synchronized and managed remotely. Management methods have evolved too, from waterfall in the 1970s to the Agile Manifesto in 2001, but she questions whether they have kept pace: "And if we think how much complexity has increased since 2001 it makes one pause and think, is it time to expand that mindset again." (3:06)
To explain why things change in one place and "go kaboom" in another, she turns to complexity theory. A system is complex when its entities are diverse, connected, interdependent and adaptable. A tech initiative ticks every box: in-house and third-party software, front and back ends, servers, networks, hardware and firmware, and above all people organized into teams, departments, vendors, customers and users. Such systems cannot be understood from their parts alone, may not follow defined rules, and can modify themselves. "And it's self organizing because of adaptability so basically it takes a life of its own so rigorously controlling a complex system is impossible." (7:01)
She shows a typical complexity graph in which predictability starts at 100% with one variable and plummets to zero at just seven, as the speaker cited. Her summary is blunt: "Predictability and control fall dramatically as complexity rises, which essentially means it's impossible to know with any certainty the effects of a single action in a complex system." (7:44) Because complexity grows with the number of variables, larger chunks of work are even less predictable.
A 5,000-year-old idea for a modern problem
Clifford's route to this thinking was personal. A few years ago she was seriously burned out, and managing that stress led her to mindfulness. She noticed that mindfulness entered the mainstream just as the complexity of our collective lives was rising, and that its ideas have long been used to manage "the most complex system of all, which is life itself."
The concept she focuses on is yin and yang from Taoism: opposites that complement each other, give rise to each other and form a dynamic whole. There is no light without dark and no exhale without an inhale. "And when these energies are out of balance that creates stress and hardship." (10:22) She illustrates the point with a light blinking on and off in a dark room, which would be stressful, compared with a sunrise, where light grows as dark recedes in a natural balance.
To connect philosophy and science, she cites complexity theorist Stuart Kauffman, author of At Home in the Universe, who describes complex adaptive systems evolving toward a state between order and chaos, a compromise between structure and surprise.
Structure and flow, detail and vision
The first balance is structure and flow. Rigorous process and disciplined Agile ceremonies move a system toward order, but too much structure chokes the free flow of ideas that innovation needs. Clifford likes to start new development with a mind mapping session, a collaborative brainstorm that builds a visual tree of ideas and uncovers hidden barriers. "And we have literally shortened projects by several months just because we allowed ideas to flow." (13:59) Only after that does the team move into structured planning.
The second is low-level detail versus visioning. Leadership often wants an annual roadmap while development teams want every detail settled before committing to a timeline. Her answer is high-level relative sizing with t-shirt estimates, framed as gut feel rather than precision. The detail comes later, when the team is ready to execute and narrows its time horizon.
Scope, time and the need to inhale
Scope and time, she says, are opposing energies that many people misunderstand: product management pushes for more features while development pushes for more time to build quality. "When that picture becomes out of balance that's when we get into trouble." (15:24) As an initiative becomes clearer, the organization needs crucial conversations about which lever it can move. A hard timeline means negotiable scope, and fixed scope means a moving timeline.
Her favorite balance is execution and reflection. "So this is one of my favorites in our modern hustle culture, we don't usually allow enough time for reflection we're usually executing on exhale forgetting to inhale." (16:15) Retrospectives are not only for scrum teams; executive leadership teams may benefit even more, since their decisions affect the whole organization. Balance also does not mean a fixed 50/50 split. It flows from 80/20 to 70/30 and back: "The natural healthy state is for the balance to move." (17:16)
Knowing when to push for change
As a long-time change agent, Clifford also balances pushing for change with accepting what already works. "Whenever I start a new job, I see so many things that I would love to jump in and improve." (17:44) She restrains herself for a few weeks to understand the organization's realities, and finds she is then far wiser about how to move the needle. Her closing message ties the framework together: "So when we become aware of these opposing and at the same time complimentary energies we become much wiser at purposefully shifting this balance in our work environments." (18:16)
Watch the talk
"The Yin and the Yang of managing complex technical projects" by Nadia Clifford (19 minutes): https://www.womentech.net/video/yin-and-yang-managing-complex-technical-projects
About the speaker
Nadia Clifford is Sr Director, Engineering Program Management at Outfront Media, with over 25 years of hands-on experience delivering complex technology initiatives and digital business transformations across retail, media, financial services and technology. Her recent endeavors include completing MIT's digital transformation program and building out program management from the ground up for a highly innovative IoT engineering organization.
Questions this talk answers
Why are complex technology projects so hard to manage?
What does yin and yang have to do with project management?
How should teams estimate an annual roadmap when details are unknown?
What should happen when scope and time conflict?
Does balance mean keeping everything at 50/50?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Yin and yang
Complexity theory
Waterfall
Agile Manifesto
Scrum retrospectives
T-shirt sizing
Mind mapping
Resources mentioned
At Home in the Universe (Stuart Kauffman)
Cite this talk
Nadia Clifford. "The Yin and the Yang of managing complex technical projects." WomenTech Network, 2023. https://www.womentech.net/video/yin-and-yang-managing-complex-technical-projects