Proactive Observability: A Strategic Imperative for Scalable Systems

In the fast-paced world of software engineering, system complexity is a constant. Services grow, user demands evolve, and architecture becomes more distributed. Amid all this change, one principle remains vital: clear, continuous observability.

While most teams have some form of monitoring in place, truly resilient, high-performing systems demand more than out-of-the-box dashboards or ad hoc alerts. It’s not about simply having visibility—it’s about having the right visibility at the right time, with actionable insights that drive real decisions.

Let’s explore why proactive observability is not just a technical best practice but a strategic cornerstone for modern engineering.

Seeing the Full Picture

To operate and scale systems effectively, engineering teams must deeply understand how their infrastructure behaves in real time. This starts with asking key questions:

• Are we aware of how load is distributed across our systems? Without this, it's impossible to plan capacity, predict bottlenecks, or scale with confidence.
• Do we have fine-grained insights into processing times? These metrics are critical for identifying slow paths and queue buildups that silently erode performance.
• Are we tracking queue levels at every stage? Queues often reveal what metrics don't: where things are stalling, which services are overwhelmed, and what components are underused.

Surface-level metrics or generic dashboards may offer a sense of control, but they rarely tell the whole story. Observability must evolve with the system, reflecting current architecture, user patterns, and operational risks.

Evolving Dashboards for Dynamic Systems

It’s easy to assume that once dashboards are built, the job is done. In reality, observability is not a one-time project—it’s an ongoing commitment.

Systems are dynamic: new services are introduced, old ones deprecated, usage patterns shift, and infrastructure evolves. As this happens, yesterday’s metrics may become irrelevant or misleading. Dashboards that once offered clarity can quickly become blind spots.

The most effective teams treat their monitoring infrastructure as a living system. They continuously audit what’s being measured, update visualizations, and refine what “normal” looks like. They ask: Are we still seeing what matters most?

Intelligent Alerting: From Noise to Signal

Monitoring without intelligent alerting is like watching the sky for storms without ever checking the radar. By the time something goes wrong, it’s already too late.

But not all alerts are created equal. Smart alerting goes beyond simple thresholds. It considers context, combines related signals, and prioritizes what actually needs attention. Done right, it reduces alert fatigue, prevents critical oversights, and empowers teams to respond quickly—with the right information at hand.

Some key practices include:

• Defining acceptable thresholds based on system behavior, not arbitrary limits.
• Implementing multi-level alerting (e.g., warning vs. critical) to avoid constant firefighting.
• Adding rich context to alerts, such as recent deployment activity or related metric trends.
• Reviewing and pruning alerts regularly to ensure relevance.

The goal is not to alert on everything—it’s to alert on what matters, when it matters.

Observability as Culture

Ultimately, the most resilient systems come from teams that prioritize observability as a core cultural value. This means:

• Building observability into the development lifecycle, not treating it as an afterthought.
• Empowering engineers at all levels to own and refine monitoring for the services they build.
• Celebrating proactive insights that prevent incidents—not just heroic recoveries when things break.

In this mindset, dashboards and alerts aren’t just tools—they’re strategic enablers. They support better decision-making, faster incident response, and more efficient scaling. They help teams move from reactive firefighting to proactive system stewardship.

Final Thoughts

As systems grow in complexity, the cost of poor visibility compounds. Latency spikes go unnoticed. Bottlenecks persist. Downtime lingers longer than it should.

Investing in robust, evolving observability—through thoughtful dashboards and intelligent alerting—is no longer optional. It’s a strategic imperative for any engineering team that values reliability, scalability, and efficiency.

Let’s make proactive observability a cornerstone of modern engineering.

Refactor or Rewrite?

As engineering leaders, we’re often faced with a familiar crossroads: the codebase is showing its age, new features are harder to ship, and technical debt is piling up. The question inevitably arises—should we refactor, or should we rewrite?

The idea of a clean slate can be appealing. Starting from scratch sounds like freedom from the past. But as leaders, it’s our responsibility to think beyond the code and consider the broader impact—on timelines, team morale, product velocity, and business continuity.

Let’s explore both paths with the strategic lens leadership demands.

Refactoring: Evolving What Works

Refactoring is about improving the structure of the code without changing its behavior. It’s incremental, continuous, and safe. Think of it as upgrading the plumbing in a functioning building—you get immediate benefits without shutting the whole operation down.

For engineering teams, refactoring allows:

• Steady delivery: Your team can ship improvements while the system remains usable.
• Knowledge retention: You’re building on years of domain context.
• Lower risk: Fewer regressions, faster testing, and fewer surprises.

Most importantly, every small win delivers value immediately. You fix a naming convention, extract a service, or simplify a function—and the system is better for it that same day.

Rewriting: The High-Risk, High-Cost Option

Rewriting is exactly what it sounds like—starting from scratch. While it promises a pristine architecture and modern tech stack, rewrites defer all value until the very end. Nothing ships until everything’s ready.

That means:

• Delayed ROI: You don’t benefit until the rewrite is complete. Rewrites take, on average, 2–3x longer than initially estimated (McKinsey Digital, 2021).
• Business risk: Your current system stagnates while the team focuses on the new one.
• Increased bug rates: New systems reintroduce old bugs or create new ones. Teams often see 15–25% higher initial defect density in rewritten systems.
• Team pressure: Rewrites often burn out teams and stall innovation. Morale drops when progress is invisible. One internal survey at a major SaaS company found a 23% increase in dev attrition during an 18-month rewrite project.

And the kicker? A classic adage in software holds true:

“You’re never just rewriting the code—you’re also rewriting every bug, edge case, and undocumented decision made over years.”

If you’re considering a rewrite, be brutally honest about your motivations. If your answer is:

• “This code isn’t written well.”
• “I can do a much better job.”
• “I don’t understand why it’s built this way.”
• “I want to build it my way.”
• “I want to create my legacy.”

— then you’re not making a strategic decision. You’re reacting emotionally.

When Is a Rewrite Justified?

There are scenarios where a rewrite is the right choice: when the architecture is fundamentally flawed, the tech stack is obsolete, or the product’s direction has shifted so drastically that the old system no longer fits.

Even then, the decision should be backed by:

• A clear cost-benefit analysis
• A realistic delivery timeline
• A plan to mitigate downtime and knowledge loss
• A strong business case supported by stakeholders

Without these, a rewrite is a gamble—one that too often ends in missed deadlines, frustrated teams, and technical debt in a different flavor.

The Leadership Perspective

As an engineering leader, your role isn’t to chase technical perfection—it’s to enable sustainable, high-impact progress. Refactoring supports that goal by allowing your team to move fast without breaking things. It gives you a path to improve the system while continuing to deliver value to the business.

Rewriting might feel like bold leadership, but unless it’s backed by data and driven by necessity, it’s often just risky.

Choose evolution over revolution—most of the time, it’s the smarter, more resilient path forward.

Tough conversations

Tough conversations — even the thought of them can make my heart race. I loathe them. They leave me jittery, shaky, and emotionally overwhelmed.

Recently, I found myself in a situation where I needed to confront someone. There was no major conflict or alarming issue, just subtle, uncomfortable moments — unspoken tension beneath polite exchanges. My usual approach would have been to let it slide, ignore the discomfort, and move on. I even tried convincing myself that addressing it wasn’t necessary. But this time, something shifted. Instead of brushing it under the rug, I chose to speak up.

The conversation didn’t start smoothly. My hands trembled, and my voice revealed my nerves. I’m sure the other person noticed. But as we talked, the tension eased, and the discussion became more grounded. I found the courage to express my feelings and clarify my needs. It wasn’t about winning or losing — tough conversations rarely are. They’re about fostering understanding, setting boundaries, and improving relationships.

For most of my life, I avoided confrontation, rationalizing that staying silent was easier. But lately, I’ve been challenging that instinct. By leaning into these difficult moments, I’m learning to articulate my expectations with honesty and courage. The results have been transformative — greater clarity, mutual understanding, and a stronger foundation for the future.

I have to credit my wife for encouraging me to take this step. Her support gave me the confidence to push through discomfort and make my expectations clear. Tough conversations may never feel easy, but they’re worth it. Moving forward, I’m embracing the discomfort, knowing that growth comes from facing these moments with an open mind and heart.

Writing and me

I have a deep passion for writing and actively explore various styles, ranging from creative and analytical to technical forms. Whether it’s crafting opinions and analyses on significant events or preparing project and design documents, I seize every opportunity to write whenever I can carve out a few minutes. This habit keeps me engaged, reflective, and intentional about the content I consume.

When I first began writing a couple of decades ago, I believed I could articulate my thoughts exactly as I imagined them. It didn’t take long to realize how much I had underestimated the complexity of translating thought into words.

As a high school student, I was introduced to some of the most beautiful poetry by great poets, with John Keats leaving the strongest impression on me. My first encounter with Keats was during my junior year, when we studied La Belle Dame sans Merci as part of our curriculum. I distinctly remember the summer mornings when my English teacher recited and explained the poem, unraveling its language and beauty. At the time, my language skills were far from polished, and I struggled to fully grasp the nuances of Keats’ words. Yet, I fell in love with his work, captivated by the depth and elegance of his poetry.

Accessing books wasn’t easy back then. Public libraries weren’t available, and most of my time was consumed by mastering high school math, physics, and chemistry. Even so, I found a poetic rhythm in solving complex math problems; they felt, in their own way, as enchanting as the verses of a poem.

A couple of years after graduating high school with excellent grades and acing the entrance exam, I embarked on a bachelor’s degree in engineering. During my first couple of years, though I devoted some attention to my studies, I often found myself lost in my thoughts. One major factor that shaped this state of mind was gaining access to a quality library.

My dad had arranged for a membership at the British Library, conveniently located on my route home from university. Some days, I would stop there, drawn to the modest section dedicated to poetry. It was during those visits that I discovered the works of several great poets, and my perspective on the world began to shift. The intricate beauty of language—how words could be woven together to create something imaginary yet deeply expressive—felt like pure magic to me. Poetry became my new math, a subject of endless fascination and exploration.

Oblivious to the world around me, I lost myself in a utopian dream, spending countless hours and days immersed in that imaginary realm. In reality, I lived like a robot, going through the motions, but in my mind’s universe, I was truly thriving. I lost all sense of time, often forgetting the hour or even the day, as I devoted myself to the solace of poetry.

After two years of being completely out of sync with my real world, I’ve come to realize the impact it has had on my future prospects. I always took pride in my ability to grasp ideas or concepts quickly—often on the first read or explanation. Such was my focus. Yet, during those two years, I failed to exercise that strength. I don’t regret the experience itself; I only regret my inability to balance it with the demands of my regular life. Following that realization, I discovered my love for writing code, particularly in the C programming language. I developed a deep fascination for automata theory and language compilers, both of which captivated me with their logic and elegance.

After graduating from the university, I was fortunate to join the research team at TCS, where I took on a pivotal role in maintaining a business language translation system that could generate error-free C++ code from structured language inputs. It was an incredibly rewarding experience that combined my passion for programming with the challenge of solving complex problems.

Drawn by the highly lucrative opportunities in the financial services industry, I transitioned from my fulfilling and intellectually rewarding role at TCS into banking, where I built a progressive career. My work primarily involved crafting comprehensive memos that detailed business, market, and financial analyses, along with well-reasoned justifications for lending or investment decisions.

Life came full circle when I transitioned back into technology, returning to writing code in C++ and eventually advancing to creating technical documents to present ideas for complex projects. This shift reignited my passion for programming and problem-solving while allowing me to combine technical expertise with clear communication.

Throughout my career, I’ve been fortunate to explore diverse avenues for expressing my thoughts and ideas through writing. Whether as an amateur poet capturing emotions, a programmer crafting efficient solutions in C/C++, or a professional drafting detailed memos to support critical financial decisions, writing has been a constant thread—one that has remained both a deeply fulfilling and integral part of my journey.

S&P 500 Vs VIX

S&P 500's Daily Returns Vs Daily VIX Change.

The daily changes have been plotted by sorting the data from the highest daily increase in VIX to highest daily fall. This plot clearly conveys a high negative correlation between the daily movements of these 2 indices -- as the Fear index (VIX) increases, the S&P 500 index falls.

 

Using VIX to predict S&P Volatility

By definition, VIX is an indication of expected market volatility in the next 30 days. Plotting standard deviation of the 30-day forward S&P standard deviation (Y) against VIX (X) on that day, indicates a decently correlated graph. Correlations between these were found to be 72%. VIX can't explain about a quarter of the volatility we observe in S&P over the next 30-days.

While VIX only indicates expected volatility and doesn't claim to predict S&P returns, I still went ahead and tried to plot 30-day forward S&P returns (Y) against VIX (X). As expected, I found that VIX can only explain about 5.8% of the 30-day forward returns of S&P. The plot indicates a poor correlation.

Analysis was based on daily closing values of VIX and S&P indices from 05-2004 to 11-2024.

Five Key Issues for the New Term of President-Elect Donald Trump

The achievements of Mr. Donald Trump, former President and now President-Elect of the United States, stand unparalleled in recent history. His return to power following a significant defeat in the previous election underscores his enduring ability to garner support across a broad spectrum of the American populace—cutting across lines of race, gender, political affiliations, and other divisions.

Both his popular vote and electoral college victories reflect a mandate that few can dispute. With such a strong endorsement, one can hope that his administration will focus on enacting transformative policies, revitalizing lawmaking, and restoring the country’s growth trajectory. In my opinion, there are five key areas that demand urgent attention: healthcare, immigration, taxation, innovation, and international trade.

1. Healthcare Reform

The U.S. healthcare system, while advanced, is deeply flawed. Despite its capability to save lives, it is one of the most inefficient systems globally, plagued by excessive bureaucracy, waste, and misuse of grants. These grants, intended to foster innovation, often fail to translate into meaningful healthcare outcomes for the taxpayers who fund them. The U.S. has historically been a leader in medical innovation, driven by science and rigorous decision-making. However, placing individuals who disregard scientific principles at the helm of healthcare only exacerbates inefficiency and disorder. Reforms must prioritize efficiency, accountability, and the promotion of tangible benefits for American citizens.

2. Immigration Policy

Immigration remains a broken facet of U.S. policy, repeatedly altered by political motives rather than comprehensive strategy. Legal immigration, which contributes to the economy and supports the tax base, should never be deprioritized in favor of leniency toward illegal immigration, which burdens public resources. Such policies contradict the foundational American principle of meritocracy. President Trump’s unconventional approach to governance uniquely positions him to address these challenges decisively and craft a fair, effective immigration system that aligns with national interests.

3. Taxation and Fiscal Responsibility

The inefficiencies in various government agencies and departments result in a significant waste of taxpayers’ money. Misallocated grants and unwarranted subsidies not only drain public funds but also encourage a culture of dependency and inefficiency. Addressing these issues would allow for a reduction in the tax burden on citizens and businesses, while simultaneously reducing national debt. A leaner, more efficient system can redirect resources toward productive uses and ensure that taxpayer contributions support meaningful and sustainable initiatives.

4. Promoting Innovation

Declining research and development (R&D) incentives since 2017 have dampened U.S. competitiveness in innovation. By reinstating robust incentives for R&D, the administration can reverse this trend and restore America’s position as a global leader in technological and scientific breakthroughs. This requires strategic policy adjustments to balance taxation and foster an environment where innovation can thrive without excessive bureaucratic hurdles.

5. International Trade Policy

President Trump’s firm stance on international trade has been a polarizing issue, but some of his measures are necessary to protect national interests. While outsourcing to low-cost countries may reduce consumer prices, it often comes at the cost of domestic employment opportunities. For outsourcing to benefit the economy, it must be accompanied by the creation of equivalent high-value jobs and accessible skill-transition pathways for displaced workers. Additionally, anti-dumping tariffs should be implemented judiciously to ensure fair trade practices without disproportionately harming domestic industries or consumers.

A Call for Strategic Leadership

With this new mandate, it is crucial for President-Elect Donald Trump to assemble a team of capable bureaucrats and advisors to address these pressing issues effectively. By focusing on these five areas, his administration can steer the country back toward a path of meritocracy, efficiency, and sustainable growth.

The next term offers an opportunity to enact policies that will not only address immediate challenges but also lay the foundation for a prosperous future.

For Richer Experiences, Never Quit!

I vaguely recollect the feeling or the emotional state we were in on that day. However, I could recollect a few key things effortlessly. It was more than 8 years back. We had yet another hectic week at IIM-K. We all had rushed to complete project reports, which took a few night outs to complete. We were planning for the last few classroom sessions, which invariably required making a presentation by our team. In most of the cases, I was to present the case in the last session of the class. For those unaware, all the teams (that had not already presented in the previous sessions) had to be ready with the presentations. Therefore, my team had to prepare presentation for all the cases and get to present the last case to the class. My professors either felt that I was the most capable so gave me the last presentation, which typically involves most of the concepts covers in the entire subject, or felt that I am the weakest and wanted me to prepare the presentation for all the topics! I would love to think it was the former. Further, we had to prepare ourselves for the final exams. That trimester, there was a “wow-feeling” and a “phew-feeling”, because, we had (based on our electives) a few days with multiple exams and a gap of 3 days with no exams in between.

That being our last trimester, I was in no mood to prepare for my exams. Yes, that was the time by when I already had a job offer and was complacent. Luckily, I was able to find a few more of my sort in the A-Top (the top floor of the A hostel). We had a crazy idea of planning one last trek before we left the school. A couple of my friends (Alok and Debu) and I planned for a simple trek – we did not want to be too adventurous in between our exams. While it was a very simple one, it was a very good place to go. Hence, we decided to bike to the base camp early morning, finish the trek in 4 hours and come back by late afternoon. We carried enough stuff to last until the lunch.

Everything was as planned until we reached the spot and we started. After about a few 100 m into the trek, we saw a watchtower nearby and went there. After enjoying the awesome view, we went back to the trek. It is from here that we got lost! We realized that after about 500 m and we thought it would connect to the right path and continued for another 1,000 m. Then we thought a bit and decided that we will not go back, even if we have to climb the mountain, we did not have any gear though! We thought we have come for an experience so it does not matter if we do the trek or not, as long as we have the experience. We went forward until we reached the foot of a hillock. We decide to go up that hillock. We used the mountain grass that grew on the hill as support and climbed our way through. It was really a very tough trek! Mountain grass blades are very sharp; we had many cuts on the palms and entire forearms. However, our focus was on surmounting the hillock! The mountain grass generally has very strong roots; that helped us!

After about an hour of such ordeal, we finally could see the normal path. Nevertheless, we could not understand how our route would connect to it. We did not lose hope! We just went further, but this time, sideways, as there was no more grass upwards, it was only a rock. Luckily, after about another 30 minutes of trek, we could reach the way that connected to the correct path. We finally concluded the trek; of course, we took about two extra hours. Did that really matter? NO! It actually was the best trek we ever had. Apart from adding another one to our list of conquered treks, we experienced a tougher trek – climbing almost 75 degrees incline, we could see the base of the hillock when finding the next place to keep our feet. Among the three of us, I rate myself the weakest who would have quit if not for Debu and Alok pushing me.

This experience taught me that if we are determined, we could surmount all the difficulties and achieve our goals. Being at it is all that really matters! In our daily life, we cannot avoid problems nor can we go back to a time from when it would have been easier to tackle. All that matters in such situation is being at it and not quitting. I always tell myself, quitting is the easiest thing and if we quit we would miss many good experiences that would shape our mental being and make us a stronger individual.

I am, because of that experience and not quitting, a stronger person and have been able to face many adversities in both my personal and professional front. I thank my friends who shared the same crazy feeling of doing the trek between the exams, pushing me and supporting me during that experience.