Why Transparency Matters

Sometimes it astounds me just how much transparency exists in the world today, and how much benefit we derive from it. In some cases, it's the incredible access to information that we have, mostly thanks to the Internet, while in others, it's simply how willing people are to share what they're doing.

As examples of the former, I started making a mental this morning of some of the data that's now almost literally "at my fingertips" that would have, even a decade or so ago, been at least a phone call or trip somewhere away:

• animated radar images of the current precipitation patterns in my locale, so that I can plan my biking forays accordingly
• real-time, up-to-the-minute sports scores, including details like who's pitching, who assisted on what goals, and how many shots have been taken
• specifications for devices like pool heaters and DVD players that you may have long since lost the paper copies of
• forms, as well as forums, related to topics like Canadian Income Tax, immigration policy and travel destinations, just to name a few
• dictionary and encyclopedia content enabling anyone to write a factual, typo-free essay (or blog post) without having to even get up out of their chair
• transit system schedules so that you know exactly when to be where to catch what (bus/train/plane)
• and thousands and thousands more

Similarly, we're able to get much greater insight into, say, the creative process behind a television show, movie or comic book than we ever could in the past, thanks to creative folks who "put it all out there" for the world to see. Not that long ago, much of that would have been closely-guarded secrets, or simply considered too pedestrian to warrant the time to write it down and the cost to publish it. Now, however, it's just a few clicks away.

So what does any of this have to do with Agile, or software development in general? Well, I think that we're realizing many of the benefits of transparency in the 21st century (as touched on above), but that doesn't necessarily always get applied to how we run our development projects. In an Agile culture, we strive for transparency in some very big ways - Burndown Charts, strictly-prioritized Product Backlogs, Retrospectives - all of which are important and positive. If you compare that to a more traditional (Waterfall-style) software project, though, you can see that the same degree of transparency is seldom achieved there. Team members may seek to hide some of what they're doing out of fear that the work won't be approved if the bosses find out about it. Project coordinators may talk about, or even hope for some sort of post-mortem to be held after the product finally ships, but rarely do those mythical meetings actually happen. Status reports all too often end up being "green-shifted" by those responsible for providing the data, in order to keep the alarm bells from sounding early on. These are all instances where the lack of transparency hurts the success of the project.

Even in an Agile environment, however, there can still sometimes be a resistance to openness. I certainly encountered many situations in my two years as Agile Manager that spoke to this point. Whether it be shutting down conversations or blogs that are bringing to light unsavoury failings among management, or a reluctance to share with others the challenges that are being encountered (because it might be used against you later), it's no easy task to adopt a culture of transparency. Politics can get in the way, as can pride, fear and an exaggerated sense of self-preservation. None of those are trivial obstacles to overcome.

Unfortunately, in virtually every case this tendency toward hiding facts ends up doing more harm than good. Decisions end up being made based on faulty data, morale problems develop thanks to a cloak of secrecy that seems to always inspire conspiracy theories much worse than the reality of the situation, and people waste time going through hoops trying to find "the person with the answer." When I think about how efficiently I can use my time today, thanks to the wealth of information available to me, compared to what life was like when I first started working... it's like night and day. And that's because, given quick access to the necessary data, it's always easier to make a reasonable, informed decision. That perspective sometimes gets lost in the heat of the office battles, though. Those of us in the Information Technology industry need to remember that, as we go about our day-to-day business of cranking out top-notch software products, information is king. And the king belongs to everyone.

So if you're reading this from your desk at work right now, ask yourself: what bit of data are you holding on to right now, that would better serve the company if it were shared? Don't betray any confidences (personal or professional), but rather look for something that you've simply been lax in getting out there, or that you know would help someone else if you made the effort to tell them about it. Sometimes that's all it takes.

Learning May Actually Help Keep Us Young

I'm reading a book called The Brain That Changes Itself, by Norman Doidge, M.D. In it, the author describes many different examples of the human brain's neuroplasticity, which is to say: its ability to change itself based on various stimuli. For decades, the belief was that each part of the brain had a certain, unchangeable function to perform. This implied that an injury to any part of the brain meant the loss of that type of functionality, forever. More recently, though, neurologists have begun to uncover many examples where the organ inside our skulls is actually much more adaptive than that rigid model would allow for.

One of the topics that Dr Doidge covers which certainly applies to Agile is the importance of continuing to learn, even after completing the scholastic period of our lives. In studies that have been done with those in their sixties and beyond, there's clear evidence that elderly individuals who seek out ways to engage their mind - such as by learning a new language or dance, taking on a new hobby or even just visiting locales that they've not been to before - are less likely to suffer from dementia and other symptoms typically related to "the aging of the brain." New neuro-connections continue to form as we get older, but only if we exercise our mental facilities during that latter stage of life. As Dr Doidge points out, it's still possible that it may be a correlation rather than causation relationship, or even causal but in the other direction. In other words, those who are actively keeping their minds fine-tuned may only be able to do so exactly because they aren't losing their edge (rather than the activities being responsible for the sharp minds). But research is making that seem less and less likely, and so I'm inclined to believe that there is more of a positive cause and effect relationship at work in those scenarios, and certainly the rest of the book is a tour of amazing brain-alterations that have been observed by researchers.

So if we accept - optimistically, perhaps! - that learning really does help to fight off some of the effects of aging (such as senility and dementia), then how do we adopt a way of life that promotes learning? In my capacity as Agile Manager between 2006 and 2008, I encountered a lot of individuals who truly epitomized the answer to that question. For them, the focus was always on discovering new ways to make their product better. Often, the people around them (especially in management) were fixated on "hitting a date", which often translated to "cutting corners." The friction caused by the intersection of those two very different approaches was probably healthy, in the final analysis, although it rarely felt that way at the time.

In life, though, we're less likely to constantly run into schedule pressures in quite the same way as we do in our vocation as software craftspeople. Granted, there may never seem to be enough time in the day, but we still get to decide - within reasonable bounds, set by family obligations - just how to spend our spare time. Do we "veg out" and turn our brain off every chance we get, or do we look for ways to stimulate our mind and imagination? Are we always satisfied with the way things are, or do we look for paths to improvement? Have we locked ourselves into a comfortable pattern, or are we still growing in ways not related to our waistlines?

I obviously have a lot more free time now than I did just a year ago. Even so, I've made a conscious effort to spend significant portions of it reading, writing and generally trying to bring my overall understanding of the world around me to a higher level. I'm also finding that my "second career" as a Math tutor is requiring no end of learning on my part (how's that for irony?) as I discover that every child really is different, and therefore no one approach works for every situation.

Will any of that do any good? Am I still destined to end up senile (or, as AgileBoy would put it, "more senile")? I don't have any answers, but I think it's important that we all all keep asking questions.

How Can Productivity Really Be Increased?

As our current Recession continues to wreak havoc across the employment ranks, one topic that we'll all surely hear more of is "increasing productivity." It's the conjoined twin of "reducing costs," after all, which has already proven to be a popular refrain since last Fall. Once a company gets to the point where it doesn't believe that it can effectively reduce the workforce any further, the next logical step is to look for ways to get more out of whatever employee base remains. I suspect that that's already happening; if it isn't, then it probably soon will be. (This, too, shall pass... but maybe not anytime soon.)

I want to say right off the top: increasing productivity by making people work longer hours isn't the answer. It may work in the very short term, but it certainly doesn't over the long haul. All kinds of studies have proven this, to the point where I'm not even going to bother citing them here (this stance against sustained overtime should be accepted by now, I hope).

So how do people become more productive? Since this is an Agile blog, I'm talking here about the so-called "knowledge worker" industry, rather than something more manual-based, where productivity improvements often take the form of automation. It's much harder to replace a Computer Programmer, Project Manager or Business Analyst with a machine or a piece of software, so just how are productivity boons realized?

The answer that was already floating around when I first hit the industry back in the mid-80s was, "Work smarter." Now, it was also the case that not too many people had any notion of specific ways in which to accomplish that, but it was still a worthwhile goal. What I learned, over time, was that working smarter required a certain amount of reflection and adaptation, both of which fit quite nicely with the principles of Agile. Specifically, you could increase productivity if you were willing to root out, and then call out, parts of your process that were inefficient. The Toyota Lean Manufacturing approach essentially crystalized this mindset and represents the most pure application of it that I've yet encountered. It's one of those things that's easier said than done, though, and it requires a superior style of leadership that's quite frankly lacking in most organizations.

But even if you work in an environment that's not based around Lean, I think that you can still apply some of the same principles. Some of your success will be limited by how much empowerment exists, since making change in an empowerment vacuum is, if not impossible, at least ridiculously difficult. Therefore assessing the latitude with which you can effect change is probably the first step to take.

Assuming that you really do want to become more productive and you have at least some wiggle room within which to operate, then the next thing to do is look for obvious inefficiencies in your current processes. In Lean, I believe these are referred to as "waste." One way to spot these opportunities involves doing some sort of workflow analysis, paying particular attention to items that either introduce delays, or which seem to have limited value. There's more to it than that, but generally speaking you're taking a critical look at "how stuff gets done" and trying to identify those areas where the effort required isn't exceeded by the reward delivered.

As an example, imagine a process in which one step requires a meeting between representatives of several different departments, during which the various attendees provide feedback, and ultimately approval, on Business Requirements. It might be the case that setting up such a meeting often introduces a delay of up to several weeks, due to the lack of availability of the players. Therefore, you might actually be able to shave a project's timeline considerably if you could find a way to gather that same feedback and approval without requiring everyone to physically attend a meeting. Perhaps it could be done electronically, or in two or more smaller groups... regardless, the goal would be to see if it was possible to accomplish the same outcome without incurring the large elapsed-time cost.

In my capacity as Agile Manager, I saw something like this done with Code Reviews. The acquisition of a product called Code Collaborator enabled two improvements to occur: first, a dramatic increase in the amount of peer reviewing of code changes (thus, improving quality of the delivered product); and second, allowing developers and testers to review code at their desks, during breaks between other activities. The win-win nature of this change, whereby both the product's quality and the impact on the schedule were positively affected, made it an outstanding success story.

One of the biggest challenges to making changes toward improved productivity, of course, is that it involves change. As anyone who's ever moved to Agile knows all too well, change is hard. Some people resist it, and many of those individuals are quite adept at throwing up obstacles to any change that they feel threatened by. The introduction of Automated Tests, for example, may represent a huge peril to anyone who's built a career around manual testing. The trick, of course, is in helping those people see what sorts of operations Automated Testing is appropriate for, as well as re-purposing the human brains that used to do that work to something that can't be automated. It's not easy, but it can be done... and the end result is improved productivity.

Other types of change that may meet resistance involve switching up how others do their job. Even something as seemingly trivial as moving from a big meeting room setting to doing something electronically may cause some folks to recoil in disgust. They may actually like attending meetings, for one thing. But beyond that, they may have legitimate concerns about how effectively a set of requirements can be reviewed without the back-and-forth discussions that they're used to in those meeting rooms. How you handle those objections will go a long way toward determining how far you get toward your goal of productivity increases. As someone well acquainted with "inspect and adapt" these days, I recommend that you set up any alteration as an experiment, and then monitor the results so that everyone can see whether it's working out or not.

I think a basic question to be prepared to ask at every step is, "What value are we getting from this part of the process?" That can be an unpopular query to pose, especially when you get to the part where, for example, Bob's entire raison d'etre is called into question. You can bet your bottom dollar that Bob will have a handful of justifications at the ready for why filling out that form in triplicate (two of which get shredded) is a key component to the checks and balances of the entire company. Being able to handle situations like that require more tact than I could ever advise on, so don't be afraid to seek out your best diplomat in management if you find yourself down that path.

The bottom line, I guess, is that you can make productivity gains, if you put your mind to it. It just requires an analytical approach and a willingness to change. One of the alternatives, of course, is that all of your jobs get outsourced to a country where the knowledge industry works at less than half your rate... and nobody wants that, I'm sure!

The Software Development Cycle

With my recent contract now complete, I thought I'd post a few non-specific details about what I did during it. While it's not completely germane to Agile, you'll see a few places where there was some overlap, at least.

There were basically three parts to complete as part of the contract:

• review the organization's current software development process
• compare that picture to what a more standard development process would look like
• provide recommendations on changes or enhancements that could be implemented to make things even better

Initially there had been some thought of doing a fourth part, as well: implement some of the recommendations, alongside the development team. However, virtually none of that came to be, through a combination of factors the most relevant of which was that the developers were in the middle of a high priority project that couldn't be delayed at the moment.

In order to get a clear idea of how development happened in this company, I read over a fair number of documents that related either to standards, or to recent projects, within the organization. From that exercise, along with a few strategically-timed interviews with personnel there, I was able to figure out what type of artifacts were produced, as well as which practices were followed. I'd describe what I saw as a fairly lightweight model, which is what I'd expected to find, given the age (fairly young), size (6 to 8 developers) and position within the company (an Information Technology group within an industry that's far removed from being an IT provider). In some areas, it was really only rigour that was missing; in others, there was simply a lack of experience with "best practices for large-scale software development", let's say. And that's the sort of thing you bring a consultant in for, anyway.

For the comparison aspect, my boss suggested a Gap Analysis in the form of a spreadsheet, which I thought was quite appropriate. The U.S. arm of the company had previously brought in-house a very comprehensive System Development Life Cycle model for the American IT groups (which are considerably larger than the Canadian one located here), and I was happy to use that as my baseline for the comparison. This model was very thorough (and very gated, as befits a Waterfall approach) and so it provided an excellent "ideal" for me to measure their setup against. There were roughly 150 artifacts or activities included in it, and so I was able to evaluate my clients' methodology in terms of which of those items they included, and (in a few cases, at least) how their versions measured up to the baseline.

Once the Gap Analysis was complete, I identified what I believed to be the highest priority gaps. This ended up being just over 20 items. For each, I then wrote up a Recommendation, in the following format:

Recommendation: xxx
Benefits: xxx
Approach: xxx
Effectiveness Metrics: xxx

In other words, I provided the what, the why, the how, and some notion of how to measure the results to make sure that they weren't wasting their time making the change.

That Gap Analysis spreadsheet, which included the 20-some Recommendations, was the primary deliverable from my contract. However, in talks with my boss we decided that a couple other concrete artifacts would be desirable to come out of this, as well. The first was a Microsoft Project Plan that was essentially a skeleton Plan that included all of the steps from the "ideal" SDLC model, customized somewhat to their environment, in order to make it easier for all of the steps to be remembered in the future. Also, because I considered a thorough and comprehensive Release Plan to be a prime example of where some additional rigour would help them out, I provided a template for that, too.

Those three documents were reviewed with most of the development staff on my last day there, and were extremely well-received. I had tried to incorporate as much feedback from the staff as I could in the content of my deliverables, and I think they appreciated that. One example was the inclusion of a recommendation that features be planned in a way that allowed them to be completed more serially than they had in the past, so as to allow at least some testing of some of the features before being passed on to the business area (who, in that environment, act as "QA"). That, and some of my unit testing suggestions, definitely benefited from my two years as Agile Manager!

All in all, it was a good learning opportunity for me, and I think that the clients received good value from my 22 years of IT experience. I think that if the timing had been different, such that the people there who needed to play a bigger part had had more time available, then it could have gone better and more immediate gains might have been realized. But as the poker enthusiasts say: you have to play the hand you're dealt, and that's what I did.

Separating The How From The What

Poor AgileBoy!

One of the most common forms of "oppression" that I used to hear about in my role as Agile Manager involved Product people (or other representatives of management) going well beyond their mandate of providing "the "what" as they ventured into the thorny territory of "the how". I doubt that it was ever intended maliciously, no matter how much conspiratorial spice was attributed to it by the development team members... but if that's true, then why did it happen?

First, though, I suppose I should explain what I'm talking about. In most Agile methodologies, there's a principle that says that the Product Owner indicates what the team should work on - in the form of a prioritized Product Backlog - and the team itself figures out how best to deliver it. Thus, "the what" is the province of Product and "the how" is up to the team. It seems like a pretty sensible, straightforward arrangement, and yet it often goes off the rails. Why?

One cause for a blurring between the two can be Product people with technical backgrounds. We certainly had that in spades with our (aptly named) Technical Product Owners, most of whom were former "star coders" who had moved up through the programming ranks into management earlier in their careers. I'm sure it must have been very difficult for some of them to limit themselves to "the what" when they no doubt had all kinds of ideas - good or bad - about what form "the how" should take. In the most extreme cases, I suspect that there were times when our TPOs actually did have better ideas on how to implement certain features than the more junior team members did... but dictating "the how" still wasn't their job. In that sort of setup (where the Product area is made up of technical gurus), what would have been preferable would have been an arrangement where those TPOs could have coached the team members on design principles and architectural considerations in general, as part of the technical aspect of their title. That healthier response would have still accomplished what I assume was their goal - leading the team to better software choices - without overstepping their bounds and causing a schism between product and development.

Another factor that can result in people not responsible for "the how" trying to exert their influence on those who are, is a simple lack of trust. Again, I saw this in action many times as the Agile Manager. I'm not going to paint a black and white picture and pretend that it was always warranted, nor that it was never warranted. But the important point here is that dictating "the how" downward in that manner usually did more harm than good, and it certainly did nothing to close the trust gap. For one thing, having "the how" taken out of their hands gave those developers who were already skeptical about management's support of Agile more ammunition with which to say, "See? It's all just lip service! We're not really being empowered here!" It was also true, in some cases at least, that "the how" being supplied wasn't particularly well suited to the new development environment that was emerging. What with all of its automated tests, refactoring and focus on peer reviews, the code was resembling what "the old guard" remembered less and less with each passing Iteration. I experienced this first hand, as I'd make some small suggestion about how to tackle something that I thought might be helpful, only to get a condescending look and "Uh, it actually doesn't work like that anymore" from someone half my age. Yeah, that's tough on the ego, but it's also completely the right response.

As for why the trust gap was there in the first place, well... as both of my AgileMan books lay out in gory detail, lots of mistakes and missteps were being made. It wasn't exactly smooth sailing, let's say. The result of that, however, was that some among management apparently felt that the teams couldn't be trusted, and so they attempted to exert more and more control over them, in whatever ways they could. Had they simply applied that energy to improving the requirements gathering techniques being used within our organization, though, it would've benefited us more. Providing clear sets of Acceptance Criteria and good, well thought out constraints would certainly have done more to improve the product features that our teams were developing than any amount of "meddling" in the mechanics of the software development process itself ever did.

I imagine that it's often difficult to keep "the how" separated from "the what" as you work in an Agile environment, but it's critical that you do. After all, the people doing the work are in the best position to know how to do it, and that principle has helped make Toyota one of the premiere auto manufacturers in the world. We could do a lot worse than to follow their example in the Information Technology business!

Agile Is In My Blood

It's striking to me how different my perspective is these days, as a result of my time as the Agile Manager at my previous job. I'm currently working in an environment that's not in the least Agile, and yet I still seem to perceive what's going on around me with Agile-Vision, as it were.

For example, I've spotted an instance where the separation between the business people and the development team has caused some issues, and so it seems fairly clear to me that that gap needs to be bridged somehow. It seems to me that there are lots of options available to be considered, even outside of an Agile methodology.

I can also see where delivering what's promising to be a fairly large feature set for the current project in smaller chunks will almost certainly help, even if the "delivery" here simply means working on the features in such a way as to allow the business folks to play around with them as they "come off the line". This won't be the Agile practice of working within short Iterations and having each small group of features "done done done", but at least it'll open up the possibility of getting feedback on many of the pieces before the development cycle is complete.

And finally, I've begun tossing around the idea of using something akin to a Retrospective for areas where the general consensus is that not enough is being learned or improved upon. That may apply to relationships between groups, or to preventing the same mistakes made in Phase N of the project from being repeated in Phase N+1. When I described what a Retrospective was to a few people, they seemed to like the idea ("like a Post Mortem, but focused on improving and learning instead of blaming and arguing"). So we'll see how that flies.

It's certainly odd to be back in a work environment that's so strangely different from what I was used to and yet still experiencing many of the same challenges. Go figure!

A Horse Of A Different Colour

I'm venturing back into the work force on Monday, but with a twist. After 22 years of full-time employment, this will be my first experience as a contractor. Also, the work I'll be doing looks to be at least somewhat unlike anything I've ever done before.

I'll be performing a review of the development environment of a local company, in order to provide some guidance as they continue to refine their practices and processes. I think that I landed the contract on the strength of a couple factors in particular: my rather far-reaching involvement throughout my last employment as the Agile Manager, as well as the general breadth and depth of my resume, to date.

On that former point, it seemed relevant in the job interview that I had had some recent experience bringing change to an organization. I talked about how much I'd learned in the process, especially regarding the importance of being patient with people as they absorb information and evaluate it through their own personal filters. There's also naturally an element of trust required in something like that. I likened it to the expression about leading a horse to water but not being able to make it drink. My job often seemed to involve repeating that process, over and over again, until the horse finally trusted me enough (or maybe just got thirsty enough!) to take its first sip. (Not that I think of co-workers as horses! It's just an analogy!!)

By the same token, I think that the wide-ranging roles that I've performed over the past two decades also helped get me a foot in this particular door. I've worked on big mainframes and small client/server infrastructures; I've coded procedural algorithms (COBOL, C, Assembly language) as well as Object-Oriented solutions (C++, Java); I've been a "do-er" and a manager ("do-nothing-er"?); I've worked within well-understood roles but at other times had to define my own job. In fact, my resume, when I take the time to actually look at it (usually when I'm updating it), speaks volumes about the fact that I like new challenges. I've tended to change "jobs" (but not employers) at a pretty predictable rate of roughly every two years or so! And that's been true all along, showing that even as a wet-behind-the-ears programmer back in the 80s I was always looking for change as soon as I got comfortable with what I was doing. I think the pattern usually went: a few months of being a newbie, a year or maybe two of competence or better, and then on to the next thing. That cycle has given me a lot of varying experiences to draw on by now, it seems.

At any rate, I'll definitely be learning about a whole new set of parameters, starting next week. I'll be comparing what I see in front of me to my mental backlog of best practices. I'll be discovering what strengths and opportunities are in effect in an environment that's quite different than anywhere I've worked before, and making recommendations on each. And I'll be building up a bunch of new relationships, right from scratch. All of which sounds like something that should be right up my alley, I hope!