March SPIN – Parasoft: What An Interesting Story

Confusion reigned with the most recent SPIN. As usual, the meeting was well worth the time, although determining what “the time” meant remained ambiguous to some extent until the presentation actually started. I arrived a bit after four o’clock, and when I stepped in the room a couple of people said, “There’s Andy Erickson,” and somehow I believe I only narrowly missed my ordination as the evening’s speaker. I’m glad I wasn’t, because then I couldn’t say the meeting was “well worth the time.”

The speaker’s flight plans wreaked havoc with the meeting’s start time. At one point 4:45PM would be the start time, and then with only a couple of days before the meeting, the start time changed to, well, I actually don’t remember now, but it was sometime later. In any case, I moved into the hallway to make something like a bazillion phone calls on my to-do list. When I finished the calls I thought I still had a few minutes to spare. I walked back in to Parasoft CEO, Adam Kolawa, pacing the front of the room preparing to start. I made my way quickly to a seat preparing to hear what would be another talk full of immensely useful and relevant material. When Kishore walked in 8 minutes later, I didn’t feel so bad 🙂

Adam began by describing the path his company took to get from where they started to today. In reality, his story was fascinating. About 20 years ago, around 1991, Dr. Kolawa and some colleagues built an automatic parallelizer which, in essence, took a serially architected system, parsed processes automatically, and ran processes in parallel. And it actually worked. Well, sort of. The system really split processes and ran them in parallel, but every parallel program ran slower than the serial original due to the enormous amount of inter-process communication necessary to keep everything synchronized. Still, learning how to actually analyze functions in ways that facilitate parallelism laid a foundation for the future work and direction of Parasoft. Today, Parasoft enjoys benefits of the industrialization of their software development process. And that is what we all came to learn about.

Around 1992, the team learned techniques to understand how programs traverse memory during mathematical operations. This led to breakthroughs in tools that help software developers find issues grounded in these types of operations, issues like array out-of-bounds errors or buffer overflows. In ’93 and ’94, build automation, something taken for granted 10 years later, first supported the development process and immediately created value. Automated make files, and nightly builds reigned in the reams of source code generated by developers. The supporting development tools required for themselves a test-first development process to ensure continuity of both the functionality and the source code of these complex parsers. Nothing could break, and test-first seemed the most logical approach to ensure continuity at the time.

Suddenly in the ’95 and ’96 time frame, ParaSoft found themselves in an enviable position of development shops *needing* their products. As software grew more complex, developers and integration testers could not successfully find all the defects in these systems in a given testing cycle. And they would discover this, say, the day before release to manufacturing. Folks would call in for licenses because they needed to release *tomorrow* with limited time to find, and fix, all the issues.

The thinking gravitated towards the abstract where folks devoted brain power to the process. So how do we *prevent* errors? Not just fix them, but prevent them from ever making their way into the source. Lint output so much garbage about our programs that we learned to simply ignore it. Compilers provided copy constructors that provided incorrect default behavior – issues that don’t resolve with memory checking. So the thinking moved to, “can we check programs automatically?” Can we create test programs from the programs themselves? Can we forget about testers?

C++ was inherently too difficult to do this successfully. Java’s introduction greatly simplified this effort. Introductory efforts focused on applets, and as these code bases grew, self checking programs proved problematic. Efforts then focused on the class level where formal unit checking originated. Kolawa’s perspective? “Unit testing is good if you can’t do anything else.” At least we know where he stands 😉

Parasoft found market success with their Jtest and C++test products, yet a disturbing pattern emerged. Developers loved the tools at first and then for some reason stopped using them. Digging deeper, it became evident that many of the folks in the industry just didn’t leverage tools. So stepping back, perhaps the process and not the tools would prove to be the core of the issue. If the process doesn’t integrate a tool set in an easy-to-use manner, a critical mass of effective users cannot be achieved. The question then became, “How can we industrialize the process of software development?”

In maufacturing, companies can setup production lines anywhere they make sense. The process is repeatable since a company can throw even the lowest skilled employee on a line and still produce high-quality output. Output is a function of the line and not the employee. Software development is not like that.

In the ’70s, US automakers had quality problems. Ford recalled more pintos than it had produced one year. At the same time, the Japanese cars also had quality problems. US manufacturers decided to push the (defective) car to a dealer then move on, essentially settling for the excuse that “we’re doing the best we can.” In contrast, Toyota listened to Demming, started looking at defects and how they correlate to the production line, and asked themselves how they could improve their manufacturing process. Software developers today generally settle for “we are doing the best we can,” when we should pay attention to how we can tweak the processes we follow in order to produce high-quality output. Okay. But what was most interesting is that in Toyota shops productivity increased right along with quality.

For some reason, software development organizations segregate development and QA. Demming says an organization cannot separate manufacturing from the responsibility for quality in manufacturing and hope to find success. Think about the process. QA can’t run regression tests because the each new build breaks the existing tests. If test developers don’t update the tests in parallel with source code updates, regression testing becomes meaningless, and QA’s only hope is to test manually. The software organization’s thinking that the software consists of the source code is simply not right. Software, rather, is the source code and the code that verifies the source code. The bottom line? Tests require the same process that source code requires, i.e. development, maintenance, and testing. Suddenly budget is required to plan for the functionality verification process, probably 50% of the development effort.

What’s this all about? We all change our software. Our businesses constantly require changes that affect the code. Our software development shops can’t control the rate of change or the changes themselves. Nor should we. Yet our processes don’t allow us to adapt as quickly as necessary to requirements changes. And all along the way, software becomes more and more complicated. Interconnected systems span enterprises. Still, for the most part, we haven’t changed or improved the software development process. Adam’s question: How is building software different than other manufacturing processes? This is a key question because the process isn’t close to the same. It’s a brain dump. Building software is essentially a brain dump, so process engineering must take the human brain, with its patterns and behviors, and all the stuff that makes us human, into account and integrate the brain as a critical part of the software development process.

Ask yourself, What are people good at? When do we perform well? What environments allow us to create high-quality work products? Bottom line: What do we need? Think about it. It’s not necessarily a deep question. It’s certainly not a trick question. People are good at creative work and not so good at repetitive work. So a software development process should support creative work and protect workers from repetitive and mundane tasks.

The outcome of processes that support creative work are better production methods, higher productivity, and, pay attention here, as a SIDE EFFECT, better quality. Said another way, to improve quality don’t think about quality. Instead, focucs on productivity. Woah, that’s a HUGE statement. Dare I say a paradigm shift? And I didn’t just want to find a way to use the word paradigm in this post 🙂 Essentially, following mundane, though very disciplined, processes will lower productivity and cause quality to plummet. Yep, you’re right to ask, “where are the numbers? Show me the evidence.”

These ideas can’t just live in the world of theory. In fact, Kolawa moves on to describe how Parasoft operates in this manner today. Parasoft operates their software development organization around the principle of managing through defect prevention. “In fact,” says Kolawa, “If you’re not using these strategies, then you’re dead.” Kolawa outlines a number of P’s, and this one is Principles.

Next are Practices: What do you do to the product? And finally Policies: What do you tell people to do? We are dealing with people.

Some quick notes about Parasoft’s Principles:

  1. Establish an infrastructure. What’s in this infrastructure? Hopefully obviously, source control (have you ever worked without version control? Yeah, thought so.), nightly builds, requirement management tool set (80% of companies don’t even have an RMS, the rest use theirs to protect themselves – not as a support in the development process), automated regression testing, and other tools. These must provide reporting. The practice of continuous build is good and needs to incorporate one additional step. Continuous build means building a project in 3000 steps that you cannot reproduce at any point in time. So every couple of days do a clean build from the bottom up. Then run the system through a regression test suite and make sure nothing broke.According to Kolawa, bugs fall into 3 categories: 1) badly implemented requirements (coded wrong, bad interpretation), and automated systems cannot find these issues, 2) missing requirements. No perfect requirement exists and those that do are generally incomplete. Agile practices go a long way towards discovering missing requirements because of the short feedback loops. Still, automated systems cannot find missing requirements. And 3) confused users who believe the application should do something that it doesn’t do. This becomes a huge problem in web services as systems expose behavior in new ways. And automation still can’t uncover bugs. Regression testing doesn’t find bugs, it only notifies you that something changed. Maybe the issue is not a bug, but for sure some disconnect exists between the application functionality and the test suite.People need investment, training, and an integrated infrastructure. That means the people need to know how to use the tools.
  2. Automate anything you can automate, otherwise your efforts are doomed. Schedule jobs. Script the check out of all code into a clean environment. Automatically prepare and build the project. Create logical policies and automatically enforce them. About 80% of security is concerned with validation and verification of user input according to OWASP. In a code scanner validate that user input flows through validation code, and if it doesn’t, flag the code as non-compliant, send it through code review, and fix it. Then run regression tests.
  3. I didn’t get Principle 3. Sorry.
  4. Measurement and tracking of project status. What? Isn’t this simple project management? I don’t know. Ask yourself a couple of questions to see how well you measure and track. How much code do you have? Most organizations don’t know. Ask any production line manager and he’ll tell you how many cars come out of his production line without a second thought. There is difference between 200,000 and 2,000,000 lines of code.Do you do code reviews? Developers HATE these and look for reasons not to show up for the review, so automate this process. How? Designate reviewers for each developer. Set scanners to look for new code on check-in. On check-in, have the scanner package up the code, send it to the reviewers, and integrate the project with the IDE. With one click, the reviewer opens the code in the IDE, allows the reviewer to comment, and then provide feedback directly to the developer in a non-intrusive way. Formal face-to-face code reviews terrify developers and waste valuable time better spent producing. Setup scanners to trigger event-driven code reviews. For instance, on a policy failure or missed requirement send the code for review. This method actually finds bugs because code reviews happen in the context of a use case. Remember, quality is a byproduct of productivity, so make the process as productive as possible.How often do builds succeed? If a shop constantly builds then builds won’t break. Breaking a build is a come-to-Jesus for a new developer when they break the code base for the first time. This is good because they understand the responsibility they have to the team. Still, expect builds to fail from time to time, so build all the time for constant integration. Then run regression tests immediately and automatically.

In the real world ParaSoft manages these processes across 4 development centers with 120 developers working on 24 million lines of source code and 36 million if you include testing code. Kolawa stressed the application of best practices. Developers are very good at reading books and doing nothing with the information. One size does not fit all, and these practices need tweaking for the individual organizations, and setting up feedback loops allow root-cause analysis to identify problems and fix them.

The process is slow and incremental to implement. Start with the smartest group of developers to implement pilot processes and your team will experience greater success. There is no big-bang here. Adapt and pilot with other groups based on your learnings.

These processes are all about productivity. So how do you measure it? Simply divide the lines of code by the number of developers. And, yes, include good code, bad code, and comments. They all even out in the end. Developers will let you know when the team needs to grow. They will do so because they will write more code and the code base will become unmanageable. The natural horizon seems to be 10K to 20K LOC per developer. To open up the horizon, add tests, measurements, charts, test suites, and more automated tools, because they allows developers to take risks and therefore become more productive. When test cases fail developers start to understand correlations within the code. Constant and consistent unobtrusive code reviews will catch issues before they become issues, and developers wind up protecting each other. With systems like this do you need status meetings? Probably not, so developers have more productive time available. Status meetings let managers feel good and they’re really only for the manager’s benefit. With systems like this in place, productivity can improve up to 10X. Quality becomes an effect of this productivity with non-intrusive processes in place.

You know? I can’t do justice to the quality of Dr. Kolawa’s presentation nor the practical work his company does to practice what they preach. You can find out all the details, along with the accompanying charts, graphs, and other reports that prove the theory in Dr. Kolawa’s book, Putting the Engineering in Software Engineering.  And check out the presentation with some of the examples.


~ by Andy on March 22, 2008.

2 Responses to “March SPIN – Parasoft: What An Interesting Story”

  1. […] an automated unit test isn’t possible, but this could be caught with static analysis tools if a policy is written that doesn’t allow dynamically constructing constant strings. If automated, then a developer doesn’t need to think about it – until the policy is […]

  2. fascinating story. too bad that nothing is said about just how bad the parasoft products are.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: