Moderator (MDR): MDR-lisa
Guest Speaker (SPK): SPK-geoffreyphipps
MDR-lisa: Hello and welcome to a special edition of Java Live. Geoffrey Phipps, PhD, is visiting us from Australia. He's been doing software metrics lately, and was curious how C++ compared to JavaTM in terms of productivity and quality. He'll be the first to admit that his results are still preliminary, but he has strong indications that Java is roughly twice as good. Geoffrey, why don't we start by talking about how you did your research, and how you became interested in the first place.
SPK-geoffreyphipps: I was interested in software metrics during my time as a grad student at Stanford. I became very disillusioned with computer "science" because it was all "argument by assertion" ( I am right, you are stupid), rather than based on scientific experimentation. My method is to use the "Personal Software Process" (PSP) invented by Watts Humphrey, which I modified, doing actual industrial projects. I used it on a four-month C++ project, and then a two-month Java project. I have hard numerical data for defect rates and productivity (lines of code per minute) for both languages.
MDR-lisa: And what did you discover?
SPK-geoffreyphipps: My defect rate is roughly halved. Defects are measured per kLoc (thousand Lines of Code). For C++, I stabilized at about 90 defects per kLoc--that's after seven years of C++. After four months of Java, I am at 50 defects per kLoc. For productivity, I am at about 1.3 lines of code per minute for C++. For Java, it is 1.8 and still rising. I expect it to stabilize at about 2.2.
MDR-lisa: So in terms of quality, Java sounds like a big win.
SPK-geoffreyphipps: Absolutely, as is PSP I should add.
glen_s: Any thoughts on the "Function Point Analysis"
(FRA) level of the language? Software Productivity Research (the Capers Jones company) gives it a level of six (the same as C++). Also they say that the function points per source line is estimated at 53. Do you have results for applying to FPA?
SPK-geoffreyphipps: I did not do Function Point analysis. Function Points are one of the attempts to quantify the difficulty of an application. You read the spec, and then write down a list of all the things it should do (function points) and count them up. It's not perfect, but it is about all we have. I'd love to see the exact reference you mentioned. I am not familar with the language levels you mentioned. My data (although limited to just me) indicates that C++ and Java are not in the same order of (binary) magnitude.
glen_s: There's a web page at http://www.spr.com/library/0langtbl.htm that gives various language levels. This is the first time I've heard of the "Personal Software Process." Where can we find out more about it?
SPK-geoffreyphipps: It was invented by Watts Humphrey of the Software Engineering Institute (SEI). He has a book, which I believe is called A Discipline of Software Engineering. The main point is that you keep very accurate counts of your defects, your time (ignoring phone calls and trips to the bathroom), and the estimates. I can now usually estimate my coding time to within 10%.
rfalk: In doing the comparison, were the projects
developing comparable software, and were you using "standard" libraries at similar levels? For example, were you in C++ using STL/Rogue wave collections, or coding your own?
SPK-geoffreyphipps: No, they were in different areas, but read on. The first project was a C++ metrics parser. I kept separate data sets for time spent working on the grammar (written in PCCTS/ANTLR), and time spent working on "support" classes. I thought that the two would have very different characteristics, but in the end I found them to be the same. The second (Java) project was a program for doing statistical analysis of telephone switch-call records. So it is possible that the domain change is what I am measuring, but I don't think so. A proper experiment would involve catching hundreds of programmers and locking them in identical cubicles, feeding them cheeseburgers, and watching them code. This is difficult to arrange. The libraries I used were "standard" PCCTS and OSE (which is standard in Sydney).
nocturne: It seems like you have done very good research on these metrics, Dr. Phipps. What do you think of the validity of comparing productivity using lines of code? In other words, does 300 lines of Java perform about the same work as 300 lines of C++? How did you collect your data? Did you use any monitoring software while coding? Or was the analysis done after the projects were completed?
SPK-geoffreyphipps: I am glad you asked that one, it always comes up in software metrics discussions. Lines of code are usually not a good measure, because they depend on the actual language (APL is very compact, COBOL is not), and also on your formatting style. However, in this experiment I am comparing my C++ against my Java, and I have a consistent formatting style in both languages. The two languages are also "close enough" in expressive power (I'll let you define that one!) for the comparison to be valid, or rather, for the error bars introduced by that assumption to be less important than other error bars. As Lisa stated in the introduction, this work is preliminary. A solid study would be a PhD thesis, and I've already done one, thank you.
rfalk: To what specific features of Java do you credit the performance gain in lines of code and quality? Syntactically the languages are similar with Java, excluding a number of C++ "features" (some of which I like, some I don't). Is it the syntax of the language, or the built-in higher-level libraries that account for most of the improvement?
SPK-geoffreyphipps: Okay, now we leave data and return to opinions again. I expect that all of you could quickly list five or ten features of Java that improve your productivity. My top features would be: garbage collection, no pointers, simpler syntax, packages, one set of standard libraries, no makefiles, and a better inheritance model. This leads me to some more data. Not only did I count defect rates, I also kept track of how long it took me to fix defects (BTW, defect i= bug + syntax error + design error). Fixing a defect during the compile phase in C++ took me six minutes. In Java, it took me 1.5 minutes. During testing, it was 30 minutes for C++ and 15 minutes for Java.
jbrasch: Does development environment effect you
outcome? Which ones did you use for your study?
SPK-geoffreyphipps: I don't have hard data, but I believe it is very important. The C++ project was done using emacs and make on HPUX. The Java project was in Symantec Cafe. I believe that C++ was much slower because the compile cycle (via make) is much slower, and it is also slower to get to the error. Plus C++ syntax questions can take time to resolve. (Pointer-to-member function on a templated class, anyone?) Plus the lack of packages means that you can spend minutes crawling around the file system looking for your header files. I did not find the syntax highlighting of Cafe to be helpful, and the editor is much less powerful (therefore probably slower to use) than emacs. As this question shows, there are so many factors it would take some thinking to design a rigorous experiment.
glen_s: Were the developers familiar with Java at the start—as familiar with Java as the C++ developers were with C++?
SPK-geoffreyphipps: The developer was me. I was much more familiar with C++ (seven years) than Java (rank beginner), so it is likely that my Java programming will improve. So my current figures that Java has half of the bugs and twice the speed of C++ are probably conservative.
MDR-lisa: So, are you working on extending your preliminary work to larger groups in the near future?
SPK-geoffreyphipps: I would like to, but I earn my money as consulting software engineer, not as a researcher. To do a proper study I would need to be full-time, and have the cooperation of a largish software company. Although I wrote these programs as part of my paid employment, all the data reduction was done in my spare time. I just got the data, so it is not written up as a paper yet.
rfalk: Were there any features missing in Java libraries, syntax, or development environment (editor already mentioned) that seemed to be holding back your Java productivity?
SPK-geoffreyphipps: Some kind of sensible templates would help (much, much less powerful than C++, all I want is list of strings and the like). The pass-by-value semantics mean that you can't pass in a pointer to int if you want to change a value, so things tend to get wrapped in classes (which is good I suppose). Better project-wide code transformation tools (even for things as simple as class name changes) would also help. I have not been coding that long, and it is not as broken as C++, so I have no major wishes.
MDR-lisa: Geoffrey's results are really "hot off the presses," or in this case, hot off the web site, so this research has not yet been published yet. Thanks to everyone for joining us, especially Geoffrey Phipps. Be sure to join us next week for our regularly scheduled Java Live, and check our new Code Clinics (follow the schedule link from the Discussion Forums page for a listing of times and topics).
