Maintenance - Repay the debt

It's time to repay some of that technical debt you have accumulated. The project has been running for a few years and often the code gets smelly. The code smells of complexity and bugs.

What do you do to approach such a problem? You clean it up of course!

Now I will tell you how you can choose what areas in the code you want to clean. For this you need a couple of tools. First one to calculate complexity and then you need something that tells you about changes in your codebase.

I have used Complexian and StatSVN in this example, but you may find other tools that does the same job just as well.

I started with Complexian and found that a few classes had a complexity above 21, which I consider as high risk. Then I combined those data with what we can get from StatSVN and came up with more that suggested that we should focus on the classes that has high complexity and many revisions. This because I reckon that a class with high complexity that is changed a lot must mean trouble. There is a lot of different approaches you can use, but this one proved to be sufficiant in my case.

From Complexian
Register.java: Complexity is 38

From StatSVN
Register.java : Lines of code: 2273
Register.java : Revisions: 32 (4th place)


Here you can of course add all the static code analysis you want, but for me this is sufficient to tell that there is something wrong with this class. In this way it is also easy to persuade the project manager that this is a risk that needs to be taken care of.

Then I dive in to the class to find if's and else's that needs a refactoring.. At the same time I notice that the class is in the wrong package as well. My goal is to reduce the risk in this class. Never refactor without a goal. Refactoring makes no sense if you do not do it to achieve something better. If the class never gets changed and you have no bugs that you know of in there, leave it alone. It would probably not show up as a candidate either since there would not be many revisions of that file.

Write some tests for those methods that have high complexity is hard, but rewarding. In this way you deal with risk in a professional manner instead of just "feeling" where to do you cleaning.

Hope this gives someone inspiration and a way to start repaying that technical debt they have accumulated during the years.

Do the rights things at the right time

Yesterday I attended the oslo xp-meetup with an expectation of hearing some inspiring thoughts on software developing using xp and agile techniques. What I did get was a very good presentation of lean startups and how to do the things that matter the most at the right time (like in xp and lean).

It was Kent Beck who presented his new ideas on what he defined as "the flight of a startup". The analogy used the different phases an aeroplane is in before it's cruising at 30.000 feet.
1. taxi
2. takeoff
3. climb
4. cruise

Kent emphasized the need to be careful about spending cash (if it was limited) in the taxi and takeoff phase. The most important issue that I noticed from my perspective was during takeoff where you actually remove functionality thats not being used because you want to get better scalability. I'd call it reduce the risk of cluttering up your good ideas with the bad ones. But as Kent said, removing features are not easy and often we tend to let them stay in there. Another interesting statement from the re-inventor of TDD was that you should ALWAYS track usage in you applications.

I will start to do just that. I will start putting in functionality for tracking usage in my applications. This to be able to get actual feedback on what is being used in the application and what is just waste. Then we have an actual map of the used functionality and with this we can start doing some real work. Removing functionality thats not being used will be removing code that slow us down.

If he is doing this talk somewhere near you, take the trip and listen in on the ideas. It's good stuff.
Another very nice post on startups here

Smidig 2008, 9-10 October

Smidig 2008 is soon here! http://smidig2008.no

Last years conference was a great success and I had good experience with the openspace discussions that I attended. This year it is even better! Since I had such a great time last year I thought that this year I could help out with the organizing. And I am!

This year we will have a even better location with more space for discussions and mingling. For you guys out there that does not understand Norwegian, do not despair. I will give you some of the lightning talks wrapped in my perspective after the conference..

More on Age of Conan and priority

It looks like someone at Funcom has found that communicating with the "product owners"/customer is a good thing. Here you can read about a Funcom employee asking for a list of bugs and new ideas.

Lets hope they keep things like that going and make AoC a great game

Priority and Age of Conan

What is priority?

It is often easier to ask people about what they want first instead of asking them about priority. This often shows when you ask the product owner about what is important and they say that all is important. Then you might try to ask for what is the most important functionality. Then they normally have problems choosing which task it is, because they are all important. They all have high priority.

Here is a trick; Ask the product owner what they want first. What do they want in the next delivery? They usually pick the most important for them. If you have multiple product owners, then let them pick their different functionality and add it all together in to the sprintlog.

Now over to something I just recently noticed about priority and product owners over at the discussionboards for Funcom's new MMORPG: Age Of Conan (AoC). The game has been in the stores for about a month and hundreds of thousands have bought it already. If you read the posts on the forum you will quickly see that many of their customers are not satisfied about the product. Even though one should be careful to believe everything on discussion boards there is one thing that really is obvious. Many of the posts talk about priorities of bug fixes and new functionality. I wonder if Funcom listen to their product owners, who in my opinion is the customers that play the game. The last patches has been critisized for not delivering the bug fixes that are important to the product owners. There have been posts about new functionality that have no value for the product owners.

To have hundreds of thousand product owners is no easy task I can imagine, but I ask myself what would I do? It seems to me that they really need to involve the product owners and actually deliver value. I propose to make it possible to vote for issues on the discussion boards and make it visible that these are the most important issues. Then when the new patch arrive they actually deliver what the product owners thought most important.

Make a list (backlog).
Vote on items from the list (1 week or until next patch).
Put the items on the list in prioritized order.
Start at the top and work you way down.
Rince and repeat.

Any comments?

Certified Scrum Master

Yep. It's true! I'm now a certified scrum master.

This week I attended "Agile Project Coaching and Project Management with Scrum Certification" which was teached by Martine Devos.
The course was more a workshop than a lecture and I really enjoyed it even though I knew most of the theory from before. The way she spoke about her practical experience in regards to getting things done, the pragmatic approach to teach us about the different techniques of agile was great. It was highly influenced by questions and people got really good answers supplied with examples and stories from the real world.

She talked about the importance of showing progress and visibility in the team, part of the solution there is big charts!

Another important issue she brought up was agile planning. Yes, there is actually more to this than just an iteration (sprint). We did an exercise and put together three sprints and made a release plan for this. The first sprint was more detailed than the second and it was more detailed than the third sprint.
The idea is not to waste your time in details of later sprints as the customer or even the world may change its course before you actually are to implement it. Maybe you even learned of a better way to do it. But you need to know something of whats coming up the road. Things like, is it big, medium or small? Is it complex? Does it involve others or only your team?
Take things into account, but don't use all your precious time on the details. Use common sense.

With all this in my backpack I'm looking forward to Wednesday where Mike Cohn is talking about agile estimation and planning. See you at xp.meetup!

Workshop: Legacy Code

It's true. We got it! We got legacy code..

To best describe legacy code I turn to one of the guru's on the subject, Michael Feathers:

[After all, there is an emotionally neutral definition of "legacy code." Legacy code is code from the past, maintained because it works. But, for people who deal with it day in and day out "legacy code" is a pandora's box: sleepless nights and anxious days poring through bad structure, code that works in some incomprehensible way, days adding features with no way of estimating how long it will take. The age of the code has nothing to do with it. People are writing legacy code right now, maybe on your project.

The main thing that distinguishes legacy code from non-legacy code is tests, or rather a lack of tests.] ( full article and book )

We now have more than 180 000 lines of java code in my project. To me that is a lot of code. How can you have knowledge of the code to safely say that your change will not break any existing functionality? You can't!

There are many reasons to why you can't know for sure. Let me give you an example. One of the classes is almost 10 000 lines long and has methods that contains more than 500 lines of code. If we get a bug report and go in to that class to change something you can be sure that there is something in there waiting for you to mess it up. The class is impossible to change and it will take forever to fix this bug. And by fixing the bug you may actually introduce another , because someone was expecting that specific value in another part of the application.

The reason for expecting the bugged value can be many, but you can't know for sure if it will break something. Without a test, we can never know if that other functionality break when we change something.

Even with tests you can't be sure not to break anything. But at least your tests will lower the risk of not finding the bug you created with your change.


The tests help us have actual knowledge of the application. Not only can we read the tests, but we can actually get feedback from them when we do something to the code. This helps us to know more of the dependencies and the relations in the code. It helps us to understand the code better. But when you have 180 000 lines of code, do you really want to add another 100 000 in tests? No. But I do want to have tests around the changes i make and take small steps to create a safer place to work. Because it is fear that keeps us from attacking this monster class with our refactoring and our changes. Get rid of your fear by learning more about the code with tests.

In this spirit we have scheduled a small workshop for tomorrow to handle that fear. We will delete unused code and we will write tests for the code we are moving out of one of our biggest classes. We will discuss and learn about our code. With this knowledge of how to handle legacy code I hope that we will conquer our fear of changing code in horrible places.

This is knowledge in action!

Estimation

Have you ever lost track of time after saying to someone (your better half maybe?), it will only take a short while. Then suddenly dinner is cold and a couple of hours have passed, at least. Is this because you did not know how long time it would take to do the things you had planned. Maybe your definition of a short while is different than your better half's definition? Maybe the planned things where not the actual things you had to do anyway.. We estimate in our living life everyday, how long will it take me to get there, how long will I use to make dinner or how long will I use to answer these emails and send them.

It's a well known fact that we tend to be overly optimistic when we do estimation, especially if we are planning to perform the task ourselves.

In our team, we just started with planning poker. Before we did some estimation with post-it notes using numbers on them , but not all in the team where involved in the process. Before this we normally have the ones that where to implement the changes, estimate the time it took, without much discussion other than project manager adding to the estimate with test and administration time (yes most people forget about time to test).

Questions that often comes up is, who are doing the work? Normally I say that we estimate how long time the team will use on it, how long will it take us to implement this feature? Often someone know something the others do not and this makes this person able to perform the task faster. This does of course create an even wider gap between the people in the team. By estimating together with the planning poker cards we open up for others to more easily try themselves on tasks they normally would not be able to get because someone in the team where better at that type of technology than they are. By this I mean that the estimate opens up for other people to have a chance on making it within the estimate. This makes the team much better suited for the day that one team member leave the team or call in sick. We are then actually still able to perform the task with less problem than if noone had ever touched a similar task or that specific technology.

Questions I want people to think about is:
How long time does it actually take? What is the estimate used for? Do we need the estimate?

These questions put a big questionmark behind the word estimation. Normally in our case the estimate is used for giving the customer a rough idea of the cost. Can this really be turned around as many people talk about in the "agile community", are we able to get the customer to come to us and tell us how much they want to use on the new functionality. Do we have enough trust to deliver what we can in the time we have available. With the Scrum approach we surely would be able to do the important tasks first, but is it enough?

Another question that shows up is how long does the estimation take? Maybe much of the estimation we do should be considered as a waste of time. David J Anderson has some nice words regarding that perspective here.

Agile @ JAOO 2007 #1

Yes, I'm attending the JAOO Conference in Denmark at the moment. Today I have seen most of the presentations in the agile track and I'm quite satisfied with the amount of information that has found its way through my ears and eyes to my brain. This information is now in a process of being mixed with my own experiences and thoughts. Hopefully on wednesday I will be able to go home and create knowledge by acting upon the informaton I now have.

I'll cut the chase and tell you. Agile Retrospectives by Diana Larsen was the last presentation on the track and its the one that still sticks in memory (The rest is written down in notes). The presentation was a small summary of Diana's experience regarding the facilitation of constructive discussions and retrospectives. One of the things she mentioned was that there is no blame in such a retrospective. The meaning is to find out what went wrong and fix it or to find ways to improve and not to finde someone to put your finger at. It is a team responsibility so there is noone to blame. Diana claims this is one of the key elements of a good retrospective, a safe environment where one can share thoughts and perspectives without handing out or be blamed for what went wrong (or right for that matter).

A second thing she mentioned was the amount of actions you should focus on for the improvement in the next iteration. This should be one or at the max two actions. These should be followed up in several ways. In the daily standing meeting was one, another being in the planning of the new iteration (ie put it on the sprint queue).

Diana Larsen inspire to a deeper look into retrospectives and from the feel of it, I think that my autumn books might be on the subject.

Have a nice night and I'll be back tomorrow with another tiny bit of the JAOO Conference

Friction and slow delivery

I was just reading up on my favourite blogs when i came over a interesting post on the From where I sit -blog. Michael S. Hyatt describes what he sees as friction when visiting a bookstore. The idea is to get the purchase happen fast enough so the customer does not change his mind or as in Michael's case, leave the shop because it took too long to get the purchase done.

Could we not draw parallel from this to our development of software and project practice? The customers want a new feature in an already developed software. We receive the request and we specify it further with the customer. We estimate the time it will take to develop the new functionality. Customer gives us clearance to start working on it. We work on it. We finish the development on the new feature and we test it. We do not deliver it, because we have more things on our list to do in this release. The customer knows this so he is patiently waiting for us to finish this as well, because he also wants the other changes. We test the other changes, we might find a bug and work more on the issue. Then we hand the release over to the customer so he can test it, this is 6 weeks after the order was placed. They find a bug which get reported back to us and we start fixing it. We deliver a patch for the release and they test this. All is well and the testing is over, the release goes to production. The users finally see the new feature which they wished for, maybe 3 months ago.

What if the process where a bit different? What if we develop the new functionality and test it and deliver this to the customer. The customer tests the new functionality while we develop the other requests on the list. The customer find no bugs in the functionality and we release it to production. Or if the customer finds a bug, he reports it and get a new release from us with the fix and maybe more new functionality in it. Maybe some of the other things on the list was finished and tested in time to get it with the fix for the new functionality. Customer gets the new delivery and test it again, now with additional functionality. It goes through and we release it to production, maybe 3 weeks from when the users wanted the new feature.

To me this sounds better than the initial way of doing things. If the functionality that gives value to the customer is done, give it to them! If the users want changes, make the change and deliver it to them as fast as you can. No need to wait.

There are of course issues around this. One of them being the ability of the customer to actually receive and test the new functionality as fast as you deliver it. Another might be that the initial software is used 24 hours a day and a production release will hinder them in their work and cost money. But I urge people to think on this and ask their customer if they really cannot receive and test small parts rather than big chunks of functionality. Do they really want to wait that long for something they ordered? Why should they order and add all these orders in to one delivery? Does 15 minute downtime on their production system really cost that much versus having users waiting for functionality for months? Do their lack of motivation towards the use of the system cost less?

In this post I have not even started talking about the cost of long feedback loops on the delivered functionality. I'll save that for another post sometime.

Agile and Post-Agile?

What is really agile? I've touched upon this topic earlier, but I have come across a few blogs on the topic again that are interesting. Especially David Anderson, Agile Management Blog where he asks the question "Are you part of the Post-Agile movement?". He also asks the question of what agile is and points further to Jason Yip and his Agility is not the point post. In this post Yip compare the so called Agile elements to those of Lean and he does a good job of it.

As Anderson says, we need to keep pushing.

To me it does not really matter if it is agile or not, as long as we can improve the way we perform our craft and deliver better solutions. Agile for me right now is to implement Scrum and see if this improves our abilities. When this process has settled I will keep pushing. Will you?

Are the customer involved enough?

A while and several blog drafts since my last post now, but here it is.

The question is really, are the customer enabled to be involved enough? While we are quick to think if the customer are participating in creating the backlog or the sprintlog. By this having ownership in the decisions made. Not long ago on InfoQ Little and Spayd gave their views on agile and organisational change and they mention the importance of the customer being part of the agile process if the deliveries are to be successful. This is very true, but are they ready in their organisation to handle the deliveries?

As the Agile Manifesto principle states: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. This assume that the customer are able to handle these deliveries at the pace they are coming. Further the manifesto claim that Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. If we are to adhere to these principles we have to enable the customer to also become agile in their process of receiving the delivieries.

To take a practicle example of this I can relate to my current project which is in the process of implementing a more agile approach. Agreements have been met and deliveries have been delivered, but the receiving end are not able to handle the deliveries at such a pace. They lack both the manpower and the routines to test the deliveries thus bringing us back to how it was before. The team cannot keep on delivering while the tests are not being managed and we have to set a new pace. A good article on Agile Journal describe this scenario as candy coming faster on the conveyor belt, it creates trouble handling them.

This shows that agile processes sets new demands to the organisations not only where the deliveries are being produced, but also in the receiving organisation. Now you have not one, but two organisations that has to adopt agile. The question is are the customer involved enough to make that change?

To be agile or not to be agile, is that the question?

What is really agile? Are we really agile or has it become a hype?

These were questions that was brought up on todays xp.meetup without coming to any conclusion, but nonetheless a fruitful discussion inspired me to write this post.

One of the issues mentioned was related to technology and how agile you can get. Many agrees that technology influence our way to work and thus how agile we are in our work. And it was in this context that J2EE was mentioned as something that really held you back from working agile. This might be well and true, but isn't the question really "How agile can you get" and at the same time deliver what the customer want with the agreed quality?

Many projects today deal with legacy technology and different aspects of the past that impose restraints on our solutions. Would it not be possible to say that one is agile in the way they are dealing with this legacy technology? Be it test driven development (TDD) with or without behaviour driven development (BDD). The world is full of legacy..