Web services need planning

Project Management Plan

Project Management Plan

Copyright perhapstoopink (Creative Commons)

I get calls all the time at home from “investment planners” and “retirement planners” and all sorts of other people who believe that I need to plan for the future.  Well, their not wrong, but planning needs to occur in other areas of your life as well.

Taking on my mantle of IT Philosopher I’m here to talk to you about planning.  Future planning for your applications.

I understand that being able to write “Developed and implemented web services” looks good on a resume, but if your application does not need a web service don’t give it one.  Sorry for the emphasis, but it’s needed.  Quite often I see application architectures that don’t make a lot of sense, other than the fact that it will look good on a resume.

Let’s take a look at a fictitious travel agency “Jessop Fantasy Tours”  (hey, my story, my travel agency).  One thing that probably makes good sense to spin off into a web service is “RetrieveTravelItinerary”.  Based on the name you would assume that this web service would retrieve the Itinerary based on a key being passed in.  In order to determine if it meets the criteria for a web service see if it satisfies these points:

  1. There is a need for this information in multiple locations in one or more applications.

OK, there was originally going to be a list of five or six points, but, for the most part, it really comes down to the above question.  Even then I could spend another couple of thousand words just talking about the permutations and combinations of items that make up the little phrase “multiple locations”.

For instance, I don’t mean two different places on the same screen.  I don’t mean in two different functions, but in two different application functions.  (“List of Itineraries” and “Itinerary Details”).  If you plan on commercializing the usage of the function then that would qualify as multiple locations.  By commercializing the usage I don’t necessarily mean external to the organization.  You could have a function, used strictly internally, that you need to promote, maintain and elicit usage within the organization.  Let’s say that my travel agency provides clients with a list of restaurants near a location that they pick.  They can do this from within the pages where they book the travel or, because I am a nice guy and I want to drive traffic to my site, I also put this on the main page.  The email system also needs it because it will generate the itinerary and associated restaurants and send that information to an email address.

There are valid reasons for creating web services (same function, different places) and poor reasons for creating web services (resume padding, bored).  Know which one is which and make sure that the reasons are valid.

Best Practices

In the past year I’ve stated a phrase so many times that I began to wonder what I really meant.  I’m sure you’ve seen the phrase many times before, both in my writing, in job postings, in management briefs, all over the place, but what does it mean?  The phrase is:

"… best practices …"

This is kind of a loaded question.  I could state that what I mean by best practices is that it is something that I think is important to do.  That would definitely boost my ego, but for the most part when I call something a "best practice" it means something more. 

For the most part when I call something a best practice it is because it is:

An established sets of practices, procedures or methods used to achieve an optimal result.

Now, who defines what an optimal result is in this context?  There are a number of different contributors:

  • Academia.  Some things are called best practices because those in the academic world have studied, debated and decided that the best practice in question meets the criteria.  While those in academia do provide some input into best practices, the problem lies in the fact that the academic world is never as complex and complicated as real life.  So, take their opinion with a good dose of realism.
  • Standards Groups.  These people create certain standards and they should, theoretically, know how to make the most effective use of the standards.  They are usually more accurate with their proclamations, but, once again, ideal situations are difficult to find and some of their best practices do not work in real life.
  • Industry Consensus.  This is something that the majority of people in an industry think is a good thing.  It is usually decided upon by a grass roots movement by people who actually do the work and not strictly academics.  Their opinion is highly valued as it is more likely to provide meaningful results when used in the correct context.
  • Organization.  Based on the way that an organization does things, there may be some specific practices that need to be done by an organization to improve the end result.  These best practices may fly in the face of other best practices read and used over the years, but because they are specific to an organization they are probably the most relevant.  (Please note that if the organization changes, but the best practice doesn’t then this becomes nothing more than a waste of electronic ink.)

All of these opinions, however, must be tempered with the fact that they were created with certain pre-conditions in place or under certain technological constraints.  A best practice is only a best practice if it actually applies to your situation.  Minor variances can sometimes be ignored, but if your naming standard was based on 6 character RPG standards, they probably aren’t applicable to .NET 2.0, so some (un)common sense needs to be used as well.

Some things should not be “added on”

When building an application there are some things that can be added on afterwards:  new functionality, better graphics and friendlier messages.  These are all things that add more value to the application.

There are some things, however, that should not be added on afterwards:

  • Error Handling.  What?  Don’t add on error handling afterwards?  No.  It needs to be done at the start.  Now, I know what some of you are saying "But, Don, we’ll add this in if we have a problem."  Face it, every new application has problems and if you don’t have error handling in at the beginning you are spending needless cycles trying to debug the application and you are causing me to drink Pepto-Bismol as if it were Dr. Pepper.  We recently had to help a project debug their application in the UAT environment, but they had no error handling at all, except the default ASP.NET error handling.  Thank goodness for Avicode, as it helped us pinpoint the problem quickly, just far too late in the development cycle.
  • Object Cleanup.  If you create an object, kill the object.  It’s simple.  It’s so simple that even Project Managers can do it.  By not cleaning up after yourself you raise the potential for memory leaks to happen.  And you know what that means?  Alka-Seltzer, Petpo’s cousin. I can’t tell you the number of applications in which we recycle the component once it hits a certain limit because it would keep you awake at night.  (Lunesta, another cousin)  Suffice to say that many of our applications are forced to recycle after a certain period of time or when they exceed a certain size. 

The scary thing is that both of these items are considered  best practices for writing code in the first place.  I know the excuses "…the project manager won’t let me do this …" or "…I don’t have enough budget to do this …" or, the one heard most frequently "… I don’t have the time to do this …:.  Very few project managers tell their staff how to code, so the first excuse is just a cop out.  As for the budget, doing these items does not add significantly to the cost of application as it usually makes debugging faster and easier, so the budget excuse is just that, an excuse.  As for the time, if you’re short on time, you need to do this as it will help you.

One of the things that many Health Organizations are putting in place is prevention of disease so that there is no need to cure the disease.  Prevention is much more cost effective.  Object Cleanup is prevention, pure and simple.  When someone has symptoms that need to be diagnosed, what does the doctor do?  Perform a seance?  Guess?  Or do they use a tool to help them out?  Ever heard of an MRI or even an X-Ray?  Think of Error Handling as a tool to help you diagnose the disease faster.  It’s better than guessing.

So, object cleanup prevents problems and error handling helps diagnose problems.  So, I guess this means that I’ll be seeing more applications with these items as an integral part of the overall application or do I need to go back to the medicine cabinet?

Post Mortem, but before Death

Justice Gray sent me an interesting note the other day in which he described a "post mortem" debriefing, but before the project had even started.  As Justice put it:  “It’s six months/a year/etc. from now and the project has failed.  Why did it fail?

One of the things that people have a hard time doing is learning from their mistakes.  George Santayana said "Those who do not learn from history are doomed to repeat it" and that is very true in the IT industry.  We work on projects that are months long, sometimes years, and during each project we make mistakes, fix them and go on to another mistake.  By the end of the project we have discovered, and fixed, dozens of mistakes.  We then make the biggest mistake of all, however, and repeat those mistakes on the very next project.  We failed to learn from our mistakes.

By setting up a post mortem before the project even starts, however, you’re asking the developers to think about the project, think about what has gone wrong on previous projects and apply those lessons to the new project.  In essence we are learning from our mistakes and applying the fixes to the next project. 

Does this work?  Well, from my personal experience I’ve had mixed results.  We did this for a couple of smaller projects and we were quite successful at identifying potential problems early on and work on risk mitigation for those items.  And that is what we are talking about:  risk mitigation.  Identifying a potential problem and what can be done early on to minimize the odds of it happening.  We also tried this same approach on a much bigger project, but the whole process blew up in our faces.  We identified so many potential problems that we spent more time mitigating risk than we did building an application.  And even after all of that advance preparation, we were hit with some nasty problems that derailed the project because of a couple of people who were not committed to the process.

Can this work, discussing potential failures before the project starts?  Most definitely, but it is really important that people think about past mistakes and problems and how to mitigate the risk of them occurring.  It also depends on the commitment of the people involved.  If the people aren’t committed to the process then the results may not be what you expect, or need.

Optimization — Part 3

There’s help you can get when you hit the wall for performance?

You bet your sweet bippy there is.  The most obvious places to look for help are with the people around you.  I find that one of the most effective resources I have is my wife.  Now, my wife is not particularly computer literate.  Seriously.  She is, however, an excellent sounding board.  When I explain something to her I have to get rid of the technical jargon and explain it to her in simple terms.  Usually, while in the process of trying to simplify the explanation, my mind goes off on a 90 degree tangent and i solve the problem that I am working on.  This doesn’t mean that this will work for everyone, but it works for me.

Or, try reaching out to other designers and asking for their opinion.  Give them a high level overview of the problem, what you’ve been looking at and let them think for a few minutes.  Don’t expect an instant answer, because if something has been troubling you for a while, don’t expect someone else to solve the puzzle instantly.  (Although, my wife does really well on those little puzzles where you have to take rings off a rope and untangle hopelessly tangled messes of metal.)

You can even talk to the Deployment Team.  Yes, I know, normally you avoid that, but you’d be surprised at how much help we can be.  By deploying and supporting over 80 applications the odds are that we’ve seen your problem before and that there are at least two, if not seventeen different ways to resolve the problem and make your life much easier.

Sometimes the answer is really obvious, but because you are so deeply involved you "can’t see the forest for the trees".  This is where reaching out to others, both technically adept and technically inept, can help.  It forces you to look at the big picture, the forest, and see whether or not there is a different path.

Secret Meeting

I got my hands on a recording of a secret meeting held between project managers last Friday.  Here it is in it’s gory glory.

"OK, so it’s agreed, we increase all of our estimates by 10% and blame it on the overhead caused by the Deployment Team.  Right?"

"Ah, hang on a moment.  There might be a problem with that."

"What is it <too garbled to understand>?  What don’t you understand about pinning the blame on the Deployment Team?  In particular that pain in the butt, Don."

"Well, don’t you think that he might notice what’s going on if everyone suddenly complains about his team costing the organization a lot of money?  I mean, I don’t think he’s a complete moron and he might figure something out."

<sound of people spitting out their beer on to the floor>

"Not again?!?!?"  <sounds like the waitress bring a mop to clean things up>

"What makes you think Jessop can figure this out?"

"Well, he has been in projects before, from a project plan creation perspective and he understands that estimates need to be complete.  And this means that they have to take into account all of the costs of the project.  Some of those costs include interacting with the Deployment Team to ensure that the application is deployed properly."

<long silence>

"You’re a plant, aren’t you?  You’re not really a project manager, your a mole!  Get HIM!!!!"

The recording breaks up at this point as the noise of chairs being tipped over and beer steins breaking on the floor seem to take up much of the remainder of the recording.  There is the faint sound of someone shouting " … no blood inside the bar … ", but that could be just my imagination.

So, Project Managers, how are those estimates coming along?

Don’t Worry About Performance!

There was a statement made a number of years ago in an ISWG (Integration Services Working Group) meeting which, to summarize, said "Don’t worry about performance, we’ll take care of that".  While that is probably going to be the epitaph of at least one person, I think it is time to set the record straight.

Worry about performance.

OK, now that we’ve gone 180 degrees it’s time to put some parameters around this.

  • Don’t worry about things over which you have no control.  The speed of a DCOM call is something you have no control over, neither is the time required to create a connection for a connection pool, the time required to retrieve session state, nor the time required to process an HTTP request.
  • Do worry about things over which you do have control.  While you can’t do anything about the speed of a DCOM call, you can control, to an extent, the number of DCOM calls that you make.  Less chattiness is better.  While you do not have control over the speed of the resources, you have control over how effectively you use those resources.

The UAT and Production environments into which your application will eventually move has a myriad of interconnected pieces that, together, create the environment in which your application will exist.  While you cannot control those resources you control how you use them.  <Al Gore>  Ineffective use of resources and our unwavering dependency on technologies that produce greenhouse gases is threatening our livelihood and our very planet.</Al Gore>  Ineffective use of resources in any ecosystem is bad, whether that ecosystem is our planet, or our clustered production environment.  Infinite resources are not available and never will be, but effective use can be made of existing technology:

  • Up until the late 1990’s the computers on board the Space Shuttle flight deck were Intel 8086 chips.
  • The original Xbox had a 733MHz CPU with 64MB of memory and yet it could (still can) outperform peoples desktops.
  • Mission critical systems have been designed with, when compared to modern technology, primitive means.
  • The first spreadsheet I used, Visicalc, ran on a 1 MHz processor.

All of these examples show that you can make something run well in limited circumstance, but you have to want to.

Identity Theft

I guess I can say that I am now a statistic.  You know, one of those millions of people who have been the victims of identity theft.  Let me tell you the story.

When I got home from work on Monday I noticed that I had supposedly been sending out emails from eBay account at lunch.  Within 15 minutes of the start of the emails I received an "A26 TKO Notice: Restored Account" from eBay UK stating that:

It appears your account was accessed by an unauthorized third party and used to send unsolicited emails to other community members, including email offers to sell items outside of eBay. It does not appear that your account was used to list or bid on any items.

The first thing I tried to do was log into my account.  Well, something either eBay UK did or something the hacker did was change my password.  I tried to enter in my answer to the Secret Question, but that didn’t work either as the information on my account had been changed.  Following the various prompts on the eBay site I ended up sending them an email telling them what had happened and what the next step should be.

A couple of hours later I got another email that I apparently sent out eight hours after the first round.  Not content to sit by and wait for the email process to work its way through the system I then started scouring the eBay site for a phone number to call.  You know, that is one of the hardest things I had to do!!!!  I followed all the usual routes and ended up with forms to fill out.  I never did get a phone number, so I had to use their "Live Help" facility.  (My reluctance to go with this approach was due in part to a 45 minute wait on the weekend for "live help" from another company, which never even connected with a human being.)  In the case of eBay, however, the wait was less than two minutes, they told me my position in the queue (started at number 5) and the approximate wait time. 

The person who was on the other end of the chat could have been anyone, anywhere in the world.  The fact of the matter is, they looked at the information on my account, the notes they had sent to me and knew that I needed to talk to the Account Security division.  Within 30 seconds I was "chatting" with someone else who had the power to help.  Two minutes later things were fixed and that included changing the password on my account to a "stronger" password.

Was it brute force hacking of my account and password?  Not if this article is correct.

This particular episode was rather benign in that all that really happened was that some emails got sent and I had to change my password.  It could have been worse.  Much worse.  Think of that the next time you sign up for a web site.  Or, more importantly, think of that the next time you are building an externally facing application.  What are you doing to safeguard the information that you keep on your clients?  What are you doing to protect their safety?  Can you honestly say that you’ve done your best?

Validity vs. Reasonableness

While most of our applications do validity checks on data, not all of them do reasonableness checks.  Let me explain the difference.

Data Validation.  Let us suppose you have a number of fields on the screen:  name, address, birth date, phone number, and spouse’s name, spouse’s address, spouse’s birth date, spouse’s phone number and a marriage date.  Data validation would ensure that if there is a birth date, it is a valid date.  So in this case it would check to ensure that all of the date fields are valid.  It would also check to ensure that the phone number follows any one of a number of different standards, but predominantly the fact that it is numeric in nature.  You can also extend data validation to more complex tasks such as determining if the postal code is correct.   In general terms, data validation serves to ensure that a single piece of data is a valid for that data type.

Data Reasonableness.  OK, now that we’ve gotten the basics out of the way, there are still a number of checks that we can perform.  If there is a marriage date, then the date must be a certain time period after the birth date of both parties.  This is not just a simple "if marriageDate > spouseBirthDate then Happiness()".  We need some additional logic to ensure that even if the data is valid, it also must make sense.  Having data make sense is as important as ensuring that it is valid.

While there are many schools of thought on this, most post secondary training lumps both data validation and data reasonableness together under the "validation" banner.  This, unfortunately, has had the effect, in most cases, of putting data reasonableness checks in the background or has the checks embedded deep within the business logic of the application.  In most cases these checks can be done at the UI level, really quickly and prevent a lot of background processing that clogs the servers.  The other big problem is that some of these reasonableness checks are missed because "I thought the client was going to do that".

Just remember, there is a lot of data checking that needs to be done and reasonableness is just another item in the list.

Temporary (Expires yyyy/mm/dd)

My wife renewed her drivers license recently, as she turned 29, again, on September 24th.  She went into the Registry near our place, filled out the form, paid her money, got her picture taken and was given a "temporary" drivers license to last her until her real license was sent to her.  If the real license takes too long her temporary license is going to expire on her and that could cause no end of grief.

My bus takes a little bit longer to get to work in the morning as there is a "temporary" detour on it’s normal route.  The city has dug a hole in the road, the entire width of the road, in order to do some emergency repair on the pipes.  They don’t have a lot of time to get this done as it impacts a lot of traffic, a lot of buses (both ETS and school) and a lot of residents.  The only way to get to certain houses is to wind your way through back alleys.  Oh yeah, this is definitely going to be temporary.

My daughter has a temporary spacer in her mouth.  One of her baby teeth, incorrectly filled by an earlier dentist, developed some severe damage and needed to be pulled.  A temporary spacer was inserted so that her teeth would grow in their proper place until the adult tooth comes in.  The spacer is going to be removed either when the adult tooth comes in, or when the baby tooth to which it is attached falls out.  No choice, it is temporary.

We have servers, both physical and virtual, which were set up for temporary purposes.  We now have to face the task of upgrading them from NT 4 to something supported.  (OK, I lied, but you get the point, don’t you?)  If things are temporary, give us a date and we will set up the system to self destruct on the day after.  If things aren’t temporary, for goodness sake, tell us!!!!!  The "Oh, it was temporary, but the client said …" story is getting old and, quite frankly, has been done better by other teams. 

Remember temporary means that it goes away.  Pick a date.  Any date.  Please….