Web services need planning

Project Management Plan

Project Management Plan

Copyright perhapstoopink (Creative Commons)

I get calls all the time at home from “investment planners” and “retirement planners” and all sorts of other people who believe that I need to plan for the future.  Well, their not wrong, but planning needs to occur in other areas of your life as well.

Taking on my mantle of IT Philosopher I’m here to talk to you about planning.  Future planning for your applications.

I understand that being able to write “Developed and implemented web services” looks good on a resume, but if your application does not need a web service don’t give it one.  Sorry for the emphasis, but it’s needed.  Quite often I see application architectures that don’t make a lot of sense, other than the fact that it will look good on a resume.

Let’s take a look at a fictitious travel agency “Jessop Fantasy Tours”  (hey, my story, my travel agency).  One thing that probably makes good sense to spin off into a web service is “RetrieveTravelItinerary”.  Based on the name you would assume that this web service would retrieve the Itinerary based on a key being passed in.  In order to determine if it meets the criteria for a web service see if it satisfies these points:

  1. There is a need for this information in multiple locations in one or more applications.

OK, there was originally going to be a list of five or six points, but, for the most part, it really comes down to the above question.  Even then I could spend another couple of thousand words just talking about the permutations and combinations of items that make up the little phrase “multiple locations”.

For instance, I don’t mean two different places on the same screen.  I don’t mean in two different functions, but in two different application functions.  (“List of Itineraries” and “Itinerary Details”).  If you plan on commercializing the usage of the function then that would qualify as multiple locations.  By commercializing the usage I don’t necessarily mean external to the organization.  You could have a function, used strictly internally, that you need to promote, maintain and elicit usage within the organization.  Let’s say that my travel agency provides clients with a list of restaurants near a location that they pick.  They can do this from within the pages where they book the travel or, because I am a nice guy and I want to drive traffic to my site, I also put this on the main page.  The email system also needs it because it will generate the itinerary and associated restaurants and send that information to an email address.

There are valid reasons for creating web services (same function, different places) and poor reasons for creating web services (resume padding, bored).  Know which one is which and make sure that the reasons are valid.


Strange things to say about code

Vim + NERDTree + Taglist

Copyright moleculea (Creative Commons)

I was looking at a SlideShare presentation on “Writing Code You Won’t Hate Tomorrow” and there were some attributes of code that I had never considered before or thought of in those terms:

  • – Code has an expiration date
  • – Code evolves
  • – Complexity kills comprehension

These are all things that we should know intuitively.  Things that we should have been taught when we were in school.  Unfortunately, most schools don’t teach the philosophy of programming, although I really think that they should.  Writing code is so much more than just combining language specific syntax into a cohesive function.  It is helping someone solve a problem, even if that someone is you.

Code has an expiration date.  When you write code you are writing it based on your understanding of the problem, your understanding of the immediate solution required and, to be honest, your skill in writing that code.  Things change, including your skill level, but, more importantly, the technology upon which the code was based also changes.  In some cases the technology no longer exists.  All code has an expiration date, but that date varies depending upon the skill of the programmer, the longevity of the tools and the understanding of the problem.

Code evolves.  The fruit fly is chosen for experimentation due to it’s proclivity to mutate quickly and spawn quickly.  Your code is like a fruit fly.  As requirements change and “service requests” are implemented the code changes.  First just a little bit, but soon it has taken on a life of its own as the changes to the system to help it accommodate new requests soon warp the code into something it was never originally intended to be.

Complexity kills comprehension.  This goes back to K.I.S.S. – Keep It Simple, Stupid – and the fact that the simpler something is the easier it is to understand.  Einstein once wrote:

It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.

This has subsequently been rephrased as “Everything should be made as simple as possible, but no simpler.”  (Apparently that applies to his statement as well.)  I have seen some very complex code over the past 30+ years of being in IT and I rarely see complex code that has been made “beautiful” in it’s simplicity.  Most often the code is complex and extremely difficult to understand.  It doesn’t have to be.  The more complex the code the less the programmer really knows about the problem.

Applications don’t last forever.  With the advent of the Internet, Service Oriented Architectures, RESTful interfaces, JSON, mobile, etc., the lifespan of code is shrinking.  No, let me rephrase that, the lifespan of poorly designed and executed code is shrinking.  Well designed and written code still has a longer lifespan as it can be more easily molded and shaped to meet new business requirements.

So, what is it that you want to get out of life?  A legacy of rewritten code or a legacy of well written code?

Some things should not be “added on”

When building an application there are some things that can be added on afterwards:  new functionality, better graphics and friendlier messages.  These are all things that add more value to the application.

There are some things, however, that should not be added on afterwards:

  • Error Handling.  What?  Don’t add on error handling afterwards?  No.  It needs to be done at the start.  Now, I know what some of you are saying "But, Don, we’ll add this in if we have a problem."  Face it, every new application has problems and if you don’t have error handling in at the beginning you are spending needless cycles trying to debug the application and you are causing me to drink Pepto-Bismol as if it were Dr. Pepper.  We recently had to help a project debug their application in the UAT environment, but they had no error handling at all, except the default ASP.NET error handling.  Thank goodness for Avicode, as it helped us pinpoint the problem quickly, just far too late in the development cycle.
  • Object Cleanup.  If you create an object, kill the object.  It’s simple.  It’s so simple that even Project Managers can do it.  By not cleaning up after yourself you raise the potential for memory leaks to happen.  And you know what that means?  Alka-Seltzer, Petpo’s cousin. I can’t tell you the number of applications in which we recycle the component once it hits a certain limit because it would keep you awake at night.  (Lunesta, another cousin)  Suffice to say that many of our applications are forced to recycle after a certain period of time or when they exceed a certain size. 

The scary thing is that both of these items are considered  best practices for writing code in the first place.  I know the excuses "…the project manager won’t let me do this …" or "…I don’t have enough budget to do this …" or, the one heard most frequently "… I don’t have the time to do this …:.  Very few project managers tell their staff how to code, so the first excuse is just a cop out.  As for the budget, doing these items does not add significantly to the cost of application as it usually makes debugging faster and easier, so the budget excuse is just that, an excuse.  As for the time, if you’re short on time, you need to do this as it will help you.

One of the things that many Health Organizations are putting in place is prevention of disease so that there is no need to cure the disease.  Prevention is much more cost effective.  Object Cleanup is prevention, pure and simple.  When someone has symptoms that need to be diagnosed, what does the doctor do?  Perform a seance?  Guess?  Or do they use a tool to help them out?  Ever heard of an MRI or even an X-Ray?  Think of Error Handling as a tool to help you diagnose the disease faster.  It’s better than guessing.

So, object cleanup prevents problems and error handling helps diagnose problems.  So, I guess this means that I’ll be seeing more applications with these items as an integral part of the overall application or do I need to go back to the medicine cabinet?

Don’t Worry About Performance!

There was a statement made a number of years ago in an ISWG (Integration Services Working Group) meeting which, to summarize, said "Don’t worry about performance, we’ll take care of that".  While that is probably going to be the epitaph of at least one person, I think it is time to set the record straight.

Worry about performance.

OK, now that we’ve gone 180 degrees it’s time to put some parameters around this.

  • Don’t worry about things over which you have no control.  The speed of a DCOM call is something you have no control over, neither is the time required to create a connection for a connection pool, the time required to retrieve session state, nor the time required to process an HTTP request.
  • Do worry about things over which you do have control.  While you can’t do anything about the speed of a DCOM call, you can control, to an extent, the number of DCOM calls that you make.  Less chattiness is better.  While you do not have control over the speed of the resources, you have control over how effectively you use those resources.

The UAT and Production environments into which your application will eventually move has a myriad of interconnected pieces that, together, create the environment in which your application will exist.  While you cannot control those resources you control how you use them.  <Al Gore>  Ineffective use of resources and our unwavering dependency on technologies that produce greenhouse gases is threatening our livelihood and our very planet.</Al Gore>  Ineffective use of resources in any ecosystem is bad, whether that ecosystem is our planet, or our clustered production environment.  Infinite resources are not available and never will be, but effective use can be made of existing technology:

  • Up until the late 1990’s the computers on board the Space Shuttle flight deck were Intel 8086 chips.
  • The original Xbox had a 733MHz CPU with 64MB of memory and yet it could (still can) outperform peoples desktops.
  • Mission critical systems have been designed with, when compared to modern technology, primitive means.
  • The first spreadsheet I used, Visicalc, ran on a 1 MHz processor.

All of these examples show that you can make something run well in limited circumstance, but you have to want to.

Validity vs. Reasonableness

While most of our applications do validity checks on data, not all of them do reasonableness checks.  Let me explain the difference.

Data Validation.  Let us suppose you have a number of fields on the screen:  name, address, birth date, phone number, and spouse’s name, spouse’s address, spouse’s birth date, spouse’s phone number and a marriage date.  Data validation would ensure that if there is a birth date, it is a valid date.  So in this case it would check to ensure that all of the date fields are valid.  It would also check to ensure that the phone number follows any one of a number of different standards, but predominantly the fact that it is numeric in nature.  You can also extend data validation to more complex tasks such as determining if the postal code is correct.   In general terms, data validation serves to ensure that a single piece of data is a valid for that data type.

Data Reasonableness.  OK, now that we’ve gotten the basics out of the way, there are still a number of checks that we can perform.  If there is a marriage date, then the date must be a certain time period after the birth date of both parties.  This is not just a simple "if marriageDate > spouseBirthDate then Happiness()".  We need some additional logic to ensure that even if the data is valid, it also must make sense.  Having data make sense is as important as ensuring that it is valid.

While there are many schools of thought on this, most post secondary training lumps both data validation and data reasonableness together under the "validation" banner.  This, unfortunately, has had the effect, in most cases, of putting data reasonableness checks in the background or has the checks embedded deep within the business logic of the application.  In most cases these checks can be done at the UI level, really quickly and prevent a lot of background processing that clogs the servers.  The other big problem is that some of these reasonableness checks are missed because "I thought the client was going to do that".

Just remember, there is a lot of data checking that needs to be done and reasonableness is just another item in the list.

Weapons of Mass Destruction

America went to war with Iraq because they wanted to find and destroy the Weapons of Mass Destruction.

In many respects that what I do by looking at the way applications are installed, operate and behave when encountering errors:  I’m looking for weapons of mass destruction.

My first IT job was with a construction company and my crowning achievement was an application that accurately allocated work site costs to various job codes in the accounting system on a daily basis.  It was rather tricky due to the fact that the company I worked for was a multinational company and each country, indeed province/state, had different holidays so it needed to take into account when costs should be allocated based on whether or not the previous day had been a holiday or not, in the location where the construction site was located.  Jobs on which 7×24 construction was occurring had other conditions that needed to be met.  All in all it was a masterpiece of software engineering.

Almost.

You see, I was under a tight timeframe for getting this done, as the VP in charge of construction had told the CEO that it would be in place by July 1st.  I only had 4 more weeks to finish the coding and then implement the application in the main production batch jobs.  Time was tight so I did what every rookie (and many seasoned professionals) do when faced with something that is difficult to compute:  I hard coded the answers.  I hard coded the holidays for all jobs sites for the next 18 months inside the application. 

It was simple to do and saved me a lot of trouble, because, you see, I left the company 2 months later, leaving behind a ticking time bomb in their production systems.  In sixteen months things were going to blow up, all because I took the easy way out instead of doing it properly.

Tick, tick, tick, tick …

Schopenhauer’s Law of Entropy

So, just what is Schopenhauer’s Law of Entropy?  Simply put, it is this:

If you put a spoonful of sewage in a barrel full of wine, you get sewage

So, what does sewage have to do with programming?  It’s not sewage that I’m looking at, but rather the concept behind it.  In IT terms, what Schopenhauer is saying is that no matter how good the overall application, if one part doesn’t work the whole application gets tarred with the bad brush. 

It is unfortunate that a single poorly designed, written or executed page can make someone believe that the entire application is poor.  Their perception of the application is what is important, not reality.  Kind of scary, isn’t it, when perceptions are more important than reality.  But this is what happens in our business and it is something that we need to understand and do our best to influence.

So what influences this perception?  Well, consider this:  two web applications side by side on your desktop.  You push a button on the left one and you get the ASP.NET error page:  unfriendly, cryptic and somewhat unnerving.  You push a button on the right one and you get an error message in English, that explains there is a problem and that steps are being taken to resolve the issue.  Which one would you perceive to be better written and robust? 

How about another example?  You push a button on the left application and you get an error message that says "Unexpected Error.  Press the OK button".  You push a button on the right application and you get an error message that says "Our search engine is currently experiencing some difficulties and is offline.  Please try again later."  Which one do you perceive to be better?  Which one do you think your business clients will think is better?

It’s not just one thing (error message or not) that gives you a feeling of confidence when dealing with an application, it is a multitude of little things.  Making things more personalized helps.  Translating from Geek ("Concurrency error") to English ("Someone else has updated the data before you") helps a lot.  Making it seem that you spent some effort to foolproof the system (i.e. don’t make every error number in your application the same error number).

No matter how good the rest of your application, one bad move can create sewage.

Initialize All Variables at Their Points of Declaration

I was reading a book recently called Code Craft – The Practice of Writing Excellent Code and one of the comments struck a particular chord with me as it brought back memories of an upgrade that went horribly wrong.  At least for me.

There was a brief section called "Initialize All Variables at Their Points of Declaration".  Now, this may seem self explanatory and quite normal to some people, but others think that this is rather strange.  "Why would I initialize a variable that I may never use?"  The problem is, that not everyone follows the same coding practices in real life.  Sometimes the compilers help/hurt us in this regard.  Back when I was predominantly working on the mainframe, we were switching from an older version of COBOL to COBOL II.  Ooh.  COBOL.  I can see your eyes glazing over.  Stay with me, there is method to my madness.

The process of conversion was really quite simple.  Recompile.  It wasn’t that hard.  However, we discovered a little bit of a problem.  When we did our testing we discovered that we were occasionally getting OC7 (data exception) errors when everything should have been working.  Indeed, running the program multiple times against the same data actually generated different results.  After a lot of head scratching we determined that the problem lay in the fact that the old compiler, by default, initialized variables when they were defined.  COBOL II did not do this by default.  When the application was loaded into memory it would occasionally access memory that had been initialized for some other purpose and the program would work.  Other times, however, it was accessing "garbage" and the program would blow up.  If the original developer had initialized the variables in the fist place we never would have had a problem.

So, we made a small change and everything was perfect.

Almost.  Because of how we were doing the upgrade process I had to baby sit the recompilation of 1900 COBOL programs in 4 different environments (7600 recompiles altogether).  Took almost 48 hours to do it and I got almost no sleep, and all because someone failed to initialize a couple of variables.

Solving the Right Problem

One of the hardest things to do it solve the right problem at the right time. 

When investigating a problem you may end up looking at a wide variety of possible solutions.  Some of these solutions are quick fixes while others require a fair amount of effort to implement.  The question is, which one do you propose?

For a crisis, the quick fix is usually the right choice.  Things need to be resolved quickly and the best solution may not be able to solve the problem fast enough.  As a result the quick fix is usually chosen for Production emergencies and rushed through into Production.  Quick fixes are not meant to be permanent solutions, but in many cases they end up being permanent for a variety of reasons.

In less crisis oriented situations, however, the best solution may actually be the resolution of a deeper, more convoluted problem that is actually the root cause of the issue.  Unfortunately, resolving the root cause of a problem may actually be a problem in and of itself.  There may be significant effort and money that needs to be spent in order to resolve the issue in the manner that it should.  Sometimes the problem is so fundamental to the application that it almost appears that you have to re-write the application to make it work as desired.  If this is the case, is this what you should propose?

As with many things in life, it comes down to a business case:  is the cost of implementing the solution less than the cost of living with the quick fix?  If this were strictly a matter of dollars and cents then the answer would be know right away.  Unfortunately the cost of living with the problem is not easily quantifiable.  How do you measure consumer lack of confidence in terms of cost?  How do you measure consumer satisfaction in terms of cost?  in many cases only the business area affected can even hope to determine the cost.  It is our job to present the facts as we know them, the costs as we know them, and let the business decide the ultimate cost.

Trust

One of the biggest things we need to deal with in applications is trust in other parts of the system.  How much trust do you have in the calling application?  How much trust do you have in the common routines that you call?

For instance, you’ve created a method that will take an XML stream and use it to update a row in a table.  How much faith do you have that the calling application has sent you the proper types?  Should you check everything to ensure that strings are strings and numbers are numbers?  Do you double check to ensure that dates are actually valid dates and that the time listed actually exists?

I used to work for a manager that insisted that every time your method gets invoked it should double check all of the data being passed in before it did any work: verification was the first thing you did.  Being young and full of myself, I didn’t follow that rule because, well, to be honest, I was writing both sides and I knew what I was passing myself!!!  Fast forward a couple of years and someone else is maintaining the code.  Well, they made some changes that didn’t follow the rules and, in production, it blew up horribly because the method did not verify that the correct type of data was being passed.  Being on the support side I was called in to troubleshoot and instantly recognized what the problem was and the solution that was required.  A quick compile and test and the application no longer died horribly, but gave a nice, easy to understand error message.

With today’s modern languages much of this work is taken care of for you by the development tool during design time as you need to ensure that you are calling with the correct types, or the the compiler won’t even compile your application for you.  However, there is a problem when you are using XML or if you are taking in a string and attempting to use it as a numeric value.  This is of particular concern to user interfaces as pretty much everything you retrieve from the UI is a string that you need to convert and use.

A user interface should place no trust in the end user entering in the correct type of data into the text box.  But, how much trust do you place in one piece of code calling another piece of code?  I guess that depends on whether or not you are going to be the person maintaining the code for the lifetime of the application.  If you are, then I guess you can trust the code.  If you aren’t then being paranoid may be beneficial.