At what point does a “data fix” do more harm than good? For that matter, what is the tipping point between doing a data fix and fixing the application? A data fix is merely a bandage for an application. It treats a symptom, not the problem.
There are many debates about this very topic but let’s get things settled at the beginning: one data fix is too many. Yes, that seems rather draconian, but if you need to go into the database and manipulate the data to achieve the correct business results then there is a failure somewhere in the process. Perhaps it was in gathering the requirements. Perhaps it was in not providing the correct testing. Or perhaps it was an unforeseen circumstance. (The users got access to the applications. It could happen.)
That all depends on which side of the fence you are standing. If you are developer things are simpler. If you support DevOps, if you are one of those people that is trying to automate processes or make things simpler for others, then your life is more complicated. Instead of one tool you’re dealing with many tools all hooked together with bubblegum and twine.
Embed from Getty Images
Back in the “good old days” when IBM 3270 was meaningful and everyone did waterfall there was kind of a set “project management” overhead that was associated to the project. For most projects I worked on at Accenture (over fifteen years ago) we charged between 10% and 12% depending upon the size of the project and the technologies we were doing. (Hint, the larger the project the higher the overhead. See my other articles on communication costs increasing based on the number of people in the group.) Continue reading “Deliver Fast”
Trust is an interesting thing. Our society, by the way we have designed it, places a lot of trust in people doing their jobs or doing the right thing. For instance, have you thought about how many people you are trusting when you buy your coffee in the morning? The grower, all of the people involved in the transport, the people at the processing plant, more transportation, the workers at the distribution centre, more transportation people and all of the employees at the coffee shop. Hundreds, if not thousands of people had the opportunity to taint your coffee, to add something to it or do something wrong so that it wouldn’t be the coffee you expected.
But none of them did it. You implicitly trusted them.
But what happens when that trust breaks? For Enron it meant the end of the company. For Bill O’Reilly it meant the end of a job. Sometimes we forgive them for breaking our trust, but if the trust has been broken repeatedly, at some point you just have to say “no”. An accountant who has repeatedly been caught laundering the books is not going to get a job with an accounting firm, no matter how much he says he is sorry. Someone who has sold state secrets is never going to get a security clearance again. For the type of work, the type of trust that was abused, there is a different threshold at which the trust can never be extended again.
Sometimes, however, revoking that trust is difficult to do, if not impossible. If the government has lost your trust you vote another one in … unless they lost your trust because they established a dictatorship and there is no more elected government. But what if it’s not the entire government, but a part of it. What if just a single section has lost your trust? What if a single group has tried, again and again, to gain your trust but has fallen down repeatedly? At what point do you say “no”? At what point do you say “this has gone on long enough”?
Sometimes you have to break things apart in order to rebuild them properly.
OK, this note is technical in nature, but the lesson is applicable to virtually everything that we do. So, if you can muddle through the geek speak I’ll have a present for you at the end. Well, maybe not a present. In the end you may know the answer already and I’m just repeating what you are mumbling in your sleep every night. Not that I’m listening to you mumble but as long as you play Candy Crush just before going to bed … oh, never mind.
In the course of some fun reading, I was reading an article on “A DLM Approach to Database Testing“. Yeah, it’s kind of sad, but this is indeed fun reading. Anyway, the author, Edward Elliot aka Phil Factor, was talking about how changing your mindset, looking at database testing from a little different perspective, you end up with more maintainable code inside the database.
Simpler database code
Generally, the need to think about how to write simple tests that will prove their code works tends to lead developers to write simpler database code, each piece with a singular, specific purpose. For example, let’s say a developer needs to devise code that will:
Retrieve a set of customer records showing their daily spending
Run a calculation on those records to get the average daily spend
Add an entry to an audit trail containing the result
Return the result
The temptation might be to simply write a single stored procedure to do all the work and return the results. Consequently, you’d then need to write a test that also does a lot of work, to prove the right records are returned, that the calculation is correct, that we always get an entry to the audit trail. If the failing test does lots of things it is often hard to see what exactly what caused the failure.
However, if we think about how to write simple tests that verify just one thing, we tend to break the task down into manageable units, for example, writing a procedure to return the result, another to add an audit log entry, an inline function to perform the calculation, and so on. Each piece of code has a singular purpose and therefore so does each corresponding test, simply needing to verify that it does exactly what it is supposed to do. Being able to test each piece of code in isolation means that we also often get much more maintainable code, as instead of having a single procedure to do everything we can separate our code and re-use in other places
When I was younger I decomposed pieces of work into smaller, more manageable chunks. And this is exactly what the author is talking about: taking one complex tasks and turning it into four simpler tasks. A rule of thumb (from many years ago) was that if you couldn’t see all of the code on the screen at the same time, whatever you were writing was too long and needed to be split up. While this is a good rule of thumb, common sense does need to be inserted in there at some point. However, large, complex stored procedures should be broken up. I recently (okay, five minutes ago as part of some research into this article) discovered a stored procedure that was, including comments, over 1300 lines long. That is way too long.
What’s worse is that there are comments, from 2008, describing a bug that exists in the code.
From 2008. And it hasn’t been fixed.
And the complexity? There are over 45 SELECT/UPDATE statements with one of the longer select statements being almost 140 lines long. The complexity is enormous. So how do you test a single stored procedure that is over 1300 lines long and has known bugs? Well, there are ways, but they’re not pretty.
Quite often in life, we are tasked with what appears to be huge tasks that take our breath away. But, if we sit back and take a look at it, we invariably find that the one task is, in reality, a bunch of smaller tasks. And those smaller tasks? Quite often they can be decomposed into additional tasks. It doesn’t matter whether you are writing code or building an 80 story tower in downtown Edmonton, the concepts are the same: take the huge task and break it down into smaller, more manageable segments. The only challenge is knowing when to stop decomposing.
Do you remember Jenga? You can still buy the game now, but it doesn’t have the same impact it did “back in the good old days.” Smartphones, gaming consoles, and the rise of personal computers have all hurt the game industry. But, hey, apparently they’re making a comeback.
Now let’s take IT for a second. (Don’t worry, we’re going to get back to Jenga in a minute.)
There is the concept of “technical debt.” Techopedia describes it as:
Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution.
So, if you had the time to do it right, you would have come up with a better solution than the one that you implemented. It sounds like almost every project out there doesn’t it?
Technical debt is like Jenga in that the more changes (turns) you make, the more the overall application (stack of bricks) becomes unstable. You can keep making changes. but at some point. it is going to fall. We have applications that are in this condition right now, where we have piled change after change after change, all of them done in relative haste and all of them adding to the technical debt of the system.
It is taking longer to introduce the simplest of changes and therefore costing more money. It will soon come to the point where the money we pour into the applications is not even enough to keep it functioning properly.
The solution, however, is not something that people want to hear: throw it away.
Yup, those solutions that cost a ton of money? Sometimes you have to bite the bullet and start from scratch. Throwing it away doesn’t mean that you throw away the data, but you throw away the application and build something fresh, something new and something free of that technical debt.
How do you prevent the technical debt? You can’t. You will always have technical debt, but you can work to keep it minimized. If you have a solid architecture, solid principles around which you develop, you will accumulate technical debt much more slowly, and you can proactively work at keeping that debt minimal. See Refactoring.
The hardest part of all this is not understanding that you have technical debt or even the architecture and standards involved in minimizing and reducing it. The hardest part of the whole process is letting go of the existing system, understanding that the tower is falling and that you need to have something in its place before it comes crashing down.
Continuous integration (CI) and continuous deployment (CD). That sounds so … fuzzy?
Let’s take a look at CI/CD from an industrialization of IT perspective. Many of our apps are built in a “custom-built” style where a group of developers sit down and build all of the various pieces by hand. Oh, occasionally a piece will be added that someone else built, either internally or externally, but by and large, it is a custom build. You usually don’t even see the end product until it is finished. In the CI/CD world, things are built incrementally, much like an assembly line. Parts one and two go together and then part three is bolted on. At any time you can take that product in the making and get it a spin. Yeah, the analogy is getting pretty thin, so I think I’ll stop here and look at the pieces separately.
Continuous Integration. In software development when you have a stable release, the release upon which all other changes are based, you call this the trunk. When you start to work on a release you call it a branch and when the branch is complete you merge it back into the trunk. Kind of like pruning a tree. Our normal method of developing is to build something so that it is complete and then merge that code back into the trunk when we release. Continuous integration is different, however, in that you quite often don’t have a branch upon which you are making changes, you are making changes to the trunk. You write code, a function or procedure, and you integrate code back into the trunk of the product when you are finished. You don’t wait for a release, you don’t wait for marketing to get all of the mouse pads printed up with the correct URL, you integrate right away. Daily in fact.
This rapid integration causes concerns for some people. Lots of concerns. What if the code is not ready to be released? Then why did you say that you were finished? You integrate code when you are finished writing and testing the code. Not before and not six days later.
But you’re not ready with all of the pieces yet! Then you implement feature flags which state that the feature is not available for public consumption. Each environment in which your application is deployed has a set of feature flags so that you can be testing something in System Test that doesn’t show up in UAT or Production. Continuous integration ensures that everyone has the latest version of the code at all times and that the code is functioning code.
Continuous deployment. Theoretically, every time someone checks code back into the repository, you would compile and send the latest version to a development server. While this is a laudable goal, with larger projects this is unrealistic. So a daily deployment to a development or system test servers works out just as well. Yes, daily. Every single day the code gets compiled and deployed. We know the code should be working (we tested it before committing it back into the repository) so there is no danger of the app not working.
I sense much fear in you. Continuous integration and continuous deployment are missing one piece: continuous testing. Yes, we test the application a lot. Indeed, in a perfect world, we would be testing immediately after we compile. When we deploy onto a system test server when we deploy to a UAT server and even some testing when we deploy to production. Continuous testing means automated testing. If you do enough testing, at enough stages, an application change can go from the developer’s machine all the way into production without a single person manually testing the change.
And how does all this get done? Through automation and standardization.
We know that we need to automate things and continuous integration and continuous deployment give us an idea as to what we need to automate. But is simple automation going to be enough? If you automate a bad process, it’s still a bad process; it just gets done faster. We need to be more flexible, more … agile.
Industrialize software and service delivery for speed and responsiveness. Industrialize so that developers can devote their energy to creating customer valua and discovering new opportunities and needs – all at a fast pace. Adoption of modern application delivery practices, continuous integration and delivery, and DevOps – depends on this industrialization.
Forrester: Reforming AD&D Organizations for Customer Obsession: The Three Models, December 23, 2016.
I’ve got days, perhaps weeks worth of material just on this one bullet point from Forrester. Let’s talk about “industrialization” and see where that takes us. Wikipedia says:
Industrialisation or industrialization is the period of social and economic change that transforms a human group from an agrarian society into an industrial one, involving the extensive re-organisation of an economy for the purpose of manufacturing.
It’s not a bad definition, but it doesn’t work for me in context with the rest of the Forrester quote.
Industrialization is the process by which an economy is transformed from primarily agricultural to one based on the manufacturing of goods. Individual manual labor is often replaced by mechanized mass production, and craftsmen are replaced by assembly lines. Characteristics of industrialization include economic growth, more efficient division of labor, and the use of technological innovation to solve problems as opposed to dependency on conditions outside human control.
OK, this fits in much better. So, Forrester says that we need to “industrialize” software and service delivery and that this is the intent of Agile, CI/CD and DevOps. Industrialize. So we need to “mechanize” or “automate” as much as possible. Instead of craftsmen, we need assembly lines and we will use technology to solve the problems.
One of the biggest things that DevOps tries to do (in conjunction with Agile and CI/CD) is to reduce the amount of time needed for identifying the need for a change and when that change is in production. And while this can be done in a number of different ways, the DevOps way would be to automate the process of deploying the application. The more automation the better off we are.
But if we have to create a custom deployment workflow for every application unless we are doing a lot of deployments, the cost/benefit is a little hard to buy. So, let’s turn this into an assembly line like Investopedia said. In an assembly line, things are not customized, they are standardized. Everything uses the same nuts and bolts, the same screws, the same individual components. So in the DevOps world, we automate through standardization. You want to automate your deployment then your deployment will follow these steps and this pattern.
Do you want to deviate? No. (OK, “no” turns into “please justify the reason why your application is different from everyone else’s application” and if you can’t justify it in 20 words or less then the odds are that it’s not a good reason. And then this gets escalated to your boss and my boss and they have a chat and then it comes down to “the only reason you’re different is that you didn’t follow standards, so follow the standards”. Wow, I can justify almost anything in my own mind.)
So, in order to automate, in order to industrialize the development of software and services we need to standardize on stuff. A lot of stuff. Probably more things than we realize right now but the entire process is an evolution so the list of standards will need to evolve as well.
Now, how about continuous integration and continuous delivery?