How one decision can turn web services to hell
In this posting I will show you how one stupid decision may turn developers life to hell. There is a project where bunch of complex applications exchange data frequently and it is very hard to change something without additional expenses. Well, one analyst thought that string is silver bullet of web services. Read what happened.
Bad bad mistake
In the early stages of integration project there was analyst who also established architecture and technical design for web services. There was one very bad mistake this analyst made:
- All data must be converted to strings before exchange!
Yes, that’s correct, this was the requirement. All integers, decimals and dates are coming in and going out as strings. There was also explanation for this requirement:
- This way we can avoid data type conversion errors!
Well, this guy works somewhere else already and I hope he works in some burger restaurant – far away from computers.
Consequences
If you first look at this requirement it may seem like little annoying piece of crap you can easily survive. But let’s see the real consequences one stupid decision can cause:
- hell load of data conversions are done by receiving applications and SSIS packages,
- SSIS packages are not error prone and they depend heavily on strings they get from different services,
- there are more than one format per type that is used in different services,
- for larger amounts of data all these conversion tasks slow down the work of integration packages,
- practically all developers have been in hurry with some SSIS import tasks and some fields that are not used in different calculations in SSAS cube are imported without data conversions (by example, some prices are strings in format “1.021 $”).
The most painful problem for developers is the part of data conversions because they don’t expect that there is such a stupid requirement stated and therefore they are not able to estimate the time their tasks take on these web services. Also developers must be prepared for cases when suddenly some service sends data that is not in acceptable format and they must solve the problems ASAP. This puts unexpected load on developers and they are not very happy with it because they can’t understand why they have to live with this horror if it is possible to fix.
What to do if you see something like this?
Well, explain the problem to customer and demand special tasks to project schedule to get this mess solved before going on with new developments. It is cheaper to solve the problems now that later.
I deal with a system (UniData) where everything coming out of it is string based. I find that dealing with strings is not that difficult if you create the right .NET base architecture.
1) I only convert what needs to be converted.
2) I created a “base type” that has a StringValue property and the “Value” be of the type decimal/double/date/time.
3) I create a class/struct and inherite the “base type” for each converted data type value I need. i.e. UdDate, UdDecimal, UdTime, etc.
I’m not sure what your architecture is with all the SSIS packages are involved, but SSIS has little issue converting strings once you are familiar which SSIS data type to use.
Secondly SSIS may not necessarally be needed, unless you need the GUI aspect of it. Writing an import class is rather simple and if done right (reusable & simple) might be even easier to maintain later on. It may even be faster than SSIS and you could multi thread it or use MARS.
Well, there are more than one development team working with those services, so SSIS is simplest choice to standardise things. Seen some code and it is better if dudes use SSIS and does not invent new rockets :)
um… If this is a webservice, isn’t all data converted to string anyway, due to the limitations of HTTP ?
Yes, James, but it is all done by frameworks and not by developers who are writing crappy code in hurry. It’s a huge difference. :)