Why public services need better architects...

So I am all for automation. "Rationalise the mundane and develop our capabilities" is one of my main mantras: i.e. stop doing things that don't give us a business edge and focus on the things that do. Removing manual process not only reduces human error, but also frees up people to add real business value.

The UK government is a sprawling mass with tonnes of systems, and a lot of people helping to keep the country moving in the right direction - hopefully forwards! With a large budget deficit and a tax paying public looking for more from their pound, you can understand why there are lot of high budget IT projects looking to streamline and make the enterprise run more efficiently. Ahem to that.

I had an interesting situation recently which showed such an efficiency, but also exposed a serious flaw in the rolled out solution. About 6 months ago I ordered a new car from a Fiat dealership in the UK, whom shall remain nameless. All was well until we went to collect the vehicle, only to discover it had the wrong colour interior... we rejected the car, and after much re-negotiation with the dealer, we ordered a replacement car.

Three months later I collected the new car, and was happy that we had waited to get the right specification. Then I received a fine from the DVLA for non-insurance of a Fiat vehicle. I thought this was odd since I had a policy out on the the car, so I checked the details, and the car in question was the one we'd rejected and never actually owned! Obviously the DVLA have a process in place to try and catch motorists that don't insure their vehicles, which in principle is a great thing to do: correlate across data-sets, and look for exceptions.

Where the process falls down is that the workflow that manages exceptions is a completely closed loop: you cannot speak to anyone at the DVLA about the fine, you can only pay it. The only way to contact them is via good old snail mail... so I write a letter, only to receive what looks like a standard automated response in an even more [legally] threatening tone. In the end it took multiple letters from both myself and the car dealer to state the car was never owned by me and that it remained in-trade. The DVLA dropped the case at the last moment, but it does make me wonder how many people pay the fine just to avoid the stress. Is that an acception of guilt?

It's an interesting one. This was either a use-case that they hadn't thought of, or the guiding principle is assume everyone is guilty because the data can't be wrong. Veracity anyone? It's the latest Big Data dimension! I also feel that there should have been some human break points in the feedback loop. Automation is great, but sometime things need a human's touch. That or a decent machine learning algo ;-)

"I am not a number, I am a free man"

An Englishman in New York...

...or an idiot abroad; i'll let you decide.

Anyhoo, just got back from a whistle stop tour of Toronto and New York, and familiarising myself with the subtle differences between us English speaking nations: "It's the little differences... Do you know what they call a Quarter Pounder with cheese?", "Royale with Cheese!".

Despite the general lack of sleep, overeating and drinking, through excessive coffee consumption I did manage to get quite a lot crammed into my 5 days.

Met some really smart guys from Apprenda, who have a very interesting offerings in the PaaS space. In particular their private PaaS solution addresses a lot of the underlying problems in IT organisations: i.e. from being infrastructure centric. They stitch together the infrastructure into a grid using their peer-to-peer fabric, which creates some interesting options in managing message flow.

Their approach enables development resources to focus on the job at hand: write code, and pushes the configuration and service level management to the platform. One thing I really liked is that they've managed to use containers not only on Linux, but also Windows...

It looks very slick, and for web apps it's a no brainer, but my challenge is that I face is a legacy of mid-range and mainframe apps, and all the [cultural] baggage that comes along with that... Any thoughts as to how to start the transition?

I also spent some time talking to the Hadoop distro vendor MapR to bounce around my IT Operations Analytics concept I'm trying to get some traction on. They also have a really interesting offering, though their marketing has let them down up to now. Basically their approach is not to champion particular products within the Hadoop ecosystem, rather they will support anything on top of their MapR-FS file system.

With MapR-FS they've basically ripped out HDFS and replaced with a file system that addresses some of the key issues in HDFS, yet still support the HDFS APIs:

1. The NameNode is a bottle neck
2. Start times on a NameNode recovery are lengthy
3. Lack of POSIX compliance
4. Supporting legacy UNIX/Linux apps
5. Small files support

You can find out more on their architect at the website, but the inclusion of an NFS server really helps get data into Hadoop, and as we are finding, the sooner we can start storing data the more we have once we work out what we're going to do with it ;-). They also have a tonne of NetApp like snapshot and replication tools, which is no surprise given their CTO is ex-NetApp.

I'm about to start kicking the tyres with MapR, so will report back once I have a bit more experience, but I'm impressed with what I've seen thus far.

Signing off.

Sting, sorry Alex.

My take on strategy, tactics and point solutions

I often hear the term tactical prefixed to a solution that has been created to address an immediate problem. Personally I don't think these are tactics. A tactic is, according to a Google search:

"An action or strategy carefully planned to achieve a specific end"

There is often very little strategy or planning around most so called tactics, so I prefer to call them point-solutions. Tactics should be stepping stones to help realise a vision or goal. They should be well thought out, and always moving the environment forwards. So, in summary:

Strategy = Long term vision and/or goal

Tactics = Shorter term sub-goals

Point Solutions = Quick-fixes with no regard for strategy

I think it's very important to clearly define this terminology and stop using tactical for poor planning and quick-fixes; these don't move the environment towards an end state, more they create further challenges for an organisation.

Can we really store everything forever?

There seems to be a growing expectation within organisations that we store every piece of data forever. If ever I raise retention policies, particularly with legal or compliance teams, the response is that we need to store data forever. Whilst I understand that some data is required to be kept in legal hold, do we need to really store everything forever? And sometimes, there could be a risk in keeping records beyond the date required by the regulators... On the flip side, in the world of Big Data we may not know that a piece of data is valuable yet?

So the challenge is that archived data takes up valuable space in an organisations data centre, and is typically not used for revenue generating purposes. It may allow a business to operate: think regulatory data, but the business doesn't make money from it, or doesn't yet... So, it's a pure bottom line cost, a cost that need to be minimised as much as possible to increase P&L. I like to call this type of data Write-Once-Read-Rarely, or cold storage.

Historically, long term archive meant magnetic tape. Tape is still the highest density storage media, but the challenge with tape is that is degrades, so ensuring data integrity is a challenge, particularly for data with long retention periods. Managing the tape pool becomes a full time job, and remember that this is a non-revenue generating post. Whilst disk is not as dense as tape, data integrity can be provided via software: parity, check-sums, continual checking. A lot of organisations have gone to pure disk storage solutions for archive and backup. One of the other benefits of disk is also faster data retrieval times.

There is another type of storage that can slot into an information life-cycle management (ILM) strategy, and that is cloud storage. The interesting thing with cloud is that the funding model typically changes from CapEx to OpEx, so it's pay-per-use, and it plays to my earlier point about non-revenue generating infrastructure: i.e. outsource it to a utility provider. Obviously there are the usual cloud concerns: security, legal, mobility etc, but if you can put in place compensating controls, the value proposition is compelling.

Not all data types are suitable for storing in the cloud: there may be regulatory or jurisdiction constraints that mandate where data is at rest. So I think we need some form of hybrid cloud storage: a mix of on and off premise storage, where the requirements and constraints dictate where the data is placed as a part of the life-cycle management. The end goal is to ensure the cost back to the business is kept as low as possible, and a company's resources are used for driving up revenue.

Now, how do I sell an Architecture Framework?

So on my path to architectural enlightenment I just passed my TOGAF examinations: Foundation and Certified, with a healthy 79% - quite chuffed with one ;-) I can now display this lovely badge:

So what do I do with it? Now that's the challenge, because my organisation hasn't embraced TOGAF, though I feel there is some value in adapting and adopting. We have a segmented approach to architecture, but I've yet to unravel the strategic architecture - see I am using TOGAF already:-)

I think my approach is to draw on areas within the TOGAF framework and show some value; in particular I like a lot of the techniques presented, and I feel most people could benefit from a more structured approach to their work efforts. As these start gaining some acceptance, I can start into introduce more concepts.

I'm interested to here how others have introduce TOGAF into their organisations? Did you use he ADM to establish a practice, or did you take another route? How did you sell the value, thus enabling you to establish a practice?

Answers on a postcard please.

CloudExpo London 2014

I went to CloudExpo Europe at ExCel in London yesterday. I only had a few hours due to needing to attend a meeting in the afternoon. I managed to traverse the floor a couple of times, pick up a load of handouts, and attend a few keynotes.

One theme that was very apparent in the infrastructure space is the ongoing disintegration of hardware and OS; by this I mean the un-bundling of a single vendor owning both: Sun/SunOS, Cisco/IOS etc. We've seen this with servers: Linux and commodity x86 servers, and we're now seeing this with both storage and networking. Take a look at Cumulus Networks as an example; if I were Cisco I would be very scared, particularly as Dell are now at the table in deals they historically haven't been.

What struck me is that we now have the potential install software like Puppet and Chef on every piece of infrastructure in the data center, and orchestrate the complete environment from one platform, and then seamlessly integrate this into the Application Life-cycle Management process. I heard a great term the other day on a DevOps Cafe podcast - "DevOpsability". I like that ;-) We're getting to a point that we can easily automate the day-to-day management and provisioning of our infrastructure - it is DevOpsable!

Interesting times ahead...

Why I don't care about server vendors any more

I come from a background of UNIX; from SunOS, Irix, Solaris, Linux and MacOS. And yes, MacOS is a UNIX for those that have never opened a terminal... Shame on you;-). Actually well done Steve, you took NextStep and brought it to the masses. Kudos.

My old favourite UNIX workstation

I used to love my hardware, particularly the purple Sun Sparc boxes, and the more colourful SGI indigo ones - loved those Indys BTW. But these days I'm not so bothered; even my much loved Macintosh doesn't really do anything for me, though part of that is probably down to me no longer rooting for an underdog: I love supporting the little guy, and love even more an obscure brand. Apple is just too big for me now, but I wish them well.

So why is this? Well the hardware is so commoditised these days that the intelligence is all in the software wrapped around it; it's this that gets me excited: auto provisioning for integration testing, nice! Burst capacity into the Cloud, smart! Deploying an app onto any portable device, wicked! This is the stuff we, as a business enablers, should be focused on. Not the colour and brand of the physical tin it all runs on.

I had an interesting dialogue with a colleague the other day about the pros of a multi vendor x86 strategy: leverage etc, but I still see the silos run deep, and the badge allegiance high. White label the lot, drive down the price, and focus on managing the service back to the business. This is where the role of the architect comes in to sell the vision and get stakeholder buy in.

It's all about business differentiation. Our focus should be on the areas within IT that can give our business partners an edge; managing commodity hardware platforms just don't provide that.

Don't mention the "C" word!

I have recently been involved with an organisation that can't say the "C" word: "Cloud". There I said it. Now, that wasn't so bad was it?

So what's the problem? I'll tell you what: organisations, particularly large ones, still don't know what the Cloud means to them... So rather than say it, they skirt around it: "Elastic Infrastructure", "On-demand Provisioning", "Capacity driven", and so on. All the while this further confuses the client as to what Cloud is, and what the organisation's strategy is to move towards all that Cloud goodness, whatever that may be. It's a funny old situation, and I'm wondering if it's unique. Are we all still struggling to define the Cloud? Should we just accept it for what it is, which is whatever you want it to be?

So what is it to me? In a word: differentiation, and in two: business differentiation. So what do I mean by this? In a nutshell it's about creating business opportunity by freeing up resources, and empowering those resources to be innovative. Innovation comes when you have a culture that enables people to not only fail fast, but to learn fast. This learning drives the innovation. Cloud technology allows us to do this in a cost effective, and timely manner. Its really that simple.

Stop worrying about saying the "C" word; just work or what your business goals and challenges are, and then see how Cloud can help you realise them.

Visualisation - a British spelling

An old colleague of mine has a great blog on BI (Business Intelligence) and it's application through a tool called Tableau. It was from reading Paul's blog that made me think I should start outputting some of my architecture ramblings; the ones I don't have the chance to during the working day, or the ones nobody wants to listen to;-)

In my head I see the visualisation piece as being the icing on the top of the cake: there is a whole data architecture that needs to underpin this, within the IT environment it is part of the ITOA movement I mentioned in a previous post.

That being said, Tableau is both a very powerful tool, and more importantly a very easy tool to get up and running to gain some valuable business insights.  Empowering a user base is one thing I look for in a platform, since it avoids service bottlenecks: i.e. a team of staff managing requests. Power to the people is what I say!

Anyhoo, recommend checking Paul's blog out for all your Tableau visualisation needs.

Going the ExtraHop

ITOA (IT Operations Analytics) is going mainstream big style in 2014. This is an area I am particularly interested in from an instrumentation, aggregation, analytics and visualisation perspective. This is an area IT folks can really add some business differentiation, but there are a number of hurdles to overcome: breaking down the silos, the usual road to DevOps enlightenment.

I had an interesting update from ExtraHop today. They have slick on-the-wire instrumentation product that does packet header analysis and correlation in real-time. They're also able to offload to Splunk to take advantage of Splunk's indexing capabilities; one nice result of this means you can use less Splunk license given you're targeting the data that's indexed using triggers (don't tell Splunk's sales people;-).

Hello world

This is the start of a little blog for me to use as an outlet for all things technology and architecture. Why undercover? Well I'm more a behind the scenes kinda guy - I like success through others. In hoping that though this blog I can sync up with other like minded technology architects working in the field.