Can we really store everything forever?

There seems to be a growing expectation within organisations that we store every piece of data forever. If ever I raise retention policies, particularly with legal or compliance teams, the response is that we need to store data forever. Whilst I understand that some data is required to be kept in legal hold, do we need to really store everything forever? And sometimes, there could be a risk in keeping records beyond the date required by the regulators... On the flip side, in the world of Big Data we may not know that a piece of data is valuable yet?

So the challenge is that archived data takes up valuable space in an organisations data centre, and is typically not used for revenue generating purposes. It may allow a business to operate: think regulatory data, but the business doesn't make money from it, or doesn't yet... So, it's a pure bottom line cost, a cost that need to be minimised as much as possible to increase P&L. I like to call this type of data Write-Once-Read-Rarely, or cold storage.

Historically, long term archive meant magnetic tape. Tape is still the highest density storage media, but the challenge with tape is that is degrades, so ensuring data integrity is a challenge, particularly for data with long retention periods. Managing the tape pool becomes a full time job, and remember that this is a non-revenue generating post. Whilst disk is not as dense as tape, data integrity can be provided via software: parity, check-sums, continual checking. A lot of organisations have gone to pure disk storage solutions for archive and backup. One of the other benefits of disk is also faster data retrieval times.

There is another type of storage that can slot into an information life-cycle management (ILM) strategy, and that is cloud storage. The interesting thing with cloud is that the funding model typically changes from CapEx to OpEx, so it's pay-per-use, and it plays to my earlier point about non-revenue generating infrastructure: i.e. outsource it to a utility provider. Obviously there are the usual cloud concerns: security, legal, mobility etc, but if you can put in place compensating controls, the value proposition is compelling.

Not all data types are suitable for storing in the cloud: there may be regulatory or jurisdiction constraints that mandate where data is at rest. So I think we need some form of hybrid cloud storage: a mix of on and off premise storage, where the requirements and constraints dictate where the data is placed as a part of the life-cycle management. The end goal is to ensure the cost back to the business is kept as low as possible, and a company's resources are used for driving up revenue.

Now, how do I sell an Architecture Framework?

So on my path to architectural enlightenment I just passed my TOGAF examinations: Foundation and Certified, with a healthy 79% - quite chuffed with one ;-) I can now display this lovely badge:

So what do I do with it? Now that's the challenge, because my organisation hasn't embraced TOGAF, though I feel there is some value in adapting and adopting. We have a segmented approach to architecture, but I've yet to unravel the strategic architecture - see I am using TOGAF already:-)

I think my approach is to draw on areas within the TOGAF framework and show some value; in particular I like a lot of the techniques presented, and I feel most people could benefit from a more structured approach to their work efforts. As these start gaining some acceptance, I can start into introduce more concepts.

I'm interested to here how others have introduce TOGAF into their organisations? Did you use he ADM to establish a practice, or did you take another route? How did you sell the value, thus enabling you to establish a practice?

Answers on a postcard please.

CloudExpo London 2014

I went to CloudExpo Europe at ExCel in London yesterday. I only had a few hours due to needing to attend a meeting in the afternoon. I managed to traverse the floor a couple of times, pick up a load of handouts, and attend a few keynotes.

One theme that was very apparent in the infrastructure space is the ongoing disintegration of hardware and OS; by this I mean the un-bundling of a single vendor owning both: Sun/SunOS, Cisco/IOS etc. We've seen this with servers: Linux and commodity x86 servers, and we're now seeing this with both storage and networking. Take a look at Cumulus Networks as an example; if I were Cisco I would be very scared, particularly as Dell are now at the table in deals they historically haven't been.

What struck me is that we now have the potential install software like Puppet and Chef on every piece of infrastructure in the data center, and orchestrate the complete environment from one platform, and then seamlessly integrate this into the Application Life-cycle Management process. I heard a great term the other day on a DevOps Cafe podcast - "DevOpsability". I like that ;-) We're getting to a point that we can easily automate the day-to-day management and provisioning of our infrastructure - it is DevOpsable!

Interesting times ahead...

Why I don't care about server vendors any more

I come from a background of UNIX; from SunOS, Irix, Solaris, Linux and MacOS. And yes, MacOS is a UNIX for those that have never opened a terminal... Shame on you;-). Actually well done Steve, you took NextStep and brought it to the masses. Kudos.

My old favourite UNIX workstation

I used to love my hardware, particularly the purple Sun Sparc boxes, and the more colourful SGI indigo ones - loved those Indys BTW. But these days I'm not so bothered; even my much loved Macintosh doesn't really do anything for me, though part of that is probably down to me no longer rooting for an underdog: I love supporting the little guy, and love even more an obscure brand. Apple is just too big for me now, but I wish them well.

So why is this? Well the hardware is so commoditised these days that the intelligence is all in the software wrapped around it; it's this that gets me excited: auto provisioning for integration testing, nice! Burst capacity into the Cloud, smart! Deploying an app onto any portable device, wicked! This is the stuff we, as a business enablers, should be focused on. Not the colour and brand of the physical tin it all runs on.

I had an interesting dialogue with a colleague the other day about the pros of a multi vendor x86 strategy: leverage etc, but I still see the silos run deep, and the badge allegiance high. White label the lot, drive down the price, and focus on managing the service back to the business. This is where the role of the architect comes in to sell the vision and get stakeholder buy in.

It's all about business differentiation. Our focus should be on the areas within IT that can give our business partners an edge; managing commodity hardware platforms just don't provide that.

Don't mention the "C" word!

I have recently been involved with an organisation that can't say the "C" word: "Cloud". There I said it. Now, that wasn't so bad was it?

So what's the problem? I'll tell you what: organisations, particularly large ones, still don't know what the Cloud means to them... So rather than say it, they skirt around it: "Elastic Infrastructure", "On-demand Provisioning", "Capacity driven", and so on. All the while this further confuses the client as to what Cloud is, and what the organisation's strategy is to move towards all that Cloud goodness, whatever that may be. It's a funny old situation, and I'm wondering if it's unique. Are we all still struggling to define the Cloud? Should we just accept it for what it is, which is whatever you want it to be?

So what is it to me? In a word: differentiation, and in two: business differentiation. So what do I mean by this? In a nutshell it's about creating business opportunity by freeing up resources, and empowering those resources to be innovative. Innovation comes when you have a culture that enables people to not only fail fast, but to learn fast. This learning drives the innovation. Cloud technology allows us to do this in a cost effective, and timely manner. Its really that simple.

Stop worrying about saying the "C" word; just work or what your business goals and challenges are, and then see how Cloud can help you realise them.

Visualisation - a British spelling

An old colleague of mine has a great blog on BI (Business Intelligence) and it's application through a tool called Tableau. It was from reading Paul's blog that made me think I should start outputting some of my architecture ramblings; the ones I don't have the chance to during the working day, or the ones nobody wants to listen to;-)

In my head I see the visualisation piece as being the icing on the top of the cake: there is a whole data architecture that needs to underpin this, within the IT environment it is part of the ITOA movement I mentioned in a previous post.

That being said, Tableau is both a very powerful tool, and more importantly a very easy tool to get up and running to gain some valuable business insights.  Empowering a user base is one thing I look for in a platform, since it avoids service bottlenecks: i.e. a team of staff managing requests. Power to the people is what I say!

Anyhoo, recommend checking Paul's blog out for all your Tableau visualisation needs.

Going the ExtraHop

ITOA (IT Operations Analytics) is going mainstream big style in 2014. This is an area I am particularly interested in from an instrumentation, aggregation, analytics and visualisation perspective. This is an area IT folks can really add some business differentiation, but there are a number of hurdles to overcome: breaking down the silos, the usual road to DevOps enlightenment.

I had an interesting update from ExtraHop today. They have slick on-the-wire instrumentation product that does packet header analysis and correlation in real-time. They're also able to offload to Splunk to take advantage of Splunk's indexing capabilities; one nice result of this means you can use less Splunk license given you're targeting the data that's indexed using triggers (don't tell Splunk's sales people;-).