Dude! Not ”those” kind of objects. 

Was at SNW last week in Dallas and had the opportunity to lead a session on object storage and describe the technology’s value from archive to cloud storage.  This is a new and emerging category in the storage industry that I believe opens new realms of possibility, not just in terms of storing stuff, but for overall information management.  There will be more to come on the subject now that I’m pretty much settled in my new gig.

The objective I’ve had with this blog is to post concepts that build upon the last and provide a context around archiving, information management and a different way of looking at things.  Things will continue in this fashion and just wanted to provide some insight into where we’ll be heading.  This is the presentation I delivered at SNW.  What’s missing is the dialog that went with it and that will be the subject of upcoming posts you’ll see here. 

Check back when you get a chance.

One of the objectives of my blog is to provide a foundation for object storage and archiving to help educate folks of the benefits of the technology.  It’s an area that isn’t well understood just yet and I expect even greater traction and visibility given Dell’s new entrant, DX6000 Object Storage Platform, into this emerging segment.  I’m excited about driving product and market strategy for it and helping to build the market. 

Stay tuned…

I’ve been talking about the importance of metadata to storage since the late 90′s and back then the “veterans” looked at me with crossed eyes.  It probably wasn’t so much that they didn’t totally get it, but that it was well outside the view people had of storage at the time, i.e. files, file systems and blocks.  However, I saw it as a way for storage vendors to increase their value to customers by adding informational value to the data they stored on disk and tape.  It was also based on a challenge I had when running several IT groups on the customer side for effectively managing an ever-increasing amount of file data coming from my end users in the mid-90s.

The issue was that my file storage was growing especially in user home directories and group shares.  Trying to prune these down to the most essential data was impossible.  There was no good way for my IT people to know what was important and what wasn’t just by looking at the file and folder tree.  There was no context for it other than who created and how “fresh” it was, i.e. created or accessed recently.  It was just a bunch of stuff (call back to my 1st and 2nd posts).  So when I jumped into the vendor world and was given the direction to simply ”build software that added value to our storage hardware,” it presented the opportunity to attack this growing problem.  Surely, I wasn’t the only customer who faced this issue.

Metadata was the one common thread I discovered when handed software projects in two different industries, health care and financial services.  Both needed to archive content for long periods and be sure it could be found and accessed in the future.  The context a richer set of metadata describing file assets (unstructured data) could provide was the key and what I would have loved to have back in my IT days.  It allowed the addition of informational value to data beyond what the file system provided that would enable search, preservation and policy-based management.

When a search is conducted through an ECM application, e-Discovery tool or using the well-known index-search engine (Google, Autonomy, etc.), it is done based on the metadata within the application.  The differences in the tools lies somewhat in the depth or richness of the metadata they use.  ECM apps typically use a database and index-search engines have optimized an index of full-text for advanced queries, e.g. proximity, natural language, semantic networks, etc.  For management purposes metadata needs to have some structure to it so file assets can be “tagged” appropriately and there are a number of metadata schemas already defined.

Source: Video Content Management in Broadcast, SMPTE Journal, February/March 2002

One method I have used to describe metadata and its depth is in a hierarchy model I developed early on and was published in the SMPTE Journal back in 2002.  There are just three levels to keep it simple, but useful. 

Base Level metadata is what you get in a traditional file system, which has almost no context (file name, date created, owner, size, last accessed, etc.)

Structured Metadata is based on a standard schema typically in name-value pairs.  There are a number already defined including Dublin Core, NBII Biological Metadata, Content Specification for Geospatial Metadata, DICOM, SMPTE, Video on Demand Content Specification, etc.  The structure provided by metadata standards enables simple rules for management such as if <tag> = blah, then replicate to clusterX.  It also enables more complex rules for managing and manipulating content such as transcoding specific MPEG files to streaming format, or transforming certain Word docs to PDFA for long-term archive.  It also provides a mechanism for creating and persisting complex relationships between content objects.

Unstructured Metadata is full-text indexing that most search engines perform.  These are indexing primary/unique words throughout a document and is not stored in a structured index.  The challenge in storage is that these indexes can be as much as 80% or more of the size of the original data.  It can be tough to sell the value of that much additional capacity.  However, full-text index/search is often provided as a critical tool on internal corporate portals enabling enterprise-wide discovery, but not as part of the storage infrastructure.

Object storage delivers a metadata capability that had previously been unavailable to application developers and end users.  The ability to add standard-based metadata and custom metadata as descriptors for files opens a whole new opportunity for information management and storage.  There remains a fair amount of work ahead of us because the traditional file system approach has become so ingrained in our perspective.  However, the massive growth of new content (unstructured data, files) and the tremendous storage capacity it will consume is necessitating a change.  Managing the amount of information foreseen requires applications and infrastructure to work more cohesively together and metadata is a common thread that extends from the app through storage.

The Web has delivered ubiquitous access to all of this content whether from the office, the home or the road.  Content can be considered the dial tone of the Web.  That’s how folks have come to see it.  Seeing HTML pages on the Web is how people know it’s working, and those pages themselves are content.  The associated images, Flash animation and text are just part of the multi-media content experience we’ve all come to know and love about the Web.  The majority of the content  is unstructured data or file-based data with greater amounts of it being available for consumption.  It’s not just on the Internet, but includes intranets as well that leverage Web-based technology for accessing, storing and sharing content, file-based data.

The challenge with content is that defining it has proven a challenge for the most part.  However, one definition I ran across during my time in the media & entertainment space delivering video content management software has proven useful.

Content = Metadata + Essence

Essence can be thought of as file data, such as a video, photo or document, and the metadata is descriptive information about the essence like producer, subject, file name, aspect ratio, date created, DRM, etc.   It was recognized that the metadata was often as important as the essence, if not moreso.  The reason being that the essence was useless if it could not be found and retrieved.  Metadata serves as searchable information allowing files to be located and enabling greater capability for intelligent management.

In this content driven world has come the need for a technology designed to handle file-based data and a rich set of metadata.  The file system is proving too limited in its capabilities and object-based storage has entered as an ideal solution to the growing challenge of content storage at massive scale.   The relationship of content to objects is simple:

Object = Metadata + File data

Therefore, content and objects can be used interchangeaby and illustrates one of the characteristics that makes object-based storage ideal for the Web and unstructured data in general.  Stay tuned.  There’s lots more coming on metadata, objects and storage…

I’ve spent a lot of time researching and thinking about this since my current employer delivers enabling technology for the storage cloud.  The “cloud” really seems to be about ubiquitous access and connectivity to a set of networked resources (public/private), which seamlessly scale based on demand for distribution of information.  It’s not an individual product or technology, but a number of interconnected technologies that delivers a complete system.  When I ran across the Wikipedia entry that says the term cloud is a metaphor for the Internet, it got me thinking.

The tech biz talks about cloud, streams and pools, as in cloud computing, streaming data and storage pools.   What all these terms have in common is a relationship to liquid or fluidity of a system that includes all of these components.  So then, perhaps the water cycle or hydrologic cycle is a better metaphor to use as it encompases a complete system.  Water evaporates from the ocean to become vapor, which accumulates as clouds that move freely around the world.  The clouds release the water as rain or snow.  Runoff from rain is carried by streams or rivers and deposited in freshwater storage such as lakes or pools and snow is stored in solid state in glaciers and snow caps.  Much of the water makes its way back to the ocean allowing the cycle to remain in constant motion.

Replace water in this metaphor with information and these two systems have very similar attributes.   For water the cloud is an access and distribution mechanism delivering preciptation all over the world.  Rivers and streams carry large capacities of water like optical fiber carries lots of information packets and bits.  Lakes and glaciers are similar storage pools for information.  The ocean is the largest store for water and the main resource for cloud creation.  One could think of a cloud as providing the most ubiquitous movement of water from the ocean to feed the glaciers, rivers and lakes just like the Web provides access to an ocean of information (content) on the Internet to feed corporate networks and consumers alike.  One could also look at the rivers and lakes as individual corporate networks that connect to the larger cloud and ocean (public network).

It’s not that I don’t think the cloud has relevance and value, but that I believe there is a broader context in which its value should be considered.   As a metaphor the water cycle isn’t a complete corelation, but it does illustrate a near perfect system that flows effortlessly.  In order to achieve a state of balance between business, information and technology it is necessary to take a holistic approach to determine how best to achieve objectives. 

The point: Before making a decision on how the cloud fits your needs, consider how it aligns with your business.  Also, it’s to realize that you need to get beyond the hype or buzz word du jour and evaluate the underlying technology to understand the benefits it can deliver within your own information ecosystem.

Example Web Map

If you think about it, the World Wide Web is simply a massive storage repository that provides ubiquitous access to content over the Internet using HTTP.  The Web browser is the primary user interface, which presents HTML pages containing hyperlinks to other HTML pages and specific content types including text, documents, images, video, etc.  The hyperlink associated with a URL (uniform resource locator), enables the user to navigate to and access another piece of content. 

There’s a lot of flexibility to name the resource/content within an HTML page such as:

“…there is an interesting item you should read to learn more about the subject.” 

The resource name or label is “interesting item”, which could be a PDF document, an image or another page.  The hyperlink’s URL is the address to access the file and below is the full URL for “interesting information.”

http://www.caringo.com/downloads/whitepapers/caringo_whitepaper_newmillennium.pdf

It is similar to the paradigm of the file system where you assign a file name, MyFile.doc, but store it in a specific location.   In order to access the file you need to know the path to get to it.

C:\MyFolder\MySubFolder’\MyFile.doc

Instead of having to traverse a folder\sub-folder hierarchy to open a particular file, one simply clicks on a hyperlink to retrieve content when surfing the Web.  In this scenario the Web is like an infinitely vast, flat address space where all content, as perceived by the end user, exists at the same level (see Web Map image above).  Sure, on the Web server itself the resource may be stored in a file system hierarchy, but that has no impact on the individual executing a hyperlink.  The Web repository is huge and grows transparently to meet demand as it comes.

What is intriguing about this is how it relates to the buzz-word du jour, “cloud storage.”  While a formal definition of cloud storage remains elusive, there does appear to be agreement that it enables access to storage resources via Internet protocol (public/private) that it is resilient, affordable, massively scalable and simple to use/manage.  As a concept, it makes great sense, but where the rubber meets the road is determining the underlying technology that’s going to deliver on the promise.  

Applications of all types have moved to embrace the Web to simplify their offerings and why shouldn’t storage.  What we’re seeing today is the emergence of object-based storage with an HTTP, REST and/or SOAP interface that behaves somewhat like the Web itself.  It is also the technology best suited to provide the underlying infrastructure for cloud storage.  Several vendors big and small have introduced products/technologies into this new category of storage. 

As an example, a file can be named or labeled, and stored through a content management system or file system.  When ingested to object storage the file name becomes a part of the object metadata and stored with the file data.  The address for retrieving that file is simply a URL linked to the file name or label for access and never changes.

One method is to link the specific file name MyFile.jpg or you can create a label as a hyperlink in a web page such as “Follow this link to my file that you’ll find quite interesting.”  Below is the URL for both of these links and the content is accessed on a storage cluster directly connected to the Web.  It’s simply raw object storage available on the Internet or maybe more appropriately, Web storage.

http://cas.caringo.com/760C9DFE1E6F5CE01AC75F715B9731CD

The cluster: standard x86 servers, massively scalable, highly resilient, self-healing, self-managing, with automated intelligent content distribution. 

Sounds to me like all you need for “cloud storage” infrastructure.

By its very nature an archive is active.  It is a living, breathing entity within an organization.  It grows, it evolves and is (or should be) a source of knowledge for future generations.  Archive is also not simply one technology or product.  It is a process comprised of human and computing systems working together to effectively manage an information lifecycle.  Yes, I said it, ILM (a little backwards though).  This is how I described the archive in developing the foundation strategy for a new solutions business unit at one of the major storage vendors many years ago (1999).  Traditional archivists themselves define their role similarly and have been using the term information lifecycle management long before it became a buzz word in the technology industry.

Back in those days archive was perceived as a static storage location that was rarely, if ever accessed, and only then in cases of emergency.  It was (sometimes still is) confused with backup or hierarchical storage management (HSM), but it is so much more and includes these components. Getting people to understand archive as a comprehensive approach to information management was a challenge, and it’s still not understood this way.  However, I believe the winds of change are blowing as we see more activity and initiatives focused on enterprise information management.  It takes a top-down approach to understand the type of information being produced within an organization and make a determination as to the importance and value of that information to develop an effective management process.

Industries that view their digital information (data) as assets fully recognize the archive’s value as an active part of their business.  Companies in health care, pharmaceutical, media & entertainment, and oil & gas understand that information created today can provide future benefits.  Research conducted during drug discovery ten years ago can have significant impact on current drug development today.  Video captured in the past can be re-purposed for new productions being developed at a broadcast company.  In these instances the archive is a critical component to business success or at least level of success.  Are traditional organizations outside these industries so different?  Is the intellectual property and knowledge contained in unstructured data (files) any less of a business asset?  No, not really.

An effective information management process will directly lead to a quality archive and vice versa as they are inextricably linked together.  Both require a top-down approach to define the information landscape starting at the point of creation and how each piece of information shall be described (metadata).  Metadata is the key to the kingdom for information management and archiving in the digital world.  It enables automation and intelligence to be applied to an individual information asset or set of assets.  It simplifies the process of finding and retrieving assets regardless of how long they’ve been stored.  (I’ll expand on the concept and value of metadata in relation to storage in a future post.)

The point is that archive is an active process, which requires a number of technologies seamlessly working together.  Storage vendors introduced the concept of archive to the digital world, but were more focused on one of the roles of an archive or archivist:

• Preserve information and evidence in a protective environment and in a format or media that will remain useable over time.

The above is more of the noun definition of archive.  The other role defined by the Society of American Archivists (SAA) makes the archive actionable.

• Provide descriptive tools such as registers, indices and databases to allow others to locate and identify information and evidence in archival holdings.

 The descriptive tools are very much dependent on metadata derived from a variety of sources where the elements are defined based on the specific needs and industry of an organization.  Applications are critical to an effective archive and ILM strategy in this area.  While some storage vendors are working to vertically integrate to include more of this functionality, a complete archive solution requires more cohesive interoperability between infrastructure and applications. 

Suffice to say that archive is more complex than just storing stuff for a long time and the better organizations get at managing information as a holistic process, from cradle to grave, the more effective they will be.

It’s often been a challenge to explain to “laymen” (folks not in the storage/tech biz) what I do.  I mean, there are a lot of people not in the industry and their experience with storage is based on the hard drive capacity of their PC.  When I talk about Petabytes, content/objects or storage clusters their eyes glaze over.  Yeah, like some of yours might have just done.  

So, when recent dinner conversation turned in this direction and a physician was asking me about my job, I took a little different tact.  The story started off by describing “digital stuff”.  You know, all that stuff one creates on their home system like e-mail, photos, audio, video, documents, graphics.  Stuff.  Digital stuff.  All this digital stuff needs to be stored somewhere.  That’s why you have a hard drive.  And when you collect too much stuff you need to get a bigger drive.  Now this was resonating with my audience.  Even the part about trying to find stuff in the big circular file cabinet, hard disk, hiding in some folder or sub-folder.

Since I wasn’t getting the deer-in-headlights expression, I pressed on.  Today, people can take some of their digital stuff on the road, in their laptop.  However, a lot of their stuff may be stored on an external drive at home, say a 1TB drive, so you can’t take all your stuff.  Just some of it.  Enter, social networking and cloud storage where you can store some of your stuff to the Web and access it from anywhere.  Of course, you don’t put all your digital stuff out there either.  The important thing is: you can still get to it, e.g. FaceBook, MySpace, etc.  I was still getting knowing nods from my dinner companions and I knew I was onto something.

It struck me a few days later that this concept wasn’t anything new.  I hadn’t suddenly hit on a ground breaking and simple explanation of storage and the problem of storing more digital stuff.  I was simply channeling George Carlin (RIP).  It was one of the first and funniest routines I saw him do and is so appropriate for drawing a real world analogy to the digital realm.  If you haven’t seen it before or if you want to re-live his brilliance again you can read the routine here or watch it here.  I recommend watching it, if for no other reason than taking a much needed break from the stresses of the day.

Stay tuned.  Maybe I’ll post something on the “seven dirty words” in storage.  Have to think on that one a bit.

As the PC appeared on the desktop, the amount of information being kept became invisible for the most part.  The circular file cabinet became the hard disk drive, which made it easier to store more stuff without the physical clutter.  However, it has also created a problem that still needs to be resolved as it relates to knowing what’s being stored, its importance and whether it even needs to be retained.  Some would say that the disk file system has replaced the physical filing systems and made managing files easier.  However, the traditional file system is rather opaque.  It provides a static, two-dimensional view of the data on disk without any context that is needed to effectively manage information. 

In the early days when directory names were limited to eight characters and file names to eight characters and a three-character extension we had to employ significant position coding schemes to provide some information about the file.  Then we got long file names that people thought would help enable more meaningful file naming, thus better management.  What we’ve wound up with is a folder, sub-folder, file name structure where often the only context for the information exists with the user that created it and the folder hierarchy.  After a short time even the end user can start to lose context for the information on their own hard drive especially if they’re not actively accessing the files. 

Now it really gets complex when you get to the data center providing shared storage for hundreds or thousands of end users.  The storage administrators are simply managing devices and have little, if any, context for the information contained in files or its business value.  File systems with thousands of folders/sub-folders and tens of thousands of files just look like a bunch of stuff, clutter.  As the volume of file-based data continues to explode and capacities to store this information grow to Petabytes, there is a major need to enable more descriptive information (metadata), context, to be stored with each file.  It’s not that file systems don’t have a place in the data center, it’s that as the repositories grow and information is retained for longer periods another dimension is required to enable more intelligent data management. 

The information contained in that circular file cabinet (spinning disk) is what’s most important and the ability to persist context with it will vastly improve management for the near and long-term.  Object-based storage is emerging as a new category of storage technology that expands the capacity of metadata and adds an intelligent management dimension to storage based on content.  Its features and functionality make it the right choice for archiving and cloud storage infrastructure.