Streamlining Enterprise Content Management with Amazon S3

• Content (Documents) Store
• Metadata Store
• Search Service

Historically, the first implementations tried to minimize complexity by using only RDBMS (Relational Database Management System) for all three purposes.

In that case, “content” is stored in some kind of BLOB (Binary Large Object) column and content metadata as fields in custom-tailored database tables. Search services are implemented as SQL queries on “metadata” columns. Although simple, that approach soon became “too simple.” In the enterprise context, storing many documents in RDBMS burdens the database too much.

A large number of different “content types” usually required different, complex database schemas for transacting (ingesting) and, at the same time, searching for different “content types.” The next generation of ECMs has tried to overcome those problems by “ditching out” content from DBs into a “file system” and keeping only the pointer (“id”) of the content in the DB.

Similarly, for scaling the “search,” Lucene-based solutions like SOLR and Elastic Search become standard practice instead of direct searches on the DB tables. Some main drivers of using ECMs, like granular security over the content and content versioning, are realized by hiding content (files) behind provider-specific APIs that “simulate” required features that file systems do not provide out-of-the-box.

In that scenario, RDBMS is often used to store additional tables (access rights tables, document versions tables, etc.).

Cloud providers have realized from the beginning that the requirements for storing BLOBs differ significantly from those of traditional file systems. A canonical example is Amazon S3 object storage.

Object storage systems have different characteristics compared to file systems. They should be seen as “key-value stores” where the value is some BLOB (Binary Large OBject) (content in the context of the ECM).

Such systems typically do not support file system features like random writes and hierarchical organization (directories). When content changes, we rewrite the whole object.

It is essential to notice that file-system features are typically not required in the ECM context, where we need versioning instead of editing, and requirements to organize content (files) in a hierarchical structure are rather a constraint than a feature in that context.

Relieved from “file system constraints,” object stores are typically more performant, especially in cloud/distributed systems.

For example, traditional ECM systems invent some artificial hierarchy, e.g. (year/month/date/hour…J) to avoid problems accessing directories with too many files. Additionally, they simulate file versions with different files “linked” with some custom application logic. Some standard features of “object storages,” like immutability, retention schedules, etc., are typical additional custom applications/modules (usually under the “Record Management” banner).

With the emergence of the “Cloud,” object storage became widely used.

Amazon S3 quickly became an industry standard for object storage in the cloud. In this post, we will not discuss the benefits of storing content in the cloud; many such analyses are readily available. Here, we are interested in using such storage in the “next generation” of ECM systems.

Why do we call S3 a standard? Although there is no official standard/body/process for Amazon S3 object storage currently, there is little doubt that S3 is the main player in this area.

For example, Gartner, in the “Magic Quadrant for Distributed File Systems and Object Storage,” under the “Standard capabilities,” categorizes “RESTful HTTP APIs such as Amazon S3.”

It is essential to mention what is “standard” in that “requirement” – “RESTful HTTP API”!

Remember that traditional ECM systems have proprietary APIs for accessing content. Transitioning between ECM vendors usually requires custom projects and expert knowledge of both platforms (export/import).

“File over app is a philosophy: if you want to create digital artifacts that last, they must be files you can control, in formats that are easy to retrieve and read. Use tools that give you this freedom. “

—Steph Ango

It is important to know that customer content in traditional ECM systems is locked under the provider’s application-specific API or, worse, the provider’s ECM application.

In S3-compatible systems, you only need the URI of your content and access permissions.

The standard REST API call returns content metadata (standard and custom(er) specific). Such URI (Uniform Resource Identifier) is based on the single “key” property of your required content (Object storage is a “key-value” store), so you do not need to know the location in the directory tree. If you use versioning, you extend the URL with the version number, simple as that.

If we step back for a minute to see a bigger picture, by only storing the content in the S3 compatible storage, we have a large chunk of ECM functionalities “out of the box”, and we do not need a box if we do not want to ;):

• “standard” REST API – your content is accessible even with the web browser only
• standard and custom properties “attached” to the content
• rich security model
• versioning
• retention/disposition schedules
• cloud-ready