The Rise of the Private Data Cloud

Corporations today face several issues in their technical architecture. Applications can be deployed on-premises or in one or more public clouds like Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP). Few have completely migrated to a single platform. Moreover, legislation in some countries can mean that users need to have control over the physical location of their data, which may be difficult (or at least expensive)  in the case of public cloud providers. While the big cloud providers invest heavily in security and fallback capability, there have been publicly noted security incidents with all the major platforms, as well as occasional outages that have affected many companies. Concerns about reliability, security and data sovereignty have meant that some organisations, especially in heavily regulated sectors or amongst government agencies, have switched some of their applications to private clouds, which have no other tenants sharing the cloud servers, where hardware can be optimised and where there are no surprising bills at the end of the month, a common complaint amongst customers regarding public cloud providers. Customers take on much of the burden of maintenance of the private cloud environment, but that may be a price worth paying in many cases. Another issue is around ownership and security of data – to what degree are you comfortable in delegating control of your data to a third-party provider?

In the case of the public clouds, customers face a choice of whether to use the database supplied by the cloud provider: Redshift for Amazon, or Synapse Analytics for Azure or BigQuery for GCP. Alternatively, they may choose to use an independent database like Snowflake, Teradata or Yellowbrick. The cloud provider databases are typically heavily optimised for their particular cloud platform, so there is portability to consider: do you want a single database that can be deployed across platforms, or multiple databases, each optimised to a particular platform? All databases are not created equal, and while the leading products are all capable, each has limitations. For example, Redshift does not enforce key uniqueness so duplicate data records can be created. There is no ability to add additional indexes to improve the performance of tricky or commonly used queries, and free space from deletions is not reclaimed automatically in Redshift. Other products also have their own limitations and issues.

Each alternative database will have pros and cons. Some will have better performance and scalability for certain use cases, and some are more intuitive to deploy and manage. Indeed, there may be a trade-off between ease of use and ability to tune and customise. Each customer needs to decide what criteria are important for their particular situation, prioritise these and carefully evaluate the alternative database products against these weighted criteria. Beyond functional criteria, they need to consider strategic issues such as the pros and cons of choosing a database that can be ported between platforms easily compared to the additional skills burden (and possible switching costs) of choosing a particular database for each platform. There is the broader choice of on-premises , one or more public clouds or a private cloud, or a hybrid approach. This choice will itself mean further decisions about the database to be used in each environment. As migration to the cloud steadily continues, organisations need to make strategic decisions about their mix of platforms and databases, bearing in mind that there has been a rapid pace of change and choice selection in recent years. A carefully considered technology architecture should aim to preserve flexibility and allow companies to adapt to future alternatives.