Marching data to cloud

Walking is controlled falling

Last year I saw an IMAX Nat-Geo documentary in the Liberty Science Center with my kid regarding robots, It showed how complex a robot is & how much code goes behind it just to make it walk. ‘Walking is controlled falling’ they said and told us that in future the robots will learn to do complex things and share their knowledge with each other over cloud…..some time back I saw another from Boston Dynamics… in 2 years robots have come from this

To this

In the last 2 years fumbling humanoids ( and dogoids) have evolved to do what is difficult for many humans to do ( almost impossible for me). Imagine the code AND data which goes behind the split-second decisions which the machine calculates regarding firing its springs and oscillators, finding the right balance between its various mechanical parts. This can only be possible when the machine learns over time how to use the data, having access to the right data is crucial here.

Data collaboration and availability are the pains which need to be alleviated to make new things, for innovation and for change. Companies are rushing towards the challenges which have been unsolved for decades, One of my favorite machine “the electric car” got an uplift last year. The new fastest car — Tesla roadster is amazing, it has a 1000 km (620 Miles)range, for comparison the highest selling electric car — Nissan leaf has a 172 km (107 Miles). Imagine transformations when innovation changes the game at the highest level.

For this kind of innovation and change, organizations and companies also need to change, Teams which develop new products cannot work in silos anymore, they need to adapt new technologies and to cloud. The next challenge comes in when we have data moving to the cloud.

AWS RDS

Amazon ( and I mostly use them as an example as they are the de-facto meaning of cloud in IT) provides cloud services which host data and allow users to move their data to cloud, This can be a hefty exercise to see, but I see this is as essential as say a 40-year-old notion of moving from a paper ledger to an excel worksheet. You have to do it.

There are many challenges here foremost is the security of data, I will come to that in detail later.

1. Architecture reworks — since databases are the center of any organization’s existence ( data is the most valuable resource to any organization), any fiddling with it is frowned upon, Moving it out: Well you need a lot of juice to prepare for it.
2. Latency — as with most IAAS/PAAS how much data you can move through the pipe becomes the bottleneck, A simple cross-reference data lookup query can generate a lot of CPU and inter-VPC traffic.
3. Cost — Don’t get swayed by “elasticity” and “on-demand” claims. Any resource in the cloud is expensive, a 4000 IOPS SSD will be $3/GB+ per minute, Add cloud-trail logging and analytics (for production)and moving that much data.. phew. I recall frantic calls from my team when I raked a $1200 bill overnight during a recent Proof-of-Value !!! ( Psst. similar instance in Google can be up to 25% cheaper than Amazon )

4. Security incidents are getting much more attention lately, The safety and security in an cloud architecture have been implemented in a shared owner model. Just as the home-owner is expected to lock the house; a policeman is expected the guard the streets. Cyber-security in cloud is nothing new compared to when I run programs & host data on someone’s computer — protecting sensitive assets is a shared obligation irrespective of who is managing H/W.

Few instances recently highlight the problem that despite all its fancies, essentially cloud is just another datacenter and users have to tweak and configure it the way they want.

Taking a cue from these data points, the cloud providers are now encouraging the model of shared responsibility to protect data. As the Cloud becomes the new normal I would focus on the how the cloud platform follows this model.

Amazon propagates the methodology as below (details here)

AWS shared security ownership model

There are many benefits when customers move to the Cloud, I will enumerate a few below:

1. Elasticity of infrastructure is big plus, Everyone knows to add a new server or disk to a system takes how much time. Even with modern architectures like VMware virtual machines, getting a VM with dedicated resources is a big deal in any organization. For comparison try asking the VM admin for an upgrade from 16 to 40 vCPU w/ resource reservation in an IT org v/s upgrading a m4.4xlarge to m4.10xlarge, any reader who has done this will get my point.
2. Certifications which comes with the underlying H/w is another major advantage especially for the CySec fraternity when using cloud, Please note that either AWS or Azure do not allow any user to inspect their data-centers ( it does not matter which brand is on your business card), No one questions their credentials as they carry almost every certification needed for PCI, SOX, HIPPA et.al and all safety recognitions like ISO 27001. Throughout my career I have seen CISO’s struggle to get the right certifications and spending millions to manage assets which are distributed under control of various groups, I am myself guilty of redundant & indiscriminate use of SUDO to make things easier.
3. Availability of services for example Key-Management using an HSM is so easy while using AWS Cloud HSM at a laughable cost, I recall my days when getting an HSM device required numerous NDAs, Custom clearance boxes and getting one patched required a courier back to Baltimore UNiCert’s Factory.
4. HA/DR is an advantage which comes ‘ very’ easily as failover and complying with a multi-region requirement which mandates a DR center in another region needs hiring local workforce and facilities and ton of work making it work. This in terms of ease is the biggest point of a cloud data center, backup your AMIs and EC2 buckets with lowest RPO needed with just the console. The power of cloud here is demonstrated by 6 sigma + business continuity benchmarks of AWS & GMAIL where they have been down only a few hours in 2017.
5. IAM is a feature which is most disruptive to the use of the cloud. Cloud being a data center the Identity, Access and Authorization realms of work From a test data perspective things become even more complicated as the entitlement and controls are blurred in a distributed setup. Privilege escalation can be either depending on the user discretion or use draconian measures similar to central PPM tools like Powerbroker etc. Cloud allows you to connect, audit-trail and manage roles centrally and enable SSO or LDAP integration using AWS CLI.

After all of the above one big hole in all the security still remains the data which is not protected, and this is your data privacy initiative without a data strategy