July 24th, 2022 - José Cruz - Wireless Rēbēl, Holt MI
Holt, Michigan USA
In 2018 rēd wireless set out with great ambitions of building our very own, in-house remote monitoring solution - the result of two previous years evaluating less-than-acceptable ‘boxed’ solutions that did not meet even the most basic of our Internet of Things (IoT) requirements. Having over two decades of experience ‘under our belt’ in designing, deploying, maintaining and troubleshooting high profile wireless networks for some of the biggest names in the industry, we knew well our own solution’s goals had to be centered primarily around securityand reliability, as well as being affordable, easy to use and implement, and well designed – not just the software and user interface, but also the hardware and wireless connectivity, which is what we know best. Lastly, we would try to host as much as possible ‘in the cloud’. Of course, looking to add all these different ‘pieces’ of our demanding ‘puzzle’ together was not going to be easy. As a startup with a strict budget, it simply made perfect sense for us to look for ‘cloud’ providers. But right from the start, some of our first pilot/clients were already having some reservations about their data being ‘on the cloud’ – and they were not the only ones. According to a survey included in a piece by Information Weekback when we started in 2016, 85% of business owners said they were “concerned” or “very concerned” about cloud security. Yet it was precisely our understanding of cloud security and its benefits outweighing risks that made us choose these solutions in the first place.These very same business owners that have some trepidations (and rightfully so!) in using cloud solutions will, inadvertently, use it in other forms almost every single day: when accessing their checking accounts via a mobile app, swiping their credit or bank card after a purchase at their favorite retailer, or taking money out from an ATM at a local gas station – they’re all using the ‘cloud’ in one way or another, whether via a public or private ‘instance’. It is, then, the understanding of the security features included in the above examples, even at its most basic level, that enables a person to evaluate and, eventually, adopt these technologies for their own personal or commercial everyday use.
We look at ‘clouds’ from both sides now
Because of these same concerns, we began developing our own ‘routine checklist’ of requirements to ask every time we met or spoke with a potential cloud service provider: Where are your servers located? What security breach practicesyou have in place? What kind of SLA/QoS(Service Level Agreement/Quality of Service) offered? Do you provide a ‘Status’ page? – the list went on. And as we continued our search, we discovered that almost every technology cloud service provider big or small will, in turn, ‘sub-cloud-host’ themselves into one or more of the three main cloud ‘giants’: Amazon Web Services(AWS), Microsoft Azureand Google Compute Platform(GCP). In other words, the very same companies we were interested in providing us with an alternativeto owning our own ‘on-premises’ servers and services were turning to AWS, Azure or GCP for their own cloud needs. In fact, it is estimated that ‘the big three’ alone account for over 65% of the total global cloud market share – let that sink in for a moment.
Defeating the [cloud] purpose
As one can imagine, having almost ¾ of the ‘pie’ for themselves opens a Pandora’s boxfull of other aspects to consider (read monopoly risks), which in essence defeats the initial purpose of what cloud computing was supposed to be. All three of them (AWS, Azure & GCP) offer the opportunity of ‘stress-free’ cloud customization with a plethora of features, flexibilities and capabilities that would, in theory, add to our collective business’s ‘agility and profitability’. This is what Forbes’ Sr. Contributor Adrian Bridgwater refers to as ‘the great cloud promise’. He argues (and so do we) that, instead, cloud computing has become inherently complicated, “the cloud model approach has been constitutionally riddled with chaotic-level mismatches…cloud is complex.” – We could not agree more.In our own journey of building what we believed to be a seemingly straightforward automation solution, that is to say, a solution that includes the 4 basic‘building blocks’ (hardware, connectivity, networkand software), has taken us more than 3 yearsto put together - and our strictest of budgets wasn’t necessarily the only one in town to blame. The ‘chaotic-level mismatches’ that Adrian mentions and that we have encountered ourselves while testing, for example, something as basic as the data exchange between various ‘sub-cloud-hosting’ platforms - even when making sure to use only standardized and globally adopted protocols - deserves an entire separate blog/discussion of its own. But even more worrying is the fact of having almost every sub-hosted provider of cloud services running on one of just three ‘giants’, which translates in the high probability of one or more of your own services being affected at some point or another by hosting them in only 1 of 3 places – the proverbial ‘single point of failure’ has only but three Olympic-size cloud ‘pools’ to ‘swim’ in. But, rest assure, no matter what your project scale’s needs are, the big three have enough ‘water’ to ‘float’ all your speedy data ‘swimmers’, right?Well, let’s take Amazon’s massive outage of last December 2021for example, where millions of users and sub-hosted big providers such as Venmo, Disney+, Ring, Instacart, Roku, DoorDash, PlayStation, Slack, Netflix, CNN, and others went down when AWS servers in two different regions(US-EAST-1, US-WEST-1) within three separate daysbegan having failures and network congestions between its own internal AWS components. Even our own MQTT communication protocol sub-hosted cloud instance was affected, and after weeks of going back and forth, we were forced to ‘switch’ to a different MQTT ‘broker’ provider (not sub-hosted on AWS) altogether - which is no easy feat since each cloud provider works differently and relies on its own infrastructure and services. Add another requirement to our ad-hoc checklist…Having seen these type of service interruptions throughout our many years of telecom involvement, we would expect this to be the kind of event that would send, as they say, ‘heads rolling’ everywhere or, at least, become a high profile ‘target’ topic for policy makers and news agencies alike. Case in point, in July 2, 2022 Japanese cellular operator KDDI faced a ‘traffic congestion’ failure that affected over 40 million usersfor two straight days, forcing Japan’s deputy Chief Cabinet Secretary to demand a detailed explanation right away. On the other side of the globe, Canada, too, would face its own cellular outage of 19hrsfrom its service provider Rogers, coincidentally around the same time as its Japanese counterpart and in the middle of a potential merger process. Canadian officials went scrambling in search for answers, and the pressure sent Rogers replacing its own Chief Technology Officer just weeks after – aka ‘heads rolled’.By contrast, when asked about the AWS ‘outagepalooza’ that also affected offices of the US government, the US Cybersecurity and Infrastructure Security Agency(CISA) said in an e-mailthat it was working with Amazon “to understand any potential impacts the outage could have on federal and other partners.” - forgive us for failing to understand ifthe same level of urgency is being applied here, especially when it was precisely AWS and their ‘public cloud evangelists’ who touted us all at Mobile World Congress 2021 (MWC21) of their strong ‘new kids’ telco position and as a replacer of sorts of the more traditional telecom service providers. Taking over the alleged multi-million dollar ‘show space’ left empty by 4G/5G equipment and solutions provider Ericsson, AWS ‘heavyweight’ Telco DR’s CEO and self-proclaimed ‘Elon Musk of telco’ (?!!), Danielle Royston, boasted in an interview with RCR Wireless, “…time for the dinosaurs to die and the new kids to come in.“ And just to enforce the point to all in attendance at the Barcelona, Spain-held event, Amazon had Dish Network’s top executives lined upto speak about their AWS’ telecom use-case of being the first major wireless network operator (in the world, it seems) to have all its functions virtualized in Amazon’s cloud: “You just cannot have a glitch in the network”, said Marc Rouanne, Chief Network Architect of Dish Network. We agree with Marc 100%, but after developing dozens of wireless networks in the Americas for over 20 years, and with all the evidence so far, we can’t help but to feel a bit concerned for Dish’s ‘larger 5G eggs’ being so optimistically placed in AWS’ grandeur-vision-of-the-future ‘basket’. Only time will tell, but be it new kidsor old dinosaurs, unforeseen ‘asteroids’ of new and untested technology affects all equally.
A secured insecurity
Many of these cloud evangelistswill often refer to these new hosted solutions by utilizing catchy terms like ‘disruptive’ or ‘decentralized’, yet it seems ironical to us that so much of this new global ‘cloud power’ has been allowed (read ‘lobbied’) to be ‘centralized’within just three main players. Even Amazon’s own Chief Technologist Ishwar Parulkar acknowledged at MWC21 the need to break up its centralized cloud computing design into ‘pieces’so it can be more easily injected into telecom providers such as Dish. And just as important as these reliability talking points, we must also address the security ramifications of such ‘disruptive centralization’ within the realities of its complex and all-encompassing cloud environment.On July 19, 2019, more than 100 million Capital One customershad their personal and/or financial data compromised – names, addresses, dates of birth, bank account ids, credit scores and/or social security numbers – which, according to the FBI, was carried out by a (now former) Amazon AWS engineerexploiting a ‘configuration vulnerability’ and, in turn and to add insult to injury, stored the illegally obtained data in rented AWS cloud servers, making it one of the biggest data breaches to ever hit a financial services company (ranking it right alongside the 2017 infamous Equifax hack). Amazon admitted the ‘misconfiguration’ but was quoted by Bloombergstating that “the data was not accessed through a vulnerability in AWS systems”. Po-tae-to, po-ta-to, to-mae-to, to-ma-to…Only a half a year later in early 2020 the US government was publicly confirming in a separate incident that hackers had infiltrated key government networks, including high level offices such as the US Treasury, State Department, Homeland Security, and the Department of Commerce among others through an ‘update mechanism’ within their IT management and observability(the irony!) software platform Orionprovided by American company SolarWinds. As reported by The New York Times, the software company allegedly “underspent on [cyber]security”, with employees/interns’ passwords leaking out the company to the public, including said update mechanism’s password for the more than 18,000 of SolarWinds Orion’s public and private customers being the same for over 2 years: “solarwinds123”. Rubbing salt to the wound, it has been reported that a security expert had tried to warn SolarWindsof the vulnerability in 2019, “[SolarWinds] update server could easily be accessed by anyone using the simple password ‘solarwinds123’…This could have been done by any attacker, easily”. Because of the sheer scale and the highly sensitive targets, ‘heads’ did‘roll’ for these non-traditional-telecom providers – or at least some officials tried to make them roll. Less than a week after the incident, the US government directed all federal civilian agencies to power down all SolarWinds Orion products: “The compromise of SolarWinds’ Orion Network Management Products poses unacceptable risks to the security of federal networks”, said CISA Acting Director Brandon Wales. In the months that followed, US senators from the Senate Intelligence Committee convened a special hearing to discuss the ‘intrusion’ and invited, both, Microsoft Azure and Amazon AWS after it was publicly revealedthe two cloud computing giants’ infrastructure seemed to have been involved in the SolarWinds attack. Microsoft denied having encountered any vulnerabilities in their services or products; AWS simply decided not to show up, “Apparently they were too busy to discuss that here with us today”, said Sen. Marco Rubio, R-FL. Sen. Mark Warner, D-VA joined the bipartisan discontent: “There may be other brand-name players that may have been penetrated that have not been as forthcoming and are leaving policymakers and potentially customers in the dark.” Sen. John Cornyn, R-TX. added, “[Amazon] declining to participate[in the hearings] …that’s a big mistake”. “I’ve got a stronger password than ‘solarwinds123’ to stop my kids from watching too much YouTube on their iPad”, noted Rep. Katie Porter, D-CA.
The [cost] devil is in the details
A few years ago, a college student decided to make good use of some “free AWS credits”still available from a previous student program. Thinking it would be a good idea, the now graduated student decided to open an account and “try out the free tier”. Nothing ever came of it, and after creating the account with an apparently less-than-ideal-not-so-strong password, the student simply forgot about it. That was until about a year or so later when the graduate was greeted by an Amazon invoice totaling a whopping US$20,000. As you might have already guessed, allegedly, hackers were able to access the account’s unused cloud ‘instances’ and/or ‘keys’and ran amok with the cloud instance. By the time Amazon’s own internal ‘account hold’ process kicked in to stop the malicious activity, thousands of dollars’ worth of cloud computing were now knocking at the college grad’s front door. Luckily, and after various phone calls, Reddits and customer support assistance, Amazon waived the bill in its entirety- a very lucky outcome!This is just but one of hundreds, if not thousands, of Reddits, Tweets and other social media nightmare stories – often not having such good endings, as it was our own case.When we set out to develop our own in-house commercial solution, we knew of horror stories like the one above and the overall ‘vendor-locked-in’ risks associated with cloud host solutions – especially with ‘proprietary’ sub-hosted ones. But giving your credit card info to a cloud provider is not necessarily the only concern – many of today’s Internet of Thingsand other hosted and sub-hosted cloud providers will offer what is referred to in the industry as a ‘sandbox’ or trial account, which in simple terms means the same as the graduate student’s case but without the worry of receiving any kind of surprise invoices. Sandbox accounts usually give a user or client a ‘feel’ for what the candidate service or platform is capable of, albeit with lots of ‘guardrails’and/or limitations. With such enticing, seemingly worry-free evaluation we were very impressed by a particular sub-hosted ‘piece of the puzzle’ provider and committed the sacrilege(against our own already established guidelines!) of evaluating their proprietaryofferings - thus, putting some of our solution’s digital ‘eggs’ in one cloud provider’s ‘nest’ for 3 full years. Just like that we had made ourselves ‘vendor-locked-in’ in the process. Having collaborated heavily with said proprietary sub-hosted cloud provider, we became extremely happy with the early results and overall progress. That was until summer of 2022, when getting ready to transfer our sandbox project to a paid subscription, that we discovered (unbeknown to us!) that the cloud sub-hosting’s monthly price from the service provider had now jumped a full order of magnitude (10x) morethan what was originally discussed. Within a split moment we realized the severity of our naiveté: 3-years’ worth of work was now ‘locked-in’; held ‘hostage’ if you will. Lesson learned.Cloud hosted and sub-hosted ‘gallery’ solutions offer ‘digital artists’, big and small, all the ‘tools, canvases, paintbrushes, and space’ that ‘creators’ like us are not able to afford on our own, and (rightfully so) will charge for said use of resources accordingly. But some often forget that clear, open, and ongoing communications are essential to any successful ‘artist/gallery’ relationship– especially when it can be argued that what’s on their ‘walls’ isn’t technically ‘theirs’. Hard, costly lessons learned from both cases above that failed to understand the ‘fine print’ and fell for the carrot along a very expensive stick. Nothing to do but to shook the dust off our backs, scratch [finally!] that last remaining proprietary piece, transfer our ‘art’ to an open standard, and start anew with the same confidence; one ‘brush stroke’ at a time.
With great power comes great responsibility - B. Parker
Lucky for us mere mortals battling out in the land of the cloud gods, the more knowledgeable folks within the ‘DevOps‘ (software development and operations) community are also voicing the same concernswe have in depositing so much of our products and services on just the same three top cloud providers. A report by Techstrong Researchshows that many of these DevOps are pushing for ‘multicloud/multiprovider’ alternative cloud(altCloud) options, citing as some of the top reasons the “lack of options, high costs and mistrust” from the big three - take special note on that last one: ‘mistrust’.As Peter Parker’s uncle, Benjamin, reminds our arachnid minded superhero, when great poweris attained certain ‘checks and