Blog
Archive Blogs
July 24th, 2022 - José Cruz - Wireless Rēbēl, Holt MI
Holt, Michigan USA
In   2018   rēd   wireless   set   out   with   great   ambitions   of   building   our   very   own,   in-house   remote   monitoring   solution   -   the result   of   two   previous   years   evaluating   less-than-acceptable   boxed ’   solutions   that   did   not   meet   even   the   most   basic   of   our Internet   of   Things   (IoT)   requirements.   Having   over   two   decades   of   experience   under   our   belt ’   in   designing,   deploying, maintaining   and   troubleshooting   high   profile   wireless   networks   for   some   of   the   biggest   names   in   the   industry,   we   knew well   our   own   solution’s   goals   had   to   be   centered   primarily   around   security    and   reliability ,   as   well   as   being   affordable, easy   to   use   and   implement,   and   well   designed   –   not   just   the   software   and   user   interface,   but   also   the   hardware   and wireless connectivity, which is what we know best. Lastly, we would try to host as much as possible ‘ in the cloud ’. Of   course,   looking   to   add   all   these   different   pieces ’   of   our   demanding   puzzle ’   together   was   not   going   to   be   easy.   As   a startup   with   a   strict   budget,   it   simply   made   perfect   sense   for   us   to   look   for   cloud ’   providers.   But   right   from   the   start,   some of   our   first   pilot/clients   were   already   having   some   reservations   about   their   data   being   on   the   cloud ’   –   and   they   were   not the   only   ones.   According   to   a   survey   included   in   a   piece   by   Information   Week    back   when   we   started   in   2016,   85%   of business    owners    said    they    were    concerned ”    or    very    concerned ”    about    cloud    security.    Yet    it    was    precisely    our understanding of cloud security and its benefits outweighing risks that made us choose these solutions in the first place. These    very    same    business    owners    that    have    some    trepidations    (and    rightfully    so ! )    in    using    cloud    solutions    will, inadvertently,   use   it   in   other   forms   almost   every   single   day:   when   accessing   their   checking   accounts   via   a   mobile   app, swiping   their   credit   or   bank   card   after   a   purchase   at   their   favorite   retailer,   or   taking   money   out   from   an   ATM   at   a   local   gas station   –      they’re   all   using   the   cloud ’   in   one   way   or   another,   whether   via   a   public   or   private   instance ’.      It   is,   then,   the understanding   of   the   security   features   included   in   the   above   examples,   even   at   its   most   basic   level,   that   enables   a   person to evaluate and, eventually, adopt these technologies for their own personal or commercial everyday use.

We look at ‘clouds’ from both sides now

Because   of   these   same   concerns,   we   began   developing   our   own   routine   checklist ’   of   requirements   to   ask   every   time   we met   or   spoke   with   a   potential   cloud   service   provider:   Where   are   your   servers   located ?   What   security   breach   practices    you have   in   place ?   What   kind   of   SLA/QoS    (Service   Level   Agreement/Quality   of   Service)   offered ?   Do   you   provide   a   ‘Status’   page ?   –   the list   went   on.   And   as   we   continued   our   search,   we   discovered   that   almost   every   technology   cloud   service   provider   big   or small   will,   in   turn,   sub-cloud-host ’   themselves   into   one   or   more   of   the   three   main   cloud   giants ’:   Amazon   Web   Services   (AWS),   Microsoft   Azure    and   Google   Compute   Platform    (GCP).   In   other   words,   the   very   same   companies   we   were interested   in   providing   us   with   an   alternative    to   owning   our   own   on-premises ’   servers   and   services   were   turning   to   AWS, Azure   or   GCP   for   their   own   cloud   needs.   In   fact,   it   is   estimated   that   the   big   three ’   alone   account   for   over   65%   of   the   total global cloud market share  – let that sink in for a moment.

Defeating the [cloud] purpose

As   one   can   imagine,   having   almost   ¾   of   the   pie ’   for   themselves   opens   a   Pandora’s   box    full   of   other   aspects   to   consider (read   monopoly   risks ),   which   in   essence   defeats   the   initial   purpose   of   what   cloud   computing   was   supposed   to   be.   All   three of    them    (AWS,    Azure    &    GCP)    offer    the    opportunity    of    stress-free ’    cloud    customization    with    a    plethora    of    features, flexibilities   and   capabilities   that   would,   in   theory,   add   to   our   collective   business’s   agility   and   profitability ’.   This   is   what Forbes ’   Sr.   Contributor   Adrian   Bridgwater   refers   to   as   the   great   cloud   promise ’.   He   argues   (and   so   do   we)   that,   instead, cloud   computing   has   become   inherently   complicated,   the   cloud   model   approach   has   been   constitutionally   riddled   with chaotic-level mismatches…cloud is complex .” – We could not agree more. In   our   own   journey   of   building   what   we   believed   to   be   a   seemingly   straightforward   automation   solution,   that   is   to   say,   a solution   that   includes   the   4   basic    building   blocks ’   ( hardware ,   connectivity ,   network    and   software ),   has   taken   us   more than   3   years    to   put   together   -   and   our   strictest   of   budgets   wasn’t   necessarily   the   only   one   in   town   to   blame.   The   chaotic- level   mismatches ’   that   Adrian   mentions   and   that   we   have   encountered   ourselves   while   testing,   for   example,   something   as basic    as    the    data    exchange    between    various    sub-cloud-hosting ’    platforms    -    even    when    making    sure    to    use    only standardized  and globally adopted protocols  - deserves an entire separate blog/discussion of its own. But   even   more   worrying   is   the   fact   of   having   almost   every   sub-hosted   provider   of   cloud   services   running   on   one   of   just three   giants ’,   which   translates   in   the   high   probability   of   one   or   more   of   your   own   services   being   affected   at   some   point   or another   by   hosting   them   in   only   1   of   3   places   –   the   proverbial   single   point   of   failure ’   has   only   but   three   Olympic-size   cloud pools ’   to   swim ’   in.   But,   rest   assure,   no   matter   what   your   project   scale’s   needs   are,   the   big   three   have   enough   water ’   to float ’ all your speedy data ‘ swimmers ’, right ? Well,   let’s   take   Amazon’s   massive   outage   of   last   December   2021    for   example,   where   millions   of   users   and   sub-hosted   big providers   such   as   Venmo ,   Disney +,   Ring ,   Instacart ,   Roku ,   DoorDash ,   PlayStation ,   Slack ,   Netflix ,   CNN ,   and   others   went down   when   AWS   servers   in   two   different   regions    (US-EAST-1,   US-WEST-1)   within   three   separate   days    began   having failures    and    network    congestions    between    its    own    internal    AWS    components.    Even    our    own    MQTT    communication protocol   sub-hosted   cloud   instance   was   affected,   and   after   weeks   of   going   back   and   forth,   we   were   forced   to   switch ’   to   a different   MQTT   broker ’   provider   ( not   sub-hosted   on   AWS)   altogether   -   which   is   no   easy   feat   since   each   cloud   provider works differently and relies on its own infrastructure and services.  Add another requirement  to our ad-hoc  checklist… Having   seen   these   type   of   service   interruptions   throughout   our   many   years   of   telecom   involvement,   we   would   expect   this to   be   the   kind   of   event   that   would   send,   as   they   say,   heads   rolling ’   everywhere   or,   at   least,   become   a   high   profile   target topic   for   policy   makers   and   news   agencies   alike.   Case   in   point,   in   July   2,   2022   Japanese   cellular   operator   KDDI   faced   a ‘traffic   congestion’   failure   that   affected   over   40   million   users    for   two   straight   days ,   forcing   Japan’s   deputy   Chief   Cabinet Secretary   to   demand   a   detailed   explanation   right   away.   On   the   other   side   of   the   globe,   Canada,   too,   would   face   its   own cellular   outage   of   19hrs    from   its   service   provider   Rogers ,   coincidentally   around   the   same   time   as   its   Japanese   counterpart and   in   the   middle   of   a   potential   merger   process.   Canadian   officials   went   scrambling   in   search   for   answers,   and   the pressure sent Rogers replacing its own Chief Technology Officer  just weeks after – aka ‘ heads rolled ’. By    contrast,    when    asked    about    the    AWS    outagepalooza ’    that    also    affected    offices    of    the    US    government,    the    US Cybersecurity    and    Infrastructure    Security    Agency     (CISA)    said    in    an    e-mail     that    it    was    working    with    Amazon    to understand   any   potential   impacts   the   outage   could   have   on   federal   and   other   partners .”   -   forgive   us   for   failing   to   understand   if   the   same   level   of   urgency   is   being   applied   here,   especially   when   it   was   precisely   AWS   and   their   public   cloud   evangelists who   touted   us   all   at   Mobile   World   Congress   2021   (MWC21)   of   their   strong   new   kids ’   telco   position   and   as   a   replacer   of sorts of the more traditional telecom service providers. Taking   over   the   alleged   multi-million   dollar   show   space ’   left   empty   by   4G/5G   equipment   and   solutions   provider   Ericsson, AWS   heavyweight ’   Telco   DR’s   CEO   and   self-proclaimed   Elon   Musk   of   telco ’   ( ?!! ),   Danielle   Royston,   boasted   in   an   interview with   RCR   Wireless ,   “… time   for   the   dinosaurs   to   die   and   the   new   kids   to   come   in. “   And   just   to   enforce   the   point   to   all   in attendance   at   the   Barcelona,   Spain-held   event,   Amazon   had   Dish   Network’s   top   executives   lined   up    to   speak   about   their AWS’   telecom   use-case   of   being   the   first   major   wireless   network   operator   (in   the   world,   it   seems)   to   have   all   its   functions virtualized   in   Amazon’s   cloud :   You   just   cannot   have   a   glitch   in   the   network ”,   said   Marc   Rouanne,   Chief   Network   Architect   of Dish   Network.   We   agree   with   Marc   100%,   but   after   developing   dozens   of   wireless   networks   in   the   Americas   for   over   20 years,   and   with   all   the   evidence   so   far,   we   can’t   help   but   to   feel   a   bit   concerned   for   Dish’s   larger   5G   eggs’   being   so optimistically   placed   in   AWS’   grandeur-vision-of-the-future   basket’ .   Only   time   will   tell,   but   be   it   new   kids    or   old   dinosaurs , unforeseen ‘ asteroids’  of new and untested technology affects all  equally.

A secured insecurity

Many   of   these   cloud   evangelists    will   often   refer   to   these   new   hosted   solutions   by   utilizing   catchy   terms   like   disruptive ’   or decentralized ’,   yet   it   seems   ironical   to   us   that   so   much   of   this   new   global   cloud   power ’   has   been   allowed   (read   lobbied’ )   to be   centralized’    within   just   three   main   players.   Even   Amazon’s   own   Chief   Technologist   Ishwar   Parulkar   acknowledged   at MWC21   the   need   to   break   up   its   centralized   cloud   computing   design   into   pieces’    so   it   can   be   more   easily   injected   into telecom   providers   such   as   Dish.   And   just   as   important   as   these   reliability   talking   points,   we   must   also   address   the   security ramifications of such ‘ disruptive centralization ’ within the realities of its complex and all-encompassing cloud environment. On   July   19,   2019,   more   than   100   million   Capital   One   customers    had   their   personal   and/or   financial   data   compromised   names,   addresses,   dates   of   birth,   bank   account   ids,   credit   scores   and/or   social   security   numbers   –   which,   according   to   the FBI,   was   carried   out   by   a   (now   former)   Amazon   AWS   engineer    exploiting   a   configuration   vulnerability ’   and,   in   turn   and   to add   insult   to   injury,   stored   the   illegally   obtained   data   in   rented   AWS   cloud   servers,   making   it   one   of   the   biggest   data breaches   to   ever   hit   a   financial   services   company   (ranking   it   right   alongside   the   2017   infamous   Equifax   hack ).   Amazon admitted   the   misconfiguration ’   but   was   quoted   by   Bloomberg    stating   that   the   data   was   not   accessed   through   a   vulnerability in AWS systems ”. Po-tae-to, po-ta-to, to-mae-to, to-ma-to Only   a   half   a   year   later   in   early   2020   the   US   government   was   publicly   confirming   in   a   separate   incident   that   hackers   had infiltrated   key   government   networks,   including   high   level   offices   such   as   the   US   Treasury,   State   Department,   Homeland Security,   and   the   Department   of   Commerce   among   others   through   an   update   mechanism ’   within   their   IT   management   and observability    (the   irony ! )   software   platform   Orion    provided   by   American   company   SolarWinds .   As   reported   by   The   New York   Times ,   the   software   company   allegedly   underspent   on   [cyber]security ”,   with   employees/interns’   passwords   leaking   out the   company   to   the   public,   including   said   update   mechanism’s   password   for   the   more   than   18,000   of   SolarWinds   Orion’s public   and   private   customers   being   the   same   for   over   2   years:   solarwinds123 ”.   Rubbing   salt   to   the   wound,   it   has   been reported   that   a   security   expert   had   tried   to   warn   SolarWinds    of   the   vulnerability   in   2019,   “[SolarWinds]   update   server   could easily be accessed by anyone using the simple password ‘solarwinds123’…This could have been done by any attacker, easily ”. Because   of   the   sheer   scale   and   the   highly   sensitive   targets,   heads ’   did    roll ’   for   these   non-traditional-telecom   providers   or   at   least   some   officials   tried   to   make   them   roll.   Less   than   a   week   after   the   incident,   the   US   government   directed   all federal   civilian   agencies   to   power   down   all   SolarWinds   Orion   products :   The   compromise   of   SolarWinds’   Orion   Network Management   Products   poses   unacceptable   risks   to   the   security   of   federal   networks ”,   said   CISA   Acting   Director   Brandon   Wales. In   the   months   that   followed,   US   senators   from   the   Senate   Intelligence   Committee   convened   a   special   hearing   to   discuss the   intrusion ’   and   invited,   both,   Microsoft   Azure   and   Amazon   AWS   after   it   was   publicly   revealed    the   two   cloud   computing giants’   infrastructure   seemed   to   have   been   involved   in   the   SolarWinds   attack.   Microsoft   denied   having   encountered   any vulnerabilities   in   their   services   or   products;   AWS   simply   decided   not   to   show   up ,      Apparently   they   were   too   busy   to   discuss that   here   with   us   today ”,   said   Sen.   Marco   Rubio,   R-FL.   Sen.   Mark   Warner,   D-VA   joined   the   bipartisan   discontent:   There   may be   other   brand-name   players   that   may   have   been   penetrated   that   have   not   been   as   forthcoming   and   are   leaving   policymakers and   potentially   customers   in   the   dark .”   Sen.   John   Cornyn,   R-TX.   added,   “[Amazon]   declining   to   participate    [in   the   hearings] that’s   a   big   mistake ”.   “I’ ve   got   a   stronger   password   than   ‘solarwinds123’   to   stop   my   kids   from   watching   too   much   YouTube   on their iPad ”, noted Rep. Katie Porter, D-CA .

The [cost] devil is in the details

A   few   years   ago,   a   college   student   decided   to   make   good   use   of   some   “free   AWS   credits”    still   available   from   a   previous student   program.   Thinking   it   would   be   a   good   idea,   the   now   graduated   student   decided   to   open   an   account   and   try   out the   free   tier ”.   Nothing   ever   came   of   it,   and   after   creating   the   account   with   an   apparently   less-than-ideal-not-so-strong password ,   the   student   simply   forgot   about   it.   That   was   until   about   a   year   or   so   later   when   the   graduate   was   greeted   by an   Amazon   invoice   totaling   a   whopping   US$20,000 .   As   you   might   have   already   guessed,   allegedly,   hackers   were   able   to access   the   account’s   unused   cloud   instances ’   and/or   keys’    and   ran   amok   with   the   cloud   instance.   By   the   time   Amazon’s own   internal   account   hold ’   process   kicked   in   to   stop   the   malicious   activity,   thousands   of   dollars’   worth   of   cloud   computing were   now   knocking   at   the   college   grad’s   front   door.   Luckily,   and   after   various   phone   calls,   Reddit s   and   customer   support assistance,    Amazon    waived    the    bill    in    its    entirety     -    a    very    lucky    outcome !     This    is    just    but    one    of    hundreds,    if    not      thousands,   of   Reddit s,   Tweet s   and   other   social   media   nightmare   stories   –   often   not   having   such   good   endings,   as   it   was   our own case. When   we   set   out   to   develop   our   own   in-house   commercial   solution,   we   knew   of   horror   stories   like   the   one   above   and   the overall   vendor-locked-in ’   risks   associated   with   cloud   host   solutions   –   especially   with   proprietary ’   sub-hosted   ones.   But giving   your   credit   card   info   to   a   cloud   provider   is   not   necessarily   the   only   concern   –   many   of   today’s   Internet   of   Things    and other   hosted   and   sub-hosted   cloud   providers   will   offer   what   is   referred   to   in   the   industry   as   a   sandbox ’   or   trial   account, which   in   simple   terms   means   the   same   as   the   graduate   student’s   case   but   without   the   worry   of   receiving   any   kind   of surprise   invoices.   Sandbox   accounts   usually   give   a   user   or   client   a   feel ’   for   what   the   candidate   service   or   platform   is capable   of,   albeit   with   lots   of   guardrails’    and/or   limitations.   With   such   enticing,   seemingly   worry-free   evaluation   we   were very   impressed   by   a   particular   sub-hosted   piece   of   the   puzzle ’   provider   and   committed   the   sacrilege    (against   our   own already   established   guidelines ! )   of   evaluating   their   proprietary    offerings   -   thus,   putting   some   of   our   solution’s   digital   eggs in one cloud provider’s ‘ nest ’ for 3 full years. Just like that we had made ourselves ‘ vendor-locked-in ’ in the process. Having   collaborated   heavily   with   said   proprietary   sub-hosted   cloud   provider,   we   became   extremely   happy   with   the   early results   and   overall   progress.   That   was   until   summer   of   2022,   when   getting   ready   to   transfer   our   sandbox   project   to   a   paid subscription,   that   we   discovered   (unbeknown   to   us ! )   that   the   cloud   sub-hosting’s   monthly   price   from   the   service   provider had   now   jumped   a   full   order   of   magnitude   (10x)   more    than   what   was   originally   discussed.   Within   a   split   moment   we realized the severity of our naiveté: 3-years’ worth of work was now ‘ locked-in ’; held ‘ hostage’  if you will. Lesson learned . Cloud   hosted   and   sub-hosted   gallery ’   solutions   offer   digital   artists ’,   big   and   small,   all   the   tools,   canvases,   paintbrushes,   and space ’   that   creators ’   like   us   are   not   able   to   afford   on   our   own,   and   (rightfully   so)   will   charge   for   said   use   of   resources accordingly.    But    some    often    forget    that    clear,    open,    and    ongoing    communications    are    essential    to    any    successful artist/gallery ’   relationship    –   especially   when   it   can   be   argued   that   what’s   on   their   walls ’   isn’t   technically   theirs ’.      Hard,   costly lessons   learned   from   both   cases   above   that   failed   to   understand   the   fine   print ’   and   fell   for   the   carrot   along   a   very expensive   stick.   Nothing   to   do   but   to   shook   the   dust   off   our   backs,   scratch   [finally ! ]   that   last   remaining   proprietary   piece, transfer our  ‘ art ’ to an open standard, and start anew with the same confidence; one ‘ brush stroke ’ at a time.

With great power comes great responsibility - B. Parker

Lucky   for   us   mere   mortals   battling   out   in   the   land   of   the   cloud   gods,   the   more   knowledgeable   folks   within   the   DevOps ( software   development   and   operations )   community   are   also   voicing   the   same   concerns    we   have   in   depositing   so   much   of our   products   and   services   on   just   the   same   three   top   cloud   providers.   A   report   by   Techstrong   Research    shows   that   many of   these   DevOps   are   pushing   for   multicloud/multiprovider ’   alternative   cloud    (altCloud)   options,   citing   as   some   of   the   top reasons the “ lack of options, high costs and mistrust ” from the big three - take special note on that last one: ‘ mistrust ’. As   Peter   Parker’s   uncle,   Benjamin,   reminds   our   arachnid   minded   superhero,   when   great   power    is   attained   certain   checks and