“…there aren’t enough skilled embedded developers –who are capable of untangling the complexdevelopment and integration challenges that haveunfortunately been a feature of many connectivitysolutions to date”
Alistair Fulton, Semtech
Thermocouple ‘Stinger’ Probe
Adeunis® Field Test Device
InBuilding RF Design
Core Samples’ Tanks
Introduction : The Internet of Things
After its inception by British pioneer Kevin Ashtonback in 1999, the term Internet of Things (IoT) has come a long way since becoming more relevant as the digital transformation age moves forward. In fact, and after two fast decades, its evolution has seen most of the protocols and associated technologies already on their way down the so called “peak of inflated expectations”, passed the “trough of disillusionment” and well into the “slope of enlightenment”, as shown below per Gartner’s Hype Cycle for IoT Standards and Protocols:With this new digital pace of ‘plateauing’from places to people to‘things’, classifying and identifying all the potential verticals and solutions can be a daunting task. Yet the overall current consensus in the industry has targeted IoT around two main applications, from which all others are branched: asset trackingand remote monitoring. And when it comes to the latter, especially on the industrialside of it, temperaturewill almost always be the #1 attributeon the list to measure, monitor and report.From monitoring precise levels in the soil for agriculture, or in critical freezer containers for Covid-19 vaccines, setting ambiance thresholds for hospitality and buildings, or mechanical variances of industrial equipment, temperaturein today’s Fourth Industrial Revolutionis no longer the trivial measurable attribute it once was since the first thermometers were invented back in the early 18th century. Such is the case of rēd wireless’ geotechnical, structural and environmental engineering colleagues and clients at HAE & Associateslocated in Canton, Michigan. Since 1994, HAE & Assoc. has been trusted by dozens of public and private clients for their multi-disciplined, cost-effective, yet practical solutions in construction, planning, surveying, management and testing services. Having their own in-house state-of-the-arttesting lab that operates 24 hours a day, 7 days a week, HAE & Assoc. is among a selected few in the industry certified to provide advanced materials evaluation services at all stages (from product development to materials already in place) and help diagnose failures, develop methods for improving existing products and/or find new uses for materials for different applications such as geotechnical engineering, environmental sciences and construction monitoring.One of their lab services involves the placement of various client’s concrete ‘cylinders’ and other construction materials’ ‘core’samples inside a tank filled with a precise pHtemperature-controlled solution to monitor the samples’ characteristics and performance before implementation in the real world. As such, it is imperative for HAE & Assoc. professionals to keep accurate, real-time records of each sample and monitor the temperature inside each tank daily, thus responding quickly to any changes in the tanks and adjusting conditions as needed – clearly not a trivialprocess by any standard. An Industrial IoT remote monitoring solution seemed like the ideal candidate for this use case, so a ‘pilot’ project was proposed by rēd wireless.
Not only this industrial temperature case was not such a trivial ‘attribute’ to collect, but also the specific pH-solution filled tanks added an additional level of complexity: the sensor’s temperature measuring device, or ‘probe’, needed to be able to chemically withstand the various alkaline/acid levels present inside the tanks’ solution without altering the measurements. Moreover, typical wireless IoT and industrial monitoring sensors only alert when a certain threshold is ‘triggered’, limiting the number of messages sent, thus, conserving battery life in the process. In this case we are requested to monitor every single hour of every single day of every single month – for 12 months. In simple math, 24 temperature readings a day would be measured, collected, and sent by the IoT temperature sensor, each day for 365 days which amounts to, at least, 8,760 temperaturedata points in a year(read ‘at least’since we are not including periodic, supervisory,and other non-temperature related messages). And since this industrial lab environment is home to not just the materials evaluation tanks but also many other chemical, environmental and geotechnical experiments and applications, the radio frequency (RF) ‘ambient noise’ and interference levels, hardware placement and antenna specifications all had to be considered when selecting the appropriate wireless connectivity technology for sensor communication quality. Finally, and probably most important of all, the pilot would be deployed right at the start of the Covid-19 pandemic and under the subsequent state and national lock-down guidelines.
Given all the requirements for this project, we needed a robust wireless technology that could withstand the harshest of RF environments, with minimal or no human intervention after setup, having the best energy efficiency given the demanding hourly/daily polling of temperature readings. Since each of the readings involves only a small number of packets ofasynchronousdata(i.e., temperature readings at least oncean hour, regardless of whenwithin that hour), technologies such as Wi-Fior Bluetoothwere not considered as these have a lot more data capacity (or bandwidth) than was needed, which in turn add range, battery, and interference limitations due to their bigger data rates and/or higher frequency RF bands. Other technologies considered that dohave lower data and power specifications with better range and lower frequency RF bands were either proprietary or not well supported – thus having the risk of typical and non-ideal ‘vendor-locked-in’ scenarios with no future-proof guarantees of any kind. After careful evaluation and considering rēd wireless have already tested and evaluated long range and low power wireless technologies in the past, it was determined that the wireless communication best suited for this project was the California-based Semtech Corporation’s patented yetroyalty-free LoRa® PHY(physical) connectivity protocol, in conjunction with the open-sourced LoRaWAN® MAC(Medium Access Control) network protocol, managed and standardized globally by the LoRa Alliance®. Not only does the LoRa® protocol offers incredible robustness against ‘noisy’ RF environments, with long range features capable of combating interference and ‘jamming’scenarios which dynamically adjust so it can still recover the small messages, but also employs a secured and encryptedsensor-to-network-and-applicationend-to-endAES-128 scheme that comes standard, and not just as a later security ‘add-on’:
After selection of the LoRa® and LoRaWAN® technologies, the next step was to choose the sensor that would collect the ambient temperature inside the tanks via an external probe capable of withstanding the various alkaline or acid liquid levels present, transform this information accurately into an electronic signal that could, then, be transmitted wirelesslyto a local ‘gateway’(or ‘concentrator’in LoRa® ‘lingo’), which in turn then sends this data to the network and application platforms, secured & encrypted, for analysis, reporting, alerting and visualizing (see diagram above). And because this is an industrial commercial setting, the device in question would need to conform not only to the challenges already described above but also to industry standards such as Ingress Protection (IP) against dust and water as well as being FCC certified.Given all these ‘checklist’items, and after careful evaluation of many ‘off-the-shelf’ options available given the broad adoption of LoRa® with many device manufacturers, we decided on Minnesota-based Radio Bridge’s industrial RBS306-TEMP-TC-USLoRaWAN® Wireless ThermocoupleArmored Sensor™. The RBS306-TEMP-TC-US can be deployed ‘standalone’, as in this case, or via the company’s sensor-to-cloudsolution. Each one comes with an external K-typethermocouple probe, which we did not use since our project required one suited for specific pHconditions. Besides K-type, the sensor can be configured for other popular industrial probe types, such as B, E, J, N, R, Sand T. With a NEMA enclosure, IP-67 rating, operating ambient temperature range between -40°C and +70°C, 16 bits precision, .06°C of accuracy, and lithium batteries capable of 200k+ messages, this Armored Sensor™ was the ideal choice for our project. Combined with Radio Bridge’s unique messaging approach, where event payloaddata messages such as temperature and humidity are separated from periodic ones such as battery status which can be scheduled independently by the user, full decoding documentation available and unmatched customer support, makes this sensor extremely easy and flexible to configure for the coders and non-coders alike.Finally, the correct pH specialized quick disconnect thermocouple‘stinger probe’was sourced locally thanks to the collaboration of our geotechnical engineering colleagues and partners at Livonia, Michigan-based Rhino Wireless, which was connected without any additional configuration or calibration needed.
Once the hardware challenges and network needs were all addressed, rēd wireless then proceeded to focus its attention on an IoT platform– which is the one ‘piece of the puzzle’that customers will interface the most. As such, we needed a highly flexible IoT user experience(UX) environment with the ability to create a uniquely branded application for clients, easily accessible without the need of special ‘apps’, secured, accurate and reliable. Moreover, this project application required real-time temperature monitoring, with on-demand, weekly scheduled reports, threshold programmable alarms and alerts via e-mail and SMS. After evaluating 6different types, from ‘boxed’and ‘closed-coded’ applications, to do-it-yourselfones, we selected ‘choice #7’ Ohio-based Losant ’s Application Enablement Platform (AEP). As a result, and with minimal coding, rēd wireless was able to provide this industrial client with a fully automated, real-time dashboardview of their pH-solution filled tanks’ exact temperature levels, thresholds, alarms, alerts, events, and other relevant data on-demand, anytimeand accessible from anywhere via secure HTTPS. The solution also allows for full historical and current records access, as well as two separate weekly reports that await the client’s e-mail inbox each Monday morning – an overall weekly reportand a detailed hourly report. They also receive SMStexts alerts if the temperature levels exceed their own programmable above/below thresholds.
Once all the hardware and software requirements were tended and the project scope, planning and goals had been discussed and approved, rēd wireless finally deployed the pilot system at HAE & Assoc. lab on March 16th, 2020 – exactly one week before Michigan’s Governor 1st statewide stay-at-home orderfor all non-essential workers due to the start of the Covid-19 pandemic. Following early mask mandates and social distancing, we validated that the location and placement of the hardware was secured, including the gateway and its connection via physical ethernet medium (notWi-Fi) and to the LoRaWAN® network, as well as the Armored Sensor® and probe which were mounted on a temporary, custom made wood frame next to the pH-solution filled tanks at the lab. rēd wireless then proceeded to conduct an RF survey studyof the building utilizing a handheld spectrum analyzerto measure and identify, among other key performance indicators(KPI), ambient noise levels and potential sources of interference to the LoRa® physical channels. Once the survey was completed and the results were analyzed to be within acceptable thresholds, we conducted an additional end-to-endnetwork bidirectional ‘live’test using a LoRa Alliance® certified Field Test Device(FTD) with confirmation of messages to validate the performance of bothcommunication directions: the ‘uplink’(UL, or sensor-to-gateway-to-network), and the ‘downlink’(DL, or network-to-gateway-to-sensor). Although many LoRaWAN® live applications will use unconfirmedmessages (meaning that, for example, the sensors will typically send periodic messages to the network without expecting a confirmation that the messages were received successfully) due to the high confidence in the LoRa® protocol and its error correction features that maximize battery and capacity efficiency, when testing commercial solutions on-premises,one must validate that the system isworking as expected by measuring metrics such as Packet Error Rate(PER) among other KPIs – a common misconception of some in the industry deemed as ‘worthless’ or ‘time consuming’. This type of network testing is key to have as a baselinefor any future problems and since no wireless network will ever be 100% error free all the time. Both, the RF survey, and network validation testing, come standardwith all rēd wireless commercial projects. rēd wireless also provides RF Indoor/Outdoor Prediction Design Studiesfor bigger and more complex commercial projects, featuring 1, 2 or more gateways with multiple sensors. This deployment took approximately 1-2 hours to complete.
The final dataset consists of 8,760 data points(hourly temperature readings, every hour, every day, for 1 year) from the Radio Bridge’s industrial RBS306-TEMP-TC-US LoRaWAN® Wireless ThermocoupleArmored Sensor™. The errors, or ‘lost’ messages, were categorized as 1. non-connectivity(non-wirelessand/or network), 2. wireless(RF) and 3. network. A mandatory network Application Programming Interface(API) configuration change on the week of October 26th, 2020, tallied 93 missing messages across 4 dayswhich were excluded from the final count since these were non-connectivityrelated (API config). For the connectivity related errors, there were a total of 24 network connectivity interruptionsspread across 4 different weeks (networkerrors) and only 7 wireless related missing messages spread across 5 different weeks (RF errors):Our original goaltarget for wireless(RF) connectivity related errors was 2% or less, taking ‘cue’from our 3G/4G cellular performance ‘old testing’ days. To say that our expectations were exceeded is an understatement, having only a .35%message ‘network/RF’ final error rate. Moreover, this pilot project’s test LoRaWAN® Network Server was notSLA’d(Service Level Agreement) or QoS’d(Quality of Service), still we were most happy with these results of just a .27%network ‘downtime’ / 99.73%network ‘uptime’ (not counting API config) - just shy of the typical industrial and commercial ‘three nines’ (99.9%) SLA standard:The Radio Bridge’s industrial RBS306-TEMP-TC-US LoRaWAN® Wireless ThermocoupleArmored Sensor™ performed brilliantly, including in the ‘energy department’, maintaining the same initial 3.1 voltsaveraged* all throughout the year without having to replace the original batteries (*values fluctuated between 3v and 3.1v). Once again, these exceeded our expectations considering the large amount of temperature data points collected, in addition to all other periodic and configuration messages, making the business case for this device on stricter scheduling requirements (less than 1hr reports) as well as its sustainability and battery waste reduction.Finally, and without even realizing it, we were able to validate some of the well-known connectivity‘corner cases’currently being addressed by many LoRa Alliance® members, including questionssuch as “what happens when a device cannot connect for over 3+ days” and “how well does a device join back the network after a 3+ days interruption” – which following our mandatory ‘API configuration’ 4 day ‘outage’in October 2020, the answer to both questions is : “this device re-joins the network flawlessly”.These results speak to, not only the resiliency of the LoRaWAN® technology, the quality of the chosen hardware components, and the overall reliability of the networks and platforms, but also of the critical importance system integrators(or any other IoT implementer for that matter) and their wireless connectivity skills and experience play in the success (or failure) of any IoT project – big or small. From Beecham Research’s own web report, whyiotprojectsfail.com, to the LoRa Alliance®’s Certification Test Tool (LCTT) and RF Performance Evaluation Procedure, it has finally become noticeable that the wireless #connectivityside remains, as industry publication IoT-NOWputs it in their 2021 Q2 edition, “a complicated jungle of telecoms industry technology”, something even ourselves have witnessed since rēd wireless’ inception back in 2016, seeing (and advising) many in the IoT industry that have chosen to overlook this critical piece of the overall IoT puzzle for the past 7 years. With over 25+ years of cellular and non-cellular wireless experience, connectivityis at the very core of all rēd wireless operations and forms the foundation for all our services and solutions – and this project’s success serves as testament of that.
The Customer Impact
After only the first 3 months of operation, rēd wireless asked Mr. Gus Haengel, President of HAE & Associates, 3 insightful key questions concerning the ongoing project results – below are his answers:1)How satisfied are you with the project so far?“I am extremely satisfied with the project. Excellent data, fast and easy to make decisions.”2)Any questions or highlights so far?“No question[s]. The data information is clear and easy to read.”3)Does this solution bring value to your organization?“Yes, as we engineers do not have extra time in our days. This type of data is very valuable and time saving for our precious time. And is excellent to make fast and clear decision on the systems.”Indeed. As rēd wireless became more familiarized with the many aspects of the geotechnical industry through this remote monitoring pilot project, we quickly realized this was no typical, trivial ‘temperature logging’ exercise. Due to the tanks’ specific heater element characteristics, lab ever-changing‘dynamics’and strict required pHlevels, once new core samples have entered the tank (as seen in the graph below at the beginning of the week), if the temperature goes above/below the specified thresholds it could take up to 16 hoursof manual daily‘adjustments’ to bring it back to ‘normal’ and in compliance:To keep the integrity of these core samples, which is critical to the success of the lab, careful inspections and methodical measurements must be accurately logged hourly, daily, and weekly. The consequences for any errors or omissions can be costly, both, in time and money and cannot simply be ‘pencil-whipped’: each core sample (which can be as many as 20 or more present in the tanks at any given time) must be meticulously extracted from the field by a specialized crew, using sophisticated and expensive ‘coring’ equipment, and shipped to the lab – to the tune of about $1,000 to $3,000 per sample. This price can easily double in the wintertime, as the extraction of the samples become more challenging.With rēd’s remote monitoring solution, all hourly, daily, and weekly trendscan now be monitored, visualized, alerted, reported, and reactedin real time before ‘hitting’the specified thresholds, thus cutting the critical time once passed the thresholds to 4 hoursor less, greatly minimizing risk while increasing productivity by about fourfold.Mr. Haengel has also expressed how invaluable it has been for him to receive via e-mail every Monday morning the two automated reports: the one-page summary weekly overview, and the multiple-pages detailed logfor each individual hourly data point. It allows him a quick and effective analysis and sharing of the data with his team on what has transpired the week prior and what’s trending in the week ahead.
Artificial Intelligence (AI)
In collaboration with our friends and colleagues at Elipsa.ai, we decided to go even further with this project. Providing Elipsa with the last 6 months’ worth of data for them to process through their ‘AI outliers’ algorithm, we wanted to see if any additional trends or other relevant information would come up. And although we were collecting just hourly temperature readings and the alerts when temperature levels were above/belowa certain specified threshold from just 1 device, an interesting ‘pattern’did come up as a result: the AI model, based on roughly 4,380 temperaturedata points collected and their associated above/belowthresholds, suggestedchangingthe present above/belowthresholds to more precise valuesfor better performance and less ‘reaction’time, lining up almost perfectly with our own findings, results and recommendations and serving as proof of the benefits and efficacy of AI services such as Elipsa’s, something we’re keen in offering as well as part of our solutions in the future.
What Is Next?
It should be no surprise to anyone reading at this point, and rēd wireless is also happy to announce, that we have officially begun the transition of this project’s status from ‘pilot’to, now, ‘commercial’implementation. In collaboration with our geotechnical engineering colleagues and partners at Livonia, Michigan-based Rhino Wireless, and after a 100% satisfaction ratingfrom HAE & Associates, this project becomes our 2nd successful industrial IoT implementation(a score of 2 out of 2) in less than 3 years – a 1stunder the rēd wireless name; the second (and prior one) in 2018 seeing our involvement only as professional contractors for one of the largest global automotive companies in Detroit, Michigan, with equally successful completion of our wireless connectivity ‘piece of the puzzle’ in just under 6 months.Below are some of the more relevant ‘next steps’ in the ongoing development of this now proven, successful industrial IoT implementation:1)Add new sensors with specialized probes to monitor the water levels of the specific pH-solution filled tanks– As the temperature rises – or changes – within each tank, the water of the specific pH-solution slowly evaporates, something which also needs remote monitoring in conjunction with the temperature levels.2)Lab environment also needs temperature and humidity remote monitoring– we are also deploying temperature and humidity ambient sensors, not only on the lab, but all throughout the building as well - which may include other attributes such as CO2.3)Develop a new Losant/Elipsa.ai routine to identify an upward/downward temperature trend– As the temperature inside each tank trendsupward (or downward) for a specific amount of time and before a specific threshold value, identify and create a sort of ‘pre-alarm’ condition just before hitting the actual threshold limit.4)Switch to an SLA’d/QoS’d, commercial and fully-technical-support-backed LoRaWAN® Network Server (LNS)– Last but not least, although the results for this project exceeded all our expectations, the current LNS used in this pilot was NOT backed by any typical Carrier Gradecommercial Service Level Agreement(SLA) and/or Quality of Service(QoS) requirements, which is perfectly normal for ‘testing’ purposes and expected of this type of so called ‘community’ LNS servers like the one chosen. But the bigger issue is the fact that with these community LNS services there is simply no guarantee of any kind of network uptime, with minimal or no technical support availableother than public ‘forum’ online spaces - regardless of problem origin or source. As such, we were ‘left in the dark’for 4 days when our non-connectivity related API mandatory configin the week of October 26th, 2020, left us ‘AWOL’ - resulting in 93 missing messages in one week. rēd wireless was forced to deal with the situation in the most creative of ways, even though it was found to be an unforeseen internal LNS problem (read ‘not rēd’) – something totally unacceptable for any/all commercial solutions (even for home/office cases as well, at least in our professional view).For this and many other reasons, and after evaluating 6 otherdifferent commercial LoRaWAN® Network Server offerings from around the globe in the last 3 years, allrēd wireless commercial remote monitoring solutions moving forward will now be hosted and supported in the US by global IoT company and Swiss-based LORIOT– including this HAE & Associates industrial commercial solution. Among the many features and advantages of selecting LORIOT as our commercial LNS of choice and, subsequently, of rēd wireless’ offerings and solutions are : Carrier Grade, 99.9% (three nines) network SLA, end-to-end encryptedbidirectional data and device protection, LoRaWAN® AES-128encryption combined with high-grade TLS v1.2communication, ‘battle-tested’security, scalability with built-in redundancy, high-availability, minimal maintenance, MQTTSintegration (rēd wireless’ protocolof choice for communication between LNS and AEP), multi-tenancy, gateway logs, alerts and alarms and many others.