I earlier covered a 3 part story on evolution of software in our 7 years long journey at Zenatix (Part1 — Early Version, Part2 — Closing the loop with control and analytics and Part3 — All about modularity and scale).
Hardware has played the role of a younger brother to Software at Zenatix. I wrote an article around 4 years ago about why doing hardware is HARD in India and our hardware philosophy at Zenatix. In another article, I covered why an integrated Hardware-Software development approach is critical. I then wrapped up the 3-part series (written in October, 2016) with an article giving a broad overview of the technology backend we were developing to support deployment and troubleshooting in a scalable manner.
Early part (initial 4–5 years) of our journey at Zenatix, has been focused on developing hardware components that can help us scale reliably. The key focus perspectives with which we developed the hardware were:
- Use off-the-shelf hardware wherever cost and feature requirements suffice: This thought process led us to use Raspberry Pi as the core edge processing element. We gave internet access to this system using a combination of a TP Link MR3020 router and a standard off the shelf Huawei dongle (plugged into the USB port of the router). We never tried to make an energy meter of our own since a RS485 based energy meter was already a commodity. For controlling large loads, we used standard contactors rather than using larger relays.
- Develop sub-components that add to hardware reliability: We developed our own USB-Modbus converter since off-the-shelf converters were not reliable (keeping in mind isolation etc.). We developed an external watchdog and an external RTC for Raspberry Pi to give ruggedness in the developing country context. We made our own RS485 based relay nodes since control was critical and we wanted to add reasonable protection in there without compromising on the price.
- Develop hardware that gives significant cost advantage compared to off-the-shelf ones: We made our own WiFi based temperature sensors since the price points in the market were 4–5x the price at which we could produce.
These perspective did not come easy. On the way, we experimented (and failed) with many other hardware designs too that never went into large scale deployment.
Early on, we tried making our own power supply to give enough current to Raspberry Pi (for handling cellular connection) only to realise that the power supply is better left to those who have been doing it for decades. For the initial few deployments, we tried controlling loads through large relays on a PCB only to realise that its not reliable at scale. Like many out there, we also conceptualised and prototyped a GPIO hat for raspberry Pi that may serve multiple purposes but realised that mechanical aspect and GPIO communication on Raspberry Pi can not be handled reliably at scale. The temperature sensors that we used with our WiFi chipset were earlier directly communicating with Raspberry Pi over GPIO. This again was not pursued further due to scalability challenges that were realised in deployment. We also extended our RS485 based relay nodes to a prototype multi-purpose RS485 based card supporting use cases of monitoring UPS battery voltage, door sensor, ground voltage (among others) only to realise that such generalised cards are often confusing to deploy (as they are often used for a subset of use cases that they support in total), difficult to iterate quickly and also difficult to maintain (from both hardware and firmware versioning perspective).
All through the journey, the focus had been on hardware that can be easily deployed and maintained at scale.
One of the biggest failures have been the WiFi based IR control card (that we designed for controlling the air conditioning). The whole project took almost a year and after the initial manufacturing (making and deploying almost 200 of the devices), we shelved the project because of the complications in deployment associated with it . This project was initially conceived to make the whole monitoring and control very easy to deploy (to an extent make it DIY). While the project was a failure (after initial field prototypes), it helped build the right perspective on how should we think about the whole plug and play hardware which eventually set the tone for what we have now built over the last 3 years (or in a way, a second phase of our hardware journey). More on this in my subsequent post later on.
The hardware has to be well supported by the firmware running on the hardware and the software running on the cloud (that will be used to maintain the hardware). We already talked (in this post) about collecting a lot of health metrics and using them for:
- Automatically verifying if the deployment was done correctly by running a set of proprietary tests at the end of each deployment and enforcing a process that the electrician on site only leaves when all tests pass.
- An automated Issue framework that analyses these health streams to then automatically create tickets for the troubleshooting team, giving them the first insight about what is wrong with a specific piece of hardware on a given site.
Remote login (using ssh) and over the air (OTA) updates were put into practice from the very early stage to ensure that we can address the problems observed in the field in a scalable manner.
Since the ambition was to scale to millions of devices, we were consciously (from the very early days) building technology components that can help deploy and maintain at scale. This integrated hardware-software development approach was first discussed in this post 4 years ago and then recently covered under Device Management Framework in this post.
Hardware/Firmware has always been a smaller team (stretching at max to 3–4 people at any point of time). We have also experimented with outsourcing hardware design to a couple of experienced individuals but failed in these engagements (for multiple reasons). A thesis we have, thus, built over the years is
Hardware design should be done by people who have reasonable experience in doing so. Even though this may be expensive upfront, overall it is more efficient from both cost and time perspective.
After we partnered with HEPL in 2018 for the next phase Zenatix journey, we got introduced to their consumer IoT arm in Qubo and the semiconductor business in Tessolve. Deeper discussions with these businesses helped us firmly develop the hardware vision for the next phase of Zenatix growth. I will cover it in a follow up article soon.
