The Four Fundamentals: Remote Software Engineering with Embedded Systems

If the COVID-19 pandemic has taught us anything about the workplace, it is how vulnerable we are to factors beyond our control. It has also shown us how lucky many of us are to have at our disposal myriad technologies that allow us to work remotely from the safety of our homes.

Software engineers are certainly fortunate in that they can continue working and contributing to the economy through lockdown and shelter-in-place orders. In fact, a recent Deloitte paper states that, “In contrast to the parts shortage and unpredictability of supply chains dragging down the industry, software is a growth catalyst.”

As reassuring as this evaluation may be, it should not blind us to the challenges we face developing and delivering software for the duration of the pandemic and beyond. ­­We must assume that for the foreseeable future our work will involve a great deal less personal contact, hence a great deal more remote work and a new definition of what it means to work in the lab.

In short, the lessons we learn and the measures we implement to get through the pandemic will also stand us in good stead after the pandemic is, thankfully, behind us.

Software development is indeed insulated, but the work we do eventually does require hardware. This is particularly true of embedded systems, where sooner rather than later, we must prove our work on specific target boards. To compound the problem, we often work with pre-production boards, which are usually rare commodities—sometimes only a handful exist, worth far more than their weight in gold, and only parsimoniously shared by the manufacturers, even inside their own organizations.

Plan to Share

Given that software engineers must work remotely, and that the boards with which they must work are likely to be even more difficult to obtain than in the past, the first order of business should be to plan to share. That is, set the boards up in secure locations with reliable and secure Internet connections so that they can be readily accessed by multiple users.

Unless other constraints, such as security, require it, these locations don’t have to be a formal lab; someone’s home may do perfectly for a board or two, as long as they aren’t accessible to the two-year-old, the parrot or the dog. If necessary, consider scheduling times slots for different users. When making schedules, remember that different people prefer different times of the day; ask for preferences, and if your organization spans multiple time zones, leverage this to minimize any inconvenience to your teams.

Connect, and Connect Securely

Wherever you locate a board, you’ll need to connect with it. Your board set up and your connection should ensure that:

• You can easily contact the board from your remote host system
  
• You have a secure tunnel through firewalls to the board
  
• You can easily reset or, if necessary, cold boot the board

Assuming it has an appropriate port—and most boards do—establishing contact with a remote board is quite simple with protocols such as Telnet or Echo. These protocols are useful to confirm that you have made contact with the board you need, but they aren’t secure.

For testing and debugging, you should probably use another port with Secure Shell (SSH) or some other technology that can provide a secure channel. This requires a bit more work than Telnet. You need to query your subnet to get the IP addresses you’ll need, and work with your router to set up port forwarding to your reserved IP address.

The brief webinar “How to Keep Your Projects on Track While Working Remotely, Part 1” offers an example of how to set up an SSH connection to an embedded target. Once you have established a secure connection, you can work with your preferred tools to test and debug your software on the hardware on which it will need to run.

Finally, plan your connection strategy for failure; specifically, plan for things going wrong with the software—after all, this is development. You will need to reset your boards remotely. Set up relays to the board reset switches, and consider connecting your boards to remote power IP switches, so that when software fails you can initiate a cold boot remotely. You can always purchase a remote relay board, but since you do embedded development, you can also build one yourself.

The commands below illustrate how easy it is to control a reset power switch remotely. This setup uses a secondary control board that can control a set of isolating relays. You can use a general purpose I/O (GPIO) driver to manage some output pins on the target like this:

# echo 2 > /dev/outs; sleep 1; echo 0 > /dev/outs

# echo 1 > /dev/outs; sleep 3; echo 0 > /dev/outs

The first command above is for a reset, while the second is for a cold reboot, so it has a three second sleep to give all the hardware time to shut down before power is applied again. To complete the setup, connect relay control on the associated GPIO pins to the relays controlling the reset pin, and the power supply of the controlled target. This simple configuration ensures that you are always able to remotely push the target’s reset button or pull the plug.

Keep Your Data

You can use stored data to advance your project without the need for field trials. For example, if you are developing a subsystem that controls high-speed trains or LRTs and are far along enough to have run some trials, you should have the live sensor data from these trials stored in a database. You can feed this data into your system as though it were coming from the sensors in real time, so you can advance development of the algorithms you use to integrate the data from diverse sensors in order to provide everything behind the sensors: position, acceleration, etc., they need to learn how to control your vehicles.

The figure below presents a vehicle control system that stores raw sensor data for use in the lab:

Figure 1. Sensor data can be stored in a database for use later in distributed labs, reducing the need for work in the field.
 

This approach allows you to leverage recorded sensor data and play it back remotely, as well as share it with a distributed team. You will of course need to return to the field, or have a remote engineer with the embedded system periodically collect additional sensor data. However, the more you can do with the data you already have to hone your algorithms and prove your applications are ready, the further along you will be with your project when field trials become possible again.

Combine Local and Remote Strategies

Just as you can use stored data, you can also use simulation to advance software development when you have limited access to hardware. UI development is an excellent candidate for simulation; not least important is that you may be developing a UI design that must be consistent across multiple targets and architectures, such as ARM and x86.

It may be best to plan a two-phase approach:

• Simulation on a development host for initial development and debugging
• Target boards for final testing, debugging and tuning

Key advantages of simulation are that it is relatively easy to implement and that in a simulated environment your software is easy to control, debug and change, which also make it an excellent strategy for storyboarding user interactions: which button to touch, what to swipe, how to respond to an unknown user action, etc.

However, simulation has its limits. The host hardware is not the target hardware, so there will never be any guarantee that software in a simulated environment will behave exactly as it would on a specific target. For starters, the host hardware is likely to be more powerful than the target hardware, to say nothing of differences in board revisions, and in peripherals such as screens. These differences make it difficult if not impossible to finish tuning a UI or any other software without running it on its intended target.

Several strategies are available for working directly with UIs on remote targets. A straightforward, non-invasive technique is to simply set up a camera to feed a video of your UI back to your host system. Your feed is unlikely to be crisp, but it will show your UI’s actual behavior while running on the target hardware. Another unobtrusive technique, but one that will provide crisper, more accurate video is to insert an HDMI capture card in the video monitor stream. Make sure you get a card that matches the specifications of your HDMI monitor, or some resolution modes may be cropped in the video feed.

A more invasive but very powerful technique includes adding a Virtual Network Computing (VNC) server on the target system. This technique allows the embedded target to share its display with your host system remotely. It provides crisper images than a video feed, and it has the advantage of allowing user interaction (e.g., keyboard, mouse) with the target UI. It is intrusive, however; you will likely have a lower frame rate than you would if you were working directly on the target or with one of the less intrusive techniques noted above.

If you are using an unobtrusive method of video capture, you still need to interact with the target. The figure below illustrates how a distributed lab can be set up for engineers to work with UI devices on a remote target. Note that for the host system’s devices, events appear on the remote target as UI events, and that the remote target UI treats these event as though they were local events:

Figure 2. A distributed lab set up to allow engineers to work remotely with devices on a target.
 

In short, your remote interface testing and tuning strategy should probably employ several methods, as each has its benefits. You can advance your development in a simulated environment, but you should also plan to complete your work with a secure tunnel to your target. The trick is to decide how much you can do with simulation, and how much you’ll need to do on the remote target, and what techniques best serve your requirements.

Plan for the long-term. Plan for a distributed lab model where targets are shared by engineers and developers across the globe. Set up secure connections, keep your data, and organize your projects so that your targets are available when they are needed. Ultimately, someone will need to spend time co-located with the targets, if only to set up some reset switches and connect the boards to remote power IP switches. Know that the more efficient you are remotely,  the more your remote strategy will contribute to the success of your projects.