Creating robotics software can be complex and challenging. This is true whether you’re building an autonomous mobile robot (AMR), an assembly-line cooperative robot (cobot), an agricultural field-task robot, or a next-generation robot designed for any of a thousand specific applications, from surgical, to search and rescue. It’s difficult to have robots see, move, and act in ways that humans do naturally while ensuring robots have the programmability, repeatability, and robustness we expect from machines.
What’s even tougher is building the right software architecture for a robot, because so many contradictory forces are at work. A good robotics software stack must support rapid prototyping, so ideas can be tested and rejected or refined quickly. The software stack must also support safety certification – dynamically at odds with rapid prototyping, yet increasingly required for situations where robots interface with or work alongside people. While the software architecture must help you get to market quickly, it shouldn’t encourage “throw-away” code. A robotics software stack needs to be stable, robust, and reusable. But it also must easily support open and collaborative development, such as integration of new vision algorithms or environmental mapping systems.
In this article, we discuss how to build a robotics software stack that addresses these sometimes-competing pressures in a balanced way. Notably, our experience is based on our robotics customers and how they use BlackBerry® QNX® software. Because we think our operating system and other BlackBerry® products are an optimum fit for robotics, we will describe specific use cases in terms of applicable BlackBerry QNX solutions. However, our requirements aren’t BlackBerry exclusive, so in most cases, you can apply the same principles to the software components and vendors of your choice.
There are eight attributes for a robotics software stack that let you build robots rapidly – and reliably:
- Compatible with ROS (robot operating system)
- Supports rapid control prototyping and “hardware-in-the-loop” testing capabilities
- Real-time capable
- Easily certified
- Contains a hypervisor
- Manages cybersecurity
- Consistent platform
- Pre-packaged distribution
Let's take a look at each of these attributes in more detail.
1. Compatible with ROS
A robot operating system (ROS) isn’t really an operating system, but rather a collection of libraries and middleware components for writing robotics software. ROS includes a pre-packaged set of the most common tools needed – geometry, mapping, goal seeking, navigation, vision systems, and diagnostics. It provides remote procedure calls (RPCs) for client/server communications as well as a decoupled publish/subscribe API (application programming interface), allowing you to develop a very modular and adaptable software architecture. ROS can also help you visualize your robot’s states, allowing you to view and debug a machine’s internal representation of its environment. It’s an excellent starting point for building out the complete software stack a robot needs.
Since the common functionality to make robots sense, plan, move, and act is already developed and available within ROS, a robotics development team can make immense progress rapidly. And since ROS software is being actively refined by a broader community, your engineering staff can benefit from other people’s fixes, improvements, optimizations, and additions, while your team focuses on the “secret sauce” that differentiates your design from others.
The libraries included with ROS run on an operating system, which means the OS you choose for your robot must support ROS. While ROS supports Ubuntu® Linux® out of the box, ROS can run on real-time operating systems, too. This can provide real-time behavior that is critically important for some applications. The QNX® Neutrino® RTOS and the QNX® OS for Safety both support ROS 2.
2. Supports rapid control prototyping and hardware-in-the-loop testing capabilities
A hardware engineer building robots is undoubtedly familiar with MathWorks® products MATLAB® and Simulink®, used ubiquitously in robotics research, algorithm development, and product deployment. Whether it’s signal processing analysis, kinesthetic input methods, perception system simulation, high-level autonomy, or hardware-in-the-loop (HIL) testing, MATLAB and Simulink products have a huge number of math tools that robots need. MathWorks software packages are used by engineers everywhere to develop the mathematics behind robotic products and prototypes.
Robotics software engineers know the value of linking these key building blocks between the workstation environment and embedded targets. By providing a pre-integrated version of Simulink on the target, robot hardware becomes a seamless part of the engineering design and development workflow. Algorithm designs can be developed and simulated virtually first, and then executed directly on the embedded targets as soon as the hardware is available. The hardware can verify the behavior of the models in the messy real world and collect data to help reshape erroneous assumptions and refine algorithms with real-life data.
MathWorks and QNX worked together to enable this powerful software combination on their respective platforms, so bringing these powerful mathematical analysis and visualization tools to your robotics project is that much easier.
3. Real-time capable
Like people, robots need real-time reactions to external inputs so they can see obstructions or targets, properly position arms and appendages, or move to a goal or away from an obstacle. However, the minimum latency that meets real-time requirements is highly dependent on the type of application and the level of reliability, as well as on the combination of OS and hardware.
What sorts of tasks require a real-time reaction? Reading encoder wheels, sampling A/D converters, running stepper motors, controlling actuators, and processing camera buffers are a few. These hardware tasks turn into higher-level activities like robot joint estimation, fine-grained map determination, and visual object categorization. Ensuring these tasks operate consistently – avoiding dropped readings, mislocated positions, skewed camera outputs, or shuddering joint motions – is one of the reasons real-time performance is so critical.
It’s notable that a non-real-time OS like Linux can often meet latency targets with enough spare and non-budgeted CPU time. However, a real-time operating system (RTOS) can often use deterministic scheduling to achieve reliable results on slower (and lower cost) CPUs. An RTOS allows the hardware team to select lower-powered hardware that can still meet latency requirements, letting the software team focus on building robotic applications rather than engineering around timing-introduced corner cases.
4. Easily certified
The next necessary attribute is an ability to certify your stack easily. A robotics software stack should be easily certified with a safety standard such as IEC 61508, IEC 62304, ISO 10218, or ISO26262. A robot should be safety-certified whenever it performs a mission-critical task. For example, an autonomous security robot, automated construction crane, space exploration robot, or personal care cobot all need to have their controlling software vetted as thoroughly as possible. This includes any applications where robots are working around or cooperatively with humans.
Safety certifications aren’t a silver bullet. They don’t magically prevent robots from making mistakes or harming people. They do ensure that software requirements, development, testing, and deployment will follow proven risk mitigation procedures that maximize the ability for the team to eliminate bugs and design flaws early. The process of pursuing a certification also significantly helps an organization build the formal software development processes that are a trademark of a mature organization. While it might be fun to be slinging code during a start-up’s birth, building consistently reliable robots requires a mature software process, once the freewheeling stage is over and the emphasis turns to testing, quality, and validation.
Many robotics engineers select initial robot software components with little thought to certification, since it can seem too far in the future when beginning development. Having a clear path to certification is advisable from the start, so it can be more easily achieved if and when it’s needed. Many clients turn to QNX when they realize it would be too impractical to certify their Linux-based software stack – too costly, lengthy, difficult, or all three. Since QNX operating systems are POSIX-compliant (API-compatible with Linux), we’re able to show them an easy path to certification. QNX helps our clients navigate the challenging certification process, and provides in-depth safety consultation, documentation, and guidance throughout.
5. Contains a hypervisor
Hypervisors make it easier to mix system criticality. With a hypervisor, you can have safety-critical software running on the same CPU as the user interface, without requiring safety certification for both. Companies building embedded systems with hypervisors do so primarily because it allows them to get a certification for just the safety-critical subset of the software, and it fulfills both safety and time-to-market goals.
While hypervisors are a commonly deployed technology in functionally safe systems, they are rare in today’s robotics. Having a hypervisor in your robotic software stack is recommended, since we expect certification will soon become more commonplace in robotics. Much of this will occur through enforced governmental regulations as cobots become more commonplace, exposing issues of unregulated machinery in a human-interacting capacity. This extends not just to human and robot coworkers, but also to robots as surgeons and caretakers. Companies that differentiate their products based on quality or safety will also drive a stronger need for robotic hypervisor solutions.
If certified software is better, why not certify the entire robotics software stack? Simply put, it’s not practical. Robot user interfaces may rely on third-party components that cannot be independently certified, like GUI technology, voice recognition engines, or speech synthesis libraries. They may have interfaces to cloud back-ends for capturing data and diagnostics, or a web-based user-control portal – parts that include massive amounts of SOUP (software of unknown provenance). It is more effective to isolate a robot’s control systems and safety critical features within a hypervisor, and keep any GUI, voice, or cloud components outside the safety critical realm.
6. Manages cybersecurity
From financial and brand-reputation standpoints, nothing is more problematic for a company than to have its products hacked on the world stage. Robots unfortunately provide a new and fruitful venue for hackers wishing to disrupt or profit from their misdeeds. With mobility and the ability to manipulate physical objects, a hacked robot potentially poses more real-world harm than hacked servers do today. Imagine a cobot attacking targeted employees in a warehouse, a medical robot that “forgets” to dispense life-saving medication, or an industrial robot that deliberately builds cars with weak welds to cause crashes.
Robotics software needs to incorporate cybersecurity. While cybersecurity is a multi-faceted discipline that must include engineer training, corporate processes, and responsive infrastructures, only the technology portion used to develop and run the robot software is discussed here. First, we recommend adding software to the robotics tool chain (such as BlackBerry® Jarvis®) that scans source code and binaries for vulnerabilities. Second, we suggest adding components on the embedded target that can detect, log and defend against unusual and unauthorized activity. Third, we recommend ensuring changes to your cybersecurity practice aren’t just tacked on at the end of your development, but are present throughout, to ensure universal adoption and better overall protection.
7. Consistent platform
Depending on how they’re architected, modern robots may contain several processors. A CPU may be dedicated to one task such as limb management, executive functioning, user interface, navigation and mapping systems, communication, or remote control. Alternatively, a CPU may bundle several (or all) of these functions.
Creating a single unified software stack and consistent hardware platform – regardless of the role of the processor in the target machine – streamlines software creation, reduces errors, and maximizes effort. A consistent software and hardware platform:
- Prevents proliferation of different development tools, software libraries, shim and abstraction layers, operating systems, and APIs that are solving the same problem
- Ensures hardware and software optimizations and errata/bug workarounds that are applied to one system will work across all, without requiring additional independent fixes
- Allows software processing tasks to be allocated and relocated within the robot as capacity needs dictate, rather than being forced by hardware limitations or capabilities
- Makes it easy to centralize computing power, something that can be done near the project end to right-size processor needs and reduce cost (often with the help of hypervisors or other partitioning techniques)
Rather than mix microprocessors and higher-power processors, it’s usually better to use several lower-powered, general-purpose processors to fulfill multiple roles. With a consistent platform, the software, tooling, debugging techniques, and test fixtures always remain the same for the development staff, no matter where in the project they’re working.
Creating such a platform also helps quicken future product development. Maybe you think your robotic vacuum could be refitted as an agricultural weed puller? In that case, keeping the same basic platform and extending it with another processor (of the same type) for image processing and arm control lets your team maximize the already developed software investment.
8. Pre-packaged distribution
Some engineers love to manage their own Linux distributions. While it gives a team ultimate control of their operating system’s destiny, it’s also a decent amount of work to keep the “distro” up to date, periodically build (and/or fix build scripts), and internally package, label, and deploy. Is it worth it? To answer that, it’s interesting to note that few large companies maintain their own distribution. Similarly, as start-ups move to scale-ups, they too usually progress from a “roll-your-own” mindset to acquiring standardized solutions.
The main reason is that someone else is already an expert at creating OS distributions, whether they be Linux, QNX, or other. Getting your OS or RTOS pre-packaged frees your team to work on tasks more directly involved with your product’s value. It also puts the responsibility for bug fixes, improvements, and vulnerability patches within complex and heavily used code onto experts who manage these aspects every day, and not your engineering staff who might occasionally need to delve into it. We always recommend using an existing packaged distribution regardless of the OS you choose.
Concluding Thoughts
As robots are increasingly used in mission-critical situations – interfacing with and working alongside people as cobots – their smooth, reliable operation is vital. Mission-critical robots include those that impact the user organization’s bottom line. A problematic robot that shuts down a factory due to defective parts might not directly harm humans, but it may decimate the company. Robots can also become mission-critical when they are able to intersect public spaces. An autonomous photography drone might not be considered mission critical until an unexpected failure causes its several pounds of rapidly spinning metal and plastic to fall from the sky into a public park.
These eight attributes provide a starting list of best practices for robotics software selection. As most developers know, a lack of planning creates more work in the long run. Software component selection is often done incidentally at a project’s onset, based on how fast a prototype can be using freely downloaded GitHub sources. Applying the same rigor used during development to the selection of software components, and having the ultimate robotics software stack right at the beginning, can save everyone time and money.
Read more about how BlackBerry QNX can help your company build state-of-the-art robotics systems.
