With the more advancement of hardware there comes the potential for drones to increase efficiency and perform new tasks, it’s the software side of things where most of the action is happening right now. Manufacturers are developing drones capable of flying for longer and with more precision. This means that the software has to keep up, not only with hardware capabilities, but with the demands of an expectant public [Source]. One of the need for software is for about making drones smarter, and includes things such as flight planning, automatic take-off and landing, obstacle avoidance, GPS-guided navigation and the ability to autonomously reach a given location via a series of waypoints. As a drone user these Software enabeles as to govern the interaction between drone and pilot, making the drones smarter and processing and analysis of the data acquired by the pilot which is pretty important. We emphasize the need for knowing more about these very important parts of the drone system in detail as follows.
By definition, software is any set of machine-readable instructions that directs a computer’s processor to perform specific operations. Generally speaking, application software gives you the ability to get the data that you are looking for, how and when you need it. The kind of software we have in or around drones performs special operations like: Optic flow, obstacle avoidance, simultaneous localization and mapping (SLAM), decision-making, flight planning…etc.
Drone software called the flight stack or autopilot sometimes. They are the magics playing roles in any hardware/unit in the drone’s hardware systems.
That makes these hardware systems coordinated or connected to each other or be enabled and transforming the hardware systems to be alive and be ready to take any order from the pilot or even to make sure communications between the pilot and the drone—to enable the drone performing the task as they ordered— as a whole transforming the drone from simple flying toys to smart flying robots and many more…
Drones are real-time systems (i.e. they have systems subject to a “real-time constraint”, example from event to system response)—that require rapid response to their changing sensor data. This also is a requirement which should be fulfilled by the different communications of software utilized in the drone’s hardware system.
Some of the main examples of the drones’ hardware systems with their systems software (operating systems) that can be mentioned here are: Raspberry Pis, Beagleboards, etc. shielded with NavIO, PXFMini, etc. or designed from scratch such as Nuttx, preemptive-RT Linux, Xenomai, Orocos-Robot Operating System or DDS-ROS 2.0.
Note the following two examples…
Raspberry Pis shielded with NavIO
Raspberry Pi—shielded with NavIO makes RTK on autopilot possible, here you just need a right GPS module. A GNSS receiver with carrier phase measurement called U-blox is used. What makes U-blox special among
other receivers is it’s ability to output not only coordinates, but also raw
measurement data that can be used as a raw data for the Raspberry Pi with Navio to RTK processing mostly controlled with their Arduino library, but can also use any software libraries to extend the functionality of your possible autonomous robotic project.
Generally talking about the main unit of storing and running a program in drones eventhough microcontrollers are efficient their development and debaging process is very complicated.
With raspbery pi its different. You can run any of the numerous existing lines of applications or you can use any programming language to create your own.
Autopilot is more than Just a computer—In it’s ways to determine its positions, to sense the world and to control actuators. And Raspberry pi is more powerful and extendable and the folks in EMLID and others take these advantages from raspberry pi to bring NAVIO and to give raspberry pi all this capability. Navio runs well proven Ardupilot flight stack and can operate in different flight modes including manual, stabilize, follow-me and auto. Code is executed directly on Raspberry Pi with real-time Linux kernel and you can run your applications alongside.
NAVIO comes with open source drivers examples and tutorials to get started easyly.
PXFMini hardware autopilot module. source: erlerobotics.com
The hardware part of PXFmini is an open autopiolt daugter-board for the Raspberry pi that allows everyone to create a ready to fly autopilot with support for dronecode’s APM flight stack. Today ArduPilot is almost entirely C++ and has evolved to run on a range of hardware platforms and operating system including the Pixhawk/PX4 line of autopilots, Parrot’s Bebop2, Raspberry Pi based flight controllers like NAVIO2, ErleBrain and Qualcomm’s SnapDragon. Eventhough it has been designed specially for the Raspberry Pi Zero it is also compatible with other models of Raspberry Pi family. The PXFmini daughterboard is smaller in size (15grams) yet embeds all the most elecronics complaying the drones components—including 3 axes gravity sensor, 3 axes gyroscope, 3 axes digital compass, pressure sensor, temperature sensor and an ADC. DIY drones community working with the others open source communities to come up with the de facto standard UAVs flight stacks—the popular APM and PX4 autopilots.
Note the definitions of following two terms
APM flight stack:
it is a flight stack for Arduino Pilot Mega Autopiolt daugter-board: A board made to carry the autopiloting system. It was started in 2007 by members of the DIY Drones community.
Flight stack is a collection of guidance, navigation and control algorithms for autonomous drones.
Dronecode is an open source project governed by the Linux Foundation—building and supporting a community of developers and providing them the resources and tools to help them make innovative advances in drone technology. The ultimate goal is to maximize adoption of the project’s code for the benefit of users with cheaper, better, and more reliable drone software. It serves as a trusted and neutral home to become the de facto standard platform for drone/robotics open projects. Take a look the following typical examples.
Xenomai is a real-time development project of a Free Software framework adding real-time capabilities to the mainline Linux kernel. project launched in August 2001. It cooperats with the Linux kernel, to provide a pervasive, interface-agnostic, hard real-time support to user space applications, seamlessly integrated into the Linux environment. In 2003 it merged with the Real-Time Application Interface (RTAI) project to produce a production-grade real-time free software platform for Linux called RTAI/fusion, on top of Xenomai’s abstract real-time operating system (RTOS) core.
Xenomai can help you in:
- designing, developing and running a real-time application on Linux.
- migrating an application from a proprietary RTOS to Linux.
- optimally running real-time applications alongside regular Linux applications
Drone software may vary based on the key factors in:
Drone software can classified according to their many unique characters and/or functionalities. For example:
- their architecture,
- app-based or desktop based,
- their application,
- their platform (military, civilian…),
- their service (Industri, Application),
- their type (VTOL, Fixed Wing, Rotor…)..etc
Drone software are developed according to the diverse needs of different applications paused by the drone users. Generally software can be seen in to their three main divisions. In the following section we’ll try to see the most general three drone flight stack layers. They are Firmware, Middleware and operating System. They seem the same seeing their basic features, because a single or a collection of computer programs assigned with some task to do on the machine—unless we see their specific applications—when, where and how we use them, and their outcomes.
Definition: firmware is a type of software that resides or embedded specifically on a piece of hardware and provides controlling, monitoring and data manipulation to the hardware (eg. engineered products and systems) on which it is hosted. i.e. It is a program which gives life to the device hardware or the combination of persistent memory and program code and data stored in it.
Firmware’s functionality ranges from performing basic tasks like minimal I/O controls to full fledged software system running with simple or no OS, with scheduler, memory management, etc. It is common in the hardware of devices such as optical drives, a router, a camera, a scanner or drones and others. Firmware is programmed into a special memory contained in the hardware itself and may be upgraded periodically to fix bugs and to add new functionality to the hardware component. It’s special operations in drones ranges from machine coding to processor execution, memory access…
STM32CubeMX, by STMicroelectronics, a freeware package for Windows, Mac OS X and Linux that is a graphical software configuration tool that allows generating C initialization code using graphical wizards. STM32CubeMX included in STM32Cube.
It also embeds a comprehensive software platform, delivered per series (such as STM32CubeF4 for STM32F4 series). This platform includes the STM32Cube HAL (an STM32 abstraction layer embedded software (NB: embedded software is computer software, written to control machines or devices that are not typically thought of as computers.), ensuring maximized portability across STM32 portfolio), plus a consistent set of middleware components (RTOS, USB, TCP/IP and graphics). All embedded software utilities come with a full set of examples.
A free companion of Pix4D software, Pix4Dcapture allows you to create flight plans for capturing image data. Post-flight, easily produce georeferenced maps and models in Pix4D desktop or cloud software.
MAVLink (Micro Air Vehicle Communication Protocol):
MAVLink is a protocol for communicating with small drones. It is used mostly for communication between a Ground Control Station (GCS) and Unmanned vehicles, and in the inter-communication of the subsystem of the vehicle. To ensure message integrity and ensuring the agreement of sender and receiver—an error-detecting code called CRC is calculated to every message. Here every message is identified by a unique ID, and the payload containes the data from the message. An XML document in the MAVlink source having the definition of the data stored in this payload also describes the logical ordering of the fields for the protocol. Where as the actual wire format (and typical in-memory representation) has the fields reordered to reduce Data structure alignment issues.
The pplication of MAVLink is to transmit the orientation of the vehicle, its GPS location and speed. Also used as the communication protocol in many Autopiolot projects like: ArduPiolot,Parrot AR.Drone,PX4FMU, pxIMU…etc also in some long range transmitters, and softwares.
PX4 Flight Stack:
PX4 is organized by a series of nodes that use a semantic channel like a “gesture” or “location” to communicate with a system state in a broadcast communication network. The software is divided into four main levels:
Application API: This interface is provided to application developers, such as ROS or DroneAPI. This API is designed to streamline, flat and hide its complexity as much as possible.
Application framework: This is the default assembly (node) for operating the underlying flight control.
Library: This layer contains all the system libraries and basic traffic control functions.
Operating system: The last layer provides hardware drivers, networks, UAVCAN and fail-safe systems.
The PX4 flight stack is a collection of guidance, navigation and control algorithms for autonomous drones. PX4 can integrate via two different APIs with ROS:
- Either natively each application as a ROS node or when running exclusively on the embedded autopilot via MAVLink mavros (MAVLink to ROS gateway with UDP proxy for Ground Control Station).
- APM/ArduPilot can integrate with ROS via MAVLink mavros.
The PX4 includes controllers for fixed wing, multirotor and VTOL airframes as well as estimators for attitude and position.It is a complete autopilot solution for multicopter and fixed wing aircraft. It consists of several customizable software packages. It is a complete flight control solution for multicopters, planes, VTOL aircraft or any ground robot.
Modern, mobile and desktop user interface to configure the system and execute flights.
Firmware hacking: This means to create an unofficial new or modified (“aftermarket”) version of firmware in order to provide new features or to unlock hidden functionality. This action can be ranging from as good as expanding the functionality for good, or as bad as drone wireless attacks like hijacking other’s drone by disconnecting the RC out of control.
Generally middleware is a piece of software that usually runs in the background. It is essentially is the “glue” that holds two other pieces of software together and allows them to effectively communicate, or is ‘a middle man between the OS and the hardware component’. In Platform as a Service industries many platforms offer software components as middleware which makes it easier for software developers to integrate into their applications to help simplify speed up the whole process and also to implement communication and input/output. Any driver installed on an OS can be considered middleware.
The distinction between operating system and middleware functionality is, to some extent, arbitrary. While core kernel functionality can only be provided by the operating system itself, some functionality previously provided by separately sold middleware is now integrated in operating systems. A typical example is the TCP/IP stack for telecommunications, nowadays included in virtually every operating system. At the other end of the scale, the boundary between middleware and application has also moved. source here…
When we come back to drones middleware used in collections of tools, drivers, sub systems and libraries that relate to flight control, navigation, radio, control management and ground control station(GCS)—communication and data sharing between them. They make software layer that lies between the operating system and applications on each side of distributed system in the over all drone. It contains device drivers that handle sensors and other peripherals. It also contains flight control libraries such as RC protocols, math utilities, and control filters. On the other hand the benefit of a middleware is that you can select which versions or implementations of specific algorithms for example to use in your Ground Control Station (GCS) solution, so you get more options for customization and integration of tasks.
Controls the drone motors, integrates the sensors, etc. It is A highly efficient, lightweight and blazing fast robotics communication toolkit.
The PX4 Middleware consists primarily of device drivers for embedded sensors and a publish-subscribe based (a messaging pattern where senders of messages, called publishers and receivers called subscribers) middleware to connect these sensors to applications that running the flight controls.
The use of the publish-subscribe scheme means that:
- The system is reactive: It will update instantly when new data is available
- It is running fully parallelized
- A system component can consume data from anywhere in a thread-safe fashion
Drone middleware is a middleware project for drones which is under development. It is with a modified version of the very permissive BSD license with aims to become a platform for developers, researchers and organizations that want to integrate uav data with their own processes, especially for more dynamic missions where the flight plan and circumstances change all the time, like in SAR operations.
The drone middleware can also be used to separate the usual concerns of a GCS and provide a testing and research bed on how those sub-processes can work together. This stimulates creating libraries in the process, which makes reuse in the same project and in different projects a lot easier.
The middleware also specifies a set of basic data messages that are received from other systems, like drawing lines, icons and areas that are used to specify search areas, no-fly zones or points of interest to circle. This supports the uav operator to replan their flight, as they have that data on-screen already. It has many very interesting objectives behind this project source…
An operating system (OS) is system software that manages computer hardware and software resources and provides common services for computer programs. All computer programs, excluding firmware, require an operating system to function.
An operating system for drones combines hardware, software to enable optic flow, obstacle avoidance, SLAM, decision-making and enable drone pilots to safely operate drones.
It includes on-aircraft hardware, and software enabling autonomous flight; Ground Control Station Software, aircraft flight planning, control, and monitoring; and Configuration Manager, software that enables operators to configure hardware and software for applications.
It is a comprehensive set of technology helps drone pilots companies in capturing and analyzing valuable aerial data for a wide variety of commercial applications, including infrastructure inspection, land management, public safety, environmental monitoring, surveying and mapping, precision agriculture, search and rescue, and wildlife conservation, photography.
ROS, Nuttx, Linux distributions, Microsoft IOT examples of operating systems.
A driver is usually part of the operating system of the flight controller that performs specific task of controlling a hardware component like display or usb controller, etc, and gives a interface to use the hardware by Operating system and applications.
Operating system will dictate a standard design and interface for the driver to adhere, like to initialize/deinitialize the hardware, or to read/write to the hardware, or to do any hardware specific operation, etc all of which performed using the middleware.
The Robot Operating System (ROS) is a set of software libraries and tools that help you build robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS is a partner project of Dronecode, with support for computer vision, navigation and drone simulation environments.
ardrone_autonomya ROS driver
ardrone_autonomy is an example of a drone software developed in autonomy Lab of Simon Fraser University by Mani Monajjemi and other Contributors .
ardrone_autonomy is a ROS driver for Parrot AR-Drone 1.0 and 2.0 quadrocopters. This driver is based on official AR-Drone SDK version 2.0.1. ardrone_autonomy is a fork of AR-Drone Brown driver.
The other example of ROS software is hector_quadrotor:
hector_quadrotor contains packages related to modeling, control and simulation of quadrotor drone systems
This stack provides packages related to modeling, control and simulation of quadrotor drone’s systems. The following packages are available:
hector_quadrotor_description provides a generic quadrotor URDF model as well as variants with various sensors.
hector_quadrotor_gazebo: contains the necessary launch files and dependency information for simulation of the quadrotor model in gazebo.
hector_quadrotor_teleop: contains a node that permits control of the quadrotor using a gamepad.
hector_quadrotor_gazebo_plugins: provides plugins that are specific to the simulation of quadrotor UAVs in gazebo simulation.
NuttX is a scalable Real-Time Operating System (RTOS) used in drones from 8-bit to 32-bit microcontroller environments. Real time systems are critical for flight control performance and safety, as they guarantee that flight control tasks will be completed in a certain amount of time, and are essential for the safety and time-critical performance of drones .
The most common NuttX-supported platforms:
- Almost all ARM from ARM xxx to ARM Cortex-xxx
- Atmel AVR
- Atmel 8-bit AVR (AT90USB, ATmega)
MicroChip PIC32MX (MIPS32 24Kc)
MicroChip PIC32MZ (MIPS32 M14k)
Zilog Z16F ZNeo
Zilog eZ80 Acclaim!
Most Common Planning and Post Processing softwares and services In Drone Data Collection
eventhough it doesn’t give you a preview of what the camera sees as it is its fine that you can switch between the two. It does a pretty good job with the 3D map.
Pix4D also seem to be the only one that offered software which you can run with your computer to do your own processing and also make it easy to make your own 3D video fly through it. It also automatically outputs a pdf with the full report of the landscape and mapping process.
FPV camera watchs it differently it takes over to different levels which gives you your 3D maps out of that. It can basically showing a time laps. It’s 3D processing can take many overlaping photographs and does a pretty good job except it sometimes has some fryking elements up in the sky togather with the real 3D objects. It still invite us so it doesn’t have a lot of options suitable for 3D mapping with the drone.
Drone deploy needed few photos to have a result of 3D good 3D view. It looks like it has nice options with nice suitable 3D maping environment with trying.It has some esues with elevations specially with less overlap persentage and fewer photos for a project. But considering what it had it is pretty good. it has functions tools like top view distance, location, elevation, area, volume or manage all these extra options with its 3D environment.
The good thing about dronedeploy is you can draw your plygons with different shaps. It doesn’t have to be a rectangle or a square pix4D does have to be a rectangle or a square so you may cover more area than you need. the more overlap is the better because if you don’t get enough overlap you’ll not gonna get the quality of accuracy the you actually need.
ALTIZURE needed few photos to have a result of 3D good 3D view as drone deploy do. Its 3D interface is very interesting and have good tools to control the 3D aspect of the project.
This is the only environment that had a pencil dronwing interface as an option, But there are no some 3D options or tools to do editing or riding some other actions like exporting the files in different file formats yet in the 3D environment.
The DroneMapper system is a cloud based imagery processing service that turns your 2D aerial images into high resolution geo-referenced Digital Elevation Models, Digital Surface Models, Orthomosaic Maps, 3D Point Clouds and more. Our system offers a wide array of photogrammetry tools including intrinsic calibration, camera adjustment, advanced aerial triangulation, bundle block adjustment, imagery rectification, geo-referencing, and flight tracking at a fraction of the traditional cost. The platform generates very high resolution results with off the shelf consumer quality cameras and affordable UAV hardware. Any UAV or Drone platform is supported with an attached flight controller log, or the pilot may simply geotag each aerial image.