Introduction
Low-end embedded devices in IoT networks typically utilize low-power communication protocols (e.g., BLE, ZigBee, sub-GHz, etc.) to extend battery life and conserve CPU/memory resources. To bridge the connection between these devices and the cloud, a more powerful device is often employed as a gateway device. Aside from collecting, manipulating, and forwarding data from the end devices, many gateways will allow for some type of device management. This could include services such as device provisioning, authentication, configuration, and/or firmware management.
In the article Implementing BLE OTA Firmware Updates on PSoC 6 with ModusToolbox, it was shown how the bootloader service could be added to the BLE GATT profile to enable OTA firmware updates on PSoC 6 devices. This article demonstrates how a gateway device can take advantage of this functionality to remotely update any PSoC device in the network that utilizes this bootloader service. To this end, an example application written in Python was developed to implement Cypress’s Device Firmware Update (DFU) host protocol and send a new firmware image to a target device. It is therefore assumed that the gateway device is a Linux-based system with Python installed.
Requirements
Any Linux-based device with BLE capability should work for this example. Note that some devices include integrated BLE chipsets (e.g., Raspberry Pi 4) while others will require the use of an external BLE adapter (e.g., BeagleBone Black). The example application is written in Python and utilizes the bluepy library to interface with the BLE device. The bluepy library was chosen because it is very simple to use and appears to still be supported. As a reference, the exact hardware and software components used to develop this application are listed below.
- BeagleBone Black (running Debian 10.6)
- BLE Dongle
- Python (version 3.7+)
- bluepy (version 1.3.0)
- crcmod (version 1.7)
Of course, a target device is also required to execute this example. Please refer to the article Implementing BLE OTA Firmware Updates on PSoC 6 with ModusToolbox for a complete walkthrough of adding the DFU library to a PSoC 6 device and generating the application image file.
The application’s source code is available from the GitHub repository.
Background
This section builds the foundation of the example application from the bottom-up. Starting with the Python bluepy library, the basics of connecting to a BLE peripheral and exchanging data will be presented. Next, Cypress’s DFU host command/response protocol is explained. Finally, a brief discussion about the .cyadc2 file containing the application data is provided.
Bluepy
The Python bluepy library provides an interface for communicating with BLE devices from a Linux system. It defines several objects (e.g., Peripheral, Service, and Characteristic objects) which makes it a very intuitive library to work with. The bluepy documentation provides a description of all objects, their public interfaces, and some basic example code.
To connect to a BLE device, start by instantiating a Peripheral object and providing the device’s MAC address as an argument. If the connection is successful, this peripheral object can be used to discover the device’s services and characteristics, read/write the values of these characteristics, disconnect from the device etc. However, to receive asynchronous notifications from the device, a delegate object must be assigned to the Peripheral instance using the Peripheral.withDelegate() method. This delegate object must be a subclass of the DefaultDelegate class. As a simple example, we define a class named “MyDelegate” to simply store the handle of the characteristic that sent the notification along with the data it sent.
from bluepy import btle
class MyDelegate(btle.DefaultDelegate):
def __init__(self):
super().__init__()
self.handle = None
self.data = None
def handleNotification(self, cHandle, data):
self.handle = cHandle
self.data = data
# Connect to the target device
target = btle.Peripheral("00:a0:50:f4:64:be").withDelegate(MyDelegate())
An example of scanning for devices and obtaining their MAC addresses is provided in the example application. For now, though, it is assumed that the device’s MAC address is known.
Once connected to the BLE device, the desired services and characteristics must be discovered. For this application, the Command characteristic provided by the Bootloader service (see the GATT profile tree on left-hand side of Figure 1) will be used to exchange data with the target device. Fortunately, the discovery process is simple because we know the UUID of this characteristic is “00060001-F8CE-11E4-ABF4-0002A5D5C51B” (see the parameters configuration section on the right-hand side of Figure 1).
The Peripheral.getCharacteristics() accepts the UUID of the characteristic in question as an argument. It returns a list of characteristic objects, which should contain exactly one item.
# Discover the Command characteristic offered by the Bootloader service
dfuCmdChar = target.getCharacteristics(uuid="00060001-F8CE-11E4-ABF4-0002A5D5C51B")[0]
Finally, to receive notifications from the target device, notifications must be enabled by setting the first bit in the Client Characteristic Configuration Descriptor (CCCD). This descriptor is also shown in the GATT profile tree on left-hand side of Figure 1. To obtain a Descriptor object for the Command characteristic’s CCCD, simply use the Characteristic.getDescriptors() method with the CCCD UUID provided as an argument. This UUID is easily accessible via the AssingedNumbers object as shown below. Note that, once again, a list with exactly one item should be returned.
# Discover the Client Characteristic Configuration Descriptor (CCCD)
dfuCCCD = dfuCmdChar.getDescriptors(forUUID=btle.AssignedNumbers.clientCharacteristicConfiguration)[0]
With the Descriptor object for the CCCD in hand, notifications can be enabled by writing a value of 0x0001 to the descriptor. Use the Descriptor.write() method with the new value provided as an argument (as a Bytes object in little-endian byte order). Also, the withResponse parameter must be set to True. The Descriptor.read() method can then be used to verify that the change was successful.
# Enable notifications from the target
dfuCCCD.write(b"\x01\x00", withResponse=True) # little-endian byte order
print(f"Client Configuration Descriptor written!\n> New Value (little-endian): {dfuCCCD.read().hex()}\n")
At this point, the host can send commands to the target device and receive the target’s responses. Listing 1 is an example of starting a DFU operation by sending the Enter DFU command. Lines 1 – 25 summarize the process of connecting to the target, discovering the Command characteristic, and enabling notifications. Lines 27 – 38 demonstrate the procedure for exchanging commands with the target. In the first step of this procedure, a DFU command is written to the Command characteristic using the Characteristic.write() method. Then, the Peripheral.waitForNotificaitons() method is used to block program execution until either a notification is received from the target or the timeout interval elapses. Lastly, the notification data is retrieved from the Delegate object and processed.
Listing 1: Using the bluepy library to connect to the target device, send the Enter DFU command, and receive the response
from bluepy import btle
class MyDelegate(btle.DefaultDelegate):
def __init__(self):
super().__init__()
self.handle = None
self.data = None
def handleNotification(self, cHandle, data):
self.handle = cHandle
self.data = data
if __name__ == "__main__":
# Connect to the target device
target = btle.Peripheral("00:a0:50:f4:64:be").withDelegate(MyDelegate())
# Discover the Command characteristic offered by the Bootloader service
dfuCmdChar = target.getCharacteristics(uuid="00060001-F8CE-11E4-ABF4-0002A5D5C51B")[0]
# Discover the Client Characteristic Configuration Descriptor (CCCD)
dfuCCCD = dfuCmdChar.getDescriptors(forUUID=btle.AssignedNumbers.clientCharacteristicConfiguration)[0]
# Enable notifications from the target
dfuCCCD.write(b"\x01\x00", withResponse=True) # little-endian byte order
print(f"Client Configuration Descriptor written!\n> New Value (little-endian): {dfuCCCD.read().hex()}\n")
# Send the Enter DFU command packet to the target
enterDFUPacket = bytes.fromhex("01 38 04 00 04 03 02 01 B9 FF 17")
print(f"Sending command to characterisitc with handle {dfuCmdChar.getHandle()}...")
print(f"> Command: {enterDFUPacket.hex()}\n")
dfuCmdChar.write(enterDFUPacket)
# Wait for response from the target
target.waitForNotifications(2) # timeout after 2 seconds
# Get the response from the target
print(f"Received resopnse from characteristic with handle {target.delegate.handle}!")
print(f"> Response: {target.delegate.data.hex()}\n")
Running the code provided in Listing 1 results in the output shown in Figure 2. The structure of the command and response packets, as well as the procedure for updating the device’s firmware, will be discussed in the next section.
Figure 2: Screenshot of the output resulting from running the program provided in Listing 1
Host Command/Response Protocol
The DFU host interacts with the target device using a simple command/response protocol, which is well-documented in Appendix B of [1]. Command packets are sent by the host and response packets are sent by the target. An example of this was just shown in the above section (Figure 2) in which the Enter DFU command packet was sent by the host and the target responded with a packet indicating successful execution of the command. Let’s look at these packets in more detail.
Figure 3 shows the structure of the command packets sent by the host. Note that the Data Length and Checksum fields are in little-endian format while the byte order of the “N bytes of data” field is dictated by the command itself. The Enter DFU command, shown explicitly in Figure 2 as “0138040004030201b9ff17”, can then be decomposed as shown in Table 1. Comparing this to the Enter DFU command documentation in section B.2.1 in [1], it becomes clear how these values where chosen. The command byte is 0x38 and 4 bytes of data are provided as input. These bytes are the product ID and will be interpreted by the Enter DFU command as 0x01020304.
Table 1: Decomposition of the Enter DFU command packet sent by the host in Figure 2
| Start of Packet | Command | Data Length (N) | N Bytes of Data | Checksum | End of Packet |
|---|---|---|---|---|---|
| 0x01 | 0x38 | 0x0004 | 0x04030201 | 0xFFB9 | 0x17 |
The packet checksum is calculated by simply adding all the preceding bytes together and taking the 2’s compliment of the result. Listing 2 provides an explicit example of a simple Python function capable of constructing command packets given only the command byte and the command payload (see the output in Figure 4).
Listing 2: An example python function which constructs host command packets
import struct
START_OF_PACKET = b"\x01"
END_OF_PACKET = b"\x17"
def createCommandPacket(cmd, payload=b''):
# Get the payload length
payloadLength = len(payload)
# Construct the packet up until the checksum field
packet = struct.pack(f"<ccH{payloadLength}s", START_OF_PACKET, cmd, payloadLength, payload)
# calculate the checksum
cs = 0
for byte in packet:
cs += byte
cs = -cs & 0xFFFF # 2's compliment
# Complete the packet by adding the checksum and end of packet byte
return packet + struct.pack("<Hc", cs, END_OF_PACKET)
if __name__ == "__main__":
CMD_ENTER_DFU = b"\x38"
dfuEnterPacket = createCommandPacket(CMD_ENTER_DFU, bytes.fromhex("04030201"))
print(f"Enter DFU Packet: {dfuEnterPacket.hex()}")
Figure 4: Screenshot of the output resulting from running the program provided in Listing 2
The response packets returned by the target device have the same structure as the command packets except a 1-byte status code takes the place of the command byte, as shown in Figure 5. A success code of 0x00 indicates that the preceding command executed successfully on the target. All other codes indicate some type of error (see Table 11 in [1] for the complete list).
In Figure 2, it was shown that the target device responded to the Enter DFU command packet with “010008000000000000000401F2FF17”. Table 2 breaks this packet down into its individual fields. Notice that the target returned a status code of 0x00 (success) and 8 bytes of data which, according to section B.2.1 of [1], can be interpreted as JTAG ID: 0x00000000, Device Revision: 0x00, and DFU SDK Version: 0x010400. The checksum can be verified in the same way as the command packet (see lines 14 – 17 in Listing 2).
Table 2: Deconstruction of the Enter DFU response packet sent by the target in Figure 2
| Start of Packet | Status Code | Data Length (N) | N Bytes of Data | Checksum | End of Packet |
|---|---|---|---|---|---|
| 0x01 | 0x00 | 0x0008 | 0x0000000000000401 | 0xFFF2 | 0x17 |
Now that the host can construct DFU commands and verify the response packets, it can now be used to update the target device’s firmware. The basic procedure for doing so is outlined with the flowchart in Figure 6. The blue elements represent DFU commands sent by the host and the others are essential data preparation steps. Note that not all the available DFU commands are used in this procedure. Again, see Appendix B of [1] for a complete list of available DFU commands.
Figure 6: Flowchart of a typical DFU operation performed by the host to update the target’s firmware (DFU commands in blue)
Enter DFU
This command starts a DFU operation. If any other commands are sent before this command, they will be ignored by the target device.
Set Application Metadata
There is a dedicated region of the target device’s flash memory set aside for storing information about its applications (see Appendix D of [1]). For each application, the application start address (4 bytes) and application length (4 bytes) are stored in this section, the values for which are provided by the host via the Set Application Metadata command. An example of such a command packet is: “014C09000100000510FCFF000099FD17”. Here, the data bytes are interpreted as Application: 1, Application Start Address: 0x10050000, and Application Length: 0x0000FFFC (65532 bytes). Note that these values were obtained from the application image file discussed in the Application Image File section.
Send Data/Program Data
At this point in the procedure, the host begins reading row data from the application image file and sending that data to the target device. A checksum must accompany this data so the target can verify its integrity before writing it to memory. This checksum is calculated using the CRC-32C algorithm. A simple way to get this value is with Python (Listing 3). Note that for simplicity, the Application object provided with the Example Application is used to read the first row of application data.
Listing 3: Calculate the CRC-32C checksum of a row of application data
import crcmod
import sys
from cydfu import Application
# Open the application image file and read the first row of data
app = Application(sys.argv[1])
rowAddr, rowData = app.getNextRow()
# Calculate the CRC-32C checksum of the row data
crc32cFunc = crcmod.predefined.mkCrcFun('crc-32c')
checksum = crc32cFunc(rowData)
After calculating the checksum for the entire row of data, it is common practice to split the data into smaller pieces and send these pieces to the target one at a time. This prevents the command packets from becoming too large and starving the communication channel. The downside of this, however, is it will take longer for the host to transfer the application to the target device. Therefore, the size of the data pieces should be as large as the application/network can tolerate.
When splitting the row data into n smaller pieces, the first n-1 pieces are sent by the host using the Send Data command. When the target receives this command, it will simply append the received piece of data to a buffer without writing it to memory. The last piece of row data (piece number n) should be sent with the Program Data command along with the checksum calculated in Listing 3 and the memory address the row of data should be written to. When the target receives this last piece of data, it will append it to the buffer to complete the row, verify the entire row with the checksum, and write the row data to the specified address.
Verify Application
After the last row of data is successfully sent to the device, it is a good idea to follow up with the Verify Application command. This command tells the target to calculate the checksum of the entire application and compare it to the one stored in flash memory. As we will see in the Application Image File section, this checksum is stored in the last 4 bytes of the file.
Exit DFU
Finally, to end the DFU operation, the host will send the Exit DFU command. Once the target receives this command, it will again ignore any commands that follow (unless it is the Enter DFU command).
Application Image File
In the article Implementing BLE OTA Firmware Updates on PSoC 6 with ModusToolbox, the process of creating a PSoC 6 application capable of being remotely loaded onto a target device is outlined. At the end of this process, after building the application, an application image file is generated as a project artifact. This file has the extension “.cyacd2” and it contains the loadable application firmware in an ASCII hex format. An example of such a file is provided in Listing 4. Note that only the first and last rows of data are shown for brevity. A complete description of the .cyacd2 file format can be found in Appendix C of [1].
Listing 4: Example of a .cyacd2 application image file (only showing first and last data rows)
010000000000000104030201
@APPINFO:0x10050000,0xfffc
:0000051000A40008D10205100D0000004503051045030510450305104503051000000000000000000000000000000000410305104103051000000000410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510
:00020510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051041030510410305104103051010B5054C237833B9044B13B10448AFF300800123237010BD342D000800000000649F051008B5034B1BB103490348AFF3008008BD00000000382D0008649F051070477047FFF7FCFF72B6144C144DAC4209DA21686268A368043BA2BFC858D050FAE70C34F3E70F490F4A00209142BCBF41F8040BFAE70D480D490860BFF34F8F00F046FC00F028FCFFF7DBFF08F03CFF08F064FC08F024FFFEE70000709F0510889F0510342D0008FC3100080024000808ED00E0FEE7FEE700B504207146084202D0EFF3098001E0EFF30880043007F0D1FEFEE70230800803D001300238FCD100BF00BF7047EFF3108072B6704780F31088704738B504461D4B1B7813B90278012A20D0012B27D0194B1B78012B02D0C3B96368B3B1174D284605F04DF96168022938BF0221C1F16401134A07EE901AF8EE677A17EE901A284605F007F90C4B01221A7038BD0B4805F036F9084B01221A7064236360D7E72378002BD4D1054805F042F90023024A13706360CCE700BF502D0008
.
.
.
:004E0810F0B4234B1B685C681968D3F8903019440CE020BF1F4B1B7803B120BF1D4B01221A700B6803F00303022B08D11A4A136943F0040313610128EBD030BFF1E7D4F80C311BBB15490E68154BD3F81855154A106841F61E770F600621C3F818153E2111604422C3F81C250D4BD3F81C35002BFADA4FF0AA33C4F80C31084B1E60A3F58473C3F8185503F582731860F0BC7047F46100087C52000800ED00E00801264000002640040126405FF800F0C99005105FF800F075AA05105FF800F0698C05105FF800F0FD8D05105FF800F07B0305105FF800F0459105105FF800F0730305105FF800F0AD8505100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002D731357
The first lines of the file do not contain application data, but rather application metadata. First is the header row with the structure shown in Table 3. The header in Listing 4 indicates that the File Version is 1, the silicon ID is “00000000”, the silicon revision is 0, the checksum type is 0, the app ID is 1, and the product ID is “01020304” (little-endian). Recall that the product ID is sent to the target using the Enter DFU command and the app ID is sent to the target using the Set Application Metadata command.
Table 3: .cyacd2 header row structure (derived from [1])
| File Version | Silicon ID | Silicon Revision | Checksum Type | App ID | Product ID |
|---|---|---|---|---|---|
| 1 byte | 4 bytes | 1 byte | 1 byte | 1 byte | 4 bytes |
The second line is the Application Verification information. It begins with the string “@APPINFO:” and provides the application start address and the application length. Recall that these are the values sent to the target device via the Set Application Metadata command. In Listing 4, the start address is 0x10050000 and the length is 0xFFFC bytes (i.e., 65532 bytes). Note that, strangely, these values are in big-endian format.
The remaining rows in the file are the application data rows. Each of these rows corresponds to a row of memory in the target device which stores the application firmware. In .cyacd2 files, these rows have the structure shown in Figure 8. Again, the row address is in little-endian format.
Table 4: .cyacd2 data row structure (derived from [1])
| Header | Address | Data |
|---|---|---|
| 1 character “:” | 4 bytes | N bytes |
The last line of row data is special because the last four bytes contain the checksum for the entire application. This can clearly be seen in Listing 4. When the host sends the Verify Application command to the target device, the target will calculate the checksum of the entire stored application firmware and compare it to these last four bytes. The target’s response to the host indicates whether the checksums where a match.
The Example Application
Using the concepts described above, an example python application called update.py has been created to demonstrate how a Linux system can be utilized as a host to update a remote target device. The source code is available from the GitHub repository.
Usage
This application depends on the bluepy and crcmod packages, so be sure they are installed in your environment. By default, the application expects a virtual environment to be setup as follows:
$ sudo apt install python3-venv python3-pip libglib2.0-dev
$ python3 -m venv env
$. env/bin/activate
(env) $ python3 -m pip install bluepy crcmod
(env) $ deactivate
Setting up the environment in a different manner may require the first line of the update.py file to be modified. The usage of the application is as follows:
$ update.py application_file [target_MAC_address]
The first argument is required and is the path to the application image file described above. The second argument is optional. If the MAC address of the target device is known, the user can provide it here and skip the process of scanning for available devices. If the second argument is not provided, the application will present the user with a list of available BLE devices from which the user must choose a device to update. An example of executing the application without the second argument is shown below in Figure 7.
If the second argument (target MAC address) is not provided, the application will have to be executed with root privileges.
Structure
The application consists of two python modules (files). The update.py module contains the main function and defines two classes: Target and ScannerUI. The Target class is simply a generalization of bluepy’s Peripheral class, inheriting all of its functionality while adding a method called updateFirmware() (Listing 5). This method implements the procedure outlined in Figure 6 to update the target device’s firmware. It also prints status information to the user as shown in Figure 8.
Listing 5: The Target.updateFirmware() method definition
def updateFirmware(self, app, maxDataLength=512):
crc32cFunc = crcmod.predefined.mkCrcFun('crc-32c')
hostCmd = cydfu.DFUProtocol(self)
# Send the Enter DFU command
print("Starting DFU operation...")
hostCmd.enterDFU(app.productID)
print(f"> Product ID: 0x{app.productID:08X}\n")
# Set Application Metadata
hostCmd.setApplicationMetadata(app.appID, app.startAddr, app.length)
print(f"Application {app.appID} is {app.length} bytes long. Will begin writing at memory address 0x{app.startAddr:08X}.\n")
# Send row data to target
print("Sending Data...")
while True:
try:
rowAddr, rowData = app.getNextRow()
except Exception:
break
# Calculate the CRC-32C checksum of the row data
crc = crc32cFunc(rowData)
# Break the row data into smaller chunks of size maxDataLength
rowData = [rowData[i:i+maxDataLength] for i in range(0, len(rowData), maxDataLength)]
# Send all but the last chunk using the Send Data command
for chunk in rowData[:-1]:
hostCmd.sendData(chunk)
# Send the last chunk using the Program Data command
hostCmd.programData(rowAddr, crc, rowData[-1])
print(f"> Sent Data Row {app.currRow}/{app.numRows}")
print("Finished sending application to target.\n")
# Send Verify Application command
print("Verifying Application...")
result = hostCmd.verifyApplication(fwImg.appID)
if result == 1:
print("> The application is valid!")
else:
print("> The application is NOT valid.")
# Send the Exit DFU command
print("Ending DFU operation.")
hostCmd.exitDFU()
At this point, the ScannerUI object is unpolished and meant for demonstration purposes only. It works in tandem with the btle.Scanner object to present a simple user interface consisting of a table of discovered devices and a prompt for the user to choose a device to connect to (Figure 7).
The second module is called cydfu.py . It also defines two classes: DFUProtocol and Application. The DFUProtocol class encapsulates the host command/response protocol described above. The Application class handles the extraction of data from the application image file.
Conclusion
This application has been designed to remotely update the firmware of a PSoC target device utilizing Cypress’s DFU middleware library. Though not perfect or complete by any means, it serves as an example of using a Python BLE library to exchange data with a peripheral device according to the DFU specification and perform OTA firmware updates. These core concepts can, of course, be repurposed to perform OTA updates on devices from other manufacturers (e.g., STMicroelectronics, Microchip, Renesas) as well. To that end, this application is a great starting point.
References
[1] M. Ainsworth, “PSoC 6 MCU Device Firmware Update Software Development Kit Guide,” Appl. Note 213924, pp. 44-53, 8 December 2018.