PRBS23

Reverse Engineering the Gravity 800 Smoker Controller

prbs23@prbs23.com (prbs23) — Thu, 13 Aug 2026 21:05:23 -0600

I recently released FreeFall 800, open source firmware for the Gravity 800 smoker/grill from Masterbuilt. The project started because I needed to open the controller to fix a broken wire. That is when I noticed that inside was a standard ESP32 module, with a plainly visible programming header. This made me very surprised to find that there were little to no details online about the controller’s internals. In this post I aim to document everything I managed to reverse engineer about the Gravity 800 controller board, and will discuss some of the very strange choices the designers made.

Gravity 800 Controller Board. (I apologize for the image quality in this post, I thought my phone camera was better)

At a high level the Gravity 800 controller board has the following major components:

ESP32 WiFi microcontroller module
Four character, 16 segment LCD display with backlight
Five button and knob encoder physical inputs
Piezo buzzer for temperature and timer alarms
Grill lid open switch
Thermocouple ADC for smoker chamber temperature, with thermistor for cold junction measurement
Four thermistor temperature probe ADC inputs
12-volt blower fan motor driver

Unfortunately on first investigation, other than the MCU which is specifically an ESP32-WROOM-32UE module, and the LCD driver which appears to be a CS1621, the rest of the ICs are either unmarked or do not have specific enough markings for me to find data sheets or documentation. This meant that not only did I need to reverse the PCB connectivity to build FreeFall 800, I also ended up having to reverse engineer the serial protocols, at least enough to emulate them.

I will go through the reverse engineering process in detail below, but in the end I did end up with a full schematic of the Gravity 800 controller board. Some component values are missing (mainly unmarked capacitors and inductors, since I didn’t want to desolder them), but references should be accurate. A KiCAD project with the schematic is published in my reverse engineering repository here: prbs23/gravity-800-reverse-engineering.

Full schematic of Gravity 800 controller board

Power Supply

The easiest place to start reversing the board was the power input. Not surprisingly, there does not appear to be anything too interesting going on with the power supply.

Power Supply Schematic

On the input side we have a reverse voltage protection diode and a couple of capacitors to produce the 12 volt rail that both feeds the 3.3 volt regulator and is used directly for the blower fan and buzzer. I was not able to identify what specific voltage regulator is being used, but it appears to be a pretty standard adjustable switching regulator configured to output 3.3V.

Microcontroller

After tracing the power supply, I moved on to the other part of the board that looked familiar: the ESP32 microcontroller. Specifically the controller uses the 8MB flash variant of the ESP32-WROOM-32UE module. In addition to the integrated WiFi and Bluetooth radios, the module has 32 GPIO that can also be used with various UART, SPI, PWM, and ADC peripherals.

Obviously the first thing to do was to connect to the programming header and dump the firmware. The programming header was conveniently labeled “Programming” and had the following pin out:

Pin	Description
1	RX
2	TX
3	GPIO0/BOOT
4	GND
5	3.3V

Like all ESP32s the programming header does double duty as a serial debug interface while the processor is running. You can boot into the programming boot loader by tying GPIO0/BOOT low while applying power to the controller. In the first (of several) annoying decisions made by the designers, the reset (EN) of the ESP32 is simply pulled high on the board, and not exposed on the programming header. This means every time we want to program the part, it’s necessary to power cycle the board. That must have been annoying for their firmware developers.

Dumping the firmware was as simple as installing espflash and running:

espflash read-flash --port /dev/ttyUSB0 --baud 115200 --chip esp32 0 0x800000 firmware_dump.bin

There is no flash protection or encryption enabled on the controller, so I now had the full flash image. I did end up doing some basic decompilation and tracing of the firmware in Ghidra and Claude. However, other than some specific thermocouple calibration functions (discussed more later), I did not find much else in the firmware that was worth the effort of tracing in detail.

The serial debug output did not have anything particularly helpful for reverse engineering. However it does expose some details of the infrastructure the mobile app is built on top of. Once connected to a WiFi network, this is a snippet of the (hopefully anonymized) messages that were logged:

(4890) wifi:new:<11,0>, old:<1,0>, ap:<255,255>, sta:<11,0>, prof:1
(5690) wifi:state: init -> auth (b0)
(5700) wifi:state: auth -> assoc (0)
(5700) wifi:state: assoc -> run (10)
(5740) wifi:connected with XXXX_SSID_XXXX, aid = 7, channel 11, BW20, bssid = xx:xx:xx:xx:xx:xx
(5740) wifi:security: WPA2-PSK, phy: bgn, rssi: -72
(5750) wifi:pm start, type: 2

event_info.disconnected:11,event_id=4
(5840) wifi:AP's beacon interval = 102400 us, DTIM period = 2
F_Rd_AdRegt[4]: 6291455
(7270) BH: b5 80 00 00 00 
(7360) Main: DRAM:101540
(8660) esp_netif_handlers: sta ip: ddd.ddd.ddd.ddd, mask: 255.255.254.0, gw: ddd.ddd.ddd.ddd
event_info.disconnected:63,event_id=7
(8660) WiFi: SYSTEM_EVENT_STA_GOT_IP
(8670) WiFi: Got IP: ddd.ddd.ddd.ddd

(8670) AWS: start aws
keyAD=4092,coseblue=0,upStep=0,showback=0
(8960) Main: wifi_shown_contr=0,b4byte3bit1=0
(9280) BH: b5 86 00 00 00 
(9670) AWS: Connecting to AWS...
rssi:-73

(10360) Main: DRAM:59132
F_Rd_AdRegt[4]: 8388607
(11280) BH: b5 86 00 00 00 
(12810) AWS: Connected to AWS ok
(12810) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/jobs/get/+
(13280) BH: b5 8e 00 00 00 
(13330) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/get/rejected
(13360) Main: DRAM:59392
keyAD=4093,coseblue=0,upStep=0,showback=0
(13960) Main: wifi_shown_contr=0,b4byte3bit1=0
F_Rd_AdRegt[4]: 8388607
(15280) BH: b5 8e 00 00 00 
(15480) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/get/accepted
(15990) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/update/document
rssi:-75

(16360) Main: DRAM:61320
(16500) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/update/rejected
(16810) AWS: SubscrTopic:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/update/delta
(17280) BH: b5 9e 00 00 00 
F_Rd_AdRegt[4]: 8388607
(17920) AWS: desired------
(17930) AWS: Update Status:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/update   Msg:{
	"state":	{
		"reported":	{
			"model":	"C:G:P26:1:D",
			"vers":	"3.0.01",
			"update":	0,
			"trigger":	0,
			"mac":	"xxxxxxxxxxxx",
			"pwrOn":	false,
			"fah":	true,
			"engaged":	true,
			"doorOpn":	true,
			"mainTemp":	999,
			"heat":	{
				"t2":	{
					"heating":	false,
					"intensity":	0,
					"trgt":	225,
					"max":	700,
					"min":	150
				}
			},
			"probes":	null,
			"errors":	[0, 0, 0, 0, 0],
			"timers":	null,
			"ssid":	"XXXX_SSID_XXXX",
			"RSSI":	-75
		},
		"desired":	null
	}
}
(17970) AWS: Update Status length=507
(18000) AWS: IoT_data_tsend 0
(18000) AWS: Request Get:$aws/things/<XXX_DEVICE_ID_XXX>/shadow/get
(18220) AWS: get accept callback

.... Repeat from keyAD line

Serial log output from factory Gravity 800 firmware

The most obvious thing we can see from this log is that they are using the AWS IoT Core managed MQTT message broker. Both from the log messages, and also some light decompiling it appears they are likely using the Espressif esp-aws-iot library for doing this. Interestingly, this trace was captured while the controller was in the “off” state. So even when “off” the controller is still sending the full device state to AWS and Masterbuilt, including your WiFi router SSID and RSSI.

Pinout

MCU Schematic

While this was interesting, it was not particularly helpful for my reverse engineering effort. The next step was tracing all the pins of the ESP32 module to find what they were connected to. This is the full list of pin connections I ended up with after fully reverse engineering:

Pin	Name	Direction	Destination	Function
25	GPIO0	Input	Programming Header	Boot mode
34	U0RXD	Input	Programming Header	UART RX
35	U0TXD	Output	Programming Header	UART TX
10	GPIO25	Input	Lid switch	Lid switch (closed=low)
27	GPIO16	Output	Buzzer	Buzzer pulse (4kHz)
28	GPIO17	Output	Fan Driver	Fan enable (active high)
30	GPIO18	Input	Knob Encoder	Encoder A
29	GPIO5	Input	Knob Encoder	Encoder B
31	GPIO19	Input	Knob Encoder	Encoder Push Button (unused)
16	GPIO13	Output	Display Controller	Backlight Enable (active low)
33	GPIO21	Output	Display Controller	Display CS (active low)
36	GPIO22	Output	Display Controller	Display WR (SCK)
37	GPIO23	Output	Display Controller	Display SDA
12	GPIO27	Output	Thermocouple ADC	Thermocouple ADC SCK
26	GPIO4	Bidir	Thermocouple ADC	Thermocouple ADC SDA
8	GPIO32	Input	UART ADC	ADC UART RX
7	GPIO35	Output	UART ADC	ADC UART TX

One thing that I was surprised NOT to find while probing the ESP32 pins was any of the control push buttons. Given the number of free pins I was expecting them to be directly connected, but after some tracing I found that the push buttons appeared to be multiplexed, and connected to what I have been calling the UART ADC.

UART ADC Subsystem

Tracing the push buttons I ended up looking at the U3 chip on the controller, an unmarked 16-pin SOIC. This chip is connected to the ESP32, through what appeared to be a 9600 baud UART interface. While both TX and RX are connected between ICs, the ESP32 factory firmware never appears to send anything, and U3 automatically starts sending out a 16 byte message every 36 milliseconds when power is applied.

Other than power and some decoupling, the remaining pins of U3 are connected to: The four temperature probe thermistor inputs, a board mounted NTC thermistor (which I eventually determined was used for the cold thermocouple cold junction temperature), and… the push buttons for the controls. This all points towards U3 being an ADC; however, it is pretty rare for an ADC to use 9600 baud UART instead of SPI or I²C. Of the few UART ADCs I could find, none matched the description of U3. My best guess is that it’s a cheap MCU programmed to behave as an ADC for the ESP32. Why would they not just use a SPI or I²C ADC? Why not just use the ADC inputs of the ESP32?

Weird design decisions aside, I captured a couple of trace samples of the 16 byte messages the “UART ADC” was sending out:

Byte #	0	1	2	3	4	5	6	7	8	9	10	11	12	15
Frame 1	E1	0F	FD	0F	FA	0F	FB	0F	FB	0F	FA	08	03	1E
Frame 2	E1	0F	FC	0F	FB	0F	FB	0F	FB	0F	FB	08	03	1F
Frame 3	E1	0F	FC	0F	FB	0F	FC	0F	FA	0F	FB	08	03	1F

Three captured frames from the UART ADC

Every message starts with a 0xE1 start byte, followed by what appears to be 7 (odd?) two byte ADC sample values, and ending with a check byte. Bits [15:12] of each sample value are zero, so everything is pointing towards this being a 12 bit ADC. The check byte appears to be calculated as: check_byte = sum(frame[0:15]) % 256. This would make the frame format look something like:

UART ADC Frame Format

The seven inputs of the ADC are mapped as follows:

Pin	Channel	Input
15	CH0	Multiplexed push buttons
14	CH1	Thermistor probe 1
13	CH2	Thermistor probe 2
12	CH3	Thermistor probe 3
11	CH4	Thermistor probe 4
10	CH5	Thermocouple cold junction thermistor
9	CH6	Unused, tied to GND

This schematic shows the whole UART ADC subsystem, with button multiplexing and thermistor conditioning.

UART ADC Subsystem Schematic

Multiplexed Push Buttons

Despite having plenty of free IO on the ESP32, and there only being 5 push buttons on the board, the designers chose to multiplex all the buttons through a single ADC input on the UART ADC. The ADC input is pulled up with a 7.5kΩ resistor, and then all 5 buttons pull the input down with a different strength resistor. The table below lists the pull down strengths, expected output voltage, as well as ADC thresholds I ended up using.

Button	Pulldown	Nominal Voltage	ADC Value	Positive Threshold	Negative Threshold	Window Size
None	N/A	3.3V	4095	N/A	3971	124
Timer	100kΩ	3.069V	3810	3931	3452	379
Up	22kΩ	2.461V	3054	3412	2375	1037
Temp	5.1kΩ	1.336V	1657	2335	1312	1023
Enter	2.2kΩ	0.748V	928	1272	624	648
Down	510Ω	0.210V	260	574	0	594

To prevent button confusion, in my FreeFall 800 firmware, I chose my thresholds such that there is a 40 ADC code window between each button’s window where I do not detect any button. Due to the structure used, detecting multiple buttons pushed at once is sort of possible, but supporting it would narrow the ADC value windows for single button presses. I have not done the math to figure out which combinations are uniquely detectable.

Thermistors

The other five (used) inputs to the UART ADC all come from NTC thermistors, which use the same structure: A voltage divider with fixed pullup and the thermistor as the pull down, followed by an RC low pass filter.

The probe inputs use a 47kΩ pull up resistor, and the thermistors have a nominal 230kΩ resistance at 25°C. From the decompiled factory firmware, it appears to use a fixed β value of 4331, so the temperature can be resolved using the following formula:

$$ T_0 = 25.0{\degree}\text{C}, R_0 = 230k\Omega, \beta = 4331, R_{high} = 47\text{k}\Omega $$ $$ R_{probe} = -\frac{R_{high} \cdot {adc\_value}}{{adc\_value} - 4096} $$ $$ T_{probe} = \frac{1}{\frac{1}{T_0 + 273.15} + \frac{1}{\beta} * \log(\frac{R_{probe}}{R_0})} - 273.15 $$

Equation for calculating probe temperature from raw ADC value

The final thermistor input is a board mounted NTC thermistor that is used as a cold junction temperature measurement for the thermocouple temperature calculation. This thermistor has a nominal 10kΩ resistance at 25°C, and uses the same input structure with a 10kΩ pull up resistor. The β value used by the factory firmware in this case is 4104. Because the same input structure is used, the same equation can be used to calculate temperature.

From my testing, the physical implementation of this cold junction thermistor has some design problems. The most obvious is that it is not located anywhere near the actual thermistor junction. A bad design decision, but not the biggest issue. The bigger problem is they used a tiny 0603 SMT thermistor, which, with a nominal 165μA passing through it, self heats to about 5-10°C above the board nominal temperature. This error then gets inferred into the thermocouple measurement, giving a constant offset.

Thermocouple ADC

Unlike the temperature probes, the Gravity 800 uses a thermocouple to measure the smoker chamber temperature. This is handled by an independent single input differential ADC IC that has the reference IC4 on the controller board. Much like the UART ADC this chip did not have any useful identifying markings, and also had a weird serial interface.

Thermocouple ADC Schematic

The thermocouple itself is connected to the ADC with a fairly standard 1MΩ biasing and low pass filter circuit. On first inspection, it appeared that the serial interface to the ESP32 was a unidirectional SPI-like interface, where the ESP32 would periodically clock a block of data out of the ADC. But after trying some initial testing under this hypothesis, the serial data line proved to use more complex bidirectional signaling.

Oscilloscope trace capture of single thermocouple ADC transaction. Green trace ESP32 side of R105, blue trace ADC side of R105.

By probing the SDA signal on either side of resistor R105 and looking at the edge rate on either side, I was able to determine which side of the interface was driving SDA. From that observation the sequence appears to be:

ADC pulls SDA low
ESP32 sends 27 clocks, during which the ADC shifts out 27 bits of sample data
On the next clock cycle the ADC stops driving SDA and the ESP32 starts driving
ESP32 drives 8 clock cycle value of 0x56
One cycle interface turn around where ESP stops and ADC starts driving SDA again.
ESP32 clocks 9 more cycles of status data out of ADC.
Wait for ADC to pull SDA low again.

From my analysis this is the most accurate representation of the protocol:

Thermocouple ADC serial protocol

However, on the first transaction after coming out of reset, the factory does something slightly different. Presumably this is initializing something in the ADC. Instead of sending a fixed 0x56 command, it sends a 16 bit value of: 0xcb 0x04

First Thermocouple ADC serial transaction after reset

Once I was able to get a 22-bit value out of the ADC, that appeared to change with temperature, I just needed to figure out how to map it to a temperature. This is where I needed to reverse engineer the factory firmware, because I had no real information on how the ADC measurements mapped to temperatures. Most of the firmware I did not fully annotate or trace, but I did identify one function which appeared to handle the calibration and correction of the thermocouple measurement.

The factory firmware appears to convert the ADC value into degrees fahrenheit with a linear scaling factor (0.00390543072717 °F/LSB), adds the cold junction temperature from the board mounted NTC, and then applies a piecewise linear correction factor. The correction function as well as a plot of its correction is shown below. Because the factory firmware does all its calculations in fahrenheit, I am showing both their coefficients in fahrenheit, and the coefficients used by FreeFall 800, which does its math in celsius.

$$ T_{corF}(T_{uncorF}) = T_{uncorF} + \begin{cases} 0 & \text{if } T_{uncorF} \leq 50{\degree}F \\ 30 \times \frac {T_{uncorF} - 50} {250 - 50} & \text{if } 50{\degree}F \leq T_{uncorF} \leq 250{\degree}F \\ 30 & \text{if } 250{\degree}F \leq T_{uncorF} \leq 400{\degree}F \\ 30 + (41-30) \times \frac {T_{uncorF} - 400} {600 - 400} & \text{if } 400{\degree}F \leq T_{uncorF} \leq 600{\degree}F \\ 41 & \text{if } 600{\degree}F \leq T_{uncorF} \\ \end{cases} $$ $$ T_{corC}(T_{uncorC}) = T_{uncorC} + \begin{cases} 0 & \text{if } T_{uncorC} \leq 10{\degree}C \\ 16.667 \times \frac {T_{uncorC} - 10} {107 - 10 } & \text{if } 10{\degree}C \leq T_{uncorC} \leq 107{\degree}C \\ 16.667 & \text{if } 107{\degree}C \leq T_{uncorC} \leq 204{\degree}C \\ 16.667 + (22.778-16.667) \times \frac {T_{uncorC} - 204} {316 - 204} & \text{if } 204{\degree}C \leq T_{uncorC} \leq 316{\degree}C \\ 22.778 & \text{if } 316{\degree}C \leq T_{uncorC} \\ \end{cases} $$

Piecewise linear correction function extracted from factory firmware.

Thermocouple correction curve

It is unclear where this correction curve comes from, it does not match any thermocouple calibration curve I was able to find. So I have to assume this is correcting for some non-linearity in the sampling circuitry, not the thermocouple itself. Additionally, since this is using a relatively coarse linear piecewise function, I suspect it is possible to get more accurate temperatures by fitting a continuous function to calibration data. However, I cannot quantify if this is worth it, since I have not yet collected the required data to test it.

This correction function is applied to the scaled and offset ADC value. The full thermocouple conversion function is shown below:

$$ T_{scaleF} = 0.00390543072717{\degree}F $$ $$ T_{scaleC} = 0.00216968373732{\degree}C $$ $$ T_{offset} = 0 $$ $$ T = T_{cor}(({ADC Sample} * T_{scale}) + T_{coldJ} + T_{offset}) $$

Full thermocouple temperature conversion from factory firmware.

Absent better calibration and correction data, this is also temperature calculation that I implemented in FreeFall 800.

Display

The last major component of the controller board is the LCD display and controller (U31). Unlike most of the other components, I was actually able to find a datasheet for the LCD controller that matched its pin out: CS1621. The English CS1621 datasheet is missing some serial protocol details, but I found that the HT1621 appears equivalent and has a better datasheet. This made reverse engineering this part much easier than the ADCs, the main thing to figure out was the mapping of segment buffer bits to specific LCD segments.

LCD display and controller schematic

The serial interface on the CS1621 is almost SPI, and fortunately it is possible to use the SPI peripheral in the ESP32 for the way it is used on this board. Technically the interface is made up of chip select (pin 9), bidirectional serial data (pin 12), and then pins 10 and 11 are inverted read and write enables. To write a data bit to the controller the WR pin is pulled low, and then data is latched on the rising edge. To read a bit from the controller the same is done on the RD pin. However if you only need to write to the controller, you can treat the WR pin as a SPI clock signal. Since the RD pin is not connected to the ESP32, clearly this is what the designers did. (I will note they probably should have a pull up on RD, but it seems to work okay un-driven)

Excerpt from CS1621 datasheet showing data and command transaction formats

Segment Mapping

The display content is updated by using the “WRITE mode” command to update the display buffer. The factory firmware actually writes the display buffer twice in a single command, first it sends 16 bytes of 0s to clear the buffer, and then 16 bytes of actual segment state data. I am not sure this is required now that I have the datasheet, but I wrote the FreeFall 800 LCD driver before I found an English version of the datasheet so I just replicated what the factory firmware did. The table below shows the full mapping of segments:

	COM0	COM1	COM2	COM3
SEG0	CH3_E	CH3_F	CH3_H	CH3_A1
SEG1	CH3_D1	CH3_K	CH3_G1	CH3_I
SEG2	CH3_L	CH3_G2	CH3_J	CH3_A2
SEG3	CH2_D2	CH2_M	CH2_C	CH2_B
SEG4	CH2_L	CH2_G2	CH2_J	CH2_A2
SEG5	CH2_D1	CH2_K	CH2_G1	CH2_I
SEG6	CH2_E	CH2_F	CH2_H	CH2_A1
SEG7	CH3_D2	CH3_M	CH3_C	CH3_B
SEG8	CH1_E	CH1_F	CH1_H	CH1_A1
SEG9	CH1_D1	CH1_K	CH1_G1	CH1_I
SEG10	CH1_L	CH1_G2	CH1_J	CH1_A2
SEG11	CH1_D2	CH1_M	CH1_C	CH1_B
SEG12	CH0_E	CH0_F	CH0_H	CH0_A1
SEG13	CH0_D1	CH0_K	CH0_G1	CH0_I
SEG14	CH0_L	CH0_G2	CH0_J	CH0_A2
SEG15	CH0_D2	CH0_M	CH0_C	CH0_B
SEG16	BT	HT2	SMK	HT1
SEG17	MP1	MP2	MP3	MP4
SEG18	TEMP	TIME	RLD	COL

The segment names map the the screen layout like the following:

  RLD        TIME        TEMP        MP1        MP2        MP3        MP4

 --A1-- --A2--      --A1-- --A2--         --A1-- --A2--      --A1-- --A2--     HT1
| \    |   /  |    | \    |   /  |       | \    |   /  |    | \    |   /  |
F  H   I  J   B    F  H   I  J   B  COL  F  H   I  J   B    F  H   I  J   B
|   \  | /    |    |   \  | /    |       |   \  | /    |    |   \  | /    |    SMK
 --G1-- --G2--      --G1-- --G2--         --G1-- --G2--      --G1-- --G2--
|   /  | \    |    |   /  | \    |       |   /  | \    |    |   /  | \    |
E  K   L  M   C    E  K   L  M   C  COL  E  K   L  M   C    E  K   L  M   C    HT2
| /    |   \  |    | /    |   \  |       | /    |   \  |    | /    |   \  |
 --D1-- --D2--      --D1-- --D2--         --D1--  -D2--      --D1-- --D2--     BT
.             .    .             .       .             .    .             .
+---- CH3 ----+    +---- CH2 ----+       +---- CH1 ----+    +---- CH0 ----+

Initialization and Configuration

When the controller comes out of reset it needs to be configured for the connected LCD. This is done with a sequence command of 12 bit command writes, where each command starts with a 0b100 command prefix. The configuration command sequence that the Gravity 800 factory firmware (and thus FreeFall 800) uses is:

	Command Word	Function
1.	`0x852`	Configure bias generator for 1/3 bias voltage with 4 commons.
2.	`0x830`	Enable internal RC oscillator with 250Khz frequency.
3.	`0x800`	Disable oscillator and bias generator.
4.	`0x80a`	Disable watchdog timer.
5.	`0x802`	Enable oscillator generator.
6.	`0x806`	Enable bias generator

Backlight

The LCD backlight is driven through a high side P-channel MOSFET, connected to GPIO13 (module pin 16) of the ESP32. Because of the high side switch circuit used the backlight is enabled when GPIO13 is pulled low, and disabled when high. Annoyingly they did not include a pull up resistor on the MOSFET gate, so until the ESP32 powers on and configures GPIO13 as an output the backlight will partially turn on.

Miscellaneous

The rest of the controller is a variety of miscellaneous passive circuits

Knob Encoder

Encoder Knob Schematic

The knob is connected to a pretty standard 2 switch mechanical rotary encoder, with pull up and series resistors driving pins GPIO18 and GPIO5. Weirdly, the board has the pads and resistors to support a push button encoder, with the push button connected to GPIO19, but the actual mounted encoder is not pushable. In theory you could swap the encoder for a pushable one and have an extra user input option.

Lid Switch

Lid Switch Schematic

The grill lid switch uses the same pull up plus series resistor input structure for driving GPIO25.

Fan Driver

Fan Driver Schematic

The blower fan is driven from the 12 volt supply using a high side 20P03 30V/20A P-Channel MOSFET. The gate of the 20P03 is driven from a second low side N-Channel MOSFET, so that the 3.3V ESP32 can switch the 12V motor using GPIO17. The motor output itself has a LC filter and clamp diode to protect the rest of the controller.

Buzzer

Buzzer Schematic

Finally, the piezo buzzer is also driven directly from the 12 volt supply, and is connected to GPIO16 of the ESP32 through a low side N-Channel MOSFET. The factory firmware always drives the buzzer with a 4kHz frequency. Presumably you could use other frequencies, but for the time being I have kept with the same configuration.

Future Work

At this point pretty much everything on the Gravity 800 controller board is fully understood. There are a few things I still want to do in this area though:

Reverse engineer more of the Gravity series controller boards. When I published the FreeFall 800 repository, one common piece of feedback I got was people wanting to run it on the Gravity XT controller. This is a substantially different controller with a full graphical LCD; but from the FCC filings, it appears to still use a (newer) ESP32 module. If I can get my hands on an XT, or any other model, gravity series controller I would be interested in reverse engineering them too.
More information on the thermocouple or UART ADCs. I have enough information to get data out of these ADCs, but if anyone has any more information about these parts, or parts with similar looking interfaces I would love to know!
There is still more work to do on FreeFall 800! I am currently looking into adding MQTT support, improved configuration and calibration, among other things to FreeFall 800. Additionally, in the process of writing this post and doing some deeper research into the LCD controller, I now suspect that there’s a more efficient way I could be updating the LCD.

Announcement: FreeFall 800 (v0.1.0)

prbs23@prbs23.com (prbs23) — Sat, 25 Jul 2026 14:20:53 -0600

Today I’m publicly releasing a project I’ve been working on for a while: FreeFall 800. FreeFall 800 is a local first, open source firmware project, that replaces the factory firmware that comes on the Masterbuilt Gravity 800 smoker grill.

A year ago I was in the market for a new grill, and ended up purchasing a Masterbuilt Gravity 800 charcoal smoker grill. The grill itself has been great. However, like most “smart” devices, it has one fatal flaw: Without the internet, manufacturer’s servers, and their mobile app, you cannot make use of any of the “smart” remote control or monitoring features.

FreeFall 800 aims to fix that. Once you replace the factory firmware with FreeFall 800 the grill will locally serve its own web app you can access from any phone or computer. This web app gives you essentially all the same remote control and monitoring features you get from the Masterbuilt app, but without any dependency on the internet or someone else’s servers. FreeFall 800 currently supports:

Monitoring the chamber and probe temperatures, with live plotting of temperatures and fan speed, and downloadable history
Settable timer and probe target temperatures, with audible alarm
PID-controlled set temperature, with tunable gains
Option to disable the lid switch interlock that normally stops the fan when the lid is open
Thermocouple and thermistor calibration, initially migrated from factory firmware calibration
Supports both standalone WiFi access point mode and joining your home network, with mDNS for automatic host name resolution
Over-the-air firmware updates

Masterbuilt Gravity 800 Smoker Grill

I started this project when I needed to repair the power cord, which had been pulled out of the controller. Taking apart the controller, I quickly found that not only was it using the popular ESP32-WROOM-32UE wireless MCU module, but the programming header for it was easily accessible on the back of the controller board. That was the point at which I knew I needed to reverse engineer this thing and write my own firmware. I will go into the reverse engineering in a separate post, but after a couple months of tinkering my custom Rust based firmware provides almost all the features of the original firmware, and I am releasing the whole thing under the GPL v3 license.

Getting Started With FreeFall 800

First, a few warnings: It is possible the FreeFall 800 firmware will damage your controller or grill. Only install this on your Gravity 800 if you are willing to take the risk. Always back up your factory firmware image, in case you need it for recovery.

This project is definitely still in an “alpha” quality state. Expect that you will likely encounter bugs. If you are willing to, please report them.
This firmware was developed specifically on a controller with hardware revision ZH-PCB-331802-C-V09, and factory firmware version 3.0.01. If you run this on a different hardware revision, no promises are given that it will work, but if it does I would love to know.
No thought has yet been put into security or authentication in this firmware. Anyone who can connect to your device can control it, so whatever you do, don’t expose it to the internet.

Building Firmware

If you are one of those people that like to build everything from source, you can do just that with the following commands:

Install Rust via rustup (this provides cargo).
Install espup ESP32 Rust toolchain installer:
```
cargo install espup --locked
```
Install the Espressif Rust toolchain:
```
espup install
```
Source additional environment variables:
```
source ~/export-esp.sh
```

Build firmware:

git clone https://gitlab.com/prbs23/freefall_800.git freefall_800
cd freefall_800
cargo build --release

If you don’t want to bother with building the firmware yourself, you can just download it from the repository release page: FreeFall 800 Releases

Flashing Firmware

To load FreeFall 800 onto your Gravity 800, you will need a 3.3 volt USB-Serial adapter and a couple of jumper wires.

Note: Using a 5V or other voltage serial adapter could damage your controller. Some adapters are configurable, make sure you are using 3.3 volts.

To get access to the programming header, remove the controller from your grill, and remove the back cover. You should find a programming header shown in the image below. The pinout for that header is the following:

RXD - USART RX
TXD - USART TX
BOOT - Boot/Flash Mode, IO0 - Pull low for boot loader
GND
+3.3V

Connect RXD, TXD and GND to a 3.3V USB-Serial adapter. You can connect BOOT directly to ground if you just want to program the controller, if you want serial debug while firmware is running, connect to GND with a switch.

Programming header connection setup.

Once connected you can run the following commands to flash FreeFall 800 onto your controller:

Install espflash:
```
cargo install espflash --locked
```

Back up current firmware:

espflash read-flash --port /dev/ttyUSB0 --baud 115200 --chip esp32 0 0x800000 firmware_dump.bin

Tie “Boot Mode” pin (pin 4) on programming header to ground, then power up the controller from its normal power supply (the serial adapter does not power the board). Screen should stay blank.
Load firmware:

If you have already built the firmware from source, you can just run:
```
cargo run --release
```
If you are not building from source, download the latest “Firmware Image” from the repository release page. And then run:
```
espflash flash --erase-parts otadata --monitor --chip esp32 --port /dev/ttyUSB0 freefall_800_fw
```
Release the boot pin, and power controller off and back on. You should see the firmware booting in the console. You can now unplug programming header and device should run on its own.

Once you have initially flashed your controller with FreeFall 800, future firmware updates can be done over the network using the OTA firmware images, also available from the release page.

Connecting

If your controller had previously been connected to your network then FreeFall 800 should automatically reconnect to the same network when it starts up for the first time. However, if your controller was not connected to a network, or was connected to a network that is not present anymore, it will fall back to operating as its own WiFi access point, creating its own network with the name freefall_800_XXXXXX (where the Xs come from the device MAC address), and the password Gravity800.

Once your computer or phone is connected to the same network as the controller, point your browser at freefall800.local and it should load the FreeFall 800 web app:

FreeFall 800 Web App

From here you can fully control the grill, monitor temperatures, set alarms, and configure the controller.

Of course the physical front panel controls still work as well. Not all the configurability available through the web app is possible through the front panel, but you can set the target and probe alarm temperatures and control the timer without opening the app.

Limitations

In its current v0.1.0 state, FreeFall 800 provides almost all of the functionality provided by the Masterbuilt app. But there are a couple of limitations:

FreeFall 800 cannot send notifications to connected devices when the timer expires or probe targets are hit. This is primarily a limitation of the web interface being a web-app, not a full mobile app. In theory web sites/applications can send notifications, but on most browsers this is limited to applications served over HTTPS, which is not really practical for this use case. To generate notifications I will likely need to create a real mobile app.
Because FreeFall 800 operates fully locally, you cannot access your grill from outside your local network, without extra effort. The Masterbuilt servers act as an intermediary for the factory firmware, so you can transparently control your grill from anywhere. There are of course options to tunnel traffic to your local network to work around this, but that is outside the scope of this project.

Future Work

There are, of course, so many things to be fixed and improved on this project going forward. Other than bug fixes and smaller improvements, these are some larger areas of improvement I would like to see:

Develop native mobile applications to connect with FreeFall 800 controllers. Mainly this is to allow for notifications to be generated for alarms, which I do miss having from the factory firmware. All control over FreeFall 800 is available through a JSON API, so no firmware changes would even be required to implement a native Android or iOS app. However, maybe the simplest solution would be to wrap the existing web app with some extensions in a simple mobile app shell.
More advanced temperature programming. The factory firmware only allows you to set a static temperature target, and this is what is implemented in FreeFall 800 as well. But now that the basics are implemented, it may be interesting to be able to program a temperature sequence, or even temperature ramps from the web app. I believe the temperature control PID loop could be made stable enough to allow for at least slow temperature changes.
Support for other Gravity series or Masterbuilt grills/smokers that use a similar controller. This one will need some community support, since I can’t justify actually buying the hardware myself. However, if anyone has other devices with similar model controllers and would like to help develop support for them, I would love a merge request!

Contributing

All source code for the project is available on the FreeFall 800 GitLab page.

If you try out FreeFall 800 and find bugs, have feature requests, or any other feedback. Please create an issue describing the bug or feature in the GitLab issue tracker.

Feel free to fork the repository and play around with the firmware! If you make improvements you think would be useful to the project as a whole, please open a merge request. I would love to see what others can do with this starting point.

Getting Started With a Private LoRaWAN Network Using ChirpStack v4 (Updated)

prbs23@prbs23.com (prbs23) — Tue, 23 Jul 2024 20:31:41 -0600

I have recently started experimenting with LoRa radios, and the LoRaWAN network protocol. Interestingly it seems that the LoRa ecosystem is simultaneously surprisingly open (both in specifications and implementations), while also having very little definitive documentation on how to actually get started as an indvidual hacker. As such I found the process of learning enough of the protocol to set up my own network surprisingly difficult.

This post will walk through the process of setting up my own private LoRaWAN network, from low cost hardware all the way up to the application level. While I would never claim to be an expert in LoRa or LoRaWAN, I hope someone finds this a useful reference.

Note:

This is an updated republication of my original 2022 Getting Started With a Private LoRaWAN Network post. Shortly after my last post I had to put the project on hold for a long time, and when I finally got back to it, I found ChirpStack had released a new architecturally incompatible version 4 of their LoRaWAN network framework. Since the configuration files and setup process have changed so significantly, the process described in my old post is no longer valid. I have rewritten the ChirpStack configuration section of this post to be compatible with the new ChirpStack v4.

LoRa and LoRaWAN Basics

LoRa (not to be confused with LoRaWAN) is a proprietary modulation and physical layer standard for low power wireless communication, originally developed in 2010 by Cycleo, which was later acquired by Semtech in 2012. The LoRa physical layer uses a “Chirp Spread Spectrum” (CSS) modulation technique to spread the signal over either 125, 250, or 500MHz bands. The use of CSS modulation, instead of the more common “Direct Sequence Spread Spectrum” (DSSS) allows LoRa to transmit over longer ranges and be more robust to many common types of interference.

The advertised range for LoRa transceivers tend to be in the 1km to 10km range, though from my research it realistically is more often on the lower end of that range. I have not tested what range I can get from my system yet, so I don’t have my own empirical data. More information about LoRa modulation and the physical layer can be found in this Application Note from Semtech, and this excellent series of blog posts.

LoRaWAN, on the other hand, is a “Low Power Wide Area Network” (LPWAN) specification, built on top of the LoRa physical layer. LoRaWAN defines a network architecture, MAC layer, and link layer protocols, primarily targeted at building low power and bandwidth sensor networks. The LoRaWAN standard is defined by the LoRa Alliance, and is published as an open standard in two primary parts: The LoRaWAN L2 Specification which defines everything except specific physical layer parameters, and the LoRaWAN Regional Parameters which defines physical layer parameters, such as frequency bands, bandwidth, etc. for specific regions of the world, in order to be compliant with local regulations.

Basic LoRaWAN Network Architecture.

The basic architecture of a LoRaWAN network is shown above. Each LoRaWAN “End-Device” communicates with one or more “Gateway” devices over the LoRa physical layer. These gateways forward messages between the end-devices and a “Network Server” over standard TCP/IP networking. The network server manages all devices connected to one or more gateways, including device activation (joining a network), congestion control, etc. Device authentication is handled by a separate “Join Server” that interfaces with one or more network servers, and finally any non-management messages from a device are sent from the network server to an application server.

Getting Started

The simplest way to start using LoRaWAN, is to buy some LoRaWAN devices, connect them to a public network such as The Things Network, and you are good to go. While this is simple because all the network and server management is handled by someone else, I wanted to set up my own network. This was partially just to understand how all the pieces of the LoRaWAN network work, but also because I don’t have coverage from any of these networks where I live anyway.

To set up your own network you need essentially four things:

End device hardware, generally a LoRa transceiver and a microcontroller.
Firmware network stack to run on the end device.
Gateway radio for the end devices to connect to.
Gateway client and network/join server software.

End-Devices

Hardware

For end-device hardware, there are many different options, from off the shelf LoRaWAN connected sensors and devices, to bare LoRa transceiver chips. For my first foray into LoRaWAN I chose to use an integrated LoRa radio/microcontroller module, since I believe it provides the most direct development path to building my own custom device PCBs in the future, while also allowing complete programmability of the device. There are several manufactures that produce these modules with different specifications, but I chose the CMWX1ZZABZ module from muRata as it was a fairly common, low cost option.

B-L072Z-LRWAN1 board for CMWX1ZZABZ module used as end-device for testing network.

The CMWX1ZZABZ module integrates a Semtech SX1276 LoRa transceiver, RF switch, and STM32L microcontroller into single surface mount RF module that is easily integrated onto a custom PCB. The integration of a STM32 MCU is convenient, as this is a microcontroller platform that I am already familiar with. Additionally, ST Microelectronics even sells the B-L072Z-LRWAN1 development board that included everything I needed to get started building a LoRaWAN end-device.

Firmware Networking Stack

In addition to the end-device hardware, I also needed firmware that implemented the end-device side of the LoRaWAN protocol. Fortunately, Semtech have published a BSD licensed LoRaWAN end-device networking stack that works with many of their LoRa transceiver chips. Even better, STMicroelectronics, provide a packaged version of this library that is already integrated with their STM32Cube embedded software stack. This makes it very simple to get some demo applications running on the B-L072Z-LRWAN1 development board with very little effort.

One of the demo applications provided as part of the STM32Cube LoRaWAN expansion makes the module behave as a fully featured LoRaWAN modem that accepts AT style commands over a UART port. This AT modem application firmware is what I ended up using to test and debug, as it gave me interactive control over my end-device with known working firmware, so I didn’t need to debug my own firmware at the same time as my network configuration.

Gateway

Hardware

Unlike the end-devices, the hardware for a Gateway turns out to be a bit more complicated. Primarily this is because while the end-device only needs to transmit or receive on one channel at a time, gateways generally must be able to receive packets from multiple end-devices on multiple channels simultaneously¹. Additionally, not only must the gateway be able to receive packets on typically several channels at once, in order to support the class B beacon protocol the gateway must use GPS to get precise time and location information, and also has to have a full TCP/IP networking stack to communicate with the network server.

Now, there are full off the shelf LoRaWAN gateways for sale, but most don’t even publish a price, and as the adage goes “if you have to ask the price you can’t afford it”. Even the few gateway devices that you can easily purchase are usually north of $400, which was a little out of my budget. After some research, it turns out that most LoRaWAN gateways are built off the same Semtech SX130x (baseband)/SX125x (front-end) chipset, connected to something like a fairly beefy ARM processor running Linux. The general structure of one of these gateways is shown in the block diagram below, from the Semtech LoRaWAN gateway “Corecell” reference design.

Semtech Block Diagram of "Corecell" gateway reference design (Semtech Corecell Reference Design Users Guide)

Since I was trying to avoid spending hundreds of dollars if I could, I did briefly consider building my own gateway PCB based on this architecture. However due to 2022 “Chipageddon”, I couldn’t actually get my hands on bare SX1308 chips. In the end I found that the Pycom makes an expansion board for their Wipy microcontroller modules that turns them into a LoRaWAN gateway. Now, I wasn’t interested in actually using a Wipy or their associated software, but the “Pygate” expansion module is essentially a copy of the Semtech reference design above that I could pick one up for $52 from Mouser.

Connections between Raspberry PI, Pygate module, and GPS.

To make a full gateway device, I connected the Pygate module to an old Raspberry Pi (Model B rev. 2) and a GP-736 GPS module that I had laying around. The GPS module unfortunately does not have a 1pps output that can be used for very precise timing. But even it’s limitations this it is sufficient to build an equivalent system to the Corecell reference design, and more importantly, works with the Basic Station software from Semtech.

Hardware setup for my low cost LoRaWAN gateway.

Gateway Software

To go along with Semtech’s Corecell reference design, they also provide the source for an packet forwarder application they call LoRa Basics Station. This software contains the code to drive the SX1308 and SX1257 chips on the Pygate board, and read timing and location information from a standard NEMA GPS data stream and. This application connects a LoRa network server to forward packets between end devices and the LoRaWAN network.

Building the Basics Station software turned out to be fairly straight forward. In fact Semtech have already put in the hard work to get it building on Raspberry Pi, so it’s as simple as:

1
2
3

git clone https://github.com/lorabasics/basicstation.git basicstation
cd basicstation
make platform=rpi variant=std

We will get into configuring and running the software a bit later, as that turned out to be the tricky bit, and requires some understanding of the rest of the network servers.

While the Basic Station software was already set up to drive the whole SX1308/SX1257 chipset at runtime, it didn’t handle everything for me. Specifically, the Pygate board has four input pins that must be driven correctly by the Raspberry Pi before it will work:

RFPWR_EN: Enable signal to voltage regulator for RF power rail. Needs to be driven high to power RF front-end.
FEM_EN: Enable signal to RF front-end. Must be driven high before enable front end circuits.
SX1308_RST: Active high reset for SX1308 chip. Can be driven high to reset SX1308 chip, must be low at runtime.
SX1257_RST: Active high reset for both SX1257 chip. Can be driven high to reset the SX1257 chips, must be low during operation.

While these possibly could have been tied high and low respectively, I connected these signals to Raspberry Pi GPIO pins and used the following shell script to power up the Pygate board and bring the chips out of reset:

#!/bin/sh

sx1308_rst=22
sx1257_rst=27
fem_en=17
rfpwr_en=4

# Export IOs
echo "$sx1308_rst" > /sys/class/gpio/export; sleep 0.1;
echo "$sx1257_rst" > /sys/class/gpio/export; sleep 0.1;
echo "$fem_en"     > /sys/class/gpio/export; sleep 0.1;
echo "$rfpwr_en"   > /sys/class/gpio/export; sleep 0.1;

# Make IOs outputs
echo "out" > /sys/class/gpio/gpio$sx1308_rst/direction; sleep 0.1;
echo "out" > /sys/class/gpio/gpio$sx1257_rst/direction; sleep 0.1;
echo "out" > /sys/class/gpio/gpio$rfpwr_en/direction;   sleep 0.1;
echo "out" > /sys/class/gpio/gpio$fem_en/direction;     sleep 0.1;

# Enable power rails and assert resets
echo "1" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$rfpwr_en/value;   sleep 0.1;
echo "1" > /sys/class/gpio/gpio$fem_en/value;     sleep 0.1;

# Deassert resets
echo "0" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "0" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;

pygate_setup.sh

One advantage of driving these enables/resets from GPIOs is that I can also reset or power down the whole LoRa chipset remotely using another shell script:

#!/bin/sh

sx1308_rst=22
sx1257_rst=27
fem_en=17
rfpwr_en=4

# Assert resets
echo "1" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;

# Disable power rails
echo "0" > /sys/class/gpio/gpio$fem_en/value;     sleep 0.1;
echo "0" > /sys/class/gpio/gpio$rfpwr_en/value;   sleep 0.1;

pygate_shutdown.sh

Servers

With the network hardware figured out, I just just needed the core LoRaWAN servers and an application back-end to send data to. For the LoRaWAN core servers ended up choosing to use ChirpStack. It provides an integrated solution for the full LoRaWAN network core that is simple to self-host, has many back-end integrations out of the box, and is open source. For my back-end application I’m starting with logging messages to a PostgreSQL database for plotting with Grafana.

All the details of the ChirpStack architecture, can be found in their architecture documentation. Depending on the size of the network, and applications being run over it, ChirpStack can be deployed in different ways, but for my initial setup, this is what I architecture ended up landing on:

ChirpStack server deployment.

You will note that compared to my previous version of this post, there are two fewer server processes. In ChirpStack v4, the network server and application server functionality were merged into a single process. Additionally, I am no longer using InfluxDB, partially this is because I have since learned that PostgreSQL is likely more than capable of handling the amount of time-series I would be generating ,data without the complexity of a separate database. But additionally the latest version of InfluxDB can no longer be self-hosted, so not appropriate for a standalone network.

ChirpStack On the Gateway

To connect the Semtech Basic Station packet forwarder into the ChirpStack infrastructure we need to use the ChirpStack Gateway Bridge. This acts as a shim between the Semtech LNS protocol used by the Basic Station packet forwarder and the MQTT protocol and API used by ChirpStack². We have the choice of running the Gateway Bridge either on the Raspberry Pi gateway or on the server with the ChirpStack Network Server, I chose to run it on the Raspberry Pi so that all the external network communication would be happening within the domain of ChirpStack protocols.

Unlike the other ChirpStack components I ended up needing to build the gateway bridge from source, this was because my Raspberry Pi board was old enough that the CPU was not supported by the pre-built ARM binaries. Fortunately this is not too difficult, I just needed to install the Go compiler and run make:

apt-get intall golang-go
git clone https://github.com/chirpstack/chirpstack-gateway-bridge chirpstack-gateway-bridge
cd chirpstack-gateway-bridge
make build

ChirpStack in Docker Compose

The rest of the chirpstack components and my back-end database I am running on a standard x86 computer running linux as docker containers, managed with docker-compose. ChirpStack conveniently publishes an example docker-compose configuration on this GitHub page: chirpstack/chirpstack-docker. This gave me a good starting place but the configuration did not match the server setup I was looking for, and did not include the Grafana for plotting received data. My final docker-compose and other configurations can be found in the prbs23/chirpstack-docker-prbs23 GitLab repository.

services:
  chirpstack:
    image: chirpstack/chirpstack:4
    command: -c /etc/chirpstack
    restart: unless-stopped
    volumes:
      - ./configuration/chirpstack:/etc/chirpstack
      - ./lorawan-devices:/opt/lorawan-devices
    depends_on:
      - postgres
      - mosquitto
      - redis
    environment:
      - MQTT_BROKER_HOST=mosquitto
      - REDIS_HOST=redis
      - POSTGRESQL_HOST=postgres
    ports:
      - 7070:7070

  postgres:
    image: postgres:14-alpine
    restart: unless-stopped
    volumes:
      - ./configuration/postgresql/initdb:/docker-entrypoint-initdb.d
      - postgresqldata:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=ruwkUXDzkGYr8Y

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server --save 300 1 --save 60 100 --appendonly no
    volumes:
      - redisdata:/data

  mosquitto:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - 7071:7071
    volumes: 
      - ./configuration/mosquitto/config/:/mosquitto/config/

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - '7072:3000'
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./configuration/grafana/provisioning:/etc/grafana/provisioning
      - ./configuration/grafana/dashboards:/etc/dashboards
    depends_on:
      - postgres
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=A37NJSGmLVLCue

volumes:
  postgresqldata:
  redisdata:
  grafana-storage:

docker-compose.yml

There are a couple main things that I customized from ChirpStack’s default docker-compose file:

Removed the ChirpStack Gateway Bridge container, because I am running the gateway bridge on my gateway device itself.
Removed the ChirpStack REST API server, as this is only neccesary for integrating with application software built for ChirpStack v3.
Changed the external facing ports. I have other applications running on this server that already used the default ports.
Configured PostgreSQL integration to log decoded payload received from end device sensors.
Added Grafana container to the configuration. Because my first application for this network will be data logging, Grafana can be used to plot data stored in the ChirpStack PostgreSQL integration database.

Configuring the Network

Now that I have all the hardware and software pieces I need for a private LoRaWAN network, I just needed to configure them to work together. This turned out to be more complicated than I expected, and also poorly documented, so I’ll walk through the important configurations for each component in the system.

Note: The following configuration is set up as a test network. There are multiple authentication and security features available in ChirpStack which I have not configured for this application, but you should be using for a real network.

ChirpStack Network Server

# Logging.
[logging]
  # Options are: trace, debug, info, warn error.
  level="info"

# PostgreSQL configuration.
[postgresql]
  # PostgreSQL DSN.
  dsn="postgres://chirpstack:<POSTGRESQL_CHIRPSTACK_PASSWORD>@$POSTGRESQL_HOST/chirpstack?sslmode=disable"

  # Connection Options
  max_open_connections=10
  min_idle_connections=0

# Redis configuration.
[redis]
  # Server address or addresses.
  servers=["redis://$REDIS_HOST/"]

  # TLS enabled.
  tls_enabled=false

  # Redis Cluster.
  cluster=false

# Network related configuration.
[network]
  # Network identifier (NetID, 3 bytes) encoded as HEX (e.g. 010203).
  net_id="000000"

  # Enabled regions.
  enabled_regions=["us915_0",]

# API interface configuration.
[api]
  # interface:port to bind the API interface to.
  bind="0.0.0.0:7070"

  # API Secret. (Generate with openssl rand -base64 32)
  secret="<API_SECRET>"

[integration]
  enabled=["mqtt", "postgresql"]

  [integration.mqtt]
    server="tcp://$MQTT_BROKER_HOST:7071/"
    json=true

  [integration.postgresql]
    dsn="postgres://chirpstack_integration:<POSTGRES_CHIRPSTACK_INTEGRATION_PASSWORD>@$POSTGRESQL_HOST/chirpstack_integration?sslmode=disable"

configuration/chirpstack/chirpstack.toml

In ChirpStack v4, the functionality of the v3 Application Server (API, and web UI) were merged into the ChirpStack network server. This does simplify the configuration some, in that there is now a single ChirpStack configuration file configuration/chirpstack/chirpstack.toml. The file now contains all the ChirpStack server configuration parameters, including LoRaWAN network settings and regions, authentication secrets, and application integration options. There are however two main LoRaWAN setting to call out.

net_id: Specifies the LoRaWAN network ID for the network server. The network ID is used to uniquely identify LoRaWAN networks when routing between network servers when supporting network roaming. If you wanted to support connecting end devices through other device’s gateways, then it is necessary to acquire a unique network ID from the LoRa Alliance. However, network IDs 0x000000 and 0x000001 are reserved for experimental networks (I consider this experimental), and networks that are not using roaming (which I am not), so this can just be set to 0x000000.
enabled_regions: Defines which LoRaWAN regions and channel plan configurations will be supported by the network server. In ChirpStack v4, the network server now simultaneously supports multiple channel plan configurations, each entry in the enabled_regions configuration list corresponds to a configuration file named region_<region>.toml in the configuration/chirpstack directory. The chirpstack/chirpstack-docker repository comes with a large set of standard channel plans for all regions. I chose the “us915_0” configuration for my network, since it is in the US, and matches the channel plan I used when setting up ChirpStack v3. More details on channel plans below.

In addition to the LoRaWAN network settings, the remainder of the settings that needed to be configured were the connection parameters to the other docker containers in the system.

Gateway Bridge

[integration.mqtt.auth.generic]
servers=["tcp://<NETWORK_SERVER_HOST>:7071"]

[integration.mqtt]
event_topic_template="us915_0/gateway/{{ .GatewayID }}/event/{{ .EventType }}"
state_topic_template="us915_0/gateway/{{ .GatewayID }}/state/{{ .StateType }}"
command_topic_template="us915_0/gateway/{{ .GatewayID }}/command/#"

[backend]
type="basic_station"

  [backend.basic_station]
  bind=":3001"

  region="US915"
  frequency_min=923000000
  frequency_max=938000000


  [[backend.basic_station.concentrators]]

    [backend.basic_station.concentrators.multi_sf]
    frequencies=[
      902300000,
      902500000,
      902700000,
      902900000,
      903100000,
      903300000,
      903500000,
      903700000,
    ]
  
    [backend.basic_station.concentrators.lora_std]
    frequency=903000000
    bandwidth=500000  
    spreading_factor=8

configuration/chirpstack-gateway-bridge/chirpstack-gateway-bridge.toml

Other than the MQTT broker host and port, the primary purpose of this configuration is to set up the channel plan that this gateway will use. This needs to match with one of the enabled_regions configurations listed in the network server configuration above. To understand how to configure the LoRaWAN channel plan should be configured we need to understand two things:

First, we need to know how the regional parameters specification defines the LoRaWAN channels for our region. In my case the US915 LoRaWAN Regional Parameters specification defines 72 uplink channels. The first 64 channels are defined as 125khz wide channels spaced every 200khz starting 902.3Mhz. The remaining 8 channels are 500khz wide spaced every 1.6Mhz starting at 902.3Mhz

The second thing to understand what is supported by the SX1308 radio baseband chip. The SX1308 has 10 independent receive demodulation data paths that operate in parallel. Channels 0-7 are fixed 125khz LoRa demodulators with independently controlled frequencies, which can demodulate all LoRa spreading factors simultaneously. Channel 8 has a configurable LoRa demodulator that supports 125khz, 250khz, and 500khz channel bandwidth and a configurable frequency and spreading factor, but can only demodulate a single data rate at a time. Finally channel 9 is a (G)FSK demodulator that is used to support the LR-FHSS channels, which use frequency hopping spread spectrum modulation instead of the LoRa chirp spread spectrum modulation used by the other channels.

Given this, these are the important settings from the gateway configuration:

region, frequency_min, frequency_max: These simply define what LoRa region that the gateway is operating in, which for me is the US915 band, which operates between 902Mhz and 928Mhz.
backend.basic_station.concentrators.multi_sf: This specifies the channel frequencies for the first eight channels of the SX1308. Note that since the regional specification defies more 125Mhz channels than the SX1308 has, we have to pick a subset of the defined channels. I am simply using the first 8 128Khz LoRaWAN channels.
backend.basic_station.concentrators.lora_std: The frequency, bandwidth, and spreading_factor values in this group configure demodulation channel 8 in the SX1308. Since we already have 8 125Khz channels configured, I am using this channel for one of the 500khz channels, specifically LoRaWAN channel 64.
integration.mqtt.*_topic_template: In the ChirpStack v4 architecture, the MQTT topics the gateway bridge uses to communicate with the network server need to be prefixed with the channel plan name used by the network server. In this case this is “us915_0”, as that is the name of the region/channel plan configuration we used in chirpstack.toml.

Gateway Packet Forwarder

{
    "SX1301_conf": {                  /* Actual channel plan is controlled by server (ChirpStack Gateway Bridge) */
        "device": "/dev/spidev0.0",   /* Default SPI device */
        "lorawan_public": false,      /* Use private sync header, must be coordinated with all devices ahead of time */
        "clksrc": 1,                  /* radio_1 provides clock to concentrator */
        "radio_0": {
            "type": "SX1257",
            "rssi_offset": -166.0,
            "tx_enable": true,        /* First SX1257 device is used for transmitter */
            "antenna_gain": 0
        },
        "radio_1": {
            "type": "SX1257",
            "rssi_offset": -166.0,
            "tx_enable": false        /* Second SX1257 is not connected to transmit path */
        },
        "pps": false                  /* My GPS doesn't have a pps output, so don't expect it */
    },  
    "station_conf": {
        "log_file":  "stderr",
        "log_level": "DEBUG",  /* XDEBUG,DEBUG,VERBOSE,INFO,NOTICE,WARNING,ERROR,CRITICAL */
        "log_size":  10000000,
        "log_rotate":  3,  
        "CUPS_RESYNC_INTV": "1s",
        "gps": "/dev/serial0",        /* GPS Connected to serial0 device */
        "pps": "gps"
    }   
}

station.conf

Finally the configuration for the Basic Station packet forwarder application just needs to be configured based on the hardware setup of our gateway. The important settings here are:

lorawan_public: Setting this option to false which changes the LoRa sync header byte from the specification’s 0x34 to 0x12. Since the sync header is used to identify the start of a packet, changing the sync header prevents packets on our private network from interfering with any public LoRaWAN network.
clksrc: This indicates which of the two SX1257 radio front-ends generates the 32Mhz radio clock for the SX1308. On the Pygate module this is the second SX1257.
radio0/radio1: These groups configure the two SX1257 front-ends connected to the SX1308. The main difference between them is that on the Pygate module, the “radio0” transmit path is hooked up, and it is not for “radio1”.
pps: Since my GPS module does not have a PPS output that I can connect to that SX1308 chip this gets set to false.

Configuring an Application Through the ChirpStack UI

Once these configurations are set up we can start the servers, but before we can start sending packets we need to configure an application that we can associate with our gateway and devices. This is done through the ChirpStack network server’s web interface.

Fortunately this is another area where ChirpStack v4 has made things simpler. Previously it was necessary to configure an “Network Server”, “Organization”, “Network”, and “Service Srofile” before setting up an application. None of that is required anymore: Version 4 comes with a default “Tenant” (somewhat equivalent to an organization in v3), and since the network and application servers have been merged, there’s not a need to define the network server connection or configure a network through the UI anymore. This means we can jump right into configuring the application.

The first thing to set up is a device profile by selecting the “Device Profiles” section on the sidebar, and clicking “Add device profile”. This configures common network parameters used by one or more types of devices connected to our application. The important settings to configure are:

General→Region and Region Configuration: Set US915 and US915 (channels 0-7 + 64), these will be the only options available, because these are the only channel plans we configured for the network server.
General→MAC version: I am using LoRaWAN 1.0.4 because it the demo firmware I am using does not support the latest LoRaWAN 1.1.0 version.
General→Regional parameters revision: Version RP002-1.0.4 is the latest specification, and is supported by my demo firmware.
General→ADR algorithm: Make sure that the LoRa only option is selected, as the gateway is not configured for LR-FHSS.

To simply logging data into the PostgreSQL integration database, I chose to enable “Cayenne LPP” codec under the “Codec” tab of the device-profile settings. When enabled, the ChirpStack application server will assume uplink packets from devices with this profile are encoded with the Cayenne LPP format, and automatically decoded them. This makes it simple encode basic binary values on the end-device and have them logged into the database in a meaningful way.

Creating a device profile.

The next step is to connect our gateway to the network server. To do this, under “Gateways” we create a new gateway with a name, gateway ID. The gateway ID is defined by the Basic Station packet forwarder application and by default is set to the MAC address of the primary network interface on the gateway. The network server and service profiles should be configured to the ones we have just created.

Add gateway.

Now we can up the application that runs on top of our LoRaWAN network. To do this we just create a new application with a name and a description. Since the PostgreSQL integration has been configured in chirpstack.toml we don’t need to set up the additional integration settings for decoded data from received packets into the database.

Create application.

Finally we can add a our end-device to the application under the “Devices” tab, by selecting create and entering a device name, device EUI, and the previously created device profile. The Device EUI is a globally unique IEEE EUI64 identifier for the end-device, the CMWX1ZZABZ transceiver module comes pre-programmed with a Device EUI which will be used by firmware when sending join requests.

Addng end-device to application.

Once the device has been created ChirpStack brings us to a page to enter or create an “Application Key”. This is a shared 128-bit key used by the end-device and network to derive encryption keys for all data sent across the network. It should be set to something random, and communicated to the end-device out of band.

Add end-device to application.

And that’s it! Our network and applications are set up and ready to go!

Sending a Packet

As previously mentioned, to test out the network I am running an example application which turns the CMWX1ZZABZ module into an AT style modem. I’m connecting to the modem firmware over the USB-Serial interface provided by the ST-Discovery development board I’m using.

When the module comes out of reset we get the following output:

APP_VERSION:        V1.1.0
MW_LORAWAN_VERSION: V2.3.0
MW_RADIO_VERSION:   V1.1.0
###### OTAA ######
###### AppKey:      2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### NwkKey:      2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### ABP  ######
###### AppSKey:     2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### NwkSKey:     2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### DevEui:  XX:XX:XX:XX:XX:XX:XX:XX
###### AppEui:  01:01:01:01:01:01:01:01
###### DevAddr: XX:XX:XX:XX
ATtention command interface
AT? to list all available functions

From here we can send the commands to have our end-device join the network. First we need to set the NwkKey to the “Application Key” we configured in the ChirpStack UI previously, then send the join command:

1
2

AT+NWKKEY=XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
AT+JOIN=1

When everything works correctly I get an output like the following, indicating that we have joined the network.

OK

299s464:TX on freq 903100000 Hz at DR 0

OK
299s895:MAC txDone
304s884:RX_1 on freq 925700000 Hz at DR 10
305s018:MAC rxDone
+EVT:JOINED
+EVT:RX_1, DR 0, RSSI -80, SNR 8

To send a packet we use the AT+SEND command which takes three arguments: The LoRaWAN port, which can be anything other than 0 (port zero indicates network management packet), a 1 or 0 indicating if we want the network to confirm receipt of the packet, and finally the data to send as a hex string. Since I have configured ChirpStack to decode packets using the Cayenne LPP protocol, the message I’m sending is formatted using Cayenne LPP. The following command will send a packet with a value of 4.00 for the “Analog Input 3” value on port 100.

`1`	`AT+SEND=100:1:03020190`

In response the firmware gives us this output indicating that it sent our message and it was confirmed by the network:

685s981:TX on freq 902500000 Hz at DR 0

OK
686s333:MAC txDone
687s323:RX_1 on freq 923900000 Hz at DR 10
687s434:MAC rxDone
+EVT:SEND_CONFIRMED
+EVT:RX_1, DR 10, RSSI -75, SNR 8

After sending a few of these messages, we can check the event_up table in the PostgreSQL integration database through Grafana and see that each packet we sent was stored into the database, with the decoded payload in the jsonb type object column.

Data from packets logged into PostgreSQL database.

Success!

Next Steps

Now that I have a working LoRaWAN network, there are several areas that I will be investigating in the future.

To start with I have just been using the built in PostgreSQL integration and Cayenne LPP packet format built into ChirpStack to log basic data. Of course the LoRaWAN network supports sending any data with any protocol, so now that I can reliably send data there are a lot of options of both custom and built in protocols that can be layered on top of LoRaWAN and ChirpStack.

However as a proof of concept I modified the one of the other STM32 LoRaWAN examples to use the internal temperature sensor inside the STM32L0 to log data using Cayenne LPP (source code here prbs23/lorawan_env_sensor_fw. Then using a Grafana dashboard I was able to make a nice dashboard that shows historical temperatures.

Simple temperature history dashboard in Grafana.

This is really a proof of concept for a more custom built data collection network I am looking to build on top of LoRaWAN. I have already started building some custom firmware from scratch using the LoRaMAC library, but there is still much to be done in terms of defining protocols, device commissioning, and custom sensor support.

Finally, as previously mentioned I have not configured all the authentication and authorization features supported by ChirpStack. I need to spend some more time investigating and learning how to configure these to better secure my network.

Technically in some regions such as US915 it is valid to only support a single channel, in which case a single transceiver could act as a gateway. A proof of concept implementation for this can be found in this “Single Channel LoRaWAN Gateway” project on Github. However in other regions such as EU868 region, for example, a gateway is required to support at a minimum of three specific channels. ↩︎
Theoretically ChirpStack also includes its own equivalent to the Basics Station application, in the form of Concentratord. However at the time, the integration between the Concentratord and the Gateway Bridge was listed as “experimental”, so I have not actually tried this out. ↩︎

Combining Dist Constarints in SystemVerilog

prbs23@prbs23.com (prbs23) — Thu, 25 May 2023 23:32:04 -0600

Basic Distribution Constraints

In SystemVerilog, almost all constraints are some form of boolean expression. The constraint solver’s job is to satisfy all these boolean expression when choosing random values. The two exceptions to this are solve before constraints, which control the order of value resolution, and dist constraints, witch control the distribution of values chosen. This latter type of constraint is what we are looking at in this post.

At a high level the distribution constraint provides a syntax for defining the probability mass function for a random variable over a given range of discrete values. Below are two examples of dist constraints and the resulting distribution of values over 100k randomizations. The first specifies linearly increasing probability over four values, while the second shows exponentially increasing probability for the same values.

Linear Distribution Function Exponential Distribution Function

constraint linear_dist_c {
  value dist {
    0 :/ 1,
    1 :/ 2,
    2 :/ 3,
    3 :/ 4,
    4 :/ 5,
    5 :/ 6,
    6 :/ 7,
    7 :/ 8
  };
}

constraint linear_dist_c {
  value dist {
    0 :/ 1,
    1 :/ 2,
    2 :/ 4,
    3 :/ 8,
    4 :/ 16,
    5 :/ 32,
    6 :/ 64,
    7 :/ 128
  };
}

Distribution of values with linear `dist` constraint

Distribution of values with exponential `dist` constraint

In both these cases the probability of the constraint solver choosing a given value is the value’s weight divided by the sum of weights. For example the probability of selecting the value 0 in our linear distribution is 1/(1+2+...+8) = 1/38 = 0.02631.

Obviously we can make these distributions more complicated, and the language has various syntax variations for specifying distributions. However, the question in this post is what happens if we have multiple different distribution constraints defined for the same variable?

Applying Multiple `dist` constraints

Our first question might be, what does the SystemVerilog LRM say about combinations of distribution constraints? Unfortunately it is completely unspecified. The only real specification about how constraint solvers should behave with regard to distribution constraints is as follows.

Absent any other constraints, the probability that the expression matches any value in the list is proportional to its specified weight. If there are constraints on some expressions that cause the distribution weights on these expressions to be not satisfiable, implementations are only required to satisfy the constraints. An exception to this rule is a weight of zero, which is treated as a constraint.

– IEEE Std 1800-2017 “IEEE Standard for SystemVerilog–Unified Hardware Design, Specification, and Verification Language” section 18.5.4

That’s not very helpful. So what happens if we just try it? The figure below shows various combinations of three different distribution constraints, and linear and exponentially increasing distributions from above, along with a uniform distribution where all values are given the same probability. For each combination of distributions I randomized a value 100k times, using three different simulators, to see how various SystemVerilog implementations compared.

Distribution of values with various combinations of distribution constraints in three simulators.

See the following EDAPlayground for how I generated the data these plots: Multiple Dist Constraints

From this experiment we can see that all three simulators combine the distribution constraints together in some way to create a new distribution (ie. they don’t discard the previous dist constraint). However, there appear to be two different methods used, as the combined distributions for Xcelium and VCS are very similar (though maybe not exactly the same), and Questa seems to do something entirely different. Let’s see if we can figure out what the tools are doing.

Combining Distributions in Xcelium and VCS

As far as I could tell, none of the simulators document how they handle combinations of distribution constraints. However, with some experimentation it wasn’t hard to come up with a function that appears to behave similarly to Xcelium and VCS. It appears in the case of both Xcelium and VCS, the probability of a given value being selected when there are multiple distribution constraints is simply the average of probabilities from all distribution constraints. Additionally, from the LRM we know that the probability of a given weight for a single dist constraint is the weight for that value divided by all weights. Therefore, more formally we can say, given N distribution constraints where the weight for a value x is given by the function weight_n(x), the resulting probability mass function is:

The figure below shows a comparison of using this formula to compute the expected distributions vs the observed distribution from Xcelium and VCS.

Comparison of Xcelium and VCS distributions to the average of probabilities.

But wait! The SystemVerilog spec says that a weight of zero in a dist constraint is considered as a hard constraint keeping the variable not equal to the value. If they were just averaging the probabilities together the average of a 0 and non-zero probability is going to be a non-zero value. If we actually test this by running the same experiment with the same sets of constraints, except that the values 1 and 6 are given a weight of 0 in the uniform dist constraint we get the results in the figure below. Note that the simple average of probabilities doesn’t match the actual measured distributions.

Comparison of Xcelium and VCS distributions to the simple average of probabilities and zero-filtered average of probabilities.

We can reproduce the same distribution we observe from Xcelium and VCS with a small modification to the average of probabilities combination technique. The averaging of probabilities appears to function the same, however when computing the probability of selecting a value for each individual distribution constraint, the weight for a value is considered as 0 if any other dist constraint has a weight of 0. Therefore a more accurate equation for the final probability mass function for a combination of N distribution constraints would be:

Combining Distributions in Questa

While Xcelium and VCS appear to use a similar function for combining distributions, the Questa simulator does something significantly different. From some experimentation, whatever they are doing appears equivalent to multiplying the weights (not probabilities) from all dist constraints for each value together, and then computing the probability from these weights. Formally we could write this probability mass function for the combination of N constraints given the weight functions weight_n(x):

Comparing this probability formulas to the actual results from Questa we get effective the same distributions.

Comparison of Questa distributions to the normalized product of probabilities.

The nice thing about this way of combining distributions is that we get for free that a weight of zero from one constraint will always result in a final probably being 0 as well. If we do a similar test as we did for the other simulators with a partial uniform distribution constraint where some values have a zero probability, we see our formula still matches the results from Questa.

Comparison of Questa distributions to the product of weights.

Conclusions

While all three simulators do something I would consider reasonable, I’m not sure either behavior is necessarily what I would want in any given application. Additionally, since there is no specification defined behavior, there’s no guarantees about how your constraint solver will handle any given set of constraints.

Overall, this investigation has made me wary of ever apply two distribution constraints to the same variable. In general I think I will be putting my distribution constraints inside their own constraint blocks, so that they can be disabled individually, and replaced, instead of combined with another constraint. It’s helpful to now know exactly what a given simulator will do, but I wouldn’t rely on it.

Finally, I have only looked into how multiple distribution constraints interact when applied to a single variable, but there’s other questions too… What if you had two distribution constraints on different variables, but a hard constraint keeping the variables equal? How do distribution constraints interact with other types of hard constraints? One day, maybe I will investigate these too.

Getting Started With a Private LoRaWAN Network

prbs23@prbs23.com (prbs23) — Thu, 07 Jul 2022 22:01:39 -0600

Note:

Since publishing this post, ChirpStack released their version 4 LoRaWAN network server, which significantly changes the architecture and configuration. This post goes through setting up a network with ChirpStack v3, for an updated version of this post using ChirpStack v4, see Getting Started With a Private LoRaWAN Network Using ChirpStack v4.

LoRa and LoRaWAN Basics

Basic LoRaWAN Network Architecture.

Getting Started

To set up your own network you need essentially four things:

End device hardware, generally a LoRa transceiver and a microcontroller.
Firmware network stack to run on the end device.
Gateway radio for the end devices to connect to.
Gateway client and network/join server software.

End-Devices

Hardware

B-L072Z-LRWAN1 board for CMWX1ZZABZ module used as end-device for testing network.

Firmware Networking Stack

Gateway

Hardware

Semtech Block Diagram of "Corecell" gateway reference design (Semtech Corecell Reference Design Users Guide)

Since I was trying to avoid spending hundreds of dollars if I could, I did briefly consider building my own gateway PCB based on this architecture. However due to the ongoing “Chipageddon”, I couldn’t actually get my hands on bare SX1308 chips. In the end I found that the Pycom makes an expansion board for their Wipy microcontroller modules that turns them into a LoRaWAN gateway. Now, I wasn’t interested in actually using a Wipy or their associated software, but the “Pygate” expansion module is essentially a copy of the Semtech reference design above that I could pick one up for $52 from Mouser.

Connections between Raspberry PI, Pygate module, and GPS.

Hardware setup for my low cost LoRaWAN gateway.

Gateway Software

Building the Basics Station software turned out to be fairly straight forward. In fact Semtech have already put in the hard work to get it building on Raspberry Pi, so it’s as simple as:

1
2
3

git clone https://github.com/lorabasics/basicstation.git basicstation
cd basicstation
make platform=rpi variant=std

We will get into configuring and running the software a bit later, as that turned out to be the tricky bit, and requires some understanding of the rest of the network servers.

RFPWR_EN: Enable signal to voltage regulator for RF power rail. Needs to be driven high to power RF front-end.
FEM_EN: Enable signal to RF front-end. Must be driven high before enable front end circuits.
SX1308_RST: Active high reset for SX1308 chip. Can be driven high to reset SX1308 chip, must be low at runtime.
SX1257_RST: Active high reset for both SX1257 chip. Can be driven high to reset the SX1257 chips, must be low during operation.

#!/bin/sh

sx1308_rst=22
sx1257_rst=27
fem_en=17
rfpwr_en=4

# Export IOs
echo "$sx1308_rst" > /sys/class/gpio/export; sleep 0.1;
echo "$sx1257_rst" > /sys/class/gpio/export; sleep 0.1;
echo "$fem_en"     > /sys/class/gpio/export; sleep 0.1;
echo "$rfpwr_en"   > /sys/class/gpio/export; sleep 0.1;

# Make IOs outputs
echo "out" > /sys/class/gpio/gpio$sx1308_rst/direction; sleep 0.1;
echo "out" > /sys/class/gpio/gpio$sx1257_rst/direction; sleep 0.1;
echo "out" > /sys/class/gpio/gpio$rfpwr_en/direction;   sleep 0.1;
echo "out" > /sys/class/gpio/gpio$fem_en/direction;     sleep 0.1;

# Enable power rails and assert resets
echo "1" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$rfpwr_en/value;   sleep 0.1;
echo "1" > /sys/class/gpio/gpio$fem_en/value;     sleep 0.1;

# Deassert resets
echo "0" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "0" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;

pygate_setup.sh

One advantage of driving these enables/resets from GPIOs is that I can also reset or power down the whole LoRa chipset remotely using another shell script:

#!/bin/sh

sx1308_rst=22
sx1257_rst=27
fem_en=17
rfpwr_en=4

# Assert resets
echo "1" > /sys/class/gpio/gpio$sx1308_rst/value; sleep 0.1;
echo "1" > /sys/class/gpio/gpio$sx1257_rst/value; sleep 0.1;

# Disable power rails
echo "0" > /sys/class/gpio/gpio$fem_en/value;     sleep 0.1;
echo "0" > /sys/class/gpio/gpio$rfpwr_en/value;   sleep 0.1;

pygate_shutdown.sh

Servers

ChirpStack server deployment.

ChirpStack On the Gateway

apt-get intall golang-go
git clone https://github.com/brocaar/chirpstack-gateway-bridge.git chirpstack-gateway-bridge
cd chirpstack-gateway-bridge
make build

ChirpStack in Docker Compose

The rest of the chirpstack components and my back-end database I am running on a standard x86 computer running linux as docker containers, managed with docker-compose. ChirpStack conveniently publishes an example docker-compose configuration on this GitHub page: brocaar/chirpstack-docker. This gave me a good starting place but the configuration did not match the server setup I was looking for, and did not include my back-end InfluxDB database. My final docker-compose and other configurations can be found in the prbs23/chirpstack-docker GitLab repository.

version: "3"

services:
  chirpstack-network-server:
    image: chirpstack/chirpstack-network-server:3
    volumes:
      - ./configuration/chirpstack-network-server:/etc/chirpstack-network-server
    depends_on:
      - postgresql
      - mosquitto

  chirpstack-application-server:
    image: chirpstack/chirpstack-application-server:3
    ports:
      - 7070:7070
    volumes:
      - ./configuration/chirpstack-application-server:/etc/chirpstack-application-server
    depends_on:
      - chirpstack-network-server

  postgresql:
    image: postgres:9.6-alpine
    environment:
      - POSTGRES_PASSWORD=<POSTGRESS_ROOT_PASSWORD>
    volumes:
      - ./configuration/postgresql/initdb:/docker-entrypoint-initdb.d
      - postgresqldata:/var/lib/postgresql/data

  redis:
    image: redis:5-alpine
    volumes:
      - redisdata:/data

  mosquitto:
    image: eclipse-mosquitto:2
    ports:
      - 7072:7072
    volumes: 
      - ./configuration/eclipse-mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf

  influxdb:
    image: influxdb:latest
    volumes:
      # Mount for influxdb data directory and configuration
      - ./influxdb2:/var/lib/influxdb2:rw
    ports:
      - "7074:8086"

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - '7075:3000'
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./grafana-provisioning/:/etc/grafana/provisioning
    depends_on:
      - influxdb
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=<GRAFANA_ADMIN_PASSWORD>

volumes:
  postgresqldata:
  redisdata:
  grafana-storage:

docker-compose.yml

There are a couple main things that I customized from ChirpStack’s default docker-compose file:

Removed the ChirpStack Gateway Bridge container, because I am running the gateway bridge on my gateway device itself.
Changed the external facing ports. I have other applications running on this server that already used the default ports.
Added InfluxDB time series database and Grafana containers to the configuration. Because my first application for this network will be data logging, I’m using the InfluxDB integration with ChirpStack as an easy time series data store, and Grafana to plot the collected data.

Configuring the Network

Network Server

[postgresql]
dsn="postgres://chirpstack_ns:<POSTGRES_NS_PASSWORD>@postgresql/chirpstack_ns?sslmode=disable"

[redis]
url="redis://redis:6379"

[network_server]
net_id="000000"

[network_server.band]
name="US915"

  [network_server.network_settings]
  enabled_uplink_channels=[0, 1, 2, 3, 4, 5, 6, 7, 64]

[network_server.gateway.backend.mqtt]
server="tcp://mosquitto:7072"

[join_server.default]
server="http://chirpstack-application-server:8003"

chirpstack-network-server.toml

The network server configuration is located in a file named by default chirpstack-network-server.toml, and primarily configures the connectivity and credentials between the network server and the other components in the ChirpStack architecture. There are however a couple interesting LoRaWAN setting to call out.

net_id: Specifies the LoRaWAN network ID for the network server. The network ID is used to uniquely identify LoRaWAN networks when routing between network servers when supporting network roaming. If you wanted to support connecting end devices through other device’s gateways, then it is necessary to acquire a unique network ID from the LoRa Alliance. However, network IDs 0x000000 and 0x000001 are reserved for experimental networks (I consider this experimental), and networks that are not using roaming (which I am not), so this can just be set to 0x000000.
band: Defines which region from the LoRaWAN Regional Parameters specification is being used for this network. Since my network is located in the US, I have to use “US915” as my band.
enabled_uplink_channels: Declares the subset of channels (as defined by the regional parameters specification for the US915 band) are to be used on the network. For reasons we will get into later, only a subset of 9 channels will be enabled.

Application Server

[postgresql]
dsn="postgres://chirpstack_as:<POSTGRES_AS_PASSWORD>@postgresql/chirpstack_as?sslmode=disable"

[redis]
url="redis://redis:6379"

[application_server.integration.mqtt]
server="tcp://mosquitto:7072"

[application_server.api]
public_host="chirpstack-application-server:8001"

[application_server.external_api]
bind="0.0.0.0:7070"
jwt_secret="<JWT_SECRET>"

chirpstack-application-server.toml

Since the application server is not directly interfacing with the LoRaWAN network, the configuration here is pretty simple. Really it just needs to configure the connections between it and the databases and MQTT server docker containers.

Gateway Bridge

# Gateway backend configuration.
[backend]
type="basic_station"

  # Basic Station backend.
  [backend.basic_station]
  bind="localhost:3001"

  # Channel Configuration
  region="US915"
  frequency_min=902000000
  frequency_max=938000000
  [[backend.basic_station.concentrators]]
    [backend.basic_station.concentrators.multi_sf]
    frequencies=[
        902300000,
	    902500000,
	    902700000,
	    902900000,
	    903100000,
	    903300000,
	    903500000,
	    903700000,
    ]
    [backend.basic_station.concentrators.lora_std]
    frequency=903000000
    bandwidth=500000
    spreading_factor=8

# Network server configuration.
[integration.mqtt.auth.generic]
  servers=["tcp://<network_server_host>:7072"]

chirpstack-gateway-bridge.toml

In addition to setting up hosts and ports for the connections to the network server and packet forwarding application, this configuration is where we define the receive channels that will be used by the gateway. To understand how these settings should be configured we need to understand two things:

Given this, these are the important settings from the gateway configuration:

region, frequency_min, frequency_max: These simply define what LoRa region that the gateway is operating in, which for me is the US915 band, which operates between 902Mhz and 928Mhz.
backend.basic_station.concentrators.multi_sf: This specifies the channel frequencies for the first eight channels of the SX1308. Note that since the regional specification defies more 125Mhz channels than the SX1308 has, we have to pick a subset of the defined channels. I am simply using the first 8 128Khz LoRaWAN channels.
backend.basic_station.concentrators.lora_std: The frequency, bandwidth, and spreading_factor values in this group configure demodulation channel 8 in the SX1308. Since we already have 8 125Khz channels configured, I am using this channel for one of the 500khz channels, specifically LoRaWAN channel 64.

Gateway Packet Forwarder

{
    "SX1301_conf": {                  /* Actual channel plan is controlled by server (ChirpStack Gateway Bridge) */
        "device": "/dev/spidev0.0",   /* Default SPI device */
        "lorawan_public": false,      /* Use private sync header, must be coordinated with all devices ahead of time */
        "clksrc": 1,                  /* radio_1 provides clock to concentrator */
        "radio_0": {
            "type": "SX1257",
            "rssi_offset": -166.0,
            "tx_enable": true,        /* First SX1257 device is used for transmitter */
            "antenna_gain": 0
        },
        "radio_1": {
            "type": "SX1257",
            "rssi_offset": -166.0,
            "tx_enable": false        /* Second SX1257 is not connected to transmit path */
        },
        "pps": false                  /* My GPS doesn't have a pps output, so don't expect it */
    },  
    "station_conf": {
        "log_file":  "stderr",
        "log_level": "DEBUG",  /* XDEBUG,DEBUG,VERBOSE,INFO,NOTICE,WARNING,ERROR,CRITICAL */
        "log_size":  10000000,
        "log_rotate":  3,  
        "CUPS_RESYNC_INTV": "1s",
        "gps": "/dev/serial0",        /* GPS Connected to serial0 device */
        "pps": "gps"
    }   
}

station.conf

Finally the configuration for the Basic Station packet forwarder application just needs to be configured based on the hardware setup of our gateway. The important settings here are:

lorawan_public: Setting this option to false which changes the LoRa sync header byte from the specification’s 0x34 to 0x12. Since the sync header is used to identify the start of a packet, changing the sync header prevents packets on our private network from interfering with any public LoRaWAN network.
clksrc: This indicates which of the two SX1257 radio front-ends generates the 32Mhz radio clock for the SX1308. On the Pygate module this is the second SX1257.
radio0/radio1: These groups configure the two SX1257 front-ends connected to the SX1308. The main difference between them is that on the Pygate module, the “radio0” transmit path is hooked up, and it is not for “radio1”.
pps: Since my GPS module does not have a PPS output that I can connect to that SX1308 chip this gets set to false.

Setting Up an Organization and Application

Even once all these configurations are set up, I found I wasn’t done. We need to configure the ChirpStack application server to set up the logical network and applications. This is done through the application server’s web interface.

Once logged into the ChirpStack application sever, we first need to setup the connection to the network server. In “Network-servers”, we add a new network server with a logical name for the network, and the host:port for network server. The host and port should be “chirpstack-network-server:8000”. This is also where we could set up TLS authentication and encryption between network and application servers.

Adding network server connection to the application server.

Next we must create an “organization” for our application network to run in. ChirpStack appears to be built from a network administrator perspective, and so provides “organizations” as a way to manage groups of users and applications. For a private LoRaWAN network this isn’t particularly useful, but we still need to set one up. Under “Organizations” we create a new organization a name, display name and permissions. For the permissions I selected “Organization can have gateways”.

Creating an organization.

Now that we have an organization, we need to configure the service and device profiles. Each gateway and application is associated with a service profile that defines how the application/gateway can consume network resources. Each device is associated with a device profile which defines network and protocol parameters that are applied to a set of devices.

First we create a new network service profile under “Service-profiles”. The profile has a name, an associated network server (we only have one option here), and other network settings. The ChirpStack defaults have the maximum allowed data rate set to DR0, which is the slowest LoRaWAN data rate. For our gateway configuration, the highest supported data rate is DR4, so setting the maximum allowed data-rate setting to 4 allows us to get the most per-device bandwidth out of the network.

Creating a service profile.

Next we create a device profile by under the “Device-profiles” sections. As with the service profile we need to give it a name and an associated network. Additionally we need to set the LoRaWAN protocol and regional parameters specification versions that will be used by the device. I’m using protocol spec version 1.0.4 because it is what was supported in the end-device firmware network stack, and regional parameters version RP002-1.0.3, since that is the latest version. Finally we need to configure the automatic data rate algorithm to “LoRa Only”, Since our gateway isn’t configured for LR-FHSS. On the “Join (OTAA/APB)” tab I also set “Device supports OTAA” to enable over the air activation protocol which allows the end-device and network server to negotiate network parameters without needing them to be configured out of band.

To simplify logging data into my InfluxDB backend, I chose to enable the “Cayenne LPP” codec under the “Codec” tab of the device-profile settings. When enabled, the ChirpStack application server will assume uplink packets from devices with this profile are encoded with the Cayenne LPP format, and automatically decoded them. This makes it simple encode basic binary values on the end-device and have them logged into the InfluxDB database in a meaningful way.

Creating a device profile.

With the profiles created we can now add a gateway to the organization. To do this, under “Gateways” we create a new gateway with a name and description, gateway ID, network server, and finally a service profile. The gateway ID is defined by the Basic Station packet forwarder application and by default is set to the MAC address of the primary network interface on the gateway. The network server and service profiles should be configured to the ones we have just created.

Add gateway to organization.

Next we will finally set up the application that runs on top of our LoRaWAN network. To do this we just create a new application with a name and the service profile that we created previously.

Create application.

Once we have an application we need to tell ChirpStack what to do with packets once they are received, by choosing one or more “Integrations”. My goal for this initial setup was simply to log data into an InfluxDB time series database, conveniently ChirpStack has an integration with InfluxDB which stores data values from each uplink packet as decoded using the Cayenne LPP protocol. To enable this, under “Applications”->“Integrations” we find the “InfluxDB” integration, add it to the application, then in the integration settings we will select InfluxDB 2.x for the protocol, http://influxdb:8086/api/v2/write for the API InfluxDB endpoint (this is the docker internal hostname and port), and specify which bucket in the database it should put received packets into.

Configuring influxdb integration.

Now we can finally add a our end-device to the application under the “Devices” tab, by selecting create and entering a device name, description, device EUI, and the previously created device profile. The Device EUI is a globally unique IEEE EUI64 identifier for the end-device, the CMWX1ZZABZ transceiver module comes pre-programmed with a Device EUI which will be used by firmware when sending join requests.

Addng end-device to application.

Add end-device to application.

And that’s it! Our network and applications are set up and ready to go!

Sending a Packet

When the module comes out of reset we get the following output:

APP_VERSION:        V1.1.0
MW_LORAWAN_VERSION: V2.3.0
MW_RADIO_VERSION:   V1.1.0
###### OTAA ######
###### AppKey:      2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### NwkKey:      2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### ABP  ######
###### AppSKey:     2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### NwkSKey:     2B:7E:15:16:28:AE:D2:A6:AB:F7:15:88:09:CF:4F:3C
###### DevEui:  XX:XX:XX:XX:XX:XX:XX:XX
###### AppEui:  01:01:01:01:01:01:01:01
###### DevAddr: XX:XX:XX:XX
ATtention command interface
AT? to list all available functions

1
2

AT+NWKKEY=XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
AT+JOIN=1

When everything works correctly I get an output like the following, indicating that we have joined the network.

OK

299s464:TX on freq 903100000 Hz at DR 0

OK
299s895:MAC txDone
304s884:RX_1 on freq 925700000 Hz at DR 10
305s018:MAC rxDone
+EVT:JOINED
+EVT:RX_1, DR 0, RSSI -80, SNR 8

`1`	`AT+SEND=100:1:03020190`

In response the firmware gives us this output indicating that it sent our message and it was confirmed by the network:

685s981:TX on freq 902500000 Hz at DR 0

OK
686s333:MAC txDone
687s323:RX_1 on freq 923900000 Hz at DR 10
687s434:MAC rxDone
+EVT:SEND_CONFIRMED
+EVT:RX_1, DR 10, RSSI -75, SNR 8

After sending a few of these messages, looking in the InfluxDB bucket I now have the data values from each packet I sent logged into the database, along with a little bit of metadata about it.

Data from packets logged into InfluxDB database.

Success!

Next Steps

Now that I have a working LoRaWAN network, there are several areas that I will be investigating in the future.

To start with I have just been using the built in InfluxDB integration and Cayenne LPP packet format built into ChirpStack to log basic data. Of course the LoRaWAN network supports sending any data with any protocol, so now that I can reliably send data there are a lot of options of both custom and built in protocols that can be layered on top of LoRaWAN and ChirpStack.

Additionally, I have just been using the CMWX1ZZABZ as a dumb modem to manually transmit packets/data over the network. For a real application of my LoRaWAN network I need to develop firmware for the CMWX1ZZABZ to actual interface with sensors and send the data to the network.

Finally, as previously mentioned I have not configured all the authentication and authorization features in the ChirpStack architecture. I need to spend some more time investigating and learning how to configure these to better secure my network.

Technically in some regions such as US915 it is valid to only support a single channel, in which case a single transceiver could act as a gateway. A proof of concept implementation for this can be found in this “Single Channel LoRaWAN Gateway” project on Github. However in other regions such as EU868 region, for example, a gateway is required to support at a minimum of three specific channels. ↩︎
Theoretically ChirpStack also includes its own equivalent to the Basics Station application, in the form of Concentratord. However on the integration between the Concentratord and the Gateway Bridge is currently listed as “experimental”, so I have not actually tried this out. ↩︎

Don't Wait For Me

prbs23@prbs23.com (prbs23) — Thu, 17 Feb 2022 22:22:26 -0700

From the perspective your more traditional software programming languages, multi-tasking in SystemVerilog is weird. On one hand, the language includes a special syntax (fork and join) that make it really easy to write code that “runs” in parallel. On the other hand, SystemVerilog “processes” aren’t really processes, or even threads, in the traditional sense. Even if you “get” the SystemVerilog model of cooperative cooperative multi-tasking, there are a number of pitfalls you can run into. One such gotcha that I recently ran into was caused by what appeared to be an innocuous wait fork statement.

An Unexpected Infinite Wait

initial begin
  bit [31:0] data;

  fork
    forever begin
      // Monitor some signals for correct behavior.
    end
  join_none

  // Do some other stuff. Then...
  do_read(32'hdead_beef, 3, data);
  $display("Done"); // <--- This print never happens
end

The initial problem code structure

The issue was in an old (I mean really old, pre-UVM) test case that hadn’t been run in a long time, and appeared to hang in a task that issued an bus read transaction. From inspecting the simulation waves and logs, I could tell that the bus transaction was completing without issue, but the task was still never returning. The basic structure of this the test code is shown above, a process is forked off, then the do_read() task is called. This obviously calls the content of do_read() into question, after digging through a few translation layers (did I mention this code was old), I found the code that actually did the bus transaction, which I’ll call do_read_internal(), looked something like this:

task do_read_internal(
  input bit [31:0] addr,
  input int byte_size,
  output bit [31:0] data,
  input flags_t flags = NONE
);
  fork
    begin
      // Wiggle bus wires and pass time as required do drive transaction
    end
  join_none

  if (flags & BLOCKING_TRANSACTION) begin
    wait fork;
  end
endtask : do_read_internal

do_read_internal() task called by test case through do_read() task

Now, if you know what you’re looking for you may already see the issue. But before I give it all away, suffice it to say, the problem here is the use of the wait fork statement. So let’s talk about what that last wait fork statement is actually doing.

A Not So Brief Introduction to `wait fork`

Lets start in the obvious place, what does the SystsemVerilog LRM (Language Reference Manual) say about wait fork? This is summary of the statement that we can find in section 9.6.1 Wait fork statement:

The wait fork statement blocks process execution flow until all immediate child subprocesses (processes created by the current process, excluding their descendants) have completed their execution.

This seems fairly simple, it is a statement that makes a process wait until all sub-processes that were started from the current process have finished. The problem with this construct is that it’s not always obvious which processes “all immediate child subprocesses” refer to. Lets start with a simple example:

initial begin : main_process
  fork
    begin : subprocess_1;
      #100 $display("%t: Hello from subprocess_1", $time());
    end : subprocess_1
    begin : subprocess_2;
      #50 $display("%t: Hello from subprocess_2", $time());
    end : subprocess_2
  join_none
  
  wait fork;
  $display("%t: Done", $time());
end : main_process

Trivial wait fork example

In this case the fork/join_none block is going to spawn two subprocesses which wait for a period of time and then print a message. Because a join_none was used, execution of the main_process block will not wait for these subprocesses to finish before continuing. Then the wait fork statement will wait for both the two subprocesses running the subprocess_1 and subprocess_2 blocks to finish, and then print “Done”. Easy enough. Now, what if we make it a bit more complicated and move the wait fork into another task.

initial begin : main_process
  fork
    begin : subprocess_1;
      #100 $display("%t: Hello from subprocess_1", $time());
    end : subprocess_1
    begin : subprocess_2;
      #50 $display("%t: Hello from subprocess_2", $time());
    end : subprocess_2
  join_none
  
  task_1();
  $display("%t: Done", $time());
end : main_process
    
task task_1();
  fork
    begin : subprocess_3;
      #25 $display("%t: Hello from subprocess_3", $time());
    end : subprocess_3
    begin : subprocess_4;
      #75 $display("%t: Hello from subprocess_4", $time());
    end : subprocess_4
  join_none;
  
  wait fork;
endtask : task_1

What happens when we move the wait fork into a task?

In this case we start subprocesses subprocess_1 and subprocess_2, and don’t wait for them to finish, just like the previous example. However we now call a task task_1() which also spawns two new processes subprocess_3 and subprocess_4, then executes a wait fork statement, before returning and printing “Done”. If wait fork followed normal scoping rules like one might naively expect it to wait for subprocesses 3 and 4 only, but if we look at the output from this example, we see this is not the case.

xcelium> run
                  25: Hello from subprocess_3
                  50: Hello from subprocess_2
                  75: Hello from subprocess_4
                 100: Hello from subprocess_1
                 100: Done

The reason that the wait fork waits for all four of the spawned subprocess to finish, is that simply that they are all subprocesses of the process from which wait fork was called, the one started from the main_process initial block. This is because calling a task or another SystemVerilog scope doesn’t start a new process, in fact the only way that a new process is spawned (from an existing one) is through a fork statement. Because the code running in task_1 is still part of the same process, “all immediate subprocesses” includes subprocess_1 and subprocess_2.

initial begin : main_process
  task_1();
  task_2();
  $display("%t: Done", $time());
end : main_process
    
task task_1();
  fork
    begin : infinite_subprocess
      forever #100 $display("%t Hello again from infinite_subprocess", $time());
    end : infinite_subprocess
    begin : subprocess_1
      fork
        begin : infinite_sub_subprocess
          forever #75 $display("%t Hello again from infinite_sub_subprocess", $time());
        end : infinite_sub_subprocess
      join_none
    end : subprocess_1
  join_none
endtask : task_1

task task_2();
  fork
    begin : subprocess_2;
      #25 $display("%t: Hello from subprocess_2", $time());
    end : subprocess_2
    begin : subprocess_3;
      #75 $display("%t: Hello from subprocess_3", $time());
    end : subprocess_3
  join_none;
  
  wait fork;
endtask : task_2

A wait fork example that will hang forever

This final example demonstrates, in a more obvious way, the bug in my problem code above. In this case, the task task_1 is first called from main_process which starts two subprocesses infinite_subprocess and subprocess_1, then immediately returns because the block is terminated with join_none. Then task_2 is executed, which forks two more processes (subprocess_2 and subprocess_3) and finally executes a wait fork statement. In this case the wait fork statement is never actually going to finish, because it has to wait for the conspicuously named infinite_subprocess (which never ends), even though it was started from a different task.

Finally, note that the wait fork in task_2 does not have to wait for the process infinite_sub_subprocess because it is not a direct subprocess of the main_process. The infinite_sub_subprocess process is forked from the subprocess_1 process, but because it uses join_none, the process does not wait for infinite_sub_subprocess, and leaves the sub-subprocess running, unattached, from any parent process.

What Went Wrong?

Going back to my hanging test case. Now that we’ve looked at some examples of how wait fork works, it’s pretty obvious if you know where to look. The main test code starts an infinite process, and then calls a task which start it’s own process and then uses wait fork to wait for it’s process, but unintentionally ends up waiting on an infinite loop to end. The do_read_internal task has no visibility to know that an infinite process has even been spawned from it’s process, but wait fork will wait for it anyway.

The thing I find devious about this bug is that it demonstrates how starting of a subprocess in one place can cause a completely unrelated piece of code to hang, potentially forever. In fact it is entirely possible that the code that starts the infinite loop subprocess is in one library and the wait fork statement is in another library. All that is required to cause havoc is that they are called from the same process. When I realized this it made me reconsider the safety of using wait fork, and at this point it seems like it’s only really safe to use in very specific applications.

Remediation

Unfortunately, there is not a version of wait fork with a more limited scope (such as waiting for subprocess of the current task or scope). This leaves us with three, not great, alternatives to implement the functionality that our do_read_internal was going for: starting a non-block process, and then conditionally waiting for it to finish.

The first, and in my opinion best, option in this case is to simply declare an event that gets emitted by the started subprocess on completion. Then to wait for the subprocess to end, the main process can just wait on the event, instead of using wait fork. Aside from needing to declare a new event variable, this works fairly well if there is only one subprocess to wait on. This implementation would look something like the following:

event done_event;
fork
  begin
    // Wiggle bus wires and pass time as required do drive transaction
    -> done_event;
  end
join_none
if (flags & BLOCKING_TRANSACTION) begin
  @done_event; // Wait or forked subprocess to report it has ended.
end

do_read_internal() task which uses a done event for waiting.

Another option is to use the SystemVerilog “fine-grain process control” features (see section 9.7 of the LRM). Essentially this feature provides an API to get a reference to a “process” object associated with the current process, which can then be used to wait on or otherwise manage the process. In this case, we can have the subprocess store it’s “process” object in a variable, and then use the await() task of the process object to wait for the subprocess to end. Something like the following:

process subprocess_handle;
fork
  begin
    subprocess_handle = process::self(); // Get process handle object for subprocess.
    // Wiggle bus wires and pass time as required do drive transaction
  end
join_none
if (flags & BLOCKING_TRANSACTION) begin
  #0;                        // Ensure that the subprocess has a chance to execute, so that the subprocess_handle will not be null.
  subprocess_handle.await(); // Wait for the subprocess to finish
end

do_read_internal() task using fine-grain process control to wait for subprocess

The two options above work fine when you are starting a single subprocess, and then want to conditionally wait for it to finish, but what if you are starting an indeterminate number of processes and want to wait for them to all finish? In this case the behavior of wait fork to wait for all subprocesses is a really convenient feature. However as we have seen, when calling wait fork you need to ensure you know which subprocesses it will actually wait for. The only way to really do this for sure, is to fork a new process within which we carefully control which processes are started. This does have the overhead of creating an extra another process, but this could be reasonable trade-off in some cases. An example of this implementation is shown below.

fork begin // Start a new process for the content of the task
  fork
    begin
      // Wiggle bus wires and pass time as required do drive transaction
    end
  join_none

  if (flags & BLOCKING_TRANSACTION) begin
    wait fork; // This call is now safe, because we are gaurenteed there is only one subprocess.
  end
end join // Block until this wrapper process finishes.

do_read_internal() task using an extra subprocess to control the scope of wait fork

Conclusions

In the end, I chose to take the first option and replace the wait fork with a wait on a done event, and everything works fine. However, I am also now taking a much more critical look at all other uses of wait fork in our repository, as I now have a better appreciation for some of the pitfalls of this statement.

There are certainly places where it is mostly safe. Use of wait fork directly within a fork/join block is generally safe because it’s harder to get an unexpected subprocess. Likewise use within a UVM sequence body() task also appears to be fairly safe, because sequence bodies are always started in a new process. However, even in these fairly well controlled uses it is important to remember that any task or function that is called could start an unexpected infinite subprocess, and then we’re right back to the same problem.

I’m not saying don’t use wait fork, but I do think it needs to be used with caution. It is easy to forget, or not be aware of, what all it will wait for.

Considering SystemVerilog Tagged Unions

prbs23@prbs23.com (prbs23) — Sat, 22 Jan 2022 11:42:00 -0700

I have recently been thinking a lot about tagged unions in SystemVerilog, since I discovered them a few months ago. In this post I present some of the ideal use cases for tagged unions, and why I think no one actually uses them.

Despite being initially proposed in 2003, and officially part of the language since the SystemVerilog 1800-2005 standard was released, at this point tagged unions appear to be a forgotten language feature. I think this is a shame, because since I accidentally ran across the tagged union section of the LRM, I keep thinking of new applications where I would like to make use of them. Don’t get me wrong, there are plenty of reasons that you don’t see tagged unions used in the wild today (language ergonomics, tool support, etc.) but that doesn’t mean they couldn’t have been useful.

Quick Introduction to Sum Types

Tagged unions, commonly also referred to as “sum types”, at least in SystemVerilog, I like to think of a combination of two more commonly used data types, unions, and enums. Like an enum, the primary value (tag) of a tagged union variable can be one of several values, as defined by the type declaration. However, unlike a normal enum, along with the “tag”, the tagged union variable can store another value of a type defined by the tag. Each tag value can define a different type of value held by the union when the variable has that tag value. The syntax of tagged unions allow the compiler to enforce that the type of value assigned to the tagged union matches the type as defined by the tag, and it is always possible to know the type of value stored in the variable by looking at the tag, unlike a regular union.

By way of example, in the following code, we declare a tagged union type int_or_float_t, when assigned the tag INT_VAL it is guaranteed by the compiler to hold an integer value, and when assigned the tag FLOAT_VAL always a float value.

typedef union tagged {
  int   INT_VAL;
  float FLOAT_VAL
} int_or_float_t;

Simple tagged union example.

Unlike a normal union, if you assign a int_or_float_t variable to INT_VAL with an associate integer value, the language will not allow us to refer to the integer value as a float. It will either be a compile time or runtime error, depending on the syntax used.

Why not to use tagged unions

Before getting into why I think tagged unions should be used more, why have they been essentially forgotten in SystemVerilog? I believe this mainly comes down to two issues: lack of support from major simulator vendors and language ergonomics.

Lacking Support

Of the three major commercial simulators I recently tested, one (Cadence Xcelium) didn’t support tagged unions at all, and the other two (Mentor Questa and Synopsys VCS) have varying degrees of partial support. In my opinion this is the biggest reason we don’t see tagged unions used, if a feature isn’t supported by all of the big three simulators then it can’t be used in any portable VIP or standard libraries. Even in the two simulators that did have support, sadly neither supported constraining or randomizing tagged union variables. Being able to randomize both the tag and values of a tagged union would provide some interesting capabilities as I will discuss later.

Ergonomics

The other problem with tagged unions is that the syntax for using them is in some ways… clunky at best. The pattern matching case and if syntax, which is (pretty much) required to check the tag and access values of the tagged union are both fairly verbose with lots of additional tagged, matches kind of keywords. Even assignments to the union value (tag) require an additional tagged keyword. Additionally, unlike other language which have an equivalent tagged union or sum type, SystemVerilog doesn’t provide a way to “parameterize” or “template” the types held by a tagged union.

Between lack of support, and generally feeling like a feature that was added to the language without a huge amount of thought, I can’t really blame anyone for not using tagged unions. However I am, none the less, disappointed that I can’t use them because they are not supported by the tools I use every day. After having used sum types in other languages, I think they could bring a lot of value to verification code written in SystemVerilog.

Uses of Tagged Unions

Now if we ignore the legitimate issues with tagged unions discussed above, what could they be used for? Note that the proposals below are pretty much all currently theoretical. I haven’t tried to implement these, and some are even impossible today due to missing support in current tools (especially around randomization and constraints). Nevertheless, I think these ideas are worth more investigation, and would love to hear from anyone who has done anything similar in SV.

Error Handling using “Result” Types

Let’s face it, error handling in most SystemVerilog code is bad. The language itself does not provide “exceptions” or any other architected error handling pattern, so most code just doesn’t handle errors. When an error does happen, most code will either log an error message then give up, or it will log an error and carry on as if nothing happened. From a software engineering perspective, this first off just bad practice, but also can lead to unpredictable behavior following the first error in a simulation.

We can do better though! Another language that doesn’t have exceptions, but does have sum types is Rust. In Rust, the error handling problem is implemented using a Result tagged union type (called enum in rust). Any function that could fail returns a Result<T, E> value, if the function succeeds then an Ok value is returned with a return value of type T. If the function fails for some reason, then an Err value is returned, along with an E typed value representing the kind of error that occurred.

The nice thing about this error handling technique is that it requires no additional features from the language (such as exceptions), but provides a consistent way for functions to return errors to their caller in a way that cannot be ignored. The compiler doesn’t allow you to use the return value from the function without either handling errors, or at least explicitly ignoring them. In my experience, this pattern works well in Rust, and I think it could be a good pattern for error handling in SystemVerilog.

Below I show a “Result” type implemented in SystemVerilog with tagged unions. Because SystemVerilog doesn’t support parametrizing tagged union types, we can’t create a generic Result<T,E>type, but a simple macro could be used to automate the creation of result types. In this case, I am declaring the handling code for deserializing a byte stream into a packet where it is necessary to check packet length and a packet CRC, either of which could result in an error.

typedef struct { int val1; int val2; } packet_t;  // Define a packet type
typedef enum { SIZE_ERR, CRC_ERR } error_e;       // Define a error type

// The Result tagged union type
typedef union tagged {
  packet_t Ok;
  error_e Err;
} packet_result_t;

// Deserialize function, returning packet_result_t type
function packet_result_t deserialize(byte data[$]);
    packet_t pkt;

    // Check for errors, returning an error enum value if there is one.
    if (data.size() != 16)
        return tagged Err SIZE_ERR;
    if (compute_crc(data[0:14]) != data[15])
        return taged Err CRC_ERR;

    // Extract the values
    pkt.val1 = {<<{data[0:3]}};
    pkt.val2 = {<<{data[8:11]}};
    
    // Return the packet value
    return tagged Ok pkt;
endfunction;

"Result" tagged union example.

The other half of this is how do you actually use the returned result type value. This is done using the “pattern matching conditional” syntax, as described in section 12.6 of the LRM. There are two forms of pattern matching conditionals, one for case statements, and one for if statements. This example shows using both to handle the “result” from the previously defined deserialize() function.

packet_result_t result;
result = deserialize(bytes);

// If pattern matching syntax. In this case errors are being ignored.
if (result matches (tagged Ok .pkt)) begin
    // Use the packet value
    // Inside this context, the `pkt` name can be used to refer to
    // the packet_t typed "Ok" value.
end

// Case matching syntax. This time we will actually do something with
//  the errors.
case (result) matches
    tagged Ok .pkt:
        begin
            // Use the packet. Once again `pkt` can be used to refer
            // to the packet_t value.
        end
    tagged Err .e &&& e == SIZE_ERR:
        begin
            // Handle specifically the packet size error condition.
        end
    tagged Err .e:
        begin
            // Handle any other error value. Once again the `e` value
            // can be used to determine the error type.
        end
endcase

Example showing how the result of the deserialize() function can be used.

Transaction, Packet, or Opcode subtypes

In design verification a common thing to do is to declare a class that represents a packet, opcode, or some other kind of “transaction”. These transaction classes contain variables that hold data from fields of the packet, or operands of the opcode, but what do you do when the transaction can have a different set of data values, depending on the type of transaction. In my experience I have seen two approaches used, either the transaction class contains a superset of all possible fields for the transaction, or sub-classes of the primary transaction type are declared for each transaction format, containing only the valid fields. The problem with including the super-set of fields is the user has to know which fields are valid in a transaction at any given time, and there is nothing to stop you from using an invalid field by mistake. On the other hand, declaring sub-classes can be very verbose, and requires a lot of explicit casts to work with values of the transaction.

However, tagged unions give us another option. As an example (inspired by one of the tagged union examples the SystemVerilog LRM), say we wanted a transaction that represents a CPU instruction where each instruction format has a different set of operands. We could define the instruction using a class that contains an anonymous tagged union as shown below:

class Instruction extends uvm_sequence_item;
    // Main opcode definition
    union tagged {
        // Binary arethmatic instruction format supports add, subtract, multiply, and divde
        // opcodes, with three operands, two source registers and one destination register
        struct {
            enum { ADD, SUB, MUL, DIV } opcode,
            bit [4:0] r1, r2, rdest;
        } BIN_FMT

        // Jump format instruction has a single destination literal operand with three
        // different opcodes.
        struct {
            enum { JMP, JEQ, JNE } opcode;
            bit [9:0] target;
        } JMP_FMT;
    } format;

    // Additional metadata fields can still be added to the transaction class.
    time start_time;
endclass : Instruction

Example of using tagged unions to express transacion with different field formats.

With this construction, the Instruction transaction class can hold all the operands for all instruction formats, but the compiler will only allow us to reference the fields for the instruction format actually used by the current value of the Instruction transaction. This sounds similar to the sub-class approach, however where with a sub-class implementation you would need to cast the Instruction to a BinFmtInstruction to access the r1 field (for example), in this implementation it is possible to refer to any field without casts as long as the format is correct. For example if we have already checked that format is BIN_FMT the expression inst.BIN_FMT.rdest will reference the rdest value of the instruction with no casts.

Is this actually better than the alternatives though? I believe it is a significant improvement over defining a transaction with a super-set of fields (the most common approach), since it ensures correctness when referencing the transaction fields. I am not sure it’s that much better than defining class sub-types for each packet format, since both require that you explicitly check the transaction format before you can reference values. However, I think the tagged union implementation would make the code easier to read with fewer $cast calls and variables of different subtypes hanging around.

Configuration

The final application where there could be real benefits from using tagged unions is in configuring sequences and components. Particularly where there is some mode setting where each mode has a different set of options that can be configured. Take the example of a sequence which generates transactions with a configurable traffic pattern. This might normally be implemented like the following:

typedef enum {PERIODIC, RAND, BURST} traffic_pattern_t;
class traffic_seq extends uvm_sequence;
    traffic_pattern_t pattern.
    rand time periodic_interval;
    rand time rand_min;
    rand time rand_max;
    rand time burst_interval;
    rand time burst_duration;
endclass : traffic_seq

Traditional implementation of a configurable traffic sequence.

In this implementation, the periodic_interval variable is only valid when pattern == PERIODIC. Likewise, the value of rand_min and rand_max are only used when pattern == RAND. At best it is confusing that you can set periodic_interval when pattern is set to BURST. A better way you could implement the same concept with tagged unions would look like the following:

class traffic_seq;
    union tagged {
        struct { time interval;                } PERIODIC;
        struct { time min;      time max;      } RAND;
        struct { time interval; time duration; } BURST;
    } pattern;
endclass : traffic_seq

Configuration traffic pattern sequence using tagged unions.

With the tagged union implementation the min and max values only exist (from the compiler’s perspective) when pattern == RAND. Additionally you cannot set pattern without setting it’s parameters. As an to set the pattern to BURST you would use an expression like:

seq.pattern = tagged BURST '{interval: 100ns, duration: 1ns};

I like this tagged union approach because not only is the assignment expression fairly concise and explicit, but it eliminates a whole class of errors and confusions. The configuration parameters for given traffic pattern type can only be accessed within the context of that pattern. There is no way to accidentally set periodic_internval instead of burst_interval. This tagged union design pattern could be applied to any use case where there is a setting where each option has it’s own parameters. For example you might use this for a configuration that controls a random distribution modes (think normal, uniform, geometric, etc.) where each distribution has it’s own unique arguments.

Wrap Up

As I mentioned at the beginning, the potential uses for tagged unions in SystemVerilog has been something I couldn’t get out of my mind for a few months. Although I have not been able to actually do a lot of work with them (due to Cadence lacking any kind of support), I think there are several good applications for tagged unions which would improve the readability and correctness of verification code, through syntax and compile time checks. Unfortunately, while tagged unions appear to have a lot of advantages, lacking tool support and poor syntax ergonomics sadly make it pretty much a non-starter from a practical perspective.

If anyone has actually used SystemVerilog tagged unions, I would love to know how you used them and how they worked out.

Building A Picture Frame From Mars

prbs23@prbs23.com (prbs23) — Sun, 14 Nov 2021 13:16:20 -0700

This whole project started when I happened across the NASA Perseverance Raw Image Library one day. Something about the constantly varying combination of images: some with scientific purposes, some for navigation, and some of the rover itself, along with the fact that they were FROM MARS kept me coming back to the page to see what Curiosity had been up to lately. This eventually lead to the idea of having a digital picture frame which would cycle through the latest raw images from the Perseverance rover. So that’s what I set out to build.

Finished Perseverance Digital Picture Frame

The Hardware

To start with, I would need a digital picture frame with internet access. There were two main directions I could have taken here; build a picture frame from scratch, or somehow modify a commercial frame. On one hand I could connect a Raspberry Pi to a large LCD panel and stick it in a picture frame, this would give me a very customizable platform to develop on, at the cost of some extra effort and being a less integrated solution. On the other hand, I could just buy an off the shelf Wi-Fi enabled digital picture frame and figure out how to write my own software/firmware for it. This would be easier from a hardware perspective, but I depending on what hardware and software ran on the picture frame it could make programming for it more difficult.

In the end I decided to use an off the shelf Wi-Fi photo frame, mainly because of cost, I was able to buy an full integrated picture frame for less than the cost of a just the LCD panel required to build my own. When I was comparing the Wi-Fi picture frames on the market I found something interesting, almost all of them used one of two platforms to enable the user to upload pictures to the frame: OurPhoto or Frameo. From what I could learn, both these platforms provide smart picture frame firmware to various manufactures which integrate with their platforms. While I was unable to find any useful information about the content of this firmware, the fact that it was designed to run across many different hardware models gave me some confidence that I wasn’t going to to find a highly specialized or low level firmware in these devices. I ended up buying an AKImart 10.1 Inch Smart WiFi Digital Photo Frame for a little over $100, as it struck a decent balance between size and being cheap enough I was okay with potentially bricking it.

“It’s an ANDROID system”

My digital picture frame turned out to be running Android 6.0.1

Overall there was very little information online about what hardware I could expect to find in the picture frame, so I didn’t really know what I was getting myself into. My best guess was I would find some kind of ARM single board computer, running either a customized Linux OS or a specialized Android ROM. It didn’t take long to figure out though, because when I first turned on my picture frame, the first thing I was greeted with was an Android setup wizard. This could either of been a really good thing (lots of development resources) or a bad thing (Android can be locked down well if properly configured).

The “Digital Picture Frame” that I bought turned out to essentially be a stripped down tablet without a battery running Android 6.0.1. While the OS was configured such one couldn’t escape the Frameo “Frame” app and use it as a normal Android device, fortunately the ADB USB debugging interface was left enabled, and even had root permissions. From the ADB shell I could install and launch any app I wanted, so all I needed now was an Android app that would show the latest pictures from Perseverance.

Perseverance Raw Image API

Now that I knew I should be able to write my own software for picture frame, I needed to figure out how to actually get the latest pictures from the raw image library. Initially, I was hoping to find public API of some kind to retrieve the pictures, maybe a simple RSS would be nice. NASA does in fact have several open APIs for various programs, they even have a Mars Rover Photos API. Unfortunately though this API does includes images from many previous mars rover missions, including Curiosity rover mission, it does not provide access to the Perseverance images. While there was no public API for me to use in the end, I did eventually find that the Perseverance Raw Image Library webpage makes requests to an undocumented but open API to retrieve a list of images to display.

Fortunately, the API used by the raw image library page was fairly simple to reverse engineer, at least as far as was required for my purposes. All requests were sent to https://mars.nasa.gov/rss/api/ with various query parameters that controlled the type of request and content returned, which in all cases was a JSON object. The first step was to understand how the various query parameters worked. There were some query parameters that were fixed, in all the requests I could observe:

feed=raw_images
Presumably there are multiple data feeds that could be served through this API.
feedtype=json
The data returned by the API was json encoded, so this makes sense, but even an obvious alternative such as feedtype=xml resulted in a 404.
ver=1.2
Likely the API version number.
category=mars2020,ingenuity
The existence of this argument implies there could be other raw image feeds, if only we knew the right string. Once again, I have yet to find other valid categories, and no way of listing the available categories.

Presumably there are alternate valid values for these parameters, but I was never able to find any that did not return a 404 error code. In addition to these fixed parameters, there were two parameters that controlled the format of JSON object, resulting in three different message formats, as indicated by the type field in the response.

Latest Images Summary

When latest=true is included in the request query parameters, a latest image summary object is returned, as indicated by a mars2020_ingenuity-latest-images-1.2 value in the type field:

{
  "type":"mars2020_ingenuity-latest-images-1.2",
  "total":161829,
  "new_count":389,
  "latest_sol":258
  "latest": "2021-11-11T15:49:19Z",
  "sol_count":55,
  "latest_sols": [
    258,
    257,
    256,
    208
  ],
}

https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&ver=1.2&latest=true

The content of this response message does not appear to change with any other query parameters. The content of the message gives a few useful pieces of information:

total
Appears to be a total number of images available through this API. It does not reflect the total number images in the Perseverance/Ingenuity category.
new_count
Number of “new” images, which appears to be the number of images published in the last 24 hours.
latest_sol
The most recent “sol” (number of mars solar days for the rover since the beginning of it’s mission) for which images have been published.
latest
UTC time that the image was received on earth from the rover. Interestingly it appears that images can be buffered for a surprisingly long time before being received, I have seen images with almost two (earth) months between being captured and received.
sol_count
Based on the name and values returned, this appears to be a count of mars solar days, but it’s not clear what it represents.
latest_sols
List of “sols” for which pictures have been published in the last 24 hours.

Image Detail Response

When an id= query parameter is included in the

 2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

https://mars.nasa.gov/rss/api/?feed=raw_imag

request, an image detail response object is returned, corresponding to the image with the given unique id. This response format for this response is indicated by the type value of mars2020-imagedetail-1.2. ht:300px;"> class="lnt"> 1 language-javascript" data-lang="javascript">{ "image": [ { "extended": { "mastAz": "21.5952", "mastEl": "24.5965", "sclk": "689864742.333", "scaleFactor": "4", "xyz": "(0.0,0.0,0.0)", "subframeRect": "(1,1,5120,3840)", "dimension": "(1280,960)" }, "link_related_sol": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?sol=258", "sol": 258, "attitude": "(0.81634,-0.0556483,-0.000426989,0.574885)", "json_link_related_sol": "https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&sol=258", "image_files": { "medium": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_800.jpg", "small": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_320.jpg", "full_res": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01.png", "large": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_1200.jpg" }, "imageid": "NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J", "camera": { "filter_name": "UNK", "camera_vector": "(0.8452750616932372,0.3358633696321663,-0.41557895401333894)", "camera_model_component_list": "(0.972477,0.396478,-2.02505);(0.843847,0.347228,-0.40911);(265.584,909.072,-265.014);(689.368,283.332,476.213);(0.843346,0.348554,-0.409015);(1.736e-06,0.0501396,-0.0171254);(-8e-09,1e-08,-2.9e-08);2.0;0.0", "camera_position": "(0.972477,0.396478,-2.02505)", "instrument": "NAVCAM_LEFT", "camera_model_type": "CAHVORE" }, "caption": "NASA's Mars Perseverance rover acquired this image using its onboard Left Navigation Camera (Navcam). The camera is located high on the rover's mast and aids in driving. \n\nThis image was acquired on Nov. 11, 2021 (Sol 258) at the local mean solar time of 18:02:21.", "sample_type": "Full", "date_taken_mars": "Sol-00258M18:02:21.470", "credit": "NASA/JPL-Caltech", "date_taken_utc": "2021-11-11T01:10:13.211", "link": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?id=NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J", "link_related_camera": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?camera=NAVCAM_LEFT&sol=258", "drive": "0", "title": "Mars Perseverance Sol 258: Left Navigation Camera (Navcam)", "site": 8, "date_received": "2021-11-11T08:34:42Z" } ], "type": "mars2020_ingenuity-imagedetail-1.2", "mission": "mars2020,ingenuity" class="p">} es&category=mars2020,ingenuity&ver=1.2&feedtype=json&id=NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J

The image object contains a lot of detailed information about the specific image. For my purpose of just displaying images the images I found the following images useful:

image_files
Object containing URLs to four differently scaled versions of the image.
sol
Mission solar day number when the image was captured.
date_taken_utc
The UTC time-stamp of when the image was captured.
date_received
UTC time-stamp of when the image was received by NASA.
title
Short title for the image. This usually contains the sol that the image was captured and the name of the camera used to capture the image.
caption
Long form auto-generated caption describing the image.

Images List Response

Finally if neither latest=true or id= query parameters are included in the request an images list response, as indicated by the mars2020_ingenuity-images-list-1.2 type value is returned. I found that this type of request responded to a few different query parameters, to either control the amount of data returned or filter the list of returned images. Below are the parameters that I was able to reverse engineer:

num=<Integer>
Number of images to return in this request. There appears to be hard maximum of 100 image per request.
page=<Integer>
Page number of results to return. By default (page=0) the returned image list contains the first num images that match the given filters. For non-zero values of page the API returns num pictures starting at offset num * page.
order=<String>
String value controlling the order of returned images. The only valid values I have found are sol desc (ordered from newest to oldest by mission solar day number), and sol asc (ordered from oldest to newest by sol number).
sol=<Integer>
Filter returned image list to a single mars solar day number.
search=<String>
Filters the returned image list based on the camera used to capture the image. This is string of pipe (|) separated camera identifiers. The response to this query includes a data structure that lists the valid camera identifiers, but the current list appears to be: HELI_NAV HELI_RTE, NAVCAM_LEFT, NAVCAM_RIGHT, FRONT_HAZCAM_LEFT_A, FRONT_HAZCAM_LEFT_B, FRONT_HAZCAM_RIGHT_A, FRONT_HAZCAM_RIGHT_B, REAR_HAZCAM_LEFT, REAR_HAZCAM_RIGHT, CACHECAM, MCZ_LEFT, MCZ_RIGHT, SKYCAM, PIXL_MCC, SHERLOC_WATSON, SHERLOC_ACI, SUPERCAM_RMI, EDL_PUCAM1, EDL_PUCAM2, EDL_DDCAM, EDL_RUCAM, LCAM.
condition_2=<String>
Generally a starting date filter. This field can either filter based on day the image was taken using the form <yyyy-MM-dd>:date_taken:gte, or filter based on the time that the image was received using the form <yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ>:date_received:gte when used in conjunction with a condition_3 sol filter parameter.
condition_3=<String>
Either an end date or sol number filter. With no condition_2 parameter specified, or when condition_2 uses the “date_taken” format, this parameter is an end date filter using the form <yyyy-MM-dd>:date_take:lt. When condition_2 is a “date_received” filter, this parameter can be a sol number filter of the form: <comma separated list of sol numbers>:sol:in.

While it would seem that the condition_2 and condition_3 parameters appear as if they could provide more generic filtering features, they appear to be locked down to only these use cases on the server side. The response to this type of request looks like the following (lists abbreviated):

{
  "nav": [
    {
      "checkboxes": [
        {
          "value": "HELI_NAV",
          "label": "Navigation Camera"
        },
        ...
      ],
      "order": "4",
      "name": "Mars Helicopter Tech Demo Cameras",
      "about_text": "Cameras onboard the <a href='https://mars.nasa.gov/technology/helicopter/'>Ingenuity Mars Helicopter</a>"
    },
    ...
  ],
  "images": [
    {
      "extended": {"mastAz": "UNK", "mastEl": "UNK", "sclk": "666952979.177", "scaleFactor": "4", "xyz": "(0.0,0.0,0.0)", "subframeRect": "(1,1,1280,960)", "dimension": "(1280,960)"},
      "sol": 0,
      "attitude": "(1.0,0.0,0.0,0.0)",
      "image_files": {
        "medium": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_800.jpg",
        "small": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_320.jpg",
        "full_res": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02.png",
        "large": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_1200.jpg"
      },
      "imageid": "FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "camera": {
        "filter_name": "UNK",
        "camera_vector": "(0.6926460219832632,-0.7044446294451989,-0.15491692056253062)",
        "camera_model_component_list": "(1.10353,-0.008945,-0.729124);(0.884935,-0.17181,0.432865);(651.291,418.924,318.224);(186.623,-81.6605,689.135);(0.885293,-0.171227,0.432363);(1e-06,0.00889,-0.006754);(-0.006127,0.010389,0.004541);2.0;0.0",
        "camera_position": "(1.10353,-0.008945,-0.729124)",
        "instrument": "FRONT_HAZCAM_LEFT_A",
        "camera_model_type": "CAHVORE"
      },
      "caption": "NASA's Mars Perseverance rover acquired this image of the area in front of it using its onboard Front Left Hazard Avoidance Camera A. \n\nThis image was acquired on Feb. 18, 2021 (Sol 0) at the local mean solar time of 15:53:58.",
      "sample_type": "Full",
      "date_taken_mars": "Sol-00000M15:53:58.302",
      "credit": "NASA/JPL-Caltech",
      "date_taken_utc": "2021-02-18T20:44:29.175",
      "json_link": "https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&id=FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "link": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?id=FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "drive": "44",
      "title": "Mars Perseverance Sol 0: Front Left Hazard Avoidance Camera (Hazcam)",
      "site": 1,
      "date_received": "2021-02-22T20:06:53Z"
    },
    ...
  ],
  "per_page": "3",
  "total_results": 67201,
  "type": "mars2020_ingenuity-images-list-1.2",
  "page": 0,
  "mission": "mars2020,ingenuity",
  "total_images": 162120
}

https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&ver=1.2&num=3&page=0&&order=sol+asc

Depending on how many images are requested, these responses can be large, but the main fields of interest are:

nav
Structure containing the valid camera identifiers for the search query parameter. Since this structure appears to be used to render the filter UI on the raw image library webpage, it not only includes the camera identifiers, it also includes a human readable name for the camera.
images
List of image details objects for images matching the query filters. This uses the same data structure as the image object in the image details response.
per_page
Number of images actually returned per “page” by this request. Should match the num query parameter, unless it’s greater than the hard limit of 100.
page
The “page” number of images returned in this request. This should be the same as the page query parameter, if specified.
total_results
Total number of images that match the filters given in the request.

The Customized Picture Frame App

Once I determined that the picture frame was running Android with an open ADB shell, I knew it should be possible to write a custom app that would query the NASA raw images API and displays them fullscreen on the frame. The only problem was that I had never written an Android app before, but there’s a first time for everything. Once I reverse engineered the API used by the NASA raw image library webpage, and learned a little bit of Android development, the actual implementation of the app turned out to be fairly simple. While I don’t recommend anyone use my code for as any kind of reference, I have published all the code on my Gitlab project here: Mars Photo Stream.

My custom Mars Photo Stream app running on the picture frame, displaying information overlay.

The app mainly consists of two periodic looper handlers. One handler is responsible for updating a list of the latest 300 images by querying the NASA API every hour. The other handler is responsible for updating the image being displayed. It randomly selects one image from the list of latest images, downloads it, and then updates an ImageSwitcher widget with the new image. Other than these periodic handlers, I just needed a few buttons to handle pausing the automatic image advancement, skipping to a new image, and managing settings..

Overall the app is functionally at a point where everything working well enough that I am happy to just have it on the wall. It’s stable, does what I set out to accomplish, and has enough configurability for daily use. Of course there are still some issues and areas for potential future improvements that I may pick up at a later time.

Next Steps

The main issue with the picture frame is that somehow Frameo configuration the custom Android ROM to completely disable the soft navigation buttons (most importantly the back button). This is not a problem when operating inside my app, but when navigating the Android settings UI, this can be problematic. To close the Android settings app and get back to the picture frame app I currently need to “force stop” the settings app… not exactly convenient. I have tried many suggestions on how to re-enable the navigation bar, but so far nothing I have tried worked. If anyone is experienced with the configuration of the Android SystemUI package and has ideas on how to re-enable it, get in touch because this really the only major issue I have.

There are also two features new features that I would like to implement, but haven’t gotten around to yet. While I find most of the raw images have something interesting about them, occasionally the app will select an image that is just “static”, or a featureless sky, or something equally uninteresting. I would like to be able to have my app somehow automatically filter these images out before showing them. It seems possible that some kind of simple contrast analysis of the images should be able to identify these “uninteresting” images. Maybe I will tackle this problem when I want to start learning about image processing.

Conversely, there will also sometimes be a really interesting image shows up that I would like to be able to save for later. My initial idea was to add a button which would send an email containing a link to the current image to a pre-configured address. Unfortunately, I wasn’t able to find any simple APIs for sending an email in the background, so I haven’t bothered to figure this out yet. I would still like to have the feature, but I’m no longer sure that email is the correct tool, maybe there’s a better alternative.

Making Your Own

Now I know what everyone is saying, how do I make one of these for myself? Overall this project turned out to be a lot less complicated than I initially expected, and should be easy to reproduce. The following steps should be all it takes:

Get yourself a Wi-Fi digital picture frame. If you want to play it safe you can get the same AKImart 10.1 Inch Smart WiFi Digital Photo Frame that I used. I would expect that at least pretty much any Frameo Wi-Fi digital picture frame would also work, but I don’t know that they will all be unlocked in the same way the AKImart one was.
Either build or download the APK for my “Mars Photo Stream” app. The Releases should always have an APK for the latest release of the app.
Install the Mars Photo Stream APK onto the picture frame using the ADB tool using the following command:
```
$ adb install com.prbs23.marsphotostream-v0.1.0.apk
```
If you don’t want to ever use the picture frame using the intended Frameo application, then you can disable it. If you don’t disable the Frameo “frame” app, then every time the picture frame boots up it will ask if you want to launch the Frameo app or Mars Photo Stream. Depending on your use case you may or may not want this.
If you want to disable the Frameo app so it always boots into the Mars Photo Stream app, then run the following commands at the ADB shell:
```
$ adb shell
shell@ZN-DP1007$ su root
root@ZN-DP1007$ pm disable net.frameo.frame
root@ZN-DP1007$ pm hide net.frameo.frame
```
Configure the Wi-Fi setting for the picture frame so that it can access the internet.

If everything has goes as expected, the Mars Photo Stream app should now be running on your picture frame, and will automatically start displaying raw images from the Perseverance rover and Ingenuity helicopter.

In theory the Mars Photo Stream app could also be installed onto any Android phone or tablet and used as a picture frame, though I haven’t tested this. In this case you can directly download the latest APK file onto the device and install it. If there is enough interest I could likely get the app on the Google Play Store, but for a personal project I didn’t bother figuring out how to do this.

Bidirectional Delay in SystemVerilog

prbs23@prbs23.com (prbs23) — Fri, 22 Oct 2021 07:07:39 -0600

I recently wrote about detecting driver strength in SystemVerilog. That work actually came out of solving a larger problem: How do you build a bidirectional digital wire model that includes propagation delay?

Motivation

The problem I had was how to model, in a digital simulation, the behavior of a “channel” (wire) for a high speed SerDes link. At the symbol rates used for modern high speed SerDes links, we start running into the physical limitations of copper as a conductor: the attenuation, frequency response, and speed of signals propagating through the channel are all factors that cannot be ignored as they might usually be. An additional requirement that my channel model had, was that it needed to be bidirectional. For my gate models to function, I not only needed to model the propagation delay of signals through the channel, but I needed drive strength resolution to resolve the value of the channel with drivers on both ends.

The protocol for the PHY I was modeling specifies that the transmitter should only be enabled when it detects that there is a receiver connected to it. The way this detection is done is by having the transmitter sense the presence of the termination resistor in the receiver. This is convenient because it lets the transmitter do endpoint detection without any extra wires, but it does complicate the channel modeling, because a lot of SystemVerilog constructs do not resolve drive strengths in both directions.

Basic schematic diagram of system.

A First Attempt

While constructing a model that could inject bit flip errors and remain bidirectional was a little tricky, its not too difficult with the right set up of tranif gates. What turned out to be the tricky part was modeling propagation delay, because as far as I could find, all SystemVerilog constructs which introduced a delay, also behave as unidirectional drivers. So, how can I make a module where a driver change on either port will appear, after a delay, on the other?

Breaking Down The Problem

An important thing to remember is that in SystemVerilog a net value is really a combination of two components, a value (0, 1, z, or x), and a strength (highz, weak, pull, strong, or supply). It’s easy to to delay the value part, but in SystemVerilog this always loses the strength part. Considering this, I hypothesized that if I could decompose the net value into two separate signals (value and strength), then recompose them on the other side of the delay, I could make a strength preserving delay module. If that works, then I could put two of those circuits back to back to make a truly bidirectional delay module.

The basic idea was to first build a strength maintaining delay element. To do this I needed three pieces: To decompose the net value I needed a strength detector which would take in a net, a output a value proportional to the strength of the net’s driver. To recompose the net value I needed a variable strength driver which would take in two “values” a net value and a net strength, and would drive an output net with the specified strength and value. Finally I would need a controllable delay element so that the delay could be controlled by my testbench. Once I had the unidirectional delay, I could put two of these back to back, as shown in the system schematic below, to create the bidirectional delay I needed.

Schematic showing structure of first bidirectional delay module concept.

First, I needed a way to convert the strength of a driver on a net to a SystemVerilog value. This turned out to be a deep hole all on it’s own, and I’ve already written about it at length in my last blog post. There are a couple ways of doing this doing this with different trade-offs, but for the purposes of this model I used the Verilog only “sense net” strength detection method because it’s a nice fast standalone implementation. It detects supply strength drivers as strong but that was okay for my use case. The code for that is shown below, for details on how it works, that can be found in my previous post.

module sense_net_strength_detect (
  inout sig,
  output reg [2:0] strength
);
  
  wire strong_sense;
  wire pull_sense;
  wire weak_sense;
  
  assign (strong1, strong0) strong_sense = (sig === 1'bx) ? 0 : ~sig;
  assign (pull1  , pull0  ) pull_sense   = (sig === 1'bx) ? 0 : ~sig;
  assign (weak1  , weak0  ) weak_sense   = (sig === 1'bx) ? 0 : ~sig;
  
  cmos strong_gate(strong_sense, sig, 1, 0);
  cmos pull_gate  (pull_sense  , sig, 1, 0);
  cmos weak_gate  (weak_sense  , sig, 1, 0);

  assign strength = (
    (sig === 1'bz) ? 0 :
    (sig === 1'bx) ? (
      (pull_sense   !== 1'bx) ? 1 :
      (strong_sense !== 1'bx) ? 2 : 3
    ) : (
      (weak_sense   === 1'bx) ? 1 :
      (pull_sense   === 1'bx) ? 2 : 3
    )
  );
endmodule

sense_net_strength_detect.sv

The next piece I needed was a module that has “value” and “strength” inputs, and will drive an output node high or low with the given strength level. There turns out to be a few ways to do this, but the cleanest approach I found was to use two assign statements for each strength level: One which can only drive a '1 value with the given strength, and one which can only drive a '0 value with the given strength. Then the value of the assign statement is the “value” input ORed or ANDed with a strength value comparison so that only the correct assign statement is active for the given value of the “strength” input. The code for this module looks like the following:

module multi_strength_driver (
  input       value,
  input [2:0] strength, // highz = 0, weak = 1, pull = 2, strong = 3, supply = 4
  inout sig
);
  assign (supply0, highz1 ) sig = value || (strength != 4);
  assign (highz0,  supply1) sig = value && (strength == 4);
  assign (strong0, highz1 ) sig = value || (strength != 3);
  assign (highz0,  strong1) sig = value && (strength == 3);
  assign (pull0,   highz1 ) sig = value || (strength != 2);
  assign (highz0,  pull1  ) sig = value && (strength == 2);
  assign (weak0,   highz1 ) sig = value || (strength != 1);
  assign (highz0,  weak1  ) sig = value && (strength == 1);
endmodule

multi_strength_driver.sv

Finally, the programmable delays could be simple assign statements with a delay value. Since I’m breaking up the strength and value of the signal, and providing independent data paths in each direction, there is no need to try to preserve drive strength in these assignments or use bidirectional gates.

Putting It All Together (For The First Time)

Now that I had all the pieces, I just needed to stick them together and try it out. For each of the two inout, I created a “sense_strength” signal that was the value representation of that net’s drive strength, and two reg signals to hold the delayed value and strength values for the multi_strength_driver, from the other side of the link.

module bidir_delay_initial #(
  parameter DELAY_WIDTH=32
) (
  inout sig_a,
  inout sig_b,
  input [DELAY_WIDTH-1:0] delay
);
  wire [1:0] sense_strength_a;
  reg        drive_value_a;
  reg  [1:0] drive_strength_a;
  wire [1:0] sense_strength_b;
  reg        drive_value_b;
  reg  [1:0] drive_strength_b;
  
  sense_net_strength_detect sense_a(.sig(sig_a), .strength(sense_strength_a));
  sense_net_strength_detect sense_b(.sig(sig_b), .strength(sense_strength_b));
  multi_strength_driver drive_a(.sig(sig_a), .strength(drive_strength_a), .value(drive_value_a));
  multi_strength_driver drive_b(.sig(sig_b), .strength(drive_strength_b), .value(drive_value_b));

  initial begin
    drive_value_a    = 1'd0;
    drive_strength_a = 2'd0;
    drive_value_b    = 1'b0;
    drive_strength_b = 2'd0;
  end
  
  always @(sig_b)            drive_value_a    <= #(delay) sig_b;
  always @(sense_strength_b) drive_strength_a <= #(delay) sense_strength_b;
  always @(sig_a)            drive_value_b    <= #(delay) sig_a;
  always @(sense_strength_a) drive_strength_b <= #(delay) sense_strength_a;
endmodule

bidir_delay_initial.sv

To test my solution I created a basic little testbench. On one side (side A) of my delay module I put another instance of my multi-strength driver module, so I could dynamically control the strength of the driver on that side, and to the other (side B), I placed weak strength driver to 0

Simple bidirectional delay test bench.

For initial turn on I used a sequence that would start by driving a strong1, decreasing strength down to highz, and then increase the strength back to strong with a value of 0 on the net a. Between each step I used a 10ns delay, and the delay element was set with a fixed 2ns delay.

Unfortunately, my initial results were not what I was intending, I expected that the b net would be initially high, transition to 1'bx 2ns after the a driver changes strength to weak1, and then transition low 2ns after the a driver transitioned to highz. The a net should have been high until 4ns after the a driver strength goes to 1, when it should to go to a 1'bx. After the a driver goes to a strength of zero it should be pulled low by the pulldown on net b. What the simulation results, as seen below, actually showed was both nets a and b being driven high (with a strong driver) until the driver on side a drove a strong0, when both sides go to strongX.

Simulation waveforms from initial bidirectional delay module showing positive feedback latch behavior.

While not the intended result, an astute observer may have predicted this behavior simply from looking at the initial simplified schematic. By putting two of the same strength preserving delays back to back, I created a positive feedback loop in the net strength. After 2x the delay duration from time 0, the variable strength driver inside the delay module wll be driving the same value and strength as the strongest external driver on either side of the delay. If the external driver changes value or strength the internal driver will continue to drive the old value for 2x the delay duration. Unless the net is externally driven with a larger strength than it was before, the net will continue to hold the old value. This is somewhat akin to the latching behavior you get from a unity gain amplifier with positive feedback.

Breaking the Feedback Loop (a more correct delay)

To actually make this delay element to behave correctly, I needed to break the positive feedback loop caused when one of the variable-strength drivers in the delay element is the strongest driver on it’s net. I found that this is possible by only enabling the driver on port b when the driver on port a is not driving the net, and vice versa. While there isn’t a (native) way to know which driver is driving a net, a close enough approximation is if the detected value and strength of the net matches the value and strength inputs to the variable-strength driver on that net. If the strength and value match, then the driver on the opposite port is disabled, which breaks the feedback loop.

Schematic showing structure of bidirectional delay module with broken feedback loop.

While this is actually a conceptually simple way to break up the feedback loop, the trick turned out to be expressing it in SystemVerilog such that it wouldn’t cause zero-time loops in the simulator when resolving the bidirectional net values. After a decent amount of experimentation, the solution turned out to be correct partitioning of always blocks and some well placed #0 delays to force net resolution order. Note that there has to be some additional special handling for 1'bx values, because they can be the result of multiple drivers on one net. The code for my final bidirectional delay module is shown below.

module bidir_delay #(
  parameter DELAY_WIDTH=32
) (
  inout sig_a,
  inout sig_b,
  input [DELAY_WIDTH-1:0] delay
);
  wire [1:0] sense_strength_a, sense_strength_b;
  reg  [1:0] drive_strength_a, drive_strength_b;;
  reg        sense_value_a, sense_value_a;
  reg        drive_value_a, drive_value_a;
  reg        driving_a, driving_b;
  
  sense_net_strength_detect sense_a(.sig(sig_a), .strength(sense_strength_a));
  sense_net_strength_detect sense_b(.sig(sig_b), .strength(sense_strength_b));
  multi_strength_driver drive_a(.sig(sig_a), .strength(drive_strength_a), .value(drive_value_a));
  multi_strength_driver drive_b(.sig(sig_b), .strength(drive_strength_b), .value(drive_value_b));
  
  initial begin
    drive_value_a    = 1'd0;
    drive_strength_a = 2'd0;
    drive_value_b    = 1'b0;
    drive_strength_b = 2'd0;
    driving_a        = 1'd0;
    driving_b        = 1'd0;
  end
  
  always @(driving_a, sig_a) if (!driving_a) drive_value_a_delay <= #(delay) sig_a;
  always @(driving_b, sig_b) if (!driving_b) drive_value_a_delay <= #(delay) sig_b;
  always @(sense_value_a_delay) drive_value_b <= (sense_value_a_delay === 1'bx) ? ~sig_b : sense_value_a_delay;
  always @(sense_value_b_delay) drive_value_a <= (sense_value_b_delay === 1'bx) ? ~sig_a : sense_value_b_delay;
  always @(driving_a, sense_strength_a) if (!driving_a) drive_strength_b <= #(delay) sense_strength_a;
  always @(driving_b, sense_strength_b) if (!driving_b) drive_strength_a <= #(delay) sense_strength_b;
  always @(posedge driving_a) drive_strength_b <= #(delay) 2'b0;
  always @(posedge driving_b) drive_strength_a <= #(delay) 2'b0;
  always @(sig_a, drive_value_a, drive_strength_a, sense_strength_a) begin
    #0; // Evaluate this after all the inputs have settled to prevent extra delta cycles.
    driving_a <= (sig_a === drive_value_a) && (drive_value_a !== 1'bx) && (sense_strength_a == drive_strength_a);
  end

  always @(sig_b, drive_value_b, drive_strength_b, sense_strength_b) begin
    #0; // Evaluate this after all the inputs have settled to prevent extra delta cycles.
    driving_b <= (sig_b === drive_value_b) && (drive_value_b !== 1'bx) && (sense_strength_b == drive_strength_b);
  end
endmodule

bidir_delay.sv

Using the same testbench as before, simulating this modified delay module resulted in the waves shown below. With the positive feedback loop broken, we now get the expected behavior on the bidriectional nets. The a node gets pulled down by the pulldown on net bafter the 2x delay period, instead of latching the strong value, as the old implementation did.

Simulation results of bidirectional delay module showing expected behavior, after breaking feedback loop.

Remaining Issues

The most significant remaining issue with this design is that if the delay port is set to 0, it can cause an infinite zero-time loop in the simulator. For my use case, the I was able to externally ensure that the delay input never was set to 0, but if this was not possible, one could also add tranif gates to disconnect and bypass the drivers when the delay input was set to zero. The other significant issue is that it cannot handle supply strength net values, mainly because of the approach used to sense the net drive strength. If this was a significant issue, the DPI drive strength detection implementation from my previous blog post could be used in place of the sense-net one I used.

Conclusion

While there are still some corner cases with this bidirectional delay implementation that don’t quite match with reality, it has proven to be fairly solid in production. The fact that it was possible to implement in pure SystemVerilog is both convenient, and means that it should be portable to any simulator. In the end, this project gave me the opportunity to really dig into the details of how net resolution, strength modeling, as well as gate primetives actually work in SystemVerilog.

If you would like to play around with my bidirectional delay module, you can find an implementation on EDA Playgound here: https://www.edaplayground.com/x/2PdE

Drive Strength Detection in SystemVerilog

prbs23@prbs23.com (prbs23) — Tue, 21 Sep 2021 22:44:59 -0600

From time to time I will run across small problems or challenges in my day job which seem as though they should be simple enough but get stuck in my mind until they are fully investigated. This is one such issue I ran into a while ago: How can a verilog model determine the drive strength of a net value in SystemVerilog?

When I initially ran into this problem, I thought it would require a quick search through the LRM and I’d be on my way. However, after some searching and not finding any useful constructs in the language that would directly provide visibility into a net’s drive strength, I ended up on google, and all the recommendations were similar to this post from stack overflow: How to check signal drive strength?. It seemed the recommended solution was to format the net value using the "%v" format string, and then do string comparisons on the formatted value.

Having run into my fair share of string and string formatting performance issues in SystemVerilog in the past, the suggested “solution” was instantly ringing some alarm bells. This is the problem that latched into my mind, and I needed to know if there was a better way… but first, a quick refresher on the drive strength modeling features in SystemVerilog.

Drive Strength Modeling

In SystemVerilog (and Verilog before it), a “net” is a special type of variable construct which is used to structurally model a wire in a circuit. The value of the “net” variable is a combination of a 4-state logical value (0,1,z,x), and a strength value (highz, weak, pull, strong, or supply). The value of a net is determined by “driver” elements connected to the net, which can be either continuous assignment expressions, or gate level modeling primitives. If semantically a SystemVerilog net represents a wire in a circuit, and the logical value represents the voltage on the wire, then the strength of the net represents how much current can that wire can source (or said another way, the impedance of the driver).

For every delta cycle of a simulation, the simulator will determine the logical and strength values of each net, by looking at the value and strength of all driver expressions on the net. If a net has no active drivers, then the value of the net is “high impedance” represented as 'z in SystemVerilog 4-state values. If a net has one driver expression or primitive, then the value and strength of the net are equal to that of the driver. However, if there are more than one driver connected to a single net, the logical value of the net is determined by comparing the strength of each driver on the net. If there is one driver on the net that has a strength lager than the rest, or there are multiple drivers that all have the same largest strength value, and the same logical value, then the logical value of the net is determined by the logical value of the driver with the greatest strength. If there are multiple drivers that all have the same greatest strength on the net, but have different values, then the value of the net is “unknown” or 'x with the strength of the strongest drivers.

Extracted figures from SystemVerilog LRM showing combinations of different strength drivers (left) and resulting net value (right)

If the strength value of a net is determined by the relative strength values of the drivers on the net, then there must be a way to define the strength of a driving expression. In SystemVerilog, by default all driver expressions have a strong strength value, which for a lot of use cases, is all you would ever need. However, the strength of logical 1 and 0 values can be individually set for both continuous assignment expressions and gate primitives by including a token of the form (<strength>1, <strength>0) after the assign keyword or primitive name. For example the expression assign (strong1, weak0) foo = ~bar; will drive the net foo with a logical value of 1 and strength of strong, when bar is a logical 0, and drive a logical value of 0 with strength value weak when bar is 1.

As an example, one might use drive strength modeling when modeling an I²C bus, which uses pull-up resistors to pull the bus high, and open drain drivers in each device to pull it low. In such a model, the pull up resistors on the bus can be simply modeled as a pull1 continuous assignments with a value of 1'b1:

`1`	`assign (pull1, highz0) scl = 1'b1;`

Additionally, the multiple I²C IO drivers can be modeled as driving the bus using a continuous assignment expressions with a (highz1, strong0) strength expression:

1
2

assign (highz1, strong0) scl = device0_scl_value;
assign (highz1, strong0) scl = device1_scl_value;

This is is not just nice because it’s a concise way of having the simulator figure out the interactions between devices on the bus, but it does so in a way that structurally mirrors how the circuits work. So if the simulator can do all the work of understanding the strengths of different drivers on a net, and resolving the net to the correct value, why would you need to be able to determine the strength of a net? There’s a few use cases I have actually run into, but continuing the I²C example, one use case would be if you wanted to write an assertion to check the bus is never driven high with a strong driver, since according to the specification, this should never happen.

Three ways of detecting drive strength

As part of this investigation I came up with two new ways for SystemVerilog code to detect or observe the strength of a net’s driver, in addition to the initial format string based solution from Stack Overflow that kicked all this off. For all three of these approaches I will be using them to implement a generic strength detection module, that accepts the target net to detect the strength of as an inout port, and outputs a value from 0-4 indicating the drive strength of the net.

The idea being if you wanted to write an assertion to check that an I²C bus was never driven high with a strong driver, you could use the module to get the drive strength, and then put a simple assertion on that value, as shown below:

1
2
3

wire scl_strength;
strength_detect(.sig(scl), .strength(scl_strength));
assert property(@(scl) not(scl == 1 && scl_strength != 2));

The String Formatting Approach: Google me up some code

As already mentioned, if you start looking into how to detect the drive strength of a net in SystemVerilog, all the recommendations I could find on the first page of Google results suggested to use the $sformatf() function with the "%v" format string, which creates a string representation of the net value, including the drive strength. For example a net being driven to a value of 1 with a strong driver, when passed to the "%v" format string results in the string “St1”.

Generalizing this approach into a module might look something like the following code. Every time the inout signal sig changes, we format the value of sig to a string, and then do a bunch of string comparisons to figure out the drive strength of the net.

module str_strength_detect (
  inout sig,
  output logic [2:0] strength
);
  always @(sig) begin
    string stren_str;
    stren_str = $sformatf("%v", sig);
    case (stren_str)
      "HiZ": strength = 0;
      "We0", "We1", "WeZ": strength = 1;
      "Pu0", "Pu1", "PuZ": strength = 2;
      "St0", "St1", "StZ": strength = 3;
      "Su0", "Su1", "SuZ": strength = 4;
    endcase
  end
endmodule

str_strength_detect.sv

Now, this does work, but I had performance concerns. SystemVerilog is good at some things, but string manipulation is not one of it’s strong suits, and I have seen simulations crawl to a halt because of particularly large or complex string formatting and manipulation tasks. We will discuss the performance of this later, but suffice it to say, when I saw this as the recommended approach to detect the strength of a net’s driver, I couldn’t help but think: There has to be a better way!

“Sense Net” Approach: Using the simulator against itself

After re-reading most of the LRM a few times, I eventually convinced myself that within the confines of the SystemVerilog language, there is no syntax or API that returns an integer value with a net’s drive strength. However, while we can’t directly query the drive strength, we can observe the effect that the strength of a driver has on net value resolution! The basic idea is to connect drivers with various strengths and opposite value to the net which we are measuring the strength of, and observe if the net value changes. The simulator’s drive strength resolution will tell us if strength of the net is stronger, equal, or weaker than the applied driver based on if the value changes, goes to 'x, or remains the same.

The trick with this, of course, is we need to do this without actually ever changing the value of the net, because that would change the result of the simulation. It turns out we can get around this by using the cmos gate primitive, which is a unidirectional model element which (when enabled) will drive it’s output net with the same strength and logical value as it’s input net. This allows us to essentially create a “duplicate” net with the same driver values (logical and strength) as that of our target net, but which we can add additional drivers to without effect the real target net.

After some optimization, I ended up up with with the following module with the same ports as the str_strength_detect module above, but doesn’t use any strings:

module sense_net_strength_detect (
  inout sig,
  output [2:0] strength
);
  wire strong_sense;
  wire pull_sense;
  wire weak_sense;
  
  assign (strong1, strong0) strong_sense = (sig === 1'bx) ? 0 : ~sig;
  assign (pull1  , pull0  ) pull_sense   = (sig === 1'bx) ? 0 : ~sig;
  assign (weak1  , weak0  ) weak_sense   = (sig === 1'bx) ? 0 : ~sig;
  
  cmos strong_gate(strong_sense, sig, 1, 0);
  cmos pull_gate  (pull_sense  , sig, 1, 0);
  cmos weak_gate  (weak_sense  , sig, 1, 0);

  assign strength = (
    (sig === 1'bz) ? 0 :
    (sig === 1'bx) ? (
      (pull_sense   !== 1'bx) ? 1 :
      (strong_sense !== 1'bx) ? 2 : 3
    ) : (
      (weak_sense   === 1'bx) ? 1 :
      (pull_sense   === 1'bx) ? 2 : 3
    )
  );
endmodule

sense_net_strength_detect.sv

Now, while this solution is almost perfect, it has one limitation: It cannot observe the difference between supply and strong strength values. The reason for this is that the cmos primitive has one special case where the strength value is not passed through, if the input drive strength is supply the output drive strength is only strong. There’s a lot of cases where this limitation doesn’t matter (including the problem that got this whole thing started), and in those cases I actually really liked this solution because it’s simple and pure SystemVerilog. However, I wanted to know if there was a way around this limitation…

DPI-C API Method: Going even deeper

Now, as already mentioned, I believe SystemVerilog has no way to directly query the strength of a driver, however, what it does have is a well defined C interface API specification. The “Direct Programming Interface” chapters of the SystemVerilog specification defines a C API that in theory allows C code to directly interact with any SystemVerilog code running in any (complaint) simulator. One thing this API provides is a method to query the value of a net, including strength value information, using the vpi_get_value() function. Since you cannot call the DPI C directly from SystemVerilog (even though you can call other linked C functions), this approach was initially less interesting to me because I would need to add yet another language to my code stack, but I was curious how this solution would look.

While the SystemVerilog specification may define the DPI, it’s far from clear how the API is intended to be used. After more experimentation than I would like to admit, I ended up with the following solution: Define two C helper methods that my SystemVerilog code can call, one to get a chandle (pointer) to the net of interest, and one to return an integer representation of the net’s drive strength from the DPI call:

vpiHandle get_handle(char *path) {
  return vpi_handle_by_name(path, NULL);
}

int unsigned get_strength(vpiHandle mod) {
  s_vpi_value v = {vpiStrengthVal};
  vpi_get_value(mod, &v);
  switch (v.value.strength->s0 > v.value.strength->s1 ? v.value.strength->s0 : v.value.strength->s1) {
    case vpiSupplyDrive : return 4;
    case vpiStrongDrive : return 3;
    case vpiPullDrive   : return 2;
    case vpiWeakDrive   : return 1;
    case vpiHiZ         : return 0;
    default             : return 0;
  }
}

dpi_strength_detect.c

Note that the vpi_get_value() function actually returns two strength values for a net: The strongest driver on the net driving a 1 value, and the strongest value on the net driving a 0 value. To get the actual drive strength that the simulator will resolve the net to, we need to return the larger strength of s1 and s0.

Using these two helper C methods, we can make a simple module that calls the get_strength() C function every time the net value changes as shown below:

module dpi_strength_detect (
  inout sig,
  output logic [2:0] strength
);
  import "DPI-C" get_handle = function chandle get_handle(string path);
  import "DPI-C" get_strength = function int unsigned get_strength(chandle handle);
  chandle h;
  initial begin
    h = get_handle($sformatf("%m.sig"));
  end
  always @(sig) begin
    strength = get_strength(h);
  end
endmodule

dpi_strength_detect.sv

This solution works as intended, and can even differentiate between supply and strong drivers, but does have the complexity downside of needing to deal with and compile in C code. Most simulators provide relatively simple ways of compiling in DPI-C code, but it’s not as clean of a solution as I would have liked. If I could find a way of directly calling the DPI-C API functions from SystemVerilog code, this solution would have been more appealing.

Performance

I started digging into this issue because I had a hunch that the string formatting approach would have too much overhead, so does this really matter? Is there really any benefit to avoiding strings?

I built a very simple test bench which would cycle through all possible drive strengths with both 0 and 1 logical values 5 million times. Without a strength detection module in the model, only the multi-strength driver use to generate the signal, 5 million iterations took 14 seconds to simulate with the Xcellium Cadence simulator. The table below shows the simulation time when one strength detection module is added to the simulation for each approach I have discussed.

	No Strength Detection	Format String Method	Sense Net Method	DPI-C Method
Simulation Duration	14 Seconds	44 Seconds	25.6 Seconds	32 Seconds
Simulation Throughput	8.571 ms/s	2.737 ms/s	4.675 ms/s	3.75 ms/s
Relative Slowdown	-	3.14x	1.83x	2.28x

The data does indeed prove out my initial expectation that string manipulation is slow. However, I was quite surprised to see that the DPI-C method was slower than my initial SystemVerilog only approach. I would have thought that the additional nets and value resolution work the simulator would have to do for the sense net approach would have out-weighed any overhead in calling out to a C library.

Conclusion

Overall this was an interesting deep dive into a little explored part of the SystemVerilog language. If you can get away with not needing to differentiate strong from supply strength values, then I would definitely recommend the “sense net” detection approach as it’s not only the fastest, but also pure SystemVerilog. If you really need to detect supply strengths and the addition of two DPI-C functions isn’t problematic, then the DPI-C approach is a good choice. Either way, is better than the commonly recommended approach of using formatting the net value as a string.

About Me

prbs23@prbs23.com (prbs23) — Mon, 01 Jan 0001 00:00:00 +0000

I am a full time ASIC verification engineer, and part time electronics and software tinkerer. This blog is a place for me to write about pretty much any topic that piques my interest. Usually this ends up being a deep dive into a niche topic related to my work in ASIC design. However, I will be trying to post some content related to my non-professional software, firmware, and hardware projects. My goal here is to be informative, and I hope that someone learns something from my mistakes and the rat-holes I go down.

What is PRBS23:

Pseudorandom Binary Sequences are deterministic bit sequences with random-like statistical distributions, commonly generated by linear-feedback shift registers (LFSR). PRBS patterns are useful for many data integrity validation techniques, because they are usually random enough to be used in place of arbitrary data, but the whole pattern can be recreated from a single short seed. There are many different possible PRBS sequences, typically defined by length and polynomial representation, but one commonly used in high speed SERDES like PCIe, Ethernet, etc. is PRBS23. It is 2²³ bit (~1 MBit) long sequence generated from the following polynomial: x²³ + x²⁸ + 1, and can be generated using the following LFSR circuit:

Fibonacci LFSR implementation of PRBS23 Generator

As far as it’s relation to this blog, the name is just as random as a PRBS23. Many things here are related, but not in an obvious way…

PRBS23

Reverse Engineering the Gravity 800 Smoker Controller

Power Supply

Microcontroller

Pinout

UART ADC Subsystem

Multiplexed Push Buttons

Thermistors

Thermocouple ADC

Display

Segment Mapping

Initialization and Configuration

Backlight

Miscellaneous

Knob Encoder

Lid Switch

Fan Driver

Buzzer

Future Work

Announcement: FreeFall 800 (v0.1.0)

Getting Started With FreeFall 800

Building Firmware

Flashing Firmware

Connecting

Limitations

Future Work

Contributing

Getting Started With a Private LoRaWAN Network Using ChirpStack v4 (Updated)

Note:

LoRa and LoRaWAN Basics

Getting Started

End-Devices

Hardware

Firmware Networking Stack

Gateway

Hardware

Gateway Software

Servers

ChirpStack On the Gateway

ChirpStack in Docker Compose

Configuring the Network

ChirpStack Network Server

Gateway Bridge

Gateway Packet Forwarder

Configuring an Application Through the ChirpStack UI

Sending a Packet

Next Steps

Combining Dist Constarints in SystemVerilog

Basic Distribution Constraints

Applying Multiple dist constraints

Combining Distributions in Xcelium and VCS

Combining Distributions in Questa

Conclusions

Getting Started With a Private LoRaWAN Network

Note:

LoRa and LoRaWAN Basics

Getting Started

End-Devices

Hardware

Firmware Networking Stack

Gateway

Hardware

Gateway Software

Servers

ChirpStack On the Gateway

ChirpStack in Docker Compose

Configuring the Network

Network Server

Application Server

Gateway Bridge

Gateway Packet Forwarder

Setting Up an Organization and Application

Sending a Packet

Next Steps

Don't Wait For Me

An Unexpected Infinite Wait

A Not So Brief Introduction to wait fork

What Went Wrong?

Remediation

Conclusions

Applying Multiple `dist` constraints

A Not So Brief Introduction to `wait fork`