Lee reports the apps are working pretty well overall, although he’s still improving stability and refining a few parts of the implementation.
It’s great for the FreeDV team to see 3rd party developers using RADE in their projects. This has recently been supercharged by the innovative pure C implementation of RADE V1 from Peter VK3TPM.
Last month I mentioned Peter’s innovative work using Claude Code to perform a C port of RADE V1, and my concerns around Quality Assurance for AI generated code. The FreeDV project leadership team discussed this and we decided to apply the same quality standards to the C port as the reference Python implementation. We adapted a suite of automated tests developed for the Python version to the C port, and wrote a few new ones. Some of these were developed with the assistance of Claude Code. Through a team effort between Peter, Mooneer and myself these tests are now passing on the C port of RADE V1, which is now ready for initial deployment.
Claude Code for RADE V2 development
I have started using Claude Code for my development and it is indeed a game changer. It’s like having another DSP engineer to work with. It is more knowledgeable than myself in many areas, and is indeed teaching me some DSP tricks like SNR estimation for OFDM. Claude is much faster and more accurate than me at coding, especially around scripting and automated tests. This month Claude helped me prototype a End of Over (EOO) system and SNR estimator for RADE V2, various automated tests, and with refactoring the V2 Python code into a class.
The EOO system is a good example of AI assisted DSP development. The transmitter sends a EOO symbol at the end of it’s transmission. The receiver detects this symbol and cleanly stops decoding, preventing a trail of R2D2 noises at the end of the over. A first pass worked, but was unreliable, only detecting about 40% of the EOO symbols on multipath channels. I then “worked” with Claude on various tests and algorithms he suggested, until we had adequate performance. The constructive criticism went both ways – some of the ideas I had were flops, but I also had objections to some of Claude’s ideas. I learnt quite a bit, as Claude’s knowledge is encyclopedic and he had algorithm ideas I had never heard of. I quite like learning so this was an added bonus. Claude did all the coding and is way faster than me. Eventually we reached an algorithm that passes all the tests we have thrown at it so far, and works down to all but the lowest SNRs.
Turning to SNR estimation, here is a plot of the SNR estimator Claude helped me come up with. Turns out SNR is related to the autocorrelation of the cyclic prefix which we already compute. This works quite well for AWGN channel but (as usual) has significant errors for multipath channels. So I asked Claude to fit a correction that minimises the error over three channel – AWGN, multipath good (slow fading), and multipath poor (fast fading).
RADE V2 SNR estimator for three channels. The x axis is the actual SNR of the channel, the dashed line an ideal SNR estimator. Over the -5 to +5 dB range the RMS error is 1.5 dB, and maximum error 2.8 dB. It is optimised for the mid range (slow fading) MPG channel.
Tx Drive Levels and ALC
The next phase V2 development is stored file over the air (OTA) tests. For each test I generate a file consisting of a chirp – SSB RADE V1 – RADE V2 in samples, which are scaled so they have the same peak power.
As a first step I connected my HF transmitter to a spectrum analyser via a RF sampler and attenuator so I could measure the actual Tx power. I encountered a significant difference in Tx power for each sample, despite the peak drive level being the same. This improved (the difference was reduced) when I increased the drive level from 60% to 80% on my IC7200 test transmitter:
Peak power against time for chirp – SSB – RADE V1 – RADE V2 with USB 60% drive level (just tickling ALC) – significant differences in peak power of each signal.Peak power against time for 80% USB drive level (ALC half scale) – peak power of four signals more uniform, and has bumped up a few dB.
The 60% drive level followed the conventional FreeDV wisdom of setting Tx drive to “just tickle the ALC”. To explore this a little more I conducted some “over the cable” tests where I connected a FT-817 receiver to the transmitter via attenuators and coax (so no HF channel or noise to deal with). In this case I measured no difference in the measured loss metric of the received RADE V1 and V2 signals between two drive and power levels. This suggests that for modern FreeDV waveforms like RADE V1 and V2, the Tx drive level and amount of ALC is not critical. Indeed driving the radio a little harder so the ALC is engaged may increase the Tx power. It also shows that the path through the two radios introduces just a small amount of distortion to the RADE signals, a pleasing result. Of course this is just one result, with one radio, so I would encourage other experiments in this area.
Test
Condition
USB Drive
Power setting
ALC (bars)
Peak Power
RADE V1 Loss
RADE V2 Loss
Control
Software
0.108
0.082
Low power
OTC
50%
L
1
20W
0.108
0.079
High Power
OTC
80%
100%
4
70W
0.107
0.087
USB/ALC drive level tests. The “loss” is the same objective measure we use for training the network, and allows us to measure distortion of the received signal. The loss barely changed across two drive/power settings, indicating ALC engagement is not critical for this radio.
RADE V2 Stored file OTA testing
So with all the algorithms for RADE V2 in place, I started testing V2 over the air (OTA) using stored file tests, similar to tests performed during RADE V1 development. As mentioned above I generated a file consisting of a chirp – SSB RADE V1 – RADE V2 in sequence, then transmitted that from my home station to various KiwiSDRs around Australia. The chirp is used to estimate SNR of the channel, and SSB and RADE V1 are included for comparison. By sending them all together, we have them passing over roughly the same HF channel.
The purpose of these tests was (a) see if RADE V2 actually works over real world HF channels (b) compare it to RADE V1 to see if we get any improvement at low and high SNRs. The simulations predict a 3 dB increase in sensitivity at low SNRs. At high SNRs the ML network “loss” is significantly lower for V2, but we are not sure what that means in terms of speech quality.
Here are some samples from the OTC/OTA testing. The first set of samples is over the cable (OTC) so very high SNR. Can you hear any difference between RADE V1 and V2? Which one sounds closer to the input speech?
Input Speechv216 high SNR RADE V1v216 high SNR RADE V2V216 high SNR SSBHigh SNR spectrogram, chirp-SSB-RADE V1-RADE V2. Note the difference in bandwidth for RADE V1 and V2
This set of samples is from a low SNR (approx -5 dB) path 2000 km across Australia on 14.230 MHz:
Objective Loss Measure for OTA tests It can be hard to tell fine differences in speech samples using ad hoc listening tests such as playing the samples above. To help with this problem, the OTA test software also measures the “loss” between the vocoder features at the input of the RADE encoder, to the output of the RADE decoder. This can be considered a number that represents the distortion of speech passed through RADE. We use the same metric when training the ML networks, where the weights are adjusted to minimise the loss. We observe during training that speech quality increases as the loss is reduced. So we can argue that in general a lower “loss” represents increased speech quality. Here is a table of the “loss” for a selection of OTA and OTC samples:
Sample
Speaker
V1 software
V2 software
Channel
V1 OTA
V2 OTA
v208
David
0.102
0.077
NVIS
0.132
0.132
v209
David
0.102
0.077
NVIS
0.141
0.125
v211
David
0.102
0.077
DX
0.520
0.154
v216
Brian
0.115
0.086
OTC
0.123
0.096
v217
Brian
0.115
0.086
DX
0.311
0.198
Loss for various samples from the OTA test campaign. The “Software” columns are the loss from software encoding only, before transmission over the channel. A perfect channel (like v216) achieves very close to these loss values.
It can be seen that when sent over the same channel at (almost) the same time, the RADE V2 samples achieve an equal of lower loss than RADE V1. It’s interesting to compare the loss values for v216 and v217 to the samples above – do you agree with the ordering suggested by the loss values?
Conclusions
My conclusions so far:
RADE V2 works over the air for the channels (40m ground wave and NVIS, 20m skip) and two speech samples tested so far. No obvious functional problems over a total of 17 tests.
At low SNRs there is some evidence of V2 performance gain (measured loss values), but it is subtle and may be hard to hear using ad-hoc listening tests. Likewise for high SNR voice quality. It is fair to say V2 is no worse than RADE V1. Both RADE V1 and V2 are doing better than SSB.
The RADE V2 waveform has a fraction of the bandwidth of RADE V1 and SSB (99% power contained in 860 Hz).
After a few laps with Claude the End of Over (EOO) system is working on most OTA samples. The SNR estimator is functional but not accurate, with performance varying across channel types. End users will need to be briefed about it’s accuracy (up to +/- 3 dB error).
Some features I still need to test/demonstrate are the lower latency and streaming text feature.
This past month was spent working on the release of version 2.2.1, which primarily consisted of bug fixes discovered after the release of 2.2.0. A few additional bug fixes and performance improvements after 2.2.1’s release also got in.
Besides the work on the FreeDV application, most of my effort this month was spent preparing for Orlando HamCation (more information on how that went here). This consisted of preparing the slides for the talk as well as booth setup/teardown and manning the booth itself, where I (along with Walter K5WH and Mel K0PFX) answered questions from people interested about FreeDV.
For this following month (March), I hope to make some additional progress on any features that are approved during our upcoming meeting. There have actually been a large number submitted since our last meeting (in January since it was cancelled last month), so thank you guys for that! Time permitting, I also hope to investigate Peter’s work on the C implementation of RADEV1 to see what gaps still remain with it.
More information can be found in the commit history below:
I started this month with writing our annual report to ARDC, who have been kindly supporting our work since 2023. It’s been a great year for FreeDV. We believe we have achieved most of the goals set for the ARDC grant, using just a fraction of our budget. Highlights include:
With RADE V1 we have a high SNR (rag chew) mode that exceeds the quality of SSB.
We have improved low SNR operation such that RADE V1 exceeds the quality of SSB at low SNRs.
Maintained and improved our freedv-gui application, and supporting technology such as the FreeDV QSO finder.
Promoted FreeDV and RADE technology online using our website and at Ham/professional conferences.
Met our goals for a greatly expanded and growing user base.
As a bonus outcome, we have developed the BBFM technology that uses machine learning for high quality, robust speech communication over VHF/UHF.
RADE V1 C Port using AI
In February Peter Marks, a member of our FreeDV project leadership team, has performed some remarkable work with Claude Code – generating a C port of RADE V1 in one day. I estimate this work would take 3 months of my time, requiring a wide range of skills. To verify the port I applied a suite of automated tests that we use for the Python version of RADE V1. Most passed, but there are few small issues that need addressing.
This availability of RADE V1 as a pure C library opens up several exciting possibilities, such as making life easier for packaging and distributing RADE, smaller downloads and embedded operation inside radios on smaller processors.
However the use of AI raises the question of maintenance and continued Quality Assurance. Even if the code is efficiently generated – we still need some assurance that it works and will keep working as the AI adds more code. An appropriately skilled human still needs to be responsible for reviewing the work of the coder (AI or human) and signing off the work. It’s not OK to have efficiently generated code that has subtle bugs. The use of AI for large coding tasks is a new and exciting paradigm, and we need to work out the best way to use the technology but retain appropriate levels of quality. The FreeDV project leadership team is currently pondering this question.
RADE V2 Test Plan, AGC and Robustness to Sine waves
A simple automatic gain control (AGC) was developed and tested to handle varying signal levels into the V2 receiver. It can handle +/- 20 dB range of input signals and based on a series of spot tests and performance curves it doesn’t appear to add any measurable distortion. ML networks tend to be sensitive to input level, hence the need for some sort of AGC.
Now that most of the RADE V2 algorithms are written, I have been proceeding through a suite of tests. Over the years we have worked out just how to break digital voice systems over the air, so we simulate each of these conditions first, as issues are much easier to find and fix in simulation that over the air. These tests and the results have been documented in a test plan/report. I moved quickly through several tests, then came to a screeching halt when testing acquisition with sine waves present.
This is a common scenario – e.g. someone tuning up on a frequency near a valid RADE signal. Turns out the RADE V2 receiver was happily obtaining sync on a sine wave, which locked out any valid RADE signals. Once again, V2 sync is more complicated than V1 due to the absence of pilot symbols. A few days of R&D followed, and now we have an acquisition system that is more robust to in-band sine waves.
The figure below illustrates the problem with sine waves. This is a mesh plot of the autocorrelation of the input signal, which we use to determine if a signal is present. Time flows from right to left in this plot, from 0 to 1000 symbols (about 20 seconds at 50 symbols/s). We peak pick this at each time step to determine the timing offset, which is between 0 and 160 samples. The central mountain is a valid RADE V2 signal with a timing offset of about 80 samples. The horizontal bar starting near 0,0 is the sine wave, which doesn’t have a clear central peak. I’m using the ratio of the maximum of the minimum at each time step to distinguish sine waves from valid RADE V2 signals.
I also worked through some issues when one V2 signal closely follows another, and performed several other tests such as modelling sample clock offsets between the transmitter and and receiver, and frequency drift of the Rx. The 99% occupied bandwidth (OBW) is just 860 Hz and the spectrum much cleaner than RADE V1.
The next step in the test plan is to send some “stored file” V2 signals over the air using SSB radios, debug the issues this throws up, and confirm the results over real world radios and channels are similar to simulation.