- Posts: 1
- Thank you received: 0
NMEA Data Output Bugs: SegFault & Runaway Memory Leak
- blane_pegasusmarineLLC
- Topic Author
- Offline
- New Member
-
Less
More
16 hours 3 minutes ago #4009
by blane_pegasusmarineLLC
NMEA Data Output Bugs: SegFault & Runaway Memory Leak was created by blane_pegasusmarineLLC
qtVlm 5.12.26 (ARM64/aarch64) — Two bugs found and diagnosed
Reporter: Blane Boynton (tachyon-rory, RORY sailing vessel navigation system)
Contact: brboynton@gmail.com
Build affected: qtVlm 64 bits 5.12-26 (meltemus) ubuntu 24.04, downloaded as qtVlm-5.12.26-rpi5_64.tar.gz (bookworm/rpi5 64-bit ARM build) from meltemus.com
Platform: Ubuntu 24.04.4 LTS, aarch64 (Qualcomm Snapdragon SoC, not an actual Raspberry Pi — a generic ARM64 build was used and runs fine), glibc 2.39
Both bugs are 100% reproducible and were caught live with gdb attached, giving exact backtraces (excerpted below; full 17-thread backtrace attached as crash_backtrace.txt).
[hr]
Bug 1: Segfault when disabling "Send active waypoint data" while NMEA is connected
Reproduction
This reproduced every time in our testing while a live feed was connected — which is the normal/expected operating condition, so the crash is effectively unavoidable via the UI toggle in normal use.
Root cause
This is a thread race with no locking, caught live by attaching gdb, letting the process continue, and catching the SIGSEGV in the act. At the moment of the crash, two threads were touching the same object graph concurrently with no synchronization between them:
Unchecking "Send active waypoint data" triggers restartGPS() -> stopRead(), which destroys the instrument panel widgets on the main thread, while the NMEA read thread is still actively decoding data against the same objects with no lock/join between the two. This is a textbook use-after-free / destroy-while-in-use race.
Suggested fix
stopRead() needs to stop and join (or otherwise synchronize with) the NMEA read thread before tearing down the InstrumentPanel/NmeaSourceTcp object graph it feeds, rather than destroying it concurrently.
Backtrace excerpt (full 17-thread capture attached as crash_backtrace.txt)
[hr]
Bug 2: Memory leak in general NMEA output ("Send NMEA" — sendNmea), ~2.5MB/s
Reproduction / isolation
Bisected via ~/.qtVlm/qtVlm.ini config flags (edited while qtVlm was closed, relaunched, then sampled RSS via ps every few seconds for 20-30s per configuration):
This isolates the leak specifically to the general NMEA output path (sendNmea=1 in [nmea]), independent of and separate from the waypoint-specific output path (sendWPtoNMEA in [Boat_real], which sends bearing/distance/XTE/VMG for the active waypoint to a separate NMEA0183 TCP target). The waypoint-output path was confirmed leak-free on its own.
Severity
Left unattended, this leak previously crashed/froze the entire host machine, not just qtVlm — system-wide available memory collapsed to <100MB, the machine became fully unresponsive (input lag climbing past 1 second, then total lockup), and required a hard power cycle to recover. This is more severe than a simple app crash; it's a host-level DoS from a single client-side application setting.
Sample of raw growth data (RSS via ps, general NMEA output enabled):
(~64MB growth in 25 seconds, consistent and unbounded — observed to continue for 30+ minutes without leveling off in an earlier full-length test.)
Suggested fix
Whatever per-sentence or per-cycle object(s) qtVlm's general NMEA output path allocates when sendNmea=1 is enabled are apparently never freed. Recommend checking for missing delete/deleteLater() or a growing container (e.g. a send queue or history buffer) that's never pruned in that code path. We don't have source access so cannot point at an exact file/line — the shipped Qt libraries are stripped of symbols, so the leak's own allocation site didn't show up in our live snapshots (the process was normally idle/blocked in poll/futex_wait at each sampling instant, as expected for an incremental per-cycle leak rather than a hot loop).
Workaround in use
We are running with sendNmea=0 permanently — this loses whatever general NMEA output does (destination/purpose not identified from the UI), but our specific use case (feeding a Garmin NMEA2000 chartplotter/instrument bus with active-waypoint bearing/distance/XTE via Signal K) only needs sendWPtoNMEA+sendWPTcp+sendWpTransTcp, which is unaffected.
[hr]
Environment / diagnostic notes
apport is configured as this system's core_pattern handler, but its default policy silently discards crash reports for binaries not installed via dpkg (like this manually-extracted qtVlm tarball) — so no .crash or core file was ever saved automatically for either bug. Both backtraces in this report were captured by manually attaching gdb (sudo gdb -p <pid>, needed due to this system's ptrace_scope=1 restricting attach to non-child processes) either at the moment of the live segfault (Bug 1) or via a scripted watchdog that captures a backtrace + generate-core-file before killing the process once available memory drops below a safety threshold (Bug 2, used only as a safety net — this bug was reproduced deliberately for diagnosis, not caught accidentally).
Full core dump files (~4GB each) from the live gdb sessions are not included here due to size, but are available on request.
Attachments: crash_backtrace.txt (full gdb thread apply all bt full output, all 17 threads, for Bug 1)
Reporter: Blane Boynton (tachyon-rory, RORY sailing vessel navigation system)
Contact: brboynton@gmail.com
Build affected: qtVlm 64 bits 5.12-26 (meltemus) ubuntu 24.04, downloaded as qtVlm-5.12.26-rpi5_64.tar.gz (bookworm/rpi5 64-bit ARM build) from meltemus.com
Platform: Ubuntu 24.04.4 LTS, aarch64 (Qualcomm Snapdragon SoC, not an actual Raspberry Pi — a generic ARM64 build was used and runs fine), glibc 2.39
Both bugs are 100% reproducible and were caught live with gdb attached, giving exact backtraces (excerpted below; full 17-thread backtrace attached as crash_backtrace.txt).
[hr]
Bug 1: Segfault when disabling "Send active waypoint data" while NMEA is connected
Reproduction
- Have a live NMEA/AIS TCP feed connected (in our case, a local Signal K server).
- Open the boat/NMEA settings dialog and uncheck "Send active waypoint data".
- Click OK/close the dialog.
- → Immediate segfault, qtVlm crashes. "core dumped" per shell, but no core file is retained by the system in our setup (apport ignores non-dpkg binaries) — this report's backtrace was captured live with gdb instead.
This reproduced every time in our testing while a live feed was connected — which is the normal/expected operating condition, so the crash is effectively unavoidable via the UI toggle in normal use.
Root cause
This is a thread race with no locking, caught live by attaching gdb, letting the process continue, and catching the SIGSEGV in the act. At the moment of the crash, two threads were touching the same object graph concurrently with no synchronization between them:
Code:
Main GUI thread (handling the dialog close, tearing down the instrument panel):
DialogParamVlm::done(int)
-> boatReal::restartGPS()
-> boatReal::stopRead(bool const&;)
-> myCentralWidget::slot_loadInstruments(bool const&, bool const&, bool const&;)
-> InstrumentPanel::~InstrumentPanel()
-> GraphicsInstrumentCOG::~GraphicsInstrumentCOG()
-> QGraphicsObject::~QGraphicsObject() / QGraphicsItem::~QGraphicsItem()
-> ... down through QGraphicsProxyWidget / QLCDNumber / QWidget destructors ...
-> QBackingStore::~QBackingStore()
NMEA network-read thread (same instant, mid-flight decoding an incoming sentence):
NmeaSourceTcp::slot_readData()
-> NmeaItemXDR::decode(QByteArray const&;) <-- SIGSEGV here
Unchecking "Send active waypoint data" triggers restartGPS() -> stopRead(), which destroys the instrument panel widgets on the main thread, while the NMEA read thread is still actively decoding data against the same objects with no lock/join between the two. This is a textbook use-after-free / destroy-while-in-use race.
Suggested fix
stopRead() needs to stop and join (or otherwise synchronize with) the NMEA read thread before tearing down the InstrumentPanel/NmeaSourceTcp object graph it feeds, rather than destroying it concurrently.
Backtrace excerpt (full 17-thread capture attached as crash_backtrace.txt)
Code:
Thread 2 "QThread" received signal SIGSEGV, Segmentation fault.
#0 NmeaItemXDR::decode(QByteArray const&;) <-- crash here
#1 NmeaSourceTcp::slot_readData()
#2-4 (Qt Core/Network internals)
#5-10 (Qt event loop: QEventDispatcherUNIX / QThread::exec)
#11-13 (pthread thread start)
Thread 1 "qtVlm" (main GUI thread, mid-teardown):
#0-10 (blocked in X server round-trip: xcb_request_check / QtXcbQpa internals)
#11 QBackingStore::~QBackingStore()
#12-19 (QWidget / QGraphicsItem / QGraphicsProxyWidget / QLCDNumber destructor chain)
#20 GraphicsInstrumentCOG::~GraphicsInstrumentCOG()
#21-22 InstrumentPanel::~InstrumentPanel()
#23 myCentralWidget::slot_loadInstruments(bool const&, bool const&, bool const&;)
#24 boatReal::stopRead(bool const&;)
#25 boatReal::restartGPS()
#26 DialogParamVlm::done(int)
#27-48 (Qt event dispatch / button-click chain — the user clicking OK on the dialog)
[hr]
Bug 2: Memory leak in general NMEA output ("Send NMEA" — sendNmea), ~2.5MB/s
Reproduction / isolation
Bisected via ~/.qtVlm/qtVlm.ini config flags (edited while qtVlm was closed, relaunched, then sampled RSS via ps every few seconds for 20-30s per configuration):
Code:
Incoming NMEA | General outgoing NMEA | Waypoint outgoing NMEA | Result
(nmeaTcp1) | (sendNmea) | (sendWPtoNMEA+sendWPTcp+ |
| | sendWpTransTcp) |
----------------+------------------------+----------------------------------+---------------------------------
ON | OFF | OFF | Stable, flat RSS
ON | ON | OFF | Leaks ~2.5MB/s (~150MB/min), CPU pegs 100%+
ON | OFF | ON | Stable, flat RSS
This isolates the leak specifically to the general NMEA output path (sendNmea=1 in [nmea]), independent of and separate from the waypoint-specific output path (sendWPtoNMEA in [Boat_real], which sends bearing/distance/XTE/VMG for the active waypoint to a separate NMEA0183 TCP target). The waypoint-output path was confirmed leak-free on its own.
Severity
Left unattended, this leak previously crashed/froze the entire host machine, not just qtVlm — system-wide available memory collapsed to <100MB, the machine became fully unresponsive (input lag climbing past 1 second, then total lockup), and required a hard power cycle to recover. This is more severe than a simple app crash; it's a host-level DoS from a single client-side application setting.
Sample of raw growth data (RSS via ps, general NMEA output enabled):
Code:
16:44:17 RSS=454584KB
16:44:21 RSS=461552KB
16:44:25 RSS=473456KB
16:44:29 RSS=484080KB
16:44:33 RSS=496240KB
16:44:37 RSS=506992KB
16:44:42 RSS=518640KB
(~64MB growth in 25 seconds, consistent and unbounded — observed to continue for 30+ minutes without leveling off in an earlier full-length test.)
Suggested fix
Whatever per-sentence or per-cycle object(s) qtVlm's general NMEA output path allocates when sendNmea=1 is enabled are apparently never freed. Recommend checking for missing delete/deleteLater() or a growing container (e.g. a send queue or history buffer) that's never pruned in that code path. We don't have source access so cannot point at an exact file/line — the shipped Qt libraries are stripped of symbols, so the leak's own allocation site didn't show up in our live snapshots (the process was normally idle/blocked in poll/futex_wait at each sampling instant, as expected for an incremental per-cycle leak rather than a hot loop).
Workaround in use
We are running with sendNmea=0 permanently — this loses whatever general NMEA output does (destination/purpose not identified from the UI), but our specific use case (feeding a Garmin NMEA2000 chartplotter/instrument bus with active-waypoint bearing/distance/XTE via Signal K) only needs sendWPtoNMEA+sendWPTcp+sendWpTransTcp, which is unaffected.
[hr]
Environment / diagnostic notes
apport is configured as this system's core_pattern handler, but its default policy silently discards crash reports for binaries not installed via dpkg (like this manually-extracted qtVlm tarball) — so no .crash or core file was ever saved automatically for either bug. Both backtraces in this report were captured by manually attaching gdb (sudo gdb -p <pid>, needed due to this system's ptrace_scope=1 restricting attach to non-child processes) either at the moment of the live segfault (Bug 1) or via a scripted watchdog that captures a backtrace + generate-core-file before killing the process once available memory drops below a safety threshold (Bug 2, used only as a safety net — this bug was reproduced deliberately for diagnosis, not caught accidentally).
Full core dump files (~4GB each) from the live gdb sessions are not included here due to size, but are available on request.
Attachments: crash_backtrace.txt (full gdb thread apply all bt full output, all 17 threads, for Bug 1)
Please Log in or Create an account to join the conversation.
- maitai
- Offline
- Administrator
-
Less
More
- Posts: 1273
- Thank you received: 25
5 hours 1 minute ago #4010
by maitai
Replied by maitai on topic NMEA Data Output Bugs: SegFault & Runaway Memory Leak
Both bugs fixed, and both related to SignalK:
Bug1: SignalK sends XTR continuously for heel so some message arrives while NMEA is restarting, resulting in a dangling pointer
Bug2: Not a memory leek, but a network feedback loop. On the same TCP socket you resend everything, and SignalK does the same, resulting in a dead loop. The relay ("transmit everything") now knows where each sentence came from and refuses to write it back to that same connection. Of course if another socket point to the same server the loop will still occur, so be careful with "resend everything".
Fixes are already in beta15a, if you want to test.
Bug1: SignalK sends XTR continuously for heel so some message arrives while NMEA is restarting, resulting in a dangling pointer
Bug2: Not a memory leek, but a network feedback loop. On the same TCP socket you resend everything, and SignalK does the same, resulting in a dead loop. The relay ("transmit everything") now knows where each sentence came from and refuses to write it back to that same connection. Of course if another socket point to the same server the loop will still occur, so be careful with "resend everything".
Fixes are already in beta15a, if you want to test.
Please Log in or Create an account to join the conversation.
Time to create page: 0.120 seconds

