Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Live Recorder: Debugging C++, Rust, and Go with Capture and Replay of Nondeterministic Data

Live Recorder: Debugging C++, Rust, and Go with Capture and Replay of Nondeterministic Data

This item in japanese

Earlier in the year the Undo team released Live Recorder 5, the latest version of their “software flight recorder technology” for C/C++, Rust, and Go applications that enables the capture of all nondeterministic data within an application’s execution for debugging purposes. The captured recording allows the recreation of an application’s entire memory and the register state, which supports the replay of execution steps, backwards and forwards, via the UndoDB debugger. At the recent QCon San Francisco conference, InfoQ sat down with Dr Greg Law, co-founder and CTO at Undo, in order to learn more.

As stated on the Undo website, debugging intermittent software failures, whether in development, test, or in production, can be a frustrating experience for both end users and engineers. Modern debuggers for languages like C++ and Go are typically feature rich, but they do not capture all relevant nondeterministic external events that may be required for the recreation of a bug, and they also only allow the debugging of processes executing in a linear fashion.

Mocks and stubs can, of course, be used to simulate external events and add determinism, and an application can also be executed multiple times during a debugging session. However, these approaches add overhead and potential complexity, and are time intensive.

Undo’s Live Recorder captures all nondeterministic data, down to the instruction level, and this can be used to recreate an application’s complete state. This recording process typically results in a 2x to 5x slowdown of the target application. Live Recorder is only used to record failed processes, and is not intended to be run constantly like a profiler or other observability tooling.

Data capture is optimized so it can be stored in memory, and the entire program’s memory and register state can be recreated on demand with minimal overhead. The recorded data can be persisted to a file and shared amongst engineers. Live Recorder comes with an integrated reversible debugger so the recording can be debugged by replaying it backwards and forwards.


Undo's Live Recorder
Image taken from Live Recorder Datasheet (PDF)

In discussion with InfoQ, Law was clear to communicate that using Live Recorder should not be seen as an alternative to emitting metrics or logging. Live Recorder does not aim to alert engineers to the presence of faults or failures within an application, and this is instead best accomplished via the use of metric collection and analysis, exception handling code, and alerting tooling. Logging can also be used by engineers to provide more context as to what is, or should be, occurring within an application at an arbitrary point in execution.

Logging is valuable, in that it provides additional context to the business problem or underlying algorithm. Logging can 'state what you know'. An exception or stack trace can also give good clues as to what’s gone wrong. But often as an engineer, you simply need to replay the scenario that caused an issue, and step-by-step build a mental model of the issue and the underlying causes.

Law stated that Live Recorder aims to facilitate the detection and fixing of bugs an engineer hasn’t previously thought about, and therefore is aiming at aiding the resolution of the “unknown unknowns”. He continued by explaining that difficulty of understanding the causes of a bug can be modelled with a two-dimensional graph, where the x axis represents how often a bug manifests itself, and the y axis how repeatable the bug is. Live Recorder excels in the bottom right of this graph, where the bug is often being seen, but it cannot easily be repeated with traditional debugging techniques.

Once a recording file has been made that captures a defect, engineers can view the recording to analyze execution history using UndoDB, Live Recorder’s "reversible debugger". UndoDB is based on GDB, the GNU Project Debugger, and supports source-level debugging for applications written in any language supported by GDB, including C/C++, Rust and Ada. UndoDB supports all the expected debugging features, with variable and memory inspection, scripting, conditional breakpoints and watchpoints, all in forwards and reverse execution of the application. Additional features include:

  • Multi-process correlation: reveals the order in which processes and threads alter data structures in shared memory.
  • Thread fuzzing: randomizes thread execution to reveal race conditions and other multi-threading defects.
  • Integration: APIs, CLI, front-end and container platform integrations.
  • Compiler- and library-agnostic: Live Recorder does not require that application to be recompiled or linked against particular libraries.

Law stated that Live Recorder 5.0 can also record and replay the execution of individual Kubernetes and Docker containers to help resolve defects faster in microservices environments.

Live Recorder currently supports C/C++, Go, Rust, Ada applications on Linux x86 and x86_64, with Java support coming soon. It is compatible with all mainstream Linux distributions, and no kernel are changes required.

Rate this Article