A feature set for debugging NetBSD applications (without threads) has been merged with upstream LLDB!
The number of passing tests this month has been increased from 267/1235 to 622/1247.
This is +133% within one month and approximately 50% of successfully passed tests in total!
As usual regular housekeeping of ptrace(2) interfaces has been done on the NetBSD side.

During this month I've finished the needed Native Process Plugin with breakpoints fully supported.
In order to achieve this I had to address bugs, add missing features and diligently debug the debugger sniffing on the GDB Remote Protocol line.
Since NetBSD-8 is approaching, I have performed the also needed housekeeping on the base system distribution side.

What has been done in NetBSD

I've managed to achieve the following goals:

Clean up in ptrace(2) ATF tests

We have created some maintanance burden for the current ptrace(2) regression tests.
The main issues with them is code duplication and the splitting between generic (Machine Independent) and port-specific (Machine Dependent) test files.
I've eliminated some of the ballast and merged tests into the appropriate directory tests/lib/libc/sys/.
The old location (tests/kernel) was a violation of the tests/README recommendation:

When adding new tests, please try to follow the following conventions.
1. For library routines, including system calls, the directory structure of
the tests should follow the directory structure of the real source tree.
For instance, interfaces available via the C library should follow:
src/lib/libc/gen -> src/tests/lib/libc/gen
src/lib/libc/sys -> src/tests/lib/libc/sys
...

The ptrace(2) interface inhabits src/lib/libc/sys so there is no reason not to move it to its proper home.

PTRACE_FORK on !x86 ports

Along with the motivation from Martin Husemann we have investigated the issue with PTRACE_FORK ATF regression tests.
It was discovered that these tests aren't functional on evbarm, alpha, shark, sparc and sparc64 and likely on other
non-x86 ports.
We have discovered that there is a missing SIGTRAP emitted from the child, during the fork(2) handshake.
The proper order of operations is as follows:

parent emits SIGTRAP with si_code=TRAP_CHLD and pe_set_event=pid of forkee

child emits SIGTRAP with si_code=TRAP_CHLD and pe_set_event=pid of forker

Only the x86 ports were emitting the second SIGTRAP signal.

The culprit reason has been investigated and narrowed down to the child_return() function in src/sys/arch/x86/x86/syscall.c:

This child_return() function was the only one among all the existing ones for other platforms to contain the needed code for SIGTRAP.
The appropriate solution was installed by Martin, as we taught our featured signal routing subsystem to handle early signals from the fork(2) calls.

PT_SYSCALL and PT_SYSCALLEMU

Christos Zoulas addressed the misbehavior with tracing syscall entry and syscall exit code.
We can again get an event inside a debugger that a debuggee attempts to trigger a syscall and later return from it.
This means that a debugger can access the register layout before executing the appropriate kernel code and
read it again after executing the syscall.
This allows to monitor exact sysentry arguments and return the values afterwards.
Another option is to fake the trap frame with new values, it's sometimes useful for debugging.

With the addition of PT_SYSCALLEMU we can implement a virtual kernel syscall monitor. It means that we can fake syscalls within a debugger.
In order to achieve this feature, we need to use the PT_SYSCALL operation,
catch SIGTRAP with si_code=TRAP_SCE (syscall entry),
call PT_SYSCALLEMU and perform an emulated userspace syscall that would have been done by the kernel,
followed by calling another PT_SYSCALL with si_code=TRAP_SCX.

This interface makes it possible to introduce the following into NetBSD: truss(1) from FreeBSD and strace(1) from Linux.
There used to be a port of at least strace(1) in the past, but it's time to refresh this code.
Another immediate consumer is of course in DTrace/libproc...
as there are facilities within this library to trace the system call entry and exit in order to catch fork(2) events.
Why to catch fork(2)? It can be useful to detach software breakpoints in order to detach them before cloning address space
of forker for forkee; and after the operation reapply them again.

What has been done in LLDB

A lot of work has been done with the goal to get breakpoints functional.
This target penetrated bugs in the existing local patches and unveiled missing features required to be added.
My initial test was tracing a dummy hello-world application in C.
I have sniffed the GDB Remote Protocol packets and compared them between Linux and NetBSD.
This helped to streamline both versions and bring the NetBSD support to the required Linux level.

As a bonus the initial code for OpenBSD support was also added into the LLDB tree.
At the moment OpenBSD only supports opening core(5) files with a single-thread.
The same capability was also added for NetBSD.

By the end of March all local patches for LLDB were merged upstream!
This resulted in NetBSD being among the first operating systems to use a Native Process Plugin framework with the debugserver capability, alongside Linux & Android.
The current FreeBSD support in LLDB is dated and lagging behind and limited to local debugging.
A majority of the work to bring FreeBSD on par could well be a case of s/NetBSD/FreeBSD/.

Among the features in the upstreamed NetBSD Process Plugin:

handling software breakpoints,

correctly attaching to a tracee,

supporting NetBSD specific ptrace(2),

monitoring process termination,

monitoring SIGTRAP events,

monitoring SIGSTOP events,

monitoring other signals events,

resuming the whole process,

getting memory region info perms,

reading memory from tracee,

writing memory to tracee,

reading ELF AUXV,

x86_64 GPR reading and writing,

detecting debuginfo of the basesystem programs located in /usr/libdata/debug

adding single step support,

adding execve(2) trap support,

placeholder for Floating Point Registers code,

initial code for the NetBSD specific core(5) files,

enabling ELF Aux Vector reading on the lldb client side,

enabling QPassSignals feature for NetBSD on the lldb client side,

enabling ProcessPOSIXLog on NetBSD,

minor tweaking.

Demo

It's getting rather difficult to present all the features of the NetBSD Process Plugin without making the example overly long.
This is why I will restrict it to a very basic debugging session hosted on NetBSD.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: