NetBSD Bloghttps://blog.netbsd.org/tnf/feed/entries/atom2017-12-01T10:14:48+00:00Apache Roller (incubating)https://blog.netbsd.org/tnf/entry/the_llvm_thread_sanitizer_hasThe LLVM Thread Sanitizer has been ported to NetBSD Kamil Rytarowski2017-11-30T19:40:41+00:002017-12-01T10:14:48+00:00During the past month I've finished my work on TSan for NetBSD/amd64.
There are still few minor issues, although the Sanitizer is already suitable for real applications and is stable.
I was able to build real applications like LLDB against TSan and get it to work to find real threading problems.
<p>
The process of stabilization and fixing TSan was challenging as there are intermixed types of issues that resulted in one big random breakage bug that is difficult to analyze.
Software debuggers need more work with threaded programs, so this was like a chicken-egg problem, to debug debugging utilities.During the past month I've finished my work on TSan for NetBSD/amd64.
There are still few minor issues, although the Sanitizer is already suitable for real applications and is stable.
I was able to build real applications like LLDB against TSan and get it to work to find real threading problems.
<p>
The process of stabilization and fixing TSan was challenging as there are intermixed types of issues that resulted in one big random breakage bug that is difficult to analyze.
Software debuggers need more work with threaded programs, so this was like a chicken-egg problem, to debug debugging utilities.
<p>
<h1>Corrections</h1>
<p>
Most of the corrections were in TSan-specific and Common Sanitizer code. There was also one fix in LSan.
<p>
<h2>TSan: on_exit()/at_exit(3)/__cxa_atexit()</h2>
<p>
There are different function types for the same purpose: to execute a callback function on thread or process termination.
The existing code in TSan wasn't compatible with the NetBSD Operating System:
<ul>
<li>on_exit() - This function is Linux-specific, I've disabled it for NetBSD.</li>
<li>at_exit(3) - It was reimplemented by TSan using __cxa_atexit(), however in an incompatible way for NetBSD.
TSan was attempting to register a wrapper callback through __cxa_atexit() with the second argument as a function pointer and the third argument (Dynamic Shared Object pointer) equal with NULL.
This approach is not portable and it broke on NetBSD, therefore I had to add a new implementation based on a stack (LIFO container).</li>
Every at_exit(3) registering function is intercepted by TSan and the sanitizer pushes it to the local LIFO container, passing its local wrapper function to the system.
During the execution of a callback by the OS, we call the wrapper, which pops the originally saved function pointer from the stack and executes it.
<li>__cxa_atexit() - This callback shared TSan internals with at_exit(3) and is functional on NetBSD.</li>
</ul>
<p>
To assure the changes, I've added a new test named atexit3, which assures the correct order of execution of the at_exit(3) callbacks.
<p>
<h2>TSan: _lwp_exit()</h2>
<p>
In order to detect a thread's termination by the TSan interceptors, a mechanism to register a callback function in the pthread(3) destructor was used.
The destructor callback was registered with pthread_key_create(3) and this approach was broken on NetBSD for two reasons.
<p>
<ol>
<li>We cannot register it during early libc and libpthread(3) bootstrap, as the system functions need to initialize.</li>
<li>The execution of callback functions is not the last event during a POSIX thread entity termination.</li>
</ol>
<p>
I was looking for a mechanism to defer the destructor callback registration to subsequent libc initialization stages, similar to constructor sections.
I've understood that this approach was suboptimal because it resulted in further breakage.
The NetBSD implementation of a POSIX thread termination notifies a parent thread (waiter for join) and still attempts to acquire mutex.
TSan assumed that no longer any thread specific function is called like a mutex acquisition and destroyed part of thread specific data to trace such events.
I've switched the POSIX thread termination event detection to the interception of _lwp_exit(2) call,
as it's truly the latest interceptable function on NetBSD, detaching the low-level
thread entity (LWP) that is the kernel context for POSIX thread.
<p>
<h2>TSan: Thread Joined vs Thread Exited</h2>
<p>
Correcting the detection of termination of a thread caused new problems,
with a race between two event notifications that happen at the same time:
<p>
<ul>
<li>Thread A sleeps waiting for joining of thread B.</li>
<li>Thread B wakes thread A notifying it as joinable.</li>
<li>Thread B terminates calling _lwp_exit().</li>
</ul>
<p>
Both events are traced by TSan: joining and exiting and they must be intercepted in the order of exiting followed by joining
(unless a thread is marked to be detached without joining).
<p>
This problem has been analyzed and fixed by the introduction of atomic-function waiters in low-level parts (not exposed to TSan or other sanitizers), that causes busy waiting
in ThreadRegistry::JoinThread for notifying the end of execution of ThreadRegistry::FinishThread.
This approach happened to be stable and so far no failures are observed.
There was a tiny breakage in ppc64-linux, as this change introduced as infinite freeze, but it was
caused by an unrelated problem and a faulty test was switched from failing to unsupported.
<p>
<h2>Sanitizers: GetTls</h2>
<p>
I've implemented the initial support for determining whether a memory buffer is allocated as Thread-Local-Storage.
The current approach uses FreeBSD code, however it's subject to future improvement: in order to make it more generic and aware of dynamic allocation
(like after dlopen(3)) TLS vectors.
<p>
<h2>Sanitizers: Handling NetBSD specific indirection of libpthread functions</h2>
<p>
I've corrected handling of three libpthread(3) functions on NetBSD:
<p>
<ul>
<li>pthread_mutex_lock(3),</li>
<li>pthread_mutex_unlock(3),</li>
<li>pthread_setcancelstate(3).</li>
</ul>
<p>
Code out of the libpthread(3) context uses the libc symbols:
<p>
<ul>
<li>__libc_mutex_lock,</li>
<li>__libc_mutex_unlock,</li>
<li>__libc_thr_setcancelstate.</li>
</ul>
<p>
The threading library (libpthread(3)) defines strong aliases:
<p>
<ul>
<li>__strong_alias(__libc_mutex_lock,pthread_mutex_lock)</li>
<li>__strong_alias(__libc_mutex_unlock,pthread_mutex_unlock)</li>
<li>__strong_alias(__libc_thr_setcancelstate,pthread_setcancelstate)</li>
</ul>
<p>
This caused that these functions were invisible to sanitizers on NetBSD.
I've introduced interception of the libc-specific functions and I have added them as NetBSD-specific aliases
for the common pthread(3) functions.
<p>
NetBSD needs to intercept both functions, as the regularly named ones are used internally in libpthread(3).
<p>
<h2>Sanitizers: Adding DemangleFunctionName for backtracing on NetBSD</h2>
<p>
NetBSD uses indirection for old threading functions for historical reasons.
The mangled names are an internal implementation detail and should not be
exposed even in backtraces.
<p>
<ul>
<li>__libc_mutex_init -> pthread_mutex_init</li>
<li>__libc_mutex_lock -> pthread_mutex_lock</li>
<li>__libc_mutex_trylock -> pthread_mutex_trylock</li>
<li>__libc_mutex_unlock -> pthread_mutex_unlock</li>
<li>__libc_mutex_destroy -> pthread_mutex_destroy</li>
<li>__libc_mutexattr_init -> pthread_mutexattr_init</li>
<li>__libc_mutexattr_settype -> pthread_mutexattr_settype</li>
<li>__libc_mutexattr_destroy -> pthread_mutexattr_destroy</li>
<li>__libc_cond_init -> pthread_cond_init</li>
<li>__libc_cond_signal -> pthread_cond_signal</li>
<li>__libc_cond_broadcast -> pthread_cond_broadcast</li>
<li>__libc_cond_wait -> pthread_cond_wait</li>
<li>__libc_cond_timedwait -> pthread_cond_timedwait</li>
<li>__libc_cond_destroy -> pthread_cond_destroy</li>
<li>__libc_rwlock_init -> pthread_rwlock_init</li>
<li>__libc_rwlock_rdlock -> pthread_rwlock_rdlock</li>
<li>__libc_rwlock_wrlock -> pthread_rwlock_wrlock</li>
<li>__libc_rwlock_tryrdlock -> pthread_rwlock_tryrdlock</li>
<li>__libc_rwlock_trywrlock -> pthread_rwlock_trywrlock</li>
<li>__libc_rwlock_unlock -> pthread_rwlock_unlock</li>
<li>__libc_rwlock_destroy -> pthread_rwlock_destroy</li>
<li>__libc_thr_keycreate -> pthread_key_create</li>
<li>__libc_thr_setspecific -> pthread_setspecific</li>
<li>__libc_thr_getspecific -> pthread_getspecific</li>
<li>__libc_thr_keydelete -> pthread_key_delete</li>
<li>__libc_thr_once -> pthread_once</li>
<li>__libc_thr_self -> pthread_self</li>
<li>__libc_thr_exit -> pthread_exit</li>
<li>__libc_thr_setcancelstate -> pthread_setcancelstate</li>
<li>__libc_thr_equal -> pthread_equal</li>
<li>__libc_thr_curcpu -> pthread_curcpu_np</li>
</ul>
<p>
This demangling also fixes several tests that expect the regular pthread(3) function names.
<p>
<h2>TSan: Handling NetBSD specific indirection of libpthread functions</h2>
<p>
I've corrected handling of libpthread(3) functions in TSan/NetBSD:
<p>
<ul>
<li>pthread_cond_init(3),</li>
<li>pthread_cond_signal(3),</li>
<li>pthread_cond_broadcast(3),</li>
<li>pthread_cond_wait(3),</li>
<li>pthread_cond_destroy(3),</li>
<li>pthread_mutex_init(3),</li>
<li>pthread_mutex_destroy(3),</li>
<li>pthread_mutex_trylock(3),</li>
<li>pthread_rwlock_init(3),</li>
<li>pthread_rwlock_destroy(3),</li>
<li>pthread_rwlock_rdlock(3),</li>
<li>pthread_rwlock_tryrdlock(3),</li>
<li>pthread_rwlock_wrlock(3),</li>
<li>pthread_rwlock_trywrlock(3),</li>
<li>pthread_rwlock_unlock(3),</li>
<li>pthread_once(3).</li>
</ul>
<p>
Code out of the libpthread(3) context uses the libc symbols
that are prefixed with __libc_, for example: __libc_cond_init.
<p>
This has caused that these functions were invisible to sanitizers on NetBSD.
Intercepting the libc-specific and adding them as NetBSD-specific aliases
for the common pthread(3) functions.
<p>
NetBSD needs to intercept both functions, as the regularly named ones
are used internally in libpthread(3).
<p>
<h2>TSan: Correcting NetBSD support in pthread_once(3)</h2>
<p>
The pthread_once(3)/NetBSD type is built with the following structure:
<p>
<code>
struct __pthread_once_st {
pthread_mutex_t pto_mutex;
int pto_done;
};
</code>
<p>
I've set the pto_done position as shifted by __sanitizer::pthread_mutex_t_sz
from the beginning of the pthread_once struct.
<p>
This corrects deadlocks when the pthread_once(3) function
is used.
<p>
<h2>Sanitizers: Plug dlerror() leak for swift_demangle</h2>
<p>
InitializeSwiftDemangler() attempts to resolve the
swift_demangle symbol. If this is not available, we
observe dlerror message leak.
<p>
<h2>LSan: Detecting thread's termination</h2>
<p>
I've fixed the same problem as has been analyzed in TSan, and I've switched to the _lwp_exit(2) approach.
<p>
<h2>Sanitizers: Handling symbol renaming of sigaction on NetBSD</h2>
<p>
NetBSD uses the __sigaction14 symbol name for historical and compat
reasons for the sigaction(2) function name.
<p>
I've renamed the interceptors and users of sigaction to sigaction_symname
and I've reused it in the code base.
<p>
<h2>TSan: Correcting mangled_sp on NetBSD/amd64</h2>
<p>
I've fixed the LongJmp(3) function on NetBSD and pointed the correct place of the RSP (stack pointer) register on NetBSD/amd64.
<p>
<h2>TSan: Supporting the setjmp(3) family of functions on NetBSD/amd64</h2>
<p>
I've added support for handling the setjmp(3)/longjmp(3) family of functions on NetBSD/amd64.
<p>
There are three types of them on NetBSD:
<p>
<ul>
<li>setjmp(3) / longjmp(3)</li>
<li>sigsetjmp(3) / sigsetjmp(3)</li>
<li>_setjmp(3) / _longjmp(3)</li>
</ul>
<p>
Due to historical and compat reasons the symbol
names are mangled:
<p>
<ul>
<li>setjmp -> __setjmp14</li>
<li>longjmp -> __longjmp14</li>
<li>sigsetjmp -> __sigsetjmp14</li>
<li>siglongjmp -> __siglongjmp14</li>
<li>_setjmp -> _setjmp</li>
<li>_longjmp -> _longjmp</li>
</ul>
<p>
This leads to symbol renaming in the existing codebase.
<p>
There is no such symbol as __sigsetjmp/__longsetjmp
on NetBSD so it has been disabled.
<p>
Additonally, I've added a comment that GNU-style executable stack
note is not needed on NetBSD. The stack is not
executable without it.
<p>
<h2>TSan: Deferring StartBackgroundThread() and StopBackgroundThread()</h2>
<p>
NetBSD cannot spawn new POSIX thread entities in early
libc and libpthread initialization stage. I've deferred this to the point
of intercepting the first pthread_create(3) call.
<p>
This is the last change that makes Thread Sanitizer functional
on NetBSD/amd64 without downstream patches.
<p>
<h1>Final TSan results</h1>
<p>
Results for the check-tsan test-target.
<p>
<pre>
********************
Testing Time: 64.91s
********************
Failing Tests (5):
ThreadSanitizer-x86_64 :: dtls.c
ThreadSanitizer-x86_64 :: ignore_lib5.cc
ThreadSanitizer-x86_64 :: ignored-interceptors-mmap.cc
ThreadSanitizer-x86_64 :: mutex_lock_destroyed.cc
ThreadSanitizer-x86_64 :: vfork.cc
Expected Passes : 290
Expected Failures : 1
Unsupported Tests : 83
Unexpected Failures: 5
</pre>
<p>
The following results present that the all crucial issues are now fixed, and this Sanitizer can be used to trace real software.
The remaining problems are minor ones and they are scheduled to be fixed in the future:
<p>
<ul>
<li>signal_block.cc - there is some race; sometimes it works sometimes it
does not work.</li>
<li>dtls.c - it looks like dynamically allocated TLS vectors are missing on the
NetBSD side.</li>
<li>vfork.cc - testing UB, it looks like NetBSD behaves the same way like
Linux does, however the test is failing.</li>
<li>mutex_lock_destroyed.cc - it is based on UB implemented in style of Linux.</li>
<li>The other tests fail for similar rare case scenarios like massive
mmap(2) calls that seem to overflow the shadow.</li>
</ul>
<p>
<h1>LLVM JIT</h1>
<p>
As noted in the previous reports, there is an ongoing process to improve NetBSD compatiblity with existing Just-In-Time frameworks in LLVM.
In the recent month the existing code has been adjusted to the point to pass all existing LLVM tests of JIT code on NetBSD under PaX MPROTECT.
<p>
<h1>Scudo hardened allocator</h1>
<p>
I've added initial support for NetBSD in the Scudo hardened allocator.
I keep this code locally in pkgsrc-wip/compiler-rt-netbsd.
<p>
More work is needed in order to correct the known failures in tests.
These are largely caused by the fact that Scudo was a Linux-only feature and the existing tests depend on GLIBC specific internals.
They need to be adapted for the default NetBSD allocator (jemalloc(3)).
<p>
<pre>
********************
Testing Time: 5.40s
********************
Failing Tests (32):
Scudo-i386 :: double-free.cpp
Scudo-i386 :: interface.cpp
Scudo-i386 :: memalign.c
Scudo-i386 :: mismatch.cpp
Scudo-i386 :: options.cpp
Scudo-i386 :: overflow.c
Scudo-i386 :: preload.cpp
Scudo-i386 :: quarantine.c
Scudo-i386 :: realloc.cpp
Scudo-i386 :: rss.c
Scudo-i386 :: secondary.c
Scudo-i386 :: sizes.cpp
Scudo-i386 :: valloc.c
Scudo-x86_64 :: alignment.c
Scudo-x86_64 :: double-free.cpp
Scudo-x86_64 :: interface.cpp
Scudo-x86_64 :: malloc.cpp
Scudo-x86_64 :: memalign.c
Scudo-x86_64 :: mismatch.cpp
Scudo-x86_64 :: options.cpp
Scudo-x86_64 :: overflow.c
Scudo-x86_64 :: preload.cpp
Scudo-x86_64 :: quarantine.c
Scudo-x86_64 :: random_shuffle.cpp
Scudo-x86_64 :: realloc.cpp
Scudo-x86_64 :: rss.c
Scudo-x86_64 :: secondary.c
Scudo-x86_64 :: sized-delete.cpp
Scudo-x86_64 :: sizes.cpp
Scudo-x86_64 :: threads.c
Scudo-x86_64 :: valloc.c
Expected Passes : 8
Unexpected Failures: 32
</pre>
<p>
<h1>Plans for the next milestone</h1>
<p>
The next goal is to finish MSan and switch back to LLDB restoration for tracing single threaded programs.
<p>
The TSan corrections indirectly increased the number of passing MSan tests.
I'm going to solve the detected problems and thanks to the experience with other sanitizers the MSan issues don't seem to be as challenging like as before finishing TSan.
<p>
<pre>
********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 30.91s
********************
Failing Tests (69):
MemorySanitizer-x86_64 :: allocator_returns_null.cc
MemorySanitizer-x86_64 :: backtrace.cc
MemorySanitizer-x86_64 :: c-strdup.c
MemorySanitizer-x86_64 :: chained_origin.cc
MemorySanitizer-x86_64 :: chained_origin_empty_stack.cc
MemorySanitizer-x86_64 :: chained_origin_limits.cc
MemorySanitizer-x86_64 :: chained_origin_memcpy.cc
MemorySanitizer-x86_64 :: chained_origin_with_signals.cc
MemorySanitizer-x86_64 :: check_mem_is_initialized.cc
MemorySanitizer-x86_64 :: death-callback.cc
MemorySanitizer-x86_64 :: dlopen_executable.cc
MemorySanitizer-x86_64 :: dso-origin.cc
MemorySanitizer-x86_64 :: dtls_test.c
MemorySanitizer-x86_64 :: dtor-base-access.cc
MemorySanitizer-x86_64 :: dtor-bit-fields.cc
MemorySanitizer-x86_64 :: dtor-derived-class.cc
MemorySanitizer-x86_64 :: dtor-multiple-inheritance-nontrivial-class-members.cc
MemorySanitizer-x86_64 :: dtor-multiple-inheritance.cc
MemorySanitizer-x86_64 :: dtor-trivial-class-members.cc
MemorySanitizer-x86_64 :: dtor-vtable-multiple-inheritance.cc
MemorySanitizer-x86_64 :: dtor-vtable.cc
MemorySanitizer-x86_64 :: fork.cc
MemorySanitizer-x86_64 :: ftime.cc
MemorySanitizer-x86_64 :: getaddrinfo-positive.cc
MemorySanitizer-x86_64 :: getaddrinfo.cc
MemorySanitizer-x86_64 :: getc_unlocked.c
MemorySanitizer-x86_64 :: heap-origin.cc
MemorySanitizer-x86_64 :: icmp_slt_allones.cc
MemorySanitizer-x86_64 :: iconv.cc
MemorySanitizer-x86_64 :: ifaddrs.cc
MemorySanitizer-x86_64 :: insertvalue_origin.cc
MemorySanitizer-x86_64 :: mktime.cc
MemorySanitizer-x86_64 :: mmap.cc
MemorySanitizer-x86_64 :: msan_copy_shadow.cc
MemorySanitizer-x86_64 :: msan_dump_shadow.cc
MemorySanitizer-x86_64 :: msan_print_shadow.cc
MemorySanitizer-x86_64 :: msan_print_shadow2.cc
MemorySanitizer-x86_64 :: origin-store-long.cc
MemorySanitizer-x86_64 :: param_tls_limit.cc
MemorySanitizer-x86_64 :: print_stats.cc
MemorySanitizer-x86_64 :: pthread_getattr_np_deadlock.cc
MemorySanitizer-x86_64 :: pvalloc.cc
MemorySanitizer-x86_64 :: readdir64.cc
MemorySanitizer-x86_64 :: realloc-large-origin.cc
MemorySanitizer-x86_64 :: realloc-origin.cc
MemorySanitizer-x86_64 :: report-demangling.cc
MemorySanitizer-x86_64 :: scandir.cc
MemorySanitizer-x86_64 :: scandir_null.cc
MemorySanitizer-x86_64 :: select_float_origin.cc
MemorySanitizer-x86_64 :: select_origin.cc
MemorySanitizer-x86_64 :: sem_getvalue.cc
MemorySanitizer-x86_64 :: signal_stress_test.cc
MemorySanitizer-x86_64 :: sigwait.cc
MemorySanitizer-x86_64 :: stack-origin.cc
MemorySanitizer-x86_64 :: stack-origin2.cc
MemorySanitizer-x86_64 :: strerror_r-non-gnu.c
MemorySanitizer-x86_64 :: strlen_of_shadow.cc
MemorySanitizer-x86_64 :: strndup.cc
MemorySanitizer-x86_64 :: textdomain.cc
MemorySanitizer-x86_64 :: times.cc
MemorySanitizer-x86_64 :: tls_reuse.cc
MemorySanitizer-x86_64 :: tsearch.cc
MemorySanitizer-x86_64 :: tzset.cc
MemorySanitizer-x86_64 :: unaligned_read_origin.cc
MemorySanitizer-x86_64 :: unpoison_string.cc
MemorySanitizer-x86_64 :: use-after-dtor.cc
MemorySanitizer-x86_64 :: use-after-free.cc
MemorySanitizer-x86_64 :: wcsncpy.cc
Expected Passes : 38
Expected Failures : 1
Unsupported Tests : 24
Unexpected Failures: 69
</pre>
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/the_strongest_kaslr_everThe strongest KASLR, ever?Maxime Villard2017-11-20T12:50:15+00:002017-11-20T12:50:15+00:00<i>latest developments in the Kernel ASLR district</i><i>latest developments in the Kernel ASLR district</i>
<p>
<h1>Initial design</h1>
<p>
As I said in the
<a href="https://blog.netbsd.org/tnf/entry/kernel_aslr_on_amd64">previous episode</a>,
I added in October a Kernel ASLR implementation in NetBSD for 64bit x86 CPUs.
This implementation would randomize the location of the kernel in virtual memory
as one block: a random VA would be chosen, and the kernel ELF sections would be
mapped contiguously starting from there.
<p>
This design had several drawbacks: one leak, or one successful cache attack,
could be enough to reconstruct the layout of the entire kernel and defeat KASLR.
<p>
NetBSD’s new KASLR design significantly improves this situation.
<p>
<h1>New design</h1>
<p>
In the new design, each kernel ELF section is randomized independently. That is
to say, the base addresses of .text, .rodata, .data and .bss are not correlated.
KASLR is already at this stage more difficult to defeat, since you would need
a leak or cache attack on each of the kernel sections in order to reconstruct
the in-memory kernel layout.
<p>
Then, starting from there, several techniques are used to strengthen the
implementation even more.
<p>
<h2>Sub-blocks</h2>
<p>
The kernel ELF sections are themselves split in sub-blocks of approximately 1MB.
The kernel therefore goes from having:
<pre>
{ .text .rodata .data .bss }
</pre>
to having
<pre>
{ .text .text.0 .text.1 ... .text.i .rodata .rodata.0 ... .rodata.j ... .data ...etc }
</pre>
As of today, this produces a kernel with ~33 sections, <b>each of which</b> is
mapped at a random address and in a random order.
<p>
This implies that there can be dozens of .text segments. Therefore, even if
you are able to conduct a cache attack and determine that a given range of
memory is mapped as executable, you don’t know which sub-block of .text it is.
If you manage to obtain a kernel pointer via a leak, you can at most guess the
address of the section it finds itself in, but you don’t know the layout of the
remaining 32 sections. In other words, defeating this KASLR implementation is
much more complicated than in the initial design.
<p>
<h2>Higher entropy</h2>
<p>
Each section is put in a 2MB-sized physical memory chunk. Given that the
sections are 1MB in size, this leaves half of the 2MB chunk unused. Once in
control, the prekern shifts the section within the chunk using a random offset,
aligned to the ELF alignment constraint. This offset has a maximum value of 1MB,
so that once shifted the section still resides in its initial 2MB chunk:
<p>
<center>
<img src="//www.netbsd.org/~maxv/kaslr/FigA.png">
<br>
<font size="1">Fig. A: Physical memory, a random offset has been added.</font>
</center>
<p>
The prekern then maps these 2MB physical chunks at random virtual addresses; but
addresses aligned to 2MB. For example, the two sections in Fig. A will be mapped
at two distinct VAs:
<p>
<center>
<img src="//www.netbsd.org/~maxv/kaslr/FigB.png">
<br>
<font size="1">Fig. B: two random, 2MB-aligned ranges of VAs point to the chunks
the sections find themselves in.</font>
</center>
<p>
There is a reason the sections are shifted in memory: it offers higher entropy.
If we consider a .text.i section with a 64byte ELF alignment constraint, and
give a look at the number of possibilities for the location of the section in
memory:
<p>
<ul>
<li>The prekern shifts the 1MB section in its 2MB chunk, with an offset aligned
to 64 bytes. So there are (2MB-1MB)/(64B)=2<sup>14</sup> possibilities for the
offset.</li>
<li>Then, the prekern uses a 2MB-sized 2MB-aligned range of VA, chosen in a 2GB
window. So there are (2GB-2MB)/(2MB)=2<sup>10</sup>-1 possibilities for the VA.</li>
</ul>
<p>
Therefore, there are 2<sup>14</sup>x(2<sup>10</sup>-1)&asymp;2<sup>24</sup>
possible locations for the section. As a comparison with other systems:
<p>
<center>
<table>
<tr>
<th>OS</th>
<th># of possibilities</th>
</tr>
<tr>
<td>Linux</td>
<td><center>2<sup>6</sup></center></td>
</tr>
<tr>
<td>MacOS</td>
<td><center>2<sup>8</sup></center></td>
</tr>
<tr>
<td>Windows</td>
<td><center>2<sup>13</sup></center></td>
</tr>
<tr>
<td>NetBSD</td>
<td><center>2<sup>24</sup></center></td>
</tr>
</table>
<br>
<font size="1">Fig. C: comparison of entropies. Note that the other KASLR
implementations do not split the kernel sections in sub-blocks.</font>
</center>
<p>
Of course, we are talking about one .text.i section here; the sections that
will be mapped afterwards will have fewer location possibilities because some
slots will be already occupied. However, this does not alter the fact that the
resulting entropy is still higher than that of the other implementations.
Note also that several sections have an alignment constraint smaller than 64
bytes, and that in such cases the entropy is even higher.
<p>
<h2>Large pages</h2>
<p>
There is also a reason we chose to use 2MB-aligned 2MB-sized ranges of VAs:
when the kernel is in control and initializes itself, it can now use large pages
to map the physical 2MB chunks. This greatly improves memory access
performance at the CPU level.
<p>
<h2>Countermeasures against TLB cache attacks</h2>
<p>
With the memory shift explained above, randomness is therefore enforced at both
the physical and virtual levels: the address of the first page of a section
does not equal the address of the section itself anymore.
<p>
It has, as a side effect, an interesting property: it can mostly mitigate
TLB cache attacks. Such attacks operate at the virtual-page level; they will
allow you to know that a given large page is mapped as executable, but you don’t
know where exactly within that page the section actually begins.
<p>
<h1>Strong?</h1>
<p>
This KASLR implementation, which splits the kernel in dozens of sub-blocks,
randomizes them independently, while at the same time allowing for higher
entropy in a way that offers large page support and some countermeasures
against TLB cache attacks, appears to be the most advanced KASLR implementation
available publicly as of today.
<p>
Feel free to prove me wrong, I would be happy to know!
<p>
<h1>WIP</h1>
<p>
Even if it is in a functional state, this implementation is still a work in
progress, and some of the issues mentioned in the
<a href="https://blog.netbsd.org/tnf/entry/kernel_aslr_on_amd64">previous blog post</a>
haven't been addressed yet. But feel free to test it and report any issue you
encounter. Instructions on how to use this implementation can still be found in
the previous blog post, and haven’t changed since.
<p>
See you in the next episode!
<p>
https://blog.netbsd.org/tnf/entry/netbsd_on_allwinner_socs_updateNetBSD on Allwinner SoCs Updatejmcneill2017-11-08T02:46:15+00:002017-11-08T10:28:56+00:00<p>
<img src="//www.netbsd.org/~jmcneill/allwinner.jpg" style="width: 83px; height: 83px; float: right;">
Since the <a href="https://blog.netbsd.org/tnf/entry/porting_netbsd_to_allwinner_h3">last update</a>, we've made a number of improvements to the NetBSD Allwinner port. The <i>SUNXI</i> kernel has grown support for 8 new SoCs, and we added many new device drivers to the source repository.
</p><p>
<img src="//www.netbsd.org/~jmcneill/allwinner.jpg" style="width: 83px; height: 83px; float: right;">
Since the <a href="https://blog.netbsd.org/tnf/entry/porting_netbsd_to_allwinner_h3">last update</a>, we've made a number of improvements to the NetBSD Allwinner port. The <i>SUNXI</i> kernel has grown support for 8 new SoCs, and we added many new device drivers to the source repository.
</p>
<h4>Supported systems</h4>
<ul>
<li>
As of November 2017, we now support the following Allwinner SoCs with the SUNXI kernel: A10, A13, A20, A31, A64, A80, A83T, GR8, H2+, H3, H5, and R8. An up-to-date list of supported SoCs with example boards is maintained on the <a href="https://wiki.netbsd.org/ports/evbarm/allwinner/?updated#index1h1">NetBSD/allwinner wiki page</a>.
</li>
<li>
Here's <a href="https://pbs.twimg.com/media/DJYA7SCXUAEvEQk.jpg">NetBSD running on a 14" Pinebook</a> from <a href="https://www.pine64.org/?page_id=3707">Pine64</a>. Since the keyboard and touchpad are connected internally to a USB controller, the device is fully functional.
</li>
<li>
A bit more effort was required to get <a href="https://pbs.twimg.com/media/DIRGmsOXcAEJu-j.jpg">NetBSD running on a PocketCHIP</a> from <a href="https://getchip.com/pages/pocketchip">NextThing Co</a>. The keyboard is wired to an I2C keypad controller, so a new <i>tcakp(4)</i> keyboard driver was required. The touchscreen is supported by the Allwinner port's <i>sunxits</i> driver and can be calibrated with the <a href="http://netbsd.gw.com/cgi-bin/man-cgi?tpctl++NetBSD-current">tpctl(8)</a> utility.
</li>
<li>
Many more devices are supported, including (but not limited to) boards from <a href="http://nanopi.io">FriendlyARM (NanoPi)</a>, <a href="https://www.olimex.com/Products/OLinuXino/open-source-hardware">Olimex (OLinuXino)</a>, <a href="https://www.pine64.org">Pine64</a>, <a href="http://www.banana-pi.org">Sinovoip (BananaPi)</a>, and <a href="http://www.orangepi.org">Xunlong (OrangePi)</a>.
</li>
</ul>
<h4>Device driver support</h4>
<p>
In addition to the countless machine-independent device drivers already in NetBSD, the following Allwinner-specific devices are supported:
</p>
<h5>Audio codec</h5>
<p>
The built-in analog audio codec is supported on the following SoCs with the <i>sunxicodec</i> driver: A10, A13, A20, A31, GR8, H2+, H3, and R8.
</p>
<h5>Ethernet</h5>
<p>
Ethernet is supported on all applicable Allwinner SoCs. Three ethernet drivers are available:
<ul>
<li>Fast Ethernet MAC (EMAC) as found in A10 and A20 family SoCs</li>
<li>Gigabit Ethernet MAC (GMAC) as found in A20, A31, and A80 family SoCs</li>
<li>Gigabit Ethernet MAC (EMAC) as found in A64, A83T, H2+, and H3 family SoCs</li>
</ul>
</p>
<h5>Framebuffer</h5>
<p>
Framebuffer console support is available wherever it is supported by U-Boot using the <i>simplefb(4)</i> driver.
</p>
<h5>Thermal sensors</h5>
<p>
Thermal sensors are supported on A10, A13, A20, A31, A64, A83T, H2+, and H3 SoCs.
</p>
<h5>CPU frequency and voltage scaling</h5>
<p>
On A10, A20, H2+, and H3 SoCs, dynamic CPU frequency and voltage scaling support is available when configured in the device tree. In addition, on H2+ and H3 SoCs, the kernel will automatically detect when the CPU temperature is too high and throttle the CPU frequency and voltage to prevent overheating.
</p>
<h5>Touch screen</h5>
<p>
The touch screen controller found in A10, A13, A20, and A31 SoCs is fully supported. The <a href="http://netbsd.gw.com/cgi-bin/man-cgi?tpctl++NetBSD-current">tpctl(8)</a> utility can be used to calibrate the touch screen and has been updated to support standard wsdisplay APIs.
</p>
<h5>Other drivers</h5>
<p>
A standard set of devices are supported across all SoCs (where applicable): DMA, GPIO, I2C, interrupt controllers, RTC, SATA, SD/MMC, timers, UART, USB, watchdog, and more.
</p>
<h4>U-Boot</h4>
<p>
A framework for U-Boot packages has been added to pkgsrc, and <a href="http://pkgsrc.se/search.php?so=u-boot">U-Boot packages</a> for many boards already exist. </p>
<h4>What now?</h4>
<p>
There are a few missing features that would be nice to have:
<ul>
<li>Wi-Fi (SDIO). There are a lot of different wireless chips used on these boards, but the majority seem to be either Broadcom or Realtek based. We recently ported OpenBSD's <i>bwfm(4)</I> driver to support the USB version of the Broadcom Wi-Fi controllers, with an expectation that SDIO support will follow at some point in the future.</li>
<li>NAND controller. Most boards have eMMC and/or microSD slots, but this would be really useful for the CHIP / CHIP Pro / PocketCHIP family of devices.</li>
<li>64-bit support for sun50i family SoCs</li>
<li>Readily available install images. A prototype <a href="http://www.invisible.ca/arm/">NetBSD ARM Bootable Images</a> site is available with a limited selection of supported boards.</li>
</ul>
</p>
<h5>More information</h5>
<p>
<ul>
<li><a href="http://mail-index.netbsd.org/port-arm/">port-arm mailing list</a></li>
<li><a href="https://wiki.netbsd.org/ports/evbarm/allwinner/">NetBSD Allwinner wiki page</a></li>
</ul>
</p>https://blog.netbsd.org/tnf/entry/one_year_checkpoint_and_threadOne year checkpoint and Thread Sanitizer updateKamil Rytarowski2017-11-01T02:19:38+00:002017-11-01T02:19:38+00:00The past year has been started with bugfixes and the development of regression tests for ptrace(2) and related kernel features,
as well as the continuation of bringing LLDB support and LLVM sanitizers (ASan + UBsan and partial TSan + Msan) to NetBSD.
<br>
My plan for the next year is to finish implementing TSan and MSan support, followed by a long run of bug fixes for LLDB, ptrace(2), and other related kernel subsystemsThe past year has been started with bugfixes and the development of regression tests for ptrace(2) and related kernel features,
as well as the continuation of bringing LLDB support and LLVM sanitizers (ASan + UBsan and partial TSan + Msan) to NetBSD.
<br>
My plan for the next year is to finish implementing TSan and MSan support, followed by a long run of bug fixes for LLDB, ptrace(2), and other related kernel subsystems
<p>
<h1>TSan</h1>
<p>
In the past month, I've developed Thread Sanitizer far enough to have a subset of its tests pass on NetBSD,
started with addressing breakage related to the memory layout of processes.
The reason for this breakage was narrowed down to the current implementation of ASLR, which was too aggressive
and which didn't allow enough space to be mapped for Shadow memory.
The fix for this was to either force the disabling of ASLR per-process, or globally on the system.
The same will certainly happen for MSan executables.
After some other corrections, I got TSan to work for the first time ever on October 14th.
This was a big achievement, so I've made a <a href="https://netbsd.org/%7Ekamil/llvm/tsan.2017-10-14.txt">snapshot</a> available.
Getting the snapshot of execution under GDB was pure hazard.
<p>
<pre>
$ gdb ./a.out
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...done.
(gdb) r
Starting program: /public/llvm-build/a.out
[New LWP 2]
==================
WARNING: ThreadSanitizer: data race (pid=1621)
Write of size 4 at 0x000001475d70 by thread T1:
#0 Thread1 /public/llvm-build/tsan.c:4:10 (a.out+0x46bf71)
Previous write of size 4 at 0x000001475d70 by main thread:
#0 main /public/llvm-build/tsan.c:10:10 (a.out+0x46bfe6)
Location is global 'Global' of size 4 at 0x000001475d70 (a.out+0x000001475d70)
Thread T1 (tid=2, running) created by main thread at:
#0 pthread_create /public/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:930:3 (a.out+0x412120)
#1 main /public/llvm-build/tsan.c:9:3 (a.out+0x46bfd1)
SUMMARY: ThreadSanitizer: data race /public/llvm-build/tsan.c:4:10 in Thread1
==================
Thread 2 received signal SIGSEGV, Segmentation fault.
</pre>
<p>
I was able to get the above execution results around 10% of the time (being under a tracer had no positive effect on the frequency of successful executions).
<p>
I've managed to hit the following final results for this month, with another set of bugfixes and improvements:
<p>
<pre>
check-tsan:
Expected Passes : 248
Expected Failures : 1
Unsupported Tests : 83
Unexpected Failures: 44
</pre>
<p>
At the end of the month, TSan can now reliably executabe the same (already-working) program every time.
The majority of failures are in tests verifying sanitization of correct mutex locking usage.
<p>
There are still problems with NetBSD-specific libc and libpthread bootstrap code that conflicts with TSan.
Certain functions (pthread_create(3), pthread_key_create(3), _cxa_atexit())
cannot be started early by TSan initialization, and must be deferred late enough for the sanitizer to work correctly.
<p>
<h2>MSan</h2>
<p>
I've prepared a scratch support for MSan on NetBSD to help in researching how far along it is.
I've also cloned and adapted the existing FreeBSD bits; however, the code still needs more work and isn't functional yet.
The number of passed tests (5) is negligible and most likely does not work at all.
<p>
The conclusion after this research is that TSan shall be finished first, as it touches similar code.
<p>
In the future, there will be likely another round of iterating the system structs and types and adding the missing ones for NetBSD.
So far, this part has been done before executing the real MSan code.
I've added one missing symbol that was missing and was detected when attempting to link a test program with MSan.
<p>
<h2>Sanitizers</h2>
<p>
The GCC team has merged the LLVM sanitizer code, which has resulted in almost-complete support for ASan and UBsan on NetBSD. It can be found in the latest GCC8 snapshot, located in pkgsrc-wip/gcc8snapshot. Though, do note that there is an issue with getting backtraces from libasan.so, which can be worked-around by backtracing ASan events in a debugger.
UBsan also passes all GCC regression tests and appears to work fine.
The code enabling sanitizers on the GCC/NetBSD frontend will be submitted upstream once the backtracing issue is fixed and I'm satisfied that there are no other problems.
<p>
I've managed to upstream a large portion of generic+TSan+MSan code to compiler-rt and reduce local patches to only the ones that are in progress.
This deals with any rebasing issues, and allows me to just focus on the delta that is being worked on.
<p>
I've tried out the LLDB builds which have TSan/NetBSD enabled, and they built and started fine. However, there were some false positives related to the mutex locking/unlocking code.
<p>
<h1>Plans for the next milestone</h1>
<p>
The general goals are to finish TSan and MSan and switch back to LLDB debugging.
I plan to verify the impact of the TSan bootstrap initialization on the observed crashes and research the remaining failures.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/google_summer_of_code_2017Google Summer of Code 2017Thomas Klausner2017-10-18T16:43:40+00:002017-10-18T16:43:40+00:00NetBSD participated in the 2017 edition of <a href="https://summerofcode.withgoogle.com/">Google of Summer of Code</a> with 3 students.
All of the students finished their projects successfully. The following links report about their activities:
<ul>
<li><a href="https://blog.netbsd.org/tnf/entry/gsoc_2017_reports_add_code">Leonardo Taccari: Add multi-packages support to pkgsrc</a>
<li><a href="http://coypu.sdf.org/2017-08-29-LFS">Maya Rashish: LFS cleanup</a>
<li><a href="https://bumblebeesky.tumblr.com/">Utkarsh Anand: Make Anita support multiple virtual machine systems</a>
</ul>
Congratulations to the students for finishing their projects successfully, and thanks to Google for sponsoring!
https://blog.netbsd.org/tnf/entry/kernel_aslr_on_amd64Kernel ASLR on amd64Maxime Villard2017-10-12T05:11:55+00:002017-10-12T05:11:55+00:00Recently, I completed a Kernel ASLR implementation for NetBSD-amd64, making
NetBSD the first BSD system to support such a feature. Simply said, KASLR is a
feature that randomizes the location of the kernel in memory, making it harder
to exploit several classes of vulnerabilities, both locally (privilege
escalations) and remotely (remote code executions).Recently, I completed a Kernel ASLR implementation for NetBSD-amd64, making
NetBSD the first BSD system to support such a feature. Simply said, KASLR is a
feature that randomizes the location of the kernel in memory, making it harder
to exploit several classes of vulnerabilities, both locally (privilege
escalations) and remotely (remote code executions).
<p>
<h1>Current design</h1>
<p>
The current design is based on a specialized kernel called the "prekern", which
operates between the bootloader and the kernel itself. The kernel is compiled
as a raw library with the GENERIC_KASLR configuration file, while the prekern
is compiled as a static binary. When the machine boots, the bootloader jumps
into the prekern. The prekern relocates the kernel at a random virtual address
(VA), and jumps into it. Finally, the kernel performs some cleanup, and executes
normally.
<p>
Currently, the kernel is randomized as a single block. That is to say, a random
VA is chosen, and the kernel text->rodata->data sections are mapped
contiguously starting from there. It has several drawbacks, but it's a first
shot.
<p>
To complete this implementation, work had to be done at three levels: the
bootloader, the prekern and the kernel. I committed several of the kernel and
bootloader patches discreetly a few months ago, to pave some way for real
changes. In the past few weeks, I changed the low-level x86 layer of the kernel
and replaced several hard-coded (and sometimes magic) values by variables, in
such a way that the kernel can run with a non-static memory layout. Finally, the
last step was committing the prekern itself to the source tree.
<p>
<h1>Future work</h1>
<p>
<ul>
<li>Randomize the kernel sections independently, and intertwine them.</li>
<li>Modify several kernel entry points not to leak kernel addresses to userland.</li>
<li>Randomize the kernel heap too (which is still static for now).</li>
<li>Fix a few other things that need some more work.</li>
</ul>
<p>
<h1>How to use</h1>
<p>
All of the patches are now in NetBSD-current. Instructions on how to
install and use this implementation can be found
<a href="https://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html">here</a>;
they are inlined below, and probably won't change in the future.
<p>
Make sure you have a v5.11 bootloader installed. If you don't, build and install
a new bootloader:
<pre>
$ cd /usr/src/sys/arch/i386/stand/boot
$ make
# cp biosboot/boot /
</pre>
Build and install a KASLR kernel:
<pre>
$ cd /usr/src
$ ./build.sh -u kernel=GENERIC_KASLR
# cp /usr/obj/sys/arch/amd64/compile/GENERIC_KASLR/netbsd /netbsd_kaslr
</pre>
Finally, build and install a prekern:
<pre>
$ cd /usr/src/sys/arch/amd64/stand/prekern
$ make
# cp prekern /prekern
</pre>
Reboot your machine. In the boot prompt, enter:
<pre>
&gt pkboot netbsd_kaslr
</pre>
The system will boot with no further user interaction. Should you encounter
any regression or unexpected behavior, please report it immediately
to tech-kern.
<p>
Note that you can still boot a static kernel, by typing as usual:
<pre>
&gt boot netbsd
</pre>
<h1>Availability</h1>
<p>
This KASLR implementation will be available starting from NetBSD 9. Once it is
stabilized, it may be backported to NetBSD 8. Until then, feel free to test it!
<p><p>https://blog.netbsd.org/tnf/entry/eurobsdcon_2017_travel_notes_afterEuroBSDcon 2017: "travel notes" after the conferenceLeonardo Taccari2017-10-11T13:05:53+00:002017-10-12T14:04:52+00:00<p>
Let me tell you about my experience at
<a href="https://2017.eurobsdcon.org/">EuroBSDcon 2017</a>
in <a href="https://en.wikipedia.org/wiki/Paris">Paris</a>, <a
href="https://en.wikipedia.org/wiki/France">France</a>. We will
see what was presented during the NetBSD developer summit on Friday
and then we will give a look to all of the
<a href="://www.NetBSD.org">NetBSD</a> and
<a href="://www.pkgsrc.org/">pkgsrc</a> presentations given during
the conference session on Saturday and Sunday. Of course, a lot of
fun also happened on the "hall track", the several breaks
during the conference and the dinners we had together with other
*BSD developers and community! This is difficult to describe and
I will try to just share some part of that with photographs that
we have taken. I can just say that it was a really beautiful
experience, I had a great time with others and, after coming back
home... ...I miss all of that! :) So, if you have never been in
any BSD conferences I strongly suggest you to go to the next ones,
so please stay tuned via
<a href="://www.NetBSD.org/gallery/events.html">NetBSD Events</a>.
Being there this is probably the only way to understand these feelings!
</p><p>
Let me tell you about my experience at
<a href="https://2017.eurobsdcon.org/">EuroBSDcon 2017</a>
in <a href="https://en.wikipedia.org/wiki/Paris">Paris</a>, <a
href="https://en.wikipedia.org/wiki/France">France</a>. We will
see what was presented during the NetBSD developer summit on Friday
and then we will give a look to all of the
<a href="//www.NetBSD.org">NetBSD</a> and
<a href="//www.pkgsrc.org/">pkgsrc</a> presentations given during
the conference session on Saturday and Sunday. Of course, a lot of
fun also happened on the "hall track", the several breaks
during the conference and the dinners we had together with other
*BSD developers and community! This is difficult to describe and
I will try to just share some part of that with photographs that
we have taken. I can just say that it was a really beautiful
experience, I had a great time with others and, after coming back
home... ...I miss all of that! :) So, if you have never been in
any BSD conferences I strongly suggest you to go to the next ones,
so please stay tuned via
<a href="//www.NetBSD.org/gallery/events.html">NetBSD Events</a>.
Being there this is probably the only way to understand these feelings!
</p>
<h2>Thursday (21/09): NetBSD developers dinner</h2>
<p>
Arriving in Paris via a night train from Italy I
literally sleep-walked through Paris getting lost again and again.
After getting in touch with other developers we had a dinner together and went
sightseeing for <del>a^W</del>several beers!
</p>
<h2>Friday (22/09): NetBSD developers summit</h2>
<p>
On Friday morning we met for the NetBSD developers summit kindly hosted by
<a href='http://www.arolla.fr/'>Arolla</a>.
</p>
<p>
<a href="https://twitter.com/ArollaFr/status/911219875678433280">
<img alt="Photograph of the NetBSD develepors summit" src='https://www.NetBSD.org/~leot/blog-posts/imgs/DKVNoHVXUAEe1X3_small.jpg' />
</a>
<br />
From left to right: <code>alnsn</code>, <code>sborrill</code>;
<code>abhinav</code>; <code>uwe</code> and <code>leot</code>;
<code>christos</code>, <code>cherry</code>, <code>ast</code> and
<code>bsiegert</code>; <code>martin</code> and <code>khorben</code>.
</p>
<p>
The devsummit was moderated by J&ouml;rg (<code>joerg</code>) and organized by
Jean-Yves (<code>jym</code>).
</p>
<h3>NetBSD on Google Compute Engine -- Benny Siegert (<code>bsiegert</code>)</h3>
<p>
After a self-presentation the devsummit presentations session started with the
talk presented by Benny (<code>bsiegert</code>) about <em>NetBSD on Google
Compute Engine</em>.
</p>
<p>
Benny first introduced
<a href='https://cloud.google.com/compute/'>Google Compute Engine (GCE)</a>
and then started describing how to run NetBSD on it.
</p>
<p>
At the moment there are no official NetBSD images and so users need to create
their own. However, <a href='https://github.com/google/netbsd-gce'>netbsd-gce</a>
script completely automatize this process that:
</p>
<ul>
<li>uses <a href='https://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/misc/py-anita/README.html'>Anita</a>
to stage an installation in <a href='https://www.qemu.org/'>QEMU</a></li>
<li>adjust several tweaks to ensure that networking and storage will work on GCE</li>
<li>packs the image into a <code>.tar.gz</code> file</li>
</ul>
<p>
The <code>.tar.gz</code> image then just need to be uploaded to a Cloud Storage
bucket, create a GCE image from it and then launch VMs based on that image.
</p>
<p>
He also discussed about GCE instance metadata, several problems founds and how
they were fixed (it's better to use NetBSD 8_BETA or -current!) and some future
works.
</p>
<p>
For more information
<a href='https://www.NetBSD.org/gallery/presentations/bsiegert/netbsd-on-gce.pdf'>slides (PDF)</a>
of the talk are also available.
</p>
<h3>Scripting DDB with Forth -- Valery Ushakov (<code>uwe</code>)</h3>
<p>
Valery (<code>uwe</code>) presented a talk about <em>Scripting DDB
with Forth</em>. It was based on a long story and actually the
first discussion about it appeared on
<a href='https://mail-index.netbsd.org/tech-kern/'>tech-kern@</a>
mailing list in his
<a href='https://mail-index.netbsd.org/tech-kern/2016/05/02/msg020509.html'>Scripting DDB in Forth?</a>
thread (<a href='http://netbsd.gw.com/cgi-bin/man-cgi?ddb+4+NetBSD-current'>ddb(4)</a>
is the NetBSD in-kernel debugger).
</p>
<p>
He showed how one can associate
<a href='https://en.wikipedia.org/wiki/Forth_(programming_language)'>forth</a>
commands/conditions with ddb breakpoints.
He used "pid divisible by 3" as an example of condition
for a breakpoint set in
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?getpid+2+NetBSD-current'>getpid(2)</a>
system call:
</p>
<pre>
db{0}&gt; forth
ok : field create , does> @ + ;
ok #300 field lwp&gt;l_proc
ok #120 field proc&gt;p_pid
ok : getpid curlwp lwp&gt;l_proc @ proc&gt;p_pid @ ;
ok : checkpid getpid dup ." &gt; PID IS " . cr 3 mod 0= ;
ok bye
-- STACK: &lt;empty&gt;
db{0}&gt; break sys_getpid_with_ppid
db{0}&gt; command . = checkpid
db{0}&gt; c
</pre>
<p>
...and then on a shell:
</p>
<pre>
# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 798 lid 1 lowest kstack 0xdabb42c0
&gt; PID IS 798
-- STACK:
0xffffffff -1
Breakpoint in pid 798.1 (ksh) at netbsd:sys_getpid_with_ppid: pushl %ebp
db{0}&gt; c
# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 823 lid 1 lowest kstack 0xdabb42c0
> PID IS 823
-- STACK:
0x00000000 0
Command returned 0
#
</pre>
<p>
If you are more interested in this presentation I strongly suggest to also give
a look to <code>uwe</code>'s
<a href='https://bitbucket.org/nbuwe/forth'>forth</a> Mercurial repository.
</p>
<h3>News from the version control front -- J&ouml;rg Sonnenberger (<code>joerg</code>)</h3>
<p>
The third presentation of the devsummit was a presentation about the recent work done by
J&ouml;rg (<code>joerg</code>) in the VCS conversions.
</p>
<p>
J&ouml;rg started the presentation discussing about the infrastructure
used for the CVS -&gt; Fossil -&gt; Git conversion and
CVS -&gt; Fossil -&gt; Mercurial conversion.
</p>
<p>
It's worth also noticing that the Mercurial conversion is fully integrated and
is regularly pushed to <a href='https://bitbucket.org/netbsd/'>Bitbucket</a> and
<a href='https://bitbucket.org/netbsd/src'>src</a> repository pushed some
scalability limits to <a href='https://bitbucket.org/'>Bitbucket</a>!
</p>
<p>
Mercurial performance were also compared to the Git ones in details for several
operations.
</p>
<p>
A check list that compared the current status of the NetBSD VCS migration to the
<a href='https://wiki.freebsd.org/VersionControl'>FreeBSD VCS wiki one</a> was
described and then J&ouml;rg discussed the pending work and answered several
questions in the Q&amp;A.
</p>
<p>
For more information please give a look to the
<a href='https://www.NetBSD.org/gallery/presentations/joerg/eurobsdcon2017'><code>joerg</code>'s presentation slides (HTML)</a>.
If you would like to help for the VCS migration please also get in touch with
him!
</p>
<h3>Afternoon discussions and dinner</h3>
<p>
After the lunch we had several non-scheduled discussions, some time
for hacking, etc. We then had a nice dinner together (it was in a
restaurant with a very nice waiter who always shouted after
every order or after accidently dropping and crashing dishes!, yeah! That's
probably a bit weird but I liked that attitude! :)) and then did some
sightseeing and had a beer together.
</p>
<p>
<a href="https://twitter.com/abhi9u/status/912330142541860864">
<img alt="Photograph of the Friday dinner, taken by Christos" src='https://www.NetBSD.org/~leot/blog-posts/imgs/DKk_Y45XUAE1CcZ_small.jpg' />
</a>
<br />
From left to right: <code>uwe</code>, <code>bad</code>, <code>ast</code>,
<code>leot</code>, <code>martin</code>, <code>abhinav</code>,
<code>sborrill</code>, <code>alnsn</code>, <code>spz</code>.
</p>
<p>
<a href="https://twitter.com/abhi9u/status/912330142541860864">
<img alt="Photograph of the Friday dinner, taken by Abhinav" src='https://www.NetBSD.org/~leot/blog-posts/imgs/DKk_aPzXcAAk0pb_small.jpg' />
</a>
<br />
From left to right: <code>uwe</code>, <code>bad</code>, <code>ast</code>,
<code>christos</code>, <code>leot</code>, <code>martin</code>,
<code>sborrill</code>, <code>alnsn</code>, <code>spz</code>.
</p>
<h2>Saturday (23/09): First day of conference session and Social Event</h2>
<h3>A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (<code>abhinav</code>)</h3>
<p>
Abhinav (<code>abhinav</code>) presented his work on the new
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?spell+1+NetBSD-current'>spell(1)</a>
implementation he's working (that isn't just a <code>spell</code> replacement
but also a library that can be used by other programs!).
</p>
<p>
He described the current limitations of old <code>spell(1)</code> (to get an
idea please give a look to <a href='https://gnats.netbsd.org/48684'>bin/48684</a>),
described the project goals of the new <code>spell(1)</code>, additions to
<code>/usr/share/dict/words</code>, digged a bit in the implementation and
discussed several algorithms used and then provided a performance comparison
with other popular free software spell checkers
(<a href='http://aspell.net/'>aspell</a>,
<a href='https://hunspell.github.io/'>hunspell</a> and <a
href='https://www.gnu.org/software/ispell/'>ispell</a>).
</p>
<p>
He also showed an interactive demo of the new <code>spell(1)</code> in-action
integrated with a shell for auto-completion and spell check.
</p>
<p>
If you would like to try it please give a look to
<a href='https://github.com/abhinav-upadhyay/nbspell'>nbspell</a> Git repository
that contains the code and dicts for the new <code>spell(1)</code>!
</p>
<p>
<a href='https://www.youtube.com/watch?v=9JPKZUcBl88'>Video recording (YouTube)</a>
of the talk and <a href='https://www.netbsd.org/gallery/presentations/abhinav/EuroBSDCon2017/a_modern_spell.pdf'>slides
(PDF)</a> are also available!
</p>
<h3>Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (<code>cherry</code>)</h3>
<p>
Cherry (<code>cherry</code>) presented recent work done with Santhosh N.
Raju (<code>fox</code>) about
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?uvm_hotplug+9+NetBSD-current'>uvm_hotplug(9)</a>.
</p>
<p>
The talk covered most "behind the scenes" work: how TDD (test driven
development) was used, how <code>uvm_hotplug(9)</code> was designed
and implemented (with comparisons to the old implementation),
interesting edge cases during the development and how
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?atf+7+NetBSD-current'>atf(7)</a>
was used to do performance testing.
</p>
<p>
It was very interesting to learn how Cherry and Santhosh worked on that and on
the conclusion Cherry pointed out the importance of using existing Software
Engineering techniques in Systems Programming.
</p>
<p>
<a href='https://www.youtube.com/watch?v=cga3yAfzxmY'>Video recording (YouTube)</a>
and <a href='https://www.netbsd.org/gallery/presentations/cherry/eurobsdcon2017/uvm_hotplug.pdf'>slides (PDF)</a>
of the talk are also available!
</p>
<h3>Hardening pkgsrc -- Pierre Pronchery (<code>khorben</code>)</h3>
<p>
Pierre (<code>khorben</code>) presented a talk about recent pkgsrc security
features added in the recent months (and most of them also active on the just
released <code>pkgsrc-2017Q3</code>!).
</p>
<p>
He first introduced how security management and releng is handled
in pkgsrc, how to use
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?pkg_admin+1+NetBSD-current'>pkg_admin(1)</a>
<code>fetch-pkg-vulnerabilities</code> and <code>audit</code>
commands, etc.
</p>
<p>
Then package signatures (generation, installation) and recent hardening
features in pkgsrc were discussed in details, first introducing them and then
how pkgsrc handles them:
</p>
<ul>
<li>SSP: Stack Smashing Protection (enabled via <code>PKGSRC_USE_SSP</code>
in <code>mk.conf</code>)</li>
<li>Fortify (enabled via <code>PKGSRC_USE_FORTIFY</code> in <code>mk.conf</code>)</li>
<li>Stack check (enabled via <code>PKGSRC_USE_STACK_CHECK</code> in
<code>mk.conf</code>)</li>
<li>Position-Independent Executables (PIE) (enabled via <code>PKGSRC_MKPIE</code>
in <code>mk.conf</code>)</li>
<li>RELRO and BIND_NOW (enabled via <code>PKGSRC_USE_RELRO</code> in
<code>mk.conf</code>)</li>
</ul>
<p>
Challenges for each hardening features and future works were discussed.
</p>
<p>
For more information
<a href='https://www.youtube.com/watch?v=os_46jVbIWk'>video recording (YouTube)</a>
and
<a href='https://www.netbsd.org/gallery/presentations/khorben/eurobsdcon2017/Hardening pkgsrc.pdf'>slides (PDF)</a>
of the talk are available. A good introduction and reference for all pkgsrc
hardening features is the <a href='https://wiki.NetBSD.org/pkgsrc/hardening/'>Hardening
packages</a> wiki page.
</p>
<h3>Reproducible builds on NetBSD -- Christos Zoulas (<code>christos</code>)</h3>
<p>
Christos (<code>christos</code>) presented the work about reproducible builds on
NetBSD.
</p>
<p>
In his talk he first provided a rationale about reproducible builds (to learn
more please give a look to
<a href='https://reproducible-builds.org/'>reproducible-builds.org</a>!), he
then discussed about the NetBSD (cross) build process, the
<a href='https://tests.reproducible-builds.org/netbsd/netbsd.html'>current status</a>
and build variations that are done in the <code>tests.reproducible-builds.org</code>
build machines.
</p>
<p>
Then he provided and described several sources of difference that were present
in non-reproducible builds, like file-system timestamps, parallel builds
headaches due directory/build order, path normalization, etc.
For each of them he also discussed in details how these problems were solved in
NetBSD.
</p>
<p>
In the conclusion the status and possible TODOs were also discussed (please note
that both <code>-current</code> and <code>-8</code> are all built with
reproducible flags (<code>-P</code> option of <code>build.sh</code>)!)
</p>
<p>
<a href='https://www.youtube.com/watch?v=x1KTMNqB3FI'>Video recording (YouTube)</a>
of Christos' talk is available. Apart the resources discussed above a nice
introduction to reproducible builds in NetBSD is also the
<a href='https://blog.NetBSD.org/tnf/entry/netbsd_fully_reproducible_builds'>NetBSD
fully reproducible builds</a> blog post written by Christos last February!
</p>
<h3>Social event</h3>
<p>
The social event on Saturday evening took place on a boat that cruised on
the Seine river.
</p>
<p>
It was a very nice and different way to sightsee Paris, eat and enjoy some
drinks and socialize and discuss with other developers and community.
</p>
<p>
<img alt="Photograph from the boat, taken by Martin." src='https://www.NetBSD.org/~leot/blog-posts/imgs/20170923_204355_small.jpg' />
</p>
<h2>Sunday (24/09): Second day of conference session</h2>
<h3>The school of hard knocks - PT1 -- Sevan Janiyan (<code>sevan</code>)</h3>
<p>
Sevan (<code>sevan</code>) presented a talk about several notes and lessons learnt
whilst running tutorials to introduce NetBSD at several events
(<a href='http://oshug.org/event/46'>OSHUG #46</a> and
<a href='http://oshug.org/event/57'>OSHUG #57</a> and
<a href='http://oshug.org/event/58'>#58</a>) and experiences from past events
(<a href='http://chiphack.org/'>Chiphack 2013</a>).
</p>
<p>
He described problems a user may experience and how NetBSD was
introduced, in particular trying to avoid the steep learning curve
involved when experimenting with operating systems as a first step,
exploring documentation/source code, cross-building, scripting in
high-level programming languages
(<a href="https://en.wikipedia.org/wiki/Lua_(programming_language)">Lua</a>)
and directly prototyping and getting pragmatic via
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?rumpkernel+7+NetBSD-current'>rump</a>.
</p>
<p>
<a href='https://www.youtube.com/watch?v=Nekb3cve1zQ'>Video recording (YouTube)</a>
of Sevan's talk and <a
href='https://www.geeklan.co.uk/files/eurobsdcon2017'>slides (HTML)</a> are
available.
</p>
<h3>The LLDB Debugger on NetBSD -- Kamil Rytarowski (<code>kamil</code>)</h3>
<p>
Kamil (<code>kamil</code>) presented a talk about the recent <a
href='https://lldb.llvm.org/'>LLDB</a> debugger and a
lot of other related debuggers (but also
non-strictly-related-to-debugging!) works he's doing in the last
months.
</p>
<p>
He first introduced debugging concepts in general, provided several examples and
then he started discussing LLDB porting to NetBSD.
</p>
<p>
He then discussed about
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?ptrace+2+NetBSD-current'>ptrace(2)</a>
and other introspection interfaces, the several improvements done and tests
added for <code>ptrace(2)</code> in
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?atf+7+NetBSD-current'>atf(7)</a>.
</p>
<p>
He also discussed about tracking LLDB's trunk (if you are more curious please
give a look to <code>wip/llvm-git</code>,
<code>wip/clang-git</code>, <code>wip/lldb-git</code> packages in <a
href='https://www.pkgsrc.org/wip/'>pkgsrc-wip</a>!) and about LLVM sanitizers
and their current status in NetBSD.
</p>
<p>
In the conclusion he also discussed various TODOs in these areas.
</p>
<p>
<a href='https://www.youtube.com/watch?v=jNuJXjD9veQ'>Video recording (YouTube)</a>
and <a href='https://www.NetBSD.org/~kamil/eurobsdcon2017.html'>slides (HTML)</a>
of Kamil's talk are available.
Kamil also regularly write status update blog posts on
<a href='https://blog.NetBSD.org/'>blog.NetBSD.org</a> and
<a href='https://mail-index.netbsd.org/tech-toolchain/'>tech-toolchain@</a> mailing
list, so please stay tuned! :)
</p>
<h3>What's in store for NetBSD 8.0? -- Alistair Crooks (<code>agc</code>)</h3>
<p>
Alistair (<code>agc</code>) presented a talk about what we will see in NetBSD
8.0.
</p>
<p>
He discussed about new hardware supported (really "new", not new "old" hardware!
Of course also support for VAXstation 4000 TURBOchannel USB and GPIO is actually
new hardware as well! :)), LLVM/Clang, virtualization, PGP signing, updated
utilities in NetBSD, new networking features (e.g. <code>bouyer</code>'s
<code>sockcan</code> implementation), <code>u-boot</code>,
<a href='http://netbsd.gw.com/cgi-bin/man-cgi?dtrace+1+NetBSD-current'>dtrace(1)</a>,
improvements and new ports testing, reproducible builds,
FDT (Flattened Device Tree) and a lot of other news!
</p>
<p>
The entire presentation was done using the Socratic method (Q&amp;A) and it was
very interactive and nice!
</p>
<p>
<a href='https://www.youtube.com/watch?v=MrtOYqJpEKI'>Video recording (YouTube)</a>
and <a href='https://www.NetBSD.org/~agc/EuroBSDcon-20170919.pdf'>slides (PDF)</a>
of Alistair's talk are available.
</p>
<h3>Sunday dinner</h3>
<p>
After the conference we did some sightseeing in Paris, had a dinner together and
then enjoyed some beers!
</p>
<p>
<a href="https://twitter.com/abhi9u/status/912039595906293760">
<img alt="Photograph of the Sunday dinner, taken by Martin." src='https://www.NetBSD.org/~leot/blog-posts/imgs/DKg3BQGVwAAL1IV_small.jpg' />
</a>
<br />
On the left side: <code>abhinav</code>, <code>ast</code>, <code>seb</code>, <code>christos</code>
<br />
On the right side: <code>leot</code>, <code>Riastradh</code>, <code>uwe</code>, <code>sevan</code>, <code>agc</code>, <code>sborrill</code>
</p>
<p>
<a href="https://twitter.com/abhi9u/status/912039595906293760">
<img alt="Photograph of the Sunday dinner, taken by Abhinav." src='https://www.NetBSD.org/~leot/blog-posts/imgs/DKg3FG4VoAAXwI0_small.jpg' />
</a>
<br />
On the left side: <code>martin</code>, <code>ast</code>, <code>seb</code>, <code>christos</code>
<br />
On the right side: <code>leot</code>, <code>Riastradh</code>, <code>uwe</code>, <code>sevan</code>, <code>agc</code>, <code>sborrill</code>
</p>
<h2>Conclusion</h2>
<p>
It was a very nice weekend and conference. It is worth to mention that
EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!).
</p>
<p>
I would like to thank the entire EuroBSDcon organising committee (Baptiste
Daroussin, Antoine Jacoutot, Jean-S&eacute;bastien P&eacute;dron and Jean-Yves
Migeon), EuroBSDcon programme commitee (Antoine Jacoutot, Lars Engels,
Ollivier Robert, Sevan Janiyan, J&ouml;rg Sonnenberger, Jasper Lievisse
Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a
wonderful conference!
</p>
<p>
I also would like to thank the speakers for presenting very
interesting talks, all developers and community that attended the
NetBSD devsummit and conference, in particular Jean-Yves and
J&ouml;rg, for organizing and moderating the devsummit and
<a href='http://www.arolla.fr/'>Arolla</a> that kindly hosted us for the
NetBSD devsummit!
</p>
<p>
A special thanks also to Abhinav (<code>abhinav</code>) and Martin
(<code>martin</code>) for photographs and locals Jean-Yves (<code>jym</code>)
and Stoned (<code>seb</code>) for helping us in not get lost in Paris'
<em>rues</em>! :)
</p>
<p>
Thank you!
</p>https://blog.netbsd.org/tnf/entry/2017_netbsd_foundation_officers2017 NetBSD Foundation OfficersWilliam J. Coldwell2017-10-04T21:39:50+00:002017-10-04T21:39:50+00:00By vote of The NetBSD Foundation Board of Directors, the officers for the 2017 term are:
<br><br>
President: William J. Coldwell &lt;billc&gt;<br>
Vice President: Jeremy C. Reed &lt;reed&gt;<br>
Secretary: Christos Zoulas &lt;christos&gt;<br>
Treasurer: Christos Zoulas &lt;christos&gt;<br>
Assistant Secretary: Thomas Klausner &lt;wiz&gt;<br>
Assistant Treasurer: Taylor R. Campbell &lt;riastradh&gt;<br>
https://blog.netbsd.org/tnf/entry/eurobsdcon_2017_paris_reportEuroBSDcon-2017 ParisKamil Rytarowski2017-10-02T00:18:45+00:002017-10-02T00:18:45+00:00This year the annual EuroBSDcon event took place in Paris (September 21st-24th).
There were BSD summits, tutorials and talks, including NetBSD & pkgsrc ones.This year the annual EuroBSDcon event took place in Paris (September 21st-24th).
There were BSD summits, tutorials and talks, including NetBSD & pkgsrc ones.
<p>
<h1>Tutorials</h1>
<p>
I came to Paris on Wednesday in order to participate in tutorials:
<ul>
<li>Thursday - DTrace for Developers: no more printfs - by George Neville-Neil</li>
<li>Friday - How to untangle your threads from a giant lock in a multiprocessor system - by Taylor R. Campbell</li>
</ul>
<p>
<h1>The NetBSD Summit</h1>
<p>
On Friday there was a NetBSD summit that gathered around 15 developers.
I have not participated in it myself because of the tutorial on SMP. There were three presentations:
<ul>
<li>News from the version control front - by joerg,</li>
<li>Scripting DDB with Forth - by uwe,</li>
<li>NetBSD on Google Compute Engine - by bsiegert.</li>
</ul>
<p>
<h1>Talks</h1>
<p>
During Saturday and Sunday there were several NetBSD and pkgsrc related presentations:
<ul>
<li>A Modern Replacement for BSD spell(1) - Abhinav Upadhyay</li>
<li>Portable Hotplugging: NetBSD's uvm_hotplug(9) API development - Cherry G. Mathew</li>
<li>Hardening pkgsrc - Pierre Pronchery</li>
<li>Reproducible builds on NetBSD - Christos Zoulas</li>
<li>The school of hard knocks - PT1 - Sevan Janiyan</li>
<li>The LLDB Debugger on NetBSD - Kamil Rytarowski</li>
<li>What's in store for NetBSD 8.0? - Alistair Crooks</li>
</ul>
<p>
During the closing ceremony there was a speech by Alistair Crooks on behalf of The NetBSD Foundation.
<p>
<h1>LLDB</h1>
At the conference I presented my work on "The LLDB Debugger on NetBSD":
<ul>
<li><a href="http://netbsd.org/~kamil/eurobsdcon2017.html">slides</a></li>
<li><a href="https://www.youtube.com/watch?v=jNuJXjD9veQ">video</a></li>
</ul>
<p>
<h1>Plan for the next milestone</h1>
<p>
Resume porting tsan and msan to NetBSD with a plan to switch back to the LLDB porting.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/the_new_tnf_board_ofThe new TNF Board of Directors are installed and patched for 2017.William J. Coldwell2017-09-27T05:33:25+00:002017-09-27T05:33:25+00:00The slate of nominees was voted for and accepted by the members of the foundation. We'd like to our team of nomcom, voting coordinator, and voting validator for putting together the slate and managing the election process.
<br><br>
We welcome <b>Pierre Pronchery</b> <khorben> and <b>Makoto Fujiwara</b> <mef> to the 2017 Board of Directors. We look forward to working with you!
<br><br>
We appreciate all of the wonderful work that <b>S.P.Zeidler</b> <spz> and <b>Erik Berls</b> <cyber> have done on the board during their time as directors, and are grateful for their excellent service to the foundation.
<br><br>
Thank you to all members participating by nominating candidates and voting on the slate.
<br><br>
<br><br>
Respectfully submitted for The NetBSD Foundation,<br>
<i>William J. Coldwell</i> <billc><br>
<i>President/Chairperson</i><br>
https://blog.netbsd.org/tnf/entry/netbsd_buildbot_in_the_binutilsNetBSD buildbot in the binutils-gdb projectKamil Rytarowski2017-09-14T21:12:24+00:002017-09-14T21:35:46+00:00The NetBSD Foundation supports projects that strive to ship the best possible support in developer oriented software. This is not exclusive to LLVM, but also includes the more traditional GNU toolchain.The NetBSD Foundation supports projects that strive to ship the best possible support in developer-oriented software. This is not exclusive to LLVM, but also includes the more traditional GNU toolchain.
<p>
Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.
<p>
A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.
<p>
The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.
<p>
After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on <a href="https://gdb-build.sergiodj.net/buildslaves/gdb-amd64-netbsd7">NetBSD/amd64</a>. This bot is hosted within The NetBSD Foundation's internal infrastructure.
<p>
The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.
<p>
Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.
<p>
The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.https://blog.netbsd.org/tnf/entry/mercurial_mirror_on_bitbucketMercurial mirror on BitbucketKamil Rytarowski2017-09-01T22:50:31+00:002017-09-01T22:50:31+00:00Joerg Sonnenberger has <a href="https://mail-index.netbsd.org/tech-repository/2017/09/01/msg000647.html">announced</a> a new set of mirrored repositories.
<p>
You can find Mercurial versions of src, pkgsrc and xsrc under
<p>
<ul>
<li><a href="https://bitbucket.org/netbsd/src">https://bitbucket.org/netbsd/src</a></li>
<li><a href="https://bitbucket.org/netbsd/pkgsrc">https://bitbucket.org/netbsd/pkgsrc</a></li>
</ul>
and
<ul>
<li><a href="https://bitbucket.org/netbsd/xsrc">https://bitbucket.org/netbsd/xsrc</a></li>
</ul>
<p>
The same rules as for the fossil and github repositories apply, i.e.
there may be occasional glitches and if it becomes too bad, they might
be recreated from scratch.
<p>
See more information in the posted <a href="https://mail-index.netbsd.org/tech-repository/2017/09/01/msg000647.html">thread</a> to tech-repository.https://blog.netbsd.org/tnf/entry/llvm_libfuzzer_and_safestack_portedLLVM libFuzzer and SafeStack ported to NetBSDKamil Rytarowski2017-09-01T22:31:03+00:002017-09-01T22:31:03+00:00This month I've finally finished upstreaming NetBSD support in ASan and UBsan.
For better coverage of the sanitizers and on user request I've ported libFuzzer and SafeStack.
There are mutual dependencies between the compiler-rt features.
NetBSD after sorting out msan and tsan shall get all the remaining ones enabled.
This is open topic after finishing LLDB.
I have also prepared better ground for the coming work on ptrace(2) enhancements with the removal of
the filesystem tracing (<code>/proc/#/ctl</code>).This month I've finally finished upstreaming NetBSD support in ASan and UBsan.
For better coverage of the sanitizers and on user request I've ported libFuzzer and SafeStack.
There are mutual dependencies between the compiler-rt features.
NetBSD after sorting out msan and tsan shall get all the remaining ones enabled.
This is open topic after finishing LLDB.
I have also prepared better ground for the coming work on ptrace(2) enhancements with the removal of
the filesystem tracing (<code>/proc/#/ctl</code>).
<p>
<h1>LLVM</h1>
<p>
The majority of the work has been done in the LLVM projects.
<p>
The developed features are not production ready and they will need productization in the future.
There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features.
There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers).
The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken.
I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).
<p>
I expect to sort out the mentioned problems after finishing LLDB.
<p>
<h2>LLVM JIT</h2>
<p>
There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT.
There have been developed and introduced cleanups in the code (like better error handling templates)
in order to prepare a draft of new API.
<p>
<h2>ASAN</h2>
<p>
All local code for <a href="https://clang.llvm.org/docs/AddressSanitizer.html">ASan</a> has been merged upstream.
This includes NetBSD patches in LLVM, Clang and compiler-rt.
<p>
All but two (one on i386 version and the other on amd64) tests (check-asan) pass.
<p>
<h2>UBSAN</h2>
<p>
Similarly to ASan, <a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html">UBsan</a> has been fully upstreamed.
All tests (check-ubsan) pass.
<p>
<h2>SafeStack</h2>
<p>
<a href="https://clang.llvm.org/docs/SafeStack.html">SafeStack</a> is a software security hardening technique that creates two stacks:
one for data that needs to be kept safe, such as return addresses and function pointers;
and an unsafe stack for everything else.
<p>
With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions)
SafeStack is an excellent candidate for inclusion in pkgsrc.
<p>
Core programs could be hardened as well,
but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.
<p>
Using SafeStack adds marginal overhead.
<p>
<h2>libFuzzer</h2>
<p>
Citing the project page, <a href="http://llvm.org/docs/LibFuzzer.html">LibFuzzer</a> is an in-process, coverage-guided, evolutionary fuzzing engine.
<p>
LibFuzzer is linked with the library under test,
and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function');
the fuzzer then tracks which areas of the code are reached,
and generates mutations on the corpus of input data in order to maximize the code coverage.
The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.
<p>
This functionality still requires more sanitizers to get aboard and is now of restricted functionality.
<p>
<h2>TSAN</h2>
<p>
Part of the <a href="https://clang.llvm.org/docs/ThreadSanitizer.html">TSan</a> code has been upstreamed.
However, a functional port isn't finished yet.
<p>
The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.
<p>
<h2>LSAN</h2>
<p>
I was also working on <a href="https://clang.llvm.org/docs/LeakSanitizer.html">LSan</a>.
This sanitizer already builds and appears to be quite completed, however there is work needed for
the implementation of StopTheWorld() function to self-introspect the process and threads.
I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.
<p>
<h2>MSAN</h2>
<p>
So far I have not been working on the <a href="https://clang.llvm.org/docs/MemorySanitizer.html">MSan</a> specific bits.
The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework,
the proper handling of the NetBSD specific process map is still to be done.
<p>
<h2>PROFILE</h2>
<p>
The profile library is used to collect coverage information.
It already passes most of the tests,
however it's not turned on,
as upstream requested additional time to be spent on the issues and it's not a priority right now.
<p>
<h1>NetBSD kernel</h1>
<p>
I've removed the filesystem tracing feature.
<p>
This is a legacy interface from 4.4BSD, and it was
introduced to overcome shortcomings of ptrace(2) at that time, which are
no longer relevant (performance). Today <code>/proc/#/ctl</code> offers a narrow
subset of ptrace(2) commands and is not applicable for modern
applications use beyond simplistic tracing scenarios.
<p>
This removal simplified kernel internals. Users are still able to
use all the other <code>/proc</code> files.
<p>
This change doesn't affect other procfs files and Linux compat
features within mount_procfs(8). <code>/proc/#/ctl</code> isn't available on Linux.
<p>
<h1>Plan for the next milestone</h1>
<p>
This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris.
I will speak about the LLDB porting to NetBSD.
<p>
Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting.
The sanitizers will be used to develop and debug the LLVM debugger.
There is also integration with sanitizers in LLDB.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>https://blog.netbsd.org/tnf/entry/gsoc_2017_reports_add_codeGSoC 2017 Reports: Add <code>SUBPACKAGES</code> support to pkgsrc, part 1Leonardo Taccari2017-08-31T17:06:53+00:002017-08-31T17:38:48+00:00<p>
In this blog post series I will discuss about <code>SUBPACKAGES</code> work
done during
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2017</a>.
</p>
<p>
In this first part I'll briefly introduce what are <code>SUBPACKAGES</code>,
why and when can be useful and finally we'll give a quick look to a trivial
<a href="https://www.pkgsrc.org/">pkgsrc</a> package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
</p><p>
In this blog post series I will discuss about <code>SUBPACKAGES</code> work
done during
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2017</a>.
</p>
<p>
In this first part I'll briefly introduce what are <code>SUBPACKAGES</code>,
why and when can be useful and finally we'll give a quick look to a trivial
<a href="https://www.pkgsrc.org/">pkgsrc</a> package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
</p>
<h2>Introduction</h2>
<p>
<code>SUBPACKAGES</code> (on some package systems they are known as
<em>multi-packages</em>, but this term for pkgsrc is already used by packages
that can be built against several versions (e.g. Python, PHP, Ruby packages))
consist in generating multiple binary packages from a single pkgsrc package.
For example, from a pkgsrc package - <code>local/frobnitzem</code> - we will
see how to generate three separate binary packages:
<code>frobnitzem-foo</code>, <code>frobnitzem-bar</code> and
<code>frobnitzem-baz</code>.
</p>
<p>
This can be useful to separate several components of binary
packages (and avoid to run the <code>extract</code> and
<code>configure</code> phase two times!),
for <a href="https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug">debugpkgs</a>
(so that all <code>*.debug</code> files containing debug symbols are contained
in a separate <code>-debugpkg</code> package that can be installed only when it
is needed), etc..
</p>
<h2>A simple <code>SUBPACKAGES</code> package: <code>frobnitzem</code>!</h2>
<p>
To understand how <code>SUBPACKAGES</code> works and can be useful let's start
to see an example of it in practice: <code>frobnitzem</code>.
</p>
<p>
<code>frobnitzem</code> is a trivial package that just install three scripts in
<code>${PREFIX}/bin</code>, let's see it:
</p>
<pre>
% cd pkgsrc/local/frobnitzem
% tree
.
|-- DESCR
|-- Makefile
|-- PLIST
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 6 files
% find . -type f | xargs tail -n +1
==&gt; ./Makefile &lt;==
# $NetBSD$
DISTNAME= frobnitzem-0
CATEGORIES= local
MASTER_SITES= # empty
DISTFILES= # empty
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT= Simple subpackages example
LICENSE= public-domain
FILESDIR= ${.CURDIR}/../../local/frobnitzem/files
WRKSRC= ${WRKDIR}/frobnitzem
NO_BUILD= yes
do-extract:
${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}
do-install:
${INSTALL_SCRIPT_DIR} ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin
.include "../../mk/bsd.pkg.mk"
==&gt; ./files/frobnitzem/frobnitzem-bar &lt;==
#!/bin/sh
echo "bar"
==&gt; ./files/frobnitzem/frobnitzem-baz &lt;==
#!/bin/sh
echo "baz"
==&gt; ./files/frobnitzem/frobnitzem-foo &lt;==
#!/bin/sh
echo "foo"
==&gt; ./PLIST &lt;==
@comment $NetBSD$
bin/frobnitzem-bar
bin/frobnitzem-baz
bin/frobnitzem-foo
==&gt; ./DESCR &lt;==
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
</pre>
<p>
Nothing fancy, just three simple scripts, <code>frobnitzem-{bar,baz,foo}</code>
that will respectively print to the standard output <code>bar</code>,
<code>baz</code> and <code>foo</code>.
Let's build and install the <code>frobnitzem</code> package:
</p>
<pre>
% make install
===&gt; Installing dependencies for frobnitzem-0
===&gt; Overriding tools for frobnitzem-0
===&gt; Extracting for frobnitzem-0
[...]
===&gt; Installing for frobnitzem-0
[...]
===&gt; Installing binary package of frobnitzem-0
[...]
</pre>
<p>
And now let's try scripts installed as part of the <code>frobnitzem</code>
package:
</p>
<pre>
% foreach w (bar baz foo)
... frobnitzem-$w
... end
bar
baz
foo
</pre>
<p>
Okay, as we expected. Despite <code>frobnitzem-{foo,bar,baz}</code> don't do
anything particularly useful we can split the <code>frobnitzem-0</code>
package in three separate subpackages: <code>frobnitzem-foo-0</code>,
<code>frobnitzem-bar-0</code> and
<code>frobnitzem-baz-0</code> (they provides different functionalities and can
also coexist if they're in separated binary packages).
</p>
<p>
To do that we need to slighty adjust <code>Makefile</code>, split the
<code>PLIST</code> in <code>PLIST.{foo,bar,baz}</code> (one for each separate
subpackage), split the <code>DESCR</code> in <code>DESCR.{foo,bar,baz}</code>.
So, at the end in <code>local/frobnitzem</code> we'll have:
</p>
<pre>
% tree
.
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 10 files
</pre>
<h3>Splitting <code>DESCR</code> and <code>PLIST</code></h3>
<p>
<code>DESCR</code> and <code>PLIST</code> splits are straightforward. We just
provide a separate <code>DESCR.&lt;spkg&gt;</code> for each subpackage,
e.g. for the <code>foo</code> subpackage:
</p>
<pre>
% cat DESCR.foo
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
This package provide the foo functionalities.
</pre>
<p>
Similarly, regarding <code>PLIST</code>s, we just provide a separate
<code>PLIST.&lt;spkg&gt;</code> for each subpackage, e.g. for the
<code>foo</code> subpackage:
</p>
<pre>
% cat PLIST.foo
@comment $NetBSD$
bin/frobnitzem-foo
</pre>
<h3><code>Makefile</code> changes</h3>
<p>
In Makefile we'll need to list all <code>SUBPACKAGES</code> hence we'll add the
following line as first paragraph:
</p>
<pre>
SUBPACKAGES= foo bar baz
</pre>
<p>
We'll then need to define a <code>PKGNAME.&lt;spkg&gt;</code> for each
subpackages:
</p>
<pre>
PKGNAME.foo= frobnitzem-foo-0
PKGNAME.bar= frobnitzem-bar-0
PKGNAME.baz= frobnitzem-baz-0
</pre>
<p>
...and similarly <code>COMMENT</code> variable should be defined for each
subpackage via <code>COMMENT.&lt;spkg&gt;</code>:
</p>
<pre>
COMMENT.foo= Simple subpackages example (foo)
COMMENT.bar= Simple subpackages example (bar)
COMMENT.baz= Simple subpackages example (baz)
</pre>
<p>
To recap here how we have adjusted Makefile, all the other lines rest unchanged:
</p>
<pre>
% sed '/LICENSE/q' &lt; Makefile
# $NetBSD$
SUBPACKAGES= foo bar baz
DISTNAME= frobnitzem-0
PKGNAME.foo= frobnitzem-foo-0
PKGNAME.bar= frobnitzem-bar-0
PKGNAME.baz= frobnitzem-baz-0
MASTER_SITES= # empty
DISTFILES= # empty
CATEGORIES= local
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT.foo= Simple subpackages example (foo)
COMMENT.bar= Simple subpackages example (bar)
COMMENT.baz= Simple subpackages example (baz)
LICENSE= public-domain
</pre>
<p>
Finally we can install <del>it^W</del>them! The usual <code>make
install</code> will generate three binary packages
(<code>frobnitzem-foo-0.tgz</code>, <code>frobnitzem-bar-0.tgz</code>,
<code>frobnitzem-baz-0.tgz</code>) and install all of them:
</p>
<pre>
% make install
===&gt; Installing dependencies for frobnitzem-0
[...]
===&gt; Overriding tools for frobnitzem-0
===&gt; Extracting for frobnitzem-0
[...]
===&gt; Installing for frobnitzem-0
[...]
=&gt; Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz
=&gt; Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz
=&gt; Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz
[...]
===&gt; Installing binary package of frobnitzem-foo-0
===&gt; Installing binary package of frobnitzem-bar-0
===&gt; Installing binary package of frobnitzem-baz-0
[...]
</pre>
<p>
Now we can try them and use
<a href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_info+1+NetBSD-current">pkg_info(1)</a>
to get some information about them:
</p>
<pre>
% frobnitzem-foo
foo
% pkg_info -Fe /usr/pkg/bin/frobnitzem-foo
frobnitzem-foo-0
% pkg_info frobnitzem-baz
Information for frobnitzem-baz-0:
Comment:
Simple subpackages example (baz)
Description:
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
This package provide the baz functionalities.
Homepage:
http://netbsd.org/~leot/gsoc2017-diary/
% pkg_info -L frobnitzem-bar
Information for frobnitzem-bar-0:
Files:
/usr/pkg/bin/frobnitzem-bar
</pre>
<p>
So we can see that <code>make install</code> actually installed three different
binary packages.
</p>
<p>
To deinstall all <code>SUBPACKAGES</code> we can run <code>make deinstall</code>
in the <code>local/frobnitzem</code> directory (that will remove all
subpackages) or we can just manually invoke
<a href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_delete+1+NetBSD-current">pkg_delete(1)</a>.
</p>
<h2>An high-level look at how SUBPACKAGES support is implemented</h2>
<p>
Most of the changes needed are in <code>mk/pkgformat/pkg/</code> hierarchy
(previously known as <code>mk/flavour</code> and then renamed and generalized to
other package formats during
<a href="http://addpackageforma.sourceforge.net/">Anton Panev's Google Summer of Code 2011</a>).
</p>
<p>
The code in <code>mk/pkgformat/${PKG_FORMAT}/</code> handle the interaction of
pkgsrc with the particular <code>${PKG_FORMAT}</code>, e.g. for <code>pkg</code>
populate meta-data files used by
<a href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_create+1+NetBSD-current">pkg_create(1)</a>,
install/delete packages via
<a href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current">pkg_add(1)</a>,
and
<a href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_delete+1+NetBSD-current">pkg_delete(1)</a>,
etc.
</p>
<p>
For more information
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/pkgsrc/mk/pkgformat/README?rev=1.3"><code>mk/pkgformat/README</code></a>
is a good introduction to <code>pkgformat</code> hierarchy.
</p>
<p>
Most of the changes done respect the following template:
</p>
<pre>
.if !empty(SUBPACKAGES)
. for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
. endfor
.else # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]
.endif # SUBPACKAGES
</pre>
<p>
In particular, in <code>mk/pkgformat/pkg/</code> targets were adjusted to
create/install/deinstall/etc. all subpackages.
</p>
<p>
Apart <code>mk/pkgformat</code> other changes were needed in
<code>mk/install/install.mk</code> in order to adjust the <code>install</code>
phase for <code>SUBPACKAGES</code>.
</p>
<p>
Regarding <code>PLIST.&lt;spkg&gt;</code> handling <code>mk/plist/plist.mk</code>
needed some adjustments to honor each PLIST per-subpackage.
</p>
<p>
<code>mk/bsd.pkg.mk</code> needed to be adjusted too in order to honor several
per-subpackage variables (the <code>*.&lt;spkg&gt;</code> ones) and
per-subpackage <code>DESCR.&lt;spkg&gt;</code>.
</p>
<h2>Conclusion</h2>
<p>
In this first part of this blog post series we have seen what are
<code>SUBPACKAGES</code>, when and why they can be useful.
</p>
<p>
We have then seen a practical example of them taking a very trivial package
and learned how to "subpackage-ify" it.
</p>
<p>
Then we have described - from an high-level perspective - the changes needed to
the pkgsrc infrastructure for the <code>SUBPACKAGES</code> features that we have
used. If you are more interested in them please give a look to the
<a href="https://github.com/iamleot/pkgsrc/tree/debugpkg">pkgsrc debugpkg
branch</a> that contains all work done described in this blog post.
</p>
<p>
In the next part we will see how to handle <code>*DEPENDS</code> and
<code>buildlink3</code> inclusion for subpackages.
</p>
<p>
I would like to thanks <a href="https://www.google.com/">Google</a> for
organizing <a href="https://summerofcode.withgoogle.com/">Google Summer of
Code</a>, the entire <a href='https://www.netbsd.org/foundation/'>The NetBSD
Foundation</a> and in particular my mentors Taylor R. Campbell, William J.
Coldwell and Thomas Klausner for providing precious guidance during these three
months. A special thank you also to J&ouml;rg Sonnenberger who provided very
useful suggestions. Thank you!
</p>https://blog.netbsd.org/tnf/entry/llvm_clang_and_compiler_rtLLVM, Clang and compiler-rt support enhancementsKamil Rytarowski2017-08-03T13:25:51+00:002017-08-04T16:05:34+00:00In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero. In the last month I started with upstream of the code for sanitizers:
the common layer and ubsan.
I worked also on the elimination of unexpected failures in LLVM and Clang.
I've managed to achieve, with a pile of local patches,
the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang)
(however these ones were caused by hardcoded environment -lstdc++ vs -lc++).
The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero.
<p>
<h1>LLVM</h1>
<p>
In order to achieve the goals of testability concerning the LLVM projects,
I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree.
The set of these projects is composed of:
llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra.
These were required to build and execute test-suites in the LLVM's projects.
Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang,
however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package
in order to develop the work environment with the ability to build and execute unit tests.
<p>
There were four threads within LLVM:
<ul>
<li>Broken std::call_once with libstdc++.
This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::call_once.
I've discovered that the llvm::call_once workaround isn't sufficient for the whole LLVM functionality,
as std::call_once can be called internally inside the libstdc++ libraries - like within the C++11 futures interface.
This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker.
</li>
<li>Unportable shell construct hardcoded in tests "&gt;&". This has been fixed upstream.</li>
<li>LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X.
This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces.
This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements.
I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream.
My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X.
</li>
<li>Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt.</li>
</ul>
<p>
The LLVM JIT API is the last cause of unexpected failures in check-llvm.
This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests.
<p>
<h1>Clang</h1>
<p>
I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64.
This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt.
<p>
<h1>compiler-rt</h1>
<p>
The current compiler-rt tasks can be divided into:
<ol>
<li>upstream sanitizer common code shared with POSIX platforms</li>
<li>upstream sanitizer common code shared with Linux and FreeBSD</li>
<li>upstream sanitizer common code shared with FreeBSD</li>
<li>upstream sanitizer common code specific to NetBSD</li>
<li>build, execute and pass tests for sanitizer common code in check-santizer</li>
</ol>
<p>
This means that ubsan, asan and the rest of the specific sanitizers wait in queue.
<p>
All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one.
<p>
The last point with check-sanitizer unveiled so far two generic bugs on NetBSD:
<ul>
<li>Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls.</li>
<li>Honor PTHREAD_DESTRUCTOR_ITERATIONS in libpthread.</li>
</ul>
These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems.
The fixes were introduced by Christos Zoulas and backported to NetBSD-8.
<p>
<h1>Plan for the next milestone</h1>
<p>
I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code.
The roadmap for the next month is to continue working on the goals of the previous months.
std::call_once is an example that every delayed bug keeps biting again and again in future.
<p>
LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT.
There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0).
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>https://blog.netbsd.org/tnf/entry/porting_netbsd_to_allwinner_h3Porting NetBSD to Allwinner H3 SoCsjmcneill2017-07-09T12:31:04+00:002017-07-10T14:17:58+00:00<p>
<img src="//www.netbsd.org/~jmcneill/quad-core-H3.jpg" style="width: 83px; height: 83px; float: right;">
A new <i>SUNXI</i> evbarm kernel has appeared recently in NetBSD -current with support for boards based on the <a href="http://www.allwinnertech.com/index.php?c=product&a=index&id=47">Allwinner H3</a> system on a chip (SoC). The H3 SoC is a quad-core Cortex-A7 SoC designed primarily for set-top boxes, but has managed to find its way into many single-board computers (SBC). This is one of the first evbarm ports built from the ground up with <a href="https://en.wikipedia.org/wiki/Device_tree">device tree</a> support, which helps us to use a single kernel config to support many different boards.
</p><p>
<img src="//www.netbsd.org/~jmcneill/quad-core-H3.jpg" style="width: 83px; height: 83px; float: right;">
A new <i>SUNXI</i> evbarm kernel has appeared recently in NetBSD -current with support for boards based on the <a href="http://www.allwinnertech.com/index.php?c=product&a=index&id=47">Allwinner H3</a> system on a chip (SoC). The H3 SoC is a quad-core Cortex-A7 SoC designed primarily for set-top boxes, but has managed to find its way into many single-board computers (SBC). This is one of the first evbarm ports built from the ground up with <a href="https://en.wikipedia.org/wiki/Device_tree">device tree</a> support, which helps us to use a single kernel config to support many different boards.
</p>
<p style="clear: both;">
To get these boards up and running, first we need to deal with low-level startup code. For the SUNXI kernel this currently lives in <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/evbarm/sunxi/">sys/arch/evbarm/sunxi/</a>. The purpose of this code is fairly simple; initialize the boot CPU and initialize the MMU so we can jump to the kernel. The initial MMU configuration needs to cover a few things -- early on we need to be able to access the kernel, UART debug console, and the device tree blob (DTB) passed in from U-Boot. We wrap the kernel in a U-Boot header that claims to be a Linux kernel; this is no accident! This tells U-Boot to use the Linux boot protocol when loading the kernel, which ensures that the DTB (loaded by U-Boot) is processed and passed to us in <i>r2</i>.
</p>
<p>
Once the CPU and MMU are ready, we jump to the generic ARM FDT implementation of <i>initarm</i> in <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/evbarm/fdt/fdt_machdep.c">sys/arch/evbarm/fdt/fdt_machdep.c</a>. The first thing this code does is validate and relocate the DTB data. After it has been relocated, we compare the <i>compatible</i> property of the root node in the device tree with the list of ARM platforms compiled into the kernel. The Allwinner sunxi platform code lives in <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/arm/sunxi/sunxi_platform.c">sys/arch/arm/sunxi/sunxi_platform.c</a>. The sunxi platform code provides SoC-specific versions of code needed early at boot. We need to know how to initialize the debug console, spin up application CPUs, reset the board, etc.
</p>
<p>
Instead of writing H3-specific code for spinning up application CPUs, I took advantage of U-Boot's <a href="http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/index.html">Power State Coordination Interface</a> implementation. A <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/arm/arm/psci.c">psci(4)</a> driver was added and the <i>allwinner,sun8i-h3</i> platform code was modified to use this code to start up all processors.
</p>
<p>
With a bit of luck, we're now booting and enumerating devices. Apart from a few devices, almost nothing works yet as we are missing a driver for the CCU. The CCU in the Allwinner H3 SoC controls PLLs and most of the clock generation, division, muxing, and gating. Since there are many similarities between Allwinner SoCs, I opted to write generic CCU code and then SoC-specific frontends. The resulting code lives in <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/arm/sunxi/">sys/arch/arm/sunxi/</a>; generic code as <i>sunxi_ccu.c</i> and H3-specific code in <i>sun8i_h3_ccu.c</i>.
</p>
<p>
Now we have a CCU driver, we can attach a <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/arm/sunxi/sunxi_com.c">com(4)</a> and have a valid console device.
</p>
<p>
After this, it's a matter of writing drivers and/or adapting existing code to attach to <i>fdtbus</i> based on the bindings used in the DTB. For cases where we had a compatible driver in the <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/arm/allwinner/">old Allwinner port</a>, I opted to make a copy of the code and FDT-ize it. A few reasons for this -- 1) the old drivers have CCU-specific code with per-SoC ifdefs scattered throughout, 2) I didn't want to break existing kernels, and 3) long term goal is to move the SoCs supported by the old code over to the new code (this process has already started with the Allwinner A31 port).
</p>
<p>
So what do we get out of this? This is a step towards being able to ship a <i>GENERIC</i> evbarm kernel. I developed the H3 port on two boards, the <a href="http://www.friendlyarm.com/index.php?route=product/product&product_id=132">NanoPi NEO</a> and <a href="http://www.orangepi.org/orangepiplus2e/">Orange Pi Plus 2E</a>, but since then users on <a href="http://mail-index.netbsd.org/port-arm/">port-arm@</a> have been reporting success on many other H3 boards, all from a single kernel config. In addition, I've added support for other Allwinner SoCs (sun8i-a83t, sun6i-a31) to the kernel and have tested booting the same kernel across all 3 SoCs.
</p>
<p>
Orange Pi Plus 2E boot log is below.
</p>
<pre>
U-Boot SPL 2017.05 (Jul 01 2017 - 17:11:09)
DRAM: 2048 MiB
Trying to boot from MMC1
U-Boot 2017.05 (Jul 01 2017 - 17:11:09 -0300) Allwinner Technology
CPU: Allwinner H3 (SUN8I 1680)
Model: Xunlong Orange Pi Plus 2E
I2C: ready
DRAM: 2 GiB
MMC: SUNXI SD/MMC: 0, SUNXI SD/MMC: 1
In: serial
Out: serial
Err: serial
Net: phy interface7
eth0: ethernet@1c30000
starting USB...
USB0: USB EHCI 1.00
USB1: USB OHCI 1.0
USB2: USB EHCI 1.00
USB3: USB OHCI 1.0
USB4: USB EHCI 1.00
USB5: USB OHCI 1.0
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 2 for devices... 1 USB Device(s) found
scanning bus 4 for devices... 1 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
reading netbsd.ub
6600212 bytes read in 334 ms (18.8 MiB/s)
reading sun8i-h3-orangepi-plus2e.dtb
16775 bytes read in 49 ms (334 KiB/s)
## Booting kernel from Legacy Image at 42000000 ...
Image Name: NetBSD/sunxi 8.99.1
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 6600148 Bytes = 6.3 MiB
Load Address: 40008000
Entry Point: 40008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 43000000
Booting using the fdt blob at 0x43000000
Loading Kernel Image ... OK
Loading Device Tree to 49ff8000, end 49fff186 ... OK
Starting kernel ...
[ Kernel symbol table missing! ]
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 8.99.1 (SUNXI) #304: Sat Jul 8 11:01:22 ADT 2017
jmcneill@undine.invisible.ca:/usr/home/jmcneill/netbsd/cvs-src/sys/arch/evbarm/compile/obj/SUNXI
total memory = 2048 MB
avail memory = 2020 MB
sysctl_createv: sysctl_create(machine_arch) returned 17
armfdt0 (root)
fdt0 at armfdt0: Xunlong Orange Pi Plus 2E
fdt1 at fdt0
fdt2 at fdt0
cpus0 at fdt0
cpu0 at cpus0: Cortex-A7 r0p5 (Cortex V7A core)
cpu0: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu0: 32KB/32B 2-way L1 VIPT Instruction cache
cpu0: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu0: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp0 at cpu0: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu1 at cpus0
cpu2 at cpus0
cpu3 at cpus0
gic0 at fdt1: GIC
armgic0 at gic0: Generic Interrupt Controller, 160 sources (150 valid)
armgic0: 16 Priorities, 128 SPIs, 7 PPIs, 15 SGIs
fclock0 at fdt2: 24000000 Hz fixed clock
ffclock0 at fdt2: x1 /1 fixed-factor clock
fclock1 at fdt2: 32768 Hz fixed clock
sunxigates0 at fdt2
sunxiresets0 at fdt1
gtmr0 at fdt0: Generic Timer
armgtmr0 at gtmr0: ARMv7 Generic 64-bit Timer (24000 kHz)
armgtmr0: interrupting on irq 27
sunxigpio0 at fdt1: PIO
gpio0 at sunxigpio0: 94 pins
sunxigpio1 at fdt1: PIO
gpio1 at sunxigpio1: 12 pins
sun8ih3ccu0 at fdt1: H3 CCU
fregulator0 at fdt0: vcc3v3
fregulator1 at fdt0: gmac-3v3
fregulator2 at fdt0: vcc3v0
fregulator3 at fdt0: vcc5v0
sunxiusbphy0 at fdt1: USB PHY
/soc/dma-controller@01c02000 at fdt1 not configured
/soc/codec-analog@01f015c0 at fdt1 not configured
/clocks/ir_clk@01f01454 at fdt2 not configured
sunxiemac0 at fdt1: EMAC
sunxiemac0: interrupting on GIC irq 114
rgephy0 at sunxiemac0 phy 0: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
rgephy1 at sunxiemac0 phy 1: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
psci0 at fdt0: PSCI 0.1
gpioleds0 at fdt0: orangepi:green:pwr orangepi:red:status
gpiokeys0 at fdt0: sw4
sunximmc0 at fdt1: SD/MMC controller
sunximmc0: interrupting on GIC irq 92
sunximmc1 at fdt1: SD/MMC controller
sunximmc1: interrupting on GIC irq 93
sunximmc2 at fdt1: SD/MMC controller
sunximmc2: interrupting on GIC irq 94
ehci0 at fdt1: EHCI
ehci0: interrupting on GIC irq 106
ehci0: 1 companion controller, 1 port
usb0 at ehci0: USB revision 2.0
ohci0 at fdt1: OHCI
ohci0: interrupting on GIC irq 107
ohci0: OHCI version 1.0
usb1 at ohci0: USB revision 1.0
ehci1 at fdt1: EHCI
ehci1: interrupting on GIC irq 108
ehci1: 1 companion controller, 1 port
usb2 at ehci1: USB revision 2.0
ohci1 at fdt1: OHCI
ohci1: interrupting on GIC irq 109
ohci1: OHCI version 1.0
usb3 at ohci1: USB revision 1.0
ehci2 at fdt1: EHCI
ehci2: interrupting on GIC irq 110
ehci2: 1 companion controller, 1 port
usb4 at ehci2: USB revision 2.0
ohci2 at fdt1: OHCI
ohci2: interrupting on GIC irq 111
ohci2: OHCI version 1.0
usb5 at ohci2: USB revision 1.0
/soc/timer@01c20c00 at fdt1 not configured
/soc/watchdog@01c20ca0 at fdt1 not configured
/soc/codec@01c22c00 at fdt1 not configured
com0 at fdt1: ns16550a, working fifo
com0: console
com0: interrupting on GIC irq 32
sunxirtc0 at fdt1: RTC
/soc/ir@01f02000 at fdt1 not configured
cpu3: Cortex-A7 r0p5 (Cortex V7A core)
cpu3: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu3: 32KB/32B 2-way L1 VIPT Instruction cache
cpu3: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu3: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp3 at cpu3: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu1: Cortex-A7 r0p5 (Cortex V7A core)
cpu1: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu1: 32KB/32B 2-way L1 VIPT Instruction cache
cpu1: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu1: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp1 at cpu1: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu2: Cortex-A7 r0p5 (Cortex V7A core)
cpu2: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu2: 32KB/32B 2-way L1 VIPT Instruction cache
cpu2: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu2: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp2 at cpu2: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
sdmmc0 at sunximmc0
sdmmc1 at sunximmc1
sdmmc2 at sunximmc2
uhub0 at usb0: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1 at usb2: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub2 at usb3: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub3 at usb1: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4 at usb4: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub5 at usb5: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
ld2 at sdmmc2: <0x15:0x0100:AWPD3R:0x00:0xec19649f:0x000>
sdmmc0: SD card status: 4-bit, C10, U1, V10
ld0 at sdmmc0: <0x27:0x5048:2&DRP:0x07:0x01c828bc:0x109>
ld2: 14910 MB, 7573 cyl, 64 head, 63 sec, 512 bytes/sect x 30535680 sectors
ld0: 15288 MB, 7765 cyl, 64 head, 63 sec, 512 bytes/sect x 31309824 sectors
(manufacturer 0x24c, product 0xf179, standard function interface code 0x7)at sdmmc1 function 1 not configured
ld2: mbr partition exceeds disk size
ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz
ld2: 8-bit width, 52.000 MHz
urtwn0 at uhub0 port 1
urtwn0: Realtek (0xbda) 802.11n NIC (0x8179), rev 2.00/0.00, addr 2
urtwn0: MAC/BB RTL8188EU, RF 6052 1T1R, address e8:de:27:16:0c:81
urtwn0: 1 rx pipe, 2 tx pipes
urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
kern.module.path=/stand/evbarm/8.99.1/modules
WARNING: clock lost 6398 days
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
Sat Jul 8 11:05:42 ADT 2017
Starting root file system check:
/dev/rld0a: file system is clean; not checking
Not resizing /: already correct size
swapctl: adding /dev/ld0b as swap device at priority 0
Starting file system checks:
/dev/rld0e: 22 files, 32340 free (8085 clusters)
random_seed: /var/db/entropy-file: Not present
Setting tty flags.
Setting sysctl variables:
ddb.onpanic: 1 -> 1
Starting network.
Hostname: sunxi
IPv6 mode: host
Configuring network interfaces:.
Adding interface aliases:.
Waiting for DAD to complete for statically configured addresses...
Starting dhcpcd.
Starting mdnsd.
Building databases: dev, utmp, utmpx.
Starting syslogd.
Mounting all file systems...
Clearing temporary files.
Updating fontconfig cache: done.
Creating a.out runtime link editor directory cache.
Checking quotas: done.
Setting securelevel: kern.securelevel: 0 -> 1
Starting virecover.
Checking for core dump...
savecore: no core dump
Starting devpubd.
Starting local daemons:.
Updating motd.
Starting ntpd.
Jul 8 11:05:58 sunxi ntpd[595]: ntp_rlimit: Cannot set RLIMIT_STACK: Invalid argument
Starting sshd.
Starting inetd.
Starting cron.
Sat Jul 8 11:06:02 ADT 2017
NetBSD/evbarm (sunxi) (console)
login:
</pre>https://blog.netbsd.org/tnf/entry/pkgsrccon_2017_reportpkgsrcCon 2017 reportSevan Janiyan2017-07-08T23:51:08+00:002017-07-09T02:22:57+00:00
<p>This years pkgsrcCon returned to London once again. It was last held in London back in 2014. The 2014 con was the first pkgsrcCon I attended, I had been working on Darwin/PowerPC fixes for some months and presented on the progress I'd made with a 12" G4 PowerBook. I took away a G4 Mac Mini that day to help spare the PowerBook for use and <a href="https://www.geeklan.co.uk/?p=1597">dedicate a machine for build and testing</a>. The offer of PowerPC hardware donations was repeated at this years con, thanks to jperkin@ who showed up with a backpack full of Mac Minis (more on that later).</p>
<p>Since 2014 we have held cons in <a href="https://pkgsrc.org/pkgsrcCon/2015/">Berlin (2015)</a> & <a href="https://pkgsrc.org/pkgsrcCon/2016/">Krakow (2016)</a>. In Krakow we had talks about a wide range of projects over 2 days, from Haiku Ports to Common Lisp to midipix (building native PE binaries for Windows) and back to the BSDs. I was very pleased to continue the theme of a diverse program this year.</p>
<p>Aside from pkgsrc and NetBSD, we had talks about FreeBSD, OpenBSD, Slackware Linux, and Plan 9.<br/>
Things began with a pub gathering on the Friday for the pre-con social, we hung out and chatted till almost midnight on a wide range of topics, such as supporting a system using NFS on MS-DOS, the origins of pdksh, corporate IT, culture and many other topics.</p>
<p>On parting I was asked about the starting time on Saturday as there was some conflicting information. I learnt that the registration email had stated a later start than I had scheduled for & advertised on the website, by 30 minutes.<br/>
Lesson learnt: register for your own event!<br/>
Not a problem, I still needed to setup a webpage for the live video stream, I could do both when I got back. With some trimming here and there I had a new schedule, I posted that to the pkgsrcCon website and moved to trying to setup a basic web page which contained a snippet of javascript to play a live video stream from Scale Engine.<br/>
2+ hours later, it was pointed out that the <acronym title="Cross-Site Scripting">XSS</acronym> protection headers on pkgsrc.org breaks the functionality. Thanks to jmcneill@ for debugging and providing a working page.</p>
<p>Saturday started off with Giovanni Bechis speaking about pledge in OpenBSD and adding support to various packages in their ports tree, <a href="http://www.netbsd.org/people/developers.html#alnsn">alnsn@</a> then spoke about installing packages from a repo hosted on the Tor network.</p>
<p>After a quick coffee break we were back to hear <a href="http://www.vitanuova.com/index.html">Charles Forsyth</a> speak about how Plan 9 and Inferno dealt with portability, building software and the problem which are avoided by the environment there. This was followed by a very energetic rant by David Spencer from the Slackbuilds project on packaging 3rd party software. Slackbuilds is a packaging system for Slackware Linux, which was inspired by FreeBSD ports.</p>
<p>For the first slot after lunch, <a href="http://www.netbsd.org/people/developers.html#agc">agc@</a> gave a talk on the early history of pkgsrc followed by <a href="https://twitter.com/docscream">Thomas Merkel</a> on using vagrant to test pkgsrc changes with ease, locally, using vagrant. <a href="http://www.netbsd.org/people/developers.html#khorben">khorben@</a> covered his work on adding security to pkgsrc and <a href="http://www.netbsd.org/people/developers.html#bsiegert">bsiegert@</a> covered the benefits of performing our bulk builds in the cloud and the challenges we currently face.<br/>
My talk was about some topics and ideas which had inspired me or caught my attention, and how it could maybe apply to my work.The title of the talk was taken from the name of Andrew Weatherall's Saint Etienne remix, possibly referring to two different styles of track (dub & vocal) merged into one or something else. I meant it in terms of applicability of thoughts and ideas. After me, agc@ gave a second talk on the evolution of the Netflix Open Connect appliance which runs FreeBSD and Vsevolod Stakhov wrapped up the day with a talk about the technical implementation details of the successor to pkg_tools in FreeBSD, called pkg, and how it could be of benefit for pkgsrc.</p>
<p><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Netflix confirms it: BSD is not dying <a href="https://twitter.com/hashtag/pkgsrcCon?src=hash">#pkgsrcCon</a></p>&mdash; Benny Siegert (@bentsukun) <a href="https://twitter.com/bentsukun/status/881188183257620481">July 1, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script></p>
<p>For day 2 we gathered for a hack day at the London Hack Space.<br/>
I had burn't some some CD of the most recent macppc builds of NetBSD 8.0_BETA and -current to install and upgrade Mac Minis. I setup the donated G4 minis for everyone in a <a href="http://mail-index.netbsd.org/port-macppc/2017/07/07/msg002425.html">dual-boot configuration</a> and moved on to taking apart my MacBook Air to inspect the wifi adapter as I wanted to replace it with something which works on FreeBSD. It was not clear from the ifixit teardown photos of cards size, it seemed like a normal mini-PCIe card but it turned out to be far smaller. Thomas had also had the same card in his and <a href="https://garbage.fm/episodes/40">we are not alone</a>. Thomas has started putting together a driver for the Broadcom card, the project is still in its early days and lacks support for encrypted networks but hopefully it will appear on <a href="https://review.freebsd.org">review.freebsd.org</a> in the future.<br/>
weidi@ worked on fixing SunOS bugs in various packages and later in the night we setup a NetBSD/macppc bulk build environment together on his Mac Mini.<br/>
Thomas setup an <a href="https://opengrok.github.io/OpenGrok/">OpenGrock</a> <a href="https://grok.pkgsrc.pub/source/">instance</a> to index the source code of all the software available for packaging in pkgsrc. This helps make the evaluation of changes easier and the scope of impact a little quicker without having to run through a potentially lengthy bulk build with a change in mind to realise the impact.<br/>
bsiegert@ cleared his ticket and email backlog for pkgsrc and alnsn@ got NetBSD/evbmips64-eb booting on his EdgeRouter Lite.</p>
<p><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/pkgsrcCon?src=hash">#pkgsrcCon</a> hackathon work by <a href="https://twitter.com/docscream">@docscream</a>: <a href="https://t.co/3GY3TRtdNV">https://t.co/3GY3TRtdNV</a> code search over everything (still indexing)</p>&mdash; Wiedi (@wied0r) <a href="https://twitter.com/wied0r/status/881568372324003842">July 2, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script></p>
<p>On Monday we reconvened at the Hack Space again and worked some more. I started putting together the talks page with the details from Saturday and the the slides which I had received, in preperation for the videos which would come later in the week. By 3pm pkgsrcCon was over. I was pretty exhausted but really pleased to have had a few days of techie fun.</p>
<p>Many thanks to <a href="http://www.netbsd.org/foundation/">The NetBSD Foundation</a> for purchasing a camera to use for streaming the event and a speedy response all round by the board. The <a href="http://ossg.bcs.org/">Open Source Specialist Group at BCS, The Chartered Institute for IT</a> and the <a href="https://london.hackspace.org.uk/">London Hack Space</a> for hosting us. <a href="https://www.scaleengine.com/">Scale Engine</a> for providing streaming facility. <a href="http://www.netbsd.org/people/developers.html#wiedi">weidi@</a> for hosting the recorded videos.<br/>
<a href="https://twitter.com/allanjude">Allan Jude</a> for pointers, Jared McNeill for debugging, <a href="http://www.nycbug.org">NYCBUG</a> and <a href="https://twitter.com/bsdtv">Patrick McEvoy</a> for tips on streaming, the attendees and speakers. This year we had speakers from USA, Italy, Germany and London E2.<br/>
Looking forward to pkgsrcCon 2018!</p>
<p>The videos and slides are available <a href="http://www.pkgsrc.org/pkgsrcCon/2017/talks.html">here</a> and the <a href="http://archive.org/details/pkgsrcCon-2017">Internet Archive</a>.</p>
https://blog.netbsd.org/tnf/entry/llvm_asan_and_ubsan_onLLVM asan and ubsan on NetBSDKamil Rytarowski2017-07-03T02:05:12+00:002017-07-04T16:48:53+00:00Over the last 30 days I was focusing on getting the environment to enable LLVM sanitizers and the Clang compiler on NetBSD.
Meanwhile I pushed forward generic parts that were needing enhancements around pkgsrc and LLVM in general to ease the future LLDB work.Over the last 30 days I was focusing on getting the environment to enable LLVM sanitizers and the Clang compiler on NetBSD.
Meanwhile I pushed forward generic parts that were needing enhancements around pkgsrc and LLVM in general to ease the future LLDB work.
<p>
<h1>dogfood</h1>
<p>
When I have realized that in order to work on the LLVM sanitizers I need to use Clang as the compiler.
A part of the compiler-rt (lowlevel LLVM compiler runtime library) has code specifically incompatible with GCC.
It was mainly related to intrinsic instructions for atomic functions.
I tried to research how much work is needed to port it to GCC.
It happened to be non-trivial and I filed a bug on the LLVM bugzilla.
<p>
These circumstances made me to switch to Clang as the pkgsrc toolchain.
I was using it to test the compilation of small bulks of packages and record build and compiler problems.
To save time, I used ccache as my cache for builds.
<p>
My options in mk.conf(5):
<pre>
PKGSRC_COMPILER= ccache clang
CCACHE_BASE= /usr/local
CCACHE_DIR= /public/ccache_tmp
CCACHE_LOGFILE= /tmp/ccache.txt
PKG_CC= clang
PKG_CXX= clang++
CLANGBASE= /usr/local
HAVE_LLVM= yes
</pre>
<p>
It's worth noting that ccache with pkgsrc won't check $HOME/.ccache for configuration, it must be placed in $CCACHE_DIR/ccache.conf.
<p>
The documented problems can be summarized as:
<ul>
<li>Broken ccache in pkgsrc for C++11 packages.<br>
The pkgsrc framework started supporting C++ languages in the USE_LANGUAGES definition.
Packages with newly added USE_LANGUAGES values (such as c++11) were not compiled with ccache because ccache.mk was not yet aware of such values.
This broke support of these packages to set these values to be built with ccache.
I've corrected it and introduced a new option CCACHE_LOGFILE to more easily track execution of ccache and detect errors.</li>
<li>Broken ccache in pkgsrc for a custom toolchain.<br>
ccache tries finding a real-compiler looking for it in $PATH.
When I have built clang within pkgsrc to work on it (installed into <code>/usr/pkg</code>), and I had my main toolchain in <code>/usr/local</code>
it was picking the one from <code>/usr/pkg</code> for new builds and it resulted in cache-misses for new builds.
I have installed a fix for it to pass ccache specific PATH to always find the appropriate compiler.
</li>
<li>Header &lt;execinfo.h&gt; cannot be included on Clang 5.0.0svn.<br>
For some reason compilers tend to install their own headers that overshadow the system headers.
This resulted in build failures in programs including plain &lt;execinfo.h&gt; header (for the backtrace(3) function).
This system header used to include &lt;stddef.h&gt; that included our &lt;sys/cdefs.h&gt;... with shadowed &lt;stddef.h&gt;
by Clang 5.0.0svn (from $PREFIX/lib/clang/5.0.0/include/stddef.h).
Christos Zoulas fixed it by making &lt;execinfo.h&gt; standalone and independent from standard libc headers.
</li>
<li>__float128 and GNU libstdc++.<br>
Our basesystem GNU libstdc++ enables __float128 on i386, amd64 and i64 ports.
As of now the LLVM equivalent library contains partial support for this type.
This results in a problem that affects 3rd party programs in the setup of Clang + libstdc++ detect __float128 support and break because the compiler does not define appropriate global define __FLOAT128__.
This issue is still open for discussion on how to solve it for NetBSD.
</li>
<li>gforth optimization problems.<br>
Upstream gforth developers ported this FORTH compiler to Clang, and triggered an optimization issue with attempting to needlessly solve a complex internal problem.
This results with compilation times of several minutes on a modern CPUs instead of getting the results immediately.
The problem has been already reported on the LLVM bugzilla and I have filed a report that it is still valid.
</li>
<li>bochs can be built with clang.<br>
A while ago, bochs was buildable only by the GCC compilers and the Clang toolchain was blacklisted.
I have verified that this is no longer the case and unmasked the package for compilers other than GCC.</li>
</ul>
<p>
<p>
<h1>LLVM and Clang testsuites</h1>
<p>
I have prepared Clang and LLVM testsuites to execute on NetBSD.
Correctness of both projects is crucial for LLDB and the LLVM sanitizers to work because their issues resound problems inside programs that depend on them.
Originally I have corrected the tests with local patches to build with GCC, and switched later to Clang.
I have restructured the packages in pkgsrc-wip in order to execute the test-suite.
I have fixed 20 test failures in LLVM implementing AllocateRWX and ReleaseRWX for the NetBSD flavor of PaX MPROTECT.
There are still over 200 failures to solve!
<p>
It's worth noting that the googletest library (used in a modified version in LLVM and in a regular one in Clang) finally accepted the NetBSD patches.
<p>
<h1>LLVM asan and ubsan</h1>
<p>
I expect to get four LLVM sanitizers working in order to move on to LLDB:
asan (address sanitizer), ubsan (undefined behavior sanitizer), tsan (thread sanitizer), msan (memory sanitizer).
The other ones like dfsan (data-flow sanitizer) or lsan (leak sanitizer) are currently to be skipped.
In general, sanitizers are part of the LLDB functionality that I want to get aboard on NetBSD, as there are plugins to integrate them within the debugger.
In the current state I require them to debug bugs inside LLDB/NetBSD.
<p>
The original work on sanitizers in GCC (with libsanitizer) has been done by Christos Zoulas.
GCC libsanitizer is a close sibling of compiler-rt/lib from the LLVM project.
I picked up his work and integrated it into compiler-rt and developed the rest (code differences, fixing bugs, Clang/LLVM specific parts in llvm/ and clang/) and I managed to get
asan and ubsan to work.
<p>
Users should pickup pkgsrc-wip in revision 3e7c52b97b4d6cb8ea69a081409ac818c812c34a and install wip/{llvm,clang,compiler-rt}-netbsd.
Clang will be ready for usage:
<pre>
/usr/pkg/bin/clang -fsanitize=undefined test.c
</pre>
and
<pre>
/usr/pkg/bin/clang -fsanitize=address test.c
</pre>
<p>
Additional compiler commands that may improve the experience:
<pre>
-g -O0 -fno-omit-frame-pointer -pie -fPIE
</pre>
<p>
These sanitizers are not production ready and are active under development.
<p>
<h1>Plan for the next milestone</h1>
<p>
Roadmap for the next month:
<ul>
<li>Finish and send upstream LLVM asan and ubsan support.</li>
<li>Correct more problems triggered by LLVM and Clang test-suites.</li>
<li>Resume msan and tsan porting.</li>
</ul>
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/pkgsrccon_2016_reportpkgsrcCon 2016 reportKamil Rytarowski2017-07-01T16:44:26+00:002017-07-01T20:18:06+00:00pkgsrcCon is the annual technical conference for people working on pkgsrc, a framework for building over 17,000 open source software packages. pkgsrc is the native package manager on NetBSD, SmartOS and Minix, and is portable across many different operating systems including Linux and Mac OS X. pkgsrcCon is the annual technical conference for people working on pkgsrc, a framework for building over 17,000 open source software packages. pkgsrc is the native package manager on NetBSD, SmartOS and Minix, and is portable across many different operating systems including Linux and Mac OS X.
<p>
The last year's pkgsrcCon 2016 event took place in Kraków, Poland.
<p>
Slides are available on the <a href="https://pkgsrc.org/pkgsrcCon/2016/">event site</a>.
<p>
Video recordings are stored at <a href="https://archive.org/details/pkgsrcCon-2016">https://archive.org/details/pkgsrcCon-2016</a>.
<p>
We would like to thank the organizers, sponsors and promotion from the Jagiellonian University, The NetBSD Foundation, Programista Magazyn, OS World, and Subcarpathian BSD User Group.https://blog.netbsd.org/tnf/entry/in_memoriam_nicolas_jolyIn Memoriam Nicolas JolyKamil Rytarowski2017-06-20T16:26:29+00:002017-06-20T16:26:29+00:00Nicolas Joly passed away on 2017-06-07.Nicolas Joly passed away on 2017-06-07.
<p>
Born in 1969, he developed a passion for computer science in his youth
but chose to study biology. He then attended a bioinformatics training
at the Pasteur Institute in 1997 at the end of which he got hired in
the scientific IT department. He has continuously worked there since.
Not only was he skilled in biology but also in system administration,
databases, and programming.
<p>
He was introduced to NetBSD in 1999 and quickly became a dedicated
user and hacker. He became a NetBSD developer in 2007 for working on
Linux/i386 (32 bits) emulation on NetBSD/amd64 (64 bits).
<p>
Nicolas leaves a wife and three children.
<p>
<i>Memorial note submitted by Marc Baudoin.</i>https://blog.netbsd.org/tnf/entry/new_home_for_the_repositoryNew home for the repository conversionKamil Rytarowski2017-06-10T20:31:43+00:002017-06-10T20:42:16+00:00Hello all,
<p>
the repository conversion setup for NetBSD CVS -> Fossil -> Git
has found a new home. Ironically, on former cvs.NetBSD.org hardware.
This provides a somewhat faster conversion cycle as well as removing
anoncvs.NetBSD.org from the process. This should avoid occasional
problems with incomplete syncs. Two other changes have been applied at
the same time:Hello all,
<p>
the repository conversion setup for NetBSD CVS -> Fossil -> Git
has found a new home. Ironically, on former cvs.NetBSD.org hardware.
This provides a somewhat faster conversion cycle as well as removing
anoncvs.NetBSD.org from the process. This should avoid occasional
problems with incomplete syncs. Two other changes have been applied at
the same time:
<p>
<ul>
<li>
The Fossil repositories have moved to using the SHA512 checksums
internally. To avoid accidents, a new project code is used. This
requires Fossil 2.x.
<li>
The Git repositories map user names to @NetBSD.org addresses
(src, xsrc) or @pkgsrc.org addresses (pkgsrc). This allows
consolidation on github with user accounts, assuming you have the
corresponding addresses as primary or secondary address.
</ul>
<p>
The new locations for the repositories are:
<p>
<table>
<tr>
<td>CVS</td>
<td>Fossil</td>
<td>Git</td>
</tr>
<tr>
<td>src</td>
<td><a href="https://src.fossil.NetBSD.org">https://src.fossil.NetBSD.org</a></td>
<td><a href="https://github.com/NetBSD/src">https://github.com/NetBSD/src</a></td>
</tr>
<tr>
<td>pkgsrc</td>
<td><a href="https://pkgsrc.fossil.NetBSD.org">https://pkgsrc.fossil.NetBSD.org</a></td>
<td><a href="https://github.com/NetBSD/pkgsrc">https://github.com/NetBSD/pkgsrc</a></td>
</tr>
<tr>
<td>xsrc</td>
<td><a href="https://xsrc.fossil.NetBSD.org">https://xsrc.fossil.NetBSD.org</a></td>
<td><a href="https://github.com/NetBSD/xsrc">https://github.com/NetBSD/xsrc</a></td>
</tr>
</table>
<p>
The old conversions will be provided for the near future, but likely
stop toward the end of the month.
<p>
A special thanks goes to Petra and Christos from the admin team for the
assistance with the machine setup and low-level CVS clean-ups.
<p>
Joerg
<p>
-- source <a href="https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html">https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html</a>https://blog.netbsd.org/tnf/entry/lldb_sanitizing_the_debugger_sLLDB: Sanitizing the debugger's runtimeKamil Rytarowski2017-06-06T15:18:20+00:002017-06-06T16:32:58+00:00This month I started to work on correcting of the ptrace(2) layer,
as test suites used to trigger failures on the kernel side.
This finally ended up sanitizing the LLDB runtime as well, addressing
LLDB and NetBSD userland bugs.This month I started to work on correcting of the ptrace(2) layer,
as test suites used to trigger failures on the kernel side.
This finally ended up sanitizing the LLDB runtime as well, addressing
LLDB and NetBSD userland bugs.
<p>
It turned out that more bugs were unveiled and this is not the final report on LLDB.
<p>
<h1>The good</h1>
<p>
Besides the greater enhancements this month I performed a cleanup in the ATF <b>ptrace</b>(2) tests again.
Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin.
<p>
It is worth noting that LLVM can run tests on NetBSD again,
the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository.
There was also an improvement in <b>ftruncate</b>(2) on the LLVM side (authored by Joerg).
<p>
Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD.
I submitted a patch to stop crashing it on startup anymore.
It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future.
Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment.
This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time.
As of now LLDB linking can take minutes on a modern amd64 machine designed for performance.
<p>
<h2>Kernel correctness</h2>
<p>
I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin.
This code revealed - when running the LLDB regression test-suite - new kernel bugs.
This unfortunately affects the usability of a debugger in a multithread environment in general and explains
why GDB was never doing its job properly in such circumstances.
<p>
One of the first errors was asserting kernel panic with <b>PT_*STEP</b>, when a debuggee has more than a single thread.
I have narrowed it down to lock primitives misuse in the <b>do_ptrace</b>() kernel code.
The fix has been committed.
<p>
<h2>LLDB and userland correctness</h2>
<p>
LLDB introduced support for <b>kevent</b>(2) and it contains the following function:
<p>
<pre>
Status MainLoop::RunImpl::Poll() {
in_events.resize(loop.m_read_fds.size());
unsigned i = 0;
for (auto &fd : loop.m_read_fds)
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
num_events = kevent(loop.m_kqueue, in_events.data(), in_events.size(),
out_events, llvm::array_lengthof(out_events), nullptr);
if (num_events < 0)
return Status("kevent() failed with error %d\n", num_events);
return Status();
}
</pre>
<p>
It works on FreeBSD and MacOSX, however it broke on NetBSD.
<p>
Culprit line:
<p>
<pre>
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
</pre>
<p>
FreeBSD defined <b>EV_SET</b>() as a macro this way:
<p>
<pre>
#define EV_SET(kevp_, a, b, c, d, e, f) do { \
struct kevent *kevp = (kevp_); \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while(0)
</pre>
<p>
NetBSD version was different:
<p>
<pre>
#define EV_SET(kevp, a, b, c, d, e, f) \
do { \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while (/* CONSTCOND */ 0)
</pre>
This resulted in heap damage, as <b>keyp</b> was incremented every time a value was
assigned to <b>(keyp)-&gt;</b>.
<p>
Without GCC asan and ubsan tools, finding this bug would be much more time consuming,
as the random memory corruption was affecting unrelated lambda function in a different part of the code.
<p>
To use the GCC sanitizers with packages from pkgsrc, on NetBSD-current, one has to use one or both of these lines:
<p>
<pre>
_WRAP_EXTRA_ARGS.CXX+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
CWRAPPERS_APPEND.cxx+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
</pre>
<p>
While there, I have fixed another - generic - bug in the LLVM headers.
The <b>class Triple</b> constructor hadn't initialized the <b>SubArch</b> field,
which upsetting the GCC address sanitizer.
It was triggered in LLDB in the following code:
<pre>
void ArchSpec::Clear() {
m_triple = llvm::Triple();
m_core = kCore_invalid;
m_byte_order = eByteOrderInvalid;
m_distribution_id.Clear();
m_flags = 0;
}
</pre>
<p>
I have filed a patch for review to address this.
<p>
<h1>The bad</h1>
<p>
Unfortunately this is not the full story and there is further mandatory work.
<p>
<h2>LLDB acceleration</h2>
<p>
The <b>EV_SET</b>() bug broke upstream LLDB over a month ago, and during this period the debugger
was significantly accelerated and parallelized.
It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke
due to threading desynchronization.
LLDB behaves differently when run standalone, under <b>ktruss</b>(1) and
under <b>gdb</b>(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug.
<p>
<h1>The ugly</h1>
<p>
There are also unpleasant issues at the core of the Operating System.
<p>
<h2>Kernel troubles</h2>
<p>
Another bug with single-step functions
that affects another aspect of correctness - this time with reliable execution of a program -
is that processes die in non-deterministic ways when single-stepped.
My current impression is that there is no appropriate translation between process and thread (LWP) states
under a debugger.
<p>
These issues are sibling problems to unreliable <b>PT_RESUME</b> and <b>PT_SUSPEND</b>.
<p>
In order to be able to appropriately address this,
I have diligently studied this month the <i>Solaris Internals</i> book to get a better image of
the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX.
<p>
<h1>Plan for the next milestone</h1>
<p>
The current troubles can be summarized as data races in the kernel and at the same time in LLDB.
I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan).
Temporarily I have removed the code for tracing processes with multiple threads to
hide the known kernel bugs and focus on the LLDB races.
<p>
Unfortunately LLDB is not easily bisectable
(build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed
on the most recent code from upstream trunk.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>https://blog.netbsd.org/tnf/entry/netbsd_8_0_release_processNetBSD 8.0 release process underwaysnj2017-06-06T08:31:15+00:002017-06-06T08:31:15+00:00<p>If you've been reading source-changes@, you likely noticed the recent creation of the netbsd-8 branch. If you haven't been reading source-changes@, here's some news: the netbsd-8 branch has been created, signaling the beginning of the release process for NetBSD 8.0.</p>
<p>We don't have a strict timeline for the 8.0 release, but things are looking pretty good at the moment, and we expect this release to happen in a shorter amount of time than the last couple major releases did.</p>
<p>At this point, we would love for folks to test out netbsd-8 and let us know how it goes. A couple of major improvements since 7.0 are the addition of USB 3 support and an overhaul of the audio subsystem, including an in-kernel mixer. Feedback about these areas is particularly desired.</p>
<p>To download the latest binaries built from the netbsd-8 branch, head to <a href="http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/">http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/</a></p>
<p>Thanks in advance for helping make NetBSD 8.0 a stellar release!</p>https://blog.netbsd.org/tnf/entry/netbsd_maintainer_in_the_qemuNetBSD maintainer in the QEMU projectKamil Rytarowski2017-05-17T01:06:38+00:002017-05-17T01:16:17+00:00<b>QEMU</b> - the FAST! processor emulator - is a generic, Open Source, machine emulator and virtualizer. It defines state of the art in modern virtualization.
<p>
This software has been developed for multiplatform environments with support for NetBSD since virtually forever. It's the primary tool used by the NetBSD developers and release engineering team. It is run with continuous integration tests for daily commits and execute regression tests through the Automatic Test Framework (ATF).<b>QEMU</b> - the FAST! processor emulator - is a generic, Open Source, machine emulator and virtualizer. It defines state of the art in modern virtualization.
<p>
This software has been developed for multiplatform environments with support for NetBSD since virtually forever. It's the primary tool used by the NetBSD developers and release engineering team. It is run with continuous integration tests for daily commits and execute regression tests through the Automatic Test Framework (ATF).
<p>
Since the projects keep researching and developing support for various modern trends in computing, the gap between the QEMU featureset in NetBSD and Linux diverged due to lack of active NetBSD maintenance resulted in breaking the default build.
<p>
The QEMU developers <a href="http://wiki.qemu.org/ChangeLog/2.9#Warning_of_unsupported_host_systems">warned</a> the Open Source community - with version 2.9 of the emulator - that they will eventually drop support for suboptimally supported hosts if nobody will step in and take the maintainership to refresh the support. This warning was directed to major BSDs, Solaris, AIX and Haiku.
<p>
Thankfully the NetBSD position has been filled - making NetBSD to <a href="https://github.com/qemu/qemu/commit/3c2bdbc1e402fa88b80fcbba7f45cd778443e5c2">restore official maintenance</a>.
<p>
The current roadmap in QEMU/NetBSD is as follows:
<ul>
<li>address all build failures [all patches sent to review, part of them already merged upstream],
<li>address all build warnings,
<li>restore the QEMU setup to run regression tests on NetBSD.
</ul>
With the goal to move on to the maintenance mode, catching up with regressions, adding NetBSD node in the regression tests cluster and reducing the featureset gap. There are various missing functions on NetBSD, including: resurrecting user-mode emulation, suboptimal kernel aio(3) support, hugepagefs support, hardware assisted virtualization, passthrough PCI and SRIOV.
<p>
This effort is spare time activity - as of now without commercial support - and possible thanks to unloading the developer (myself) from more urgently pending tasks in NetBSD thanks to the <a href="https://blog.netbsd.org/tnf/entry/funded_contract_2016_2017">contract</a> for enhancing debuggers in business hours.https://blog.netbsd.org/tnf/entry/announcing_netbsd_and_the_googleAnnouncing NetBSD and the Google Summer of Code Projects 2017Hubert Feyrer2017-05-05T19:16:02+00:002017-05-05T19:16:02+00:00We are very happy to announce that the selection process in this year's Summer of Code with its bargaining of slots and what student gets assigned to which project is over. As a result, the following students will take on their projects:
<p>
<ul>
<li> Leonardo Taccari will work add multi-packages support to pkgsrc.
<li> Maya Rashish will work on the LFS cleanup.
<li> Utkarsh Anand will make Anita support multiple virtual machine systems and more architectures within them to improve testing coverage.
</ul>
What follows now is a community bonding period until May 30th, followed by a coding period over the summer (it's Summer of Code, after all <img src="https://blog.netbsd.org/images/smileys/smile.gif" class="smiley" alt=":-)" title=":-)" />) until August 21st, evaluations, code submission and an announcement of the results on September 6th 2017.
<p>
Good luck to all our students and their mentors - we look forward to your work results, and welcome you to The NetBSD Project!
<p>
https://blog.netbsd.org/tnf/entry/new_synchornization_mechanism_localcount_9New synchronization mechanism - localcount(9)Paul Goyette2017-05-03T07:20:38+00:002017-05-03T22:35:30+00:00A new localcount(9) reference-counting mechanism will soon be available to provide improved protection against having a device or driver "disappear" while it is being used.
<p>Coming soon we have a new set of kernel synchronization routines - localcount(9) - which provide a medium-weight reference-counting mechanism. From the manual page, "During normal operations, localcounts do not need the interprocessor synchronization associated with atomic_ops(3) atomic memory operations, and (unlike psref(9)) localcount references can be held across sleeps and can migrate between CPUs. Draining a localcount requires more expensive interprocessor synchronization than atomic_ops(3) (similar to psref(9)). And localcount references require eight bytes of memory per object per-CPU, significantly more than atomic_ops(3) and almost always more than psref(9)."</p>
<p>We'll be adding localcount(9) reference counting to the device driver cdevsw and bdevsw structures, to ensure that a (modular) device driver cannot be removed while it is active. Modular drivers with initializers for these structures need to be modified to initialize their localcount members, using the DEVSW_MODULE_INIT macro (this change is mandatory for all loadable drivers). To take advantage of the reference counting, the drivers also need to replace all calls to bdevsw_lookup() and cdevsw_lookup() with bdevsw_lookup_acquire() and cdevsw_lookup_acquire() respectively, and then release the reference using bdevsw_release() and cdevsw_release().</p>
<p>We'll also be using localcount(9) to provide reference-counting of individual device units, to prevent a unit from being destroyed while it is active. To implement device unit reference-counting, all calls to device_lookup(), device_find_by_driver_unit(), and device_lookup_private() need to be replaced by their corresponding *_acquire() variant; when the caller is finished using the device, it must release the reference using device_release().</p>
<p>More details and examples can be seen by examining the prg-localcount2 branch in cvs including the new localcount(9) manual page!</p>
https://blog.netbsd.org/tnf/entry/lldb_netbsd_process_plugin_enhancementsLLDB: NetBSD Process Plugin enhancements Kamil Rytarowski2017-05-02T02:08:44+00:002017-05-02T02:08:44+00:00Last month I have worked on features of the Process Plugin on NetBSD and support for threads in <b>core</b>(5) files.Last month I have worked on features of the Process Plugin on NetBSD and support for threads in <b>core</b>(5) files.
<p>
<h1>What has been done in NetBSD</h1>
<p>
I've managed to achieve the following accomplishments:
<p>
<h2>Introduction of <b>PT_SETSTEP</b> and <b>PT_CLEARSTEP</b></h2>
<p>
This allows to:
<ul>
<li>singlestep particular threads,</li>
<li>combine <b>PT_STEP</b> with <b>PT_SYSCALL</b>,</li>
<li>combine <b>PT_STEP</b> and emission of a signal.</li>
</ul>
<p>
There are equivalent operations in FreeBSD with the same names.
<p>
<h2>Introduction of helper macro <b>PTRACE_BREAKPOINT_ASM</b></h2>
<p>
This code was prepared by Nick Hudson and it was used in ATF tests to verify behavior of software breakpoints.
<p>
<h2>Addition of new <b>sysctl</b>(2) functions</h2>
<p>
Add new defines in <b>sysctl</b>(2) on <b>amd64</b> and <b>i386</b> ports.
These values are defined in <b>&lt;x86/cpu.h&gt;</b>:
<ul>
<li><b>CPU_FPU_SAVE</b> (15)<br>
<pre>
int: FPU Instructions layout
* to use this, CPU_OSFXSR must be true
* 0: FSAVE
* 1: FXSAVE
* 2: XSAVE
* 3: XSAVEOPT
</pre>
</li>
<li><b>CPU_FPU_SAVE_SIZE</b> (16)<br>
<pre>
int: FPU Instruction layout size
</pre>
</li>
<li><b>CPU_XSAVE_FEATURES</b> (17)<br>
<pre>
quad: FPU XSAVE features
</pre>
</li>
<li>Bump <b>CPU_MAXID</b> from 15 to 18.</li>
</ul>
<p>
These values are useful to get <b>FPU</b> (<i>floating point unit</i>) properties in e.g. a debugger.
This information is required to properly implement <b>FPR</b> (<i>floating point register</i>) tracer operations
on <b>x86</b> processors.
<p>
<h2>Corrections in ptrace(2) man-page</h2>
<p>
Few mistakes were corrected to make the documentation more correct.
<p>
<h2>ATF tests cleanup in ptrace(2)</h2>
<p>
There were added new tests for new <b>ptrace</b>(2) operations (<b>PT_SETSTEP</b> and <b>PT_CLEARSTEP</b>).
<p>
Also several tests were updated to reflect the current state of "<i>successfully passed</i>" and "<i>expected failure</i>".
This is important to mark issues that are already known and quickly catch new regressions in future changes.
<p>
<h2>F_GETPATH in fcntl(2)</h2>
<p>
It was decided that NetBSD will not introduce new <b>fcntl</b>(2) function for compatibility with certain other systems.
This means that once LLDB will require this feature, we will need to introduce a workaround in the project.
<p>
<h1>What has been done in LLDB</h1>
<p>
The NetBSD Process Plugin in LLDB acquired new capabilities.
Additionally enhancements in LLDB were developed such as handling threads in <b>core</b>(5) files.
<h2>Floating point support</h2>
<p>
The <b>x86_64</b> architecture supports in default properties <b>FXSAVE</b> processor instructions.
The <b>FXSAVE</b> feature allows to operate over floating point registers.
A thread state (context) is composed of (and not restricted to) general and floating point registers.
<p>
The NetBSD Process Plugin acquired the functionality to read these registers and optionally set new values for them.
<p>
<h2>Watchpoint support</h2>
<p>
A programmer can use hardware assisted watchpoints to stop execution of a tracee whenever a certain variable or instruction
was read/written/executed.
The support for this feature has been implemented on NetBSD with <b>ptrace</b>(2) operations <b>PT_SETDBREGS</b> and <b>PT_GETDBREGS</b>.
These operations are now available in the LLDB Process plugin.
<p>
<h2>Threads support in core(5) files</h2>
<p>
I've included support for LWPs in <b>core</b>(5) files.
This means that larger programs with threads,
like Firefox that emitted coredump for some reason (usually during crash) can be investigated postmortem.
<p>
<h1>Demo</h1>
<p>
I've prepared a <a href="https://netbsd.org/~kamil/lldb/firefox-core.typescript">recording</a> with the script(1) utility from the NetBSD base system. To replay it:
<pre>
script -p ./firefox-core.typescript
</pre>
<p>
This recording shows a debugging session of a Firefox <b>core</b>(5) file.
<p>
(I was kind to prepare a Linux version of the NetBSD script(1) <a href="https://netbsd.org/~kamil/lldb/nbscript.c">here</a>).
<p>
<h1>Plan for the next milestone</h1>
<p>
The plan for the next milestone is continuing development of threads in the NetBSD Process Plugin.
I will need to work more on correctness of <b>ptrace</b>(2) calls as new issues were detected
in setups with threads that resulted in crashes.
<p>
There is also ongoing work on a new build node running NetBSD-current (prerelease of 8) and building LLVM+Clang+LLDB.
I'm working on enabling unit tests to catch functional regressions quickly.
The original LLDB node cluster was privately funded by myself in the last two years and has been switched to a
<a href="http://lab.llvm.org:8011/builders/lldb-amd64-ninja-netbsd8?numbuilds=200">machine</a> hosted by
The NetBSD Foundation.
<p>
To keep this machine up and running (8 CPU, 24 GB RAM) community support through donations is required.
This is crucial to actively maintain the LLVM toolchain (Clang, LLDB and others) on NetBSD.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/netbsd_the_first_bsd_introducingNetBSD: the first BSD introducing a modern process plugin framework in LLDBKamil Rytarowski2017-04-04T17:12:05+00:002017-04-04T21:30:29+00:00A feature set for debugging NetBSD applications (<i>without threads</i>) has been merged with upstream LLDB!
The number of passing tests this month has been <b>increased</b> from 267/1235 to 622/1247.
This is <b>+133%</b> within one month and approximately <b>50%</b> of successfully passed tests in total!
As usual regular housekeeping of <b>ptrace</b>(2) interfaces has been done on the NetBSD side.A feature set for debugging NetBSD applications (<i>without threads</i>) has been merged with upstream LLDB!
The number of passing tests this month has been <b>increased</b> from 267/1235 to 622/1247.
This is <b>+133%</b> within one month and approximately <b>50%</b> of successfully passed tests in total!
As usual regular housekeeping of <b>ptrace</b>(2) interfaces has been done on the NetBSD side.
<p>
During this month I've finished the needed Native Process Plugin with breakpoints fully supported.
In order to achieve this I had to address bugs, add missing features and diligently debug the debugger sniffing on the GDB Remote Protocol line.
Since NetBSD-8 is approaching, I have performed the also needed housekeeping on the base system distribution side.
<p>
<h1>What has been done in NetBSD</h1>
<p>
I've managed to achieve the following goals:
<p>
<h2>Clean up in ptrace(2) ATF tests</h2>
<p>
We have created some maintanance burden for the current ptrace(2) regression tests.
The main issues with them is code duplication and the splitting between generic (Machine Independent) and port-specific (Machine Dependent) test files.
I've eliminated some of the ballast and merged tests into the appropriate directory <b>tests/lib/libc/sys/</b>.
The old location (<b>tests/kernel</b>) was a violation of the <b>tests/README</b> recommendation:
<p>
<pre>
When adding new tests, please try to follow the following conventions.
1. For library routines, including system calls, the directory structure of
the tests should follow the directory structure of the real source tree.
For instance, interfaces available via the C library should follow:
src/lib/libc/gen -> src/tests/lib/libc/gen
src/lib/libc/sys -> src/tests/lib/libc/sys
...
</pre>
<p>
The <b>ptrace</b>(2) interface inhabits <b>src/lib/libc/sys</b> so there is no reason not to move it to its proper home.
<p>
<h2>PTRACE_FORK on !x86 ports</h2>
<p>
Along with the motivation from Martin Husemann we have investigated the issue with <b>PTRACE_FORK</b> ATF regression tests.
It was discovered that these tests aren't functional on evbarm, alpha, shark, sparc and sparc64 and likely on other
non-x86 ports.
We have discovered that there is a missing <b>SIGTRAP</b> emitted from the child, during the <b>fork</b>(2) handshake.
The proper order of operations is as follows:
<ul>
<li>parent emits <b>SIGTRAP</b> with <b>si_code</b>=<b>TRAP_CHLD</b> and <b>pe_set_event</b>=pid of forkee
<li>child emits <b>SIGTRAP</b> with <b>si_code</b>=<b>TRAP_CHLD</b> and <b>pe_set_event</b>=pid of forker
</ul>
<p>
Only the x86 ports were emitting the second <b>SIGTRAP</b> signal.
<p>
The culprit reason has been investigated and narrowed down to the <b>child_return()</b> function in <b>src/sys/arch/x86/x86/syscall.c</b>:
<pre>
void
child_return(void *arg)
{
struct lwp *l = arg;
struct trapframe *tf = l->l_md.md_regs;
struct proc *p = l->l_proc;
if (p->p_slflag & PSL_TRACED) {
ksiginfo_t ksi;
mutex_enter(proc_lock);
KSI_INIT_EMPTY(&ksi);
ksi.ksi_signo = SIGTRAP;
ksi.ksi_lid = l->l_lid;
kpsignal(p, &ksi, NULL);
mutex_exit(proc_lock);
}
X86_TF_RAX(tf) = 0;
X86_TF_RFLAGS(tf) &= ~PSL_C;
userret(l);
ktrsysret(SYS_fork, 0, 0);
}
</pre>
<p>
This <b>child_return()</b> function was the only one among all the existing ones for other platforms to contain the needed code for <b>SIGTRAP</b>.
The appropriate solution was installed by Martin, as we taught our featured signal routing subsystem to handle early signals from the <b>fork</b>(2) calls.
<p>
<h2>PT_SYSCALL and PT_SYSCALLEMU</h2>
<p>
Christos Zoulas addressed the misbehavior with tracing syscall entry and syscall exit code.
We can again get an event inside a debugger that a debuggee attempts to trigger a syscall and later return from it.
This means that a debugger can access the register layout before executing the appropriate kernel code and
read it again after executing the syscall.
This allows to monitor exact sysentry arguments and return the values afterwards.
Another option is to fake the trap frame with new values, it's sometimes useful for debugging.
<p>
With the addition of <b>PT_SYSCALLEMU</b> we can implement a virtual kernel syscall monitor. It means that we can fake syscalls within a debugger.
In order to achieve this feature, we need to use the <b>PT_SYSCALL</b> operation,
catch <b>SIGTRAP</b> with <b>si_code</b>=<b>TRAP_SCE</b> (syscall entry),
call <b>PT_SYSCALLEMU</b> and perform an emulated userspace syscall that would have been done by the kernel,
followed by calling another <b>PT_SYSCALL</b> with <b>si_code</b>=<b>TRAP_SCX</b>.
<p>
This interface makes it possible to introduce the following into NetBSD: <b>truss</b>(1) from FreeBSD and <b>strace</b>(1) from Linux.
There used to be a port of at least <b>strace</b>(1) in the past, but it's time to refresh this code.
Another immediate consumer is of course in DTrace/libproc...
as there are facilities within this library to trace the system call entry and exit in order to catch <b>fork</b>(2) events.
Why to catch <b>fork</b>(2)? It can be useful to detach software breakpoints in order to detach them before cloning address space
of forker for forkee; and after the operation reapply them again.
<p>
<h1>What has been done in LLDB</h1>
<p>
A lot of work has been done with the goal to get breakpoints functional.
This target penetrated bugs in the existing local patches and unveiled missing features required to be added.
My initial test was tracing a dummy hello-world application in C.
I have sniffed the GDB Remote Protocol packets and compared them between Linux and NetBSD.
This helped to streamline both versions and bring the NetBSD support to the required Linux level.
<p>
As a bonus the initial code for OpenBSD support was also added into the LLDB tree.
At the moment OpenBSD only supports opening <b>core</b>(5) files with a single-thread.
The same capability was also added for NetBSD.
<p>
By the end of March all local patches for LLDB were merged upstream!
This resulted in NetBSD being among the first operating systems to use a Native Process Plugin framework with the debugserver capability, alongside Linux & Android.
The current FreeBSD support in LLDB is dated and lagging behind and limited to local debugging.
A majority of the work to bring FreeBSD on par could well be a case of <b>s/NetBSD/FreeBSD/</b>.
<p>
Among the features in the upstreamed NetBSD Process Plugin:
<ul>
<li>handling software breakpoints,
<li>correctly attaching to a tracee,
<li>supporting NetBSD specific ptrace(2),
<li>monitoring process termination,
<li>monitoring SIGTRAP events,
<li>monitoring SIGSTOP events,
<li>monitoring other signals events,
<li>resuming the whole process,
<li>getting memory region info perms,
<li>reading memory from tracee,
<li>writing memory to tracee,
<li>reading ELF AUXV,
<li>x86_64 GPR reading and writing,
<li>detecting debuginfo of the basesystem programs located in <b>/usr/libdata/debug</b>
<li>adding single step support,
<li>adding execve(2) trap support,
<li>placeholder for Floating Point Registers code,
<li>initial code for the NetBSD specific <b>core</b>(5) files,
<li>enabling ELF Aux Vector reading on the lldb client side,
<li>enabling QPassSignals feature for NetBSD on the lldb client side,
<li>enabling ProcessPOSIXLog on NetBSD,
<li>minor tweaking.
</ul>
<p>
<h1>Demo</h1>
<p>
It's getting rather difficult to present all the features of the NetBSD Process Plugin without making the example overly long.
This is why I will restrict it to a very basic debugging session hosted on NetBSD.
<p>
<pre>
$ cat crashme.c
#include <stdio.h>
int
main(int argc, char **argv)
{
int i = argv;
while (i-- != 0)
printf("argv[%d]=%s\n", i, argv[i]);
return 0;
}
$ gcc -w -g -o crashme crashme.c # -w disable all warnings
$ ./crashme
Memory fault (core dumped)
chieftec$ lldb ./crashme
(lldb) target create "./crashme"
Current executable set to './crashme' (x86_64).
(lldb) r
Process 612 launched: './crashme' (x86_64)
Process 612 stopped
* thread #1, stop reason = signal SIGSEGV: address access protected (fault address: 0x7f7ffec88518)
frame #0: 0x000000000040089c crashme`main(argc=1, argv=0x00007f7fffdd6420) at crashme.c:9
6 int i = argv;
7
8 while (i-- != 0)
-> 9 printf("argv[%d]=%s\n", i, argv[i]);
10
11 return 0;
12 }
(lldb) frame var
(int) argc = 1
(char **) argv = 0x00007f7fffdd6420
(int) i = -2268129
(lldb) # i looks wrong, checking argv...
(lldb) p *argv
(char *) $0 = 0x00007f7fffdd6940 "./crashme"
(lldb) # set a brakpoint and restart
(lldb) b main
Breakpoint 1: where = crashme`main + 15 at crashme.c:6, address = 0x000000000040087f
(lldb) r
There is a running process, kill it and restart?: [Y/n] y
Process 612 exited with status = 6 (0x00000006) got unexpected response to k packet: Sff
Process 80 launched: './crashme' (x86_64)
Process 80 stopped
* thread #1, stop reason = breakpoint 1.1
frame #0: 0x000000000040087f crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:6
3 int
4 main(int argc, char **argv)
5 {
-> 6 int i = argv;
7
8 while (i-- != 0)
9 printf("argv[%d]=%s\n", i, argv[i]);
(lldb) frame var
(int) argc = 1
(char **) argv = 0x00007f7fff3d17c8
(int) i = 0
(lldb) n
Process 80 stopped
* thread #1, stop reason = step over
frame #0: 0x0000000000400886 crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:8
5 {
6 int i = argv;
7
-> 8 while (i-- != 0)
9 printf("argv[%d]=%s\n", i, argv[i]);
10
11 return 0;
(lldb) p i
(int) $1 = -12773432
(lldb) # gotcha i = argv, instead of argc!
(lldb) bt
* thread #1, stop reason = step over
* frame #0: 0x0000000000400886 crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:8
frame #1: 0x000000000040078b crashme`___start + 229
(lldb) disassemble --frame --mixed
4 main(int argc, char **argv)
** 5 {
crashme`main:
0x400870 <+0>: pushq %rbp
0x400871 <+1>: movq %rsp, %rbp
0x400874 <+4>: subq $0x20, %rsp
0x400878 <+8>: movl %edi, -0x14(%rbp)
0x40087b <+11>: movq %rsi, -0x20(%rbp)
** 6 int i = argv;
7
0x40087f <+15>: movq -0x20(%rbp), %rax
0x400883 <+19>: movl %eax, -0x4(%rbp)
-> 8 while (i-- != 0)
-> 0x400886 <+22>: jmp 0x4008b3 ; <+67> at crashme.c:8
** 9 printf("argv[%d]=%s\n", i, argv[i]);
10
0x400888 <+24>: movl -0x4(%rbp), %eax
0x40088b <+27>: cltq
0x40088d <+29>: leaq (,%rax,8), %rdx
0x400895 <+37>: movq -0x20(%rbp), %rax
0x400899 <+41>: addq %rdx, %rax
0x40089c <+44>: movq (%rax), %rdx
0x40089f <+47>: movl -0x4(%rbp), %eax
0x4008a2 <+50>: movl %eax, %esi
0x4008a4 <+52>: movl $0x400968, %edi ; imm = 0x400968
0x4008a9 <+57>: movl $0x0, %eax
0x4008ae <+62>: callq 0x400630 ; symbol stub for: printf
** 8 while (i-- != 0)
0x4008b3 <+67>: movl -0x4(%rbp), %eax
0x4008b6 <+70>: leal -0x1(%rax), %edx
0x4008b9 <+73>: movl %edx, -0x4(%rbp)
0x4008bc <+76>: testl %eax, %eax
0x4008be <+78>: jne 0x400888 ; <+24> at crashme.c:9
** 11 return 0;
0x4008c0 <+80>: movl $0x0, %eax
** 12 }
0x4008c5 <+85>: leave
0x4008c6 <+86>: retq
(lldb) version
lldb version 5.0.0 (http://llvm.org/svn/llvm-project/lldb/trunk revision 299109)
(lldb) platform status
Platform: host
Triple: x86_64-unknown-netbsd7.99
OS Version: 7.99.67 (0799006700)
Kernel: NetBSD 7.99.67 (GENERIC) #1: Mon Apr 3 08:09:29 CEST 2017 root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC
Hostname: 127.0.0.1
WorkingDir: /public/lldb_devel
Kernel: NetBSD
Release: 7.99.67
Version: NetBSD 7.99.67 (GENERIC) #1: Mon Apr 3 08:09:29 CEST 2017 root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC
(lldb) # thank you!
(lldb) q
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y
</pre>
<p>
<h1>Plan for the next milestone</h1>
<p>
I've listed the following goals for the next milestone.
<p>
<ul>
<li>watchpoints support,
<li>floating point registers support,
<li>enhance <b>core</b>(5) and make it work for multiple threads
<li>introduce <b>PT_SETSTEP</b> and <b>PT_CLEARSTEP</b> in <b>ptrace</b>(2)
<li>support threads in the NetBSD Process Plugin
<li>research <b>F_GETPATH</b> in <b>fcntl</b>(2)
</ul>
<p>
Beyond the next milestone is x86 32-bit support.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>
https://blog.netbsd.org/tnf/entry/netbsd_7_1_releasedNetBSD 7.1 releasedsnj2017-03-15T09:52:19+00:002017-03-15T09:52:19+00:00<p>The NetBSD Project is pleased to announce NetBSD 7.1, the first feature update of the NetBSD 7 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements.</p>
<p>Some highlights of NetBSD 7.1 are:
<ul>
<li>Support for Raspberry Pi Zero.</li>
<li>Initial DRM/KMS support for NVIDIA graphics cards via nouveau (Disabled by default. Uncomment nouveau and nouveaufb in your kernel config to test).</li>
<li>The addition of vioscsi, a driver for the Google Compute Engine disk.</li>
<li>Linux compatibility improvements, allowing, e.g., the use of Adobe Flash Player 24.</li>
<li>wm(4):
<ul>
<li>C2000 KX and 2.5G support.</li>
<li>Wake On Lan support.</li>
<li>82575 and newer SERDES based systems now work.</li>
</ul></li>
<li>ODROID-C1 Ethernet now works.</li>
<li>Numerous bug fixes and stability improvements.</li>
</ul>
</p>
<p>For more details, please see the <a href="http://www.NetBSD.org/releases/formal-7/NetBSD-7.1.html">release notes</a>.</p>
<p>Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at <a href="http://www.NetBSD.org/mirrors/">http://www.NetBSD.org/mirrors/</a>.</p>https://blog.netbsd.org/tnf/entry/ptrace_2_tasks_segment_finishedptrace(2) tasks segment finishedKamil Rytarowski2017-03-01T07:14:26+00:002017-03-01T07:14:26+00:00During this month I've finished the needed work in the base distribution in order to host fully featured LLDB.
Currently the ptrace(2) interfaces in NetBSD are, in terms of features, closely related to FreeBSD and Linux.
There are only few bugs left with filed Problem Reports and alerting regression tests,
however they do not interfere with the needed functions to move the port of the debugger forward.During this month I've finished the needed work in the base distribution in order to host fully featured LLDB.
Currently the <b>ptrace</b>(2) interfaces in NetBSD are, in terms of features, closely related to FreeBSD and Linux.
There are only few bugs left with filed Problem Reports and alerting regression tests,
however they do not interfere with the needed functions to move the port of the debugger forward.
<p>
This time I will shortly iterate over the finished tasks as I would like to present slides on <b>ptrace</b>(2) in NetBSD.
<p>
As usual hundreds of ATF tests were introduced, few Problem Reports filed and two patches upstreamed to LLDB.
<p>
<h1>What has been done in NetBSD</h1>
<p>
I've managed to achieve the following goals:
<ol>
<li>Marked <b>exect</b>(3) obsolete in libc</li>
<li>Removed <b>libpthread_dbg</b>(3) from the base distribution</li>
<li>Added new <b>ptrace</b>(2) operations <b>PT_SET_SIGMASK</b> and <b>PT_GET_SIGMASK</b></li>
<li>Switch <b>ptrace</b>(2) <b>PT_WATCHPOINT</b> API to <b>PT_GETDBRGS</b> and <b>PT_SETDBREGS</b></li>
<li>Validation of <b>ptrace</b>(2) <b>PT_SYSCALL</b></li>
</ol>
<p>
The <b>exect</b>(3) interface is no longer functional and it's the proper time to obsolete it.
The <b>libpthread_dbg</b>(3) library is no longer needed, it became unnecessary along with the M:N thread model removal.
<p>
I originally added the <b>PT_*ET_SIGMASK</b> interface in order to help the <a href="https://criu.org/Main_Page">criu</a> port for NetBSD.
This software is used to checkpoint programs on Linux.
Debuggers can also have checkpointing support, for example GDB/Linux has this ability.
<p>
The debug registers are finally in the proper form on NetBSD.
The previous API had defects as it was designed to be safe, but keeping it safe on the kernel side was impractical.
I've finally decided to adapt it to the existing FreeBSD semantics and reuse <b>PT_*ETDBREGS</b> operations.
I've verified this new API on real hardware amd64 (Intel i7) and i386 (Intel Pentium IV) with newly written 390 tests.
It's worth noting that this API renders GDB to support hardware watchpoints on NetBSD almost out-of-the-box!
However it's not free of bugs, after catching a watchpoint, GDB enters on a trap in a dynamic linker (?).
<p>
<pre>
(gdb) c
Continuing.
Watchpoint 2: traceme
Old value = 0
New value = 16
main (argc=1, argv=0x7f7fff79fe30) at test.c:8
8 printf("traceme=%d\n", traceme);
(gdb) c
Continuing.
Watchpoint 2 deleted because the program has left the block in
which its expression is valid.
0x00007f7e3c000782 in _rtld_bind_start () from /usr/libexec/ld.elf_so
(gdb) c
Continuing.
Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007f7e3c0007b5 in _rtld_bind_start () from /usr/libexec/ld.elf_so
(gdb) c
Continuing.
traceme=16
traceme=17
[Inferior 1 (process 28797) exited normally]
</pre>
<p>
Sadly the <b>PT_SYSCALL</b> needs more work.
The current implementation is currently broken and cannot stop the process on syscall entry.
<p>
<h1>What has been done in LLDB</h1>
<p>
As usual, I've pushed some ready patches to upstream LLDB.
The first one covers Debug Register accessors and the second one adds proper thread identity detection on NetBSD.
<p>
<h1>Porting ptrace(2) software to NetBSD</h1>
<p>
I have prepared this presentation to illustrate the new code in the context of Linux and other BSDs.
<p>
To browse the slides one must use arrow buttons or the mouse scroll wheel. Press the <b>h</b> key for help.
<p>
<i><a href="http://netbsd.org/~kamil/ptrace-netbsd/presentation.html">Porting <b>ptrace</b>(2) software to NetBSD</a></i>.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>