#+BEGIN_COMMENT
.. title: Bitbake recipes: part 1
.. slug: bitbake-recipes
.. date: 2018-10-31 11:05:43 UTC+05:30
.. tags: embedded, yocto, databases
.. category:
.. link:
.. description:
.. type: text
#+END_COMMENT
In this two part posts, I am trying to document the process of
deciding on a issue, and how it evolves while trying to resolve or
_"close"_ it.
After initial release of [[https://www.senic.com/en/nuimo-click][Nuimo Click]], we looked at the pain points we
had on our backend stack. Currently we use lot of config file across
processes, some applications write to them, others read/need them to
connect to smart home devices. We use threads which "watches" changes
to these config file so applications update themselves. This overall
is becoming pain and also leaving us in nondeterministic states, which
resulted in bad UX and is hard to reproduce and debug. We had thought
of using Database or better key-value storage system for one sprint
and I tried to take a shot at it.
#+BEGIN_QUOTE
Title of issue: "Add DB to senic-os"
#+END_QUOTE
[[https://medium.com/@avichalp][Avichal]] my colleague, had already done some [[https://gist.github.com/avichalp/7de5cf31ac54f268de219e84ab72049a][benchmarking]] with options
available and I started with this reference point. There was initial
consensus on using popular [[http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html][BerkeleyDB]], said library(~libdb.so~) was
already part of the SenicOS and we just wanted python3 [[https://pypi.org/project/bsddb3/][library]] recipe
for it to start using it.
#+BEGIN_QUOTE
Title of issue: "Add BerkeleyDB to senic-os"
#+END_QUOTE
I started exploring on how to get the library installed but got stuck
with following bug:
#+BEGIN_SRC
user@senic-hub-02c0008185841ef0:~# ls -la /usr/lib/libdb-5.so
lrwxrwxrwx 1 user user 12 May 15 13:44 /usr/lib/libdb-5.so -> libdb-5.3.so
user@senic-hub-02c0008185841ef0:~# pip3.5 install bsddb3
Collecting bsddb3
Downloading https://files.pythonhosted.org/packages/e9/fc/ebfbd4de236b493f9ece156f816c21df0ae87ccc22604c5f9b664efef1b9/bsddb3-6.2.6.tar.gz (239kB)
100% |████████████████████████████████| 245kB 447kB/s
Complete output from command python setup.py egg_info:
Can't find a local Berkeley DB installation.
(suggestion: try the --berkeley-db=/path/to/bsddb option)
----------------------------------------
#+END_SRC
Small background, [[https://www.yoctoproject.org/docs/][Yocto]] is custom embedded linux distribution which
can be tailored for any SoC to create a small footprint, lightweight
distribution. For libraries and packages which needs to be shipped
with the OS, we need to write [[https://blogs.mentor.com/chrishallinan/blog/2012/04/27/more-on-yocto-terminology-recipes-and-packages/][recipes]]. They are small snippet of
config file, which defines the components, source of the package,
dependency, tools needed to compile the recipe etc. Most/many packages
already have recipes for them in [[http://layers.openembedded.org/layerindex/branch/master/layers/][layer-index]], where they can be
searched and integrated out of the box, but at times, we need to write
one.
I am familiar with and writing/figuring out how to put together a
recipe which should work but I needed some more understanding of
internals. While python package, ~bsddb3~ is actively maintained,
Berkely DB itself was available for download behind oracle sign-in. I
wasn't able to get these dependencies sorted out hence I started
looking at alternatives.
[[https://github.com/google/leveldb][leveldb]] was a good option, it is actively maintained, has a recipe
available for its core package and its [[https://github.com/wbolster/plyvel][python library]] is also well
maintained.
#+BEGIN_QUOTE
Title of issue: "Add leveldb to senic-os"
#+END_QUOTE
As I tried to put together recipe for ~leveldb~, I got stuck on trying
to figure out how to make ~cython~ compile header files from the
library for the native platform we used. I reached out in IRC
channel(#oe on irc.ubuntu.com) and shared my recipe and doubt, folks
there helped me understand how to use, enable ~cython~ to compile for
native platform. Here is how the final recipe looked like:
#+BEGIN_SRC
DESCRIPTION = "Plyvel is a fast and feature-rich Python interface to LevelDB."
HOMEPAGE = "https://github.com/wbolster/plyvel/"
SECTION = "devel/python"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://LICENSE.rst;md5=41e1eab908ef114f2d2409de6e9ea735"
DEPENDS = "leveldb \
python3-cython-native \
python3-setuptools-native \
"
RDEPENDS_${PN} = "leveldb"
# using setuptools3 fails with make command
# python3native is needed to compile things using python3.5m
inherit setuptools python3native
S = "${WORKDIR}/git"
SRC_URI = "git://github.com/wbolster/plyvel.git;tag=1.0.5 \
file://setup.patch \
"
PV = "git-${SRCPV}"
#+END_SRC
I needed to add a small patch to python package to get it compiled
with ~python3~:
#+BEGIN_SRC
diff --git a/Makefile b/Makefile
index 2fec651..2c2300a 100644
--- a/Makefile
+++ b/Makefile
@@ -3,8 +3,8 @@
all: cython ext
cython:
- cython --version
- cython --cplus --fast-fail --annotate plyvel/_plyvel.pyx
+ cython3 --version
+ cython3 --cplus --fast-fail --annotate plyvel/_plyvel.pyx
ext: cython
python setup.py build_ext --inplace --force
diff --git a/setup.py b/setup.py
index 3a69cec..42883c6 100644
--- a/setup.py
+++ b/setup.py
@@ -1,7 +1,10 @@
from os.path import join, dirname
from setuptools import setup
+from distutils.command.build import build
from setuptools.extension import Extension
import platform
+from distutils.command.install import install as DistutilsInstall
+from subprocess import call
CURRENT_DIR = dirname(__file__)
@@ -14,6 +17,16 @@ def get_file_contents(filename):
return fp.read()
+class BuildCython(build):
+ def run(self):
+ cmd = 'make'
+ call(cmd)
+ build.run(self)
+ # do_pre_install_stuff()
+ # DistutilsInstall.run(self)
+ # do_post_install_stuff()
+
+
extra_compile_args = ['-Wall', '-g']
if platform.system() == 'Darwin':
extra_compile_args += ['-mmacosx-version-min=10.7', '-stdlib=libc++']
@@ -53,5 +66,8 @@ setup(
"Topic :: Database",
"Topic :: Database :: Database Engines/Servers",
"Topic :: Software Development :: Libraries :: Python Modules",
- ]
+ ],
+ cmdclass={
+ 'build':BuildCython
+ }
)
#+END_SRC
[[https://baali.muse-amuse.in/posts/bitbake-recipes-part-2.html][To be continued...]]