PythonComments Off on MicroPython on ESP32: Tools – virtual environments vs system installs

Sep232017

These are my notes on using some MicroPython specific tools in relation to a ESP32-DevKitC board.

There are many tutorials and youtube videos that constantly encourage users to install tools and packages into their system-level libraries. (If you need to use sudo when you pip install foo, you are installing it as a system level library.) Please, Please, Please take the time to learn the basics of virtual environment. If you are a developer/hacker/maker – save yourself lots of frustration by using virtual environments.

A virtual environment is an isolated Python environment it contains all the necessary executables to use the packages that a Python project would need. It allows you use the desired version of a specific library and isolates that library from other virtual environments and the system and user level libraries. It allows you to easily define what packages are required to reproduce your work.

PythonComments Off on Hello World on a naked ESP32-DevKitC Board using MicroPython

Sep172017

Every now and again, I get the bug to build something. Lately, I’ve been following MicroPython and the microcontrollers that it supports. The new hotness is the Expressif ESP32 chip. These are available from a number of different sources, many supplying a breakout board. Prices are all over the place from 20+ to 8+ depending on where you shop and how patient you are.

Now in the world of microcontrollers, blinking an LED is the “Hello World” program. However, the boards I purchased only had an LED that lit if the board was receiving power. No other LEDs on the board connected to a GPIO pin like some other breakout boards. It does have 2 switches, one of which, Switch 1(SW1) is connected to the GPIO0 pin. In the image, SW1 is the button on the top right, labeled boot.

So I write some code to figure out the initial state of GPIO and then toggle the button a couple times.

I did sort some of this out using the serial REPL, but for this post, I wrote up a script to demonstrate my findings.

Using the adafruit ampy tool, we’ll run the code.

pip install adafruit-ampy

Note: you will need to press sw1 twice before you see anything after the ampy cmd.

$ ampy -p /dev/ttyUSB0 run sw1_1.py
sw1p0 initial value is 1
sw1p0 was 1 is now 0
sw1p0 was 0 is now 1
sw1p0 was 1 is now 0
sw1p0 was 0 is now 1

As you can see from the results, the initial state of GPIO0 was high(or 1). When sw1 is pressed/closed it goes low(0) and goes back high(1) when it is released/open. If you look at the board schematic, in the Switch Button section, you’ll see that when sw1 is closed, it shorts out GPIO0 to ground. This would indicate that you were pulling it low from a high state. So our observations match the schematic.

If you look at the schematic, you will see a capacitor from R3 to Ground that is used to debounce the switch. You should assume that all mechanical switches bounce and that bouncing needs to be dealt with in either the circuit or code. Life is much easier if you debounce the circuit with hardware.

Conclusions:

Success! While we don’t have an onboard LED to blink, we can do something with the board without extraneous components, a Hello World app.

The app is very naive since it uses polling to monitor state changes and spins in a tight loop most of the time. Often the reason for using a microprocessor has a power element to it. Sitting and spinning would be counter to a goal of low power usage.

We covered a lot of ground in this article, skipping or very lightly going over how to load MicroPython and the other tools I used. There are lots of very good resources for them on the interwebs.

If you liked this article, and you want to get an ESP32 board, you can use the Amazon affiliate link above as an expression of your support.

In an upcoming article, I’ll rework the example to be more energy conscious by using an interrupt to signal the state change.

PythonComments Off on Raspian: Configuring for editing Python with vim

Mar312016

We have a new Raspberry Pi 3 in the house and I’ve been having some fun getting it setup to edit Python with customized vim. You can see my dotfiles here.

So in a nutshell, I like using tmux + vim + plugins and customizations. The following are some snags I ran into and how to get them sorted.

To get tmux and vim with Python enabled for plugins:

sudo apt-get install tmux vim-nox

I use syncthing to keep my dotfiles in sync on the different systems I develop on.

cd ~/sync/common/config/dotfiles
./updot.sh

cd ~
vim +PluginInstall

Before trying to build YouCompleteMe you will need to increase your swapfile size otherwise you’ll die a strange an convoluted death.

sudo vim /etc/dphys-swapfile

#edit this line
#CONF_SWAPFILE=100
CONF_SWAPFILE=1000

Then reboot. switch to

.vim/bundle/YouCompleteMe

run

./install.py

I returned the CONF_SWAPFILE back afterwards.

sudo apt-get install python-virtualenv

To get the requests package to run without the Insecure Platform you’ll need to install some supporting libs

sudo apt-get install libffi-dev libssl-dev python2.7-dev

Finally, the default /etc/vim/vimrc has syntax highlighting commented out. So either uncomment it, or as in my case, update your .vimrc to enable highlighting. — First platform that I have had to do this on. It won’t hurt my other installs so I kept it in my own dotfiles.

PythonComments Off on Birthdays, Ramanujan but in the end it’s just Python.

Feb102016

It’s Sunday morning and I get some time to myself, so I’m listening to some blues and catching up on my newsfeeds when I come across this interesting article on calculating what size group of people would be necessary to have a 50/50 chance of two of them sharing the same birthday.

The primary focus of the article is on this equation to calculate the probability of uniqueness given a sample size of r from a group of N things to choose.

$$p = \frac {N!} {N^r (N-r)!}$$

In the article, he is concerned with the possibility of overflow with the size of the factorials involved and since scipy doesn’t have log factorial he implemented his solution with gamma log — see above article for his code. So I say to myself, “Wonder if I can do this with just straight Python?”

A quick google on log factorial found this approximation of log factorial by Srinivasa Ramanujan on math.stackexchange If you have not heard of Ramanujan before — stop and google him immediately. Wow!

Ok, so no scipy needed to follow along with this article plus I get to use some very cool math. I like Sunday morning fun time. However, I then start thinking, “overflow”, hmmmm. What is the upper bound of math.factorial anyway. Since it’s the birthday problem lets see what happens with 365

Well it took a bit but it ran, with over 40 screens of numbers. Python is still going strong, so what is the upper bound of math.factorial? A search brought me here http://bugs.python.org/issue8692 and specifically this message. Which means that the max size of the result can not exceed sys.maxsize - 1 digits, or on a 64bit platform, 2**63 – 1 digits of capability. Thanks to some dedicated individuals who seemed to be having as much fun as I was, math.factorial is up to the task.

The take-away from this article is not the cool math, or the approximations of log n! — it is…

Don’t underestimate the power of Python.

Try straight Python before you move on to something more complex. The approximations of $\log n!$ were unnecessary. All that was needed was just Python and we get the following implementation of the Probability of Uniqueness:

So hopefully this does not surprise you. If it does, please see the two links in the footnotes(*1) as they explain it quiet well.

My question deals with the implicit nature of the mutation. Not that Python is somehow wrong, it is the fact that the function usage does not convey the mutation to the reader as pointedly as I want. Coming from other languages that are call by value, a function that wanted to mutate an argument and get it back into the caller’s scope had to return the mutated value.

So why does toggle_explicit() give me the warm fuzzies? Where as toggle() requires the reader to know what is going on. Is it just me shaking the cruft of call-by-value languages off? What do you do, when you mutate state within a function? Is the toggle_explicit() form not/less Pythonic?

In a recent post Amit talked about temporary files and gave a number of scenarios where they can be quite handy. In testing, I occasionally need temporary files and prefer to use mkstemp however, the clean up of the file was bothersome and I found that I often needed to write something into the files before the test.

from dhp.test import tempfile_containing
contents = 'I will not buy this record, it is scratched.'
with tempfile_containing(contents) as fname:
do_something(fname)

Thus tempfile_containing was my solution. It uses mkstemp, writes contents to the file and is implemented as a context manager so it returns a path and file name that is cleaned up when it goes out of scope. (Note: NamedTemporary and others, return a file-like object tempfile_containing returns a path/filename of an existing file) After the file is written to it is closed so no inadvertent lock contentions occur on some OSs. If it sounds like something that would make writing tests faster for you, check it out at:

A comment starts with a hash character (#) that is not part of a string literal, and ends at the end of the physical line. A comment signifies the end of the logical line unless the implicit line joining rules are invoked. Comments are ignored by the syntax; they are not tokens.

Seems pretty straight forward to me. Parser finds a comment token, and ignores everything to the new-line token. (Except for when implicit line joining rules are invoked.) So why was I struck with a hmmm, when I opened up a Python Console and entered a comment then pressed Enter?

>>> # this is a comment, guess what happens next?
...

If you look closely, the next line is prefixed with an ellipsis (…) and not a new prompt (>>>). Well there must be a reason, but this feels “unexpected.” So I pull up a 2.7 console and try it again.

>>> # this is a comment, guess what happens next?
...

So it looks to be intentional, although it doesn’t seem to feel correct. Ok, let’s give pypy a try and see what happens there.

>>>> # this is a comment, guess what happens next?
>>>>

Ok, now I am confused. CPython treats it as an unclosed statement of some kind, although you would think it would be closed because a new-line token should have been encountered thus closing off the comment. However, when pypy gave a different and less surprising result, I put a hand to my chin and said, “Hmmm.”

Ok interweb – does anyone know what is going one here?

1) Why the unexpected ellipsis in CPython ?
2) Why does pypy not return an ellipsis ?
3) Are they both correct and are just slightly different implementations of the same reference? Or is one more correct than the other?

A search for information on string interpolation in Python will inevitably lead you to comments and links to old documentation that the string modulo operator is going to be deprecated and removed. However, that is just outright FUD. I need not make a case for the modulo operator, I’ll just let the code do the talking.

Note that test_format_explicit is the form most commonly found on the web. However, the implicit version is a much closer equivalent to test_modulo. In this case, there is an apparent price for being explicit.

Until .format is on par, speed-wise, with % there is no chance of it being deprecated. I support .format‘s existance, in some battle grounds it is superior. You shouldn’t bring regex‘s to the fight when .starstwith, .find, .endswith or in can handle the challenge cleanly. The same is True for .format and %.

An as PEP 461 demonstrates, the string modulo operator is not going quietly into the night.

I personally don’t like Python’s type annotations, the completely mask out what was a human friendly function signature. For something that was being proposed and skunk-worked in to stub files, according to Guido’s keynote at PyCon, it is spreading inside of source files, at an alarming rate, through out Python’s upper echelon.

Just this weekend Type Annotations infested a blog post on Why print is now a function in Python 3. Luckily Brett, took out the hedge trimmers and cleared away the mess. It is a interesting blog post now that you can see what he is talking about. Now I see that they’ve spread to PEP8.

The usefulness of Static Type Checking against Dynamic code is questionable at best. The damage that can and may be done to readable function signatures is frightening on scale similar to the Kudzu infestation and decimation of indigenous plants in the US. I encourage you to help fight this invasive and damaging trend by keeping your Type Annotations where they belong, in stub files.

Why doesn’t PEP8 encourage the use of stub files over obfuscating your code? A considerable number of people don’t like lint droppings ( # pylint:disable=… ) in their code and it goes in a comment, this trash is being put right in the function’s signature.

What is the practical upside to Static Type Checking? Guido talked about it in big terms and hand waving, in his keynote and people use glowing buzz words, but, seriously with examples, what can it actually do that writing testable code and tests can’t? The upside is as hard to see as a type annotated signature.

I’ve decided to lend out my robot to other group members who are interested in the topic. I am going to document how to get the Robot set up and create an environment to interact with it. It had been a while since I last used the robot (python2.4 or so) and I had to do a few things to get things fixed up with the current version of python and the supporting packages. I got a lot of help from this article but I am going to condense that information to what needs to be done on a linux platform.

NOTE: These instructions are for python2.7, I’ve read that python3.x is problematic, although I’ve not tried.

Software Environment

First setup a virtual environment

virtualenv robot

Now change to the directory and activate the virtual environment

cd robot; source bin/activate

Now install the dependencies

pip install numpy pyserial Pillow

Now checkout the latest myro source

svn co http://svn.cs.brynmawr.edu/Myro/trunk myro

change into the myro/myro subdirectory and edit the graphics.py file. You will need to change the line import ImageTk to from PIL import ImageTk

Now change back up one directory and run the setup for the myro library

python setup.py install

now go back one more directory so you are in robot

Bluetooth setup

My laptop didn’t have Bluetooth built-in so I used a dongle. Do what you need to do and open up your Bluetooth manager, then turn on the robot (with the fluke board attached). Robot requires 6 AA batteries and will run fine with rechargeable batteries if you have them. Install them in bottom compartment. Install fluke board by mating it to the rs-232 connector on the top of the S2. Power switch is a black slider by the comm port.

Look in your bluetooth manager for a device that has IPRE in the string. Pair with it and use the following code. 1234 Make a note of the device that is set up. On my rig it was /dev/rfcomm0NOTE: on mine the /dev/rfcomm0 was root:dialout but I couldn’t access it. I was too lazy to check my groups so I just pulled out the hammer and hit it withsudo chmod 666 /dev/rfcomm0

Ready to Test

Inside your virtualenv, fire up python and enter the following:

from myro import *
initialize('/dev/rfcomm0')

repeat the initialize command until your are successfully connected. You’ll hear the beeps once you connect. Now lets test it. Assuming we are starting where we left off from above. (If not, start the python interpreter up again and issue the import and initialize commands from above.)

forward(1, 1)

See the manual for a list of commands or just dir() and help(interestingCommand)

If you run into any issues please let me know. I am going to keep this post updated so as others borrow the robot they’ll have some up to date instructions to get them started. If you are a member of the Omaha Python Group and would like to arrange to borrow the robot, please contact me.

I like the idea of listing changes to my distribution in the long_desciption in setup.py. So a release a go, I started appending docs/changes.rst to my README.rst file that I am using for pypi. It was a simple doc with bulleted lists. The world was good.

However, I grew unsatisified with my changes.rst file and wanted it to link, ala cross-references to the documentation, so when reading the docs a user could quickly go see the docs for that item. For example:

Sphinx liked it, I liked it.
However, python setup.py check --restructuredtext --strict gagged when it saw it, so would PyPi.

Harumph! I mutter to myself. I want it all, I want Don’t Repeat Yourself(DRY), I want my change log to display on PyPi, I want cross-references in my docs. However, cross-references don’t make sense in the long_description, what they link to isn’t there. I do not want to update changes in two different places, I am already vexed with making sure that just the one document is updated. After all, who likes writing docs more than writing code?

Well, how do I get the cross-references scrubbed out for the long_description? Here is what I coded up:

It is a Decorate, Scrub, Transform, Undecorate kind of pattern. Stripping out the :roles: tags left the single ` markers. So, if I wanted to change those to ``, I have to hide all of the existing ``, scrub, transform, and then unhide the original ``. And so that is what I did.

I imagine a determined individual could create a regex to cover the entire breadth of sphinx directives and make a sphinx_to_pypi converter. But for me, my itch is scratched. Maybe it will help someone else too.

2014-05-05, I am updating this post to give the short answer: use python setup.py register to update your meta-data on pypi. It can be run repeatedly and will modify the meta-data on pypi for the distribution.

and now the original blog post…

During the first launch of YamJam, I went through a series of releases because of rendering issues of README.rst on pypi. WAT!, I say to myself and a few other choice words. I had just created 6 pages of rst, that compiled just fine for my sphinx generated documentation. I had cut and pasted the top portion of my index.rst with some text edits — what can be going on? I uploaded my package with twine – no errors. pypi was seeming to say, everything is peachy, then turning its back and mumbling FU. (Things should not fail silently.) I pulled up my shiny new package on pypi and was met with unseemly, unformatted text instead of a spiffy display.

I google for answers, I review other dists in pypi, pulling up their repos and reading through their readmes — Aha! I say to myself, I don’t see anyone else using an :alt: on their build status badge. I remove mine, go through the release procedures again, I upload and was slapped down yet again.

I grow angry and frustrated, I recheck my readme, I google for issues related. I find mention of “run the docutils command on it”, but no mention of what command. I review even more dists on pypi, seeing other broken readme renderings. I am unsure of what to do. I think — “It must be my .. code:: python, I see lots of :: , I rip out mine and replace with ::. I try again and FAIL.

I reread the rst, I put it through sphinx just by itself and see a warning about duplicate references. I had `view <url1>`_ for docs and `view <url2>`_ . One of those little edits, I mentioned earlier. You think, looks like an href, smells like an href — but no, it is different. That did it, 3 revisions later, I’m happy with the display of my readme on pypi.

It was a day later, while enhancing my release and CI scripts that I found the python setup.py check and python setup.py check --restructuredtext commands. I tested it locally, and sure enough, it would “warn” but not set an return code, so my release script couldn’t detect the failure. I figure, that messing up your pypi page, should equal FAIL not warn. Ok, so I’ll submit a patch to make it ERROR and set a non-Zero return code. I find the code repos and start reading through the code and discover an option I hadn’t seen. -s, strict. That will cause it to FAIL and set a non-Zero return code. So off to the CI script I go, I add the strict option and the test passes when it should have failed. If you don’t have docutils installed, and I didn’t on my CI, it just returns the same response as if it had passed. FACE PALM. pip install docutils

As it stands now, I can detect rst that will cause pypi to fail silently so I am good in that regard, and you should be to, now that you have read this.

Sidebar: why dosen’t setup.py upload and twine upload automatically run the checks supplied by setup.py in their strictest mode? Fail early, the cost is less

Which brings me to unnecessary binding. Why is the description on pypi so tightly and unnecessarily bound to a distribution release. Forcing a new release to fix render problems and typos? Pypi will let us upload packages but not let us edit the description in a web interface WITHOUT having to do an entire re-release? Use the readme as a starting point, let us edit without re-releasing.

Alex G, if you happen to read this — please make this possible on warehouse. Mr Gaynor, Tear Down this binding! Also, Give us download stats, with as much info as possible so we can weed out mirror requests.

Anyone who is thinking, “I’m going to rewrite pip because of X, DON’T” We have had too many installers, to many distutils and setuptools. setup.py is a cacophony of knobs, buttons, dials, many fighting each other. “There should be one– and preferably only one –obvious way to do it.” Unless it has to do with setup.py, then it should be as confusing as possible. There needs to be a pypi/setup.py BDFaW (Benevolent Dictator For a While). If the new solution doesn’t solve the current problem, or creates new problems, it is not a solution, it is just change for the sake of change. Eggs and wheels – harumph, I say.

I have a project, YamJam, that I have been using since 2009. The main idea is a framework that allows you to factor out sensitive data from your code before you upload to a 3rd party repos. I’ve got internal and external projects that have been using it for quite a while and it makes these refactorings a breeze. I have other open source projects that get a lot more attention but yet have a niche audience, so I am experimenting to see if it is the lack of documentation and being a conforming pypi dist that is limiting it’s appeal. Yamjam should be popular, because it can scratch an itch caused by Djano. That itch being, “What do you do with settings.py?” It’s got lots of sensitive data that shouldn’t be checked into to a repos but it also has a lot of code that should. It also makes deployment between dev, staging and production easy to do with a checkout.

To that end, I’ve been creating a proper and complete test suite (was doctests) using py.test and tox, Continuous Integration with drone.io, documentation via readthedocs.org and sphinx, spiffing up the setup and dist with the latest distutils and uploading with twine instead of setup.py upload.

Through this whole process, I’ve had a lot of new experiences that I am going to be blogging about in the upcoming weeks. Things I like, really like, things that are annoying and some things that are counter to my way of thinking. During this process I’ve also been filing bug reports when I’ve encountered them and sending out feedback for improvements along the way.

After moving my code from subversion on google code to mercurial on bitbucket (hg convert), I started looking for a CI service to use. Off the bat, I looked at Travis-ci, but unfortunately, travis is a github snob. If you are not hosting your code on github or mirroring off of github then travis-ci is not an option. Some google searching showed that pylint (a tool I like very much and use) moved from internal tools to bitbucket and http://drone.io/ . So off to drone I go.

http://drone.io/ took about 10 minutes from signing in via my bitbucket account to running my first integration. When I learned that drone.io allows you to view the settings for other open source projects build environments, I had python 27, 32, 33 and 34 tests running via tox less than an hour later. What higher praise could I give a service than saying from 0 to testing in 10 minutes? I really, really like drone.io and recommend that you check them out. Unlike Travis-ci, drone.io supports bitbucket, github and google code. Options, I like. It is the same reason I prefer bitbucket to github. Bitbucket supports mercurial and git. I use both dvcs systems. I prefer mercurial. I like that bitbucket allows me to make the choice. Choice makes me happy. Check out my setup on drone.

I’ve run in to this situation a few times and end up having to query my gBrain for the answer. When using json as a transport from python to html/javascript, I frequently end up needing to move date and time data. However, the builtin json module is not happy when you ask it to serialize a date/time object.

So this means we need to figure out a work around. The trick is to let the json module know what it should do with a date/time object while leaving the rest of it in place. So, no replacing the default handler.

What we need to do is subclass the JSONEncoder and override the default method

Using the hasattr and looking for ‘isoformat’ method will allow this to handle both date objects and datetime objects. So all that is left to demonstrate is how to put it together with the json.dumps method.

Ok, so now you have this ISO formatted string containing date information, how do you get that converted to a javascript date object once it has been transmitted across the abyss and now resides in a javscript callback function?

So, I’m hacking on a Python for Android project, which is built over top of the SL4A project. I’m currently using the remote method of development where you fire up an interpreter and share it in public mode. Then you import android.py and instantiate an instance of Android with IP and Port information of your public server. You can then hack in your favorite editor on a laptop instead of using a thumb-board or other. It looks like this:

What happens is the __getattr__ method of the Android object uses magic to change droid.getenvironment() into an RPC call to the public server and then return the result back as a named tuple. Nice. Being the nosey bugger that I tend to be, I modified the code to add a debug param to the __init__ method, that when set, would print out what was being sent out over RPC and then the raw tuple results. A snippet of the modification goes like this:

You can easily see where I put my “if self._debug” logic in place. Now if I use my modified android.py I can turn on the debug flag and get some 411 on the magic that is going on. It ends up looking like this: