kdev-python 1.4 stable released!

I’m happy to announce the release of the first stable version of kdev-python, version 1.4! As this is the first stable release, this post is supposed to be an overview about what kdev-python actually does.

KDevelop with kdev-python 1.4

First of all, kdev-python is a plugin for KDevelop. Its purpose is to make development of python applications more convenient. The main focus of the program is static analysis of source code, and providing features which use the information gathered, such as

Intelligent code completion, depending on where the cursor is and what variables exist etc.

Navigation features, such as Jump to Declaration, searching for functions/classes, class browser, …

and many more

There’s also a few features other than static code analysis, such as debugger integration, but the former is clearly the main focus of the project. kdev-python itself is written in C++.

Dependencies and language support

The 1.4 release is only meant to be used with python 2.7. It does not depend on python 2.7, but scripts containing syntax which is incompatible with python 2.7, such as stuff new in python 3.x, will be marked as containing syntax errors. You can work around some things with __future__ imports, tough. A python 3.x version of the plugin is currently being developed; it’s almost done, and will soon be merged to the master branch. The next release (1.5) will support python 3.
kdev-python 1.4 is meant to be used with kdevplatform 1.4 (kdevelop 4.4), which is also where the version number comes from. Versions 1.3 or older or 1.5 or newer of kdevplatform are not compatible due to API changes.

Installation

Installing kdev-python 1.4 should be possible through your distribution’s package management system soon-ish. The package will probably be called “kdev-python” or similar. The sources are available from here. Compilation instructions can be found in the tarball.
The checksums for the file are:

After installing, make sure to add the installation directory to your $KDEDIRS environment variable, and run kbuildsycoca if necessary. You may want to add those two commands to /etc/profile to have them executed at login automatically.

Using kdev-python

To use kdev-python, just start up KDevelop and open any python file. You can verify that kdev-python is working by looking at Help » Loaded plugins:

kdev-python in the plugins list

If you want to open a whole python project, just click Project » Open/Import project, and select your project’s root folder.
As soon as you do so, the background parser will start analyzing your code. You can observe its progress by looking at the right bottom corner of the KDevelop main window:

Background parser progress bar

Depending on how large your project is and on how many external libraries it imports, this may take up to several minutes the first time you open a particular project (on my computer it takes roughly 15 minutes per million lines of code). The computed information is stored to a persistent cache, so subsequent startups will be much faster. You can already start coding while the background parser is still analyzing your project, but the information being displayed might not be complete yet.
The following sections will give an introduction on how to use the most common features of kdev-python.

Code completion

kdev-python does context-sensitive code completion; there will be different suggestions depending on where in the program your cursor is located. It will open up automatically when the program thinks it would be useful, or you can force it to show by pressing Ctrl+Space (by default). Here’s a few examples of what it does:

Even in complex cases, there often is useful completion suggestions

Slices are interpreted correctly

If a function which was just completed takes no arguments, the cursor is placed behind the brackets…

… and if it takes arguments, it’s placed inside the brackets.

Full completion (Ctrl+Space) shows types of objects next to their names.

Generator expressions are supported by this handy little feature.

There’s also completion for importing objects from libraries

An example of context-sensitive completion: for inheritance, only classes are listed in the completion list.

The “implement function” completion feature

For the “raise” statement, only names that will give a valid exception object are suggtested.

Another example of a not-so-simple case where completion still works.

When invoking full completion inside the brackets of a function, you are presented with a list of arguments the function takes

The most basic case: All matching objects from the local and global scope are suggtested.

Obviously, all this is based on the static language analysis framework. Thus, in obscure cases (such as when using exec), it will not be able to do anything useful. In quite a lot of cases it turns out to be very nice, tough.

Note that it is very difficult to get proper support for python libraries written in C. Currently, only a few select ones are supported. The support for PyQt4 / PyKDE4 is very good, as can be seen from the above screenshots; there’s also some very basic support for numpy and a few others. Help is very welcome here, it’s quite easy to get your favourite library supported (I think it can well be done in one afternoon without prior knowledge, and you can do it in just python probably).

Navigation widget

The navigation widget is another feature of KDevelop which allows you to navigate code easily. To show the navigation widget, move your mouse over any highlighted use:

The navigation widget.

Another example of the navigation widget. While the widget is shown, all uses of the selected object are highlighted with a different background color

You can navigate inside the widget; all the blue things are clickable. You can for example click the “Material” in the first screenshot to look at the navigation widget for that class. You can also click the place of the object’s declaration to jump there, or make KDevelop find all uses of an object.

The navigation widget currently does not work for import statements, that needs to be implemented in future releases.

QuickOpen

Another very useful feature of KDevelop (which is not really specific to kdev-python) is QuickOpen. It allows you to instantly jump to any class or method declaration in your whole project. Just click into the filter bar at the top which says “QuickOpen” (or use the shortcut shown in the menu), and type parts of a function or class name, then press Enter:

The Quickopen feature

“Outline” does the same, but only in the current file.

Output marks & debugging

In Run » Configure launches, you can select “Script Application” for a launch in the left toolview. Then, enter “python” for the interpreter, and provide a full path to your application in the “Script” line edit (note that this dialog has already been totally refactored, it’s much nicer in kdevelop master. The changes have not yet been released tough). If you now run your application and it encounters an error, you can use F4 and Shift+F4 to jump to the next or previous line in a backtrace:

Press F4 or click on an output mark to jump to that position in the respective file.

KDevelop will automatically open that file and jump to the correct place.

There is also (somewhat basic) support for the python debugger, pdb; just click the “debug” button after configuring your script as described in the previous paragraph:

pdb plugin in action

The navigation widget is replaced with the debugger widget, which displays the current value of the object your mouse is moved over. The left toolbar displays all current variables; you can exapand the trees to see properties of objects. You can also enter some arbitrary python expression in the line edit in the bottom left and press enter; it will be evaluated on every debugger step and will be displayed there, too. The right toolview shows a stack trace. Note that since this is based on a thin wrapper around pdb, threads are not supported (since they’re not supported in pdb either). There is some special code to support debugging of Qt applications with event loops correctly.

Future plans

There’s two big things which need to be done next: Proper support for Python 3, and proper support for frequently used libraries. Support for Python 3 is being worked on, and some ideas for e.g. better django support are also there. So, you can be excited about what will happen in future releases! 😉

Final words

So, that’s about it — as every application’s first release, kdev-python 1.4 is totally bug-free; it will never crash or display something which is incorrect! If it still does, you probably found a hidden feature; you can report it to bugs.kde.org in the kdev-python product, so I can add it to the official feature list. 🙂

If you’d like to help this project, there’s lots of things to be done! Just write me an email, or visit us in #kdevelop on irc.freenode.net.

bpy is a C library, so it will not work automatically. To get it supported, you need a script which creates something like C's header files, e.g. python files which contain no actual code but only the function / object declarations. You can for example do this by introspection. If you have such files, just put them in the documentation_files directory, and it'll work in kdev-python.

I'm still thinking about how to do this best in a more general way, so if you're interested in working on this, tell me 🙂Everything which is required for this could probably be done in python and would not need any knowledge about kdevelop's or kdev-python's internals.

Probably won't help, unless you manage to generate pydoc from the C source code. That might be possible, I didn't investigate that yet. If it is possible, then using KDevelop's nice C/C++ parsing capabilities would be the easiest way.

The way I would have suggested is looking for how the blender people generate their (HTML) docs, and then hack that to print python instead of HTML. But I'm not at all sure it's the best way.

Basically all you need is a bunch of python files with empty definitions for all the classes / methods, something like this:

This is just a rough example but I think you'll see what I mean.If you have this, then kdev-python will support whatever is written there. How to generate it… I don't know, you'll need to investigate 🙂For PyQt, we generate it from the SIP files with some scripts, you can find those in documentation_src/pyqt/, for example.

It looks like blender uses Sphinx to generate its documentation. It has a bunch of output formats, so there might be one you can use. Or perhaps you can write your own, or intercept the documentation data before it is written. I don't know much about the internals of Sphinx, but I do build a lot of python packages and see it used very often, especially for large projects like numpy/scipy, matplotlib, django, and blender. So supporting it is probably a good idea.

Hi, really great work. Thanks. I'm using kdev-python now in production until september. I really would like to help. I can't write C or C++, but I have pretty good python skills. So if I can do something let me now.

Some way to generate documentation in a machine-readable format from python libraries written in C would be very important. One could for example attempt to build this upon pydoc. or a similar python documentation framework. This could be done in python entirely and would have a hugely positive effect on the usefulness of kdev-python overall.

If you're interested in working on this, drop by in #kdevelop and ping me! 🙂

That sounds good. The question is, tough, what information does it provide? Especially important for kdev-python would be "likely return type of a function", which is something most documentation generators don't have — since it's not really well-defined for python in general (altough the return type is actually fixed for most functions).

I want it because this information is vital for static language analysis.Consider for example the etree module:

root = etree.fromstring(some_xml_data)root.

What would you display as completion items if you don't know the return type of etree.fromstring()?

Of course you can in most cases not be sure what a function returns, but almost always there's only a few possibilities (for example "an etree.element instance or None on failure" — but the function will never return, whatever, an int). kdev-python has a nice concept for those "unsure" types and they can be handled in a proper way. But you still need exactly that information ("what might be returned").

Not expicitly. I also don't really know that package. But from quickly reading what it does, you should be able to emulate support for it by setting different values for $PYTHONPATH, then starting KDevelop.

The include path KDevelop uses are those which are in the default python interpreter's sys.path.

Agreed, pep8 syntax highlighting would be a killer feature. This may be something more for the kate editor component in general rather than kdevelop specifically, though. Does kdevelop support kate pate plugins?

Someone else wanted PEP 8 checking too, there's even a wish in the tracker. I personally wouldn't really know what to do with it, but if so many people are interested then I'll implement it (will probably be in the 1.5 release then). It's not really very difficult I think, at least for the simple aspects.

I'll have a look at it. Integrating a ready-made PEP8 checker into kdevelop should be a matter of minutes, I just want it to be configurable so it doesn't spam warning messages people don't want (for example I don't want it 😉

Yeah I fixed it meanwhile, it should be okay now. It'll only do the checks for documents which are actually open, so it shouldn't slow down parsing.If you experience extraordinary sluggishness with large documents, tell me, it might be related.

Compile without -Werror=shadow. Scope shadowing is a valid feature in C++; of course the warning can be useful at times but setting it as an error condition in your compiler is bound to cause a load of problems when building stuff.

No; kdevpdb.so is the library which contains the PDB debugger support plugin. The plugin itself is contained in the kdevpythonlanguagesupport.so file, but there's a few more libraries and files which are required to run the plugin. Why do you want to know? It shouldn't matter to setting it up.

No, the expected result is that it loads the kdevpdb and kdevpythonlanguagesupport plugin.

If you have further questions I suggest you come to our IRC channel (#kdevelop on irc.freenode.net) and ping me, since discussing this in blog post comments is a bit tedious and not very likely to be of use for others. I'm sure the problem can be quickly resolved there.

First i wanna say thanks for creating this great plugin for Kdevelop! I work with both C++ and Python on a daily basis now and being able to use them so seamlessly in the same editor is really great.

Second, you said earlier that generating auto completion for c-libraries would involve creating python "header" files. I was wondering if this is really necessary. Couldn't you just load the symbols from the library directly, like Python does?

I suppose you could even invoke the python interpreter to load the module, obtain a list of all symbols and docstrings, and unload the module. This seems to be a common strategy for generating header files from C-libraries.

For the documentation stuff: You could do that, yes. It has two problems, tough:1) Introspecion does by far not provide as much information as you want it to. It will, for example, never give you return types of functions, which are very important for this IDE (imagine lxml — if the function parsing a string has no return type set, then you can forget the whole module, since your XML tree's root object has an unknown type and everything that is derived from that will have an unknown type too). It also often doesn't give you stuff like argument count, argument names, or proper docstrings.2) Introspection is unsafe. Since on an "import", code from the library being loaded is executed, all this needs to be done in a good sandbox in order to not introduce horrible security flaws — and sandboxing python isn't exactly an easy thing to do, as far as I can see.

I see your point. However, i recon that even a list of symbols exposed by the the module would be better than nothing. I use OpenCV a lot and while the Python bindings exist purely in an .so file the docstrings will tell me the function arguments. It's not perfect, but it's much quicker than having to check the online documentation (and it would be even quicker if i could do it directly from KDevelop 😉 )

I hadn't considered your second point though and i agree that this would be quite risky to do automatically. Still, it could be made optional with appropriate warning signs and confirmation prompts to assure that it only happens at the users approval.

Yeah… but, in many cases, modules use functions as entry points, such as lxml, and then it's all useless 🙁

And, doing this warning-message based will give the user a huge lot of warning messages at times where he doesn't want them… plus, I hate warning messages where you have to click "yes" or "no" 😉

I'm actually currently working on a different approach for solving this problem: kdev-python should ship with a set of tools which import a library and convert it to a python "header" with the information retrieveable from introspection. So, you could generate your opencv kdevelop docs by yourself easily with one command (or even, GUI button). Additionaly, I want to implement GetHotNewStuff-support for those files, so you could easily upload your result and others could download it. Additionally, it is quite easy to edit the resulting "header files" by hand and make the most important corrections, e.g. add return types for the entry point functions. So, people interested in a specific library could get introspection-based documentation generated for it automatically, and if they want to improve it, they could just do that and share their result with others. Eventually, I would merge the best additions to upstream kdev-python on each release in case the creators agree with that.

That should work just as well :-). I guess it really just comes down to how you choose to store the information obtained from the module. I can see many advantages in being able to improve and share these "header" files.

I'm curious as to how kdev-python currently determines the return type of a function? Does it look for return statements and try to determine the objects type directly?Since docstrings often contain this information and more, perhaps it would be possible to parse these automatically? Of course there are no strict conventions for docstrings but they tend to at least be consistent within each module. It could be as simple as having a script for every module that parses the docstring and returns any useful information such as argument descriptions and return type.

Just an idea of course :-). If you can use any help i wouldn't mind lending a hand.

Argh blogger destroyed my reply text… I hate this software ._.I also hate that the world has in 40 years not invented a web browser which is capable of detecting that you spent time typing a text and saving it for you… well.

—

For python code, kdev-python has a somewhat sophisticated (~10k LOC) static language analyzer. The main purpose of that analyzer is to guess the type of each object in your project's python code as accurately as possible. Main sources of information for this analysis are of course variable initializations (myfoo = Foo() -> myfoo is of type Foo). The analyzer then tries to track this type as far as possible, through lists, tuples, assignments, loops,… and also function return types. So, yeah, the return type of a function type is determined from the types of the objects which appear in the return statements in that function.

Parsing docstrings would be a good idea, definitely. I'll definitely provide some pre-made docstring parsers in the header file generator I mentioned, and you could add your own too.

If you're willing to help, that would be very much appreciated — this area is where you can really help this project much, because a) it's somewhat easy and b) it's likely to improve the usefulnes of the software by a large deal.

I'm currently struggling with this up/download stuff, it's sort of frustrating… maybe I'll put it down for a while and write that generator script first. Either way, if something useful turns up, I'll blog about it, so if you follow planet KDE, you won't miss it 😉

The advantage with this is that you don't really need to understand any of the code, you can do everything in a python script which has nothing to do with the plugin itself 😉I started writing said script, if you like you can look at it: https://projects.kde.org/kdev-python click repository, documentation_src, introspection, introspect.py. Download it and run e.g. python2 introspect.py cv > .kde4/share/apps/kdevpythonsupport/documentation_files/cv.py (create the dir if it doesn't exist). That should give you some basic support for cv already.

Also, if you want to help, make sure to come by in #kdevelop on irc.freenode.net!

Sweet! There were a few problems with OpenCV, but nothing that couldn't be hacked away. It seems to work very well 🙂

Is there a way to tell KDevelop to look for modules in specific paths? I'd rather not have to put the cv2.py file in the same directory since my script will then import this file rather than the actual module.

Oh, also, if you have KDevelop 4.4 (and probably KDevPlatform 1.4), make sure to "git checkout 1.4" in kdev-python, because master/1.5 are not compatible with KDevPlatform 1.4 (shouldn't even compile, though, so I suppose you got that right).

I can't access the URL (403). It works for me: http://i.imgur.com/4FQnGgR.pngPlease make sure you have a recent version of kate, that the correct highlighting is selected, and if it still doesn't work report a bug with kate.

I'm building it from master, but I do not remember it ever being broken recently, so it should work. Try reinstalling all kate-related packages (kate, katepart, …)Sorry, this is not the correct place to discuss the issue — please report a bug with kate.

I just took up programming Python again. I'm a hobby programmer and I always liked Python best of all the languages I have tried. Since I'm very much into KDE and into 'integrated DE' madness, I wondered if there was Python support for KDevelop, and Google led me here.

I haven't tried it yet, but it looks just awesome! Thank you very much for your great work and effort. If it works just half as nicely as it looks, I shall be the happiest of all Python hobby programmers!