random thoughts on PureBasic development

It’s already the end of 2017 and no post has appeared this year on the blog. Syed Ibrahim (@TI-994A) brings up the idea and the post to fill the gap ! Have a nice reading and see you in 2018, the 20th year of PureBasic existance (I can’t really believe it) !

A BRIEF HISTORY
1998 to 2018

The coming new year is a very special one for PureBasic, as it marks twenty years since the language first appeared, in 1998. So, it seems fitting that the last blog entry of 2017 be dedicated to this glorious history, and its advancements to date.

PureBasic was first commercially available in 1998, for the Commodore Amiga, and shortly after that, in 2000, for the PC. It uses a predominantly BASIC-like syntax, with some similarities to C and other mainstream languages. Built upon of an assembly language compiler, it has the unique ability to include inline assembly code right alongside its own PureBasic code. Needless to say, this also results in very small compiled binaries, and blazing-fast execution of the compiled executables.

PLATFORM EXPANSIONS
From Amiga, to Windows, to Linux, to MacOS, and Beyond

In 2001, a Linux version was added to the line, and a few years after that, in 2005, a version for Apple’s PowerPC. The PureBasic Suite, as it stands today, was completed in 2008, when another version for Apple’s x86 operating system was added, making it the only non-RAD independent programming language in the world to support all the major desktop platforms.

The language has no real parallel as it is not a rapid application development tool (RAD). It is a true platform-native compiler that produces independent, standalone applications that do not require any additional supporting libraries, frameworks, or virtual machines. The compiled binaries remain very small and tight as no such dependencies are bundled within the final packages either. PureBasic compiled programs would run on the intended platforms right out of the box. Simply copy the compiled executable and drop it into any Windows, MacOS, or Linux PC, and it will run, and run fast.

UNPARALLELED PERFORMANCE
A true, native, multi-platform compiler

Besides old-school powerhouses like Assembly and C, all other popular programming languages today fall victim to burdensome dependencies. Java applications will not run on machines that do not have their corresponding JVMs (Java virtual machines) pre-installed, and .NET programs require compatible versions of the framework in order for their applications to work. Ruby and Python are interpreted scripting languages, and Swift and Objective-C are purpose-built for the MacOS. RAD tools like Xamarin and Xojo have no place in this arena as they tend to bundle massive libraries within the final binaries, resulting in bloated and sometimes slow executables.

This leaves PureBasic as the sole forerunner in its class, as a truly multi-platform compiler, that produces native executable binaries which are small and fast, without requiring any additional resources whatsoever.
Over the years, the language has undergone many modifications, as the developers attempt to keep it updated and relevant with the latest trends and technologies. It has introduced a plethora of tools and libraries to aid the programmer in simplifying development tasks, without sacrificing performance. While some upgrades have come at the price of legacy-compatibility, the end-product leaves nothing to be desired. PureBasic gets better with every new iteration, and the diligence and dedication of the development team to bug-fixes and feature implementations are highly commendable.

PureBasic also conforms to the industry’s strictest policies, with its dual-release model. The compilers are released in beta and stable LTS (long-term support) versions, which allow for legacy developments to continue unabated. Furthermore, every version of PureBasic, from publication year 2000, is still available for download from the site’s unique “museum” page. That’s pure dedication.

The team has recently started development on a web-platform tool, known as SpiderBasic, which is syntax-compatible with PureBasic, and also as easy to use. That will definitely hold many new promises for the future as well.

FEATURE CHARTPlatforms, Syntax, Toolkits, Designers, and true 64-bit

– simple, readable, terse syntax
– purely procedural paradigm with true multithreading
– supports both 32-bit and true 64-bit environments
– true cross-platform compilation of unmodified code on Windows, MacOS, Linux
– includes comprehensive libraries usable on all platforms
– includes 2D & 3D game and multimedia engines for all platforms
– implements native UI of each OS without any external libraries
– includes visual form designer that automatically generates UI code
– OS API functions, constants, structures accessible without import
– able to import and utilise external third-party libraries
– able to utilise inline assembly for speed-critical code
– compiles to very small, very fast binary executables
– executables require absolutely no dependencies, frameworks, or VMs
– executables can be run out of the box without any installation
– lifetime license on single purchase that includes all three platforms

LOYALTY LICENSING
A Lifetime of Power Performance

Last, but not least, is the unbelievable cherry on this cake. The entire PureBasic Suite, which includes the compilers for Windows, MacOS, and Linux, costs just €79.00, and comes with LIFETIME upgrades! That’s right. Even the first licensee of the very first PureBasic version in 1998 is still eligible for upgrades to the latest current versions today, absolutely free-of-charge.

Now, that’s one feature that no other commercially licensed language has ever offered.

Thank you, TEAM PureBasic, and specifically its commander-in-chief, Mr Frédéric Laboureur, for your diligence and generosity through the years.

Since PureBasic 5.40 LTS, GTK3 is the default GUI toolkit used by PureBasic on Linux. It seems like a no brainer update for our users, but the fact is than GTK3 doesn’t fit well the PureBasic GUI library and we have several bad hacks here and here to make it work at all. We decided from the start in PureBasic to handle pixel perfect sizing of the GUI elements to allow maximum flexibity. For example, we added a Dialog library on top of it, which adds automatic layout depending of the font size, and allow to design complex resizable windows. It’s nice, but not mandatory, and depending of your needs, you will or will not use it. Up to GTK2, it was the same: you could define pixel perfect size and the toolkit would honor it. Now, it’s a different story: the toolkit can decide if the size you gave programmatically is too small or not. It’s a big change, as you could end up with a deformed GUI and more important, the running program doesn’t reflect the code entered by the programmer. You may wonder why such change has been implemented ? My guess is to force use of layout for everything, so the toolkit can support themes more efficiently and without change from the programmer. It may be true, but it’s one of the first toolkit to do so and forcing the programmer hand isn’t always a great thing, especially when it removes flexibility: you can write a layout engine using pixel perfect sizing, but you can’t do pixel perfect sizing with a layout engine. And for some specific apps, running on proprietary OS, it can be an issue as theming is irrelevant.

As Fred has explained here, supporting both the creation of ascii and unicode executables in the compiler is becoming a burden and we would like to end support for ascii compilation in order to streamline the library code and make it easier to maintain the PureBasic package in the future. However, the above thread has shown that people wish for a longer transition period, and we would like to honor this wish.

So we decided to remove the ability to compile in ascii mode in the PB version that follows after the next LTS version (that is, the LTS version coming after the 5.2x LTS cycle ends).

There are no exact dates for the releases, but the timeline looks like this:

The current 5.2x LTS version will be supported until at least 09/17/2015

After that date, a new LTS version will be released with support for 2 years. This version will still have full ascii compilation support.

The first non-LTS version released after the next LTS cycle starts will have no ascii compilation support

This means that the first non-LTS version without ascii support will be released in about a year. By staying with LTS releases, you have the ability to use an ascii mode compiler with a fully supported PB version for at least another 3 years starting from today. This should give enough time for a smooth transition.

No changes will be made to the language or data types (The .c, .a and .u as well as the pseudo-types and library commands will remain as they are). The only difference is that the “compile in unicode mode” switch in the compiler will be permanently set to “on”.

Please understand that we do listen to the concerns voiced in the discussion thread and that we do not make this decision lightly. I think we have a quite good track record of supporting older technology as is evidenced by the very long support for Windows 9x which just recently ended and by the fact that we still support Windows XP even after MS dropped all support for it. However, in order for us to be able to introduce new technologies (like x64 compilation in the past, or now support for the web with Spiderbasic), we simply cannot support old stuff indefinitely. In order to move forward into the future, we have to leave some stuff behind from time to time. We hope you can understand that.

That’s it, PureBasic 5.30 final version is out ! The beta phase was longer than usual and even if it’s holiday and summer time, it’s not a valid excuse ;). So let’s jump a bit in PureBasic development internals, so you can better understand why.

First of all, the IDE is now fully unicode. It looks like a no brainer as in PureBasic it’s more or less a checkbox to enable to have your program working in unicode, right ? Well, not this time. The (new) IDE is around for almost 10 years now (09/21/2005) and it grew out of proportion by including tons of features. We heavily use raw pointers to process string buffers quickly, and when switching to unicode, all the strings went to UCS-2 (16-bit character). The stuff here is Scintilla is working internally in UTF-8, so all our comparisons between PB strings and Scintilla internal
didn’t work anymore as one side was UTF-8 and the other UCS-2. Also, all the Scintilla commands which needed a buffer as parameter needed to be changed as we mostly used a pointer to a PB string for this. We also had API calls here and here which required some adaptation work, and there was no compiling error or runtime warning, so usually a nice crash was awaiting us when missing one and reaching this code section. To be honest, it was quite a bit depressing to see a solid IDE get back to a shacky shape. This looks brighter now, and the unicode switch allowed us to fix long standing IDE issues which is cool. It was a needed update, especially for the internal debugger which can now handle unicode programs. There is will be probably some quicks left but it should be ironed out in future versions.

We also made important internal compiler changes to enable runtime serialization trough the XML and new JSON lib. It doesn’t seem like a big deal from the user point of view, but actually it is. What is runtime serialization ? In a nutshell: you have your nice linked list of complex structured elements (which can contain dynamic map, array etc.) and you want to save it, so when you quit your program, you wont loose these data. To do that, you had mostly only one way until now: walk the list, write all the fields of your structure to either a file or a database. It can be time consumming to write this, and if you add a new field in your structure, you have to update both your load and save routines. With 5.30 serialization functions, you can do that with only one command. The compiler will flag your structure as a runtime one so it gets all needed information (like fieldname, fieldtype etc.) and build special information table to allow full recursive analysing and dumping of the structure data. We made this change generic in the compiler, so any command (even user libs) can use this feature. To allow this, we needed to support ‘shadow parameters’ in PB function call. Basically, when we declare a command parameter as “Runtime Structure” like in InsertJSONStructure(), one hidden parameter is added to the function call to add more information to the structure. The same is done to when a runtime list, map or array is need like in InsertJSONList/Map/Array(). Adding such a core feature in the compiler is always tricky and can creates new hard to track issues (like stack mis-aligment), and that’s why the IDE didn’t started anymore on some Windows x64 computers.

And finally, a word about LTS (Long Time Support). The 5.30 is the first version which is released in parallel than an LTS version. That means than we had to update our build servers and deployment tools to handle 2 concurrents versions (when building and when releasing). The website also needed some update to handle this properly. It’s small things but it does add-up and it in the end, it takes time to get everything right and properly working.

As a side note, all the doc are now updated to the new format in every languages and I would like to thank Andre and Mesa for the huge work done on this. A programming langage without a good doc is not a good programming langage.

I hope you didn’t fell asleep when reading this, enjoy the new version !

The first public beta version of the upcoming PureBasic version is available for download on your download account.

Some notable new features include:

A new JSON library including functions to serialize complex data structures directly from PB code to JSON and back. The same functionality has also been added to the XML library.

New functions for the 3D engine.

The IDE and debugger are now full unicode programs

Improvements for the IDE such as a “plain-text” editing mode, the highlighting of repeated occurrences of the current selection, a new ‘Issues’ tool to track TODOs in the source code and more.

A new /PREPROCESS compiler flag which will generate a PB code with all compiler directives and macros resolved.

… and much more!

The english documentation is already up to date for the new features, so you can find a description and examples there.

This version also includes some changes that may break compatibility with existing code:

The (X)IncludeFile and IncludeBinary have changed their behavior for relative paths.

The #PB_Event_SizeWindow and #PB_Event_MoveWindow events are no longer reported in realtime in the main event loop. This is to reduce problems with flickering on resize. To get realtime events you have to use the BindEvent() function and a callback in the future.

CreateXMLNode() has a new mandatory parameter.[/list]

This is regular release which means it brings new features and will have the regular support cycle. The 5.2x LTS release will continue to be maintained and a 5.23 LTS release is planned for shortly after this release becomes final.

Here is the full list of changes:

Added: New "Issue" tool for IDE to build todo/issue lists easily from comments.
Added: JSON library
Added: ParseXML(), ComposeXML(), InsertXMLArray/List/Map/Structure(), ExtractXMLArray/List/Map/Structure() to XML lib
Added: OpenGLGadget() with native opengl commands ande constants support for Windows, OSX and Linux
Added: ExamineRegularExpression() and related commands to process regex matches step by step (with group support)
Added: ClipOutput(), UnclipOutput(), SetOrigin(), GetOriginX(), GetOriginY()
Added: GetWindowData(), SetWindowData()
Added: AllocateStructure(), FreeStructure()
Added: #PB_Default support to WindowsBound() to reset min/max size
Added: 'Format' parameter to Read/WriteProgramString(), WriteProgramStringN() and ReadProgramError()
Added: /PREPROCESS compiler flag to create a big single source with all macros, compilerif and file include resolved. Can be combined with /COMMENTED to get the original source with comments as well.
Added: optional '#Server' parameter to NetworkServerEvent() to check events only on a specific server
Added: #PB_String_NoZero flag support to PokeS() to avoid writting the ending null character
Added: #PB_Enumeration support for Defined()
Added: Optional 'Type' parameter to CreateBillboardGroup()
Added: BillboardGroupCommonDirection(), BillboardGroupCommonUpVector()
Added: #PB_Entity_MinVelocity and #PB_Entity_ForceVelocity to SetEntityAttribute()
Added: SetMaterialAttribute() with #PB_Material_DepthCheck and #PB_Material_DepthWrite constants
Added: #PB_Material_DepthCheck support for GetMaterialAttribute().
Added: Engine3DStatus() with these constants: #PB_Engine3D_NbRenderedTriangles, #PB_Engine3D_NbRenderedBatches
#PB_Engine3D_CurrentFPS, #PB_Engine3D_MaximumFPS, #PB_Engine3D_MinimumFPS, #PB_Engine3D_AverageFPS, #PB_Engine3D_ResetFPS
Added: #PB_Absolute / #PB_Relative support to CameraDirectionX/Y/Z(), CameraX/Y/Z(), EntityX/Y/Z(), LightX/Y/Z(), LightDirectionX/Y/Z()
BillBoardGroupX/Y/Z(), NodeX/Y/Z(), ParticleEmitterX/Y/Z() and FetchOrientation()
Added: ParticleSpeedFactor(), DisableParticleEmitter()
Added: GetEntityCollisionMask(), GetEntityCollisionGroup(), SetEntityCollisionFilter()
Added: WaterHeight(), FreeWater()
Added: Fully unicode IDE
Added: Highlighting of repeated occurrences of the currently selected word in the IDE
Added: Plain-text editing mode to edit non-PB files in the IDE
Added: 'Issues' IDE tool to collect and display TODO/FIXME markers inside the code
Added: Ctrl+E and Ctrl+Shift+E shortcut to align/shift comments in a selected code block
Added: Ctrl+M and Ctrl+Shift+M shortcut to select the current code block (repeated presses select the next code block)
Added: PopupMenu to IDE error log for clear/copy operation
Added: %HOME and %PROJECT to IDE tool commandline options, added PB_TOOL_Project to available env vars
Added: Automatic code indentation can align comments at the end of code lines in the IDE
Added: AutoComplete remembers last selection for Structure/Module AutoComplete
Added: Context sensitivity for current module/procedure for variable display and expression eval in the debugger
Changed: FormatXML() with #PB_XML_ReFormat no longer adds newlines inside single-line elements for a more readable output
Changed: DeleteElement() now returns the data pointer to the new current element (if any)
Changed: SetXMLAttribute() to accept newline characters in attributes (will be encoded as character entities)
Changed: Added a mandatory "name" parameter to CreateXMLNode() as some parser needs it at node creation time
Changed: The way (X)IncludeFile and IncludeBinary works: it's now relative to the file which contains these statements (which is easier to handle)
Changed: #PB_Event_SizeWindow and #PB_Event_MoveWindow are no more realtime on Windows, use BindEvent() to get real time update. It should fixes ugly flickering when realtime resizing on Windows.
Changed: DataSection label within Procedure are now local labels.
Changed: ASM local label prefix has been changed from "l_" to "ll_" ("ll" for local label), to avoid possible clash with main labels.
Changed: #PB_LinkedList constant has been renamed to #PB_List for better consistancy
Changed: Removed the "Billboard" parameter from AddBillboard() as it was not used. Now returns the new billboard index.
Changed: Updated Scintilla to version 3.4.2
Removed: MaterialDepthWrite() (replaced by SetMaterialAttribute())
Removed: CountRenderedTriangles() and Engine3DFrameRate(), replaced by Engine3DStatus()

Since PureBasic 5.20, we introduced a new LTS suffix after the version name. Some of you wondered what it means, and what was the purpose of it. Basically, a LTS (Long Term Support) version will be supported for 2 years trough bug fixes release without any command or compiler modification. If you need stability in your development, you can choose to stick to an LTS version and then when the support is over go to the next one. We will provide clear migration path when going from one LTS version to another one. The advantage here is you don’t have to worry about code adpation when developping and still have stabilisation releases to have a rock solid development enviromnent. The disadvantage is you don’t get the new features of new major release until the next LTS version is out. That’s a balance to find and a decision to make, but now the choice is yours !

PureBasic uses a commandset built on top of the OS API, using generic commands to abstract it from you, the developer. One of the most daunting (and boring) task for us is to follow underlying API changes. A famous example was Cocoa support replacing old Carbon API on PureBasic for OS X. Another one is the DirectX11 support for the Sprite/Screen library on Windows. Linux isn’t left behind, with the recent GTK3 support for the whole commandset. All this requiers a lot of work, new knownledge to learn, and tons of testing, just to provide functionnalities which already work in the current API. From the PureBasic developer point of view, there is nothing fancy as the commands works as usual, only the implementation under the hood has been completely reworked. That’s one of the great strength of PureBasic: you can upgrade your program without worrying it about the technology used behind by just using a new version. OK, you will probably have some adjustements to do here and here, but it’s really nothing compared to migrate a whole program from one API to another.

Which brings us to a somewhat recurrent issue with PureBasic upgrade: commands are sometimes renamed, changed, or even removed. It often makes the developer angry as a working code needs to be modified to run again with the new version, and we don’t like to change working code. Agreed. But I would like to explain why it happens, and why it is important for PureBasic future. One important thing to understand: we don’t change the commandset unless it’s really necessary. It’s not our goal to mixup the whole stuff only to bother you. That said, the technology is always evolving and an high level programming language like PureBasic needs to follow the evolution as well. If not, it will be left behind, and a new language, using more modern API will take over. Let me use the sprite library as an example to illustrate my thoughs:

– API designed on Amiga in 1998 with no 3D acceleration. Was quite simple: sprite only, no zoom, no rotation, and many software only routines.
– Later, with DirectX7 on Windows, we got some 3D accelerated functions for some cards, so we added the ‘sprite3d’ library to support it. It was optional, as it didn’t work everywhere. That made the sprite lib a bit more hard to use, as you had to create 3D sprite using a real sprite with texture mode. DirectX7 mainly worked with 2D acceleration using DirectDraw, so we couldn’t switch to full 3D
– Nowaday DirectX9/DirectX11/OpenGL offers much better support for 3D only sprites, and 2D acceleration is not supported anymore. So all the sprites are now natively in 3D, even if we use it in 2D mode. What the purpose of having a ‘sprite3d’ lib as regular sprites are 3d anyway ? This one was obsolete, so we had to remove it to cleanup the library and have easier sprite handling, for you and for newcomers.

A new language won’t even think about separating sprite in 2 libs as it makes no sens: all is 3D now, even cheap phones handle its GUI in 3D mode. That was legacy and if we want to have PureBasic around in the future, we have to think ‘new’, and adapt the current commandset to new technology, even if it slightly breaks code. If we stick with older functionalities/design for compatibility sake, we will be slowing dying. Indeed, old PureBasic versions are available in the museum section of your online account, if you just need to do maintainance on a software.

Since the last beta version (5.20 beta8) the IDE finally has support for Modules as well (AutoComplete, QuickHelp, ProcedureBrowser etc). Please test these features well, as it was quite a change in the IDE to implement this. Now, to get the best experience when using the IDE and Modules together, there are some simple rules that have to be followed. I will explain them below. But first, let me explain why these limitations exist.

As the AutoComplete feature became smarter and more helpful in past versions, it had to grow from a simple scanning of the source for strings into a full grown analysis of the source code. This is especially required for modules, because the IDE needs to know what procedures/constants/structures are exposed from each module in order to present the correct choices in AutoComplete. So the IDE in essence needs to understand your code. The best way to fully understand code is to compile it. The IDE does not actually compile the code for the following reasons:

Speed: The AutoComplete feature cannot introduce delays because that would make it unusable. So compiling the entire code from beginning to end is not an option.

The code is usually not compilable anyway: Remember, the purpose of the IDE is to edit code. While you edit a line, it will be in a state that cannot be compiled most of the time. In fact, if you look at all characters typed in a source code, the instances in which that typed character puts the source in a compilable state are rather rare compared to those that put it in a state with errors (from a compiler perspective).

There are IDEs that actually have a builtin compiler that can tolerate this and still compile the code. Eclipse is such an example. However, writing something like that is quite complex and is actually harder for PB because of features such as macros and compiler directives.

Instead, the IDE tries its best to (quickly) understand the code without a full compilation. For that, it parses the code and extracts all information that can be understood without context (from a single line of code) into an indexed form for fast search. This organization has the benefit that if a line of code is changed, only this line needs to be parsed again and not the rest of the code which is quite fast. Then when the type of a variable is needed or something like that, the IDE searches through this indexed data to put it in the proper context to answer the question. So it is not a full compilation, but rather a “fuzzy” analysis of the code. In essence, the IDE collects the information it can understand, and stuff that only a compiler would know (like what an expanded macro would look like) is ignored.

So how does this affect Modules? One thing that the IDE does not do is resolve which source file is included where in the final program. It only looks at each source file individually. This was no big limitation before modules existed, but with modules it means you have to follow some rules so the IDE understands where a module begins and ends and what it contains.

These are the rules:

The entire DeclareModule/EndDeclareModuleblock should be in the same source file.

The entire Module/EndModuleis best also written in a single file, but it does not have to be the same one as the one that contains the DeclareModule

At least the file containing the DeclareModuleblock should be included in the Project and scanned for AutoComplete. You don’t have to include the actual module implementation since the stuff in there is not visible on the outside anyway.

If you use UseModule, all modules that are open within a source file should be opened within that source file. The best was is simply to have the UseModulecalls right at the top.

Don’t hide the essential module commands behind macros. The IDE does not expand macros so it will not know that the keywords are in there.

The explanation is rather simple. Consider this code:

DeclareModule Foo
XIncludeFile "FooDeclaration.pb"EndDeclareModule

Since the IDE only looks at each source file individually, it does not know that the contents of FooDeclaration.pb are part of the module declaration. So the IDE sees an empty Foo module, plus a file FooDeclaration.pb with stuff that is outside of any module.

In the same way, if you have a UseModulecall outside of the current source file, the IDE does not know that stuff from the opened module is available while you edit your file and therefore cannot help you with AutoComplete in this case.

Other languages enforce such rules on the compiler level (one class per file etc). We chose to not limit the compiler in this way so you can organize your code as you like. And you don’t have to follow these rules to use the IDE. It just cannot be as helpful as it could be if you don’t. Anyway, in my opinion these rules also make sense from a code organization standpoint so i don’t think it is that hard to follow them.

Ironic isn’t it ? Nowaday, anti-virus are becoming more and more intrusives, and ‘thanks’ to heuristic approaches, the false positives raise a lot, while eradication or real virus are less and less common. When was your last real virus detection ? Mine was may be 5-6 years ago. Since then, i got many alerts, but all were just dumbs. Microsoft has made giant progress to counter virus spreading and execution with modern iterations of Windows (see ASLR, DEP, user mode, etc.).

My concern about anti-viruses is the famous heuristic approach which often flag legit PureBasic programs as contamined. I don’t know exactly how works an anti-virus, but I bet it builds a database with some code patterns found in real virus, and if this code pattern is found in an executable it will raise an alert. And here is really the problem: if PureBasic is used to make a trojan or a virus, then chances are high than the code pattern will be actually a purelibrary command, meaning than every other PureBasic program using this command will be flaged as well. It’s really a poor detection mechanism and I can’t understand why anti-virus maker don’t try to find other methods to detect threats. Since several years now, PureBasic programs (and even the official IDE) are regulary flagged as virus with no reasons, and such threads often popup on officials forums. I can imagine the face of a potential customer wanting install the demo version of PureBasic when a big alert saying ‘Warning Trojan.bigvirus.1337’ popup.

So what to do ? Actually a very few things can be done:
– always send false positives to the your anti-virus maker. The more they get, bigger are the chances they will improve their detection routines.
– change your anti-virus. well, some are better than other. If it doesn’t work with PureBasic, you can consider it as a bad one 🙂