Subscribe

Tag: Python

Badgeyay is in development stage and is frequently seen to encounter bugs. One such bug is the Internal Server Error in Badgeyay.

What was the bug?

The bug was with the badge generator’s backend code. The generator was trying to server the zip file that was not present. After going through the log I noticed that it was because a folder was missing from Badgeyay’s directory.

I immediately filed an issue #58 which stated the bug and how could it be resolved. After being assigned to the issue I did my work and created a Pull Request that was merged soon.

Resolving the bug

With the help of extensive error management and proper code and log analysis I was able to figure out a fix for this bug. It was in-fact due to a missing folder that was deleted by a subsequent code during zipfile/pdf generation. It was supposed to be recreated every time it was deleted. I quickly designed a function that solved this error for future usage of Badgeyay.

How was it resolved?

First I started by checking if the “BADGES_FOLDER” was not present. And if it was not present then the folder was created using the commands below

if not os.path.exists(BADGES_FOLDER): os.mkdir(BADGES_FOLDER)

Then, I added docstring to the remaining part of the code. It was used to empty all the files and folder inside the “BADGES_FOLDER”. We could have to delete two things, a folder or a file.

So proper instructions are added to handle file deletion and folder deletion.

Here “os.unlink” is a function that is used to delete a file. And “shutil.rmtree” is a function that deletes the whole folder at once. It is similar to “sudo rm -rf /directory”. Proper error handling is done as well to ensure stability of program as well.

Challenges

There were many problems that I had to face during this bug.

It was my first time solving a bug, so I was nervous.

I had no knowledge about “shutil” library.

I was a new-comer.

But I took these problems as challenges and was able to fix this bug that caused the INTERNAL SERVER ERROR : 500 .

With all the major functionalities packed into the badgeyay web application, it was time to add some automation testing to automate the review process in case of known errors and check if code contribution by contributors is not breaking anything. We decided to go with Selenium for our testing requirements.

What is Selenium?

Selenium is a portable software-testing framework for web applications. Selenium provides a playback (formerly also recording) tool for authoring tests without the need to learn a test scripting language. In other words, Selenium does browser automation:, Selenium tells a browser to click some element, populate and submit a form, navigate to a page and any other form of user interaction.

First things first:
To install these package run this code on the CLI:

pipinstallselenium==2.40pipinstallnose

Don’t forget to add them in the requirements.txt file

Web Browser:
We also need to have Firefox installed on your machine.

Writing the Test
An automated test automates what you’d do via manual testing – but it is done by the computer. This frees up time and allows you to do other things, as well as repeat your testing. The test code is going to run a series of instructions to interact with a web browser – mimicking how an actual end user would interact with an application. The script is going to navigate the browser, click a button, enter some text input, click a radio button, select a drop down, drag and drop, etc. In short, the code tests the functionality of the web application.

We can also use the Phantom.js package along with Selenium for UI testing purposes without opening a web browser. We use this for badgeyay to run the tests for every commit in Travis CI which cannot open a program window.

The Elasticsearch instance in the current Open Event Server deployment is currently just used to store the events and search through it due to limited resources.

The project uses a PostgreSQL database, this blog will focus on setting up a job to create the events index if it does not exist. If the indices exists, the job will delete all the previous the data and rebuild the events index.

Although the project uses Flask framework, the job will be in pure python so that it can run in background properly while the application continues its work. Celery is used for queueing up the aforementioned jobs. For building the job the first step would be to connect to our database:

The next step would be to fetch all the events from the database. We will only be indexing certain attributes of the event which will be useful in search. Rest of them are not stored in the index. The code given below will fetch us a collection of tuples containing the attributes mentioned in the code:

We will be using the the bulk API, which is significantly fast as compared to adding an event one by one via the API. Elasticsearch-py, the official python client for elasticsearch provides the necessary functionality to work with the bulk API of elasticsearch. The helpers present in the client enable us to use generator expressions to insert the data via the bulk API. The generator expression for events will be as follows:

We will now delete the events index if it exists. The the event index will be recreated. The generator expression obtained above will be passed to the bulk API helper and the event index will repopulated. The complete code for the function will now be as follows:

Flask is a microframework for Python, which is mostly used in web-backend development.There are projects in FOSSASIA that are using flask for development purposes such as Open Event Server, Query Server, Badgeyay. Optimization is indeed one of the most important steps for a successful software product. So, in this post some few off- the-hook tricks will be shown which will make your flask-app more fast and reliable.

3.That’s it! All it takes is just few lines of code to make your flask app optimized .To know more about the module check out flask-compress module.

Requirements Directory

A common practice amongst different FOSSASIA projects which involves dividing requirements.txt files for development,testing as well as production.

Basically when projects either use TRAVIS CI for testing or are deployed to Cloud Services like Heroku, there are some modules which are not really required at some places. For example: gunicorn is only required for deployment purposes and not for development.

So how about we have a separate directory wherein different .txt files are created for different purposes.

Below is the image of file directory structure followed for requirements in badgeyay project.

As you can see different .txt files are created for different purposes

Badgeyay is a web application which takes a CSV file, an image file and an optional config.json file, and converts them into a PDF file which consist of a set of badges as per the data in the CSV and the image as its background. In order to contribute to the badgeyay repository, a contributor is expected to have some knowledge of Python Flask, HTML and CSS. An understanding of git version control system is inevitable in open source.

Flask – Web development in baby steps

First things first – Having a local copy

Sign up for GitHub and head over to the Badgeyay repository. Then follow these steps.

Go ahead and Fork the repository

Star the repository

Get the clone of the forked version on you local machine using git clone https://github.com/<username>/badgeyay.git

This snippet starts the flask server at localhost:5000 and index.html template gets rendered on visiting the root url. All the templates reside in templates folder while the static asset files are stored in static folder.

Steps:

First. we imported the Flask class and a function render_template.

Next, we created a new instance of the Flask class.

We then mapped the URL / to the function index(). Now, when someone visits this URL, the function index() will execute.

The function index() uses the Flask function render_template() to render the index.html template we just created from the templates/ folder to the browser.

Finally, we use run() to run our app on a local server. We’ll set the debug flag to true, so we can view any applicable error messages if something goes wrong, and so that the local server automatically reloads after we’ve made changes to the code.

The template consists of a base layout which is extended by the pages.

PSLab device is made useful with applications running on two platforms. One is Android and the other one is a desktop application developed using Python frameworks. Desktop application uses half a dozen of dependent libraries and they are required to be installed prior to installing the application itself.

For someone with zero or less knowledge on how to install packages in a Linux environment, this task will be quite difficult. To ease up the process of installing the desktop application in a computer, we can use a script to run specific commands which will install the dependencies and the application.

Dependencies required by PSLab Desktop app

PyQt 4.7

Python 2.6, 2.7 or 3.x

NumPy, Scipy

pyqt4-dev-tools

Pyqtgraph

pyopengl and qt-opengl

iPython-qtconsole

These dependencies can be made installed using a bash script running with root permission. A bash script will have the file extension “.sh” and a header line;

#!/bin/bash

A bash script needs to be made executable by the user himself. To do this, user needs to type a one line command in the terminal as follows and enter his password;

sudo chmod +x <Name_of_the_script>.sh

The keyword “sudo” interprets as “Super User DO” and the line follows will be executed with root permission. In other words with administrative privileges to modify system settings such as copying content to system folders.

The keyword “chmod” stands for “Change Mode” which will alter the mode of a file. In current context, the file is made executable by adding the executable property to the bash script using “+x” syntax.

Once the script is made executable, it can be executed using;

sudo ./<Name_of_the_script>.sh

An installer can be made attractive by using different colors rather than the plain old text outputs. For this purpose we can use color syntax in bash script. They are represented using ANSI escape codes and following is a list of commonly used colors;

As in any programming language, rather than using the same line in many places, we can define variables in a bash script. The syntax will be the variable name followed by an equal sign with the value. There cannot be spaces around the equal sign or it will generate an error.

GREEN='\033[0;32m'

These variables can be accessed using a special syntax as follows;

${GREEN}

Finally we can output a message to the console using the “echo” command

echo -e "${GREEN}Welcome to PSLab Desktop app installer${NOCOLOR}"

Note that the keyword “-e” is used to enable interpretation of the following backslash escapes.

In order to install the packages and libraries, we use two package management tools. One is “apt” which stands for “Advanced Packaging Tool” and the second is “pip” which is used to download python related packages from “Python Package Index”. The following two lines illustrates how the two commands can be accessed.

apt-get install python-pip python-dev build-essential -y

pip install pyqtgraph

The keyword “-y” avoids the confirmation prompt in console to allow installation by pressing “Y” key every time it installs a package from “apt”.

Open Event API Server Orders API is one of the core APIs. The permissions in Orders API are robust and secure enough to ensure no leak on payment and ticketing.The permission manager provides the permissions framework to implement the permissions and proper access controls based on the dev handbook.

The following table is the permissions in the developer handbook.

List

View

Create

Update

Delete

Superadmin/admin

✓

✓

✓

✓

✓

Event organizer

✓ [1]

✓ [1]

✓ [1]

✓ [1][2]

✓ [1][3]

Registered user

✓ [4]

Everyone else

Only self-owned events

Can only change order status

A refund will also be initiated if paid ticket

Only if order placed by self

Super Admins and admins are allowed to create any order with any amount but any coupon they apply is not consumed on creating order. They can update almost every field of the order and can provide any custom status to the order. Permissions are applied with the help of Permission Manager which takes care the authorization roles. For example, if a permission is set based on admin access then it is automatically set for super admin as well i.e., to the people with higher rank.

Self-owned events

This allows the event admins, Organizer and Co-Organizer to manage the orders of the event they own. This allows then to view all orders and create orders with or without discount coupon with any custom price and update status of orders. Event admins can provide specific status while others cannot

Registered User

A registered user can create order with basic details like the attendees’ records and payment method with fields like country and city. They are not allowed to provide any custom status to the order they are creating. All orders will be set by default to “pending”

Also, they are not allowed to update any field in their order. Any status update will be done internally thus maintaining the security of Order System. Although they are allowed to view their place orders. This is done by comparing their logged in user id with the user id of the purchaser.

Creating Attendees Records

Before sending a request to Orders API it is required to create to attendees mapped to some ticket and for this registered users are allowed to create the attendees without adding a relationship of the order. The mapping with the order is done internally by Orders API and its helpers.

In the ordering system of Open Event API Server, there is a requirement to send email notifications to the attendees. These attendees receive the URL of the pdf of the generated ticket. On creating the order, first the pdfs are generated and stored in the preferred storage location and then these are sent to the users through the email.

Generating PDF is a simple process, using xhtml2pdfwe can generate PDFs from the html. The generated pdf is then passed to storage helpers to store it in the desired location and pdf-url is updated in the attendees record.

Sample PDF

PDF Template

The templates are written in HTML which is then converted using the module xhtml2pdf.To store the templates a new directory was created at app/templates where all HTML files are stored. Now, The template directory needs to be updated at flask initializing app so that template engine can pick the templates from there. So in app/__init__.py we updated flask initialization with

The remote-lab framework of the pocket science lab has been designed to enable user to access their devices remotely via the internet. The pslab-remote repository includes an API server built with Python-Flask and a webapp that uses EmberJS. This post is a guide for users who wish to test the framework. A series of blog posts have been previously written which have explored and elaborated various aspect of the remote-lab such as designing the API server, remote execution of function strings, automatic deployment on various domains etc. In this post, we shall explore how to execute function strings, execute example scripts, and write a script ourselves.

Create an account

Signing up at this point is very straightforward, and does not include any third party verification tools since the framework is under active development, and cannot be claimed to be ready for release yet.

Click on the sign-up button, and provide a username, email, and password. The e-mail will be used as the login-id, and needs to be unique.

Login to the remote lab

Use the email-id used for signing up, enter the password, and the app will redirect you to your new home-page, where you will be greeted with a similar screen.

Your home-page

On the home-page, you will find that the first section includes a text box for entering a function string, and an execute button. Here, you can enter any valid PSLab function such as `get_resistance()` , and click on the execute button in order to run the function on the PSLab device connected to the API server, and view the results. A detailed blog post on this process can be found here.

Since this is a new account, no saved scripts are present in the Your Scripts section. We will come to that shortly, but for now, there are some pre-written example scripts that will let you test them as well as view their source code in order to copy into your own collection, and modify them.

Click on the play icon next to `multimeter.py` in order to run the script. The eye icon to the right of the row enables you to view the source code, but this can also be done while the app is running. The multimeter app looks something like this, and you can click on the various buttons to try them out.

You may also click on the Source Code tab in order to view the source

Create and execute a small python script

We can now try to create a simple script of our own. Click on the `New Python Script` button in the top-bar to navigate to a page that will allow you to create and save your own scripts. We shall write a small 3-line code to print some sinusoidal coordinates, save it, and test it. Copy the following code for a sine wave with 30 points, and publish your script.
import numpy as np
x=np.linspace(0,2*np.pi,30)
print (x, np.sin(x))

Create a button widget and associate a callback to the get_voltage function

A small degree of object oriented capabilities have also been added, and the pslab-remote allows you to create button widgets and associate their targets with other widgets and labels.
The multimeter demo script uses this feature, and a single line of code suffices to demonstrate this feature.

button('Voltage on CH1 >',"get_voltage('CH1')","display_number")

You can copy the above line into a new script in order to try it out.

Associate a button’s callback to the capture routines, and set the target as a plot

The callback target for a button can be set to point to a plot. This is useful if the callback involves arrays such as those returned by the capture routines.

Example code to show a sine wave in a plot, and make button which will replace it with captured data from the oscilloscope:

We wanted to add more features to Melix Generator web app to be able to customize Meilix ISO with more features so we thought of sending every customization we want to apply as a different variable and then use the scripts from Meilix Generator repo to generate ISO but that idea was bad as many variables are to be made and need to be maintained on both Heroku and Travis CI and keep growing with addition of features to web app.

So we thought of a better idea of creating a combined script with web app for each feature to be applied to ISO and send it as a variable to Travis CI.

Now another problem was how to send script as a variable after generating it as json do not support special characters inside the script. We tried escaping the special characters and the data was successfully sent to Travis CI and was shown in config but when setting that variable as an environment variable in Travis CI the whole value of variable was not taken as we had spaces in the script.

So to eliminate that problem we encoded the variable in the app as base64 and sent it to Travis CI and used it using following code.

For this we have to import base64 module and open the script generated in binary mode and using base64 we encode the script and using Travis CI API we send variable as script to the Travis CI to build the ISO with script in chroot we were also required to make changes in Meilix to be able to decode the script and then copy it into chroot during the ISO build.