Subscribe

Category: loklak

The Loklak search creates a website using the Loklak server as a data source. The goal is to get a search site, that offers timeline search as well as custom media search, account and geolocation search.

In order to run the service, you can use the API of http://api.loklak.org or install your own Loklak server data storage engine. Loklak_server is a server application which collects messages from various social media tweet sources, including Twitter. The server contains a search index and a peer-to-peer index sharing interface. All messages are stored in an elasticsearch index.

The site of this repo is deployed on the GitHub gh-pages branch and automatically deployed here: http://loklak.org

In this blog, we will talk about how to install Loklak_Search locally and deploying it to Surge (Static web publishing for Front-End Developers).

How to clone the repository

Sign up / Login to GitHub and head over to the Loklak_Search repository. Then follow these steps.

2. After installing angular-cli we need to install our required node modules, so we will do that by using the following command:

npm install

3. Deploy locally by running this

ng serve

Go to localhost:4200 where the application will be running locally.

How to Deploy Loklak Search on Surge :

Surge is the technology which publishes or generates the static web-page demo link, which makes it easier for the developer to deploy their web-app. There are a lot of benefits of using surge over generating demo link using GitHub pages.

We need to install surge on our machine. Type the following in your Linux terminal:

npm install –global surge

This installs the Surge on your machine to access Surge from the command line.

In your project directory just run

surge

After this, it will ask you three parameters, namely

EmailPasswordDomain

After specifying all these three parameters, the deployment link with the respective domain is generated.

Auto deployment of Pull Requests using Surge :

To implement the feature of auto-deployment of pull request using surge, one can follow up these steps:

Create a pr_deploy.sh file

The pr_deploy.sh file will be executed only after success of Travis CI i.e. when Travis CI passes by using command bash pr_deploy.sh

Codeheat is a coding contest for developers interested in contributing to Open Source software and hardware projects at FOSSASIA. Join development of real world software applications, build up your developer profile, learn new new coding skills, collaborate with the community and make new friends from around the world! Sign up for #CodeHeat here now and follow Codeheat on Twitter.

The contest runs until 1st February 2018. All FOSSASIA projects take part in Codeheat including:

Grand prize winners will be invited to present their work at the FOSSASIA OpenTechSummit in Singapore from March 23rd -25th 2018 and will get 600 SGD in travel funding to attend, plus a free speaker ticket and beautiful Swag.

Our jury will choose three winners from the top 10 contributors according to code quality and relevance of commits for the project. The jury also takes other contributions like submitted weekly scrum reports and monthly technical blog posts into account, but of course awesome code is the most important item on the list.

Other participants will have the chance to win Tshirts, Swag and vouchers to attend Open Tech events in the region and will get certificates of participation.

Team mentors and jury members from 10 different countries support participants of the contest.

Participants should take the time to read through the contest FAQ and familiarize themselves with the introductory information and Readme.md of each project before starting to work on an issue.

Developers interested in the contest can also contact mentors through project channels on the FOSSASIA gitter.

In Loklak Search the post items contain links, which are either internal or external. These links include the hashtags, mentions, and URLs. From the backend server we just received the message in the plain text format, and thus there is need to parse the plain text and render it as clickable links. These clickable links can be either internal or external. Thus we need an auto-linker component, which takes the text and render it as links.

The API of the Component

The component takes as the input the plain text, then four arrays of strings. Each containing the text to be linked. These are hashtags, mentions, links and the unshorten attribute which is used to unshorten the shortened URLs in the post. These attributes are used by the component to render the text in the appropriate format.

The next part of the logic is to generate the shard array, an array which contains each chunk, once. To do this we iterate over the Sorted Indexed array created in the previous step and use it split the text into chunks. We iterate over the text and take substrings using the indexes of each element.

I recently added multiscraper system which can scrape data from web-scrapers like YoutubeScraper, QuoraScraper, GithubScraper, etc. As scraping is a costly task, it is important to improve it’s efficiency. One of the approach is to index data in cache. TwitterScraper uses multiple sources to optimize the efficiency.

This system uses Post message holder object to store data and PostTimeline (a specialized iterator) to iterate the data objects. This difference in data structures from TwitterScraper leads to the need of different approach to implement indexing of data to ElasticSearch (currently in review process).

These are the following changes I made while implementing ‘indexing of data’ in the project.

1) Writing of data is invoked only using PostTimeline iterator

In TwitterScraper, the data is written in message holderTwitterTweet. So all the tweets are written to index as they are created. Here, when the data is scraped, Writing of the posts is initiated. Scraping of data is considered a heavy process. This approach keeps lower resource usage in average traffic on the server.

2) One object for holding a message

During the implementation, I kept the same message holder Post and post-iterator PostTimeline from scraping to indexing of data. This helps to keep the structure uniform. Earlier approach involves different types of message wrappers in the way. This approach cuts the processes for looping and transitioning of data structures.

3) Index a list, not a message

In TwitterScraper, as the messages are enqueued in the bulk to be indexed. But in this approach, I have enqueued the complete lists. This approach delays the indexing till the scraper is done with processing the html.

Creating the queue of postlists:

// Add post-lists to queue to be indexedqueueClients.incrementAndGet();try{postQueue.put(postList);}catch(InterruptedExceptione){DAO.severe(e);}queueClients.decrementAndGet();

Indexing of the posts in postlists:

// Start indexing of data in post-listsfor(Timeline2postList:postBulk){if(postList.size()<1)continue;if(postList.dump){// Dumping of data in a filewriteMessageBulkDump(postList);}// Indexing of data to ElasticSearchwriteMessageBulkNoDump(postList);}

4) Categorizing the input parameters

While searching the index, I have divided the query parameters from scraper into 3 categories. The input parameters are added to those categories (implemented using map data structure) and thus data fetched are according to them. These categories are:

a) Get the parameter– Get the results for the input fields in map getMap.

// Result must have these fields. Acts as AND operatorif(getMap!=null){for(Map.Entry<String,String>field:getMap.entrySet()){query.must(QueryBuilders.termQuery(field.getKey(),field.getValue()));}}

b) Don’t get the parameter- Don’t get the results for the input fields in map notGetMap.

// Result must not have these fields.if(notGetMap!=null){for(Map.Entry<String,String>field:notGetMap.entrySet()){query.mustNot(QueryBuilders.termQuery(field.getKey(),field.getValue()));}}

c) Get if possible- Get the results with the input fields if they are present in the index.

// Result may preferably also get these fields. Acts as OR operatorif(mayAlsoGetMap!=null){for(Map.Entry<String,String>field:mayAlsoGetMap.entrySet()){query.should(QueryBuilders.termQuery(field.getKey(),field.getValue()));}}

By applying these changes, the scrapers are shifted from a message indexing to list of messages indexing. This way we are keeping load on RAM low, but the aggregation of latest scraped data may be affected. So there will be a need to workaround to solve this issue while scraping itself.

During orientation change i.e. from portrait to landscape mode in Android, the current activity restarts again. As the activity restarts again, all the defined variables loose their previous value, for example the scroll position of a RecyclerView, or the data in the rows of RecyclerView etc. Just imagine a user searched some tweets in Loklak Wok Android, and as the user’s phone is in “Auto rotation” mode, the orientation changes from portrait to landscape. As a result of this, the user loses the search result and has to do the search again. This leads to a bad UX.

Saving state in onSavedInstanceState

The state of the app can be saved by inserting values in a Bundle object in onSavedInstanceState callback. Inserting values is same as adding elements to a Map in Java. Methods like putDouble, putFloat, putChar etc. are used where the first parameter is a key and the second parameter is the value we want to insert.

The values can be retrieved back when onCreate or onCreateView of the Activity or Fragment is called. Bundle object in the callback parameter is checked, whether it is null or not, if not the values are retrieved back using the keys provided at the time of inserting. The latitude and longitude of a location in TweetPostingFragment are retrieved in the same fashion

publicvoidonViewCreated(Viewview,@NullableBundlesavedInstanceState){...if(savedInstanceState!=null){// checking if bundle is null// extracting from bundlemLatitude=savedInstanceState.getDouble(PARCELABLE_LATITUDE);mLongitude=savedInstanceState.getDouble(PARCELABLE_LONGITUDE);// use extracted value}}

Restoring Custom Objects, using Parcelable

But what if we want to restore custom object(s). A simple option can be serializing the objects using the native Java Serialization or libraries like Gson. The problem in these cases is performance, they are quite slow. Parcelable can be used, which leads the pack in performance and moreover it is provided by Android SDK, on top of that, it is simple to use.

The objects of class which needs to be restored implements Parcelable interface and the class must provide a static final object called CREATOR which implements Parcelable.Creator interface.

writeToParcel and describeContents method need to be override to implement Parcelable interface. In writeToParcel method the member variables are put inside the parcel, in our case describeContents method is not used, so, simply 0 is returned. Status class which stores the data of a searched tweet implements parcelable.

NOTE: The order in which variables are pushed into Parcel needs to be maintained while variables are extracted from the parcel to recreate the object. This is the reason why no “key” is required to push data into a parcel as we do in bundle.

The CREATOR object implements the creation of object from a Parcel. The CREATOR object overrides two methods createFromParcel and newArray. createFromParcel is the method in which we implement the way an object is created from a parcel.

publicstaticfinalParcelable.Creator<Status>CREATOR=newCreator<Status>(){@OverridepublicStatuscreateFromParcel(Parcelsource){returnnewStatus(source);// a private constructor to create object from parcel}@OverridepublicStatus[]newArray(intsize){returnnewStatus[size];}};

The private constructor, note that the order in which variables were pushed is maintained while retrieving the values.

The status objects are restored the same way, latitude and longitude were restored. putParcelableArrayList in onSaveInstaceState and getParcelableArrayList in onCreateView methods are used to push into Bundle object and retrieve from Bundle object respectively.

@OverridepublicvoidonSaveInstanceState(BundleoutState){super.onSaveInstanceState(outState);ArrayList<Status>searchedTweets=mSearchCategoryAdapter.getStatuses();outState.putParcelableArrayList(PARCELABLE_SEARCHED_TWEETS,searchedTweets);...}// retrieval of the pushed values in bundle@OverridepublicViewonCreateView(LayoutInflaterinflater,ViewGroupcontainer,BundlesavedInstanceState){...if(savedInstanceState!=null){...List<Status>searchedTweets=savedInstanceState.getParcelableArrayList(PARCELABLE_SEARCHED_TWEETS);mSearchCategoryAdapter.setStatuses(searchedTweets);}...returnview;}

Imagine working on a large source code, and as a new developer you are not sure whether the available source code works properly or not, you are surrounded by questions like, Are all these methods invoked properly or the number of times they need to be invoked? Being new to source code and checking manually already written code is a pain. For cases like these unit-tests are written. Unit-tests check whether the implemented code works as expected or not. This blog post explains about implementation of unit-tests of Presenter in a Model-View-Presenter (MVP) architecture in Loklak Wok Android.

Setup for Unit-Tests

The presenter needs a realm database and an implementation of LoklakAPI interface. Along with that a mock of the View is required, so as to check whether the methods of View are called or not.

The LoklakAPI interface can be mocked easily using Mockito, but the realm database can’t be mocked. For this reason an in-memory realm database is created, which will be destroyed once all unit-test are executed. As the presenter is required for each unit-test method we instantiate the in-memory database before all the tests start i.e. by annotating a public static method with @BeforeClass, e.g. setDb method.

NOTE: The in-memory database should be closed once all unit-tests are executed. So, for closing the databasse we create a public static method annotated with @AfterClass, e.g. closeDb method.

@AfterClasspublicstaticvoidcloseDb(){mDb.close();}

Now, before each unit-test is executed we need to do some setup work like instantiating the presenter, a mock instance of API interface generated by using mock static method and pushing in some sample data into the database. Our presenter uses RxJava and RxAndroid which depend on IO scheduler and MainThread scheduler to perform tasks asynchronously and these schedulers are not present in testing environment. So, we override RxJava and RxAndroid to use trampoline scheduler in place of IO and MainThread so that our test don’t encounter NullPointerException. All this is done in a public method annotated with @Before e.g. setUp.

Some fake suggestion queries are created which will be returned as observable when API interface is mocked. For this, simply two query objects are created and added to a List after their query parameter is set. This is implemented in getFakeQueries method.

Lastly, the mocking part is implemented using Mockito. This is really simple, when and thenReturn static methods of mockito are used for this. The method which would provide the fake data is invoked inside when and the fake data is passed as a parameter to thenReturn. For example, stubSuggestionsFromApi method

Finally, Unit-Tests

All the tests methods must be annotated with @Test.

Firstly, we test for a successful API request i.e. we get some suggestions from the Loklak Server. For this, getSuggestions method of LoklakAPI is mocked using stubSuggestionFromApi method and the observable to be returned is obtained using getFakeSuggestions method. Then, loadSuggestionFromAPI method is called, the one that we need to test. Once loadSuggestionFromAPI method is invoked, we then check whether the method of the View are invoked inside loadSuggestionFromAPI method, this is done using verify static method. The unit-test is implemented in testLoadSuggestionsFromApi method.

Similarly, a failed network request for obtaining is suggestions is tested using testLoadSuggestionsFromApiFail method. Here, we pass an IOException throwable – wrapped inside an Observable – as parameter to stubSuggestionsFromApi.

Lastly, we test if our suggestions are saved in the database by counting the number of saved suggestions and asserting that, in testSaveSuggestions method.

@TestpublicvoidtestSaveSuggestions(){mPresenter.saveSuggestions(queries);intcount=mDb.where(Query.class).findAll().size();// queries is the List that contains the fake suggestionsassertEquals(queries.size(),count);}

Loklak search is a web application which is built on latest web technologies and is aiming to be a progressive web application. A PWA is a web application which has a rich, reliable, fast, and engaging web experience, and web API which enables us to get these are Service Workers. This blog post describes the basics of service workers and their usage in the Loklak Search application to act as a Network Proxy to and the programmatical cache controller for static resources.

What are Service Workers?

In the very formal definition, Matt Gaunt describes service workers to be a script that the browser runs in the background, and help us enable all the modern web features. Most these features include intercepting network requests and caching and responding from the cache in a more programmatical way, and independent from native browser based caching. To register a service worker in the application is a really simple task, there is just one thing which should be kept in mind, that service workers need the HTTPS connection, to work, and this is the web standard made around the secure protocol. To register a service worker

This piece of javascript, if the browser supports, registers the service worker defined by sw.js. The service worker then goes through its lifecycle, and gets installed and then it takes control of the page it gets registered with.

What does service workers solve in Loklak Search?

In loklak search, service workers currently work as a, network proxy to work as a caching mechanism for static resources. These static resources include the all the bundledjs files and images. These bundled chunks are cached in the service workers cache and are responded with from the cache when requested. The chunking of assets have an advantage in this caching strategy, as the cache misses only happen for the chunks which are modified, and the parts of the application which are unmodified are served from the cache making it possible for lesser download of assets to be served.

Service workers and Angular

As the loklak search is an angular application we, have used the @angular/service-worker library to implement the service workers. This is simple to integrate library and works with the, CLI, there are two steps to enable this, first is to download the Service Worker package

npm install --save @angular/service-worker

And the second step is to enable the service worker flag in .angular-cli.json

"apps": [ { // Other Configurations serviceWorker: true }]

Now when we generate the production build from the CLI, along with all the application chunks we get, The three files related to the service workers as well

sw-register.bundle.js : This is a simple register script which is included in the index page to register the service worker.

worker-basic.js : This is the main service worker logic, which handles all the caching strategies.

ngsw-manifest.json : This is a simple manifest which contains the all the assets to be cached along with their version hashes for cache busting.

Future enhancements in Loklak Search with Service Workers

The service workers are fresh in loklak search and are currently just used for caching the static resources. We will be using service workers for more sophisticated caching strategies like

Dynamically caching the results and resources received from the API

Using IndexedDB interface with service workers for storing the API response in a structured manner.

Using service workers, and app manifest to provide the app like experience to the user.

Direct URL is a web address which redirects the user to the preset customized media wall so that the media wall can directly be used to be displayed on the screen. Loklak media wall provides direct URL which has information related to customizations set by the user included in the web address. These customizations, as the query parameters are detected when the page is initialized and actions are dispatched to make changes in the state properties, and hence, the UI properties and the expected behaviour of media wall.

In this blog, I would be explaining how I implemented direct URL in loklak media wall and how customizations are detected to build on initialization of component, a customized media wall.

Flow Chart

Working

Media Wall Direct URL effect

This effect detects when the WALL_GENERATE_DIRECT_URL action is dispatched and creates a direct URL string from all the customization state properties and dispatches a side action WallShortenDirectUrlAction() and stores direct URL string as a state property. For this, we need to get individual wall customization state properties and create an object for it and supply it as a parameter to the generateDirectUrl() function. Direct URL string is returned from the function and now, the action is dispatched to store this string as a state property.

Generate Direct URL function

This function generates Direct URL string from all the current customization options value. Now, keys of the object are separated out and for each element of the object, it checks if there is some current value for the elements and it then first parses the value of the element into URI format and then, adds it to the direct URL string. In such a way, we are creating a direct URL string with these customizations provided as the query parameters.

Creating a customized media wall

Whenever the user searches for the URL link on the web, a customized media wall must be created on initialization. The media wall component detects and subscribes to the URL query parameters using the queryParams API of the ActivatedRoute. Now, the values are parsed to a required format of payload and the respective actions are dispatched according to the value of the parameters. Now, when all the actions are dispatched, state properties changes accordingly. This creates a unidirectional flow of the state properties from the URL parameters to the template. Now, the state properties that are supplied to the template are detected and a customized media wall is created.

Now, the state properties are rendered accordingly and a customized media wall is created. This saves a lot of effort by the user to change the customization options whenever uses the loklak media wall.

MVP stands for Model-View-Presenter, one of the most popular and commonly used design pattern in android apps. Where “Model” refers to data source, it can be a SharedPreference, Database or data from a Network call. Going by the word, “View” is the user interface and finally “Presenter”, it’s a mediator between model and view. Whatever events occur in a view are passed to presenter and the presenter fetches the data from the model and finally passes it back to the view, where the data is populated in ViewGroups. Now, the main question, why it is so widely used? One of the obvious reason is the simplicity to implement it and it completely separates the business logic, so, easy to write unit-tests. Though it is easy to implement, its implementation requires a lot of boilerplate code, which is one of its downpoints. But, using Dagger2 the boilerplate code can be reduced to a great extent. Let’s see how Dagger2 is used in Loklak Wok Android to implement MVP architecture.

Adding Dagger2 to the project

Implementation

First a contract is created which defines the behaviour or say the functionality of View and Presenter. Like showing a progress bar when data is being fetched, or the view when the network request is successful or it failed. The contract should be easy to read and going by the names of the method one should be able to know the functionality of methods. For tweet search suggestions, the contract is defined in SuggestContract interface.

A SuggestPresenter class is created which implements the SuggestContract.Presenter interface. I will not be explaining how each methods in SuggestPresenter class is implemented as this blog solely deals with implementing MVP. If you are interested you can go through the source code of SuggestPresenter. Similarly, the view i.e. SuggestFragment implements SuggestContract.View interface.

So, till this point we have our presenter and view ready. The presenter needs to access the model and the view requires to have an instance of presenter. One way could be instantiating an instance of model inside presenter and an instance of presenter inside view. But, this way model, view and presenter would be coupled and that defeats our purpose. So, we just INJECT model into presenter and presenter into view using Dagger2. Injecting here means Dagger2 instantiates model and presenter and provides wherever they are requested.

ApplicationModule provides the required dependencies for accessing the “Model” i.e. a Loklak API client and realm database instance. When we want Dagger2 to provide a dependency we create a method annotated with @Provides as providesLoklakAPI and providesRealm.

If we look closely providesLoklakAPI method requires a Retrofit instance i.e. a to create an instance of LoklakAPI the required dependency is Retrofit, which is fulfilled by providesRetrofit method. Always remember that whenever a dependency is required, it should not be instantiated at the required place, rather it should be injected by Dagger2.

After preparing the source to provide the dependencies, it’s time we request the dependencies.

Dependencies are requested simply by using @Inject annotation e.g. in the constructor of SuggestPresenter @Inject is used, due to which Dagger2 provides instance of LoklakAPI and Realm for constructing an object of SuggestPresenter.

@Inject can be used on the fields also. When @Inject is used with a constructor the class also becomes a dependency provider, this way creating a method with @Provides is not required in a Module class.

Now, it’s time to connect the dependency providers and dependency requesters. This is done by creating a Component interface, here ApplicationComponent. The component interface defines where are the dependencies required. This is only for those cases where dependencies are injected by using @Inject for the member variables. So, we define a method inject with a single parameter of type SuggestFragment, as the Presenter needs to be injected in SuggestFragment.

Imagine an Activity popping out of nowhere suddenly in front of the user. And even more irritating, the user doesn’t even know whether a button was clicked. Though these are very small animation implementations but these animations enhance the user experience to a new level. This blog deals with the animations in Loklak Wok Android, a peer message harvester of Loklak Server.

Activity transition animation

Activity transition is applied when we move from a current activity to a new activity or just go back to an old activity by pressing back button.

In Loklak Wok Android, when user navigates for search suggestions from TweetHarvestingActivity to SuggestActivity, the new activity i.e. SuggestActivity comes from right side of the screen and the old one i.e. TweetHarvestingActivity leaves the screen through the left side. This is an example of left-right activity transition. For implementing this, two xml files which define the animations are created, enter.xml and exit.xml are created.

NOTE: The entering activity comes from right side, that’s why android:fromXDelta parameter is set to 100% and as the activity finally stays at extreme left, android:toXDelta parameter is set to 0%.

As the current activity, in this case TweetHarvestingActivity, leaves the screen from left to the negative of left. So, in exit.xml the android:fromXDelta parameter is set to 0% and android:toXDelta parameter is set to -100%.

Now, that we are done with defining the animations in xml, it’s time we apply the animations, which is really easy. The animations are applied by invoking Activity.overridePendingTransition(enterAnim, exitAnim) just after the startActivity method. For example, in openSuggestActivity

Touch Selectors

Using touch selectors background color of a button or any clickable can be changed, this way a user can see that the clickable responded to the click. The background is usually light accent color or a lighter shade of the icon present in button.

There are three states involved while a clickable is touched, pressed, activated and selected. And a default state, i.e. the clickable is not clicked. The background color of each state is defined in a xml file like media_button_selector, which is present in drawable directory.