Tag: javascript

Boat Licenses

NOTE: June 17th 2016 – I received a cease and desist letter from boat-ed.com and they have fixed the way in which the script in this post circumvents the wait. I am not aware of any other way to bypass it.

Summer is fast approaching and me and a group of friends decided to take the water on some ridiculously fun jet skis. In the state of Washington, a boat license is required to operate any PWC (personal watercraft) which thankfully can be done online. As a procrastinator I put aside 30 minutes of my time the night before the excursion to do the online exam on www.boat-ed.com. I was in for a large shock when I discovered that you had to sit through all the course material on the website prior to attempting the exam; average time for completion 6-8 hours.

Time is precious

Time is precious. It’s constantly fleeting and the last thing I wanted to do was sit through 6-8 hours of instructions that I knew and are largely common sense. The final exam should be indicative of whether you know the material or not, therefore it is not necessary to force people to sit through the course (all though you can Google the answers to the exam…). As my roomate decided to sit through the course material blindly hitting next when the button becomes available as he watched tv shows on Netflix, I decided to investigate if my knowledge of Computer Science can save me!

Chrome Extensions FTW

Having some prior experience to writing Chrome Extensions and knowing more than my fair share of web services, I was able to dig into their JavaScript source files and find how they were forcing the users to sit through the course material. I will leave future blog posts perhaps detailing how one might go about “hacking” websites to reverse engineer their web services and fiddle around with it, briefly the steps were:

A constant rat race

Honestly, I don’t even use hypem.com anymore since my music taste has changed, however I am constantly trying to keep my hypem chrome extension and python download script up to date with the changes to the site. People have been overall extremely generous in their compliments which is why I keep updating the script.

Although I may not use the site anymore, I am finding their constant attempts to thwart similar extensions interesting and have enjoyed the constant rate race. I’ve documented on my blog already several of their numerous attempts to thwart similar projects and looks like they released their latest incarnation. I hope this post will serve those who are attempting to enjoy their music beyond the confines of the site helpful.

So, What’s Changed?

Previously, the link to the mp3 file was very easy to generate and scrape from their HTML page. Every song use to contain it’s unique identifiers right in the ‘div’ element of that song. One simply had to parse out the ‘key’ and ‘id’ and construct a URL to http://hypem.com/serve/play/id/key.mp3 and your content was served on a platter. They’ve now added a few more tricks to their sleeve but it’s still trivial getting access to the mp3.

Firstly, they’ve moved where the ‘key’ is stored for each song. They’ve embedded the ‘keys’ of the current page in a ‘script’ object on the site. The interior of the script object is simply JSON.

After the page loads, they read the JSON data into a javascript variable and remove the script tag from the HTML body. So you won’t be able to inspect a current page and find the code

Secondly, they’ve changed the whole request mechanism for serving songs. Before as I mentioned, the /serve/play url immediately served up the Mp3. This is no longer the case. There is now an intermediate request which returns some JSON data and then finally the URL to the Mp3 is retrieved and served!

Thirdly, this hasn’t changed but don’t forget about setting the cookie! Hypemachine checks for a AUTH cookie and without it will not authorize your get request for the song data! Simply store the cookie when you retrieve the HTML and use the same one when you make further requests. Voila!

Warning: I am relatively new to Javascript. If the solution presented below seems trivial, please forgive.

Once again, I’m isolated

I remember the frustration of hacking things into a web page due to Chrome’s Isolated World for their extensions. The problem I originally faced was that HypeMachine was storing the info for their tracks in variables that are restricted to the content scripts loaded by the extension.

Isolated worlds completely separate the JavaScript on the page from the JavaScript in extensions.

The second time around, my problem was not with accessing local data but getting access to jQuery instance used to make Ajax requests on the web page. The webpage for which I am writing the extension, refreshes its page via Ajax requests; after which it reconstructs the DOM programmatically. I could load jQuery myself into my extension however it would be a different instance! Any attempt to hook into the global Ajax event handlers would do no such good…

Isolated worlds allow each content script to make changes to its JavaScript environment without worrying about conflicting with the page or with other content scripts.

Breaking Free

It was pretty easy how I circumvented the issue the first time around,byinjecting the whole content script right into the web page! This let me write the script and have access to all the goodies of the page. This time however the solution is not doable for the new extension since I also wanted to make cross domain browser requests. My new solution would have to allow me to write the code in the extension’s script but be notified of all Ajax requests…

Javascript on the main page are bound to make requests only to the original domain from where they came (Same Origin Policy), however scripts in the extension are allowed to make cross domain requests.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

varmain=function(){

varmyEvent=document.createEvent('Event');

myEvent.initEvent('CustomEvent',true,true);

functionfireCustomEvent(){

document.body.dispatchEvent(myEvent);

};

jQuery(document).ajaxComplete(function(event,request,settings){

console.log("Ajax complete.");

fireCustomEvent();

});

};

// Lets create the script objects

varinjectedScript=document.createElement('script');

injectedScript.type='text/javascript';

injectedScript.text='('+main+')("");';

(document.body||document.head).appendChild(injectedScript);

document.body.addEventListener('CustomEvent',function(){

console.log("Received event!");

});

The solution above (somewhat described by Google here) demonstrates that the common denominator between the content scripts and the web page is the DOM. Though modification of the DOM you can pass data & propagate events. In the example above, I’ve injected my personal .ajaxComplete() handler which fires my custom event. The extension then listens for my custom event which is now technically a wrapper for the ajaxComplete event!

I know you can pass data through InnerText and by user jQuery’s data method. However is it possible to serialize the actual jQuery instance from the web page and pass it to the extension?

Recap

I’ve written previously about a Chrome Extension I had in mind for this website. The idea I had was to facilitate for people who purchase from the online store a means of making sure they are getting the best price.

The great thing about the online music store is that they specialize in electronic music and therefore most DJs and electronic music lovers look to their to find latest releases. Most of the music found on on the site is not offered by some of the more giant music stores (i.e. iTunes or Amazon) however there is some overlap!

The idea for the extension came when I noticed that many of the more mainstream songs on the store could be found cheaper (usually -50%) on other electronic stores (i.e. iTunes or Amazon).

This websiteis an online music store specializing in electronic dance music and culture.

Beatprice Chrome Extension

About time

It’s been a while since I’ve been only working on my HypeMachine extension and although it was simple (although learning how to get around the isolated world was interesting). The passion to make the HypeMachine extension was largely driven by my passion for music, and that’s why I’m excited about the new extension I’m working on!

What is @REMOVED@?

For those unfamiliar with this site, it is an online music store specializing in electronic/dance music. What’s great about the music store is that the whole spectrum of popularity range is on the store, anyone from as well known such as Tiesto to smaller local DJs in your hometown.

The current catalog is made up of more than 700,000 tracks, 160,000 releases, and 24,000 artist charts from over 120,000 artists and remixers.

The Catalyst

The songs sold on the site are ahem pretty steep. For a user who’s gotten use to either purchasing their music from larger music stores (i.e. iTunes or Amazon), any track more than 0.99$ seems a lot.

Most tracks on the site are 2.49$ for instance

The goal in writing this new extension was to quickly offer a way for someone browsing @REMOVED@ to see if it’s available on another online music store at a cheaper price. It was while browsing the store with a friend and remembering having seen the same track offered at almost -50% the cost did the drive and idea to write this extension come about.

The Extension

My goal with the extension was to inject something onto the online music stores web page (as the user browsed), to notify them that a cheaper alternative had been found. The ultimate goal is to have a list of supported music stores to check against and then present the cheapest price. The current workflow model however is:

The script will then attempt to find matching songs and present them in a popup window

Here is a quick video to show what I’ve done so far with the extension. Please ignore the Screenflow watermark. I upgraded my version and it wanted me to purchase a new license. >

Future

There are a bunch of fixes and polish I’d like to make, however I am pretty excited that the idea overall is plausible and somewhat functional! Things I’d like to change or add to the extension are:

I’d like to inject the notification or link to the cheaper site directly next to the buy icon on the webpage rather than having the user click the button in the URL tray

I’d like to support a bunch more sites (depending on whether they offer a search API) since iTunes only really sells the more notable artists. Some of the sites I’d like to include are juno.co.uk and stompy.com

Figure out a way to better perform the matching/searching for each track or release. I’m currently matching 100% on the track’s title and artist, however it would be nice to figure out a more lenient model perhaps involving regular expressions

Create a settings page where the user can insert their country of origin and other information (perhaps even how strong they’d like the pattern matching to be. The country of origin is important, since some stores price the content differently and may not be available in certain countries.

Contribute!

I’ve learnt already a great deal more about Chrome Extension development, however my Javascript skills are far from amazing. If you are a fan of the site and would like to contribute please fork my repository and contribute changes! Just like my previous extension, I’ve uploaded this extension on GitHub and would love to see feedback and contribution from others. Hopefully I have made a great base to get the ball rolling!

Ninja Update

I haven’t been using my HypeMachine Extension lately, which has caused me to fall behind in maintaining the extension. Someone had just e-mailed me recently regarding how the extension is no longer working.

I should really integrate a settings option for the extension that points people to the GitHub EPR system.

I’ve updated the Extension on my GitHub and you can follow these steps for installing the extension:

Finding usefulness out of annoyingness

My girlfriend has the habit of posting something on someone’s wall and then immediately deleting her post for a variety of reasons. Recently she had done so to my own wall. What is interesting about deleted posts are that I still received a Facebook notification in the top left corner stating that she had posted something however clicking the item does nothing.

If you’ve signed up for Facebook e-mail notifications however, you can track the notification down via the e-mail sent and it should contain a short summary of the post; allowing me to see the deleted content (which seems kind of odd). The whole process of deleting Facebook posts really seemed odd and as a result I wrote a quick web page in order to try and play around with the functionality of deleting messages.

I’ve struck copper, now I want to strike gold.

It’s funny how that of all the small software projects I’ve written and released into the public domain, non of them have garnered any attention or use quite like my HypeMachine extension.

By no means was it insanely popular, however it did manage to get roughly 3,000 downloads from the Google App store + any additional downloads I had sent via e-mail or downloaded through my GitHub.

The most shocking and intoxicating aspect of having this mildly popular extension however were the numerous e-mails I’d get thanking me for developing it and all the other miscellaneous work I had done around at annoying the people at HypeMachine (i.e. batch downloader etc..). It’s been about a month since I’ve taken down my extension from the Google App Store yet I probably still receive about 1-2 e-mails a day such as:

Hi Farid,

Thank you so very much for all your work!

Yours truly,

I’m still shocked whenever I receive an e-mail and I wonder whether people who write really popular software are just flooded with e-mails. I’m only curious as I’ve never really sent an e-mail myself to show appreciation. Maybe I should….

EDIT: This post is now irrelevant..I’ve since found the new placement of the ‘Pretty Print’ button

Making debugging easier

One of my biggest paints when developing/debugging my HypeMachine extension was that they minify all most of their scripts they use. That’s why it was a pleasant surprise to hear in Chrome 12.X they had inserted the option to de-obfuscate the scripts in the developer tools window since debugging minified JavaScript is almost impossible because it makes stepping and tracing through the code nearly impossible.

With Minification

Post De-Obfuscation/Minification

Minificiation is the process of removing all unnecessary characters from source code, without changing its functionality. These unnecessary characters usually include white space characters, new line characters, comments, and sometimes block delimiters, which are used to add readability to the code but are not required for it to execute.

Two For One Ajax Request

I believe in an attempt to stop browser extensions that inject download buttons onto HypeMachine’s website, they’ve recently changed how the songs are delivered. This is annoying once again because it means not only my extension is broken but all my scripts + Java Batch Loader as well.
Let’s take a look at what they’ve changed.

Prior the requests to the song were simply one Ajax Request

Before

1

2

3

4

5

6

varsong_url="/serve/play/"+activeItem.id+"/"+activeItem.key;

try{

currentPlayerObj[0]=soundManager.createSound({

id:'currentPlayer'+currentTrack,

url:song_url

}

Now

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

functionplayTrack(skip_prompts){

//removed bunch of code not related to grabbing the song content

r=newAjax.Request("/serve/source/"+activeItem.id+"/"+activeItem.key,{

method:'get',

onSuccess:function(transport){

varsource_data;

source_data=transport.responseText.evalJSON();

try{

currentPlayerObj[0]=soundManager.createSound({

id:'currentPlayer'+currentTrack,

url:source_data.url,

}

//removed a bunch of settings set for creating the sound object for sake of brevity

});

}

The request to “/serve/source/” + activeItem.id + “/” + activeItem.key now results in a JSON object that contains the appropriate URL and some additional data. Making retrieving song requiring two Ajax requests.

For those that are curious to look at the HypeMachine’s javascript, you will notice that it has been minified in order to save space. I’ve been suing JSBeautifier , in order to ‘un-minify’ it. A very clean and wonderfully helpful website, kudos.

What does this mean for my extension and scripts ?

Previously the buttons I injected were linked to straight MP3 but now instead when I inject the buttons I’ll have to perform an Ajax request for each one to get the proper URL to the MP3 file. I haven’t implemented this yet but hopefully it doesn’t cause the injection to be exceedingly slow and unusable. I’ll work on it sometime this week and hopefully can fix it !

The change is less severe for the scripts & java program as I just need to make another request to get the URL before downloading the file.