Hi! In this post I would like to introduce you to an application called Hightail.

Hightail is an automatic tester for programming contests such as CodeForces rounds. It will parse the problem statement, extract sample test cases (inputs and outputs) from it, and verify the correctness of your program against them. It is built to provide maximum automation and to relieve the contestant as much as possible.

A list of Hightail's features:

parsing problem statements

parsing entire contests (autoloading all problems)

scheduling contest parsing ahead of time

ability to comfortably view, edit and add test cases

handling of all verdicts: WA, TLE, RE, even AC

customizable time limits

ability to create in/out files in your working directory

detection of floating point values (comparing 0.1 vs. 0.10 does not give WA)

easy-to-use UI, keyboard shortcuts

resilient: it is multi-threaded so it will not let your program hang it; it will withstand large amounts of output from your program

Hightail started out as a simple program that I wrote for myself quite some time ago, then it became a small open source project when some other people contributed code to it. I think that it is now sufficiently developed that it can be released to the public, i.e. announced publicly here, rather than being hidden somewhere on the net, possible to find only if you already know the name. (It's probably long overdue, actually.)

If you see bugs, have feature suggestions etc., please use GitHub to its fullest extent :) And look at the readme — it has a tutorial on how to contribute to Hightail.

First,thank you for the awesome tool. Second, one small advise, you may add this feature in the future version. Some problems just allow multiple correct answers, with some words like "if multiple answer exists, print any of them". If this tool can sense this and remind me "Though your answer differs with the standard answer, but it might be correct.",this tool will be more lovely. Thanks again.

To get a fairly up-to-date version — in the form of a .jar file which is ready to run if you have Java (JRE) installed — visit Downloads. Just save it somewhere on your computer and run it from there. It will create a config file in its directory.

Well, I think it's right — the executable does not exist (unless you like to name your executables like A.exe even under Linux :) ). Point it to the compiled executable file (if you compile it with g++ with no options, it is usually called a.out).

This is more of a philosophical question :) I think that testing on the samples is a nuisance and a distraction which is better eliminated so that you can focus on the problems. It can actually save a lot of time sometimes when you don't get the program right from the start, but every time you correct it, there is one testcase (not always the same one) which doesn't pass... and so on. The time you gain will not only give you slightly more points, but sometimes you might be able to solve one more whole problem in the last minute ;)

Of course it's just that. It won't actually make you a better coder in any way.

My question is that why this application use executable file to test our code ? Why not main source code ? Coz ,when we debug our code, we need to update our executable file by compilation . But if this app uses our source file directly, we just need to save our new code only.

It's a design decision which might change in later versions (but probably not anytime soon). The downside is that the configuration (and the entire application) would be made more complicated. And when there are compilation errors, we would have to display them somewhere, and we couldn't do it as nicely as an IDE would (pointing the user to the line with the error etc.). Also, should we display warnings or not? The assumption is now just that the user will compile the source code themselves.

Hi Jakub, I tried using Hightail with java language, but it's not working. I think, I have made some path configuration or setting mistake. For your info, I am adding link of screenshot of my setting. Can you please tell me what's that mistake ? Link of screenshot

Thanks a lot for this comment, I see that there indeed was a problem there. I have put up a new version now, where it should be possible to run Java programs (see notes on the config panel for more info). Please let me know if it works for you. :)

It's still not working for me. I don't know where I have made the mistake. Did it work for you when you tried ? If yes, then please share screenshot of your settings. By the way thanks for giving your time. :)

Thank you for sharing this tool to the rest of us, i'll try to contribute to this project as soon as i can. Though i did find a bug, if the file path has any spaces in it, the program doesn't run at all. It keeps showing "running..." in the last column. Just letting you know, so you can add a issue on gihub.

does Hightail also work well with virtual contests? since i haven't used it before, i would like to test it on a virtual contest (or maybe even practice) and check its performance, before deciding whether if i'll use it in tomorrow's contest.

Hightail is a wonderful tool, thank you very much, it will help me a lot automatizing my submissions and training. Thank you and apologize my mistake configuring the settings, I read the instructions on the config screen, it's correct :)

After digging through the source, this is the minimal configuration needed for Java.
In Settings:
1. Leave the "Working directory" field empty.
2. Leave the "Path from working directory to executable" field empty.
3. In the "Command to prepend the executable with" field, put "java -cp" (without the quotes)
4. Untick all the checkboxes.
5. Now, click on "Save settings".
Given that you can run your class file using the java command line tool as follows:
java -cp path/to/your/project/directory package.your.classname
...then you're almost done. When doing a new contest, copy the arguments of your "java -cp" command (i.e. path/to/project/directory package.your.classname) and paste it in the field labelled "Path to executable file".

Sorry for not replying for so long, but I didn't get a notification email from CodeForces...

I agree this would be a good thing. Are you suggesting to unify the interfaces of the parsers, or also the libraries that they use (Hightail parsers now use a HTML parser library that I don't enjoy too much, but it gets the job done)? Do you have a more concrete proposal of what this should look like?

Though the idea of converting lines that start with "DBG:" to green doesn't entirely solve the problem, it would be very useful; adding "DBG:" to my template before each round is a very low price for the great benefits of being able to use Hightail both debugging and testing.

It seems that hightail leaves the tests hanging on round #288 B with the code I submitted on contest. It tests the first case and sometimes returns the result but the rest seems to be untested. It shows as I could abort the testing but nothing happens. It works fine on the other problems from that round though.

First of all thanks for making this software, it seems very good. However I do have an issue right now. I would like to use Hightail to run python codes. This is all well and good, and when I put python in the "Command to prepend executable with" the hightail software successfully runs the Python 2.7 I have installed on my computer. However I would like to use Python 3 that I have. Now in the terminal I can run a python file as python 3 by just typing python3 and the name of the python file. However when I put python3 rather than python in the dialog box for hightail it does not work, as when I run the tests it simply says running... for infinite time. Do you have any advice for how I can get hightail to run my Python codes as Python 3? Many thanks in advance. Also, I am on mac if this makes a difference.

Could you raise this issue on GitHub? Try to give an example of an input and a program for which this behaviour appears.

For the second one, please also open an issue. Although the chances that someone will implement this are not so high :) I always envisioned Hightail rather as a tool for situations where you solve a problem in one shot (without restarting your computer etc.).

I tried the example problem [A] given by mike. And it didn't work. Because in interactive tasks the input is coming after you output a number but hightail takes the example input and output which are just random.

WOOOWW!!! I was just watching petr's screencast for February lunchtime on codechef to know how they code fast ,, what kind of ide they use..and I saw him testing problems on the editor itself...i wondered if there is something like this for c++ and this 3 year old post came alive....just WOWW

Well, at most ~30 seconds after the scheduled time something should happen :) If it doesn't work for you, could you make a bug report (issue) on GitHub (with details, like which contest and what you are entering in the fields)?

Hightail Gives Runtime Error When i use it with codeblocks ( which creates both a .exe and .o file) but works perfectly fine with devc++ ( which creates only .exe file) can you look into this as codeblocks is my primary ide :)

Hi, when i am parsing individual problem, i am not able to give my own name for the problem , even if i edit the name from "A" to some "contestcodeA" , its still appears as "A" , but in settings when i check parse with full name, it gives "problem name" , what i wanted to do is to parse individual problems with "contestCode" prefix so that i dont have to delete the previous contests A,B,C,D from hightail. Thankyou

Is there any way to show error stream (cerr in C++ instead of cout) in Hightail?Thanks for this awesome program btw!

EDIT: Say a problem has x time limit, when a user creates a new test, (imho) it's better to set default time limit of the new test to x instead of default 3 seconds. So we don't need to manually set the time limit for each new test. Thanks for hearing my opinion :)

I recently started using it and I love it. I use Sublime to code in C++ on my Mac, and I was trying to use the prepend command feature to compile the file. Eg. I have in the path A.cpp, and a command "g14 name" that compiles name.cpp into name.exe I tried prepend command = g14 %P; But this doesn't work because %P does not evaluate to "A" in the prepend command. Is there any way to do this?

You would need to make a script that both compiles and runs your file (like make a && ./a on unix). Then have the prepend command be g14 (with a space after) and the path from ... be %P. That should work...

Note that it runs like: r14 /Users/.../a rather than /Users/.../r14 a. So you should make sure that r14 is in your PATH, for example. Try running r14 /Users/.../a from the console and see if that works.

There's a small bug running on my win10. When I use cmd to compile my C++ code, it works perfectly. However, when I use bash on win10 to compile my code, it just keep running. I just downloaded Hightail-v0.9.7.jar here. Is there anything I missed?

When you compile with bash on win10, the executable's target is Linux, so it can't be runned on Windows. However you can run Hightail on Linux (with bash on w10) by installing an X server like Xming. Here is a screenshot