challenge

ChallengeThe CVPR 2019 Conceptual Captions Challenge is based on two separate test sets:

T1) a blind test set that participants do not have direct access to. This test set consists of about 12.5K image & captions pairs approved by human annotators, but otherwise preserving the distribution and style of images present in the Conceptual Captions dataset. You must submit your model to get your official scores. Submissions are handled via the Google Container Registry. Your model must be packaged in a Docker image and then uploaded to the Google Container Registry so that we can run it against the test set. Please see Prerequisites below for detailed info.

The results obtained by the submitted model(s) are made available on a leaderboard as part of this competition. You will have access to the global leaderboard that will rank the successful submissions we receive.

T2) an open test set that participants have access to and can download. This test set consists of 1000 images, taken from the Open Image Dataset. You must run the same model (the one submitted for T1) on the T2 images, and submit the resulting captions to us.

If your model submitted for T1 test set scores within the top-10 on the Challenge leaderboard and you have provided us the captions for your T2 open test set, we will evaluate your T2 captions using human evaluators for accuracy. The final model rankings for the Challenge will be based on the rankings achieved based on the human evaluation results of the T2 captions.

Submitting a Model The submission platform accepts two official submission for each participant (unlimited test submission can be attempted to make sure the submission mechanics work as intended). You can also use the provided T2 images to make sure that you get the right outputs from your docker image.

Each submission must process all T1 test images in under 24 hours or they will be considered failed. The execution environment has 32GB available memory and limited disk. Exceeding the amount of storage on the system will be considered a failed submission.

Sign up for a Google Cloud Platform account at cloud.google.com. Few important points below

If this is a new account, you will be following the sign up process which may require billing information.

For the purpose of this competition, your GCP account will be used only for the storage of the container image(s), which allows you to maintain all ownership of your container image(s). The compute cost of model evaluation will not be charged to your GCP account. You can start with free credit, if not already exhausted.

Install Cloud SDK. This is needed to interact with Google Cloud Project using command-line tools such as gcloud.

Install Docker. This is needed to build a Docker image for your model.

Building your Docker image You must submit your model as a Docker image. You are allowed to use whatever software dependencies you want, but those dependencies must be included in your Docker image. Let's say your model is a Tensorflow model. For this, you can use the official tensorflow Docker container as the base container image:

The first line of this Dockerfile says to use the official Tensorflow Docker image as the starting point of the image. The second line says to add the contents of a directory called "conceptual-captions" to a folder called "/conceptual-captions" inside the image. Read the Docker manual for more details on how to use Dockerfiles.

The folder "/conceptual-captions" is expected to contain a script called "submission.sh". The submission.sh script should call your model to get captions for each input image, and write them to file. Here is an example submission.sh.