Using and configuring SQS with FME Cloud

Why would you use SQS with FME Cloud?

FME Cloud runs on Amazon Web Services (AWS). You can use SQS with FME Desktop but most clients use it as part of an enterprise deployment alongside FME Server or FME Cloud.
There are several ways you can use FME Server with SQS.

Distributed processing

If you have a process that is taking a long time to run in FME Workbench, you can use FME Server to dramatically reduce runtime by batching the work and running the jobs in parallel. If you want to scale even further and process on a cluster of FME Servers then you will need FME Cloud (cloud deployment of FME Server). Since you pay for the compute you need with FME Cloud, it ideal if you have infrequent large data processing events. SQS forms a crucial part of this architecture.

With FME Cloud the largest instance you can launch is an Enterprise which has 16 cores. If this is not enough compute you can launch several of these instances and then distribute tasks between them. You use SQS as a workflow queue sending jobs to it, the cluster of FME Cloud instances then pull jobs from the SQS queue once they have capacity. Read more here

Decouple services

When designing fault tolerant applications, decoupling services is a key design pattern. That is, the producer continually generates messages for the broker, completely unaware of the message consumer.

Let’s say you are sending an asynchronous message to FME Server from a mobile device. If there is an issue with FME Server preventing it from accepting a request, the request will be lost. Rather than sending it to FME Server directly, you send it to SQS and then configure FME Server to pull messages from the SQS queue. With SQS you can almost guarantee the service will be running as the service runs across multiple servers and data centers.

The key part here is that FME Server pulls the message from the queue in a specific way to increase reliability — otherwise it is no better than sending the message direct. Pulling a message from an SQS queue is in two stages, you firstly pull the message from the queue which leaves the message in the queue but in a locked state. You can then do the work on FME Server and once that work is completed delete the message from the queue. If the job fails to complete for whatever reason and you don’t delete the message, the message will reappear in the SQS queue after a configurable period.

Working with FME and SQS

FME Desktop

SQSReceiver and SQSDeleter
This transformers allows you to pull and delete messages from an SQS queue. When setting the parameters on this transformer think about if you wish to delete the message on receiving it. If you do then set Delete on Receive to Yes. Often however, you want to perform work and then if the work succeeds you then want to delete the message from the queue. In this scenario you set Delete on Receive to No. Then once the job has finished you can use the SQSDeleter transformer to remove the job from the queue. I often use the SQSReceiver and SQSDeleter in combination with the the FMEServerJobSubmiiter.

SQSSender This transformer allows you to send messages to an SQS queue. You may want to send a message to SQS in a translation to trigger a tasks that needs to be completed by another application. You can also keep notifications of significant events in an Amazon SQS queue. Each event can have a corresponding message in a queue, and applications that need to be aware of the event can read and process the messages.

SQSMessageCounter You can count both the total number of messages in the queue and the number of visible messages in the queue. This is useful if you want to use FME Server to distribute messages to another system or even provision more capacity if the queue gets too long.

FME Server

SQS Publisher The SQS publisher is part of the FME Server notification server. It allows you to poll an SQS queue for messages continuously and then on receipt of a message publish it to a FME Server topic. Usually you then have an FME Workspace subscribed to the topic which you use to read the message and then initiate a task.

SQS Subscriber The SQS subscriber is the opposite of a publisher. It lets you send a message to an SQS queue upon receiving a notification from a Notification Service topic.

For more information on how you can leverage the FME Server notification server read here.