It will be 10 years ago next month that I wrote my first line of Apex code. Visualforce was in beta (though not yet packageable), the only valid use of the “extends” keyword in Apex was for custom exceptions, and S-Controls were still alive and kicking… happy days.

Earlier in 2007, and of slightly (British understatement) more significance than my first Apex trigger handler, Heroku was founded.

By 2009 my trigger handler class now belonged to a FinancialForce product, and in the same year Heroku had commercially launched as a deployment platform for Ruby apps.

In 2010 Salesforce acquired Heroku, then in the following year Heroku added support for Node.js, Clojure, Java, Python and Scala. The acquisition of Heroku by Salesforce was interesting. Heroku and the Lightning Platform or “the platform formerly known as Force.com”, are very different beasts.

The Lightning Platform is a high productivity Application Platform as a Service (aPaaS) designed for “power admins” and “citizen developers” and is primarily model-driven – based around Salesforce Objects (SObjects). Apex, Visualforce and Lightning Components provide coding capabilities for advanced requirements, but this is a “low-code” platform in technology and culture.

Heroku on the other hand is a high-control Platform as a Service (PaaS) designed for developers, and which firmly embraces a message-driven architecture. We might call it a “high-code” platform, designed to be elastically scalable to very high throughputs.

It was a few years before we saw any real bridge between Heroku and the Lightning Platform. Heroku Connect was introduced in 2014 as an automated data synchronisation service which is often used by freestanding Heroku web apps or services to bring data into Salesforce without the Heroku developer knowing too much about the Lighting Platform.

Lighting Platform apps have become bigger and much more complex, to the point where “low-code” is perhaps no longer a very good description of what is going on. Apex governors are a “one size fits all” way of managing platform resources, and may well be suited to a “low-code” platform. However with the very complex logic now being written in Apex, governors are driving engineers to extreme lengths to get the job done.

Some Apex developers have looked enviously at the Heroku environment which not only offers computational power which can be scaled up to the needs of the task, but the ability to choose the most suitable language, and the availability of a plethora of libraries and frameworks which mean that the developer need not reinvent the wheel on a daily basis.

Some intrepid developers have decided to straddle the gap between the Lighting Platform and Heroku to take advantage of Heroku compute power within their applications – embedding Heroku power within their Salesforce applications.

However, despite the advantages of using Heroku compute with the Lightning Platform there are challenges to be overcome.

Identification and Authentication

When starting a Heroku compute process from within the Lighting Platform, your Heroku process will most likely want to check the identity of the caller – the Salesforce Organization and User – and whether the Org Admin has authorised the user to perform the action.

Similarly, when the Heroku process needs to call back into the Lightning Platform, Salesforce will likewise need to authenticate the request and identify the Salesforce User – and Heroku will need to provide the necessary credentials. This ensures that the data written back to Salesforce is tied to an appropriate Salesforce user – which will often be the user who initiated the process.

Asynchronicity

To a large extent, Heroku’s scalability comes from the way it can handle processes asynchronously – using a message-driven architecture. Ok, so you don’t have to write asynchronous services in Heroku, but you’ll be limiting your application’s scalability if you don’t. Worker Dynos, Background Jobs and Queueing does a good job of explaining this, but here’s a real world illustration:

Imagine you walk into a pizza place, you make your order and the person who takes it pops on an apron, washes their hands and heads back into the kitchen to prepare your food. What happens to the next person who comes in? There’s no one to take their order! That’s synchronous processing. There’s a lot of waiting. I want to get my order taken right away, and maybe step next door to pick up some gelato while my pizza is being prepared. Maybe I’ll pop in to ask how it’s going – or better yet get a notification on my phone when my order ready.

With asynchronous processing, work is put into a queue. In Heroku if the queue gets too long, then you can either increase the power available for your process, or use a greater number of “processors” to get through the work faster. Either way you are not causing anything upstream to wait while the work is done.

The downside of running asynchronous processes is that it involves a different way of looking at the implementation and a little more effort – publishing and subscribing messages to a queue, and considering how you’re going to respond to the process ending.

For Identification and Authentication there’s a package which makes the necessary calls to the Salesforce Identity Provider, to validate the requests coming from Salesforce, and also to authenticate with Salesforce in order to call back in when needed. There’s also a command line tool which configures both the Salesforce org (with a Connected App) and the Heroku app – wiring up the authentication both ways.

With the asynchronous processing side of things, Orizuru aims to make this as simple as writing Batch Apex. You, the developer, write the code that does the actual work you want done in Heroku – maybe pulling in some super libraries in Node or Java, and the Orizuru toolkit does the plumbing: Apex API client, API service, validation of requests, publishing messages to a queue and subscribing from the queue.

It’s early days, so there are bound to be some gaps, but it’s an open source project, and the team would welcome both constructive criticism and contributions!

When running a Batch Apex job which implements Database.Batchable<sObject>, you specify a query in the start method (in which you can select up to 50M records) and define an activity to be performed based on these records in the execute method.

The platform runs the start method, queries the data, and feeds the data to your execute method, in batches.

Ch-Ch-Changes

Originally, the way this worked was that the data was queried in the start, and the entire query results (all the queried fields) put into temporary storage, to be chopped into batches and processed in the execute method (stale batches).

However, some time ago (around 2012) this was changed so that only the IDs of the records were put into temporary storage. The platform will retrieve the records by ID just-in-time to be processed for each batch.

Apex developers need to be aware of what this means for them. If any of the records are modified by another user or process while the Batch job is running, it is likely that it will be the modified (fresh) version of the records which will be passed into the execute, and not the (stale) version of the records as they were when the start method was called.

I have seen some discussion about this recently and so set about creating a test to verify this behaviour. I have shared my test code on github.

Testing 1… 2…3…

In my test I created 5,000 test Account records with sequential AccountNumbers, and then ran two concurrent Batch jobs over the same records. In the one job the records were sorted in ascending order of AccountNumber, and the other job, in descending order.

Both jobs simply update the Site (text) field, appending a simple tag to identify the batch process (whether it’s the ascending or descending job) and also adding a batch number.

Once both processes are finished, if the records had been read into temporary storage from the start method, then when examining the records with the lowest AccountNumbers the Site field would show the tag for the descending batch job. The “low” records would have been updated by the ascending batch job first, and then later on the changes would appear to have been overwritten by the descending batch job – because the descending job would be using stale data which does not contain the ascending tag.

On reviewing the Account records I found that the records at the start of the set showed both tags – the ascending tag first, and then the descending tag. This means that the descending job must have selected these records after the ascending job had already updated them. Similarly, the records at the end of the set showed the descending tag followed by the ascending tag.

In the middle of the record set were some records which had only one tag – either ascending or descending. These records would have been read at the same time by both processes and so each process appended its tag to an empty field (the initial state).

Conclusions… this slice of data is only fit for toasting

The test would appear to confirm that the records are not passed into temporary storage at the start of the job.

What was interesting was that by using a smaller batch size I found that these single-tagged records spanned multiple batches, which would indicate that the platform queries the records in larger chunks than the batch size and breaks these query chunks into batches.

This is also quite important, as it implies that sometimes, but not always, the platform will read-ahead and so your batch data may or may not be stale.

This is a good thing – it means that a bad tenant (who thinks that bulkification is a dietary term and object orientation refers to feng shui for the desktop) doesn’t affect the good tenants – that’s you and me.

Making Light Work of Heavy Lifting

Sometimes though, the limits may be too restrictive – when we have some heavy lifting to do. Here Asynchronous Apex comes to the rescue, which makes light work of heavy jobs.

As the name implies, Asynchronous Apex means that the work is not done immediately / synchronously, as it is in the normal interactive context for your users. There’s an exchange, a compromise to be made – you accept that you will be prepared to wait a little for the work to be done, which makes it easier for Salesforce to manage demand on resources, and in return Salesforce will give you higher governor limits, as the resources are easier for the platform to allocate. Currently with Async Apex the governor benefits are:

x6 CPU time

x2 SOQL queries

x2 heap size

Now, because the work isn’t done immediately, you need to put some more thought into what happens when the work is finished. With synchronous Apex you can provide immediate feedback to your user – for example if you’re working in a Visualforce page or a Lightning Component. But with async Apex the execution has split off from the user’s path and there’s no direct way of providing them with a response.

You can of course send user notifications by email or Chatter, or otherwise you would have to provide some means in your application to check back on the work being done in the async process.

Future Methods

The future method was the first means provided by Salesforce to do Asynchronous Apex. These methods are really simple to use – annotate a static method in Apex, and when you call it, the method returns immediately, but the work will be done at some point in the not too distant future:

We get the higher async governor limits within the execution of the static method, but there are some limitations / considerations:

we cannot directly monitor the status of future method execution once we’ve fired it off – we’d have to look for the effects of the method on our data (but what if it fails? – there would be no effects – queue tumbleweed…)

we cannot be sure of the order that multiple future methods are executed

parameters can only be primitive types (or collections of primitives)

there is a limit of 50 future method calls in a single Apex execution

recommended best practice for future methods is that methods should be designed to execute quickly; therefore if you need to make a (potentially slow) HTTP callout from a future method, you need to declare it in the annotation with “callout=true”

Batch Apex was provided to handle situations where a similar operation needs to be executed iteratively many times, most typically an operation to be performed on a large set of records (up to 50M records).

A batch job has three parts:

start by defining the overall scope of the job – typically (but not exclusively) using a SOQL query to select your records

execute a method repeatedly, each iteration needs to handle a batch of the work that was scoped in the start (again, typically a batch will be a list of sObjects)

You fill out the three methods required in a Batch job, and the platform will call them for you, once you’ve submitted the job with the executeBatch call.

Each batch can process between 1 and 2,000 items of work (you choose). The governor limits of course are applied to each batch – that is to a single execute method (rather than to each item in the batch).

You might therefore think that a batch size of 1 would be perfect, as you get the highest governor limits per item of work, but you should avoid really small batch sizes – as the overall job will take much longer to process owing to the “overhead” in starting each batch.

We can also programmatically monitor a batch job while it is running, and abort it if we need to:

Whilst batch Apex is most often used to process many sObject records via a SOQL query, it can also be used with an Iterable class so that you can design how you want to define the job and divide it into batches. In this case there’s no batch size limit, but you will likely be restricted by what can be accomplish within the start method (in which you construct your Iterable).

Batch Chaining

But what if 50M records isn’t enough? Or, you have a composite process in mind where you need to first process Opportunities and then Accounts? We will want to create several batch jobs and chain them together, so that one starts on completion of another – by calling Database.executeBatch within the finish method of a batch job.

When batch Apex was first made available we had to cheat to achieve this – we couldn’t do it directly. We would use the Schedulable interface and call System.schedule to kick off the next batch process, say in 5 minutes time.

Today however, we don’t need this workaround; we can chain jobs together by calling Database.executeBatch for one job in the finish method of another.

Whoah there…

The main limitation of batch Apex is that there can be only 5 concurrent batch jobs running in an organisation; this should be a major consideration when developing batch processes as this limit applies across all users and applications.

Given this, we might infer that batch Apex is provided primarily for administrative processes rather then end-user processes, but it is very common to use Batch Apex for routine rather than admin processes.

unlike future methods, execution can be monitored: when a job is submitted, an ID is returned to you which you can use to query the AsyncApexJob table in the same way as batch Apex

future methods were static and took only primitive arguments, but a queueable Apex implementation can effectively pass in complex types like sObjects and custom Apex types (an Account in the above example)

in common with future methods, there is a limit of 50 executions enqueued within a single Apex execution

as of Spring 15, queueable executions can be chained together (although in DE orgs the chain limit is 5 jobs)

although not documented at the time of writing, as with future methods and batch Apex we must declare when we intend to make HTTP callouts – with Database.AllowsCallouts (same syntax as batch Apex)

Hot on the heels of queueable Apex, in the Spring 15 release we have the FlexQueue. At the time of writing, this feature can be seen purely as an enhancement to batch Apex – but at some point in the future we should see queueable Apex join the FlexQueue (see Josh Kaplan’s blog post, linked below).

The FlexQueue provides you with a backlog of up to 100 batch Apex processes, in addition to the 5 “live” concurrent batch Apex jobs.

Say you have 5 batch jobs preparing or processing, and nothing in the FlexQueue, you can submit 100 new batch Jobs and they will go straight into the FlexQueue – previously this would have caused an error.

When jobs are picked from the FlexQueue to be processed, this frees up space in the FlexQueue for further jobs to be submitted.

Jobs in the FlexQueue can be seen in the existing Apex Jobs admin page, and from here they can also be aborted. They can also be seen programatically when querying AsyncApexJob and also aborted programmatically in the same way as any running batch job.

Note that we should take care when aborting a job as the same method aborts a job in the FlexQueue or the currently running Batch jobs – you may set out to abort a job waiting in the FlexQueue but by the time you make the call, the job has started processing and so you end up cancelling a part-complete running batch job.

There’s also a new FlexQueue admin page is provided which can be used to change the order of jobs that aren’t yet being processed.

In the Summer 15 release, a new system method was piloted for controlling the order of jobs in the FlexQueue programmatically. This mirrored the functionality already provided in the FlexQueue UI – specifying a new position for a job in the FlexQueue as an integer:

Boolean isSuccess = System.moveFlexQueueJob(jobId, positionNumber);

The weakness of this is that position numbers change as jobs are picked off the queue – so the method call may not produce the expected results. As a result, this method was not made generally available.

Instead, in the Winter 16 release we have several methods on a new FlexQueue class which provide for control over the order of items by relative rather than absolute position, which is much better:

Unfortunately however, there is something missing, which is the ability to determine programmatically the current order of the FlexQueue. This severely limits our ability to build functionality to manage the ordering of jobs. However, from what I understand, this gap is due to be plugged soon.

One final feature worth mentioning in passing is Enhanced Futures. This is not yet generally available, but it has been in pilot since the Summer 14 release. Enhanced futures allows a specific future method to increase one (and only one) governor limit by doubling or tripling. That’s doubling/tripling the already superior async governor limits. The enhanceable governors are:

Heap size

CPU time

Number of SOQL queries

Number of DML statements

Number of DML records

There is a fuzzy warning linked to this feature: “Running future methods with higher limits might slow down the execution of all your future methods”. I imagine we will get more detail when the feature is released, but clearly the intention is that enhanced limits should be used sparingly, when you really really need them.

And finally… in addition to the limits which apply to each async type, there is an overarching limit on the number of asynchronous method executions which can be made in a 24 hour period. This limit is typically 250,000 calls / 24 hours, but higher in large organisations. See Apex Governor Limits for more details.

Note that this limit encourages us to obtain high value from each call – rather than firing off a higher number of lower value calls. For example – avoiding small / overly pessimistic batch sizes for Batch Apex jobs.

Most of the content for this post came from the Dreamforce 15 DevZone session: Apex Liberation: The evolution of Flex Queue which I had the great pleasure of presenting with Carolina Ruiz Medina. The session slides are available on SlideShare.

Share this:

Like this:

Q&A WITH STEPHEN WILLCOCK OF FINANCIALFORCE.COM

Monday 27th April 2015

We talked with Stephen Willcock from our client Financialforce.com, and had the opportunity to discuss a number of subjects such as his role as Director of Product Innovation, what it’s like to be part of a technology company going through hyper-growth and he fills us in on his favourite tech gadgets.

With 16 years’ experience in consulting and software development, including 7 years developing and architecting Force.com applications, in his role as Director of Product Innovation Stephen has a remit to ensure that product development effectively harnesses the Salesforce1 platform and complementary technologies to solve business challenges through delivery of innovative products.

HI STEPHEN, THANKS FOR TAKING THE TIME TO HAVE A CHAT WITH US TODAY. SO, YOUR CURRENT ROLE IS “DIRECTOR, PRODUCT INNOVATION” WHICH SOUNDS GREAT – WHAT ARE YOU RESPONSIBLE FOR IN THIS ROLE?

Not a problem! Well, it’s genuinely a “dream job”. I have a small team of extraordinary developers, and we get to build prototypes for new products or product features, experimenting with the very latest Salesforce technologies. The work is varied; we might be presented with a tricky problem to solve, or given a new technology or feature to evaluate and come up with ideas around how we can put them to best use in our products. Continue reading →

Share this:

Like this:

Sometimes using JSON.serialize on a custom Apex type does not provide sufficient control over how the JSON is serialized. For example, serializing an Apex type will include null values for all fields that haven’t been set, when you might prefer to omit these properties from the JSON altogether. Or perhaps you need to use a property name in the JSON which is invalid for a class member name Apex. Fortunately JSON.serialize works on any Object, so you can serialize JSON from any structure you can assemble in Apex, for example:

Share this:

Like this:

I recently wanted to create an extension controller for a custom object which I could use with both single records and a set of records. In other words, a list and detail controller extension, with actions that could be applied to one or many records.

I started out by creating my controller with two constructors, one which accepted a standard controller and a second that accepted a standard set controller so that it could be bound to Visualforce pages for either list or detail pages:

So far, so good. But before using any methods on either standard controller I need to check which controller has been assigned, which I think is a little messy, and also seems like hard work if the same method exists on both controllers – save() in this case.Continue reading →

Share this:

Like this:

Salesforce user security. Its great. As well as being one of the things customers value highly, its a massive advantage for application developers to be building upon a trusted platform with robust and well considered security features.

Restricting logins to specific IP addresses or ranges of addresses is a valuable user security feature, although it can be slightly confusing.

… Salesforce then checks whether the user’s profile has IP address restrictions. If IP address restrictions are defined for the user’s profile, any login from an undesignated IP address is denied, and any login from a specified IP address is allowed.

If profile-based IP address restrictions are not set, Salesforce checks whether the user is logging in from an IP address they have not used to access Salesforce before:

If the user’s login is from a browser that includes a Salesforce cookie, the login is allowed. The browser will have the Salesforce cookie if the user has previously used that browser to log in to Salesforce, and has not cleared the browser cookies.

If the user’s login is from an IP address in your organization’s trusted IP address list, the login is allowed.

If the user’s login is from neither a trusted IP address nor a browser with a Salesforce cookie, the login is blocked.

Whenever a login is blocked or returns an API login fault, Salesforce must verify the user’s identity:

For access via the user interface, the user is prompted to enter a token (also called a verification code) to confirm the user’s identity.

I have diagrammed the flow to (hopefully) make this a little easier to follow, focussing on a user logging in via a web browser (rather than API access):

A few points that I think are worth mentioning:

There are two places where an admin can specify IP addresses, or ranges of addresses for the login process. Whilst these look very similar, they purposes are distinct.

User Profile – Login IP Ranges

We can set up IP ranges (or individual addresses) on each user profile. When no IP address ranges are specified, the user’s profile does not restrict IP addresses in any way. However as soon as we specify any IP addresses as login IP ranges, then all other IP addresses become invalid for users with the profile and will result in access being denied.

Network Access – Trusted IP Ranges

In Security Settings, we can also set up trusted IP addresses for the entire org.

When a user logs in from an IP address they haven’t used to login to Salesforce, they will need to go through a verification process. This is where they are challenged and need to ask for a token to be sent to them, which they can use to gain access.

Trusted IP Ranges simply allow users to bypass the verification process when logging in from a trusted IP address.

Each trusted IP range is limited however, so 0.0.0.0 – 2.0.0.0 is acceptable, but 0.0.0.0 – 3.0.0.0 is not. Larger groups of trusted IP addresses would need to be specified my adding several ranges.

“Login IP Range” beats “Trusted IP Range”

From my flow diagram we can see that any profile login IP ranges are enforced before trusted IPs are considered. This means that “trusting” your IP address has no effect if your profile blocks it (by specifying some ranges, but not one that includes your address).

Engage Demo Mode

IP address restrictions and the verification process are generally a good thing. However, there may be times where the feature is unnecessary and perhaps frustrating – where you are giving demonstrations form a developer edition org of work in progress, or of new Salesforce features, for example.

In order to disable the verification process, we cannot add all IP addresses to a single Trusted IP Range, as the size of the range is limited. We could add many ranges, e.g. 0.0.0.0 – 0.255.255.255, 1.0.0.0 – 1.255.255.255, 2.0.0.0 – 2.255.255.255, etc.

However, there is no limit in size for to a Login IP Range, so you could specify all IP addresses: 0.0.0.0 – 255.255.255.255. In a demo org you may have have one, or at worst a handful of profiles that you will be using, so it would be easy to add an all-IP-addresses Login IP Range to each profile.

This will prevent login challenges and verification in the middle of a demo, but is not good practice in production.