znProjects Blog

Software is beautiful again!

Daily Coding Problem 4 - Time and Space Complexity

This problem was asked by Stripe.

Given an array of integers, find the first missing positive integer in linear time and constant space. In other words, find the lowest positive integer that does not exist in the array. The array can contain duplicates and negative numbers as well.

… O(1) denotes constant (10, 100 and 1000 are constant) space and DOES NOT vary depending on the input size (say N).

What do these constraints mean?

Linear time complexity

Linear time complexity limits the algorithms that can be used to solve this problem. For example, a comparison-based sort cannot perform better than O(n log n) in the worst case, which means that we cannot use any comparison-based sorting algorithm to solve this problem.

Constant space complexity

This means that we cannot change the space used by the algorithm as a function of the input size. Or, in other words, I can’t simply create a min-heap, for example, and ask for the minimum value because the heap’s size will change each time depending on the size of the input array. In case someone is thinking about creating a massive data structure to use for each execution of the algorithm (e.g. always create an Array of size MAX_INT for execution, regardless of the size of the input array) - while I guess that this is theoretically constant space, we run into the following problems:

This is a hugely wasteful approach to solving the problem.

This solution will fail if we can use a data structure larger than addressable memory, for example a data structure that can be stored on disk.

I don’t believe this is in the spirit of the problem.

On a side note, the final solution will not be pretty because:

Most of F#’s data structures, by default, are immutable. Luckily, Arrays are not.

F# encourages a functional programming style and, I believe, the final code will have to be more procedural.

So, how do we solve it?

I thought of a lot of solutions to solve this problem that can meet the given constraints.

This can work if I can use a linear time and constant space sort. As far as I know, there is no such sort.

Comparison sorts violate time constraint

Non-comparison sorts violate space constraint

Recognize that if an array has length n, the answer can be, at most, n + 1.

Consider an array that is 1-indexed.

If all the values in the array are 0 or negative, the answer is 1.

If all the values in the array are >= 2, the answer is 1.

If the value at each index of the array is the same as the index, the answer is n + 1.

While I haven’t done anything close to a formal proof for this, I believe I am right about this.

It took me a long time to arrive at this. I had been working on this problem in my head for a couple of weeks (mostly while commuting to and from work) before I came to this realization. I also had to force myself to think procedurally, like in C/C++, to get to this answer.

Code

The following is the solution I came up with.

Strategy for the code

At a high level, the code adopts the following strategy:

First pass. Places values in their corresponding indices.

for each entry do
while the entry's value is not 0 and is not index + 1 do
swap values with the index represented by the value in the entry

Second pass. Find the first entry where the value is 0, return index + 1.

Scan the array for the first entry with value of 0
If found, return index + 1
Else, return array length

When reading the code, there are two items to remember. These may seem obvious, but I had to keep reminding myself courtesy of 0-based arrays in a situation where 0 has special meaning:

Given an index, the value there should be index + 1, i.e.value = index + 1

Given a value, the index it belongs to should be value - 1, i.e. index = value - 1

/// The main function.letsolverarr=/// swap the value in between 2 indices.letswap(arr:int[])i=letorigVal=arr.[i]arr.[i]<-arr.[origVal-1]arr.[origVal-1]<-origVal// Process a single index.letprocessOneIndex(arr:int[])i=// Invalid, negative or zero value (doesn't influence end result)ifarr.[i]<=0thenarr.[i]<-0// Value is greater than the array lengthelifarr.[i]>Array.lengtharrthenarr.[i]<-0// Value is already set to the array index, no action requiredelifarr.[i]=i+1then()// Requires swapping but the target has the same value alreadyelifarr.[i]=arr.[arr.[i]-1]thenarr.[i]<-0// Value is valid and requires swappingelseswaparri/// First pass, put each value in its corresponding indexforiin0..Array.lengtharr-1dowhilenot(arr.[i]=i+1||arr.[i]=0)doprocessOneIndexarri/// Second pass, find the first 0 or return (array.length + 1) as the resultletzeroIdx=Array.tryFindIndex((=)0)arrmatchzeroIdxwith|Some(i)->i+1|None->Array.lengtharr+1

Alternate solutions

In order to test this solution, we need to write a few alternate solutions to ensure that the results are correct.

I suggested three solutions above, and have implemented two of them below.

Set-based solution

The first alternate solution solves the problem by creating two sets and performing set subtraction. Sets will automatically remove duplicate values, so I do not need to handle that situation explicitly.

letsetSolverarr=// The set of original values, without any negative numbers.letorigSet=arr|>Array.filter(fune->e>=0)|>Set.ofArray// The set of "solution" values, with the minimum possible solutionletplusOneSet=origSet|>Set.map(fune->e+1)|>Set.add1// Return the minimum value from the solution set that wasn't in the original// setSet.differenceplusOneSetorigSet|>Set.minElement

Sort-based solution

In the second alternate solution, I solve the problem by sorting the array and finding the first pair of values that is not consecutive. Duplicate values are explicitly removed to avoid issues with comparison later.

letsortSolverarr=// Filter, sort, and de-duplicate the arrayletsortedArray=arr|>Array.filter(fune->e>0)|>Array.sort|>Array.distinctifArray.contains1sortedArraythenifArray.lengthsortedArray>1then// Find the first pair of numbers in the array that is not consecutive.sortedArray|>Seq.windowed2|>Seq.filter(funa->a.[1]-a.[0]<>1)|>Seq.tryHead|>function|Some(a)->a.[0]+1|None->Array.lengthsortedArray+1elifArray.lengthsortedArray=1thensortedArray.[0]+1else1else1

Utility functions

Some utility functions to work with 3-tuples and to collect performance data using Windows Performance Counters.

Testing

Base tests

Considering the “index math” happening in these solutions, we need to thoroughly test them to find any issues. The first tests are to ensure that the two given test cases work correctly for each of the algorithms.

After much struggling with FsCheck, I’m happy to start using a property-based testing tool that works with Jupyter and FSharp.Literate, Hedgehog.

I am going to use a stopwatch to help measure runtimes since I am using FSharp.Literate instead of Jupyter for this post, and I can’t figure out how to enable the #time directive through FSharp.Literate.

openHedgehog/// The stopwatch that will measure run-times for the rest of the code.letstopWatch=Stopwatch()stopWatch.Start()

Performance Testing

Now that the algorithms seem to be working correctly, the next step is to capture some performance metrics. For this portion, I will be using System.Diagnostics.Stopwatch to measure each algorithm’s speed. I believe my last post proved that, as of now, I cannot benchmark memory usage reliably.

First, we can setup a generator to generate random values for testing. Similar to the base testing, let’s benchmark the runtime of the test data generation.

Now that we have our profiling data, the next step is to measure performance.

First, let’s create a bare minimum set of utility ‘tools’ to enable the performance monitoring.

/// The number of runs to perform, so average the performance.letnumRuns=500/// Function that helps measure the average runtime based on the number of runsletmeasurementnumRunsalglst=// Measures actual passage of time, not just time spent in the algorithmletswRT=Stopwatch()// Measures GC runtimeletswGC=Stopwatch()swRT.Start()// Reset the stopwatch when starting a new measurement.stopWatch.Reset()for_in1..numRunsdo// Perform a forced, blocking garbage collection with compaction, but don't// penalize the algorithm.swGC.Start()System.GC.Collect(System.GC.MaxGeneration,System.GCCollectionMode.Forced,true)swGC.Stop()// Start the stopwatch.stopWatch.Start()// Run the algorithm on the entire input list.foriinlstdoalgi|>ignore// Stop the stopwatchstopWatch.Stop()// Stop the measurement of "real passage" of time.swRT.Stop()// Return algorithm, GC, and total runtimesSystem.TimeSpan.FromTicks(stopWatch.ElapsedTicks/(int64numRuns)).Ticks,swGC.ElapsedTicks,swRT.ElapsedTicks

Solver

Let’s start by benchmarking the base solver algorithm.

letsolverRuntime=measurementnumRunssolvertcprintfn"Solver algorithm time %s, gc time %s, and total time %s"(TimeSpan.FromTicks(mfstsolverRuntime).ToString("G"))(TimeSpan.FromTicks(msndsolverRuntime).ToString("G"))(TimeSpan.FromTicks(mtrdsolverRuntime).ToString("G"))

Solver algorithm time 0:00:00:00.0646408,
gc time 0:00:01:10.1015987, and
total time 0:00:01:42.4224170

Set Solver

The next benchmark is with the setSolver algorithm.

letsetSolverRuntime=measurementnumRunssetSolvertcprintfn"Set Solver algorithm time %s, gc time %s, and total time %s"(TimeSpan.FromTicks(mfstsetSolverRuntime).ToString("G"))(TimeSpan.FromTicks(msndsetSolverRuntime).ToString("G"))(TimeSpan.FromTicks(mtrdsetSolverRuntime).ToString("G"))

Set Solver algorithm time 0:00:00:33.7109077,
gc time 0:00:01:10.8180741, and
total time 0:04:42:06.2721556

Sort Solver

And finally, the benchmark for the sortSolver algorithm.

letsortSolverRuntime=measurementnumRunssortSolvertcprintfn"Sort Solver algorithm time %s, gc time %s, and total time %s"(TimeSpan.FromTicks(mfstsortSolverRuntime).ToString("G"))(TimeSpan.FromTicks(msndsortSolverRuntime).ToString("G"))(TimeSpan.FromTicks(mtrdsortSolverRuntime).ToString("G"))

Sort Solver algorithm time 0:00:00:00.7639951,
gc time 0:00:01:11.5495339, and
total time 0:00:07:33.5473061

Results

So, what were the results? First, let’s chart the runtimes as measured by the stopwatches (algorithm Alg, garbage collection GC, and total time TT) and some comparative percentages.

Performance Counter statistics

While executing the workbook, I collected a number of values using Performance Counters on Windows. These were collected throughout the life of the book’s execution. For descriptions of these counters, please refer to the .NET Performance Counters page.

.NET CLR LocksAndThreads

Number of current logical Threads

Number of current physical Threads

.NET CLR Memory

Number Gen 0 Collections

Number Gen 1 Collections

Number Gen 2 Collections

Gen 0 heap size

Gen 1 heap size

Gen 2 heap size

Large Object Heap size

Process

Private Bytes

% Processor Time

I thought that it would be interesting to chart these values out.

However, the first thing we need to do is to stop collecting the data.

cancelToken.Cancel()

.NET CLR LocksAndThreads, # of Current Threads

letminThreadSize=[counterResults.[0].Count;counterResults.[1].Count]|>List.min[yieldList.initminThreadSize(funi->i,counterResults.[0].Itemi)yieldList.initminThreadSize(funi->i,counterResults.[1].Itemi)]|>Chart.Line|>Chart.WithOptions(Options(title=".NET CLR LocksAndThreads, # of current Threads",curveType="function",legend=Legend(position="bottom")))|>Chart.WithLabels["# of current logical Threads";"# of current physical Threads"]

Conclusion

So, I think the place to begin is with a seemingly obvious statement - time and space complexity matters and should be balanced in conjunction with other considerations such as conceptual “simplicity” and maintainability. For example, the set solver was one of the simplest solutions (for me) to understand. However, the algorithm runtime was 521x slower and the total runtime was 165x slower than the fastest solution, with the total runtimes being 4 hours 42 minutes compared to less than 2 minutes on the same dataset.

The memory statistics present an interesting story, with the number of Gen 0 collections growing at a linear rate through the life of the program. Now, most of the program’s time was spent in the Set Solver’s performance test. At the end, there is a sharper uptick in Gen 0 collections, possibly due to the Sort Solver’s performance test. However, as I did not implement a method to correlate counter readings to the currently running code block, I can’t say that with absolute certainty.

In general, I am extremely pleased with how Hedgehog was able to take on a large percentage of the burdens related to property testing and random test data generation. Windows Performance Counters were also a pleasant find and a good alternative to BenchmarkDotNet. I definitely plan to use these going forward, at least until FsLab comes to .NET 3.0 and I can use something like dotnet-counters.