NativeScript Now Supports Vector Types

Up until NativeScript 4.1.0 creating AR (augmented reality) apps was possible but somewhat limited due to the unsupported iOS vector types. I won’t get into details about AR and how you can use NativeScript for creating augmented reality apps (there are already some great articles on this topic - https://www.nativescript.org/blog/getting-started-with-augmented-reality-in-nativescript) but rather show you how to use vector type values provided by ARKit.

Vectors, Matrices, SIMD Types and Vectorization

Matrices are a fundamental part of AR and anything related to computer graphics, as they are a great way to represent image information. While a vector can be defined as a list of numbers, an n-matrix is a collection of numbers, organised in rows and columns. Each column is represented by a vector so that Nx1 matrix is same as vector with N elements.

For example, if we have two vectors a = (6, 4, 24) and b = (1, -9, 8) we can put them in the columns of a matrix:

The rows and columns determine a matrix’s dimension - the one we just drew is a 2x3 matrix. Dimensions are really important as some operations are possible only with matrices with identical dimensions. They can be added, subtracted, multiplied by a scalar or among themselves. However, such operations can take quite a long time and this where SIMD comes in.

A SIMD (https://en.wikipedia.org/wiki/SIMD) (Single-Instruction-Multiple-Data) instruction set describes an extension in microprocessors that allows them to operate data in parallel. The method for achieving parallelism by a single core is called Vectorization. What SIMD instructions basically do is operating on multiple pieces of data at the same time. While SISD (Single-Instruction-Single-Data) uses a separate register for each value to be added (e.g. addition operation), the SIMD instruction will fill one register with all the values(“packing” the register). The following image should illustrate the example:

This is extremely useful for multimedia applications, as they demand significant processing speed for video processing, speech recognition, MP3 etc. SIMD registers and operations are the low-level ingredients to SIMD programming. Higher-level abstractions are built on top of these such as simd_float4x4, simd_float3x3 etc. (matrix_float and matrix_double types are just typedef-s based on simd_float and simd_double) in iOS frameworks. For detailed SIMD library documentation see https://developer.apple.com/documentation/accelerate/simd.

This should be enough theory for now (though we have some more later). I will list some useful resources on this topic at the bottom of the article.

And now back to the vectors. We usе the Pythagorean theorem to derive a formula for finding the distance between to points A and B in the 2D space:

AB = ((x2 - x1)2 + (y2 - y1)2)1/2

So for three dimensions we just add the third (z) axis and the formula becomes:

AB = ((x2 - x1)2 + (y2 - y1)2 + (z2 - z1)2)1/2

Cool, but how do we find the coordinates of our two objects - the camera and the drone?

For the camera we will use ARCamera’s transform property:

This transform creates a local coordinate space for the camera that is constant with respect to device orientation. In camera space, the x-axis points to the right when the device is in landscapeRight orientation—that is, the x-axis always points along the long axis of the device, from the front-facing camera toward the Home button. The y-axis points upward (with respect to landscapeRight orientation), and the z-axis points away from the device on the screen side.

If we run the app again, the label will be updated every time we move the drone.

Calculating distance between objects in 3D space is slightly scratching the surface of everything possible to be done using matrices data. The great news is it can now be done with NativeScript, so go hack some AR!

Enterprise

Stay connected with NativeScript

I agree to receive email communications from Progress Software or its Partners, containing information about Progress Software’s products. Consent may be withdrawn at any time.

We see that you have already chosen to receive marketing materials from us. If you wish to change this at any time you may do so by clicking here.

Thank you for your continued interest in Progress. Based on either your previous activity on our websites or our ongoing relationship, we will keep you updated on our products, solutions, services, company news and events. If you decide that you want to be removed from our mailing lists at any time, you can change your contact preferences by clicking here.