.. _converter_functions:
Transformation Function Overview
--------------------------------
Type Conversions
~~~~~~~~~~~~~~~~
- ``::int`` or ``::integer``
- ``::long``
- ``::float``
- ``::double``
- ``::boolean``
- ``::r``
- ``stringToInt`` or ``stringToInteger``
- ``stringToLong``
- ``stringToFloat``
- ``stringToDouble``
- ``stringToBoolean``
String Functions
~~~~~~~~~~~~~~~~
- ``strip``
- ``stripPrefix``
- ``stripSuffix``
- ``stripQuotes``
- ``replace``
- ``removeChars``
- ``length``
- ``trim``
- ``capitalize``
- ``lowercase``
- ``regexReplace``
- ``concatenate``
- ``substring``
- ``toString``
- ``emptyToNull``
- ``printf``
Date Functions
~~~~~~~~~~~~~~
- ``now``
- ``date``
- ``dateTime``
- ``basicIsoDate``
- ``isoLocalDate``
- ``basicDateTime``
- ``isoLocalDateTime``
- ``isoOffsetDateTime``
- ``basicDateTimeNoMillis``
- ``dateHourMinuteSecondMillis``
- ``millisToDate``
- ``secsToDate``
Geometry Functions
~~~~~~~~~~~~~~~~~~
- ``point``
- ``multipoint``
- ``linestring``
- ``multilinestring``
- ``polygon``
- ``multipolygon``
- ``geometrycollection``
- ``geometry``
- ``projectFrom``
ID Functions
~~~~~~~~~~~~
- ``stringToBytes``
- ``md5``
- ``murmur3_32``
- ``murmur3_128``
- ``uuid``
- ``uuidZ3``
- ``uuidZ3Centroid``
- ``base64``
Math Functions
~~~~~~~~~~~~~~
- ``add``
- ``subtract``
- ``multiply``
- ``divide``
- ``mean``
- ``min``
- ``max``
List and Map Functions
~~~~~~~~~~~~~~~~~~~~~~
- ``list``
- ``mapValue``
- ``parseList``
- ``parseMap``
Control Functions
~~~~~~~~~~~~~~~~~
- ``try``
- ``withDefault``
.. _convert_scripting_functions:
Functions defined using scripting languages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can define functions using scripting languages that support JSR-223.
This is currently tested with JavaScript only as it is natively
supported in all JREs via the Nashorn extension. To define a JavaScript
function for use in the converter framework, create a file with the
``.js`` extension and the function definition as the contents of the file.
For instance, if you have defined a function such as
.. code-block:: javascript
function hello(s) {
return "hello: " + s;
}
you can reference that function in a transform expression as
``js:hello($2)``
Installing Custom Scripts
~~~~~~~~~~~~~~~~~~~~~~~~~
Custom scripting functions are made available to GeoMesa comamnd line tools or
distributed (map-reduce) ingest via including them on the classpath or
setting a system property.
For local usage, geomesa defines the system property ``geomesa.convert.scripts.path``
to be a colon-separated list of script files and/or directories containing scripts.
This system property can be set when using the command line tools by setting the
``CUSTOM_JAVA_OPTS`` environmental variable:
.. code-block:: bash
CUSTOM_JAVA_OPTS="-Dgeomesa.convert.scripts.path=/path/to/script.js:/path/to/script-dir/"
A more resilient method of including custom scripts is to package them as a JAR or ZIP
file and add it to the ``GEOMESA_EXTRA_CLASSPATHS`` environmental variable. If using
maven you can simply package them in a folder under ``src/main/resources/geomesa-convert-scripts/``
which will create a folder in your jar file named ``geomesa-convert-scripts`` with
the scripts inside. You can manually create a jar with this folder as well. An easier way
is often to package them as a zip archive with a folder similary named ``geomesa-convert-scripts``
inside the archive containing the scripts:
.. code-block:: bash
$ unzip -l /tmp/scripts.zip
Archive: /tmp/scripts.zip
Length Date Time Name
--------- ---------- ----- ----
0 2017-03-09 11:33 geomesa-convert-scripts/
42 2017-03-09 11:33 geomesa-convert-scripts/my-script.js
--------- -------
42 2 files
For either zip or jar files add them to the extra classpaths in your environment to
make them available for the tools or map-reduce ingest:
.. code-block:: bash
GEOMESA_EXTRA_CLASSPATHS="/path/to/my-scripts.jar:/tmp/scripts.zip"
A example of ingest with a scripts on the classpath is below:
.. code-block:: bash
GEOMESA_EXTRA_CLASSPATHS="/tmp/scripts.zip:/path/to/my-scripts.jar" bin/geomesa-accumulo ingest -u
-p -s -C -c geomesa.catalog hdfs://localhost:9000/data/example.csv
You can also verify the classpath is properly configured with the tools:
.. code-block:: bash
GEOMESA_EXTRA_CLASSPATHS="/tmp/scripts.zip:/path/to/my-scripts.jar" bin/geomesa-accumulo classpath
CQL Functions
~~~~~~~~~~~~~
Most of the basic CQL functions are available as transformations. To use
one, invoke it like a regular function, prefixed with the ``cql``
namespace. For example, you can use the CQL buffer function to turn a
point into a polygon:
::
cql:buffer($1, 2.0)
For more information on the various CQL functions, see the GeoServer
`filter function
reference `__.
JSON/Avro Transformations
~~~~~~~~~~~~~~~~~~~~~~~~~
See :ref:`json_converter_functions` and :ref:`avro_converter_functions`.
Enrichment Functions
~~~~~~~~~~~~~~~~~~~~
The converter framework provides a mechanism for setting an attribute based on a lookup
from a cache. The cache can be a literal cache in the system or in an external system such
as Redis.
- ``cacheLookup``