2014-03-26

Imagine that there are hundreds of JPEG files stored in a specific folder.
The goal is to create world files (*.wld) so that the images will be arranged in tile shapes when displaying them on a map viewer (e.g. ArcMap).
Width and height of a tile, and number of columns are specified as constants. Aspect ratio of every image is approximately equal to the ratio of a tile, but their sizes are various.

Required result (simplified example: number of images = 9, number of columns = 3)

A world file is a plain text file in which these 6 parameter values have been written.
Line 1: pixel size in the x-direction in map units/pixel
Line 2: rotation about y-axis
Line 3: rotation about x-axis
Line 4: pixel size in the y-direction in map units, almost always negative
Line 5: x-coordinate of the center of the upper left pixel
Line 6: y-coordinate of the center of the upper left pixel
-- World file, Wikipedia

Line 2 and 3 can be always 0 since it's not necessary to rotate images in this case. I have to calculate other values (Line 1, 4, 5, 6), and write the lines into text files for every image.
Except extension, path of a world file has to be same as associated JPEG file.

OK. The requirement and conditions have been clarified. This image shows a workspace example.

Firstly I defined these user parameters for the constants.
TILE_WIDTH (Float): horizontal lenghth of a tile
TILE_HEIGHT (Float): vertical length of a tile
TILE_COLUMNS (Integer): number of columns of tiles

The JPEG reader reads every image, the Counter appends 0-based sequential number (_count) to each raster feature, and the RasterPropertiesExtractor extracts raster properties.
"_count" and these two properties will be used in the next step.
_num_columns: width of the image (pixels)
_num_rows: height of the image (pixels)

The AttributeCreator calculates values which should be written into a world file. Here I assigned the values to elements of a list attribute named "text_line_data{}".

The AttributeKeeper removes unnecessary attributes, and the GeometryRemover removes the geometry (raster). These transformers are not essential, but removing unnecessary objects is effective to reduce memory usage especially when exploding the feature.

The ListExploder explodes the feature to create features each of which has a text line.
Finally, the TEXTLINE writer with fanout option writes the text lines into each world file for every image. This image shows the fanout setting.

All the transformers and the TEXTLINE writer can be replaced with a PythonCaller.
-----
# Python Script Example: Create World Files to Arrange Images in Tiles
import fmeobjects

2014-03-25

I tested some FME Functions using the TclCaller in the previous article, but any FME Function can be called also in a Python script embedded in a PythonCaller.
FMEFeature.performFunction() method can be used to call any FME Function on the feature.
Here I give examples of Python edition.

Example 3: Set Z Values to Coordinates
Assume input feature has a list attribute named "_z{}" storing numeric values. This script set Z values stored in the list to coordinates of the feature when number of the list elements is equal to number of the coordinates.
-----
import fmeobjects
def setZValues(feature):
n = int(feature.performFunction('@NumCoords()'))
# The line above can be replaced with:
# n = feature.numCoords()
if 0 < n and n == int(feature.performFunction('@NumElements(_z{})')):
feature.performFunction('@Dimension(2)')
# The line above can be replaced with:
# feature.setDimension(fmeobjects.FME_TWO_D)
feature.performFunction('@ZValue(_z{})')
feature.setAttribute('_result', 'success')
else:
feature.setAttribute('_result', 'failure')
-----

I've never used the FMEFunctionCaller transformer, performFunction method (Python) and FME_Execute procedure (Tcl) in any practical workspace, since the documentation about FME Functions had not been accessable.
But now, the documentation has become available. FME Functions could be used effectively in some cases.

2014-03-22

In a workspace, any FME Function can be called with the FMEFunctionCaller transformer. If the function returns a value, it can be also used in an expression for value setting of many transformers such as AttributeCreator, ExpressionEvaluator etc.. And also, as I mentioned before, they can be called directly from a Tcl script embedded in a TclCaller.
-----
2014-03-25: Of course any FME Function can be also called using the PythonCaller.
See the next article > FME Function and PythonCaller
-----

I tested some functions with TclCaller. Use FME_Execute proc. to call a function in the script.
The syntax is:
FME_Execute <function name without @> [<arg1> <arg2> ... ]

Example 1: Merge Multiple List Attributes
Using @MergeLists function, two or more list attributes can be merged at once.
@RemoveAttributes function removes all the specified attribute(s) including list.
Assume input feature has two list attributes named "_list1{}", "_list2{}". This script creates a new list attribute named "_merged{}" by merging them and removes the original lists.
-----
proc mergeLists {} {
FME_Execute MergeLists "_merged{}" "_list1{}" "_list2{}"
FME_Execute RemoveAttributes "_list1{}" "_list2{}"
}
-----
If you don't need to remove the original lists, the FMEFunctionCaller can be used with this parameter setting.
-----
FME Function: @MergeLists(_merged{},_list1{},_list2{})
-----
# Why isn't there "ListMerger" transformer?

Example 2: Remove Duplicate Vertices from Polygon
@GeometryType function is interesting.
If no argument was given, it returns a geometry type identifier of the feature. e.g. fme_point, fme_line, fme_polygon etc..
If fme_polygon was given as its argument, it removes duplicate vertices from the polygon.
Other than above, this function has also several options.
This script identifies geometry type of input feature and removes duplicate vertices when the type is fme_polygon.
-----
proc removeDuplicateVertices {} {
if {[string compare [FME_Execute GeometryType] fme_polygon] == 0} {
FME_Execute GeometryType fme_polygon
}
}
-----
If geometry type of input feature is always fme_polygon, the FMEFunctionCaller can be used with this parameter setting.
-----
FME Function: @GeometryType(fme_polygon)
-----

Example 3: Set Z Values to Coordinates
@NumCoords returns number of coordinates of the feature.
@NumElements <list name> returns number of the list elements.
@Dimension 2|3 forces the feature to 2D|3D.
@ZValue <list name> set Z values stored in the list to coordinates of the feature.
Assume input feature has a list attribute named "_z{}" storing numeric values. This script set Z values stored in the list to coordinates of the feature when number of the list elements is equal to number of the coordinates.
-----
proc setZValues {} {
set n [FME_Execute NumCoords]
if {0 < $n && $n == [FME_Execute NumElements "_z{}"]} {
FME_Execute Dimension 2
FME_Execute ZValue "_z{}"
return "success"
} else {
return "failure"
}
}
-----
In my testing, @ZValue function did nothing if coordinates had Z values already. So I've used @Dimension function to force the feature to 2D beforehand.

2014-03-17

The green shape is a 3D polygon. Consider calculating its perimeter and area in 2D or 3D.

The calculation itself is easy. Just use a LengthCalculator and an AreaCalculator.

Parameter Settings

Transformer

Parameter Name

Choice for 2D

Choice for 3D

LengthCalculator

Length Dimension

2

3

AreaCaluculator

Type

Plane Area

Sloped Area

Now, assume that there is a requirement that either 2D or 3D has to be determined through a user parameter at run-time.
It's easy to create two published parameters linked to the two transformer parameters separately, but the user will have to always be careful of their consistency. Ideally, it's better that the two parameter values can be specified simultaneously through setting just one published parameter. In other words, the two parameter values should be synchronized.

If value choices of the two transformer parameters were same, you could link both of them to the same user parameter. However, as shown in the table, those are different in fact.
In such a case, I would create a published parameter and a private scripted parameter so that value of the private parameter would be determined depending on value of the published parameter.
For example:

And then, specify $(DIM) to "Length Dimension" of the LengthCalculator, $(AREA_TYPE) to "Type" of the AreaCaluclator. User doesn't need to be careful of the consistency of the two parameter values no longer.

Other than the LengthCalculator and AreaCalculator, there are several transformers which have a parameter choosing 2D or 3D. Parameter names and value choices are various.

Transformer

Parameter Name

Choice for 2D

Choice for 3D

MeasureGenerator

Length Dimension

2

3

Snipper

Measurement Mode

2D

3D

GeometryValidator
(Duplicate Consecutive Points)

Check Z Values

No

Yes

etc.

=====
At first, I expected that I could do that with "Choice with Alias" parameter, rather than script.
-----
Type: Choice with Alias
Name: AREA_TYPE
Configuration: 2,Plane<space>Area%3,Sloped<space>Area
Default Value: $(DIM)
-----
But unfortunately value of this parameter became the same value as $(DIM), i.e. 2 or 3.
Why not?

2014-03-15

From an actual job which I've completed this week. The requirement was to transform 3D polyline like this image into pipe-like solids.

Through the following manipulations, cylindric solids along the line can be created.

Step 1

Chopper divides the polyline into individual line segments.LengthCaluculator calculates 2D length of the line segment (_length).
Two CoordinateExtractor extract coordinates of start and end points (x0, y0, z0), (x1, y1, z1).2DEllipseReplacer and 3DForcer create horizontal circle whose center is located at (x0, y0, z0).

Step 2

AttributeCreator calculates components of vector representing direction of the line segment.
vx = x1 - x0
vy = y1 - y0
vz = z1 - z03DRotator rotates the circle so that the face turns to direction of the vector. PI is a user parameter whose value is the circular constant.
Note: There is a bug on the 3DRotator (Custom Axis mode, FME 2014 build 14235), it doesn't work correctly when more than one axis direction has been specified. But it works fine if the rotational origin was (0, 0, 0). So I used two Offsetter transformers in the workflow as a workaround for the bug. After the bug having been fixed, remove the Offsetters and specify (x0, y0, z0) to "Origin X, Y, Z" of the 3DRotator.
Regarding the bug, see also here (Community members only) > Chatter: 3DRotatorProblem

2014-03-08

A Python script example for the task.
I think the getVertices function in this script can be used for general purpose.
-----
# PythonCaller Script Example: Classify Polygon Shape
# 2013-03-09 Added triangle classification.
# Classify a polygon into triangle, quadrilateral, and other.
# Triangle will be further classified into right, isosceles, and equilateral.
# Quadrilateral will be further classified into trapezoid, parallelogram, rectangle, and square.
import fmeobjects

The mission was to create a raster based on the data. As a requirement, each cell of the resultant raster would have to contain the original grid point at its center. That is, I had to create a raster whose cell spacing matches with the spacing of the grid points.
It was known that x, y spacings are constant, but the exact values were unknown. So I needed to calculate the spacings based on the point geometries.

At first, inspecting the data with the Data Inspector, I determined approximate ranges, and defined them as user parameters.
APPROX_MIN_X_SPACING, APPROX_MAX_X_SPACING
APPROX_MIN_Y_SPACING, APPROX_MAX_Y_SPACING

Then, I created a workflow like this to calculate the spacings.

In the actual dataset, x-spacing and y-spacing were the same value, but the workflow works even if they are different values.

There might be a quick function which calculates the spacing of grid points in FME, but I wasn't able to find it. If you know, tell me that!

2014-03-01

Think about setting measure values to every vertex of line geometries. Assume that the measure value has to be distance from start node of the line which the vertex belongs to.If the coordinate system for measuring is same as the source coordinate system, a MeasureGenerator can be used simply.If not, a quick way is to perform reprojection before and after the MeaureGenerator.Reprojector -> MeasureGenerator -> ReprojectorBut output line geometry may not be strictly same as input line, since interpolation is performed in reprojection process in general. Although the error caused by reprojection is slight amount, cannot be avoided.If such an error will not be allowed, this workflow would be a workaround.

The pattern of the workflow - adding temporary ID, branching feature flow into multiple streams, and merging them after processing - is used in many cases. I think there are many general patterns which frequently appear in FME workspaces. I personally call them "Standard methods".
The pattern above is typical one. I call it "Counting-Branching-Merging" method, provisionally.
How do you call it?

=====
2014-03-04: I got a great comment on this subject. That is a suggestion regarding another workaround with the GeometryExtrator and GeometryReplacer like this.

In my quick test for a large dataset, this method was apparently more efficient in both memory usage and processing speed (Geometry Encoding: FME Binary). And also the original geometries have been restored exactly. I'm going to research further more in another opportunity.
I've never noticed such an effective usage of GeometryExtractor / Replacer. I would add the pattern of "ExtractGeometry-Processing-RestoreGeometry" to my "Standard methods".
Thanks for the input!

P.S. It is a basic usage of the GeometryExtractor / Replacer. I was blind ..."This transformer is often used to make a copy of the feature's geometry into an attribute before some temporary geometry change is made, so that it can later be restored."-- Help on the GeometryExtractor, FME 2014"This transformer is typically used to restore geometry previously extracted into an attribute by the GeometryExtractor."-- Help on the GeometryReplacer, FME 2014

-----
If "ExtractGeometry-Processing-RestoreGeometry" method would be used generally in many scenarios, it might be more convenient that the GeomeryExtractor and GeometryRepalcer have an optional parameter "Coordinate System Attribute", so that the method can extract and restore coordinate system without using the CoordinateSystemExtractor / Setter if necessary.