======================================================================
Endeca Deployment Template
GENERAL (3.2.2 Revision A)
- Updated: March 14, 2012
- Revision A consists of an update to this release note file. The
software has not changed since the 3.2.2 release.
KNOWN ISSUES (3.2.2 Revision A)
Documentation:
- The instructions in the Installation Guide do not reflect the
correct installation file name. For this release, the
installation package is named according to the Oracle part
number.
======================================================================
GENERAL (3.2.2)
- Release Date: March 2012
- The release notes may have been updated since the release date.
Contact Oracle Endeca Customer Support for the most recent version
of the release notes.
MIGRATION NOTES (3.2.2)
- If you are upgrading from a version of the Deployment Template
earlier than 3.2.1, refer to the MIGRATION NOTES (3.2.1) below.
======================================================================
GENERAL (3.2.1)
- Release Date: November 2011
- The release notes may have been updated since the release date.
Access the Product Downloads section of the Endeca Developer Network
(https://eden.endeca.com) for the most recent version of the release
notes.
KNOWN ISSUES (3.2.1)
- The documentation for this release has not been updated. Updated
documentation may be provided in a future release if necessary.
Please see the EDeN knowledge base for the most recent documentation.
MIGRATION NOTES (3.2.1)
- If you are upgrading to the 3.2.1 release of the Deployment Template,
you must also upgrade your Platform Services to version 6.1.1.
- The Deployment Template relies on a version of the Wget utility being
installed in Platform Services. This executable is not provided with
Platform Services 6.1.1. You must download and install it separately.
For Windows installations, download and deploy a Windows implementation
of GNU Wget 1.9.1 to:
%ENDECA_ROOT%\bin\wget.exe
For Linux and Solaris installations, download and deploy GNU Wget 1.9.1 to:
$ENDECA_ROOT/bin/wget
You can obtain the utility freely from the GNU project, or a Windows
implementation such as MSYS, a component of MinGW (Minimalist GNU for Windows).
======================================================================
GENERAL (3.2.0)
- Release Date: March 2010
- This release is compatible with IAP 6.1.x, MDEX 6.1.x, Workbench 2.1.x,
and CAS Server 2.2.x.
- The release notes may have been updated since the release date.
Access the Product Downloads section of the Endeca Developer Network
(https://eden.endeca.com) for the most recent version of the release
notes.
This release introduces the following bug fixes and enhancements:
* ENH: LocalMoveUtility will now throw an error if you tell it to move
a file or directory that doesn’t exist.
* BUG1270: CleanDirUtility will now throw an error if it fails to
re-create the cleaned directory.
* ENH: CAS crawls will now throw an error if they fail to complete
successfully.
======================================================================
GENERAL (3.1.0 Revision A)
- Release Date: May 2009
- This release is compatible with IAP 6.0.x, MDEX 6.1.x, Workbench 2.0.x,
and CAS Server 2.0.x.
- The release notes may have been updated since the release date.
Access the Product Downloads section of the Endeca Developer Network
(https://eden.endeca.com) for the most recent version of the release
notes.
This release introduces the following documentation bug fixes:
* BUG15819: The Deployment Template Usage Guide has been revised
with better procedures in "Specifying a CAS Server host" and
"Specifying a CAS Server as a custom component for Record Store output".
* BUG15818: The Deployment Template Usage Guide has been revised
with better procedures in "Editing a baseline CAS crawl script for a
Record Store instance" and in "Editing an incremental CAS crawl script for
a Record Store instance".
=========================================================================
GENERAL (3.1.0)
- Release Date: May 2009
- This release is compatible with IAP 6.0.x, MDEX 6.1.x, Workbench 2.0.x,
and CAS Server 2.0.x.
- The release notes may have been updated since the release date.
Access the Product Downloads section of the Endeca Developer Network
(https://eden.endeca.com) for the most recent version of the release
notes.
This release introduces the following bug fixes and enhancements:
* ENH: The Deployment Template Usage Guide has been revised to
describe working with CAS 2.0 and Record Store instances.
* ENH: Upgraded CAS integration to support CAS 2.0.
* ENH: Added new "updateGroup" property for DgraphComponent. Partial updates
will respect this new property, instead of the "restartGroup" property,
when applying updates. Index updates, configuration updates, provisioning
updates, and xquery updates will continue to respect the "restartGroup"
property. This allows users to, for example, apply partial updates to all
dgraphs simultaneously (because MDEX 6.1 supports query processing while
applying updates), while applying other updates to one dgraph at a time
(allowing a load balancer to distribute traffic while dgraphs are
temporarily unavailable).
* ENH: Improved parallelization of utility invocations during DgraphCluster
operations. Now, the DgraphCluster's applyIndex, applyPartialUpdates,
applyConfigUpdate, and reloadXqueryModules will execute operations in
parallel for all dgraphs in the same group. For example, during a
partial update, all dgraphs in the same update group will invoke the call
to the admin?op=update administrative url simultaneously.
* ENH: Previously, each server that hosted a Dgraph would maintain a copy
of all partial updates applied since the last baseline update. This can
improve recovery speed when restoring a failed dgraph or rolling back
an unwanted update, but can cause update performance to degrade over time,
as updates accumulate and all are applied to the Dgraph, instead of the
new update. The new default behavior is to only maintain the full copy of
cumulative updates on the ITL server (and just the latest update on each
MDEX server), improving update performance.
* ENH 1168: Added config/lib/xquery directory for users to store XQuery
modules. Added new LoadXQueryModules script and supporting DgraphComponent
and DgraphCluster methods to distribute XQuery modules to Dgraph servers
and instruct Dgraphs to load the modules.
* ENH 1175: Upgrade to Spring Framework 2.5.6.
* ENH 1192: Added new "--version" argument for runcommand.bat (and .sh)
script, which prints the version number of the Deployment Template's EAC
Development Toolkit.
* ENH 3436: Updated sample wine pipeline and sample CAS pipeline to remove
the primary dimension and the manipulator that associated records with the
primary dimension. These are no longer necessary, as MDEX 6.1 removed the
requirement for the primary dimension.
* ENH 3470: Removed support for the "warmupseconds" argument when invoking
the admin?op=update administrative URL on the Dgraph, as this parameter is
no longer supported by the MDEX Engine. The argument is now ignored and
a warning is logged.
* BUG 1157: baseline_update.bat now returns a non-zero status code when
the script fails. This change removes the use of tee.pl, which used to
route standard error and output to the console as well as to a file.
Logging configuration (managed in logging.properties) provides equivalent
functionality.
* BUG 1159: Removed unused --unix argument from deploy.pl script.
* BUG 1195: Resolved issue that prevented scripts from executing on Solaris.
Scripts now resolve the absolute path of the deployed application on
supported platforms when using common shells (e.g., bash).
* BUG 1198: CleanDirUtility's default behavior includes an optimization to
(quickly) test directory contents and skip cleanup if the directory is
empty. Added ability to skip this test and always execute the cleanup.
Added new "skipTestingForFilesDuringCleanup" property that can be set to
"true" for components to have them skip the directory contents test
when cleaning their directories.
* BUG 1204: Absolute path resolution failed on Linux, when scripts were
invoked in a new shell (e.g., sh runcommand.sh). Scripts now resolve
the application's absolute path relative the runcommand.sh without
requiring runcommand.sh to be on the user's PATH.
* BUG 2127: BeanShell script interpreters are now set up prior to
updating definitions, so that Agraph/Dgraph components with pre-shutdown
and post-startup scripts can execute those scripts if they need to be
restarted to pick up a provisioning update.
* BUG 2278: A DgraphCluster configured with copyInParellel set to "true"
could fail if multiple Dgraphs in the same cluster specify a different
localIndexDir, as both copies to the same server are constructed with
the same token (identifier). Tokens now include the name of the Dgraph,
and utilities invoked with Utility.runInParallel will automatically
attempt to avoid conflicts by appending a timestamp to avoid invoking
utilities with the same identifier at the same time.
* BUG 3004: load_partial_test_data.sh script updated to use "=" instead
of "==" operator, allowing the script to run on Solaris.
* BUG 3162: deploy.sh script updated to remove references to undefined
"usage" function.
* BUG 3573: Sample wine pipeline and sample CAS pipeline updated to use
recommended wilcard configurations. Both pipelines now specify a value
of 3 for DICTIONARY_MAX_NGRAM_LENGTH and MAX_NGRAM_LENGTH.
* BUG 4199: Enhanced ForgeComponent.timestampPartials method to match
a broader set of output file names when applying timestamps. Now, a
timestamp is applied to any file in the Forge output directory that
matches the following pattern: [dataPrefix](-partN)-sgmtX.records.xml(.gz).
The method now supports a compression-enabled update adapter, Agraph
deployments (with and without parallel Forge), as well as multiple
file segments when output exceeds Forge's configurable maximum file size.
* BUG 4342: Enhanced WgetUtility and its subclasses to support communication
with a secure MDEX Engine by specifying client and CA certificates.
DgraphComponent will now invoke administrative URLs (admin?op=update,
config?op=update, admin?op=flush, and admin?op=reload-services) with the
appropriate certificates when the Dgraph is enabled for SSL.
=========================================================================
(3.0.0)
* Updated for use with IAP 6.0. This release introduces the following
enhancements:
- Updated installer to ask the user if they want to integrate with
IAP Workbench (formerly Web Studio) and, if so, what port to use.
Updated default install_config.xml file to include settings to
enable Workbench integration and specify the default Workbench
port (8006).
- ConfigManagerComponent includes new "webStudioEnabled" property.
All interaction with IAP Workbench hinges on this property, taking
no action if the property is not set to "true." For backward
compatibility, applications that don't specify this setting have
this property set to "true" by default.
- Default BaselineUpdate and ConfigUpdate scripts interact with
IAP Workbench only when Workbench integration is enabled.
- initialize_services.[bat|sh] now calls a new InitialSetup script,
rather than uploading configuration to IAP Workbench. If Workbench
integration is enabled, the InitialSetup script uploads configuration
as before.
- runcommand.[bat|sh] no longer finds and specifies eac.ks and ca.ks
Java keystores and truststores for use when communicating with an
SSL-enabled EAC Central Server. These default keystores are no
longer provided by Endeca. For information about enabling SSL and
using Endeca's certificate utilities to generate custom SSL
certificates, see the Endeca Security Guide.
- Removed CrawlerComponent and references to EAC type
CrawlerComponentType. This deprecated component has been removed
from the EAC.
- Updated report generation scripts to specify the directory from
which Web Studio should read its reports. Reports are now delivered
to a location specified in a "webstudio-report-dir" property on a host
that uses the reserved Workbench host name "webstudio."
- Updated DgraphComponent and DgraphUpdateUtility to discontinue support
for the "offlineUpdate" property and the "&offline=[true|false]"
parameter it used to create on the admin?op=update URL. MDEX 6.0
no longer supports online updates and returns an error if
"offline=false" is specified, so the Deployment Template now logs a
warning and ignores the unsupported behavior.
KNOWN ISSUES:
- BUG 1195: On Solaris, logic added in 2.2 to support path independence
causes EAC provisioning to fail. This occurs because, on this OS,
set_environment.sh does not resolve ENDECA_PROJECT_DIR to
an absolute path. A simple workaround is to edit AppConfig.xml and
set the element under to specify the absolute
path to the deployed project directory, rather than the variable
${ENDECA_PROJECT_DIR}. This workaround effectively reverts the
feature described in ENH 5616 below.
- BUG 14341: On Linux and Solaris, provisioning may fail under rare
circumstances when attempting to create custom directories. For
example, when provisioning the "webstudio" host, attempting to
create the "webstudio-report-dir" custom directory may return an
error like "Directory webstudio-report-dir
(/localdisk/apps/myApp/control/.././reports) could not be created."
Users can work around this issue by editing the AppConfig.xml file
and setting the element under to specify the
absolute path to the deployed project directory, rather than the
variable ${ENDECA_PROJECT_DIR}.
=========================================================================
(2.3.0)
* Migration Notes from 2.2 to 2.3: Due to the flexible nature of the
DT and the opportunities for customization, specifying a universal
migration path from DT 2.2 to DT 2.3 is impossible. It is the job
of the DT user to know how they have customized their deployment
so that the appropriate modifications can be retained. The updates
in DT 2.3 include support for CAS 1.2. These updates have affected
numerous files through renaming and as actual code changes. The
following is a full list of the affected files and can act as a
launching point for migration. Below, the left side of --> is the
2.2 filename and the right side is the 2.3 filename. The paths are
defined relative to an unmodified application directory.
Changed only:
./config/lib/java/casStubs.jar
./config/lib/java/eacComponents.jar
./config/crawl_script_templates/README
Renamed only:
./data/complete_filesys_crawl_output/
--> ./data/complete_cas_crawl_output/
Renamed and changed:
./config/crawl_script_templates/@CRAWL_NAME@FsCrawlConfig
--> ./config/crawl_script_templates/@CRAWL_NAME@CasCrawlConfig
./config/fs_crawl_pipeline/
--> ./config/cas_crawl_pipeline/
./config/fs_crawl_pipeline/pipeline.epx
--> ./config/cas_crawl_pipeline/pipeline.epx
./config/fs_crawl_pipeline/pipeline.lyt
--> ./config/cas_crawl_pipeline/pipeline.lyt
./config/lib/perl/make_fs_crawl_scripts.pl
--> ./config/lib/perl/make_cas_crawl_scripts.pl
./config/script/endecaFsCrawlConfig.xml
--> ./config/script/endecaCasCrawlConfig.xml
./config/script/fetchFsCrawlDataConfig.xml
--> ./config/script/fetchCasCrawlDataConfig.xml
./control/cas/*_fs_*.bat
--> ./control/cas/*_cas_*.bat
./control/cas/*_fs_*.sh
--> ./control/cas/*_cas_*.sh
In your old ./config/script/, any instantiations of
@CRAWL_NAME@FsCrawlConfig should be remade using
./control/cas/make_cas_crawl_scripts.
* ENH: Updated to support CAS 1.2 servers.
* BUG 1606: Updated to handle all CAS output file formats, including
compressed and uncompressed XML.
=========================================================================
(2.2.0)
* ENH: Scripts, shell/batch wrappers, sample pipeline and documentation
added to integrate (and document implementation practices for) file
system crawls powered by the CAS.
* ENH 5817: Enhanced deploy.pl installer to support custom token
substitution. Users can now specify in app descriptor XML files
the names of tokens to substitute when filter-copying files, and
the question to prompt the user (or the option to parse in an
install config file) to retrieve the value for that token. By
default, the app descriptors for the dgraph and agraph deployments
now use this functionality to prompt the user for agraph, dgraph,
logserver and forge server ports.
* ENH 5616: Deployed application enhanced to be "path independent," by
locating deployed location relative to the invoking scripts. App
can now be moved post-deployment and will re-provision itself
to the new location. Custom log handler implemented to support
relative file paths.
* BUG 5358: Documented sample EndecaMDEXUpdateScript fixed to use
isSslEnabled boolean argument when instantiating GetWSDgraphFilesUtility.
* BUG 5314: Update default report_settings.xml to inclued A/B test report.
* ENH 4879: Adding default java argument of -Xmx1G to all report
generator components.
* BUG 3875: Documentation updated to use isSslEnabled boolean argument
when instantiating CopyUtility.
* BUG 3755: AppConfig.xml now properly references "numStateBackups" for
Forge component, instead of "numStateArchives."
* BUG 3750: AppConfig.xml fixed to determine ".bat" or ".sh" script suffix for
provisioned report generator scripts at deployment time.
* EDT ENH: New custom component ContentAcquisitionServerComponent implemented
to support CAS-based file system crawls.
* EDT ENH: New property "numIdleSecondsAfterStop" allows configurable
pause after a component stops, ensuring file locks are released
before proceeding.
* EDT BUG: Fixed init() method for RollbackUtility to set directory specified
as init argument.
* EDT ENH 5830: Component object now holds a reference to the Host object
on which the component runs, providing easier access to hostname/port
information.
* EDT BUG 5829: ConfigManager.updateWsConfig method no longer filters
inactive merch rules when retrieving Web Studio files. New methods
downloadWsConfig and downloadWsDgraphConfig now accept a boolean arg
to specify whether they should filter merch rules. No-arg methods
continue to filter inactive merch rules when invoked.
* EDT BUG 5822: ConfigManager.updateWsConfig method now uploads config files
to Web Studio, even if no files are specified as being maintained by
Web Studio (i.e. the method will upload dev studio-maintained files).
* EDT ENH 5818: Introduced caching of application definition during provisioning
check, to require fewer requests of the app definition from the EAC.
* EDT ENH 5701: Components now allow a configurable number of failed status
queries before assuming a component has failed. By default, 5
consecutive failed status queries are allowed before a component
is assumed failed.
* EDT ENH 5639: Upgraded Spring version used to 2.0.8 (latest 2.0.x. release).
* EDT ENH 5560: Enhancing eacToolkit.xsd to allow ${token} values to
be specified for boolean, int and long values.
* EDT ENH 5513: Dgraphs now support "offlineUpdate" property (defaults to
"true") to specify whether dgraph should remain online while
processing partial updates. Also supports "numCacheWarmupSeconds"
(incompatible with online updates) to specify number of seconds
to spend warming cache before going back online.
* EDT BUG 4939: Utility tokens (IDs) will be automatically truncated to
ensure the token name and the associated log directory (i.e.
ENDECA_CONF/logs/.../app.token.log) are shorter than 255 characters,
to avoid Windows path length limitations and pre-5.1.3 EAC data
store field length limits.
* EDT ENH 4889: Components can now specify numLogBackups=-1 to clear the
log directory, rather than archiving it. If numLogBackups=0, log dir
is not archived, but most components will append, so the new -1
setting may be useful for dev environments, to save disk space.
* EDT BUG 4282: Fixed SSL-argument handling for agraph, dgraph, forge to
treat "cipher" is an optional argument.
* EDT ENH 3974: Agraph and dgraph components allow scripts to be executed
before the component is stopped and/or after the component is
started. New "pre-shutdown-script" and "post-startup-script"
attributes can specify the names of Scripts defined in AppConfig.xml.
Component will invoke the BeanShell script before shutdown or
after startup, allowing common actions like signaling a load balancer
that the dgraph should be added or removed. BeanShell scripts used
for this purpose have a new variable, "invokingObject," which
references the dgraph or agraph that invoked the script, allowing
the same script to be executed for multiple dgraphs.
* EDT ENH 3957: ConfigManager's webStudioMaintainedFile now detects when
names include data prefix, adding it only when not specified (e.g.
wine.thesaurus.xml and thesaurus.xml can now be specified).
* EDT ENH 3837: Added wildcard support for specifying webStudioMaintainedFiles
in ConfigManager custom component.
* EDT BUG 3821: Provisioning must always set/update hosts before
components, as the EAC will error when components are added
before their associated host is defined.
* EDT ENH 3691: Enhanced logging to include host IDs in "starting" messages
for all components and utilities (e.g.
[02.24.08 21:10:25] INFO: [ITLHost] Starting component 'Dgidx'.)
* ENH 3505: Updated javadoc to clarify that Host ID (not hostname) must be
specified to FileUtility method invocations.
* EDT ENH 2952: Server components will now poll for status stopped or
failed after sending asynchronous "stop" command. New property
"shutdownTimeout" to specify non-default time to wait.
=========================================================================
(2.1.0)
* Implemented EAC agraph deployment powered by Java EAC Development
Toolkit, with support for baseline index updates and Web Studio
integration.
* ENH: Added support for agraph and parallel forge.
* ENH: Implemented AgidxComponent.
* ENH: Implemented AgraphComponent.
* ENH: Implemented AgraphCluster.
* ENH: Implemented ForgeCluster.
* ENH: Implemented IndexingCluster.
=========================================================================
(2.0.1)
* BUG: Fixed a bug in the installer that caused it to stop copying if any
copy pattern matched no files.
* ENH: Added report generation scripts.
* EDT BUG: LocalMoveUtility updated to strip trailing slash from source path.
* EDT BUG: Updated Controller JVM args and Utility invocations to support SSL-
enabled communication with the EAC.
* EDT ENH: Added acquireLockBlocking and waitForFlagSet methods to allow blocking
while waiting to acquire a lock.
* EDT ENH: Added support for multiple --app-config files.
* EDT ENH: Added support for --config-override properties files.
* EDT ENH: Exposed all ForgeComponent methods as public.
* EDT ENH: Implemented CrawlerComponent.
* EDT ENH: Added support for an auto-executed BeanShell script to run for each
BeanShell Interpreter, allowing common imports to be defined centrally
and made available in all scripts.
* EDT ENH: Implemented isPartialDataReady, getPartialIncomingData and
timestampPartials methods on ForgeComponent, removing the need to shell
out to Perl scripts for these methods.
* EDT ENH: Enhanced LogServerComponent with methods for identifying logs by
data range, rolling the log and delivering logs for a requested date
range for report generation.
* EDT ENH: Enhanced ReportGeneratorComponent with methods for updating
RG provisioning to output files/charts named with a timestamp (to conform
to Web Studio's naming convention) and for delivering output reports/charts
to Web Studio.
=========================================================================
(2.0.0)
* Removed legacy Perl EAC and Control Interpreter deployments from the
2.0 release.
* Implemented EAC dgraph deployment powered by Java EAC Development
Toolkit, with support for baseline and partial index updates and
Web Studio integration.
=========================================================================
(1.8.0)
* Added support for partial updates to Perl EAC dgraph deployment.
* (BUG2688) Updated agraph control script to remove duplicate agraph
and dgraph process/request logs. All logs are now written to
logs/dgraphs/[CLUSTER]_[DGRAPH] and
logs/agraphs/[CLUSTER]_[AGRAPH]. Initial setup and archiving
bricks updated to match.
* (BUG2613) Updated set_environment.[bat|sh] scripts to avoid setting
irrelevant ENDECA_PROJECT_OVERRIDE environment variable for EAC
deployments.
=========================================================================
(1.7.0)
* Updated EAC dgraph deployment to support changes in the EAC WSDL
introduced in IAP 5.0.1. Refer to the Migration Guide (Chapter
5: Incremental Changes from 5.0.0 to 5.0.1) for details about
the changes.
* (BUG2526) Updated .sh/.bat wrapper scripts that execute baseline and
partial updates to redirect both STDERR and STDOUT to screen and
to a log file.
* (BUG2454) Updated installer to replace indexer adapter's output
prefix with the name of the installed project. This removes a
warning when opening the pipeline in Dev Studio.
=========================================================================
(1.6.0)
* Added agraph deployment type, including support for parallel forge
and baseline index updates. Sample pipelines are provided for
both parallel and non-parallel forge deployments.
* Updated EAC deployment to conform to IAP 5.0.1 component/utility
status message formats (e.g. "NotRunning" instead of
"NOT_RUNNING".
=========================================================================
(1.5.0)
* Added support for partial updates, including control script
enhancements and test data.
* Added support for file-based configuration of the deployment
template installer. New install_config.xml file may be used
instead of interactively configuring the installer at runtime.
* Enhanced installer to use a config file to define the application
being installed. External modules and additional features are
installed by providing new/additional application descriptor files.
The installer now also supports use for installing/deploying custom
or project-specific deployments by allowing developers to write
their own installer configuration, including directory structure
and files to distribute.
* Added EAC-powered deployment template for dgraph, supporting
baseline index updates.
* Added documentation to describe the procedure for installing a
pipeline into a deployment template, including required changes
to pipeline paths and file prefix names.
* (ENH2193) Updated installer to validate application name against
regex /^[a-zA-Z0-9]+$/, to avoid special characters that may
cause problems for some processes.
* (ENH1880) Updated installer to update filenames for the sample wine
config files, changing the file prefix during installation to
match the project name.
* (ENH2191) Added config/lib directory with perl and java
subdirectories.
* (ENH2189) Added --logDir flag for forge, to redirect logs generated
by forge components to the baseline logs directory.
* (ENH2253) Added --inputDir flag for forge, to specify where forge
should read data and dimensions. This will enable cleaner
integration with Manager, as the --inputDir flag is always
specified by Manager.
* (BUG1945) Enhanced shell/batch wrapper scripts to reference Perl
relative to the ENDECA_ROOT environment variable and to validate
paths and executables before proceeding.
=========================================================================
(1.1.0)
* (ENH1829) Updated documentation to describe (1) configuring a Unix
deployment to run SSL-enabled JCD, (2) updating the deployment
control script to use SSL for communication with the JCD (especially
when deploying on Windows, where SSL is enabled by default).
* (ENH1836) The 4.8 Report Generator has been updated to use a
report_settings.xml configuration file (replacing the old
report_config.xml) and the control script brick has been updated to
use a "settings" parameter (replacing the old "config" parameter).
Updated the control script and deployment scripts to support use of
the template on IAP 4.7 and 4.8.
=========================================================================
(1.0.0)
* Initial release