Archive

For faster and better search functionalities EMC has developed a Full Text Index Server which is installed separately with the Content management software to provide an index based search capability. In version 5.2.5 SPx the full text search engine was using Verity which has been now changed to FAST (Fast Search & Transfer) in 5.3 SPx onwards and xPlore replaced further.

In Verity we have to explicitly define the attributes to be indexed, in the content server configuration, whereas one of the salient features of FAST is that by default, all the attributes are indexed along with the content of the document.Since, FAST is no longer tightly coupled with the installation of the content server; one has the option of not installing Index Server. If the Full Text Index server is not installed, simple search will perform a case sensitive database search against object_name, title and subject attributes of dm_sysobject and its subtypes.

This post describes the various components of Index Server and their operations.

1. Software Components

Full-text indexing in a Documentum repository is controlled by three software components:

Content Server, which manages the objects in a repository, generates the events that trigger full-text indexing operations, queries the full-text indexes, and returns query results to client applications.

The index agent, which exports documents from a repository and prepares them for indexing.

The index server, which is a third-party server product that creates and maintains the full-text index for a repository. The index server also receives full-text queries from Content Server and responds to those queries.

2. Set Up Configuration

a) Basic Set Up

The basic indexing model consists of a single index agent and index server supporting a single repository. The index agent and index server may be installed on the Content Server host or on a different host.

b) Consolidated Set Up

In a consolidated deployment, a single index server provides search and indexing services to multiple repositories. The repositories may be in the same Content Server installation or on different hosts. However, all repositories must be of the same Content Server version.

3. Index Server Processes

The index server consists of five groups of processes that have different functions.

a) Document processors

Document processors (also sometimes called procservers) extract indexable content from content files, convert DFTXML to FIXML (a format that is used directly by the indexer), and merge the indexable content with the metadata during the DFTXML conversion process. Document processors are the largest consumer of CPU power in the index server.

b) Indexer

The indexer creates the searchable full-text index from the intermediate FIXML format. It consists of two processes. The frtsobj process interfaces with the document processor and spawns different findex processes as necessary to build the index from FIXML.

c) Query and Results servers

The QR Server (Query and Results Server) is a permanently-running process that accepts queries from Content Server, passes queries to the fsearch processes, and merges the results when there are multiple fsearch processes running.

The index server can run in continuous mode or in a special mode called suspended mode. In suspended mode, FIXML is generated for any updates to the index but not integrated into the index. When the index server is taken out of suspended mode, the index is updated. Running in suspended mode; speeds up the indexing process. Suspended mode should be used when the requirement is to index large volume of documents or to re-index an entire repository.

Navigate to “Matching Engines” tab for details on total no. of documents in all the filestores (if there are multiple repositories) and the no. of documents processed by Index Server. It also provides a link to Index Server log file.

5. How to determine Index Server ports

Using the Index server base port we can determine the ports for various Index Server processes

Index Server admin console: Base Port + 3000

FAST Search console: Base Port + 2100

6. Fulltext indexing queue messages

When a document has been marked and submitted for fulltext indexing, it is queued to the Index Agent/Index Server.

‘’ – The item is available to be picked up by an Index Agent for indexing.

‘acquired’ – The item is being processed. If an Index Agent stops abruptly, a queue item can be left in this state until the Index Agent is restarted or an Administrator clears the queue item.

‘warning’ – The item was indexed with a warning. Often it indicates that the content of the object failed to index but the meta-data was successfully indexed.

The ‘message’ attribute and the Index Agent log will have further details.

‘failed’ – The item failed to index, please refer to the ‘message’ attribute and the Index Agent log for more information.

‘done’ – Successfully indexed the item

7. Index Agent Modes

An index agent may run in one of three operational modes:

normal

In normal mode, the index agent process index queue items and prepares the SysObjects associated with the queue items for indexing. When the index agent successfully submits the object for indexing, the index agent deletes the queue item from the repository. If the object is not submitted successfully, the queue item remains in the repository and the error or warning generated by the attempt to index the object is stored in the queue item.

migration

In migration mode, the index agent processes all SysObjects in a repository sequentially in r_object_id order and prepares them for indexing. A special queue item, the high-water mark queue item, is used to mark the index agent’s progress in the repository.

An index agent in normal mode and an index agent in migration mode cannot simultaneously update the same index.

file

In file mode, a file is used to submit a list of objects IDs to the index agent when a new index is created and index verification determines which objects are missing from the index.

8. Switching modes of Index Agent

At the time of Index Agent set up the wizard gives an option to start the Index Agent under “Normal” or “Migration” mode.

The following steps should be performed to change the Index Agent from one mode to another.

3. Now change the Index Agent mode from Normal to Migration or Migration to Normal as the case may be.

4. Click on OK

5. Start the Index Agent again.

Note:

i) While in Migration mode Index Agent doesn’t appear in DA under Indexing Management tab. On the Index Agent admin screen it will provide the details of the no. of documents processed out of the total no. of documents.

ii) If the Index Agent service is restarted from services console then start the Index Agent from Index Agent Admin console or through DA under Indexing Management tab.

9. Re-configuring Index Agent and FAST

Configuring another IA and FAST to a repository previously configured to work with one IA and FAST doesn’t modify dm_ftengine_config object and IA fails to start displaying error to connect to old FAST machine.

To resolve:

Manually update the dm_ftengine_config object based on the settings from the new machine

1. Go to IAPI and execute the following API –

iapi> retrieve,c,dm_ftengine_config

…

2. Note the object_id retrieved by the above API and use it to execute the following API –

iapi> dump,c,l

…

3. In the dump results note the following param_name, param_value pairs

fds_base_port should match 13000 or the base port number for Index Server Install

fds_config_host should match the host name where the Index Server is installed.

and so on….

4. The param_name/param_value pairs should be changed to match the values for the index server install.

5. Delete the following via dql:

delete dm_ftengine_config object where r_object_id = ‘old_value’

delete dm_ftindex_agent_config object where r_object_id = ‘old_value’

6. Run the index agent configuration program to create new index agent.

10. Relocating fulltext indexes in Index Server

The following steps describe how we can change the location of fulltext indexes

1. Shutdown the Index Agent

2. Shutdown the Index Server

3. Copy the indexes to the target location ( both the fixml and the index directories)

In Windows: %DOCUMENTUM%/fulltext/IndexServer/etc/config_data/RTSearch/ webcluster/rtsearchrc.xml. Change fixmlpath and fsearchdatasetdir to the new path

In Unix: $DOCUMENTUM/fulltext/IndexServer/etc/config_data/RTSearch/webcluster/rtsearchrc.xml. Change fixmlpath and fsearchdatasetdir to the new path

7. Startup IndexServer, Index Agent

11. dm_FTCreateEvents Job

The Create Full-Text Events tool (dm_FTCreateEvents) may be used in two ways:

a) To complete an upgrade by causing any objects missed by the pre-upgrade indexing operations to be indexed.

The job generates events for each index able object added to a repository between the time a new 5.3 or later full-text index is created for a 5.2.5 repository and when the repository is upgraded to 5.3.

This is the out-of-the-box behavior of the job.

b) To generate the events required to re-index an entire 5.3 SP1 or later repository.

Re-indexing the repository does not require deleting the existing index.

Please refer to the screenshot for the configuration of dm_FTCreateEvents Job –

To generate the events required to re-index an entire 5.3 SPx or later repository the -full_reindex argument must be set to TRUE to generate the required events.

The first time the job runs in its default mode, the job determines the last object indexed by an index agent running in migration mode and the date on which that object was indexed. The job searches for objects modified after that date and before the job runs for the first time and generates events for those objects. On its subsequent iterations, the job searches for objects modified after the end of the last iteration and before the beginning of the current iteration.

Before the job is run in a 5.3 SP1 or later repository with argument –full_reindex set to TRUE, you must create a high-water-mark queue item (dmi_queue_item) manually using the API –

create,c,dmi_queue_item

save,c,l

and specify the r_object_id of the queue item as the -high_water_mark_id argument of the dm_FTCreateEvents Job.

In case you get the following error message in the job’s report –

FTCreateEvents was aborted. Error happened while processing job. Error: No high water mark found for qualification:

Verify the -high_water_mark_id attribute and check whether the API was executed after installation or re-installation of the index server to get the required r_object_id argument.

Disable the job if the application is not using Full Text Index Server.

The job can also be de-activated if the following events are registered for ‘dm_fulltext_index_user’

dm_save

dm_destroy

dm_readonlysave

dm_checkin

dm_move_content

Execute the following query to verify the same:

select event from dmi_registry where user_name=’dm_fulltext_index_user’

Note:The instruction above save a Superuser name and password to the file system in a plain text parameter file. For security reasons, you may wish to remove that information from the file after running the FTIntegrity tool. It is recommended that you save the parameter file in a location accessible only to the repository Superuser and installation owner.

To run the index verification tool:

Navigate to %Documentum%\fulltext\IndexServer\bin (Windows) or

To verify both completeness and accuracy, open a command prompt and execute

cobra ftintegrity.py -i ftintegrity.params.txt -m b

To verify completenes only, open a command prompt and execute

cobra ftintegrity.py -i ftintegrity.params.txt -m c

To verify accuracy only and query all indexed objects, open a command prompt and execute

cobra ftintegrity.py -i ftintegrity.params.txt -m a

FT Integrity generates 3 reports

res-comp-common.txt – object id of all documents that are found in both index and repository.

res-comp-dctmonly.txt – object id of documents that are in repository but not indexed

res-comp-fastonly.txt –object id of documents in index but not in repository.

It also generates ftintegrityoutput.txt file which is nothing but the console output generated in the text format.

ARBITRARY

In addition to reliability, here are 5 reasons why WordPress is the best platform for any
online business:

1. Free. If you have never heard about open source software, you might be wondering why something as great as WordPress is free. WordPress is free because it has been developed by a worldwide community of developers, who do it as a hobby and because they love the open source spirit. Not only WordPress itself is free but also most of its themes and plugins are free, which means that you can create a really rocking site for free.
2. Easy to use. Ease of use is another reason why WordPress is a great choice for an online platform. This means that someone with little experience will be able to deal with WordPress with no or just a little help from the pros.
3. Secure. As far as online business is concerned, security is of the greatest importance. If your platform is insecure, this is a threat to everything you do and you just can’t afford to go for an insecure platform. Fortunately, in regards to security, WordPress tops the list. It is true that there is no unhackable software and nobody can guarantee you that even with the most stringent security measures your WordPress site will never be hacked but it does make a difference whether you install a security fortress or an easy target for a beginner hacker.
4. Highly customizable. You will certainly want your online platform to have a unique look. With WordPress this is easy to achieve. There are thousands of themes you can choose from and thousands of plugins to add in order to get exactly the functionality you need. If you are a web designer and/or web developer, you can also use existing themes and plugins and modify them exactly to your liking. Not many other CMS offer such customization freedom!
5. Great community. Last, but not least, another reason why WordPress is such a great online platform is its community. The forums at WordPress.org and the other WordPress-related sites contain tons of threads about everything WordPress-related. Additionally, almost always when you are stuck with a problem and ask for help, knowledgeable and skillful WordPress gurus rush to your help. The WordPress community is simply great and you should experience it yourself.

WordPress is a top CMS. It is easy to start using, yet it is powerful and you can build your site exactly the way you dreamed it. If you have never tried WordPress, get it today and try it out. Just don’t forget to browse through wordpress hosting guides to see which WordPress hosts that are worth dealing with and which aren’t. WordPress is reliable and secure but a poor web host is more than it can stand, so don’t take any chances.

EMAIL SUBSCRIPTION

Enter your email address to subscribe to this blog and receive notifications of new posts by email.