RequestHandlerBase.init will parse the request hander definition in solrconfig.xml, and put them into defaults, appends, invariants accordingly.There is only one instance of request hander for a core, be careful of its thread safety.SolrCore constructor calls loadSearchComponents, then initPlugins to load all components defined, will also put all classed defined in solrconfig.xml which implement SolrCoreAware int waitingForCore.Then later it will cal SolrResourceLoader.inform(SolrCore), to call inform(SolrCore) defined in each class.For a non-shard query, SearchHandler.handleRequestBody will call prepare on all components, and then call process on all components.QueryComponent.prepare user QParser to prepare parameter to get query, sort.FacetComponent.prepare check whether facet it enabled: facet=trueMoreLikeThisComponent.prepare does nothingHighlightComponent.prepare check whether highlight it enabled: hl=trueStatsComponent.prepare check whether stats it enabled: stats=trueDebugComponent.prepare does nothingQueryComponent.process do the real search stuff, groupingSolrIndexSearcher searcher = req.getSearcher();searcher.search(QueryResult qr, QueryCommand cmd)SearchHandler.inform will create SearchComponent, shardHandlerFactory.Solr.postDecorateResponse will put status, QTime, and possibly request handler and params, in the response header.Then SolrDispatchFilter will choose QueryResponseWriter and write response.Classes: XMLResponseWriter, XMLWriter; CSVResponseWriter, CSVWriter.ClassesQParser: Parse query, sortQParserPlugin.DEFAULT_QTYPE=luceneQueryParsing.LOCALPARAM_STARTHow to parse Local ParamQParser.getParser(String, String, SolrQueryRequest)if(qstr.startsWith(QueryParsing.LOCALPARAM_START))localParamsEnd = QueryParsing.parseLocalParams(qstr, 0, localMap, globalParams);ThreadLocal: SolrRequestInfoprotected final static ThreadLocal<SolrRequestInfo> threadLocal = new ThreadLocal<SolrRequestInfo>();It is cleared in finally bock of SolrDispatchFilter.doFilter.How Solr Executes Shard RequestsSame as non-shard request, through SolrDispatchFilter to SearchHandler.handleRequestBody.It will call prepare on all components, it defines different stages for distributed request. In method distributedProcess of each component, it checks the current stages and responds accordingly.If needed, the component may create ShardRequest, and call modifyRequest on each components in ResponseBuilder.addRequest(SearchComponent, ShardRequest).Then it will remove parameter like shards, set distrib false, use completionService and Callable task to send request to all shards using ShardHandler, later will get ShardResponse back, which wraps SolrResponse. Then call handleResponses on all components, which usually merges response from multiple servers.Then call finishStage on all components.ShardHandler shardHandler1 = shardHandlerFactory.getShardHandler();shardHandler1.checkDistributed(rb);for (String shard : sreq.actualShards) {shardHandler1.submit(sreq, shard, params);}handler.component.HttpShardHandler.submit will use completionService to submit a Callable task to call HttpSolrServer to send the request,SolrServer server = new HttpSolrServer(url, httpClient);ssr.nl = server.request(req);In the shards parameter, for each shard, we can use | to specify multiple solr server to balance request in multiple servers.handler.component.HttpShardHandler.getURLs(String)urls = StrUtils.splitSmart(shard, "|", true);Resourceshttp://wiki.apache.org/solr/DistributedSearch

EntranceSolrDispatchFilter.initnew CoreContainer.Initialize().initialize();CoreContainer.Initialize.initialize()1. SolrResourceLoader.locateSolrHome()The order to find Solr Home: jndi lookup: java:comp/env/solr/home, system environment: solr.solr.home, otherwise solr/, relative to current directory.2. new CoreContainer(solrHome).load(solrHome, solr.xml)This will create add jars in solr-home/lib to class loader, parse solr.xml, use ThreadPoolExecutor and CompletionService to load cores defined in parallel, define a CoreDescriptor for each core, in CoreDescriptor, we can see the default value for core properties: loadOnStartup=true, isTransient = false.CompletionService completionService = new ExecutorCompletionService(coreLoadExecutor);Set> pending = new HashSet>();pending.add(completionService.submit(task));In the Callable task, it will call CoreContainer.createFromLocal(String, CoreDescriptor).CoreContainer.createFromLocal1. Create SolrConfig which represents solrconfig.xml, this will read the xml, load the jars into classloader, create SolrIndexConfig for indexConfig section, create CacheConfig, HttpCachingConfig, load requestHandler, queryParser, transformer etc.2. Creat IndexSchema which represents schema.xml, this will parse schema,xml. create field types, SchemaField, read SimilarityFactory.3. Create one SolrCore: core = new SolrCore(dcore.getName(), null, config, schema, dcore);This will initialize listeners defined in solrconfig.xml, initIndex, initQParsers, initValueSourceParsers, initTransformerFactories. It will initialize RequestHandlers, create one instance for each request handler defined, and put it into a map. So there will be only one instance for each request handleRequestBody. So be careful of the thread safty when write our own request handler.reqHandlers = new RequestHandlers(this);reqHandlers.initHandlersFromConfig(solrConfig);Class in IndexSchema: DynamicField, DynamicCopyCoreContainer.cfg represents solr.xml:, in CoreContainer, you can find out all available configuration in solr.xml.Example: String dcoreName = cfg.get("solr/cores/@defaultCoreName", null);SolrCore.SolrCore(String, String, SolrConfig, IndexSchema, CoreDescriptor, UpdateHandler, SolrCore)reqHandlers = new RequestHandlers(this);reqHandlers.initHandlersFromConfig(solrConfig);SolrResourceLoaderAdd solr-home/lib jars to class loader.this.classLoader = createClassLoader(null, parent);addToClassLoader("./lib/", null);reloadLuceneSPI();Learned1. Add jars into class loader?SolrResourceLoader.replaceClassLoader2. CompletionService

When import big csv file, we can split it to multiple small csv files, then use multiple threads to import them, this can improve import performance.Test with one 882mb csv file, using the new threaded request handler with 2GB memory, it takes 6 mins 32 seconds.If we import the 882mb file directly - no split, it takes 8 mins 22 seconds.So with this new request handler, it is 32% faster.ImplementationThe code is like below: you can review the complete code at Github.

In this article, I would like to introduce how to use Apache commons-daemon to install it as a Windows Service.First download latest commons-daemon-*-bin-windows.zip from here.Download latest commons-daemon-*-src.tar.gz from here.

In src\samples folder in commons-daemon-*-src.tar.gz, you can learn how to create a service, and the scripts to install and remove windows services:ProcrunService.java, ProcrunServiceInstall.cmd, ProcrunServiceRemove.cmd.The java code to start/stop embedded jetty server is here.Next we create 2 scripts to install it as a windows service, and remove the windows services. installEmbededJettyService.bat

@echo off
setlocal
set MYPATH=%~dp0
echo %MYPATH%
set PATH_PRUNSRV=%MYPATH%
set SERVICE_JAVA=EmbededJetty
set "PR_LOGPATH=%MYPATH%/../logs"
if "%PRUNSRV%" == "" set PRUNSRV=%PATH_PRUNSRV%prunsrv
echo Removing %SERVICE_JAVA%
%PRUNSRV% //DS//%SERVICE_JAVA%
if not errorlevel 1 goto removed
echo.
echo Failed uninstalling '%SERVICE_JAVA%' service
goto end
:removed
echo The service '%SERVICE_JAVA%' has been removed
:end
endlocal
@echo on

We can copy prunmgr.exe to the bin folder, and rename it as ${service-name}w.exe, so user can run it to edit it, or start, stop it like below:

2 Scripts to start and stop the windows servers.startService.bat

@echo off
setlocal
set MYPATH=%~dp0
set PATH_PRUNSRV=%MYPATH%
set SERVICE_JAVA=EmbededJetty
if "%PRUNSRV%" == "" set PRUNSRV=%PATH_PRUNSRV%prunsrv
if [%1] == [] goto startService
echo Changing the parameters for %SERVICE_JAVA%
%PRUNSRV% //US//%SERVICE_JAVA% --StartParams=start;-dynamicPort;true;%1;%2;%3;%4;%5;%6
if not errorlevel 1 goto updated
echo Failed updating '%SERVICE_JAVA%' service
goto end
:updated
echo The service '%SERVICE_JAVA%' has been updated.
:startService
"%PRUNSRV%" //ES//%SERVICE_JAVA%
if not errorlevel 1 goto started
echo Failed starting '%SERVICE_JAVA%' service
goto end
:started
echo %SERVICE_JAVA% is started.
:end
endlocal
@echo on

stopService.bat

@echo off
setlocal
set MYPATH=%~dp0
set PATH_PRUNSRV=%MYPATH%
set SERVICE_JAVA=EmbededJetty
if "%PRUNSRV%" == "" set PRUNSRV=%PATH_PRUNSRV%prunsrv
:startService
%PRUNSRV% //SS//%SERVICE_JAVA%
if not errorlevel 1 goto stopped
echo Failed stopping '%SERVICE_JAVA%' service
goto end
:stopped
echo %SERVICE_JAVA% is stopped.
:end
endlocal
@echo on

Examples we find on web are usually to import csv files into Solr, here I want to to show how to import CSV string into Solr.Why we want to import CSV string?1 Compared to importing csv files: importing csv string would be much faster, as no unnecessary IO: no need to write the csv files in client side and read it at server side.2 Compared to importing XML String, importing csv string should be also faster: It is faster than writing xml string in the client would and reading/parsing it in server side.3. CSV string is usually much smaller than XML, less time spend on network transfer, also can save a little bandwidth.How to import CSV String into Solr?1. Use stream.body to specify csv data, one thing we should pay attention, we have to use ASCII code %0D%0A to separate lines - use \r\r wouldn't work.2. In the stream body, you have to escape special character, change “ to \”, \ to \\.The following request would import 2 lines into Solr.curl -d "stream.body=2,0,1,0,1,\"c:\\\",1,0,\"c:\",0,1,16 %0D%0A 2,0,1,0,1,\"x:\\\",2,0,\"x:\",0,1,16 &separator=,&fieldnames=omiited&literal.id=9000&stream.contentType=text/csv;charset=utf-8&commit=true" http://localhost:8080/solr/update/csvIn order to simplify client code, we can create a new requestHandler, which sets fieldnames, so client need not specify fieldnames in each request. Solr Code

Why getProperty returns null?The reason is that Properties extends Hashtable, put its value into a Hashtable. its getProperty method will only return String value, it first use call super.get(key) to search the Hashtable, if the key doesn't exist, or its value is not a String, it will try to get from the string value from defaults.

As we usually will read Properties from a configuration file, and later save it to the file, so it's better we always use getProperty and setProperty, but use put, and get methods.defaults in Propertiesdefaults is also a Properties. When construct a new properties:newProp, we can pass a Properties:oldProp, it will save it as an instance value, and leave it here. When get value of a key, it will first search the Properties itself, it not found, it will search the defaults. When you use remove to remove a key, it will only remove from the Properties's Hashtable, not from defaults. Later get(key) will return null as it only searches the Properties's Hashtable, but getProperty will return the value in the defaults.If you don't like this behavior, you can use properties.putAll(aProperties); this will copy all element into the Hashtable of this new peoperties.SummaryJava Properties is not well designed, as it extends Hashtable directly, and expose many mehtods of Hashtable which it should not. But it's still the simple way to read and save properties from/to a file. Just be careful when we use it, always use getProperty and setProperty, not put nor get methods.The test code is like below: