2IntroductionBest estimate of the atmospheric state from instrument synergyUse a variational framework / optimal estimation theorySome important measurements are integral constraintsE.g. microwave, infrared and visible radiancesAffected by all cloud types in profile, plus aerosol and precipitationHence need to retrieve different particle types simultaneouslyFunded by ESA and NERC to develop unified retrieval algorithmFor application to EarthCAREWill be tested on ground-based, airborne and A-train dataAlgorithm componentsTarget classification inputState variablesMinimization techniques: Gauss-Newton vs. Gradient-DescentStatus of forward models and their adjointsProgress with individual target typesIce cloudsLiquid clouds

3Ingredients developed before In progress Not yet developedRetrieval framework1. New ray of data: define state vectorUse classification to specify variables describing each species at each gateIce: extinction coefficient , N0’, lidar extinction-to-backscatter ratioLiquid: extinction coefficient and number concentrationRain: rain rate and mean drop diameterAerosol: extinction coefficient, particle size and lidar ratio3a. Radar modelIncluding surface return and multiple scattering3b. Lidar modelIncluding HSRL channels and multiple scattering3c. Radiance modelSolar and IR channels4. Compare to observationsCheck for convergence6. Iteration methodDerive a new state vectorEither Gauss-Newton or quasi-Newton scheme3. Forward modelNot convergedConvergedProceed to next ray of data2. Convert state vector to radar-lidar resolutionOften the state vector will contain a low resolution description of the profile5. Convert Jacobian/adjoint to state-vector resolutionInitially will be at the radar-lidar resolution7. Calculate retrieval errorError covariances and averaging kernelIngredients developed beforeIn progressNot yet developed

4Target classificationIn Cloudnet we used radar and lidar to provide a detailed discrimination of target types (Illingworth et al. 2007):A similar approach has been used by Julien Delanoe on CloudSat and Calipso using the one-instrument products as a starting point:More detailed classifications could distinguish “warm” and “cold” rain (implying different size distributions) and different aerosol types

6Results: radar+lidar onlyRetrievals in regions where radar or lidar detects the cloudRetrieved visible extinction coefficientRetrieved effective radiusLarge error where only one instrument detects the cloudRetrieval error in ln(extinction)

9The cost function + Smoothness constraintsThe essence of the method is to find the state vector x that minimizes a cost function:Each observation yi is weighted by the inverse of its error varianceThe forward model H(x) predicts the observations from the state vector xSome elements of x are constrained by an a priori estimateThis term penalizes curvature in the extinction profile+ Smoothness constraints

10Gauss-Newton methodSee Rodgers’ book (p85): write the cost function in matrix form:Define its gradient (a vector):…and its second derivative (a matrix):Approximate J as quadratic and apply this:Advantage: rapid convergence (instant convergence for linear problems)Another advantage: get the error covariance of the solution “for free”Disadvantage: need the Jacobian matrix of every forward model: can be expensive for larger problems

11Gradient descent methodsJust use gradient information:Advantage: we don’t need to calculate the Jacobian so forward model is cheaper!Disadvantage: more iterations needed since we don’t know curvature of J(x)Use a quasi-Newton method to get the search direction, such as BFGS used by ECMWF: builds up an approximate form of the second derivative to get improved convergenceScales well for large xDisadvantage: poorer estimate of the error at the endWhy don’t we need the Jacobian H?The “adjoint” of a forward model takes as input the vector {.} and outputs the vector Jobs without needing to calculate the matrix H on the wayAdjoint can be coded to be only ~3 times slower than original forward modelTricky coding for newcomers, although some automatic code generators existTypical convergence behaviour

16Gradient constraintA. Slingo, S. Nichols and J. Schmetz, Q. J. R. Met. Soc. 1982We have a good constraint on the gradient of the state variables with height for:LWC in stratocu (adiabatic profile, particularly near cloud base)Rain rate (fast falling so little variation with height expected)Not suitable for the usual “a priori” constraintSolution: add an extra term to the cost function to penalize deviations from gradient c:

18ProgressDone:C++: object orientation allows code to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to different observing systemsCode to generate particle scattering libraries in NetCDF filesAdjoint of radar and lidar forward models with multiple scattering and HSRL/Raman supportRadar/lidar model interfaced and cost function can be calculatedIn progress / future work:Interface to BFGS algorithm (e.g. in GNU Scientific Library)Implement ice, liquid, aerosol and rain constituentsInterface to radiance modelsTest on a range of ground-based and spaceborne instrumentsTest using ECSIM observational simulatorApply to large datasets of ground-based observations…