I am a full-time consultant and provide services related to the design, implementation and deployment of mathematical programming, optimization and data-science applications. I also teach courses and workshops. Usually I cannot blog about projects I am doing, but there are many technical notes I'd like to share. Not in the least so I have an easy way to search and find them again myself. You can reach me at erwin@amsterdamoptimization.com.

Monday, June 17, 2013

I was running some neural networks with a back propagation algorithm using a rather simple optimization tool (gradient descent). This actually ran pretty decent. Now lets if we can formulate something like this in GAMS. I tried something like:

I know the formulation is not great but may be better high-performance sparse solvers would help me out here and still do a decent job. Unfortunately, the result was dramatically poor. Most solvers ran out of memory, and MINOS was still running after 3 hours.

The basic problem is that we really only want THETA2 and THETA3 as variables and all the rest just as temporary intermediate values. That makes the problem very much smaller (the thetas are not depending on i which is the observation number of the training set). With AMPL we probably could have used defined variables to make this work but GAMS does not offer this.

I got some feedback that users were not really familiar with how to write this in GAMS. So I thought this would be a good thing to share.

$ontext

Example fromUsing PROC IML to do Matrix BalancingCarol Alderman, University of KansasInstitute for Public Policy and Business ResearchMidWest SAS Users Group MWSUG 1992In this version we stop on convergence$offtext

I often use a large set to loop over: “loop(iter,”. This will make sure we always terminate (no infinite loop). The $ condition on the loop can be used to terminate the loop when convergence is observed: “loop(iter$continue,”. Finally we easily can check whether the loop terminated with or without convergence and give an appropriate message.

table IO_1(*,*)'Row and column totals of the new, to be updated, input–output table'GoodsServicesConsumptionNetExports TotalOutputGoods15Services26NetTaxes-1 TotalUse91617-240ValueAdded6100016TotalInput152617-2;

In the models below the row and column totals are known while the inner part of the matrix is estimated. The goal is to find a nearby matrix such that the row and column sums are equal to the given row and column totals. For many economic problems it is important we preserve the zeros.

The RAS algorithm implementation can easily be improved by iterating until convergence is observed instead of the fixed number of iterations. One big advantage of using an Entropy model instead of an algorithm is that it is easy at add side-constraints. This can often be very important in practical situations.

RAS Algorithm

$ontext

Example fromUsing PROC IML to do Matrix BalancingCarol Alderman, University of KansasInstitute for Public Policy and Business ResearchMidWest SAS Users Group MWSUG 1992$offtext