Dimitriy Masterov <dvmaster@gmail.com> wrote:
>I am trying to load a very large table from a Teradata database. It's slower
>than molasses on a cold winter morning, and then gives me an error message that
>I have exceeded the maximum number of observations. However, the table only
>contains only 55% of that limit in terms of rows (1,186,316,312 out of a
>theoretical 2,147,483,647). I have Stata MP 12.1 on a powerful Linux machine
>with 1 TB of RAM, so I hoped I could get close to the theoretical limit.
When -odbc load- issues the SELECT command to the database the entire table is
returned back from the database into memory. Next, the table data in memory is
converted into a Stata dataset in memory. Dimitriy has two datasets in memory,
the table data and Stata data, so his computer is running out of memory.
The solution for Dimitriy is to divide the table data into multiple
sections using a WHERE clause with his SELECT statement. Next, import each
section of data into Stata using -odbc load, exec- and save the data.
Last, append the sections of data together using the -append- command.
Dividing the data into multiple sections will resolve this memory error
and speed up the process.
Kevin
-Kevin Crow
kcrow@stata.com
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/