System limit exceeded - see manual r(1000); While using odkmeta to merge ODK data set

Support
1

Dear ODK Fourm,

I was using the packege odkmeta for merging ODK datasets. I am using ODK briefcase v 1.12.2 to download the data from odk server and using STATA 14.1 to merge. As I run the command
replace temp' = "examplename(07)" if var' == "examplename"
in the generated do file, I get the following error.
system limit exceeded - see manual r(1000);

Thank you in advance.

2

Hi @Amrita_Pal!

This is a bit of a challenging issue to troubleshoot, because the error message isn't very informative. We've had some related discussion here, but we never got to the bottom of the issue. It's hard to imagine how this replace command would exceed a system limit, but the other discussion also pointed to the issue coming from that section of the do-file.

What flavor of Stata are you using (Stata/IC, SE, or MP)?

I have access to Stata/IC, and I can take a look if you share your form along with fake, example data that reproduces the error message.

Let me know if you have thoughts about what could be triggering this error! For reference, the list of system limits is here.

3

Thanks @Matthew_White,

Is it happen because of memory capacity of STATA? I have set
set maxvar 32767
set matsize 11000;
set niceness 10
set min_memory 0
set max_memory 1600g
But i am unable to set segmentsize to its upper-limit (32g), its now 3g in my STATA. Is this error coming because of this?

4

I had the thought that it could be a memory issue, because when the string "examplename" is replaced with "examplename(07)", that could increase the length of the string variable (for example, from str11 to str15). However, unless you have a large number of observations, an increase from, say, str11 to str15 shouldn't significantly increase your memory usage. Stata also usually issues an informative error message when it hits a memory limit.

Stata allocates memory in blocks whose size equals segmentsize, but I don't think segmentsize affects your overall memory capacity. So I don't think that's likely the source of the error message. help memory also notes:

We recommend resetting segmentsize only if your computer has large amounts of memory.

There are a couple of other things to try to figure out whether this could be a memory issue.

First, after the error message is issued, you can display a memory usage report by typing memory. Does it indicate that you are at or near your limits?

Second, how many observations does your dataset have? Does the do-file complete successfully if you try to import a smaller number of observations? Note that you can use ODK Briefcase to export only submissions from a particular date range. You can also manually edit a CSV file in a text editor, removing rows that way. Just don't use Excel to edit the CSV file, because Excel will automatically make some formatting-related changes to it.

5

Another possibility to consider, based on this Statalist discussion, is the size of the do-file. How many lines is the do-file? What's the size of the do-file in bytes? If you have a lot of unneeded char commands, you can reduce the size of the do-file by using the keepattrib() and dropattrib() options of odkmeta. For example, you could specify keepattrib(type name label disabled) to import only the most essential attributes.