It seems eol is applied after delims and tokens. Change "tokens=*" to "delims=" and run the program again to see what I mean. Just one more thing to be mindful of when you’re processing lines of text with a for /f loop!

Sorry for not being clearer. The difference in the output from your first two examples is, as you inferred, what I was trying to draw attention to. When "delims=" is in effect, lines beginning with a semi-colon will be ignored. But when it’s "tokens=*", lines where the first non-whitespace character is a semi-colon will be passed over.

The behavior is strictly controlled by DELIMS and EOL - it has nothing to do with the TOKENS value.

When FOR /F processes a line, it first breaks the line into tokens as per DELIMS, and then skips the line if the first character of the first token is the EOL character. This explains why the indented line is skipped when using the default DELIMS value of <space><tab>. It also explains the known behavior that setting EOL to one of the DELIMS characters effectively disables EOL - any EOL character at the start will have been consumed by DELIMS processing by the time the EOL check is made.

Only after all of the above occurs are the appropriate token values assigned to FOR values as per TOKENS. So TOKENS has no impact on whether a line is skipped due to EOL.

I do find the indented case interesting - I was not fully aware of the order of operations until you showed that case - thanks

dbenham wrote:When FOR /F processes a line, it first breaks the line into tokens as per DELIMS, and then skips the line if the first character of the first token is the EOL character. This explains why the indented line is skipped when using the default DELIMS value of <space><tab>.

Under the A line in my screenshot shows that the indented line is not skipped. Only the line starting with the default EOL

dbenham wrote:When FOR /F processes a line, it first breaks the line into tokens as per DELIMS, and then skips the line if the first character of the first token is the EOL character. This explains why the indented line is skipped when using the default DELIMS value of <space><tab>.

Under the A line in my screenshot shows that the indented line is not skipped. Only the line starting with the default EOL

Yes - this is mostly consistent with the rules as I had stated them. When using the default DELIMS of <space><tab> (TOKENS value is insignificant), the DELIMS strips the leading spaces, so the first remaining character in the first token is ";", which matches the default EOL, so the line is skipped. When DELIMS is disabled, the leading spaces are preserved, so the first remaining character of the first token does not match EOL, so it is not skipped.

I did make one mistake - The DELIMS token splitting must not be done to completion before TOKEN selection, otherwise * would not work properly. The token selection must be interleaved with the delimiter parsing.

npocmaka_ wrote:So the the prio seems to be:

useback>skip>tokens=*>delims>eol>tokens=[ something that is not an asterisk ]

Not quite. The tokens=* belongs at the end along with all other tokens values.

1 - apply USEBACKQ to IN() clause to determine the appropriate content source2 - Line Loop - while not end of content { 3 - If SKIP not defined or line number > SKIP value then { 4 - Remove leading delimiters as per DELIMS 5 - If first remaining character does not match EOL then { 6 - Token Loop - while not end of TOKENS { 7 - If next specified token is not *, then split at next set of DELIMS characters 8 - If current token number matches the next token specified by TOKENS, then assign FOR variable and advance the TOKENS list } } }}

Test file xll.txt contains six tokens. Tokens 3 and 5 are 10,001 hashes long. The for /f loop processes the extremely long line token by token. It doesn’t throw an error when it gets to the 3rd or 5th token and increments the nth variable without complaint. Only when it looks for the 7th token and finds none does it execute the || claus.

This magic is fragile, however, and falls apart if you attempt to access the %%A loop variable. Even so, an interesting way to find the number of tokens in a string.

dbenham wrote:Sponge Belly's point is still valid, as long as you understand the definition of a FOR /F token.

I am not understanding. Are you saying my comment is wrong and his code will work regardless of the number of repeat delimiters and we don't have to use your PARSECSV.bat file to properly parse the number of correct tokens?

It creates one line with param1=100.000 "X", param2 and param3 with a space as delimiter.And as you can see the parameters 2 and 3 work as expected.It seems that there is no limit for the length of a single parameter, the code also works with 1MB or also with 10MB.

Btw. I'd never found a way to access the ultra long parameters itself.

All modifiers seem to fail, even when I build a ulp (ultra long parameter) of the form "C:\XXXXXXX....100k...XXXX.txt"%%~dA, %%~xA or %%~zA results in a block failure, so the complete block isn't executed