This is my attempt to get over the issue of UTF-8 characters on PCs [only] when filepaths etc return incorrect results in some of Ruby's native File and Dir methods.It uses win32ole.so, which you need to have an appropriate version of in Plugins[ viewtopic.php?p=380121#p380121 ]It returns correct results as far as I've tested it...Can some of you try testing it too...e.g. replace File.exist?() with PCFile.exist?()and see if filepaths with accented characters etc are then returned 'true' [properly]...These methods mimic the native tools fairly well and there are also some 'better' copy/move/rename methods; but there's no 'binmode' and the read/write syntax etc is slightly different [but still logical]...###

PCFileTools.rb

###(c) TIG 2012Permission to use, copy, modify, and distribute this software for any purpose and without fee is hereby granted, provided that the abovecopyright notice appear in all copies.THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS ORIMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIEDWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

PCFileTools.rb

Adds new PC ONLY tools similar to the Ruby 'File' and 'Dir' tools, BUT which avoid issues with UTF-8 characters in filepaths etc...[NOTE: MACs have no equivalent issues so they don't need this!]

PCFile... & PCDir...

They generally work like the Ruby equivalentSo File.exist?("C:/Temp/boué.txt")which will WRONGLY return 'false,' is replaced by PCFile.exist?("C:/Temp/boué.txt")which correctly returns 'true'...

###

PCFile.basename( fileName [, aSuffix] ) returns the basename of the file or folder, e.g. "C:/Temp/file.txt" -> "file.txt" ; if the 'aSuffix' is used then any matching extension is stripped, using ".*" removes ANY file extension - e.g. -> "file".

PCFile.delete( fileName ) deletes the given file; if it's not an 'absolute path' the file is sought in the pwd.

PCFile.rename( fileName, newName ) renames the file 'fileName' as 'newName'; unlike File.rename(...) it can 'move' a file to a new folder, if the specified folder exists, if no folder path is given it assumes the same folder as 'fileName'.

PCFile.move( fileName, nname ) equivalent to PCFilename.rename(...)

PCFile.copy( fileName, newName, [overwrite=false] ) makes a copy of the file 'fileName' as 'newName'; it can 'copy' a file to a new folder, if the specified folder exists, if no folder path is given it assumes the same folder as 'fileName'. If a 'true' 3rd argument is given then any existing file is overwritten, if a 'false' or no 3rd argument is given then any existing file is NOT overwritten.

PCFile.new( fileName, [modeString="r"] ) opens a file as f=PCFile.new(...), making it or using it if it exists, the mode defaults to "r" = 'read', the options are: "r" =read "r+" =read+write "w" =write "w+" =write+read "a" =append "a+" =append+read

# he can always refer to the base class, # by using the toplevel scope prefix, thus: ::File.basename(pathstring)

# or if he's done using the PC specific class, # he can make the local constant to point at # the base class again, by reassignment: File = ::File # # or by removing the local constant, thus: remove_const(:File)

I'm sure that the 'use' of the core of the code can be made better - although I'd be wary of overwriting base class ?A lot of this is 'above my pay grade'...

I've kept everything separated for testing - so it still works provided you substitute PCFile.exist?(filepath) for File.exist?(filepath) and so on...

What I'm asking initially is for developers to test these new/replacement PCFile methods and see if they are buggy and/or work as hoped.

The biggest bind with 1.8~ Ruby File on PCs is that many operations like File.exist?() can return false when you know the file really does exist, these 'failures' are caused by UTF-8 characters in the filepath string [typically these are 'accented letters' in FR/DE/ES/PT/NO etc, but results from Chinese users etc would be great too...].

I've tried to write simple code using pack/unpack and equivalent Win32OLE methods that return 'true' correctly when the filepath exists and it contains such characters...

There are many other failures with File that I hope I have now trapped with these equivalent PCFile methods. I've had to make slightly different read/write code to get around some Win32OLE vagaries etc... and I can't see currently how to do the equivalent of 'binmode' etc. Any suggestions/additions gratefully received...

At this stage I really just want feedback on the efficacy of these methods...We can then debate later how the finalized 'fixed' methods are shoehorned into Ruby...

I didn't say anyone should. I said they should be subclasses of the base class. If they are, Ruby would not let anyone make the subclass overwrite the baseclass, because it would be a circular reference.

EDIT: I tested this at the console. Ruby does not check for circular references using the c-side = operator. Anyway, it does not actually create a circular reference. The reference to original baseclass becomes un-identified by any constant, but the object can still be got, via the superclass() method.

Making a LOCAL constant point at an object (in this case a Class definition,) does not overwrite anything.

Now obviously, if some stupid 'newb' types File = SKX::WIN::File in the Ruby Console, or in an unwrapped script, THAT global constant that references the base class definition object, is changed (and affects all scripts that are also using it.) So, in order to set it back [without restarting,] it might be a good idea to have a 'secret' reference to the base class definition, kept "out of sight."

TIG wrote:I've kept everything separated for testing - so it still works provided you substitute PCFile.exist?(filepath) for File.exist?(filepath) and so on...

You won't find many who are willing to go through their code and search and replace, "File." with "PCFile.", besides you haven't provided aliases for the methods you did not implement, which makes a simple editor search and replace, into a tedious manual edit session.

You might consider, overriding the method_missing callback, whilst you still have them as custom modules:

# Now he can use File as he's always done. # # On Mac, there is no local constant "File", and the # call is evaluated to the object, that is referenced, # by the toplevel constant "File". # # But on PC, within this namespace ONLY, the call # is evaluated, to the object pointed to, by the # local (constant) reference "File", or fully # qualified: "Author::FancyTool::File", # which does not touch ::File (aka: Object::File)

end#classend#module

There is no way, scripters will want to changed all calls to methods of class File or Dir, into platform conditional statements, like:

No doubt that goes for any single ONE rubyist. Something like this needs to be a group project.Others have done some work in this area already. ThomThom on wide strings (which also need to be addressed, because many of the base class String methods, will garble unicode strings.)Dan Berger's "win32-api" toolkit plays a bit with creating a WideString class. (I think he considers it beta.)Also, we should not ignore the extended class Pathname (which is actually a wrapper class, not a String subclass.)

TIG wrote:We can then debate later how the finalized 'fixed' methods are shoehorned into Ruby...

OK, I made my point, on this issue.

The only thing to add is to state the obvious. There are really only 2 alternatives.

My suggestion is the least invasive, with the least responsibility. (Authors are left to decide to use the extension on a namespace by namespace basis.)

The other option, to actually redefine any Ruby base classes that will use unicode strings, on the Windows platform(s).This would mean taking responsibility for string functionality of ALL plugins (running on Windows.)Not what (I think,) you would wish to do. Nor any project group. (Not at least without compensation, of some kind,.... and a lot of time to devote to the "cause.")Even the Ruby Core guys are taking forever to implement unicode support.

TIG wrote:... and I can't see currently how to do the equivalent of 'binmode' etc. Any suggestions/additions gratefully received..

Ya know.. it's weird that the Core guys made this switch, without a way to test later IF the stream was IN binmode or not.

I checked the docs, and the Ruby 1.8.7 branch is still the same.

BUT... in the 1.9.x trunk, they have added a binmode?() boolean query method. (See the online docs for IO class. They also added methods binread and binwrite, special open methods.)Of course, they have added a bunch of options, to read and write in several encodings.

I found that if I didn't do that change to the string then any tests of UTF-8 strings, like PCFile.exist?(path) do not work properly and return 'false' when it should be 'true'... just like the File.exist?(path) version; BUT making that change to the string before testing it seems to return correct results - consistently 'true' when it should be 'true' and 'false' when it should be 'false'. For a simple ANSI character string it works fine either way [the unpack/join has no affect], but if you test with a UTF-8 string with accented characters [perhaps obtained from a UI.openpanel()], that is unpack/joined etc then you can see the difference between what the File.. and PCFile.. versions return...There's probably a more elegant way to do this... BUT it seems to work the way I've bodged it together, so now perhaps we can think of better ways of achieving the same difference...

I thot about using Dan Berger's String subclass(es) WideString or whatever he called them, but it seem like it would be combersome. Unless they converted themselves automatically similar to how Numerics use coerce().Currently the interpreter always makes ANSI strings from " " and ' ' literals. (and their %dilimeter equivs.)

I wonder if possible to create a %u function that creates UTF8 strings. And maybe a %U that creates UTF16 ?(Are these defined in Kernel, or are they C-side interpreter functions?

The underlying problem in Ruby 1.8 under windows is that it calls the A version of the file functions instead of the W versions. If a function is called FileFunction is used in C/C++ - when compiled it will translate to FileFunctionA or FileFunctionW depending on whether UNICODE is defined.I was thinking that a C Extension that would forcefully call the FileFunctionW variants would be sure to work as it would be the system doing all the work. Meddling with the string in Ruby is quite likely to cause data to be lost or corrupted.

I only works for UTF-8 [i.e. 'European' accented-characters etc] - the more complex Chinese/Japanese return false when it should be true However, if we have a way of resolving one hopefully the other will follow...

Strings are stored as sequences of 8-bit bytes,[For use in Japan, the jcode library supports a set of operations of strings written with EUC, SJIS, or UTF-8 encoding. The underlying string, however, is still accessed as a series of bytes.] and each byte may contain any of the 256 8-bit values, including null and newline. The substitution mechanisms in Table 18.2* on page 203 allow nonprinting characters to be inserted conveniently and portably.

So it seems that (in my mind,) since Sketchup sets $KCODE to UTF8 when it loads the interpreter, we may not actually have as much of a problem on the Rubyside as I thought.

So we have a choice...

1) A pure-Ruby patch, that either accesses the system calls (for File functions,) via WIN32OLE or WIN32API (the so libraries.)

2) A compiled C patch, ie: "Cut out" the c code files that define classes IO, Dir and File (perhaps also FileTest,) and recompile with either UNICODE #defined, or change the C function calls explicitly to the wide versions. These would be ".so" files, and they would redefine the old methods. (What happens on the C-side when you re-define a C function that has already been defined? Do the C functions that the new Ruby wrappers call, need to be renamed as well?)

I was thinking that a C extension that did most of the functions used, like File.exist?, read, write, delete and list files in folders would go a long way. It wouldn't be as extensive as a complete rewrite - therefor more quicker to develop. Then off course not replacing the existing methods - as it'd just open up a vast pool of possible problems which would require even more testing and development.

Trying to rewrite the entire File, Dir and IO classes to call the W variant file function just seems like such a massive undertaking that it'd probably never be completed.

Does Sketchup set $KCODE to "UTF8" in all locales - I know it does in UK/US and probably European language locales BUT what about Chinese ? I'll ask someone...If we get a consistent code for their locale can we modify the pack/join code to say use 'u' not 'U' and get an appropriate conversion in different $KCODE cases ?

thomthom wrote:Then of course not replacing the existing methods - as it'd just open up a vast pool of possible problems which would require even more testing and development.

Well what I said before still goes... The new class(es) are in a Library namespace, need to be require(d), and then referenced within an author's namespace via an alias, (as I showed in the examples above.)

I mispoke when I said redefine, they would have the same identifiers, but be within, say SKX::WIN module namespace.

thomthom wrote:Trying to rewrite the entire File, Dir and IO classes to call the W variant file function just seems like such a massive undertaking that it'd probably never be completed.

I was hoping (without yet digging into the C source,) that it might be easy to use Notepad++ search and replace to stick "W" where they need to be.