Is it possible to really delete a file (ie, bypass the Trash) from Finder? Currently, when I want to reclaim space from a USB drive, I rm files/folder from the Terminal, but I'd like to do it from the GUI (similar to Shift+Delete in Windows).

I came across an AppleScript that does something similar to what @bmike has suggested. User Jolly Jimmy listed in a forum post:

tell application "Finder"
set theSelection to selection
if theSelection is {} then
beep
activate current application
display alert "No files are selected for deletion." message "Select one or more files to delete." as warning
else
activate current application
display alert "Delete item(s)?" message "The selected item(s) will be deleted immediately. Are you sure you want to continue?" buttons {"OK", "Cancel"} as warning
set buttonpressed to button returned of result
if the buttonpressed is "OK" then
repeat with i from 1 to (count theSelection)
set item1 to (the quoted form of POSIX path of (item i of the theSelection as alias))
set deleteit to "rm -rf " & item1 & ""
try
do shell script deleteit
on error
do shell script deleteit with administrator privileges
end try
end repeat
end if
end if
end tell

Add this script to the services menu, and you should be able to delete by right-clicking.

This is basically muscle memory for me. Any time I trash something (which I always use command+del) I immediately empty the trash using command+shift+del. It's still two steps but it only takes a fraction of a second.
–
Mr RabbitApr 4 '14 at 14:15

In a more recent OS X versions there is a Backspace key instead of Del used.
–
TranslucentCloudJun 23 at 6:30

Not a fully-GUI answer, but you can make the procedure much easier by using a Finder+Terminal+Drag'n'drop approach, i.e. given that you have a terminal window opened, and a non-overlapping Finder window:

type rm -rf in the terminal (with a trailing space).

Drag the required folder from Finder to the terminal window and press Enter.

Hint: To remove the current folder in Finder, drag the blue icon in the Finder's title bar. This will also work with text editor windows or any OSX app that has a normal title bar displaying the current path or file name.

Hint2: If you are dragging the file/folder from a window that overlaps the Terminal window, then drag the file to the Terminal's icon in the dock, wait until the Terminal windows are displayed and then continue dragging into the the one you need.