Codrspace
-
Latest posts by glenbot

Some of you may have already used expect in linux. It's a nifty little tool. A tool that allows you to automate command line interfaces that require interactive prompts. It's simple, let's look at an example of expect in action.

Notice the shebang line #!/usr/bin/expect -f. Expect is an interactive shell. If you just type expect from the command line you will be placed in a shell that gives you commands such as spawn, send, and expect.

Let's look at what expect is doing in the code snippet above.

We tell expect to spawn a child process of passwd

spawn passwd

Next, we tell expect to wait for some stout from the child process where the text says Enter New UNIX password

expect "Enter new UNIX password: "

Then we tell expect to send a password with a carriage return, the \r idiom.

send "mynewpassword\r"

Finally, we expect the termination of the child process.

expect eof

So, how do we implement this in Chef? It's more simple than you think.

Removing stopwords is a common thing to do if you are indexing data. Typically you don't want words like and, the, and or to show up in your index. It waters down the results. I have modified an existing script by Durden to show small and large corpus of text and its benchmarks. Durden has already done a great job of setting up the code now let's see some test results.

"""Demonstration of ways to implement this API: sanitize(user_input, stop_words)Related discussions: - Modifying a list while looping over it: - http://stackoverflow.com/questions/1207406/remove-items-from-a-list-while-iterating-in-python - Remove all occurences of a value in a list: - http://stackoverflow.com/questions/1157106/remove-all-occurences-of-a-value-from-a-python-list"""importsysfromtimeitimporttimeitfromfunctoolsimportpartialfromcollectionsimportdequedefsanitize_1(user_input,stop_words):"""Sanitize using set subtraction then wrapped in list()"""# Downsides:# - Sets are unordered so if user_input was a sentence we lose the# ordering because set difference will create a new set and not# maintain ordering.returnlist(set(user_input)-set(stop_words))defsanitize_2(user_input,stop_words):"""Sanitize using intersection and list.remove()"""# Downsides:# - Looping over list while removing from it?# http://stackoverflow.com/questions/1207406/remove-items-from-a-list-while-iterating-in-pythonstop_words=set(stop_words)forswinstop_words.intersection(user_input):whileswinuser_input:user_input.remove(sw)returnuser_inputdefsanitize_3(user_input,stop_words):"""Sanitize using standard lists"""new_list=[]forwinuser_input:ifwnotinstop_words:new_list.append(w)returnnew_listdefsanitize_4(user_input,stop_words):"""Sanitize using standard list comprehension"""return[wforwinuser_inputifwnotinstop_words]defsanitize_5(user_input,stop_words):"""Sanitize using collection.deque and list comprehension"""user_input=deque(user_input)stop_words=deque(stop_words)return[wforwinuser_inputifwnotinstop_words]def_sanitize_funcs():"""Get all the sanitize functions in scope"""module=sys.modules[__name__]forobjinsorted(vars(module).values()):ifcallable(obj)andobj.__name__.startswith('sanitize_'):yieldobjdefget_functions(user_input,stop_words):"""Build a list of the sanitize functions with parameters"""functions=[]forfin_sanitize_funcs():functions.append(partial(f,user_input,stop_words))returnfunctionsdefcheck_results(functions):"""Test the results of each function"""forfinfunctions:print'%s - %s'%(f.func.__name__,f())defcheck_performance(functions,number=1000000):"""Test the performance of each function"""forfinfunctions:print'%s [%f] %s'%(f.func.__name__,timeit(f,number=number),f.func.__doc__)defmain():# number of iterations for timeit# timeit defaults to 1,000,000 but change this number# to see different variationsnumber=10000# a list of stop words to be removedstop_words=['the','that','to','as','there','has','and','or','is','not','a','of','but','in','by','on','are','it','if']user_input='the cat walked down the road.'.split()functions=get_functions(user_input,stop_words)print'Timing with %d iterations'%numberprint'----- Function Results -----'check_results(functions)print'----- Function Performance -----'check_performance(functions,number)print'-'*10user_input=""" Proficient reading depends on the ability to recognize words quickly and effortlessly.[2] If word recognition is difficult, students use too much of their processing capacity to read individual words, which interferes with their ability to comprehend what is read. Many educators in the USA believe that students need to learn to analyze text (comprehend it) even before they can read it on their own, and comprehension instruction generally begins in pre-Kindergarten or Kindergarten. But other US educators consider this reading approach to be completely backward for very young children, arguing that the children must learn how to decode the words in a story through phonics before they can analyze the story itself. During the last century comprehension lesson/s usually comprised students answering teachers' questions, writing responses to questions on their own, or both.[citation needed] The whole group version of this practice also often included "Round-robin reading", wherein teachers called on individual students to read a portion of the text (and sometimes following a set order). In the last quarter of the 20th century, evidence accumulated that the read-test methods assessed comprehension more than they taught it. The associated practice of "round robin" reading has also been questioned and eliminated by many educators. Instead of using the prior read-test method, research studies have concluded that there are much more effective ways to teach comprehension. Much work has been done in the area of teaching novice readers a bank of "reading strategies," or tools to interpret and analyze text.[3] There is not a definitive set of strategies, but common ones include summarizing what you have read, monitoring your reading to make sure it is still making sense, and analyzing the structure of the text (e.g., the use of headings in science text). Some programs teach students how to self monitor whether they are understanding and provide students with tools for fixing comprehension problems. Instruction in comprehension strategy use often involves the gradual release of responsibility, wherein teachers initially explain and model strategies. Over time, they give students more and more responsibility for using the strategies until they can use them independently. This technique is generally associated with the idea of self-regulation and reflects social cognitive theory, originally conceptualized by """.lower().split()functions=get_functions(user_input,stop_words)print'----- Function Results -----'check_results(functions)print'----- Function Performance -----'check_performance(functions,number)if__name__=="__main__":main()

#!/usr/bin/env python# Due Date Notifier - Notify VIA SMS (SendHub) when a bill is going to be due# * Requires a SendHub account (free) - http://sendhub.com# * Has python package dependencies: simplejson, requests# * Recommend putting this on a crob job running once a day# I like my text messages at at 11am## NOTE: Sendhub free accounts allow only 500 requests to the API per month.# This should suffice assuming the amount of bills you pay arent loco.## Author: Glen Zangirolami - http://github.com/glenbot# GIST: https://gist.github.com/4445947importsimplejsonimportrequestsimportloggingfromdatetimeimportdatetimelogging.basicConfig(level=logging.ERROR)log=logging.getLogger(__name__)# List of bills in the following format:# (bill_name, day_of_month, [notify_x_days_before, notify_x_days_before, ...])# Example: ('Macys Credit Card', 5, [7,3,1]) will notify you that# the Macys card is due 7, 3, and 1 day before the 5th of any monthBILLS=(('My Bill #1',5,[7,3,1]),('My Bill #2',7,[7,3,1]),('My Bill #4',12,[7,3,1]),)# Phone numbers of people you want to send the text message to# in the format "+12815555555"SEND_SMS_TO=["+12815555555"]# change me to a real number# SendHub API settings# https://www.sendhub.com/settingsSENDHUB_USERNAME='2815555555'# 10 digit number you signed up withSENDHUB_API_KEY='xxxxxxx'# api key from settings pageSENDHUB_API_URL='https://api.sendhub.com/v1'defsendhub_request(endpoint,payload={},_type='GET'):"""Make a request to the sendhub API"""# create the API urlurl='%s/%s/'%(SENDHUB_API_URL,endpoint)validation={'api_key':SENDHUB_API_KEY,'username':SENDHUB_USERNAME}if_type=='GET':# inject the api key, and username in the query stringpayload.update(validation)log.debug('HTTP %s to %s with payload %s'%(_type,url,payload))returnrequests.get(url,params=payload)if_type=='POST':headers={'content-type':'application/json'}log.debug('HTTP %s to %s with payload %s'%(_type,url,payload))returnrequests.post(url,params=validation,data=simplejson.dumps(payload),headers=headers)returnNonedefsendhub_get_contacts():"""Get a list of contacts from the sendhub API"""request=sendhub_request('contacts')ifrequestandrequest.status_code==200:data=simplejson.loads(request.content)returndata['objects']return[]defsendhub_send_sms(data):"""Send an SMS text"""returnsendhub_request('messages',data,'POST')defget_notifications():"""Run through the dates and build a notification list"""# placeholder for all notificationsnotifications=[]# get the current day of the monthday_of_month=datetime.now().dayforbillinBILLS:# unpack the bill databill_name,bill_day,bill_notify=bill# see if we need to send a notificationifday_of_month<bill_day:days_until_due=bill_day-day_of_monthifdays_until_dueinbill_notify:sms="A reminder that your %s is due in %d days"%(bill_name,days_until_due)notifications.append(sms)returnnotificationsdefsend_notifications(notifications):"""Send the notifications to the receipients in SEND_SMS_TO"""sendhub_contacts=sendhub_get_contacts()contacts_to_send_to=[]# parse a list of contacts to send to, we aren't using# groups from sendhub here although the code is easy enough# to modify to use groupsforsendhub_contactinsendhub_contacts:ifsendhub_contact['number']inSEND_SMS_TO:contacts_to_send_to.append(sendhub_contact['id'])ifcontacts_to_send_to:fornotificationinnotifications:data={'contacts':contacts_to_send_to,'text':notification}log.debug('Sending payload %s to sendhub'%data)response=sendhub_send_sms(data)ifresponse.status_code!=201:log.error('Could not send text message, HTTP Status %s'%(response.status_code))else:log.debug('Could not find any contacts to send to')if__name__=='__main__':notifications=get_notifications()send_notifications(notifications)