My Howtos and Projects

This blog is about: Open Source,
Operating Systems (mainly linux), Networking
and Electronics
The information here is presented in the form of howtos.
Sometimes the information migth be in portuguese!
As vezes a informação pode estar em portugues!

Tuesday, November 7, 2017

Expect is a natural and intuitive automation scripting language that operates in much the same way humans do when interacting with a system. You type in commands and expect a certain response to your command. When you receive the expected response, you enter another command and so on. Expect works in the same way, except you have to provide the script with commands and expected responses to those commands. Basically, you have to script out the entire two-way “conversation.”

You can think of an Expect script as a dialog script written for two actors: a sender and a receiver. One of the more popular activities to automate is an SSH session between two hosts, in which one host is the sender (local host) and the other is the receiver (remote host). Being able to emulate every keystroke and create a true interactive session between two systems via a script is an exciting proposition.

Expect Setup

Most Linux distributions include Expect as part of the available and installable software packages. In other words, you won’t have to download and install from source code. Use your system’s package manager to download and install Expect and any required dependencies or associated packages. For example:

$ sudo yum install expect
or
$ sudo apt-get install expect

Once you have Expect installed, you can begin writing scripts.

Creating an Interactive SSH Session

As stated previously, you must provide both sides of the conversation in your script because you’re setting up an interactive system. Look at a few essential items before diving right into a script.

To make an Expect script executable as a standalone program, you must do two things: Make the script executable, and supply the path to the script for expect . The path on my system is: /usr/bin/expect ; therefore, enter that path on the first line of your script with a preceding “shebang” (#! ):

#!/usr/bin/expect -f

The -f switch tells Expect that it is reading commands from a file.

The spawn command spawns or launches an external command for you. In this case, ssh to a remote host (aspen ):

spawn ssh aspen

Change the host aspen to your remote host. When you SSH to a remote system, you’re prompted for a password. This password prompt is what you “expect” from the remote system; therefore, you enter that expected response:

expect "password: "

From the local side, you have to enter your password at the password prompt. To send anything to the remote system, it must be included in double quotes and must include a hard return (\r ). Change PASSWORD to your password:

send "PASSWORD\r"

Again, you have to enter the expected response from the remote system, which in this case is a user prompt ($ ).

expect "$ "

Now that you’re logged in to the remote system, you can begin your interactive session on that remote host. The following send command issues the ps -ef |grep apache command:

send "ps -ef |grep apache\r"

Output will appear as STDOUT. After the command has executed, you’re returned to a prompt, so tell the Expect script that bit of information:

expect "$ "

Finally, send the exit command to the remote system to log out. Don’t forget that hard return (\r ):

Change permissions on the script so that it is executable; for example,

$ chmod 755 script.sh

and try it for yourself.

Expect Caveats

If your script hangs and doesn’t continue, try the command manually yourself and look for the response. If the remote system drops you to a prompt as its final act, then place that in your script (e.g., expect "$ " ). Be sure you have entered the hard return (\r ) inside the closing quotation mark in your send line. You might also find that your system needs two backslashes on the send line for a hard return (\r ).

Sometimes Expect scripts execute too fast, and you won’t see your expected response. If that happens, place a sleep command and a number of seconds for the command preceeding it to wait for a response, or your data might be ignored.
For example, if you connect to a remote system and there’s a delay in creating that connection, your script will continue to execute and fail because it sends commands before the remote system has time to respond.

You have to think about network delays, shell responses and system timing when scripting in Expect. Like any scripting language, Expect has its quirks, but you’ll find that it’s an easy way to automate those repetitious keystrokes and procedures. The time you spend debugging your scripts is well worth the effort.

Autoexpect

Of course, some lazy system administrators take lazy to a higher level and even cheat at writing Expect scripts by invoking a shell “watcher” or recorder script named Autoexpect. Once invoked, Autoexpect watches your every keystroke and records it to a file named, script.exp by default. You’ll almost certainly have to edit and prune this script to achieve your desired results; however, it can save hours of script debugging to have an almost complete script from which to work.

If you simply run a freshly created Autoexpect script, it will likely fail because, if you issued a command that answers your request by displaying information to the screen, the script picks up that answer, too, and copies it into the script file.
For example, if during your Autoexpect session, you type, ls , the result of that command appears in your script.exp file as well. After you’ve created a few Expect scripts by hand, you’ll appreciate the cleanup editing you have to do in an Autoexpect-created script.
To install Autoexpect, issue a command like:

$ sudo apt-get install expect-dev

You’ll likely require many more dependencies for this feature, so prepare yourself for a slight delay while everything installs.

Creating an Interactive SSH Session with Autoexpect

After installing Autoexpect and all of its required packages, you’re ready to create Expect scripts automatically by stepping through the procedures you want to automate. Using the above example, SSH to a remote system and run a

ps -ef |grep apache

command and then log out.
Invoking Autoexpect is easy:

$ Autoexpect

Autoexpect started, file is script.exp
$

Although it looks as if nothing has happened or is happening, every keystroke you type will be recorded into script.exp . Every STDOUT response you receive will also be copied into that same file. Your entire session is recorded – but not just recorded, it is also formatted in Expect script style. To stop recording keystrokes to your script, press Ctrl+D on your keyboard to stop Autoexpect and copy the buffer to your file.

The complete transcription of this simple procedure is very long and includes a lot of commentary from the author, Don Libes:

#!/usr/bin/expect -f
#
# This Expect script was generated by Autoexpect on Thu Oct 11 15:53:18 2012
# Expect and Autoexpect were both written by Don Libes, NIST.
#
# Note that Autoexpect does not guarantee a working script. It
# necessarily has to guess about certain things. Two reasons a script
# might fail are:
#
# 1) timing - A surprising number of programs (rn, ksh, zsh, telnet,
# etc.) and devices discard or ignore keystrokes that arrive "too
# quickly" after prompts. If you find your new script hanging up at
# one spot, try adding a short sleep just before the previous send.
# Setting "force_conservative" to 1 (see below) makes Expect do this
# automatically - pausing briefly before sending each character. This
# pacifies every program I know of. The -c flag makes the script do
# this in the first place. The -C flag allows you to define a
# character to toggle this mode off and on.

#
# 2) differing output - Some programs produce different output each time
# they run. The "date" command is an obvious example. Another is
# ftp, if it produces throughput statistics at the end of a file
# transfer. If this causes a problem, delete these patterns or replace
# them with wildcards. An alternative is to use the -p flag (for
# "prompt") which makes Expect only look for the last line of output
# (i.e., the prompt). The -P flag allows you to define a character to
# toggle this mode off and on.
#
# Read the man page for more info.
#
# -Don

The prompt still works because Expect looks for the last few characters in an expect line and not the entire string. You could shorten the line that expects the password prompt from:

expect -exact "password: "
to
expect -exact ": "

A word of caution against shortening your expect lines too much – it makes the script more difficult, not easier, to read and interpret in the future when you try to figure out what’s going on.

You might not realize that ": " is a password prompt. Unless you’re great at including comments in your scripts, you might spend hours debugging this shortened version.

Summary

To be perfectly honest, I only use Autoexpect when building an Expect draft script. To sit down and attempt writing Expect line-by-line just isn’t appealing after being seduced and ruined by the ease of removing unwanted lines from an Autoexpect-created script. Autoexpect makes using Expect fun and more intuitive by letting you perform a procedure one time instead of many. After discovering and using Autoexpect, my Expect scripting creation time and debug time has been cut by at least two-thirds. I suspect you’ll have much the same return on your time as well.

3. Hit OK. Then navigate to the Board Manager by going to Tools > Boards > Boards Manager. There should be a couple new entries in addition to the standard Arduino boards. Look for esp8266. Click on that entry, then select Install.

4. The board definitions and tools for the ESP8266 include a whole new set of gcc, g++, and other reasonably large, compiled binaries, so it may take a few minutes to download and install (the archived file is ~170MB). Once the installation has completed, an Arduino-blue “INSTALLED” will appear next to the entry

5. Select the Generic ESP 8266 Module Board.

6. Set the Upload Speed and CPU Frequency.

7. Select the device port. (Device should be connected to the PC)

8. To test the ESP8266, open up serial monitor (Ctrl + Shift + M)

9. Set baud rate to : 115200 baud and Both NL & CR with AutoScroll

10. Type AT command in the serial monitor and click enter. If you see an OK message, you are good to go.

Tuesday, April 18, 2017

Here I’ going to show you how to stream media (in this the example is video but it should be similar to other media) and share files with a Smart TV.

Smart TVs support a standard protocol called DLNA that allows you to stream video and share files to stream, among other things. Normally TV brands give it another name like AllShare (Samsung).

Windows Media Player and Windows itself support DLNA, wich is very usefull in order to display you media in you TV.

Enable Streaming

The DLNA streaming on Windows is off by default, so we need to enable it, the easiest is from Windows Media Player, just go to:

Turn on you TV and wait a bit (20s) after click on “Turn on media streaming”:

now just click OK (check that you TV is on the list and is allowed), additionally you can change you media library name and disallow devices in your network:

later on if you want to comeback to this menu to Allow or Disallow a device you can just go to Windows Media Player again:

An alternative to get to this menu without Windows Media Player is just to go to the Start Windows icone and type “Media streaming options”, and click on the icon with that name:

Stream From the PC to the TV

First put your files here:

now this video will be available on the Videos section in Windows Media Player, you can select it and cast the video to your TV.

The first time you do this a popup will show on your TV asking if you allow the PC to stream, just select OK, the next time the streaming will automatically.

when the streaming starts, it will show you the controlls and the progress bar:

and you should be seeing the video on you TV .

The inconvenience with this is that you have to go to you PC to start the video and maintain the player window open (in the next topic we are going to see how to start videos from the TV).

By default TV you only have the pause control over the video, but you can enable all the controls (Back, Forward, Previous Video, Next Video) with this option:

Start Videos From The TV (or another PC)

At this point if you already can stream from the PC to the TV, what you don’t know is that when you enabled streaming in the above section you are also started sharing the files in the Video library (by default only the Video folder) on your PC via DLNA.

Now on the TV you can get the the Videos library’s folders that your PC is sharing.

In my TV (Samsung Smart TV) you can find the PC by pressing on the “Sources” in the TV remote.

As you can see bellow your PC is represented by a media icon, which has you PC name plus the library name you gave when you enabled streaming.

# Parameters that are managed by the system and usually should not need# changing:# SHELLINABOX_DATADIR=/var/lib/shellinabox# SHELLINABOX_USER=shellinabox# SHELLINABOX_GROUP=shellinabox

# Any optional arguments (e.g. extra service definitions). Make sure# that that argument is quoted.## Beeps are disabled because of reports of the VLC plugin crashing# Firefox on Linux/x86_64.SHELLINABOX_ARGS="--no-beep"

Monday, February 6, 2017

Docker is a “container” platform, which allows applications to be run in their own sandboxed world. These applications share resources, e.g. things like hard drive space or RAM, but otherwise can’t interfere with programs running on the host system. For corporate servers this means an attacker may not be able to use a compromised web server to get at the database holding customer data.

For the desktop user, it means the bleeding-edge app you’re trying out can’t accidentally delete all your cat’s selfies.

Pros and Cons of Using Docker

There are several good reasons to try out new programs via Docker, including the following:

They are safely isolated from your system, without the means to do damage in most cases.

Docker containers have a mechanism to keep them up-to-date, meaning it’s easy to make sure you have the latest and greatest versions.

You’re not installing anything on your “real” system, so you won’t run into conflicts with your “regular” versions on the application. You could, for example, run LibreOffice on your host system, but run OpenOffice in a container (you know, in case you don’t believe the project is shutting down).

Speaking of versions, you can even have multiple (but different) copies of the same version running on your machine at once. Try that with Word 2016!

Some Docker apps run their own minimized version of Linux. This means even if the app isn’t normally compatible with Mac or Windows it may still work for you within a Docker container. Try them out before you switch to Linux full time.

They’re easy to clean up. Don’t like the way things turned out? Just trash the container and create a new one.

On the other hand, there are some caveats to using applications this way:

As they operate in their own little world, they don’t have access to your files unless you give it to them. That means if you want to try the brand new version of LibreOffice via Docker, you may need to do some additional work to make your files accessible.

In general, Docker apps ship with everything they need to run, which often includes libraries that could be re-used with other programs. Some even ship with a full operating system behind them. So you may be doubling up on disk space usage.

They don’t provide convenient icons and other desktop-centric niceties. While we’ll show you a GUI you can use to download and run these Docker containers, they won’t show up in your main application launcher unless you create an entry by hand.

Like many things open source, it’s members of the community who have been creating these Docker applications from their upstream releases. This means your access to the latest version and/or any bugfixes is at the mercy of these peoples’ free time.

Installation and Usage

Getting things up and running involves three preliminary steps:

First, get Docker installed and running on your system (including a graphical interface for it, if you want one).

Next, find and download an image for the application you want to run. While you normally install an application, you get one (and only one) copy of it. Think of an image as a template for the application — you can create as many installs from this template as you like.

Lastly, create one of those copies, called a container, and run it.

Let’s look at each of these in detail.

Installation

Most Linux distribution have Docker available in repositories for easy installation. In Ubuntu, the following command will get you what you need:

The Docker daemon will start up with your system automatically by default, but you can set that differently if you know how to adjust your systemd settings.

If you’re interested, you can also grab the Simple Docker UI Chrome app. Follow the instructions here to get things set up so you can connect to the Docker daemon on your machine.

Note: If you use Simple Docker UI, make sure you add yourself to the “docker” user group as described here. If you’re not part of this group, you won’t be able to use Docker commands from your normal (non-root) user account, the one with which you’ll be running Chrome and its apps, without using sudo all the time.

Finding and Installing Desktop Applications With Docker

Now that you’ve got a nice UI going, it’s time to find something to install. Your first stop should be the Hub, a repository of applications hosted by the docker project. Another straightforward way to find some interesting applications is to Google for them. In either case look for a “Launch Command” along the lines of the following:

docker run -it -v someoptions \

-e more options \

yet even more options...

Paste this into a terminal and it will download and launch the application for you.

You can also “pull” the application, then launch it yourself. If you’re using the Simple UI app, it can search Docker Hub automatically for your keyword.

Once you’ve found what you’re looking for, click its listing, then the Pull Image button in the pop-up dialog to download the image of the application.

Remember, an image is a “template” of sorts. Next you’ll need to create a container that uses your new image. Switch over to the Images tab. Clicking the Deploy Container button will create a new, runnable copy of your application.

Running Your New Docker Container

From the command line, you can view a list of all your docker containers with the command:

docker ps -a

This lists the containers with some of their stats — note the “NAMES” column to the far right. To restart one of your containers, pick the name of the container you want and issue the following:

docker start [containername]

Using the app, go the “Containers” screen, select the container you want, and click the “Start” button in the upper left of the screen. Your application will start in a new window on your desktop, just like a “normal” application.

Your application should open in a new window, just as if you had installed it normally. But remember, it exists in isolation from your other applications. This allows you to do some neat things, like run LibreOffice and OpenOffice in parallel (their dependencies usually conflict with one another):

Try Docker-ized Apps for Fun and Profit

Docker provides an easy way to get an app up and running so you can try it out, and an equally easy way to clean it from your system. Once you get through the initial set-up of Docker, a single run command is often all you need to download an image, create a container from it, and launch it on your desktop.