What I am trying to do is with arduino get values from a web site, and from the values close or open a relay (turn a digital pin high or low). While at the same time host a simple HTML page that displays the current position of the relay and the values I am reading from the website.

I am using an arduino mega (ATmega1280) board and an Ethernet shield.

I have managed to set up a simple HTML page that can be viewed on the local network and externally over internet, using the web server example. In other words I have set up the port forwarding for the arduino.

I need help or suggestions (even feasibility) on how to get values from a web site and displaying them on the html page hosted by the arduino.

The web site is http://80.85.104.41/ its a sunny webbox, say I would like to get the total yield value.

Quote

Is the Arduino supposed to to get data mining in addition to serving up a page?

I have no idea, that`s why I am posting here to see if anyone has done it before or if it is even possible. If its not possible, I would have to settle for getting the values and activating the relay only (would still need help to do this).

That site requires a password to access. What happens when you enter a password?

Making the Arduino-as-server serve up a web page that contains a form, and making the Arduino process the form when the submit button is pressed is a piece of cake, it you understand client/server/web programming.

If not, zoomkat will come along shortly and post some code. Mine's on my other computer.

What I am trying to do is with arduino get values from a web site, and from the values close or open a relay (turn a digital pin high or low). While at the same time host a simple HTML page that displays the current position of the relay and the values I am reading from the website.

Below is some combined server/client code you can experiment with. You will need to develop the code required to perform your specific needs.

//zoomkat 12-08-11, combined client and server//simple button GET with iframe code//for use with IDE 1.0//open serial monitor and send an g to test client and//see what the arduino client/server receives//web page buttons make pin 4 high/low//use the \ slash to escape the " in the html //address will look like http://192.168.1.102:84 when submited//for use with W5100 based ethernet shields

Does the target page have a RSS feed or does the website provide an API? Otherwise you need to look at scraping the value from the site/page, here's a Wikipedia article on web scraping. http://en.wikipedia.org/wiki/Web_scraping

Should the led (pin4) remain off(low) once the off button in the web page has been clicked? ....as it is only going off for a moment and back on, i.e one blink.

Pin 4 should stay low until another request tells it to go high. There should be no blink (no pin state changeunless commanded). Do note that using your browser's back button will resend the previous command.

Quote

what is this part of the code doing?

This tells the browser to not expect a new page to be sent. I use this with a javascript video feed to prevent disruption of the feed. You can remove it if you don't need it.

Quote

Also this part of the code is confusing me,

This is the client part of the code. It sends a request to a server for the arduino.txt file.

Google forum search: Use Google Advanced Search and use Http://forum.arduino.cc/index in the "site or domain:" box.

Picking up where I left off, the following code is working fine. With one exception when I click the on or off button using google chrome as the browser it only blinks off the led and the turns the led back on. So if anyone knows why this is happening I would be happy to learn about it, but for now I will use explorer and safari as browser when communicating with arduino.

Also if anyone know about a good tutorial on ethernet syntax would be much appreciated.

//George-Xrik 31-03-12, updated//zoomkat 12-08-11, combined client and server//simple button GET with iframe code//for use with IDE 1.0//open serial monitor and send an g to test client and//see what the arduino client/server receives//web page buttons make pin 4 high/low//use the \ slash to escape the " in the html //address will look like http://192.168.1.177 or http://88.203.15.76 when submited//for use with W5100 based ethernet shields

Picking up where I left off, the following code is working fine. With one exception when I click the on or off button using google chrome as the browser it only blinks off the led and the turns the led back on. So if anyone knows why this is happening I would be happy to learn about it, but for now I will use explorer and safari as browser when communicating with arduino.

Also if anyone know about a good tutorial on ethernet syntax would be much appreciated.

//George-Xrik 31-03-12, updated//zoomkat 12-08-11, combined client and server//simple button GET with iframe code//for use with IDE 1.0//open serial monitor and send an g to test client and//see what the arduino client/server receives//web page buttons make pin 4 high/low//use the \ slash to escape the " in the html //address will look like http://192.168.1.177 or http://88.203.15.76 when submited//for use with W5100 based ethernet shields

With one exception when I click the on or off button using google chrome as the browser it only blinks off the led and the turns the led back on. So if anyone knows why this is happening I would be happy to learn about it, but for now I will use explorer and safari as browser when communicating with arduino.

Sounds like chrome is being non compliant with the returned 204 status code. 204 tells the browser to not refresh the currrent page. Looks like chrome is refreshing orreloading a prevous page instead of doing nothing. The below code refreshes the current page and doesn't have te 204 code included.

//zoomkat 2-13-12//DNS and DHCP-based web client test code//for use with IDE 1.0//open serial monitor and send an e to test//and to see test result//for use with W5100 based ethernet shields//browser equivelant URL: // http://web.comporium.net/~shb/arduino.txt//note that the below bug fix may be required// http://code.google.com/p/arduino/issues/detail?id=605