If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

jquery/ajax to load a big text file in real time

Hi folks,

One of our research project requires a real-time-updated (big!) text file to be loaded to a table for visualization. I am a researcher with some (very limited) basic coding and I thought of using javascripts/html/php to archive the goal. So far I was able to load a test text file (in real time) to a html table. Please check it here: http://grih.org:8888/test/refresh/. The test text is here: http://grih.org:8888/test/refresh/data.txt. The idea is pretty basic:

+ since in the real project, the text file will be big, and loading big file will give an ugly visualization of data. We want to do some other kinds of dynamic content so that for example, all data will be loaded to pages like the Ajax Pagination script, and the changes from text file will be shown in the first page. All other pages can be loaded later if users click on them.

+ is there a better way of doing this, with only html/javascript? I know a bit of php and coded some, so that's why I did it with the help of php.

controls the frequency of the update. Here 9 seconds (9 times 1000 milliseconds) But, in actual practice, if a lot of people are viewing the page, there would be more frequent updates because an interval would be established for each visitor and they would likely be firing off at various times. It might be a better idea to make the updating (the part that writes the separate pages) a 'chron job'. I'm not sure how to do that yet. Or we could make a flag file that disappears when a user leaves the page to tell this page whether or not to set up its own interval, but that could get messy because it would have to check periodically if that flag disappeared, make its own at that point and set up its own interval. I'll think about it.

For now I've added the LOCK_EX flag to the file_put_contents() function so that if two visitors try to write at the same time, that should at least prevent file corruption.

If this is more for internal use by an organization and not too many people would be viewing it at one time, more frequent updates are probably not a big issue. The LOCK_EX flag should be all that's required to deal with it.

It took me some time to find out how John's solution works. I then thought it might be possible without PHP. I created a text file (data.txt) and put 'arie' in it. I also created a html file (index.html) and put the following in it:

I opened index.html and there it was (in index.html), after 4 seconds: 'arie'.
Then I added 'molendijk' to the text file. After 4 seconds, my index.html showed 'arie molendijk', which proves that the content in index.html was dynamically changed in real time.

Added border color change for hover of the .pages links and for the active page, see style section in the above code.

This line (highlighted):

Code:

jQuery(function($){
...
});

controls the frequency of the update. Here 9 seconds (9 times 1000 milliseconds)

John, the updated code works perfectly!

Originally Posted by jscheuer1

But, in actual practice, if a lot of people are viewing the page, there would be more frequent updates because an interval would be established for each visitor and they would likely be firing off at various times. It might be a better idea to make the updating (the part that writes the separate pages) a 'chron job'. I'm not sure how to do that yet.

You are absolutely right about this. I also think that creating htm chunks for each viewer is not as good as having "fixed" htm chunks for all viewers (meaning viewers can only view, not create) - this also probably helps in security. In our real project, there is a unix bash script which is called by an unix daemon if there is new data. This bash script then appends new data to the data.txt file. I can modify that bash code so that after appending to data.txt, it will delete old htm chunks (if created before) and create new ones using the updated data.txt. The php/html/javascript simply loads it up for visualization.

Or we could make a flag file that disappears when a user leaves the page to tell this page whether or not to set up its own interval, but that could get messy because it would have to check periodically if that flag disappeared, make its own at that point and set up its own interval. I'll think about it.

For now I've added the LOCK_EX flag to the file_put_contents() function so that if two visitors try to write at the same time, that should at least prevent file corruption.

I dont really understand neither the "flag" part you mentioned nor the LOCK_EX flag. Does it like a "write" and "read" permission flags in unix system?

Originally Posted by jscheuer1

If this is more for internal use by an organization and not too many people would be viewing it at one time, more frequent updates are probably not a big issue. The LOCK_EX flag should be all that's required to deal with it.

The real project can be huge, but I doubt there will be many people view the page at the same time. Anyway, learning different solutions and "playing safe" are always good, right?

We are going to have the real system in the next few days and I gonna test what you suggested. I will post back the results for sure.

It took me some time to find out how John's solution works. I then thought it might be possible without PHP. I created a text file (data.txt) and put 'arie' in it. I also created a html file (index.html) and put the following in it:

I opened index.html and there it was (in index.html), after 4 seconds: 'arie'.
Then I added 'molendijk' to the text file. After 4 seconds, my index.html showed 'arie molendijk', which proves that the content in index.html was dynamically changed in real time.

I am not a coder, but from what I understand, you are loading an iframe onto a div, and in turns the iframe loads and refreshes its data - the data.txt file - using html refresh mechanism. If the data.txt is big, the load/refresh can be very long. Not to mention that converting from txt to html table will create large html size. I remember I tried to load about 1k-2k lines from a text file to a html table and the browser seemed to be frozen.

I can modify that bash code so that after appending to data.txt, it will delete old htm chunks (if created before) and create new ones using the updated data.txt.

That would be the way to go. In fact, it should be used to create the chunks in the first place. So that part of the PHP code would be dropped from the page. Instead of creating the files, PHP would only count them to determine how many .pages links there should be. That could be done either by parsing the main data.txt file as an array like we're doing now into the same chunk size as the bash script uses, without writing the files, or by actually counting the chunk files the bash script makes. I think parsing the data.txt file would be more efficient, maybe even just counting its lines. Then, instead of PHP writing the chunks (bash has already done that), have javascript update the number of .page links if different than before on the basis of PHP reporting how many that there need to be. When each chunk file is fetched via AJAX, it will be the current one created by bash because the AJAX routine is set to not cache the responses.

So we would still set an interval of 9 seconds (or less or more), but simply to check if the data.txt file has changed by having recorded its file date and time on page load as a reference, and comparing and updating that each time the interval fires, and if it has changed, then count the chunks, update the nav if required, and refresh the currently displaying chunk just in case it has changed. Again, any other chunks that are then fetched via AJAX will be the current ones because we're not caching them.

When I get a bit of time, I will write that all up.

One thing puzzles me though. You say that bash will add to the data.txt file. Do you mean that literally? I mean, does the old data ever get taken away or changed? If not, we don't ever have to update the currently displaying chunk file. But it defies logic to think that old data would never change or be removed, so I think we should keep that part anyway, even if it's not anticipated to be frequently needed.

Oh, and I almost forgot - the LOCK_EX flag means lock exclusive, so that while writing to the file, no other process may write to the file. It's not so much a flag as an argument or parameter of the file_put_contents function. PHP calls it a flag though. If bash is doing the writing, though, there's no need to be concerned with that. The other flag I mentioned would be a file we would create in an attempt to keep track of whether or not an update loop is already in process when a given user loads the page and if so not to initialize a new one. But if bash does the updating of the chunks, a flag file for that would no longer be needed.

One more thing though before I could write that, will bash be making the the td tags in the chunk flies:

I am not a coder, but from what I understand, you are loading an iframe onto a div, and in turns the iframe loads and refreshes its data - the data.txt file - using html refresh mechanism. If the data.txt is big, the load/refresh can be very long. Not to mention that converting from txt to html table will create large html size. I remember I tried to load about 1k-2k lines from a text file to a html table and the browser seemed to be frozen.

I changed my code a little bit. It works better now. There's a demo HERE.
I extract the contents of a hidden iframe with src=data.txt and write them onto the div. The iframe refreshes at regular intervals, ensuring that the div always contains the latest data.
I don't have a javascript-mechanism comparable to the PHP-fgets, so the text file is not converted to a html table.
But I made sure that the data are readable enough by putting them inside pre-tags.Index.html looks like this:

I've been playing around with this. I think that it would be a good idea to have the bash script write/overwrite a flag file when it's finished making the new chunks and to use that, rather than the main data.txt as the file for the live page to look at to tell it when things have updated. That way we should be able to avoid a situation where the data.txt file has been updated but the chunks haven't all been updated yet.

A fine point. For now I have it working off of the data.txt file, like so:

If you have any problem outputting chunks like that, let me know. However, it would be best to use the above format.

Also - The bash need not delete the old chunk files, though it can. All it has to do is overwrite them. Any extra ones (like if there end up being less chunks after an update) will be ignored by this script.

Added Later:

I played around with this a bit more. This time with a flag file. It really is the way to go because it can hold the new number of chunk files - all the live page needs to know in order to update, which saves a lot of time when refreshing. I have it worked out, along with a PHP version of what the bash would need to do. If you're interested, let me know.