Page 1 of 2

HTML from remote PWS

Posted: Thu Jan 07, 2010 5:39 pm
by daddylito
My idea, is to create a small program, that periodically run on my computer
which creates the YoWindow xml-file with data fetched from a PWS not controlled by me
combined with data from a METAR station.

I have no PWS of my own, but there is one 15 km away.
Of course, these values are better than the airport 70 km away.
However, as Pasha has mentioned you only get sky description data from METAR stations!

From the PWS there is a HTML-file with a layout that seems to be stable.
And the file has most of wanted values.
It should be possible to extract data from that file as well as from the METAR station.

But I have questions:
1) The program, that periodically run.
- Could it be a script?
- Must it run periodically, is there a another solution to keep data updated?

2) Anyone seeing any other problems?


Re: Personal Software PWS with data from other PWS and METAR

Posted: Fri Jan 08, 2010 12:41 pm
by gemini06720
You do not mention anything about the local PWS near you, such as which software is being used...

If the software is Weather Display, then, you might eventually be able to convince the station operator to use the PHP script for Weather Display that is being developed by Jim and myself (although not cooperatively yet).

If the software is Cumulus, then, eventually, there might/will be a PHP script for that software.

Now, a try at answering your questions:
But I have questions:
1) The program, that periodically run.
- Could it be a script?
- Must it run periodically, is there a another solution to keep data updated?
A PHP script is not running all the time - only when YoWindow updates its information. But that PHP script must be able to get its weather data from a file that is continuously updated.
2) Anyone seeing any other problems?
What do you mean by this, such as getting the YoWindow data from an HTML file? Although it is possible to get/extract information from and HTML page you have to remember that a page is not updated that often...

Re: HTML from remote PWS

Posted: Fri Jan 08, 2010 5:42 pm
by par
Hi, Bernt!
I have changed the subject for a better one, hope you don't mind.

You have proposed a good feature.
I think we need a way for people without PWS to access the data from the closest PWS they know somehow.

It is possible to create a program that would download HTML, extract the weather data and create YoWindow XML file periodically.
I hope someone in our community would create this software.
Any decent programmer could create such a program.

Sorry, right now I have to direct my energy to YoWindow specific tasks.
If nobody will help us, I will, but not now.

Script or Program?

Posted: Sat Jan 09, 2010 2:52 pm
by daddylito
Ray and Pasha, thanks for your answers!

The PWS is a Davis Vantage Pro, 5 years old (my guess) .
I don't know what software is used!
The cyclic update time of the HTML-file is 5 minutes.

My idea of combining PWS data and METAR data, is handled, as I understand it,
of YoWindow, since YoWindow will fetch missing values from the Metar.

So the solution is to create some code that reads the HTML-file and creates/updates the XML-file.

I havn´t studied how YoWindow will handle a PWS, but is it possible to make YoWindow
run a supplied script or must it be a standalone program?

Re: HTML from remote PWS

Posted: Sat Jan 09, 2010 5:01 pm
by nico
Best way would be to convince that station to use YoWindow! Then the xml-file would already be there.

However I can imagine that the traffic generated by say a couple of 1000 people downloading the xml-file every time would be considerably. Perhaps the pws would not be so happy about that, remember that most of them have the website as a guidance to a few local people and traffic is low. So please get in contact with them.

Re: HTML from remote PWS

Posted: Sun Jan 10, 2010 10:47 am
by daddylito
Hi Nico!
I´m not sure I understand what you mean.
The traffic I will generate is reading the HTML file once every 5 minutes!

Why do you think 999 others would do the same as me?

Re: HTML from remote PWS

Posted: Sun Jan 10, 2010 10:05 pm
by par
Yes, it's possible to create a script in PHP or Perl and send automatic requests from YoWindow to fetch the data from PWS HTML through the script. Eh... Am I sound clear? :)

Re: HTML from remote PWS

Posted: Mon Jan 11, 2010 6:10 am
by gemini06720
Guys, most weather software produce a data file, such as (those are the only 2 files I have access to):
- Weather Display == 'clientraw.txt' file produced every 5 secondes with 166 bit of data;
- Cumulus == 'realtime.txt' file produced every 5 seconds with 50 bits of data.

Both files are text only and both are space separated. The 2 files are uploaded to the Web host servers by the weather software as both files are used by either PHP scripts and/or AJAX/Java scripts.

There is no way that those files can be automatically sent to individual users as CRON would be needed on the Web host server and most free and low cost Web host servers do not offer that option. Also, sending files to individual user would involved additional traffic (at additional costs) something most weather station operators cannot afford (luckily, some of us, the station operators, are using a Web host server with unlimited traffic, but those Web host servers are mostly within North America).

The only practical solution would be a program that would connect to the Web host server, download one of the files mentioned above, process it and eventually produce the required XML file needed by YoWindow - obviously, that file would have to be saved somewhere onto the user's local hard drive AND YoWindow would need the facility/option to read the XML file directly from the user's local hard drive.

But, still, the PWS station operator would have to be contacted and would have to give his/her approval - as I indicated above, the additional download of the files would create additional traffic and, if that traffic exceed the maximum monthly allowable number of bytes available, then access to the Web host server would be cut off until the beginning of the next month or until the weather station operator pays for the additional traffic - many/most Web hosts operate that way.

Trying to use an HTML file produced by a PWS software is not that easy - I should know as I have 4 different PHP scripts downloading, reading and decoding HTML files available from some Web host servers and produced by my other software... Pasha, if you are willing to write such a PHP or Perl script to decode an HTML file, go ahead, and good luck, as the script will only be working fine as long as the HTML page is not modified ... and that would be a problem with most of my Web files/pages as some of them are continuously updated via AJAX/JavaScript...

Re: HTML from remote PWS

Posted: Mon Jan 11, 2010 2:23 pm
by daddylito
Hi Ray!

I guess that the sizes of the 2 files you mentioned was in bytes and not in bits!

I agree with you that this HTML concept might not work in all cases and can you find
some kind of raw data file, that you can interpret, that is better.

But I don't agree that this might cause traffic problems.

The HTML-file is a very small file! The file has only references to images (if any),
it doesn't hold any images!

In my case would I download 744 kB per day, cyclic update time 6 minutes, file size: 3.1 kB.
Of course, it´s a additional load, but this load is about the same as viewing the HTML with a web-browser once every day.

Reading one ascii file and producing another, seems as a pretty simple task (if you have learned
correspondig programming language).

As for me, I have never done any PHP programming.

Re: HTML from remote PWS

Posted: Mon Jan 11, 2010 3:40 pm
by jmcmurry

What you are asking is certainly possible, but it would be a custom script for just that one site and not good for anywhere else. Then, even though you say the page is stable, if the author were to make any changes, it's likely that you'd need to start all over again.

I'd recommend that you contact that individual and see what software he is using and whether there's a "standard" file being uploaded as is the case with Weather Display and Cumulus. And also ask his permission to utilize it.

- Jim