File: massage.txt by M Gunter This file outlines the manipulation (or "massaging") of data collected on site at the Breamlea wind generator. The file logger.txt in this directory is a listing of the "schedule" (or program) which configures the DataTaker DT50 data logger to record data in its RAM. Normally this data would be periodically downloaded to a PC using a serial interface and proprietary software. Just to be a bit different, I decided to interrogate the logger using "Telix" running a simple SALT script. This was set up on COM1 of an old Toshiba T1100 which a relative happened to own (subsequently donated to the ATA). As luck would have it, a 2400 bps internal modem was available for that same computer. When plugged in, it appears on COM2. Thus it is possible to run Telix as a sequence of calls from a DOS batch file, each one configured to talk either to the logger or the modem. The batch file loops back on itself every 4 days, so on the fourth day when it exits the "daily" script for the fourth time, it rearranges the on-board backup files before getting telix (configured with the "determ.cnf" file) to unload the memory of the data logger to a telix capture file (good old "telix.cap"!). So here is the listing of "breamlea.bat": :START A: TELIX SDAILY.SLC TELIX SDAILY.SLC TELIX SDAILY.SLC TELIX SDAILY.SLC B: IF EXIST TELIXCAP.BAK DEL TELIXCAP.BAK IF EXIST TELIX.CAP REN TELIX.CAP TELIXCAP.BAK A: IF EXIST NEWDATA.BAK DEL NEWDATA.BAK IF EXIST NEWDATA.ZIP REN NEWDATA.ZIP *.BAK TELIX CDETERM.CNF SUNLOAD.SLC TELIX SMICHAEL.SLC GOTO START The next listing is the timing file "daily.slc" which uses the computer's clock to jump to the next line of the batch file once daily. If you're familiar with the Telix script language SALT you will know that the *.slt files must be compiled using cs.exe to *.slc files before telix can interpret them. All the scripts here are *.slt versions. main() { cputs_tr("~~~+++~~~ATH0^M"); start: delay_scr(100); if (thour(curtime())==20 && tmin(curtime())==50) exittelix(0,1); goto start; } The next script, unload.slc prompts the data logger to send a status report and unload its most recent data: that is done by the string in the "cputs_tr" line. Remember that Telix is talking to the logger on COM1 at this time, not to the modem on COM2, because it was run using the "determ.cnf" configuration file which specifies which COM port, baud rate, etc main() { capture("b:\telix.cap"); delay_scr(20); cputs_tr("^M~/U^M~STATUS^M~~~/u^M~U(LAST)^M"); delay_scr(3000); capture("*CLOSE*"); exittelix(0,1); } The second last line of the batch file contains telix in default config- uration (telix.cnf) talking to the modem using the "michael.slc" script which rings my computer at home: main() { run("pkzip.exe","newdata b:\telix.cap"); cputs_tr("ATDT0393767515^M"); delay_scr(80); cputs_tr("~~~+++~~~ATH0^M"); delay_scr(1200); dial("m0393767515",3); send('Z',"newdata.zip"); cputs_tr("~~~+++~~~ATH0^M"); exittelix(0,1); } Looking at the above script as I write this note (finally documenting something I patched together over 7 months ago!), I cannot imagine why I decided to compress the capture file by running pkzip as a "shell" out of telix. It would be much more logical to run it BEFORE invoking the modem-talking telix program i.e. straight after the file captured from the data logger is written to the b: floppy (yes - this setup doesn't even need a hard disk!) Anyhow, it works! Note that it gives a short warning ring, and then waits 120 seconds before ringing again to send the zipped data as a Zmodem transfer. We had terrible trouble as complete novices getting THAT to work, until it finally dawned on us that you MUST disable xon/xoff flow control when sending zipped files, as they are liberally sprinkled throughout with ASCII characters 17 and 19 (you guessed it - the xon/xoff characters!!) No wonder the file transfers kept crashing after about half a second! Meanwhile, back at home, roused by the first ring, I have staggered out of bed, put the kettle on for a cuppa and fired up Telix in auto-Zmodem- download-mode. Within a few seconds of the second call, the file is safely tucked away on my home computer's hard disk, awaiting the assault of dBaseIII+ But wait ! There's more ! (preparation, that is) First I unzip the data file and decapitate it: that is remove the status report "header". This is a fiddly manual operation, but fairly easily achieved using the "gather" function of Xtree Gold. If you have never had cause to use it, it is a subfunction of file viewing: text is highlighted or "gathered" and then appended to any file name you specify - I call it "new.log" and it consists of all of the capture file except the status report at the top. Now for the big finale ! Am I the only computer Luddite still using dBaseIII+ in the whole universe? Any dBase "Trekkies" out there can Email me at mickgg@suburbia.net for some old fashioned dBase banter such as "go bott", "appe from" and my favourite "set rela to". The following routine is actually a concatenation of three smaller routines that evolved over an 8 month period, and my word, DOESN'T IT SHOW ! Who cares - it does what I want semi-automatically, and just imagine how much cutting and pasting would be required to achieve one tenth of the result using Excel (I know nothing of newer generation databases so I cannot comment about them - someone enlighten me please) !copy newdt.dbf *.bak set talk on set stat on set date brit use newdt go bott mrecno=recno() append from new.log deli with blan goto mrecno+1 * the next line creates a character field which sorts "alphabetically" by date and time repl datetime with righ(chardate,4)+subs(chardate,4,2)+left(chardate,2)+left(time,2)+subs(time,4,2)+righ(time,2) whil .t. set filter to time='00:00:00' * the data logger's main fault is the way it records the date and time: any * summing function which is logged at midnight on day x will actually carry * the date of day x+1. The next line converts all 0000 hrs loggings on * day n to be recorded as 2400hrs on day n-1. Try doing that with Excel ! repl datetime with '19'+righ(dtoc(ctod(chardate)-1),2)+subs(dtoc(ctod(chardate)-1),4,2)+left(dtoc(ctod(chardate)-1),2)+'2400' all set filt to * you still have a local backup in the zip file, so............ !del telix.cap !del new.log goto mrecno+1 repl no_shadow with av_ws whil .t. goto mrecno set filt to kwh>0 .and. ((dirn>0 .and. dirn<90) .or. dirn>330) skip * the following polynomial expression is used to calculate (from energy * data) what the anemometer would have been reading if it was not in * the shadow of the wind generator's rotor. The anemometer is directly * behind the rotor when wind is from the northeast (about 30 degrees) repl no_shadow with .0006*(kwh**3)-.0251*(kwh**2)+.5584*kwh+3.5256 whil .t. set filter to av_ws>no_shadow goto mrecno skip * this next line is statistically invalid (??) repl no_shadow with av_ws whil .not. eof() set filt to ((dirn>0 .and. dirn<90) .or. dirn>330) ?'Winds at 22 metres (raw & corrected) are as follows' ?'for the shadowed sector 330 deg to 90 deg.' goto mrecno skip set talk on aver av_ws,no_shadow for ((dirn>0 .and. dirn<90) .or. dirn>330) whil .not. eof() ?'Winds at 22 Metres (raw and corrected) for directions 90 to 330 degrees:' set filt to dirn>89 .and. dirn<331 goto mrecno skip aver av_ws,no_shadow whil .not. eof() wait set safety off set filt to inde on righ(datetime,4) to time sele 2 use timeoday if .not. bof() zap endif sele 1 go top do whil .not. eof() Mtime=righ(datetime,4) aver av_ws,no_shadow,2*kwh to Mraw,Mcorr,Mkw whil righ(datetime,4)=Mtime sele 2 appe blan repl time with Mtime,raw_ws with Mraw,no_shadow with Mcorr,kw with Mkw sele 1 enddo use sele 2 go top brow clos all set safety on retu Well that's about it. The rest is just your usual sorting and charting, for which I use Excel. The results of that can be seen in some of the gif charts such as history.gif and dec95.gif. Shrinking the charts for display on the Web would be to chronicle yet another tedious operation (using the software that I have to hand, anyway). A precis of how I do it is on the chart demo.gif M Gunter 2-2-1996