Chapter 16: Automatic communication

=================================== Automatic data communication as a development strategy. To get a lead on your competitors. To avoid duplication of effort. To reduce costs. To reduce boring and repetitive work. To avoid having to remember technical details. Automatic communication is both for professionals and amateurs. First, because it keeps the costs down. Second, because it lets you do the job faster and safer. We all have different needs --------------------------- Automation will never be the same for everybody. Our needs are too different. Some get excited when a program can dial a bulletin board, retrieve a program, and then disconnect without them having to touch the keyboard. Some want an "answering machine" that can respond to and forward email when he or she is away from the office. Others want a communications system that can tap selected news sources, search databases, and do postprocessing on the retrieved material. For most professionals, doing things manually takes too much time. Time is better spent reading, digesting, and using, rather than on stupid technical retrieval work. Computers can do that. To others again, automation is a question of being able to use the online resource at all. If it takes 60 seconds to get a piece of information, it may be possible to get before running for the next meeting. If it takes 15 minutes, however, there may not be enough time. If you also need to read a help text to find out how to do it, you may not even consider it. The mind is full of other things right now. | When using a system for automatic communication, you do not | | have to learn and remember online commands. The system will | | do it for you. | The minimum solution -------------------- Automatic data communication in its simplest form entails the following: * One keypress to get the communications program to dial a number, and send user name/password when the online service requests this information. * Macro commands (like in a word processor) for navigating through an online service, searching, and to send complex commands by pressing one key. Most communication programs have a macro language or a script language. You will probably never regret time spent on learning how to use these features. At a minimum, you should be able to have your system log on to a service automatically. Autologon spares you the task of remembering your user name and password. Besides, most people are only able to use the keyboard at a low speed. They easily get frustrated by having to correct typing errors. Auto-logon with Procomm ----------------------- Procomm is one of the most popular communications program in use today (see appendix 2). A Procomm script file is a text file, which can contain a list of commands for dialing and navigating on an online service. When writing a Procomm script for auto-logon, your first step is to list the commands that you believe required. Enter them in a text file (as DOS or ASCII text). In such scripts, you can test for the occurrence of a small piece of information that the online service is supposed to send at a given time (like the question "Password?"). When this information is found, Procomm can be set to send the proper response or command (here, your secret password). Scripts can be tied to your favorite online services through Procomm's dialing directory. Press a key to start the appropriate script file for access to a service. The following is a simple PROCOMM script file. It can be used to access my bulletin board in Norway. It assumes that your name is Jens Mikkelsen, and that the secret password is FOXCROOK4. You'll have to change this before testing. ; ;Script file for auto-logon to SHS with PROCOMM and PROCOMM PLUS ; WAITFOR "our FIRST Name? " PAUSE 1 TRANSMIT "Jens^M" WAITFOR "our LAST Name? " PAUSE 1 TRANSMIT "Mikkelsen^M" WAITFOR "ots will echo)? " PAUSE 1 TRANSMIT "foxcrook4^M" WAITFOR "^JMore (Y),N,NS? " PAUSE 1 TRANSMIT "n^M" WAITFOR "^JMore (Y),N,NS? " PAUSE 1 TRANSMIT "n^M" WAITFOR "R] to Continue? " PAUSE 1 TRANSMIT "^M" It is not difficult. You probably understand a lot already. Here is the explanation: * the ";" character at the beginning of a line identifies it as a comment line. Procomm is to ignore it. We use such lines for notes. * WAITFOR "our FIRST Name? " has Procomm wait for the text string "our FIRST NAME?" from my BBS. It is a part of the question "What is your first name?". * PAUSE 1 halts the execution of the script file for one second. * TRANSMIT "Jens^M" sends the name "Jens" followed by a Return (the code ^M in Procomm). * WAITFOR "our LAST Name? " makes Procomm wait for the question "What is your LAST Name?" The script continues like this. In WAITFOR commands, we use part of the text that is displayed on our screen once the scrolling stops. Make sure that the search term is unique. It must not appear elsewhere in the text coming from the host computer. If it does, your name and password may be sent too early. You can call the script HORROR.CMD, and attach it to the entry for my board in your Procomm phone directory. When you call it the next time, Procomm will execute the commands in the file and "turn the keyboard over to you" when done. Macros in Procomm ----------------- Above, we used a script to log on automatically to a service. When Procomm gives us access to the keyboard again, we must continue manually. What we want to do online varies. Sometimes, we want to read new messages in conferences. In other cases, the purpose is to check new programs in the file library. If we find programs of interest, we may want to download them. Shorthand macros can help you do this faster and safer. For example, one macro can take you quickly to a conference for new messages. You can make Procomm start this macro whenever you press ALT-0 (keep the ALT key down, then press 0). You can have the macro key ALT-1 send other commands when in the file archives. When I started using MS-DOS computers for data communications, PC-TALK became my favorite program. It has many of the same macro capabilities that Procomm has. With PC-TALK, I did autologon to NewsNet. Macro number one sent commands that gave me the contents of various newsletters. Macro #2 picked up the contents in another group. Macro #3 picked up stories from my mailbox, and macro #4 logged me off the service. My mission was completed by pressing four or five keys. Automating the full task ------------------------ It's a long way from automated logon scripts and the use of macros to automating the whole task. The major difference is that with full automation, you do not have to look at the screen while the script is working. You can do other things. Sometimes, you may not even be present when the job is being done. On a typical morning, I go directly from bed to my office to switch my communications computer on. While I visit the bathroom, my communications program calls three online services, retrieve and send information. When the script has disconnected from the first service, which is my bulletin board, it analyzes the received data. I want an alphabetic list of visitors since my last visit, a sorted list of downloaded programs, and names of those calling in at 9600 bps or higher. Sometimes, the unexpected happen. There may be noise on the line, or a sudden disconnect. Usually, my script can solve this without manual intervention. It is therefore allowed to work unattended most of the time. When I get to my office after breakfast, it is all done. My communications program is set for reading and responding to today's email. I can sit down, and immediately get to work. After having written all my replies, I say "send" to my system. For me, it's time for another cup of coffee. I am not needed by the keyboard while my mail is being sent. This is what an automatic communications system can do. My scripts also help plan and prepare online visits, and ease my work by postprocessing results. | When your communication is fully automated, you need not | | read incoming data while it scrolls over your screen, and | | then again after logging off the service. You do it only | | once. | How to get it? Here are some alternatives: Alternative 1: Write your own system ------------------------------------ You can write procedures for powerful script-driven programs like ProYam (from Omen Technology) and Crosstalk MK IV. I started writing scripts for ProYam over seven years ago. The system is constantly expanded to include new services, refined to include more functions, and enhanced to become more robust. The scripts make my system work like an autopilot. It calls online services, navigates, retrieves and sends data. Postprocessing includes automatic reformatting of retrieved data, transfers to various internal databases, statistics, usage logs, and calculation of transfer costs. Such scripts can do quite complex operations online. For example, it can - Buy and sell stock when today's quotes are over/under given limits, - Select news stories and other types of information based on information found in menus or titles. Script writing is not for everybody. It is complicated, and takes a lot of time. Therefore, it is only for the specially interested. On the other hand, those going for it seldom regret. Tailor- made communication scripts give a wonderful flexibility. The software does not cost much, but again, it takes a lot of time! | Do not use large and complex script files before you know the | | online service well. The scripts let you do things quicker and | | safer, but there is always a possibility for unexpected | | problems. | | | | Test your scripts for a long time to make them robust by | | "training" them to handle the unexpected. Leave them to work | | unattended when you are reasonably certain that they can do | | the job. - It may take months to get to that point. | | | | Build a timeout feature into your scripts, so that they don't | | just hang there waiting for you after an encounter with fate. | Alternative 2: Use scripts made by others ----------------------------------------- Some script authors generously let others use their creations. Earle Robinson of CompuServe's IBM Europe Forum, share his ProYam scripts for automatic usage of CompuServe with others. They are available from the IBM Communication Forum library. Enter GO XTALK on CompuServe to find advanced script files for Crosstalk Mk.4. ZCOMM and ProYam scripts for visiting my board automatically can be freely downloaded there. They split access up into these three phases: Phase 1: Menu driven offline preparation. Phase 2: Automatic logon, navigation through the system, and automatic disconnection. Phase 3: Automatic offline postprocessing. You will find scripts for other programs on many online services. Alternative 3: Special software ------------------------------- Several online services sell communication programs with built-in functions that provides you with automation. They can have offline functions for reading and responding to mail. The degree of automation varies. There are also many programs written by third parties. Most programs assume that you use 'expert' as your default operating mode on the online service. TapCIS, Autosig (ATO), OzCIS, CISOP, CompuServe Navigator (for Macintosh), AutoPilot (for Amiga), ARCTIC (for Acorn Archimedes), and QuickCIS (for Atari) are popular choices on CompuServe. TapCIS is my personal favorite. (CIM does not offer much automation!) Aladdin is for GEnie. It automates your use of RoundTables (conferences), file areas, and mail. Dialog users turn to Dialog- Link. Nexis News Plus (for Nexis, US$50) has pull-down menus and detailed selection of commands. This MS-DOS program helps users set up detailed search commands before logging on to the Mead Data Central. Your search results will be downloaded automatically. Personal Bibliographics Software, Inc. (Ann Arbor, Mich, U.S.A. Tel.: +1-313-996-1580) sells Pro-Search to Dialog and BRS users (for Macintosh and MS-DOS). Pro-Search will lead you through menus to find information on both services. It translates your plain English search commands into the cryptic search language used by the services. It logs on automatically, connects to these services, finds your information, and shows you the hits. Alternative 4: Offline readers ------------------------------ The alternatives above have one important weakness. Noise on the line can prevent the "robot" from doing the job. All it takes is for noise to give a prompt another content than is expected by your program or script (as in "En@er a number:" instead of "Enter a number:"). You can avoid noise problems by using get commands (see Chapter 15), and by making the online service use its minimum prompts ('expert mode') . Still, this does not give full protection. The best is to let the online service do the navigation. Think of it as logging on to run a batch file on the remote computer. Combine this with automatic transfers of your commands, transmitted in of one stream of data with automatic error correction (in the software and in the modem), and you have a very robust system. The program logs on to the service. Then the service takes over. It registers your user identity, checks your user profile for personal interests, retrieves and packs all messages, news and files into one compressed file, and sends it to you at high speed. Your outgoing messages, search commands, commands to join or leave conferences, and more, are transferred to the remote computer in a similar packet (compressed file). When received by the remote computer, it unpacks the transfer file and distributes messages and commands to various services following your instructions. Your "physical" contact with the service is when your modem is disconnected. The help menus that you read belong to your program, and not the online service. You read and respond to mail in a reading module (ref. the term "offline reader"). Some offline readers give the caller access to more tools than is available on the online service itself. They may have spelling checkers, multimedia support, let you use your favorite editor or word processor, and offer various storage, search, and printing options. Using offline readers is probably the easiest, cheapest, and safest way of using online services. These "readers" are popular among bulletin board users, and some commercial services are also starting to accommodate them. There are many offline reader programs. The most advanced take over completely upon logon, and manage transfers of commands and compressed information files to and from the host. (Example: Binkley Term on FidoNet) Global Link is an offline reader for EcoNet. Bergen By Byte offers the BBS/CS Mail Grabber/Reader, a script system used with the communications program Telix and the service's "auto-get" function. The most popular systems on the PCBoard based Thunderball Cave BBS are Offline Express, Megareader, Session Manager, Rose Reader and EZReader. These are used with scripts written for various communication programs. Some of them have built in communications (and script) modules. EZReader from Thumper Technologies (P.O. Box 471346, Tulsa, OK 74147-1346, U.S.A.) lets users retrieve mail from several online systems using transfer formats such as QWK, PCBoard capture files, ProDoor ZIPM files, XRS, MCI Mail, and others. Cost: US$49 (1992). 1stReader from Sparkware (Post Office Box 386, Hendersonville, Tennessee 37077, U.S.A.) is my personal favorite for accessing Qmail based online systems. | Note: Some offline readers contain all the features required | | for fully automated communications. Some bulletin boards allow | | up- and downloading to start right after CONNECT. | | Off-Line Xpress, an offline mail reader for QWK (Qwikmail) | | packets, does not contain a communications module. It just does | | pre- and postprocessing of mail packets. | | You can use the Off-Line Xpress as one element in a larger | | automated system. For example, a system for access to PCBoard | | bulletin boards may consist of Off-Line Xpress software, PKZIP | | and PKUNZIP (popular shareware programs to compress/decompress | | mail packets), the QMODEM communications program, and a script | | to navigate to/from the QWK packet send and receive area on the | | BBS. | | 1stReader (version 1.11) contains a powerful script based | | communications module. It lets you compose replies, set search | | commands, subscriptions to services, add and drop conferences, | | and enter download commands offline. | Automatic automation -------------------- We have explained how to write scripts with Procomm. However, there are simpler and quicker ways. Many communication programs can make scripts automatically using a learning function. It goes like this: Start the learning function before calling the online service. Then log on, navigate to the desired services, do what you want to automate, and disconnect. The learning feature analyzes the received data and builds a script file for automatic communication. If you call again with the new script, it will "drive the same route one more time." ZCOMM and ProYam have a learning feature. This is how I made a script for accessing Semaforum BBS using ZCOMM: ZCOMM asked for a phone number. I entered +47-370-11710. It asked for speed, and I entered 2400 bps. Next, I had to choose one of the following: (1) System uses IBM PC (ANSI) line drawing (2) 7 bits even parity (3) 8 bits no parity My choice was 1. ZCOMM dialed the number. When the connection was established, I entered my name and password, navigated to the message section, read new messages, browsed new files in the library, and entered G for Goodbye. This was the "tour" that I wanted to automate. When disconnected, I pressed the F1 key. This prompted the learning process based on a record of the online tour. The log described everything that had happened in detail, including my pauses to think. Now I was prompted by the following question: 'newscr.t' exists. Append/replace/quit? I selected append. Then: Do you want this script file as a new entry in your telephone directory (y/n)? I entered "y," and named it "semaforum." After a few seconds, my new script was ready: Your new script is in the file 'newscr.t' !! You can append the file to your current script file (for example PHODIR.T) or have the commands executed by entering: call semaforum.newscr.t It was time to test the new wonder. I entered call semaforum.newscr.t at the ZCOMM command line, hit the Enter key, and off it went. ZCOMM called the BBS and repeated everything - at far higher speed than I had done it manually. It went on-hook as planned when done. Limitations ----------- Auto-learn programs can create a script file that let you "drive the same route." For some applications this is enough. For others, it's just part of the way. You have to refine the script manually to get what you want. Example: If you call my bulletin board with an auto-learned script made yesterday, chances are that everything works well. If you call twice on the same day, however, you're in for a surprise. The board greets you differently on your second visit. You will not get the menu of available bulletins. It will take you directly to the system's main menu. Your script must take this into account. On most online services, many things can happen at each "junction of your road." At one point in one of my scripts, up to twenty things may happen. Each event needs its own "routing." Twenty possible events are an extreme, but three to four possibilities at each system prompt is not unusual. All of them need to be handled by your script, if you want it to visit online services unattended while asleep. It is quicker and simpler to use other people's scripts and programs, although this might force you to use a different program for each service. Personally, I prefer offline readers on services where such are able to do the job. On other services, I usually depend on my own tailor-made scripts.