RSS Feeds, Twitter, and Log Files

 amazon s3, condor, google apps, log files, microblog, rss, syslog, twitter  Comments Off on RSS Feeds, Twitter, and Log Files
Jun 292008

I keep having the same thought lately: RSS Feeds, microblogging services like Twitter, and computer system log files are very similar, and maybe are even different faces of the same animal.

All three have the following features:

  • A list of messages is provided.
  • The messages are listed in chronological order.
  • Each message is available for a limited time, after which it is deleted or archived.
  • The messages are short, generally under a few hundred characters.
  • The messages proxy for or imply longer communications.
  • There is a well-advertised central repository for the messages.
  • Theme-specific subrepositories can be created as desired.
  • The method for adding new messages is well-defined.

There are some differences, and, perhaps, ways to think about how to improve all three by cross-pollinating their features:

RSS feeds provide both short messages and links to more detailed messages. They are relatively expensive to create, since their format is more complex and the detailed messages require separate handling. They are usually created by webmasters in parallel with web pages, or are generated automatically when new web pages are created. Some RSS message systems give the creator the ability to “tag” messages–to add keywords that allow for easy topical searches.

Twitter-style messages (or “tweets”) provide only the message itself. They are relatively easy to create. They are generally created by non-technical users, and often are created on standard cell phones.

Log files messages provide only the messages themselves. They are easy for computers to create. They’re generally created automatically, as a result of applications and system services running on computers. They tend to have a low signal-to-noise ratio; the reader may have to wade through a lot of unimportant messages to find the critical ones. But most logging facilities have the ability to control the “importance” level of messages that are actually added to the log files, and the applications that create the log messages can tag the importance level of each message.

There are currently many cross-creations between RSS feeds, microblogs, and logfiles. For example, some logfiles are available as RSS feeds. Some robots create microblogs.

But I think it might be useful to have a service that supports all three: a super-RSS service. On the message creation side, the server could accept new messages via standard RSS files; via tweets created on cell phones or web browsers; or via syslog calls. The messages could be simple–just a line of text. Or they could be complex, and include “importance” tags, keywords, and links to more detailed information sources.

On the message reading side, the reader application could filter the messages by importance, author / source / friend, and keywords. Messages above an importance threshold could be handled differently than those below the threshold. For example, “critical” messages could be forwarded to a cell phone via SMS, while less important ones would simply be queued up in the reader.

One area where this super-RSS system could be very useful would be for users of cloud / grid / mesh / distributed application servers, like Condor, Google App Engine, or Amazon’s S3 Web Services. Users would have a single place to look where they could find messages from others on the same team (“IMPORTANT: I’ve just updated the user interface, let me know what you think–Joe”), warnings from system administrators (“CRITICAL: The servers will be down Sunday for maintenance”), and status updates from the computers running in their cloud (“CRITICAL: Error. Simulation halted. Cannot access database.”; “DETAIL: Node 165 opened file ‘profile3.txt'”). The messages would be in chronological order, and the user could choose to view only those that are of “critical” or “important” level, and ignore the “general information” or “detail” messages.

More thoughts on this later.

 Posted by at 4:41 AM