RSS Feeds, Twitter, and Log Files

 amazon s3, condor, google apps, log files, microblog, rss, syslog, twitter  Comments Off on RSS Feeds, Twitter, and Log Files
Jun 292008
 

I keep having the same thought lately: RSS Feeds, microblogging services like Twitter, and computer system log files are very similar, and maybe are even different faces of the same animal.

All three have the following features:

  • A list of messages is provided.
  • The messages are listed in chronological order.
  • Each message is available for a limited time, after which it is deleted or archived.
  • The messages are short, generally under a few hundred characters.
  • The messages proxy for or imply longer communications.
  • There is a well-advertised central repository for the messages.
  • Theme-specific subrepositories can be created as desired.
  • The method for adding new messages is well-defined.

There are some differences, and, perhaps, ways to think about how to improve all three by cross-pollinating their features:

RSS feeds provide both short messages and links to more detailed messages. They are relatively expensive to create, since their format is more complex and the detailed messages require separate handling. They are usually created by webmasters in parallel with web pages, or are generated automatically when new web pages are created. Some RSS message systems give the creator the ability to “tag” messages–to add keywords that allow for easy topical searches.

Twitter-style messages (or “tweets”) provide only the message itself. They are relatively easy to create. They are generally created by non-technical users, and often are created on standard cell phones.

Log files messages provide only the messages themselves. They are easy for computers to create. They’re generally created automatically, as a result of applications and system services running on computers. They tend to have a low signal-to-noise ratio; the reader may have to wade through a lot of unimportant messages to find the critical ones. But most logging facilities have the ability to control the “importance” level of messages that are actually added to the log files, and the applications that create the log messages can tag the importance level of each message.

There are currently many cross-creations between RSS feeds, microblogs, and logfiles. For example, some logfiles are available as RSS feeds. Some robots create microblogs.

But I think it might be useful to have a service that supports all three: a super-RSS service. On the message creation side, the server could accept new messages via standard RSS files; via tweets created on cell phones or web browsers; or via syslog calls. The messages could be simple–just a line of text. Or they could be complex, and include “importance” tags, keywords, and links to more detailed information sources.

On the message reading side, the reader application could filter the messages by importance, author / source / friend, and keywords. Messages above an importance threshold could be handled differently than those below the threshold. For example, “critical” messages could be forwarded to a cell phone via SMS, while less important ones would simply be queued up in the reader.

One area where this super-RSS system could be very useful would be for users of cloud / grid / mesh / distributed application servers, like Condor, Google App Engine, or Amazon’s S3 Web Services. Users would have a single place to look where they could find messages from others on the same team (“IMPORTANT: I’ve just updated the user interface, let me know what you think–Joe”), warnings from system administrators (“CRITICAL: The servers will be down Sunday for maintenance”), and status updates from the computers running in their cloud (“CRITICAL: Error. Simulation halted. Cannot access database.”; “DETAIL: Node 165 opened file ‘profile3.txt'”). The messages would be in chronological order, and the user could choose to view only those that are of “critical” or “important” level, and ignore the “general information” or “detail” messages.

More thoughts on this later.

 Posted by at 4:41 AM

A Quick And Dirty Guide to RSS

 aggregator, atom, feed, google reader, my yahoo, newsfeed, opml, rss, subscription, xml  Comments Off on A Quick And Dirty Guide to RSS
May 252008
 

RSS icons

I love RSS. I am a self-confessed, dyed-in-the-wool RSS junkie. I can’t imagine surfing the web without it.

You may be thinking: Well, that’s really nice. Good for you. It’s important to have something that you believe in.

And you may also be thinking: What the heck is RSS? And why should I care?

I’ll answer the second question first, since it is the more important one. There’s also a lot of RSS-related jargon to wade through, but I’ll leave that for last.

What RSS Is

RSS is a convenient way of getting lists of fresh, up-to-the-minute things from the web. These things might, for example, be:

  • The current sales and new items at your favorite on-line stores.

  • Bands, concerts, nature walks, you name it–events that are happening around town in the next few days.

  • Today’s comic strips and political cartoons–just the ones you want, without needing to wade through the stupid ones.

  • The latest news, from your favorite on-line papers.

  • The most recent rant or how-to or funny picture by your favorite bloggers.

  • New podcasts or videos from the artistes that you give your patronage to.

  • Current weather reports–very useful for travellers.

  • Job postings, customized for what you’re looking for.

All of these things are pulled from the web automatically for you, daily, or multiple times a day, without you having to lift a finger. They’re presented to you as lists of links with descriptions. Click on one of the links in the list, and the web page that it refers to is opened in your browser.

Once you’ve looked at an item on the list, your computer will keep track of it and won’t show that item to you again. It’ll only show you new, unread items on the list. (Unless you explicitly tell it to show you everything on the list, even the old stuff that you’ve already looked at.)

I personally find that the most valuable lists are for those websites that I really enjoy but that aren’t updated very often, and that my pea-brain forgets to check without a little RSS help. For example, I really like Tim Eagan’s comics, but he generally only puts out one a week, so I usually forget to go to a website that carries his stuff and look for the latest one. With RSS, I don’t need to remember. Tim’s comics are in my lists, and whenever he publishes a new one, it’ll show up for me.

The other type of RSS list that I find useful is one that replaces a lot of hassle. For example, I love Chad Vader videos. (If you haven’t checked out Darth’s younger and less successful brother, he’s a hoot.) Chad’s videos are posted on YouTube, but it is kind of a hassle to go to YouTube, search for “Chad Vader”, sort through the thousands of videos that have been posted by Chad’s actual creators and legions of fans, try to guess from the sometimes misleading titles if each video is one I’ve seen or not, and so on. Or, to try to remember the name of the people that make the Chad Vader videos (Blame Society Films, which is a funny name but I still can’t remember it), and search for “blame society”, wade through the videos, etc. It’s a hassle.

But YouTube lets me create a custom RSS list just for Blame Society Films. I create it once, and done. Now whenever there are new Chad antics to watch, they show up in my list, and I watch them. So simple. Ahhhh. Yeah, thanks, RSS, and YouTube, and thanks especially, Blame Society Films.

RSS lists still aren’t available for every website, but more and more sites are adding RSS support all the time. And many sites allow you to really customize your list just the way you want it. Sadly, some websites that do provide RSS lists don’t seem to really get the idea, and provide lists that are kind of…useless. (NetFlix, I’m looking at you.)

How To “Do” RSS

So, it sounds pretty good, huh? Setting yourself up with RSS is fairly easy. The basic idea is that you get yourself an RSS reader. If you use the Mac Safari web browser, or Firefox, or the very latest version of Internet Explorer on Windows, you’ve already got an RSS reader built right into your browser. There are exactly twelve zillion other RSS readers that you can install on your PC or Mac; most are free, but some you have to pay for. Personally, though, I like to use Google Reader, which is web-based. It’s fairly easy to set up, I can use it from any computer, and it’s free. If you have a Google Mail account, you’re already set up to use Google Reader. Another popular free RSS reader is My Yahoo!, which you’re already signed up for if you have an e-mail or IM account through Yahoo!. There are roughly thirteen quintillion zillion other web-based RSS readers out there.

OK! Sounds good! Let’s go get us some lists!

The jargon, the lingo, the RSS insiders’ secret codewords

Hold up, there, pardner. We’re at the point where we need to jump in and deal with some jargon. Being a relatively new computer technology, RSS has a pretty good heap of jargon already. Some of it is pretty misleading, and can really trip you up. Let’s go through a few of the more common terms:

  • RSS – Short for Really Simple Syndication. RSS is the computer file format of the lists of things we want to look at. RSS lists are designed to be easily passed around and comprehended by computers, but they’re also just text files that you can read yourself if you really want to. Not surprisingly, an RSS list stored in a file will probably have a filename like “AmazonSales.RSS”. Note that there are multiple versions of RSS, since RSS is still relatively new and is still changing. The latest version of any good RSS reader can handle all of them.

  • Feed – Websites that provide RSS lists usually call them Feeds. They’re not really feeds of any sort. Nothing is being fed to your computer; the feeds aren’t forced down your computer’s throat. They’re just lists of things on the web, and your computer needs to go get them just like it does with normal web pages.

  • Subscription – Another misleading term. When you use an RSS list, you’re not really subscribing to anything. You don’t need to provide any personal info to anyone, or pay anyone anything to use the RSS list. When you subscribe to an RSS feed, you’re simply adding the RSS list to your existing lists. That’s all.

  • Newsfeed – A list or feed of…news stories. Since news bloggers and newspapers were some of the earliest publishers of RSS lists, the term newsfeed and feed became somewhat synonymous.

  • Aggregator – A nerdy name for your RSS reader. Since the reader aggregates–or collects–the lists of stuff, it is called an aggregator.

  • OPML – Short for Outline Processor Markup Language. A list of lists; a computer format for passing around multiple RSS lists in a single file. The OPML list doesn’t contain the items in the RSS lists, but just refers to the lists themselves.

  • XML – Short for eXtensible Markup Language. XML is just the “mama” of RSS; it is an (intentionally vague) computer file format that RSS is derived from. Many websites that provide RSS lists will call them XML instead, but they’re still RSS, or one of it’s siblings, like…

  • Atom – A sibling of RSS, and another way for computers to pass lists around. Atom has a different internal format from RSS, but works pretty much the same way. Any decent RSS reader can handle Atom too.

Subscribe to those fine RSS Feeds

OK, so, now we’re down to it. You’ve got yourself an RSS reader, you’re up on the jargon, and you’re ready to start makin’ some lists, baby. If you use Safari, or Firefox, or fresh fresh Internet Explorer, you just keep your eyeballs peeled for a little link that looks like this, or one that says “RSS” or “XML” or “Feeds” and click on it. It’ll either cause your browser to get excited and ask if you want to aggregate the feed to your bookmarks, or will lead you to a web page full of RSS lists. And you click on the list that you’re interested in, and then your browser adds it to your bookmarks. That’s it. Done. Here are some examples of the icons used for RSS lists:

A galaxy of RSS icons

If you’re using an on-line reader like Google Reader or My Yahoo!, there are three ways to get the RSS list into your reader:

  • Go to the reader’s web page, and search for what you’re interested in. The easiest way to do this in Google Reader is to click on the “Add Subscription” item at the left of the the window, then when the text box pops up, just type in the name of whatever it is you’re interested in, for example, “chad vader”. Reader will then show a list of feeds that are related to your search terms. Click on the feed that sounds closest, and that’s it.

  • Many websites provide links that are already set up for your RSS reader. These are nice, since you just click on the name of your RSS reader, and the feed will be added for you automatically.

  • Most websites still only provide basic links to their RSS lists. To get these ones into your web-based reader, you need to right-click on them, then select “Copy Link Location” from the pop-up list, then go to your reader’s web page, and paste in the link in the correct place. For Google Reader, click on the “Add Subscription” item, then paste the link into the pop-up text box.

RSS: Admitting There’s A Problem is the first step

All right, now you’re an RSS junkie too! And since you’ve had a taste of the good life and are hooked, it is time to reveal the dark side of RSS. Yeah, you just knew that something this good had to be pretty bad too, and you’re absolutely right.

Here are, in no particular order, the biggest problems that I see with RSS feeds:

  • Firehose. It’s very easy to subscribe to a lot of feeds, especially fun feeds. And many of the websites that publish those feeds put out several updates per day, which means several RSS items per day per site. That can add up to hundreds of new RSS items per day. Breaking the feeds up by category helps, but ultimately, it’s just a lot of information flowing at you. It can be overwhelming, yet you still want to keep up with your favorites. This can be stressful. Not stressful like being in a car accident, but just a little stressful. And we’re all already stressed out as it is.

  • Blandness. Each RSS reader has its own look, and RSS feeds have no real “look” to them at all. The result is that every item from every feed looks just like every other item. It is really easy to lose focus and just breeze past an item that you might be very interested in. Looking at individual feeds rather than all of them at once helps, but some feeds have dozens or hundreds of new items each day all by themselves, so it is still easy to zone out while reading through a list of hundreds of look-alike items from a single feed.

  • Ads. RSS feeds don’t have many ads in them so far, but there are a few, and there will be more. Some day soon, we’ll probably need to install special features in our RSS readers that block some or all ads. This might be tricky, since ads are no different from other RSS items, except that they talk about stuff that someone wants you to buy, rather than stuff you’re interested in.

  • Bloat. Some websites provide RSS items that are essentially the entire content of the web pages that they refer to, rather than just a link and a summary. The items will include thousands of words of text, dozens of large pictures, and even videos. Besides being an annoyance to scroll past, these items can really slow down or even crash some RSS readers.

The first taste is always free

And now, a few example RSS feeds:

The latest “Deep Cover” comics from Tim Eagan.

The latest “Chad Vader” videos on YouTube (and other comic creations from Blame Society Films)

Dave Barry’s Blog. (Not as good as his column used to be, but there is still a lot of funny stuff.)

The latest news and financial analysis from The Economist magazine.

Today’s Gold Box deals from Amazon.com

Well, that’s it, pardner. See you out at the feedlot.

 Posted by at 11:00 AM

Stupid Amazon RSS Tricks

 amazon, amazon power search, dover, onfocus, rss  Comments Off on Stupid Amazon RSS Tricks
May 172008
 

I’ll admit it. I’m an RSS junkie. I wrote a web-based RSS feed reader when RSS barely existed, and nowadays my Google Reader subscriptions number close to 100.

One of the things I think RSS feeds are (or should be) useful for is announcing when manufacturers or sellers have added a new gizmo or book or game or whatever to their store. Many retailers are catching onto this: ThinkGeek and American Science and Surplus both provide RSS feeds of new items for sale at their stores. But other sellers are slow to add these services. For example, I love Dover books. They’re incredible bargains, and every year or so I go on a binge where I buy like $100 of them or so (which is a lot of books at Dover’s prices!). But their website is pretty old-school, and they don’t provide a “new publications” RSS feed.

Enter Amazon, which provides a useful combination of partnering with just about every company around, customizable RSS feeds, and power searches. Problem is, Amazon doesn’t make them easy to use. So a whole cottage industry has sprung up which provides feeds for items on sale at Amazon, sorted by discount, coupon codes, you name it.

One such site is OnFocus, which provides a free web interface that allows you to roll your own Amazon RSS feeds. For example, let’s say that I want to create a feed that lists, say, new Dover books for sale at Amazon.

I could use OnFocus’ “Amazon Feed Generator” with a keyword of “Dover”, with Store set to “Books” and the sort method set to “Publication Date: New To Old”. This gets me what I want: an RSS feed of the latest (and yet-to-be-released) “Dover” books available at Amazon. However, it includes too many books; not just the ones published by Dover, but any with Dover in the title or written by someone named Dover, etc.

But there’s another, better way to create the RSS feed: using the “Power Search Feed Generator”. This uses Amazon’s “Power Search”, which isn’t as user-friendly (and only works for books, not video games or DVDs or anything else Amazon sells), but is useful for creating a very specific search.

Amazon’s Power Search works by letting you tie a search term to a specific area. For this example, I want to tie my search term “Dover” to publishers, not authors nor titles nor any other fields. So, I type this into the Power Search window:

publisher: Dover

and then set the sort method to “Publication Date: New To Old”, then click the “Create Feed” button. Voila! My new RSS feed is exactly what I want: new books at Amazon from Dover Publications.

The Power Search has several different areas you can limit the search by:

  • author: name, for example, author: Wolfe and Gene

  • isbn: number

  • keyword: words, I have yet to figure out how to get this field to work

  • language: language, for example, language: Spanish

  • pubdate: date, for example, pubdate: 2008

  • publisher: name, for example, publisher: Penguin or Tor or Baen or Dover

  • subject: subject, for example, subject: history

  • title: words, for example, title: Ringworld

You can use “and”, “or”, and “not” plus parentheses to combine these. For example, to search for books on Portugese Fado music that are written in English:

subject: fado and language: English

More details on Power Search terms can be found by scrolling to the “Power Searches” section near the bottom of this Amazon page.

BTW, here’s the Atom feed for this blog.

 Posted by at 7:55 AM