Unterbahn

I use Amazon S3 for backup, and find that it’s hard to use because there’s no search function, so I can’t see what I’ve backed up.

So, I extended s3cmd.rb (comes with the WONDERFUL s3cmd.rb rsync clone for Amazon S3) with a search command.

Now I can use s3sync.rb to rsync my files up, then type:

s3sync.rb search bucket_name:path/to/restrict/search search_term

and I get, for example:

backup/2009/spatial/jeff-spatial-vertical.key
backup/2009/spatial/spatial-assignment-1-jeff.pdf
backup/2009/spatial/spatial-assignment-1.key
backup/2009/spatial/spatial-assignment-1.key.zip
backup/2009/spatial/spatial-interface
backup/2009/spatial/spatial-lecture-1
backup/2009/spatial/spatial-lecture-1.key
backup/2009/spatial/spatial-lecture-1.key.zip
backup/2009/spatial/spatial-lecture-1/spatial-lecture-1.key
backup/2009/spatial/spatial-lecture-1/spatial-lecture-1.pdf
backup/2009/spatial/spatial-lecture-3.key
backup/2009/spatial/spatial-lecture-3.mov
backup/2009/spatial/spatial-lecture-3.pdf
backup/2009/spatial/spatial-uploads
backup/2009/spatial/spatial-wayfinding.key
you just spent 6.0e-05 cents

The last line is because the price of making a list request in s3 is $0.01 per 1000 requests in the US; so I thought it’d be a good idea to tell people how much they’re spending.

I’ll try to commit this to the main s3sync.net codebase, but for now you can just download it here: s3sync

I’ve been wondering how much of the data transferred by the OpenStreetMap API is actual geometric data as opposed to timestamp and author data. I ran some rough numbers on a typical API response (in JSON, not XML, though these are relative measurements, so it shouldn’t matter too much). The file I examined is here: dr5ru0.json but my count is by number of characters and I did include formatting.

I just tested the native JSON parser in Safari (this is also implemented in Firefox now, and IE8) and it dramatically speeds up the parsing of Cartagen’s JSON data files! Take a look at these times for parsing six typical data plots in milliseconds:

prototype.js’s .evalJSON():
parsed: 43
parsed: 7
parsed: 47
parsed: 63
parsed: 37
parsed: 53
Native JSON.parse():
parsed: 8
parsed: 6
parsed: 6
parsed: 9
parsed: 5
parsed: 7

That’s a sixfold improvement!

And for Firefox 3.5:

prototype.js’s .evalJSON():
parsed: 174
parsed: 157
parsed: 198
parsed: 349
parsed: 155
parsed: 455
parsed: 323
parsed: 276
parsed: 272
Native JSON.parse():
parsed: 13
parsed: 13
parsed: 14
parsed: 20
parsed: 12
parsed: 25
parsed: 20
parsed: 18
parsed: 18

That’s more than 15 times the speed!

Rails performs an SQL SELECT statement like “SELECT * FROM table_name WHERE table_name.id IN (1,2,3,4,5)” when its .find() method is passed an array of IDs. But it fails if one of those ids is not found. Sometimes it’s useful to find all _available_ records, but not fail if not all are found. I adjusted the search for this scenario in my working copy of the OpenStreetMap Rails port:

# @nodes += Node.find(nodes_to_fetch, :include => :node_tags)
@nodes += Node.find(:all,:conditions => ["id IN ("+nodes_to_fetch.join(',')+")"], :include => :node_tags)

I was surprised that Google News has no API, so I built my own: Using HTTParty, I wrote a small Ruby library for Google News. Put this in a ‘googlenews.rb’ file in the /lib/ directory of your Rails application to grab google news stories and topic lists.

Haven’t figured out how to get more than 10 items… Safari seems to be able to with feed://news.google.com?output=rss… but I’m not sure how.

gem "httparty"
require "httparty"

class Googlenews
  include HTTParty
  base_uri 'news.google.com'

  def self.items
    options = { :query => { :output => 'rss' } }
    features = self.get('/news', options)
    features['rss']['channel']['item']
  end

  def self.extract_topic(item)
    # ncl=duPfj30A5WoxBHMupoOtlciO1jY1M&
    item['description'].match(/ncl=([a-zA-Z0-9]+)&/)[1]
  end

  def self.topic(item)
    options = { :query => { :output => 'rss', :ncl => self.extract_topic(item) } }
    features = self.get('/news/more', options)
    features['rss']['channel']['item']
  end

  # store in database (unused)
  # def self.story(item)
    # item['category']
    # item['title']
    # item['guid']
    # item['description']
    # item['link']
    # item['pubDate']

    # t.string :category, :default => ''
    # t.string :title, :default => ''
    # t.string :guid, :default => ''
    # t.text :description, :default => ''
    # t.string :link, :default => ''
  # end

end

Once you’ve dropped this in /lib/, you can use it in a Rails controller like this (I needed JSON, to work with the feed in JavaScript):

class DataController < ApplicationController

  def latest
    render :json => Googlenews.items
  end

end

Update: to get more than 10 items, add a GET parameter ‘num=20′ to the feed URL. This works both for topic and for the main feed. I haven’t confirmed this but I think it’s still limited to 30 items. Again, Safari manages to get more, but perhaps it does repeated queries. Still investigating. Anyways, that makes it:

 def self.items
    options = { :query => { :output => 'rss', :num => '20' } }
    features = self.get('/news', options)
    features['rss']['channel']['item']
  end 

Update: This now has a real home on Github, where it’s called ruby-googlenews.

Excited to work with the New York Times API

Well, SVG doesn’t scale well to large numbers of objects, but Canvas doesn’t scale well to large screens:

Here are the results of the first fruitful experiment, which clearly shows that SVG performance degrades quickly (exponentially on Safari?) in the number of objects, but Canvas performance remains at a near-constant low. This makes sense, since Canvas is just a bitmap buffer, while SVG has to maintain additional references to each object that it renders. Also, though not pictured, note that performance in clearing an SVG element also decreases in the number of drawn objects.

Via Boris Smus

OpenStreetMap.org‘s RESTful API allows anyone to access data on their continually growing collaborative map of the world… in XML. This is great for most applications, but if you’re working in JavaScript (as we are), XML might as well be greek. We need JSON.

To offer OSM-JSON along with of OSM-XML, we added a route to accept a “.format” suffix, and split up the render call based on the params[:format] part of the route:

# /config/routes.rb:46-50

map.connect "api/#{API_VERSION}/geohash/:geohash.:format", :controller => 'api', :action => 'geohash'
map.connect "api/#{API_VERSION}/geohash/:geohash", :controller => 'api', :action => 'geohash'

map.connect "api/#{API_VERSION}/map.:format", :controller => 'api', :action => 'map'
map.connect "api/#{API_VERSION}/map", :controller => 'api', :action => 'map'

Notice we also added a ‘geohash’ route. Whereas the /map call requires a bbox parameter (‘bbox=min_lon,min_lat,max_lon,max_lat’), we can use a geohash (Geohash in JavaScript, Geohash in Rails) which defines a bounding box as a sequence of letters and numbers. This fits Cartagen’s needs well, and since it doesn’t require any parameters, we can page cache it in Rails. (Remember that page caching bypasses Rails entirely, letting Apache handle these cached files at high speed – that saved us when we were on BoingBoing).

Read the rest of this entry »

We know we just released Cartagen 0.5 a couple weeks ago, but after testing it extensively in the wild, we really wanted a fast, low-resource release so that users of netbooks, older computers, and older browsers could use Cartagen too.

So, as well as including some general cleanup, this version hums along on a variety of machines. Please send feedback on speed/load times/CPU usage, etc; we tested it on an 800mhz G4 iMac and while it wasn’t super responsive, it did load and was somewhat usable in Firefox 3.5. For reference, Hulu.com’s Flash player does not run on that machine. On more powerful machines (4x3ghz Intel Xeon, 2500×1600 monitor), it can load over 10,000 objects in the viewport without hiccuping. For most users, a typical browser window size yields a responsive and reasonably low-resource experience.

Download Cartagen 0.6.1 (We did 0.6 and just rushed directly into 0.6.1 before announcing.)

There’s definitely a lot more we can do to improve speed, but for now we feel good about our optimizations. We identified the high CPU usage as generally poor management of timers in JavaScript, and wrote a TimerManager class which adjusts its timer intervals automatically, easing the CPU load. We elected to maintain a relatively high CPU usage to keep things rendering smoothly, but users of TimerManager can ‘turn up the heat’ or turn down CPU usage by tweaking a ‘spacing’ parameter. We also broke more intense tasks up by creating a TaskManager class, which preserves UI responsiveness and framerate while staying in a single JavaScript thread. We plan to use the multithreaded, asynchronous Web Workers spec in HTML5 when available, but for now we wanted to work on older hardware/software.

We hope that TimerManager and TaskManager will be of use to others working with JavaScript animation and we’ll be packaging them up separately for download.

For now, look at http://code.google.org/p/cartagen for source and http://wiki.cartagen.org for update docs.

Today we’d like to announce the 0.5 release of the dynamic mapping framework Cartagen. In this release, we’ve taken a shot at rendering any location on the planet (drawing, of course, on data from OpenStreetMap), and at rendering a much greater density of map features with richer styles. The has come a long way, and we hope it makes designing your own map easier while allowing even greater possibilities for unique and compelling designs.

Download Cartagen 0.5

This is the first release we consider to be useable by the general public; as such we are releasing a video to demonstrate how to download Cartagen, get a data set, and write your own GSS stylesheet in just a few minutes:

More information and setup instructions are on Cartagen’s wiki, and our live demonstration site at cartagen.org has been updated.

Cartagen is still incomplete: it needs more optimization for slower browsers and devices like the iPhone and Android phones, it can make better use of HTML5 features such as Web Workers for multi-threaded JavaScript for a more responsive UI, and we hope to continue our early efforts to combine it usefully with OpenLayers and other mapping platforms.

Special thanks to Ben Weissmann who’s joined the Cartagen team and has been instrumental in bringing it this far.