Making today worse so tomorrow seems better.
Estimating Key Collisions (Birthday Problem)
Thanks to a somewhat well known statistical behavior known as the Birthday Problem, collisions of randomly generated keys are much more likely than you would expect. You can estimate how likely with some quick Ruby.
I ran into real world application of the Birthday Problem while I was building the mBlox adapter for Vocel’s SMS product. Most APIs for sending SMS give you back a randomly generated tracking ID when you submit a message. This tracking ID can then be used to report the delivery status of the message.
The mBlox API doesn’t automatically return a tracking ID, but it allows you to set one of your own. Could we get away with just randomly generating our own without checking to see if we had used the ID yet?
After some research online, I built this little method to determine the chance of having at least one collision in a set of randomly distributed keys ‘k’ out of a range of ‘n’ possible values. While calculating the exact probability for large numbers isn’t easy, it turns out estimating it is.
The real problem with the mBlox API is that the maximum value for their “BatchID” parameter is 99,999,999. That would make a big salary, but it’s not a big keyspace. For the expected trial run of 70,000 messages, the chance of at least one collision is over 99.999%.
Clickatell (another SMS aggregator), let’s you specify a 32-character string as the tracking ID. Using that for 16-bytes of hex, that is a big enough space that the collision rate is effectively 0 even with over 100 Billion messages.
So the random ID plan didn’t work for mBlox, although we could use it for Clickatell. In the end we used a MySQL DB table with an autoincrement key as the tracking ID for mBlox messages. This avoids dupe-checking and is synchronized on all the machines in the cluster.
- About
- Technology, programming, the interwebs and other topics by members of the slackworks community.
- Contributors
- All Posts
-
- Aug 05 2009 Using New Relic add_method_tracer with Class methods
- Jun 30 2009 Estimating Key Collisions (Birthday Problem)
- Jun 29 2009 The Blarg is Back!
- Jul 23 2008 JNDI with Rails
- Jul 06 2008 JRuby Service in Rails
- May 16 2008 Wildcard DNS and Rails
- Jan 30 2008 Sprint Error "911" and the "Download Domain"
- Jan 16 2008 Getting a Good View of Your Couch
- Jan 12 2008 Bring Selenium to the integration party
- Jan 12 2008 Bj Makes Attachment_fu Happy
- Jan 03 2008 Restore Finder magic to a sparsebundle directory
- Dec 18 2007 First Post
0 comments