Mike Dirolf

A _New Perspective_ on _MongoDB_

2011-03-28T00:00:00+00:00

I’ve been nominally using MongoDB since 2008, but most of my experience has been as a MongoDB developer, not a real end user. Over the past few months I’ve made the transition to other side, using MongoDB rather than developing it. The project I’ve been working on, fiesta, uses MongoDB exclusively for data storage. Here are some thoughts on MongoDB from my new perspective.

Flexibility

In the presentations I’ve done about MongoDB, I have always made a point to talk about how the document-oriented data model makes the database very flexible. I’d generally cite the example of how, when working with the folks at Business Insider, we’d seen the need to run any sort of migrations basically disappear. I’d explain how documents in a collection can have any “shape”, and how doing migrations was often as simple as adding an extra key when saving new documents. Sometimes people didn’t quite buy this. Now that I have some concrete examples from fiesta, maybe I can explain a little better.

When I first implemented fiesta, lists would always “munge” the Reply-To header when distributing a message. I like this behavior. Some others don’t. It pretty quickly became apparent that deciding either way would leave roughly 50 percent of my users unhappy, so reply-to munging became one of the (few) configurable bits of functionality in fiesta.

To implement this, all I needed to do was add a boolean "munge" field to the documents in the users collection. That key is set when a user chooses their reply-to preference. When sending a message to a user, we have to be careful to check if that key exists, though, since some users predate the munge setting:

if user.get("munge", True):
    do_munge(...)

That’s it! No migration.

Dynamic Queries

Another feature I always tout when presenting MongoDB is its dynamic query language. The claims:

Being able to create queries on the fly makes development really fast.
The document-based query language looks and feels totally natural in dynamic languages.
Dynamic queries make administration easier.

After using MongoDB in the real world for a while, I feel those claims have proven to be pretty accurate:

When I first started development on fiesta, the queries I was running changed probably every five minutes or so. Being able to quickly experiment with new queries (especially from the shell) kept the pace of development fast.
I write all of my queries directly against PyMongo, and it just feels natural. I have noted, however, that my memory of some of the $ operators has started to fade a bit. From time to time I’ll have to go check the reference – makes me wish I had held onto one of those quick reference cards for myself!
To be honest I can’t think of any administrative tasks that I’ve had to do that have been much easier with dynamic queries then they would’ve been with a few well-designed views. So this one might be a push – I’ll keep this updated if I think of any.

Replication

I stopped working full-time on MongoDB right around the time that replica sets went out in a production release. Even since then they have come a long way; at this point they really couldn’t be much easier to work with.

When I first deployed fiesta in beta, it was running on a single EC2 micro instance. I had set up nightly backups (using the fsync and lock command), but didn’t have replication set up. Needless to say, I was given a bit of a hard time when the 10gen gang heard that – hadn’t we (myself included) always said to never run without replication? Their ribbing was enough to convince me to take the time to set up replication, which took all of about an hour. And most of that time was spent playing around with puppet (time well spent, for the record). Now fiesta is running on a three-node replica set, with one passive node. The passive node is responsible for taking nightly backups.

Administration

Another thing I’ve often talked about is the MongoDB admin interface / web console. This has been really useful, especially the replica set UI. The one issue I’ve run into with it is documented here, but it’s far from a show stopper.

For administration I’ve found Elliot’s munin plugins to be even more useful than I expected. If you’re running MongoDB in production you should definitely check them out.

Performance

So far, so good. More updates here once fiesta hits its millionth user. Oh, by the way, you should give it a shot if you haven’t already :).

Create a _Mailing List_ With _fiesta.cc_

2011-03-09T00:00:00+00:00

</a>

Over the past few months I’ve been working on a project called fiesta.cc. I thought I’d do a quick post to explain what fiesta.cc is, and share an invite code so you can try it out!

The goal with fiesta.cc is to make it easy to communicate with groups using email. The current state of the art here seems to be a choice of:

Reply-all chains (Someone sends an email to a bunch of addresses. To send a message to the group others reply-all). These are great because creating a group is trivial. These are bad because:
- Group members can never unsubscribe, be removed, or change their email address, and new members can’t be added: someone ends up replying to an old message with the old set of addresses. This is really bad for groups like sports teams, which might be relatively consistent but have a few roster changes each season.
- Replying to a thread is easy, but sending a new message to the list is hard. You need to find an old message and reply to that, or retype the addresses. Either of those options is frustrating in general and pretty much unusable from a mobile device.
Client-side group expansion. Some mail clients allow users to create a list of addresses that can be aliased to a single group name. For everybody but the original sender, these have all the problems of a reply-all chain; the group isn’t shared to other list members.
Services like google/yahoo groups or posterous, or old school listservs (set up manually or through an IT department). These options solve all of the problems with (1) and (2) but they are much more work than something like (1).

An Easier Way

Here’s how to create a mailing list for your soccer team using fiesta.cc:

Send an email to the members of your list (just like you would in a reply-all chain).
CC soccer@fiesta.cc (or any other list name @fiesta.cc).
You’re done!

This is great because it’s as easy to set up as a reply-all chain (just add one more address), but it gives you all of the benefits of a real mailing list! When any list member emails soccer@fiesta.cc it’ll go to the whole list. People can unsubscribe or be added/removed at any time. They can also change their email address.

The really cool part is that you don’t need to reserve a group name or worry about conflicts with other groups: fiesta.cc ties group names to the sender of a message. That means that you can have a list called family@fiesta.cc which is totally separate from my family@fiesta.cc list. The site has more details about how to create and manage mailing lists with fiesta.cc.

Beta Invite

I think that fiesta.cc is a really useful service already, and the feedback from users so far has been very positive. I’d love for readers of this blog to check it out and send any feedback my way.

Click here for an invite.

The invite will let you see the homepage, just follow the steps there to create your first list and get an account. It’s simple enough that it makes sense to use even for small groups (like family@fiesta.cc or roommates@fiesta.cc) and temporary groups (like birthdayparty@fiesta.cc).

_MongoDB_ Contest _Winners_

2011-01-07T00:00:00+00:00

It has been about a week since I announced that I was giving away of copies of MongoDB: The Definitive Guide. So, it’s time to announce the winners!

I did a random drawing of the people who commented on the post (only including those who actually listed their favorite restaurant!). The winners are (drum roll):

Rob D (Robdoherty2)
Shehab Hamad
Sam Sen

If you’re one of the winners, please send me an email (address is in the header of this site) with your address and we’ll figure out how to get you your copy! If you didn’t win, don’t lose all hope: if any of the winners don’t pipe up within the next week, I’ll pick a new name. If all the prizes are claimed, or they aren’t but you still don’t win, send me an email and I’ll set you up with a nice discount on the book (contest participants only, please).

Thanks for playing!

_MongoDB_: The Definitive Guide _Giveaway_

2010-12-30T00:00:00+00:00

Updated 2011-1-7: The contest is closed and the winners have been announced.

I don’t think I’ve mentioned it here, but a few months ago Kristina and I finished writing MongoDB: The Definitive Guide. Writing the book was a big undertaking, and for a while it seemed like it would take a miracle just to finish it on time. Well, to make a long story short, the book is finished and it’s on the shelves.

I have a couple of extra copies of TDG that have been collecting dust in my apartment, and, after tripping over them for the past two months, I’ve decided to give them away! There are three copies up for grabs, each of which will be signed by me (and hopefully by Kristina too, if I see her soon enough). To enter the contest, leave a note in the comments with your name and the name of your favorite restaurant. In a week or so I’ll (randomly) select the winners and announce them in another post. I’ll pay for shipping in the US, but if you’re in a different country I’ll probably ask you to foot that bill. Good luck!

Connecting to a _Replica Set_ from _PyMongo_

2010-08-16T00:00:00+00:00

PyMongo makes working with “replica sets”:http://dochub.mongodb.org/core/rs easy. Here we’ll launch a new replica set and show how to handle both initialization and normal connections with PyMongo. This blog post will also be going up as an example in the “PyMongo docs”:http://api.mongodb.org/python/current/examples/index.html - check that page out for more examples of how to get things done with PyMongo.

Replica sets require server version = 1.6.0. Support for connecting to replica sets also requires PyMongo version = 1.8.0.

Starting a Replica Set

The main replica set documentation contains extensive information about setting up a new replica set or migrating an existing MongoDB setup, be sure to check that out. Here, we’ll just do the bare minimum to get a three node replica set setup locally.

Replica sets should always use multiple nodes in production - putting all set members on the same physical node is only recommended for testing and development.

We start three mongod processes, each on a different port and with a different dbpath, but all using the same replica set name “foo”. In the example we use the hostname “morton.local”, so replace that with your hostname when running:

$ hostname
morton.local
$ mongod --replSet foo/morton.local:27018,morton.local:27019 --rest

$ mongod --port 27018 --dbpath /data/db1 \
  --replSet foo/morton.local:27017 --rest

$ mongod --port 27019 --dbpath /data/db2 \
  --replSet foo/morton.local:27017 --rest

Initializing the Set

At this point all of our nodes are up and running, but the set has yet to be initialized. Until the set is initialized no node will become the primary, and things are essentially “offline”.

To initialize the set we need to connect to a single node and run the initiate command. Since we don’t have a primary yet, we’ll need to tell PyMongo that it’s okay to connect to a slave/secondary:

>>> from pymongo import Connection
>>> c = Connection("morton.local:27017", slave_okay=True)

We could have connected to any of the other nodes instead, but only the node we initiate from is allowed to contain any initial data.

After connecting, we run the initiate command to get things started (here we just use an implicit configuration, for more advanced configuration options see the replica set documentation):

>>> c.admin.command("replSetInitiate")
{u'info': u'Config now saved locally.  ...',
 u'info2': u'no configuration explicitly specified -- making one',
 u'ok': 1.0}

The three mongod servers we started earlier will now coordinate and come online as a replica set.

Connecting to a Replica Set

The initial connection as made above is a special case, for an uninitialized replica set. Normally we’ll want to connect differently. A connection to a replica set can be made using the normal Connection() constructor, specifying one or more members of the set. For example, any of the following will create a connection to the set we just created:

>>> Connection("morton.local")
>>> Connection("morton.local:27018")
>>> Connection("morton.local", 27019)
>>> Connection(["morton.local:27018", "morton.local:27019"])
>>> Connection("morton.local:27017,morton.local:27018,morton.local:27019")

The nodes passed to Connection() are called the seeds. As long as at least one of the seeds is online, the driver will be able to “discover” all of the nodes in the set and make a connection to the current primary.

Handling Failover

When a failover occurs, PyMongo will automatically attempt to find the new primary node and perform subsequent operations on that node. This can’t happen completely transparently, however. Here we’ll perform an example failover to illustrate how everything behaves. First, we’ll connect to the replica set and perform a couple of basic operations:

>>> db = Connection("morton.local").test
>>> db.test.save({"x": 1})
ObjectId('...')
>>> db.test.find_one()
{u'x': 1, u'_id': ObjectId('...')}

By checking the host and port, we can see that we’re connected to morton.local:27017, which is the current primary:

>>> db.connection.host
'morton.local'
>>> db.connection.port
27017

Now let’s bring down that node and see what happens when we run our query again:

>>> db.test.find_one()
Traceback (most recent call last):
pymongo.errors.AutoReconnect: ...

We get an AutoReconnect exception. This means that the driver was not able to connect to the old primary (which makes sense, as we killed the server), but that it will attempt to automatically reconnect on subsequent operations. When this exception is raised our application code needs to decide whether to retry the operation or to simply continue, accepting the fact that the operation might have failed.

On subsequent attempts to run the query we might continue to see this exception. Eventually, however, the replica set will failover and elect a new primary (this should take a couple of seconds in general). At that point the driver will connect to the new primary and the operation will succeed:

>>> db.test.find_one()
{u'x': 1, u'_id': ObjectId('...')}
>>> db.connection.host
u'morton.local'
>>> db.connection.port
27018

If you have any questions on working with Replica Sets and PyMongo, feel free to ask in the comments!

_Hackabit_ and _be bit.ly_

2010-07-15T00:00:00+00:00

Last night the folks at bit.ly hosted a hackathon, called hackabit, to celebrate their 2nd anniversary (congratulations guys!). It was awesome; there were a bunch of us there from 10gen, the bit.ly people are great, and it was fun just hanging out and working on stuff. There was a lot of coding being done, but I don’t have links to much of it (one cool project was a Python url normalization library by Jehiah). If anybody has links to other projects people were working on, let me know.

I wanted to do yet another quick JS game, and also wanted to do something bit.ly themed. The result is be bit.ly - a game where you play the role of the bit.ly website, by shortening incoming urls and turning shortened urls back into long ones. It sounds a bit strange, but I think it came out okay - hope you enjoy it! I know there are some rough edges, but time was short. Let me know what you think in the comments.

Edit: Chris also created a bit.ly inspired game at hackabit, Bitly Sea - it’s pretty fun, check it out!

_PyMongo_ _1.7_ Released

2010-06-17T00:00:00+00:00

A new release for “PyMongo”:http://api.mongodb.org/python has been long overdue – the last release (1.6) was made on May 11th, over two months ago! It’s been a busy couple of months with travel and “writing”:http://oreilly.com/catalog/9781449389536/ (and life), so a release hasn’t been on the top of the list. I finally made the effort to get a release out today, though, and the result is PyMongo 1.7. For a full list of changes, check the “changelog”:http://api.mongodb.org/python/1.7/changelog.html; in this post I’ll talk about a few of them in detail.

Using a `dict` to Specify Fields to Return

One of the smallest changes in 1.7 is that the fields argument to find/find\_one can now take a dict in addition to a list. So, the following two find\_ones are equivalent:

>>> db.test.save({"x": 1, "y": 2, "z": [1, 2, 3, 4]})
ObjectId('...')
>>> db.test.find_one(fields=["y"])
{u'y': 2, u'_id': ObjectId('...')}
>>> db.test.find_one(fields={"y": 1})
{u'y': 2, u'_id': ObjectId('...')}

The second call should look familiar to those who have used the MongoDB shell extensively, while the first is a little less verbose (and has always been the PyMongo way of doing things). Since PyMongo was first implemented, however, there have been some new features added to the server that are best exposed through the dict interface, like specifying keys that we don’t want returned:

>>> db.test.find_one(fields={"y": 0})
{u'x': 1, u'_id': ObjectId('...'), u'z': [1, 2, 3, 4]}
>>> db.test.find_one(fields={"z": 0})
{u'y': 2, u'x': 1, u'_id': ObjectId('...')}

Or using the new $slice operator to only return portions of an array:

>>> db.test.find_one(fields={"z": {"$slice": 2}})
{u'y': 2, u'x': 1, u'_id': ObjectId('...'), u'z': [1, 2]}
>>> db.test.find_one(fields={"z": {"$slice": -2}})
{u'y': 2, u'x': 1, u'_id': ObjectId('...'), u'z': [3, 4]}
>>> db.test.find_one(fields={"z": {"$slice": [1, 2]}})
{u'y': 2, u'x': 1, u'_id': ObjectId('...'), u'z': [2, 3]}

`datetime` Handling

PyMongo 1.7 also improves support for working with datetime instances. The first change is that timezone aware datetimes are now properly encoded, by converting them to UTC before saving them. Naive datetimes will still be assumed to be in UTC.

Since BSON is not currently capable of storing datetimes with timezone information, all datetimes will still decode as UTC. Currently the datetime instances returned will be naive, but that will probably change in the future as well.

A final change in 1.7 is that the PyMongo C extension now uses the y2038 project’s implementation of time.h. This allows the C extension to properly support dates beyond January 19, 2038. There might still be issues for users on 32-bit platforms using the pure-Python encoder, at least for older versions of Python.

max_scan

Version 1.7 adds support for the server’s new max\_scan functionality. max_scan allows a developer to set the maximum number of documents to scan for each individual query. This functionality can be useful when a developer wants guaranteed performance, even if it means a partial result set.

max\_scan requires a MongoDB server version = 1.5.1.

max_scan can be passed as an argument to find/find\_one, or can be added to a Cursor using chaining; the following two queries are equivalent:

db.test.find({"a": 1}, max_scan=50)

db.test.find({"a": 1}).max_scan(50)

Custom Classes for Returned Documents

The BSON decoder in PyMongo normally decodes all documents as dicts. Version 1.7 adds the ability to specify a custom class to decode to. All the class needs is to have a *setitem* method, which will be called for each key/value pair in the BSON being decoded.

This ability is exposed through the as_class argument to find/find\_one, and a new default class can be set using a Connection’s document_class attribute. The two queries below both decode to SON, so that the resultant documents maintain key order:

from pymongo.son import SON

db.test.find_one(as_class=SON)

connection.document_class=SON
db.test.find_one()

There are a couple of cool things that fall out of this functionality. First, third party libraries like MongoKit should have an easier time re-hydrating their models, and will probably be a little more efficient. Second, the class used doesn’t even need to represent a document: it can just treat *setitem* like a hook, allowing custom code to run on each key/value pair. Here, we create a custom class that just prints key/value pairs to stdout:

>>> class DocPrinter(object):
...     def __setitem__(self, key, value):
...         print "(%r, %r)" % (key, value)
... 
>>> db.test.save({"x": 1})
ObjectId('4c1a4ee9e6fb1b5adb000001')
>>> db.test.find_one(as_class=DocPrinter)
(u'_id', ObjectId('4c1a4ee9e6fb1b5adb000001'))
(u'x', 1)
<__main__.DocPrinter object at 0x1012d7bd0>

This example is a bit pointless, but I’m sure readers can think of much cooler things to do with it!

There’s a whole bunch more that’s new in PyMongo 1.7: check the changelog for the full list, and go upgrade!

Videos from _MongoNYC_

2010-06-02T00:00:00+00:00

MongoNYC happened in the end of May and was one of my favorite conferences of the past year. All of the talks I went to were very good, and I also had a chance to meet and talk with a ton of interesting people. The videos from the conference were just posted, and they can all be found here. The two talks I did are embedded below, but I’d also recommend checking out at least the talks by Kushal and Jay.

[**Mastering the MongoDB Shell**](http://blip.tv/file/3700829) </embed>

[**Inside PyMongo**](http://blip.tv/file/3701097) </embed>

If you have any questions about my talks or the conference in general, leave a note in the comments!

Inside _MongoDB_

2010-05-27T00:00:00+00:00

Updated 2010-10-29: There is now video of this talk (from when it was given at Mongo Boston) available here.

This year at Gluecon I was asked to speak about MongoDB, and specifically to give a “very technical” talk about it. I realized that this would be a good opportunity to do a talk other than the usual introduction to MongoDB and decided speak about MongoDB internals. Here’s the deck I used:

</embed>

If you have any questions or interest in hearing more about this (perhaps a series of blog posts on MongoDB internals?), let me know in the comments!

_Introduction_ to _MongoDB_

2010-04-09T00:00:00+00:00

I just finished giving a presentation introducing MongoDB at PhillyETE. Here’s the deck I used; I think video will be available soon and I’ll post that when it’s up:

</embed>

I think the presentation went well, and overall the conference was great. My favorite talk (besides the MongoDB one, of course) was one by Geir. His talk was ostensibly about Project Voldemort, but ended up being more about infrastructure at Gilt and the myriad approaches they’ve taken to tackling their unique traffic patterns - very entertaining stuff.

If you have any questions lingering from the presentation at PhillyETE, or about MongoDB in general, pipe up in the comments!

Working With _Stored_ _JavaScript_ in _MongoDB_

2010-04-05T00:00:00+00:00

Stored JS in MongoDB is saved in the special system.js collection. The documents there should be structured like:

{_id: "sum",
 value: function (x, y) { return x + y; }}

_id is the name of the function, and value is the JS code defining the function. Here’s an example of saving such a function and then using it in an eval. This can be run using the MongoDB shell:

> db.system.js.save({_id: "sum",
...                  value: function (x, y) { return x + y; }});
> db.eval("return sum(2, 3);");
5

Once sum is defined we can use it from all of MongoDB’s JS contexts, including Map/Reduce and $where. In this example we use sum within a $where clause to get all documents where the sum of x and y is 6:

> db.test.save({x: 4, y: 2});
> db.test.save({x: 4, y: 3});
> db.test.save({x: 3, y: 3});
> db.test.find({$where: "sum(this.x, this.y) == 6"});
{ "_id" : ObjectId("4bba376231d25858659843f7"), "x" : 4, "y" : 2 }
{ "_id" : ObjectId("4bba376e31d25858659843f9"), "x" : 3, "y" : 3 }

Since system.js is a collection, we can perform normal MongoDB operations on it in order to administer our stored JS functions. We might want to list the stored functions:

> db.system.js.distinct("_id");
[ "sum" ]

or remove a function we previously stored:

> db.system.js.remove({_id: "sum"});
> db.eval("return sum(2, 3);");
Mon Apr  5 15:14:45 JS Error: uncaught exception: {
    "errno" : -3,
    "errmsg" : "[...] sum is not defined [...]",
    "ok" : 0
}

Stored JS and PyMongo

In version 1.5 PyMongo added some helpers to make working with stored JS incredibly easy. The API is accessed through the Database.system\_js property, which returns an instance of the SystemJS helper class. The SystemJS class allows us to add, call, and remove stored JS functions using a nice Pythonic interface:

>>> db.system_js.sum = "function (x, y) { return x + y; }"
>>> db.system_js.sum(3, 2)
5.0
>>> del db.system_js.sum
>>> db.system_js.sum(3, 2)
Traceback (most recent call last):
pymongo.errors.OperationFailure: [...] sum is not defined [...]

The implementation of the SystemJS class is actually pretty trivial. The trickiest bit is getting the argument passing right between the Python lambda returned by *getattr* and the invocation of the stored JS function within the eval.

_Gluecon_ Early Bird _Discount_

2010-03-30T00:00:00+00:00

I’ll be speaking at “Gluecon”:http://www.gluecon.com/2010/ in the end of May. The early bird pricing for registration ends on April 2nd, and you can get an additional 10% off of early bird registration as a reader of this blog if you use the code “spkr12”!

The conference looks to have a great lineup — Dwight will be speaking as well so there will be a couple of members of the 10gen team in attendance. If you’re going to be at Gluecon let me know in the comments. I look forward to seeing you there!

A _New_ _GridFS_ Implementation for _PyMongo_

2010-03-29T00:00:00+00:00

The API (Application Programming Interface) for PyMongo, the Python driver for MongoDB, has been pretty stable for quite some time now. In general, I think that the driver does a great job of exposing all of MongoDB’s functionality in a way that is both MongoDB-ish and Pythonic (of course I’m biased — if you have suggestions for improvements please let me know). There have been some minor tweaks recently, like an improvement for the command API, but there haven’t been any huge changes in a while.

All that said, I think there is one place where PyMongo has been deficient for a long time: its support for GridFS. There are a couple of problems with the GridFS implementation in previous versions (= 1.5.2) of PyMongo:

It is slower and less concurrency-friendly than it needs to be.
It could be much simpler and easier to work with.
It allows some operations (modifying existing files) that are incorrect according to the GridFS semantics.

I think that all of these deficiencies stem from one fatal flaw in the original API design: it was trying too hard to mimic Python’s filesystem API. Exposing file-like objects for writing and reading is great, but things like focusing on filename as a file-handle (filename is less important in GridFS) and allowing users to modify files (which is not a concept supported by GridFS) were bad decisions. This post introduces a new GridFS API for PyMongo which tries to address all of the deficiencies in the old implementation.

The New API

The new GridFS API is available in PyMongo versions = 1.6. There are API docs available for the new version of GridFS, but this post walks through the API in a little more detail.

Almost all of the API is exposed through the GridFS class, here’s an example instantiation:

>>> from pymongo import Connection
>>> from gridfs import GridFS
>>> db = Connection().test_database
>>> fs = GridFS(db)

You can also use an alternate root collection for GridFS by passing a collection name as the second argument to GridFS. Given an instance of GridFS, creating new files and getting data from existing ones is easy:

>>> file_id = fs.put("hello world")
>>> fs.get(file_id).read()
'hello world'

The put method takes a string or file-like object containing data to be written to a GridFS file, and returns the \_id of the newly created file. It also accepts keyword arguments for any of the fields available in the GridFS file spec. So to insert the contents of the local file myimage.jpg with the content type “image/jpeg” and the filename “myimage” we would do:

>>> with open("myimage.jpg") as myimage:
...   oid = fs.put(myimage, content_type="image/jpeg", filename="myimage")
...

The equivalent of doing this using the old API would be:

>>> from pymongo.objectid import ObjectId
>>> with open("myimage.jpg") as myimage:
...   with fs.open({"filename": "myimage",
...                 "contentType": "image/jpeg",
...                 "_id": ObjectId()}, "w") as grid_file:
...     grid_file.write(myimage.read())
...     oid = grid_file._id
...

The get method takes an \_id of a file in GridFS, and returns a file-like object (an instance of gridfs.grid\_file.GridOut) that can be used to read that file’s data. If there is no file with the given \_id an exception is raised:

>>> oid = fs.put("hello world")
>>> fs.get(oid)
<gridfs.grid_file.GridOut object at ...>
>>> fs.get("non-existant _id")
Traceback (most recent call last):
gridfs.errors.NoFile: ...

To delete a file in GridFS, pass its \_id to the delete method:

>>> fs.get(oid)
<gridfs.grid_file.GridOut object at ...>
>>> fs.delete(oid)
>>> fs.get(oid)
Traceback (most recent call last):
gridfs.errors.NoFile: ...

Advanced Usage

While put should cover most use cases that opening a file in write mode used to, there may be some cases where you don’t want to write all of the file’s data at once. In that case you can use the new\_file method to get a new gridfs.grid\_file.GridIn instance which you can write to. When you’re done with the instance call close (or just use Python 2.6’s with statement to handle it for you). Like put, new\_file takes keyword arguments for any of the values in the GridFS file spec. A cool note about both methods is that any unrecognized keyword arguments (in this example, location) will automatically be set as attributes on the underlying file document:

>>> myfile = fs.new_file(location=[-74, 40.74])
>>> myfile.write("hello ")
>>> myfile.write("world")
>>> myfile.writelines([" and have a ", "good day!"])
>>> myfile.close()
>>> out = fs.get(myfile._id)
>>> out.read()
'hello world and have a good day!'
>>> out.location
[-74, 40.740000000000002]

Most of the examples above have been referencing files in GridFS by their \_id. The old API made more of a point of emphasizing filename as the primary mode of access. While the new API encourages the better practice of referencing by \_id, there are still some cases where you might need to work with files based on filename, or where your application treats filename as a unique identifier.

The way to work with filenames in the new API is to create a new GridFS file with the same filename every time you need to modify a file. Since files in GridFS store their upload date, we can always get the most recent version of a file by filename. We can also reference by \_id to get any previous version — we can treat GridFS as a versioned filestore. The method to get the last version of a file by name is get\_last\_version:

>>> a = fs.put("foo", filename="test")
>>> fs.get_last_version("test").read()
'foo'
>>> b = fs.put("bar", filename="test")
>>> fs.get_last_version("test").read()
'bar'
>>> fs.delete(b)
>>> fs.get_last_version("test").read()
'foo'

Performance

I’ve only done some very basic benchmarking of the new GridFS implementation. It doesn’t appear to be a drastic improvement when writing large files, but writing small files is about four times faster in a simple benchmark of mine. This is because the simplification of the API has reduced the per-file overhead dramatically. There are also huge performance improvements when uploading a large file from disk or some other file-like source (especially if it was being done naively using the old API), as the new API automatically handles streaming from a file-like source into chunk-sized buffers.

Reading small files is about 10% faster with the new API, while there doesn’t seem to be much difference with reading large files. A big difference in performance for both reads and writes is that concurrency is vastly improved. Since the GridFS semantics are handled correctly, there is no longer a per-file lock for access - concurrent operations are fully supported and safe now (with the notable exception of deleting a file, which could cause concurrent readers to see partial data).

If anybody has any ideas for benchmarks, or existing benchmarks for the old API, I’d love to see how they compare — please leave a note in the comments.

Deprecation of the Old API

In the current state of the master branch I have removed the old GridFS API completely, raising an UnsupportedAPI acception when attempts are made to use it. Normally I would go through a deprecation window, but I’m considering just releasing like this. My reasoning is that it’s a big change and I want people to be making it as soon as possible. There is also a problem that mixed use of the APIs could be unsafe (in general, any use of the old API can be unsafe, if it’s used to overwrite existing files). Let me know what your thoughts on this decision are as well. Hoping to get this right as I think it will be a big improvement to PyMongo!

_pyporter2_

2008-09-22T00:00:00+00:00

This “post” is actually just the pyporter2 homepage, reformatted as a blog post. This is mainly so that the link doesn’t disappear on anybody.

about

This is an implementation of the “Porter2 (english) stemming algorithm”:http://snowball.tartarus.org/algorithms/english/stemmer.html in Python. It was born out of some academic work I did on clustering algorithms in the spring of 2008. The “Porter Stemming Algorithm”:http://tartarus.org/~martin/PorterStemmer/ was first published in “this”:http://tartarus.org/~martin/PorterStemmer/def.txt 1979 paper - it is now one of the most widely known and used “stemming”:http://en.wikipedia.org/wiki/Stemming algorithms. An “implementation”:http://tartarus.org/~martin/PorterStemmer/python.txt of the Porter stemmer already existed in Python, but not of the updated Porter2 stemmer. I decided to implement a Python version of Porter2 as an exercise.

Python bindings for the official C version of the Porter2 stemmer exist here. If using these bindings is an option, it will probably be much more efficient than using the pure Python implementation here. pyporter2 is useful when the C bindings are not an option (like in Jython, IronPython, Babble or App Engine).

download

pyporter2 is open source software, released under an MIT-style license. The latest version of pyporter2 is available here. To check out the source, install Git and run:

$ git clone git://github.com/mdirolf/pyporter2.git

usage

The new API (Application Programming Interface) matches that of PyStemmer. Here is an example of how to use pyporter2:

>>> import Stemmer
>>> print Stemmer.algorithms()
['english']
>>> stemmer = Stemmer.Stemmer('english')
>>> print stemmer.stemWord('cycling')
cycl
>>> print stemmer.stemWords(['cycling', 'cyclist'])
['cycl', 'cyclist']
>>> print stemmer.stemWords(['cycling', u'cyclist'])
['cycl', u'cyclist']

testing

pyporter2 includes a test suite written using unittest. To run the tests, do:

$ python Stemmer.py

questions

Feel free to leave a comment with any questions. It’d also be cool to let me know if you find pyporter2 useful for anything.

_Zoomix_ -- Another _JavaScript Game_

2007-11-30T00:00:00+00:00

Updated 2009-12-13: Adding the actual game itself to this page, mainly so that Aaron has a place to play.

GAME OVER - Press Spacebar to Restart Score: 0

)

Yesterday I wrote another game using JavaScript and the canvas tag, which I’ve named Zoomix. This time the design of the game is completely original, and I think it came out pretty nicely. Hopefully you should be able to figure out the rules / object of the game after playing with it for a few minutes. Let me know what you think! (And post your high scores in the comments).

The look and feel of the game was inspired by the Rubik’s Cube. After creating a static image of a face of the cube, zooming in on it seemed to be the only logical thing to do. After that was implemented, the mechanics of gameplay just fell out.

The game is open-source (but those of you who care probably viewed the source already).

Enabling _Cross-Domain AJAX_ in _Firefox_

2007-06-11T00:00:00+00:00

Updated 2009-12-13: This post seems to be the most popular of those from the old incarnation of this blog. As such I’ve decided to migrate it over. Unfortunately I can’t migrate the comments as well. One comment which seems that it might be useful comes from [stefano](http://myblog.digitalemagine.com/) (I’ve editorialized somewhat): > Good trick, but there’s an easier way to enable cross domain without > editing config files by hand. Type “about:config” in your URL bar, > then right click on the list of preferences and select the *New* -> > *String* contextual menu. Add > **capability.policy.default.XMLHttpRequest.open** as key name and > **allAccess** as value. I do not remember if this is taken into > account immediately or if you have to reboot. Done! I’m also not positive if this works in newer versions of Firefox, and I haven’t tested it. Feel free to let everybody know in the comments.

Tonight I have finally conquered one of the biggest annoyances of the past year for me (in terms of development at least). Developing web applications with Firefox is a pleasure because of the “firebug extension”:http://getfirebug.com/. Nothing comes close in Safari. Unfortunately, Firefox doesn’t allow cross-domain XMLHttpRequests for security reasons. While good security is a plus, this restriction can make development and testing a real chore. For those of us willing to risk the security vulnerability, here is how to bypass the cross-domain restriction once and for all.

Close Firefox \#

Edit the file **prefs.js** in your Firefox user profile folder \#

Add the following line anywhere in the file:

user_pref("capability.policy.default.XMLHttpRequest.open",
          "allAccess");

Save the file and re-open Firefox. You can now risk your life and limb by doing XHR (XMLHttpRequests)’s to whatever domains you want - congratulations!

_MDCave_ -- A _JavaScript Game_

2007-06-07T00:00:00+00:00

Updated 2009-12-13: Adding the actual game itself to this page, mainly so that Aaron has a place to play. Also changing the title to MDCave — A JavaScript Game from A Fun Game.

GAME OVER - Press Spacebar to Restart Score: 0

)

In lieu of a real post here’s a time-waster for those I haven’t showed already. This is a game I wrote to procrastinate about a month or two ago, meant to be a larger version of “this game”:http://www.sfcave.com/javaCave.php. It should work in Safari and Firefox. Also, cut me some slack for the lack of polish, as it really was a one night affair. The whole thing is written using Apple’s canvas tag and JavaScript, feel free to use the “source”:http://github.com/mdirolf/mdcave for whatever you want. Post your high scores in the comments.

Mike Dirolf

A _New Perspective_ on _MongoDB_

Flexibility

Dynamic Queries

Replication

Administration

Performance

Create a _Mailing List_ With _fiesta.cc_

An Easier Way

Beta Invite

Click here for an invite.

_MongoDB_ Contest _Winners_

_MongoDB_: The Definitive Guide _Giveaway_

Connecting to a _Replica Set_ from _PyMongo_

Starting a Replica Set

Initializing the Set

Connecting to a Replica Set

Handling Failover

_Hackabit_ and _be bit.ly_

_PyMongo_ _1.7_ Released

Using a dict to Specify Fields to Return

datetime Handling

max_scan

Custom Classes for Returned Documents

Videos from _MongoNYC_

Inside _MongoDB_

_Introduction_ to _MongoDB_

Working With _Stored_ _JavaScript_ in _MongoDB_

Stored JS and PyMongo

_Gluecon_ Early Bird _Discount_

A _New_ _GridFS_ Implementation for _PyMongo_

The New API

Advanced Usage

Performance

Deprecation of the Old API

_pyporter2_

about

download

usage

testing

questions

_Zoomix_ -- Another _JavaScript Game_

Enabling _Cross-Domain AJAX_ in _Firefox_

_MDCave_ -- A _JavaScript Game_

Using a `dict` to Specify Fields to Return

`datetime` Handling