Sep 122013

I have come across this problem quite recently. I usually sort BsonDocuments server side, by providing order by parameters to queries, but then there was a situation where I had BsonDocuments in an array that arrived as a parameter, and I couldn’t guarantee that they were sorted by the key I needed them to be. I could only guarantee that they had that key and they also might have had duplicate such keys, which would be valid for my scenario.

Because I would allow duplicate sorting keys, I had to exclude the default .Net SortedList and SortedDictionary implementations, because they don’t allow that. So I came with two valid solutions: one using the Sort method of the List class and the other using OrderBy extension of the IList interface. I chose the second simply because I like methods that apply to interfaces rather than classes only. Why didn’t they add the Sort method to the IList interface?! Having a method be applied only to a class whilst it implements such an obvious-named interface (IList => List) gives me a sense of implementation specific, changed quite often over time, and I don’t want that idea. Well, I’ve included that line, too, just in case :) I know this might seem a superficial decision, but as long as I’m keeping the array small (say a maximum of 100 elements), I’m not worried for performance hits, which could be one of the most important differences between the two versions. And, by the way, the timing was better when using the OrderBy extension with 100 elements in the list.

Let’s see an example:

public class BsonDocumentSortedListTest {
public void test() {
IList<BsonDocument> list = new List<BsonDocument>();
Random random = new Random();
for (int i = 0; i < 100; i++) {
list.Add(new BsonDocument("so", random.Next(100)));
DateTime startTime = DateTime.Now;
BsonDocumentComparer<BsonValue> documentComparer = new BsonDocumentComparer<BsonValue>("so");
foreach (BsonDocument value in list) {
Console.WriteLine("Time for list: " + DateTime.Now.Subtract(startTime).TotalMilliseconds.ToString("#,##0.00"));
startTime = DateTime.Now;
IOrderedEnumerable<BsonDocument> orderby = list.OrderBy(documentComparer.KeySelector, documentComparer);
foreach (BsonDocument value in orderby) {
Console.WriteLine("Time for orderby: " + DateTime.Now.Subtract(startTime).TotalMilliseconds.ToString("#,##0.00"));
public class BsonDocumentComparer<T> : IComparer<T> where T : BsonValue {
private string _keyName { get; set; }
public BsonDocumentComparer(string keyName) {
_keyName = keyName;
public int Compare(T doc1, T doc2) {
int result = -1;
if (doc1.IsBsonDocument && doc2.IsBsonDocument && doc1.AsBsonDocument.Contains(_keyName) && doc2.AsBsonDocument.Contains(_keyName)) {
result = doc1[_keyName].CompareTo(doc2[_keyName]);
} else {
result = doc1.CompareTo(doc2);
return result;
public T KeySelector(T doc) {
return (T)(doc.IsBsonDocument && doc.AsBsonDocument.Contains(_keyName) ? doc[_keyName] : 0);

Both methods used for comparing in those two cases check for the presence of the key in the document, just in case one of them is missing it. And I used the generic version so that I don’t have to code two Comparer-s for the two versions (Sort and OrderBy).

Using the Sort method, the documents that don’t contain the sorting key will be the first in the resulting list (just return 1 to be the last). Using the OrderBy method, the documents that don’t contain the sorting key will be returned first (well, if the keys are equal or greater than zero; if you want to be certain, than return some big negative number instead of zero); if you want to be returned last, just return a big positive number instead of zero.

This BsonDocumentComparer can be used with non-numeric keys, too, as long as they can be wrapped as BsonValue-s, so they could happily expose a CompareTo method.

Although this kind of sorting should hardly be needed in normal operations, because the best sorting is that used in the query itself, server-side, this might help in some situations, where it’s about more than just ordering, selecting and displaying.

Later UPDATE: If the documents had only one key, as in this example, we could use directly the BsonDocument comparer instead of our own. But in most cases, the documents we handle are more complex, so we need a specific key in it to sort by.

Sep 072013

In our times, when the consumer applications run on devices out there, having an API to consume is the best approach instead of using persistent connections to a database and copying the logic to each and every application instance. We probably all agree that we’re deploying smaller and smaller applications, from the business logic point of view. They use the resources to be more user-friendly rather than having more business logic; from the latter point of view, they rely on an external service which stores, processes and returns the data ready to be displayed.

Also, instead of us developing an application that would use that data, we might let / invite others to conceive their own application, so that their customized purpose is fulfilled; thus, we hardly know the specific client that uses the exposed methods and data. We’ve separated the concerns between who stores and processes the data and who displays it, without necessarily being in the same team. Or even in the same organization.

So here comes the API. As any interface, it forces the implementer to use each method with certain parameters of certain types. It creates a strong and explicit binding between the interface and it’s implementers. But, unlike the traditional OOP case when a class implements an interface, the API client is concerned to follow the API rules only at method level. In the first case, the class implementing an interface needs to implement all its methods, whilst the API is interested only in the current request. It’s like a late-bound, runtime relationship. Each request is a distinct entity, the API makes no assumptions (based, for example, on an http session).

The question at hand is: how can we accomplish our goals, i.e. serve a myriad of clients, which integrate in various businesses? A lot of RESTful APIs, which act on the data with CRUD operations, have most of their exposed methods (let’s admit that they’re mostly Read effectively), have this question answered by default: they don’t need more than they already have. They are so generic that they don’t need anything special.

The more specialized the API, the more behavioural it is. That is, the client not only needs to follow certain rules when making a (i.e. one) request, it most probably needs to make more consecutive requests and must behave in a certain way during that process. Also, the API itself must behave in a certain way during such conversations. We’ll find ourselves in situations where we must define rules that are cross-requests, not only request-specific.

Let’s take an example of an on-line store that has an underlying API which serves the on-line portal, the mobile application and the internal accounting application. Also, the API is ready to serve requests to third party businesses which want to reward their clients products from the store that meet certain criteria. This kind of architecture can take various shapes, so we won’t dig into much detail; the main idea is that there are certain flows that require API clients to follow rules that would not make much sense if they were request specific: an authentication token, products and shopping cart identifiers and so son. There are rules around these concepts that I would call cross-requests rules or, my preferred term, behavioural rules. That’s because these rules determine a certain behaviour of the clients throughout their journey with the API; those request-specific rules combine themselves to serve a cross-request purpose through a behaviour.

Why would we care about defining such a concept? As any behaviour (in its common sense), it is educated. The API is not only a server, it educates its clients behaviour. Those simple CRUD, RESTful, APIs are just point in time acquaintances in the clients’ business logic, whilst these APIs that expose behaviour contain themselves business logic that the clients must be aware of, must be educated to comply to. In this latter case, the responsibility of the main business logic resides server-side.

Being aware of this perspective will have implications in both parties, separating the server-business layer from the client-business/presentation layer; we’re having here two distinct projects, project management plans and so on so forth. If the client-business needs a certain behaviour, it is the main-business’ responsibility to integrate (or not) the new business request and to expose the behaviour that it requires, which might be more or less different from the initial client-business’ request-for-work. This is because the API-business rules are the ones to educate their clients behaviour, even if the so called “request-for-work”  originated form the client-business.

A great talk (although an old one) about creating succesful APIs you can find here: How to Design a Good API and Why It Matters

Aug 212013


Don’t we all miss the procedure level atomicity? Having more than one year of programming upon MongoDB, I have already learned to live without transactions, although there were (and still are) times when I would really use one. The dual step commit advice is also not always possible and, besides, it’s not 100% guaranteed either. I would really want try { run(); } catch(Exception ex) { rollback(); } finally { happilyEverAfter(); }

The challenge

Right, so how can we programatically implement a transactional system that would also be abstract enough to be re-used but would also need little specific implementation for each case? As always, divide and conquer:

  1. A transaction has multiple transaction-able (i.e. rollback-able) items. Each item has three parts / paths: one that runs, one that commits and one that does the rollback. This transaction-able item is the only one who knows what and how to rollback; it’s deciding whether it can run, thus rollback, multiple database operations or only one. If a transaction-able item runs multiple database operations, then it’s his duty to know what to rollback (depending on what operation has run and what not yet at the rollback moment) and how to rollback (what documents to change, what keys in those documents to change). So here, we’re having an atomic object from the operational point of view. We’ve already spotted three methods for it: run(), commit() and rollback(). So this transaction-able item / atomic object has already an interface that we can attach to it. Note that it’s not the item that decides when to rollback. It only has methods ready to be called in those three moments of a transaction: run(), commit() and rollback(). And also it has logic to determine what and how to rollback, but, again, not when. The run() method can throw an exception, but it’s not it’s duty to call the rollback() method, because the item can be part of a multi-item transaction, so we need an external decider, thus we’re letting the exception propagate to the caller. Also, it’s the item’s duty to not make the items visible to other operations until the commit() method has finished executing.
  2. A transaction is an object that manages the transaction items. It allows the calling code to add items, it runs them, intercept any exception thrown by those items, calls the commit() or rollback()  methods, respectively. It should also remember what exception has been thrown and what item has thrown that exception, so that the calling code can check and make decisions upon that. A transaction (and also a transaction item) has a status, too: notStarted, started, committed, cancelled, ended, error, cancelling, errorCancelling.
  3. We might also need a transaction factory, which will serve us transactions based on items that we’re creating and also will run those transactions, calling their commit or rollback methods, respectively.

These steps hopefully seem quite straightforward. But, the real challenge arises: how to commit() the items in a transaction atomicly? If we have a single-item transaction and that single item has only one operation (usually write operation) to make, then … well, we might not even need transactions, a try-catch block would be enough.

The proposal

What we need here is a write lock managed by the MongoDB server. Depending on the transaction items involved, we might need a collection level write lock or a database level write lock. The idea is to initiate those locks explicitly from the user (probably with some timeout, to avoid infinite locks due to application bugs).

Without this kind of lock, we could still have intermediate reads while commiting transaction items. I know that this kind of lock is almost impossible in a sharded cluster (well, I said almost), but it might open great ways towards implementing transactions, even natively at the server level.

The solution

Since were talking about a lock, there might be a solution we can think of. That is, to implement a lock (like lock / synchronized keywords in C# / Java, respectively) at the application level. The requirements of such kind of lock are:

  • it has to apply only to writes
  • it must refer only to certain collections or databases
  • multiple such locks can coexist, as long as they don’t intersect (they don’t have collections neither databases in common)

This logic leads us to the idea of having two types of locks: collection locks and database locks. Of course, a database lock will conflict with any collection lock that refers to a collection in that database, so the intersection rule applies. Also, another idea is that those locks will be in fact arrays of collection and database names, respectively. They are application / domain scoped, thus static.

Each item of the transaction must expose the type of locks it requires (collection and/or database) and the names of those collections and/or databases, respectively. The transaction will have an initial loop through the items, collecting the locks arrays, then it will try to acquire those locks (waiting for them to be released by other transactions, if any) then execute its items properly, rolling them back as necessary and finally releasing the locks.

Doesn’t this sound like a feasible solution?

And some code, please!

Of course, let’s see how would this look from a C# perspective (Java much alike).

Define the statuses enums somewhere:

public enum MongoTransactionStatusEnum {
public enum MongoTransactionItemStatusEnum {

The transactionable item interface:

interface IMongoTransactionable {
bool run();
bool rollback();
string name { get; set; }
int order { get; set; }
MongoTransactionItemStatusEnum status { get; }
string[] lockableCollections { get; }
string[] lockableDatabases { get; }

The Mongo Transaction interface:

public interface IMongoTransaction {
string id { get; }
MongoTransactionStatusEnum status { get; }
bool start();
bool commit();
bool rollback();
IMongoTransactionable[] items { get; }
bool addItem(IMongoTransactionable item);
Exception exception { get; }
IMongoTransactionable exceptionItem { get; }
string[] lockableCollections { get; }
string[] lockableDatabases { get; }
void releaseLocks();

The method to handle a transaction from a transaction factory:

public void handle(IMongoTransaction transaction) {
try {
transaction.start(); // start will acquire the necessary locks
} catch {
} finally {

Acquiring and releasing the locks from within the transaction are in fact wrappers for two static methods that will effectively do those tasks:

static object collectionLockObject = new object(), databaseLockObject = new object();
static IDictionary<string, IList<string>> collectionsLocks = new Dictionary<string, IList<string>>();
static IList<string> databasesLocks = new List<string>();
static void acquireCollectionLock(string databaseName, string collectionName) {
while(collectionsLocks.Keys.Contains(databaseName) && collectionsLocks[databaseName].Contains(collectionName)){
collectionsLocks.Add(databaseName, new List());
static void acquireDatabaseLock(string databaseName){
lock(databaseLockObject) {
static void releaseCollectionLock(string databaseName, string collectionName){
if(collectionsLocks.Keys.Contains(databaseName) && collectionsLocks[databaseName].Contains(collectionName)){
static void releaseDatabaseLock(string databaseName){

This is not an easy implementation to do and to test, but seems like a good plan to me. Will probably post more on this while I’m testing this approach. Meanwhile, waiting for your input, too.


Aug 192013

The challenge

One of the most interesting challenges that we face while running our MongoDB based application is that of having no downtime due to primary changes in the replica set. During that 15-40 seconds of no-primary-window we need to still be responsive, to still capture the writes. In fact, capturing the writes is the big challenge; the reads can either be postponed / ignored (if they are set to read from the primary only), but the writes.. I can hardly think of an application or two who can afford losing the writes. Not even the so-called least important ones.

So what do we do? I came up with a solution that seems to work for quite a while now: delay the writes asynchronously.

What does that mean?

That means that we are capturing the writes, storing them in programmable objects in memory, then release the application flow as if we have executed them. Obviously, no reads can reflect those writes until they will be effectively written, but we won’t lose them either. We need to be able to live without those writes for the time that the primary change occurs (say 40 seconds) plus the time that will take for them to be written (probably another minute or so after the new primary has been acknowledged). This be-able-to-live condition is the first downside of the scenario.

What do we need to accomplish that?

a) A synchronized queue mechanism which we can add writes to. So, each write becomes an object by itself, which we add to the queue.

b) A synchronous way of checking for the availability of the primary, of retrieving the writes from the queue and executing them. Obviously, the writes must be executed in the order they arrived, otherwise we’ll mess up the business logic.

The second challenge is a quite classic programming challenge – having a synchronized queue of objects and a manager that takes the objects on the other end of the queue one by one and calls a method on each of them, so I won’t be stressing on that now.

The next challenge is to encapsulate a write operation into a programmable object. Following the natural flow of things, I have imagined:

– a base write object (which will be common to all types of writes)

– naturally inherited objects, which will contain specifics for each type of write (insert, update, remove)

For each write operation, we have to keep:

– the query part of the write (an IMongoQuery object), which selects the document(s) to be written against (for update and remove)

– the update part of the write (an IMongoUpdate object), which specifies the fields to be set, pushed into, etc. and their corresponding values.

– any other update parameters (UpdateFlags, UpdateOptions, InsertOptions, RemoveFlags, etc).

– the document to insert

We might think of other useful information to keep together with the object. The main thing is that we have an object that is subject to execute a write operation against the database, once it has been asked so by a queue-management mechanism. This mechanism is to decide when the primary is available and when is the time to start unqueue-ing the write objects.

The last challenge will be for the queue to still accept writes to be added while the manager is taking objects from it, so the application will act as if the primary is still absent while the queue is not empty.  When the queue is emptied by the manager, the application takes control again and the writes are sent as usual.

The main downside

The drawback is that we might need a lot of memory for those objects to live in during the primary transition. And I cannot stress enough the importance of garbage collection (for the programming languages that this makes sense for) while and after the queue is emptied: we need to make sure that our objects are efficiently garbage collected, so no reference kept to those objects after the processing is done. Once we add them to the queue, only the queue should reference the write objects.

What about the implementation details?

I’ll let you imagine the proper implementation according to your specific needs. If the scenario seems at least good, then the implementation should be fun to create.


Aug 112013

We might be tempted to use an index for each type of query we use. But creating more and more indices will lead to a bad write performance, so we always ask ourselves whether we really need to query this way. A great solution is to create our queries in such a way that they make heavy use of the compound indices. For example, if we often search for a product in a certain category, we will have a compound index for category and product name. But if we also often search for products in any category, then we might rethink our index: what if we create a compound index for product name and category instead? The order of the fields in the compound index does matter.

An index like

{ name: 1, category : 1 }

will allow us to search either for name only or for name and category, whilst

{ category : 1, name : 1 }

will allow us to search either for category only or for category and name, in this order.

If we also have the fields in that order for the sort by part of the query, then our advantage is double.

Both cases will be efficient, because will eliminate the need for a second index.

Aug 052013

The application data access patterns

By using MongoDB, the link between database administrators and application developers is tightened so much, that they really should sit close to each other not only at design time, but during the whole development process. Database design and application design interfere so much, that one might say that the database is merely a part, a module of the software system, instead of a separate layer.

Why would that be true? Let’s see the following ideas:

the database structure does not have a standard design pattern, but it highly depends on how the application accesses the data

the application can dynamically change the database structure, by easily adding / removing new keys, without making special operations

– most of the logic that took place in SQL Stored Procedures are now part of the application logic

Thus, the application developer is much more aware of the database structure and operations and viceversa, the database administrator is much more aware of the application development.

Now, let’s see some practical aspects:

Asynchronous updates for de-normalized documents (and other situations)

I have asserted. in the first part of this article, that normalization is still a good idea. But there are times when de-normalization is a much better idea. For example, you have a products collection and each product is part of one or more category. One example of such a document can be:

name : "Product One",
description: "Product One Description",
categories: [ "Some Category", "Some Other Category" ],
price : 10.25

Let’s say you already added 10000 products and 7300 of them have “Some category” amongst their categories. There is now a reason to change the name of this category. We don’t want to say “Some category” anymore, but “Some product category”. It is a simple update that we all know, i.e.

db.test.update({categories:"Some category"},{$set:{"categories.$":"Some product category"}},{multi:true})

What if this update is a time consuming one? And, more than that, what if the array that we are updating might contain the value we’re changing more than once? The $ operator in the update section refers only to the first value found in the array, so in the latter case not all values would be updated.

This, in my opinion, calls for an asynchronous approach, if the business rules allow for delayed updates. In the asynchronous method, the update logic would be:

modified_documents = db.coll.update(..);
while modified_documents > 0

In fact, the asynchronous (and thus delayed) updates are welcome wherever possible, so that the application responsiveness is greatly improved. An application-wide thread pool would help, too, because the main application thread won’t waste time on thread creation.

For Java and C#, I recommend using the Quartz scheduler for tasks management, even if you’re not going to use it’s repetitive tasks features, but you won’t have to worry about thread pools and other asynchronous programming related matters.

Use the aggregation framework

This is as simple as it reads. Make sure you have indices created for the $match and $sort parts of the aggregation pipeline, and use $project as often as possible in the pipeline, to limit the keys / fields you’re operating against (thus narrowing the documents sizes). Be aware that if the result of an operation of the pipeline grows bigger than 16 MB (the maximum document size), the operation will throw an error, so you must consider limiting the results as early and as much as possible using $match and $project. It will improve speed and lower the chance of unwanted document growth.

Another tip is to try the aggregation framework to count the documents matching a certain criteria instead of using the more “traditional” count method. It might prove to be a lot faster (not always, but it’s worth trying it, especially with very big counts and probably depending on the criteria used). For example, instead of
db.coll.count({name:"Some name"})
db.coll.aggregate([{$project:{_id:0,name:1}},{$match:{name:"Some name"}},{$group:{_id:null,count:{$sum:1}}}])

Use precalculated fields wherever possible. This will be the fastest way to have ready-made stats. For example, say you have an user login history, instead of counting the number of documents in the history collection, filtered by the user id, you can have a field on the user collection that will be incremented each time the user logs in.

A related improvement can be to have a count field for each array in a document. So, if a product has many categories, the only way to count the categories is either retrieve the document and count the elements in the array, or use the aggregation framework to do the same. But you can also have a count_categs key which is incremented atomically with each push into the array:

db.coll.update({_id:1},{$push:{categories:"new category"},$inc:{count_categs:1})

Thus, counting the categories is done by simply reading that field, which can prove to be much faster than reading the entire array then counting its elements.


The only thing to be emphasized here, apart from what the documentation already does, is that for each collection you’ll have to trade having a lot of indices which means faster reads to having fewer indices to favour faster writes.

As an example, for the users collection, having faster reads is essential; a new user won’t be disturbed if his account is created with, say, 100 ms delay, and profile updates are rare enough, whilst we always read from the users collection to get various user information used in various sections of our application. On the other side, a write-heavy collection might be the stats collection; we will also need quite a few indices on this collection, too, so this is a tough decision. But lucky enough, we usually can live with stats delayed with even several seconds, so stats is a good candidate for asynchronous writing. We still need to be careful, though, because stats are updating so frequently that even asynchronous writing won’t help if they take dangerously long, locking our database anyway. So make extensive use of the aggregation framework and pre-calculated fields for stats with as few indices as possible.

Application-wide database “locks”

What if we run our application on multiple servers, but we want to run a task only once at a time? We would need a way to communicate to other application instances that the task is already running so it doesn’t start again.

This can be accomplished by having a flag in the database, which will be set to true when the task is already running. The task first checks this flag and exits immediately if it is not supposed to run. We’re taking advantage of the atomicity of the findAndModify command:
This wil return null if the flag is already set.

Other aspects

Decide whether you can live with delayed reads. If this is the case, do setup the connection strings to read from the secondaries first (readPreference=secondaryPreferred). This would benefit from using more than 2 secondaries.

With MongoDB, SQL injection becomes history for good. But you can fall into some traps related to the query string parameters and database / collection / field naming issues. For example, you might find useful to receive the sort by field name in the query string and return your results sorted by that field name. In this case, case sensitivity may be an issue if you don’t make sure that the received parameter has the same case as the key name. This has also implications for index use, because it needs to consider indices for all the possible keys that are being received.

For normalized collections, you might be in the situation where, for each master row, you need to get something from it’s corresponding children rows. This would mean to query the child collection for each row in the master collection, which will lead to a lot of trips to the database. The C#-like solution for this might be:
IDictionary<ObjectId, BsonDocument> masterDocuments = new Dictionary<ObjectId, BsonDocument>();
foreach(BsonDocument masterDocument in db.masterCollection.find({...})){
masterDocuments.Add(masterDocument["_id"].AsObjectId, masterDocument);
IDictionary<ObjectId, IList> children = new Dictionary<ObjectId, IList>();
foreach(BsonDocument childDocument in db.childCollection.find(Query.In("master_id", new BsonArray().AddRange(children.keys)))){
children.Add(childDocument["master_id"].AsObjectId, new List());
// do whatever you need with the resulted documents

So basically you select all your children documents by using the $in operator, then do the math you require application-side.  Although this implies writing more code, it brings your application a lot of gain in speed.

I hope this article brings some help and clarifying to both novice and experimented developers who use MongDB. Please do post comments to either correct or to add to what I have said.

Thank you!

Aug 022013

Communicating with the database

I decided to talk about indices after we cover the application / client side of things, because indices greatly depend on the data access patterns used application side. So this part is about the communication principles.

We all know that network is one of the biggest uncertainties when it comes to programming. We must always take into consideration all sorts of issues: unavailability, lags, interruptions, speed.

The main idea is around making as few trips to the database as possible. As few queries as possble and each query and each update has to do as much as possible, but not more than strictly needed.

Queries must always use indices. I will repeat this in the next section: create several groups of queries (i.e. query by name, query by id, etc) for each collection and try to have as less groups as possible, but always adjust your queries to use indices each time. When considering this, also take the sort part into consideration, because sorting uses indices, too. On the test database, I always insert some dummy data and try my queries there so I make sure they use an index – see explain.

Expect the unexpected – i.e. don’t trust the network. You must have a strategy in the application that will handle unavailability (a short one, say 30-40 seconds). Reads will most probably timeout, but the writes can prove to be invaluable, so try to detect any network issue (and also any primary switch situation, in which case you will also not be able to write) and have an asynchronous queuing mechanism that will immediately return to the client but will keep the writes in the queue until the primary is available and they can be successfully sent to it.

In the SQL world, it is a good practice not to use SELECT * FROM table, but rather explicitly specify the fields to be returned, and preferably as few fields as possible. The same idea is valid with MongoDB: always retrieve only the needed keys of your resulting documents, not all of them.

Also, make sure the application uses paging when retrieving results. This is a must, so always use paging and each page should contain a small number of results. Your users can’t read more than one row at once anyway. They would most probably “wait” 100 ms for the next page with 10 results rather than 5 seconds for everything on one page.

More on the application techniques to have an efficient way to communicate with the database in the next part.

(to be continued)

Jul 312013

(continued from here)

The automated backup strategy

For regular backups, I recommend an approach similar to 10gen backup strategy, three sets of backups: daily, weekly and monthly. Add three RAID10 mount points for each of these backups; create folders for 4 daily backups (one every 6 hours), 4 folders for weekly backups and 12 folders for monthly backups, respectively. Daily backups will be valid for 7 days, weekly ones for 4 weeks, monthly ones for 12 months. Each backup will have its own folder, named after the date and time when it’s taken.

All these backups will be created automatically by cron jobs.

Here is an example of such crontab:
0 0  * * * /bin/bash /root/
0 6  * * * /bin/bash /root/
0 12 * * * /bin/bash /root/
0 18 * * * /bin/bash /root/
0 2  * * * /bin/bash /root/
0 4  * * 0 /bin/bash /root/

And here is an example of a backup script:

Note: This script creates folders named using the timestamp of the creation time, so it can easily delete them when they expire.

# change these to match the current backup parent folder (daily/ weekly/ monthly/ , respectively)
# and the amount of time back to remove backups from
# for daily backups use
yesterday=$(date --date="7 days ago" +"%s")
# for hourly backups use
yesterday=$(date --date="24 hours ago" +"%s")
# for weekly backups use
yesterday=$(date --date="4 weeks ago" +"%s")

# this is the absolute backup parent folder
# data to connect to the database
# the current date and time to create the folder name
dnow=$(date +”%Y%m%d_%H%M%S_%s”)
# this is the backup parent folder (absolute path)
# this is the backup folder
# get the folders array
dirs=`ls -lah –time-style=”long-iso” $pfolder | egrep ‘^d’ |  awk ‘{print $8}’`
# loop through siblings folders
for dir in $dirs; do
# get the timestamp part
adir=`echo $dir| cut -d’_’ -f 3`
# if it’s and old backup, remove it
if [ $adir -le $yesterday ]; then
rm -r — “$pfolder$dir”*
# check whether we’re primary. If yes, we don’t start the backup
checkResult=`mongo –quiet –username=$dbusername –password=$dbpassword –port=$dbport –eval \
“checkPrimary(); function checkPrimary(){ var result=false; rs.status().members.forEach(function(m){ if(m.self && m[‘stateStr’] == ‘PRIMARY’){ result=true; }}); return result; }” admin`
if [ $checkResult == ‘false’ ]; then
# create the current backup, redirecting the output to a specific dumplog.log file
if [ ! -d $folder ]; then
mkdir $folder
mongodump –port $dbport -u $dbusername -p $dbpassword –journal –oplog -o $folder > $folder/dumplog.log

The script can be easily changed (the javascript part) to use a database lock, so it doesn’t run on more than one secondary at a time. Until I make and test that change, a temporary solution is to setup a secondary with priority zero and install the jobs only on that member.

(to be continued)

Jul 282013

There are lots of readings about this subject, but I found them hardly complete. It’s definitely a complex and arguable subject, but hopefully I could find a synthetic way to check the most important aspects.

As with any database, it’s a separate layer from the application and it most probably resides on a separate server. There are concerns about:

  • inner database configuration
  • communication with the database
  • the data access patterns used application-side

The database documentation covers a lot in what the database configuration is concerned. The main ideas will spawn through more articles here.. So let’s start

Part I: Planning for a good system-wide database configuration.

Making the switch from a relational database can be tricky even from the first step. A database administrator should closely take a programmer’s mindset: the database is to be used by an application, not just queried from the command line or some GUI client. This might be seen as a trivial idea, but it’s a very important one. We are designing for specific business rules which will be enforced at the application layer. We could hardly enforce business rules at the database level; we’re merely designing and preparing an efficient storage system. Even if MongoDB comes with stored javascript support, we’ll probably rarely use it. And even if we’ll use it, javascript is still seen as a client, so the main concern for our design is an efficient storage followed by an efficient communication strategy. Efficient storage means: store as less as possible, normalize as much as possible.

storing less is as simple as it reads: don’t store everything you might think of. Any information that you store you must use somewhere in the application. Take advantage of the dynamic schema that MongoDB offers: you will be able to add more keys later, if necessary. This is a huge advantage. Use as short keys as possible for your document keys, but meaningful ones. You’ll most probably won’t have time to document your database thoroughly; short keys are very efficient for large collections (many keys and many records). For smaller collections, it will be useful to have more meaningful, explicit key names. Pay attention to data types, too: you will always want to use a numeric data type where a numeric value is to be stored. Try as much as possible to not mix data types, even if MongoDB lets you do so. In fact, this might be a trap you can fall into: you’ll find indexing and querying quite inefficient when mixing the data types. Also, keep in mind that since you decide you need a key, you should always put a value in it, application-side, be it null or not. Checking for existing keys/values will also prove to be a headache.

normalization is still a good idea. Generally, you’ll prefer embedding small documents and documents you’re not referring to from other collections, as long as you can live with duplicates not updating instantly when one of them is updated. This last sentence will prove to be a very important one, so always have it in mind when deciding to embed or not to embed. You can either normalize or use application-side techniques (such as asynchronous updates, will talk later about them) if you need those duplicates to be updated. This is one of the effects of the document level update atomicity.

plan for special keys: pre-calculated keys (for faster aggregation), case-insensitive ordering keys (duplicate a key and store a case insensitive value of the original key, for case insensitive sorting, until 10gen will provide a native solution) for those string valued keys you know you’ll need to sort by. You might also want to use friendly ID-s in some cases, so that you present the user with values uniquely identifying the documents but in a friendlier way than the ObjectId data type appears to be. This will imply having a unique value generator application side by using a database locking system, which we’ll discuss later.

– use replication. Definitely. Even on the test environment, because you’ll most probably need to test operations that depend on the fact that not all reading sources contain the exact same data. The test members should be started with the –smallfiles and a small value for –oplogSize (see Configuration File Options), so that they don’t take a lot of space.

– plan for regular, automated backups (another separate discussion needed).

– for each replica set member, and also for each backup set, use a convenient RAID10 configuration; it’s very efficient and reliable. Here is a good tutorial for CentOS which can be probably extended to other distributions, too.

(to be continued)