DynamoDB’s shortcomings (and our work arounds)

At Dailycred we use DynamoDB as a data store, and there’s a lot to like about it:

• scalability and simplicity of NoSQL
• consistent performance
• low learning curve
• it comes with a decent object mapper for Java

It’s a great service considering how new it is, but it definitely still has some rough edges that make some things harder than we expected.

DynamoDB’s not ideal for storing events

Like most websites, we store a variety of user events. A typical event has an event ID, a user ID, an event type and other attributes that describe actions performed by users. At Dailycred, we needed an event storage that is optimized for reads. For our dashboard we need to quickly filter events by type, sort events by time and group events by user. However, we don’t need to record events as soon as they happen. A second of delay is fine.

Using a relational database, we can store events in a denormalized table. Add indices to columns that answer query predicates. In this setup, writes are not very fast, but reads are extremely fast. We can use the richness of SQL to query events easily.

A big limitation of DynamoDB and other non-relational database is the lack of multiple indices. In an events table, we could use the event ID as the hash key, the event time as the range key. This schema would enable us to retrieve most recent events, but we can’t filter event by type without doing a full table scan. Scans are expensive in DynamoDB. You could store events in many tables, partitioned by event type or by user ID. Perhaps for each predicate, create an index table with the predicate’s value as the key and the event ID as an attribute. Is the added complexity worth it? Probably not. DynamoDB is great for lookups by key, not so good for queries, and abysmal for queries with multiple predicates.

SQL was a better tool in this case, so we decided not to use DynamoDB at all for storing events.

DynamoDB overhead (compared to SQL)

DynamoDB supports transactions, but not in the traditional SQL sense. Each write operation is atomic to an item. A write operation either successfully updates all of the item’s attributes or none of its attributes. There are no multi-operation transactions. For example, you have two tables, one to store orders and one to store the user-to-order mapping. When a new order comes in, you write to the order table first, then the mapping table. If the second write fails due to network outage, you are left with an orphaned item. Your application has to recognize orphaned data. Periodically, you will want to run a script to garbage collect those data, which in turn involve a full table scan. The complexity doesn’t end here. Your script might need to increase the read limit temporarily. It has to wait long enough between rounds of scan to stay under the limit.

One strike, and you are out

While DynamoDB’s provisioned throughput lets you fine tune the performance of individual tables, it doesn’t degrade gracefully. Once you hit the read or write limit, your requests are denied until enough time has elapsed. In a perfect world, your auto-scaling script will adjust throughput based on anticipated traffic, increasing and decreasing limits as necessary, but unexpected traffic spikes is a fact of life. Say you bump up the limits as soon as DynamoDB throws a ProvisionedThroughputExceededException, the process could take a minute to complete. Until then, you are at the mercy of retries, a feature that is thankfully enabled by default by the official SDK.

At Dailycred, it’s hard to accurately anticipate resource usage, due to the fact that we do not know when our clients are hit with a wave of visitors. What if the wave happens to multiple clients at once?

Backups: slow or expensive (pick one)

Another annoyance with provisioned throughput is that you can only decrease your provisioned limit once a day. Say your daily backup script temporarily increases the read limits for the duration of the job. After the limits were reduced, your site gets a burst of traffic. After increasing the limits for the second time, you are stuck with the new limits for up to a day. In that regard, DynamoDB isn’t as elastic as other AWS services, where resources can be deallocated with no daily limits.

Unit tests: slow or expensive (pick one)

We also run a lot of tests that use DynamoDB, which means a lot of items are written and read very quickly. We run the tests several times a day, which means our development database tables are sitting completely idle most of the time, only to be hammered with reads and writes when we run our unit tests. From a cost perspective, this isn’t ideal. However, it’s even worse if a developer has to wait extra time for his unit tests to complete.

Our solution: Dynamonito, a lightweight caching utility for DynamoDB

To work around some of these limitations, we built Dynamonito. Dynamonito is an open source tool we built to cache DynamoDB data, and save us an unnecessarily large bill.

Dynamonito is a drop in replacement for the high level mapper. It intercepts the DynamoDB save operations, serializes the object into DynamoDB’s native wire protocol format in json, and puts the json in cache. The cache is a write-through cache.

There are two pluggable cache implementations: in-memory and Redis. The in-memory cache is backed by Guava’s cache which is in-process and is extremely fast. The cache cannot be shared among multiple application hosts, so its use is limited to tests. The Redis cache resides on a Redis server and is out-of-process and so it can be accessed by many hosts. Optionally, you can set a time-to-live duration for cached objects.

Currently, Dynamonito can only cache objects that are mapped through the Java SDK, not through low level GetItem requests. It does not cache objects returned by the scan operation or the query operation.

Check out the Dynamonito project on Github.

Dynamonito was written by Shaun Ma.

How we’re fixing pricing

[background: DailyCred is the best way to handle user accounts on your website]

When we announced our big update on HackerNews recently, we also changed our pricing. We did a couple things wrong that scared away people who wanted to sign-up.

How we screwed up:

• no free tier to try the service
• a credit card screen that scared people away

This seems obvious (but we got it wrong): Developers need to try your service before they will buy it. It’s crucial that they can easily sign up and try it without risk, so they can learn about it and see if it’s right for their project.

How we’re fixing it:

• now with free tier up to 50 users
• no credit card required to try it out

Thank you:

As a thanks for giving DailyCred a look and signing up today, here’s a 50% off coupon. If you like the service (and we’re sure you will!) you’ll get a discounted rate when your site grows into one of the paid plans. There’s no risk - sign up today to lock in the discounted price!

2 new ways for users to join your site

DailyCred now supports and Instagram and Disqus, so users have more options in how they join your site.

Adding these options is easy: simply enter the app id & secret to your settings page and choose which providers you want to offer.

More options coming soon! If you’d like to vote for your favorites, drop us a line: support@dailycred.com

Dailycred now supports Facebook, Twitter, Google, and Email user accounts

Now you can let users join your site with their Facebook, Twitter, or Google accounts. No additional coding required!

Simply visit your DailyCred dashboard and choose the providers you want to add. If you need help getting set up, please contact us at support@dailycred.com.

More options means happier users and more signups!

Discuss on HackerNews

Surprise! People hate being forced to use Facebook

When you sign up for a new website, do you use the convenient “Connect with Facebook” option?  Or do you just make an email & password account?

In this post we take a look at signups for our previous startup and conduct a “man on the street” survey to see how people feel about Facebook Connect.

Survey says… “hell no”

We wanted to know how real people feel about this, so we hit the streets to ask random tourists in Seattle’s Pike Place market for their reaction.

Don’t worry, we used pictures to make sure they knew what we were talking about.

Here are the results:

The most surprising thing about the survey was 1) people actually understood what we were asking them, 2) the pure vitriolic hatred of being forced to use the Sign Up with Facebook option.  Some people said “of course - I love Facebook”, but many more said, “No - never. Never never never.”

Actual user signup behavior

We shouldn’t trust small un-scientific surveys too much, and there’s a difference between what people say and what they do. So we looked back over a year’s worth of signups for one of our sites, Shopobot, to get some more meaningful numbers. This is a better test, because the signup screen offers both options with equal weight and we have over 70,000 organic signups.

Younger & Tech Savvy use Facebook More

Back in early 2011, the beta testers were somewhat open to signing up using their Facebook account, with 42% choosing to connect.  Apparently the tech-savvy HackerNews / TechCrunch crowd is either not super concerned about their privacy, or we did a great job of convincing them we were trustworthy.

Mainstream Audiences use Facebook Less

But once the site was released publicly in June and more mainstream users started coming in from NYTimes, CNN, and other big media sites, you can see the preference to not connect with Facebook.  Only 23% took this choice, and the other 77% created an email & password account instead.

It turns out real people actually are concerned about their privacy and wall spam, and would rather not connect to Facebook on their first experience with your site.

tl;dr

If you’re building a consumer facing site, you should really offer an email & password signup option.  The vast majority of users prefer this.

Discuss on HackerNews

Tutorial: Creating a Rails To-Do app with Dailycred

Today I’ll be showing you guys how to add DailyCred to a fully functional Ruby on Rails application to get comprehensive authentication functionality. This tutorial assumes you have a basic understanding of Ruby on Rails (RoR) and can follow along basic tutorials. Before you get started you need to sign up for dailycred so you can get API keys, which you can obtain on your settings page.

The first step is to clone the wonderful demo To-Do app that Engine Yard has provided on github as an example RoR app. This app already includes everything you need to create tasks and save them to a list. We will be adding the functionality to authenticate into the app and save tasks to specific user accounts. To start, open your terminal and execute:

git clone https://github.com/engineyard/todo.git todo
cd todo

Next, add the dailycred gem to your gemfile.

gem 'dailycred'

I also removed the line "gem 'mysql2', '~> 0.2.7'" as I don’t wish to use or install the mysql2 gem (and you don’t either to follow along with this tutorial). When you’re all set, bundle your gems by running:

bundle

Now have your dailycred client_id and secret_key ready and run:

rails g dailycred your_client_id your_secret_key

This will generate everything you need to get going with authentication, including a user model, session controller, omniauth initializer, javascript tracking code, and many helper variables.

Congratulations!!

You’ve officially set up authentication on your rails site. Start your server with rails s and point your browser to locahost:3000/auth. Click the ‘Sign Up’ button and go ahead an create an account.

But you may feel like something is missing. While you can sign in and out of your app, the experience isn’t personal at all! We need a way to allow users to have their own lists and tasks.

Next let’s add a property to tasks and lists to associate them with a specific user. We will use the classic “belongs_to” relationship, where a user has many tasks and lists. We will you the “has_many :through” relationship to describe how users have many tasks through lists. First run some migrations to add a user_id property to tasks and lists:

rails g migration AddUserIdToLists user_id:string
rake db:migrate

open the file app/models/list.rb and add the line:

belongs_to :user

open app/models/user.rb and add

has_many :tasks , :through => :lists
has_many :lists , :dependent => :destroy

That takes care of our relationships. Open up app/controllers/lists_controller.rb and edit line 4 to

@list = current_user.lists.new(params[:list])    

This changes the creation of the list to belong to the current signed in user. Open app/controllers/tasks_controller.rb and add the following before_filter methods.

before_filter :authenticate

This makes it so that a user who isn’t yet authenticated is automatically redirected to the login screen. It also creates creates a @user instance variable that is available in all controller methods for convenience. Finally change the index method in tasks_controller.rb to the following:

def index
  @todo   = current_user.tasks.where(:done => false)
  @task   = current_user.tasks.new
  @lists  = current_user.lists
  @list   = current_user.lists.new

  respond_to do |format|
    format.html
  end
end

Congratulations!!!

You are all set up. You have a fully functional to-do application where users can sign in to your app and create lists and tasks. You should be proud of yourself.

Finishing up

Custom Events

Wouldn’t it be fun to be able to keep track of when your users created new tasks and lists? Dailycred has a custom events feature that lets you do just that.

Go back to your app/controllers/tasks_controller.rb and edit the create method to look like the following:

def create
  @list = List.find(params[:list_id])
  @task = @list.tasks.new(params[:task])
  if @task.save
      dailycred.event(current_user.uid, "New Task", @task.name)
      flash[:notice] = "Your task was created."
  else
      flash[:alert] = "There was an error creating your task."
  end
  redirect_to(list_tasks_url(@list))
end

And do something similiar in app/controllers/lists_controller.rb:

def create
  @list = current_user.lists.new(params[:list])
  if @list.save
      dailycred.event(current_user.uid, "New List", @list.name)
      flash[:notice] = "Your list was created"
  else
      flash[:alert] = "There was an error creating your list."
  end
  redirect_to(list_tasks_url(@list))
end

Now when new tasks and lists are created by your users, you can see the events being created in your dashboard

Finishing up

You’re all set up! I hope this was helpful in getting more acquainted with Dailycred and how to work with our service. Thanks for checking us out and always email us at support@dailycred.com For any questions!

Get an email when press, investors, or competitors use your app

During our fund-raising and launch process, we always wanted to know when a journalist or potential investor signed up for our service. Starting today we’ll email you the instant a VIP signs up for your app or website.

But we don’t stop there. We’ll also link you to their user profile in DailyCred’s CRM so you can watch them use your service in real-time. Don’t go into another pitch meeting blind; go in knowing what kind of experience they had using your service.

We also include the tag in your app’s subsequent API requests. Why? So you can give press and investors free perks of course. :-) For example including extending free trials. (But you might want to validate emails first, which we also do.)

Our default tagging config includes:

• investor (VCs and investment banks)
• press (newspapers and prominent blogs)
• bureaucrat (US legislators and bureaucrats)
• incubator (techstars and the like)

If you’d like to set up your own tagging system, feel free to fork our full list on Github and we’ll get you set up.

Need some ideas for custom tags? Here are some fun ones:

• competitor: find out when competitors and clone factories log in to your service to copy all your ideas.
• fakemail if your service needs a legit email, add services like mailinator to a tag. When they attempt to do a sensitive action, like posting links, ask them to update their email.

Discuss on HackerNews

Hello world! Introducing DailyCred

Startups work best when we focus on our unique value, and don’t waste time reinventing the wheel.  We don’t need to build our own servers, payment, or monitoring systems anymore - those are all available off the shelf, and are great, cheap, and easy to use. 

The #1 thing that startups still waste their time building: User Sign in.  
(#2 is analytics, so we can watch our users sign in.)

We’re launching DailyCred to fix this.  DailyCred handles user authentication, so you get a polished and secure account system for your website or app, instantly, without having to build it yourself.