Quora’s Technology Examined

Quora has taken the tech and entrepreneurial world by storm, providing a system that works so fluidly that it is sometimes hard to see what the big fuss is all about. This slick tool is powered, not only by an intelligent crowd of askers and answerers, but by a well-crafted backend created by co-founders who honed their skills at Facebook.

It is not surprising that, with all the smart people using this smart tool, there are many pondering on how it works so well. The NoSQL boffins scratch their heads and ponder such questions as, “Why does Quora use MySQL as the data store rather than NoSQLs such as Cassandra, MongoDB, CouchDB, etc?“.

In this blog post I will delve into the snippets of information available on Quora and look at Quora from a technical perspective. What technical decisions have they made? What does their architecture look like? What languages and frameworks do they use? How do they make that search bar respond so quickly?

• Components Of Quora
• What’s Cooking Under That Hood?
• The Search-Box
• Webnode2 And LiveNode
• Amazon Web Services
• Ubuntu Linux
• Static Content
• HAProxy Load-Balancing
• Nginx
• Pylons And Paste
• Python
• Thrift
• Tornado
• Long Polling (Comet)
• MySQL
• Memcached
• Git
• Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”
• Conclusion
• Recommended Reading
• Resources

Components Of Quora

The general components that make up Quora are…

• You can ask questions
• You can answer questions (anonymously if you desire)
• You can comment on answered questions
• You can vote-up or vote-down answers to questions
• Questions can be assigned to topics
• You can write a post (a informative statement, rather like a orphaned answer or blog post)
• You can follow questions, topics or other users
• A super-fast auto-complete search-box at the top, which doubles as the method for entering new questions

The last point, the super-fast auto-complete search-box, is one of the defining features of Quora. You can see immediately, as you begin to enter a question, whether somebody else has already asked that question or if there is a topic or post on the subject. Let’s start there…

What’s Cooking Under That Hood?

The Search-Box

Only the questions, topic labels, user names or post titles are indexed and served up to the search-box. There is no full-text search, so searching the content of questions and answers will not work. The text that is indexed is tokenized so that words in a different order will still be matched. Prefix matching enables best matches to be shown before the entire word is entered. For instance, typing “mi” might immediately show “Microsoft” in the results.

There is some simple stemming of words, since “nears” matches “near”, but “pony” does not match “ponies”. “Topic-aliases” allow for similar matches on topic names, such as “startup” and “start-up”. These topic-aliases have been manually entered by users. Otherwise these would not match.

If a duplicate question is redirected to another question (a feature of Quora), then that original question will still appear in the search results, since it increases the chances of a match. There is no n-gram indexing, so slight mis-spellings will not match. For instance, “gooogle” (with an extra “o”) finds nothing.

Previously, they did use an open source search server, called Sphinx. It supports the features they are using above, but they have since moved from this due to real-time constraints. Their new solution is built in-house and allows them better prefix indexing and control over the matching algorithms. They built this in Python.

What libraries does Quora use for search?
Adam D’Angelo, Quora Founder (Nov 13, 2010)
Our search is custom-written. It doesn’t use any libraries aside from Thrift, and Python’s unicode library, which we use for unicode normalization.

Speedy Queries

Did I mention that the search-box is fast? I did some tests and found the responses to be impressive. Queries are sent over AJAX as a GET request. Responses come back as JSON with the rendered HTML embedded inside the JSON. Rendering of the results on the server-side, as opposed to rendering them in JavaScript, seems to be due to the need to highlight matching words in the text. This is sometimes too complex for JavaScript. For instance, typing “categories” might highlight the world “category” in the result text.

I was seeing responses of roughly 50 milliseconds per query from my Linode machine. Quora does not short-change you when sending requests. From within the browser, I found typing “Microsoft” (9 characters) would result in nine requests to the Quora search server, no matter how fast you type. As you will see later, the server is in control, so if it did become over-loaded, then it could update the results less frequently without changing the JavaScript.

Quora uses persistent connections. A HTTP connection is established with the server when you start typing the search query. This connection is kept open and further requests are made on this same open connection. The connection will terminate (times-out) if not used for 60 seconds. If a connection times-out then a new connection is established when typing begins.

To simulate the typing of a word into the search-box, I sent the following requests, character-by-character, across a persistent connection. For instance “butler” is six requests (“b”, “bu”, “but” … “butler”).

"butler" (6 chars) duration: 0.393 secs 0.065 secs per query
"butler monkeys" (14 chars) duration: 0.672 secs 0.048 secs per query
"fasdisajfosdffsa" (16 chars) duration: 0.746 secs 0.046 secs per query

That last query was used to test if there was a slow-down for a word that would obviously not be in a caching layer. I saw no slow-down. This means that they are not caching, caching is only used to take the load off the backend search engine or they are doing something smarter (e.g. if there is no match for “fasd” then there will be no match for “fasdi”, so abort).

Is Quora going to implement full-text search?
Adam D’Angelo, I made a lot of the early Quora … (Sep 1, 2010)
Yes, eventually. We haven’t implemented this yet because we’ve prioritized other things, but we will definitely do it in the future.

Webnode2 And LiveNode

Webnode2 and LiveNode are some of Quora’s internal systems, which were built for managing the content. Webnode2 generates HTML, CSS and JavaScript and is tightly coupled with LiveNode, which is responsible for managing the display of the content on the webpage. Charlie Cheever says that he were to start a similar project without LiveNode, then the first thing he would do is rebuild it.

They seem very pleased with the technology they have built and struggled to find its weaknesses. One weakness is that it is tricky for LiveNode to keep track of what is happening within the browser as it pushes changes from the server. If users A and B are viewing the same question then ones interactions will affect the other. For instance, if user A up-votes an answer then that answer will be promoted and will visibly move up the page. This display change will be pushed over AJAX to user B’s browser. Any prior browser-side change that user B made, such as expanding a comments section, might be lost.

LiveNode is written in Python, C++, and JavaScript. jQuery and Cython is also used.

While they would like to open-source LiveNode and have tried to keep code separation, doing so right now would be too much work and would take time away from their main goal, which is making Quora better.

Charlie Cheever points out that webnode2 is unrelated to the “free and easy website builder” called Webnode at webnode.com.

Amazon Web Services

Amazon EC2 and S3 is used for their hosting. While this is not as cost-effective in the long-term as running your own servers, it is perfectly designed for fast growing companies like Quora.

Ubuntu Linux

Quora uses Ubuntu Linux as its OS of choice. No major surprises there. It is easy to deploy and manage on Amazon EC2. Adam D’Angelo points out that he used Debian Linux at high school and college and stuck with it because “it works and there hasn’t been a compelling reason to switch”.

Static Content

You only need to look at the source HTML of any Quora webpage to see that they are using Amazon’s distributed content delivery network, Cloudfront. URLs are in the form…

https://d2o7bfz2il9cb7.cloudfront.net/main-thumb-670336-25-7kmigSSkkdusoE6gHRkdQsXfjuTCaxQs.jpeg

CloudFront is used for all static images, CSS and JavaScript (except for Google’s Analytics JavaScript, which is hosted by Google). Images are uploaded to the EC2 machine, then resized and uploaded to S3. This is managed using the Python S3 API.

HAProxy Load-Balancing

Quora uses HAProxy at the front-line, which load-balances onto the distributed Nginx servers behind them.

Nginx

Behind the load-balancer, Nginx is used as a reverse-proxy server onto the web-servers.

To understand more about this setup I recommend reading “Using Nginx As Reverse-Proxy Server On High-Loaded Sites“.

Pylons And Paste

Pylons, a lightweight web framework, is used as their main web-server behind Nginx. They use the default Pylon + Paste stack.

Pylons was selected much like you would select a pumpkin for Haloween. They scooped out the insides, such as templates and the ORM, and replaced it with their own technology, written in Python. This is where LiveNode and webnode2 reside.

MochiMedia was also one of the inspirations for using Pylons, since they were using it themselves.

Python

Coming from Facebook, it was a good bet that Charlie and Adam would choose PHP for their development language. As Adam points out, “Facebook is stuck on that for legacy reasons, not because it is the best choice right now“. From this experience they knew that choosing technologies, especially programming languages, for the long-run was very important. They also looked at C#, Java, and Scala. Discounting Mono, C# would be a choice of more than just the language. It would require them to build on-top of a Microsoft stack. Python won over Java because it is more expressive and quicker to write code than Java. Scala was too new. Adam mentions speed and the lack of type-checking as drawbacks with Python, but they both already knew the language reasonably well. Where Python lacks speed for performance critical backend components, they opt to write them in C++. They saw Ruby as a close match to Python, but their experience with Python and lack of experience in Ruby, made Python the winner. Python 2.6, to be precise.

Additional benefits for using Python are the fact that data-structures that map well to JSON, code readability, there is a large collection of libraries and the availability of good debuggers and reloaders. Browser-server communication using JSON is major component of what Quora does, so this was an important factor.

No IDEs are used for development as most use the Emacs text editor. Obviously this is personal choice, and would change as the team grows.

PyPy, a project that aims to produce a flexible and fast Python implementation, was also mentioned as something that might give them a speed-boost.

Thrift

Thrift is used for communications between backend systems. The Thrift service is written in C++.

Why would you write a Thrift service in C++?
Adam D’Angelo, I’ve written a lot of Python, in… (Sep 4, 2010)
Mainly if you want to keep data in memory between requests, and want to keep your Python code stateless. Writing a Python wrapper around a C library involves some memory management with reference counting that requires some understanding of the Python internals, but writing a thrift interface is simple. You also isolate failures this way – if the service goes down it won’t take the Python code down with it.

Tornado

The Tornado web framework is used for live updating. This is their Comet server, which handles the large volumes of open connections used for long-polling and pushes updates to the browsers.

Long Polling (Comet)

Quora does not display just static web pages. Each page will update will new content as questions, answers and comments are submitted by other others. As Adam D’Angelo points out, one of the best ways to do this currently is with “long polling”. This is different to “polling”. With polling the browser will repeatedly send requests to the server saying “Any updates?” and the server will respond with “No”. A few seconds later it will ask again, “How about now?”. “No”. “How about now?”, “No, already!”. This puts the client (web-browser) in the driver’s seat. This is backwards because the client does not how long to wait before asking again. If the client asks the server too frequently then it will unduly overload the server. If it pings the server too infrequently, then server will be sitting on updates while it waits for the client to request them and the end-user will not see updates immediately.

Long polling, also known as Comet, puts the server in control, by making the client wait for responses. The conversation between the client and server is the same, but instead of the client waiting before making another request, the server waits before it makes the response. The server can keep the connection open for a long period of time (e.g. 60 seconds) while it waits to see if any updates come in. When updates do come in it can respond immediately to the client. On receiving the update, the client then immediately sends a new request for more updates. The server, once again, delays responding until it knows something worth telling the client or enough time has past that it would be rude not to respond.

The benefit to long-polling is that there is less back-and-forth between the client and server. The server is in control of the timing, so updates to the browser can be made within milliseconds. This makes it ideal for chat applications or applications that want really snappy updates for their users.

The down-side is that you are going have lots of open connections between the clients and your servers. If you have a million users (Quora will soon) and, if only 10% of them are online on your site, you will need an architecture that can hold open at least 100,000 concurrent connections. This assumes they only have one tab open to your site. Right now I have 7 tabs open for quora.com in my browser. Each tab usually has multiple connections open to quora.com. In short, Quora must maintain a lot of open connections.

The good news is that there are technologies specifically designed for this. It costs very little to hold open connections in memory if you free up all the resources used for that connection. For instance, Nginx (Quora uses this for proxying requests) is a single-threaded event-based application and uses very little memory for each connection. Each Nginx process is actively dealing with only one connection at a time. This means it can scale to tens of thousands of concurrent connections.

How do you push messages back to a web-browser client through AJAX? Is there any way to do this without having the client constantly polling the server for updates?
Adam D’Angelo, Quora (Sep 29, 2010)
There is no reliable way to do this without having the client polling the server. However, you can make the server stall its responses (50 seconds is a safe bet) and then complete them when a message is ready for the client. This is called “long polling” and it’s how Quora, Gmail, Meebo, etc all handle the problem.

If you have a specialized server that uses epoll or kqueue, you should be able to hold on the order of 100k users per server (depending on how many messages are going). This is called the “c10k” problem. https://www.kegel.com/c10k.html

MySQL

Just like Facebook, where co-founder Adam D’Angelo previously worked, Quora heavily uses MySQL. In answer to the Quora question “When Adam D’Angelo says “partition your data at the application level”, what exactly does he mean?“, D’Angelo goes into the details of how to use MySQL (or relational-databases generally) as a distributed data-store.

The basic advice is to only partition data if necessary, keep data on one machine if possible and use a hash of the primary key to partition larger datasets across multiple databases. Joins must be avoided. He sites FriendFeed’s architecture as a good example of this. FriendFeed’s architecture is described by Bret Taylor in his post “How FriendFeed uses MySQL to store schema-less data“. D’Angelo also states that you should not use a NoSQL database for a social site until you have millions of users.

It is not only Quora and FriendFeed who are heavily using MySQL. Ever heard of “Google”? It is hard to imagine, since everything Google does has to scale so well, but in the words of Google, “Google uses MySQL […] in some of the applications that we build that are not search related”. Google has released patches for MySQL related to replication, syncing, monitoring and faster master promotion.

How does one evaluate if a database is efficient enough to not crash as it’s put under increasing load?
Adam D’Angelo, Quora (Oct 10, 2010)
One option is to simulate some load. Write a script that mimics the kinds of queries your application will be doing, and make sure it can handle the amount of load you want it to be ready for (especially as the size of the dataset changes).

Memcached

Memcached is used as a caching layer in front of MySQL.

Git

Git is used for version control.

JavaScript Placement

If you look at Quora’s source you will see that the JavaScript comes at the end of the page. Charlie Cheever suggests that this give the feeling of a quicker loading page, since the browser has content to display before the JavaScript has be seen.

Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”

Steve Souders, author of High Performance Web Sites and Even Faster Web Sites, lists the following rules for making websites faster. This list is mentioned by Charlie Cheever, the co-founder of Quora, as one of the reasons for Quora’s speed.

“One resource we used as a guide is Steve Souders’ list of rules for high performance websites: https://stevesouders.com/hpws/rules.php”
– Charlie Cheever, Quora

Steve Souders’ 14 rules are…

• Make Fewer HTTP Requests
• Use a Content Delivery Network
• Add an Expires Header
• Gzip Components
• Put Stylesheets at the Top
• Put Scripts at the Bottom
• Avoid CSS Expressions
• Make JavaScript and CSS External
• Reduce DNS Lookups
• Minify JavaScript
• Avoid Redirects
• Remove Duplicate Scripts
• Configure ETags
• Make AJAX Cacheable

Conclusion

Quora is a great example of a modern tech start-up. They are very small team who understand the technologies they are using very well. They have made considered choices in the technology they have selected and have a good vision of which components would be better written from scratch. They seem keen to share these in-house technologies with the open-source community and I look forward to when they have the time to make this a reality.

I intend to keep following Quora and writing about them more in future blog posts.

If you found this post useful then please leave a comment or follow this blog.

Resources

• Quora.com
• How FriendFeed uses MySQL to store schema-less data
• Steve Souders’ 14 Rules for Faster-Loading Web Sites

Comments