Category Archives: 架构

myspace架构及大型网站架构经验

架构,

着中国大型IT企业信息化速度的加快，大部分应用的数据量和访问量都急剧增加，大型企业网站正面临性能和高数据访问量的压力，而且对存储、安全以及信息检索等等方面都提出了更高的要求……
本文中，我想通过几个国外大型IT企业及网站的成功案例，从Web技术人员角度探讨如何积极地应对国内大型网站即将面临的扩展（主要是技术方面，而较少涉及管理及营销等方面）矛盾。
一、 国外大型IT网站的成功之道 (一) MySpace      今天，MySpace已经成为全球众口皆碑的社区网站之王。尽管一流和营销和管理经验自然是每个IT企业取得成功的首要因素，但是本节中我们却抛弃这一点，而主要着眼于探讨在数次面临系统扩张的紧急关头MySpace是如何从技术方面采取应对策略的。 第一代架构—添置更多的Web服务器      MySpace最初的系统很小，只有两台Web服务器（分担处理用户请求的工作量）和一个数据库服务器（所有数据都存储在这一个地方）。那时使用的是Dell双CPU、4G内存的系统。在早期阶段，MySpace基本是通过添置更多Web服务器来对付用户暴增问题的。但到在2004年早期，在MySpace用户数增长到五十万后，其数据库服务器已经开始疲于奔命了。
第二代架构—增加数据库服务器      与增加Web服务器不同，增加数据库并没那么简单。如果一个站点由多个数据库支持，设计者必须考虑的是，如何在保证数据一致性的前提下让多个数据库分担压力。
MySpace运行在三个SQL Server数据库服务器上—一个为主，所有的新数据都向它提交，然后由它复制到其它两个；另两个数据库服务器全力向用户供给数据，用以在博客和个人资料栏显示。这种方式在一段时间内效果很好——只要增加数据库服务器，加大硬盘，就可以应对用户数和访问量的增加。
这一次的数据库架构按照垂直分割模式设计，不同的数据库服务于站点的不同功能，如登录、用户资料和博客。垂直分割策略利于多个数据库分担访问压力，当用户要求增加新功能时，MySpace只需要投入新的数据库加以支持。在账户到达二百万后，MySpace还从存储设备与数据库服务器直接交互的方式切换到SAN（存储区域网络）—用高带宽、专门设计的网络将大量磁盘存储设备连接在一起，而数据库连接到SAN。这项措施极大提升了系统性能、正常运行时间和可靠性。然而，当用户继续增加到三百万后，垂直分割策略也变得难以维持下去。
第三代架构—转到分布式计算架构      几经折腾，最终，MySpace将目光移到分布式计算架构——它在物理上分布的众多服务器，整体必须逻辑上等同于单台机器。拿数据库来说，就不能再像过去那样将应用拆分，再以不同数据库分别支持，而必须将整个站点看作一个应用。现在，数据库模型里只有一个用户表，支持博客、个人资料和其他核心功能的数据都存储在相同数据库。
既然所有的核心数据逻辑上都组织到一个数据库，那么MySpace必须找到新的办法以分担负荷——显然，运行在普通硬件上的单个数据库服务器是无能为力的。这次，不再按站点功能和应用分割数据库，MySpace开始将它的用户按每百万一组分割，然后将各组的全部数据分别存入独立的SQL Server实例。目前，MySpace的每台数据库服务器实际运行两个SQL Server实例，也就是说每台服务器服务大约二百万用户。据MySpace的技术人员说，以后还可以按照这种模式以更小粒度划分架构，从而优化负荷分担。
第四代架构—求助于微软方案      2005年早期，账户达到九百万，MySpace开始用微软的C#编写ASP.NET程序。在收到一定成效后，MySpace开始大规模迁移到ASP.NET。      账户达到一千万时，MySpace再次遭遇存储瓶颈问题。SAN的引入解决了早期一些性能问题，但站点目前的要求已经开始周期性超越SAN的I/O容量——即它从磁盘存储系统读写数据的极限速度。
第五代架构—增加数据缓存层并转到支持64位处理器的SQL Server 2005      2005年春天，MySpace账户达到一千七百万，MySpace又启用了新的策略以减轻存储系统压力，即增加数据缓存层——位于Web服务器和数据库服务器之间，其唯一职能是在内存中建立被频繁请求数据对象的副本，如此一来，不访问数据库也可以向Web应用供给数据。
2005年中期，服务账户数达到两千六百万时，MySpace因为我们对内存的渴求而切换到了还处于beta测试的支持64位处理器的SQL Server 2005。升级到SQL Server 2005和64位Windows Server 2003后，MySpace每台服务器配备了32G内存，后于2006年再次将配置标准提升到64G。
事实上，MySpace的Web服务器和数据库仍然经常发生超负荷，其用户频繁遭遇“意外错误”和“站点离线维护”等告示，他们不得不在论坛抱怨不停……
MySpace正是在这样不断重构站点软件、数据库和存储系统中，才一步步走到今天。事实上，MySpace已经成功解决了很多系统扩展性问题，其中存在相当的经验值得我们借鉴。MySpace系统架构到目前为止保持了相对稳定，但其技术人员仍然在为SQL Server支持的同时连接数等方面继续攻坚，尽可能把事情做到最好。
(二) Amazon      亚马逊书店无疑是电子商务发展的里程碑。2000年到现在，世界网络业腥风血雨。Amazon曾经成为网络泡沫的头号代表。如今，当这个“最大的泡沫”用几经易改的数字把自己变成了坚实的IT巨人。
历览Amazon发展过程，其成功经验在于，它创造性地进行了电子商务中每一环节的探索，包括系统平台的建设，程序编写、网站设立、配送系统等等方面。用Amazon当家人贝索斯的话说就是，“在现实世界的商店最有力的武器就是地段，地段，地段，而对于我们来说最重要的三件事就是技术，技术，技术。”
(三) eBay      eBay是世界闻名的拍卖网站，eBay公司通信部主管凯文•帕斯格拉夫认为，“eBay成功的最重要原因在于公司管理和服务。”      其成功的奥秘可以列举为以下几点：      ①敢为天下先—在网络尚不普及的时代，eBay率先进入网络拍卖领域；      ②依托虚拟商场所产生的特有的“零库存”是eBay公司取得成功的另一个重要原因。该公司的核心业务没有任何库存风险，所有的商品都是由客户提供，它只需要负责提供虚拟的拍卖平台—网络和软件。所以，eBay公司的财务报表上不会出现“库存费用”和“保管费用”等。 ③自eBay公司成立开始，它就一直遵循两条“黄金原则”：建设虚拟社区，给网民以家的感觉；保证网站稳定安全地运行。
二、 国内大型网站开发时的几点建议      从本节开始，我们将结合国内外大型IT网站在技术扩展方面的沉痛教训和成功经验，探讨在如今刚刚开始的Web 2.0时代如何应对国内网站即将面临的数据访问量增加（甚至是急剧膨胀）的问题，并提出一些供参考的策略和建议。
(四) 搭建科学的系统架构      构建大型的商业网站绝对不可能像构建普通的小型网站一样一蹴而就，需要从严格的软件工程管理的角度进行认真规划，有步骤有逻辑地进行开发。对于大型网站来说，所采用的技术涉及面极其广泛，从硬件到软件、编程语言、数据库、Web服务器、防火墙等各个领域都有了很高的要求，已经不是原来简单的html静态网站所能比拟的。以著名的Yahoo!为例，他们的每一个大型网站工程都需要大量相应专业人员的参与。
(五) 页面静态化      可不要小看纯静态化的HTML页面！其实在很多情况下，HTML往往意味着“效率最高、消耗最小”，所以我们尽可能使我们的网站上的页面采用静态页面来实现。但是，对于大量内容并且频繁更新的网站，我们无法全部手动实现，因此可以开发相应的自动化更新工具，例如我们常见的信息发布系统CMS。像我们经常访问的各个门户站点的新闻频道，甚至他们的其他频道，都是通过信息发布系统来管理和实现的。信息发布系统可以实现最简单的信息录入自动生成静态页面，还能具备频道管理、权限管理、自动抓取等功能，对于一个大型网站来说，拥有一套高效、可管理的CMS是必不可少的。
(六) 存储问题      存储也是一个大问题，一种是小文件的存储，比如图片这类；另一种是大文件的存储，比如搜索引擎的索引。 大家知道，对于Web服务器来说，不管是Apache、IIS还是其他容器，图片是最消耗资源的，于是我们有必要将图片与页面进行分离，这是基本上大型网站都会采用的策略，他们都有独立的图片服务器，甚至很多台图片服务器。这样的架构可以降低提供页面访问请求的服务器系统压力，并且可以保证系统不会因为图片问题而崩溃，在应用服务器和图片服务器上，可以进行不同的配置优化以保证更高的系统消耗和执行效率。
(七) 数据库技术—集群和库表散列      对于大型网站而言，使用大型的数据库服务器是必须的事情。但是，在面对大量访问的时候，数据库的瓶颈仍然会显现出来，这时一台数据库将很快无法满足应用，于是我们需要借助于数据库集群或者库表散列技术。
在数据库集群方面，很多数据库厂商都有自己的解决方案，Oracle、Sybase、SQL Server等都有很好的方案，常用的MySQL提供的Master/Slave也是类似的方案。因此，你使用了什么样的数据库，就参考相应的解决方案来实施即可。
上面提到的数据库集群由于在架构、成本、扩张性方面都会受到所采用数据库类型的限制，于是我们需要从应用程序的角度来考虑改善系统架构，其中，库表散列是常用并且最有效的解决方案。我们在应用程序中安装业务和应用或者功能模块将数据库进行分离，不同的模块对应不同的数据库或者表，再按照一定的策略对某个页面或者功能进行更小的数据库散列，比如用户表，按照用户ID进行表散列，这样就能够低成本的提升系统的性能并且有很好的扩展性。在这一方面一个现成的例子就是搜狐。它的论坛就是采用了这样的架构，将论坛的用户、设置、帖子等信息进行数据库分离，然后对帖子、用户按照板块和ID进行散列数据库和表，最终可以在配置文件中进行简单的配置便能让系统随时增加一台低成本的数据库进来补充系统性能。
(八) 缓存策略      这绝对不单指低级的缓存技术相关的编程，应从整个架构角度着眼，深入研究Web服务器、数据库服务器的各层级的缓冲策略，最后才是低级的缓冲技术的编程。不同的Web服务器、数据库服务器及Web编程语言都有自己不同的缓冲策略。例如数据库存储方面，SQL Serve 2005中的主动式缓存机制，Oracle数据的cache group技术，Hibernate的缓存包括Session的缓存和SessionFactory的缓存；Web服务器方面，Apache提供了自己的缓存模块，也可以使用外加的Squid模块进行缓存，这两种方式均可以有效的提高Apache的访问响应能力，IIS缓冲器技术；至于web开发语言，所用缓存技术更存在很大不同，例如ASP.NET 2.0中提出了两种缓存应用程序数据和缓存服务页输出的策略，这两种缓存技术相互独立但不相互排斥，PHP有Pear的Cache模块，等等。
(九) 镜像      镜像是大型网站常采用的提高性能和数据安全性的方式，镜像的技术可以解决不同网络接入商和地域带来的用户访问速度差异。在镜像的细节技术方面，这里不阐述太深，有很多专业的现成的解决架构和产品可选。也有廉价的通过软件实现的思路，比如Linux上的rsync等工具。
(十) 负载均衡      负载均衡将是大型网站解决高负荷访问和大量并发请求采用的终极解决办法。 负载均衡技术发展了多年，有很多专业的服务提供商和产品可以选择，基于LAMP解决方案的Lighttped+Squid是相当不错的解决负载均衡和加速系统的有效方式。
(十一) 硬件四层交换      第四层交换使用第三层和第四层信息包的报头信息，根据应用区间识别业务流，将整个区间段的业务流分配到合适的应用服务器进行处理。第四层交换功能就象是虚IP，指向物理服务器。它传输的业务服从的协议多种多样，有HTTP、FTP、NFS、Telnet或其他协议。这些业务在物理服务器基础上，需要复杂的载量平衡算法。在IP世界，业务类型由终端TCP或UDP端口地址来决定，在第四层交换中的应用区间则由源端和终端IP地址、TCP和UDP端口共同决定。
在硬件四层交换产品领域，有一些知名的产品可以选择，比如Alteon、F5等，这些产品很昂贵，但是物有所值，能够提供非常优秀的性能和很灵活的管理能力。Yahoo中国当初接近2000台服务器使用了三四台Alteon就搞定了。
(十二) 软件四层交换      大家知道了硬件四层交换机的原理后，基于OSI模型来实现的软件四层交换也就应运而生，这样的解决方案实现的原理一致，不过性能稍差。但是满足一定量的压力还是游刃有余的。
一个典型的使用负载均衡的策略就是，在软件或者硬件四层交换的基础上搭建squid集群，这种思路在很多大型网站包括搜索引擎上被采用，这样的架构低成本、高性能还有很强的扩张性，随时往架构里面增减节点都非常容易。
(十三) 软件投资问题      据报导，目前国内除了一些上市企业和特别大知名大公司以外，很少有企业在成本中考虑正版软件的购置费用。这种思维极有可能给中国互联网带来噩梦。如果一些公司真正面临软件资金方面的困难，完全可以考虑使用开源世界的LAMP解决方案（Linux＋Apache＋MySQL＋Perl、PHP或者Python Web编程语言）；否则，随着我国加入WTO范围的不断扩大，盗版打击必然越来越严。因此，“苟且偷生”必将自食其果。
另外，随着网络带宽日渐提升，WEB 2.0技术必将影响到网络世界的几乎每一个角落。因此，如何积聚技术人员进行技术攻关并进一步加强安全防范也成为一个日益严峻的问题，宜尽早纳入到公司的议事日程。   四、 总结      中国电子商务真正理性发展的一个标志，是大量的传统企业实实在在地开始用互联网来处理商务、做生意，而现在这样的浪潮已经开始。北京发行集团，联合SINA、6688.com等单位共同推出的网上虚拟书店—新新书店就是这样的一个标志。
随着网络带宽日渐提升，随着网络理念和WEB 2.0技术的断深入人心，各种B2B、B2C、C2C等电子商务模式很可能以立体交叉方式整合到各种大型商务网站中来。因此，作为公司的技术人员，作为临危救驾的“白衣骑士”，如何应对海量存储、海量访问问题，海量信息检索的问题，日益严峻的安全问题，等等，已经刻不容缓。

QQ空间技术架构之深刻揭密

架构

QQ空间作为腾讯海量互联网服务产品，经过近七年的发展，实现了从十万级到亿级同时在线的飞跃。在这个过程中，QQ空间团队遇到了哪些技术挑战？其站点前后台架构随着业务规模的变化又进行了怎样的演进与变迁？成长背后有过怎样的心酸和快乐……

4.26日晚间，腾讯大讲堂·深圳首站《QQ空间技术架构之峥嵘岁月》活动在腾讯总部腾讯大厦多功能厅举办，腾讯Web前端研发领域专家、腾讯最早的专职Web开发团队核心成员、Qzone核心架构研发总监徐晓在两小时的演讲中揭开了QQ空间技术架构背后的秘密。

正式分享之前，徐晓介绍了QQ空间目前的海量服务数据规模（图1）：

  1. 拥有5.5亿的活跃用户
  2. 过万台的设备
  3. 数千万级别的同时在线
  4. 数十亿级别的全站PV
  5. P级的UGC存储量
  6. 每天千亿级别的服务请求

图1——QQ空间海量服务数据规模

接下来，徐晓把QQ空间成长和发展的历程分成三个阶段：石器时代、冷兵器时代和现代战争时代，分享了QQ空间从开始甚至难以生存，到逐步发展成国内最大的SNS社区产品背后的酸甜苦辣。

石器时代–从十万在线到百万在线

石器时代是QQ空间1.0到3.0稳健上线的过程。

徐晓说：“QQ空间在石器时代遇到的最大难题就是如何生存？如何让这个襁褓中的产品活下来？如何积累第一批宝贵的用户资源？”， 2005年QQ空间首发，最初是采用传统的发放邀请码的方式征集第一批用户。第一款QQ空间是嵌在独立的客户端软件中，看似是个客户端，其实是一个浏览器内核，里面都是HTML页面，以网站的方式存在。 当初的架构非常简单，采用了Apache搭建Web Server，Mysql保存最终数据。用户通过{uin}.qzone.qq.com(uin就是用户的QQ号码)这样的方式访问自己的空间。最开始上线时，仅邀请了电信用户。但上线之后才发现很多北方网通用户也非常希望来体验。但如果让北方网通用户跨网来访问电信的服务，那将是一个非常糟糕的体验。因为当时的跨网之间的带宽是非常有限的。

为了满足网通用户的需求，QQ空间团队在网通搭建了一套与电信一模一样的服务，电信的用户数据只存在电信的服务器上，网通用户数据只存在网通服务器上。看似异地分布，其实两部分数据是互相独立、不能冗余的。在用户注册开通空间时，就确定了他的数据存在于哪个服务器上。

用户访问空间的逻辑抽象来说是这样的（如图2）：用户通过浏览器发起请求访问我们的第一个CGI，这个CGI再串行的获取用户数据，比如：装扮数据，日志数据等。等获取到全部数据之后，这个CGI再将所有的数据拼装成一个完整的HTML页面输出给用户。

图2——Qzone2.0典型载入逻辑

但同时，空间第一版的独立客户端也带来了各种问题和烦恼：

第一：非常不利于定位问题。

一旦页面元素填写失败，定位问题将是一件麻烦的事情。因为它不是Web页面，所以只能用其他抓包软件，比如Ethereal（Wireshark的前身）来抓取网络包做分析。但那个时候网络抓包工具的能力还比较弱，没有高亮和HTML语法检测等功能，所以整体环境比较受限。

第二：服务接入层压力大。

由于不仅要接受用户的请求，还要向后访问不同的数据。当后端某一个接口超时严重的情况下，很多用户的请求都会被挂起。这样就会消耗更多的连接资源，对Web服务器CPU的消耗很大。

第三：服务集群没有发挥出应有的服务能力

当时共用30-40台服务器，也只能支持50万左右的用户。无奈之下，团队想了一个没办法的办法：有意限制超过五十万用户，提出了一个排队等待机制，学习海底捞模式，当在线人数达到50w的时候，QQ空间会给用户展示一个Flash小游戏，让用户在等待的时候玩小游戏。

针对当时存在的种种问题，团队做了以下改进来解决：

第一：采用Web RIA。

当时大量使用了Ajax技术来减轻服务器的负载。其中一个极端的例子是：整个空间首页都是采用JS绘制，这样能够将用户访问的CGI足够轻量化。尽量简化了CGI逻辑，让CGI程序运行更加强壮、效率更高。同时，当服务异常时，还可以通过JS脚本绘制一个错误信息给予用户提示。而且，采用JS开发效率更高，调试更方便。 Web RIA化后，降低了服务器CPU消耗的40%，节约了DC的输出带宽约30%（部分JS脚本都可以被浏览器Cache住）。

第二：采用动静分离策略。

静态资源都放在自研的Web服务器qhttpd上，它具备当时ApacheSelect模型的两个数量级以上的接入能力。

第三：采用自研的Web Server：Qzhttp，主要用于动态分离的动态服务的需要。

由于Qzhttp是腾讯自研的、轻量的、更适合业务自己逻辑的Web Server，从而保证了它的高性能，它的接入能力比Apache (非FastCGI模式)提高了3倍。

石器时代做的最大的优化：

当用户申请QQ空间多个服务时，每个服务都有独立的存储和处理逻辑。只有当所有的服务处理完成，才会由QQ空间框架服务器返回给用户。这样对于同时依赖日志、相册、音乐、留言的服务就会产生两大问题：

  1. 短板效应一损俱损；
  2. 具体业务容量无法差异化。

针对这两个问题，做了首页载入性能优化及首页内容静态化系统两方面的优化：

研究用户访问模型。拉取展现与UGC内容的变更比例是7：1。用户之间互访频繁，看他人与看自己的比例是5：1.。通过以上数据，可以看出用户的UGC更新很少，但是用户之间互相访问很频繁。根据用户的这种访问模型，空间团队研发了一套静态化系统（图3）。这套系统会将用户首页的内容全部Cache，然后根据用户的访问，以及用户自己UGC内容的更改，采用一定策略更新静态化系统的Cache数据。 通过增加首页内容静态化系统，首页展现速度从5s提升到3s，用户也不用再玩小游戏进行等待。同时在设备没有扩容的前提下，同时在线用户访问数突破100w。

图3—首页内容静态化系统

通过不断的改进和优化，QQ空间3.0于2006年中稳健上线。

冷兵器时代——功能的磨砺

冷兵器时代是在线人数从百万阵营向千万阵营过渡的过程。这个阶段做的更多的是功能的磨砺，提高可用性的过程。

但与此同时，QQ空间团队又面临了新的问题：

  1. 网通，教育网用户体验很差；
  2. 版本迭代快速，导致外网bug不断；版本发布之后，所有开发必须留守2个小时；
  3. 后端服务质量不稳定，导致服务经常宕机。

之前有谈到，空间的用户数据是分布在电信与网通两套体系中的。但由于公司在网通的服务设备有限，随着用户量的不断增加，很快就达到了网通设备服务的上限。网通不能扩容设备，但用户量还在不断增加。面对这种情况，只好将网通和电信的数据合并为一套。当网通用户访问服务的时候，会通过一个代理将用户的请求转发到内网电信服务上。采用这种方式，提供了一套通用的解决方案，可以解决海外，网通，教育网，铁通等运营商的问题。

但这种跨网访问的静态资源下载量非常大，静态资源请求次数 : CGI请求数接近10：1，所以采用CDN提供静态资源下载，来提高用户访问速度。具体逻辑是：根据用户客户端IP判断用户属于哪一个ISP服务商，通过URL方式将用户的静态资源访问定向到该ISP的服务设备上。

这样，跳出CDN系统的拘束、优化思路，解决了大部分教育网用户问题，同时多级支持的思路成为公共方案。 但同时还是有很多的故障：

  1. 低ARPU、低成本、低价机器、大集群
  2. 版本迭代快每周发布版本
  3. 用户对故障和低效的容忍度非常低 通过调研，我们发现用户对Web的容忍度是很低的（如图4）：

图4——用户容忍度曲线

一般页面在1s内打开，用户会感觉体验非常流畅。所以我们通过以下策略来保证服务质量：

  1. 区分关键路径,对关键路径的服务质量我们要求4个9以上，非关键路径的服务失败之后做有损体验；
  2. 采用动态超时时间控制算法，保证整个程序在可控的时间内做出响应；
  3. 多级容错机制，从后端服务到CGI，到前台JS脚本都对错误做了容错处理；
  4. 采用柔性可用策略,失败接口采用默认数据。

为了保证版本的服务质量，空间采用了灰度发布策略。新特性可能按照用户尾号发布,每次新特性只对一小部分人可见。一段时间内的用户反馈后，不断修复和改进问题，再逐步扩大用户群对新特性的可见性，最后新特性才对全量用户可见。同时，也通过控制JS版本号来达到了灰度发布的目的。

通过这些优化，QQ空间成功踏入千万级在线的阵营，QQ空间5.0正式上线。其中主要变化：后端服务进行了重构，前端页面也进行了优化。

现代战争时代–数千万在线向亿级在线

经过反复的改进和优化，QQ空间服务质量大幅提高，但是，新的问题还是随之而来：

不满足于发日志、传照片的用户，想要玩游戏、使用某些web工具。同时，公司其它团队要在QQ空间上开发应用，但QQ空间的应用配置都写死在QQ空间平台逻辑里面。为配合其他团队上线，发版本需要一周，工作不可并行，非常耗时，受到了很大挑战。针对这种情况，我们采取了两大措施：平台和应用架构分离（如图5）、简单配置后上线（如图6）。

图5——平台和应用架构分离

图6——简单配置后上线

如果某天深圳的IDC不可用了怎么办？

为了解决这个问题，空间团队做了全国IDC的异地部署，采用“一点写，多点读”模式的架构模型，将服务部署到深圳、上海、天津和西安。深圳是一个写入点，通过QQ空间自己的一套同步系统同步到全国各个IDC（如图7）。同时QQ空间提供了全方位的立体监控，7*24小时对服务做监控。

图7——IDC分布

如何快速发现和解决用户的问题？

定位用户问题需要花很大的时间成本，这就要求团队做很多的监控工作：服务器流量监控、接口调用监控、前端测速监控、前端调用错误监控。

同时，在千万级到亿级在线的过程中，团队要有全国甚至全球的分布和匹配分布的运营能力；出问题可以快速切割，全方位的立体化监控能力；各个逻辑层持续优化能力。只有持续优化，用户才能感知到性能的增加，用户才会增加。

经过无数次的推敲和不断优化，QQ空间的服务能力已经能够支持千万级别的用户在线请求，并且能够给用户提供7*24小时的不间断服务。向亿级在线阵营冲刺也将指日可待！

总结

百万在线：

当时如何撑住服务，让用户进来，从而积累QQ空间第一批用户，根据用户模型进行优化，让QQ空间架构有更好的接入能力，做到高性能。

千万在线：

通过各层的柔性服务和灰度发布的策略，让服务更加稳定，让用户数量级再上一个新台阶。

亿级在线：

服务要有更灵活、更敏捷的变更手段。同时有更好的监控和可运营的能力。

接下来是半个小时的Q&A环节，同学们踊跃提问，徐晓做出了详细专业的解答。最后，徐晓和大家分享了“腾讯《海量服务之道》系列的优质课程和秉承的原则”：

Qzone是全球第二大SNS社区，仅次于Facebook。徐晓说：“海量服务的价值观和用户对我们的厚爱造就了现在的QQ空间。我们会持续的发展，攻克各种难题，不断前进。”

from:http://www.infoq.com/cn/articles/qzone-architecture

数据库架构的升级和变更

SQL Server, 架构,

SQLServer2008在数据的高安全、高性能、高可用方面的技术已经比较成熟，这些技术和方案都是随着很多公司的业务和数据访问压力的增加而不断的升级和

变迁的，同时经历了方方面面的考验，证明了它们都是成熟可靠的，下面就这方面的技术方案和变迁过程来做一些分析。

阶段一：

裸奔时代：

优点：裸奔最大的好处就是简单，成本低。

缺点：一旦服务器出现问题，恢复起来比较麻烦；如果访问压力变大，服务器可能不堪重负。

 

阶段二：

单库+Mirror+BackUp方案：

说明：Mirror有两种方式，同步和异步；同步方式能保证主库和Mirror端数据的一致性，而且不需要使用企业版，但是对主库的性能影响也比较大；异步方式需要

企业版才支持，绝大部分时刻能保证数据的一致性，但是也有丢失小部分的数据可能，不过它主库的影响比较小。

 

优点：此方案对主库的数据提供了可靠的保护，一旦主库出现问题，从库能在比较短的时间内恢复，尤其是数据库很大时（从备份恢复需要的时间会很长），能尽

快的恢复业务使用，而且Mirror端能生成快照，能给实时性要求不高的业务使用。

 

缺点：Mirror会影响主库的部分性能（异步方式影响比较小），主库出现问题后，前端需要更改访问的IP地址（或者将从服务器的IP地址改成主服务器的IP地址），

还需要账号、权限和作业等信息迁移过去。

 

单库+Replication+ BackUp方案：

优点：Replication端可以提供给前段访问，可以将读操作放到从库，分担主库的部分压力，还能提供数据库的备份功能，不过这种备份很可能数据会有丢失。

 

缺点：不能提供安全的数据保护功能，对主库有一些性能影响。

 

阶段三：

单库+Replication+Mirror+BackUp方案：

优点：这种方案是前面两种方案的结合，既能够解决数据保护的问题，也能够提供读写分离的功能。

缺点：主库上既有Mirror又有Replication，这种方式对主库影响会比较大，而且实际证明，Mirror和Replication在同一台机器上部署，在一个出现问题时，

会对另一个造成影响。

 

阶段四：

Cluster（双A）+BackUp方案：

说明：图中矩形部分代表存储，两台服务器做了双A的群集。

 

优点：Cluster能确保其中的一个服务器出现问题时所有的数据和服务能切换到另外一台机器，切换的时间很短，能尽快的恢复业务访问。

缺点：双A群集一般要求配置比较好，价格比较高；因数据都存放在存储上，所以群集不能保护数据，一旦数据或者存储出现问题，需要从备份中恢复数据；

SQLServer的群集不能提供负载均衡的功能。

 

阶段五：

Cluster（双A）+Mirror+BackUp方案：

 

说明：双A群集再加两个服务器上库的Mirror保护。

 

优点：这个方案能对数据提供可靠的保护，无论是服务器故障还是存储故障，都能保证数据的安全，而且数据恢复的时间比较短。

缺点：Mirror会消耗主服务器的部分性能，多了两台Mirror机器，成本会增加，如果存储出现问题，快速恢复的方案是启用Mirror机器，后面可能需要重做群集。

 

阶段六：

Cluster（双A）+Mirror+BackUp+Replication+单分发方案：

 

说明：双A 群集，Mirror保护，单分发机器和读写分离方案。

 

优点：群集和Mirror能充分保护数据的安全，读写分离能提高系统整体的性能。

缺点：成本较高，单分发机存在单点故障，如果分发机器出现问题，将需要重建，此时读和写都将集中到主库，压力会比较大。

 

Cluster（双A）+Mirror+BackUp+Replication+双分发方案：

 

 

优点：与单分发机相比，没有单点故障，即使某台分发机出现问题，也能保证读写分离机制继续运行。

缺点：成本增加，维护方面更复杂。

 

阶段七：

Cluster（双A）+双存储+BackUp+Replication+双分发方案：

 

优点：双存储方案使得数据能得到有效的保护，而且避开了Mirror和Replication同时在主库运行对主服务器造成的影响，节省主服务器资源，而且恢复比较方便。

缺点：成本增加。

 

阶段八：

Cluster（双A）+双存储+BackUp+Replication+双分发+SSB异步方案：

 

此方式的主要优势是将数据流异步处理，缓解瞬时高流量主库的压力，因为此方案比较复杂，暂时不做说明，可以参考数据库架构。

 

阶段九：

拆分业务和数据、采用分布式数据库、使用能负载均衡集群功能的数据库等。

 

此文档大致描述了随公司的发展、服务器压力的增加，数据库架构方面的变迁阶段，当然我们应该根据公司的具体情况，选择性的采用其中的技术，也可能是

直接跳过某些阶段，而上更高效的方案（如果成本能够接受），因此技术和方案的选择应该根据实际情况，灵活应对。

from：http://www.cnblogs.com/fygh/archive/2012/03/23/2413164.html

Introduction to IIS Architecture

IIS, 架构

Introduction

Internet Information Services (IIS) 7 and above provides a request-processing architecture that includes:

  • The Windows Process Activation Service (WAS), which enables sites to use protocols other than HTTP and HTTPS.
  • A Web server engine that can be customized by adding or removing modules.
  • Integrated request-processing pipelines from IIS and ASP.NET.

This article describes the components, modules, and request-processing architecture in the following sections:

Components in IIS

IIS contains several components that perform important functions for the application and Web server roles in Windows Server® 2008 (IIS 7.0) and Windows Server 2008 R2 (IIS 7.5). Each component has responsibilities, such as listening for requests made to the server, managing processes, and reading configuration files. These components include protocol listeners, such as HTTP.sys, and services, such as World Wide Web Publishing Service (WWW service) and Windows Process Activation Service (WAS).

Protocol Listeners

Protocol listeners receive protocol-specific requests, send them to IIS for processing, and then return responses to requestors. For example, when a client browser requests a Web page from the Internet, the HTTP listener, HTTP.sys, picks up the request and sends it to IIS for processing. Once IIS processes the request, HTTP.sys returns a response to the client browser.

By default, IIS provides HTTP.sys as the protocol listener that listens for HTTP and HTTPS requests. HTTP.sys was introduced in IIS 6.0 as an HTTP-specific protocol listener for HTTP requests. HTTP.sys remains the HTTP listener in IIS 7 and above, but includes support for Secure Sockets Layer (SSL).

To support services and applications that use protocols other than HTTP and HTTPS, you can use technologies such as Windows Communication Foundation (WCF). WCF has listener adapters that provide the functionality of both a protocol listener and a listener adapter. Listener adapters are covered later in this document. For more information about WCF, see Windows Communication Foundation on MSDN.

Hypertext Transfer Protocol Stack (HTTP.sys)

The HTTP listener is part of the networking subsystem of Windows operating systems, and it is implemented as a kernel-mode device driver called the HTTP protocol stack (HTTP.sys). HTTP.sys listens for HTTP requests from the network, passes the requests onto IIS for processing, and then returns processed responses to client browsers.

In IIS 6.0, HTTP.sys replaced Windows Sockets API (Winsock), which was a user-mode component used by previous versions of IIS to receive HTTP requests and send HTTP responses. IIS 7 and above continue to rely on HTTP.sys for HTTP requests.

HTTP.sys provides the following benefits:

  • Kernel-mode caching. Requests for cached responses are served without switching to user mode.
  • Kernel-mode request queuing. Requests cause less overhead in context switching because the kernel forwards requests directly to the correct worker process. If no worker process is available to accept a request, the kernel-mode request queue holds the request until a worker process picks it up.
  • Request pre-processing and security filtering.

World Wide Web Publishing Service (WWW service)

In IIS 7 and above, functionality that was previously handled by the World Wide Web Publishing Service (WWW Service) alone is now split between two services: WWW Service and a new service, Windows Process Activation Service (WAS). These two services run as LocalSystem in the same Svchost.exe process, and share the same binaries.

Note   You may also see the WWW Service referred to as W3SVC in documentation.

How WWW Service works in IIS 6.0

In IIS 6.0, WWW Service manages the following main areas in IIS:

  • HTTP administration and configuration
  • Process management
  • Performance monitoring

HTTP Administration and Configuration

The WWW Service reads configuration information from the IIS metabase and uses that information to configure and update the HTTP listener, HTTP.sys. In addition, WWW service starts, stops, monitors, and manages worker processes that process HTTP requests.

Performance Monitoring

The WWW Service monitors performance and provides performance counters for Web sites and for the IIS cache.

Process Management

The WWW Service manages application pools and worker processes, such as starting, stopping, and recycling worker processes. Additionally, the WWW Service monitors the health of the worker processes, and invokes rapid fail detection to stop new processes from starting when several worker processes fail in a configurable amount of time.

How the WWW Service works in IIS

In IIS, the WWW service no longer manages worker processes. Instead, the WWW Service is the listener adapter for the HTTP listener, HTTP.sys. As the listener adapter, the WWW Service is primarily responsible for configuring HTTP.sys, updating HTTP.sys when configuration changes, and notifying WAS when a request enters the request queue.

Additionally, the WWW Service continues to collect the counters for Web sites. Because performance counters remain part of the WWW Service, they are HTTP specific and do not apply to WAS.

Windows Process Activation Service (WAS)

In IIS 7 and above, Windows Process Activation Service (WAS) manages application pool configuration and worker processes instead of the WWW Service. This enables you to use the same configuration and process model for HTTP and non-HTTP sites.

Additionally, you can run WAS without the WWW Service if you do not need HTTP functionality. For example, you can manage a Web service through a WCF listener adapter, such as NetTcpActivator, without running the WWW Service if you do not need to listen for HTTP requests in HTTP.sys. For information about WCF listener adapters and about how to host WCF applications in IIS 7 and above by using WAS, see Hosting in WCF on MSDN.

Configuration Management in WAS

On startup, WAS reads certain information from the ApplicationHost.config file, and passes that information to listener adapters on the server. Listener adapters are components that establish communication between WAS and protocol listeners, such as HTTP.sys. Once listener adapters receive configuration information, they configure their related protocol listeners and prepare the listeners to listen for requests.

In the case of WCF, a listener adapter includes the functionality of a protocol listener. So, a WCF listener adapter, such as NetTcpActivator, is configured based on information from WAS. Once NetTcpActivator is configured, it listens for requests that use the net.tcp protocol. For more information about WCF listener adapters, see WAS Activation Architecture on MSDN.

The following list describes the type of information that WAS reads from configuration:

  • Global configuration information
  • Protocol configuration information for both HTTP and non-HTTP protocols
  • Application pool configuration, such as the process account information
  • Site configuration, such as bindings and applications
  • Application configuration, such as the enabled protocols and the application pools to which the applications belong

If ApplicationHost.config changes, WAS receives a notification and updates the listener adapters with the new information.

Process Management

WAS manages application pools and worker processes for both HTTP and non-HTTP requests. When a protocol listener picks up a client request, WAS determines if a worker process is running or not. If an application pool already has a worker process that is servicing requests, the listener adapter passes the request onto the worker process for processing. If there is no worker process in the application pool, WAS will start a worker process so that the listener adapter can pass the request to it for processing.

Note: Because WAS manages processes for both HTTP and non-HTTP protocols, you can run applications with different protocols in the same application pool. For example, you can develop an application, such as an XML service, and host it over both HTTP and net.tcp.

 

Modules in IIS

IIS provides a new architecture that is different from previous versions of IIS. Instead of keeping the majority of functionality within the server itself, IIS include a Web server engine in which you can add or remove components, called modules, depending on your needs.

Modules are individual features that the server uses to process requests. For example, IIS uses authentication modules to authenticate client credentials, and cache modules to manage cache activity.

The new architecture provides the following advantages over previous versions of IIS:

  • You can control which modules you want on the server.
  • You can customize a server to a specific role in your environment.
  • You can use custom modules to replace existing modules or to introduce new features.

The new architecture also improves security and simplifies administration. By removing unnecessary modules, you reduce the server’s attack surface and memory footprint, which is the amount of memory that server worker processes use on the machine. You also eliminate the need to manage features that are unnecessary for your sites and applications.

Native Modules

The following sections describe the native modules that are available with a full installation of IIS 7 and above. You can remove them or replace them with custom modules, depending on your needs.

HTTP Modules

Several modules in IIS 7 and above perform tasks specific to Hypertext Transfer Protocol (HTTP) in the request-processing pipeline. HTTP modules include modules to respond to information and inquiries sent in client headers, to return HTTP errors, to redirect requests, and more.

Module Name Description Resource
CustomErrorModule Sends default and configured HTTP error messages when an error status code is set on a response. Inetsrv\Custerr.dll
HttpRedirectionModule Supports configurable redirection for HTTP requests. Inetsrv\Redirect.dll
ProtocolSupportModule Performs protocol-related actions, such as setting response headers and redirecting headers based on configuration. Inetsrv\Protsup.dll
RequestFilteringModule Added in IIS 7.5. Filters requests as configured to control protocol and content behavior. Inetsrv\modrqflt.dll
WebDAVModule Added in IIS 7.5. Allows more secure publishing of content by using HTTP over SSL. Inetsrv\WebDAV.dll

Security Modules

Several modules in IIS perform tasks related to security in the request-processing pipeline. In addition, there are separate modules for each of the authentication schemes, which enable you to select modules for the types of authentication you want on your server. There are also modules that perform URL authorization, and a module that filters requests.

Module Name Description Resource
AnonymousAuthenticationModule Performs Anonymous authentication when no other authentication method succeeds. Inetsrv\Authanon.dll
BasicAuthenticationModule Performs Basic authentication. Inetsrv\Authbas.dll
CertificateMappingAuthenticationModule Performs Certificate Mapping authentication using Active Directory. Inetsrv\Authcert.dll
DigestAuthenticationModule Performs Digest authentication. Inetsrv\Authmd5.dll
IISCertificateMappingAuthenticationModule Performs Certificate Mapping authentication using IIS certificate configuration. Inetsrv\Authmap.dll
RequestFilteringModule Performs URLScan tasks such as configuring allowed verbs and file name extensions, setting limits, and scanning for bad character sequences. Inetsrv\Modrqflt.dll
UrlAuthorizationModule Performs URL authorization. Inetsrv\Urlauthz.dll
WindowsAuthenticationModule Performs NTLM integrated authentication. Inetsrv\Authsspi.dll
IpRestrictionModule Restricts IPv4 addresses listed in the ipSecurity list in configuration. Inetsrv\iprestr.dll

Content Modules

Several modules in IIS perform tasks related to content in the request-processing pipeline. Content modules include modules to process requests for static files, to return a default page when a client doesn’t specify a resource in a request, to list the contents of a directory, and more.

Module Name Description Resource
CgiModule Executes Common Gateway Interface (CGI) processes to build response output. Inetsrv\Cgi.dll
DefaultDocumentModule Attempts to return a default document for requests made to the parent directory. Inetsrv\Defdoc.dll
DirectoryListingModule Lists the contents of a directory. Inetsrv\dirlist.dll
IsapiModule Hosts ISAPI extension DLLs. Inetsrv\Isapi.dll
IsapiFilterModule Supports ISAPI filter DLLs. Inetsrv\Filter.dll
ServerSideIncludeModule Processes server-side includes code. Inetsrv\Iis_ssi.dll
StaticFileModule Serves static files. Inetsrv\Static.dll
FastCgiModule Supports FastCGI, which provides a high-performance alternative to CGI. Inetsrv\iisfcgi.dll

Compression Modules

Two modules in IIS perform compression in the request-processing pipeline.

Module Name Description Resource
DynamicCompressionModule Compresses responses and applies Gzip compression transfer coding to responses. Inetsrv\Compdyn.dll
StaticCompressionModule Performs pre-compression of static content. Inetsrv\Compstat.dll

Caching Modules

Several modules in IIS perform tasks related to caching in the request-processing pipeline. Caching improves the performance of your Web sites and Web applications by storing processed information, such as Web pages, in memory on the server, and then reusing that information in subsequent requests for the same resource.

Module Name Description Resource
FileCacheModule Provides user mode caching for files and file handles. Inetsrv\Cachfile.dll
HTTPCacheModule Provides kernel mode and user mode caching in HTTP.sys. Inetsrv\Cachhttp.dll
TokenCacheModule Provides user mode caching of user name and token pairs for modules that produce Windows user principals. Inetsrv\Cachtokn.dll
UriCacheModule Provides user mode caching of URL information. Inetsrv\Cachuri.dll

Logging and Diagnostics Modules

Several modules in IIS perform tasks related to logging and diagnostics in the request-processing pipeline. The logging modules support loading of custom modules and passing information to HTTP.sys. The diagnostics modules follow and report events during request processing.

Module Name Description Resource
CustomLoggingModule Loads custom logging modules. Inetsrv\Logcust.dll
FailedRequestsTracingModule Supports the Failed Request Tracing feature. Inetsrv\Iisfreb.dll
HttpLoggingModule Passes information and processing status to HTTP.sys for logging. Inetsrv\Loghttp.dll
RequestMonitorModule Tracks requests currently executing in worker processes and reports information with Runtime Status and Control Application Programming Interface (RSCA). Inetsrv\Iisreqs.dll
TracingModule Reports events to Microsoft Event Tracing for Windows (ETW). Inetsrv\Iisetw.dll

Managed Support Modules

A couple of modules in IIS support managed integration in the IIS request-processing pipeline.

Module Name Description Resource
ManagedEngine Provides integration of managed code modules in the IIS request-processing pipeline. Microsoft.NET\Framework\v2.0.50727\webengine.dll
ConfigurationValidationModule Validates configuration issues, such as when an application is running in Integrated mode but has handlers or modules declared in the system.web section. Inetsrv\validcfg.dll

Managed Modules

In addition to native modules, IIS enables you to use managed code modules to extend IIS functionality. Some of the managed modules, such as UrlAuthorization, have a native module counterpart that provides a native alternative to the managed module.

Note   Managed modules depend on the ManagedEngine module.

The following table lists the managed modules that are available with a full installation of IIS 7 and above. For more information about the managed modules, see the .NET Framework SDK 2.0 on MSDN.

Module Name Description Resource
AnonymousIdentification Manages anonymous identifiers, which are used by features that support anonymous identification such as ASP.NET profile. System.Web.Security.AnonymousIdentificationModule
DefaultAuthentication Ensures that an authentication object is present in the context. System.Web.Security.DefaultAuthenticationModule
FileAuthorization Verifies that a user has permission to access the requested file. System.Web.Security.FileAuthorizationModule
FormsAuthentication Supports authentication by using Forms authentication. System.Web.Security.FormsAuthenticationModule
OutputCache Supports output caching. System.Web.Caching.OutputCacheModule
Profile Manages user profiles by using ASP.NET profile, which stores and retrieves user settings in a data source such as a database. System.Web.Profile.ProfileModule
RoleManager Manages a RolePrincipal instance for the current user. System.Web.Security.RoleManagerModule
Session Supports maintaining session state, which enables storage of data specific to a single client within an application on the server. System.Web.SessionState.SessionStateModule
UrlAuthorization Determines whether the current user is permitted access to the requested URL, based on the user name or the list of roles of which a user is a member. System.Web.Security.UrlAuthorizationModule
UrlMappingsModule Supports mapping a real URL to a more user-friendly URL. System.Web.UrlMappingsModule
WindowsAuthentication Sets the identity of the user for an ASP.NET application when Windows authentication is enabled. System.Web.Security.WindowsAuthenticationModule

 

Request Processing in IIS

In IIS, the IIS and ASP.NET request pipelines combine to process requests with an integrated approach. The new request-processing architecture consists of an ordered list of native and managed modules that perform specific tasks in response to requests.

This design provides several benefits over previous versions of IIS. First, all file types can use features that were originally available only to managed code. For example, you can now use ASP.NET Forms authentication and Uniform Resource Locator (URL) authorization for static files, Active Server Pages (ASP) files, and all other file types in your sites and applications.

Second, this design eliminates the duplication of several features in IIS and ASP.NET. For example, when a client requests a managed file, the server calls the appropriate authentication module in the integrated pipeline to authenticate the client. In previous versions of IIS, this same request would go through an authentication process in both the IIS pipeline and in the ASP.NET pipeline.

Third, you can manage all of the modules in one location, instead of managing some features in IIS and some in the ASP.NET configuration. This simplifies the administration of sites and applications on the server.

Application Pools in IIS

Application pools separate applications by process boundaries to prevent an application from affecting another application on the server. In IIS 7 and above, application pools continue to use IIS 6.0 worker process isolation mode. In addition, you can now specify a setting that determines how to process requests that involve managed resources: Integrated mode or Classic mode.

Note: In IIS 6.0, worker process isolation mode and IIS 5.0 isolation mode are set at the server level. This makes it impossible to run both isolation modes on the same server. However, in IIS 7 and above, Integrated mode and Classic mode are set at the application pool level, which enables you to run applications simultaneously in application pools with different process modes on the same server.

Integrated application pool mode

When an application pool is in Integrated mode, you can take advantage of the integrated request-processing architecture of IIS and ASP.NET. When a worker process in an application pool receives a request, the request passes through an ordered list of events. Each event calls the necessary native and managed modules to process portions of the request and to generate the response.

There are several benefits to running application pools in Integrated mode. First the request-processing models of IIS and ASP.NET are integrated into a unified process model. This model eliminates steps that were previously duplicated in IIS and ASP.NET, such as authentication. Additionally, Integrated mode enables the availability of managed features to all content types.

Classic application pool mode

When an application pool is in Classic mode, IIS 7 and above handles requests in the same way as in IIS 6.0 worker process isolation mode. ASP.NET requests first go through native processing steps in IIS and are then routed to Aspnet_isapi.dll for processing of managed code in the managed runtime. Finally, the request is routed back through IIS to send the response.

This separation of the IIS and ASP.NET request-processing models results in duplication of some processing steps, such as authentication and authorization. Additionally, managed code features, such as Forms authentication, are only available to ASP.NET applications or applications for which you have script mapped all requests to be handled by aspnet_isapi.dll.

Be sure to test your existing applications for compatibility in Integrated mode before upgrading a production environment to IIS 7 and above and assigning applications to application pools in Integrated mode. You should only add an application to an application pool in Classic mode if the application fails to work in Integrated mode. For example, your application might rely on an authentication token passed from IIS to the managed runtime, and, due to the new architecture in IIS 7 and above, the process breaks your application.

HTTP Request Processing in IIS

IIS 7 and above have a similar HTTP request-processing flow as IIS 6.0. The diagrams in this section provide an overview of an HTTP request in process.

The following list describes the request-processing flow that is shown in Figure 1:

  1. When a client browser initiates an HTTP request for a resource on the Web server, HTTP.sys intercepts the request.
  2. HTTP.sys contacts WAS to obtain information from the configuration store.
  3. WAS requests configuration information from the configuration store, applicationHost.config.
  4. The WWW Service receives configuration information, such as application pool and site configuration.
  5. The WWW Service uses the configuration information to configure HTTP.sys.
  6. WAS starts a worker process for the application pool to which the request was made.
  7. The worker process processes the request and returns a response to HTTP.sys.
  8. The client receives a response.

Figure 1: Overview of an HTTP Request In a worker process, an HTTP request passes through several ordered steps, called events, in the Web Server Core. At each event, a native module processes part of the request, such as authenticating the user or adding information to the event log. If a request requires a managed module, the native ManagedEngine module creates an AppDomain, where the managed module can perform the necessary processing, such as authenticating a user with Forms authentication. When the request passes through all of the events in the Web Server Core, the response is returned to HTTP.sys. Figure 2, below, shows an HTTP request entering the worker process.

Figure 2: Detail of a HTTP request inside the Worker Process

Stack Overflow Architecture Update – Now at 95 Million Page Views a Month

架构,

A lot has happened since my first article on the Stack Overflow Architecture. Contrary to the theme of that last article, which lavished attention on Stack Overflow’s dedication to a scale-up strategy, Stack Overflow has both grown up and out in the last few years.

Stack Overflow has grown up by more then doubling in size to over 16 million users and multiplying its number of page views nearly 6 times to 95 million page views a month.

Stack Overflow has grown out by expanding into the Stack Exchange Network, which includes Stack Overflow, Server Fault, and Super User for a grand total of 43 different sites. That’s a lot of fruitful multiplying going on.

What hasn’t changed is Stack Overflow’s openness about what they are doing. And that’s what prompted this update. A recent series of posts talks a lot about how they’ve been handling their growth: Stack Exchange’s Architecture in Bullet Points, Stack Overflow’s New York Data Center, Designing For Scalability of Management and Fault Tolerance, Stack Overflow Search — Now 81% Less, Stack Overflow Network Configuration, Does StackOverflow use caching and if so, how?, Which tools and technologies build the Stack Exchange Network?.

Some of the more obvious differences across time are:

  • Just More. More users, more page views, more datacenters, more sites, more developers, more operating systems, more databases, more machines. Just a lot more of more.
  • Linux. Stack Overflow was known for their Windows stack, now they are using a lot more Linux machines for HAProxy, Redis, Bacula, Nagios, logs, and routers. All support functions seem to be handled by Linux, which has required the development of parallel release processes.
  • Fault Tolerance. Stack Overflow is now being served by two different switches on two different internet connections, they’ve added redundant machines, and some functions have moved to a second datacenter.
  • NoSQL. Redis is now used as a caching layer for the entire network. There wasn’t a separate caching tier before so this a big change, as is using a NoSQL database on Linux.

Unfortunately, I couldn’t find any coverage on some of the open questions I had last time, like how they were going to deal with multi-tenancy across so many diffrent properties, but there’s still plenty to learn from. Here’s a roll up a few different sources:

The Stats

  • 95 Million Page Views a Month
  • 800 HTTP requests a second
  • 180 DNS requests a second
  • 55 Megabits per second
  • 16 Million Users  – Traffic to Stack Overflow grew 131% in 2010, to 16.6 million global monthly uniques.

Data Centers

  • 1 Rack with Peak Internet in OR (Hosts our chat and Data Explorer)
  • 2 Racks with Peer 1 in NY (Hosts the rest of the Stack Exchange Network)

Hardware

  • 10 Dell R610 IIS web servers (3 dedicated to Stack Overflow):
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz Quad Core with 8 threads
    • 16 GB RAM
    • Windows Server 2008 R2
  • 2 Dell R710 database servers:
    • 2x Intel Xeon Processor X5680 @ 3.33 GHz
    • 64 GB RAM
    • 8 spindles
    • SQL Server 2008 R2
  • 2 Dell R610  HAProxy servers:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 4 GB RAM
    • Ubuntu Server
  • 2 Dell R610 Redis servers:
    • 2x Intel Xeon Processor E5640 @ 2.66 GHz
    • 16 GB RAM
    • CentOS
  • 1 Dell R610 Linux backup server running Bacula:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 32 GB RAM
  • 1 Dell R610 Linux management server for Nagios and logs:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 32 GB RAM
  • 2 Dell R610 VMWare ESXi domain controllers:
    • 1x Intel Xeon Processor E5640 @ 2.66 GHz
    • 16 GB RAM
  • 2 Linux routers
  • 5 Dell Power Connect switches

Dev Tools

  • C#: Language
  • Visual Studio 2010 Team Suite: IDE
  • Microsoft ASP.NET (version 4.0): Framework
  • ASP.NET MVC 3: Web Framework
  • Razor: View Engine
  • jQuery 1.4.2: Browser Framework:
  • LINQ to SQL, some raw SQL: Data Access Layer
  • Mercurial and Kiln: Source Control
  • Beyond Compare 3: Compare Tool

Software and Technologies Used

  • Stack Overflow uses a WISC stack via BizSpark
  • Windows Server 2008 R2 x64: Operating System
  • SQL Server 2008 R2 running Microsoft Windows Server 2008 Enterprise Edition x64: Database
  • Ubuntu Server
  • CentOS
  • IIS 7.0: Web Server
  • HAProxy: for load balancing
  • Redis: used as the distributed caching layer.
  • CruiseControl.NET: for builds and automated deployment
  • Lucene.NET:  for search
  • Bacula: for backups
  • Nagios: (with n2rrd and drraw plugins) for monitoring
  • Splunk: for logs
  • SQL Monitor: from Red Gate – for SQL Server monitoring
  • Bind: for DNS
  • Rovio:  a little robot (a real robot) allowing remote developers to visit the office “virtually.”
  • Pingdom:  an external monitor and alert service.

External Bits

Code that is not included as part of the development tools:

  • reCAPTCHA
  • DotNetOpenId
  • WMD – Now developed as open source. See github network graph
  • Prettify
  • Google Analytics
  • Cruise Control .NET
  • HAProxy
  • Cacti
  • MarkdownSharp
  • Flot
  • Nginx
  • Kiln
  • CDN: none, all static content is served off the sstatic.net, which is a fast, cookieless domain intended for static content delivered to the Stack Exchange family of websites.

Developers and System Administrators

  • 14 Developers
  • 2 System Administrators

Content

  • License: Creative Commons Attribution-Share Alike 2.5 Generic
  • Standards: OpenSearch, Atom
  • Host: PEAK Internet

More Architecture and Lessons Learned

  • HAProxy is used instead of Windows NLB because HAProxy is cheap, easy, free, works great as a 512MB VM “device” on a network via Hyper-V. It also works in front of the boxes so it’s completely transparent to them, and easier to troubleshoot as a different networking layer instead of being intermixed with all your windows configuration.
  • A CDN is not used because even “cheap” CDNs like Amazon one are very expensive relative to the bandwidth they get bundled into their existing host’s plan. The least they could pay is $1k/month based on Amazon’s CDN rates and their bandwidth usage.
  • Backup is to disk for fast retrieval and to tape for historical archiving.
  • Full Text Search in SQL Server is very badly integrated, buggy, deeply incompetent, so they went to Lucene.
  • Mostly interested in peak HTTP request figures as this is what they need to make sure they can handle.
  • All properties now run on the same Stack Exchange platform. That means Stack Overflow, Super User, Server Fault, Meta, WebApps, and Meta Web Apps are all running on the same software.
  • There are separate StackExchange sites because people have different sets of expertise that shouldn’t cross over to different topic sites. You can be the greatest chef in the world, but that doesn’t qualify you for fixing a server.
  • They aggressively cache everything.
  • All pages accessed by (and subsequently served to) annonymous users are cached via Output Caching.
  • Each site has 3 distinct caches: local, site, global.
  • local cache: can only be accessed from 1 server/site pair
    • To limit network latency they use a local “L1” cache, basically HttpRuntime.Cache, of recently set/read values on a server. This would reduce the cache lookup overhead to 0 bytes on the network.
    • Contains things like user sessions, and pending view count updates.
    • This resides purely in memory, no network or DB access.
  • site cache:  can be accessed by any instance (on any server) of a single site
    • Most cached values go here, things like hot question id lists and user acceptance rates are good examples
    • This resides in Redis (in a distinct DB, purely for easier debugging)
    • Redis is so fast that the slowest part of a cache lookup is the time spent reading and writing bytes to the network.
    • Values are compressed before sending them to Redis. They have plenty of CPU and most of their data are strings so they get a great compression ratio.
    • The CPU usage on their Redis machines is 0%.
  • global cache: which is shared amongst all sites and servers
    • Inboxes, API usage quotas, and a few other truly global things live here
    • This resides in Redis (in DB 0, likewise for easier debugging)
  • Most items in the cache expire after a timeout period (a few minutes usually) and are never explicitly removed. When a specific cache invalidation is required they use Redis messaging to publish removal notices to the “L1” caches.
  • Joel Spolsky is not a Microsoft Loyalist, he doesn’t make the technical decisions for Stack Overflow, and considers Microsoft licensing a rounding error. Consider yourself corrected Hacker News commentor.
  • For their IO system they selected a RAID 10 array of Intel X25 solid state drives . The RAID array eased any concerns about reliability and the SSD drives performed really well in comparision to FusionIO at a much cheaper price.
  • The full boat cost for their Microsoft licenses would be approximately $242K. Since Stack Overflow is using Bizspark they are not paying near the full sticker price, but that’s the max they could pay.
  • Intel NICs are replacing Broadcom NICs and their primary production servers. This solved problems they were having with  connectivity loss, packet loss, and corrupted arp tables.

Related Articles