Re-arquitetando oRe-arquitetando oStack OverflowStack Overflow

ou como construímos o Stack Overflow for Teams

Roberta Arcoverde1

/whois /whois 

recifenseprogramadora há 15 anosprincipal softwaredeveloper na stackoverflowco-host do hipsters.tech@rla4

2

desde 200850+ milhões de usuários únicos/mês18 milhões de perguntas27 milhões de respostastop 50 sites mais acessados do mundo

3

3k Teams criados, 50k usuários10 meses em desenvolvimentolançado em maio/2018equipe tinha originalmente 3 devs,agora são 7

melhor nome de time da história:Teams Team �

4

https://stackoverflow.com/c/demo 5

https://stackoverflow.com/c/demo 5

https://stackoverflow.com/c/demo 5

https://stackoverflow.com/c/demo 5

https://stackoverflow.com/c/demo 5

https://stackoverflow.com/c/demo 5

https://stackoverflow.com 6

https://stackoverflow.com 6

>170 sites>170 sites

7

números do dia 03/05números do dia 03/05

278.912.108 HTTP requests67.188.355 page views3.506.670.995.363 bytes (3.5 TB) enviados953.860.308 SQL queries executadas5.250.697.564 redis hits600.000 websockets ativos19ms de tempo de renderização da Question page

54.290.431 page views, ou 80% do total123ms de tempo de renderização geral

8

9 WEB SERVERS

4 SQL SERVERS

LIVE HOT STANDBY LIVE HOT STANDBY

9

Stack Exchange, Meta, TalentStack Overflow

~350 req/s

por servidor

528 Mqueries/dia

498 Mqueries/dia

~5% CPU

imagem gentilmente cedida por Marco (@sklivvz) em http://www.slideshare.net/howtoweb/marco-cecconi-stack-overflow-architecture

10

11

como?como? 

spoilers: é boring12

performance performance é umaé umafeaturefeature

13

tech stacktech stackc#asp.net mvc*sql server

dapper, ef coretypescript

vanillarediselasticsearchha proxy

*migrando pra .NET Core

14

15

�♀ 

15

multi tenant applicationmulti tenant application

um único app pool paratodos os sitesroteado via host headers

16

17

https://nickcraver.com/blog/2016/02/03/stack-overflow-a-technical-deconstruction/ 18

Q&A pra dadosQ&A pra dadosprivados?privados?

19

(o nome original do SO for Teams era Channels)

nasce uma ideia! (sim, o screenshot é legítimo)20

times são sites que existem

dentro do Stack Overflowtratá-los como se fossem novos sites narede, porém visíveis apenas a partir do 

public class Post { public int Id { get; } public string Title { get; } public int? TeamId { get; } ... } // reusar banco // criar novo código

public class Post { public int Id { get; } public string Title { get; } ... } // criar novo banco // reusar código

21

[StackRoute("help/search-inline")]public async Task<ActionResult> SearchInline(string q){ var searchSite = GetSearchSite(); var results = await searchSite.HelpPostIndex.SearchAsync(searchSite, q); var sm = new SearchModel { SearchString = q, Results = results }; return PartialView("~/Views/Help/SearchInline.cshtml", sm);}

123456789

10111213

https://stackoverflow.com/help/search-inlinehttps://askubuntu.com/help/search-inlinehttps://stackoverflow.com/c/demo/help/search-inline

22

ModeloModeloevitar forks, DRY, minimizar alterações no core do projeto

23

ModeloModelo

EscalabilidadeEscalabilidade

evitar forks, DRY, minimizar alterações no core do projeto

capacity planning, o que acontece se tivermos 1k, 10k, 100ktimes?

23

ModeloModelo

SegurançaSegurança

EscalabilidadeEscalabilidade

evitar forks, DRY, minimizar alterações no core do projeto

default private, mudança de mindset, crash na aplicação >vazamento de dados

capacity planning, o que acontece se tivermos 1k, 10k, 100ktimes?

23

��

Bases isoladas entreTeamsDados isolados dosdados públicosMínimo de alteraçõesno código (usar modeloexistente pra novossites)

��

Escalabilidade. AGdistribuídos começam adegradar rapidamentea partir de 1k bancosHardware einstrumentação paragerenciar milhares debases de dados

Plano A: um banco paraPlano A: um banco paracada Teamcada Team

24

��

EscalabilidadeDados isolados dosdados públicos

��

Sem isolamento entreTeamsReescrever boa partedas consultasConsultas não são maisas mesmas para sites vsTeams

Plano B: um banco paraPlano B: um banco paratodos os Teamstodos os Teams

25

��

Dados isolados entreTeamsDados isolados dosdados públicosEscalabilidade é...decenteBaixo custo de reescrita

��

Precisamos escreverinfra deprovisionamentodinâmico

Plano C: um schema porPlano C: um schema portime no mesmo bancotime no mesmo banco

26

27

28

basicamente: saindo de 170 para 10k+ sites

SQL Server1 banco per-site1 banco pra todos os Teams, 1 schema per-Team

Elasticsearch1 índice per-site1 índice per-team, até 5k

Provisionamentotarefa agendada cria sempre um buffer de 100schemas para futuros Teams

EscalabilidadeEscalabilidade

29

onde manter os dados dos Teams?como comunicar o site público com o Team?migrar *tudo* pra lugares seguros

notificaçõesemailsmonitoramentointernal APIwebsocketstags

SegurançaSegurança

30

31

como as redes secomo as redes secomunicam?comunicam?

32

ProxyingProxyingJá usávamos no /jobsRequisição é "clonada" e enviada para a CFZResponse é jogada direto no stream de saída800 LoCPor que não usar APIs/serviços?

custo de serializaçãomais código, menos uniformidade

33

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

123456789

101112131415161718192021222324252627282930313233

34

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

123456789

101112131415161718192021222324252627282930313233

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]

12

public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

34

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

123456789

101112131415161718192021222324252627282930313233

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]

12

public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9

101112131415

16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

34

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

123456789

101112131415161718192021222324252627282930313233

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]

12

public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9

101112131415

16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

return await this.BlindProxy(channelSite, path);

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24

25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

34

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

123456789

101112131415161718192021222324252627282930313233

[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]

12

public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9

101112131415

16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

return await this.BlindProxy(channelSite, path);

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24

25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33

// BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D

[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27

282930313233

34

// No, you can't: // - Use a CookieCollection (it'll get headers, but not pass them here) // - Set the Set-Cookie header on the response (ASP.Net strips it) // - Set an additional Set-Cookie (also stripped) // - Take the raw header and pass it (comma delimited, only the first cookie wil // - Use Headers.GetValues(string) (it screws up on commas) // - Maintain your sanity working with ASP.Net and cookie headers // Fun fact: half of the cookie BS here is supporting IIS6 and IE5. Not kidding. if (cResponse.Headers["Set-Cookie"].HasValue()) { var nvc = cResponse.Headers; var result = new List<string>(); for (var i = 0; i < nvc.Count; i++) { if (nvc.GetKey(i) == "Set-Cookie") { // Don't ask. You'll cry. var vals = nvc.GetValues(i); if (vals != null) result.AddRange(vals); } } // ... } 35

liçõesliçõesentenda seus cenários de escalabilidadequando não souber: capacity planningsegurança vai além de proteger dados de acessoexterno

36

outras palestrasoutras palestrasinstrumentação

adaptamos todos os nossos sistemas demonitoramento pra incluir Teams

proxy v2protobufgrpcstructured model

single sign-onre-arquitetando o modelo de autenticação eautorização

modelo de segurançadados (perguntas, respostas, tags)metadados (traffic logs, IPs, urls)external endpoints (ads, APIs, emails) 37

obrigada!obrigada!

rla4roberta at stackoverflow.comrla4.comhipsters.tech 38

instância privada, standalone do StackOverflowSLA, priority supportsingle sign-onon premise ou Azurereleases trimestraiscompletamente customizávelapropriado para grandes empresas$$$

39