Linux numa evolution

Fredrik Teschke, Lukas Pirl seminar on NUMA, Hasso Plattner Institue, Potsdam

Fredrik Teschke, Lukas Pirl seminar on NUMA, Hasso Plattner Institue, Potsdam 2


http://storage.pardot.com/6342/95370/lf_pub_top500report.pdf

http://upload.wikimedia.org/wikipedia/commons/e/e1/Linus_Torvalds,_2002,

_Australian_Linux_conference.jpg

UNIX Linux

3



`diff -up`

https://www.kernel.org/doc/Documentation/SubmittingPatches (20.11.2014)

5

Fredrik Teschke, Lukas Pirl seminar on NUMA, Hasso Plattner Institue, Potsdam http://thread.gmane.org/gmane.linux.kernel/1392753 6


http://upload.wikimedia.org/wikipedia/commons/e/e1/Linus_Torvalds,_2002,_Australian_Linux_conference.jpg

…


https://www.kernel.org/doc/Documentation/development-process/2.Process, http://www.linuxfoundation.org/sites/main/files/publications/whowriteslinux.pdf

8


https://www.kernel.org/doc/Documentation/development-process/2.Process

“9





http://en.wikipedia.org/wiki/Scheduling_%28computing%29

13


http://lwn.net/Articles/254445/

?

14



✔

✘

https://www.cs.sfu.ca/~fedorova/papers/usenix-numa.pdf

16


✔

✔

✘

https://www.cs.sfu.ca/~fedorova/papers/asplos284-dashti.pdf

Traffic Imbalance

17


→

18



…

20


http://lse.sourceforge.net/numa/topology_api/in-kernel/

21


asm/topology.h

int __cpu_to_node(int cpu);

int __memblk_to_node(int memblk);

unsigned long __node_to_cpu_mask(int node);

int __parent_node(int node); # /!\ supports hierarchies

http://lse.sourceforge.net/numa/topology_api/in-kernel/

22



int __cpu_to_node(int cpu);

http://home.arcor.de/efocht/sched/

keep task

& mem on same node

24


⇒

L = local_node();

# regular load balancing as for multicore

# (O(1) scheduler):

balance_node(L);

N = most_loaded_node();

C = most_loaded_cpu(N);

if load(L) <= system_load()

steal_tasks_from_cpu(C);

http://home.arcor.de/efocht/sched/

25




27


→


Node

Physical CPU

CPU0 CPU1

Physical CPU

CPU2 CPU3

Node Node Node

minimize cost

of moving task (& mem)

28


node node

node node

29


…

node

node node

node

node

node

30


…

node

node node

node

node

node

31


…

node

node node

node

node

node

32


…

33


?

34


…

mem

node

mem

node

35


…

mem

node

mem

node

36


…

mem

node

mem

node

37


…

mem mem

nodenode

38


…

mem

node

mem

node

39


…

mem

node

mem

node

40


…

mem

node

mem

node

41


⇒


42



mem follows task

43



task follows mem

44


http://lwn.net/Articles/486858/tasks w/ shared

mem on same node

45



46



memmem

nodenode

⚡

⚡⚡

47


Unless you're going to listen to feedback I give you, I'm going to completely stop reading your patches, I don't give a rats arse you work for the same company anymore. You're impossible to work with.


“48


http://lwn.net/Articles/522093/, http://lwn.net/Articles/524535/

49



http://lwn.net/Articles/524977/, http://thread.gmane.org/gmane.linux.kernel/1392753

51



autonuma

53


⚡

http://lwn.net/Articles/568870/scheduling: NUMA

and load balancing and groups

54


http://thread.gmane.org/gmane.linux.kernel/1631332

mem

node

mem

node

55



mem

node

mem

node

56



57





60


⇔

∃


node node

node node

61


⇔


backplanecontroller

node

backplanecontroller

backplanecontroller

backplane

node

node

node

node

node 62


…


63



64



65



node node

mem

NIC

DMA?IO

adapter

mem

66





→ ≈ →

CPU

CPU

CPU CPU

CPU

node

node

node node

node

Software

Linux numa evolution