"Is parallel programming hard?", first edition Posted Mar 11, 2014 13:09 UTC (Tue) by Kluge (subscriber, #2881) [Link] (5 responses) Very generous of Paul McKenney, unless his answer is "No it isn't, what's your problem?" "Is parallel programming hard?", first edition Posted Mar 11, 2014 13:15 UTC (Tue) by corbet (editor, #1) [Link] (1 responses) Even Paul would have a hard time taking 500+ pages to say that! :) "Is parallel programming hard?", first edition Posted Mar 11, 2014 18:11 UTC (Tue) by dfsmith (guest, #20302) [Link] It would have been a single page, but a recursive subroutine accidentally fired off too many threads. (Is there a parallel version of LaTeX?) "Is parallel programming hard?", first edition Posted Mar 12, 2014 7:55 UTC (Wed) by fotoba (guest, #61150) [Link] (2 responses) I do not think so. It depends on group you are member of. Parallel programmers not prepared for the glorious revolution By Wily Ferret Tue Nov 27 2007, 12:28 INTEL RECKONS barely one per cent of software programmers are prepared to face the challenge of parallel programming, which the hardware giant (unsurprisingly) reckons is the future of development. http://www.theinquirer.net/inquirer/news/1026585/programm... If only one percent of pragrammers is able to do so, it may be hard. And is was not changed since that time till today Software needs meaty cores, not thin, stringy ARMs, says Intel By Simon Sharwood, 26 Feb 2014 “The world has a big issue around vectorisation and parallelisation of code,” Graylish said. “99% of code isn't written that way.” Graylish also feels “defining a workload that can run in 1000 cores is hard.” Most software, Graylish added, “still requires a big meaty core” and Intel is happy to provide them. http://www.theregister.co.uk/2014/02/26/software_needs_me... If you are memeber of TOP one percent it is not problem, but if not it is problem.. "Is parallel programming hard?", first edition Posted Mar 12, 2014 10:15 UTC (Wed) by vonbrand (guest, #4458) [Link] (1 responses) I'd be surprised if much more than 1% are competent at programming, period. "Is parallel programming hard?", first edition Posted Mar 12, 2014 11:11 UTC (Wed) by fotoba (guest, #61150) [Link] But this amount of people coreesponds to Linux desktop user count. "Is parallel programming hard?", first edition Posted Mar 11, 2014 13:50 UTC (Tue) by tsmithe (guest, #57598) [Link] (4 responses) No discussion of OpenCL and similar technologies? "Is parallel programming hard?", first edition Posted Mar 11, 2014 16:44 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (3 responses) Sorry, but no. One of the great many things I learned from Donald Knuth was to keep my books quite focused. :-) But you might check out Norm Matloff's PDF entitled "Programming on Parallel Machines" which covers high-performance-computing. He mostly focuses on CUDA, but has at least some coverage of OpenCL, along with OpenMP and MPI. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:16 UTC (Tue) by tsmithe (guest, #57598) [Link] (2 responses) That's understandable. I was just surprised to see a book on parallel programming not even mention a fashionable and powerful parallel programming paradigm! The opening and title of the book suggest that it's going to cover parallel programming (in general, or at least a representative sample), which has many forms, but it looks like it should be called instead something like "Is POSIX parallel programming hard?" or "Is multi-threaded programming hard?". "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] I figure that it is easy enough for people to download the book and scan the table of contents. :-) "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:27 UTC (Tue) by Tara_Li (guest, #26706) [Link] My take on it was more parallel programming in general, rather than looking at any particular implementation. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:07 UTC (Tue) by stijn (subscriber, #570) [Link] (7 responses) I am at the moment gathering data for a program that does sparse matrix multiplication using POSIX threads. The scale of data is approximately 5M rows, 1G matrix entries (zeroes not stored). For this data (and, admittedly, certain hardware), memory access seems to kill any gains from additional CPUs going above 8 (on a 24-core machine). I assume this depends on the NUMA layout (which I can see using lstopo), and I aim to test this by ordering the data in such a way that memory access will predominantly refer to memory that was allocated locally, i.e. rows that 'refer' to each other should ideally cluster together as much as possible (there are reasons to assume that such clustering is achievable). There is a worry that this could prove to be futile. I have not managed to research this succesfully so far - or assemble the required insights from the various aspects that affect this scenario. Does anyone have any insights or pointers? I am aware this is not a great question (somewhat underspecified, not enough data), but to some extent it is a hard question to bootstrap. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:30 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (2 responses) There has been quite a bit of research done on that topic. Might be worth a websearch for something like "sparse matrix NUMA memory locality" if you have not already done so. If you are running on Linux, tools like "perf" can help identify which part of the program is having difficulty and why. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:51 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) Thanks Paul, I had done quite a number of searches, but perhaps focused a bit too much on pthreads. This does lead to relevant research. Fantastic book, looks like I'll be alternatively reading and searching it a lot. "Is parallel programming hard?", first edition Posted Mar 11, 2014 18:10 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad it helped, and best of everything with the big sparse matrices! "Is parallel programming hard?", first edition Posted Mar 11, 2014 19:25 UTC (Tue) by darthscsi (guest, #8111) [Link] (2 responses) For this size machine, NUMA aware placement and scheduling will help, but simply doing round-robin memory placement and placement-oblivious scheduling will perform decently. More likely, you need to consider contention on your task (row) scheduler. Differential profiling likely will pinpoint your problem (use some hardware-counter, sampling-based, full-system profiler). "Is parallel programming hard?", first edition Posted Mar 11, 2014 23:42 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) My current thinking is it very likely entirely memory access, disregarding the algorithm and implementation specifics of the per-thread task that is being executed (which is orthogonal to the observed thread/CPU count vs timing). This task is a matrix - vector multiplication, and the way it is paralellised is the most standard way possible, almost like a textbook case with no complications: One loop that calls pthread_create(), followed by a loop that calls pthread_join(). Each thread computes a number of tasks (M v multiplication). I'd love to be wrong though. "Is parallel programming hard?", first edition Posted Mar 12, 2014 12:58 UTC (Wed) by etienne (guest, #25256) [Link] > One loop that calls pthread_create(), followed by a loop that calls pthread_join(). > Each thread computes a number of tasks (M v multiplication). Should work fine, the problem is how to schedule each pthread in between the _create() and the _join(). Using the standard scheduler means that you assume the physical memory cache is very good, and can plan the future access pattern to keep processor busy. You can try to write the code carefully (i.e. write a complex logic) so that you can guess what is in the memory cache and so just one more memory fetch will enable to quickly calculate that other block. You can use a processor-accelerator like a video card + OpenCl so that all/most of the threads are started, and its scheduler is so good it is not even planning the future but can stop a thread in the middle of an instruction (a read or write to main memory) to run other ready thread (one where its read from main memory just arrived). Thousands of threads with a lot more read/write queues to main memory helps hiding latencies. Unfortunately transparent pthread interface to OpenCl does not seem to be ready yet... "Is parallel programming hard?", first edition Posted Mar 13, 2014 11:11 UTC (Thu) by ehiggs (guest, #90713) [Link] I strongly recommend that you take a look at OpenMP for sparse matrix multiplication. This is a decent page demonstrating the differences: http://www.futurechips.org/tips-for-power-coders/open-mp-... > For this data (and, admittedly, certain hardware), memory access seems to kill any gains from additional CPUs going above 8 (on a 24-core machine) Do you mean 8 CPUs or 8 cores? If your memory bandwidth is the limiting factor it doesn't really matter how many cores you have but how well you can feed those cores from the memory bus(es). "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:18 UTC (Tue) by mday_ii (guest, #25315) [Link] (1 responses) I learn something valuable every time I read a section of this work. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:31 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad you like it! "Is parallel programming hard?", first edition Posted Mar 12, 2014 20:30 UTC (Wed) by riccieri (guest, #94794) [Link] Awesome! I only use very high level languages (ruby and javascript), on the very top of the stack. It's very interesting to understand to poke at what happens on the lower levels, and maybe be able to carry some parallelism patterns upwards. "Is parallel programming hard?", first edition Posted Mar 15, 2014 6:09 UTC (Sat) by lemmings (guest, #53618) [Link] Fantastic work Paul. It looks very comprehensive and well written. "Is parallel programming hard?", first edition Posted Apr 9, 2014 17:15 UTC (Wed) by mikachu (guest, #5333) [Link] (1 responses) I fully expected the body of this article to just say "Yes." "Is parallel programming hard?", first edition Posted Apr 11, 2014 6:29 UTC (Fri) by mathstuf (subscriber, #69389) [Link] Shh, you'll give away the ending.
Posted Mar 11, 2014 13:09 UTC (Tue) by Kluge (subscriber, #2881) [Link] (5 responses)
"Is parallel programming hard?", first edition Posted Mar 11, 2014 13:15 UTC (Tue) by corbet (editor, #1) [Link] (1 responses) Even Paul would have a hard time taking 500+ pages to say that! :) "Is parallel programming hard?", first edition Posted Mar 11, 2014 18:11 UTC (Tue) by dfsmith (guest, #20302) [Link] It would have been a single page, but a recursive subroutine accidentally fired off too many threads. (Is there a parallel version of LaTeX?) "Is parallel programming hard?", first edition Posted Mar 12, 2014 7:55 UTC (Wed) by fotoba (guest, #61150) [Link] (2 responses) I do not think so. It depends on group you are member of. Parallel programmers not prepared for the glorious revolution By Wily Ferret Tue Nov 27 2007, 12:28 INTEL RECKONS barely one per cent of software programmers are prepared to face the challenge of parallel programming, which the hardware giant (unsurprisingly) reckons is the future of development. http://www.theinquirer.net/inquirer/news/1026585/programm... If only one percent of pragrammers is able to do so, it may be hard. And is was not changed since that time till today Software needs meaty cores, not thin, stringy ARMs, says Intel By Simon Sharwood, 26 Feb 2014 “The world has a big issue around vectorisation and parallelisation of code,” Graylish said. “99% of code isn't written that way.” Graylish also feels “defining a workload that can run in 1000 cores is hard.” Most software, Graylish added, “still requires a big meaty core” and Intel is happy to provide them. http://www.theregister.co.uk/2014/02/26/software_needs_me... If you are memeber of TOP one percent it is not problem, but if not it is problem.. "Is parallel programming hard?", first edition Posted Mar 12, 2014 10:15 UTC (Wed) by vonbrand (guest, #4458) [Link] (1 responses) I'd be surprised if much more than 1% are competent at programming, period. "Is parallel programming hard?", first edition Posted Mar 12, 2014 11:11 UTC (Wed) by fotoba (guest, #61150) [Link] But this amount of people coreesponds to Linux desktop user count.
Posted Mar 11, 2014 13:15 UTC (Tue) by corbet (editor, #1) [Link] (1 responses)
"Is parallel programming hard?", first edition Posted Mar 11, 2014 18:11 UTC (Tue) by dfsmith (guest, #20302) [Link] It would have been a single page, but a recursive subroutine accidentally fired off too many threads. (Is there a parallel version of LaTeX?)
Posted Mar 11, 2014 18:11 UTC (Tue) by dfsmith (guest, #20302) [Link]
(Is there a parallel version of LaTeX?)
Posted Mar 12, 2014 7:55 UTC (Wed) by fotoba (guest, #61150) [Link] (2 responses)
Parallel programmers not prepared for the glorious revolution By Wily Ferret Tue Nov 27 2007, 12:28
INTEL RECKONS barely one per cent of software programmers are prepared to face the challenge of parallel programming, which the hardware giant (unsurprisingly) reckons is the future of development. http://www.theinquirer.net/inquirer/news/1026585/programm...
If only one percent of pragrammers is able to do so, it may be hard.
And is was not changed since that time till today
Software needs meaty cores, not thin, stringy ARMs, says Intel By Simon Sharwood, 26 Feb 2014
“The world has a big issue around vectorisation and parallelisation of code,” Graylish said. “99% of code isn't written that way.” Graylish also feels “defining a workload that can run in 1000 cores is hard.”
Most software, Graylish added, “still requires a big meaty core” and Intel is happy to provide them. http://www.theregister.co.uk/2014/02/26/software_needs_me...
If you are memeber of TOP one percent it is not problem, but if not it is problem..
"Is parallel programming hard?", first edition Posted Mar 12, 2014 10:15 UTC (Wed) by vonbrand (guest, #4458) [Link] (1 responses) I'd be surprised if much more than 1% are competent at programming, period. "Is parallel programming hard?", first edition Posted Mar 12, 2014 11:11 UTC (Wed) by fotoba (guest, #61150) [Link] But this amount of people coreesponds to Linux desktop user count.
Posted Mar 12, 2014 10:15 UTC (Wed) by vonbrand (guest, #4458) [Link] (1 responses)
I'd be surprised if much more than 1% are competent at programming, period.
"Is parallel programming hard?", first edition Posted Mar 12, 2014 11:11 UTC (Wed) by fotoba (guest, #61150) [Link] But this amount of people coreesponds to Linux desktop user count.
Posted Mar 12, 2014 11:11 UTC (Wed) by fotoba (guest, #61150) [Link]
Posted Mar 11, 2014 13:50 UTC (Tue) by tsmithe (guest, #57598) [Link] (4 responses)
"Is parallel programming hard?", first edition Posted Mar 11, 2014 16:44 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (3 responses) Sorry, but no. One of the great many things I learned from Donald Knuth was to keep my books quite focused. :-) But you might check out Norm Matloff's PDF entitled "Programming on Parallel Machines" which covers high-performance-computing. He mostly focuses on CUDA, but has at least some coverage of OpenCL, along with OpenMP and MPI. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:16 UTC (Tue) by tsmithe (guest, #57598) [Link] (2 responses) That's understandable. I was just surprised to see a book on parallel programming not even mention a fashionable and powerful parallel programming paradigm! The opening and title of the book suggest that it's going to cover parallel programming (in general, or at least a representative sample), which has many forms, but it looks like it should be called instead something like "Is POSIX parallel programming hard?" or "Is multi-threaded programming hard?". "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] I figure that it is easy enough for people to download the book and scan the table of contents. :-) "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:27 UTC (Tue) by Tara_Li (guest, #26706) [Link] My take on it was more parallel programming in general, rather than looking at any particular implementation.
Posted Mar 11, 2014 16:44 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (3 responses)
But you might check out Norm Matloff's PDF entitled "Programming on Parallel Machines" which covers high-performance-computing. He mostly focuses on CUDA, but has at least some coverage of OpenCL, along with OpenMP and MPI.
"Is parallel programming hard?", first edition Posted Mar 11, 2014 17:16 UTC (Tue) by tsmithe (guest, #57598) [Link] (2 responses) That's understandable. I was just surprised to see a book on parallel programming not even mention a fashionable and powerful parallel programming paradigm! The opening and title of the book suggest that it's going to cover parallel programming (in general, or at least a representative sample), which has many forms, but it looks like it should be called instead something like "Is POSIX parallel programming hard?" or "Is multi-threaded programming hard?". "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] I figure that it is easy enough for people to download the book and scan the table of contents. :-) "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:27 UTC (Tue) by Tara_Li (guest, #26706) [Link] My take on it was more parallel programming in general, rather than looking at any particular implementation.
Posted Mar 11, 2014 17:16 UTC (Tue) by tsmithe (guest, #57598) [Link] (2 responses)
The opening and title of the book suggest that it's going to cover parallel programming (in general, or at least a representative sample), which has many forms, but it looks like it should be called instead something like "Is POSIX parallel programming hard?" or "Is multi-threaded programming hard?".
"Is parallel programming hard?", first edition Posted Mar 11, 2014 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] I figure that it is easy enough for people to download the book and scan the table of contents. :-) "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:27 UTC (Tue) by Tara_Li (guest, #26706) [Link] My take on it was more parallel programming in general, rather than looking at any particular implementation.
Posted Mar 11, 2014 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link]
Posted Mar 11, 2014 17:27 UTC (Tue) by Tara_Li (guest, #26706) [Link]
Posted Mar 11, 2014 17:07 UTC (Tue) by stijn (subscriber, #570) [Link] (7 responses)
There is a worry that this could prove to be futile. I have not managed to research this succesfully so far - or assemble the required insights from the various aspects that affect this scenario. Does anyone have any insights or pointers? I am aware this is not a great question (somewhat underspecified, not enough data), but to some extent it is a hard question to bootstrap.
"Is parallel programming hard?", first edition Posted Mar 11, 2014 17:30 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (2 responses) There has been quite a bit of research done on that topic. Might be worth a websearch for something like "sparse matrix NUMA memory locality" if you have not already done so. If you are running on Linux, tools like "perf" can help identify which part of the program is having difficulty and why. "Is parallel programming hard?", first edition Posted Mar 11, 2014 17:51 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) Thanks Paul, I had done quite a number of searches, but perhaps focused a bit too much on pthreads. This does lead to relevant research. Fantastic book, looks like I'll be alternatively reading and searching it a lot. "Is parallel programming hard?", first edition Posted Mar 11, 2014 18:10 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad it helped, and best of everything with the big sparse matrices! "Is parallel programming hard?", first edition Posted Mar 11, 2014 19:25 UTC (Tue) by darthscsi (guest, #8111) [Link] (2 responses) For this size machine, NUMA aware placement and scheduling will help, but simply doing round-robin memory placement and placement-oblivious scheduling will perform decently. More likely, you need to consider contention on your task (row) scheduler. Differential profiling likely will pinpoint your problem (use some hardware-counter, sampling-based, full-system profiler). "Is parallel programming hard?", first edition Posted Mar 11, 2014 23:42 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) My current thinking is it very likely entirely memory access, disregarding the algorithm and implementation specifics of the per-thread task that is being executed (which is orthogonal to the observed thread/CPU count vs timing). This task is a matrix - vector multiplication, and the way it is paralellised is the most standard way possible, almost like a textbook case with no complications: One loop that calls pthread_create(), followed by a loop that calls pthread_join(). Each thread computes a number of tasks (M v multiplication). I'd love to be wrong though. "Is parallel programming hard?", first edition Posted Mar 12, 2014 12:58 UTC (Wed) by etienne (guest, #25256) [Link] > One loop that calls pthread_create(), followed by a loop that calls pthread_join(). > Each thread computes a number of tasks (M v multiplication). Should work fine, the problem is how to schedule each pthread in between the _create() and the _join(). Using the standard scheduler means that you assume the physical memory cache is very good, and can plan the future access pattern to keep processor busy. You can try to write the code carefully (i.e. write a complex logic) so that you can guess what is in the memory cache and so just one more memory fetch will enable to quickly calculate that other block. You can use a processor-accelerator like a video card + OpenCl so that all/most of the threads are started, and its scheduler is so good it is not even planning the future but can stop a thread in the middle of an instruction (a read or write to main memory) to run other ready thread (one where its read from main memory just arrived). Thousands of threads with a lot more read/write queues to main memory helps hiding latencies. Unfortunately transparent pthread interface to OpenCl does not seem to be ready yet... "Is parallel programming hard?", first edition Posted Mar 13, 2014 11:11 UTC (Thu) by ehiggs (guest, #90713) [Link] I strongly recommend that you take a look at OpenMP for sparse matrix multiplication. This is a decent page demonstrating the differences: http://www.futurechips.org/tips-for-power-coders/open-mp-... > For this data (and, admittedly, certain hardware), memory access seems to kill any gains from additional CPUs going above 8 (on a 24-core machine) Do you mean 8 CPUs or 8 cores? If your memory bandwidth is the limiting factor it doesn't really matter how many cores you have but how well you can feed those cores from the memory bus(es).
Posted Mar 11, 2014 17:30 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (2 responses)
If you are running on Linux, tools like "perf" can help identify which part of the program is having difficulty and why.
"Is parallel programming hard?", first edition Posted Mar 11, 2014 17:51 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) Thanks Paul, I had done quite a number of searches, but perhaps focused a bit too much on pthreads. This does lead to relevant research. Fantastic book, looks like I'll be alternatively reading and searching it a lot. "Is parallel programming hard?", first edition Posted Mar 11, 2014 18:10 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad it helped, and best of everything with the big sparse matrices!
Posted Mar 11, 2014 17:51 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses)
"Is parallel programming hard?", first edition Posted Mar 11, 2014 18:10 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad it helped, and best of everything with the big sparse matrices!
Posted Mar 11, 2014 18:10 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link]
Posted Mar 11, 2014 19:25 UTC (Tue) by darthscsi (guest, #8111) [Link] (2 responses)
Differential profiling likely will pinpoint your problem (use some hardware-counter, sampling-based, full-system profiler).
"Is parallel programming hard?", first edition Posted Mar 11, 2014 23:42 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses) My current thinking is it very likely entirely memory access, disregarding the algorithm and implementation specifics of the per-thread task that is being executed (which is orthogonal to the observed thread/CPU count vs timing). This task is a matrix - vector multiplication, and the way it is paralellised is the most standard way possible, almost like a textbook case with no complications: One loop that calls pthread_create(), followed by a loop that calls pthread_join(). Each thread computes a number of tasks (M v multiplication). I'd love to be wrong though. "Is parallel programming hard?", first edition Posted Mar 12, 2014 12:58 UTC (Wed) by etienne (guest, #25256) [Link] > One loop that calls pthread_create(), followed by a loop that calls pthread_join(). > Each thread computes a number of tasks (M v multiplication). Should work fine, the problem is how to schedule each pthread in between the _create() and the _join(). Using the standard scheduler means that you assume the physical memory cache is very good, and can plan the future access pattern to keep processor busy. You can try to write the code carefully (i.e. write a complex logic) so that you can guess what is in the memory cache and so just one more memory fetch will enable to quickly calculate that other block. You can use a processor-accelerator like a video card + OpenCl so that all/most of the threads are started, and its scheduler is so good it is not even planning the future but can stop a thread in the middle of an instruction (a read or write to main memory) to run other ready thread (one where its read from main memory just arrived). Thousands of threads with a lot more read/write queues to main memory helps hiding latencies. Unfortunately transparent pthread interface to OpenCl does not seem to be ready yet...
Posted Mar 11, 2014 23:42 UTC (Tue) by stijn (subscriber, #570) [Link] (1 responses)
"Is parallel programming hard?", first edition Posted Mar 12, 2014 12:58 UTC (Wed) by etienne (guest, #25256) [Link] > One loop that calls pthread_create(), followed by a loop that calls pthread_join(). > Each thread computes a number of tasks (M v multiplication). Should work fine, the problem is how to schedule each pthread in between the _create() and the _join(). Using the standard scheduler means that you assume the physical memory cache is very good, and can plan the future access pattern to keep processor busy. You can try to write the code carefully (i.e. write a complex logic) so that you can guess what is in the memory cache and so just one more memory fetch will enable to quickly calculate that other block. You can use a processor-accelerator like a video card + OpenCl so that all/most of the threads are started, and its scheduler is so good it is not even planning the future but can stop a thread in the middle of an instruction (a read or write to main memory) to run other ready thread (one where its read from main memory just arrived). Thousands of threads with a lot more read/write queues to main memory helps hiding latencies. Unfortunately transparent pthread interface to OpenCl does not seem to be ready yet...
Posted Mar 12, 2014 12:58 UTC (Wed) by etienne (guest, #25256) [Link]
Should work fine, the problem is how to schedule each pthread in between the _create() and the _join(). Using the standard scheduler means that you assume the physical memory cache is very good, and can plan the future access pattern to keep processor busy. You can try to write the code carefully (i.e. write a complex logic) so that you can guess what is in the memory cache and so just one more memory fetch will enable to quickly calculate that other block. You can use a processor-accelerator like a video card + OpenCl so that all/most of the threads are started, and its scheduler is so good it is not even planning the future but can stop a thread in the middle of an instruction (a read or write to main memory) to run other ready thread (one where its read from main memory just arrived). Thousands of threads with a lot more read/write queues to main memory helps hiding latencies.
Unfortunately transparent pthread interface to OpenCl does not seem to be ready yet...
Posted Mar 13, 2014 11:11 UTC (Thu) by ehiggs (guest, #90713) [Link]
http://www.futurechips.org/tips-for-power-coders/open-mp-...
> For this data (and, admittedly, certain hardware), memory access seems to kill any gains from additional CPUs going above 8 (on a 24-core machine)
Do you mean 8 CPUs or 8 cores? If your memory bandwidth is the limiting factor it doesn't really matter how many cores you have but how well you can feed those cores from the memory bus(es).
Posted Mar 11, 2014 17:18 UTC (Tue) by mday_ii (guest, #25315) [Link] (1 responses)
"Is parallel programming hard?", first edition Posted Mar 11, 2014 17:31 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link] Glad you like it!
Posted Mar 11, 2014 17:31 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link]
Posted Mar 12, 2014 20:30 UTC (Wed) by riccieri (guest, #94794) [Link]
It's very interesting to understand to poke at what happens on the lower levels, and maybe be able to carry some parallelism patterns upwards.
Posted Mar 15, 2014 6:09 UTC (Sat) by lemmings (guest, #53618) [Link]
Posted Apr 9, 2014 17:15 UTC (Wed) by mikachu (guest, #5333) [Link] (1 responses)
"Is parallel programming hard?", first edition Posted Apr 11, 2014 6:29 UTC (Fri) by mathstuf (subscriber, #69389) [Link] Shh, you'll give away the ending.
Posted Apr 11, 2014 6:29 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
Copyright © 2014, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds
LWN