Xeon - not utilising more than 32 threads

Forum to discuss and compare Hardware profiles and Benchmarking
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#1 Xeon - not utilising more than 32 threads

Post by noetus »

Hi, I'm new to BOINC and to this forum, having just joined this team!

I have two Xeon builds with 52 threads each. BOINC only runs 32 threads, so it only utilises a max of 57% of available CPU. I've tried changing the local setting to 100% of CPU, and I've tried increasing the number of CPU cores to 56 in the profile on my web portal settings page. Alas, to no avail. Any suggestions as to what is going on?

I'd sorely like to be contributing the lost 43% of potential points to the Team....
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#2 Re: Xeon - not utilising more than 32 threads

Post by Alez »

Welcome to the forum noetus.
As McShane asked, give us a bit more info and we'll see how we can help you.

What user name do you crunch for the team on and which project ?

That will enable us to check out the stderr from the project and get what system you are running etc on which operating system. If you have your systems hidden on the project , then unhide them. You can change it back later. It could be as simple as the project will only send you so many units at one time. Once we have the basic info we can figure out what the problem is.

Again welcome.
Image
The best form of help from above is a sniper on the rooftop....
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#3 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

One possible answer is system memory. There are some projects that are memory intensive and require a lot of RAM (>1G) for each WU. BOINC will only run as many WU as will fit within the available memory.

1. What project are you running?

2. In the BOINC manager, at the top is "Options" select that and then "Computing Preferences". You then get a pop up and the 3rd tab is "Disk and memory". Open that and make sure you have the Memory set to something like 90-95% utilization in the 2 boxes. BOINC will only allow that percentage of your total system memory to be used. Even if you have a LOT of system RAM BOINC will only use the percentage that is set in that tab.

3. Do you have hyper-threading enabled in BIOS?

4. As Pete mentioned, install and run CPU-Z (CPUID). The bottom right 2 boxes will show you how many cores and threads are being detected by Windows.

5. What CPUs are you running?

If you are on Linux, then run "htop". That will show you how many threads it sees the machine having.
Image
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#4 Re: Xeon - not utilising more than 32 threads

Post by Alez »

I'm presuming it's server class equipment. My first though was to check which power mode is enabled and to ensure performance ( or equivelent ) is selected rather than eco or power save.
We'll know more when we get some of the questions answered.

Also restart BOINC and post the startup script . It should show something like this

| Starting BOINC client version 7.4.36 for windows_x86_64
| log flags: file_xfer, sched_ops, task
| Libraries: libcurl/7.39.0 OpenSSL/1.0.1j zlib/1.2.8
| Data directory: C:\ProgramData\BOINC
| Running under account Administrator
| CUDA: NVIDIA GPU 0: Quadro K5200 (driver version 341.21, CUDA version 6.5, compute capability 3.5, 4096MB, 4096MB available, 3553 GFLOPS peak)
| OpenCL: NVIDIA GPU 0: Quadro K5200 (driver version 341.21, device version OpenCL 1.1 CUDA, 8192MB, 4096MB available, 3553 GFLOPS peak)
| Host name: WIN-99OC5SRSPAM
| Processor: 72 GenuineIntel Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz [Family 6 Model 63 Stepping 2]
| Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss htt tm pni ssse3 fma cx16 sse4_1 sse4_2 movebe popcnt aes f16c rdrandsyscall nx lm avx avx2 vmx smx tm2 dca pbe fsgsbase bmi1 smep bmi2
| OS: Microsoft Windows Server 2012 R2: Standard x64 Edition, (06.03.9600.00)
| Memory: 255.89 GB physical, 257.85 GB virtual
| Disk: 185.97 GB total, 133.88 GB free
| Local time is UTC -7 hours
| VirtualBox version: 4.3.12
| Config: simulate 72 CPUs
World Community Grid | URL http://www.worldcommunitygrid.org/; Computer ID 3277834; resource share 100
World Community Grid | General prefs: from World Community Grid (last modified 07-Mar-2015 15:09:57)
World Community Grid | Host location: none
World Community Grid | General prefs: using your defaults
| Reading preferences override file
| Preferences:
| max memory usage when active: 196525.90MB
| max memory usage when idle: 235831.09MB
| max disk usage: 10.00GB
| (to change preferences, visit a project web site or select Preferences in the Manager)
| Not using a proxy
Image
The best form of help from above is a sniper on the rooftop....
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#5 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Oh guys, thanks for all the responses! I'm overwhelmed - never thought that I would see so much activity in response to my question.

Thanks also for the welcome to the Team. I am going to respond to that thread as well to tell you all a little bit about myself (in case anyone's interested).

You can assume I'm fairly tech savvy and know my way around CPU-Z, the BIOS, etc. I'm programming these number-crunching machines myself in C# for my own research project, so with that said...

Windows 10 Enterprise, the two systems are hardware dentical, each has 32GB RAM and 2x Intel Xeon E5-2658 v4 which is 2 x 14 hardware cores for a total of 56 threads on each system. For my own research I commonly use up all CPU cycles with 56 separate threads, so I know the hardware and OS setup is not the issue.

I'm going to check the memory allocation for BOINC but as far as I can see I shouldn't have to mess with cc_config.xml to get all cores fully utilised for BOINC. The fact is that there are exactly 32 processes running in the BOINC client on each machine, and this is the exact default assignment on my default profile on the web portal. This isn't a coincidence. Upping the number in the profile (done more than 24 hours ago) should have upped the number of processes running on the client, but it hasn't, so something is wrong there. This should be fixable without having to do anything low-level. Either I'm missing a setting somewhere or somethiung is not happening with the client's communication with the servers.
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#6 Re: Xeon - not utilising more than 32 threads

Post by noetus »

And to answer the other questions: username is same as here on the forums, and project is Smash Childhood Cancer. Yeah, and despite my self-affirmed "tech savvy" nature I got the number of threads wrong on my original post - indeed it is 56 and not 52.
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#7 Re: Xeon - not utilising more than 32 threads

Post by noetus »

I checked the system memory and settings. Each system has 32 GB RAM, and client is allowed to use 50% of it (75% when idle). Since each WU is using about 40 MB of RAM, that is a total RAM usage right now of 32 x 40 = 1280 MB, which is less than 4% of system RAM, so I don't think RAM is the issue.
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#8 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

It sounds as if you have installed the 32 bit version of WIn 10 which only supports 32 cores/threads.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#9 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Nope, as I mentioned earlier with my own projects (.NET) I can run 56 threads no problem, and it is definitely the 64 bit version of the OS that is installed! 8-)

By the way, I also mentioned each system has 32 GB RAM. Since when does a 32 bit OS address 32 GB of RAM?

PS. Name of process: wcgrid_scc1_vina_7.08_windows_x86_64
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#10 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

Do your systems have a cc_config.xml file? If so, post the contents. And WCG doesn't show computer info. If you attach to some other project we can look at the computer details get a little more info about what the Boinc client is seeing/reporting. That and see if a different project will use all 56 threads. Perhaps a screen shot of your Boinc Client computing preferences, processor usage and disk and memory usage tabs.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#11 Re: Xeon - not utilising more than 32 threads

Post by noetus »

cc_config.xml doesn't seem to have anything useful:

<cc_config>
<log_flags>
</log_flags>
<options>
<client_download_url>http://www.worldcommunitygrid.org/reg/m ... wnload_url>
<client_version_check_url>http://www.worldcommunitygrid.org/downl ... _check_url>
<network_test_url>http://www.ibm.com</network_test_url>
</options>
</cc_config>

global_prefs.xml does however contain some useful stuff:

<global_preferences>
<source_project>http://www.worldcommunitygrid.org/</source_project>
<source_scheduler>https://scheduler.worldcommunitygrid.or ... _scheduler>

<mod_time>1496049590</mod_time>
<cpu_scheduling_period_minutes>120</cpu_scheduling_period_minutes>
<disk_interval>60.0</disk_interval>
<disk_max_used_gb>10.0</disk_max_used_gb>
<disk_max_used_pct>50.0</disk_max_used_pct>
<disk_min_free_gb>0.5</disk_min_free_gb>
<end_hour>0</end_hour>
<idle_time_to_run>3.0</idle_time_to_run>
<leave_apps_in_memory/>
<max_bytes_sec_down>0.0</max_bytes_sec_down>
<max_bytes_sec_up>0.0</max_bytes_sec_up>
<daily_xfer_period_days>0</daily_xfer_period_days>
<daily_xfer_limit_mb>0.0</daily_xfer_limit_mb>
<max_cpus>88</max_cpus>
<max_ncpus_pct>100.0</max_ncpus_pct>
<suspend_cpu_usage>0.0</suspend_cpu_usage>
<net_end_hour>0</net_end_hour>
<net_start_hour>0</net_start_hour>
<run_if_user_active/>
<start_hour>0</start_hour>
<cpu_usage_limit>100.0</cpu_usage_limit>
<ram_max_used_busy_pct>50.0</ram_max_used_busy_pct>
<ram_max_used_idle_pct>75.0</ram_max_used_idle_pct>
<vm_max_used_pct>50.0</vm_max_used_pct>
<work_buf_min_days>0.2</work_buf_min_days>
<work_buf_additional_days>0.3</work_buf_additional_days>
<suspend_if_no_recent_input>0.0</suspend_if_no_recent_input>
<venue name="home">
<cpu_scheduling_period_minutes>120</cpu_scheduling_period_minutes>
<disk_interval>60.0</disk_interval>
<disk_max_used_gb>10.0</disk_max_used_gb>
<disk_max_used_pct>50.0</disk_max_used_pct>
<disk_min_free_gb>0.5</disk_min_free_gb>
<end_hour>0</end_hour>
<idle_time_to_run>0.0</idle_time_to_run>
<leave_apps_in_memory/>
<max_bytes_sec_down>0.0</max_bytes_sec_down>
<max_bytes_sec_up>0.0</max_bytes_sec_up>
<daily_xfer_period_days>0</daily_xfer_period_days>
<daily_xfer_limit_mb>0.0</daily_xfer_limit_mb>
<max_cpus>88</max_cpus>
<max_ncpus_pct>100.0</max_ncpus_pct>
<suspend_cpu_usage>0.0</suspend_cpu_usage>
<net_end_hour>0</net_end_hour>
<net_start_hour>0</net_start_hour>
<run_if_user_active/>
<start_hour>0</start_hour>
<cpu_usage_limit>100.0</cpu_usage_limit>
<ram_max_used_busy_pct>50.0</ram_max_used_busy_pct>
<ram_max_used_idle_pct>75.0</ram_max_used_idle_pct>
<vm_max_used_pct>50.0</vm_max_used_pct>
<work_buf_min_days>0.1</work_buf_min_days>
<work_buf_additional_days>0.3</work_buf_additional_days>
<suspend_if_no_recent_input>0.0</suspend_if_no_recent_input>
</venue>
</global_preferences>
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#12 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Is there a way of posting screenshots on this forum without using an image hoster?
Save the image to a file, click the "Attachments" tab at the bottom of the post dialog, click "Add files", select your image file, click "Place inline", make sure attachment tag is placed where you want it.
attachment.jpg
What are these tags doing? Any particular reason you use them?

<client_download_url>http://www.worldcommunitygrid.org/reg/m ... wnload_url>
<client_version_check_url>http://www.worldcommunitygrid.org/downl ... _check_url>
<network_test_url>http://www.ibm.com</network_test_url>
You do not have the required permissions to view the files attached to this post.
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#13 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

Too many variables :-)

It sounds like you know what you are doing at the computer level so we may be able to assume it is a BOINC implementation problem. As Alez suggested, restart BOINC and post the startup message so we can see what BOINC thinks your system looks like.

In the BOINC manager at the top under "Tools" select "Event Log". That will show the same type of info that Alez posted earlier. That hopefully will give us a clue as to what is going on.

At the Windows level, launch the task manager, select "Performance". At the bottom of the screen is "Launch Resource Monitor". Do that and then select "CPU". It then shows the loading for each thread Windows is seeing. Does it show 56 CPUs?
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#14 Re: Xeon - not utilising more than 32 threads

Post by noetus »

There is no "Event Log" under "Tools". There is only, in advanced view, "Add project or account manager...", "Options...", "Computing preferences..." (When in Simple View there is only "Computing preferences..." and "Options...", in a different order for some reason.

However, under "Advanced" (only available in Advanced View) I do find "Event log...". Is this what you meant?

Looking at the Event Log gives me a headache. For starters it says the OS is Windows 8.1, which it most certainly is not. And it says the CPU is only 32 cores, which it most certainly is not.

Resource Monitor shows 28 CPUs, and hyperthreading is enabled of course, which means there are 56 virtual CPUs available (threads). In case you are in any doubt about this, here is a screenshot of Task Manager, which is funny 'cos in fact BOINC IS using all 56 vcpus, just not at 100% in any of them (despite only seeing 32 when it starts up -WTH).

https://ibb.co/nfNNSa

Here is a dump of the log, first few lines:

30.05.2017 13:27:26 | | Starting BOINC client version 7.2.47 for windows_x86_64
30.05.2017 13:27:26 | | log flags: file_xfer, sched_ops, task
30.05.2017 13:27:26 | | Libraries: libcurl/7.25.0 OpenSSL/1.0.1 zlib/1.2.6
30.05.2017 13:27:26 | | Running as a daemon
30.05.2017 13:27:26 | | Data directory: C:\ProgramData\BOINC
30.05.2017 13:27:26 | | Running under account boinc_master
30.05.2017 13:27:26 | | No usable GPUs found
30.05.2017 13:27:26 | | Host name: Petra
30.05.2017 13:27:26 | | Processor: 32 GenuineIntel Intel(R) Xeon(R) CPU E5-2658 v4@ 2.30GHz [Family 6 Model 79 Stepping 1]
30.05.2017 13:27:26 | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss htt tm pni ssse3 fma cx16 sse4_1 sse4_2 movebe popcnt aes nx lm vmx smx tm2 dca pbe
30.05.2017 13:27:26 | | OS: Microsoft Windows 8.1: x64 Edition, (06.03.9600.00)
30.05.2017 13:27:26 | | Memory: 31.89 GB physical, 36.64 GB virtual
30.05.2017 13:27:26 | | Disk: 118.75 GB total, 65.78 GB free
30.05.2017 13:27:26 | | Local time is UTC +2 hours
30.05.2017 13:27:26 | World Community Grid | URL http://www.worldcommunitygrid.org/; Computer ID 3958430; resource share 100
30.05.2017 13:27:26 | World Community Grid | General prefs: from World Community Grid (last modified 29-May-2017 11:19:50)
30.05.2017 13:27:26 | World Community Grid | Computer location: home
30.05.2017 13:27:26 | | General prefs: using separate prefs for home
30.05.2017 13:27:26 | | Reading preferences override file
30.05.2017 13:27:26 | | Preferences:
30.05.2017 13:27:26 | | max memory usage when active: 16328.59MB
30.05.2017 13:27:26 | | max memory usage when idle: 24492.88MB
30.05.2017 13:27:26 | | max disk usage: 10.00GB
30.05.2017 13:27:26 | | don't use GPU while active
30.05.2017 13:27:26 | | (to change preferences, visit a project web site or select Preferences in the Manager)
30.05.2017 13:27:26 | | Not using a proxy
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#15 Re: Xeon - not utilising more than 32 threads

Post by noetus »

And in case you are in doubt that I hit my funny bone and installed Win 8.1 when I thought I was installing Win 10 Enterprise, here's a screenshot of of HWiNFO64 too:

https://ibb.co/ffLbna

Like I said, this whole thing is giving me a headache. For something that is supposed to be easy to install and start using, I am wasting an awful lot of my time on it.
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#16 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

You are running Boinc version 7.2.47 which, based on other forum posts, defaults to only using 32 cpus/threads. Upgrade to 7.6.33 found here https://boinc.berkeley.edu/dl/boinc_7.6 ... 4_vbox.exe

Might want to dump these options in your cc_config.xml

<client_download_url>http://www.worldcommunitygrid.org/reg/m ... wnload_url>
<client_version_check_url>http://www.worldcommunitygrid.org/downl ... _check_url>
<network_test_url>http://www.ibm.com</network_test_url>
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#17 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Why did the 'Download' button on the world community thread website give me an out of date client?
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#18 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Testing this now. You might want to put this as a sticky somewhere obvious. My issue - 32 threads only - was right there in the topic title, and it took all this time to figure out it is a known/documented issue.
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#19 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

Don't know why WCG only has a link for that version. It appears it's a 32 bit version of boinc which is what I assume is causing the 32 core/thread limit.
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#20 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

Where did you get that version of BOINC? The reason I ask is that it doesn't show as ever having been released .... https://boinc.berkeley.edu/dl/. If you want to install BOINC w/o VirtualBox you can download the correct 64 bit version from the site listed.

With the problems you are having I would suggest a "clean" install. Meaning after you uninstall from Windows then DELETE the 2 BOINC folders ... c:\Program Files\BOINC and c:\ProgramData\BOINC.

EDIT: As usual I'm a day late and a dollar short ... looks like you guys figured it out while I was typing.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#21 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Yeah, I figured out that the non Virtual Box version can be downloaded here:

http://boinc.berkeley.edu/download.php

And I also figured out that I need to do a clean install, because it installed in 32 bit location last time.

I downloaded the earlier version from:

https://www.worldcommunitygrid.org/viewLogin.do
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#22 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Download link is on top right.

The file name is wcg_boinc_7.2.47_windows_intelx86, and the file is dated 27-05-2017 16:26 CET, which is when I downloaded it.

So indeed it was the 32 bit version, hosted by WCG.

Pretty weird. Apologies for blaming you good people for not figuring this out earlier - seems this was caused by out of date client and moreover 32 bit client inadvertently hosted by WCG. What do you get if you click on that download link now?
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#23 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

No problem. We're glad to have you crunching with us. :-)
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#24 Re: Xeon - not utilising more than 32 threads

Post by noetus »

OK, so this is a completely different client - different look, and I'm assuming different menu structure, which is why your directions to find the "Event Log" did not work in my case.

Also provides connection to a much wider variety of Projects - there is more out there than provided by World Community Grid. I wasn't even aware of this.

Might be worth telling newbies about this in a welcome post, or putting a sticky somewhere. More of the story: don't download the client hosted by WCG, use the Berkeley client instead, which gives access to a larger range of projects, is more uptodate and less buggy ;-)
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#25 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

The latest Boinc client project list is not complete or up to date either. You can go to the BoincStats website here https://boincstats.com/en/stats/-1/team/list/

This is the Team Stats page but look/scroll down the left side and look for the "Active projects" list. Some projects have limited or intermittent work and can go without any work for a while. Look at the project "Credit Overview" tab and see how many credits have been crunched over a period of time. That will give you an idea of project work availability.

As an example, here is the credit overview for WCG, which has been pretty steady except for the few days during their recent server migration https://boincstats.com/en/stats/15/proj ... ail/credit

Here is Lattice which ran out of work at the beginning of May but will likely have work in the future if you keep checking back https://boincstats.com/en/stats/7/project/detail/credit

If you click on the project name in the top right section of the stats overview page it will take you to the project website where you can learn more about the project. We have a section in our forum we try to keep up to date with project info also...https://www.tsbt.co.uk/forum/viewforum.php?f=171
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#26 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Thanks for that info.

Seems like the client hosted by WCG is a fork of IBM's.

Can confirm that I now have 56 threads. However a different issue has surfaced, which is that CPU usage is jumping around all over the place. Seems like the client keeps on throttling back (which it wasn't before). Is there an easy fix for this in settings somewhere or should I post about that as a separate thread?
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#27 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

Welcome to the world of BOINC noetus ... did you happen to notice how many releases of "stable" code they've had over the years? :roll:

Welcome to the team and enjoy your new found projects.

BTW, on some of the WCG projects (like SCC) the WU you turn in will require a "wingman" to also turn in the same WU before it will be accepted as valid and you are awarded credits. Once you have 12-15 WU validated then it will put your machine into the "trusted" category and from that time forward it will issue WU that won't require a wingman. The sub-projects I can remember that work this way (trusted) are SCC, Zika, FAAH2 ... there may be a couple of others. The remainder of the sub-projects do not have a trusted mode.

The best technique is to get your machine trusted before opening up the machine so it "caches" a bunch of WU - meaning it downloads WU and puts them into a waiting state so when you finish a WU and new one begins crunching immediately. There are 2 ways to do this:

1. In the WCG preferences you can set resource share to 0 (zero). When a project is set to 0 share then BOINC will only request enough work to keep your available threads busy. It will finish a WU and request a new one.

2. In your BOINC preferences (on the client) you can set the min/max work to .01/.01 and it will accomplish the same thing.

Once the project starts showing that you have 12-15 valid WU and the "in progress" WU are not showing the WU being issued to 2 users then you can open up your cache because the machine has become "trusted" and is golden. You can check the status of "in progress" WU by selecting Results Status in WCG, selecting the computers, project, and then in progress. Click on any of the WU that are in progress and it will give you a popup window that will either show 2 users or 1.

One other thing is that most of us use a program called BoincTasks to control our machines. You install it on one machine and then you can control all other machines from that one location. It is easier that having to bring up a monitor etc every time you want to change something on a machine.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#28 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

And since you are new to Boinc, got ask if you have used any systems for this type of high CPU utilization processing before, have decent heatsinks on your CPUs and good air flow in the cases. Would hate to see those CPUs get toasted. Some folks pay more attention to temps than others. Some not at all. So much to cover. We haven't even touched on GPUs yet :lol:
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#29 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Oh, don't worry about CPU cooling in my case - I make sure to have excellent cooling. When running my own cpu intensive tasks I have been having these machines sitting at 100% for weeks at a time, temps never go out of the 50s. Built the systems myself from scratch, too.

I figured out the whiplashing CPU usage, by writing in 100% for *both* "Use at most 100% of the CPUs" and "Use at most 100% of CPU time" (advanced view). The latter is supposed to prevent the OS from throttling frequency based on high usage/temps but (a) I don't need this on the servers, and (b) it doesn't work anyway on my laptop which does do this sort of frequency throttling.

However now CPU usage is stuck at a steady 85%, which is better than 57% it was before. But why not 100%? On machines with lower core count it sits steady at 100% no problem, so why doesn't it do this on a machine with high core count? The full count of 56 threads is present, just not utilised to full 100%. There must be something more I can do, but what?
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#30 Re: Xeon - not utilising more than 32 threads

Post by noetus »

Restared machine and client, everything good now!

Thanks for all the other info!
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#31 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

The use cpu time is intended to be a throttle used mostly for temperature control. I've never had it below 100% myself.

I'm not in a position right now to move a machine over to WCG and check those particular WU. Depending on the particular project/executable you will see some variation in CPU utilization. Typically they will range from 93-100% loading however. The worst are some of the multi-threaded apps. On those it is very common for it to use X number of threads and at times a number of those threads will be sitting idle.

It is not uncommon for a WU to start slowly on utilization because it is doing some initialization stuff etc. As it progresses the usage will increase. Most of the projects will hammer the CPU from the beginning.

I did just put a 12 thread machine on WCG SCC. 8 threads on WCG and 4 threads reserved for feeding a GPU. When the 8 WU 1st started they were showing 82% utilization. Now at the 10 minute mark they are up to 90-92% so the behavior you are seeing appears to be normal.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#32 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

Bryan had a good idea to go ahead and install BoincTasks. It allows you to manage Boinc running on many different systems and provides extra info not available in the Boinc Manager program such as displaying the CPU % a task is using. Need to verify you have 56 WUs running and what the CPU % is on each of the running WUs.
http://efmer.com/b/?q=boinctasks_download
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#33 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

noetus, I'll give you a little more information about BOINC. BOINC was designed with the intention that everyday users would donate "spare" CPU cycles to science. There are a lot of people who do this and they typically fall into the set-and-forget class. These are folks who hear about Seti (the biggest) or maybe like yourself WCG. They attach to the project, use their spare CPU cycles, and never pay any attention to the project or how many credits they earn.

The 2nd class are the folks that crunch a project because they have a strong desire to help the particular science being done. It might be a cure for a disease or finding a new prime number or ..... Some of these people will be like the set-and-forget guys and others will fully dedicate machines to running a project.

Then there is the hobbyist crowd. These are the folks who may have multiple machines that are dedicated to only crunching BOINC projects. They violate the original intent/concept of BOINC to use only "spare" cpu cycles. These people build machines and then use them for nothing except running BOINC. Many of these people are credit chasers, badge whores, or they even participate in team challenges ... ie Scole would be an example of that crowd. :lol: There might be a few more of those types running around - I certainly wouldn't know. :roll:

The point is, regardless of how you wish to proceed you can have fun with it. Having fun is what it is all about.

Some of the projects issue badges. For example, when you get 14 days worth of validated WU on WCG SCC they will award you with a badge. When you hit 45 days the badge will change color ... etc etc etc. If you run another of the WCG sub-projects then at the 14 day mark you get another (different) badge.

If you think you may do something besides being a set-and-forgetter then you should attach each machine to a project called WuProp which stands for WU Properties. This project is NCI (non computer intensive) so it runs along side whatever you are crunching. It collects data about the projects ie amount of memory required etc etc. Additionally when you run a sub-project for 100 hours you will be awarded a "little" star. If you run another sub-project for 100 hours you will get a 2nd "little" star. When you have 20 of those little stars then it will give you a big star added to a ring. The more hours you have the different color of your little star etc etc.

Attach each machine to: http://wuprop.boinc-af.org/
Image
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#34 Re: Xeon - not utilising more than 32 threads

Post by Alez »

I think you've now got most of it sorted, but to run things smoothly, never have use at most % of cpu time at less than 100%. Set anything less and it simply ramps the cpus up and down, either on or off. Why it's still in manager I don't know as it's and extremely poor way to control your systems.
Secondly, don't get greedy. Leave a core or two free for the system to run on and to handle i/o etc. For example, set use at most % of the cpu's to 93.75% will use 30 of the 32 cores.
All active projects have some info about them on the forum. Try a few different ones and run those you like, discard the others that don't fit with what you want.
Image
The best form of help from above is a sniper on the rooftop....
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#35 Re: Xeon - not utilising more than 32 threads

Post by noetus »

My machines aren't dedicated to the project. I only donate spare CPU cycles. The BOINC processes are all set to idle priority and don't interfere with my day-to-day work at all. Don't notice a thing, even at 100% CPU usage, so I'm fine with that. And as mentioned earlier, I solved the ramping up and down issue by also setting "Use at most ___ % of CPU time" to 100% (in addition to "Use at most ___ % of the CPUs" at 100%). It's all steady as a rock now, 100% usage all the time, system is as fast and responsive for my own tasks as before.

The primary purpose of the machines is my own crunching (fintech big data project) and for that my own processes take priority, and also keep the machines all working at 100% when they are working on my own data. In order to make sure that 100% of my CPU cycles are dedicated to my own work when the machines are occupied with that, I have simply set the "Suspend when non-BOINC CPU usage is above ___ %" to 50%. That means when my machines ramp up with the local jobs all the BOINC stuff suspends and only takes up memory (of which there is plenty).

:-)
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#36 Re: Xeon - not utilising more than 32 threads

Post by noetus »

OK, not all smooth. Since installing the new client each machine now has two entries in my device management page on the WCG portal. Seeems like maybe the new client didn't recognise the machine and set a new machine ID or something? Bizarrely, though, it didn't do this for every machine - one of them wasn't doubled up like this (despite also getting the new client). It's rather aggravating to see double listings for (almost) all my devices, and also to see an inaccurate report of the number of devices I have on my info page.

Here's what I mean:

https://ibb.co/c9akfv

Anything can be done about this? Presumably it would require manual intervention by WCG technical staff - not a very likely prospect I'm guessing.
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#37 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

I don't think there is anything you can do on WCG. On projects that run the standard BOINC supplied server code you can "merge" computers. That function will check the machines if it has 2 different IDs for a machine it will merge the stats and show only 1 machine.

WCG runs there own server code and I don't think they have that capability.
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#38 Re: Xeon - not utilising more than 32 threads

Post by Dirk Broer »

With the Intel® Core™ i9-7980XE (not to mention CPUs with an even more EPYC number of cores/threads) just around the corner,
nice to know that the number of threads should not be a problem (but wasn't Windows limited to 64 threads? Well, we have Linux for that problem...)
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#39 Re: Xeon - not utilising more than 32 threads

Post by noetus »

And the upcoming Xeon Platinum 8180 with 28 cores. That's 112 threads on a dual processor motherboard.

Windows 10 supports up to 256 cores (512 threads with hyperthreading), so we don't have anything to worry about for the time being...
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#40 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

noetus wrote: Wed May 31, 2017 10:08 pm And the upcoming Xeon Platinum 8180 with 28 cores. That's 112 threads on a dual processor motherboard.

Windows 10 supports up to 256 cores (512 threads with hyperthreading), so we don't have anything to worry about for the time being...
You might want to take a look at the Windows NUMA implementation which is only 64 threads. That applies to ALL versions of Windows including the server series.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#41 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

Bryan wrote: Thu Jun 01, 2017 1:35 am
noetus wrote: Wed May 31, 2017 10:08 pm And the upcoming Xeon Platinum 8180 with 28 cores. That's 112 threads on a dual processor motherboard.

Windows 10 supports up to 256 cores (512 threads with hyperthreading), so we don't have anything to worry about for the time being...
You might want to take a look at the Windows NUMA implementation which is only 64 threads. That applies to ALL versions of Windows including the server series.
The thing to point out is Windows will see all the cores/threads but the OS architecture limits the number of concurrent pipes to memory at 64. You will see 72 threads active on a 72 thread system but those processes must share 64 pipelines to memory and you will see less than 100% utilization on those threads.
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#42 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

@Pete, nope! :P

NUMA is the new memory architecture used by both Windows and Linux. Its purpose is to speed things up (which it does) by having the thread use the memory that is tied to the CPU that the thread is running on. If it were to use memory on the other CPU it would have to access that by having the memory controller contact the other CPU's memory controller over the QPI link which is much slower because the QPI link is not as fast and 2 memory controllers get involved. In normal operation this would never happen unless the process needed more memory than was available on the local CPU.

On Windows there are only 64 NUMA channels - period. It doesn't matter how many threads the CPU's have because they will be sharing the 64 NUMA pipes. If you run more than 64 threads simultaneously it will start "sharing" the memory pipes. That means that it has to load the memory pipe for 1 thread and then reload for the 2nd thread. This is inefficient and means that you will actually get less results from running 72 threads than by just leaving it at 64.

In the real world where people who buy and use machines with > 64 threads you can rest assured they know what they are doing (I don't). There are commands in NUMA that allow you to set up CPU nodes and each node would have 64 NUMA pipes. In the case of my 72 thread machines I would assign CPU0 to Node 0 and assign CPU1 to another node, Node 1. Then I would assign an instance of BOINC to run on CPU0 and another instance of BOINC to run on Node1. In effect you have a machine with 2 CPUs but they run independent of one another. Some of the E7 Xeons allow you to run 8 CPUs at a time. This is the way it would be done on those machines ... or they would use sophisticated VM hypervisors.

If I put my 72 thread machines on Win7 and use ncpus to limit BOINC to 64 threads it still doesn't like it. On those machines I turn off HT if I need to run Windows. That is the reason the last 2 machines I built have only 64 threads. On those I can switch between Linux and Win and not have to screw with it.

BTW, I'm sure that Linux has some kind of limit on NUMA threads but I can't find what it is. I know they will handle 88 thread machines without a problem.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#43 Re: Xeon - not utilising more than 32 threads

Post by noetus »

"On Windows there are only 64 NUMA channels - period. It doesn't matter how many threads the CPU's have because they will be sharing the 64 NUMA pipes. If you run more than 64 threads simultaneously it will start "sharing" the memory pipes. That means that it has to load the memory pipe for 1 thread and then reload for the 2nd thread. This is inefficient and means that you will actually get less results from running 72 threads than by just leaving it at 64."

I don't see how this can be right as a matter of principle. Once there are more than 64 logical cpus and thus multiple processor groups and sharing of NUMA channels of the same NUMA node, how much performance suffers will be a matter of empirically testing. I can't see the justification for making a blanket statement that there is no point in using more threads because the performance will be the same or worse. It is going to depend on the specifics of the architecture, how the application is coded, and just how much performance gain there is anyway from NUMA in the first place.

I will soon be able to test this empirically on an 88 thread machine, in a couple of weeks probably, so I'll test and report back here.
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#44 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

Have fun :) If you find it isn't a problem then I know a LOT of people that will be purchasing Win10 Enterprise! You can turn off the hardware NUMA in your BIOS and then you need to turn it off in Win as well.

You might want to install Boinc Tasks ... it makes it real easy to see the loading on each of the threads (or you can use Task Manager).

Here is some "light" reading for you: https://msdn.microsoft.com/en-us/librar ... s.85).aspx

In essence the problem is NOT the architecture, nor is it the executable from the project. The problem is that the BOINC client does not specifically take into account the systems NUMA nodes when it launches a thread. Ideally it would specifically assign a new thread to a NUMA node but instead it relies on the Windows scheduler to make that determination. The Win scheduler uses 64 NUMA threads.
The reason for initially limiting all threads to a single group is that 64 processors is more than adequate for the typical application. An application that requires the use of multiple groups so that it can run on more than 64 processors must intentionally determine where to run its threads. The application is responsible for setting thread affinities to the desired groups.


Then of course Windows may work differently with your 88 threads machines than it does with my 72 thread Xeon V4s ... hopefully that will be the case.

Linux doesn't have the same limitation BTW. It will run your 88 threads at 100% loading /thread.
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#45 Re: Xeon - not utilising more than 32 threads

Post by Dirk Broer »

Bryan wrote: Fri Jun 02, 2017 4:04 pmThe reason for initially limiting all threads to a single group is that 64 processors is more than adequate for the typical application.
Where have I heard this line of argument before? Sounds like "640K ought to be enough for anybody" (supposedly said by Bill Gates around 1981) or "I think there is a world market for maybe five computers" (supposedly said by Thomas J. Watson of IBM in 1943).
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#46 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

@Dirk ... I think you broke the code :lol:
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#47 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

Bryan, Read this https://support.microsoft.com/en-us/hel ... -windows-7

Might be solution. Haven't tried it yet.
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#48 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

scole250 wrote: Fri Jun 02, 2017 7:08 pm Bryan, Read this https://support.microsoft.com/en-us/hel ... -windows-7

Might be solution. Haven't tried it yet.


Now that is interesting - very very interesting. Obviously the "hotfix" is irrelevant since it is dated 2010 and is incorporated in later releases/updates. However, the info on how to assign a process to a node is great.

You wouldn't be able to assign boinc.exe to 2 nodes (probably), however if you ran 2 instances of boinc with one being boinc.exe and the 2nd being boinc2.exe and then using ncpus to limit each to (total threads/2) then it might work :D That is worth playing with.

My machines are totally tied up for the next 4-5 days on Amicable Numbers - doing run to 100M. So when will you get a chance to try it Steve? :lol:

BTW, how much do you know about Python?

@noetus - you can install and simultaneously run multiple instances of BOINC on your computer. There are times when this is beneficial ... like when running a team competition. The "ncpus" is a command that can be placed into your cc_config.xml file and it tells BOINC to simulate that many CPUs/threads. For this discussion and your upcoming machine, you would set ncpus to 44 in each of your boinc instances.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#49 Re: Xeon - not utilising more than 32 threads

Post by scole of TSBT »

I can't recall if I reinstalled win7 on the 72 thread system last time it flaked out but I'll try to give it a whirl this weekend.
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#50 Re: Xeon - not utilising more than 32 threads

Post by Bryan »

scole250 wrote: Fri Jun 02, 2017 8:26 pm I can't recall if I reinstalled win7 on the 72 thread system last time it flaked out but I'll try to give it a whirl this weekend.
That was too easy, I should have asked for more :roll:
Image
Post Reply Previous topicNext topic

Return to “Benchmarking and Hardware”