Xeon server build

Forum to discuss and compare Hardware profiles and Benchmarking
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#51 Re: Xeon server build

Post by Bryan »

Excellent vendor ... I've bought 3 pairs of CPUs from them.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#52 Re: Xeon server build

Post by noetus »

pinhodecarlos wrote: Sun Jul 02, 2017 4:04 am Entire system.
On that budget I am guessing you will go single-socket.

I would figure out how much it will cost to buy the system minus the CPU, then see what is left over for the CPU. For example, you might already have a computer case lying around, or other bits and pieces such as a PSU you can put into it without having to spend much. Perhaps you are going to cannabilise an old system of yours for parts. Or have a look on eBay and see if you can find a PC with bad motherboard someone is trying to flog, that could form the basis of your system. Think creatively!

Then try pricing the components for various types of motherboard you are thinking of For example, if you price a used LGA-2011 motherboard (for v2 or v3 E5 processors) on eBay for the kind of case you envisage (eg. ATX or ITX) then you can view the selection of processors for that board and try to figure out which will get you the best bang for the buck with what you still have left in your budget. Try also LGA-2011 v3 (for later v3 and v4 processors), but you might find that pops your budget when you add in other components.

When you have a possible configuration, post it here (with prices for each component) for some advice. At this stage it would just be an idea, a suggestion to start off some discussion, nothing written in stone.
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#53 Re: Xeon server build

Post by Dirk Broer »

Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#54 Re: Xeon server build

Post by noetus »

That's a great article - it's what got me into these Xeon server builds in the first place! Thanks for posting it.
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#55 Re: Xeon server build

Post by noetus »

How about the following 24 core / 48 thread build:

Component         Model & Source             Price (£)

Motherboard:      ASRock EP2C612D8 Dual Socket     300
CPU:          2 x Intel E5-2648L 1.8GHz 12 Core    270
Memory:       4 x Kingston KVR21R15D8/8 (32 GB)    280
CPU cooler:   2 x Noctua NH-U12DX i4               110
Power supply:     Seasonic G450 450W 80+ Gold       75
PWM Fans:     2 x Noctua NF-A14 PWM                 50
ULN Fan:          Noctua NF-A14 ULN                 20
System Drive:     Samsung 850 EVO 120 GB            50
Data Drive:       Western Digital Red 3 TB         100
Case:             Fractal Design Define R4          75
TOTAL:                                            1330
  • Obviously the total is a bit more than you proposed as a budget, but I just wanted to showcase what is possible, and there are various ways of saving costs. If you already have a case and drives lying about or that can be cannabilised, that could save over 200. You can also drop down to a single socket board (EPC612 series), and/or get a processor with fewer cores. That could save a couple of hundred. You didn't specify your memory needs, and memory is expensive for these servers. You could save another 140 by going for 16GB instead of 32GB. The above list is just a starting off point, and offers some flexibility.
  • The article cited by Dirk features the ESRock EP2C602 series of boards. I recommend avoiding these boards as they are the by now outdated LGA 2011 boards which only let you install CPUs that are v3 (less power efficient) and only up to 8 cores (they maxed out their build at 16 cores / 32 threads). By getting the EP2C612 series (LGA 2011 v3) you can use v3 or v4 (with a BIOS update) processors with up to 22 cores each. That gives you much more flexibility now and in the future if you ever want to upgrade (just pop in replacement processors) and I found the cost to be practically the same for the two board variants when I did a quick price check.
  • The processors I only gave as an example. I linked to a QS (Qualification Sample) listing on eBay, but you can go for the cheaper ES (Engineering Sample) processors although I do not recommend doing that. You can choose a lower clock or higher clock, or fewer cores or more cores, Haswell or Skylake, and so on. There is really a lot of flexibility here, and it will depend on your specific usage scenario. I do think that the 12 core CPUs offer a very good performance/price tradeoff, especially if you install two of them. Before pulling the trigger on any processor purchase, make sure you post about it here with a specific link. And always check with the vendor about motherboard compatibility before purchasing, and tell them the specific motherboard you have or are planning on buying so they can 100% check availability. This is a bit of a hornet's nest, with all the processor variants about.
  • The memory I suggested is on the motherboard manufacturer's QV (qualified vendor) list and is proper ECC server memory. I would recommend using only QVL memory for any server board; you can find the list for this board here. It is clocked at 2133 and you could go for 2400 but there is little point; it is more expensive and offers only fractionally better performance.
  • I like Seasonic as a PSU manufacturer. Corsair is better known, but Seasonic supply PSUs for a lot of the OEMs and the quality is good (better than Corsair imho which is just well-marketed).
  • I've suggested purchasing three case fans for this build, along with the Fractal Design R4 case. I really like this case; it is cheap but doesn't feel cheap, and it offers lots of flexibility. It already comes with two servicable front fans which you can just leave in place, and the additional 3 Noctua fans will provide plenty of air circulation. I would mount the two PWM fans on the top of the case as exhaust fans, and the one ULN fan at the rear as an inlet fan. You could consider the purchase of an additional ULN fan as an inlet for the case side panel, but it isn't necessary (just leave the cover off). It is better to run more fans each at lower RPM; if you run the PWM fans with Noctua's low noise adapters (supplied in the fan packaging) you will find that the build is extremely silent, silent enough to be in your living room with all cores maxed out.
  • With some components it is worth ordering from the USA. I would do this with the motherboard and the memory; these are both easy to pack and ship, and the US prices seems significantly better. The Newegg deal on that motherboard is especially good. To order from Newegg, you might have to use a US-based reshipper such as reship.com to get the thing sent to the UK (with minimal or no customs charges) as they won't ship internationally; I've factored in an extra $50 to cover this. If you can't do this, that motherboard is available from Germany on the German eBay site, though for a bit more.
  • Definitely install your OS on a separate drive and make sure it is an SSD.

    I'll finish off by saying I've had personal experience building using all the above components (except for that particular processor and that particular Seasonic PSU; I've built with other Seasonics though) and I have never had any issues.
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#56 Re: Xeon server build

Post by Alez »

Jeez, I've just finished upgrading /updating all my GPU's ..... now you've got me thinking of updating my servers from gen 6 / 7 HP's #sadface and depleted wallet.
Image
The best form of help from above is a sniper on the rooftop....
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#57 Re: Xeon server build

Post by pinhodecarlos »

Thank you all. I've read everything and basically I need to invest in all parts since I only own laptops. I'm considering those old HP Z200 but I need to read more.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#58 Re: Xeon server build

Post by scole of TSBT »

See what it cost to for shipping and customs for these systems...
Z600, 2x X5675 3.06Ghz , 48GB RAM

Z800, 2x X5690 3.46Ghz
48GB RAM for this system

Or perhaps a series new LGA 2011 system...
Dell T7600, 1x E5-2687W 3.1/3.4Ghz, 32GB RAM
2nd E5-2687W CPU
CPU cooler
additional 32 GB RAM (assuming base system using 4x8GB sticks)

If the apps you want to run have more need for Ghz than newer chipsets, memory speed and other chip features such as AVX2, then those series systems/cpus are not bad for the money
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#59 Re: Xeon server build

Post by noetus »

Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#60 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Thu Jul 06, 2017 5:14 pm Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
Newer would be better. What do you recommend for £1,000?
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#61 Re: Xeon server build

Post by Dirk Broer »

noetus wrote: Thu Jul 06, 2017 5:14 pm Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
You have to carefully select your Xeons in each generation. TDP varies wildly, together with the MHz.
Socket 2011 (DDR3) doesn't suffer much compared to the newer Socket 2011-3 (DDR4).
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#62 Re: Xeon server build

Post by scole of TSBT »

From my testing running Carlos' app, 14 threads on a Xeon V4 clocking at 2.8Ghz with 28GB RAM reserved runs about the same as a 2x X5675 (24 threads) using 36GB RAM. A 10c/20t V4 ES system clocking at 2.4Ghz on all threads should do as much. I'd estimate $1,200 to build a single cpu system w/32GB RAM, maybe $100 more for 12c/24t system. That's if you buy it here in the US though. If the parts can be bought and shipped to the UK for £1,000, then it would be worth it. Would use 50%-66% of the lecky a Z600 2x X5675 system uses (~350 watts?). If can save 150 watts, then assuming your power rate is .15 per kwh, you could save around 197 per year.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#63 Re: Xeon server build

Post by noetus »

Mains electricity costs vary in the EU. In US dollars (using Eurostat figures for 2016), in the UK it is 0.20 per kWh, 33% more than what you were basing your calculations on, which is significant. So the payoff for using more recent hardware is considerably more. In Denmark and Germany the price is even higher - 0.35 and 0.34 respectively, more than double what you were assuming! Just using your figures over 2.5 years we are talking about a cost saving of almost the original cost of the system. No wonder the big tech companies turf out all their old systems after a few years and invest in the latest hardware.

Hungary (where I live) and Bulgaria are the cheapest EU countries for power, 0.13 and 0.11 US dollars respectively. So the payoff for newer hardware partly depends where you live. You also have to factor in the higher chance of hardware failure for older equipment, especially server equipment that has seen 24/7 use (after 6 years some equipment, such as SSDs, might be reaching the end of their design life).

The cheapest place in Europe is Kosovo, a mere .07.

A quick glance at electricity prices in the US (from the US EIA) reveals wide state-by-state differences there too, from .27 in Hawaii to .09 in Washington, with prices generally lower than in the EU.

Another factor to consider is that if you buy one of those old servers you're going to get yourself a pretty noisy system, in all likelihood. If you build a system yourself you can make it near silent. And don't forget - you'll have manufacturer's warranties on all those new components of your build (excluding the CPU if you get a used/ES/QS version). I don't think you get any warranty purchasing used server equipment on eBay, maybe DOA that's all.

What is Carlos' app? I wouldn't mind running some tests of my own.
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#64 Re: Xeon server build

Post by pinhodecarlos »

Guys,

I'm an energy consultant and I trying to understand what's the most efficient machine to run several things like prime hunting, factoring (msieve, nfs@home) and prime gaps.

For example, the new intel desktop cpu's throttle when you use avx512.
So prime hunting uses avx, avx2.0 and in the near future avx512. When using Prime95/LLR with lots of cores the memory is the bottleneck.

Factoring (msieve client, or NFS@Home) is pure GHz needed as well as the gaps prime application.

What I don't understand is the example of my ivybridge laptop when running prime gaps on 4 threads (30 W, 18-20e9 n/s, depends on the range being processed) is more efficient than these systems:

6e9 n/s i3-530 2.9GHz 3 thread (Windows/Cygwin)
19e9 n/s E5-2666v3 2.9GHz 8 thread
27e9 n/s i7-4770k 4.3GHz 8 thread
32e9 n/s i7-6700k 4.2GHz 8 thread
34e9 n/s E5-2666v3 2.9GHz 16 thread
59e9 n/s E5-2686v4 2.7GHz 48 thread

My laptop, sandybridge i7- 3630QM.
Untitled.jpg
Carlos
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#65 Re: Xeon server build

Post by pinhodecarlos »

Prime gaps application:
gap5_g_linux.zip
gap5_g.zip
cygwin1.zip



gap5_g -n1 9000e15 -n2 9001e15 -gap 1346 -delta 155 -sb 24 -bs 18 -t 4 -mem 12

Windows:

start /low /min gap5_g.exe -n1 9000e15 -n2 9001e15 -gap 1364 -delta 155 -sb 24 -bs 18 -mem 12 -t 4

-t number of threads
-mem memory allocated
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#66 Re: Xeon server build

Post by pinhodecarlos »

gap5_g_linux_static.zip
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#67 Re: Xeon server build

Post by pinhodecarlos »

I'm also considering the prime gaps is not well optimised for full threads or it has memory bottleneck. I suppose it will be quicker on a multi core machine run several instances of the client even set to a lower 1 thread.
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#68 Re: Xeon server build

Post by Dirk Broer »

pinhodecarlos wrote: Fri Jul 07, 2017 12:27 pmWhat I don't understand is the example of my ivybridge laptop when running prime gaps on 4 threads (30 W, 18-20e9 n/s, depends on the range being processed) is more efficient than these systems:

6e9 n/s i3-530 2.9GHz 3 thread (Windows/Cygwin)
19e9 n/s E5-2666v3 2.9GHz 8 thread
27e9 n/s i7-4770k 4.3GHz 8 thread
32e9 n/s i7-6700k 4.2GHz 8 thread
34e9 n/s E5-2666v3 2.9GHz 16 thread
59e9 n/s E5-2686v4 2.7GHz 48 thread
Make sure each thread has the same amount of RAM, when you make a comparison like this.
IMHO a 48 thread Xeon should have 192GB of RAM (4GB/thread), a 16 thread Xeon 64GB, a 8 thread Xeon or i7 32GB.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#69 Re: Xeon server build

Post by scole of TSBT »

Verify how the different options affect performance...
Usage: gap [options]

Options:
-n1 x First number to check is x
-n2 y last number to check is y
-gap g searching for gap>=g (default=1346)
-delta d sieving on k*m+[0,d) intervals (default=196)
-sb u sieve uses 2^u bits of memory (default=25)
-bs v one bucket size is 2^v bytes (default=14)
-mem m the maximal memory usage is m GB (m can be any real number)
-t k use k threads
Maybe different values for -gap, -delta, -sb, and -bs give better performance on higher thread or memory systems.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#70 Re: Xeon server build

Post by pinhodecarlos »

We are all using the same gap, delta, sb and bs sometimes.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#71 Re: Xeon server build

Post by pinhodecarlos »

Reviving and old thread. Soon my wife will start her PhD which will be based on model simulation. I hope she will have access to high end computers to run gprooms or aspen, otherwise we will have to purchase a machine. I suppose both softwares will run better with higher number of threads and lots of memory. Could anyone advise me on tower racks, budget £1200. Anyone here is an U.K. is seller?

This is much appreciated.

Carlos
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#72 Re: Xeon server build

Post by Dirk Broer »

Image
Post Reply Previous topicNext topic

Return to “Benchmarking and Hardware”