Project

General

Profile

Feature #1493

AVX2 detection causes make to give assembler errors

Added by James Barnett over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Low
Assignee:
Category:
build system
Target version:
Difficulty:
uncategorized
Close

Description

When compiling Gromacs 5.0-rc1 I use the following cmake command in a subdirectory of the source:

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/bin/gromacs-5.0-rc1 -DCMAKE_C_COMPILER=/usr/local/bin/gcc-4.9.0 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-4.9.0

Then running make I get several errors. Here are the first few lines:

Scanning dependencies of target gmock
Scanning dependencies of target analysisdata-test-shared
Scanning dependencies of target onlinehelp-test-shared
Scanning dependencies of target tng_compress
[  0%] Building C object src/external/tng_io/src/compression/CMakeFiles/tng_compress.dir/bwlzh.c.o
[  0%] [  0%] Building CXX object src/external/gmock-1.7.0/CMakeFiles/gmock.dir/src/gmock-all.cc.o
Building CXX object src/gromacs/onlinehelp/tests/CMakeFiles/onlinehelp-test-shared.dir/mock_helptopic.cpp.o
[  0%] Building CXX object src/gromacs/analysisdata/tests/CMakeFiles/analysisdata-test-shared.dir/datatest.cpp.o
/tmp/ccOnlP2l.s: Assembler messages:
/tmp/ccOnlP2l.s:477: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:479: Error: suffix or operands invalid for `vpsrld'
/tmp/ccOnlP2l.s:480: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:482: Error: suffix or operands invalid for `vpsrld'
/tmp/ccOnlP2l.s:483: Error: suffix or operands invalid for `vpor'
/tmp/ccOnlP2l.s:484: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:485: Error: suffix or operands invalid for `vpsrld'
/tmp/ccOnlP2l.s:486: Error: no such instruction: `vpermq $216,%ymm6,%ymm7'
/tmp/ccOnlP2l.s:487: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:488: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:489: Error: suffix or operands invalid for `vpsrld'
/tmp/ccOnlP2l.s:490: Error: suffix or operands invalid for `vpor'
/tmp/ccOnlP2l.s:491: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:492: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:493: Error: no such instruction: `vpermq $216,%ymm10,%ymm11'
/tmp/ccOnlP2l.s:494: Error: suffix or operands invalid for `vpor'
/tmp/ccOnlP2l.s:495: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:496: Error: no such instruction: `vpermq $216,%ymm6,%ymm7'
/tmp/ccOnlP2l.s:497: Error: suffix or operands invalid for `vpor'
/tmp/ccOnlP2l.s:498: Error: no such instruction: `vpermq $216,%ymm14,%ymm15'
/tmp/ccOnlP2l.s:499: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:500: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:501: Error: suffix or operands invalid for `vpor'
/tmp/ccOnlP2l.s:502: Error: suffix or operands invalid for `vpshufb'
/tmp/ccOnlP2l.s:503: Error: no such instruction: `vpermq $216,%ymm10,%ymm11'
/tmp/ccOnlP2l.s:504: Error: suffix or operands invalid for `vpshufb'

If I use the following command for cmake I am able to compile and install sucessfully:

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/bin/gromacs-5.0-rc1 -DGMX_CPU_ACCELERATION=AVX_256 -DCMAKE_C_COMPILER=/usr/local/bin/gcc-4.9.0 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-4.9.0

However, then when I run gmx mdrun I get this notification:

Compiled SIMD instructions: AVX_256 (Gromacs could use AVX2_256 on this machine, which is better)

I can go back and run cmake with the following command when compiling and get the same assembler messages as above (it will not compile):

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/bin/gromacs-5.0-rc1 -DGMX_CPU_ACCELERATION=AVX2_256 -DCMAKE_C_COMPILER=/usr/local/bin/gcc-4.9.0 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-4.9.0

I am able to compile and install Gromacs 5.0-beta2 successfully with the following cmake command:

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/bin/gromacs-5.0-beta2 -DCMAKE_C_COMPILER=/usr/local/bin/gcc-4.9.0 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-4.9.0

AVX2_256 was not an option for DGMX_CPU_ACCELERATION in beta2 (trying to do so gives an error). Also there is no output line when running mdrun for beta2 to incdicate if AVX2_256 was used or if AVX_256 was used.

I am running on CentOS 6.5, 64-bit. The output of /proc/cpuinfo shows a flag "avx2" for all of my processors.

make.log (21.9 KB) make.log make VERBOSE=1 output James Barnett, 05/02/2014 02:33 PM

Associated revisions

Revision ef0530b7 (diff)
Added by Erik Lindahl over 5 years ago

Make it harder for compiler to optimize away SIMD CMake tests

On at least one old version of Linux, a new compiler in
combination with an old assembler led to the compiler
understanding the SIMD code but optimizing it away, which made
the test pass even though the assembler could not handle it.
This changes the return value of the CMake tests to be based on
the SIMD operations, which should make them a lot more
difficult to optimize away.

Fixes #1493.

Change-Id: I3e021c3c718cf54afaadf131c5fa911b3933f61e

History

#1 Updated by Mark Abraham over 5 years ago

AVX2 was added after 5.0-beta2, so that part of the behaviour is normal.

As far as I know, we haven't been testing the compile with GCC 4.9, but we will do so now that it is out. Can you please provide the output from

make VERBOSE=1

in the failing case?

#2 Updated by Roland Schulz over 5 years ago

I just tested it with gcc 4.9 on OpenSuse 13.1 and it works. Thus I don't think this is related to gcc 4.9. I have GNU assembler "as" 2.23.2. I suspect that your "as" version is too old to support AVX2. What as version do you have? Does "man as" list avx2?

We could add a test that we actually try to compile a program with avx/avx2 during cmake. That would not only detect if the compiler supports the avx/avx2 flag but that the assembler does too. We had the same problem on Mac with avx (#1021). I suggested already then to add such a test.

#3 Updated by James Barnett over 5 years ago

Mark Abraham wrote:

AVX2 was added after 5.0-beta2, so that part of the behaviour is normal.

As far as I know, we haven't been testing the compile with GCC 4.9, but we will do so now that it is out. Can you please provide the output from

make VERBOSE=1

in the failing case?

I've attached the output from make VERBOSE=1 for the failing case.

#4 Updated by Mark Abraham over 5 years ago

That confirms that the -march=core-avx2 is present as expected, so I think GROMACS is doing OK, but your assembler is too old, per Roland's suggestion. What is its version, please?

#5 Updated by James Barnett over 5 years ago

Roland Schulz wrote:

I just tested it with gcc 4.9 on OpenSuse 13.1 and it works. Thus I don't think this is related to gcc 4.9. I have GNU assembler "as" 2.23.2. I suspect that your "as" version is too old to support AVX2. What as version do you have? Does "man as" list avx2?

We could add a test that we actually try to compile a program with avx/avx2 during cmake. That would not only detect if the compiler supports the avx/avx2 flag but that the assembler does too. We had the same problem on Mac with avx (#1021). I suggested already then to add such a test.

I have GNU as version 2.20.51.0.2. I don't see avx or avx2 in the manpage.

So I decided to get a newer version of the GNU assembler, and upgrading GNU binutils seems to solve this issue. I compiled version 2.24 of binutils and installed it in my home directory. I then used the following command to find out where gcc searches for as and ld:

gcc -print-search-dirs | grep '^programs:'

Since I have gcc 4.9.0 installed side by side with the original gcc I placed a symbolic link to both as and ld in this directory, which was the first one listed from the above command.

Running cmake with -DGMX_CPU_ACCELERATION=AVX2_256 and then make is now successful. Additionally removing this flag also results in a successful build.

#6 Updated by Mark Abraham over 5 years ago

OK great. We should probably look at some kind of configure-time test that checks a full link succeeds. We have a few of these by now...

#7 Updated by Erik Lindahl over 5 years ago

  • Tracker changed from Bug to Feature

Technically not a bug since the problem is in the assembler binary rather than Gromacs, but I'll try to add detection for it.

#8 Updated by Erik Lindahl over 5 years ago

  • Priority changed from Normal to Low

I finally had a look at our tests (in cmake/gmxTestSimd.cmake), and we actually do test linking for each instruction set since we use the cmake macro check_c_source_compiles(). I'm not entirely sure why it accepts your assembler, but it might be that a smart compiler optimizes away the entire simple constant test expression.

If anybody has a system where we can reproduce this, and where they are happy to give me a login, I can look into it - otherwise I'll put this on the back burner for the time being.

#9 Updated by Mark Abraham over 5 years ago

Hmm yes, it seems like the check is insufficient. I was assuming the checks were based on something that only does a compile, because lately we had some such issues with find_package() doing only that. But check_c_source_compiles() is implemented by try_compile() and that does a real link. We could do the SIMD checks at -O0?

#10 Updated by Erik Lindahl over 5 years ago

  • Status changed from New to Fix uploaded

I thought a bit about Mark's suggestion, but decided it would not be a very good idea. In theory it might help us provide better error messages, but at the same time we're just opening a new potential source of compiler bugs if we are not testing the SIMD instructions with the same compiler flags we use for the actual Gromacs compile.

However, I've modified at least the x86 SIMD testcode to return a result based on the SIMD operations. This will make it a lot more difficult for the compiler to optimize away those instructions.

#11 Updated by Gerrit Code Review Bot over 5 years ago

Gerrit received a related patchset '1' for Issue #1493.
Uploader: Erik Lindahl ()
Change-Id: I3e021c3c718cf54afaadf131c5fa911b3933f61e
Gerrit URL: https://gerrit.gromacs.org/3649

#12 Updated by Erik Lindahl over 5 years ago

  • Status changed from Fix uploaded to Resolved

#13 Updated by Erik Lindahl over 5 years ago

  • Status changed from Resolved to Closed

#14 Updated by Teemu Murtola over 5 years ago

  • Assignee changed from Mark Abraham to Erik Lindahl
  • Target version set to 5.0

Also available in: Atom PDF