git index.lock issue
We still have the problem that the index.lock is sometimes left behind and then subsequent build fail. This usually happens on Windows and when the previous build crashes with "socket closed". An example is: http://jenkins.gromacs.org/job/Gromacs_Gerrit_master/5970/OPTIONS=Compiler=icc%20CompilerVersion=12.1.2%20GMX_GPU=OFF%20GMX_DLOPEN=OFF%20GMX_THREAD_MPI=OFF%20GMX_FFT_LIBRARY=fftpack%20CMAKE_BUILD_TYPE=Release%20host=bs_Win2008_64,label=bs_Win2008_64/consoleFull (socket closed) and http://jenkins.gromacs.org/job/Gromacs_Gerrit_master/5972/OPTIONS=Compiler=icc%20CompilerVersion=12.1.2%20GMX_GPU=OFF%20GMX_DLOPEN=OFF%20GMX_THREAD_MPI=OFF%20GMX_FFT_LIBRARY=fftpack%20CMAKE_BUILD_TYPE=Release%20host=bs_Win2008_64,label=bs_Win2008_64/consoleFull (next build using the same workspace which failed).
The reason for the socket closed is probably a reboot of the VM. I'm not sure we still do this for backup. If so that would be something we could tune but even than the VM could still reboot for other reasons. https://issues.jenkins-ci.org/browse/JENKINS-12513 tracks this bug for Jenkins but there is no progress there. We could either implement this ourselves in the git jenkins plugin or find some workaround.
#1 Updated by Roland Schulz over 5 years ago
One option would be to use https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin
#3 Updated by Roland Schulz over 5 years ago
I think so. Maybe better would be to delete the whole workspace (in which case we automatically do a full clone). I had it at least once that the lock was there and the repository was corrupted. But just deleting the lock would cover at least 95% of the problems. BTW: We only have the problem because we do not always delete the workspace, because otherwise we put a lot of stress on gerrit (and make it slower) if we do a full clone everytime.
#4 Updated by Roland Schulz over 5 years ago
We today had a similar issue. The .git/config file was corrupted causing builds to fail. I think the only way to avoid any of those kind of errors is to use "Wipe out workspace before build" and then redownload git every time. This could be made faster by using "Use shallow clone". Also Gerrit Trigger 2.12.0-beta-3 allows to wait until replication is done. Thus we don't need to download from gerrit but could download from git.gromacs.org to avoid overloading gerrit. We could even replicate to all jenkins nodes and then download from a local folder. But that last option might have the same problem as the current solution. I assume the corrupted file today was caused by Windows restarting/crashing. If that happens during the replication that repository might get corrupted too and we might have to manually fix that too.