Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between pages "Scripting Eclipse" and "EMFT Build Server Setup"
m (more debug details) |
|||
Line 1: | Line 1: | ||
− | + | = EMFT Build Server Setup = | |
+ | |||
+ | You will need to be root for most of these tasks. | ||
+ | |||
+ | == Set up web content == | ||
+ | |||
+ | * Copy content from an existing server setup. Most of the content exists in CVS as well (<tt style="color:DarkGreen">[http://dev.eclipse.org/viewcvs/indextech.cgi/*checkout*/org.eclipse.emft/releng/common/ /cvsroot/technology/org.eclipse.emft/releng/common]</tt>): | ||
+ | |||
+ | <table> | ||
+ | <tr><td><b>FROM</b> (emf.torolab.ibm.com)</td><td></td><td><b>TO</b> (emft.eclipse.org)</td></tr> | ||
+ | <tr><td colspan="3"><hr noshade="noshade" size="1"/></td></tr> | ||
+ | <tr><td>/home/www-data/build/emft</td><td>--></td><td>/home/www-data/build/emft</td></tr> | ||
+ | <tr><td>/var/www/emft</td><td>--></td><td>/var/www/html/emft</td></tr> | ||
+ | <tr><td>/var/www/technology/emft</td><td>--></td><td>/var/www/html/technology/emft</td></tr> | ||
+ | </table> | ||
+ | |||
+ | == Fix permissions & ownership == | ||
+ | |||
+ | <pre> | ||
+ | cd /var/www/html; find . -type f -exec chmod 664 {} \; | ||
+ | cd /var/www/html; find . -type d -exec chmod 775 {} \; | ||
+ | cd /var/www/html; find . -exec chown apache:www {} \; | ||
+ | |||
+ | cd /home/www-data/build; find . -type f -exec chmod 664 {} \; | ||
+ | cd /home/www-data/build; find . -type d -exec chmod 775 {} \; | ||
+ | cd /home/www-data/build; find . -exec chown apache:www {} \; | ||
+ | cd /home/www-data/build/scripts; find . -type f -name "*.sh" -exec chmod 755 {} \; | ||
+ | </pre> | ||
+ | |||
+ | <p>Apache 2 w/ PHP 5 was already installed. Only a few additional programs needed to be installed as well.</p> | ||
+ | |||
+ | == Install via <tt style="color:DarkGreen">yum</tt> using [http://people.redhat.com/~katzj/yum-xen.conf yum-xen.conf file] == | ||
+ | |||
+ | <pre>yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Java Development" | ||
+ | yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Development Tools"</pre> | ||
+ | |||
+ | == Install & symlink == | ||
+ | |||
+ | * [http://w3.hursley.ibm.com/java/jim/ibmsdks/latest/ IBM JDK 1.4.2sr4-1] (symlink to <tt style="color:DarkGreen">/opt/ibm-java2-1.4</tt>) | ||
+ | * [http://java.sun.com/j2se/1.5.0/download.jsp Sun JDK 1.5.0_06] (symlink to <tt style="color:DarkGreen">/opt/sun-java2-5.0</tt>) | ||
+ | * [http://ant.apache.org/bindownload.cgi Apache Ant 1.6.5] (symlink to <tt style="color:DarkGreen">/opt/apache-ant-1.6</tt>) | ||
+ | |||
+ | == Fix web user (apache) == | ||
+ | |||
+ | * Edit <tt style="color:DarkGreen">/etc/group</tt>. Add <tt style="color:DarkGreen">www</tt> group: | ||
+ | |||
+ | ==== /etc/group ==== | ||
+ | <pre>www:x:501:user,apache</pre> | ||
+ | |||
+ | * Edit <tt style="color:DarkGreen">/etc/passwd</tt>. Change user's home directory and shell: | ||
+ | |||
+ | ==== /etc/passwd ==== | ||
+ | <pre>apache:!!:48:48:Apache:/home/www-data:/bin/bash</pre> | ||
+ | |||
+ | * Edit <tt style="color:DarkGreen">/etc/sudoers</tt> to let you run commands and switch to the web user w/o needing a password. Add the following lines: | ||
+ | |||
+ | ==== /etc/sudoers ==== | ||
+ | <pre>%www ALL = (apache) NOPASSWD: ALL, (root) /usr/bin/su apache</pre> | ||
+ | |||
+ | * Switch to the web user. You should NOT be prompted for a password. | ||
+ | |||
+ | <pre>sudo -u apache bash</pre> | ||
+ | |||
+ | * Create an ssh key, WITH NO PASSPHRASE. Store in <tt style="color:DarkGreen">~/.ssh/id_rsa</tt> and <tt style="color:DarkGreen">~/.ssh/id_rsa.pub</tt>: | ||
+ | <pre>ssh-keygen -b 2048 -t rsa</pre> | ||
+ | |||
+ | * Copy contents of <tt style="color:DarkGreen">~/.ssh/id_rsa.pub</tt> into ~/.ssh/authorized_keys file for user on dev.eclipse.org who | ||
+ | will be running builds. This is so that the web user can commit changes to cvs | ||
+ | (tagging, updating map files) for I, M, S & R builds. | ||
+ | |||
+ | * Test by ssh'ing to dev.eclipse.org, where <tt style="color:DarkGreen">_username_</tt> should be replaced with your actual username: | ||
+ | |||
+ | <pre>ssh _username_@dev.eclipse.org</pre> | ||
+ | |||
+ | * Run <tt style="color:DarkGreen">newgrp www</tt> so that when new files are created, they will use the group id <tt style="color:DarkGreen">www</tt> instead of <tt style="color:DarkGreen">apache</tt>: | ||
+ | |||
+ | <pre>newgrp www</pre> | ||
+ | |||
+ | * Set <tt style="color:DarkGreen">umask 022</tt> so that files will be created with group write perms 664 (see [[#/etc/bashrc]]): | ||
+ | |||
+ | <pre>umask 022</pre> | ||
+ | |||
+ | * Set the remote shell connection method for CVS to be <tt style="color:DarkGreen">ssh</tt> instead of the default <tt style="color:DarkGreen">rsh</tt> (see [[#/etc/bashrc]]): | ||
+ | |||
+ | <pre>export CVS_RSH=/usr/bin/ssh</pre> | ||
+ | |||
+ | * Set an <tt style="color:DarkGreen">ANT_HOME</tt> and <tt style="color:DarkGreen">JAVA_HOME</tt>, and add ant to the <tt style="color:DarkGreen">PATH</tt> (see [[#/etc/bashrc]]): | ||
+ | |||
+ | <pre>export ANT_HOME=/opt/apache-ant-1.6 | ||
+ | export JAVA_HOME=/opt/sun-java2-5.0 | ||
+ | export PATH=${PATH}:${ANT_HOME}/bin | ||
+ | </pre> | ||
+ | |||
+ | * Switch to the root user. | ||
+ | |||
+ | * Append the following into <tt style="color:DarkGreen">/etc/bashrc</tt>, where <tt style="color:DarkGreen">_username_</tt> should be replaced with your actual dev.eclipse.org username: | ||
+ | |||
+ | ==== /etc/bashrc ==== | ||
+ | <pre>umask 022 | ||
+ | export ANT_HOME=/opt/apache-ant-1.6 | ||
+ | export JAVA_HOME=/opt/sun-java2-5.0 | ||
+ | export PATH=${PATH}:${ANT_HOME}/bin | ||
+ | export CVSROOT=:ext:_username_@dev.eclipse.org:/cvsroot/technology | ||
+ | export CVS_RSH=/usr/bin/ssh</pre> | ||
+ | |||
+ | * Add the following to <tt style="color:DarkGreen">.bashrc</tt> and <tt style="color:DarkGreen">.bash_profile</tt> files: | ||
+ | |||
+ | ==== ~/.bashrc ==== | ||
+ | <pre>if [ -f /etc/bashrc ]; then | ||
+ | . /etc/bashrc | ||
+ | fi | ||
+ | |||
+ | if [ "$PS1" ]; then | ||
+ | # enable color support of ls and also add handy aliases | ||
+ | eval `dircolors -b` | ||
+ | alias ls='ls --color=auto' | ||
+ | alias ll='ls -l --color=auto' | ||
+ | |||
+ | # set a fancy prompt | ||
+ | # 1;30 - grey, 1;31 - red, 1;32 - green, 1;33 - yellow, 1;34 - blue, etc. | ||
+ | PS1="\[\033[<b>1;30</b>m\]\u@\h:\w\\[\033[0;39m\] \$ " | ||
+ | export PS1=$PS1"\[\e]30;\u@\H:\w\a\]" | ||
+ | PATH=$PATH:/opt/ibm-java2-1.4/bin:/opt/apache-ant-1.6/bin | ||
+ | export PATH | ||
+ | fi | ||
+ | |||
+ | source ~/.alias | ||
+ | cat ~/.alias</pre> | ||
+ | |||
+ | ==== ~/.bash_profile ==== | ||
+ | <pre>if [ -f ~/.bashrc ]; then | ||
+ | . ~/.bashrc | ||
+ | fi | ||
+ | |||
+ | PATH=$PATH:$HOME/bin | ||
+ | export PATH | ||
+ | unset USERNAME</pre> | ||
+ | |||
+ | == Secure build script == | ||
+ | |||
+ | * Since the webserver is public but builds should only be run by authorized users, we must secure access to the <tt style="color:DarkGreen">build.php</tt> script. | ||
+ | |||
+ | * Edit Apache config file to allow <tt style="color:DarkGreen">.htaccess</tt> rule changes to take affect. Change <tt style="color:DarkGreen">None</tt> to <tt style="color:DarkGreen">All</tt>: | ||
+ | |||
+ | ==== /etc/httpd/conf/httpd.conf ==== | ||
+ | <pre># AllowOverride controls what directives may be placed in .htaccess files. | ||
+ | # It can be "All", "None", or any combination of the keywords: | ||
+ | # Options FileInfo AuthConfig Limit | ||
+ | # | ||
+ | AllowOverride All</pre> | ||
+ | |||
+ | * Create password file, where <tt style="color:DarkGreen">_username_</tt> should be replaced with an actual username and <tt style="color:DarkGreen">/path/to/password/file</tt> should be replaced with an actual path: | ||
+ | |||
+ | <pre>httpasswd -c /path/to/password/file _username_</pre> | ||
+ | |||
+ | * You can add additional users to the password file like this: | ||
+ | |||
+ | <pre>httpasswd /path/to/password/file _username2_</pre> | ||
+ | |||
+ | * Create <tt style="color:DarkGreen">.htaccess</tt> file, replacing <tt style="color:DarkGreen">/path/to/password/file</tt> with the same path used in the previous step: | ||
+ | |||
+ | ==== /var/www/html/emft/.htaccess ==== | ||
+ | <pre>AuthType Basic | ||
+ | AuthName "EMFT Build Server @ Eclipse.org" | ||
+ | AuthUserFile /path/to/password/file | ||
+ | Require valid-user</pre> | ||
+ | |||
+ | * Restart apache, eg.: | ||
+ | |||
+ | <pre>/usr/sbin/httpd -k restart | ||
+ | - or - | ||
+ | apache2ctl restart</pre> | ||
+ | |||
+ | * For more on htaccess, see [http://httpd.apache.org/docs/2.0/howto/auth.html http://httpd.apache.org/docs/2.0/howto/auth.html]. | ||
+ | |||
+ | == Run a build == | ||
+ | |||
+ | Using the username and password set up in the previous step, go here: [http://emft.eclipse.org/emft/build.php http://emft.eclipse.org/emft/build.php]. | ||
+ | |||
+ | Check the build log while it's running or after it completes. If you see any messages such as <tt style="color:DarkGreen">permission denied</tt> or the following, | ||
+ | something is amiss. | ||
+ | |||
+ | === Debugging tips === | ||
+ | |||
+ | <pre>execvp: Permission denied</pre> | ||
+ | * Shell scripts must contain unix line endings. Run <tt style="color:DarkGreen">dos2unix</tt> to make sure, if copying them from a non-unix filesystem. | ||
+ | * Shell scripts <tt style="color:DarkGreen">/home/www-data/build/emft/scripts/*.sh</tt> must be executable. See [[#Fix permissions & ownership]]. | ||
+ | |||
+ | -- | ||
+ | |||
+ | * Directories, eg. <tt style="color:DarkGreen">/home/www-data/build/emft/jet/downloads/drops/1.0.0</tt> must be writable by the web user | ||
+ | (or group <tt style="color:DarkGreen">www</tt>). See [[#Fix permissions & ownership]]. | ||
+ | |||
+ | -- | ||
+ | |||
+ | <pre>eclipse.org: Connection refused | ||
+ | cvs [checkout aborted]: end of file from server (consult above messages if any) | ||
+ | </pre> | ||
+ | * CVS connection refusals are the result of not being able to automatically ssh as the web user to _username_@dev.eclipse.org. This | ||
+ | could be a problem with ssh keys (see above - [[#Fix web user (apache)]]) or the method CVS uses to | ||
+ | connect (rsh instead of ssh - see [[#/etc/bashrc]]). | ||
+ | |||
+ | -- | ||
+ | |||
+ | * For any compilation-related problems, missing file problems, or other issues not touched upon in this document, see [[EMFT_Procedures]]. | ||
+ | |||
+ | == Display build logs, details & test results == | ||
+ | |||
+ | Builds - including unpublished Nightly builds - are listed on the downloads page here: [http://emft.eclipse.org/technology/emft/downloads/ http://emft.eclipse.org/technology/emft/downloads/]. | ||
+ | |||
+ | Published builds are located here: [http://downloads.eclipse.org/technology/emft/downloads/ http://downloads.eclipse.org/technology/emft/downloads/]. | ||
+ | |||
+ | == Add additional users == | ||
+ | |||
+ | * As root, run the following script: | ||
+ | |||
+ | <pre>#!/bin/bash | ||
+ | |||
+ | if [ $# -lt 1 ]; then | ||
+ | echo "Usage: ./adduser <username>" | ||
+ | else | ||
+ | last=`tail -2 /etc/group | head -1`; last=${last##*:x:}; last=${last%%:*}; (( last++ )); #echo $last | ||
+ | newgrp=$1":x:"$last":"$1; | ||
+ | |||
+ | echo -n "Adding: " | ||
+ | echo $1"::"$last":"$last"::/home/"$1":/bin/bash" | ||
+ | echo $1"::"$last":"$last"::/home/"$1":/bin/bash" >> /etc/passwd | ||
+ | |||
+ | wwwgrp=`tail -1 /etc/group`","$1; #echo $wwwgrp; | ||
+ | wc=`wc -l /etc/group`; wc=${wc%% *}; (( wc-- )); wc="-"$wc; #echo $wc; | ||
+ | head $wc /etc/group > /etc/group.new | ||
+ | |||
+ | echo -n "Adding: " | ||
+ | echo $newgrp | ||
+ | echo $newgrp >> /etc/group.new | ||
+ | |||
+ | echo -n "Adding: " | ||
+ | echo $wwwgrp | ||
+ | echo $wwwgrp >> /etc/group.new | ||
+ | |||
+ | mv /etc/group.new /etc/group | ||
+ | |||
+ | echo $1 | passwd --stdin $1 | ||
+ | echo "Creating home dir: /home/$1" | ||
+ | cp -r /home/user /home/$1 | ||
+ | echo "Setting ownership: $1:$1" | ||
+ | chown -R $1:$1 /home/$1 | ||
+ | fi</pre> | ||
+ | |||
+ | --[[User:Nickb|Nick Boldt]] 21:48, 14 March 2006 (EST) |
Revision as of 22:48, 14 March 2006
Contents
EMFT Build Server Setup
You will need to be root for most of these tasks.
Set up web content
- Copy content from an existing server setup. Most of the content exists in CVS as well (/cvsroot/technology/org.eclipse.emft/releng/common):
FROM (emf.torolab.ibm.com) | TO (emft.eclipse.org) | |
/home/www-data/build/emft | --> | /home/www-data/build/emft |
/var/www/emft | --> | /var/www/html/emft |
/var/www/technology/emft | --> | /var/www/html/technology/emft |
Fix permissions & ownership
cd /var/www/html; find . -type f -exec chmod 664 {} \; cd /var/www/html; find . -type d -exec chmod 775 {} \; cd /var/www/html; find . -exec chown apache:www {} \; cd /home/www-data/build; find . -type f -exec chmod 664 {} \; cd /home/www-data/build; find . -type d -exec chmod 775 {} \; cd /home/www-data/build; find . -exec chown apache:www {} \; cd /home/www-data/build/scripts; find . -type f -name "*.sh" -exec chmod 755 {} \;
Apache 2 w/ PHP 5 was already installed. Only a few additional programs needed to be installed as well.
Install via yum using yum-xen.conf file
yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Java Development" yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Development Tools"
Install & symlink
- IBM JDK 1.4.2sr4-1 (symlink to /opt/ibm-java2-1.4)
- Sun JDK 1.5.0_06 (symlink to /opt/sun-java2-5.0)
- Apache Ant 1.6.5 (symlink to /opt/apache-ant-1.6)
Fix web user (apache)
- Edit /etc/group. Add www group:
/etc/group
www:x:501:user,apache
- Edit /etc/passwd. Change user's home directory and shell:
/etc/passwd
apache:!!:48:48:Apache:/home/www-data:/bin/bash
- Edit /etc/sudoers to let you run commands and switch to the web user w/o needing a password. Add the following lines:
/etc/sudoers
%www ALL = (apache) NOPASSWD: ALL, (root) /usr/bin/su apache
- Switch to the web user. You should NOT be prompted for a password.
sudo -u apache bash
- Create an ssh key, WITH NO PASSPHRASE. Store in ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub:
ssh-keygen -b 2048 -t rsa
- Copy contents of ~/.ssh/id_rsa.pub into ~/.ssh/authorized_keys file for user on dev.eclipse.org who
will be running builds. This is so that the web user can commit changes to cvs (tagging, updating map files) for I, M, S & R builds.
- Test by ssh'ing to dev.eclipse.org, where _username_ should be replaced with your actual username:
ssh _username_@dev.eclipse.org
- Run newgrp www so that when new files are created, they will use the group id www instead of apache:
newgrp www
- Set umask 022 so that files will be created with group write perms 664 (see #/etc/bashrc):
umask 022
- Set the remote shell connection method for CVS to be ssh instead of the default rsh (see #/etc/bashrc):
export CVS_RSH=/usr/bin/ssh
- Set an ANT_HOME and JAVA_HOME, and add ant to the PATH (see #/etc/bashrc):
export ANT_HOME=/opt/apache-ant-1.6 export JAVA_HOME=/opt/sun-java2-5.0 export PATH=${PATH}:${ANT_HOME}/bin
- Switch to the root user.
- Append the following into /etc/bashrc, where _username_ should be replaced with your actual dev.eclipse.org username:
/etc/bashrc
umask 022 export ANT_HOME=/opt/apache-ant-1.6 export JAVA_HOME=/opt/sun-java2-5.0 export PATH=${PATH}:${ANT_HOME}/bin export CVSROOT=:ext:_username_@dev.eclipse.org:/cvsroot/technology export CVS_RSH=/usr/bin/ssh
- Add the following to .bashrc and .bash_profile files:
~/.bashrc
if [ -f /etc/bashrc ]; then . /etc/bashrc fi if [ "$PS1" ]; then # enable color support of ls and also add handy aliases eval `dircolors -b` alias ls='ls --color=auto' alias ll='ls -l --color=auto' # set a fancy prompt # 1;30 - grey, 1;31 - red, 1;32 - green, 1;33 - yellow, 1;34 - blue, etc. PS1="\[\033[<b>1;30</b>m\]\u@\h:\w\\[\033[0;39m\] \$ " export PS1=$PS1"\[\e]30;\u@\H:\w\a\]" PATH=$PATH:/opt/ibm-java2-1.4/bin:/opt/apache-ant-1.6/bin export PATH fi source ~/.alias cat ~/.alias
~/.bash_profile
if [ -f ~/.bashrc ]; then . ~/.bashrc fi PATH=$PATH:$HOME/bin export PATH unset USERNAME
Secure build script
- Since the webserver is public but builds should only be run by authorized users, we must secure access to the build.php script.
- Edit Apache config file to allow .htaccess rule changes to take affect. Change None to All:
/etc/httpd/conf/httpd.conf
# AllowOverride controls what directives may be placed in .htaccess files. # It can be "All", "None", or any combination of the keywords: # Options FileInfo AuthConfig Limit # AllowOverride All
- Create password file, where _username_ should be replaced with an actual username and /path/to/password/file should be replaced with an actual path:
httpasswd -c /path/to/password/file _username_
- You can add additional users to the password file like this:
httpasswd /path/to/password/file _username2_
- Create .htaccess file, replacing /path/to/password/file with the same path used in the previous step:
/var/www/html/emft/.htaccess
AuthType Basic AuthName "EMFT Build Server @ Eclipse.org" AuthUserFile /path/to/password/file Require valid-user
- Restart apache, eg.:
/usr/sbin/httpd -k restart - or - apache2ctl restart
- For more on htaccess, see http://httpd.apache.org/docs/2.0/howto/auth.html.
Run a build
Using the username and password set up in the previous step, go here: http://emft.eclipse.org/emft/build.php.
Check the build log while it's running or after it completes. If you see any messages such as permission denied or the following, something is amiss.
Debugging tips
execvp: Permission denied
- Shell scripts must contain unix line endings. Run dos2unix to make sure, if copying them from a non-unix filesystem.
- Shell scripts /home/www-data/build/emft/scripts/*.sh must be executable. See #Fix permissions & ownership.
--
- Directories, eg. /home/www-data/build/emft/jet/downloads/drops/1.0.0 must be writable by the web user
(or group www). See #Fix permissions & ownership.
--
eclipse.org: Connection refused cvs [checkout aborted]: end of file from server (consult above messages if any)
- CVS connection refusals are the result of not being able to automatically ssh as the web user to _username_@dev.eclipse.org. This
could be a problem with ssh keys (see above - #Fix web user (apache)) or the method CVS uses to connect (rsh instead of ssh - see #/etc/bashrc).
--
- For any compilation-related problems, missing file problems, or other issues not touched upon in this document, see EMFT_Procedures.
Display build logs, details & test results
Builds - including unpublished Nightly builds - are listed on the downloads page here: http://emft.eclipse.org/technology/emft/downloads/.
Published builds are located here: http://downloads.eclipse.org/technology/emft/downloads/.
Add additional users
- As root, run the following script:
#!/bin/bash if [ $# -lt 1 ]; then echo "Usage: ./adduser <username>" else last=`tail -2 /etc/group | head -1`; last=${last##*:x:}; last=${last%%:*}; (( last++ )); #echo $last newgrp=$1":x:"$last":"$1; echo -n "Adding: " echo $1"::"$last":"$last"::/home/"$1":/bin/bash" echo $1"::"$last":"$last"::/home/"$1":/bin/bash" >> /etc/passwd wwwgrp=`tail -1 /etc/group`","$1; #echo $wwwgrp; wc=`wc -l /etc/group`; wc=${wc%% *}; (( wc-- )); wc="-"$wc; #echo $wc; head $wc /etc/group > /etc/group.new echo -n "Adding: " echo $newgrp echo $newgrp >> /etc/group.new echo -n "Adding: " echo $wwwgrp echo $wwwgrp >> /etc/group.new mv /etc/group.new /etc/group echo $1 | passwd --stdin $1 echo "Creating home dir: /home/$1" cp -r /home/user /home/$1 echo "Setting ownership: $1:$1" chown -R $1:$1 /home/$1 fi
--Nick Boldt 21:48, 14 March 2006 (EST)