diff --git a/mozilla/webtools/tinderbox2/ChangeLog b/mozilla/webtools/tinderbox2/ChangeLog index 6bc4ec4754f..43f84af33e1 100644 --- a/mozilla/webtools/tinderbox2/ChangeLog +++ b/mozilla/webtools/tinderbox2/ChangeLog @@ -1,3 +1,18 @@ +2005-10-08 Mike Taylor + + * ChangeLog + removed release note text and put them into ReleaseNotes file + + * Todo + added my Todo list + + * Contact + removed tabs + + * ReadMe, Policies, Overview, Goals, Install + edited to read in the third person. Fixed typos, spelling + and reformatted for readability + 2005-10-06 Mike Taylor * Contact @@ -11,166 +26,3 @@ fixed spelling error in generated html content s/BPRDER/BORDER/ bug 230348 patch supplied by Todd Endicott -New Time columns for use in environments with multiple time zones. - -Spelling fixes - -Add support for popup windows which work "on click" - -Add support for users who want to see tinderbox's notion of what was -checked in not just Bonsai's query. - -Add support for historical Tinderbox images - (flames, stars) - -Mozilla layout preferences are now -standard -(non uniform row spacing, - truly empty cells, - legend on bottom of page). - -Allow users to backdate notices - -automate regeneration of HTML when user uses CGI -interface - -Put the C in all build cells. - -VC code did the wrong thing when generating old pages and there was -no TreeState data. - -Fix Spelling mistakes throughout code. - -Add support for build machines to tell the server to add text/links -into build cells (TinderboxPrint). - -When builds are started too frequently (more then once every five minutes) -newest build is now kept and oldest build is thrown away. - -New Configure script contributed by Stephen Lamm (slamm@yahoo-inc.com) -This includes a fix to allow Perl to be in different locations on the -machine where make is run and on the machine where the webserver runs. - -Now it is possible to include the time column on the left and right -side of the status table. - -There is now a working Perforce module for those who use this product -instead of CVS. - -There is a module to interface with the req ticketing system. - -New Summary format and number of errors displayed on build page by - Adam Di Carlo , - -Spelling and consistency fixes from: - Adam Di Carlo , - Orion Poplawski , - John L. Villanlovos , - -Change HTML generation so that the text browser 'links' -(http://artax.karlin.mff.cuni.cz/~mikulas/links/) can -work with tinderbox. - -Allow the build script to use a single process and alternate builds -of different types. - -Add a sample /etc/rc.d/init.d file for the RedHat Build machines. - -Mail processing programs were not writing error files correctly. -This has been broken since I separated out the mail processing module. - -Add script to cull old builds from a directory of completed builds. - -Let the VC and BT headers be configurable. Too many companies find -this an important feature. - -Allow Tinderbox to run via suid wrappers. This is needed for some -times of mail delivery systems - -No longer exclude 'privileged id's from running Tinderbox. -Now we require tinderbox to run as the id which the user specifies -is the tinderbox id. - -Better errors for failures in Storable.pm - -BasicTxtHeader.pm had a bug which prevented it from working when using Storable - -Rework the BT modules to allow use when URL is not easily mapped from the bugid - -Begin work to allow tinderbox admin pages to open and close bonsai trees - -fix code which allowed tinderbox to determine the state of bonsai trees - -Major bug in the untainting code, fixed. This was a security -problem. I suspect people may have trouble with the fix but this was -how the code was intended to work. I do not understand how this got -through my unit tests. - -Give a useful example of how use the new build script. - -Improve the specfiles, split out build machine packages. - - -Release 0.9 - -Add full Bonsai integration functionality. - -Allow users to define a display order for the buildnames on a per tree basis. -This adds a new function into TreeData.pm check your local files. - -Fix computation of 'deadtime' for eazel.com's build setup, - this will improve their estimate of when the next build will be - complete. - -Develop portable client build scripts. - -Add RPM specfiles. - -Relax check of root id for all redhat users. - uid nobody is 99 and by default redhat apache runs all cgi as nobody. - -VC_CVS.pm was not removing excess treestates from the Database correctly, - now fixed. - -Release 0.8 - -This release moved some files into new directories update with - cvs update -d -P - -Remove the strict time checking on mail, it caused too many problems. - -Fix the Unmonitored Builds in admintree.cgi and in status.html page. - -Storable.pm has been tested as a Persistence implementation. - -Improve the prediction of when builds will end by introducing the -notion of 'deadtime'. - -Separate out the BuildStatus table for easier configuration of the -status without changing the build display library. - -Better comments in the ReadMe files - -Better security: - - All scripts run in taint perl mode - - Separation of HTML and DATA directories in - addition to the CGI directory - - Additional checks of ownership for the data directories - - Check the network address of all administrative commands - -Improved Makefile/Configure support - - New permissions help prevent accidental edit of files in the - build or install directory - - Interpositioning of libraries, for local configuration, - fully supported - - Test arguments added to Makefile - - Test compiles of the perl are no longer required - diff --git a/mozilla/webtools/tinderbox2/Contact b/mozilla/webtools/tinderbox2/Contact index 94ee9b09f05..9f5f0f68b3e 100644 --- a/mozilla/webtools/tinderbox2/Contact +++ b/mozilla/webtools/tinderbox2/Contact @@ -1,9 +1,9 @@ Bug Tickets can be filed at: - http://bugzilla.mozilla.org/show_bug.cgi?id=113212 + http://bugzilla.mozilla.org/show_bug.cgi?id=113212 - Product: Webtools - Component: Tinderbox2 + Product: Webtools + Component: Tinderbox2 Tinderbox2 is currently being co-maintained by: diff --git a/mozilla/webtools/tinderbox2/Goals b/mozilla/webtools/tinderbox2/Goals index e1c4a971aff..eb9ef461bb3 100644 --- a/mozilla/webtools/tinderbox2/Goals +++ b/mozilla/webtools/tinderbox2/Goals @@ -4,107 +4,109 @@ USERS It is helpful to consider the types of users for this system and their special needs. -1) Project Managers: not very technical will be administering the -project and need a GUI to help change various types of project -information (treestate, message of the day, etc). They will also need -summary pages which will show them the current status of all the -projects which they are working on. They may occasionally "drill -down" into the detailed status page but this will not be their primary -view of the tinderbox system. + * Project Managers: not very technical will be administering the + project and need a GUI to help change various types of project + information (tree state, message of the day, etc). They will also + need summary pages which will show them the current status of all + the projects which they are working on. They may occasionally + "drill down" into the detailed status page but this will not be + their primary view of the Tinderbox2 system. -2) Build Administrator: A system administrator who will be in charge -of setting up the build machines, configuring tinderbox and other -build systems (bugzilla, cvs, bonsai, etc). A GUI would not be -helpful as local customizations may require small changes to the code. -Configurations need to be kept (mostly) in files which are separate -from the Tinderbox source code so that they can be version controled -and will not get stepped on when tinderbox is upgraded. + * Build Administrator: A system administrator who will be in charge + of setting up the build machines, configuring Tinderbox2 and other + build systems (bugzilla, cvs, bonsai, etc). A GUI would not be + helpful as local customizations may require small changes to the code. -3) Developers: need to view the "state of development" and add notices -to the notice board. + Configurations need to be kept (mostly) in files which are separate + from the Tinderbox2 source code so that they can be version controlled + and will not get stepped on when Tinderbox2 is upgraded. + + * Developers: need to view the "state of development" and add notices + to the notice board. Improvements needed from Tinderbox1 ----------------------------------- -highly configurable design with multiple Version Control systems -possible (bonsai, raw cvs, perforce, continuous, clearcase) and -multiple modes of running possible (with no version control system -with no builds display). + * Highly configurable design with multiple Version Control systems + possible (bonsai, raw cvs, perforce, continuous, clearcase) and + multiple modes of running possible (with no version control system + with no builds display). -clear programatic interfaces and better separation of functionality -into separate modules. + * Clear programatic interfaces and better separation of functionality + into separate modules. -Modules should not have circular dependences. Care should be taken on -adding 'use' statement. I worry very much about which modules depend -on which other modules. It is important to consider which modules -need to work in isolation and which modules need to share data. + * Modules should not have circular dependences. Care should be taken + on adding 'use' statement as you should worry very much about which + modules depend on which other modules. It is important to consider + which modules need to work in isolation and which modules need to + share data. -It should be possible to add hooks so that users get beeped when the -next good build goes through or that trouble tickets are automatically -opened when the builds fail. + * It should be possible to add hooks so that users get beeped when the + next good build goes through or that trouble tickets are automatically + opened when the builds fail. -Greater flexibility in setting status of builds. We may need more -gradations of failure then just 'busted' or 'test-failed' to -distinguish the types of tests which have failed. + * Greater flexibility in setting status of builds. We may need more + graduations of failure then just 'busted' or 'test-failed' to + distinguish the types of tests which have failed. -generated html must be readable and help isolate programming errors. + * Generated HTML must be readable and help isolate programming errors. -all programmable configuration parameters should be stored easy change -and configure for novice users. + * All programmable configuration parameters should be stored easy change + and configure for novice users. -make better use of the perl data structures to mirror the way we wish -to use the data. This will allow easier maintainability and allow for -more expansion of features. + * Make better use of the Perl data structures to mirror the way we wish + to use the data. This will allow easier maintainability and allow for + more expansion of features. -display should work on many different browsers. + * Display should work on many different browsers. -popup windows should not be netscape specific. + * Popup windows should not be Netscape specific. -Permanent data should be stored via datadumper so that the data and -data structures are easy to read and debug. Currently this is a -performance bottle neck with a large percentage of our cpu time during -testing being spent in Data::Dumper::Dump. The perl module Storable -is much faster. I wish to not add additional module requirements at -this time, this will be configurable. + * Permanent data should be stored via datadumper so that the data and + data structures are easy to read and debug. Currently this is a + performance bottle neck with a large percentage of cpu time during + testing being spent in Data::Dumper::Dump. The Perl module Storable + is much faster and there is no need to add additional module + requirements at this time, this will be configurable. -# dprofpp says that: -# %64.8 of elapsed real time which is 66.25 seconds -# (out of 102.15 Seconds) -# was spent in 3 calls to TinderDB::VC::apply_db_updates() + # dprofpp says that: + # %64.8 of elapsed real time which is 66.25 seconds + # (out of 102.15 Seconds) + # was spent in 3 calls to TinderDB::VC::apply_db_updates() -# %58.0 of user time which is 11.05 seconds -# (out of 19.03 User/102.15 Elapsed Seconds) -# was spend in 32878 calls to Data::Dumper::_dump() + # %58.0 of user time which is 11.05 seconds + # (out of 19.03 User/102.15 Elapsed Seconds) + # was spend in 32878 calls to Data::Dumper::_dump() -# System Time was negligible at 2.49 Seconds + # System Time was negligible at 2.49 Seconds -All errors should be trapped and sent to log files. Strange program -states should be explicitly checked for. + * All errors should be trapped and sent to log files. Strange program + states should be explicitly checked for. -Databases should update atomically, no information should be lost due -to race conditions. + * Databases should update atomically, no information should be lost due + to race conditions. -All column modules (processmail, build, VC, Notices) should be able to -be run individually. Modules should accept well defined text files as -input and produce text files as output. This will greatly enhance the -ability to test each module in isolation and to quickly port modules -to new architectures. + * All column modules (processmail, build, VC, Notices) should be able to + be run individually. Modules should accept well defined text files as + input and produce text files as output. This will greatly enhance the + ability to test each module in isolation and to quickly port modules + to new architectures. -The source code should be able to run using the standard Perl -libraries, as it can be difficult for some users to add libraries -which are not included in the perl distribution onto production -machines. + * The source code should be able to run using the standard Perl + libraries, as it can be difficult for some users to add libraries + which are not included in the perl distribution onto production + machines. -Put CVS keywords into all the source files so that when the software -is deployed, there is no doubt what version was checked out and where -the files are stored in the local version control system. + * Put CVS keywords into all the source files so that when the software + is deployed, there is no doubt what version was checked out and where + the files are stored in the local version control system. -Pay closer attention to security. Use taint perl to ensure that -harmful input does not get used in unexpected ways. Keep the flow of -control clear and simple. + * Pay closer attention to security. Use taint Perl to ensure that + harmful input does not get used in unexpected ways. Keep the flow of + control clear and simple. + * Allow for use of any text browser which can display tables. + Currently this is true for the browser + links (http://artax.karlin.mff.cuni.cz/~mikulas/links/) but not the + browser lynx. -Allow for use of any text browser which can display tables -currently this is true for the browser links: - http://artax.karlin.mff.cuni.cz/~mikulas/links/. -but not the browser lynx. diff --git a/mozilla/webtools/tinderbox2/Install b/mozilla/webtools/tinderbox2/Install index 527cb7def70..04deb3fbfbe 100644 --- a/mozilla/webtools/tinderbox2/Install +++ b/mozilla/webtools/tinderbox2/Install @@ -2,50 +2,50 @@ Tinderbox Documentation: ------------------------ -*) Every directory has a ReadMe file which tells about all the files -in that directory. + * Every directory has a ReadMe file which tells about all the files + in that directory. -*) Every executable can be run with --help so that you can see what -the program does and what its arguments are. If you have trouble -running the file just look at the file in a text editor and find -the string "usage". + * Every executable can be run with --help so that you can see what + the program does and what its arguments are. If you have trouble + running the file just look at the file in a text editor and find + the string "usage". -*) The top of every file (both code and configuration) has a short -explanation of what this file does. + * The top of every file (both code and configuration) has a short + explanation of what this file does. -*) The code has lots of documentation so you can read the code if you -need more details about how something works. + * The code has lots of documentation so you can read the code if you + need more details about how something works. To install: ----------- +NOTE: You will need to ensure that Perl is installed on your web server. + It would be a good idea to have these modules installed: -*) You will need to ensure that Perl is installed on your webserver. -It would be a good idea to have these modules installed: - Storable, Date::Format, + Storable, Date::Format, -You can easily check if these are installed by executing these two -commands. + You can easily check if these are installed by executing these + following commands. If the module is installed, you'll get no + output: -If the module is installed, you'll get no output: - perl -e 'use Storable' - perl -e 'use Date::Format' - + perl -e 'use Storable' + perl -e 'use Date::Format' -*) Read the Policies and Overview documents found in this directory to +Read the Policies and Overview documents found in this directory to help you get a feel for the scope of this installation. -*) The process id which receives and process the mail must be the -same id which runs the tinderbox cron job to prepare the web pages. I -prefer to manage my webserver so that all CGI scripts do not run as -the same user. Using one user id can cause security problems which -are hard to detect. If you must run all web applications as a single -user it would be safer to ensure that this user is a specific web user -(apache, cgiuser, webuser) so that this is not mixed up with other -unix system users (daemon, nobody, bin) since this could cause -security interactions with other programs which use these ids. +The process id which receives and process the mail must be the +same id which runs the Tinderbox cron job to prepare the web pages. + +A good practice is to configure the web server so that all CGI scripts +do not run as the same user. Using one user id can cause security +problems which are hard to detect. If you must run all web applications +as a single user it would be safer to ensure that this user is a specific +web user (apache, cgiuser, webuser) so that this is not mixed up with other +unix system users (daemon, nobody, bin) since this could cause security +interactions with other programs which use these ids. It may take some thought as to how the user id will be configured to run when receiving mail and when receiving web requests and not be a @@ -55,224 +55,230 @@ These products will help partition your web application to run as different users. (See http://www.w3.org/Security/Faq/wwwsf4.html for more info) -CGIWrap - http://cgiwrap.unixtools.org/ - http://sourceforge.net/projects/cgiwrap/ - -sbox: Put CGI Scripts in a Box - http://stein.cshl.org/software/sbox/ - + CGIWrap + http://cgiwrap.unixtools.org/ + http://sourceforge.net/projects/cgiwrap/ + sbox: Put CGI Scripts in a Box + http://stein.cshl.org/software/sbox/ The Apache Web server comes with its own wrapper script called suEXEC. -*) The src/default_conf directory contains perl libraries which are +The src/default_conf directory contains Perl libraries which are specific to an individual users site configuration. No two users of -Tinderbox will have the identical sets of files. +Tinderbox2 will have the identical sets of files. The files found in default_conf are sample files which are used at Mozilla.org. Other users are expected to customize these libraries and install the modified libraries in local_conf. Files found in local_conf will be used before any file found in default_conf. -The distribution of Tinderbox will never place any files in the +The distribution of Tinderbox2 will never place any files in the local_conf directory as this is assumed to be under local control. A quick overview of the files: - TinderConfig.pm: general configuration settings - (HTML directories, Log files, implementations of libraries - to use, etc). - TreeData.pm: version control (CVS, Bonsai) configuration. - Error_Parse.pm: the regular expressions for identifying - errors in build logs. - BTData.pm: bug tracking configuration - FileStructure.pm: filesystem (storage) and global/per-project - settings. Most users will not have to change this. + TinderConfig.pm + general configuration settings: HTML directories, Log files, + implementations of libraries to use, etc + TreeData.pm + version control (CVS, Bonsai) configuration + + Error_Parse.pm + the regular expressions for identifying errors in build logs. + + BTData.pm + bug tracking configuration + + FileStructure.pm + filesystem (storage) and global/per-project settings. + Most users will not have to change this. In particular you will need a TinderConfig.pm and a TreeData.pm which describes your local setup. You may need to make local versions of the other files depending on how you wish to have Tinderbox configured. -I install Tinderbox via RPM. I have three RPMS. One tracks changes in -the Tinderbox server source code just as I get it from CVS. One RPM -contains my local_conf files. The clientbin files get put in their -own RPM which is installed on the buildmachine along with the -local_conf files. I do not currently have all the other configuration -details worked out (crontab files, /etc/rc.d/init.d, .cvspass,) but -most of the state of my machines is under RPM control. +1. Run ./configure. When configure is done you will have a Makefile + and a config.out. Although Tinderbox2 consists entirely of Perl + scripts we must substitute some values into the source code to make it + executable. You may wish to change the default directories in + configure for some of the Makefile variables. Please read config.out + and make any changes which need to be made for your system. Configure + also accepts command line options to change some default variables. + Please look at the configure source code for variable details, but the + most common changes are: -*) Run ./configure. When configure is done you will have a Makefile -and a config.out. Although Tinderbox consists entirely of perl -scripts we must substitute some values into the source code to make it -executable. You may wish to change the default directories in -configure for some of the makefile variables. Please read config.out -and make any changes which need to be made for your system. Configure -also accepts command line options to change some default variables. -Please look at the configure source code for variable details, but the -most common changes are: + --prefix + the directory for most of the tinderbox files. + Defaults to /home/tinderbox2 - --prefix: the directory for most of the tinderbox files, - defaults to /home/tinderbox2 + --cgibin-prefix + The directory where cgi scripts will be run. + This depends on how your web server is configured. + For security you may wish to ensure that this directory is + disjoint from the prefix directory. + Defaults to /var/www/cgi-bin/tinderbox. - --cgibin-prefix: The directory where cgi scripts will be - run. This depends on how your webserver is configured. - For security you may wish to ensure that this - directory is disjoint from the prefix directory. This - defaults to /var/www/cgi-bin/tinderbox. + --html-prefix + The directory where html files should be written. + This depends on how your web server is configured. + This is where the web server will pick up Tinderbox2 output. + Defaults to /home/httpd/html/tinderbox. - --html-prefix: The directory where html files should be - written. This depends on how your webserver is configured. - This is where the webserver will pick up tinderbox2 - output. This defaults to /home/httpd/html/tinderbox. +2. Run 'make' to create executable versions of the source code in the + ./build directory. + +3. Run 'make compile_bin_code' to ensure that the code will compile + when it is installed. This will not work if your target machine is + very different from the machine you are running make on. + +4. If you wish run the test programs as described in /build/test/ReadMe + to ensure that you have configured the program correctly. This will + require a TinderConfig.pm file in your local_conf directory. + + See the ReadMe file in the test directory for detailed information + on how the tests work. + +5. Run 'make install' to install Tinderbox2 on your system. You may + wish to run this with a different 'prefix' + + make install --prefix=/opt/tinderbox + + It is currently assumed that the bin directory of your installation + will be where the web server will run the Tinderbox2 cgi's from. + You can copy the *.cgi files to another directory if this is not + the case. + +5. There are some gifs located in the gif directory which have + historically been used by Tinderbox2. The installation via + 'make install' does not install these images. Put them + somewhere in your web server's html directory if you wish to + use them. Samples of their use are in the configuration files. + +6. Set up a cron job to run the following every five minutes: + + $cgi-bin/bin/tinder.cgi --daemon-mode + + This generates the static tinderbox pages which users see. + + Set up a cron job to run the following at least once a day: + + $prefix/bin/rmlogs + + Six am is a good time to run this as the machine load is usually + light and will avoids any Daylight Savings problems. This keeps + the archive of compressed build logs from growing without limit. + +7. You may recieve data into the $prefix/bin/processmail* programs via + http post. Typically this means that these programs must be installed + in the $cgi-bin directory and have a name which ends in *.cgi. Instead + of mailling the logs to these programs, use the HTTPPost program to + post the data to the correct URL. + +8. If using the mail system, set up the $prefix/bin/processmail* + programs to receive the incoming tinderbox mail. The process id which + receives and process the mail must be the the same id which runs the + Tinderbox2 cron job to prepare the web pages. Usually this set up is + accomplished by having the MTA (Sendmail) pass mail for particular + accounts into a script. This can be configured via a global + configuration file (Sendmail alias file) or via a .forward file (each + account gets the same user id but a different home directory, each + home directory gets a .forward to cause incoming mail to be delivered + through the correct tinderbox mail processing program). + + The following configuration has been tested for the mail server + Postfix. The postfix aliases file /etc/postfix/aliases contains the + following lines. This file must be processed with a /usr/sbin/postalias + command which must be run by the same id which will run the Tinderbox2 + cgi scripts. + + tinderbox_builds: |/home/tinderbox2/bin/processmail_builds + tinderbox_bugzilla: |/home/tinderbox2/bin/processmail_bugs + + If it is not possible to have mail delivery on your web machine and to + have your web machine deliver specific mail through a program then you + could you use fetchmail to simulate this delivery format. + + Download fetchmail (from http://tuxedo.org/~esr/fetchmail) and install + it on your web server. Have the mail sent to any pop/imap mail server + which is accessible from the web server. Configure fetchmail to gather + the mail from the mail server and pass it through to the correct mail + processing program. You should keep the polling interval short + (30-200 seconds) as long polling intervals will cause delays in the + Tinderbox2 system and limit how quickly users see the new data. If + the polling interval is too short then you will overload your pop + server with needless queries. Fetchmail should run under the same + id as the Tinderbox2 server. The fetchmail configuration file will + look something like: + + poll mail.provider.net with proto pop3: + user "tinderbox_builds" there has password "u can't krak this" + is tinder here and wants mda "$prefix/bin/processmail_builds" + + A common mail address is 'tinderbox_builds' for build information + destined for the web server. Similarly the bug tracking system should + send mail to 'tinderbox_bugs'. + +9. For CVS systems - make sure that when Tinderbox2 runs it has all the + permissions that it needs to access the Version Control database. + This usually means having an environment variable set when Tinderbox2 + is run from cron. If you are using VC_CVS.pm then you will need to put + a ~/.cvspass for the Tinderbox2 server to use. + + To create this file you will need to Log into the CVS repository + yourself, once for each tree you have defined. The login command must + use the hostname/modules exactly as you wrote it in VCData. Then copy + your ~/.cvspass into the Tinderbox2 server user id's home directory. + This must be the REAL home of the Tinderbox2 daemon, as listed in + /etc/passwd/ and set in the $HOME environmental variable for tinder.cgi. + The file must not be world readable or writable or executable. It is + recommended that VC_CVS.pm is only used as an interim step in setting up + VC_Bonsai. VC_Bonsai is not hard to setup and provided many useful + features which are absent in VC_CVS.pm. + +10. Configure the build machines to mail their build logs (with Tinderbox2 + variables on the top) to the web server machine. New builds must not + start earlier then 6 minutes after the last build started. Each build + machine mails the build log of each build and puts some build data at + the top of the log. Build information includes whether the build was + a success, which error parsers to use on the log file and what build + this is. + + Run 'processmail_builds --help' to learn about how the mail + processing system works and what the mail messages are expected to + look like. There is an example mail in the test directory called + samplelog. The clientbin directory contains code which can be used on + the buildmachine. + + See the README to help setup your build machine. + + http://www.ntsendmail.com/ has a Perl program which can be used to + send mail on systems where there is no sendmail installed. + + There is nothing special about sending mail. Any data transport layer + would work including HTTP Post and SSH the only requirement is that + the data is sent to the STDIN of the mail processing programs upon + delivery and that the process which runs this is the same uid as the + Tinderbox2 server. + + Alternate delivery mechanisms have not been explored thoroughly but + they should not be too hard to setup. There is a set of instructions + for this in lwpcook.pod which suggest you will need to install + HTTP::Request::Common and LWP::UserAgent; + +11. Set up the bug tracking system to send mail to 'tinderbox_bugs' on + the web server machine whenever a Bug ticket changes state. It is not + interesting to see which tickets are being worked on, so restrict + mailing to changes in ticket state not updates (edit) of a ticket. + +12. Check that the time on your web server, your version control + machine, your bug tracking machine and your build machines are all in + sync. Check that if build mail bounces on any of the above + machines that it will be received by someone who can act on it. -*) run 'make' to create executable versions of the source code in the -./build directory. +Contact information for questions or updates or gripes is kept in the +Contact file. -*) run 'make compile_bin_code' to ensure that the code will compile -when it is installed. This will not work if your target machine is -very different from the machine you are running make on. - -*) If you wish run the test programs as described in -/build/test/ReadMe to ensure that you have configured the program -correctly. This will require a TinderConfig.pm file in your local_conf -directory. See the ReadMe file in the test directory for detailed -information on how the tests work. - -*) run make install, to install Tinderbox on your system. You may -wish to run this with a different 'prefix' - - make install --prefix=/opt/tinderbox - -It is currently assumed that the bin directory of your installation -will be where the webserver will run the tinderbox cgi's from. You -can copy the *.cgi files to another directory if this is not the case. - -*) There are some gifs located in the gif directory which have -historically been used by tinderbox. The installation via 'make -install' does not install these images. Put them somewhere in your -webserver's html directory if you wish to use them. Samples of their -use are in the configuration files. - -*) set up a cron job to run $cgi-bin/bin/tinder.cgi --daemon-mode - every five minutes. This generates the static tinderbox pages - which users see. - set up a cron job to run $prefix/bin/rmlogs at least once a day. - six am is a good time to run this as the machine load is usually light - and will avoids any day light savings problems. This keeps the - archive of compressed build logs from growing without limit. - -*) you may recieve data into the $prefix/bin/processmail* programs via -http post. Typically this means that these programs must be installed -in the $cgi-bin directory and have a name which ends in *.cgi. Instead -of mailling the logs to these programs, use the HTTPPost program to -post the data to the correct URL. - -*) If using the mail system, set up the $prefix/bin/processmail* -programs to receive the incoming tinderbox mail. The process id which -receives and process the mail must be the the same id which runs the -tinderbox cron job to prepare the web pages. Usually this set up is -accomplished by having the MTA (Sendmail) pass mail for particular -accounts into a script. This can be configured via a global -configuration file (Sendmail alias file) or via a .forward file (each -account gets the same user id but a different home directory, each -home directory gets a .forward to cause incoming mail to be delivered -through the correct tinderbox mail processing program). - -I have used the following configurations for the mail server Postfix. -The postfix aliases file /etc/postfix/aliases contains the following -lines. This file must be processed with a /usr/sbin/postalias command -which must be run by the same id which will run the tinderbox cgi scripts. - - tinderbox_builds: |/home/tinderbox2/bin/processmail_builds - tinderbox_bugzilla: |/home/tinderbox2/bin/processmail_bugs - - - -If it is not possible to have mail delivery on your web machine and to -have your web machine deliver specific mail through a program then I -suggest you use fetchmail to simulate this delivery format. Download -fetchmail (from http://tuxedo.org/~esr/fetchmail) and install it on -your webserver. Have the mail sent to any pop/imap mail server which -is accessible from the webserver. Configure fetchmail to gather the -mail from the mail server and pass it through to the correct mail -processing program. You should keep the polling interval short (30-200 -seconds) as long polling intervals will cause delays in the tinderbox -system and limit how quickly users see the new data. If the polling -interval is too short then you will overload your pop server with -needless queries. Fetchmail should run under the same id as the -tinderbox server. The fetchmail configuration file will look -something like: - - poll mail.provider.net with proto pop3: - user "tinderbox_builds" there has password "u can't krak this" - is tinder here and wants mda "$prefix/bin/processmail_builds" - - - - -We use the mail address 'tinderbox_builds' for build information -destined for the webserver. Similarly the bug tracking system should -send mail to 'tinderbox_bugs'. - - -*) Make sure that when tinderbox runs it has all the permissions that -it needs to access the Version Control database. This usually means -having an environmental variable set when tinderbox is run from -cron. If you are using VC_CVS.pm then you will need to put a -~/.cvspass for the tinderbox server to use. To create this file you -will need to Log into the CVS repository yourself, once for each tree -you have defined. The login command must use the hostname/modules -exactly as you wrote it in VCData. Then copy your ~/.cvspass into the -tinderbox server user id's home directory. This must be the REAL home -of the Tinderbox daemon, as listed in /etc/passwd/ and set in the -$HOME environmental variable for tinder.cgi. The file must not be -world readable or writable or executable. It is recommended that -VC_CVS.pm is only used as an interim step in setting up -VC_Bonsai. VC_Bonsai is not hard to setup and provided many useful -features which are absent in VC_CVS.pm. - - -*) Setup the build machines to mail their build logs (with tinderbox -variables on the top) to the web server machine. New builds must not -start earlier then 6 minutes after the last build started. Each build -machine mails the build log of each build and puts some build data at -the top of the log. Build information includes whether the build was -a success, which error parsers to use on the log file and what build -this is. Run 'processmail_builds --help' to learn about how the mail -processing system works and what the mail messages are expected to -look like. There is an example mail in the test directory called -samplelog. The clientbin directory contains code which can be used on -the buildmachine. See the README to help setup your buildmachine. - -http://www.ntsendmail.com/ has a perl program which can be used to -send mail on systems where there is no sendmail installed. - -There is nothing special about sending mail. Any data transport layer -would work including HTTP Post and SSH the only requirement is that -the data is sent to the STDIN of the mail processing programs upon -delivery and that the process which runs this is the same uid as the -tinderbox server. I have not thought too much about how to set up -these alternate delivery mechanisms so I have no practical advice but -It should not be too hard to setup. There is a set of instructions for -this in lwpcook.pod which suggest you will need to install -HTTP::Request::Common and LWP::UserAgent; - -*) Set up the bug tracking system to send mail to 'tinderbox_bugs' on -the webserver machine whenever a Bug ticket changes state. It is not -interesting to see which tickets are being worked on, so restrict -mailing to changes in ticket state not updates (edit) of a ticket. - -*) Check that the time on your webserver, your version control -machine, your bug tracking machine and your build machines are all in -sync. Check that mail if build mail bounces on any of the above -machines that it will be received by someone who can act on it. - - - - -I keep my email up to date in the Contact file. If you have trouble -with these instructions please drop me a line. - -Ken Estes. +Originally written by Ken Estes. +Updated by Mike Taylor. diff --git a/mozilla/webtools/tinderbox2/Overview b/mozilla/webtools/tinderbox2/Overview index b72f42a56f8..267c82150be 100644 --- a/mozilla/webtools/tinderbox2/Overview +++ b/mozilla/webtools/tinderbox2/Overview @@ -1,157 +1,172 @@ - Overview of the Tinderbox System -------------------------------- -Tinderbox is an information display system. It runs on a machine with -a webserver and will periodically write static HTML files to the disk -so that the webserver can serve these documents. Tinderbox is run out -of cron every five minutes. It gathers up information from various -databases including: CVS Logs, Bonsai, and Perforce. It will also -process mail which is sent to it. Mail is sent from Bug Ticketing -software and Build/Test Machines. All this information is combined to -produce the HTML pages. +Tinderbox2 is an information display system. It runs on a machine with +a web server and will periodically write static HTML files to the disk +so that the web server can serve these documents. +Tinderbox2 is run out of cron as often as your needs require and gathers +up information from various databases including: Repository Logs, Bonsai +and Perforce. It will also process mail which is sent to it from Bug +Ticketing software and Build/Test Machines. All this information is +combined to produce the HTML pages. Since no two companies will structure their development processes the -same way, the tinderbox code has to be highly configurable to account +same way, the Tinderbox2 code has to be highly configurable to account for most possible uses. There is a main configuration file which allows most of the major user configurable variables to be set. Novice users can expect to edit only this file and get a working -tinderbox system. Additionally each library has been broken into two -parts. One part is the library specific configurations. This file is -expected to need modifications in some installations. I have put all -the library configurations into one directory to make it easy to find -the parts of tinderbox which are easy to modify. Each configuration -library can be thought of as a table which might need to be edited or -extended for use at your company. I have provided a working system -but the defaults may not suit your needs. These tables can be easily -changed in small ways by simply looking at the file and making obvious -changes. I have also allowed for the possibility of making complex -changes that only a competent perl programmer could define. Changes -are not made to the files which I have provided. Rather the changes -are made to copies of the files which are stored in a local +Tinderbox2 system. + +Additionally each library has been broken into two parts. One part is +the library specific configurations. This file is expected to need +modifications in some installations. All of the library configurations +are located in one directory to make it easy to find the parts of Tinderbox2 +which are easy to modify. Each configuration library can be thought of +as a table which might need to be edited or extended for use at your company. +The defaults provide a working system but may not suit your needs. These +tables can be easily changed in small ways by simply looking at the file +and making obvious changes. + +Changes are not made to the files which have been provided but rather +the changes are made to copies of the files which are stored in a local configuration directory. This ensures that you can easily version the -Tinderbox code as it is provided to you from the official distribution +Tinderbox2 code as it is provided to you from the official distribution and you can separately version the local configurations which you make. It is also easy to see the local configurations since you have both the original and the modified code on the same server and can -difference the two. As an example you might need to change the -BuildStatus - I assume that you have the following possible build -outcomes (Build in progress, Build failed, Build succeded but tests -failed, Build and all tests were successful) You may have additional -outcomes to specify which kind of tests failed (unit test failed, not -enough unit test coverage, performance tests failed). Similarly you -may have unusual requirements for how the filesystem should be laid -out. I provide a outcomes to specify which kind of tests failed (unit -test failed, not enough unit test coverage, performance tests failed). -Similarly you may have unusual requirements for how the filesystem -should be laid out. I suggest that you read through the files to see -how they are laid out and what types of changes are possible. +difference the two. +As an example you might need to change the BuildStatus - assuming that +you have the following possible build outcomes (Build in progress, +Build failed, Build succeded but tests failed, Build and all tests were +successful.) You may have additional outcomes to specify which kind of +tests failed (unit test failed, not enough unit test coverage, performance +tests failed). Similarly you may have unusual requirements for how the +filesystem should be laid out. -The build machines are not considered part of the tinderbox server. -They are clients just like Bug Ticketing systems and Version control +The build machines are not considered part of the Tinderbox2 server. +They are clients just like Bug Ticketing systems and Version Control systems are clients. Build machines mail their build logs to the server in a special format. This format specifies that name/value pairs must appear at the top of the mail message followed by the -complete build log. Scripts for setting up a tinderbox build client -can be found in the clientbin directory but you may have other build -needs and may use any build methods you choose. +complete build log. + +Scripts for setting up a Tinderbox2 build client can be found in the +clientbin directory but you may have other build needs and may use any +build methods you choose. + +The central concept of the Tinderbox2 system is the notion of a 'Tree'. -The central concept of the Tinderbox system is the notion of a 'Tree'. When several different groups are working out of the same version control system often the files are partitioned into separate modules with each group working on one or more disjoint modules. Over time the developers need to branch their code because several different versions of the files are under development at the same time. A tree -is a module/branch pair. This corresponds to a set of files which can -be checked out and built. Tinderbox makes one page for each tree and -displays what work is being done on that tree. CVS has a notion of -branches and of modules but not of trees. It is not possible to give -a branch/module pair a name. The tinderbox TreeData provides the -mappings between treenames and branch/module pairs. Tinderbox -displays the updates to bug tickets on the appropriate tree page. -This requires an easy mapping between bug tickets and trees. One -example of a complex function to determine tree name would be if each -of the product product types listed in the bug tracking data base -refers to one development project, except for a particular -feature/platform of one particular project which is being developed by -a separate group of developers. So the version control notion of -trees (a set of modules on a branch) may not have a direct map into -the bug tracking database at all times. In large projects it is -sometimes convenient to have a tree called 'ALL' which is used to -display all checkins performed on any trees and all bug tickets worked -on by any programmers. It is not possible to build or test the 'ALL' -tree and neither the version control nor bug ticketing system knows of -its existence. +is a module/branch pair and corresponds to a set of files which can +be checked out and built. Tinderbox2 creates one page for each tree and +displays what work is being done on that tree. +While most version control systems know about modules and branches, they +do not know about trees and it is not possible to give a branch/module pair +a name. The Tinderbox2 TreeData provides the mappings between treenames +and branch/module pairs. + +Tinderbox2 displays the updates to bug tickets on the appropriate tree +page. This requires an easy mapping between bug tickets and a tree. One +example of a complex function to determine tree name would be if each +of the product types listed in the bug tracking database refers to one +development project, except for a particular feature/platform of one +particular project which is being developed by a separate group of +developers. So the version control notion of trees (a set of modules +on a branch) may not have a direct map into the bug tracking database +at all times. + +In large projects it is sometimes convenient to have a tree called 'ALL' +which is used to display all checkins performed on any trees and all bug +tickets worked on by any programmers. It is not possible to build or +test the 'ALL' tree and neither the version control nor bug ticketing +system knows of its existence. The Bug Tracking code was intended to be as general as possible. Most bug ticketing systems send mail when tickets change state. The mail is often of the same form. It is a name/value pair which the -separator being the string ": ". Tinderbox will parse mail of this +separator being the string ": ". Tinderbox2 will parse mail of this form and display the interesting fields on the appropriate tree page. The configuration of this module involves specifying which bug ticket -names are interesting and should be displayed. Also you will need to -specify how to map a bug ticket into a a tree. This could be very -simple if each bug ticket has a field which represents the tree it is -applicable to (in this case tree could equal project) or can be very -complex if the tree must be computed by the values of a set of fields. -Also tinderbox keeps track of which bugs are "reopened" and displays +names are interesting and should be displayed. + +You will need to specify how to map a bug ticket into a tree, this +could be very simple if each bug ticket has a field which represents +the tree it is applicable to, in this case the tree would equal the +project. Or it can be very complex if the tree must be computed by the +values of a set of fields. + +Also Tinderbox2 keeps track of which bugs are "reopened" and displays them in a different column. The idea is that some bugs are moving backwards and creating duplicate work. These tickets are particularly -troublesome and should be watched specially. So possible all ticket +troublesome and should be watched specially. So possibly all ticket status are partitioned into "progress" or "slippage" categories. You will need to specify what status values are possible for your ticket system and you will also need to specify the set of columns which you would like to see on the status page. -The heart of the tinderbox system is the 'status table'. This is an +The heart of the Tinderbox2 system is the 'status table'. This is an HTML table which graphically shows how the changes made to the development databases. It will show what is going on in the version control system, the bug tracking system, the build system, automatic regression tests and provide a notice board for developers to inform -each other of current news. By placing all this information in the -same table it is possible to correlate and cross check how different -types of changes effected each other and what was going on with the -whole project at different times in the day. The rows of the table -represent time with the most current events at the top of the page. -There are different sets of columns for each database which needs to -be displayed. The sets of columns are managed by independent modules. +each other of current news. + +By placing all this information in the same table it is possible to +correlate and cross check how different types of changes effected each +other and what was going on with the whole project at different times +in the day. + +The rows of the table represent time with the most current events at the +top of the page. There are different sets of columns for each database +which needs to be displayed. The sets of columns are managed by +independent modules. + There is one module for each version control system and each bug -tracking system which tinderbox knows how to interface with. It is +tracking system which Tinderbox2 knows how to interface with. It is easy to port the system to new databases by just adding a new module using the same style as the existing modules. Modules never share or -peek at each others data all combining of data is done by the humans -who stare at the table and interpret what is going on. The main -tinderbox system does not know how many columns the final table will -have. It only knows about a list of table modules. Each module in the -list is called in turn to generate the complete row then the entire -row is displayed. The user must configure tinderbox with the list of -modules which are of important to their own environment. There is no -restriction on the number of modules which may be configured, though -due to implementation details each module can only appear once in the -table. There are many pop up windows embedded in the status table -these will provide extra level of detail when a mouse is placed over -the link. By moving your mouse around the page you may effectively -drill down into an item of interest and learn more about it without -leaving the page. Most of the links will click through to the -appropriate database. Thus if you need more data about an item you -can click on the link and query the database directly. +peek at each others data and all combining of data is done by the humans +who stare at the table and interpret what is going on. + +The main Tinderbox2 system does not know how many columns the final +table will have - it only knows about a list of table modules. +Each module in the list is called in turn to generate the complete row +then the entire row is displayed. The user must configure Tinderbox2 with +the list of modules which are of important to their own environment. +There is no restriction on the number of modules which may be configured, +though due to implementation details each module can only appear once +in the table. + +There are many pop up windows embedded in the status table these will +provide extra level of detail when a mouse is placed over the link. +By moving your mouse around the page you may effectively drill down +into an item of interest and learn more about it without leaving the page. +Most of the links will click through to the appropriate database. +Thus if you need more data about an item you can click on the link and +query the database directly. Besides the status table there is one other feature of the status -page. The page displays some information which is not correlated +page. The status page displays some information which is not correlated through time and with other data. This information is called status -table headers. The main headers are the message of the day (MOTD), -and the Tree State though there are a few others headers of mainly -historical interest. The important issue with the headers is that -they are not optional. Tinderbox can render a table with as little or -as many columns in the status table as you wish but each of the -headers has a particular place on the status page and needs to be -rendered in a particular way (font size, font type, etc) thus the -tinderbox server must know where each header must go and how to -specify the appropriate html context for this header. Users may set -null defaults for headers that they do not need but it is much harder -for a user to add new headers to the code in a modular fashion. +table headers. + +The main headers are the message of the day (MOTD), and the Tree State +though there are a few others headers of mainly historical interest. +The important issue with the headers is that they are not optional. +Tinderbox2 can render a table with as little or as many columns in the +status table as you wish but each of the headers has a particular place +on the status page and needs to be rendered in a particular way +(font size, font type, etc) thus the Tinderbox2 server must know where +each header must go and how to specify the appropriate HTML context for +the header. Users may set null defaults for headers that they do not +need but it is much harder for a user to add new headers to the code +in a modular fashion. diff --git a/mozilla/webtools/tinderbox2/Policies b/mozilla/webtools/tinderbox2/Policies index 6e8c702a6e5..29e41efbfbd 100644 --- a/mozilla/webtools/tinderbox2/Policies +++ b/mozilla/webtools/tinderbox2/Policies @@ -1,277 +1,296 @@ -Preparations you will need to make and -policies you will need to set: ------------------------------------ - -To install tinderbox you will need some information about your +To install Tinderbox2 you will need some information about your existing computer systems and some idea about what your goals are. -Here is a list of questions to help get you started, some of these + +Here is a list of topics to help get you started, some of these ideas may not be appropriate for your environment. +Web server configuration +------------------------ -The webserver will serve the tinderbox pages. -Webserver configuration is a bit of an art and you will need to -understand the policies which are used to administer your webserver. +The web server is what delivers the Tinderbox2 information to the world. +Web server configuration is a bit of an art and you will need to +understand the policies which are used to administer your web server. -*) You will need to decide the directory where tinderbox should write -the static HTML pages. This will depend on how your webserver is -configured. The default location is based on the RedHat 7.1 -(apache-1.3.19-5) installation and is: /var/www/html/tinderbox2. You -will also need to know what the URL browsers will need to use to find -this directory. Since tinderbox generates static web pages, it is -possible to run tinderbox and not run a web server. One way this -could be done is if you have a network file system and all users have -browsers which can read from the HTML directories. In this case all -URL's should begin with "file:/" instead of the usual "http://". + * You will need to decide the directory where Tinderbox2 should write + the static HTML pages. This will depend on how your web server is + configured. The default location is based on the RedHat 7.1 + (apache-1.3.19-5) installation and is: /var/www/html/tinderbox2. -*) Project level administration is done via cgi scripts. These -scripts allow administrators to set the message of the day, and the -state of the tree (open, closed, restricted). Also all users can post -notices to the web pages via a cgi script. CGI programs are often -restricted to a portion of the file system which is disjoint from the -HTML files. You will need to figure out where the CGI programs will -go. Tinderbox takes its defaults from RedHat 7.1 and uses: -/var/www/cgi-bin/tinderbox2. You will also need to know what the URL -browsers will need to use to find this directory. + You will also need to know what the URL browsers will need to use + to find this directory. Since Tinderbox2 generates static web + pages, it is possible to run Tinderbox2 and not run a web server. + One way this could be done is if you have a network file system + and all users have browsers which can read from the HTML directories. + In this case all URL's should begin with "file:/" instead of the + usual "http://". -*) CGI scripts will run as an unauthenticated user on your system. -You will need to decide which user will run the tinderbox CGI scripts. -The same user id must be used for running the scripts as for tinderbox -mail delivery. The Tinderbox Configuration files will define this -user id and as a security precaution check that it is running as the -required id. It is suggested that this id not be a privileged id -(higher ids are better, please make this number be greater than 10 and -bigger than 100 is recommended). Smaller ids are often assumed to -have more privileges on a Unix box then larger ids. It is not a good -idea for an unauthenticated user to have any privileges so a large id -is recommended. It is also recommended that you not use the id 'nobody' -as this id is over used and it would be better to partition the -unauthenticated user into separate ids in case of security problems. -RedHat runs all its CGI scripts as the user 'apache', this is an -acceptable user. I would prefer to have a separate user to run the -tinderbox CGI scripts but this would require recompiling apache to -enable suEXEC, and it is more effort then most groups can afford. + * Project level administration is done via cgi scripts. These scripts + allow administrators to set the message of the day, and the state + of the tree (open, closed, restricted). Also all users can post + notices to the web pages via a cgi script. + + CGI programs are often restricted to a portion of the file system + which is disjoint from the HTML files. You will need to figure out + where the CGI programs will go. Tinderbox2 takes its defaults from + RedHat 7.1 and uses /var/www/cgi-bin/tinderbox2. You will also need + to know what the URL browsers will need to use to find this directory. -*) Tinderbox Files. There are other tinderbox files which need to be -placed on the webserver. These include libraries and non-cgi -programs. You will need to decide where to place these files. Most -users put them in /home/tinderbox2. + * CGI scripts will run as an un-authenticated user on your system. + You will need to decide which user will run the Tinderbox2 CGI + scripts. The same user id must be used for running the scripts as + for Tinderbox2 mail delivery. -*) Tinderbox Data. Tinderbox stores its data in the file system. For -security it is often a good idea to keep this data out of the HTML and -CGI directories so that malicious users can not directly access this -data. The compressed build logs can grow quite large, so it is -recommended to put the data on a file system with room. The default -is to put them in the directory /home/tinderbox2/data. + The Tinderbox2 configuration files will define this user id and, + as a security precaution, check that it is running as the required + id. It is suggested that this id not be a privileged id (higher + ids are better, please make this number be greater than 10 and + bigger than 100 is recommended). Smaller ids are often assumed to + have more privileges on a Unix box then larger ids. It is not a + good idea for an un-authenticated user to have any privileges so a + large id is recommended. + It is also recommended that you not use the id 'nobody' as this id + is over used and it would be better to partition the un-authenticated + user into separate ids in case of security problems. + + RedHat runs all its CGI scripts as the user 'apache', this is an + acceptable user. I would prefer to have a separate user to run the + Tinderbox2 CGI scripts but this would require recompiling apache to + enable suEXEC, and it is more effort then most groups can afford. + + * Tinderbox2 Files. There are other Tinderbox2 files which need to + be placed on the web server. These include libraries and non-cgi + programs. You will need to decide where to place these files. + Most users put them in /home/tinderbox2. + + * Tinderbox2 Data. Tinderbox2 stores its data in the file system. + For security it is often a good idea to keep this data out of the + HTML and CGI directories so that malicious users can not directly + access this data. The compressed build logs can grow quite large, + so it is recommended to put the data on a file system with room. + The default is to put them in the directory /home/tinderbox2/data. Mail ---- -*) Many of the tinderbox modules (Bug Ticket, Build, CVS) receive -their data via mail. The mail system on you web server machine must -be configured to deliver the mail into the tinderbox mail processing -programs. You should spend some time understanding how your mail -delivery system can be configured to allow user mail to be delivered -into a program and how to set the user id under which this delivery -occurs. If you do not wish to configure your mail delivery program -then you can use fetchmail to pull the mail out of a mail box and push -it into the programs on a periodic basis. See the install page for -details on what I have learned about mailing systems. +Many of the Tinderbox2 modules (Bug Ticket, Build, CVS) receive their +data via mail. + * The mail system on your web server machine must be configured to + deliver the mail into the Tinderbox2 mail processing programs. + You should spend some time understanding how your mail delivery + system can be configured to allow user mail to be delivered into a + program and how to set the user id under which this delivery occurs. + + If you do not wish to configure your mail delivery program then you + can use fetchmail to pull the mail out of a mail box and push it into + the programs on a periodic basis. See the install page for details + on different aspects about mailing systems. Production Version Control ------------------------- -One of the biggest responsibilities which a "buildmaster" has is the +One of the biggest responsibilities which a "build master" has is the requirement that all code should be reproducible. That is that at any point in the future, even more than one year later, the current binaries should be able to be rebuilt byte for byte from sources. + This requirement can be broken down as follows: -1) The build machine must be reproducible. + * The build machine must be reproducible. -We must be able to get back the same build machine we had at any point -in the past. This means that all OS libraries, all header files, all -compilers, all build tools (make, grep, sed) must have some mechanism -to roll back. It is common to use a backup of the build machine to -reconstruct it. Most OS will give you a list of the software packages -which are installed on the machine and their version numbers. I like -to keep the list of software packages which are installed on the -machine checked into version control. This allows me to compare the -state of the build machine at any two points in time. I have tools to -recreate the build-machine from just a list of packages with version -numbers. It is considered a best practice to limit the amount of -software which is available on the build machine. A build machine -with too much installed will only make it difficult to reproduce older -builds should the need arise. I recommend not installing any -web servers or graphical window managers on your build machine. It -should be clear that the build machine should not be the same machine -where the tinderbox server runs. + You must be able to get back the same build machine configuration + you had at any point in the past. This means that all OS libraries, + all header files, all compilers, all build tools (make, grep, sed) + must have some mechanism to roll back. It is common to use a backup + of the build machine to reconstruct it. Most OS will give you a + list of the software packages which are installed on the machine and + their version numbers. Maintaining a the list of software packages + which are installed on the machine and check it into version control. + This allows you to compare the state of the build machine at any two + points in time. + + It is considered a best practice to limit the amount of software which + is available on the build machine. A build machine with too much + installed will only make it difficult to reproduce older builds should + the need arise. A build machine should not have installed any web + servers or graphical window managers on your build machine. It should + be clear that the build machine should not be the same machine where + the Tinderbox2 server runs. -2) The build process must be reproducible. That is all the steps -which are used to create the application must be reproducible. + * The build process must be reproducible. That is all the steps which + are used to create the application must be reproducible. -*) Build Interface: We must be able to run exactly the same build -process in the future including: all commands with command line -arguments, all environmental variables. I recommend that the entire -build process be viewed as something outside of the build master -control. Developers are responsible for ensuring that there is a -simple build master interface to construct all the software products -which go into a build. Typically there is a makefile in a standard -place where the buildmaster can run something like "make all; make -install;" and be guaranteed that this will build the product. The -build interface should be viewed as something which never changes and - are part of the build machine, like the OS and are changed only -rarely. It is hard enough to track all the parts of the build process -which we expect to change, we should not need to track complex build -procedures. The build procedures should have a standard interface. -By keeping the build instructions in one makefile which is checked -into the same version control system as the sources it is easy to -recreate any previous build even if the commands used to build the -software fluctuate rapidly between releases. There must be a simple -interface to construct the software which will hide all the complexity -of the actual construction. + * Build Interface - You must be able to run exactly the same build + process in the future including all commands with command line + arguments, all environmental variables. I recommend that the entire + build process be viewed as something outside of the build master + control. Developers are responsible for ensuring that there is a + simple build master interface to construct all the software products + which go into a build. Typically there is a Makefile in a standard + place where the build master can run something like "make all; make + install;" and be guaranteed that this will build the product. -*) Build Environment: The makefile will code all the build commands -and all the environmental variables (PATH, UMASK, LD_LIBRARY_PATH, -CLASSPATH) needed to build the software though it may rely on some -well defined command line arguments (PREFIX, CCFLAGS, JAVA_LIBS) to -make these prematurely. These command line arguments should not -change between versions of the software but should be a fixed set of -build parameters. The parameters may be needed to specify where some -files are found on the build machine (Ideally the build machine is set -up the same as developers machines so these directories can be -hard-coded into the makefiles but often there is a need for some -directories to be specified at build time) or where files are to be -created/installed on the build machine (typically a subdirectory of -/var/tmp but there may be several builds running at once and each will -need a different directory) or what kind of build is being created. -Each part of the build which needs a particular environmental variable -set or a special header file in some path should have tests which -ensure that the build environment is valid. I keep my build scripts -installed on the build machine and they are always started by running -/etc/rc.d/init.d/build start this ensures that I am not relying on any -build environmental variables which are set by logging into the build -account and are thus not tracked and versioned. + The build interface should be viewed as something which never changes + and are part of the build machine, like the OS and are changed only + rarely. It is hard enough to track all the parts of the build process + which are epected to change, let alone trying to track complex build + procedures. -*) Environmental safety issues: + The build procedures should have a standard interface. By keeping the + build instructions in one Makefile which is checked into the same + version control system as the sources; it is easy to recreate any + previous build even if the commands used to build the software + fluctuate rapidly between releases. There must be a simple interface + to construct the software which will hide all the complexity of the + actual construction. -If the build environment can not be used to build the software then a -human readable error message should be generated. My makefiles often -run various checks on the environmental variables before they -construct the code. They check that all required environmental -variables are set, that the required libraries are found, that -directories which must be disjoint (build and install directories) do -not overlap. This test suite becomes a build regression test and as I -discover additional possible build problems I add new tests to the -makefile. I make it a habit to explicit set all environmental -variables so that there is no doubt as to their expected values. It -is important for the QA group to only use Builds which were created by -an automated process so that we are sure that there are no -undocumented steps in either the test builds or the released build. + * Build Environment - The Makefile will code all the build commands + and all the environmental variables (PATH, UMASK, LD_LIBRARY_PATH, + CLASSPATH) needed to build the software though it may rely on some + well defined command line arguments (PREFIX, CCFLAGS, JAVA_LIBS) to + make these prematurely. These command line arguments should not + change between versions of the software but should be a fixed set of + build parameters. -3) Track the Build numbers. Given a clean install of your product you -should have all the information necessary to reproduce the executable -from sources. If a customer shows you the application binaries you -must be able to get the source code which build the application, -reconstruct the build machine which created the application and -possibly rerun the build exactly the same way as the application was -created before, this may include making some minor source code changes -before the build is run. I like to keep a file which contains: + The parameters may be needed to specify where some files are found + on the build machine (Ideally the build machine is set up the same + as developers machines so these directories can be hard-coded into + the Makefile but often there is a need for some directories to be + specified at build time) or where files are to be created/installed + on the build machine (typically a subdirectory of /var/tmp but there + may be several builds running at once and each will need a different + directory) or what kind of build is being created. - The product release name + Each part of the build which needs a particular environmental variable + set or a special header file in some path should have tests which + ensure that the build environment is valid. The build scripts could be + installed on the build machine and started by running: + "/etc/rc.d/init.d/build start". This would ensure that the build script + does not rely on any build environmental variables which are set by + logging into the build account and are thus not tracked and versioned. - The sources 'as of date'. (I always checkout my sources using - cvs -D 'date time' so that exactly the same sources - can be recovered knowing only the 'data time' which - was used to check them out. I am sure a similar trick - could be used with a perforce 'change set number'.) + * Environmental safety issues. If the build environment can not be used + to build the software then a human readable error message should be + generated. The Makefile can often run various checks on the environmen + variables before they construct the code. They check that all required + environment variables are set, that the required libraries are found, + that directories which must be disjoint (build and install directories) + do not overlap. - The branch name. + This test suite becomes a build regression test and as additional possible + build problems are discovered, new tests can be added to the Makefile. + It is a good habit to explicit set all environment variables so that there + is no doubt as to their expected values. It is important for the QA group + to only use Builds which were created by an automated process so that you + are sure that there are no undocumented steps in either the test builds or + the released build. - The module name. + * Track the Build numbers. Given a clean install of your product you + should have all the information necessary to reproduce the executable + from sources. If a customer shows you the application binaries you + must be able to get the source code which build the application, + reconstruct the build machine which created the application and + possibly rerun the build exactly the same way as the application was + created before, this may include making some minor source code changes + before the build is run. + + One method is to keep a file which contains: + - The product release name + - The sources 'as of date'. (Always checkout sources using + a known datetime or revision # (cvs -D or svn -R) so that + exactly the same sources can be recovered knowing only the + 'date/time' which was used to check them out. + - The branch name + - The module name. -This can be stored as a file in the product (encrypted if necessary) -or may be stored in some secure build master database where the data -can be looked up by release name. My preference is to keep all data -necessary to reproduce a build in the build output and delivered as -part of the product. This means that I can generate as many builds as -I want automatically and not need to keep track of any of them. When -the QA team deems that a certain build is 'important', by making a -particular build the official released copy then I can take a look at -its contents and tag/branch the code at the sources which I used to -build it. + This can be stored as a file in the product (encrypted if necessary) + or may be stored in some secure build master database where the data + can be looked up by release name. A good practice is to keep all data + necessary to reproduce a build in the build output and delivered as + part of the product. This means that you can generate as many builds as + needed automatically and not need to keep track of any of them. When + the QA team deems that a certain build is 'important', by making a + particular build the official released copy then you can take a look at + its contents and tag/branch the code at the sources which were used to + build it. -4) Build Prefix: It is a good idea to familiarize yourself with the -makefile conventions regarding the make variable PREFIX. It is -easiest to understand if you think about what RedHat does when they -build their distribution of RPM's but this will apply in many -different systems including the Andrew File System (AFS) and most -packaging systems. This variable is used during the build process -"make all PREFIX=/home/apache" to tell the package where it will be -installed (examples include /usr, /usr/local, /home/apache). I -suggest reading a few RedHat Spec files to see how this works in -practice. The application may need to hard-code this value into its -object code. When the application is installed it must not be -installed into its proper place on the build machine. The package we -are constructing could cause the build machine to stop working -correctly if it is a buggy version of a system library or major OS -application. Instead the makefile will install "make install -PREFIX=/var/tmp/build-root/home/apache" the package into some other -directory with a similar tree structure to its final destination. The -packaging system will then move the files into the correct place -during an installation step on the target machine. The installation -step only moves files and sets permissions. The makefile is not -supposed to use the installation directories to hard code values into -the application since the application will never be run from this -installation directory. The hard part of the build including any -PREFIX magic is in the build section. Notice the clear separation -between build machine / target machine and installation on the build -machine and installation on the target machine and construction of the -application binaries and installation of the application binaries. -This is one of the reasons why building an application on a build -machine is different from the way in which developers build their code -on their personal development machines. This PREFIX issue will arise -when you try and build the Tinderbox system and also when you -construct the makefiles for your own application. Since the build -machine is not the target machine it can not be assumed that files -will always be in the same places on both (for example perl). + * Build Prefix: It is a good idea to familiarize yourself with the + Makefile conventions regarding the make variable PREFIX. It is + easiest to understand if you think about what RedHat does when they + build their distribution of RPM's but this will apply in many + different systems including the Andrew File System (AFS) and most + packaging systems. -5) Application Architecture: + This variable is used during the build process + "make all PREFIX=/home/apache" to tell the package where it will be + installed (examples include /usr, /usr/local, /home/apache). Reading + a few RedHat Spec files to see how this works in practice. The + application may need to hard-code this value into its object code. + When the application is installed it must not be installed into its + proper place on the build machine. The package that is being + constructed could cause the build machine to stop working + correctly if it is a buggy version of a system library or major OS + application. Instead the Makefile will install "make install + PREFIX=/var/tmp/build-root/home/apache" the package into some other + directory with a similar tree structure to its final destination. -*) The build process should mimic the architecture of the code. It should -be a final test that the code was coded to the same specifications -that it was designed. It is a common problem for code to turn into -spaghetti with each piece of code using functions and creating -dependencies on every other piece of code. For example it is probably -a mistake for code in the database abstraction layer to be implemented -in terms of code in the HTML generation layer. These two libraries -should probably be independent of each other, though they both might -depend on a common string library. The code architecture should limit -the dependency graph between code modules. The BuildMaster must -enforce the restrictions on information flow between components. Thus -no libraries should be in the path unless the architecture allows this -module to depend on those libraries. + The packaging system will then move the files into the correct place + during an installation step on the target machine. The installation + step only moves files and sets permissions. The Makefile is not + supposed to use the installation directories to hard code values into + the application since the application will never be run from this + installation directory. -*) The architecture must not have circular dependencies. Circular -dependencies not only make upgrading individual libraries difficult -but also make testing components nearly impossible. That is it should -be possible to build some set of libraries L0 which depend on no -libraries and then build some other set of libraries L1 which depend -only on L0 libraries then build L2 which depend only on the L0 and L1 -libraries. This "build chain" will prevent circular dependencies and -help keep your code testable and the dependencies understandable. -More information about why this is a good practice is available in -"Large-Scale C++ Software Design" (Addison-Wesley Professional -Computing Series) by John Lakos + The hard part of the build, including any PREFIX magic, is in the + build section. Notice the clear separation between build/target + machine, installation on the build machine, installation on the target + machine, construction of the application binaries and installation of + the application binaries. -*) I enforce the convention that developers are not allowed to overload -standard system libraries. I always put standard libraries in the -path before any library our company develops. I build the application -in stages to ensure that parts of the application which are not -intended to depend on other code will not have other header files on -the build machine at the time that they are constructed. Build -dependencies between modules which are expected are explicitly -controlled with build scripts and version numbers. + This is one of the reasons why building an application on a build + machine is different from the way in which developers build their code + on their personal development machines. This PREFIX issue will arise + when you try and build the Tinderbox2 system and also when you + construct the Makefiles for your own application. Since the build + machine is not the target machine it can not be assumed that files + will always be in the same places on both (for example Perl.) + + * Application Architecture. The build process should mimic the + architecture of the code. It should be a final test that the code + was coded to the same specifications that it was designed. It is a + common problem for code to turn into spaghetti with each piece of + code using functions and creating dependencies on every other piece + of code. + + For example it is probably a mistake for code in the database + abstraction layer to be implemented in terms of code in the HTML + generation layer. These two libraries should probably be independent + of each other, though they both might depend on a common string library. + The code architecture should limit the dependency graph between code + modules. The Build Master must enforce the restrictions on information + flow between components. Thus no libraries should be in the path unless + the architecture allows this module to depend on those libraries. + + The architecture must not have circular dependencies. Circular + dependencies not only make upgrading individual libraries difficult + but also make testing components nearly impossible. That is it should + be possible to build some set of libraries L0 which depend on no + libraries and then build some other set of libraries L1 which depend + only on L0 libraries then build L2 which depend only on the L0 and L1 + libraries. This "build chain" will prevent circular dependencies and + help keep your code testable and the dependencies understandable. + More information about why this is a good practice is available in + "Large-Scale C++ Software Design" (Addison-Wesley Professional + Computing Series) by John Lakos + + You should enforce the convention that developers are not allowed to + overload standard system libraries. Always put standard libraries + in the path before any library our company develops. Build the + application in stages to ensure that parts of the application which + are not intended to depend on other code will not have other header + files on the build machine at the time that they are constructed. + Build dependencies between modules which are expected are explicitly + controlled with build scripts and version numbers. diff --git a/mozilla/webtools/tinderbox2/ReleaseNotes b/mozilla/webtools/tinderbox2/ReleaseNotes new file mode 100644 index 00000000000..0538223e925 --- /dev/null +++ b/mozilla/webtools/tinderbox2/ReleaseNotes @@ -0,0 +1,163 @@ +Release ??? + + New Time columns for use in environments with multiple time zones. + + Spelling fixes + + Add support for popup windows which work "on click" + + Add support for users who want to see tinderbox's notion of what was + checked in not just Bonsai's query. + + Add support for historical Tinderbox images + (flames, stars) + + Mozilla layout preferences are now standard + (non uniform row spacing, + truly empty cells, + legend on bottom of page). + + Allow users to backdate notices + + automate regeneration of HTML when user uses CGI interface + + Put the C in all build cells. + + VC code did the wrong thing when generating old pages and there was + no TreeState data. + + Fix Spelling mistakes throughout code. + + Add support for build machines to tell the server to add text/links + into build cells (TinderboxPrint). + + When builds are started too frequently (more then once every five minutes) + newest build is now kept and oldest build is thrown away. + + New Configure script contributed by Stephen Lamm (slamm@yahoo-inc.com) + This includes a fix to allow Perl to be in different locations on the + machine where make is run and on the machine where the web server runs. + + Now it is possible to include the time column on the left and right + side of the status table. + + There is now a working Perforce module for those who use this product + instead of CVS. + + There is a module to interface with the req ticketing system. + + New Summary format and number of errors displayed on build page by + Adam Di Carlo , + + Spelling and consistency fixes from: + Adam Di Carlo , + Orion Poplawski , + John L. Villanlovos , + + Change HTML generation so that the text browser 'links' + (http://artax.karlin.mff.cuni.cz/~mikulas/links/) can + work with tinderbox. + + Allow the build script to use a single process and alternate builds + of different types. + + Add a sample /etc/rc.d/init.d file for the RedHat Build machines. + + Mail processing programs were not writing error files correctly. + This has been broken since I separated out the mail processing module. + + Add script to cull old builds from a directory of completed builds. + + Let the VC and BT headers be configurable. Too many companies find + this an important feature. + + Allow Tinderbox to run via suid wrappers. This is needed for some + times of mail delivery systems + + No longer exclude 'privileged id's from running Tinderbox. + Now we require tinderbox to run as the id which the user specifies + is the tinderbox id. + + Better errors for failures in Storable.pm + + BasicTxtHeader.pm had a bug which prevented it from working when using Storable + + Rework the BT modules to allow use when URL is not easily mapped from the bugid + + Begin work to allow tinderbox admin pages to open and close bonsai trees + + Fix code which allowed tinderbox to determine the state of bonsai trees + + Major bug in the untainting code, fixed. This was a security + problem. I suspect people may have trouble with the fix but this was + how the code was intended to work. I do not understand how this got + through my unit tests. + + Give a useful example of how use the new build script. + + Improve the spec files, split out build machine packages. + + +Release 0.9 + + Add full Bonsai integration functionality. + + Allow users to define a display order for the buildnames on a per tree basis. + This adds a new function into TreeData.pm check your local files. + + Fix computation of 'deadtime' for eazel.com's build setup, + this will improve their estimate of when the next build will be + complete. + + Develop portable client build scripts. + + Add RPM spec files. + + Relax check of root id for all redhat users. + uid nobody is 99 and by default redhat apache runs all cgi as nobody. + + VC_CVS.pm was not removing excess treestates from the Database correctly, + now fixed. + +Release 0.8 + + This release moved some files into new directories update with + cvs update -d -P + + Remove the strict time checking on mail, it caused too many problems. + + Fix the Unmonitored Builds in admintree.cgi and in status.html page. + + Storable.pm has been tested as a Persistence implementation. + + Improve the prediction of when builds will end by introducing the + notion of 'deadtime'. + + Separate out the BuildStatus table for easier configuration of the + status without changing the build display library. + + Better comments in the ReadMe files + + Better security: + + - All scripts run in taint perl mode + + - Separation of HTML and DATA directories in + addition to the CGI directory + + - Additional checks of ownership for the data directories + + - Check the network address of all administrative commands + + Improved Makefile/Configure support + + New permissions help prevent accidental edit of files in the + build or install directory + + Inter-positioning of libraries, for local configuration, + fully supported + + Test arguments added to Makefile + + Test compiles of the perl are no longer required + diff --git a/mozilla/webtools/tinderbox2/Todo b/mozilla/webtools/tinderbox2/Todo index daf15f65299..6a5d221b0f0 100644 --- a/mozilla/webtools/tinderbox2/Todo +++ b/mozilla/webtools/tinderbox2/Todo @@ -1,3 +1,12 @@ +Bear's Todo List: + + * Integrate SVN for blame column - bug 303333 + * Allow Tinderbox2 to use a Bonsai on another computer - bug 237289 + * Bring perl code up to some of the newer Mozilla styles + * Create more export streams - rdf, atom feed, pub/sub??? + + +Original Todo list: It would be really nice if CVS/Bonsai could mail tinderbox the updates which apply to the branches of interest, then tinderbox would not have