Bommels05/Mozilla
Bommels05/Mozilla
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated info files so they read in the third person

Browse Source 
fixed typos and grammer
changed layout to be a bit more readable
Moved release notes into a ReleaseNotes file


git-svn-id: svn://10.0.0.236/trunk@181822 18797224-902f-48f8-a5cc-f745e15eee43
This commit is contained in:
bear%code-bear.com 2005-10-08 18:57:16 +00:00
parent 611c5ee3e2
commit 5bfae9cd23
8 changed files with 873 additions and 807 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

178
mozilla/webtools/tinderbox2/ChangeLog
View File

@ -1,3 +1,18 @@
2005-10-08 Mike Taylor <bear@code-bear.com>
* ChangeLog
removed release note text and put them into ReleaseNotes file
* Todo
added my Todo list
* Contact
removed tabs
* ReadMe, Policies, Overview, Goals, Install
edited to read in the third person. Fixed typos, spelling
and reformatted for readability
2005-10-06 Mike Taylor <bear@code-bear.com>
* Contact
@ -11,166 +26,3 @@
fixed spelling error in generated html content s/BPRDER/BORDER/
bug 230348 patch supplied by Todd Endicott
New Time columns for use in environments with multiple time zones.
Spelling fixes
Add support for popup windows which work "on click"
Add support for users who want to see tinderbox's notion of what was
checked in not just Bonsai's query.
Add support for historical Tinderbox images
(flames, stars)
Mozilla layout preferences are now
standard
(non uniform row spacing,
truly empty cells,
legend on bottom of page).
Allow users to backdate notices
automate regeneration of HTML when user uses CGI
interface
Put the C in all build cells.
VC code did the wrong thing when generating old pages and there was
no TreeState data.
Fix Spelling mistakes throughout code.
Add support for build machines to tell the server to add text/links
into build cells (TinderboxPrint).
When builds are started too frequently (more then once every five minutes)
newest build is now kept and oldest build is thrown away.
New Configure script contributed by Stephen Lamm (slamm@yahoo-inc.com)
This includes a fix to allow Perl to be in different locations on the
machine where make is run and on the machine where the webserver runs.
Now it is possible to include the time column on the left and right
side of the status table.
There is now a working Perforce module for those who use this product
instead of CVS.
There is a module to interface with the req ticketing system.
New Summary format and number of errors displayed on build page by
Adam Di Carlo <adam@onshore-devel.com>,
Spelling and consistency fixes from:
Adam Di Carlo <adam@onshore-devel.com>,
Orion Poplawski <OPoplawski@cqg.com>,
John L. Villanlovos <john.l.villalovos@intel.com>,
Change HTML generation so that the text browser 'links'
(http://artax.karlin.mff.cuni.cz/~mikulas/links/) can
work with tinderbox.
Allow the build script to use a single process and alternate builds
of different types.
Add a sample /etc/rc.d/init.d file for the RedHat Build machines.
Mail processing programs were not writing error files correctly.
This has been broken since I separated out the mail processing module.
Add script to cull old builds from a directory of completed builds.
Let the VC and BT headers be configurable. Too many companies find
this an important feature.
Allow Tinderbox to run via suid wrappers. This is needed for some
times of mail delivery systems
No longer exclude 'privileged id's from running Tinderbox.
Now we require tinderbox to run as the id which the user specifies
is the tinderbox id.
Better errors for failures in Storable.pm
BasicTxtHeader.pm had a bug which prevented it from working when using Storable
Rework the BT modules to allow use when URL is not easily mapped from the bugid
Begin work to allow tinderbox admin pages to open and close bonsai trees
fix code which allowed tinderbox to determine the state of bonsai trees
Major bug in the untainting code, fixed. This was a security
problem. I suspect people may have trouble with the fix but this was
how the code was intended to work. I do not understand how this got
through my unit tests.
Give a useful example of how use the new build script.
Improve the specfiles, split out build machine packages.
Release 0.9
Add full Bonsai integration functionality.
Allow users to define a display order for the buildnames on a per tree basis.
This adds a new function into TreeData.pm check your local files.
Fix computation of 'deadtime' for eazel.com's build setup,
this will improve their estimate of when the next build will be
complete.
Develop portable client build scripts.
Add RPM specfiles.
Relax check of root id for all redhat users.
uid nobody is 99 and by default redhat apache runs all cgi as nobody.
VC_CVS.pm was not removing excess treestates from the Database correctly,
now fixed.
Release 0.8
This release moved some files into new directories update with
cvs update -d -P
Remove the strict time checking on mail, it caused too many problems.
Fix the Unmonitored Builds in admintree.cgi and in status.html page.
Storable.pm has been tested as a Persistence implementation.
Improve the prediction of when builds will end by introducing the
notion of 'deadtime'.
Separate out the BuildStatus table for easier configuration of the
status without changing the build display library.
Better comments in the ReadMe files
Better security:
All scripts run in taint perl mode
Separation of HTML and DATA directories in
addition to the CGI directory
Additional checks of ownership for the data directories
Check the network address of all administrative commands
Improved Makefile/Configure support
New permissions help prevent accidental edit of files in the
build or install directory
Interpositioning of libraries, for local configuration,
fully supported
Test arguments added to Makefile
Test compiles of the perl are no longer required

6
mozilla/webtools/tinderbox2/Contact
View File

@ -1,9 +1,9 @@
Bug Tickets can be filed at:
http://bugzilla.mozilla.org/show_bug.cgi?id=113212
http://bugzilla.mozilla.org/show_bug.cgi?id=113212
Product: Webtools
Component: Tinderbox2
Product: Webtools
Component: Tinderbox2
Tinderbox2 is currently being co-maintained by:

156
mozilla/webtools/tinderbox2/Goals
View File

@ -4,107 +4,109 @@ USERS
It is helpful to consider the types of users for this system and their
special needs.
1) Project Managers: not very technical will be administering the
project and need a GUI to help change various types of project
information (treestate, message of the day, etc). They will also need
summary pages which will show them the current status of all the
projects which they are working on. They may occasionally "drill
down" into the detailed status page but this will not be their primary
view of the tinderbox system.
* Project Managers: not very technical will be administering the
project and need a GUI to help change various types of project
information (tree state, message of the day, etc). They will also
need summary pages which will show them the current status of all
the projects which they are working on. They may occasionally
"drill down" into the detailed status page but this will not be
their primary view of the Tinderbox2 system.
2) Build Administrator: A system administrator who will be in charge
of setting up the build machines, configuring tinderbox and other
build systems (bugzilla, cvs, bonsai, etc). A GUI would not be
helpful as local customizations may require small changes to the code.
Configurations need to be kept (mostly) in files which are separate
from the Tinderbox source code so that they can be version controled
and will not get stepped on when tinderbox is upgraded.
* Build Administrator: A system administrator who will be in charge
of setting up the build machines, configuring Tinderbox2 and other
build systems (bugzilla, cvs, bonsai, etc). A GUI would not be
helpful as local customizations may require small changes to the code.
3) Developers: need to view the "state of development" and add notices
to the notice board.
Configurations need to be kept (mostly) in files which are separate
from the Tinderbox2 source code so that they can be version controlled
and will not get stepped on when Tinderbox2 is upgraded.
* Developers: need to view the "state of development" and add notices
to the notice board.
Improvements needed from Tinderbox1
-----------------------------------
highly configurable design with multiple Version Control systems
possible (bonsai, raw cvs, perforce, continuous, clearcase) and
multiple modes of running possible (with no version control system
with no builds display).
* Highly configurable design with multiple Version Control systems
possible (bonsai, raw cvs, perforce, continuous, clearcase) and
multiple modes of running possible (with no version control system
with no builds display).
clear programatic interfaces and better separation of functionality
into separate modules.
* Clear programatic interfaces and better separation of functionality
into separate modules.
Modules should not have circular dependences. Care should be taken on
adding 'use' statement. I worry very much about which modules depend
on which other modules. It is important to consider which modules
need to work in isolation and which modules need to share data.
* Modules should not have circular dependences. Care should be taken
on adding 'use' statement as you should worry very much about which
modules depend on which other modules. It is important to consider
which modules need to work in isolation and which modules need to
share data.
It should be possible to add hooks so that users get beeped when the
next good build goes through or that trouble tickets are automatically
opened when the builds fail.
* It should be possible to add hooks so that users get beeped when the
next good build goes through or that trouble tickets are automatically
opened when the builds fail.
Greater flexibility in setting status of builds. We may need more
gradations of failure then just 'busted' or 'test-failed' to
distinguish the types of tests which have failed.
* Greater flexibility in setting status of builds. We may need more
graduations of failure then just 'busted' or 'test-failed' to
distinguish the types of tests which have failed.
generated html must be readable and help isolate programming errors.
* Generated HTML must be readable and help isolate programming errors.
all programmable configuration parameters should be stored easy change
and configure for novice users.
* All programmable configuration parameters should be stored easy change
and configure for novice users.
make better use of the perl data structures to mirror the way we wish
to use the data. This will allow easier maintainability and allow for
more expansion of features.
* Make better use of the Perl data structures to mirror the way we wish
to use the data. This will allow easier maintainability and allow for
more expansion of features.
display should work on many different browsers.
* Display should work on many different browsers.
popup windows should not be netscape specific.
* Popup windows should not be Netscape specific.
Permanent data should be stored via datadumper so that the data and
data structures are easy to read and debug. Currently this is a
performance bottle neck with a large percentage of our cpu time during
testing being spent in Data::Dumper::Dump. The perl module Storable
is much faster. I wish to not add additional module requirements at
this time, this will be configurable.
* Permanent data should be stored via datadumper so that the data and
data structures are easy to read and debug. Currently this is a
performance bottle neck with a large percentage of cpu time during
testing being spent in Data::Dumper::Dump. The Perl module Storable
is much faster and there is no need to add additional module
requirements at this time, this will be configurable.
# dprofpp says that:
# %64.8 of elapsed real time which is 66.25 seconds
# (out of 102.15 Seconds)
# was spent in 3 calls to TinderDB::VC::apply_db_updates()
# dprofpp says that:
# %64.8 of elapsed real time which is 66.25 seconds
# (out of 102.15 Seconds)
# was spent in 3 calls to TinderDB::VC::apply_db_updates()
# %58.0 of user time which is 11.05 seconds
# (out of 19.03 User/102.15 Elapsed Seconds)
# was spend in 32878 calls to Data::Dumper::_dump()
# %58.0 of user time which is 11.05 seconds
# (out of 19.03 User/102.15 Elapsed Seconds)
# was spend in 32878 calls to Data::Dumper::_dump()
# System Time was negligible at 2.49 Seconds
# System Time was negligible at 2.49 Seconds
All errors should be trapped and sent to log files. Strange program
states should be explicitly checked for.
* All errors should be trapped and sent to log files. Strange program
states should be explicitly checked for.
Databases should update atomically, no information should be lost due
to race conditions.
* Databases should update atomically, no information should be lost due
to race conditions.
All column modules (processmail, build, VC, Notices) should be able to
be run individually. Modules should accept well defined text files as
input and produce text files as output. This will greatly enhance the
ability to test each module in isolation and to quickly port modules
to new architectures.
* All column modules (processmail, build, VC, Notices) should be able to
be run individually. Modules should accept well defined text files as
input and produce text files as output. This will greatly enhance the
ability to test each module in isolation and to quickly port modules
to new architectures.
The source code should be able to run using the standard Perl
libraries, as it can be difficult for some users to add libraries
which are not included in the perl distribution onto production
machines.
* The source code should be able to run using the standard Perl
libraries, as it can be difficult for some users to add libraries
which are not included in the perl distribution onto production
machines.
Put CVS keywords into all the source files so that when the software
is deployed, there is no doubt what version was checked out and where
the files are stored in the local version control system.
* Put CVS keywords into all the source files so that when the software
is deployed, there is no doubt what version was checked out and where
the files are stored in the local version control system.
Pay closer attention to security. Use taint perl to ensure that
harmful input does not get used in unexpected ways. Keep the flow of
control clear and simple.
* Pay closer attention to security. Use taint Perl to ensure that
harmful input does not get used in unexpected ways. Keep the flow of
control clear and simple.
* Allow for use of any text browser which can display tables.
Currently this is true for the browser
links (http://artax.karlin.mff.cuni.cz/~mikulas/links/) but not the
browser lynx.
Allow for use of any text browser which can display tables
currently this is true for the browser links:
http://artax.karlin.mff.cuni.cz/~mikulas/links/.
but not the browser lynx.

452
mozilla/webtools/tinderbox2/Install
View File

@ -2,50 +2,50 @@
Tinderbox Documentation:
------------------------
*) Every directory has a ReadMe file which tells about all the files
in that directory.
* Every directory has a ReadMe file which tells about all the files
in that directory.
*) Every executable can be run with --help so that you can see what
the program does and what its arguments are. If you have trouble
running the file just look at the file in a text editor and find
the string "usage".
* Every executable can be run with --help so that you can see what
the program does and what its arguments are. If you have trouble
running the file just look at the file in a text editor and find
the string "usage".
*) The top of every file (both code and configuration) has a short
explanation of what this file does.
* The top of every file (both code and configuration) has a short
explanation of what this file does.
*) The code has lots of documentation so you can read the code if you
need more details about how something works.
* The code has lots of documentation so you can read the code if you
need more details about how something works.
To install:
-----------
NOTE: You will need to ensure that Perl is installed on your web server.
It would be a good idea to have these modules installed:
*) You will need to ensure that Perl is installed on your webserver.
It would be a good idea to have these modules installed:
Storable, Date::Format,
Storable, Date::Format,
You can easily check if these are installed by executing these two
commands.
You can easily check if these are installed by executing these
following commands. If the module is installed, you'll get no
output:
If the module is installed, you'll get no output:
perl -e 'use Storable'
perl -e 'use Date::Format'
perl -e 'use Storable'
perl -e 'use Date::Format'
*) Read the Policies and Overview documents found in this directory to
Read the Policies and Overview documents found in this directory to
help you get a feel for the scope of this installation.
*) The process id which receives and process the mail must be the
same id which runs the tinderbox cron job to prepare the web pages. I
prefer to manage my webserver so that all CGI scripts do not run as
the same user. Using one user id can cause security problems which
are hard to detect. If you must run all web applications as a single
user it would be safer to ensure that this user is a specific web user
(apache, cgiuser, webuser) so that this is not mixed up with other
unix system users (daemon, nobody, bin) since this could cause
security interactions with other programs which use these ids.
The process id which receives and process the mail must be the
same id which runs the Tinderbox cron job to prepare the web pages.
A good practice is to configure the web server so that all CGI scripts
do not run as the same user. Using one user id can cause security
problems which are hard to detect. If you must run all web applications
as a single user it would be safer to ensure that this user is a specific
web user (apache, cgiuser, webuser) so that this is not mixed up with other
unix system users (daemon, nobody, bin) since this could cause security
interactions with other programs which use these ids.
It may take some thought as to how the user id will be configured to
run when receiving mail and when receiving web requests and not be a
@ -55,224 +55,230 @@ These products will help partition your web application to run as
different users. (See http://www.w3.org/Security/Faq/wwwsf4.html for
more info)
CGIWrap
http://cgiwrap.unixtools.org/
http://sourceforge.net/projects/cgiwrap/
sbox: Put CGI Scripts in a Box
http://stein.cshl.org/software/sbox/
CGIWrap
http://cgiwrap.unixtools.org/
http://sourceforge.net/projects/cgiwrap/
sbox: Put CGI Scripts in a Box
http://stein.cshl.org/software/sbox/
The Apache Web server comes with its own wrapper script called suEXEC.
*) The src/default_conf directory contains perl libraries which are
The src/default_conf directory contains Perl libraries which are
specific to an individual users site configuration. No two users of
Tinderbox will have the identical sets of files.
Tinderbox2 will have the identical sets of files.
The files found in default_conf are sample files which are used at
Mozilla.org. Other users are expected to customize these libraries and
install the modified libraries in local_conf. Files found in
local_conf will be used before any file found in default_conf.
The distribution of Tinderbox will never place any files in the
The distribution of Tinderbox2 will never place any files in the
local_conf directory as this is assumed to be under local control.
A quick overview of the files:
TinderConfig.pm: general configuration settings
(HTML directories, Log files, implementations of libraries
to use, etc).
TreeData.pm: version control (CVS, Bonsai) configuration.
Error_Parse.pm: the regular expressions for identifying
errors in build logs.
BTData.pm: bug tracking configuration
FileStructure.pm: filesystem (storage) and global/per-project
settings. Most users will not have to change this.
TinderConfig.pm
general configuration settings: HTML directories, Log files,
implementations of libraries to use, etc
TreeData.pm
version control (CVS, Bonsai) configuration
Error_Parse.pm
the regular expressions for identifying errors in build logs.
BTData.pm
bug tracking configuration
FileStructure.pm
filesystem (storage) and global/per-project settings.
Most users will not have to change this.
In particular you will need a TinderConfig.pm and a TreeData.pm which
describes your local setup. You may need to make local versions of
the other files depending on how you wish to have Tinderbox configured.
I install Tinderbox via RPM. I have three RPMS. One tracks changes in
the Tinderbox server source code just as I get it from CVS. One RPM
contains my local_conf files. The clientbin files get put in their
own RPM which is installed on the buildmachine along with the
local_conf files. I do not currently have all the other configuration
details worked out (crontab files, /etc/rc.d/init.d, .cvspass,) but
most of the state of my machines is under RPM control.
1. Run ./configure. When configure is done you will have a Makefile
and a config.out. Although Tinderbox2 consists entirely of Perl
scripts we must substitute some values into the source code to make it
executable. You may wish to change the default directories in
configure for some of the Makefile variables. Please read config.out
and make any changes which need to be made for your system. Configure
also accepts command line options to change some default variables.
Please look at the configure source code for variable details, but the
most common changes are:
*) Run ./configure. When configure is done you will have a Makefile
and a config.out. Although Tinderbox consists entirely of perl
scripts we must substitute some values into the source code to make it
executable. You may wish to change the default directories in
configure for some of the makefile variables. Please read config.out
and make any changes which need to be made for your system. Configure
also accepts command line options to change some default variables.
Please look at the configure source code for variable details, but the
most common changes are:
--prefix
the directory for most of the tinderbox files.
Defaults to /home/tinderbox2
--prefix: the directory for most of the tinderbox files,
defaults to /home/tinderbox2
--cgibin-prefix
The directory where cgi scripts will be run.
This depends on how your web server is configured.
For security you may wish to ensure that this directory is
disjoint from the prefix directory.
Defaults to /var/www/cgi-bin/tinderbox.
--cgibin-prefix: The directory where cgi scripts will be
run. This depends on how your webserver is configured.
For security you may wish to ensure that this
directory is disjoint from the prefix directory. This
defaults to /var/www/cgi-bin/tinderbox.
--html-prefix
The directory where html files should be written.
This depends on how your web server is configured.
This is where the web server will pick up Tinderbox2 output.
Defaults to /home/httpd/html/tinderbox.
--html-prefix: The directory where html files should be
written. This depends on how your webserver is configured.
This is where the webserver will pick up tinderbox2
output. This defaults to /home/httpd/html/tinderbox.
2. Run 'make' to create executable versions of the source code in the
./build directory.
3. Run 'make compile_bin_code' to ensure that the code will compile
when it is installed. This will not work if your target machine is
very different from the machine you are running make on.
4. If you wish run the test programs as described in /build/test/ReadMe
to ensure that you have configured the program correctly. This will
require a TinderConfig.pm file in your local_conf directory.
See the ReadMe file in the test directory for detailed information
on how the tests work.
5. Run 'make install' to install Tinderbox2 on your system. You may
wish to run this with a different 'prefix'
make install --prefix=/opt/tinderbox
It is currently assumed that the bin directory of your installation
will be where the web server will run the Tinderbox2 cgi's from.
You can copy the *.cgi files to another directory if this is not
the case.
5. There are some gifs located in the gif directory which have
historically been used by Tinderbox2. The installation via
'make install' does not install these images. Put them
somewhere in your web server's html directory if you wish to
use them. Samples of their use are in the configuration files.
6. Set up a cron job to run the following every five minutes:
$cgi-bin/bin/tinder.cgi --daemon-mode
This generates the static tinderbox pages which users see.
Set up a cron job to run the following at least once a day:
$prefix/bin/rmlogs
Six am is a good time to run this as the machine load is usually
light and will avoids any Daylight Savings problems. This keeps
the archive of compressed build logs from growing without limit.
7. You may recieve data into the $prefix/bin/processmail* programs via
http post. Typically this means that these programs must be installed
in the $cgi-bin directory and have a name which ends in *.cgi. Instead
of mailling the logs to these programs, use the HTTPPost program to
post the data to the correct URL.
8. If using the mail system, set up the $prefix/bin/processmail*
programs to receive the incoming tinderbox mail. The process id which
receives and process the mail must be the the same id which runs the
Tinderbox2 cron job to prepare the web pages. Usually this set up is
accomplished by having the MTA (Sendmail) pass mail for particular
accounts into a script. This can be configured via a global
configuration file (Sendmail alias file) or via a .forward file (each
account gets the same user id but a different home directory, each
home directory gets a .forward to cause incoming mail to be delivered
through the correct tinderbox mail processing program).
The following configuration has been tested for the mail server
Postfix. The postfix aliases file /etc/postfix/aliases contains the
following lines. This file must be processed with a /usr/sbin/postalias
command which must be run by the same id which will run the Tinderbox2
cgi scripts.
tinderbox_builds: |/home/tinderbox2/bin/processmail_builds
tinderbox_bugzilla: |/home/tinderbox2/bin/processmail_bugs
If it is not possible to have mail delivery on your web machine and to
have your web machine deliver specific mail through a program then you
could you use fetchmail to simulate this delivery format.
Download fetchmail (from http://tuxedo.org/~esr/fetchmail) and install
it on your web server. Have the mail sent to any pop/imap mail server
which is accessible from the web server. Configure fetchmail to gather
the mail from the mail server and pass it through to the correct mail
processing program. You should keep the polling interval short
(30-200 seconds) as long polling intervals will cause delays in the
Tinderbox2 system and limit how quickly users see the new data. If
the polling interval is too short then you will overload your pop
server with needless queries. Fetchmail should run under the same
id as the Tinderbox2 server. The fetchmail configuration file will
look something like:
poll mail.provider.net with proto pop3:
user "tinderbox_builds" there has password "u can't krak this"
is tinder here and wants mda "$prefix/bin/processmail_builds"
A common mail address is 'tinderbox_builds' for build information
destined for the web server. Similarly the bug tracking system should
send mail to 'tinderbox_bugs'.
9. For CVS systems - make sure that when Tinderbox2 runs it has all the
permissions that it needs to access the Version Control database.
This usually means having an environment variable set when Tinderbox2
is run from cron. If you are using VC_CVS.pm then you will need to put
a ~/.cvspass for the Tinderbox2 server to use.
To create this file you will need to Log into the CVS repository
yourself, once for each tree you have defined. The login command must
use the hostname/modules exactly as you wrote it in VCData. Then copy
your ~/.cvspass into the Tinderbox2 server user id's home directory.
This must be the REAL home of the Tinderbox2 daemon, as listed in
/etc/passwd/ and set in the $HOME environmental variable for tinder.cgi.
The file must not be world readable or writable or executable. It is
recommended that VC_CVS.pm is only used as an interim step in setting up
VC_Bonsai. VC_Bonsai is not hard to setup and provided many useful
features which are absent in VC_CVS.pm.
10. Configure the build machines to mail their build logs (with Tinderbox2
variables on the top) to the web server machine. New builds must not
start earlier then 6 minutes after the last build started. Each build
machine mails the build log of each build and puts some build data at
the top of the log. Build information includes whether the build was
a success, which error parsers to use on the log file and what build
this is.
Run 'processmail_builds --help' to learn about how the mail
processing system works and what the mail messages are expected to
look like. There is an example mail in the test directory called
samplelog. The clientbin directory contains code which can be used on
the buildmachine.
See the README to help setup your build machine.
http://www.ntsendmail.com/ has a Perl program which can be used to
send mail on systems where there is no sendmail installed.
There is nothing special about sending mail. Any data transport layer
would work including HTTP Post and SSH the only requirement is that
the data is sent to the STDIN of the mail processing programs upon
delivery and that the process which runs this is the same uid as the
Tinderbox2 server.
Alternate delivery mechanisms have not been explored thoroughly but
they should not be too hard to setup. There is a set of instructions
for this in lwpcook.pod which suggest you will need to install
HTTP::Request::Common and LWP::UserAgent;
11. Set up the bug tracking system to send mail to 'tinderbox_bugs' on
the web server machine whenever a Bug ticket changes state. It is not
interesting to see which tickets are being worked on, so restrict
mailing to changes in ticket state not updates (edit) of a ticket.
12. Check that the time on your web server, your version control
machine, your bug tracking machine and your build machines are all in
sync. Check that if build mail bounces on any of the above
machines that it will be received by someone who can act on it.
*) run 'make' to create executable versions of the source code in the
./build directory.
Contact information for questions or updates or gripes is kept in the
Contact file.
*) run 'make compile_bin_code' to ensure that the code will compile
when it is installed. This will not work if your target machine is
very different from the machine you are running make on.
*) If you wish run the test programs as described in
/build/test/ReadMe to ensure that you have configured the program
correctly. This will require a TinderConfig.pm file in your local_conf
directory. See the ReadMe file in the test directory for detailed
information on how the tests work.
*) run make install, to install Tinderbox on your system. You may
wish to run this with a different 'prefix'
make install --prefix=/opt/tinderbox
It is currently assumed that the bin directory of your installation
will be where the webserver will run the tinderbox cgi's from. You
can copy the *.cgi files to another directory if this is not the case.
*) There are some gifs located in the gif directory which have
historically been used by tinderbox. The installation via 'make
install' does not install these images. Put them somewhere in your
webserver's html directory if you wish to use them. Samples of their
use are in the configuration files.
*) set up a cron job to run $cgi-bin/bin/tinder.cgi --daemon-mode
every five minutes. This generates the static tinderbox pages
which users see.
set up a cron job to run $prefix/bin/rmlogs at least once a day.
six am is a good time to run this as the machine load is usually light
and will avoids any day light savings problems. This keeps the
archive of compressed build logs from growing without limit.
*) you may recieve data into the $prefix/bin/processmail* programs via
http post. Typically this means that these programs must be installed
in the $cgi-bin directory and have a name which ends in *.cgi. Instead
of mailling the logs to these programs, use the HTTPPost program to
post the data to the correct URL.
*) If using the mail system, set up the $prefix/bin/processmail*
programs to receive the incoming tinderbox mail. The process id which
receives and process the mail must be the the same id which runs the
tinderbox cron job to prepare the web pages. Usually this set up is
accomplished by having the MTA (Sendmail) pass mail for particular
accounts into a script. This can be configured via a global
configuration file (Sendmail alias file) or via a .forward file (each
account gets the same user id but a different home directory, each
home directory gets a .forward to cause incoming mail to be delivered
through the correct tinderbox mail processing program).
I have used the following configurations for the mail server Postfix.
The postfix aliases file /etc/postfix/aliases contains the following
lines. This file must be processed with a /usr/sbin/postalias command
which must be run by the same id which will run the tinderbox cgi scripts.
tinderbox_builds: |/home/tinderbox2/bin/processmail_builds
tinderbox_bugzilla: |/home/tinderbox2/bin/processmail_bugs
If it is not possible to have mail delivery on your web machine and to
have your web machine deliver specific mail through a program then I
suggest you use fetchmail to simulate this delivery format. Download
fetchmail (from http://tuxedo.org/~esr/fetchmail) and install it on
your webserver. Have the mail sent to any pop/imap mail server which
is accessible from the webserver. Configure fetchmail to gather the
mail from the mail server and pass it through to the correct mail
processing program. You should keep the polling interval short (30-200
seconds) as long polling intervals will cause delays in the tinderbox
system and limit how quickly users see the new data. If the polling
interval is too short then you will overload your pop server with
needless queries. Fetchmail should run under the same id as the
tinderbox server. The fetchmail configuration file will look
something like:
poll mail.provider.net with proto pop3:
user "tinderbox_builds" there has password "u can't krak this"
is tinder here and wants mda "$prefix/bin/processmail_builds"
We use the mail address 'tinderbox_builds' for build information
destined for the webserver. Similarly the bug tracking system should
send mail to 'tinderbox_bugs'.
*) Make sure that when tinderbox runs it has all the permissions that
it needs to access the Version Control database. This usually means
having an environmental variable set when tinderbox is run from
cron. If you are using VC_CVS.pm then you will need to put a
~/.cvspass for the tinderbox server to use. To create this file you
will need to Log into the CVS repository yourself, once for each tree
you have defined. The login command must use the hostname/modules
exactly as you wrote it in VCData. Then copy your ~/.cvspass into the
tinderbox server user id's home directory. This must be the REAL home
of the Tinderbox daemon, as listed in /etc/passwd/ and set in the
$HOME environmental variable for tinder.cgi. The file must not be
world readable or writable or executable. It is recommended that
VC_CVS.pm is only used as an interim step in setting up
VC_Bonsai. VC_Bonsai is not hard to setup and provided many useful
features which are absent in VC_CVS.pm.
*) Setup the build machines to mail their build logs (with tinderbox
variables on the top) to the web server machine. New builds must not
start earlier then 6 minutes after the last build started. Each build
machine mails the build log of each build and puts some build data at
the top of the log. Build information includes whether the build was
a success, which error parsers to use on the log file and what build
this is. Run 'processmail_builds --help' to learn about how the mail
processing system works and what the mail messages are expected to
look like. There is an example mail in the test directory called
samplelog. The clientbin directory contains code which can be used on
the buildmachine. See the README to help setup your buildmachine.
http://www.ntsendmail.com/ has a perl program which can be used to
send mail on systems where there is no sendmail installed.
There is nothing special about sending mail. Any data transport layer
would work including HTTP Post and SSH the only requirement is that
the data is sent to the STDIN of the mail processing programs upon
delivery and that the process which runs this is the same uid as the
tinderbox server. I have not thought too much about how to set up
these alternate delivery mechanisms so I have no practical advice but
It should not be too hard to setup. There is a set of instructions for
this in lwpcook.pod which suggest you will need to install
HTTP::Request::Common and LWP::UserAgent;
*) Set up the bug tracking system to send mail to 'tinderbox_bugs' on
the webserver machine whenever a Bug ticket changes state. It is not
interesting to see which tickets are being worked on, so restrict
mailing to changes in ticket state not updates (edit) of a ticket.
*) Check that the time on your webserver, your version control
machine, your bug tracking machine and your build machines are all in
sync. Check that mail if build mail bounces on any of the above
machines that it will be received by someone who can act on it.
I keep my email up to date in the Contact file. If you have trouble
with these instructions please drop me a line.
Ken Estes.
Originally written by Ken Estes.
Updated by Mike Taylor.

229
mozilla/webtools/tinderbox2/Overview
View File

@ -1,157 +1,172 @@
Overview of the Tinderbox System
--------------------------------
Tinderbox is an information display system. It runs on a machine with
a webserver and will periodically write static HTML files to the disk
so that the webserver can serve these documents. Tinderbox is run out
of cron every five minutes. It gathers up information from various
databases including: CVS Logs, Bonsai, and Perforce. It will also
process mail which is sent to it. Mail is sent from Bug Ticketing
software and Build/Test Machines. All this information is combined to
produce the HTML pages.
Tinderbox2 is an information display system. It runs on a machine with
a web server and will periodically write static HTML files to the disk
so that the web server can serve these documents.
Tinderbox2 is run out of cron as often as your needs require and gathers
up information from various databases including: Repository Logs, Bonsai
and Perforce. It will also process mail which is sent to it from Bug
Ticketing software and Build/Test Machines. All this information is
combined to produce the HTML pages.
Since no two companies will structure their development processes the
same way, the tinderbox code has to be highly configurable to account
same way, the Tinderbox2 code has to be highly configurable to account
for most possible uses. There is a main configuration file which
allows most of the major user configurable variables to be set.
Novice users can expect to edit only this file and get a working
tinderbox system. Additionally each library has been broken into two
parts. One part is the library specific configurations. This file is
expected to need modifications in some installations. I have put all
the library configurations into one directory to make it easy to find
the parts of tinderbox which are easy to modify. Each configuration
library can be thought of as a table which might need to be edited or
extended for use at your company. I have provided a working system
but the defaults may not suit your needs. These tables can be easily
changed in small ways by simply looking at the file and making obvious
changes. I have also allowed for the possibility of making complex
changes that only a competent perl programmer could define. Changes
are not made to the files which I have provided. Rather the changes
are made to copies of the files which are stored in a local
Tinderbox2 system.
Additionally each library has been broken into two parts. One part is
the library specific configurations. This file is expected to need
modifications in some installations. All of the library configurations
are located in one directory to make it easy to find the parts of Tinderbox2
which are easy to modify. Each configuration library can be thought of
as a table which might need to be edited or extended for use at your company.
The defaults provide a working system but may not suit your needs. These
tables can be easily changed in small ways by simply looking at the file
and making obvious changes.
Changes are not made to the files which have been provided but rather
the changes are made to copies of the files which are stored in a local
configuration directory. This ensures that you can easily version the
Tinderbox code as it is provided to you from the official distribution
Tinderbox2 code as it is provided to you from the official distribution
and you can separately version the local configurations which you
make. It is also easy to see the local configurations since you have
both the original and the modified code on the same server and can
difference the two. As an example you might need to change the
BuildStatus - I assume that you have the following possible build
outcomes (Build in progress, Build failed, Build succeded but tests
failed, Build and all tests were successful) You may have additional
outcomes to specify which kind of tests failed (unit test failed, not
enough unit test coverage, performance tests failed). Similarly you
may have unusual requirements for how the filesystem should be laid
out. I provide a outcomes to specify which kind of tests failed (unit
test failed, not enough unit test coverage, performance tests failed).
Similarly you may have unusual requirements for how the filesystem
should be laid out. I suggest that you read through the files to see
how they are laid out and what types of changes are possible.
difference the two.
As an example you might need to change the BuildStatus - assuming that
you have the following possible build outcomes (Build in progress,
Build failed, Build succeded but tests failed, Build and all tests were
successful.) You may have additional outcomes to specify which kind of
tests failed (unit test failed, not enough unit test coverage, performance
tests failed). Similarly you may have unusual requirements for how the
filesystem should be laid out.
The build machines are not considered part of the tinderbox server.
They are clients just like Bug Ticketing systems and Version control
The build machines are not considered part of the Tinderbox2 server.
They are clients just like Bug Ticketing systems and Version Control
systems are clients. Build machines mail their build logs to the
server in a special format. This format specifies that name/value
pairs must appear at the top of the mail message followed by the
complete build log. Scripts for setting up a tinderbox build client
can be found in the clientbin directory but you may have other build
needs and may use any build methods you choose.
complete build log.
Scripts for setting up a Tinderbox2 build client can be found in the
clientbin directory but you may have other build needs and may use any
build methods you choose.
The central concept of the Tinderbox2 system is the notion of a 'Tree'.
The central concept of the Tinderbox system is the notion of a 'Tree'.
When several different groups are working out of the same version
control system often the files are partitioned into separate modules
with each group working on one or more disjoint modules. Over time
the developers need to branch their code because several different
versions of the files are under development at the same time. A tree
is a module/branch pair. This corresponds to a set of files which can
be checked out and built. Tinderbox makes one page for each tree and
displays what work is being done on that tree. CVS has a notion of
branches and of modules but not of trees. It is not possible to give
a branch/module pair a name. The tinderbox TreeData provides the
mappings between treenames and branch/module pairs. Tinderbox
displays the updates to bug tickets on the appropriate tree page.
This requires an easy mapping between bug tickets and trees. One
example of a complex function to determine tree name would be if each
of the product product types listed in the bug tracking data base
refers to one development project, except for a particular
feature/platform of one particular project which is being developed by
a separate group of developers. So the version control notion of
trees (a set of modules on a branch) may not have a direct map into
the bug tracking database at all times. In large projects it is
sometimes convenient to have a tree called 'ALL' which is used to
display all checkins performed on any trees and all bug tickets worked
on by any programmers. It is not possible to build or test the 'ALL'
tree and neither the version control nor bug ticketing system knows of
its existence.
is a module/branch pair and corresponds to a set of files which can
be checked out and built. Tinderbox2 creates one page for each tree and
displays what work is being done on that tree.
While most version control systems know about modules and branches, they
do not know about trees and it is not possible to give a branch/module pair
a name. The Tinderbox2 TreeData provides the mappings between treenames
and branch/module pairs.
Tinderbox2 displays the updates to bug tickets on the appropriate tree
page. This requires an easy mapping between bug tickets and a tree. One
example of a complex function to determine tree name would be if each
of the product types listed in the bug tracking database refers to one
development project, except for a particular feature/platform of one
particular project which is being developed by a separate group of
developers. So the version control notion of trees (a set of modules
on a branch) may not have a direct map into the bug tracking database
at all times.
In large projects it is sometimes convenient to have a tree called 'ALL'
which is used to display all checkins performed on any trees and all bug
tickets worked on by any programmers. It is not possible to build or
test the 'ALL' tree and neither the version control nor bug ticketing
system knows of its existence.
The Bug Tracking code was intended to be as general as possible.
Most bug ticketing systems send mail when tickets change state. The
mail is often of the same form. It is a name/value pair which the
separator being the string ": ". Tinderbox will parse mail of this
separator being the string ": ". Tinderbox2 will parse mail of this
form and display the interesting fields on the appropriate tree page.
The configuration of this module involves specifying which bug ticket
names are interesting and should be displayed. Also you will need to
specify how to map a bug ticket into a a tree. This could be very
simple if each bug ticket has a field which represents the tree it is
applicable to (in this case tree could equal project) or can be very
complex if the tree must be computed by the values of a set of fields.
Also tinderbox keeps track of which bugs are "reopened" and displays
names are interesting and should be displayed.
You will need to specify how to map a bug ticket into a tree, this
could be very simple if each bug ticket has a field which represents
the tree it is applicable to, in this case the tree would equal the
project. Or it can be very complex if the tree must be computed by the
values of a set of fields.
Also Tinderbox2 keeps track of which bugs are "reopened" and displays
them in a different column. The idea is that some bugs are moving
backwards and creating duplicate work. These tickets are particularly
troublesome and should be watched specially. So possible all ticket
troublesome and should be watched specially. So possibly all ticket
status are partitioned into "progress" or "slippage" categories. You
will need to specify what status values are possible for your ticket
system and you will also need to specify the set of columns which you
would like to see on the status page.
The heart of the tinderbox system is the 'status table'. This is an
The heart of the Tinderbox2 system is the 'status table'. This is an
HTML table which graphically shows how the changes made to the
development databases. It will show what is going on in the version
control system, the bug tracking system, the build system, automatic
regression tests and provide a notice board for developers to inform
each other of current news. By placing all this information in the
same table it is possible to correlate and cross check how different
types of changes effected each other and what was going on with the
whole project at different times in the day. The rows of the table
represent time with the most current events at the top of the page.
There are different sets of columns for each database which needs to
be displayed. The sets of columns are managed by independent modules.
each other of current news.
By placing all this information in the same table it is possible to
correlate and cross check how different types of changes effected each
other and what was going on with the whole project at different times
in the day.
The rows of the table represent time with the most current events at the
top of the page. There are different sets of columns for each database
which needs to be displayed. The sets of columns are managed by
independent modules.
There is one module for each version control system and each bug
tracking system which tinderbox knows how to interface with. It is
tracking system which Tinderbox2 knows how to interface with. It is
easy to port the system to new databases by just adding a new module
using the same style as the existing modules. Modules never share or
peek at each others data all combining of data is done by the humans
who stare at the table and interpret what is going on. The main
tinderbox system does not know how many columns the final table will
have. It only knows about a list of table modules. Each module in the
list is called in turn to generate the complete row then the entire
row is displayed. The user must configure tinderbox with the list of
modules which are of important to their own environment. There is no
restriction on the number of modules which may be configured, though
due to implementation details each module can only appear once in the
table. There are many pop up windows embedded in the status table
these will provide extra level of detail when a mouse is placed over
the link. By moving your mouse around the page you may effectively
drill down into an item of interest and learn more about it without
leaving the page. Most of the links will click through to the
appropriate database. Thus if you need more data about an item you
can click on the link and query the database directly.
peek at each others data and all combining of data is done by the humans
who stare at the table and interpret what is going on.
The main Tinderbox2 system does not know how many columns the final
table will have - it only knows about a list of table modules.
Each module in the list is called in turn to generate the complete row
then the entire row is displayed. The user must configure Tinderbox2 with
the list of modules which are of important to their own environment.
There is no restriction on the number of modules which may be configured,
though due to implementation details each module can only appear once
in the table.
There are many pop up windows embedded in the status table these will
provide extra level of detail when a mouse is placed over the link.
By moving your mouse around the page you may effectively drill down
into an item of interest and learn more about it without leaving the page.
Most of the links will click through to the appropriate database.
Thus if you need more data about an item you can click on the link and
query the database directly.
Besides the status table there is one other feature of the status
page. The page displays some information which is not correlated
page. The status page displays some information which is not correlated
through time and with other data. This information is called status
table headers. The main headers are the message of the day (MOTD),
and the Tree State though there are a few others headers of mainly
historical interest. The important issue with the headers is that
they are not optional. Tinderbox can render a table with as little or
as many columns in the status table as you wish but each of the
headers has a particular place on the status page and needs to be
rendered in a particular way (font size, font type, etc) thus the
tinderbox server must know where each header must go and how to
specify the appropriate html context for this header. Users may set
null defaults for headers that they do not need but it is much harder
for a user to add new headers to the code in a modular fashion.
table headers.
The main headers are the message of the day (MOTD), and the Tree State
though there are a few others headers of mainly historical interest.
The important issue with the headers is that they are not optional.
Tinderbox2 can render a table with as little or as many columns in the
status table as you wish but each of the headers has a particular place
on the status page and needs to be rendered in a particular way
(font size, font type, etc) thus the Tinderbox2 server must know where
each header must go and how to specify the appropriate HTML context for
the header. Users may set null defaults for headers that they do not
need but it is much harder for a user to add new headers to the code
in a modular fashion.

487
mozilla/webtools/tinderbox2/Policies
View File

@ -1,277 +1,296 @@
Preparations you will need to make and
policies you will need to set:
-----------------------------------
To install tinderbox you will need some information about your
To install Tinderbox2 you will need some information about your
existing computer systems and some idea about what your goals are.
Here is a list of questions to help get you started, some of these
Here is a list of topics to help get you started, some of these
ideas may not be appropriate for your environment.
Web server configuration
------------------------
The webserver will serve the tinderbox pages.
Webserver configuration is a bit of an art and you will need to
understand the policies which are used to administer your webserver.
The web server is what delivers the Tinderbox2 information to the world.
Web server configuration is a bit of an art and you will need to
understand the policies which are used to administer your web server.
*) You will need to decide the directory where tinderbox should write
the static HTML pages. This will depend on how your webserver is
configured. The default location is based on the RedHat 7.1
(apache-1.3.19-5) installation and is: /var/www/html/tinderbox2. You
will also need to know what the URL browsers will need to use to find
this directory. Since tinderbox generates static web pages, it is
possible to run tinderbox and not run a web server. One way this
could be done is if you have a network file system and all users have
browsers which can read from the HTML directories. In this case all
URL's should begin with "file:/" instead of the usual "http://".
* You will need to decide the directory where Tinderbox2 should write
the static HTML pages. This will depend on how your web server is
configured. The default location is based on the RedHat 7.1
(apache-1.3.19-5) installation and is: /var/www/html/tinderbox2.
*) Project level administration is done via cgi scripts. These
scripts allow administrators to set the message of the day, and the
state of the tree (open, closed, restricted). Also all users can post
notices to the web pages via a cgi script. CGI programs are often
restricted to a portion of the file system which is disjoint from the
HTML files. You will need to figure out where the CGI programs will
go. Tinderbox takes its defaults from RedHat 7.1 and uses:
/var/www/cgi-bin/tinderbox2. You will also need to know what the URL
browsers will need to use to find this directory.
You will also need to know what the URL browsers will need to use
to find this directory. Since Tinderbox2 generates static web
pages, it is possible to run Tinderbox2 and not run a web server.
One way this could be done is if you have a network file system
and all users have browsers which can read from the HTML directories.
In this case all URL's should begin with "file:/" instead of the
usual "http://".
*) CGI scripts will run as an unauthenticated user on your system.
You will need to decide which user will run the tinderbox CGI scripts.
The same user id must be used for running the scripts as for tinderbox
mail delivery. The Tinderbox Configuration files will define this
user id and as a security precaution check that it is running as the
required id. It is suggested that this id not be a privileged id
(higher ids are better, please make this number be greater than 10 and
bigger than 100 is recommended). Smaller ids are often assumed to
have more privileges on a Unix box then larger ids. It is not a good
idea for an unauthenticated user to have any privileges so a large id
is recommended. It is also recommended that you not use the id 'nobody'
as this id is over used and it would be better to partition the
unauthenticated user into separate ids in case of security problems.
RedHat runs all its CGI scripts as the user 'apache', this is an
acceptable user. I would prefer to have a separate user to run the
tinderbox CGI scripts but this would require recompiling apache to
enable suEXEC, and it is more effort then most groups can afford.
* Project level administration is done via cgi scripts. These scripts
allow administrators to set the message of the day, and the state
of the tree (open, closed, restricted). Also all users can post
notices to the web pages via a cgi script.
CGI programs are often restricted to a portion of the file system
which is disjoint from the HTML files. You will need to figure out
where the CGI programs will go. Tinderbox2 takes its defaults from
RedHat 7.1 and uses /var/www/cgi-bin/tinderbox2. You will also need
to know what the URL browsers will need to use to find this directory.
*) Tinderbox Files. There are other tinderbox files which need to be
placed on the webserver. These include libraries and non-cgi
programs. You will need to decide where to place these files. Most
users put them in /home/tinderbox2.
* CGI scripts will run as an un-authenticated user on your system.
You will need to decide which user will run the Tinderbox2 CGI
scripts. The same user id must be used for running the scripts as
for Tinderbox2 mail delivery.
*) Tinderbox Data. Tinderbox stores its data in the file system. For
security it is often a good idea to keep this data out of the HTML and
CGI directories so that malicious users can not directly access this
data. The compressed build logs can grow quite large, so it is
recommended to put the data on a file system with room. The default
is to put them in the directory /home/tinderbox2/data.
The Tinderbox2 configuration files will define this user id and,
as a security precaution, check that it is running as the required
id. It is suggested that this id not be a privileged id (higher
ids are better, please make this number be greater than 10 and
bigger than 100 is recommended). Smaller ids are often assumed to
have more privileges on a Unix box then larger ids. It is not a
good idea for an un-authenticated user to have any privileges so a
large id is recommended.
It is also recommended that you not use the id 'nobody' as this id
is over used and it would be better to partition the un-authenticated
user into separate ids in case of security problems.
RedHat runs all its CGI scripts as the user 'apache', this is an
acceptable user. I would prefer to have a separate user to run the
Tinderbox2 CGI scripts but this would require recompiling apache to
enable suEXEC, and it is more effort then most groups can afford.
* Tinderbox2 Files. There are other Tinderbox2 files which need to
be placed on the web server. These include libraries and non-cgi
programs. You will need to decide where to place these files.
Most users put them in /home/tinderbox2.
* Tinderbox2 Data. Tinderbox2 stores its data in the file system.
For security it is often a good idea to keep this data out of the
HTML and CGI directories so that malicious users can not directly
access this data. The compressed build logs can grow quite large,
so it is recommended to put the data on a file system with room.
The default is to put them in the directory /home/tinderbox2/data.
Mail
----
*) Many of the tinderbox modules (Bug Ticket, Build, CVS) receive
their data via mail. The mail system on you web server machine must
be configured to deliver the mail into the tinderbox mail processing
programs. You should spend some time understanding how your mail
delivery system can be configured to allow user mail to be delivered
into a program and how to set the user id under which this delivery
occurs. If you do not wish to configure your mail delivery program
then you can use fetchmail to pull the mail out of a mail box and push
it into the programs on a periodic basis. See the install page for
details on what I have learned about mailing systems.
Many of the Tinderbox2 modules (Bug Ticket, Build, CVS) receive their
data via mail.
* The mail system on your web server machine must be configured to
deliver the mail into the Tinderbox2 mail processing programs.
You should spend some time understanding how your mail delivery
system can be configured to allow user mail to be delivered into a
program and how to set the user id under which this delivery occurs.
If you do not wish to configure your mail delivery program then you
can use fetchmail to pull the mail out of a mail box and push it into
the programs on a periodic basis. See the install page for details
on different aspects about mailing systems.
Production Version Control
-------------------------
One of the biggest responsibilities which a "buildmaster" has is the
One of the biggest responsibilities which a "build master" has is the
requirement that all code should be reproducible. That is that at
any point in the future, even more than one year later, the current
binaries should be able to be rebuilt byte for byte from sources.
This requirement can be broken down as follows:
1) The build machine must be reproducible.
* The build machine must be reproducible.
We must be able to get back the same build machine we had at any point
in the past. This means that all OS libraries, all header files, all
compilers, all build tools (make, grep, sed) must have some mechanism
to roll back. It is common to use a backup of the build machine to
reconstruct it. Most OS will give you a list of the software packages
which are installed on the machine and their version numbers. I like
to keep the list of software packages which are installed on the
machine checked into version control. This allows me to compare the
state of the build machine at any two points in time. I have tools to
recreate the build-machine from just a list of packages with version
numbers. It is considered a best practice to limit the amount of
software which is available on the build machine. A build machine
with too much installed will only make it difficult to reproduce older
builds should the need arise. I recommend not installing any
web servers or graphical window managers on your build machine. It
should be clear that the build machine should not be the same machine
where the tinderbox server runs.
You must be able to get back the same build machine configuration
you had at any point in the past. This means that all OS libraries,
all header files, all compilers, all build tools (make, grep, sed)
must have some mechanism to roll back. It is common to use a backup
of the build machine to reconstruct it. Most OS will give you a
list of the software packages which are installed on the machine and
their version numbers. Maintaining a the list of software packages
which are installed on the machine and check it into version control.
This allows you to compare the state of the build machine at any two
points in time.
It is considered a best practice to limit the amount of software which
is available on the build machine. A build machine with too much
installed will only make it difficult to reproduce older builds should
the need arise. A build machine should not have installed any web
servers or graphical window managers on your build machine. It should
be clear that the build machine should not be the same machine where
the Tinderbox2 server runs.
2) The build process must be reproducible. That is all the steps
which are used to create the application must be reproducible.
* The build process must be reproducible. That is all the steps which
are used to create the application must be reproducible.
*) Build Interface: We must be able to run exactly the same build
process in the future including: all commands with command line
arguments, all environmental variables. I recommend that the entire
build process be viewed as something outside of the build master
control. Developers are responsible for ensuring that there is a
simple build master interface to construct all the software products
which go into a build. Typically there is a makefile in a standard
place where the buildmaster can run something like "make all; make
install;" and be guaranteed that this will build the product. The
build interface should be viewed as something which never changes and
are part of the build machine, like the OS and are changed only
rarely. It is hard enough to track all the parts of the build process
which we expect to change, we should not need to track complex build
procedures. The build procedures should have a standard interface.
By keeping the build instructions in one makefile which is checked
into the same version control system as the sources it is easy to
recreate any previous build even if the commands used to build the
software fluctuate rapidly between releases. There must be a simple
interface to construct the software which will hide all the complexity
of the actual construction.
* Build Interface - You must be able to run exactly the same build
process in the future including all commands with command line
arguments, all environmental variables. I recommend that the entire
build process be viewed as something outside of the build master
control. Developers are responsible for ensuring that there is a
simple build master interface to construct all the software products
which go into a build. Typically there is a Makefile in a standard
place where the build master can run something like "make all; make
install;" and be guaranteed that this will build the product.
*) Build Environment: The makefile will code all the build commands
and all the environmental variables (PATH, UMASK, LD_LIBRARY_PATH,
CLASSPATH) needed to build the software though it may rely on some
well defined command line arguments (PREFIX, CCFLAGS, JAVA_LIBS) to
make these prematurely. These command line arguments should not
change between versions of the software but should be a fixed set of
build parameters. The parameters may be needed to specify where some
files are found on the build machine (Ideally the build machine is set
up the same as developers machines so these directories can be
hard-coded into the makefiles but often there is a need for some
directories to be specified at build time) or where files are to be
created/installed on the build machine (typically a subdirectory of
/var/tmp but there may be several builds running at once and each will
need a different directory) or what kind of build is being created.
Each part of the build which needs a particular environmental variable
set or a special header file in some path should have tests which
ensure that the build environment is valid. I keep my build scripts
installed on the build machine and they are always started by running
/etc/rc.d/init.d/build start this ensures that I am not relying on any
build environmental variables which are set by logging into the build
account and are thus not tracked and versioned.
The build interface should be viewed as something which never changes
and are part of the build machine, like the OS and are changed only
rarely. It is hard enough to track all the parts of the build process
which are epected to change, let alone trying to track complex build
procedures.
*) Environmental safety issues:
The build procedures should have a standard interface. By keeping the
build instructions in one Makefile which is checked into the same
version control system as the sources; it is easy to recreate any
previous build even if the commands used to build the software
fluctuate rapidly between releases. There must be a simple interface
to construct the software which will hide all the complexity of the
actual construction.
If the build environment can not be used to build the software then a
human readable error message should be generated. My makefiles often
run various checks on the environmental variables before they
construct the code. They check that all required environmental
variables are set, that the required libraries are found, that
directories which must be disjoint (build and install directories) do
not overlap. This test suite becomes a build regression test and as I
discover additional possible build problems I add new tests to the
makefile. I make it a habit to explicit set all environmental
variables so that there is no doubt as to their expected values. It
is important for the QA group to only use Builds which were created by
an automated process so that we are sure that there are no
undocumented steps in either the test builds or the released build.
* Build Environment - The Makefile will code all the build commands
and all the environmental variables (PATH, UMASK, LD_LIBRARY_PATH,
CLASSPATH) needed to build the software though it may rely on some
well defined command line arguments (PREFIX, CCFLAGS, JAVA_LIBS) to
make these prematurely. These command line arguments should not
change between versions of the software but should be a fixed set of
build parameters.
3) Track the Build numbers. Given a clean install of your product you
should have all the information necessary to reproduce the executable
from sources. If a customer shows you the application binaries you
must be able to get the source code which build the application,
reconstruct the build machine which created the application and
possibly rerun the build exactly the same way as the application was
created before, this may include making some minor source code changes
before the build is run. I like to keep a file which contains:
The parameters may be needed to specify where some files are found
on the build machine (Ideally the build machine is set up the same
as developers machines so these directories can be hard-coded into
the Makefile but often there is a need for some directories to be
specified at build time) or where files are to be created/installed
on the build machine (typically a subdirectory of /var/tmp but there
may be several builds running at once and each will need a different
directory) or what kind of build is being created.
The product release name
Each part of the build which needs a particular environmental variable
set or a special header file in some path should have tests which
ensure that the build environment is valid. The build scripts could be
installed on the build machine and started by running:
"/etc/rc.d/init.d/build start". This would ensure that the build script
does not rely on any build environmental variables which are set by
logging into the build account and are thus not tracked and versioned.
The sources 'as of date'. (I always checkout my sources using
cvs -D 'date time' so that exactly the same sources
can be recovered knowing only the 'data time' which
was used to check them out. I am sure a similar trick
could be used with a perforce 'change set number'.)
* Environmental safety issues. If the build environment can not be used
to build the software then a human readable error message should be
generated. The Makefile can often run various checks on the environmen
variables before they construct the code. They check that all required
environment variables are set, that the required libraries are found,
that directories which must be disjoint (build and install directories)
do not overlap.
The branch name.
This test suite becomes a build regression test and as additional possible
build problems are discovered, new tests can be added to the Makefile.
It is a good habit to explicit set all environment variables so that there
is no doubt as to their expected values. It is important for the QA group
to only use Builds which were created by an automated process so that you
are sure that there are no undocumented steps in either the test builds or
the released build.
The module name.
* Track the Build numbers. Given a clean install of your product you
should have all the information necessary to reproduce the executable
from sources. If a customer shows you the application binaries you
must be able to get the source code which build the application,
reconstruct the build machine which created the application and
possibly rerun the build exactly the same way as the application was
created before, this may include making some minor source code changes
before the build is run.
One method is to keep a file which contains:
- The product release name
- The sources 'as of date'. (Always checkout sources using
a known datetime or revision # (cvs -D or svn -R) so that
exactly the same sources can be recovered knowing only the
'date/time' which was used to check them out.
- The branch name
- The module name.
This can be stored as a file in the product (encrypted if necessary)
or may be stored in some secure build master database where the data
can be looked up by release name. My preference is to keep all data
necessary to reproduce a build in the build output and delivered as
part of the product. This means that I can generate as many builds as
I want automatically and not need to keep track of any of them. When
the QA team deems that a certain build is 'important', by making a
particular build the official released copy then I can take a look at
its contents and tag/branch the code at the sources which I used to
build it.
This can be stored as a file in the product (encrypted if necessary)
or may be stored in some secure build master database where the data
can be looked up by release name. A good practice is to keep all data
necessary to reproduce a build in the build output and delivered as
part of the product. This means that you can generate as many builds as
needed automatically and not need to keep track of any of them. When
the QA team deems that a certain build is 'important', by making a
particular build the official released copy then you can take a look at
its contents and tag/branch the code at the sources which were used to
build it.
4) Build Prefix: It is a good idea to familiarize yourself with the
makefile conventions regarding the make variable PREFIX. It is
easiest to understand if you think about what RedHat does when they
build their distribution of RPM's but this will apply in many
different systems including the Andrew File System (AFS) and most
packaging systems. This variable is used during the build process
"make all PREFIX=/home/apache" to tell the package where it will be
installed (examples include /usr, /usr/local, /home/apache). I
suggest reading a few RedHat Spec files to see how this works in
practice. The application may need to hard-code this value into its
object code. When the application is installed it must not be
installed into its proper place on the build machine. The package we
are constructing could cause the build machine to stop working
correctly if it is a buggy version of a system library or major OS
application. Instead the makefile will install "make install
PREFIX=/var/tmp/build-root/home/apache" the package into some other
directory with a similar tree structure to its final destination. The
packaging system will then move the files into the correct place
during an installation step on the target machine. The installation
step only moves files and sets permissions. The makefile is not
supposed to use the installation directories to hard code values into
the application since the application will never be run from this
installation directory. The hard part of the build including any
PREFIX magic is in the build section. Notice the clear separation
between build machine / target machine and installation on the build
machine and installation on the target machine and construction of the
application binaries and installation of the application binaries.
This is one of the reasons why building an application on a build
machine is different from the way in which developers build their code
on their personal development machines. This PREFIX issue will arise
when you try and build the Tinderbox system and also when you
construct the makefiles for your own application. Since the build
machine is not the target machine it can not be assumed that files
will always be in the same places on both (for example perl).
* Build Prefix: It is a good idea to familiarize yourself with the
Makefile conventions regarding the make variable PREFIX. It is
easiest to understand if you think about what RedHat does when they
build their distribution of RPM's but this will apply in many
different systems including the Andrew File System (AFS) and most
packaging systems.
5) Application Architecture:
This variable is used during the build process
"make all PREFIX=/home/apache" to tell the package where it will be
installed (examples include /usr, /usr/local, /home/apache). Reading
a few RedHat Spec files to see how this works in practice. The
application may need to hard-code this value into its object code.
When the application is installed it must not be installed into its
proper place on the build machine. The package that is being
constructed could cause the build machine to stop working
correctly if it is a buggy version of a system library or major OS
application. Instead the Makefile will install "make install
PREFIX=/var/tmp/build-root/home/apache" the package into some other
directory with a similar tree structure to its final destination.
*) The build process should mimic the architecture of the code. It should
be a final test that the code was coded to the same specifications
that it was designed. It is a common problem for code to turn into
spaghetti with each piece of code using functions and creating
dependencies on every other piece of code. For example it is probably
a mistake for code in the database abstraction layer to be implemented
in terms of code in the HTML generation layer. These two libraries
should probably be independent of each other, though they both might
depend on a common string library. The code architecture should limit
the dependency graph between code modules. The BuildMaster must
enforce the restrictions on information flow between components. Thus
no libraries should be in the path unless the architecture allows this
module to depend on those libraries.
The packaging system will then move the files into the correct place
during an installation step on the target machine. The installation
step only moves files and sets permissions. The Makefile is not
supposed to use the installation directories to hard code values into
the application since the application will never be run from this
installation directory.
*) The architecture must not have circular dependencies. Circular
dependencies not only make upgrading individual libraries difficult
but also make testing components nearly impossible. That is it should
be possible to build some set of libraries L0 which depend on no
libraries and then build some other set of libraries L1 which depend
only on L0 libraries then build L2 which depend only on the L0 and L1
libraries. This "build chain" will prevent circular dependencies and
help keep your code testable and the dependencies understandable.
More information about why this is a good practice is available in
"Large-Scale C++ Software Design" (Addison-Wesley Professional
Computing Series) by John Lakos
The hard part of the build, including any PREFIX magic, is in the
build section. Notice the clear separation between build/target
machine, installation on the build machine, installation on the target
machine, construction of the application binaries and installation of
the application binaries.
*) I enforce the convention that developers are not allowed to overload
standard system libraries. I always put standard libraries in the
path before any library our company develops. I build the application
in stages to ensure that parts of the application which are not
intended to depend on other code will not have other header files on
the build machine at the time that they are constructed. Build
dependencies between modules which are expected are explicitly
controlled with build scripts and version numbers.
This is one of the reasons why building an application on a build
machine is different from the way in which developers build their code
on their personal development machines. This PREFIX issue will arise
when you try and build the Tinderbox2 system and also when you
construct the Makefiles for your own application. Since the build
machine is not the target machine it can not be assumed that files
will always be in the same places on both (for example Perl.)
* Application Architecture. The build process should mimic the
architecture of the code. It should be a final test that the code
was coded to the same specifications that it was designed. It is a
common problem for code to turn into spaghetti with each piece of
code using functions and creating dependencies on every other piece
of code.
For example it is probably a mistake for code in the database
abstraction layer to be implemented in terms of code in the HTML
generation layer. These two libraries should probably be independent
of each other, though they both might depend on a common string library.
The code architecture should limit the dependency graph between code
modules. The Build Master must enforce the restrictions on information
flow between components. Thus no libraries should be in the path unless
the architecture allows this module to depend on those libraries.
The architecture must not have circular dependencies. Circular
dependencies not only make upgrading individual libraries difficult
but also make testing components nearly impossible. That is it should
be possible to build some set of libraries L0 which depend on no
libraries and then build some other set of libraries L1 which depend
only on L0 libraries then build L2 which depend only on the L0 and L1
libraries. This "build chain" will prevent circular dependencies and
help keep your code testable and the dependencies understandable.
More information about why this is a good practice is available in
"Large-Scale C++ Software Design" (Addison-Wesley Professional
Computing Series) by John Lakos
You should enforce the convention that developers are not allowed to
overload standard system libraries. Always put standard libraries
in the path before any library our company develops. Build the
application in stages to ensure that parts of the application which
are not intended to depend on other code will not have other header
files on the build machine at the time that they are constructed.
Build dependencies between modules which are expected are explicitly
controlled with build scripts and version numbers.

163
mozilla/webtools/tinderbox2/ReleaseNotes Normal file
View File

@ -0,0 +1,163 @@
Release ???
New Time columns for use in environments with multiple time zones.
Spelling fixes
Add support for popup windows which work "on click"
Add support for users who want to see tinderbox's notion of what was
checked in not just Bonsai's query.
Add support for historical Tinderbox images
(flames, stars)
Mozilla layout preferences are now standard
(non uniform row spacing,
truly empty cells,
legend on bottom of page).
Allow users to backdate notices
automate regeneration of HTML when user uses CGI interface
Put the C in all build cells.
VC code did the wrong thing when generating old pages and there was
no TreeState data.
Fix Spelling mistakes throughout code.
Add support for build machines to tell the server to add text/links
into build cells (TinderboxPrint).
When builds are started too frequently (more then once every five minutes)
newest build is now kept and oldest build is thrown away.
New Configure script contributed by Stephen Lamm (slamm@yahoo-inc.com)
This includes a fix to allow Perl to be in different locations on the
machine where make is run and on the machine where the web server runs.
Now it is possible to include the time column on the left and right
side of the status table.
There is now a working Perforce module for those who use this product
instead of CVS.
There is a module to interface with the req ticketing system.
New Summary format and number of errors displayed on build page by
Adam Di Carlo <adam@onshore-devel.com>,
Spelling and consistency fixes from:
Adam Di Carlo <adam@onshore-devel.com>,
Orion Poplawski <OPoplawski@cqg.com>,
John L. Villanlovos <john.l.villalovos@intel.com>,
Change HTML generation so that the text browser 'links'
(http://artax.karlin.mff.cuni.cz/~mikulas/links/) can
work with tinderbox.
Allow the build script to use a single process and alternate builds
of different types.
Add a sample /etc/rc.d/init.d file for the RedHat Build machines.
Mail processing programs were not writing error files correctly.
This has been broken since I separated out the mail processing module.
Add script to cull old builds from a directory of completed builds.
Let the VC and BT headers be configurable. Too many companies find
this an important feature.
Allow Tinderbox to run via suid wrappers. This is needed for some
times of mail delivery systems
No longer exclude 'privileged id's from running Tinderbox.
Now we require tinderbox to run as the id which the user specifies
is the tinderbox id.
Better errors for failures in Storable.pm
BasicTxtHeader.pm had a bug which prevented it from working when using Storable
Rework the BT modules to allow use when URL is not easily mapped from the bugid
Begin work to allow tinderbox admin pages to open and close bonsai trees
Fix code which allowed tinderbox to determine the state of bonsai trees
Major bug in the untainting code, fixed. This was a security
problem. I suspect people may have trouble with the fix but this was
how the code was intended to work. I do not understand how this got
through my unit tests.
Give a useful example of how use the new build script.
Improve the spec files, split out build machine packages.
Release 0.9
Add full Bonsai integration functionality.
Allow users to define a display order for the buildnames on a per tree basis.
This adds a new function into TreeData.pm check your local files.
Fix computation of 'deadtime' for eazel.com's build setup,
this will improve their estimate of when the next build will be
complete.
Develop portable client build scripts.
Add RPM spec files.
Relax check of root id for all redhat users.
uid nobody is 99 and by default redhat apache runs all cgi as nobody.
VC_CVS.pm was not removing excess treestates from the Database correctly,
now fixed.
Release 0.8
This release moved some files into new directories update with
cvs update -d -P
Remove the strict time checking on mail, it caused too many problems.
Fix the Unmonitored Builds in admintree.cgi and in status.html page.
Storable.pm has been tested as a Persistence implementation.
Improve the prediction of when builds will end by introducing the
notion of 'deadtime'.
Separate out the BuildStatus table for easier configuration of the
status without changing the build display library.
Better comments in the ReadMe files
Better security:
- All scripts run in taint perl mode
- Separation of HTML and DATA directories in
addition to the CGI directory
- Additional checks of ownership for the data directories
- Check the network address of all administrative commands
Improved Makefile/Configure support
New permissions help prevent accidental edit of files in the
build or install directory
Inter-positioning of libraries, for local configuration,
fully supported
Test arguments added to Makefile
Test compiles of the perl are no longer required

9
mozilla/webtools/tinderbox2/Todo
View File

@ -1,3 +1,12 @@
Bear's Todo List:
* Integrate SVN for blame column - bug 303333
* Allow Tinderbox2 to use a Bonsai on another computer - bug 237289
* Bring perl code up to some of the newer Mozilla styles
* Create more export streams - rdf, atom feed, pub/sub???
Original Todo list:
It would be really nice if CVS/Bonsai could mail tinderbox the updates
which apply to the branches of interest, then tinderbox would not have