build

package make produced a load of compiler diagnostics. Should I be concerned?
Check the make errors link for a list of known make failures by architecture. The make failures and other hard errors should be limited to 3rd party (e.g., X11) library mismatches. We have tried to reduce the warnings across all architectures, but in some cases, like signed/unsigned warnings, we stopped short of casting every blasted statement just to please a particular compiler implementation. In general, if the regression tests pass, don't worry. How's that for a plan?
Is there an easy way to build debug (cc -g) binaries?
You can build -g binaries for debugging in the default INSTALLROOT by: 
package make debug=1
You can build -g binaries for debugging in a separate INSTALLROOT ($(package)-debug here) by: 
package make debug=1 HOSTTYPE=$(package)-debug
Is package make the only way to build?
You can build individual components by first setting up the environment (PACKAGEROOT INSTALLROOT PATH VPATH CDPATH): 
bin/package use
and then running nmake(1) or mamake(1) to rebuild the component: 
cd libast
nmake [ -n ] [ -d1 ] [ target ... ]
 or
mamake [ -n ] [ -d1 ] [ target ... ]
Remember to use a copy of nmake when building or installing nmake, and to export COSHELL=/bin/sh when rebuilding ksh; package make does this by copying selected programs to the $INSTALLROOT/bin/ok directory and running with PATH=$INSTALLROOT/bin/ok:$PATH.
Why is there an $INSTALLROOT/bin/cc command?
Some C compilers process different dialects or generate different binary formats based on command line options. The defaults are not always chosen wisely. Worse, vendors have a bad habit of changing the defaults between releases. This can lead to binary incompatibilities between hosts from the same vendor within a local network. So on some systems a cc script is generated to nail the compiler down. The script also takes the opportunity to cut out some of the noise from some compilers.
Can multiple cc(1)s or multiple HOSTTYPEs per cc(1) exist under one $PACKAGEROOT?
It is important for an $INSTALLROOT to present a consistent set of binaries and objects. For binary formats other than the default produced by cc(1) or for alternate cc(1) implementations an $INSTALLROOT/bin/cc command must be provided. This will ensure that all binaries produced under $INSTALLROOT are compatible:

  1. invent a HOSTTYPE name for the binary format; ideally it should use the default HOSTTYPE listed by package as a prefix 
    HOSTTYPE=hosttype-alternative

  2. create the architecture-specific INSTALLROOT and bin directories 
    mkdir -p $PACKAGEROOT/arch/$HOSTTYPE/bin

  3. create a cc script that coaxes the real cc to build $HOSTTYPE binaries; this script must contain a line matching ^HOSTTYPE=hosttype-alternative$ and ideally a -dumpmachine option intercept: 
    cat > $PACKAGEROOT/arch/$HOSTTYPE/bin/cc <<!
: $HOSTTYPE cc wrapper : `date +%Y-%m-%d` :
HOSTTYPE=$HOSTTYPE
case $* in
-dumpmachine) echo $HOSTTYPE; exit ;;
esac
full-path-to-the-real-cc  HOSTTYPE-specific-option(s) "$@"
!
chmod +x $PACKAGEROOT/arch/$HOSTTYPE/bin/cc

  4. export HOSTTYPE=hosttype-alternative or add a HOSTTYPE=hosttype-alternative argument to the package(1) command to build or use hosttype-alternative binaries 
    package make HOSTTYPE=hosttype-alternative
package use HOSTTYPE=hosttype-alternative
Is there an easy way to build the binaries on all my hosts?
  1. select one host to be the master -- this is where the multi-host build script will run from
  2. select a package root directory; this will be referred to as packagedir, for example, packagedir=ast; each host may have its own packagedir
  3. run the following to set up the master package source 
    cd packagedir
mkdir -p bin lib/package/tgz lib/package/admin
# copy the download package script to bin
# copy download source .tgz to lib/package/tgz
bin/package read

  4. create the file lib/package/admin/admin.db with one line for each host containing: 
    hosttype	hostname	packageroot
    where hosttype is either the host architecture type as displayed by the package command (with no arguments) or a host architecture type invented by you, hostname is the host name, and packageroot is one of the following forms:
    • ssh::packagedir if hostname:packagedir is mounted on the same network filesystem as $HOME/packagedir (hostname:packagedir and $HOME/packagedir are always in sync)
    • ssh:packagedir if hostname:packagedir is mounted on a different filesystem than $HOME/packagedir (hostname:packagedir requires a remote copy to be in sync with $HOME/packagedir)
    • ssh:otherhostname:packagedir if otherhostname has its own admin.db entry specifying remote sync and hostname:packagedir is mounted on the same network filesystem as otherhostname:packagedir (hostname:packagedir and otherhostname:packagedir are always in sync)
    NOTES:
    • a hosttype and/or hostname may appear more than once in admin.db; the only restriction is that every hostname:hosttype pair must be unique
    • at minimum the INIT and ast-base packages must be manually installed (once) on hosts requiring remote sync in order for the remote syncs to work
    • the minimal amount of remote copies are done to sync the remote hosts with the master source host

  5. run 
    package admin make
    to sync source and build and 
    package admin results [ on hostname ]
    to check results

  6. the builds are done concurrently; each host build is done by ssh(1)ing to the host, sourcing the .profile, sourcing packagedir/lib/package/admin/admin.env if it exists, and then running package make

Our master source admin.db currently controls 31 architectures. On build-from-scratch day the amazing dr.ek (with a different admin.db) has a large screen displaying a dozen or so concurrent builds -- its a thing of beauty when it all comes together.

The package command generates the wrong HOSTTYPE.
HOSTTYPE is used to name the target architecture for binaries generated by the C compiler. The package command determines HOSTTYPE by first checking the binaries that $CC (default cc(1)) generates. This may be different than the type reported by uname(1) or the local OS. This is the how package handles cross-compilation.
How do I cross compile a source package?
See cross compilation.
I followed all of the source installation instructions and it still crashed and burned; what now?
  1. Make sure $PATH points to cc and ar for the local host architecture.
  2. If you see core dumps from $SHELL then get into a different shell, export SHELL, and try package make again. We have seen PDKSH core dump in the package script.
  3. Look at the output of package results to see if it hints at the problem.
  4. To dig deeper, look at the unfiltered output of package verbose results, or edit the file named by package results path.
  5. We have seen NFS problems (surprise) on our local network when NFS-2 and NFS-3 are mixed between different vendors, or when we fire off package make on a half dozen hosts on the same $PACKAGEROOT. To rule this out, run package make from a host that has $PACKAGEROOT in a native filesystem. One workaround that works on our sol7.i386 and unixware.i386 hosts is to keep $PACKAGEROOT as a master cross mount on all hosts, and to make $INSTALLROOT a symlink to a local filesystem on the hosts that have NFS trouble.
I made the binaries; how do I distribute them to other hosts?
Generate a binary package archive by: 
cd $PACKAGEROOT/lib/package
package write binary
Then read the archive in the new place: 
cd new-INSTALLROOT
pax -m -rvf package.YYYY-MM-DD.NNNN.hosttype.tgz
If the installation directory is for binaries of one type then you can chop off the arch/hosttype/ path prefix with this pax(1) option: 
-s ',arch/[^/]*/,,'
How long does it take to build the ast-open source package?
The fastest build time is ~5 minutes on a 3Ghz linux.i386, on a local filesystem. The slowest times are ~1 hour on older sparcs, on an NFS filesystem. When everything is up-to-date it should take less than 3 minutes, even on NFS.
foo didn't build because the bar headers and libraries were not found; what now?
First, make sure that the libraries and headers are installed on your systems. Sometimes only a runtime environment is installed for a particular package (as opposed to a development environment). If your system really has the libraries and headers, here is a quick workaround: 
export PACKAGE_bar=bar-root-dir
where bar-root-dir contains an include and lib subdirectory or 
export PACKAGE_bar_include=bar-include-dir
export PACKAGE_bar_lib=bar-lib-dir
if the include and lib directories do not share a common root. For a more permanent workaround, use 
:PACKAGE: foo
in the nmake makefile and refer to $INSTALLROOT/lib/make/pkg-X11.mk as a template for a new $INSTALLROOT/lib/make/pkg-foo.mk.
I installed the sol8.* binaries and all commands fail at runtime with _libc_register_forkhandler diagnostics.
This is a sun backwards compatibility bug between /lib/libc.so and /lib/libnsl.a. Somewhere between 5.8 Generic_108528-11 and 5.8 Generic_108528-14 sun dropped the _libc_register_forkhandler symbol from libc.so; a bad move since any executable linked against the old libnsl.a needs this symbol at runtime because of hidden references, outside of user control, in /lib/libnsl.a[mt_misc.o]. The fix is to force a dynamic link for nsl by changing +lnsl to -lnsl in the ksh93 Makefile. This has been done since the 2002-09-15 release.
I installed the ast version of libz.so in the system standard directory and a bunch of commands began to fail.
The ast libz.so handles 64 bit file sizes and offsets. The first attempt at this introduced binary incompatibilities; these have been fixed since the 2002-09-15 release.
The build on our sun4 museum piece failed with a bunch of errors.
This may be related to a /bin/sh here document or temp file bug. To bootstrap the source build you can download the latest standalone ksh binary, or try SHELL=path-to-bash, or resurrect an earlier sunos /bin/sh.

download

I downloaded the foo.bar binary package and didn't find any source. Where do I get the foo.bar source?
You need to download the corresponding source package. Source packages contain generic source that builds on all architectures.
Why don't you post binaries for the foo.bar architecture?
With few exceptions we only post binary packages for architectures that we have direct login access to. If your foo.bar isn't on our list we'll be happy to assist you in getting the source package built on it.
Why don't you have an ftp download site like everyone else?
Even though the packages are licensed as Open Source, our legal department requires that each download recipient provably accept the associated license(s) before downloading. See Specht vs. Netscape for details on manifestation of agreement with respect to software downloads. HTTP provides the hooks we need for the manifestation of agreement, FTP does not.

We mainly need to protect our pockets from liability claims based on "what license?". Since the packages are Open Source, a 3rd party site, not affiliated with the original providers, can accept the license on our site, download the packages, and then post the packages on their site, still under the original license, but using whatever form of license acceptance they are comfortable with -- at that point we are out of the liability loop.

gunzip says the archive files I downloaded are corrupt.
The archive files are given the application/octet-stream mime type on the server side, but sometimes text translation is still applied. Compare the archive size with the size posted in the download selection list. If your copy is a few hundred bytes larger then your browser probably did a text transfer (with \n => \r\n translation) instead of a binary transfer. Make sure the browser download does a binary transfer.

Some browsers automatically apply gunzip, so check if the file is already uncompressed. This has happened on some .gz standalone package downloads.

licensing

Is the software ``Open Source?''
Yes, it is OSI Certified Open Source Software, released under the .

You didn't give me a name and password to download the software.
Everyone gets the same name and password, listed at the bottom of this copy of the license.
Why are the downloads password protected? GNU doesn't do that.
Its all about deep pockets. Even though the packages are licensed as Open Source, our legal department requires that each download recipient provably accept the associated license(s) before downloading. This is for protection from liability claims based on ignorance of the license. See Specht vs. Netscape for details on manifestation of agreement with respect to software downloads. We could have done a mouse click to agree interaction, but that would have encumbered wget(1) and curl(1) style command line access.
This name/password nonsense will never catch on with my user base.
Since the software is licensed by you are free to accept the license on our site, download the software, and then post it on your site with whatever manifestation of agreement suits you. At that point the liability is between you and your user base.

nmake

Is nmake(1) really the greatest thing since sliced bread?
Well, basing the entire build and packaging process on the AT&T nmake does put us way out on a lonely limb. But the benefits far outweigh the initial learning curve:

  • no need for autoconf, autheader, automake, libtool, or configure makefile generators and support tools
  • no need for make depend or make config; with nmake(1) and iffe(1) all dependencies are brought up to date each time nmake is executed
  • source and binary manifests are a byproduct of the build process
  • maintaining separate source and binary hierarchies requires no makefile change
  • :PACKAGE: assertions place a uniform view on top of a chaotic namespace
  • :MAKE: assertions dynamically determine recursive make order
  • new components (directories controlled by makefiles) can be added and will be built in the proper order

See nmake for more details.

Is there any online documentation?
Yes.
Isn't recursive make considered harmful?
That is a good paper and we agree with the conclusions for the given thesis. Those conclusions, however, are based upon the limitations of /bin/make(1) and its clones. Substitute a different make implementation and the same conclusions can be dead wrong. With nmake(1) and the ast build hierarchy, new component source directories, controlled by an nmake makefile, can be dropped anywhere under $INSTALLROOT/src, and the next package make or nmake will make everything, including the new component directories, in the proper order. This is an essential part of the ast build process, and its how we maintain sanity across packages, releases, and architectures. So our response is: Recursive nmake considered essential.
Is there an easy way to build debug (cc -g) binaries?
You can build -g binaries for debugging by adding --debug-symbols in Makeargs or by adding it to the nmake command line: 
nmake --debug-symbols
Static libraries will be installed with -g appended to the base name, e.g., libast-g.a. In addition, -llib and +llib references will bind to the static lib-g version if possible. Note that --debug-symbols replaces all binaries for the current component with -g versions. You can build -g binaries in a subdirectory by: 
mkdir cc-g
nmake cc- # or cc-g or install cc-g
This is done for most of the library components on the master ast build host so that debugging versions are always available. Use --nolib-type to disable the current component library renaming.
Are there any other nmake(1) debugging tips?
To generate the preprocessed output of foo.c in foo.i: 
nmake -n -g debug.mk foo.i
To generate the nested include structure of foo.c in foo.inc: 
nmake -n -g debug.mk foo.inc
To see the binding for a specific file or variable use the interactive interface (make> is the interactive prompt, lines wrapped for readability): 
nmake -n query
make> bind sys/types.h
make> sys/types.h

/usr/include/sys/types.h==sys/types.h : [2003-03-06+09:48:05.687128] \
	.LCL.INCLUDE .STD.INCLUDE .SCAN.c dontcare global regular

()sys/types.h : [2003-03-06+09:48:05.687128] .LCL.INCLUDE .STD.INCLUDE \
	.SCAN.c event=[2004-12-15+11:22:05.450622] compiled state
 prerequisites: sys/nodemask.h sys/cpumask.h sys/bsd_types.h sys/pthread.h \
	sgidefs.h standards.h 

make> print $(CC.HOSTTYPE)
sgi.mips3
make> q
Any makefile statement can be entered and >>>>> is the secondary prompt for statements that span more than one line. Action blocks are terminated by a blank line. The example shows that sys/types.h binds to /usr/include/sys/types.h, the file time stamp is 2003-03-06+09:48:05.687128, the time that nmake noticed that the file time changed (the event time) was 2004-12-15+11:22:05.450622, and sys/types.h includes the files listed after prerequisites:.
Why does nmake(1) provide its own cpp(1)?
nmake generates -I options based on the source and headers visible in its view (i.e., visible in $VPATH), so it is important that the header bindings be exactly specified to the compiler. The #include search rules are notoriously non-portable; the nmake cpp allows them to be completely specified using the -I- option. Since gcc has the -I- (added in the late 80's), nmake does not override the gcc cpp by default. You can force the native preprocessor by adding --nativepp=1 on the nmake command line, or --nativepp=-1 to inhibit the warning.
What is a mamfile?
A mamfile contains the Make Abstract Machine equivalent of a makefile. Mamfiles are generated by nmake (we also have a version of gmake that generates MAM) and are used to bootstrap nmake on systems that don't have it using the standalone mamake(1) command. See A Make Abstract Machine for details.
The probe(1) information is wrong for my compiler; how can I fix it?
See probe --?override.

packaging

What does AST stand for?
AT&T Software Technologies. In older distributions you may have seen Advanced Software Technologies or Advanced Software Tools; these originate from the name of our old Bell Labs department in the late 80's. The department name has changed dozens of times since, but remarkably the people involved haven't.
Is there a recommended installation directory?
Although there are no hard-coded pathnames in ast software, installing official releases in an FHS compliant directory like /opt/ast may lead to some uniformity when hopping from host to host. See /opt/ast for details.
How do I read packages on an EBCDIC host?
Follow the source or binary download instructions using the ratz(1) command to read the INIT package tarball. ratz recognizes three EBCIDIC variants and automatically converts each text file tarball member from ASCII to the native character set.
Why does package read fail even though there are unread packages in lib/package/tgz?
The default .0000 base version was dropped starting on 2001-10-20, causing older package(1) commands to fail. To get a new package command download the INIT package and manually read it in. package read should work again.
Why don't you use rpm(1) for packaging?
We are finally at the point where we can automatically generate source and binary packages for any collection of software components that are controlled by nmake(1) makefiles. The only extra work is a manually edited per-package makefile that lists the components in the package. So, to answer the question, we understand makefiles, and with that understanding we get packaging for free. Now, see the next question.
Can you generate rpm(1) packages?
Yes, in theory. The conjecture is that nmake(1) can produce the information needed by most of the popular packaging schemes. The main problem now is for one of us to understand rpm details enough to do the transformation. This will be done by the package(1) command. It's documented but not implemented (now that's a first around here).
I just want ksh(1); why isn't there a ksh tarball?
Because of its popularity, you may now get just the ksh source by downloading the INIT and ast-ksh packages. However, if you want the same build environment as ours, you should download the ast-base package (that also includes ksh).
Why do you require more than one package in some cases?
All the software on this site is built on a common set of libraries. We could have taken the GNU *utils approach and packaged a separate copy of the libraries with each command tarball. In our experience this leads to splintering that is hard to undo, and we would end up with n slightly different copies of getopt(3), for example. Instead we package the libraries once in the ast-base package. ast-base also includes ksh(1), pax(1), and nmake(1), which are used to maintain and generate new packages and deltas (patches). BTW, ast-open contains all of our open source commands, so you don't need ast-base if you download ast-open. Other AT&T Labs Research packages that require ast-base were posted in 2004, including CIAO and graphviz.

software issues

I included <ast_foo.h> and it failed miserably.
The <ast_foo.h> headers are helpers for the public ast headers and should not be explicitly included. The one exception is <ast_std.h> which may be explicitly included.
Are any of the libraries thread-safe?
aso, cdt and vmalloc are thread-safe; sfio is next on the todo list.
Do I need to run package(1) and export a bazillion environment vars just to run ksh(1)?
No. ksh(1) is a standalone executable, so you can run it from any environment. Other commands in a package, however, may be linked with shared or dynamic libraries, and these may require environment incantations to run. ksh(1) compensates by referring to the file .paths in each directory on $PATH. This file may contain definitions for FPATH and LD_LIBRARY_PATH (or the local equivalent). An FPATH=directory definition causes the specified directory to be search for function definitions. The default value is FPATH=../fun. An LD_LIBRARY_PATH=directory (or its local equivalent) definition causes the specified directory to be prepended on LD_LIBRARY_PATH for any command executed from the directory containing the .paths file. $INSTALLROOT/bin/.paths is automatically initialized when source and binary packages are installed.

The upshot is that if you export $INSTALLROOT/bin anywhere in PATH and run $INSTALLROOT/bin/ksh then any other commands in $INSTALLROOT/bin dynamically linked with shared or dynamic libraries in $INSTALLROOT/lib will find their libraries with no other environment incantations. In addition, all of the files under $INSTALLROOT/bin and $INSTALLROOT/lib are position independent: no other changes are required as long as you copy both directory trees as a unit. And yes, this eliminates the need for package use.

packge test produced a lot of failures. Should I be concerned?
Some tests are susceptible to locale and basic C type size differences and these may cause some failures. Check the test errors link for a list of known test failures by architecture.
I'd rather not install setuid commands; why is probe(1) setuid?
probe(1) generates tool-specific compiler information, which can be a relatively expensive operation. probe can operate setuid or not. If it is setuid (to the owner of the $INSTALLROOT/lib/probe tree, not root) then the generated information is shared by all users in $INSTALLROOT/lib/probe. If it is not setuid then the information is generated per-user in the bindir/../lib/probe tree, where bindir must appear in $PATH before $INSTALLROOT/bin, or in the $HOME/.probe directory as a last resort. As an alternative you can manually generate the probe information for each compiler and install the output in a writable file named by probe -k language tool processor. See probe --?override for details.
ast commands fail with libast shared library not found when run setuid -- is there a workaround?
Running setuid triggers a security feature of runtime linkers that searches for shared libraries in a restricted set of directories, typically /usr/local/lib, /usr/lib and /lib. A symbolic link to the fully qualified libast shared library path (including version numbers) from one of the restricted directories should do the trick.
Why do 3d(1), ie(1) and warp(1) fail on my system?
These commands are implemented as DLLs (aka dynamically linked libraries, shared libraries, dynamic shared objects) that change the semantics of system calls. This is done by intercepting the calls before calling the real (non-intercepted) calls. For this to work the intercept DLL must load before the local system call interface DLL (usually the C library) loads. This is a non-standard operation, but most modern UNIX variants provide an environment variable hook into the runtime dynamic linker. sun provided an early implementation using LD_PRELOAD; sgi followed with _RLD_LIST, _RLD32_LIST or _RLD64_LIST. Other vendors use one of these. 3d(1), ie(1) and warp(1) initialize both environment variables, but do no further checks.
Why do you have a foo command when the GNU one is available?
Some of our research cuts across the section 1 commands. Much of the work involves library interfaces for common operations. UNIX commands have a tradition of independence, which for the most part has been a boon to programming. The downside, though, is that for any particular algorithm, chances are it is currently implemented n>1 possibly independent ways on your system. For example, find(1), ls(1) -R, diff(1) -R, and chmod(1) -r probably use separate, hand-coded mechanisms for descending a directory tree. We developed the fts(3) directory stream interface and then re-implemented all of the commands that traverse directory trees to use the interface. This provides consistent behavior across all commands and all host instantiations for those commands, including loop detection when symbolic links are traversed. In other cases we have re-implemented commands to use sfio(3), some with performance improvements of 2 to 5 times. In some cases a command re-implementation may be used as a harness for algorithm comparison. Our sort(1) and sum(1) are built on top of libraries that provide different methods (algorithms) that can be selected at runtime. See sort --?method and sum --?method for more details. As a final example, the command line option library interface optget(3) provides a consistent mechanism for parsing, error message generation, online help and documentation.
Will you track non-standard GNU additions to standard interfaces?
Some GNU contributors have taken GNU's Not UNIX to heart. Contemptuous GNU-isms, like POSIXLY_CORRECT and ls --dereference-command-line-symlink-to-dir won't see the light of day in ast. Its a good thing the core of UNIX was designed, implemented and adopted before GNU got on board (man 2 open_file_dereference_symlink_if_dir?) That said, we do appreciate gcc, gdb, --long-options, and open source initiatives fueled by GNU.
I don't see a sign of perl or C++ anywhere; are you guys in the dark ages?
The software on this site spans the 80's, 90's, 00's and 10's. In the mid 80's we made a conscious decision to use C rather than C++ for portability reasons. The alternative library designs (e.g., discipline/method interface) have served us well enough that we have remained with C. However, the software is compatible with C++ (e.g., no true or false identifiers), and the library interface headers are passed through proto(1) to provide simultaneous compatibility with K&R C, ANSI C, and C++. As far as perl, we're keeping an alternative path alive.
I installed ksh in /bin/ksh and made it my login shell, and now I can't ftp(1) into my host.
Make sure there is a /bin/ksh entry in /etc/shells.
The ast foo man page claims that it accepts options, but no options are listed.
This seems misleading, but there is an explanation. There are some options common to all ast commands (via optget(3)) and these are not listed with the individual command documentation. You can list these global options by passing the --??? option to any ast command.
ast commands on { cygwin EMX winix } don't handle DOS format paths and path lists like { pdksh bash foo }. Why?
Well, because we are concerned with { UNIX POSIX X/Open } portability. If the underlying native system has a non-POSIX path syntax then it is the responsibility of the POSIX emulation layer { cygwin EMX UWIN } to handle the translation. POSIX emulation layers that force windowisms on POSIX applications abdicate their responsibility to emulate POSIX. See http://www.research.att.com/sw/download/win32/ for details on how ast software deals with POSIX/Windows emulation shortcomings. We do recognize emulation limitations: case-ignorant filesystem workarounds have crept into ast ksh(1) and nmake(1) code proper, but the remaining Windows workarounds are in the ast library, preserving the POSIX API for the rest of ast.
show all answers hide all answers

March 13, 2012