Sorcerer Booting Explained Copyright 2008 by Kyle Sallee BIOS When a box boots the BIOS, basic input output system, runs. BIOS detects and checks RAM, initializes the CPU and hardware, and finishes by passes control to the first disk sector of the primary fixed disk. Although modern BIOS can boot from USB devices, removable devices and network devices, for simplicity this documentation assumes that a fixed disk or flash drive is booting. The start or beginning area is called the MBR, master boot record. The master boot record contains partition information. And it may contain a bootloader. Bootloaders installed to the MBR typically do nothing more than pass control to the first sector of the first partition marked as bootable in the partition table. The first sector of the boot partition will contain a bootloader such as LiLo. LiLo, the Linux Loader, presents a menu of kernels that can be loaded. It also accepts kernel parameters when provided with the correct password. The default password is sorcerer. The default password is stored in /etc/lilo.conf and /etc/lilo.conf.head The SA should change the default password to something more secure, yet easily remembered. LiLo After a kernel image is selection in LiLo and optional kernel parameters entered then LiLo will load the kernel and initramfs images from disk into memory. Finally, LiLo passes control to the Linux kernel. /init Modern Sorcerer boxes boot using an initramfs image. initramfs is short for Initialization Random Access Memory File System. The initramfs exists only to run sysinit After sysinit completes the initramfs is cleared from RAM. After the Linux kernel finishes loading the kernel initializes some hardware and then decompresses and unpacks the initramfs image. Finally, the Linux kernel passes control to /init on the initramfs image. /init has a few complex tasks to perform. However, /init 's primary task is to execute /etc/init.d/rcS . /etc/init.d/rcS executes sysinit. More precisely sysinit is the execution of the order symbolic links in /etc/rc.d/rcS.d The purpose of sysinit is to check and mount the real root filesystems. After sysinit completes then /init resumes execution. /init may now create a more precise initramfs image for a boot that will be labled "fast" The next time the box boots, fast may appear on LiLo's boot menu. The Linux version for the fast image that is created will be the same version as Linux booted when the fast image is created. In almost all cases the fast image will also be the most recently installed Linux kernel. Finally, /init has but a few tasks left to accomplish. /init must deleted the contents of the initramfs image from memory. Then, /init must pass control to /sbin/init located in the real root filesystem. After that happens the initramfs is as good as gone since there is no way back to it. About a minute after the runlevel begins the content of the initramfs is removed from RAM. During sysinit execution the initramfs is / and the real root filesystem will eventually be mounted as /media/root However, after /init pass control to /sbin/init on the real root filesystem then the real root filesystem becomes / System V Init /sbin/init /init starts as process 1. However, when /init finishes /sbin/init becomes process 1. Since sysinit has already run the next task for /sbin/init is to start the runlevel Runlevels # Runlevels # 0: halt # 1: sequential init # 2: parallel init + multi-user + networking # 3: sequential init + multi-user + networking # 4: parallel init + multi-user + networking + GUI login # 5: sequential init + multi-user + networking + GUI login # 6: reboot The runlevel to execute is the runlevel specified by initdefault in /etc/inittab Boxes may have more than 6 runlevels. However, Sorcerer only defines and uses 6 runlevels. And only one runlevel can be the initdefault. Runlevel 0 is for halting. Runlevel 6 is for rebooting. Therefore, runlevels 0 and 6 are never initdefault. Runlevel 1 is for single user mode. Runlevel 1 is rarely the initdefault. Runlevels 2 3 4 5 are typically initdefault. Runlevel 3 is most common for console only boxes. However, boxes with a GUI, graphical user interface, Xwindows login will probably use runlevel 5. Runlevels 3 and 5 are recommended for new system administrators. For faster booting runlevels 2 and 4 replicate runlevels 3 and 5, respectively. Unlike runlevels 3 and 5 which run init-scripts sequentially, runlevels 2 and 4 start programs concurrently. Consequently, runlevels 2 and 4 complete a few seconds faster. However, the runlevel log stored in /etc/init.d/log.d/ becomes more difficult to read. The definition of runlevels 2 and 4 as parallel 3 and 5 may be unique to Sorcerer. Most distributions either reserve runlevels 2 and 4 or those runlevels are considered experimental. However, with Sorcerer system administrators have the choice between sequential and parallel executing runlevels. Just as /init delegates the task of starting sysinit to /etc/init.d/rcS so does /sbin/init delegate the task of starting the runlevel to /etc/init.d/prc Directories /etc/rc.d/rc${RUNLEVEL}.d/ contain symbolic links to a init-scripts that should be executed for each runlevel. Therefore, symbolic links to init-scripts required to run for runlevel 5 would be located in /etc/rc.d/rc5.d/ Login Boxes not yet prepared for GUI login will have a console login prompt. If sysinit and runlevel scroll was redirected to virtual consoles 9 and 10 then the login prompt will be visible. Otherwise, it will not appear until enter is tapped. More precisely, it did appear right as sysinit finished and the initdefault runlevel began. However, runlevel scroll can overwrite the login prompt or scroll it off the top of the console. Console login becomes available immediately before runlevels 1 through 5 start. However, GUI login becomes available later in the execution of runlevels 4 and 5. Often both console and GUI login are available prior to runlevel completion. Once the X desktop displays the box is responsive and ready to use. No illusion is hiding another lengthy minute or two delay. Sorcerer is a fast booter with no bull. Boot Logs Boot logs are stored in /etc/init.d/log.d/ and not /var/log/ /etc/init.d/log.d/ is a tmpfs, virtual RAM based filesystem, that is created early in the boot process. Eventually, /etc/init.d/log.d/ is moved from it's mount point on the initramfs to the real root filesystem. This way logs can be created and stored long before /var/log/ becomes writeable. Whether boot logs scroll virtual console 1 during boot depends on what the SA selected when configuring the sysvinit spell. To reconfigure sysvinit execute at a root prompt: augur forget sysvinit; cast sysvinit The recommendation is to have sysinit scroll on /dev/tty9 and runlevels scroll on /dev/tty10 which are also virtual consoles 9 and 10 respectively. From a Linux virtual console they can be reached by pressing the alt key and appropriate function key together. ALT and F9 for virtual console 9 Alt and F10 for virtual console 10 Also ALT and the left and right arrow keys progresses forward and backwards through all available virtual consoles. Typically, only the current console can be scrolled back by pressing shift page up. Therefore, if important scroll already departed virtual consoles 9 and 10 then the log files in /etc/init.d/log.d/ can still display all the scroll that happened during boot. Booted After the runlevel completes then the box has finished booting. All services normally started during boot have started. Checking the logs in /etc/init.d/log.d/ may be prudent. If services failed to start the logs in /etc/init.d/log.d/ will show which ones. However, most boxes boot fast and near flawlessly. Unlike other distributions Sorcerer boxes are not clones. The init-scripts stored in /etc/init.d/ and executed during sysinit and runlevels are precisely the init-scripts that are required for the box given the software that is installed upon that box. Consequently, if nfs-utils is not installed then no init-scripts for NFS are installed and therefore no init-scripts for NFS run during boot and shutdown. The same is true for other programs and init-scripts such as samba. The precise installation and execution of tiny eloquent init-scripts gives Sorcerer an uncommonly fast boot time. Other distros may execute an entire wardrobe of init-scripts during boot and shutdown. Most of those init-scripts are for controlling software that is not installed. In contrast Sorcerer's init-scripts is like the attire of a the person who is appropraitly dressed for function that they are intending. Would someone wearing everything they own, amounting to +40 KG of clothes and shoes, look strange? We agree. And that is why Sorcerer development decisions are impelled by that sort of common sense. Precise and concise is nice. Init-Scripts A name like init-script implies a program only used for startup. However, init-scripts handle: start; stop; status; restart; and other requests. init-scripts are installed in /etc/init.d/ Usually, only spells install init-scripts. And only spells should modify init-scripts installed in /etc/init.d/ However, SAs are invited to toggle on and off the automatic execution of init-scripts. Init-scripts with mode 700 execute automatically. Init-scripts with mode 600 execute only when the system administrator executes them. The simple commands to turn on and off an init-script called $file are: chmod 700 /etc/init.d/$file chmod 600 /etc/init.d/$file A safe and convenient way to select the automatic execution status of nearly all installed init-scripts is to execute: augur forget init-functions; cast init-functions However, please be careful about disabling automatic execution of init-scripts. Some init-scripts may be absolutely essential for a box to properly boot. Init-Scripts in Detail Sorcerer supports execution of standard LSB style init-scripts as described at: http://refspecs.Linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/tocsysinit.html Therefore, Sorcerer is LSB compliant in regards to supporting init-scripts. However, Sorcerer init-scripts are not LSB compliant. Sorcerer init-scripts extend the functionality beyond what is specified by the LSB. Consequently, Sorcerer init-scripts tend to be extremely terse. Only after reading several init-scripts does the factorization of code become apparent as to how Sorcerer init-scripts can be so small. Most init-scripts are provided with spells. Rarely will a SA have to create an init-script. If an init-script should be created then a LSB compliant style init-script will work fine. However, if the factorization of code makes sense then go ahead and use Sorcerer style init-scripts. Unfortunately, the exact specifications for Sorcerer style init-scripts is not contained in this document for the following reasons: init-scripts are rarely a concern for SAs; most spells install init-scripts when required; the factorization of what would otherwise be common code among many init-scripts requires a very lengthy explanation. However, as an example the following is the init-script for mounting the virtal filesystem proc onto the /proc mount point #!/bin/dash ### BEGIN INIT INFO # Default-Start: S # Short-Description: proc mounts proc on /proc ### END INIT INFO . /lib/lsb/init-functions NAME=/proc OPTS='-n -t proc proc' parse "$@" The /etc/init.d/proc init-script contains only the above few lines. However, the majority of the work is done from functions defined in the file /lib/lsb/init-functions. Reboot and Shutdown Eventually a box must be rebooted. Almost all software on a running box can be restarted. However, restarting into a new kernel requires a reboot. An uptime of 1 year straight is not worth running a year old kernel. So reboot at least quarterly. Many SAs were surprised and upset when new versions of coreutils began to use functions that were only available in a kernel released within the last 6 months. But those boxes only had to be re booted into moderately recent kernels in order to avoid the problem entirely. Therefore, rebooting is good, not bad, and should be done quarterly. At the very least rebooting will fsck drives which may find and repair errors in the filesystem. Sorcerer boxes reboot very fast. Boxes with kexec-utils installed can reboot directly into a newer kernel and initramfs without entering the LiLO boot loader, and without having to go through the BIOS POST, Basic Input Output System Power On Self Test. Rebooting begin by issuing the command: shutdown -r now Shutdown begins by issuing the command: shutdown -h now Rebooting causes /sbin/init to switch to runlevel 6 To accomplish this /sbin/init executes /etc/init.d/prc Then /etc/init.d/prc executes the scripts in /etc/rc.d/rc6.d/ The execution of init-scripts during shutdown and reboot is always sequential and never concurent. Shutdown and reboot should be allowed to run to completion. Sorcerer has convenient automatic maintainence that is performed during shutdown and reboot. When necessary the following events can occur: 1. depmod can execute to update module dependency lists 2. Symbolic links located in /etc/rc.d/rc?.d/ to init-scripts in /etc/init.d/ are regenerated. 3. /etc/lilo.conf can be regenearted from /etc/lilo.conf.{head,tail} and kernel images /boot/*/linux 4. Initramfs images such as /boot/*/initrf can be regenerated 5. /sbin/lilo can be run to install a new LiLo into a boot block The above events involve filesystem modification. Also if new initramfs images are generated then /sbin/lilo must and will be automatically run. If a box is forcefully powered down or reset during the execution of runlevels 6 or 0 then that box may be unable to boot. A box with an interrupted shutdown or interrupted reboot may respond during the following boot with error messages from LiLo or Linux. LiLo may claim that the initramfs image does not have correct magic. LiLo may complain about the kernel. LiLo may print numbers instead of an error message or only part of the lilo name. LiLo may present an incorrect boot menu. The Linux kernel may crash when loaded. The Linux kernel may complain about the initramfs and immediately halt. The Linux kernel may halt and blink some keyboard lights repeatedly. Those situations are not necessarily doom. But they will require booting from the Sorcerer Install/Rescue disk. And they also require skillful manual repair by a system administrator. 1. Boot using the I/R 2. Mount the real root as /media/root 3. Mount the real /boot as /media/root/boot 4. chroot /media/root 5. mount -t proc proc /proc 6. mount -t sysfs sysfs /sys 7. Check /etc/lilo.conf 8. Manually run init-scripts in /etc/rc.d/rc6.d/ with the stop parameter 9. However, do not run halt or reboot init-scripts 10. umount /proc 11. umount /sys 12. exit 13. umount /media/root/boot 14. umount /media/root 15. try rebooting normally. While easier than making tea for an exprienced SA a novice SA could fill a tea cup with sweat alone while trying to solve a boot problem. Best to let Sorcerer do what it does automatically. That way a boot problem might never happen. The benefit of Sorcerer shutdown may not be speed. However, it is definitely convenient. System adminsitrators never have to learn which commands to type in order to manually regenerate initramfs images, update lilo.conf, and to regenerate symbolic links in directories /etc/rc.d/rc?.d/ But if the system adminsitrators wants then those commands can always be executed manually at any time. Either the init-scripts can be executed directly with stop parameters. Or the locations defined by the LSB for those programs can be executed. For regenerating initramfs: /sbin/mkinitrd or /sbin/mkinitrf For regenerating /etc/rc.d/: /lib/lsb/install_initd For re-installing LiLo: /sbin/lilo To execute those commands is so mundane that Sorcerer makes it entirely automatic and part of the normal shutdown/reboot process so that system administrators are not bothered to even think about such trivial matters. Trouble Shooting Fortuantely, the text editor called "nano" is installed and available on the initramfs. Therefore, init-scripts and configuration files can be edited. If a box can run linux and begin sysinit then it can be troubleshoot without an Install/Rescue disk. Otherwise, please consult the section on "Reboot and Shutdown" The first step to trouble shooting a problem that is occuring on the initramfs is to gain manual control of the initramfs as soon as possible. To accomplish this give the kernel parameter rdinit=/bin/bash With the kernel parameter rdinit=/bin/bash Linux will not execute /init Instead /bin/bash executes /bin/bash grants a root shell command prompt. From this prompt changes can be made to files and init-scripts. After examining /init , /etc/init.d/rcS , and /lib/lsb/init-functions then debugging init-scripts is becomes easy to understand. The SA can manually execute init-scripts one at a time to precisely locate the probable initial point of failure. Also init-scripts can be executed such as: bash -x $SCRIPT_NAME That will show a bash trace of the init-script. Fortunately, the Sorcerer Development Team does excellent testing of init-scripts. Rarely are there any serious problems. Distribution wide derailments of boxes during boot never happens. Even if minor problems do happen Sorcerer init is sufficiently robust that most boxes continue booting without requiring system administrator interaction. After sysinit completes and the runlevel begins nothing almost nothing can prevent console login since gettys are started even before the runlevel begins. Knowing how to fix a boot problem as it occurs is potentially useful for system administrators who write init-scripts that run during sysinit. However, almost all Sorcerer system administrators will never use kernel parameter rdinit=/bin/bash and debug a boot problem. Yet, at least 5 people sometimes do. Therefore, if help would be useful then please ask on the Sorcerer email list. Those experienced Sorcerer SAs can assist others trouble shooting problems. They make the detection and resolution of nearly any problem seem no more complicanted than plugging the power cord back into the UPS, uninterruptible power supply. No need to feel alone and paniced. These system administrators have been there, seen it, and solved it. They give detailed responses that confirm the importance of each question. Nobody should feel ignorant or embarrassed. Helping our fellow SAs, system administrators, is the Sorcerer way. Conclusion Congratulations. Most of the information presented in this document the average Sorcerer SA will either not learn or will learn as needed by examination their boxes and by asking questions on the email list. However, you are now in the know. You are no longer a Sorcerer SysVInit neophyte. Consequently, you are invited to provide answers to other SA's questions on the email list and to provide revised and improved documentation for your fellow Sorcerer SAs. Thank you for allocating time to read this document.