What I Know About Linux That You May Not Know

Found a nice write up here. Just thought it's worth sharing as most people are unaware of this. There is one more nasty reason for df and du to show different output which is not discussed there. May not be worth discussing there.

Me had a case where an administrator installed a system with just 5GB allocated to /. Later he figured out the log files in /var is quickly growing up and / will fill up very soon. So far /var has logs of size 2GB. He just created another 10GB partition, copied the current contents of /var to it, then mounted the new partition on /var without deleting the current contents in /var.

This administrator did never document this event and quit the company. A new guy stepped in. Later, when he scanned the / filesystem (may be when the / was 100% next time) he found df and du output of / is different. (showing 2GB difference). You can imagine what is the cause?

The worst question is how a Technical Support Engineer figure this out? Wild guesses? But I had to.

I was about to submit claims for my expenses. I had to scan the receipts and email them to blah blah blah. The scanner gave me multiple pdfs and I preferred to send only one pdf. I had to merge the pdfs and did that by using the below command in my Fedora 10 box.

# gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=merged.pdf -dBATCH first.pdf second.pdf

I got this solution through a search in the web.

Today when I logged into one of my test system and did "ls -l /root", I found a "+" in the output against each file and directory immediately after the permission bits are displayed. One example is given below.

-rw-r-xr--+ 1 root root 151 Jul 31 20:38 test.sh

I had no idea what this "+" indicates about the file or directory. I just searched google to find out without any luck. Every docuemnt that I referred speaks about all other fields displayed in the output, but kept silent about "+". "man ls" has nothing to say about it. But I was not ready to give up, I found out myself what that field indicates. You may already know what is meant by this +, but this blog is intended to explain how did I find it out myself which may be useful for you also if you face a similar situation in future. Below is the method that I followed.

I created a file in /tmp named file.txt. When I did "ls -l" on that file, I didn't see the "+" in the output. Now I have a file which has a + in the "ls -l" output and one which doesn't have.

Now I did strace on "ls -l" while listing both the files. Strace was executed as below.

# strace -fvvv -s 1024 -o output-file ls -l file-name

Analyzed both straces and compared them. This comparison helped me to see what is different between these two files.

For the file which has + in its output, I found the below system call in strace.

29608 getxattr("/root/test.sh", "system.posix_acl_access", 0x0, 0) = 44

For the file which doesn't have + in the output, I found the same system call as below.

29616 getxattr("/tmp/file.txt", "system.posix_acl_access", 0x0, 0) = -1 ENODATA (No data available)
29616 getxattr("/tmp/file.txt", "system.posix_acl_default", 0x0, 0) = -1 ENODATA (No data available)

The difference in the output of getxattr() told me that the file which has a "+" in the output has a filesystem acl on it where as the file which doesn't have a "+" in the output has no acls set on it (This is indicated by the "-1 ENODATA (No data available").

I verified this by running "getfacl " on both files. Then I did "man acl" and started reading that and found the below details.

"For files that have a default ACL or an access ACL that contains more than the three required ACL entries, the ls(1) utility in the long form produced by ls -l displays a plus sign (+) after the permission string."

Is "man acl" the right place to have this info?

I am running Fedora 10 on my Laptop. My sister brought a vcd and requested me to play that. I inserted the vcd, mounted on /mnt and started to play it. To my surprise, it didn't work.

$ cd /mnt/

$ ls
cdi ext mpegav segment vcd

$ cd mpegav/

$ ls
avseq01.dat

$ mplayer avseq01.dat

Playing avseq01.dat.
Seek failed

Exiting... (End of file)

Then I tried to copy the avseq01.dat to a local folder.

$ cp avseq01.dat ~/
cp: reading `avseq01.dat': Input/output error

Tried to copy using dd, but no luck.

$ dd if=avseq01.dat of=~/vcd.dat
dd: reading `avseq01.dat': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00805312 s, 0.0 kB/s

Problem is neither with cdrom nor with the drive, but with the format of the data which need to be converted to another format which is free from the proprietry stuffs. So my friend Ritesh came to my help. Below is how I did it.

- Installed "vcdimager" package. That version that I installed was vcdimager-0.6.2-1.i386. It's available via yum.

- Create a folder named ~/vcd and cd into it.

- Run vcdrip (vcdxrip in the latest version of vcdimager package) to copy the video file to mpg format.

$ vcdrip --rip --cdrom-device=/dev/cdrom

- The above command will copy the avseq01.dat from cd to avseq01.mpg in the current working directory. This process will take sometime and will take more time if there are a lot of scratches on the cd. Once it exits, run "mplayer avseq01.mpg" in the current directory to play it.

Below are the steps that I followed to clone an RHEL5 guest running under vmware ESX-3.5 to Xen environment over the network. The method that I followed can be used to clone any virtual machine (except windows?) running under any virtualization product to any other virtualization product (Correct?). I agree that there are specific tools provided by vendors for p2v and v2v conversions, but they are limited to their virtualization product. Eg, Vmware tools may not be used to convert a virtual machine running under vmware to run on Xen/KVM and vice-versa.

Pre-requisites.

- Linux based LiveCD. (Which should have coreutils and nc - netcat - packages installed).

Below are the steps that I followed:

- Downloaded the LiveCD for Fedora 10 from here (You can use whatever Linux LiveCD you want.)

- Booted the rhel5 virtual machine already available in Vmware ESX from this ISO by attaching this ISO to the guest and selecting cdrom as the first boot device. (Consult the concerned documentation for more details)

- Created a new blank guest in Xen and assigned it a hard disk with the same size of vmware disk. I just created a fully virtualized guest using virt-manager and when it started the anaconda installation, aborted the installation and shutdown the guest. Now I have a guest with a blank image.

- Then started the blank guest under Xen from the Fedora LiveCD. (If not sure how to do it, please consult xen documentation).

Now we have both Vmware guest (will be called GuestA from now onwards) and a guest with blank image in Xen (will be called GuestB from now onwards) booted of the Fedora 10 Live CD.

- On both guests flush the iptables firewall.

# iptables -F

- Networking should be enabled automatically by the LiveCD. Make sure that networking is working as expected on both guests and they can ping each other.

- Run fdisk -l on both guests and identify how the hard disk has been detected. I had them detected as "/dev/sda" on both GuestA and GuestB.

- On GuestB, run the below command:

# nc -l 7000 | dd of=/dev/sda bs=16M

- On GuestA, run the below command.

# dd if=/dev/sda bs=16M | nc ip-of-GuestB 7000

Replace ip-of-GuestB with the actual ip of GuestB and replace /dev/sda with the actual block device in both commands. 7000 is the port number, you can use other unused ports as well.

The process of copying the hard disk image will take sometime depending upon the network bandwidth and the IO bandwidth availability of storage. Please be patient and restart GuestB once the process is over from the hard disk.

- Went to bed and had a good sleep. When I was up in the morning, I had the guest cloned successfully and started off the new guest.

Below are the steps which explain how to set this up in a simple environment.

This setup involves three systems.

1. Windows 2003 as KDC
2. NFS server
3. NFS client

Windows server is Windows 2003 R2 SP2. NFS server and NFS client are running RHEL-5.3

Pre-requisites

- Working DNS server
- Working NTP server so that time in all the machines are in sync.

These are requirements for Kerberos to work.

In Windows server create a user "nfsuser" with some password.

How to setup NFS server?

1.1- Run "authconfig-tui". In the "Authentication" section, select "Use Kerberos" -> Next -> Type in the realm name, IP of Windows AD for KDC and admin server.

1.2- Add a user named "nfsuser" with uid 2000 without setting up any password for that user.

# useradd -u 2000 nfsuser

1.3- Create 2 groups named "group1 and group2" with gid 3001-3002 on the system.

1.4- Add "nfsuser" as member of both the groups.

# usermod -G group1,group2 nfsuser

1.5- Create a directory to share using nfs.

# mkdir /nfs

Create 2 directories and each writable by one group.

# mkdir /nfs/group1
# chgrp group1 /nfs/group1
# chmod g+w /nfs/group1

# mkdir /nfs/group2
# chgrp group2 /nfs/group2
# chmod g+w /nfs/group2

1.6 - Edit /etc/sysconfig/nfs and un-comment the below line.

SECURE_NFS="yes"

1.7 - Edit /etc/exports and share /nfs as below.

/nfs gss/krb5p(rw,sync,fsid=0)

1.8- Now create the keytab on windows server in command prompt using the below coammnd.

ktpass -princ nfs/nfsserver.example.com@EXAMPLE.COM -mapuser nfsuser -pass **** -out nfsrv.keytab /crytp rc4-hmac-nt /ptype KRB5_NT_PRINCIPAL

* Please note that I have used "rc4-hmac-nt" crypto.

RC4-HMAC-NT is the default and employs 128-bit encryption.

You can find more details about ktpass and types of crypto here

1.9- Copy nfsrv.keytab on NFS server as /etc/krb5.keytab
1.10- Verify with klist -ke

# klist -ke /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
2 nfs/nfsserver.example.com@EXAMPLE.COM (ArcFour with HMAC/md5)

1.11- Start nfs server service and rpcidmapd.

# service nfs start
# service rpcidmapd restart

How to configure NFS client?

2.1 - Repeat step 1.1
2.2 - Repeat step 1.2
2.3 - Repeat step 1.3
2.4 - Repeat step 1.4

2.5- Edit /etc/sysconfig/nfs and un-comment the below line.

SECURE_NFS="yes"

2.6- Create the keytab on windows server

ktpass -princ nfs/nfsclient.example.com@EXAMPLE.COM -mapuser nfsuser -pass **** -out nfscli.keytab /crytp rc4-hmac-nt /ptype KRB5_NT_PRINCIPAL

2.7- Copy nfscli.keytab on NFS client as /etc/krb5.keytab

2.8- Verify with klist -ke

# klist -ke /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
2 nfs/nfsclient.example.com@EXAMPLE.COM (ArcFour with HMAC/md5)


2.9- Start "rpcgssd" and "rpcidmapd".

# service rpcgssd start
# service rpcidmapd restart

2.10 - Mount the share as below.

# mount -t nfs4 nfsserver.example.com:/ -o sec=krb5p

# mount | grep nfs4
nfsserver.example.com:/ on /mnt type nfs4 (rw,sec=krb5p,addr=10.65.209.189)

# ls /mnt
group1 group2

I have a windows Xp virtual machine running in my laptop on top of Fedora 10 KVM. One day I accidentally deleted the sound card from virt-manager -> Hardware tab for that guest. I then understood my mistake and tried to add it via "Add Hardware" wizard. To my dismay, there was no option to add a sound card.

Then how did I add the sound card back to the guest?

- I did "virsh dumpxml anotherguest" which has sound card attached to it and was able to see the below line in the output.

....................
sound model='es1370'
....................

- I then did "virsh dumpxml xp" and was not able to see the above line in it. So fixing this is as simple as adding the above line to winxp configuration file.

How to do that?

- Dump the xml file to a file in the disk.

# virsh dumpxml xp> xp.xml

- Edit the xp.xml and add the below line to it.

sound model='es1370'

I added this immediately before the "devices" line.

- Then redefine the guest using the new configuration file.

# virsh define xp.xml

Restart the guest and the network card would be present in the guest. Hope we can add the sound card in F11 through virt-manager GUI.

In simple steps:

1 - Collect the below details about the system from where the crash dump was generated.

- Architecture of the system. i686, x86_64 and etc.
- Version of the kernel running while the system crashed.

2 - Prepare a system which runs the same OS as of the crashed system and the same arch. (Not mandatory, but below steps depends on this). Using the same system which crashed is also fine.

3 - Download the kernel-debug package matching the version of the kernel which was running while the system crashed and install it using "rpm -ivh packagename" command.

Debug info packages for RHEL can be downloaded from here

4 - Run crash as below.

# crash /usr/lib/debug/lib/modules/kernel-version/vmlinux path-to-the-vmcore

Eg, I had a customer recently reported a kernel crash with rhel5 kernel "2.6.18-128.1.1.el5xen". Below is the steps that I did.

- Downloaded debug kernel from here

- Installed it using "rpm -ivh kernel-xen-debuginfo-2.6.18-128.1.1.el5.x86_64.rpm"

- Executed crash as below.

# crash /usr/lib/debug/lib/modules/2.6.18-128.1.1.el5xen/vmlinux /root/vmcore

If anyone of you are facing complete network failure or unacceptable packet loss in guest communication while using RHEL5 for Virtualization using xen + bonding (only mode0/round robin) + briding, please refer redhat kbase to know how to solve or work around it. This can also be applicable for KVM virtualization in Fedora if public bridge is used for networking with bonding in mode0

Red Hat has introduced support for autofs5 in RHEL4. The default version of autofs in RHEL4 is autofs4 which has a lot of limitations. The biggest limitation of autofs4 is that doesn't have support for direct mounts. With the introduction of autofs5 in RHEL4, one can achieve all the features of autofs5 in RHEL4.

I wanted to run the autofs configuration that I explained here with ldap support which includes examples for both direct and indirect mounts in RHEL4 and below are the steps that I followed.

- The minimum update level to support autofs5 in RHEL4 is U7. The kernel version required is 2.6.9-78 and above. So I updated my system to RHEL4 U7 and the kernel running is 2.6.9-78.0.13.EL

- Install "autofs5" rpm.

up2date autofs5 or rpm -ivh autofs5-blah-blah.rpm

- It's best to turn off autofs v4 while using v5 to avoid conflict. Autofs5 supports everything in autofs4 and there is no need to run both of them at the same time.

chkconfig autofs off
service autofs stop

- Edit /etc/sysconfig/autofs5 and uncomment the schema definitions that you are following in your network. I uncommented the below lines to follow schema III that I explained here

- Edit /etc/auto.master and add the below line into it.

+auto.master

This step is very important as the default rhel4 /etc/auto.master does not include this line unlike RHEL5. So if you forget this step, maps will not be read from LDAP.

- Start "autofs5" service.

chkconfig autofs5 on
service autofs5 start

It should work just like in RHEL5.