5.5.8 How to Add Additional JFS Logs

5.6 Troubleshooting File System Problems

This section will discuss some of the problems encountered while managing LVM and how to resolve them.

5.6.1 Recovering from Super Block Errors

If you receive one of the following errors from the fsck or mount commands, the problem may be a corrupted superblock.

fsck: Not an AIX3 file system
fsck: Not an AIXV3 file system
fsck: Not an AIX4 file system
fsck: Not an AIXV4 file system
fsck: Not a recognized file system type
mount: invalid argument

The problem can be resolved by restoring the backup of superblock over the primary superblock using one of the following commands.

dd count=1 bs=4k skip=31 seek=1 if=/dev/lv00 of=/dev/lv00

The following command works only for AIX 4.x.

fsck -p /dev/lv00

Once the restoration process is completed, check the integrity of the file system by issuing the fsck command.

fsck /dev/lv00

In many cases, restoration of the backup of superblock to the primary superblock will recover the file system. If this does not resolve the problem, recreate the file system, and restore the data from a backup.

5.6.2 Cannot Unmount File Systems

A file system cannot be unmounted if any references are still active within that file system and the following error message will be displayed.

Device busy

or

A device is already mounted or cannot be unmounted

The following situations can leave an open references to a mounted file system.

• Files are open within a file system. Close these <files> before the file system can be unmounted.The fuser command is often the best way to determine what is still active in this file system. The fuser command will return the process IDs for all processes that have open references within a specified file system as shown in the following example.

  fuser -xc /tmp
  /tmp: 2910 3466 11654 26400
  

  The process having an open reference can be killed by using the kill command and the unmount can be accomplished.

• If the file system is still busy and not getting unmounted, this could be due to a kernel extension that is loaded, but exists within the source file system. The fuser command will not show these kinds of references, since a user process is not involved. However the genkex command will report on all loaded kernel extensions.

• File systems are still mounted within that file system. Unmount those file systems before this file system can be unmounted. If any file system is mounted within a file system, this leaves open references in the source file system at the mount point of the other file system. Use the mount command to get a list of mounted file systems. Unmount all the file system which are mounted within the file system to be unmounted.

5.6.3 When the df Command Hangs

If the system hangs when you use the df command, the process described in this section may assist you.

Run the df command to check the file system. Note that the raven:/u2 entry at the bottom is a remote file system that is locally mounted.

# df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           40960     19384   53%     1252    13% /
/dev/hd2          663552     13288   98%    13170    16% /usr
/dev/hd9var       401408    262528   35%    22288    45% /var
/dev/hd3           65536     35336   47%       70     1% /tmp
/dev/hd1          360448     19704   95%       98     1% /home
raven:/u2          16384         0  100%       18     1% /u2

If the remote system raven is down and you enter the df command, the following will be your response:

# df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           40960     19384   53%     1252    13% /
/dev/hd2          663552     13288   98%    13170    16% /usr
/dev/hd9var       401408    262528   35%    22288    45% /var
/dev/hd3           65536     35336   47%       70     1% /tmp
/dev/hd1          360448     19704   95%       98     1% /home
NFS server raven not responding still trying

Break out of the error by pressing Ctrl-c.

Correct the problem with the remote system and once communication has been reestablished, you can run the df command to completion.

Some tips to check the remote system follow.

• To check if the remote system is reachable through the network, enter:

  # ping raven
  PING raven.itsc.austin.ibm.com: (9.3.1.215): 56 data bytes
  64 bytes from 9.3.1.215: icmp_seq=0 ttl=255 time=9 ms
  64 bytes from 9.3.1.215: icmp_seq=1 ttl=255 time=1 ms
  ^C
  ----raven.itsc.austin.ibm.com PING Statistics----
  2 packets transmitted, 2 packets received, 0% packet loss
  round-trip min/avg/max = 1/5/9 ms
  

  This shows that the problem is on the remote machine. If ping failed, it could be a break in the network or a problem with the network adapters in either machine. Choose another address on the same network segment to test.

• Stop and restart the NFS daemon on the remote system:

  # stopsrc -s nfsd 
  0513-044 The stop of the nfsd Subsystem was completed successfully.
  # startsrc -s nfsd
  0513-059 The nfsd Subsystem has been started. Subsystem PID is 10178.
  

5.6.4 Deleting an Application File That Is Locked

In this section, you are trying to delete an application file that is locked. When you try to delete the file, you receive an error message telling you that it cannot be deleted because the file is locked.

If you get an error trying to delete the file /userdirectory/myfile enter:

# fuser /userdiectory/myfile
myfile:   104772

The number 104772 is the process ID (PID) of the file. To kill the process ID enter:

kill -9 104772

Now, to delete the file and free the disk space for use by the file system enter:

rm myfile

5.7 Summary of LVM Commands