File Storage:

Introduction: To help with data file management some new practices are being implemented.  These practices will make data files more easily available to VS120 users at a location which will be accessible over the network. 

Overview: In order to backup users' data files on the E: drive of the Olympus VS120 scanner will be copied to the data server \\minsky.med.unc.edu\vs120 nightly at ~4 AM.  This will make files on the E: drive of the VS120 system available on the \\minsky.med.unc.edu\vs120  server the next day.  The files on the server will be accessible over the network using the usual shared directory mapping procedure and an mhmicroscopy domain account. Thus it will not be necessary to be at the VS120 system to access data on the next day after scanning.

What is being done?

• Every evening files will be automatically copied from the E: drive if they do not exist at \\minsky.med.unc.edu\vs120 or if they are newer than the ones already at \\minsky.med.unc.edu\vs120.  Date & time stamps for files and directories will be updated at the server as well.  Files will remain on the E: drive until removed by the owner or after 3 months by the system manager.

What are the data implications for users?

• Files will be at 2 locations initially
• The copy process is cumulative.  Files which are newer on the E: drive & files on the E: drive that do not exist at the \\minsky\vs120 server will be copied nightly
• Deletion of files at the VS120 E: drive will leave copies on the \\minsky.med.unc.edu\vs120 server and also on a hidden backup server

What are the limitations?

• The VS120 C: & D: drive files will not be copied to the server
• Newer files will displace older files with the same name in each respective directory on the \\minsky.med.unc.edu\vs120  server storage space
• No archiving is carried out. Older files with the same name as newer files in each respective directory will not be recoverable.
• At some future time someone will have make the executive decision to remove very old data on the server which will inevitably fill up.
• Files deleted from the E: drive will remain at \\minsky.med.unc.edu\vs120.  This good since the data is effectively backed up.  This is also bad since files deleted from E: will remain on the server and may be unwanted, and, hence, take up valuable server storage space.
• If files are deleted from  \\minsky.med.unc.edu\vs120  they should be deleted from the VS120 E: drive.  This represents an extra step for users.  If the files are not also deleted from E: they will reappear at \\minsky.med.unc.edu\vs120 after the next evenings copy process.
• Copying will slow down slide scans being run ~4 AM.  The slow down should not be too significant, and be over in less than one hour.
• This is all necessarily rather complicated.  But it solves the E: drive filling up problem and preserves files which may be needed by users who have not copied their data of the system E: drive.

What will be done with data on the E: drive?

• Files older than 3 months old will be subject to summary deletion on the E: drive after it is made sure copies exist on the \\minsky.med.unc.edu server or potentially another backup server.

What users can do to help with data management?

• If files are deleted from the \\minsky.med.unc.edu\vs120 they should also be deleted from the VS120 E: drive.  This represents an extra step for users.  A work around is to add "-DELETE" to the top directory or directories on the E: if the data is not needed in both the E: drive and the \\minsky.med.unc.edu\vs120 server (It will be up to management to notice the message and handle deletion of all files along the storage chain.  This minor overhead has the benefit of letting data managers know the files are not needed and hence can be deleted along the server storage chain)
• Do not rename subdirectory names on the E: drive since that will create two directories on the \\minsky.med.unc.edu server creating duplicated data and wasting storage space.  Exception is adding "-DELETE".

General Data Storage Notes