Vyapin Blogs

Most of the content migration to SharePoint involves migration of archived and active contents from different systems such as file servers, network shared folders and other document management systems (e.g., Humming Bird, DocuShare, DocsOpen). This is an impending task apart from migrating frequently changing folders and files in the current storage system on a day-to-day basis.

It is a well-known fact that the contents stored in these systems might have their folder and file names coined as per legacy naming conventions or standards prevailed at that time. There also exists a possibility of the folder and file names having special characters that are valid in legacy systems but invalid (disallowed) in SharePoint. During migration of contents from legacy systems to SharePoint, users might want to get rid of the old names and replace them with new names, rename invalid to valid characters in folder and file names.

Apart from this, users may have a possible need for renaming folders and files that are already available in SharePoint.

In such cases, organizations and users who migrate bulk contents into SharePoint need user-friendly means with varied options to rename those folders and files in bulk during migration / post migration.

In a nutshell, we highlight the following general and specific cases of renaming requirements:

	Rename folders & files while migrating from legacy systems to SharePoint
	Rename folders & files with invalid characters to valid characters in SharePoint
	Rename folders & files that already exist in SharePoint

Folder level requirements:

	Create a new folder hierarchy (restructure existing contents) and import files into them during migration
	Create new document sets by way of renaming folders

Document / File level requirements:

	Rename files of different names and related contents and import them as document versions (maintain version history)
	Rename files having names for each version (appended with version number) and import them as document versions

To cater to the above mentioned renaming requirements, DocKIT for SharePoint 2010 product provides easy renaming of folders and files in the following two ways:

1. Rename folders & files using the values specified in an external metadata file or batch descriptor file.
2. Rename folders & files using folder & file naming rules.

Rename folders & files using the values specified in an external metadata file / batch descriptor file

I will explain how to use the external metadata file or a batch descriptor file to rename folders and files in the most common cases using a few examples below:

1. Rename files simply by providing alternate names

2. Rename folders simply by providing alternate names

3. Rename files that have same extension

Let’s take an example where files generated by an external system with the same extension (e.g., “.01”). In this case, the end user could be interested in providing suitable alternate file names for meaningful representation. Moreover, since MS Office files (like - .docx, .doc, .xlsx, .ppt, .doc, .xls,) are handled by SharePoint differently, renaming those files would be useful if they are stored with correct file extension.

It would most helpful if the end user is able to provide suitable alternate name by appending the extension corresponding to the native application that generated the file / document.

You can achieve this by specifying (source file path, new file name) in an external metadata file or the batch descriptor file.

By using an external metadata file

4. Create a new folder hierarchy and import files into them during migration

One other specific usage scenario that you may encounter is to migrate all files in a folder to a new sub-folder in the destination location. This requirement can be easily achieved through “Batch Descriptor file” task option in DocKIT.

You can achieve this by stating the desired folder hierarchy under New Folder column while specifying batch file entries. Also, you have to set ‘No’ for ‘Do you want to create the top-level folder(s) included for import?’ in ‘Folder Options’ step in DocKIT Task Wizard.

5. Create a new document set by way of renaming folders on the fly and import files into them during migration
   

In order to leverage the use of newly introduced Document Set feature in SharePoint 2010, you would be interested in creating Document Set by using the existing folders during migration into SharePoint. DocKIT comes in handy to address this unique requirement.

To accomplish this, you have to specify the new folder name which has to be created as document set and the corresponding folder content type (created using ‘Document Set’ as its parent content type) to be used for creating the new folder as document set.

After creating the specified folder as document set, DocKIT will create its sub-folders as document sets and migrate the files under the respective sub-folders.

Adding custom document set content type to SharePoint library:

6. Rename files that already exist in SharePoint by providing alternate names

DocKIT provides the option to rename the files that are already available in SharePoint. It works in a similar fashion like new files that are imported into SharePoint, but for the Path column. The
Path column should be replaced by Destination Path column and should contain the file URL of the list item available in SharePoint.

You can use only batch file syntax to rename files that are already available in SharePoint. In the batch file, apart from the New Name column, only Destination Path and not the Path will be specified.

7. Rename folders that already exist in SharePoint by providing alternate names

Similar to renaming an existing file in SharePoint, DocKIT facilitates existing folder rename option. As indicated already, you can use only the batch file syntax to rename folders and files that are already available in SharePoint. In the batch file, you have to specify the New Folder column, apart from the Destination Path column.

Rename folders & files using folder & file naming rules

We will discuss about another interesting and useful option in DocKIT to rename folders / files based on patterns, remove illegal characters in folders/file names while migrating to SharePoint in the next blog post.

Read: DocKIT: Rename folders/files based on patterns using renaming rules

In continuation with the previous blog post that deals with the folder and file rename options, here we discuss how to deal with renaming of folders/files that contain illegal characters.

As said earlier, folders and files residing in legacy systems will have having special characters that are invalid (disallowed) in SharePoint. DocKIT facilitates this renaming by providing folder and file naming rules while migrating into SharePoint.

Certain users might also need migration of different versions of a file with their version named with certain patterns. DocKIT also supports this file version history creation seamlessly.

Rename folders & files using naming rules

This option is intended /recommended for renaming folders & files that have characters not accepted in SharePoint as part of their names.

SharePoint disallows certain invalid characters in the folder and file names. In order to overcome this behavior, you need to replace certain characters that appear in Windows folder & file names and are invalid folder / file characters in SharePoint libraries, with valid characters (all characters except ~ # % & { } * | \ : < > / ?) in SharePoint.

For this, you can use the “Folder & File Renaming Rules” feature in DocKIT. This feature in DocKIT uses ‘Regular Expressions’ technique that is widely used in software applications where character pattern matching is crucial.

Consider the following cases:

Example 1: Order of entries when constructing renaming rules

The Folder and File Renaming rule in DocKIT requires the rule entries to be correctly placed for successful and meaningful replacement of invalid characters. The order of the entries (sequence) is important depending on the name of the files and folders. For example, if you want dot and double underscore OR triple dots to be treated as a single entity, then a rule entry with this value has to be given first. After this, you should provide dot and single underscore OR double dots value as entry.

Example:

Equivalent Naming Rules:

Figure 1: Naming rule as specified in DocKIT application

Example 2: Usage of reserved characters

The ‘Regular Expressions’ technique used by DocKIT for pattern matching categorizes some of the operators as reserved and has its own meaning for the operators in the pattern syntax. In order to treat the reserved characters as normal characters like ‘A’, ‘1′, you have to use ‘\’ (backslash) as an escape sequence character in DocKIT Renaming rule as given below:

Create file version history by using certain naming patterns

We will discuss a situation where certain files in a folder have their names appended with version numbers. If you would like these version numbers to stack up nicely as version history of the file / document, this is what you can do.

Example 1: Uniform file naming pattern for file version

Example 2: Random file naming pattern for file version:

Note: Entries under the Find section should be in the same order as listed.

Figure 2: File Settings in DocKIT Task Wizard

To create version history, select ‘Create new version’ option in ‘File Settings’ step in DocKIT Task Wizard.

For more information, please refer the following help sections in the online help document. You can launch the help document from Help menu (Help -> Contents menu) in the application.

• DocKIT Features -> Folder & File Renaming Rules
• DocKIT Special Cases -> Rename Documents in File System

SharePoint Server 2010’s yet another new capability Document Sets facilitate managing documents as an entity based on their metadata (document property). The entity can be categorized based on the need or task which shares a common set of attributes and can be versioned as a whole (instead of individual documents) in SharePoint. The Content Type forms the underlying technique for Document Set and hence multiple document sets can be maintained in SharePoint.

As a natural course for a SharePoint migration product, our DocKIT for SharePoint 2010 product supports creation of Document Sets and classifying documents within them. With the existence of document sets in SharePoint, it optionally creates versions.

Here is a ready reckoner for migrating folders as document sets and importing the folder contents into them using DocKIT:

Let us consider the following folder structure maintained for plumbing system in the file system for migration into SharePoint.

• Fixtures
  • bathtub.dwg
  • coldwater.dwg
  • hotwater.dwg
  • piperack.dwg
  • shower.dwg
  • washsink.dwg
    

• Plan
  • drillguide.dwg
  • floorplan.dwg
  • pipingplan.dwg

1. In DocKIT, create an import task using Import folders, files and metadata to SharePoint Libraries (Explorer Mode) or Import folders, files and metadata to SharePoint Libraries (Batch File Mode) option.

2. In Import folders, files and metadata to SharePoint Libraries (Explorer Mode) option: Select the desired SharePoint site to be used as destination.

In Select items to import dialog, add the desired folders that you want to create as document sets in the destination SharePoint libraries.

In Import folders, files and metadata to SharePoint Libraries (Batch File Mode) option:

In Batch File Options step, select the desired batch descriptor file to be used as reference for import folders and files.

Note:Ensure that you specify the desired folders that you want to create as document sets in the destination SharePoint libraries, destination path.

3. Click Next to go to Metadata Options step. In this step, you can provide an external metadata file in (CSV/TXT/XLS) format to assign custom metadata for columns included in document set.

4. Click Next twice to go to System Dates step. This step will assist you to retain the original date field (Created Date, Modified Date) values as in source file system to destination SharePoint.

5.  Proceed with the subsequent steps to reach Content Type Settings step

6. In this step, under Folder Content Type Options, specify a content type that is derived from ‘Document Set’ content type. You can either enter a content type name or select a column in the external metadata file (that contains the content type name) to be used.

7. Proceed with the subsequent steps to reach Folder Options step.  

8. In this step, select a suitable folder option so as to create the added (top level / its sub-folders) folders as document sets in destination SharePoint library.  

9. Proceed with the subsequent steps to save and run the task.

10. On completion of the import process, the folders will be created as document sets with files imported underneath them. 

For further information on DocKIT for SharePoint 2010, please visit our product home page at:

http://www.vyapin.com/products/sharepoint/sharepoint-2010/dockit/sharepoint-migration.htm 

You can download a 30-day trial version from the above product web page.

The latest major release of our DocKIT software comprehensively addresses content migration from file shares and servers to SharePoint 2007 and SharePoint 2010. We have combined all features of DocKIT of SharePoint 2010 / 2007 v4.x and SPList Manager for SharePoint 2007 v1.x into DocKIT v5.0, along with a few other useful tools for content migration. This new version will be useful to users who are looking to migrate external data or contents to any SharePoint list type such as Document libraries, Picture libraries, Custom Lists, Tasks, Calendar, Contacts, Survey, Discussion Boards, Announcements, Links, Issue Tracking etc. using one comprehensive solution.

In a nutshell, you can do the following with respect to content migration and SharePoint metadata (list data) management using DocKIT v5.x:

• Migrate documents and metadata properties to SharePoint libraries.
• Import (or migrate) new data or metadata properties including file attachments to SharePoint lists
• Update data or metadata properties to existing items (including file attachments) / documents in SharePoint lists.
• Delete data or metadata properties in SharePoint lists.
• Manage existing data or metadata properties in SharePoint lists.

Let us look at DocKIT’s features in further detail below:

Migrate documents and metadata properties to SharePoint libraries

Import folders and documents along with metadata (stored in external metadata file and file properties) from file shares to SharePoint libraries using DocKIT’s explorer interface or using a simple batch descriptor file.

DocKIT allows you to associate metadata of documents stored in an external file, thereby eliminating the burden of manually entering values to the documents while checking in.

DocKIT associates file system properties (’Summary’ properties) - Title, Subject, Author, Category, Keywords, and Comments and applies it to the respective documents in SharePoint automatically during content migration. DocKIT imports file properties such as Manager, Company, Date Last Printed, Date Last Saved, Revision Number, Version, Word Count etc. and all other custom file properties in the case of MS-Office documents. DocKIT retains the original Created Date, Last Modified Date, Created By (Author) and Modified By (Editor) and assigns it to the respective SharePoint columns in a library.

Import data or metadata properties to SharePoint lists in bulk

Import data or metadata properties to a SharePoint List based on the line entries in the batch descriptor file.

DocKIT imports metadata and its associated file attachments to SharePoint lists such as Tasks, Calendar, Contacts, Survey, Discussion Boards, Announcements, Links, Issue Tracking, Custom Lists etc.

DocKIT can also update metadata properties (column values) for existing SharePoint list items by adding them as a new version or retain the existing version. You can also delete the items, including file attachments in the SharePoint list during the content migration task. Basically, you can manage list items residing in multiple SharePoint lists using a single batch file. 

DocKIT retains the original Created Date, Last Modified Date, Created By and Modified By fields to the equivalent column values in SharePoint for each list item. You can also update SharePoint’s built-in fields, namely - Content Approval Status & related Comments, Content Type, Created Date, Modified Date, Created By and Modified By fields for each folder and list item in a SharePoint list. 

Manage metadata in SharePoint lists

Update or Clean-up (Delete) SharePoint list items using DocKIT’s Explorer interface using the quick metadata management tool. You can specify the conditions based on which you would like the updates to happen. This is a useful tool to clean-up existing SharePoint list items before commencing a fresh migration task.

DocKIT’s unique features

DocKIT makes content migration to SharePoint much easier by providing granular control and automation using its unique feature set:

New tools bundled in DocKIT v5.x

Metadata Extractor tool, bundled in DocKIT v5.x, addresses the specific pain of generating or preparing metadata for your folders and documents in the file share. This tool can generate metadata for a list of files / documents residing in file shares or local drives. Mine out metadata from file summary properties and custom properties for various file formats. You can extract document properties associated with Microsoft Office and PDF and file properties from various file types such as JPEG, PNG, GIF, BMP, TIFF, HTML, TXT etc. This feature will be useful to auto-generate folder structures, file list and metadata in various file share locations that are to be migrated to SharePoint.

List Export tool generates a list of all list items (metadata properties) in a SharePoint list with their corresponding SharePoint URL and exports the results to a CSV file. The URLs along with the corresponding metadata properties will be useful to clean-up or delete items in a SharePoint list. This CSV file can be provided as input to manage your metadata properties in the SharePoint list.

We are excited about the new features in DocKIT v5.0 and hope you are two. Our SPList Manager for SharePoint 2007 customers upgrading to SharePoint 2010 can transition to DocKIT v5.x and enjoy the larger feature set.

You can download a 30-day free trial copy from the product web page and read more about the latest release at: http://www.vyapin.com/products/sharepoint/sharepoint-2010/dockit/sharepoint-migration.htm

Our much anticipated major release of DocKIT for SharePoint 2010 version 5.0 is finally here with the following new features included,

• Supports Microsoft SharePoint Server 2010 / Microsoft SharePoint Foundation 2010 and Microsoft Office SharePoint Server 2007 (MOSS 2007) / Microsoft Windows SharePoint Services v3.0 (WSS 3.0).
• Migrate contents to all SharePoint list types, including Document libraries, Picture libraries, Tasks, Calendar, Contacts, Survey, Discussion Boards, Announcements, Links, Issue Tracking, Custom Lists etc. using a single application toolset.
• Supports Business Connectivity Services (BCS), Managed Metadata, Document Sets, Document ID and other new features in SharePoint Server 2010.
• Built-in metadata extractor tool to auto-generate folder structures, file list and metadata in various file share locations that are to be migrated to SharePoint.

For further information on DocKIT for SharePoint 2010, visit our product home page at http://www.vyapin.com/products/sharepoint/sharepoint-2010/dockit/sharepoint-migration.htm

We have come out with a significant major release of our flagship SharePoint document migration tool, DocKIT for SharePoint 2010 / 2007.

DocKIT v4.0 now supports Microsoft SharePoint Server 2010 and Microsoft SharePoint Foundation 2010 in addition to its support to the earlier versions of Microsoft Office SharePoint Server 2007 (MOSS 2007) and Microsoft Windows SharePoint Services (WSS 3.0).

DocKIT in addition to its support to native SharePoint 2010 environment has come out with the following new features:

1. Extended support to Business Connectivity Services: Business Connectivity Services (BCS) is all about keeping the users connected to external databases and other sources of enterprise data (LOB systems). BCS is an evolution of Business Data Catalog (BDC) feature in SharePoint 2007, allowing the users to better integrate external data to the SharePoint portal. DocKIT through its support to BCS / BDC ensures that users can map their existing line of business applications into the SharePoint portal, thereby allowing the users to leverage on integrated information.
2. Support to Managed metadata: Managed metadata resembles the ‘Lookup Columns’ of SharePoint 2007. Unlike Look Up columns where it was impossible for the users to share metadata across site collections, Managed metadata is not site specific. Managed metadata has a centralized repository of pre-defined metadata that can be replicated for use throughout the SharePoint environment without having to define them every single time. DocKIT helps users retrieve the metadata values from ‘term sets’ associated with the Managed Metadata column.
3. Supports Document Sets: Document sets allow users to bundle related documents and treat them as a single entity. With DocKIT, the users can migrate multiple documents from file repositories and manage them better by assigning metadata to these ‘compound documents’ through an external metadata source.

DocKIT with its support to SharePoint 2010 features such as ‘Ratings Column’ data type, ‘Asset Library’ etc. would help organizations achieve a better collaborative environment by allowing the users to leverage on the extended features of SharePoint 2010.

The interesting information is the latest version of DocKIT supports both SharePoint 2010 and SharePoint 2007 environment within the same version of the product. This means our DocKIT customers do not have to invest twice (lower TCO) in a SharePoint document migration solution, one for each SharePoint version and can also quickly adapt / transition to the new version using the familiar interface.

For a free trial version of DocKIT for SharePoint 2010/2007 please visit our product home page at http://www.vyapin.com/products/sharepoint/sharepoint-2010/dockit/sharepoint-migration.htm.

Our flagship SharePoint migration solution, DocKIT is out with a beta version for SharePoint 2010. DocKIT for SharePoint 2010 fully supports SharePoint 2010 environment with the following additional new features:

1.Support to Business Continuity Services: Business Continuity Services (BCS) is an extension to the Business Data Catalog provision of Microsoft Office SharePoint Server 2007 to integrate external data to the portal. BCS with out-of-the box features, services and tools allows deep integration of external data both online and offline. BCS is all about connectivity and is all set to provide connectivity support to external databases and other data sources. DocKIT with its extended support to BCS would allow users to leverage on the external data source.

2.Support to Managed Metadata type: Managed metadata type is about having a centralized collection of pre-defined metadata and managing them independently from SharePoint columns. Metadata values are assigned by referring to the centralized metadata ‘repository ‘. DocKIT helps users assign metadata to the corresponding documents through a metadata file by retrieving the specific metadata value from the term set corresponding to the Managed metadata column.

3.Supports Document Sets: Document Sets are special containers allowing users to treat a document collection as a single entity. It is yet another ‘Folder Content Type’. DocKIT allows users to migrate documents from specific file locations and get them grouped as a document set. Assigning metadata to the document set could be easily done through an external metadata file.

4.Support to Asset Library and other SharePoint built-in columns such as Ratings: DocKIT extends its support to Asset library, a powerful content management feature in SharePoint 2010 allowing the users to share rich media assets without necessarily reproducing them. New SharePoint Built-In columns like Ratings column are also supported by DocKIT. Like the other SharePoint columns, ratings for documents can be updated using an external metadata file.

DocKIT ‘s new features are certain to extend the out-of-the-box functionalities of SharePoint 2010 and add value to enterprise content management at an enterprise level.

For a free trial version of DocKIT beta for SharePoint 2010, visit the product home page at http://www.vyapin.com/products/sharepoint/sharepoint-2010/dockit/sharepoint-file-migration-tool.htm

Our flagship SharePoint migration product, DocKIT for SharePoint 2007 has come out with a feature rich new version. This new version of DocKIT for SharePoint 2007 v3.9 is a significant release from the solution’s usability standpoint. Usability is something that we keep constantly improvising on.

Our DocKIT customers would now be bestowed with the following provisions with the new release:

1. Web Services Component Indicator: DocKIT has in it now a provision to indicate to the user about the existence of DocKIT Web Services component in the destination SharePoint server. The user has to specify the SharePoint site URL and its corresponding credentials to verify the existence of DocKIT Web Services component. It can be accessed from the Tools Menu of the application (Tools–> Test DocKIT Web Service). The new feature is expected to avoid any Web Services Component mismatch. DocKIT Web Services component indicator has been incorporated based on suggestions from some of our DocKIT’s customers.

2. Support for Forms Authentication: DocKIT now supports a Forms Authentication validating the users against the corresponding SharePoint environment. With this feature, DocKIT extends its support to websites configured with Forms Authentication validation type. DocKIT would automatically initiate Windows Authentication or Forms Authentication depending on the destination SharePoint site configuration settings.

We also added forms authentication to our SharePoint to Windows file system migration tool, SPListX quite recently: http://www.vyapin.com/blogs/index.php/2010/03/10/splistx-for-sharepoint-2007-v31-released/.

3. Elapsed Time Display: Users can view the time taken for a particular task to migrate documents to the destination SharePoint document library. DocKIT now keeps the user informed about the time involved in migrating documents and this would help the user in scheduling migration tasks.

4. Profile Manager Tool: Users now can define the User Profiles and their credentials through the Profile Manager tool. Windows-based credentials and Forms-Based credentials can be defined through this. Using this, the users can just select their corresponding credentials without manually entering them every time.

For a 30-day free trial version or to upgrade to the latest version of DocKIT software, visit the product home page at http://www.vyapin.com/products/sharepoint/moss-2007/dockit/sharepoint-2007-file-migration.htm.

One of the most common scenarios in SharePoint migration is the migration of File shares and folders from various sources – file servers, desktops and other storage devices. While the need for migrating files and folders is quite common, the task of getting this accomplished becomes very complex if there are a large number of files and folders to migrate. You will need an automated solution that will take care of some of these complexities. A large migration typically involves migrating and tagging tens of thousands of files (documents) from legacy file shares and folders into a new SharePoint repository while retaining the existing taxonomy or moving to a completely new taxonomy.
Most file system users will not be familiar with the concept of document metadata. They are more familiar with the term file properties. The concept of document metadata originates from Document Management Systems (DMS), where documents are stored and made searchable on a wider set of keywords and phrases. Document libraries in SharePoint are akin to Document Management Systems in the way they store documents and properties for search and retrieval. However, these properties need to be presented to SharePoint during migration in a certain manner that makes the search and retrieval more powerful and elegant. You may migrate / propagate metadata fields from the document properties stored within the file or using an external metadata file / database. Mapping and tagging existing file properties and adding additional properties to documents in SharePoint, especially for a well structured taxonomy can be a laborious process. Thus, migrating files and folders along with metadata needs an appropriate solution that will break down the migration into several different tasks (automated batch processes), with several different rules to process the files and folders.
Our SharePoint migration tool for File shares and folders, DocKIT, allows users to move folders and files along with metadata (using an external metadata file and file properties) from file system to SharePoint libraries based on the folders/files added by the user using the DocKIT user interface. Some of the supported metadata elements that typically define the source content are:

• Custom properties defined in the external metadata file
• Summary file system properties - Title, Subject, Author, Category, Keywords and Comments
• File properties such as Manager, Company, DateLastPrinted, DateLastSaved, RevisionNumber, Version, WordCount etc. in the case of MS-Office documents
• Original Created Date & Last Modified Date of source documents
• Author (Created By) and Editor (Modified By) of source documents
• Content Approval Status & Approval Comments

Please click the following link to know more about Vyapin’s DocKIT: http://www.vyapin.com/products/sharepoint/moss-2007/dockit/sharepoint-2007-file-migration.htm

The much anticipated release of SharePoint 2010 is on the cards. It’s almost official that the final cut would be released by June 2010. The beta version is already out and is certainly creating a buzz among the business community. The latest version that is coming up has flexible deployment options and has kindled our excitement as well. Several customers have inquired about our products’ compatibility with the latest version of SharePoint. Our answer to them is yes, we have all our products ready for the next big release of SharePoint.

Our reporting tool for SharePoint, ARKSP is ready for its major upgrade for the SharePoint 2010 release and should be up and running by next week. DocKIT, our flagship document migration solution is to follow suit with its upgraded version for SharePoint 2010. We have been getting a lot of queries from keen customers about DocKIT’s compatibility with SharePoint 2010 and we are not going to make them wait any further than the first week of Feb 2010. The SharePoint to file system migration solution, SPListX is gearing up for the release with additional ‘Search Criteria’ features, which, the users are going to find it extremely useful in pulling out precise ‘spot-on’ information. Our SharePoint list management solution, SPListM comes out with a beta version by March 2010. XPlica, our solution to your migration needs (from SharePoint to SharePoint libraries) would be the last to be released. We have scheduled it for April 2010. We are on full speed to keep our solutions abreast with the biggest release of SharePoint and to continue helping our customers leverage their SharePoint resources. The beta version of our products has already started to roll out and is being evaluated by some of our renowned customers. We have already started to listen to some excellent feedback from our customers and we are looking forward to keep the momentum going on. The next couple of months are going to be extremely significant for us as we deliver the latest versions to our customers and prospects.
Looking forward to an absorbing month ahead…