Genii Weblog

Out of Notes Part 5: To SharePoint

Wed 24 Jul 2013, 12:51 PM



by Ben Langhinrichs
This is the fifth in my series, Out Of Notes (see Part 1 - The Challenge, Part 2 - Capsule Views, Part 3 - ePublish from XPages and  Part 4 - Linked Collections), which shows how the Midas Rich Text LSX can be used to take data out of Notes either temporarily or permanently. In this part, I show how to export an entire database to SharePoint via CSV (comma-separated values), including both rich text fields and all other fields, as well as all attachments and images. This code has been used already by early adopter customers to export many gigabytes of data to SharePoint. In a very short video, I demonstrate how easy this can be with Midas V5. More information and the script used are at http://mini-lessons.info/Midas12 
 
Note: Accurate closed captions have been provided, and if English is not your first language, you might want to turn on the auto-translate. I also advise turning on HD and watching full screen so the captions show under the video.
 

Copyright © 2013 Genii Software Ltd.

What has been said:


1028.1. Lance Spellman (10/24/2013 09:36 AM)

Ben,

What have your customers done to import this data? Are they using custom .NET programs to work with the CSV file, using Powershell, going through the UI to import?

The issues I see are not with the export, but the proper way to structure the SP side to receive the import. Importing via spreadsheet (with the CSV) creates brand new lists with datatypes sometimes chosen arbitrarily (date fields not recognized as dates and turned to text, etc...). Once a large list is created like this, it is hard to modify (over 5,000 records). Index fields can't be created, filters are hard to setup.

So the preferred scenario is to have an empty list structure defined with indicies prior to importing data when you're talking about large data sets.

Given that, using SP web services and a piece of java code in ND that leverages it, you can get exactly what you want out of Notes with assured datatype translations and either go to a pre-defined list, or even create it via the webservices at runtime.

Thoughts?


1028.2. Ben Langhinrichs (10/24/2013 09:56 AM)

Excellent questions, Lance, and I don't exactly have the answers. I have reached out to a SharePoint person to help me better understand the choices. The customers I have had were not very forthcoming about their side of it, though as I understood it, they were pre-defining the fields and types and then importing to those so that SP would know what types were coming in. I would create a small sample, then they would set up SharePoint to receive those fields, then they could run the full database.

The biggest problems to be solved (why they came to me) were 1) how to ensure that the rich text was rendered with good fidelity and with links/images/etc where they wanted them, and 2) how to import from a Notes document library which could have multiple attachments in one file to a SharePoint document library which would only have one per "file". Midas handles these two scenarios differently, but you won't get the rich text fidelity or flexibility via Java.

Of course, the web services approach is one reason why I added JSON generation to Midas as well, so you can retrieve data that way and still get the fidelity and flexibility.

I'm not entirely sure I understand the gist of your last paragraph, so I'm not sure I addressed it fully. I'd be happy to continue that discussion on-line or off-line.


1028.3. Lance Spellman (10/24/2013 10:37 AM)

Thanks Ben,

I assumed right away that the Midas involvement was to handle the pesky RT fidelity, since there is no other good way (shameless plug for you)!!!

If you do hear anything about how they then handled the import, I'd be curious :-)

Go Midas!