Throttling output to Sharepoint

[PuffinPanic]

This is the first time I've had to actually post a question on here, because this amazing community has already answered every question I've had so far. But this is an odd one that I've not been able to find an answer to. I think from what I have been able to find that I should be able to do what I need, but it is a bit beyond me.

Background:

We have an enquiry system (created by an external development company) that runs on sharepoint. I have been able to extract information from it using the sharepoint input tool, but I would now like to upload some further data in bulk. Sounded straightforward to me - I can see and work out how the sharepoint output tool works. However, I have been told by the developers of the enquiry system that uploading in bulk would (paraphrasing) break it because each update causes an event flag and a bulk upload would basically cause it to fall over. (My understanding, apologies for lack of technical details - I have no experience of developing sharepoint systems)

The question:

Is there a way that I can force the sharepoint output tool to only upload x number of records per minute?

(I can see that there is a throttle tool, but I'm not sure if I can combine that with the sharepoint output tool to somehow achieve my aim)

Details:

I have 2 fields, the key field for each record on the sharepoint list and the field I need updating.

The list of updates could be several hundred long. (At the moment the updates are being done manually, which is madness)

I don't have scheduler.

Any help would be greatly appreciated. Even if it's just to tell me 'in theory this is possible' in the first case.

Thanks

Puffin (not panicking at the moment)

[danilang · Accepted solution]

Hi @PuffinPanic 

I'm sure we've all wanted to throttle SharePoint at one point😀

You should be able to put the throttle tool just before your SharePoint Output tool.  It limits the records going through it to X/minute.   The default is 60/minute which gives about one per second.  Even SharePoint should be handle that insertion speed.

To see it in action, check out the attached workflow

With the container minimized the workflow runs in about 0.1 seconds.  With it maximized and the throttle tool in effect it takes about 7 seconds to run.  The message tool outputs the time that each record passes through it.  As you can see they are spaced about a second apart 

Dan

[PuffinPanic]

Thanks Dan,

That looks great. Thanks for the example workflow.

The only question left is about the sharepoint output tool - does it update to sharepoint as it receives the data, or does it wait until all data is received and then update?

If anyone could answer this for me, I would be eternally grateful. 

Thanks in advance,

PuffinPanic

[pedrodrfaria]

Hi @PuffinPanic 

It is my belief that It should be updating everything as it moves along with the workflow. If it waited for it to be finished and then update everything at once, the throttle would be useless in this case.

[PuffinPanic]

---

@pedrodrfaria wrote:

It is my belief that It should be updating everything as it moves along with the workflow. If it waited for it to be finished and then update everything at once, the throttle would be useless in this case.

---

Thanks for this confirmation. My concern was that throttling would indeed be useless as you suggest. I didn't want to go back to my line manager with this great solution, if it was going to fall over because I hadn't considered the potential issues at all stages.

But, now that I have this useful information to take back to our system designers I'm sure, after discussion with them, I will be able to do some testing with their co-operation to get this to work for us. 

Thanks both for your help, it is much appreciated.

PuffinPanic