Python SDK: Create temporary data table and output to downstream tools

[Chriszou]

Hi,

I am new to the Python SDK and trying to use it to develop a few plugins.

Based on my understanding, the ii_init and ii_push_record function in the IncomingInterface class process data line by line. However, is it there a way to store incoming data into a temporary table, apply logic and then ouputput the final product to downstream (line by line or all at one time).

The reason I am asking is that. Some of our client's data required to be processed in a table instead of line-by-line. For example, an upstream tool pass a 4 by 4 table (4 columns and 4 rows) to the Python plugin. The plugin will add a column and conduct calculation based of data in current row, previous row and next row (just like a multi-row formula tool in Alteryx but much more complex than that). Is there an easy way to read all incoming data to a temporary table and then output the data to downstream tools ?

Another question I have is around the "in_record" argument in the ii_push_record function. Does anyone know how to create an instance of that object using the Python SDK codes (I tried to found reladted documentation but no luck yet). Attached is pic is the ii_push_record function I am referring to.

Thank you very much for all your help !

Thank you very much.

[Chriszou]

I think I figured out how to write upstream data in to a array (which can be served as the temporary table) using below 

snippet codes from the output tool sample. However, I am still not sure about how to output this list to a downstream tool. Does anyone know ?  Thanks. 

[NeilR]

As you suggest, you should be able to add incoming records into a data structure of your choice (data frame, array, etc) in  ii_push_record (called when an input record is being sent to the plugin).  Then you can do your data processing to the complete array and output your transformed records in ii_close ( called when the incoming connection has finished passing all of its records).

[NeilR]

As to your other question about how to handle the in_record argument, this blog post gives an example...

    def ii_push_record(self, in_record: object) -> bool:
        """
        Responsible for pushing records out
        Called when an input record is being sent to the plugin.
        :param in_record: The data for the incoming record.
        :return: False if method calling limit (record_cnt) is hit.
        """
        # Copy the data from the incoming record into the outgoing record.
        self.record_creator.reset()
        self.record_copier.copy(self.record_creator, in_record)
                                               
        if self.parent.input_field.get_as_string(in_record) is not None:
            url = self.parent.input_field.get_as_string(in_record)
            article = Article(url)
            article.download()
            article.parse()
            article.nlp()
            result = article.summary
            self.parent.summary.set_from_string(self.record_creator, result)          
            out_record = self.record_creator.finalize_record()
 
        # Push the record downstream and quit if there's a downstream error.
        if not self.parent.output_anchor.push_record(out_record):
            return False
 
        return True

[Chriszou]

Hi Neil,

Thanks for all your guidance. I was able to figure a way to output the temp table to downstream tools know. Especially like your idea to process and output data in the ii_close method. Again, thanks for all your inputs ! 

[MichaelCh]

---

@NeilR wrote:

As you suggest, you should be able to add incoming records into a data structure of your choice (data frame, array, etc) in  ii_push_record (called when an input record is being sent to the plugin).  Then you can do your data processing to the complete array and output your transformed records in ii_close ( called when the incoming connection has finished passing all of its records).

---

I've reached out to @TashaA to see if we can put some work in the backlog to extend the SDK with the record caching classes that our C++ tools use for this kind of design.

[Chriszou]

That will be awesome ! Thank you so much guys ! 

[RuchikaMangla]

Hi @Chriszou 

I am also looking for similar thing. I need to store data into a data frame. So, could you please help me out? Thanks