Skip to main content

How to move multiple files in ssis and also rename simultaneously

There are two ways to achieve this.
1) We can move the flat files and then rename it.
2) While moving files itself, automatic rename should be done.

We will do the second type. The criteria is to rename the files while moving from source to destination.
So for that, we need FILE SYSTEM TASK to be included. Secondly since we need to move many files, we will use FOR EACH LOOP CONTAINER.

To fetch all the files, we can use FOR EACH LOOP task in SSIS.
In collection tab, we can select FOREACH FILE enumerator option for fetching files and we can change enumerator configuration
Folder option: Points to source where we need to fetch files.
Files: will give us idea whether we need to fetch all the files (*.*) or if we give extension like *.txt, it is going to fetch only  .txt files .
Once I give Source name in FOR EACH LOOP container, It is going to fetch all the files corresponding to that path.

Retrieve file name:
This option is used to let the variables mentioned in VARIABLE MAPPING tab, like how fetch the file and in what format.

This is supported by 3 options:
1) Fully qualified
2) Name and extension
3) Name only

1) Fully qualified: Fully qualified will assign path and file name with its extension to variable present in VARIABLE MAPPING.

2) Name and extension: If we use name and extension then, Only name and extension will be assigned to variable at VARIABLE MAPPING option

3) Name Only: Name only option enables variable at VARIABLE MAPPING option to fetch only name of file (Not even the extension)

Say example: If we have three text files in our source like txt1, txt2, txt3 and we need to fetch all three text files.

Now these files which are actually my source are in the path C:\Documents and Settings\swami\Desktop\test\source

Now I want to move these files to C:\Documents and Settings\swami\Desktop\test\dest.

My requirement is that while moving these files, I should rename these files and should place to destination folder 'Dest'.


Now at FOR EACH LOOP container, We will give source path where our files resides at Folder option from collection tab.


secondly I'm going to give *.txt in files position to retrieve only .txt files.

Retrieve file name is selected as Name and extension and only name and extension will be passed to any variable I assign to VARIABLE MAPPING tab.

Now I'm assigning a dummy variable named by me as 'Filename' to VARIABLE MAPPING tab.

Now variable Filename will take only Name of file and extension from given path.

ie,. Values of Filename taken by foreach loop container at first iteration during run will be only txt1.txt
and on second iteration, it will be txt2.txt and so on.. Iteration counts are handled by foreach loop container internally.

now for renaming, Im going to create one more variable with values being the path of source. ie., I create a variable named 'source' and value is C:\Documents and Settings\swami\Desktop\test\source.\

|||rly Im creating one more variable for destination named 'destination' and values assigned will be C:\Documents and Settings\swami\Desktop\test\dest.

Now I'm adding a FILE SYSTEM TASK inside foreach loop container.

I'm creating one more variable called 'sourcepath'.
Reason for creating this variable is  that 'souce' variable assign only source path name and filename variable have only files name with  its extension.

to integrate path and then corresponding file loop fetches with extension, I created this variable.
The expression for variable sourcepath is given as user::source+user::filepath.

ie,. on first iteration, Sourcepath will have value as C:\Documents and Settings\swami\Desktop\test\source\txt1.txt.
 and on second iteration, it will have values like  C:\Documents and Settings\swami\Desktop\test\source\txt2.txt. and so on..

I'm assigning this variable Source path to File System task in source column.

Similar things are done on destination also by creating new variable 'destinationpath' where its expression is given as user::destination+'newname_'+user::filepath

where filepath is txt1.txt, txt2.txt and so on variable destination is indicates to destination path.

After running package, The files are transferred with their name are changed. The complete snippets are shown follows:-



 


Comments

  1. Hi Dhinakaran S,

    Great article,
    and how to do the file name extend with date. In above example txt1_2012-05-10.txt,txt2_2012-05-10.txt and txt3_2012-05-10.txt.

    This is an urgent issue, Please anyone could help me on this.


    Thanks
    Anil

    ReplyDelete
    Replies
    1. How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Now

      >>>>> Download Full

      How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download LINK

      >>>>> Download Now

      How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Full

      >>>>> Download LINK W1

      Delete
  2. Hello Anil.

    In Foreach loop editor, Under Retrieve Filename, instead of Name and Extension, you can use Name Only and user::filename variable will hold only the name of file.
    Now use this variable to append the date and extension at last.
    Example: if ur filename is abc.txt, then variable user::filename will hold 'abc' and then in FILESYSTEM task,
    Use this to append the date function as I explained earlier in same post with Extension ('.txt' in your case).

    Hope you are cleared.

    ReplyDelete
  3. I think I'm about to go crazy! Thanks so much for the example... I've replicated it (3 times from scratch so far)...but it won't run successfully. Here is the error I get. Any idea what I'm doing wrong? Oh... so frustrated.

    TITLE: Package Validation Error
    ------------------------------

    Package Validation Error

    ------------------------------
    ADDITIONAL INFORMATION:

    Error at File System Task: Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\Source\" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".

    Error at File System Task [File System Task]: An error occurred with the following error message: "Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\Source\" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
    ".

    Error at File System Task: Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\dest\_" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".

    Error at File System Task [File System Task]: An error occurred with the following error message: "Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\dest\_" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
    ".

    Error at File System Task: There were errors during task validation.

    (Microsoft.DataTransformationServices.VsIntegration)

    ------------------------------
    BUTTONS:

    OK
    ------------------------------

    ReplyDelete
    Replies
    1. Hello Jim,

      This error means, that you have not configured your variable properly in FILE SYSTEM task. Please check the issue what variable you have used.

      Delete
    2. I am also facing same error and couldn't resolve it.

      Delete
  4. Nice information. Thanks for sharing content and such nice information for me. I hope you will share some more content about. Please keep sharing!
    big data training in chennai

    ReplyDelete
  5. Your blog is very interesting. And the point describing the information very effectively.
    Big Data Institutes in Pune

    ReplyDelete
  6. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    Best PHP Training Institute in Chennai|PHP Course in chennai

    Best .Net Training Institute in Chennai
    Software Testing Training in Chennai
    Blue Prism Training in Chennai
    Angularjs Training in Chennai

    ReplyDelete
  7. I think there are actually more and more useful aspects of SSIS and has more functions in order to solve complex IT problems.

    SSIS Upsert

    ReplyDelete
  8. Thanks for the nice Post. I must suggest your readers to Visit Big data course in coimbatore

    ReplyDelete
  9. How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Now

    >>>>> Download Full

    How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download LINK

    >>>>> Download Now

    How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Full

    >>>>> Download LINK X2

    ReplyDelete

Post a Comment

Popular posts from this blog

Zip/Unzip multiple files and also include password for zipped file using SSIS

We have many scenario that we need to Zip many files which we come across and then so some operations like either sending it as a email or just moving zipped file to some other destinations etc. But we were using manual method to zip multiple files. In this post, I tried to create a package which will zip multiple files using SSIS. Here for Zipping files purpose, I'm using 7-ZIP which is free software available in google sites. Download files and install onto your system. First let me show how to Zip on file and later I will show how to zip multiple files using SSIS and 7Zip tool. Compressing Single file. Here I'm trying to Zip one single flat file which is of 40MB size. I kept this file in C:\Documents and Settings\\Desktop\test\source folder. Now to compress this file, I will open my SSIS and I'm dragging and dropping EXECUTE PROCESS TASK from Control Flow. Now right click on Execute Process task and go for edit and select Process option. In process tab,

SSIS: The Value Was Too Large To Fit In The Output Column

I had a SSIS package where I was calling a stored procedure in OLEDB Source and it was returning a “The Value Was Too Large to Fit in the Output Column” error. Well, My Datatype in OLEDB source was matching with my OLEDB Destination table. However, when I googled, we got solutions like to increase the output of OLEDB Source using Advanced Editor option . I was not at all comfortable with their solution as my source, destination and my intermediate transformation all are having same length and data type and I don’t want to change. Then I found that I was missing SET NOCOUNT ON option was missing in Stored Procedure. Once I added it, my data flow task ran successfully.