Skip to main content

How to Copy or Move Multiple Files from One Folder to Another Folder using Talend

Hello all,

In this Post, I will explain how to move Multiple Files from One Folder (Say Source) to Other folder (Say Destination).

This Post will also helps you to understand How to Declare Variable and Use it.

To Declare a variable, We are go to use Contexts option in repository.

Lets say we have two .txt files in Path D:/Source/. My Requirement is to move the files from Source Folder (D:/Source/) to Destination Folder (D:/Dest/).

Step 1: Open a New job
Step 2: Now right click and Create a New Contexts from Repository. Give some Name and give Next.




Step 3: Now Fill in the Source Directory Details where the loop on files should happen as shown in the snippet and give finish.




Step 4: Now Context is created and The values will be changing based on each file in Folder.

Step 5: Click and Drag the context from Repository to Context Job Window below the Job Designer.



Step 6: If we Expand the Contexts, We can find the variable SourcePath is holding value the Path of Source Folder and Also, Script Code is generated.

This Script code is what we are going to use in coming process.



Step 7: In Order to iterates on files or Folder, We are going to use  tFileList component. This  tFileList component is found under Files>Management from the Palette on Right Side.

tFileList component acts as a Foreach Loop Container with File Enumerator as in SSIS.
Click and Drag the  tFileList Component to designer Space Since we need to work on multiple files from Source Path D:/Source/ We will configure  tFileList component to fetch each file one by one.

Now Since we are working on only TEXT Files/Flat Files, I will configure to fetch only .txt files from the folder. Any other files other than txt files will be ignored.

This can be done by simply adding "*.txt" in FileMask option under Files Block in  tFileList component.
Don't forget to use " " Double Quotes. Its a Mandatory.


Since variable FilePath which is already holding path of the Source, We simply can give context.FilePath in Directory option in component which now tFileList understands which folder it should work.

Also FileListType is selected as FILES from the Drop Down.
To Summarize, The values to be configured in tFileList components are 

Directory: context.FilePath 
FileListType: Files
Files: "*.txt"

Step 8: Now our tFileList  is configured to fetch all the files. Now to copy the files from D:/Source/ to D:/Dest/ folder, we will drag and drop tFileCopy Component from Files>Management under palette.

tFileCopy Copies a source file into a target directory and can remove the source files if required.

Also we will first connect our tFileList component to tFileCopy component.
(Note: The connection will be iterate as shown in snippet)



Now, we lets configure tFileCopy to copy the file.

  • We know that path for the file is present in variable context.FilePath. But to get the exact file name, we should write a small command in tFileCopy.
  • This can be done by simply adding ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) in filename of tFileCopy component.
  • Configure the Destination Directory as shown in the snippet.



And Thats it. Run the job and check the file move.


Comments

  1. Thank you. Detailed description helps. :)

    ReplyDelete
  2. Hello Dhinakaran, How do you send two different files 1) Error file 2) Successful files to two different folders -- let us say both are coming from above process -- ( one folder multiple files)

    ReplyDelete
  3. Hello Dhinakaran, thanks. How can i do to list all files copied and send their names in a mail??

    ReplyDelete
  4. Nice information. Thanks for sharing content and such nice information for me. I hope you will share some more content about. Please keep sharing!
    big data training in chennai

    ReplyDelete
  5. I feel this is a very good information and post about SSIS and many other useful aspects.On the other hand it is a very useful tool to solve complex IT problems.

    SSIS Upsert

    ReplyDelete
  6. Thanks for such informative Post. I must suggest your readers to Visit Big data course in coimbatore

    ReplyDelete
  7. Really very nice information,thank you for sharing your thoughts with us .
    thank you....

    Big data online training

    Big data hadoop training

    ReplyDelete
  8. with us to get some idea about that content.Thanks for sharing
    Msbi Online Course
    Msbi Online Training

    ReplyDelete
  9. Great article. Thank you for sharing such informative content. If you are looking for Digital marketing company in Dubai then go through it.

    ReplyDelete

Post a Comment

Popular posts from this blog

BIG Data, Hadoop – Chapter 2 - Data Life Cycle

Data Life Cycle The data life cycle is pictorial defined as show below:     As we see, in our current system, we capture/ Extract our data, then we store it and later we process for reporting and analytics. But in case of big data, the problem lies in storing and then processing it faster. Hence Hadoop takes this portion, where it stores the data in effective format (Hadoop distributed File System) and also process using its engine (Map Reduce Engine). Since Map Reduce engine or Hadoop engine need data on HDFS format to process, We have favorable tools available in market to do this operation. As an example, Scoop is a tool which converts RDBMS to HDFS. Likewise we have SAP BOD to convert sap system data to HDFS.

SSIS: The Value Was Too Large To Fit In The Output Column

I had a SSIS package where I was calling a stored procedure in OLEDB Source and it was returning a “The Value Was Too Large to Fit in the Output Column” error. Well, My Datatype in OLEDB source was matching with my OLEDB Destination table. However, when I googled, we got solutions like to increase the output of OLEDB Source using Advanced Editor option . I was not at all comfortable with their solution as my source, destination and my intermediate transformation all are having same length and data type and I don’t want to change. Then I found that I was missing SET NOCOUNT ON option was missing in Stored Procedure. Once I added it, my data flow task ran successfully.