Nsryjdtyk

I am interested in using ImageDataGenerator in Keras for data augmentation. But it requires that training and validation directories with sub directories for classes be fed in separately as below (this is from Keras documentation). I have a single directory with 2 subdirectories for 2 classes (Data/Class1 and Data/Class2). How do I randomly split this into training and validation directories

    train_datagen = ImageDataGenerator(

    rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True)



    test_datagen = ImageDataGenerator(rescale=1./255)



    train_generator = train_datagen.flow_from_directory(

    'data/train',

    target_size=(150, 150),

    batch_size=32,

    class_mode='binary')



   validation_generator = test_datagen.flow_from_directory(

    'data/validation',

    target_size=(150, 150),

    batch_size=32,

    class_mode='binary')



   model.fit_generator(

    train_generator,

    steps_per_epoch=2000,

    epochs=50,

    validation_data=validation_generator,

    validation_steps=800)

I am interested in re-running my algorithm multiple times with random training and validation data splits.

Thank you guys! I was able to write my own function to create training and test data sets. Here's the code for anyone who's looking.

import os

source1 = "/source_dir"

dest11 = "/dest_dir"

files = os.listdir(source1)

import shutil

import numpy as np

for f in files:

    if np.random.rand(1) < 0.2:

        shutil.move(source1 + '/'+ f, dest11 + '/'+ f)

Unfortunately, it's impossible for the current implementation of keras.preprocessing.image.ImageDataGenerator (as for October 14th, 2017) but as it's a really requested feature I expect it to be added in the nearest future.

But you could do this using standard Python os operations. Depending on the size of your dataset you could also try to first load all images to RAM and then use a classical fit method which could split your data randomly.

You will need to either manually copy out some of your training data and paste it into a validation directory, or create a program to randomly move data from your training directory to your validation directory. With either of these options, you will need to pass in the validation directory to your validation ImageDataGenerator().flow_from_directory() as the path.

Details for organizing your data in the directory structure are covered in this video.

https://stackoverflow.com/a/52372042/10111155 provided the easiest way: ImageDataGenerator now supports splitting into train/test from a single directory with subdirectories directly.

This is copied directly from that answer with no changes. I take no credit. I tried it and it worked perfectly.

Note that train_data_dir is the same in the train_generator and validation_generator. If you want a three-way split (train/test/valid) using ImageDataGenerator, the source code will need to be modified --- there are nice instructions here.

train_datagen = ImageDataGenerator(rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True,

    validation_split=0.2) # set validation split



train_generator = train_datagen.flow_from_directory(

    train_data_dir,

    target_size=(img_width, img_height),

    batch_size=batch_size,

    class_mode='binary',

    subset='training') # set as training data



validation_generator = train_datagen.flow_from_directory(

    train_data_dir, # same directory as training data

    target_size=(img_width, img_height),

    batch_size=batch_size,

    class_mode='binary'

    subset='validation') # set as validation data



model.fit_generator(

    train_generator,

    steps_per_epoch = train_generator.samples // batch_size,

    validation_data = validation_generator, 

    validation_steps = validation_generator.samples // batch_size,

    epochs = nb_epochs)

Here's my approach:

# Create temporary validation set.

with TemporaryDirectory(dir=train_image_folder) as valid_image_folder, TemporaryDirectory(dir=train_label_folder) as valid_label_folder:

    train_images = os.listdir(train_image_folder)

    train_labels = os.listdir(train_label_folder)



    for img_name in train_images:

        single_name, ext = os.path.splitext(img_name)

        label_name = single_name + '.png'

        if label_name not in train_labels:

            continue

        if random.uniform(0, 1) <= train_val_split:

            # Move the files.

            shutil.move(os.path.join(train_image_folder, img_name), os.path.join(valid_image_folder, img_name))

            shutil.move(os.path.join(train_label_folder, label_name), os.path.join(valid_label_folder, img_name))

Don't forget to move everything back.

You solution worked, thanks.

   import os

   import shutil

   import numpy as np



   sourceN = base_dir + "\train\NORMAL\"

   destN = base_dir + "\val\NORMAL"

   sourceP = base_dir + "\train\PNEUMONIA"

   destP = base_dir + "\val\PNEUMONIA"



   filesN = os.listdir(sourceN)

   filesP = os.listdir(sourceP)       



   for f in filesN:

       if np.random.rand(1) < 0.2:

       shutil.move(sourceN + '\'+ f, destN + '\'+ f)



   for i in filesP:

       if np.random.rand(1) < 0.2:

       shutil.move(sourceP + '\'+ i, destP + '\'+ i)



   print(len(os.listdir(sourceN)))

   print(len(os.listdir(sourceP)))

   print(len(os.listdir(destN)))

   print(len(os.listdir(destP)))