How to load a custom dataset with tf.data [Tensorflow]

Hi great video, im trying to set up a dataset for multiclass classification. Would this process work for that?

Hi Daniel could you please share a similar excercise on a tabular data with mixed data types where we can have the pre processing steps as well in the tensorflow instead of any other platform. I tried to follow the tensorflow documentation but they did not clarify kuch as they were working one dataset but failing on the other. A guide similar to this will be helpful

This is fantastic, but I can't seem to figure out how to use my new tfRecords in the mnist DC Gan tutorial. Do you know if this is possible?

First of all, thank you Daniel for this excellent resource. Like others I too found the tensorflow documentation very difficult to understand. Secondly, please do a video walk-through of the code for implementing tf.keras.ctc_decode as well.

Thanks a lot!
The only update needed in tf2 for createdataset.py(in github) would be "tf.python_io.TFRecordWriter" to "tf.io.TFRecordWriter"

Thank you and Subscribed ! :)

Awesome!

How to create text based tensorflow dataset?

Thank you so much for uploading this! I've been stuck for a few days trying to import pictures for my thesis and the documentation on the TF lousy...

Thank you it was very helpful but i have a question. In this video you demonstrated how to transform a dataset of images into TFRecords. But i'm having troubles transforming a dataset of videos into TFRecords. Do you have any idea how to define the features for this task?

Thank you for taking the time to make this video. It was exactly what I needed in order to start my development, like a spark to start an engine. I wasn`t able even to start a simple training. Now I discovered, for example, that in order to obtain smaller records files, you can use tf.gfile.GFile to encode the image. Thanks again!

you are a absolute legend Thank_You

This video has won a subscriber. I was literally pulling my hairs out. Thanks a million.

Hi, how do I create a tensorflow dataset (tfds) with pictures that I alread have? Currently have around 16,000 images on my computer, but I don't know how to use them with the tensorflow code. I'm doing a binary classification testing wether or not there is a human in the picture. Thank you

Great video Daniel. Very clear. Would've loved to hear more depth into the TF specifics like the AdamOptimizer, convolution/max pooling, etc. But solid work.

Hello Daniel, I have dataset which contains very similar features, I am trying to recognize granite tiles, when I use this model my accuracy stuck at %8.33. I couldn't find your mail, I would be very happy if you help

Does this work for video data sets? Haven’t completed the vid yet

Hi Daniel,
I'm new in tensorflow.
I can't understand that there is only 2 classes (Cat and Dog), why the num_classes = 3 ?(in the github)
Could you tell me the reason? thanks!

Hi Daniel, I find other tensorflow programs really hard to understand as data preparation and modelling was written in a single file. Here it is separated using tfrecords. Thank you so much!!. I am trying to enhance it to include Tensorboard now. Do you have any TensorBoard samples/examples to share?

I Search this for not Image Data =(

Amazing! Do you have the same example of tf record and training with tensorflow for object detection (detecting more than one label in one image)?

Hi Daniel,
Thanks for the videos. Really helpful. I believe you have very good experience so allow me to ask some questions. Let's say I want to make a binary classifier. What's the minimum number of pictures I should get to train a (Convolutional Neural Net) classifier and get above 90% accuracy. Also in case of multi-classification, what's the minimum number of pictures I should have in each class to get also above 90% accuracy. Also another question: Does increasing the number of classes affect negatively the accuracy? Last question: Let's say I don't have enough number of pictures for CNN classifier. Do you suggest other methods?
Many thanks for your support

Hi Daniel, Thank you so much for your video. Could you please take a look at these tries and let me know what i am doing wrong. As you said, it is not well documented. Please see below the code that works with tf.data.TextLineDataset and fails with tf.contrib.data.make_csv_dataset.

This is the CSV file (3 images that were copied to the local directory, once the code is working, they will be moved back to gs://):

image,label,feat1,feat2
image1.PNG,1,x1,y1
image2.PNG,2,x2,y2
image3.PNG,3,x3,y3

This code that works:

import tensorflow as tf
tf.reset_default_graph()
dataset = tf.data.TextLineDataset(‘file1.txt’).skip(1)
dataset = dataset.map(lambda row: tf.string_split([row],delimiter=”,”).values)
dataset = dataset.map(lambda row: tf.image.decode_png(tf.read_file(row[0]), channels=3))
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
print(sess.run(next_element))

The code that doesnt work:

import tensorflow as tf
tf.reset_default_graph()
dataset = tf.contrib.data.make_csv_dataset(‘file1.txt’,batch_size=1,header=True)
dataset = dataset.map(lambda row: tf.image.decode_png(tf.read_file(row[‘image’]), channels=3))
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
print(sess.run(next_element))

I get the following error:

ValueError: Shape must be rank 0 but is rank 1 for 'ReadFile' (op: 'ReadFile') with input shapes: [1].
What am I missing (syntax? access to the tensor values?)

Thanks, eilalan

very useful, thanks a lot

How many epochs are used in this algorithm? I dont see the number of epochs defined?

'image_raw': _bytes_feature(img.tostring())
in this line why you are converting img.tostring() .??

Hi Daniel, thank you for this video!

Sorry for this question, but how I can run this model after training with some image?

Hi Daniel,
thank you very much for making this video and for publishing the code on github! this is exactly what I was looking for

I'm new to python and neural networks and have a problem:
When I'm running create_dataset, I get this Error: [Errno 2] No such file or directory

I saved the folder "PetImages" into my Project folder, which I suppose is my current working directory
I also used the same code you did.
what did I do wrong? do I have to save the folder somewhere else?

Sorry to bother you with something like this, but I would be glad, if you could help me

Fantastic thank you!

It is helpful. Thanks for it! :)

Hi, Daniel.
This video is great and helpful. Thank you. And I have a problem about the filenames. Should we put the file path before the filenames? When I run my code, I am trying to print the images and labels and got this 'InvalidArgumentError: Expected image (JPEG, PNG, or GIF), got empty file'.
import tensorflow as tf
from PIL import Image
import numpy as np
import os

def train_input_fn():
filenames = ["mytrain.tfrecords"]
dataset = tf.data.TFRecordDataset(filenames)

def parser(record):
keys_to_features = {
"image_data": tf.FixedLenFeature((), tf.string, default_value=""),
"date_time": tf.FixedLenFeature((), tf.int64, default_value=0),
"label": tf.FixedLenFeature((), tf.int64,
default_value=tf.zeros([], dtype=tf.int64)),
}
parsed = tf.parse_single_example(record, keys_to_features)

image = tf.image.decode_jpeg(parsed["image_data"])
image = tf.reshape(image, [128, 128, 3])
label = tf.cast(parsed["label"], tf.int32)

return {"image_data": image, "date_time": parsed["date_time"]}, label

dataset = dataset.map(parser)
dataset = dataset.shuffle(buffer_size=10000)
dataset = dataset.batch(32)
dataset = dataset.repeat(1)
iterator = dataset.make_one_shot_iterator()

features, labels = iterator.get_next()
return features, labels

images, labels = train_input_fn()

init_op = tf.global_variables_initializer()
# I am trying to print the images and labels
with tf.Session() as sess:
sess.run(init_op)
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord = coord)
for i in range(230):
image, label = sess.run([images, labels])
img = Image.fromarray(image, 'RGB')
img.save(cwd+str(i) + '_''Label_'+str(l)+'.jpg')
print(image, label)
coord.request_stop()
coord.join(threads)

Very helpful video!!
You're certainly true about the Tensorflow doc haha
Instead of struggling through all the pages I'd recommend your video
Nice work :)

Hi Daniel, Thanks for this great video. I am trying to make a GAN network with custom dataset. Can I use the train.tfrecord for input data for GAN network? If yes, how can I load the dataset for GAN

Thank you so much. This presentation helped me to get myself going. While trying your example I observed that tf.identify() was used. May I know what is the purpose of tf.identity() in the context of this example? Thank you so much

Sorry to bother you again Daniel,
I got everything to work but Im getting an accuracy of around 95% staright off the bat, which to me seems unreasonable. Any chance that this could be?
Also you mentioned in the video that you printed the images, if possible could you please put that on your github.
Thank you :)

Thank you so much. After more than 20 years of developer experience, TF makes me feel that I know nothing, I feel wasted.

Hi, how can I load png images to a tf dataset pipeline without going through the fuss of making it to string and back again. Also is there a easier way to create tf. Records?

This is great stuff, thank you! :) I'm a bit of a python newbie (more of a coding newbie in general), and for my Masters I'm trying to get a basic deep learning system set up through Tensorflow to learn from a custom dataset. I'm learning from a pluralsight tutorial on basic Tensorflow usage, half of which I don't actually understand but am hoping I can make some tweaks to get it working without having a 100% understanding of what every step is doing.
I'm preparing image files for the dataset (non-standard size of approx 25x256 pixels, greyscale), and hoping I can find a fairly easy way to use them in the tutorial code instead of the MNIST dataset already being used.
Should I be able to use the code from your video to prepare the files and then load them into the tutorial code I'm working with (making a few changes to account for the different size of the images) and have it work properly? Or is it not as simple as that? Sorry if this is a stupid question.

Im sorry if this seems like a rather trivial question, I am very new too both python and tensorflow, but is "addr" and "vaule" supposed to be replaced with actual values and addresses. I have a gut feeling they shouldn't be, in which case I dont really understand when the algrothim loads the images.

Thank you so much for the video, and for taking the time!

does it work with more than just 2 types of images?
i want to create a dataset with 26 types of images

For more than two clasess what changes should be done in above program

Hey Daniel, Awesome vid , thank you !
I have a question : image = tf.decode_raw(parsed["image_raw"], tf.uint8)
Why don't you set tf.float32 as an output format directly ?

I found Tensorflow documentation quite hard to follow in general

You look like an absolute unit

Very helpful!

i was in the edge of having a mental breakdown until i saw your video.

Thank you so much. I'm really interested in this video. Hope to see more videos that explain tensorflow from you in the future! ^^

Your video is really nice. I need help in setting labels. I want to set string type labels up to 26. You were just setting 0 and 1. I want to change that. So How can I do that in code?? As I have not worked much in python...