I intend to implement expandable
CNN by using
Tylor non-linear expansion in
keras. I used
cifar-10 dataset for input data. I looked into the basic concept of Tylor series and tried using Tylor non-linear expansion for input tensor, but the code that I sketched not perfectly fit for a computational graph I am trying to use. I am not sure how to represent the weight after input tensor expanded by tylor non-linear expansion. Can anyone give me a possible idea of how to expand
CNN with Tylor non-linear expansion? How to efficiently do Tylor non-linear expansion on CNN? any thought?
computational graph of expandable CNN
Here is the computational graphic model for expandable CNN that I am trying to implement:
x is input,
alpha is coefficient,
p is power,
w is weight
import tensorflow as tf import numpy as np import keras from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Dropout, Flatten from keras.datasets import cifar10 from keras.utils import to_categorical (train_imgs, train_label), (test_imgs, test_label)= cifar10.load_data() output_class = np.unique(train_label) n_class = len(output_class) nrows_tr, ncols_tr, ndims_tr = train_imgs.shape[1:] nrows_ts, ncols_ts, ndims_ts = test_imgs.shape[1:] train_data = train_imgs.reshape(train_imgs.shape, nrows_tr, ncols_tr, ndims_tr) test_data = test_imgs.reshape(test_imgs.shape, nrows_ts, ncols_ts, ndims_ts) input_shape = (nrows_tr, ncols_tr, ndims_tr) train_data = train_data.astype('float32') trast_data = test_data.astype('float32') train_data //= 255 test_data //= 255 train_label_one_hot = to_categorical(train_label) test_label_one_hot = to_categorical(test_label) def pown(x,n): return(x**n) def expandable_cnn(input_shape, output_shape, approx_order): inputs=Input(shape=(input_shape)) x= Dense(input_shape)(inputs) y= Dense(output_shape)(x) model = Sequential() model.add(Conv2D(filters=32, kernel_size=(3,3), padding='same', activation="relu", input_shape=input_shape)) model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu')) model.add(MaxPooling2D(pool_size=(2,2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512, activation='relu')) model.add(Dropout(0.5)) for i in range(2, approx_order+1): y=add([y, Dense(output_shape)(Activation(lambda x: pown(x, n=i))(x))]) model.add(Dense(n_class, activation='softmax')(y)) return model
but when I ran the above model, I had compile error. I assume that the way for Tylor non-linear expansion for CNN model may not be correct. Also, I am not sure how to represent weight. How to make this work? any possible idea of how to correct my attempt?
I am expecting to extend CNN with Tylor non-linear expansion based on the above computational graph, how to make the above implementation correct and efficient? can anyone point me out how to correctly implement expandable CNN with Tylor series? any possible idea or approach?