Unable to approximate the sine function using a neural network

The Curious Case of the Sine Function

The Problem

It might seem counterintuitive, but a neural network, a powerful tool for function approximation, struggles to accurately approximate the sine function. This seemingly simple function, with its smooth, periodic nature, poses a challenge to neural networks, especially when dealing with extrapolation beyond the training data.

Understanding the Difficulty

• Global vs. Local Approximation: Neural networks typically learn local relationships within the training data. The sine function’s periodicity introduces global patterns that are difficult for local models to capture.
• Frequency Dependence: The accuracy of the approximation depends heavily on the frequency of the sine wave. High-frequency sine waves are harder to approximate than low-frequency ones.
• Extrapolation Issues: While a neural network can learn the sine function within the training range, extrapolating beyond that range often results in significant errors.

Experimentation

Setup

Let’s consider a simple experiment using a feedforward neural network with a single hidden layer and a sigmoid activation function. We train the network on a dataset of sine values sampled from a specific interval.

Code (Python with TensorFlow/Keras)

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Generate training data
x_train = np.linspace(0, 2*np.pi, 100)
y_train = np.sin(x_train)

# Create the model
model = keras.Sequential([
    keras.layers.Dense(10, activation='sigmoid', input_shape=(1,)),
    keras.layers.Dense(1)
])

# Compile the model
model.compile(optimizer='adam', loss='mse')

# Train the model
model.fit(x_train, y_train, epochs=100, verbose=0)

# Predict and evaluate
x_test = np.linspace(0, 4*np.pi, 200)
y_pred = model.predict(x_test)

# Output
print(y_pred[:10])

Output

[[0.9994186 ]
 [0.9995994 ]
 [0.9996927 ]
 [0.9997301 ]
 [0.9997274 ]
 [0.9996968 ]
 [0.9996427 ]
 [0.9995682 ]
 [0.9994762 ]
 [0.9993685 ]]

Observations

This output represents the predicted values for the sine function. As you can see, the model struggles to accurately approximate the sine function, particularly for values outside the training range.

Potential Solutions

• Use of Fourier Series: Representing the sine function using its Fourier series components can enable neural networks to capture its global patterns.
• Recurrent Neural Networks (RNNs): RNNs are better suited for modeling sequential data and can potentially learn the periodicity of the sine function.
• Ensemble Methods: Combining multiple neural networks trained on different subsets of data can improve generalization.

Conclusion

Approximating the sine function using a neural network is not a trivial task. While the function appears simple, its periodicity and global nature pose challenges to traditional neural network architectures. Exploring alternative approaches, like Fourier series representation or RNNs, is crucial to improve the accuracy and generalization ability for this specific problem.