Traveling forward in time is actually quite easy and not impossible by any means. Just go very fast and relativity takes care of the rest.
Traveling backwards in time is trickier, but there are many suggested ways to how this would be accomplishable. For example through a wormhole which not only is connected by to points in space, but also by two points in time. To be able to pass and also return, you'd need a transversable wormhole, which would require negative energy. This the most promising design for an actual time machine. Sadly it requires so much energy (like that of an entire galaxy) that we aren't even close to be able to construct one. This kind of time machine would also only work back to the point in time when it was built. This would solve the problem of time travelers not visiting us all the time.
Stephen Hawking said "time travel may be possible, but it's not practical".