| Hi all,
I am in a bit of a pickle, I'm afraid. A few months ago, I was hired to do Data Engineering for a small startup. When I came on-board, I discovered that the data infrastructure consists of undocumented bash scripts and large flat files (mostly csv) on a server.
I only have 1 year of experience in engineering and I am a bit lost. Because of the lack of infrastructure, most of my time is spent extracting and cleaning data and I often barely have any time to do modelling or any sort of 'analysis'.
I would like to build out a good infrastructure for them, but my problem is: there are too many unknowns and I am not experienced enough. My immediate superior does not have a background in data science and is happy with flat files (which I personally don't agree with).
Can someone share some stories from the trenches/help me out?
I'd like to see this as an opportunity to challenge myself and grow, but I'm afraid that my newbieness will do more harm than good.
Thank you. |