My mentor said my training data curation was too clean, here's what I changed

I was filtering out every single outlier and duplicate from my dataset for a sentiment analysis model, spending hours making it perfect. She told me that real world text is messy and noisy, and by removing all that I was actually making the model less useful for actual conversations. Has anyone else found that adding back a little bit of noise improved their results?

3 comments

3 Comments

taylor.hayden26d ago

Adding back 5% random duplicates made my model way better with real user input.

the_jessica26d ago

Used to be dead set against adding any noise like that, but a similar test completely flipped my perspective. Your mileage may vary, but it worked for me too.

blair_webb25d ago

My buddy runs a small spam filter project on the side. He spent a month scrubbing his training data until it was totally spotless, just perfect emails. When he tested it on actual inboxes, it completely flopped. It was letting through obvious spam because the real stuff never looked like his clean samples. He called me frustrated and I told him to just dump some raw, unfiltered emails back in. After he added back about 10% of the messy duplicates and weird formatting, his filter started catching way more actual spam. He said it was like the model finally understood how people actually type.