Want to conditionally load CSS for responsive layout

I'm trying to use CSS as so:

<link rel="stylesheet" media="screen and (max-width: 991.98px)" href="/files/mobile_css/daniweb.css" type="text/css">
<link rel="stylesheet" media="screen and (min-width: 992px)" href="/files/css/daniweb.css" type="text/css">

I want to load one CSS file for our mobile layout and a different one for our desktop layout. The problem is the web browser always downloads both files, which completely eliminates the performance advantages of having a more lightweight mobile CSS.

What I'm thinking of doing is using a <link rel> to download all of the CSS that's common between mobile and desktop. Then, conditionally, via javascript, injecting CSS via @import (I guess?) to load the additional CSS that the desktop requires. Pretty obnoxious, no? Is there a better way? And why aren't web browsers smarter here?