Wow.
I’ve been processing a couple of billion rows of data on my machine, the fans didn’t even come on. WTF are they teaching “experts” these days, or has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I’ve been seeing with younger programmers; they’ve siloed themselves so specifically into whatever programming they “specialize” in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they’ve grown up with systems that “just work”. Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don’t even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
Hey! Thats offensive to 19-25 year olds, there are many who just finished college/university and are more than aware.
They’re just role playing like in movies, with no idea of the consequences.
How on earth is it offensive to say they’re “not experts”? They’re not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
There is nothing wrong with being 19-25. There’s something wrong with being wholly incompetent.
There’s not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That’s not Musk’s minions though.
Bunch of 1337 hax0rs script kiddies who don’t understand anything but they suck elon’s balls or something idk.
has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
There’s two issues going on:
- Elmo’s sociopathic approach to laying people off is public knowledge, and top experts have the luxury of not even applying for his jobs.
- Elmo’s ability to judge engineering talent has likely been wildly exaggerated thanks to how he has successfully bought organizations full of talented people, in the past.
I’ve read a story on the forbidden website where a “database” was a single table with a single column holding a single row that contained the actual data as a CSV blob. I’m willing to bet the muskies are not beyond such acts of genius.
You have to understand that the average Trump voter probably knows everything they know about computers from watching the ‘wacky-zaney hacker with personality issues/quirks’ “hack” into things by tippity tapping their fingies on a keyboard in your average copaganda performance.
This is something those types of people will believe.
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.
I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I’ll update, “one day”.
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
60k isn’t that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
i mean its even excel sized depending on how many columns. This is seriously sad and alarming
How about a 6.4TB sqlite database?
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
Don’t know what Elmos minions are doing, but I’ve written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I’m better now (but I’m not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it’s been a while) and it needed conversion to XML (or something similar).
The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).
But even then I didn’t need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.
But, as I already mentioned, my hard drive didn’t overheat on that load.
Seriously - I can parse multiple tables of 5+ million row each… in EXCEL… on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
Excel have a limit of 2^20 rows, something more that 1M. Curious what version of excel are you using for that.
You’re correct - the standard tabs can only hold roughly 1.2 million rows.
The way to get around that limitation is to use the Data Model within Power Pivot:
It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):
You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.
The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you’re going to build anything for professional use.
The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it’s a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it’s a nice way to build reports for end-users that’s harder for them to fuck up by deleting a formula or hiding a column.
My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That’s neither really fast, nor cool. But stable.
Unless I’m misreading it which is possible it’s awfully late, he said he processed 60,000 rows didn’t find what he was looking for but his hard drive overheated on the full pass.
Discs don’t overheat because there was load. Even if he f***** up and didn’t index the data correctly (I assume it’s a relational database since he’s talking about rows) The disc isn’t just going to overheat because the job is big. It’s going to be lack of air flow or lack of heatsink.
I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he’s on a desktop and he slept a raw NVMe on his motherboard without a heatsink
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Can’t be a relational database, Musk said the government doesn’t use SQL.
Please remember that he is a genius. Only geniuses say a lot of things.
I rest my case.
yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.
It’s bait.
They probably have an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.
Based on all that has been going on, I feel like they don’t really have the capacity to think more than one step ahead. They do sth stupid and then they usually follow up with “lol joke” or “lol you can’t understand”
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Honestly, any sweet, white-haired old lady who keeps pictures of her dogs and grandkids on her desk who’s been doing data entry for 15 years could do circles around these clowns.
But she might also have the wisdom and perception to know we’re not supposed to be doing this “work” at all, which is why he recruits naive teenagers and college kids who are still emotionally immature to think that this is going to be their “destiny” or their opportunity to get into the big leagues of business.
I keep hearing things about these hires he has, I don’t think they’re naive, At least not as such. They seem to be more power hungry trust fund babies.
But yeah, people with a few years in them would be a moral liability in that line of work.
Yeah if you read more of these guys tweets they are clearly in politics. One message tried to claim trump loves kids (to be clear: in the abstract sense, not in the he definitely fucked kids on an island with Epstein sense). Then they tried to twist the words to say “why don’t you love kids”. It was clumsy like you’d expect from someone who is practically a teenager, but the core is an attempt to follow the usual right wing playbook.
60k of rows is nothing. Fuck, where do you find these “geniuses”?
The IRS just switched columns and rows. So there’s 60k rows and 330 million columns /s
