The Rust Programming Language

Rules

We strive to treat others with respect, patience, kindness, and empathy.

Keep criticism constructive

Criticism is encouraged, though ensure that your criticism is useful and actionable.

No zealotry

Stay mindful of the fact that different technologies have different goals and exhibit fundamentally different tradeoffs in pursuit of those goals. Keep it civil; no flamewars.

Keep it chill

In the grand scheme of things, there are better things to get tilted at than a programming language.

Keep it on-topic

All submissions must explicitly reference Rust or relate to things using Rust. If you wish to submit a link that you believe would be of interest to the community but does not meet the above criteria, then please wrap the link in a self-post that explains its relevance.

We are porting Reddit-flavored Markdown parser from our open sourced C implementation to Rust and need someone to lead this project. This project will impact millions of users and pages. You should be comfortable in Rust, compilers / programming languages, and be willing to teach Rust to others.

We don't have a specific job opening for this position but the closest posted job description is senior backend eng.

Please do not apply through the previous link because we need to route applicants to the specific team. Please email me your resume (w at reddit.com) or PM me.

It is very impressive. They're following Mozilla's lead pretty closely in identifying the most critical bits to rewrite in Rust and deploying these, without in the least jumping on the silly RIIR bandwagon. This demonstrates quite well what I like best about Reddit - it seems to have a very real and enduring culture of making solid, sensible technical decisions that ultimately benefit the whole ecosystem around the site - the polar opposite of what so many other "Internet communities" did that are now pretty much footnotes in computer history.

It's not 'questionable', it's just 'prototype-grade'. I mean the website itself is written in Python, ffs (or was when this sort of stuff was routinely being discussed in the open, for all we know this may be a misleading description nowadays). And I'm not going to comment on the other languages that OP has mentioned here, you can see that there's been some improvement but the 'prototyping', 'iterate quickly', 'move fast and you-know-the-deal' etc. use case is very much the one that's being optimized for. And I know that the Web evolves quickly and some of this can't be avoided, but this stuff is so much more mature today than it was 10 years ago.

We evaluated using this project to convert to CommonMark but eventually went with a straight port because of well... Reddit-specific features (e.g. spoiler tagging, username mentions). More specifically it would have made the AST more complicated with new node types. Keep in mind we need to backport this parser to work on all historical Reddit data as well.

Snudown forked from Sundown back in 2011. What was the rationale behind the fork in the first place? Have you ever discovered memory safety vulnerabilities in it (I'd love to see a torturous test case)? Why decide to port it to Rust? Are there any other internal libraries that might also be candidates for porting if this goes well?

The fork was just to add reddit-specific features on top (e.g. r/subreddit and u/username autolinking). We otherwise kept pretty close to mainline and upstreamed anything non-reddit-specific back.

I did a bunch of testing before we switched over and we did find some interesting bugs. Most were about letting funky HTML through rather than memory issues, but there was a segfault and a really fun one with varargs that u/slyf fixed.

Snudown forked from Sundown back in 2011. What was the rationale behind the fork in the first place?

I wasn't at Reddit then so... ¯\(ツ)/¯

Have you ever discovered memory safety vulnerabilities in it (I'd love to see a torturous test case)?

Most of our existing test suite has been focused on parsing correctness rather than security vulnerabilities. Also the C version of the parser is mostly maintained by our security team. It means that it's pretty safe, but feature velocity is quite detrimental as they typically have higher priorities on their plate.

Why decide to port it to Rust? Are there any other internal libraries that might also be candidates for porting if this goes well?

Reddit is primarily a CPython + JS shop with projects in C, Cython, Rust, Go, Scala, Java when appropriate. Rust is a natural extension to C making Python -> Rust bindings possible for performance critical areas of the codebase.

We might explore porting other relevant libraries to Rust but it depends on the success of this project and how easy we can onboard other devs. Possible candidates might be performance critical areas like sorting posts and building comment trees.

We do but it's not a lot. I'd be more confident extolling the virtues when we have something more substantial to point at (and we're on our way there!). Otherwise it feels like damning with faint praise :)

Reddit has a large history of open sourcing projects including the primary monolith codebase but we've changed that policy to be more focused on frameworks and libraries. Since this parser is primarily a library there's a strong possibility we can open source it.

It really depends if the person we hire is comfortable taking on community and pull request management duties as well.

Just thought I'd throw out that I'd love if it was FOSS. I've created a site that relies on reddit to render the markdown before it's shown to the end user (it's a reddit post, I'm not abusing your servers). If it was FOSS, I would likely compile to wasm or run the code on my server, instead.

Yup, that bothers me all the time too. Often I end up writing repies in vim before copy/pasting into the comment box. One of the longterm goals of this project is to make it easier to add and extend new features. :)

We are porting Reddit-flavored Markdown parser from our open sourced C implementation to Rust and need someone to lead this project.

Can someone give me a response that explains the motivation here? I want to understand why Rust was chosen to replace something that is working in C (so that in the future I have ammo for when I want to do the same thing...).

Also the C version of the parser is mostly maintained by our security team. It means that it's pretty safe, but feature velocity is quite detrimental as they typically have higher priorities on their plate.

Indeed, and with C it's easy to conclude that your code wasn't safe after all once it's been exploited, whereas with Rust it's easy to conclude it may not have been safe once the compiler (borrow checker) refuses to proceed.

And no, I'm not claiming all Rust code is secure. Just that it catches some vulnerabilities before they exist.

There's a significant cost to porting a code base from one language to another. And type safety is something that should already be handled in a production code base that's years in the making. So I wanted to understand the motivation for moving. It sounds like feature velocity is one of the considerations. And I can appreciate that, though I am of the opinion that it's a weak argument for a critical move like that.

Well, there's a lot more to writing a robust parser than just memory safety. If it was my decision to make, I'd still want the Rust version to be maintained by the security folks, and I wouldn't want any interns committing features with no proper review either! This doesn't mean that there's no advantage in being memory safe, but Java is memory safe too and abusing hand-rolled parsers for things they weren't supposed to do is absolutely rampant there.

Replace the reddit.com in your URL with ceddit.com and press enter, keeping all the other components the same. It will show you all the moderator removed comments as well as their content. I can figure why the comment was removed... :).

Mixed feelings. Reddit is one of the biggest vehicles for propaganda on the net and their original values such as allowing free speech no longer exist. I would not go after this position. If programmers displayed some ethical awareness in their choice of work, the internet wouldn't have become such a crappy place.