Waterfall Staff

Staff blog for Waterfall. To ask questions, go to @ask-staff instead, but first read the FAQ. For tutorials on basic features, head to @support.

Please report bugs in the Discord, not here.

Waterfall FAQ

Welcome to all our new users!

You've come at a great time for the site, with major updates around the corner.

We realise things are barebones right now, but we hope you like the place enough to stick around and see what's coming! Feel free to hop into the Discord (the best way to see what's coming soon and get a chance to test it!) or follow us on Twitter and we'll do our best to help you with any trouble or questions you might have.

In the meantime, enjoy your stay!

staff reblogged thellere

Do you ever feel fatigued and feel like you've not gotten anything done?

I remembered reading about a neat tool a while back called Gource (https://gource.io) and decided to run it against the currently live version of Waterfall's code.

The result was a fun little animated thing that shows the progress over some of the last two years. One day is about three seconds.

This video represents only the source code changes to Waterfall. It excludes everything not included in the Git repository (i.e. files and folders specified in the .gitignore file), so represents only code written by me or another staff member. Directory labels have been scrubbed for clarity, but you'll see three main branches - these correspond to the Core site, Blogs, and Common files shared between them. A little later on you see a smaller branch for the webhooks API used for payments and the commission market.

Some notes - this doesn't actually go back to the start of Waterfall's development. It wasn't until a few months in that I realised "hey, this is actually a serious project now, I should treat it like one" and set up Git. You'll also notice it tops out at March 2020 - that's roughly when development focus shifted properly towards Raven, the rewrite, and the other new stuff that I'm hoping to get out before the end of the year.

You can figure out my total lack of sleep schedule from this if you try!

We'll be doing server maintenance in 50 minutes to try and solve some problems, so the site will be down for the duration while we figure it out. Sorry for the inconvenience!

Planned Maintenance

There'll be a downtime at 6pm UK time today for server maintenance. We don't expect it to take long, but join the Discord for updates just in case!

Welcome to all our new users!

You've come at a great time for the site, with major updates around the corner.

We realise things are barebones right now, but we hope you like the place enough to stick around and see what's coming! Feel free to hop into the Discord (the best way to see what's coming soon and get a chance to test it!) or follow us on Twitter and we'll do our best to help you with any trouble or questions you might have.

In the meantime, enjoy your stay!

Community Update - October 2020

Site Stats

As of writing, we have 32,192 users, 52,051 blogs, and 418,334 posts.

Site News

Bit of a change this month - while last month was entirely code, this month has been spent doing code and research into how various bits of the site are doing.

One major change that's worth noting separately is I'm changing the UI strategy. Those who've been around since the start know that one of the original ideas with the UI of the site was that it's snappy and simple. As time has gone on, it's become obvious that it's currently a bit too simple, so I started rewriting it in React. But then I realised... actually, that goes counter to what I wanted. React is a very heavy Javascript framework, and I specifically wanted to avoid too much Javascript. So, that UI is scrapped - I'm still rebuilding it from scratch, but it'll be a little easier to maintain, and still nice and simple. There'll just be fewer page switches and a little more interactivity (alongside quick reblog etc. That's still coming).

Additionally, one breaking change is coming - when this update hits, we will officially be dropping support for Internet Explorer, and will not be attempting or offering any support for bugs discovered while using that browser. This doesn't necesarily mean the site won't work in it - but Internet Explorer is outdated and officially end of lifed by Microsoft, so you really should upgrade to a modern browser if you still use it. My recommendation is Firefox - it's lightweight, privacy focused, and doesn't transmit all your data to Google like Chrome does.

So - since last month people said that actually, they are interested in getting into the lower level, technical stuff around the infrastructure and policy decisions of the site, this month you get a longer post going into some of that. I'll also be taking this opportunity to answer a few questions that come up every so often, since I assume that means more people are interested than are asking. I'll hold some stuff back for next month. Oh, final note - still on track for the update being launched in December.

All this stuff will be going under a readmore with a table of contents just above it.

  • Writer's Block - The New Text Editor

  • Accessibility - Fonts and Image ID

  • Image Formats - WebP is the future

  • How does Waterfall make money?

So, let's go!

Writer's Block - The New Text Editor

We're probably not calling the new text editor Writer's Block.

But, since everyone hates the current one and, frankly, the licencing conditions for it aren't great, I've wanted to build our own for quite a while. I'm happy to announce that we finally have one.

This screenshot is a prototype, and there's still all the styling work to be done. Bear that in mind.

Our Kickstarter backers were given a live preview and report that, so far, it seems to work very well on mobile too.

Right off the bat, you'll notice you'll have more formatting options available to you. These will be extended - you're likely familiar with markdown, it's the thing Discord uses for its text formatting. We're working on a custom implementation that, eventually, will add support for colours and some other fancy formatting. Both the text editor and the Falldown parser will be open sourced, so that anyone can contribute to improving them.

Accessibility - Fonts and Image IDs

Something that's been largely neglected so far is the accessibility of the site. That's on me, and is largely a symptom of trying to do too much at once. It's also something I need to take seriously if I want Waterfall to grow. Given I'm about to rewrite the UI from scratch anyway, this is the perfect time to make that happen. While I'm not an expert in accessibility - I'll be bringing someone in who is to tell me what a bad job I've done so far - there are three things that are going to be addressed.

First, font sizes. They're going to be made configurable without messing with browser zoom. This may be a per-device setting, or iit may be a global account one - implementation details will dictate which I end up going with.

Secondly, something that's been requested a lot is the option for a dyslexia friendly font. I've been looking around and found two promising options. Dyslexie is the one that commonly gets suggested, but the licencing terms for that are unfriendly and we have to pay by both the staff member and, as a website, by the visitor, so that's out. Instead, we'll be going with OpenDyslexic for now, and possibly forking it to add in any missing glyphs we encounter as the site grows. This will be a toggleable option in user settings.

Third, I noticed a lot of people putting Image ID stuff in reblogs and under images and such. I'd never really seen them before but it didn't take long to figure out that actually, those are a great idea, so I'll be building native support for those into the site. The image editor is being overhauled anyway, so support for regular image captions was planned already. Now, it just has Image ID support too. While we can't force artists and image uploaders to use the feature, we do highly recommend it. As with the font, viewing image IDs will be a toggleable option.

Image Formats - WebP is the future

WebP has been a thing for years now. It's better than PNG, it's better than GIF. Why hasn't it taken off? Apple is why it hasn't taken off. They're the only major browser to not support it (edit while proofreading: apparently they do now. But only for some people).

The only real limitation of it is the image size - it can't exceed 16,383 pixels on either dimension. This is unlikely to be too much of a problem, but it's something we had to bear in mind - as such, with the update, images will be limited to 8,192 pixels on any dimension. We don't anticipate many being bigger than that, but we think that's the sweet spot for any hyper-detailed artists to still be able to get their stuff in a high quality format.

When the update hits, image transcoding will finally be a thing, meaning image sizes can be shrunk a little bit. Ideally, we'd be going WebP only - but, thanks to Apple... WebP supports both static and animated images, at a significantly lower file size. When testing, a GIF file we used started at 5.7MB, and, once converted to WebP by our encoder, came out at only 628KB with no loss in perceptual quality. When testing with a 16,000 x 8,000 pixel PNG file, it shrank from 478MB (yes, megabytes) down to 8.7MB. Obviously, WebP is a good idea.

However, not all browsers support it yet. According to CanIUse's page for WebP, support is at only 80% coverage, and Apple have decided to artificially restrict the version of Safari that will support it to only supporting it on the latest Macs for... some reason. Unfortunately this probably means that support on iPhones will be similarly artificially constrained. All these factors combined means that we'll be keeping both WebP and legacy (hereby defined as GIF and PNG) copies of images for the foreseeable future. If your browser supports WebP - awesome, you'll get the WebP version. If not, you'll get the bandwidth sucking legacy formats instead.

You can expect this to be a trend going forward - I hate the idea of constraining Waterfall to using legacy technology, so as new stuff becomes widely available and we've tested it, we will start using it and, eventually, phase the older stuff out, since for reasons outlined below, supporting 1% of the population who can't use the more modern stuff is a resource drain we can't realistically handle.

For now, it'll be a couple of years (most likely - if the support matrix changes to a high enough usage, it may be earlier) before support for PNG and GIF are dropped, and our converter will continue creating those formats. We'll give plenty of notice before we finally drop them too. AVIF recently launched, we'll likely add support for that when we drop PNG and GIF. Similar with x264 - it's a legacy format now, but not enough support for the technology we want to use (in fact, here it's worse - there's been VP8, VP9, and now AV1 whereas regular images haven't advanced as quickly. We're ignoring HEVC since that's patent encumbered and basically nothing supports it. VP9 has about the same support level as WebM currently, and AV1 sits at around 30% - unfortunately, video is a lot costlier to store than images, so we have to stick with one format for now) is available yet. It's an incredibly frustrating scenario as both a developer and the one who has to pay for the storage space to keep all this.In an ideal world, we could just drop the old formats immediately, but we aren't in an ideal world.

How does Waterfall make money?

Alternative title: How do we plan to make money going forward?

Right now, 90% of the income Waterfall itself brings in is from the Patreon, sitting at $80 a month. The other 10% is from blog slots and commission market fees. Yeah, it's not much, we know, we're still very much in the "passion project" phase of the site and likely will be for a while longer.

The rest of the company's income is largely made up of a couple of other small sites and freelance work (the latter being the vast majority of what pays our bills). Additionally, we're game devs, so we're working on a project there too. The games subsidise Waterfall, and Waterfall subsidises the games.

Going forward, I'm committed to one thing in particular - fierce independence. I don't want shareholders or stock options or an IPO or whatever. I want Waterfall funded by the community, not advertisers. This, obviously, makes things a bit difficult, since most sites sell out fairly quickly to that stuff. To understand how I want us to work going forward, you need to remember the plan I laid out last year - to make Waterfall a collective of sites rather than just one.

That's where the subscriptions come in. From the beginning, the idea was one subscription across the entire network, however large that ends up being. Three sites or three hundred, your single $10 subscription gets you the bonus features across all of them, no double-dipping, no "oh but you subscribed to Glacier, not Waterfall Social" or any other crap like that. There'll always be some people who adamantly refuse to pay a penny, but the goal here is basically to make subscriptions give you so much for your money that from a sense of pure practicality, it's just worth it. Whether we'll succeed or not remains to be seen.

Something we're actually getting rid of that makes us money is blog slots. I've always felt weird about them to begin with and they're not a huge seller, so the limitation will be lifted in the update. People who hoard URLs will still have them nuked, obviously, but I think we're outgrowing that kind of restriction now. Anyone who's bought blog slots already will be given a free subscription, rounded up to the nearest number of months it'd cover.

We're also doubling down on the commission market. It's been in alpha for a year and we've had some reasonbly good feedback so far, so that too is getting a total rewrite. New payment options, new search options, one-and-done templates, adoptables support, all being added. We'll reach out to some artists on the site soon asking if they want to help test it. Additionally, we'll be giving the fee structure another look - our bottom line didn't move much with the coronavirus special, so we're considering making that permanent even after the crisis has ended.

In addition to that, we're still looking at the patron subscription stuff to provide a safe space for artists of all kinds to offer exclusive work and have fans subscribe. As with the commission market, we'll take a small fee - likely less than Patreon - to help sustain it. Glacier's monetisation model is still up in the air.

Most of this relies on us building something good enough that people actively want to support it. Some people won't no matter what - I still remember our first hate mail about the commission market fee structure and that we shouldn't bother trying to offer protections - but ultimately, it feels the right balance of independence and incentive to make something good that it might work.

That's it for this month, see you in November!

There's a decent chance that, finally, I've fixed emails.

If you've not verified your email or you have a friend who forgot their password, now's the time to get in on it and see if I did!

Community Update - September 2020

In August, we...

  • Began testing the new database

  • Finished the video transcode system

  • Finalised the spec of Waterfall 1.0

Site Stats

As of writing, we have 32,029 users, 51,875 blogs, and 407,230 posts.

Site News

This month has been very busy in terms of code stuff, but also very boring. The most interesting stuff - we've finally finished the video transcoder, and it works very well! Uploaded videos will now have multiple qualities to choose from, which should help mobile users out a lot. This also, for those who've been keeping up, forms the basis of the video management software stack for Glacier. Currently, it's only producing two qualities, but is easily extendable to have more.

In other news, I made the decision to migrate to a new database stack. There are many reasons for this - mostly being performance and reliability - and there'll be a full post-mortem after the fact, since it'll be a long post. But, this is sort of the right timing to do it - we're doing an overhaul and getting it out of the way now is better than doing it later when it'll take so much longer to do. The only thing that DOESN'T seem to work is... avatars. Considering everything else is so overcomplicated compared to those, that's pretty hilarious actually.

These two things complete the trifecta of things we needed to get the new stuff running, and, as I type this post, a test run is happening on the dev server. If it works - amazing, I can get on with developing the core site, and you can expect performance to be a LOT better going forward.

Speaking of which - the spec for Waterfall 1.0 has been finalised. Assuming the test above passes, the next few days will be spent auditing the existing code and seeing which of the 183 open issues can be closed because the code is already there. I'm expecting a fair few to be closable since it's basically just a big list of everything on the site and that we wanted to add, and just needs powering through, effectively.

Sorry this one is so short - I'm not sure whether you guys would be interested in more technical posts or not. If you are, let me know in the comments and I'll do a supplemental!

Oh right, release date for the overhaul - I'm aiming for it being done by the second anniversary, on the 10th December. Or 17th if you want to go by the first ad post. Either way, that's my personal aim.

Community Update - August 2020

The bees are imminent

In July, we...

  • Got more server hardware

  • Finished the image transcode system

  • Started the new text editor

  • Published our public roadmap

Site Stats

As of writing, we have 31,804 users, 51,610 blogs, and 396,653 posts.

Site News

There's a lot this month. We'll start with some Commission Market news - we finally got approved by PayPal, so it'll be an option going forward once the rewrite hits. We are considering removing Stripe entirely - users have reported some bad experiences that are shaking our faith in them, as well as being too complicated to use, so some feedback here would be great. Additionally, we've had a couple reports of them trying to make you use unexpected transaction methods - if you try and buy a blog slot or pay for a commission and get redirected to Apple or Google Pay, do not proceed with the transaction - hop in the Discord and let us know so we can figure it out.

To answer a concern that was brought up last month - using PayPal will not change the NSFW policy of the site. Business and partners have different rules from individuals, and Waterfall's existing content policy puts is well within the safe zone by all the interpretations and second opinions we got. If PayPal elects to go back on that, we'll simply look for another option or roll our own.

I needed a break this month because there's only one of me and mental fatigue is a little bitch, so I switched focus some to work on Springwater's blog analytics dashboard.

This is still a prototype (you can tell by the labelling) but even with the test data, I can already tell people will find this very useful, especially artists. I'll do a writeup next month about what's included, since at the minute I'm still surprising myself with what breakdowns I'm able to do with it.

I'm considering doing a mockup of the dashboard to see how it'd look with the stylesheet used there - certainly it'd look a bit more modern. You guys let me know what you think before I start?

Next up, I finally started work on a custom text editor. It's SO much better. Looks better, behaves better, and, since I accidentally backspaced when I finished writing this post and had to start over, it'll also save state so if your browser crashes or you acidentally leave you don't lose everything. I don't have screenshots of it since it's still effectively a glorified text box, but I do have screenshots of the new audio player.

There's still a few kinks to work out, mostly related to mobile, and I'm not sure whether we'll keep the visualisation there or not, but it works and I'm pleased. Album art support! It'll auto-populate with whatever ID3 tags the file has and then you can edit the picture, track name, artist etc. It'll also eventually have playlist support for musicians to use in art posts.

Final piece of UI news - we'll be adding a bunch of accessibility options. There'll be stuff like a colour blind mode, dyslexia mode, font size options etc. We'll run these by members of the community before launch to make sure we're covering the largest cross-section of users possible. They'll all be off by default, but users can pick and choose which ones they find useful and turn them on.

Oh, and something about how we've started working on chats too, but nobody cares about that.

Back on the backend, the main roadblock now is making posts.

Legacy code sucks.

In theory I could port it wholesale, but there's a few improvements I want to make in a couple of places, as well as fixing mentions code and whatever is happening with queue time calculations. Purely text based posts work, but file based posts need some more work, as well as link posts, because I'm just not happy with those at the minute.

That's all for now, so see you next month, and give the roadmap a look!

Community Update, Supplemental

With work going on behind the scenes, we thought it was time for two things we've been meaning to do for a while to come out.

Public Roadmap

We now have a public roadmap of features we're working on, and have planned for the future. The hope is that this communicates more effectively what we're doing behind the scenes during periods of silence, as well as let you know what to expect in the coming updates. Go have a look!

Developer Applications

Now we have our internal API mostly working, it's time to consider a public API as well. This would allow for fun bots and fancyu additions made by you guys to do things that we either never considered or didn't feel were a good fit for the core site. If you're a developer, and you're interested in applying to be in the beta of the public API, join the Discord - the application process will be outlined there tonight! Get your ideas ready.

See you on the 1st for the regular update!

Community Update - July 2020

In June, we...

  • Started prototyping the new art system

  • Began the overhaul of the Commission Market

  • Cleared the first major roadblock for the apps and new UI

Site Stats

As of writing, we have 30,392 users, 50,095 blogs, and 383,159 posts.

Site News

This month was a bit slower due to some freelance work that took up a considerable chunk of time, but we're thrilled to announce that the new UI now has a fully functioning dashboard! This means one of three major hurdles are now cleared. The remaining two are overhauling the image uploading system, and building a new text editor. The ones we have right now are... fine, I guess, but that's it - they're not particularly good, just... fine. These two elements will be left till last - its perfectly valid to (and may be the case that we do) leave them as is in favour of getting the new UI out and start testing that without the added variables of weird new stuff that could also be breaking it. It actually makes sense to do it that way even - given the look of the UI isn't changing at all, other than being slightly more consistent with how it displays things, we'll know that any bugs are a logic error rather than a display error.

On the side, we've been looking at the CM. The biggest thing missing right now is stuff that I wanted to include from the beginning, in that artists can set templates up. The second biggest thing is getting PayPal support in some way. Both of these are making headway, but we're not quite ready to talk about those yet until details are finalised.

The next big thing is the art system. We've talked about the new anti-theft system for images we want, but that's a long way off yet - we don't have the budget for the specialised hardware we need. What we are doing though is preparing to extend art posts to cover writing as well, including an anti-theft implementation. There's a lot of work to do on it still, and we'll likely need to tinker with how it scores things a fair bit. However - with the advent of that system, we'll be having a dedicated mod in place, whos only moderation duties (barring emergencies) will be handling any issues the art/writing theft detection causes, or disputes over who wrote what. We'll be writing a more in-depth dev blog on the new writing tools soon. We've always wanted the art system to cover more than just images, and this is the first step towards that goal.

We'd also like to extend a welcome to the new users we got today coming from Tumblr after their latest update! We hope you enjoy your stay, and feel free to join the Discord to ask any questions you might have.

Apologies for the unplanned downtime this morning! We had to do some maintenance to prepare for the API rollout, and it took significantly longer than expected.

This is the first of two backend patches to prep for the overhaul. The second one will be done at the time of the rollout - retrofitting the existing site to handle the data structure changes would be, in effect, wasted work.

There's still a fair bit to do before then, but in the meantime, you should notice some performance improvements. Thanks for your patience!

staff reblogged wyvern-king
wyvern-king -

Idk if everyone knows but I just thought I'd share that itch.io is having a massive bundle for BLM rn where all the profits are being donated to NAACP and the Community Bail Fund 50/50. It costs 5USD, but it encourages you donate 10USD. It's hit over 2M USD so far in funds.

It's got a ton of good stuff from games to songs to assets and some engines if y'all wanna check it out here.

Community Update - June 2020

36.2215405525 squared

In May, we...

  • Upgraded the servers in preparation of the app

  • Made more headway on the new UI

Site Stats

As of writing, we have 30,105 users, 49,755 blogs, and 367,979 posts.

Site News

Most work right now is related to getting the new UI done. We're happy to report that we've gotten the first set of data from the site to display on there, signifying the first tangible progress, and frankly, giving a big morale boost. It's still ugly as sin, but real progress is being made now. 99% of code written so far is directly transferrable to the app versions too.

As a micellaneous site stat -the Commissions Market is now consistently handling an average of $100 of transaction volume a week. We're hoping that the reduced fees are helping those artists using it. We've also not had any bug reports for a while, which is excellent - we'll be keeping the Alpha label until we've figured out PayPal integration and written commissions though.

On a more serious note, things are a bit fucked at the minute. You can probably infer our stance if you used a calculator on the number at the beginning, but to be explicit on our stance - Waterfall is a safe space to discuss and organise. While we can't legally ignore any warrants or court orders, and so can't guarantee it's perfectly safe, we invite anyone who can make use of it to use Waterfall to post resources. If we DO recieve a warrant, we will fight it where we're able to do so. Our hope is that not being US-based makes it not worth their time to try and get us to comply.

Provided this line is present, Waterfall has not recieved or complied with any warrant in relation to the protests. Should this line not be present, assume we've been forced to hand information over.

While not in a position to protest directly ourselves, our lead dev is taking a break for a couple of days to work on software tools specifically to assist those protesting, but in the meantime, please feel free to use Waterfall for masterposts, or anything else - as a matter of policy, staff will not remove any posts, users, or blogs providing resources or organising protests.

If you are involved in protesting - please stay safe out there.

Community Update - May 2020

Cough and sneeze into your elbows!

In April, we...

  • Finished the basics of the API, formally starting the frontend work for both the app and the new desktop site

  • Beefed up our internal security and started planning the new infrastructure

  • Added pride badges and themes for all users

Site Stats

As of writing, we have 29,657 users, 49,251 blogs, and 351,093 posts.

Community Shoutouts

This month's shoutout goes to @dohz who is posting puzzles every week! Giving your brain a workout is essential to your mental health with the world as it is right now, so give it a go!

Site News

The first is the API stuff - at long last, we're in a position where we can start making it actually look like something. There's still some features missing, but everything relating to users and blog interactions is now tested, as is retrieving posts. Making posts still needs some work, since we're introducing a new post type, and I want to redesign the workflow to be a bit nicer on different post types.

Additionally, some improvements have been made along the way - for example, tag blocking now reads the tags of the OP as well for the purposes of both blocklists and NSFW checks. Code to get a tag history has been added, and on the technical side, the number of database calls has been halved. In lab conditions, getting my main blog's dashboard takes 67ms on average with the new system, compared to the current system's 180ms average. This is assuming the posts have been loaded by someone recently, however - if every post is new to the system that day, it takes an average of 130ms instead. Still an improvement! It also means less bandwidth usage for people.

Second of all - most of you are probably aware of stuff going down with the bisexual flag right now. Obviously, it's stupid, and the actual creator has declared the flag free to use by anyone. To that end, we've gone ahead and got off our asses to add pride flag badges, and are working on themes to go with some of them. Most badges are up - there's a couple (the most commonly requested being aro) that still need graphics making, but those'll be added within a few days. Themes are also being actively worked on by the design team while the programming team works their magic so they can do stuff for the new site, and we'll be adding those periodically too. So far, the trans pride theme is available, with the lesbian pride theme coming as soon as they stop crying long enough to fix my attrocious colouring abilities. NB pride will be next, followed by at a minimum, four others. It's likely not every flag will get its own theme purely because of time and resource constraints, but we're hoping we'll get most of them. The rewrite might provide facilities for you guys to fill in the rest of the blanks as well as other themes, we'll see how that goes.

We realise "Hey! We wrote code!" isn't super exciting, and it's a far cry from the old days where we were pushing updates and new features out every other day. Hopefully, with the app finally starting to take shape and the eldritch, spaghetti like horror of the code I wrote a year ago finally being turned into something approaching modern standards and being more manageable, those days will return again soon. In the meantime, thanks for sticking with us, and stay safe!

Welcome to all our new users! We hope you'll enjoy it here.

Waterfall is a small site, still in development, but the site should be pretty stable. Over the last 15 months we've grown from a site built for a handful of friends by one guy, to a site of almost 30,000 users and 50,000 blogs, and a staff of one developer, three community mods, and an infrastructure engineer. Hopefully, that carries on and we can keep growing.

Behind the scenes, we're doing a full rewrite of the site's code to get a mobile app out to you guys, which is where all our development effort is currently focused, but please let us know of any site-breaking bugs regardless! For a rough how-to on how things work, check out the Support blog, where we've put a few tutorials until we can develop a proper one. As a quick note - emails are currently inconsistent and may not be delivered. Don't worry, no functionality is locked by not verifying for now until we figure out the cause.

For the artists among you, use the "artists on waterfall" tag to start getting your work out there, it's the one the community checks most often. And please remember - the "nsfw" tag is mandatory on stuff that qualifies, even if your blog is flagged adult only!

If you have any questions or need help, feel free to join the Discord where we'll do our best to get you straightened out!

staff reblogged thellere

Art Theft 2.0 - Overhauling the System

thelldev -

As far as I'm aware, no other site has an art theft prevention system. Waterfall's is the first, and I'm quite proud of it. However, it's an imperfect system, and I have some improvements I want to make to it.

Art Theft 1.0 - An Overview

Art Theft 1.0 (or, to be more accurate, the current version is 1.3 or so) is an extraordinarily simple piece of code. Unfortunately, extraordinarily simple means extraordinarily easy to defeat.

When a piece is marked as art, three things happen in order. First, the image is hashed to MD5. Then, a list of MD5s for all art uploaded that don't belong to the user uploading is retrieved, and the hash is checked against each one. Quick and painless.

MD5 is best described as a "signature" that can be applied to something to see if it's the same thing as something else. The problem is (while it's old and collision attacks have been demonstrated now), changing a single byte in the file results in a different MD5 signature.

This means that lazy thieves get caught, but anyone else doesn't. I've been thinking about this a lot over the last 6 months, and I think that - after the app is done - it's time to address these shortcomings.

Before I continue - while the system is easy to defeat, you'll still be banned if you crop out a watermark and upload it as your own thing.

Art Theft 2.0 - Rise of the Machines

The current system is instantaneous. You press upload, it tells you in a couple of seconds whether you're naughty or not. After a brief bit of thinking, I realised this cannot be the case with any sufficiently advanced system that can be called "decent". So first things first - when Art Theft 2.0 rolls out, there'll be two states to art posts. Unverfied, and verified. The main difference is that unverfied posts will just have a yellow icon instead of green for the art symbol in the corner of the header. If a post passes verification - no problem, nothing happens. If a post fails verification, it works the same way as it does now, it'll be silently converted into a reblog of the artist's original post. The major difference is that since there's a chance that post will be reblogged while it's awaiting verification, any reblogs of that post will need to be converted too. Luckily, the way the site stores post chains means this is not a problem at all.

While a post is in unverified state, the site will be running the process in the background. Let's go over what it'll be. It's SIGNIFICANTLY more complex that the current method, and requires some special hardware, so before the cutoff, I'm going to link our Patreon. Ordinarily it's hidden down in the site footer because we feel weird about taking money without giving anything in return (other than... the site I guess?) but the faster we get the hardware to run this (and the more of it), the faster we can improve things.

Here's our test image, you can find the art post for it here.

The first stage of the process will be to convert it into a grid. On each of these grids, an MD5 (or perhaps a SHA256) is generated. It'll then search for images where there's matching MD5s. In theory, there should never be any unless the image is a straight duplicate. This is the low hanging fruit part - it can stop the process if it finds a match here.

But what if we modify a square?

We drew a line in. That square now gets a different MD5. But, since collisions CAN happen - for example, if someone does a plain white background, or transparency, and whatever grid size we use ends up capturing fully transparent/white squares - that's enough to throw it off. So instead, it'll go off how many squares are the same. This is a good time to introduce confidence scoring.

Being Confident

If all but one square returns an MD5 match to an existing image, we can say pretty confidently it's a repost. Let's say 98% confident. The system will reclassify that post as stolen, and change it to a reblog.

But what about more complicated scenarios? The grid system isn't the only method we'll be using, and they're less clear cut. We need to assign a cutoff or two - how confident should the system be to act autonomously? If it's less confident, it should ask a mod to review it manually. At the same time, at some point, the system should be confident enough that it's not a repost that it doesn't bother us. We'll settle on 90% confidence for autonomous action and 20% confidence for not bothering us until we've refined the system. Now, let's go over the other methods we'll be using.

Size Matters

Resizing something is a common way to get around filters of this kind, so we need to keep records of different sizes of the art too. We'll also need to check for images being flipped - the same way that people uploading TV episodes on YouTube do to get around the copyright filters. Ading borders is something we need to check as well.

Why so Blue?

Another common way of getting around filters is changing the colours of something - either the colour itself, or the saturation of it. Checking this will be a pain in the ass, but is essential to a comprehensive theft prevention system.

I See You

Finally, the part that'll take the longest - visual comparisons. If all the above fail, there's a chance the thief is skilled. So, we pass it to an AI to look at. This, once we've gotten it working right, is all the above on steroids. It'll be able to look at it and say whether it's seen it before, as well as a confidence score. If it gets to this stage, unless it's 100% certain, a mod will likely be required for intervention - after all, we've seen what visual recognition is like with Tumblr's porn filter.

This step has some nice bonuses, however. It'll be able to see if it's been blurred, is a cropped version blown up to full size, or whether it's seen something similar before but watermarks have been removed or text altered. It might even be able to tell whether it's a trace of someone else's work or what's been used as a reference - however, we're consciously choosing not to intervene on that stuff, and it's a waste of resources to try.

Here's what it makes of our test piece (rendered in paint - right now, we get a text readout of pixel areas that I've had to translate into something easy to read). As you can see, it's far from perfect, which is why manual mod intervention will be mandatory for this stage.

Closing Thoughts

The above is about half the system we're implementing, excluding the experiments that are more curiosity rather than something serious to include. We're not listing them all here because the post will drag on a bit, and because we want there to be some element of secrecy so you can't find holes to defeat it.

It's a pretty complex system and our aim is that any given art piece should take no longer than 20 minutes to verify. In an ideal world, it'd be 5 minutes - but we're unlikely to have the budget for that any time soon, and as more art is uploaded, the longer it'll inherently take.

Suffice it to say - art theft, while a unique system, is flawed. We want to fix it, and we want to share with you what we're doing to improve on it.

Thanks for reading!

staff -

From our lead dev, talking about the overhauled, upgraded, and immeasurably improved art theft system currently in the works behind the scenes!

It's now easier to ask us questions!

Go ahead and ask whatever you like, we’ll do our best to answer them all. It also lets us do our more rambling answers without maing folks’ dashboards overly long.

Answers will be crossposted to our Twitter! You’ll also be able to get live support and status updates there, so go give it a follow!

Community Update, Supplemental

We were going to all in on the Star Trek joke here, but apparently, stardates don't start until 2323.

Regardless, here's the update we said might show up mid-month!

App Testing Raffle

The app raffle is now closed, and 100 winners have been picked. We'll be messaging these people soon once we've figured out the best way to get everyone in the right place to give us feedback and bug reports, and figure out how to get the info we need to grant access. In addition, we're picking a handful of people who didn't win as "staff picks" - either because of their contributions to the site, or because they do something that none of the other entrants do and it's something we want to cultivate. These people will be messaged at the same time as the raffle winners.

Changes to Art Theft and New Tools for Artists Coming Soon

We've only touched on it briefly before, but with the upcoming API upgrades, we'll be in a position to start working on Art Theft 2.0, and we've gotten some fancy hardware to help with it. We're on version 1.3 or so right now, but we want to take it significantly beyond what we currently have. We'll do a writeup on the new system a bit later on, but it'll crank it up to 14.

We're also investigating extremely experimental features to allow you to track how your art is being spread across the web on other sites, though these may not be available depending on how well they (or whether they actually do) work.

In addition, those in the Discord might know that we've recently acquired a very large, very expensive printer. We're happy to announce that, following the launch of the app, we'll be adding the ability for artists to sell on-demand prints, similar to another artist hub site. Artists will be able to set their own price, but the key difference is Waterfall can do it a LOT cheaper than the main competitor - for a 24" poster, I believe we figured out that our fixed price take can be as low as $8 per print, compared to dA's $20 minimum. Artists can either adopt a similar price as they would on dA, or lower the price for the same take but to encourage sales. We'll release more info about the programme nearer the time, and after the lockdowns end for us to get spare parts and supplies.

We'll also be adding PayPal to the commission market as soon as is feasible.

API Development Update

We've started making some real headway now, with some of the boring technical implementation stuff finally over. There were four main areas of rewriting for the API that we expected would give us trouble - logins, media, emails, and the new suite of moderation and security tools. In a traditional, simple site, logins aren't a problem - however, given we're taking Waterfall multi-node instead of being on a single server to resist downtimes (such as when we have to reboot the server to install updates), it gets a lot harder. Luckily, this problem is now solved. Theoretically, the other three can wait since logins were the only "critical" part - media can be made do with for now (though we're still overhauling that because I'm just not happy with the current setup), and emails "work", but as you've seen, they seem to be doubling up at the minute and we're not 100% why. That's why this bit is here - this is a heads up that we'll be working on the email server today, so emails for new followers or new accounts won't be getting through for a little while. We'll be setting up an entirely new pipeline for emails to make things much more efficient and streamlined, and give us better control over how it works. We'll let you know when the new server is up, if you don't figure it out yourselves from the emails you start getting, though the server may be down for up to 24 hours.

The current phase of development is still under "The Basics" banner, my internal denotation of the core stuff - think roughly how the site was in March of last year. Once the basics of posts, users, and blogs are implemented, we can start work on the new UI, and then add all the modern stuff we currently have as we go. Now we're getting into the swing of things with how the new technology we're adopting works, this stage should go a lot quicker now, though we anticipate a slowdown when we start building the UI as we get familiar with having to actually use JavaScript. Luckily - the rest of the team are now being formally trained in how to code, so this will soon no longer be a one-man show with the rest of the admittedly gorgeous team sitting around looking pretty.

An accurate representation of the team's thoughts on attempting to reverse engineer the existing code to learn from, via The Profound Programmer.

[text: “thousands of lines and zero comments am I being punished for something”, photograph of an infinite distant horizon ready to ignore your pleas for help or death.]

Site Usage

We got an interesting ask earlier asking about how the site usage has changed during the quarantine, so I thought I'd mention it here!

Overall usage is up about 31.6% in terms of daily visitors, with a week of particularly increased activity when the new Animal Crossing game released. After that week, there was a small dip, which then slowly increased back to normal levels, then kept rising.

Commission Market Usage

Likewise, usage on the Commission market has increased. We don't have figures for how many new artists joined, but we do have figures for the number of commissions and money changing hands.

There's been a 15% increase in the number of commissions taken per week since we adopted our new fee structure, and, if our math is right, a 35% increase in the amount of cash artists are taking home. Waterfall's income remained about the same with the new fees in place, so we're considering this a success for now. We anticipate these numbers to climb once we get PayPal integrated.

staff reblogged coffee

General Strike 2020 - No Rent. No Debt. No Labor.

No Rent. No Debt. No Labor.

website for general strike info

there's a lot of resources here for starting strikes, strikes already happening in certain areas, and ways to connect with people in your area and encourage striking

very informative and helpful if you're considering that but don't know where to help

Community Update - April 2020

Hello, good evening, and remain indoors!

This month's update comes a few days earlier thanks to the first of the month being April Fools Day, we decided to get it out of the way to prevent any confusion. We're not intending on doing anything for it, but why risk it?

In March, we...

  • Made significant headway on the backend rewrite

  • Lowered our commission marketplace fees to help artists cope during the current crisis

Site Stats

As of writing, we have 29,211 users, 48,751 blogs, and 333,277 posts.

The Waterfall Folding Team has crunched 1,128 work units for a total of 10,381,579 points, and is currently 3,276 of 244,538 teams on the leaderboard!

Community Shoutouts

This month, some honourable mentions are due! This is a new section for these updates, and might not be present in all of them. However, sometimes we come across something that deserves wider recognition, and that's what this section is for.

First, @fancybits gets a shoutout for producing some excellent tutorial posts on a range of things useful in everyday life, but particularly useful right now! Everything from art to clothes repair is covered, and there's some pretty tasty looking recipes too.

Second, @sentiniel gets a shoutout for developing a Quick Reblog extension for Waterfall! While we still intend to build it into the site proper eventually, this is an excellent stopgap solution, and we're looking forward to feedback on how it works.

Site News

First off, a reminder - the raffle for the app beta is still open! It'll close at midnight UK time on the 1st April, with the drawing taking place later that week. To enter, simply follow @app-dev from one of your blogs. Note that following multiple times does NOT give you more entries, and empty blogs will be excluded - only active users, please.

This month has mostly been spent working on the API logic. There's a couple of ways we can proceed from where we are now. First, we can hold off on getting new stuff out unttil it's done and dusted. A downside of that is that is that the desktop and app versions get delayed launches, since the changes made will require both to be in sync. The second alternative is hold off on the UI update and gradually update the current build with any new features. This means the app beta can start sooner since the desktop site will be in line with the app's changes, but means the new UI will be further off, since it'd be a bit much for the resources available to do all of that at once. The second option is the most likely solution - while that way the desktop version wouldn't be using the API itself, it would be hooking directly into the site logic itself, the same way it does now. That way, we can test that everything works the way it's meant to. A third option is a sort of hybrid - we modify the current build of the site to use the API as we go and, once tested, begin the UI rewrite proper. This one has advantages in that testing is done "properly" and there's less wasted code, but does mean that the site will likely be a little bit buggier. Given how quickly we fix bugs usually, this shouldn't be a major issue, but it's still something that we're hestitant on. We'd love to hear you guys' thoughts on whether to go with option two or three!

Another significant portion of time has been spent both relaxing and getting set up for true remote working. While nobody ever comes to the office and everyone works remotely anyway other than the guy who literally lives there, we're on a big push internally to get everyone on staff knowing at least basic coding - the theory there is that then, the artists on staff don't need to use finger puppets to explain what they want implemented, and can have a go at it themselves, with just cleanup and testing done by the more experienced coders. This should, in theory, mean the art and commission systems get a big boost in utility since those on staff who use them most will be able to work on them, as well as, hopefully, making hiring new staff easier in future.

In other news, the Folding team has taken off! Details are in the last community update on how to join the effort. We'll be making the team a permanent presence even after COVID-19 is no longer an issue, and badge awarding will be automated in future too. We'll also be adding a few tiered badges to it to encourage people to participate permanently instead of only running it for one work unit. In the meantime, those who've contributed so far - you'll be awarded your badges in the next few days. Thank you for taking part!

There'll likely be another update in the middle of next month. See you then!

COVID-19, Quarantine, and the Commission Marketplace

Hello everyone,

By now, you're almost certainly acutely aware of the global health emergency going on. Chances are you're also being forced to self-isolate, or your workplace has shut down.

We're very aware that this is a concerning time for everyone and for many, financial uncertainty is almost as big a concern as being ill right now. While we can't help everyone, we're doing what we can for our artists, for some of whom commissions are likely to be their only source of income for the near future.

With immediate effect, the Commission Marketplace fees have been adjusted. For lower priced tiers, we've lowered the take as far as we can push it without it costing us - in other words, just enough to cover the payment processors fees. On all milestones over $10, we now take 5%, including the payment processor fee. This means artists will now recieve up to 95% of the list price, which should be putting us better than PayPal in some cases.

On milestones costing less than $6, Waterfall will still take 10% (remember - this 10% includes the fees for Stripe, which we pay on your behalf). For those between $6 and $7, we'll take 9%. Less than $8, we'll take 8%, etc etc, until less than $10 where we'll take 6%. After that, we only take 5%. For non-USD users, the same rates and thresholds apply in your local currency. For users billing in Japanese Yen, the treshholds are ¥599, ¥699, etc through ¥999.

We're also looking into adding PayPal support as soon as possible for those unable to use Stripe.

While we'd love to say "screw it, fees waived", we're unfortunately not large enough or in a financial position to do that right now, so we're hoping that the rates being pushed as low as we can go without bankrupting ourselves gives our artists the boost they need.

The reduced rates will remain in place until the outbreak slows down and infection control measures begin to be lifted.

After doing some math to see how Waterfall was doing for artists, we realised the results are completely invalid because we forgot to ever put featured posts on the dashboard of mobile users. This meant almost half our users weren't seeing a ton of great art!

Now that it is on the dashboard for those users, we're looking forward to seeing how the numbers change next month when we run the numbers again!

Hey everyone!

As part of the internal redesign for the apps, we have an unparalleled opportunity to redesign systems that are failing our users. To that end, we've decided to hold a workshop session where anyone can drop by with suggestions, issues, or gripes, and discuss them!

The main purpose of these is to identify areas of shortcomings and figure out how to fix them.

We've identified three major categories we want to focus on for this - writers, artists, and fandom content. Each of these has their own workshop and discussion channels. There's also a misc channel for stuff that doesn't fit into these but you feel is still worth mentioning.

Join the Discord and have your say!

Community Update - March 2020 (and the App raffle!)

In February, we...

  • Rewrote the blocking system from scratch, giving us - as far as we know - the most comprehensive and foolproof blocking system of any social media site that we've tested

  • Rewrote the activity page to be vaguely functional

  • Started distributing the first badges

  • Added Invite Links (you can make them in your blog settings!)

  • Released our final update before the backend rewrites to bring you the app!

Site Stats

As of writing, we have 29,024 users, 48,488 blogs, and 315,154 posts.

Site News

Two big things we want to talk about this month! The app beta, and some science stuff. We'll do the science stuff first.

While we're using our computers, it's not really doing anything. As a species, we've managed to trick rocks into thinking for us, and it can think a few billion things a second, but most of the time, for most of us, it's not having to. Why not give it something to do? Enter Folding@Home!

The team behind the project recently took up the fight against COVID-19, but traditionally has focused on conditions such as cancer, Parkinson's, Alzhiemer's, and Huntington's Disease. The basic gist of the project - use your computer's spare resources to simulate how molecules interact with each other, in the hopes of figuring out the cause of these illnesses and potential drugs to target them. On a personal level, I've been running the project on my computers since the mid 2000s, and it's a good feeling knowing I'm contributing. So, we're taking it a step further - Waterfall wants to help!

For those so inclined, you can install the Folding software and let it run - either while you're working, or just set it to run while you're idle (even the half hour you're cooking dinner helps!). Give yourself a name (one of your blogs would be a good choice, so we can find you for rewards) and set your Team ID to 235659 to join Waterfall's team! Occasionally, we'll hand out special badges (and potentially, other rewards) for those helping out. We're doing this as a trial run, but we're hoping to expand to other distributed computing projects in future.

And now the bit most of you care about.

App Beta Raffle

It's finally happening. We've given ourselves a week off to decompress and study, and over the next few days, we'll gradually be easing in to working on the new framework. After some core stuff is designed, we'll be designing the app in parallel with the API. This means a beta isn't far behind at all, and, to make sure we're getting it right, we need testers! We have 150 or so from the Kickstarter, but we'd like more. So, raffle it is! Entering is pretty simple. Just go follow @app-dev.

That's it, you're entered.

There are a few rules though on eligibility:

  • One entry per user. We can tell which blogs belong to which user, so don't try it.

  • Users must be active. We're not putting any limitations on what kind of content you post, but we do need you to actually be someone who uses the site.

  • There's no minimum account age - users new and old (including joining after this is posted) can enter.

  • Android and iOS only. We're not doing BlackBerry or Windows Phone, sorry.

We'll do the drawing on 1st April, and exactly how many are drawn is still up in the air - but we're looking for at least 300 total testers, given our current user count. For those not lucky enough to get picked, don't despair - we'll be noting a few names down for handpicking as well based on their content and activity levels. The actual drawing amount will likely be a percentage of who enters if it goes above that figure, so please - spread the announcement around so we get a higher number of testers!

And unrelated - we rolled out invites! There's a couple badges up for grabs by bringing new people in, and anyone joining through one of your invite links will be automatically following you. Make them in your blog settings! Note that badge options won't show until you're elligible for one.

That's it for now!