Monday, 26 January 2015

Page_Load called twice and another hole in the HTML specification

I was debugging a new site I have created in ASP.Net and noticed that Page_Load was called twice. I was a bit worried because with all the new bells and whistles, I didn't know how easy it would be to debug why this was happening.

Fortunately, a quick Google search revealed the problem. An image tag with a # symbol for the href.

I actually wanted to do that because my image is loaded asynchronously in Javascript and the HTML 5 spec does not allow an empty src attribute, which seems pretty dumb. What is worse is that HTML 4 did allow an empty src but browsers treated it differently!

Anyway, what happens in HTML 5 is that you are still not allowed to use blank but if you use a # symbol or a slash at the start, then the browser treats it has a relative image source. That's pretty dumb, I mean if I was pointing an image source at the same location as the page itself, I would be doing something wrong probably. I'm guessing that the problem is related to shared URL resolution code i.e. not treating the image src url any differently but since the need for empty images at start up is present and the need to load from the current page is probably not, it would be nice if the spec allowed some special way of saying, "don't load the image".

I could of course create the image element when the image is loaded but I think that is more messy compared to having the image sized correctly waiting for the image to load. Everything else would be a workaround.

I realised I had to use a small blank image (1x1) just to give the tag a valid image source but get this: the 1x1 white png in 8 bit colour is a whopping 963 bytes! I tried GIF and BMP and both were similar so either the formats are very bulky or Paint.Net is saving a load of information I don't need with the image.

Thursday, 8 January 2015

Why complexity is always the enemy of good design

I was reading an article about why mobile payments hadn't taken off and why, in the words of the article, they would soon succeed. People quote ApplePay and other attempts to infiltrate the market but all of these, currently, are on a losing battle. I can't find the exact article I was reading but there are loads of them out there!

ApplePay doesn't actually bring anything new to the market other than Apple's name and brand and perhaps a bit of marketing muscle but otherwise they have the same problems as everyone else.

The article said that the barrier to uptake was security but I don't actually think that's true. As in the case of magnetic strips on older credit cards and even chip-and-pin, which wasn't reviewed and therefore as some have said it is not really very strong but we still use/used it because it was a definite advantage over cash in certain situations, namely:

  1. Paying for high value goods didn't require a briefcase full of cash
  2. When cash is stolen, it is usually irretrievable. Cards can be blocked fairly easily.
  3. Credit companies insure most of your purchases, including fraudulent ones.
  4. You can pay for stuff remotely e.g. over the phone or internet
  5. Compared to cheques, the receiver knows that they will receive payment

So, I think we choose to use things that give us an advantage over the current system. So there are certain practical barriers to using mobile payments:

  1. Shops need the kit to handle them - most don't even have credit card proximity readers
  2. Customers need smart phones
  3. Mobiles will generally require a data connection to be available - good luck at Tescos
  4. The customer will need to set up 1 or more potentially incompatible apps
But even if those barriers were suitably overcome, there are still reasons why a mobile payment does not really cut the mustard for us the user.
  1. I don't really have a problem with my credit cards, they work, generally quickly
  2. I need my wallet when I go shopping anyway since it contains store cards, cash and cards. This way, I also need to remember my mobile
  3. What if my phone is faulty or the battery has run out? No purchases for you son!
  4. If it isn't working, the responsibility is now blurred. Is my phone working? My data connection? The payment app? The shop's reader kit? At the moment, there is a card and a reader. I have never had a problem working out what is wrong. Doesn't work? Try another card.
There are of course a whole raft of security issues when it comes to mobile payments. I have found similar problems creating mobile apps. The mobile device is in the red zone - I have limited control over the software on it and how it might be modified or corrupted intentionally or due to hardware problems and I cannot protect anything on the device unless it has a Secure Element, something lacking on many smart phones. What does this mean? It means that people can potentially work out how an app works and then either put some malware on a phone that watches the process and send the data to an attacker or perhaps find a hole in the process for an attacker to create fake transactions and get money from the banks. This is much easier to do than it is to clone cards and some attacks wouldn't even require the attacker to steal the phone, they potentially install malware over usb, bluetooth or nfc or otherwise via a malicious web site that the user visits on their mobile, none of which you can do with a credit card.

What the mobile payment providers are suggesting is a complex raft of design, counter-measures, fraud detection and insurance in order to make it work but complexity is always the enemy of good design. Sure, some technology is complex in one sense but things work well when the complexity has been tested as a unit and then a simple interface provided to the user but in the case where there are various players, various communication links, various design decisions (mostly taken behind closed doors) and a whole lot of trust, I think it is just another problem waiting to happen. We have seen so many high-profile security failures in large organisations that we cannot trust people to safely look after anything more than the simplest systems.

That said, I still don't see people wanting to implement a new system that is generally slower and more fiddly than credit cards just because some bank or whatever can potentially save some credit card transaction fees.

What we really need is something to replace the, now pointless, credit system where people like Visa and Mastercard charge a fortune to do little more than provide credit. Their networks are defunct now that we have an internet to most retail premises, certainly in the Western world. I think these companies would be better off creating their own shared credit network which would run not-for-profit and take payments directly from banks for a fraction of what current credit cards charge.


Tuesday, 6 January 2015

Regular Expressions, Unicode, PHP and pain

You have all kinds of people saying we should do this and that to make our sites more accessible and secure. Whether it's high quality SSL certificates or stuff that makes our sites easier for people with disabilities but some of these are so tricky to do.

Take internationalisation (or i18n for short). Sounds good right? Make your web site translatable into other languages. One of the results of this? My input fields now need to validate all unicode characters, not just a-z.

I have learned this is a pain.

Javascript does not handle Unicode hardly at all, you can only match specific unicode characters. That is shocking in this day and age but hey what can you do? What I had to do was disable client validation for any text field regexes on my site.

PHP is slightly weird. You have to specify u at the end of the regex to make it Unicode. The reason, apparently, is that Unicode can make things slower so it is opt-in but that's not too bad since it is documented.

Now how do I something simple like match a set of Unicode characters?

You would think it would be easy but no. The dot character does not match unicode characters. This is not that surprising because a Unicode character or rather a grapheme - what is displayed on the screen, is not a byte, it is not even a few bytes, it might be a set of byte groups or code points (each of 1 or more bytes) if you like that produce a single grapheme. For instance, the two characters U+0061 (a) and U+0300 (grave accent) will combine to produce a single grapheme that looks like this: à. To add to the confusion, there is also a single character/code point from the standard ASCII character set that is also this same character (for historical reasons). The dot would match the single character but it would match the preceding example twice: once for the a and once for the accent - one of the many "combining marks" that get added to the preceding letter. This is unlikely to be what you want to do.

So what? In my case, I just want a sequence of Unicode characters - I won't be counting them - so why not just use the dot anyway? Because if the user typed in a load of characters that were not actually proper letters but were just combining marks, the regex should fail but with the dot it would pass.

So I could just use \X right? That works in PHP. Well, except that it doesn't seem to work at all even though it is specifically what I wanted to use. No idea why, it just doesn't work.

What I ended up having to do was use the long-hand version of \X (or close enough), which is (?>\P{M}\p{M}*) and which basically says find any Unicode character that is not a combining mark followed by zero or more combining marks.

As with any regex work, test it and test it some more. Think of the types of data you want to and don't want to accept. Also remember that most regexes are not perfect so if you need them to be, either be prepared to do a LOT of testing or have another system that can check the data after it has been passed by the regex to perform other automated tests.

The mysterious Chrome padlock warning symbol

So you visit a site with SSL and Chrome shows you a yellow warning triangle on the SSL padlock. You click on the icon and sometimes it might tell you that you have http links on the https page (a big no no) but sometimes it doesn't say anything, it just shows the warning.

Don't worry, you might not have done anything wrong but this is part of Google's attempt to force web site owners to keep up with security.

If your SSL certificate (or the one on the site you are visiting) is more than 6 months or 1 year old, it is likely to be signed with a hash signature called SHA1 and the problem is that it is old and theoretically possible to fake. This means that somebody could theoretically fake a certificate with a matching signature, insert a man-in-the-middle machine between the site and its users and the people visiting the site would not know because the certificate chain would still match up.

You can now issue ssl certs using SHA256 (also called SHA2, the family it is from) and if you do this, Chrome thinks you are being good and removes the warning. The warning only appears if the certificate has a long time left to run - I guess they are assuming ones that run out soon will be replaced soon with SHA256. SHA256 is much stronger and will buy us another good few years hopefully.

Anyway, re-issuing your SSL cert to change it from SHA1 to SHA256 is usually free, although naturally some work will be required to update these on your servers, especially if you have lots of sites.

Another reminder that security doesn't stand still!

Monday, 5 January 2015

Yii 2 not translating messages

The short version is: "developer error" but I thought a subtle piece of logic might also catch some of you guys out like it did to me.

I was just testing a simple translation using DbMessageSource. The DB tables were setup and they had one entry - my site name. This was because I don't yet know what my product is going to be called but don't want to find and replace it everywhere. It seemed like a good idea to call it "sitename" everywhere and use the translation system to give it a real name.

It didn't work, Even the missing translation event handler didn't fire and I was really confused. There isn't that much logging built into the Yii framework classes to work out what is going on but after a few hours of digging and around and trying things out, I realised:

The translation system won't do anything if your source and target languages are the same!

I hadn't got as far as actually setting the languages up, I just wanted the translation system ready to go. That means that my source and target language are both set at the default (en-US) so the translation system thinks it has nothing to do. It turns out you can enable forceTranslation but the experience made me realise I was using it for the wrong thing. Instead of using the translation system for basically a global variable, I put it instead into my params array and created a single function in the application to return the parameter translated if needed and which could be used throughout the app by calling \Yii::$app->getSiteName()

Easy when you know how.

Sunday, 4 January 2015

Cheating performance on large web applications

One of the biggest problems that a developer or DBA should never face is replicating across multiple databases. If you only ever write to one database and copy to a set of read-only databases, that is not so bad but if you ever have to replicate in two directions, you have a whole world of pain.

But what do you do when you simply cannot get decent enough performance from your one master database? Do you have to accept the replication burden? Not necessarily. You can sometimes cheat.

One of the skills of a good developer is thinking about things from a usability point of view, rather than a purely technical point of view. For example, imagine you are writing a post on Facebook. Most engineers would think that a successful performance would be when the data has been written to the master database and propagated across all other copies of the data right? If that took longer than, say, 3 seconds, you would start to get into problem land. But let us consider this from a user point of view. What do I actually want here? I want to add a post and know that it has been sent to Facebook - I simply want feedback that my job has been done - I don't actually care, or expect, my friends to read it straight-away or 30-60 seconds later if it takes that long to move around the world.

This is a good Facebook example of cheating performance and it is not actually massively complicated. The user types in their text, perhaps attaches an image and what happens next? Before the data has actually been written to the master database, the web page is simply modified in Javascript and the text and image, that both live on the client anyway, are displayed to the user as if they already live on their timeline. That suits the user since it is feedback that the data is accepted. What then happens asynchronously is that the data is queued to write to the master database that could take 30 seconds or more if it is busy so no-one else will see the post until that has happened but it doesn't matter. The user perceives speed and responsiveness, the master database can take its time consuming the data and replicating it across the other databases and Facebook can avoid two-way replication.

It can be hard taking a step back from coding problems and this is where it can be useful to get other people's ideas. Thinking about what you actually need from the user's perspective helps ask the question of which parts need to actually be quick and which parts can we queue asynchronously to happen at some suitable point?

Yii 2 not serving css files

I recently copied a Yii 2 site from an old hard disk to a new installation of Windows 7. You forget about all the things you had to set up the first time and the hacks and permission settings etc.

Anyway, I installed PHP and ran up index.php and although it displayed in the browser, it was obvious that the stylesheet(s) weren't being loaded so the page was laid out wrong. Seemed weird and Yii does have an asset bundling system but one of the files that didn't load was just a static CSS file.

The issue was nothing to do with Yii but my IIS installation. For some unknown reason, when you install IIS, there is an optional feature called "Static Content". Guess what? If you don't install that feature, IIS will not serve static content. Why would you ever want that? Maybe if you were just running a web service, it might offer some hacking protection but anyway, just had to install it - no reboot or anything required - reloaded the page and boom - it worked.

Saturday, 3 January 2015

Large Databases, Primary Keys, Replication and Pain

So I am thinking of some fairly fundamental issues with the design for my new project which could scale to millions of users and 10s of millions of database rows. I don't even know at this point (and don't want to decide) how to handle it when I am running at the limits of the system but I also don't want to spend lots of time and money until I know the system is popular and is being used enough to make it a problem.

So I need to make some smart choices now that might help later even though I cannot tell what will happen. One of these is database primary keys.

A database primary key is very powerful, it is by definition unique and provides a way to uniquely identify a row in a database table. You don't have to have one but in most cases, it is a no-brainer. In many cases, it is a simple integer value that auto-increments every time a row is inserted and which can be used easily in SQL statements like "UPDATE table_name SET column_name = 'something' where id = 123". Of course, this would work without a primary key but the primary key does something significant, it defines how the data is ordered in memory or on disk. This has an affect on both the write time and the read time.

Imagine I am looking for a house number on a road. If the numbers are ordered, as they usually are, it is very easy to find the house (usually!). In the same way, if the database is fetching a record with a given primary key, it can very quickly find it since the data is ordered by the primary key. Imagine if the house numbers were not ordered? You could find it but you would on average have to travel half the length of the road, checking all houses on the way, to find any given house.

For this reason, it should be easy to see why an auto-increment integer works well. It is a simple value to find but also, it is by definition ever increasing so new rows are always added to the end of the existing data. If this was not the case, the database would have to re-order data to accommodate the new rows. For larger tables, this additional write time could be minutes or hours!

So we are happy? Nope, there is something that an integer with its predictability is not good at. Replication. If you have multiple master databases (data can be written to more than one database) but you want that data to be copied across the other databases, you have a problem (usually). You end up with two different rows with the same primary key value in two separate databases. What do you do? You have to copy each unique row to the other database but it will end up with a different primary key value. Row A could be 1 on database X and 2 on database Y. Conversely Row B could be 2 on X and 1 on Y, how confusing! If you have lots of related data in other tables, these also need to be replicated and have their foreign keys modified in the process. What it means is that the master databases are not actually exact copies and you cannot just swap between them from the web servers because their ids will not match.

Of course, you can sometimes avoid these problems (and if possible you should!) by having either completely separate databases that don't replicate (perhaps split by geographic region) or otherwise direct all writes to a single database and then copy across many read-only databases. This way, the replication is only ever in one direction and there will never be conflicts but it does imply a lot of work being done by a single database and that might present very real performance problems. You could also mitigate the problem by seeding the primary keys with different values on different databases so that database A starts at 1 and database B starts at, say, 1000000 so that theoretically the ids will never conflict but again, that is an assumption and if database A ended up with a million rows, you would be stuffed.

This brings us to an interesting alternative to integers: GUIDs Globally Unique Identifiers that use the date and time and some hardware information to generate theoretically unique values. In fact, most are not guaranteed to be 100% unique so you still need to handle the edge case where a duplicate occurs but that might be easier to deal with. Replication becomes easy because for the most part, different rows will always have a different primary key, they will be unique so they are usable for a primary key to uniquely identify a row and although some people seem to dislike the idea of guids in URLs (for some reason), they do solve the replication problem.

So we should just use GUIDs? Eh, no! Guids bring their own problem. Although they are unique (more or less!), they are NOT sequential. What does this mean? When we insert a new row into the database, we would almost always have to re-order the database table on disk to accommodate the new GUID id in the correct order. This is very slow and would be like building a new house in the middle of a road and having to renumber every other house to keep the ordering correct! So GUIDs mostly solve replication issues but are not actually Primary Key friendly on any table that has more than a couple of hundred rows (and not even then if the table is heavily used). Guids also take up more database space (either 36 characters if stored as a string or 16 if stored in raw binary compared to a 4 or 8 byte integer) which kind of adds up over time.

So are we stuck between a rock and a hard place? Well there is one other alternative that merits mention and that is sequential ids. They are still supposed to be globally unique (more or less!) but the generation puts the date and time as the first part of the identifier so that newer ids are always sequentially higher in value and will therefore work well with primary keys but their unique nature makes good work for replication also.

So we should just use these sequential ids? Not necessarily. If you are never going to replicate in two directions across databases, just stick with the integer, it is fastest and takes up the least space in the database. If you will (or might - gulp) have to replicate in two directions, sequential ids are a good choice but there are implications. Firstly, SQL Server for instance provides a method (NEWSEQUENTIALID), but this exists in the database layer and one of the cool things about guids is you can generate them in the business layer or client without having to hit the database first so you could alternatively use another way to generate these sequential ids in code but do your homework and check how unique they are and decide how you will detect and repair a duplicate guid. Perhaps you could do something hybrid and use an id and an integer to permit a few duplicate ids. The primary key would use both - not sure if that's a good idea but it avoids the need to detect duplicates.

Why do software projects overrun?

I was talking to my sister the other day about some work she had done on her house. The builder said it would take 2 weeks and it ended up taking 5. She asked why this was and I said that it was the same reason that a software project (read web site for example) always takes longer than expected.

A customer approaches you and asks how long to build their site. You ask a few ball questions about the design, content and functionality make a load of assumptions and come up with a ballpark figure. About 4 weeks. Someone wrote (can't remember who, sorry) that engineers only have 3 measures of time, short, medium and long. Short for me would always be a day or two, medium would always be a month and long would be 6 months. Everything I estimate is one of those three units. That's stupid but that's what engineers do.

So when I say 4 weeks, what I am actually saying is, "4 weeks assuming that you know exactly what you want, that you don't change your mind or want something different once you've seen the prototype. I assume that the framework will behave as expected and I won't find any weird bugs. Whoever works on it will know exactly how to do everything and won't need any help from anyone else. They will not be sick or on holiday and will never be called in to help on another project. None of my equipment will fail temporarily or permanently, I will not lose any code and I won't hurt my hand playing tennis causing my typing to slow down. The design will be finished and usable and will not require any clarification or rework, it will also include relevant information for the mobile version of your site, which you probably assume will just work. All those things being correct, the web site will take about 4 weeks"

The customer hears, "It will take 4 weeks, even if the apocalypse starts, the office burns down or I suddenly realise that what I asked you to do is not strictly correct and needs to be modified. There is no way it will take even a minute longer than 4 weeks and I will plan its deployment for the day following the 4 week plan".

What happens? We all know what happens. Most customers don't actually know what they want, they mostly don't understand the web domain and sometimes don't even understand their own business! Perhaps they only have one or two people who are design authorities and they take too long to reply to queries, each one holding up the process. Like most customers, they know roughly what design they want but they don't know exactly what they want until the first design is basically finished and they start to change colours here and there, want to move things around like it was MS Publisher rather than the web. They, of course, don't even think about the fact that the site needs to be different on mobile and even if their side is perfect (it never is), you have your own battles to fight. Holiday, sickness, skill shortage, equipment problems, strange software bugs that only occur in one browser during a full moon on a Tuesday and everything else.

4 Weeks EASILY becomes 10 weeks and the customer gets funny, especially if they have to pay for 10 weeks instead of 4. Most software houses are wary of quoting fixed prices because there are so many variables so it becomes a lose-lose. Customers don't want to think they are paying for incompetence and software houses don't want to lose money to customers who can't specify what they want properly.

For some reason, we don't seem able or willing to fix this. It seems to happen time and time again, right up to the higher echelons of government who are possibly the worst of all at paying out stupid money for normal quality work.

We should take some advice from the building industry. They are certainly not perfect but a good builder and reasonable customer can get a decent quality building without losing too much sleep. Even an inexperienced customer who has money and no sense can usually be reasoned with because a building is slightly easier for most people to understand than a web site.

So you want a new house. Do you find a builder and ask for a price and schedule them in? No way! In the UK, you need planning permission, which takes time and requires drawings. The drawings don't have to be full architectural drawings but they need to be of a suitable quality to be submitted for planning consideration. The planning department can give general advice and might even have policy but they do not make the final decision. They might for instance recommend making the building 1 metre further back from the road because, "that's the norm". You would be advised to take their advice and modify your drawings because although the planning committee could overrule the advice from the department, they are more likely to take their side. This stage can be very long and drawn out, especially if you are doing something in an unusual area of the country or your proposed building is non-standard. Importantly, you spend time and money before you even know whether you will be able to build the building at all! This can cost thousands depending on who you employ to help you but what you (hopefully) end up with is the knowledge that your design is approved and will work and it is now in a position to be either further detail designed by the architect/engineer or otherwise priced for construction.

We rarely do this as a separate job in software and it is a shame. This is probably mostly due to the unrealistic speed that customers expect their web sites to be built in - and there are people out there who will imply this is possible. Perhaps one of your friends (who is not picky) got something on wix.com and tells everyone it was quick, easy and cheap so your customer expects the same for their fully bespoke site. Why not have two distinct work packages? Don't be pushing your customer to start paying you for build but cost out the design process separately, let them know that after the design work is complete, they can design not to proceed now or ever and it isn't costing any more than if it was all lumped together but it is more clear what is happening. If it slows down because the customer is not replying to queries, it is obvious what the delay is and the customer won't assume that you are just working on some other part of their site while you are waiting. What you end up with is an approved design that the customer is happy with and the designer is basically verifying as correct and acceptable for web - it could even include some simple web mockups to show how it works and what it looks like on mobile.

After that, the customer can then either get their own people to code it up (if they have any) or even get a number of quotations from developers to implement the site based on the fact that it is designed and complete and won't be changed. Developers need to invoice weekly for their work to ensure they are never out of pocket and if the customer changes their mind on something, it becomes a variation. If it needs re-designing, that's fine, it goes back to the designers who might OK it or tell the customer it will be very difficult. When it eventually comes back, the developers need to talk to the customer about how much of what has already been done can be kept and how much needs breaking down - this should be an easy conversation to have in the same way as a builder who has to knock a wall down after the customer changes their mind can explain why the wall needs knocking down.

One of the reasons why this also makes things easier is that the build phase is much shorter when you have a signed-off design. The implication is that assuming the developers have carefully checked that the design is complete, they should not have to ask the customer anything, they can just build it and release it for testing.

I know the theory sounds easy and I know that the reality is not that easy but having something is better than nothing and just because it isn't perfect doesn't mean that it isn't loads better than what we currently have. The thing I struggle with the most is that we call ourselves engineers but most of us can't engineer our way out of the minefield that is customer management. We need to be good at spotting weaknesses and conceiving solutions. They won't all work as expected but progress is littered with failure.

Friday, 2 January 2015

More Scaling Considerations

In a previous post here, I talked about some scaling decisions when implementing a new web application but I just realised I missed a whole earlier set of design questions when considering scaling and had just assumed that you already had the site architecture decided.

So let's take a step back. I am looking at a new project, not based on anything previously built. My canvas is blank, I have no reason to choose any architecture, framework or deployment other than what is right for the job. The issue is, this site could become super popular so the following lists some things I am already thinking about this new super site. (not in any order)


  1. You cannot predict the future and you won't even correctly predict exactly how your site will perform when it has loads of users. You can perform load testing but that will only match what you think will happen, not necessarily what will. That is fine. Do NOT worry about a perfect solution - you will not be able to create it. However, you can make some sensible decisions early on to help with the scaling, at least to the point when you can make more specialised decisions when you are really scraping the limits of performance at scale.
  2. Consider translation. If your site will become very popular, chances are you will want it available in at least a few other languages. Chinese, Spanish, Portuguese, German, French, Russian perhaps. You should build the functionality in from the beginning and consider how it might work practically i.e. how will you extract the text to translate and inject the translations back in again. What if you are changing the text too quickly, how will you handle that?
  3. Consider your database design early on. If your application is quite data-heavy (many are), a single database will only last you so long but there are various ways to scale out, each with their own challenges. Replication/geo-caching; write-once, read-many; vertical and horizontal sharding. Part of this decision will be based on whether your site is mainly reading data, in which case, writing to a single master and replicating across many read-only databases is fairly straight-forward. You might also create multiple instances of your application across the world so that US data is stored in one database, European data in another etc. Although you shouldn't take too much time worrying about this in the early days, deciding what you will eventually have will dictate some of your decisions now. For instance, integer primary keys are not replication friendly (what happens when two database instances have two different rows at index 123?).
  4. How will you handle shared session when your site is likely to have multiple web servers?
  5. In what ways can you cache data across web servers?
  6. Most bottleneck is in I/O so how can you reduce traffic between web servers and databases (or any other servers you might be using?). Can you reduce the number of database calls by merging requests? Pre-emptively get data from the database in case the user then wants to read that data, avoiding another call to the database?
  7. Consider how the largest organisations will use your system. In my case, the simple case is simple and does not require a lot of functionality but if I start there and then consider the more complicated corporate case, I would then have to retro-fit a load of functionality that is almost certainly needed for the large case.
  8. Do not be afraid to jump around between design type documents. Hand-drawn pictures might work for something but then you think a data definition document or translation spreadsheet might be helpful. Don't feel forced to totally finish a particular document if it isn't really doing what you want. 100% documentation sounds great but is incredibly time consuming and often contains a lot of additional work with formatting/copy etc. that doesn't actually make it into the product.
  9. Spend the most time thinking about the most important stuff. I have a User Management section for this new site but for the most part, this is fairly well-known and common. Another section about object lifecycle, entry and exit points with transitions and the ability to edit it is slightly more involved and I need to consider more things - permissions, widows and orphan objects, cascading changes (or not) etc.
  10. Make sure that just because your site is designed to scale big, that you don't make it too complicated and unusable. Twitter is simple but scalable and your site can be too. You might want to separate the more advanced options out or disable them by default so that people can use the simple case out of the box before trying other things out.
  11. If you are successful and start to scale big, investment or sales should be close behind at which point, you can then start worrying about whether MySql is really up to the job rather than Cassandra or whether you need an asynchronous queueing system to speed things up. The point is that although you haven't predicted the future, you have produced enough slack to buy you the time to decide what you need to change and when.
  12. Consider early on what you will do with side-channels like support. In some ways, it can be easier to include these in the main site but if you are scaling big the chances are that you will want this separated to reduce the burden on the main system. Only join systems that have to be joined!