Difference between revisions of "Summer of Code 2010"

Revision as of 19:19, 16 March 2010

Welcome to Dreamwidth!

An Introduction to Dreamwidth Development

If you're totally new to Dreamwidth, you can find out more about us by reading our Guiding Principles, our Diversity Statement, and our About Us guide. We're a blogging platform and social network, forked in 2008 from the code originally developed by LiveJournal.com and considerably extended. We've developed a strong mentoring program that has brought dozens of new programmers into the world of open source, and we'd love to include you as a part of that.

Getting started in Dreamwidth development is documented in the 'Category:Development' category on this Wiki. We offer free hosted development environments, if you don't have to install the server code yourself, so you can get straight to hacking, or if you prefer, you can install from scratch. (Our system documentation being perpetually behind, we really do recommend that you use our hosted development environment service, but if you prefer to install to your own space, we'll give you as much of a hand as you need -- we have many available mentors who are used to doing a step-by-step install.)

As a project, we have a pretty well-developed network of mentoring, coaching, and tutoring, so if you aren't already involved in Dreamwidth development, our first step will be to introduce you to our development community. We're a close-knit team full of lots of enthusiasm! You'll be assigned an official mentor as part of the Google Summer of Code program, but everyone on the project will be available to answer questions, walk you through things, or give you a few tips and hints.

You can browse or download our code at our Mercurial repositories: the dw-free branch is the branch where main development happens, while the other branches are backend utilities or site-specific branding. Technical and development discussion takes place on Dreamwidth itself: dw_dev for general development discussion, dw_dev_training for questions, tips, and useful tricks, and changelog for our project changelog. (There's also dw_news for general site announcements.)

If you'd like an account creation code for Dreamwidth, so you can explore the site, just email Denise (denise@dreamwidth.org) or Mark (mark@dreamwidth.org), the project owners.

General Skill Requirements

Most of the projects outlined here are going to be user-facing improvements to the web site. Unless otherwise stated, you will want to have some knowledge of modern HTML and CSS in order to create the pages that the users of the site will be interacting with. We're in the process of converting our entire site to use the Template Toolkit templating system, but many of our existing pages use a custom templating system with HTML/CSS resting on top of it.

Perl knowledge is also very helpful given that nearly all of the code you write will be in this language. You aren't expected to be a master, and we certainly hope that you learn a few tricks in your time working with us. Similarly, if you've never coded in Perl in your life, you'll probably do just fine if you have experience with Python or a similar language and a desire to learn -- we regularly provide Perl coaching and training for people who want to get started with us.

Other than that, we don't have high requirements. There are additional skills that would be of benefit to you: MySQL, the architecture of a web site in general, Apache, Linux (Ubuntu), scalability, memcached, Gearman, etc etc. None of these are required, however.

Summer of Code Project Ideas

This page serves as a beginning ideas list for the Google Summer of Code project -- it isn't the sole list of projects that might interest a new developer who's looking for a summer-long project, but it's a list of things that we think are fairly standalone and would work well as modular projects. If you don't see something here to strike your fancy, you might also want to look over our unassigned 'effort-major' bugs, or our list of unassigned user suggestions. (Or, if you're already a Dreamwidth user and have an idea for something in particular you want to work on, we'd love to hear about it.)

Most of these ideas are flexible -- so you'll have lots of leeway no matter what you choose. Still, if you're interested in a more self-guided experience, you might want to choose something that's less specified, while if you're the type who prefers to work to a spec, you should pick a project that has a functional spec written. (Or, if you're really interested in a project that doesn't have a functional spec yet, but you work better with one, just let us know. We'll get you one.)

Likewise, if you want to work on a project that has both backend and frontend elements, but you're only interested in the backend bits, come talk to us before you write your proposal, and we'll see what we can work out. We may be able to find you a partner who can work with you on frontend presentation.

Some of these projects are more standalone than others -- we've marked the ones that are more tied in to how the site's culture works, or things we think might require significant user input. Those projects would be a good choice for someone who's interested in getting experience gathering, working with, and incorporating end-user feedback. (We'll do as much of the end-user communication for you as you'd like, or we'll coach and guide you through the process if you'd like to get experience doing it yourself. You can check the dw_biz and dw_suggestions communities for some examples of our user feedback.)

Project:

Description:

Code area to look at:

Difficulty level:

Skills needed:

Spec available:

Mentor help needed:

Clients & Protocols

LiveJournal has always had, and Dreamwidth would like to have as well, desktop or mobile applications that connect to the web site and allow you to write posts or do other actions that normally you need the site to do. One of our most requested features is for the ability to use the site without having to access it via a web browser.

This is a fairly broad category. It would be a valid project to do a mobile client (iPhone, Android, etc), or similarly, one for the desktop (Mac, Windows, Linux). We've made several changes to the LiveJournal client protocol in order to update it, as we've proceeded along our fork, so many of the existing LiveJournal clients work imperfectly for Dreamwidth, or not at all. We'd love to have a set of DW-native clients.

Likewise, we'd love to implement some more of the more widely popular standard blogging APIs: Blogger, MetaWeblog, Atom, etc, as part of our drive towards interoperability. We'd love to hear your ideas on any of these.

Here are some sample clients and protocols projects:

Desktop clients

Project: Desktop client

Description: Program a downloadable client (for the platform of your choice) that will allow users to interact with their journal remotely, without needing a web browser. Ideally, a client program will allow users to post entries to their journal, read their "reading page", and manage the settings of their account (to the extent available through the protocol), but at a bare minimum, the client program should allow the user to post entries to their journal.

Code area to look at: The LiveJournal client protocol contains all of the functions you'd need to implement.

Difficulty level: Medium

Skills needed: Desktop software development, proficiency in application language of your choice

Spec available: No formal spec, but the LiveJournal download page contains examples of client programs written for LiveJournal. Clients written for Dreamwidth should function in roughly the same way.

Mentor help needed: Your mentor will help guide you through the client protocol -- specifically, the changes Dreamwidth has made to the LiveJournal client protocol -- and help you to determine what further changes need to be made to the client protocol to best serve client authors. (There are also several LiveJournal client authors who have expressed interest in adapting their clients to work with Dreamwidth.)

Mobile clients

Project: Mobile client

Description: Program a mobile/handheld client (for the platform of your choice -- we're particularly interested in an iPhone application, but any mobile platform would be awesome) that will allow users to interact with their journal through their mobile device. Ideally, a client program will allow users to post entries to their journal, read their "reading page", and manage the settings of their account (to the extent available through the protocol), but at a bare minimum, the client program should allow the user to post entries to their journal.

Code area to look at: The LiveJournal client protocol contains all of the functions you'd need to implement.

Difficulty level: Medium

Skills needed: Mobile platform software development, proficiency in the mobile programming language of your choice

Spec available: No formal spec, but the LiveJournal download page contains examples of client programs written for LiveJournal. Clients written for Dreamwidth should function in roughly the same way.

Mentor help needed: Your mentor will help guide you through the client protocol -- specifically, the changes Dreamwidth has made to the LiveJournal client protocol -- and help you to determine what further changes need to be made to the client protocol to best serve client authors.

API extensions

Project: API Extensions

Description: In addition to the existing LiveJournal protocol, implement and integrate one (or more) additional open blogging APIs: Blogger, MetaWeblog, Atom, etc. We have rudimentary Blogger and Atom support, but no MetaWeblog support.

Code area to look at: The LiveJournal protocol is where the existing protocol lives -- all transactions with the site go through the protocol, even web-based transactions. Additional APIs would need to be implemented in a way that would tie in to the existing protocol. We have existing bare-bones implementations of the Blogger and Atom APIs, but they aren't very fully-featured and could use a lot of expansion.

Difficulty level: Medium

Skills needed: Backend programming knowledge, Perl, familiarity with chosen API

Spec available: No Dreamwidth-specific spec, but the specs for various APIs are widely available.

Mentor help needed: Your mentor will answer questions about the existing client protocol and help you make decisions about how to translate other APIs' features to Dreamwidth's feature set.

OAuth

Project: Implement OAuth Support

Description: Right now the various APIs that Dreamwidth supports require users to provide their password to third party tools and web sites. This is not up to date with modern best practices, so we'd like to add support for OAuth.

Code area to look at: The crossposting system that we've developed (in which Dreamwidth acts as a native client for other blogging platforms) is in cgi-bin/DW/External.

Difficulty level: Medium

Skills needed: In addition to our general requirements, this project requires you to read and understand OAuth and be able to implement it on Dreamwidth in Perl -- you can of course use modules distributed on CPAN.

Spec available: No spec on the Dreamwidth side, but the OAuth spec is available.

Mentor help needed: Your mentor will explain and walk you through the existing crossposting functionality and be available to answer questions about the details of OAuth implementation.

Wordpress or Movable Type integration

Project: Wordpress or Movable Type import/crosspost

Description: Dreamwidth allows importing archives from, and crossposting entries to, LiveJournal-based platforms already. We would like to add support for Wordpress and Movable Type so people don't need to rely on third-party plugins/extensions/scripts, and so people can post from Dreamwidth if they choose.

Code area to look at: The crossposting system that we've developed (in which Dreamwidth acts as a native client for other blogging platforms) is in cgi-bin/DW/External. Currently, the system only speaks to LJ-based sites, but Wordpress support would go there. Code for the importer is in a few different places, but primarily cgi-bin/DW/Worker.

Difficulty level: Low-Medium, depending on how much detail you want to get into. Both the crossposting system framework and the importing system framework have had a lot of bugs worked out of them over the past year. Adding WP or MT support would plug into this existing framework and expand it.

Skills needed: Our basic requirements, plus familiarity with (or desire to learn) the other platform.

Spec available: Not currently

Mentor help needed: Your mentor will explain and walk you through the existing crossposting and importing functionality and be available to answer questions about the details of WP/MT integration.

Facebook Connect

Project: Implement support for Facebook functionality.

Description: We would like the ability to support some of the Facebook Connect functionality. For example, allow our users to opt-in to sharing content they post on Facebook, or giving people the ability to post something interesting they see on Dreamwidth to Facebook (Share This type functionality), or even live chat functionality.

There are a lot of potential modes of integration. This project will require some independent research and thought into what makes the most sense for Dreamwidth's particular userbase and use cases, so it would be a good choice for someone who's particularly interested in working with end-users to create and implement a spec.

Code area to look at: Depends on the particular implementation you choose, it's likely you might wind up touching everything. The crossposting system might be a good place to start looking, though.

Difficulty level: Medium to High

Skills needed: Knowledge of Facebook and an interest in connecting it with Dreamwidth. Given the breadth of this project it would help to have experience with Perl and MySQL already.

Spec available: No; there has been some some discussion in the dw_suggestions (Crossposting of non-public items to Facebook, Tool to post links to DW entries on Facebook), so there's some initial user feedback, but this would require a lot of careful thought.

Mentor help needed: Your mentor will help walk you through the process of gathering and processing user feedback (or take point, if you're less comfortable), as well as helping you form a spec that takes site culture into consideration.

ping.fm support

Project: Implement ping.fm support for Dreamwidth.

Description: To make it easier for our users to post, we'd like to support the ping.fm API. Whether it's just ping.fm API support or a more tight integration with ping.fm's services is up to you.

Code area to look at: The LiveJournal protocol is where the existing protocol lives -- all transactions with the site go through the protocol, even web-based transactions. Additional APIs would need to be implemented in a way that would tie in to the existing protocol.

Difficulty: Medium-high, depending on how detailed you get with your implementation.

Skills needed: Our basic requirements, plus a good grasp of how ping.fm works (or should work). You'll be making a lot of the integration choices yourself, since we're less familiar with the ping.fm service, so you should be comfortable with working without specs.

Spec available: Not yet, although this is bug 1675 in our Bugzilla.

Mentor help needed: Your mentor will help walk you through the process of gathering and processing user feedback (or take point, if you're less comfortable), as well as helping you form a spec that takes site culture into consideration.

Big Features

Media Hosting

Project: Implement a system for storing and displaying user submitted media (photos, video, audio, etc).

Description: Dreamwidth has a lot of functionality for manipulating, storing, and displaying text. One of our serious downsides right now is not having the ability for users to store other media on the site and share it with the people that subscribe to them. This is something that we consider a high priority project for us, and we'll provide a lot of support to make this happen.

This project will be considered successful if, at the end, users are able to upload images to the site, post those images in their journals, and other users can view them. Other types of media (audio, video, etc) and other functionality (automatic rotation, resizing, transcoding, etc) will be considered as bonus objectives that we will plan but may not implement.

Code area to look at: We don't currently have any media hosting, so all of this would be written from scratch.

Difficulty level: High, you're implementing an entirely new feature on the site and will have to touch many parts of our codebase.

Skills needed: For this project you should already be familiar with Perl, SQL, and have an interest in media. You will be working with us to implement everything from backend storage (using the MogileFS system) to image manipulation (using distributed tasks in Gearman) to front-end management pages.

Another option might be to take an existing free-software image hosting/gallery system and integrate it with our code. This variation of the project would involve more backend work (MogileFS storage, Gearman distributed tasks, etc) and less front-end management.

Spec available: Partial; we've solicited user feedback on the project, but haven't written a formal spec yet.

Mentor help needed: Your mentor will help guide you through the design stages of the feature, as well as teaching you about how to design for high availability and maximizing database/webserver performance for production code.

Calendar Functionality

Project: Implement calendar/event support.

Description: Our users want the ability to maintain a calendar of events on Dreamwidth. These events can be located in the real world (groups of people who want to schedule a party, f.ex) or they could be entirely online (a fiction challenge). The basic idea of a calendaring system is well understood, but Dreamwidth has some special requirements around security, support for online only events, etc.

We also want the ability for people to integrate their local calendaring application (iCal, Google Calendar, whatever) with their Dreamwidth account. Bonus points if the user can schedule events through their application of choice and have those events show up on the web site.

Code area to look at: We don't currently have any calendaring support, so this would be written from scratch.

Difficulty level: High; you will be working with Dreamwidth at its most basic level.

Skills needed: It would help if you are already familiar with calendaring, but that is certainly not required. It would also be useful given the breadth of the project that you are already familiar with Perl, MySQL, and have some web development experience.

Spec available: Not currently

Mentor help needed: Your mentor will help guide you through the design stages of the feature, as well as teaching you about how to design for high availability and maximizing database/webserver performance for production code.

Embedding Dreamwidth

Project: Drastically improve the possibilities and options for embedding your DW journal into your external website.

Description: As it stands, there's limited support for embedding your DW journal into your external website, but when we say limited, we mean "limited" -- it's possible, for instance, to 'forward' your domain to your DW account through the use of DNS manipulation, but once you get past the first page of the embedded journal, the URLs revert to the Dreamwidth version. (So, comment pages, entry pages, etc, etc, all reflect the dreamwidth.org URL.)

We want people to be able to embed their DW account into their website seamlessly, so the journal is hosted on DW (and has access to all the community-based features such as access and privacy controls), but appears to be hosted on the user's website. This project will involve taking the existing skeletal support and significantly expanding it.

Code area to look at: cgi-bin/LJ/Setting/DomainMapping.pm, htdocs/manage/domain.bml, cgi-bin/LJ/Session.pm, and cgi-bin/Apache. (Limited support for this is scattered throughout all of the codebase to an extent, and the project will certainly touch on a lot of it, but that's a good place to start.)

Difficulty level: High. This project will touch on a lot of areas throughout the site, so it's best suited for someone who already has strong Perl and backend skills -- it's not a beginner project.

Skills needed: Because this touches on authentication issues, cookie use, etc, this should be a project for someone who already has very strong skills with internet security and desigining code that's well-hardened against scripting attacks.

Spec available: Not as yet; if you're interested in this project, we'll help you build one.

Mentor help needed: Your mentor will concentrate with you on security issues, since this is a high-risk feature.

Pingback support

Project: Add and integrate pingback.

Description: Everybody likes to know when people are talking about them, so we'd like to add pingback support when someone links to a user's Dreamwidth entry, both natively on Dreamwidth and elsewhere on the internet.

The full pingback system will ideally involve a pingback publishing/moderation queue, treatment of pingbacks as separate from comments, and integration of pingback display into S2, our custom journal styling system. If frontend isn't your thing, we'd also be happy with just the backend framework, but we'd prefer the full package.

Code area to look at: We don't have any pingback support in the code, so this will be working from scratch. You'll likely be adding a new worker to cgi-bin/DW/Worker, and display views into our custom journal-styling system, S2, specifically core2.s2. (Don't worry, we'll teach you S2.) LiveJournal did a pingback implementation, but we want to do ours in a different way -- still, their code is there to examine as a starting point, since it was released in their open source branch.

Difficulty level: Medium-high to high. S2 is fairly easy to learn, but does take a bit of time and effort, while this will also involve building a user-facing admin frontend for managing and moderating their pingback queue.

Skills needed: The usual, plus frontend/presentational skills if you want to take on the display and user-facing end of things.

Spec available: No, but we have bug 270 for it, plus the pingback specs. If you're interested in this project, we'll help you gather and process user feedback about the best way to implement things in order to combine the basic functionality with the site's commitment to protecting users' privacy and upholding privacy concerns.

Mentor help needed: Your mentor will help walk you through the process of gathering and processing user feedback (or take point, if you're less comfortable), as well as helping you make decisions about the best way to implement the pingback spec to take into account the site culture.

Systems/Administration

Production Monitoring

Project: Implement additional tools for monitoring of the production environment.

Description: All sites require production monitoring: keeping an eye on what is going on inside the servers. Disk space, CPU/memory usage, backups, RAID health status, etc etc. Dreamwidth has need of someone with the ability to code and an interest in the production end of the stack to implement some new tools for monitoring.

We use industry standard environments such as Nagios and Cacti, so most of the code written will be in the form of Nagios/Cacti plugins. However, we have a few of our own tools that we use that do things such as analyzing the packets sent to the MySQL servers to look at the queries being executed, or looking at the HTTP requests being sent to our load balancers live.

Code area to look at: The dw-ops repository has our existing monitoring plugins and configs.

Difficulty level: Low to Medium

Skills needed: You must be interested in production environments. (Future sysadmins?) You should also have at least a rudimentary understanding of Linux servers and really strong attention to detail. Difficulty varies by the exact tasks you work on.

Spec available: No formal spec, but we have a good sense of what additional monitoring tools we need that we're happy to talk over with you.

Mentor help needed: Your mentor will help coach you through the existing tools and the process of writing additional ones, as well as tutoring you on the basic principles of what sort of code-based support is useful for monitoring high-availability websites.

Medium Difficulty Improvements

WAI-ARIA Improvements

Project: Implement support for WAI-ARIA to improve site accessibility.

Description: Dreamwidth has a strongly stated goal of supporting accessibility whenever it can. We would love if someone with a passion for this area would be interested in helping us to achieve a better level of accessibility for all of our site's functionality. In particular, we will determine what the most popular and hardest to use pages are and then you will work to modernize and make those pages functional for more people.

Code area to look at: We've been working on adding basic WAI-ARIA landmark roles to many of our site pages in bug 820, as well as a discussion here in WAI-ARIA, but there's a lot more work to be done.

Difficulty level: Medium

Skills needed: Interest in what makes a site accessible or not and how to get it there. Understanding accessibility is an ongoing process but the skills you learn will help make you a better web developer for the rest of your career.

Spec available: The WAI-ARIA specs are available, but how to implement them on Dreamwidth is something you'll get to decide, with consultation with the dw-accessibility project team.

Mentor help needed: This project will likely wind up with two mentors, one helping to guide you on the code and someone from the dw-accessibility project team guiding you on accessibility guidelines. This is a great project for someone who's interested in learning more about web accessibility, since we have the resources in place to train you on various assistive devices and how to work with them. (We may also be able to provide some help with obtaining assistive technology devices/programs/etc for testing and design purposes, but at the very minimum we can put you in touch with users of those technologies.)

Account/Usage Statistics

Project: Implement site admin usage tracking and statistics.

Description: Dreamwidth has a few site-admin tools for tracking information about account types and site usage, but we're still far behind where we need to be. For example, we would love to know exactly how many accounts are created per day, where they were referred in from, how many accounts are purchased, account turnover, average size of posts, usage of various post features (tags, cuts, images, etc), and many more.

There are a thousand things we can track, and most of the infrastructure for recording the data is already in place. What we lack most is connecting new data points to the statistics system and then a visualization frontend so we can look at the data. We want to be able to select data to view, pick a timeframe, and then view the resulting graph.

An extension to this project would be nightly/weekly/etc reports. I.e., using the administrative interface, configure that we want to see reports with certain data at certain intervals. (This is a bonus objective.)

Code area to look at: The backend stats framework is in cgi-bin/DW/StatData.pm and StatStore.pm, with individual stat-gathering types in cgi-bin/DW/StatData/.

Difficulty level: Medium to High, depends on how fancy the frontend is.

Skills needed: You should be interested in data, statistics, and graphing. If you already have experience with GD or a similar graphing library, so much the better.

Spec available: A list of "wanted" stats is in bug 124, as the meta-bug for the stats project. Individual projects will tie in to that existing meta-project.

Mentor help needed: Your mentor will help coach you through the existing stats system, as well as providing you with feedback about what sort of stats-gathering would be most helpful from a project perspective.

Rich Text Editor overhaul

Project: Replace the existing rich text editor with a more modern package and integrate it into the site posting page.

Description: Currently, the rich text editor we use is an older (and glitch-filled) version of FCKEditor, and we'd like to upgrade to something more fully-featured and modern. This project will involve comparing available packages, selecting one, and doing all of the work to integrate it into our codebase, including (especially) the customization needed to support our custom site-specific markup and posting formats. There's a lot of directions you could take this (such as offering a limited version of the RTE for commenting in addition to journal updates, etc), but at the very least it will involve a lot of frontend work.

Code area to look at: The existing FCKEditor files are in htdocs/rte, with the javascript in htdocs/js (richtext.js and rte.js). We'd like to replace it entirely, but that's a good place to start looking for some of the details on what was done to the original RTE to integrate and customize it.

Difficulty level: Medium, but likely to involve a lot of careful testing and cross-browser work. This would be a good project for somebody who likes frontend work, bugfixes, and investigating/reproducing trouble reports, and has the careful attention to detail necessary to perform extensive regression testing.

Skills needed: HTML/CSS and strong Javascript skills, along with familiarity with standards compliance. Dreamwidth supports a variety of modern browsers (see Design Philosophy for more details), so there'll be a lot of cross-browser regression testing. Basic familiarity with accessibility needs for assistive technology is also a plus, although the dw-accessibility project team will work with you to make sure that your work is as broadly accessible in with as many assistive devices and technologies as possible.

Spec available: No, but we will provide a list of the DW-specific markup that needs to be integrated, as well as suggestions on what additional features would be useful.

Mentor help needed: Your mentor will help you select an alternate rich text package and coach you through the process of integrating it with Dreamwidth.

"Named guest" commenting level

Project: Add support for "input your name & URL at time of commenting" form of named-anonymous/pseudonymous commenting.

Description: Currently, there are only three forms of authentication supported at the time of commenting: purely anonymous, OpenID-authenticated, and DW user. We'd like to add a "guest" commenting level in between anonymous and OpenID-authenticated, for someone to provide a name/URL (without authentication) but allowing for persistence of pseudonym through the conversation.

Code area to look at: The commenting system is in cgi-bin/LJ/Comment.pm and cgi-bin/LJ/Talk.pm.

Difficulty level: Low to medium. This would be a good beginner project, since there's a solid spec and the code paths are fairly well commented.

Skills needed: Our basic skillset should serve you well here.

Spec available: Yes, at bug 759.

Mentor help needed: Your mentor will help coach you through implementing the spec and tweaking it as necessary to suit the community needs and feedback.

Polling system

Project: Significantly improve the existing poll system.

Description: Posting polls is a useful and compelling paid-user enhancement we offer, and we'd like to upgrade the end-user controls/user experience, as well as providing the user tools necessary to do advanced interpretation and data analysis on their poll data. This project can include either or both: maybe you'd like to work on adding a statistical analysis/data-processing system for processing poll data, or maybe you'd like to implement more question types, views for perusing the results, etc.

Code area to look at: cgi-bin/LJ/Poll.pm is the poll framework.

Difficulty level: Low to medium-high, depending on how detailed you'd like to get.

Skills needed: Depending on which direction you'd like to take this in, this could be an easy project for someone with good frontend/HTML skills, all the way to a complex project suitable for someone with a strong statistics and data analysis background.

Spec available: We have a number of poll-related bugs open, all with varying details and levels of spec available. You'll get to pick and choose which ones you'd like to include in your project, since any or all of them would be a great improvement.

Mentor help needed: Your mentor will help you choose which elements to include in your project, as well as giving you help with collecting and processing user feedback.