Difference between revisions of "Dev Getting Started"

Latest revision as of 22:09, 17 December 2022

Welcome to Dreamwidth code development! This page is a guide to how to get started with hacking on the Dreamwidth code. It's intended to cover all steps of the process from the very beginning, but each individual step is a brief overview -- follow the links for more detail. If you're confused about anything here, you can ask us in dw-dev-training or the latest dw-dev question thread, or come talk to us in the Discord Chat channels. One-on-one mentorship may be available if time permits.

This is geared at people who have little to no development experience. If you've got more experience and just need the details of how our project differs from other open source projects out there, check out the Dev Quick Start list.

The languages and skills Dreamwidth development uses are:

Perl (the majority of the project)
BML (custom templating language inherited from LiveJournal which we are currently phasing out)
TT (Template Toolkit - better templating language, used for new pages)
S2 (custom templating language used in journal styles)
HTML / CSS / Foundation
Javascript / JQuery
MySQL and DBI (the Perl interface to MySQL and other SQL databases)
Test::More (for writing automated tests)

You don't need to know them all! Some familiarity with at least one will be helpful, but we welcome people at all levels, and are happy to answer questions and give advice as long as you're willing to learn.

One time only: Initial setup

What you need:

A working Dreamwidth installation

To hack on the Dreamwidth code, you'll need a development environment: a place where you can install the Dreamwidth code yourself, to make changes and submit them. We offer hosted sandbox development environments where we'll install the code for you. We call them Dreamhacks, and they're available free of charge for anyone who's interested in contributing to the project. (It's possible for you to install the code yourself on your own server, but we don't recommend it for beginners. Even the most experienced Dreamwidth developers mostly work on Dreamhacks.)

Apply for a Dreamhack: this will give you a hosted development environment with everything already ready for you to get started.

Go through the Beginning dev checklist and follow along with the steps for getting your Dreamhack account set up. Most of these steps are tasks you'll only have to do once to make sure your development environment is set up to hack on.

A GitHub account

GitHub is the version control system we use. It keeps the "master version" of the Dreamwidth code, keeps track of the changes you make, and allows you to submit those changes to us for us to review and incorporate them.

Create a GitHub account with your preferred username.

Fork the dreamwidth repository from the Dreamwidth organization.

A basic sense of the code structure and how to find things

You aren't going to know everything right up front, of course, but there are a few things you can look through and familiarize yourself with.

Familiarize yourself with Perl, the language that the majority of DW development is done in. The best reference that we've found to point people at is Picking Up Perl. You absolutely don't need to be a Perl guru to submit a patch, but you should be familiar with the basic syntax.

If you'd like to practice first, check out OpenHatch's Training Missions -- they'll let you practice with some of the tools you might need! (In particular, you might find their Git training mission helpful.)

Browse through some of the Dreamwidth source code to get a good sense of where things are and what the setup looks like. The basic directory structure, as well as some guidelines about what goes where, can be found in Directory Structure.

Read the Programming Guidelines. (You may not understand all of the items there, but you can familiarize yourself with them, and start noticing when existing code doesn't follow the rules as you read.)

If you start to get overwhelmed, don't worry! Go do something else for a bit and come back to this later -- this doesn't all have to be done at once. You can read an old dev pep-talk. Also, have a look through the Epic List of Things Real Dreamwidth Programmers Do if you start feeling a bit of impostor syndrome. Finally, you can always ask for help in irc or in dw-dev-training -- everybody who's working on the project now was in your shoes at one point, and we all remember what it's like to feel overwhelmed and over our heads. We want to help you!

If you run into problems following any of these instructions, we also want to know about it so we can fix this documentation. You can let us know in irc or in dw-dev-training, or make a note on the Dev Wanted How-To or Installation Wanted How-To pages to describe what you found confusing.

Finding something to work on

Once you've gone through the initial setup, pick an issue to work on. The issues are kept in GitHub Issues.

Look through the curated:beginner label for issues that someone thinks would make a good first issue to work on. From time to time, people also post lists of "babydev bait" to dw_dev_training, and there is a masterlist.

Let people know that you're working on the bug by assigning it to yourself. Comment on the issue you've chosen with some variation on the word "claim" ("claim", "claimed" and "claiming" all work, and can be surrounded by any other text). This will trigger our bot to assign the issue to you.

Create a new branch in git for this issue so you have a place to make your changes.

Making your changes

Make sure your git repository is up to date.

Make your changes, probably in your Dreamhack (editing text files, though the tricky part is finding things).

Restart Apache and test your changes on the Dreamhack. It's a good idea to test multiple use cases, including logged in/not logged in, community vs personal journal (if applicable), personal journal with different settings, etc.

Commit your changes to git.

Push your new git branch to your github account.

Submitting your changes

Once you're satisfied that your changes fix the issue and don't introduce any new bugs, open a pull request. This sends your changes to us for us to look over in review.

A senior Dreamwidth developer will look over your changes and see if they can be accepted or if there are things you need to change. If there are things that could be improved, the reviewer will comment on the commit and give you notes on what can be improved. Don't worry if your changes don't pass review on the first time -- it's really common for pull requests to go through several versions before they're committed.

If your pull request needs more changes, you can go back to the branch you made for the change and make those changes. Once you commit them to your branch and push your changes to make them public, the pull request will update automatically. However, this doesn't fire off a notification to the folk reviewing the code, so please leave a comment on your pull request to let people know that the requested changes have been made!

When your pull request is accepted, the person who accepts it will close the pull request and the issue that the pull request fixed.

The next fix!

Once your pull request is accepted and merged into the main branch, you can delete the branch for that issue on your GitHub account.

Switch back to the 'master' branch on your Dreamhack and update to the most current code.

Pick the next issue to work on!

What'd we miss?

If you run into anything in the process that isn't well-documented, or you have questions, you can ask in the dw_dev_training community or in the #dreamwidth-dev irc channel. You should also make a note on the Dev Wanted How-To or Installation Wanted How-To wiki pages so we can document it better for the next person to come along!

@@ Line 1: / Line 1: @@
-== Welcome to Development ==
+Welcome to Dreamwidth code development! This page is a guide to how to get started with hacking on the Dreamwidth code. It's intended to cover all steps of the process from the very beginning, but each individual step is a brief overview -- follow the links for more detail. If you're confused about anything here, you can ask us in <dwcomm>dw-dev-training</dwcomm> or the latest <dwcomm>dw-dev</dwcomm> [https://dw-dev.dreamwidth.org/tag/questions question thread], or come talk to us in the [https://dw-dev.dreamwidth.org/209778.html Discord] [[Chat]] channels. One-on-one mentorship may be available if time permits.
-This document is designed to give you a rough guide to setting up the DW code on your server.  There is, currently, a high expectation of "tech savvy."  If you do not know what sudo is or how to get a root console, then you probably are going to get pretty lost in this.
+This is geared at people who have little to no development experience. If you've got more experience and just need the details of how our project differs from other open source projects out there, check out the [[Dev Quick Start]] list.
-Doesn't mean you shouldn't try, but forewarned is forearmed!
+The languages and skills Dreamwidth development uses are:
-Help can be found on IRC (#dw on irc.zhzh.org) or on the mailing lists.
+* Perl (the majority of the project)
+* [[BML]] (custom templating language inherited from LiveJournal which we are currently phasing out)
+* [[Routing and Template Toolkit|TT]] (Template Toolkit - better templating language, used for new pages)
+* [[Styles|S2]] (custom templating language used in journal styles)
+* HTML / CSS / [[Foundation]]
+* Javascript / JQuery
+* MySQL and DBI (the Perl interface to MySQL and other SQL databases)
+* Test::More (for writing automated tests)
-The original LJ Server documentation is [http://www.livejournal.com/doc/server/ here].
+You don't need to know them all! Some familiarity with at least one will be helpful, but we welcome people at all levels, and are happy to answer questions and give advice as long as you're willing to learn.
-== Requirements ==
-Here is what you'll want for a development environment.  Sure, you can probably get away with less or different, but that's not what we're going to support.  The DW code is known to work with the following.  But we'd love to hear of your results using something else.
+== One time only: Initial setup ==
-=== Linux based server ===
+What you need:
-We are presently using Ubuntu.  This may stay or become Debian or Centos at some point, that's undecided.  But for now, Ubuntu is the supported distro.  If you use anything else, you will need to be familiar enough with the differences to work out any different steps on your own.  (But we'd love to hear success stories and get documentation for other ones!)
+=== A working Dreamwidth installation ===
-Notes for other OSes:
+To hack on the Dreamwidth code, you'll need a development environment: a place where you can install the Dreamwidth code yourself, to make changes and submit them.  We offer hosted sandbox development environments where we'll install the code for you. We call them [[Dreamhacks]], and they're available free of charge for anyone who's interested in contributing to the project. (It's possible for you to install the code yourself on your own server, but we don't recommend it for beginners. Even the most experienced Dreamwidth developers mostly work on Dreamhacks.)
-*[[Running Dreamwidth on a Mac OS X system]]
-=== 64-bit (x86_64 typically, NOT i686/i386) ===
+* [http://hack.dreamwidth.net/ Apply for a Dreamhack]: this will give you a hosted development environment with everything already ready for you to get started.
-This is something you can probably get away without doing, but you'll be unable to test some things and others might be a bit wonky.  Notably, you need the 64 bit version of Perl in order for the pack/unpack operations to work for memcache.
+* Go through the [[Beginning dev checklist]] and follow along with the steps for getting your Dreamhack account set up. Most of these steps are tasks you'll only have to do once to make sure your development environment is set up to hack on.
-But if you absolutely do not have a 64 bit system, then you might still be okay.  Don't use memcache and don't try to use more than 30 friend groups and you shouldn't run into any big issues.
+=== A GitHub account ===
-If you're not sure whether you have a 64 bit system or not, check out the collection of tips on this page: http://www.stata.com/products/64bit.html
+[[Git Getting Started | GitHub]] is the version control system we use. It keeps the "master version" of the Dreamwidth code, keeps track of the changes you make, and allows you to submit those changes to us for us to review and incorporate them.
-=== 512mb RAM ===
+* Create a GitHub account with your preferred username.
-I have gotten away with less.  Depends on how much traffic you expect and what else you are running on the box.  Also, if you intend on running TheSchwartz workers or not.  If you do not, and you only run the web site code, then it will probably work on 256.
+* [https://help.github.com/articles/fork-a-repo/ Fork] the [https://github.com/dreamwidth/dreamwidth/ dreamwidth] repository from the Dreamwidth organization.
-=== Dedicated! ===
+===A basic sense of the code structure and how to find things===
-The LJ/DW code is pretty heavy, and doesn't like other things running.  You should expect that if you are not that technical, you will need a fairly blank server to run it on.  But if you are technical you can do some pretty convoluted setups...  (This one is more of a recommendation and not a hard requirement.)
+You aren't going to know everything right up front, of course, but there are a few things you can look through and familiarize yourself with.
-=== root ===
+* Familiarize yourself with Perl, the language that the majority of DW development is done in. The best reference that we've found to point people at is [http://www.ebb.org/PickingUpPerl/ Picking Up Perl]. You absolutely don't need to be a Perl guru to submit a patch, but you should be familiar with the basic syntax.
-If you do not have root on the box, you are in for a rough road.  Sure, the code doesn't need root privileges to run.  But there are a lot of libraries that need to be installed on the system.  If you are using a shared development environment where the owner has installed the modules for you, then you should be fine.
+* If you'd like to practice first, check out [https://openhatch.org/missions/ OpenHatch's Training Missions] -- they'll let you practice with some of the tools you might need! (In particular, you might find their Git training mission helpful.)
-=== MySQL ===
+* Browse through some of the [http://github.com/dreamwidth/dw-free Dreamwidth source code] to get a good sense of where things are and what the setup looks like. The basic directory structure, as well as some guidelines about what goes where, can be found in [[Directory Structure]].
-You will need a MySQL database.  It can be local or remote, it doesn't matter really.  As long as you have the ability to create/drop/alter tables in a directory, that's what matters.
+* Read the [[Programming Guidelines]]. (You may not understand all of the items there, but you can familiarize yourself with them, and start noticing when existing code doesn't follow the rules as you read.)
-== Preparation ==
+* If you start to get overwhelmed, don't worry! Go do something else for a bit and come back to this later -- this doesn't all have to be done at once. You can read an old [http://azurelunatic.dreamwidth.org/6731755.html dev pep-talk].  Also, have a look through the Epic List of [[Things Real Dreamwidth Programmers Do]] if you start feeling a bit of [https://en.wikipedia.org/wiki/Impostor_syndrome impostor syndrome]. Finally, you can always ask for help in irc or in <dwcomm>dw-dev-training</dwcomm> -- everybody who's working on the project now was in your shoes at one point, and we all remember what it's like to feel overwhelmed and over our heads. We want to help you!
-Okay, for the sake of argument, let's say you have a machine (virtual or otherwise) that matches the above requirements.  Getting the DW code setup is fairly straightforward.
+* If you run into problems following any of these instructions, we also want to know about it so we can fix this documentation. You can let us know in irc or in <dwcomm>dw-dev-training</dwcomm>, or make a note on the [[Dev Wanted How-To]] or [[Installation Wanted How-To]] pages to describe what you found confusing.
-=== User Account ===
+== Finding something to work on ==
-Create a user account that will be used for the code.  This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code.  That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.
+Once you've gone through the initial setup, pick an issue to work on. The issues are kept in [http://wiki.dreamwidth.net/wiki/index.php/Github_Issues GitHub Issues].
-=== Setup User Account ===
+* Look through the [https://github.com/dreamwidth/dw-free/labels/curated:%20beginner curated:beginner] label for issues that someone thinks would make a good first issue to work on. From time to time, people also post lists of "[http://dw-dev-training.dreamwidth.org/tag/babydev+bait babydev bait]" to <dwcomm>dw_dev_training</dwcomm>, and there is [http://dw-dev-training.dreamwidth.org/45772.html a masterlist].
-I'm assuming that you use the bash shell.  If you do not, then this section does not apply.  You will have to find the shell specific way of setting environment variables.
+* Let people know that you're working on the bug by assigning it to yourself. Comment on the issue you've chosen with some variation on the word "claim" ("claim", "claimed" and "claiming" all work, and can be surrounded by any other text). This will trigger our bot to assign the issue to you.
-For bash, you should look in the home directory of the user account you just created for a file named .bashrc or .bash_profile or maybe even .profile.  One of these files will have some commands in it (alias commands usually).  You will want to add one more line to this file:
+* [[Git How To#How to create a new branch for feature development / bugfixes|Create a new branch in git for this issue]] so you have a place to make your changes.
-    export LJHOME=/home/dw
+== Making your changes ==
-Or whatever your user account's home directory is.  Yes, the variable is named LJHOME.  We'll live with it for now.  ;-)
+* Make sure your git repository is [[Dev_Maintenance#Updating_dw-free|up to date]].
-Test this.  Log out of your user account and log back in, then type:
+* Make your changes, probably in your [[Dreamhacks|Dreamhack]] ([[Beginning_dev_checklist#Editors|editing text files]], though the tricky part is [[Dev Finding Things|finding things]]).
-    echo $LJHOME
+* [[Dev_Maintenance#Restart_the_server|Restart Apache]] and test your changes on the Dreamhack. It's a good idea to test multiple use cases, including logged in/not logged in, community vs personal journal (if applicable), personal journal with different settings, etc.
-You should see /home/dw or whatever you set it to.  If you don't, then you didn't get the export line in the right place.  (Hey, I'm not here to teach basic sysadmin stuff.  You should know how to set environment variables.)
+* [[Git How To#The_Basics|Commit your changes to git.]]
-=== Prep Packages ===
+* [[Git How To#The_Basics|Push your new git branch to your github account.]]
-Next, you will need two package management systems installed: subversion and mercurial.  This is typically done on an Ubuntu system with:
-    apt-get install mercurial subversion
+== Submitting your changes ==
-You will need to be root to run that command.
+* Once you're satisfied that your changes fix the issue and don't introduce any new bugs, [[Git How To#How to submit a pull request|open a pull request.]] This sends your changes to us for us to look over in [[Code Review|review]].
-=== Get and Run Bootstrap ===
+* A senior Dreamwidth developer will look over your changes and see if they can be accepted or if there are things you need to change. If there are things that could be improved, the reviewer will comment on the commit and give you notes on what can be improved. Don't worry if your changes don't pass review on the first time -- it's really common for pull requests to go through several versions before they're committed.
-Now, log in as the user account you created in the first step.  We have put together a bootstrap script.  You can obtain this file with your favorite tool (wget, curl, etc):
+* If your pull request needs more changes, you can go back to the branch you made for the change and make those changes. Once you commit them to your branch and [[Git How To#The_Basics|push your changes]] to make them public, the pull request will update automatically. However, this doesn't fire off a notification to the folk reviewing the code, so please leave a comment on your pull request to let people know that the requested changes have been made!
-    wget http://hg.dwscoalition.org/dw-free/raw-file/e4be2e864b95/bin/bootstrap.pl
+* When your pull request is accepted, the person who accepts it will close the pull request and the issue that the pull request fixed.
-The next step is fairly automatic.  Assuming that your environment variable is good to go, you should be able to just do this:
+== The next fix! ==
-    perl bootstrap.pl
+* Once your pull request is accepted and merged into the main branch, you can delete the branch for that issue on your GitHub account.
-The script will start doing some work.  It should start out by saying 'seems we need to start at the beginning', which is what you expect.  Then you will sit around for half an hour while it checks out various packages and repositories from various locations.
+* Switch back to the 'master' branch on your Dreamhack and update to the most current code.
-When it's done, it will say so.  It will also say to delete the bootstrap script.  Do a quick ls in $LJHOME to see if you have a bunch of new directories like cvs, bml, cgi-bin, htdocs, etc.  If you do, huzzah!  You now have the code checked out.
+* Pick the next issue to work on!
-If you don't, figure out what step didn't work, and try again.  All else fails, hit the mailing list or find someone on IRC.
+== Further reading ==
-=== dw-nonfree ===
+This is a brief overview of the process. There are plenty of places to go for more information. Useful wiki pages include:
-If you are doing development on Dreamwidth custom files (what few there are), then you will need to do another step.  This is fairly manual, the bootstrap script does not handle this process for you.  But assuming that you did the above steps, you can get dw-nonfree code like this:
+* [[Dev Finding Things]] - getting around the code base and figuring out where to start when looking for a specific feature
+* [[Dev Programming Guidelines]] - programming style
+* [[Design]] - things to keep in mind when designing features. Developers, please take note of this page!
+* [[Dev Maintenance]]  - how to keep your code updated
+* [[Git How To]] - basic commands for keeping track of your changes using version control. Also includes instructions for submitting those changes via a pull request
+* [[Git Getting Started]] - a collection of resources for using Git
+* [[Github Issues]] - instructions for filing and claiming bugs to work on
+* [[Development]] - another list of stuff that might be useful for you at some point, in particular [[Development#Customising_Your_Dev_Environment | Customising Your Dev Environment]] (for instructions on setting up the Support Board, enabling Beta Features, and so on)
+* [[:Category:Development]] - every page tagged with the Development category on the wiki
-    cd $LJHOME/cvs
+==What'd we miss?==
-    hg clone http://hg.dwscoalition.org/dw-nonfree dw-nonfree
-    cp dw-nonfree/cvs/multicvs-local.conf .
-    cd $LJHOME
-    bin/cvsreport.pl -s
-You should see a number of files get copied around.
+If you run into anything in the process that isn't well-documented, or you have questions, you can ask in the <dwcomm>dw_dev_training</dwcomm> community or in the #dreamwidth-dev irc channel. You should also make a note on the [[Dev Wanted How-To]] or [[Installation Wanted How-To]] wiki pages so we can document it better for the next person to come along!
-=== Database Setup ===
-I'm assuming that you have a local MySQL installation.  I'm assuming that you have the root login on this database, or know how to get a root shell to the MySQL console.  If you don't, you'll have to figure out how to create a database in your setup.
-But assuming you do have root, this is how you can easily create your development database:
-    create database dw_dev_1;
-    grant all on dw_dev_1.* to 'dw_dev_user'@'localhost' identified by 'somePassword';
-Of course, you might want to pick an appropriate username/database/password instead of using the crap I made up here.
-=== ljconfig.pl Setup ===
-Next you need to configure the site configuration script.  This is probably the most tricky part of the whole process, since there are so many things you can tweak.  (Note to self: get the DW ljconfig cleaned up and posted...)
-    cd $LJHOME
-    cp doc/ljconfig.pl.txt etc/ljconfig.pl
-    vim etc/ljconfig.pl
-Or use your editor of choice, of course.  Do a search for the phrase 'CHANGE THIS' which is in three places.  You will want to change at least your $DOMAIN and the %DBINFO structure.  It should be fairly self-explanatory.
-Note that %DBINFO contains the authentication credentials you chose in the database setup step above!
-=== checkconfig.pl ===
-Now, you need to see if everything is working.  If you've setup the files as indicated above, run this command:
-    cd $LJHOME
-    bin/checkconfig.pl --no=ljconfig
-It should take a few minutes to run and then tell you that you are missing modules.  This is where you need some systems specific knowledge for your system.  You will need to install whatever modules it's missing - I can't really help you there at this time.  (But if you get well and truly stuck, find someone on IRC or the mailing list!)
-If all goes well, then take heart, you're almost done!
-=== Install Data ===
-There are a few commands you can now run to install the database.  Just run these and watch for errors.
-    cd $LJHOME
-    bin/upgrading/update-db.pl -r -p --innodb
-    bin/upgrading/update-db.pl -r --cluster=all --innodb
-    bin/upgrading/texttool.pl load
-    bin/upgrading/make_system.pl
-(That step will ask you for a password for the system account.  You can change it later by logging in as system, so just give it something for now.)
-    bin/upgrading/copy-emailpass-out-of-user
-And that's it for the code!
-=== Configure Apache ===
-apache
-...
+[[Category: Getting Started]]
 [[Category: Development]]

Difference between revisions of "Dev Getting Started"

Latest revision as of 22:09, 17 December 2022

Contents

One time only: Initial setup

A working Dreamwidth installation

A GitHub account

A basic sense of the code structure and how to find things

Finding something to work on

Making your changes

Submitting your changes

The next fix!

Further reading

What'd we miss?

Navigation menu

Personal tools

Namespaces

Variants

Views

Actions

Search

Important Info

Wiki Navigation

Main Categories

Tools

Other Places