Understanding how data is prioritized and managed is crucial in network and server administration. One such concept that plays a significant role in this process is the Differentiated Services Code…
Web Hosting Geeks’ Blog
Code
A Web based system to push your SVN code through development, staging and production environments
Note the files in this post are now on GitHub
Hello there!
In development, having a seamlessly integrated process where you can propagate your code through whatever QA, testing and development policy you have is invaluable and a definite time saver.
We work with SVN as well as GIT code repository systems and have developed a web based system to “Export” or “Push” the code through development, staging and production environments as such.
I have already talked about sanitizing your code during the commit process, to ensure commit messages are standard and there are no PHP fatal errors, so now I will be showcasing you a simple web based system for propagating your code through development, staging and production servers.
This system should be on a secure web accessible page on each server. For the sake of argument , I’ll call each server the following :
dev.server.com — development server
staging.server.com — staging server
www.server.com — production server
We will be using PHP for the web based interface, and we will assume that you will be password protecting access to this page via htpasswd, as well as forcing SSL. I am also assuming that within your SVN repository, you have multiple “sites” that you will be individually pushing or exporting (svn export). Once you have the secure, password protected page (lets call it https://dev.server.com/svn) , the following PHP page will be the main index :
svnupdate.php
$ sites[] = array( "name" => "Site A", "url" => "http://site-a.server.com", "path" => "/usr/local/www/site-a.server.com", "source" => "svn://svn.server.com/repository/branches/site-a", "login" => "svnlogin", "base" => "1.00", "notes" => "Standard build for Site A" ); "name" => "Site B", "url" => "http://site-b.server.com", "path" => "/usr/local/www/site-b.server.com", "source" => "svn://svn.server.com/repository/branches/site-b", "login" => "svnlogin", "base" => "1.00", "notes" => "Standard build for Site B" ); ?>SVN Update Page
Server: | Development Server | Staging Server | Production Server |
View Development Export Log
Site | Source | UN/PW | Base | Revision | Export | Pending Updates | Notes |
If you carefully look at the above code, you will see that this page will be dependent on 3 external scripts, which I will describe below. The page itself generates a list of whatever sites you want to include in the push process, within a PHP based Array. The array details important info per site such as the name, svn location, location of the files on the server as well as whatever other notes and additional info you want to provide.
Each time a site is “exported” by clicking the export button, it calls an external script called svnupdate_process.php. This executes the SVN EXPORT command, as well as logging which user requested the action into a simple text based log file. The user is determined by the authentication user that is accessing the page. The htpassword credentials you will be providing to your users should be set per-user so that it can be easier to determine who pushed the code and whatnot.
The other two external scripts are one that will view the log file in an iframe on the same page, as well as a script to extrapolate the pending commits that are in the queue since the LAST code push / svn export. That is really useful, as you can imagine.
Script to view the export log
This script, log.php is used to dump the contents of the log.txt export log file. Very simple
log.php
Untitled Development Export Log:
include("./functions.inc.php"); $ logfile = new logfile(); $ logfile->display(); ?>
Simple, right? The log.php code includes a functions.inc.php file, used for writing and reading the log.txt file. The above code depends on it, as well as the svnupdate_process.php code (described below), in order to log each time someone hits the export code button
functions.inc.php
class logfile { function logfile() { $ this->filename = "log.txt"; $ this->Username = $ _SERVER['PHP_AUTH_USER']; $ this->logfile = $ this->filename; } function write($ data) { // write to logfile $ handle = fopen($ this->logfile, "a+"); $ date = date("Y-m-d H:i:s"); $ IP = getenv('REMOTE_ADDR'); $ data = "[$ date] {$ this->Username}:{$ IP} - " . $ data . "n"; $ return = fwrite($ handle, $ data); fclose($ handle); } function display() { // display logfile $ handle = fopen($ this->logfile, "a+"); while(!feof($ handle)) { // Pull lines into array $ lines[] = fgets($ handle, 1024); } $ count = count($ lines); $ count = $ count - 2; for($ i=$ count;$ i>=0;$ i--) { echo $ lines[$ i] . "
"; } fclose($ handle); } } ?>
The code of the svn export process is handled by the following script below. Again its self explanatory. PHP executes a shell command to export the svn code based on the array variables defined in the very first script. Make sure all the variables match up to whats in svn and the location of the files, and even execute a test run of the command manually with the variables. If there are problems, you can modify the command to pipe the output to a log file for further analysis. Additionally you may need to alter the permissions of the apache user so that the command can be properly run. Avoid setting the apache user to have a shell (big no-no) ,but maybe a nologin shell or something along those lines. Its completely up to you , but be very careful about the choices you make to get it to run properly.
svnupdate_process.php
Update/Status Window
include("./functions.inc.php"); $ logfile = new logfile(); if($ _POST['submitbutton']) { if($ _POST['revision'] != "") { $ revision = "-r ".$ _POST['revision']; } $ command = "/usr/bin/svn export --force --username svnuser --password 'svnpassword' $ revision --config-dir /tmp ".$ _POST['source']. " " . $ _POST['site']." 2>&1"; if($ _POST['submitbutton'] == "Export") { $ output = shell_exec("umask 022;".$ command); } echo "$ output";
$ logtext = "Exported to {$ _POST['site']}";
$ logfile->write($ logtext);
eaccelerator_clear();
}?>
Finally the last script will be the script that parses the SVN log output with a date/time range from the last time the export button was pushed, until the current date and time. This will load the output in the same iframe log window on the svn page so the user can see what pending commits are in the code since the last time it was exported. Invaluable information, right?
Note that this has a function to filter out additional illegal characters to avoid cross site scripting injections. This code should be completely 100% restricted from outside public use, however it might be worth it to put this function in the svnupdate_process.php script as well. Can’t be too careful. I thought I’d include it here for you to use.
viewcommit.php
if(($ _GET['svn'] != "") && ($ _GET['path'] != "") && ($ _GET['name'] != "")) { // Cross Site Script & Code Injection Sanitization function xss_cleaner($ input_str) { $ return_str = str_replace( array('<',';','|','&','>',"'",'"',')','('), array('<',':','|','&','>',''','"',')','('), $ input_str ); $ return_str = str_ireplace( '%3Cscript', '', $ return_str ); return $ return_str; } $ xss_path=xss_cleaner($ _GET['path']); $ xss_svn=xss_cleaner($ _GET['svn']); $ xss_name=xss_cleaner($ _GET['name']); echo "Viewing Pending Updates For : ". $ xss_name . ""; echo "
"; $ command = "/usr/bin/svn --username svnuser --password 'svnpassword' --config-dir /tmp log " . $ xss_svn . " -r {"`grep "" . $ xss_path . "" log.txt | tail -n 1 | awk -F " " '{printf "%s %s", $ 1,$ 2}' | sed -e 's/[//g' -e 's/]//g'`"}:{"`date "+%Y-%m-%d %H:%M:%S"`"}"; $ output = shell_exec("umask 022;".$ command); echo "$ output";
}
else {
echo "No queries passed!";
}?>
Lets break down the SVN log command, so you know whats going on. I’m grabbing the SVN site array variables when the “view log” link is clicked on the svn page. I am also parsing the export log text file to get the last entry for the particular site in question, grabbing the date and time.
I am then getting the current date and time to complete the date/time range in the svn log query. The finished query should look something like this :
svn --username svnuser --password 'svnpassword' --config-dir /tmp log svn://svn.server.com -r {"2013-01-01 12:01:00"}:{"2013-02-01 12:01:00"}Note the files in this post are now on GitHub
The post A Web based system to push your SVN code through development, staging and production environments appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.
Managed WordPress Hosting | Managed VPS Hosting | Stack Star
Web based system to push your GIT code
Hello!
Since posting recently about our Web based SVN push system , we have decided to take what we did there one step further and implement a very similar system for GIT, but with more options!
The web based GIT push system is, as mentioned, very similar to the web based SVN push system, with the exception that you can select branches before exporting the code.
I should stress before continuing that this system is not intended to be publicly visible on a website. Strict access controls need to be implemented in front of this implementation to protect the integrity and protect from malicious users. For example, only making this system available on a Development LAN, or putting it behind an IP restricted firewall, with IP restricted apache/nginx rules, web authentication and SSL will allow for a much more secure implementation of this system. My advice is to always assume everything is vulnerable at any time. Working backwards with that assumption has always been a good policy for me.
First of all the entire solution is available on GitHub for you to preview.
I’ll go through each file individually, briefly explaining what each file does.
index.php
This is a straightforward file. There is a small amount of php code embedded in this file with HTML to present the push page in a simple HTML table. An array is built for all the sites you want to push (in this example case its a Dev and Prod site). The array makes it very easy to add additional sites. Each array propagates a source, destination, site name and site url within.
The only field that is really used is the “pushname” variable in each site array. That variable gets passed to the shell script that actually takes care of the pushing mechanism.
The remaining php code in this file builds a list of sites based on the array, as well as pulling the current branch by running a function included in functions.inc.php that pulls all the branches associated with a repository and saves it to a text file for easy parsing. The other function pulls the last time the site was pushed or “exported”, giving an easy reference when dealing with multiple developers.
It should be noted that it is best to implement apache/nginx web based access on a per-user basis in order to access this page. This is because the index.php file parses the username of who is accessing the site for logging purposes. So every user that needs to access this needs an htpasswd user/password created for them for security and accountability purposes.
functions.inc.php
This file is where many of the functions lie (obviously). There is a crossite scripting function that is used to filter any submit input. I realize this is not very secure, but with the security considerations I mentioned in the beginning of this post, it should suffice. A good systems administrator would implement many hardware, software and intrusion layers to prevent malicious users from injecting content such as snort and mod_security. Nothing beats the security of a completely offline web accessible page on an internal LAN, obviously.
Next we have some functions that grab the branches, get the current branch that the site has been previously pushed on, some log file functions for storing the log file info and writing the log data and displaying it as well. All of these functions are intended to help keep the development process very organized and easy to maintain.
gitupdate_process.php
This file is where the index.php file POSTS the data of the site you want to push. This file receives the data as a $ _POST (with the XSS cleaner function mentioned earlier sanitizing as best as it can) and then passes that variable to the push bash shell script in order to do the actual file synchronization.
It might be possible to do all the file synchronization in php, but I felt that separating the actual git pulling and rsync process into a separate shell script made the process less obfuscated and confusing. The shell script rarely needs to change unless a new site is added obviously.
log.php
This file is simply loaded as an iframe within index.php when someone clicks to view the export log. It parses the log.txt file and displays it. The export log format can be customized obviously, but usually would contain the site name, username who pushed, date and time as well as the branch pushed.
log.txt
This is self explanatory and contains the log information detailed in log.php
push.sh
This is the push bash shell script that gitupdate_process.php calls. Again this can be consolidated to be 100% PHP but I felt segmenting it was a good idea. You can see that the command line arguments are parsed from a $ _POST in gitupdate_process.php and then passed to the shell script as an argument. This is very simple and shouldn’t be too hard to understand. The arguments would basically be the site name ($ 1) and the git branch name that was selected from the dropdown box before hitting the export button ($ 2).
That’s it! This package for GIT has made many developers’ life easier and caused less headaches when diagnosing problems or even rolling back to a stable branch. Keeping a stable and organized development environment is key here, with the security considerations I mentioned earlier being paramount above everything else.
I hope that this script was helpful and would welcome any suggestions to improve it further 🙂
The post Web based system to push your GIT code appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.
Managed WordPress Hosting | Managed VPS Hosting | Stack Star
A Web based system to push your SVN code through development, staging and production environments
Note the files in this post are now on GitHub
Hello there!
In development, having a seamlessly integrated process where you can propagate your code through whatever QA, testing and development policy you have is invaluable and a definite time saver.
We work with SVN as well as GIT code repository systems and have developed a web based system to “Export” or “Push” the code through development, staging and production environments as such.
I have already talked about sanitizing your code during the commit process, to ensure commit messages are standard and there are no PHP fatal errors, so now I will be showcasing you a simple web based system for propagating your code through development, staging and production servers.
This system should be on a secure web accessible page on each server. For the sake of argument , I’ll call each server the following :
dev.server.com — development server
staging.server.com — staging server
www.server.com — production server
We will be using PHP for the web based interface, and we will assume that you will be password protecting access to this page via htpasswd, as well as forcing SSL. I am also assuming that within your SVN repository, you have multiple “sites” that you will be individually pushing or exporting (svn export). Once you have the secure, password protected page (lets call it https://dev.server.com/svn) , the following PHP page will be the main index :
svnupdate.php
$ sites[] = array( "name" => "Site A", "url" => "http://site-a.server.com", "path" => "/usr/local/www/site-a.server.com", "source" => "svn://svn.server.com/repository/branches/site-a", "login" => "svnlogin", "base" => "1.00", "notes" => "Standard build for Site A" ); "name" => "Site B", "url" => "http://site-b.server.com", "path" => "/usr/local/www/site-b.server.com", "source" => "svn://svn.server.com/repository/branches/site-b", "login" => "svnlogin", "base" => "1.00", "notes" => "Standard build for Site B" ); ?>SVN Update Page
Server: | Development Server | Staging Server | Production Server |
View Development Export Log
Site | Source | UN/PW | Base | Revision | Export | Pending Updates | Notes |
If you carefully look at the above code, you will see that this page will be dependent on 3 external scripts, which I will describe below. The page itself generates a list of whatever sites you want to include in the push process, within a PHP based Array. The array details important info per site such as the name, svn location, location of the files on the server as well as whatever other notes and additional info you want to provide.
Each time a site is “exported” by clicking the export button, it calls an external script called svnupdate_process.php. This executes the SVN EXPORT command, as well as logging which user requested the action into a simple text based log file. The user is determined by the authentication user that is accessing the page. The htpassword credentials you will be providing to your users should be set per-user so that it can be easier to determine who pushed the code and whatnot.
The other two external scripts are one that will view the log file in an iframe on the same page, as well as a script to extrapolate the pending commits that are in the queue since the LAST code push / svn export. That is really useful, as you can imagine.
Script to view the export log
This script, log.php is used to dump the contents of the log.txt export log file. Very simple
log.php
Untitled Development Export Log:
include("./functions.inc.php"); $ logfile = new logfile(); $ logfile->display(); ?>
Simple, right? The log.php code includes a functions.inc.php file, used for writing and reading the log.txt file. The above code depends on it, as well as the svnupdate_process.php code (described below), in order to log each time someone hits the export code button
functions.inc.php
class logfile { function logfile() { $ this->filename = "log.txt"; $ this->Username = $ _SERVER['PHP_AUTH_USER']; $ this->logfile = $ this->filename; } function write($ data) { // write to logfile $ handle = fopen($ this->logfile, "a+"); $ date = date("Y-m-d H:i:s"); $ IP = getenv('REMOTE_ADDR'); $ data = "[$ date] {$ this->Username}:{$ IP} - " . $ data . "n"; $ return = fwrite($ handle, $ data); fclose($ handle); } function display() { // display logfile $ handle = fopen($ this->logfile, "a+"); while(!feof($ handle)) { // Pull lines into array $ lines[] = fgets($ handle, 1024); } $ count = count($ lines); $ count = $ count - 2; for($ i=$ count;$ i>=0;$ i--) { echo $ lines[$ i] . "
"; } fclose($ handle); } } ?>
The code of the svn export process is handled by the following script below. Again its self explanatory. PHP executes a shell command to export the svn code based on the array variables defined in the very first script. Make sure all the variables match up to whats in svn and the location of the files, and even execute a test run of the command manually with the variables. If there are problems, you can modify the command to pipe the output to a log file for further analysis. Additionally you may need to alter the permissions of the apache user so that the command can be properly run. Avoid setting the apache user to have a shell (big no-no) ,but maybe a nologin shell or something along those lines. Its completely up to you , but be very careful about the choices you make to get it to run properly.
svnupdate_process.php
Update/Status Window
include("./functions.inc.php"); $ logfile = new logfile(); if($ _POST['submitbutton']) { if($ _POST['revision'] != "") { $ revision = "-r ".$ _POST['revision']; } $ command = "/usr/bin/svn export --force --username svnuser --password 'svnpassword' $ revision --config-dir /tmp ".$ _POST['source']. " " . $ _POST['site']." 2>&1"; if($ _POST['submitbutton'] == "Export") { $ output = shell_exec("umask 022;".$ command); } echo "$ output";
$ logtext = "Exported to {$ _POST['site']}";
$ logfile->write($ logtext);
eaccelerator_clear();
}?>
Finally the last script will be the script that parses the SVN log output with a date/time range from the last time the export button was pushed, until the current date and time. This will load the output in the same iframe log window on the svn page so the user can see what pending commits are in the code since the last time it was exported. Invaluable information, right?
Note that this has a function to filter out additional illegal characters to avoid cross site scripting injections. This code should be completely 100% restricted from outside public use, however it might be worth it to put this function in the svnupdate_process.php script as well. Can’t be too careful. I thought I’d include it here for you to use.
viewcommit.php
if(($ _GET['svn'] != "") && ($ _GET['path'] != "") && ($ _GET['name'] != "")) { // Cross Site Script & Code Injection Sanitization function xss_cleaner($ input_str) { $ return_str = str_replace( array('<',';','|','&','>',"'",'"',')','('), array('<',':','|','&','>',''','"',')','('), $ input_str ); $ return_str = str_ireplace( '%3Cscript', '', $ return_str ); return $ return_str; } $ xss_path=xss_cleaner($ _GET['path']); $ xss_svn=xss_cleaner($ _GET['svn']); $ xss_name=xss_cleaner($ _GET['name']); echo "Viewing Pending Updates For : ". $ xss_name . ""; echo "
"; $ command = "/usr/bin/svn --username svnuser --password 'svnpassword' --config-dir /tmp log " . $ xss_svn . " -r {"`grep "" . $ xss_path . "" log.txt | tail -n 1 | awk -F " " '{printf "%s %s", $ 1,$ 2}' | sed -e 's/[//g' -e 's/]//g'`"}:{"`date "+%Y-%m-%d %H:%M:%S"`"}"; $ output = shell_exec("umask 022;".$ command); echo "$ output";
}
else {
echo "No queries passed!";
}?>
Lets break down the SVN log command, so you know whats going on. I’m grabbing the SVN site array variables when the “view log” link is clicked on the svn page. I am also parsing the export log text file to get the last entry for the particular site in question, grabbing the date and time.
I am then getting the current date and time to complete the date/time range in the svn log query. The finished query should look something like this :
svn --username svnuser --password 'svnpassword' --config-dir /tmp log svn://svn.server.com -r {"2013-01-01 12:01:00"}:{"2013-02-01 12:01:00"}Note the files in this post are now on GitHub
The post A Web based system to push your SVN code through development, staging and production environments appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.
Managed WordPress Hosting | Managed VPS Hosting | Stack Star
Web based system to push your GIT code
Hello!
Since posting recently about our Web based SVN push system , we have decided to take what we did there one step further and implement a very similar system for GIT, but with more options!
The web based GIT push system is, as mentioned, very similar to the web based SVN push system, with the exception that you can select branches before exporting the code.
I should stress before continuing that this system is not intended to be publicly visible on a website. Strict access controls need to be implemented in front of this implementation to protect the integrity and protect from malicious users. For example, only making this system available on a Development LAN, or putting it behind an IP restricted firewall, with IP restricted apache/nginx rules, web authentication and SSL will allow for a much more secure implementation of this system. My advice is to always assume everything is vulnerable at any time. Working backwards with that assumption has always been a good policy for me.
First of all the entire solution is available on GitHub for you to preview.
I’ll go through each file individually, briefly explaining what each file does.
index.php
This is a straightforward file. There is a small amount of php code embedded in this file with HTML to present the push page in a simple HTML table. An array is built for all the sites you want to push (in this example case its a Dev and Prod site). The array makes it very easy to add additional sites. Each array propagates a source, destination, site name and site url within.
The only field that is really used is the “pushname” variable in each site array. That variable gets passed to the shell script that actually takes care of the pushing mechanism.
The remaining php code in this file builds a list of sites based on the array, as well as pulling the current branch by running a function included in functions.inc.php that pulls all the branches associated with a repository and saves it to a text file for easy parsing. The other function pulls the last time the site was pushed or “exported”, giving an easy reference when dealing with multiple developers.
It should be noted that it is best to implement apache/nginx web based access on a per-user basis in order to access this page. This is because the index.php file parses the username of who is accessing the site for logging purposes. So every user that needs to access this needs an htpasswd user/password created for them for security and accountability purposes.
functions.inc.php
This file is where many of the functions lie (obviously). There is a crossite scripting function that is used to filter any submit input. I realize this is not very secure, but with the security considerations I mentioned in the beginning of this post, it should suffice. A good systems administrator would implement many hardware, software and intrusion layers to prevent malicious users from injecting content such as snort and mod_security. Nothing beats the security of a completely offline web accessible page on an internal LAN, obviously.
Next we have some functions that grab the branches, get the current branch that the site has been previously pushed on, some log file functions for storing the log file info and writing the log data and displaying it as well. All of these functions are intended to help keep the development process very organized and easy to maintain.
gitupdate_process.php
This file is where the index.php file POSTS the data of the site you want to push. This file receives the data as a $ _POST (with the XSS cleaner function mentioned earlier sanitizing as best as it can) and then passes that variable to the push bash shell script in order to do the actual file synchronization.
It might be possible to do all the file synchronization in php, but I felt that separating the actual git pulling and rsync process into a separate shell script made the process less obfuscated and confusing. The shell script rarely needs to change unless a new site is added obviously.
log.php
This file is simply loaded as an iframe within index.php when someone clicks to view the export log. It parses the log.txt file and displays it. The export log format can be customized obviously, but usually would contain the site name, username who pushed, date and time as well as the branch pushed.
log.txt
This is self explanatory and contains the log information detailed in log.php
push.sh
This is the push bash shell script that gitupdate_process.php calls. Again this can be consolidated to be 100% PHP but I felt segmenting it was a good idea. You can see that the command line arguments are parsed from a $ _POST in gitupdate_process.php and then passed to the shell script as an argument. This is very simple and shouldn’t be too hard to understand. The arguments would basically be the site name ($ 1) and the git branch name that was selected from the dropdown box before hitting the export button ($ 2).
That’s it! This package for GIT has made many developers’ life easier and caused less headaches when diagnosing problems or even rolling back to a stable branch. Keeping a stable and organized development environment is key here, with the security considerations I mentioned earlier being paramount above everything else.
I hope that this script was helpful and would welcome any suggestions to improve it further 🙂
The post Web based system to push your GIT code appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.
Managed WordPress Hosting | Managed VPS Hosting | Stack Star
Google Summer of Code 2019
The phpMyAdmin project is excited to announce our student projects for the 2019 Google Summer of Code. We’ve had many great applicants and it is unfortunate that we were only able to accept these three.
- Apoorv Khare will be working on a series of general improvements, fixing bugs and adding improvements to the interface,
- Mohit Kuri’s projoct is to to refactor and improve the Designer interface, and
- Nupur Agrawal has been selected to convert the user interface to Bootstrap 4. These changes will make it easier for developers to maintain a consistent style and appearance throughout the application.
This year, the mentors from phpMyAdmin were pleased with the amount of student support; many students joined the conversation on Gitter and other forums to help other students with their applications and offer their support of each other. This response was overwhelming and reflects wonderfully on the student applicants.
Students who were not selected are welcome to continue to contribute to the phpMyAdmin community and, for those who will be eligible next year, we hope you’ll apply again.
Summer of Code is a Google initiative where Google funds college students getting paid for real-world experience and mentorship through open source projects. For many students, this is the first exposure to an open source project. Several phpMyAdmin team members have started as GSoC students. This marks phpMyAdmin’s tenth year of participation in GSoC.