Project: “짤방”( Memes )

Github: https://github.com/kimsup10/memes

1. Motivation

Images called “meme” have become major measurements for communication among people. But More images we have in our smart phone or desktop, Harder to find and use proper image at funny timing. We designed web application to help store, share and quickly use the meme.

2. Features

  • Session
    • Sign-up
    • Sign-in
    • Friend request / Retrieve the request
    • Accept the request / Deny the request
  • Meme management
    • Store ( Create ) memes with description
    • Read it
    • Update it
    • Delete it
  • Meme Exploration
    • Hot trending memes
    • Keyword search

3. Technical Issues

  • Simple UX which is user-friendly and focus on service motivation(Simple and fast search and use of memes!) ( We have no designer …)
  • Database management for many feature.
    • Have to design major database for CRUD of memes and user data.
    • Have to design other databases such as ElasticSearch and Redis for the similar data to implement trending and search.
  • Disk storage store for uploaded images.
  • Session implementation
  • Quickly serving many images
  • Quickly calculating and uploading hot trending
  • Fast and accurate search
  • Designing Restful API
  • Reducing duplicate images to efficiently manage disk storage

4. Technical stack for the solution스크린샷 2016-12-26 오후 5.09.01.png

  • HTML and Bootstrap
  • jQuery and Ajax
    • Simple meme copy
    • Counting the number of copy
  • Node.js & Express frame work
  • Nginx
    • Reverse proxy
    • Fast static file serving
    • Load balancing
  • Postgresql and Sequelize ORM
    • Easy connection with postgresql
    • Easy database synchronization between Redis, Elastic search and Postgresql using addHook function.
  • Multer.js
    • Easy disk storage store.
  • Redis
    • Session store
    • Hot trending
  • Elastic search
    • Fast and powerful search
  • Docker
    • Containerization
    • Scale

5. My role in the team and Code.

  • Meme CRUD
  • Hot trending
  • Meme copy
  • User page
  • Performance tuning with Nginx, Docker

1) Meme CRUD

%e1%84%89%e1%85%b3%e1%84%8f%e1%85%b3%e1%84%85%e1%85%b5%e1%86%ab%e1%84%89%e1%85%a3%e1%86%ba-2016-12-26-%e1%84%8b%e1%85%a9%e1%84%92%e1%85%ae-5-13-34
Screenshot before choosing image
%e1%84%89%e1%85%b3%e1%84%8f%e1%85%b3%e1%84%85%e1%85%b5%e1%86%ab%e1%84%89%e1%85%a3%e1%86%ba-2016-12-26-%e1%84%8b%e1%85%a9%e1%84%92%e1%85%ae-5-15-16
Screenshot after choosing image
// public/javascripts/uploadingFile.js
$(document).ready(readyListener);

// Show a preview and relative information input
function readyListener(){
    var imgTarget = $('.input-group .uploadImg');

    imgTarget.on('change', function() {

        var files = $(this)[0].files;
        if (!$(this)[0].files[0].type.match(/image\//))
            return;

        for (var i in files ){
            if(files.hasOwnProperty(i)) {
                showPreview(files[i]);
            }
        }
    });

    function showPreview(file){
        var reader = new FileReader();
        var thumbnailId = file.name.replace(/(\.)|(\s)/g, '-');

        reader.onload = function myOnload(e){
            var src = e.target.result;
            $('.uploadImgList').append('

‘); }; reader.addEventListener(‘load’, function () { $(‘.uploads-display#’+thumbnailId).append(” + ‘
전체공개
친구공개
비공개
‘ +’취소하기
‘ ); }); reader.readAsDataURL(file); } } function removeAttachment(id) { var thumbnail = document.getElementById(id); var oldInput = document.getElementById(‘attachFile’); var newInput = document.createElement(“input”); // Replace old input=”file” to new one to clear filelist. newInput.type = “file”; newInput.id = oldInput.id; newInput.className = oldInput.className; newInput.name = oldInput.name; newInput.style = “display: none;”; newInput.multiple = oldInput.multiple; oldInput.parentNode.replaceChild(newInput, oldInput); $(document).ready(readyListener); // Remove the thumbnail of the canceled update. document.getElementsByClassName(‘uploadImgList’)[0].removeChild(thumbnail); if ($(‘.uploads-display’).length==0) document.getElementsByTagName(‘button’)[0].disabled=true; } function enableApply(){ var apply = $(‘button[name=apply]’)[0]; if($(‘input[name=description]’)[0].value) apply.disabled=false; else apply.disabled=true; }

// Code for RUD
// routes/memes.js
router.get('/new', function (req, res) {
    if (req.session.user_id){
        res.render('newMeme');
    } else {
        // No authorization
        req.flash('error', '로그인 하세요.');
        res.redirect('/')
    }
});
(...)
router.post('/', upload.array('image'), function (req, res) {
    var attachments = req.files.map(function (f) {
        return {
            filesize: f.size,
            filepath: f.filename
        };
    });
    if (attachments.length==1){
        m.Attachment.create(attachments[0]).then(function (a) {
            meme = {
                user_id: req.session.user_id,
                attachment_id: a.id,
                description: req.body.description,
                privacy_level:req.body.privacy_level
            };
            m.Meme.create(meme).then(function () {
                res.redirect('/');
            })
        })
    } else {
        m.Attachment.bulkCreate(attachments, {returning: true}).then(function (a) {
                var memes = [];
                for (var i in a) {
                    memes.push(
                        {
                            user_id: req.session.user_id,
                            attachment_id: a[i].id,
                            description: req.body.description[i],
                            privacy_level: req.body.privacy_level[i]
                        }
                    );
                }
                m.Meme.bulkCreate(memes, {returning: true}).then(function () {
                    res.redirect('/');
                });
            }
        )}
});
// in utils/multer.js
var multer = require('multer');
var uuid = require('uuid');

var storage = multer.diskStorage({
    destination: function (req, file, cb) {
        cb(null, __dirname+'/../public/uploads');
    },
    filename: function (req, file, cb) {
        var extension = file.mimetype.split('/')[1];
        cb(null, uuid()+'.'+extension)
    }
});
// routes/memes.js
router.get("/:id",function(req,res,next) {
    m.Meme.find({
        include: [m.Meme.associations.attachment, m.Meme.associations.user],
        where: {id: req.params.id}
    }).then(function (meme) {
        res.render('meme', {meme: meme});
    });
});

router.post('/:id/copy', function (req, res) {
    var key = 'meme:'+req.params.id;
    m.Meme.findOne({
        where: {
            id: parseInt(req.params.id),
            privacy_level:'public'
        }
    }).then(function (meme) {
        if (!meme) {
            res.send(200);
        } else {
            redis.hincrby(key, 'copy_count', 1, function (err, reply) {
                if (err) {
                    res.send(500);
                } else {
                    redis.hgetall(key, function (err, hash) {
                        if (err) {
                            res.send(500);
                        } else {
                            var score = Math.log10(Number(hash.copy_count)) + Math.round(meme.created_at/(1000*45000));
                            redis.zadd('trending', score, meme.id);
                        }
                    });
                    res.send(200);
                }
            });
        }
    });
});

router.get('/:id/edit', function (req, res) {
    m.Meme.findOne({
        where:{id:req.params.id},
        include:[m.Meme.associations.user, m.Meme.associations.attachment]
    }).then(function (m) {
        if (m.user_id == req.session.user_id){
            res.render('editMeme', {meme:m});
        } else {
            req.flash('error', '권한이 없습니다.');
            res.redirect('/');
        }
    });
});

router.post('/:id/edit', function (req, res) {
    m.Meme.update({
        description: req.body.description,
        privacy_level:req.body.privacy_level
    }, {where:{id:req.params.id}, returning:true}).then(function (m) {
        res.redirect('/memes/'+req.params.id);
    });
});

router.get('/:id/delete', function (req, res) {
    m.Meme.findOne({where:{id:req.params.id}}).then(function (m) {
        if (req.session.user_id == m.user_id){
            redis.del('meme:'+req.params.id, function (err) {
                redis.zrem('trending', req.params.id, function (err) {
                    m.destroy({where:{id:req.params.id}}).then(function (r) {
                        res.redirect('back');
                    });
                })
            });
        }else {
            req.flash('error', '권한이 없습니다.');
            res.redirecte('back');
        }
    });
});
// models/memes.js
var Meme = db.define('meme', {
    id: {
        type: Sequelize.INTEGER,
        primaryKey: true,
        autoIncrement: true
    },
    user_id: Sequelize.INTEGER,
    description: Sequelize.STRING,
    attachment_id: Sequelize.INTEGER,
    privacy_level: {
        type: Sequelize.ENUM('public', 'friends', 'private'),
        defaultValue: 'public'
    }
}, {
    underscored: true
});

Meme.addHook('afterCreate', 'saveES', function(meme, options) {
  es.create({
    index: 'meme',
    type: 'meme',
    id: meme.id,
    body: {
      user_id: meme.user_id,
      privacy_level: meme.privacy_level,
      description: meme.description
    }
  }, function(error, response) {
  });
});

Meme.addHook('afterBulkCreate', 'saveES', function(memes, options) {
  memes.forEach(function(meme){
    es.create({
      index: 'meme',
      type: 'meme',
      id: meme.id,
      body: {
        user_id: meme.user_id,
        privacy_level: meme.privacy_level,
        description: meme.description
      }
    }, function(error, response) {
    });
  });
});

Meme.addHook('afterDestroy', 'saveES', function(meme, options) {
  es.delete({
    index: 'meme',
    type: 'meme',
    id: meme.id
  }, function(error, response) {
  });
});

2) Hot trending

Designed a hot trending algorithm for which I refer to Reddit algorithm( https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9#.ii7q6zoyu ). Used Redis for the needed data structure store.

router.get('/trending', function (req, res, next) {
    var page = parseInt(req.query.page || '1');
    redis.zcard('trending', function (err, count) {
        redis.zrevrange('trending', ((page - 1)*pageLimit), (page*pageLimit)-1, function (err, memeIds) {
            if (err) {
                res.send(500);
            } else {
                m.Meme.findAll({
                    where: { id:{ $in : memeIds } },
                    include: [m.Meme.associations.attachment, m.Meme.associations.user]
                }).then(function (memes) {
                    console.log(memes);
                    var sortedMemes = memeIds.map(function(id) {
                        var i = memes.findIndex(function(meme){
                            return meme.id == parseInt(id);
                        });
                        console.log(i);
                        return memes.splice(i, 1).pop();
                    });
                    res.render('index', { memes: sortedMemes, total: count, page: page, limit: pageLimit });
                });
            }
        });
    });
});

3) Meme copy

// public/javascripts/copy.js
//Cross-browser function to select content
function SelectText(element) {
    var doc = document;
    if (doc.body.createTextRange) {
        var range = document.body.createTextRange();
        range.moveToElementText(element);
        range.select();
    } else if (window.getSelection) {
        var selection = window.getSelection();
        var range = document.createRange();
        range.selectNodeContents(element);
        selection.removeAllRanges();
        selection.addRange(range);
    }
}

function dragged(id){
    window.addEventListener("dragend", function( event ) {
        $.post("/memes/"+id+"/copy");
    }, false);
}

function copy(id) {
    //Make the container Div contenteditable
    var img = $($('#meme-'+id));
    img.attr("contenteditable", true);
    //Select the image
    SelectText(img.get(0));
    //Execute copy Command
    //Note: This will ONLY work directly inside a click listenner
    document.execCommand('copy');
    //Unselect the content
    window.getSelection().removeAllRanges();
    //Make the container Div uneditable again
    $($(img)).removeAttr("contenteditable");
    //Success!!
    $.post("/memes/"+id+"/copy");
}

4) User page

%e1%84%89%e1%85%b3%e1%84%8f%e1%85%b3%e1%84%85%e1%85%b5%e1%86%ab%e1%84%89%e1%85%a3%e1%86%ba-2016-12-26-%e1%84%8b%e1%85%a9%e1%84%92%e1%85%ae-5-34-30

Designed a simple user page which is showing simple user profile and memes list he uploaded. I used a gravatar to show a profile image of the user.

in models/users.js
var User = db.define('user', {
  id: {
    type: Sequelize.INTEGER,
    primaryKey: true,
    autoIncrement: true
  },
  (...)
}, {
  underscored: true,
  instanceMethods: {
    authenticate: function(value) {
      if (bcrypt.compareSync(value, this.password))
        return this;
      else
        return false;
       // Make possible to use gravatar profile image with user model instance
    }, getProfileUrl: function() {
      return gravatar.url(this.email, {s: '100', r: 'x', d: 'retro'}, true);
    }
  }
});
// in routes/memes.js
router.get('/:username', function (req, res, next) {
    m.User.findOne({
        where: {username: req.params.username}
    }).then(function (user) {
        function render(memes) {
            memes = memes.map(function (m) {
                m.user=user;
                return m;
            });
            res.render('index', {title: user.username + '의', user: user, memes: memes});
        }

        // separate listing based on authorization such as friend relation or privacy level of the memes.
        var opts = {include: [m.Meme.associations.attachment]};
        if (user.id == req.session.user_id) {
            user.getMemes(opts).then(render);
        } else {
            opts.where = {$or: [{privacy_level: 'public'}]};
            if (req.session.user_id) {
                m.Friend.findOne({
                    where: {
                        user_id: user.id,
                        friend_id: req.session.user_id,
                        status: "accepted"
                    }
                }).then(function (f) {
                    if (f) {
                        opts.where.$or.push({privacy_level: 'friends'});
                    }
                    user.getMemes(opts).then(render);
                });
            } else {
                user.getMemes(opts).then(render);
            }
        }
    });

5) Performance tuning

Node.js is a event driven, single thread web server framework. So, even if it happens in a same machine, load balancing can improve performance of the service. Based on this idea, I scaled up web application container using docker and set load balancing between web application containers with nginx.

upstream docker-webapp {
    server webapp_1:3000;
    server webapp_2:3000;
    server webapp_3:3000;
}

server {
        listen 80;
        server_name localhost;

        location / {
                client_max_body_size 20M;
                proxy_pass http://docker-webapp;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}
version: '2'
services:
 webapp:
 build:
 context: .
 dockerfile: Dockerfile
 volumes:
 - meme-volume:/webapp/public
 depends_on:
 - postgres
 - redis
 - elasticsearch
 postgres:
 image: postgres
 container_name: meme-db
 redis:
 image: redis
 container_name: meme-redis
 nginx:
 build:
 context: .
 dockerfile: Dockerfile.nginx
 hostname: nginx
 ports:
 - "80:80"
 - "443:443"
 links:
 - webapp

volumes:
 meme-volume:
$ dokcer-compose up --scale webapp=3

 

답글 남기기

아래 항목을 채우거나 오른쪽 아이콘 중 하나를 클릭하여 로그 인 하세요:

WordPress.com 로고

WordPress.com의 계정을 사용하여 댓글을 남깁니다. 로그아웃 /  변경 )

Google photo

Google의 계정을 사용하여 댓글을 남깁니다. 로그아웃 /  변경 )

Twitter 사진

Twitter의 계정을 사용하여 댓글을 남깁니다. 로그아웃 /  변경 )

Facebook 사진

Facebook의 계정을 사용하여 댓글을 남깁니다. 로그아웃 /  변경 )

%s에 연결하는 중

search previous next tag category expand menu location phone mail time cart zoom edit close