Archive for the ‘Reserch And Development’ Category

My Developer tools

Posted: May 8, 2015 by Narendra Shah in Reserch And Development

Basic Tools which any Developer should have, these are my tools, which i always keep on my system.

  1. Editor:
    1. Eclipse
    2. Editplus
    3. Notepad++
  2. Java Software: Any java developer will have following tools
    1. JDK
    2. Tomcat:
  3. Remote connection
    1. Remote connection manager: Remote connection,default program is very poor to connect multiple remote desktops, this tool saves all information in categorized way and can save user/password, so just one click will connect to remote machine.
    2. Putty: Connect over ssh.
    3. WinSCP: For sftp, ssh data transfer
  4. Database
    1. Any one database(Postgres/Mysql/Mongo): Good to have one database setup in maching, i used started with using mysql, then postgresql then back to mysql, but now my love is mongo.
    2. HeidiSql/Squirrel-Database client
  5. Browser: Every one knows about what browser are.
    1. Chrome
      1. Advanced Rest Client
      2. YSlow
      3. Instant Dictionary
    2. Internet Explorer
  6. Compare/Merge
    1. Winmerge: Compare and merge 2 files, must tool for any developer.
  7. Zip/Archiving
    1. WinRAR: Dont like winzip for some reason, this tool is much more faster to open, view, create zips and can open broken zips also sometimes.
  8. System debug
    1. TCP View: View current connection from your machin/system/laptop to outside world, beuty of this tools is it will show your connection, which program made connection and resolving host, you can end established connection etc.
    2. Process Explorer: This tools is really help me to understand, in depth information about running processes like, what is running in your machine, which forked/created this process, what are threads or dlls or files used by the process, many times this is being my life saver during debugging.
    3. Cygwin: Dont use much, but though for linux lover this is the tool to enable linux shell in windows maching. With entry of powershell, this is being less popular
  9. Utility
    1. Adobe Flash: Flash viewing in browser.
    2. Adove Acrobat reader: Reading pdf
    3. VLC Player: Media player to play any video or audio format,
    4. K-Lite Codec: Media player codecs to any video, audio format
    5. Lightshot for taking easy screenshot, print screen with edit on the fly,

I was looking for log analyser for logs from our sytem, i found this was interesting.

1. Scribe – Real time log aggregation used in Facebook
Scribe is a server for aggregating log data that’s streamed in real time from clients. It is designed to be scalable and reliable. It is developed and maintained by Facebook. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups.

2. Logstash – Centralized log storage, indexing, and searching

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use. Logstash comes with a web interface for searching and drilling into all of your logs.

3. Octopussy – Perl/XML Logs Analyzer, Alerter & Reporter
Octopussy is a Log analyzer tool. It analyzes the log, generates reports and alerts the admin. It has LDAP support to maintain users list. It exports report by Email, FTP & SCP. Scheduled reports could be generated. RRD tool to generate graphs.

4. Awstats – Advanced web, streaming, ftp and mail server statistics
AWStats is a powerful tool that generates advanced web, streaming, ftp or mail server statistics graphically. It can analyze log files from all major server tools like Apache log files, ebStar, IIS and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages.

5. nxlog – Multi platform Log management
nxlog is a modular, multi-threaded, high-performance log management solution with multi-platform support. In concept it is similar to syslog-ng or rsyslog but is not limited to unix/syslog only. It can collect logs from files in various formats, receive logs from the network remotely over UDP, TCP or TLS/SSL . It supports platform specific sources such as the Windows Eventlog, Linux kernel logs, Android logs, local syslog etc.

6. Graylog2 – Open Source Log Management
Graylog2 is an open source log management solution that stores your logs in ElasticSearch. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser. Take a look at the screenshots or the latest release info page to get a feeling of what you can do with Graylog2.

7. Fluentd – Data collector, Log Everything in JSON
Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

8. Meniscus – The Python Event Logging Service

Meniscus is a Python based system for event collection, transit and processing in the large. It’s primary use case is for large-scale Cloud logging, but can be used in many other scenarios including usage reporting and API tracing. Its components include Collection, Transport, Storage, Event Processing & Enhancement, Complex Event Processing, Analytics.

9. lucene-log4j – Log4j file rolling appender which indexes log with Lucene
lucene-log4j solves a recurrent problem that production support team face whenever a live incident happens: filtering production log statements to match a session/transaction/user ID. It works by extending Log4j’s RollingFileAppender with Lucene indexing routines. Then with a LuceneLogSearchServlet, you get access to your log using web front end.

10. Chainsaw – log viewer and analysis tool
Chainsaw is a companion application to Log4j written by members of the Log4j development community. Chainsaw can read log files formatted in Log4j’s XMLLayout, receive events from remote locations, read events from a DB, it can even work with the JDK 1.4 logging events.

11. Logsandra – log management using Cassandra
Logsandra is a log management application written in Python and using Cassandra as back-end. It is written as demo for cassandra but it is worth to take a look. It provides support to create your own parser.

12. Clarity – Web interface for the grep
Clarity is a Splunk like web interface for your server log files. It supports searching (using grep) as well as trailing log files in realtime. It has been written using the event based architecture based on EventMachine and so allows real-time search of very large log files.

13. Webalizer – fast web server log file analysis
The Webalizer is a fast web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser. It andles standard Common logfile format (CLF) server logs, several variations of the NCSA Combined logfile format, wu-ftpd/proftpd xferlog (FTP) format logs, Squid proxy server native format, and W3C Extended log formats.

14. Zenoss – Open Source IT Management
Zenoss Core is an open source IT monitoring product that delivers the functionality to effectively manage the configuration, health, performance of networks, servers and applications through a single, integrated software package.

15. OtrosLogViewer – Log parser and Viewer
OtrosLogViewer can read log files formatted in Log4j (pattern and XMLL yout), java.util.logging. Source of events can be local or remote file (ftp, sftp, sa ba, http) or sockets. It has many powerful features like filtering marking, formatting, adding notes, etc. It could also format SOAP messages in logs.

16. Kafka – A high-throughput distributed messaging system
Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by “logging” and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

17. Kibana – Web Interface for Logstash and ElasticSearch
Kibana is a highly scalable interface for Logstash and ElasticSearch that allows you to efficiently search, graph, analyze and otherwise make sense of a mountain of logs. Kibana will load balance against your Elasticsearch cluster. Logstash’s daily rolling indicies let you scale to huge datasets, while Kibana’s sequential querying gets you most relevant data quickly, with more as it becomes available.

18. Pylogdb

A Python-powered, column-oriented database suitable for web log analysis pylogdb is a database suitable for web log analysis.

19. Epylog – a Syslog parser
Epylog is a syslog parser which runs periodically, looks at your logs, processes some of the entries in order to present them in a more comprehensible format, and then mails you the output. It is written specifically for large network clusters where a lot of machines (around 50 and upwards) log to the same loghost using syslog or syslog-ng.

20. Indihiang – IIS and Apache log analyzing tool
Indihiang Project is a web log analyzing tool. This tool analyzes IIS and Apache Web logs and generates real time reports. It has Web Log Viewer and analyzer. It is capable to analyze the trend from the logs. This tool also integrate with windows Explorer so you can attach a log file in to indihiang tool via context menu.

Make your Eclipse Faster

Posted: February 11, 2013 by Narendra Shah in Java Box, Reserch And Development

Once we start working on Eclipse, After some time, it usually became slow. To make it fast, there is way to clean up history and indexes which eclipse has created. (if both are not important to you). This helped me to speed up eclipse, and decrease RAM and HDD usage on my system

Here are the 2 folders, which should be cleanup

Index info

Delete the files under both the folder. This is big impact on eclipse performance.

Note: Both folder are part of your Eclipse workspace and not of your Projet source or eclipse installation. You can view workspace location information under Windows->Preferences->(General->Startup and Shutdown->workspaces).

You can even follow below links, if you running slow machine

There some good part ref:

1.  Add following in eclipse.ini


2. Configure Xmx and Xms and permsize according to your system. I have configured mine with following in eclipse.ini


This are all working parameters.

About OS install on USB pen Drive

Posted: March 23, 2009 by Narendra Shah in Linux Box, Reserch And Development

After long time i am back with the Operating System on USB or pen drive. See i am not forcing you to install the linux on pen drive. But some times it is very helpful to you to have Any OS handy on your hand. Just put your usb in any machine and your likely OS is ready to use.

Today let me instroduce one software which works with both linux and windows to install iso to your pendrive. Really very nice tool and very user friendly also. I am looking for such tool from long time. One more thing this is open source.

UNetbootin allows you to create bootable Live USB drives for a variety of Linux distributions from Windows or Linux, without requiring you to burn a CD. You can either let it download one of the many distributions supported out-of-the-box for you, or supply your own Linux .iso file if you’ve already downloaded one or your preferred distribution isn’t on the list.

And one more thing i am currently studying debian OS named Backtrack 4, which includes tool for hacking. If you are interested get it from the net and work with again USB and enjoy Hacking and not cracking. and know your network activity.

–Narendra Shah.

Cache Memory

Posted: November 17, 2008 by Narendra Shah in Reserch And Development
Tags: ,

Every one which is from IT industry is some how knows about Cache Memory. Then also let me introduce about it with little bit different way as explained in HOWSTUFFWORKS, it is some how related librarirain meance non technical.

E.g In any library(not online), there is person with post Librarian(i.e CPU, Harddisk etc.).His job is to issue and take back books(data digital data). What he will do when ever any user comes, he goes to libarary store room take the book required by user and issue him. When some other user comes for submitting book, he will put that book back to library. Think about the time required by libararian to put and take book from Library.

As such he knows that every one is generally coming for requirement of very known book, why he will go to store and come back. Rather when any use submit book then he will put that in the Rack(CACHE) itself. If any one comes to take book he will return that book from the Rack itself. This is very popular solution which generally many libararian are following.

Now coming to our computer world, with coming of Dual core and Core 2 Duo processor, they do have cache which will store frequently used operation in the cache. and it will make that processor being faster than any other CPU which do have higher clock rate. and Performance hikes. Now intel is thinking only for cache to increase speed of his CPU. So he is now ahead of AMD. AMD do have processor which are much more faster in clock speed then also, Intel chip will give more performance because he have cache. And now intel wins.

When you think about writing cache for your application then you will get the performance difference of your own application.

Let me show you how? In many application we do have databases, why dont we cache the connection in connection pool. Why dont we design a cache architecture which will store frequently used database operation will be on database as well as on cache also. When you write in frequently used table then write it in cache and database both. So when ever any operation requested by your application from table it will return from Cache itself and not from table. So decrease I/O read from Harddisk.

Same is true for file operation also.

There are some thing before incorporating cache with your application like if you have good RAM then your application goes faster. But if it is not then it is not good to use it.